`US 6,909,443 B1
`(10) Patent No.:
`(12)
`Robertson etal.
`(45) Date of Patent:
`Jun. 21, 2005
`
`
`US006909443B1
`
`(75)
`
`.
`
`(54) METHOD AND APPARATUS FOR
`PROVIDING A THREE-DIMENSIONAL TASK
`GALLERY COMPUTER INTERFACE
`Inventors: cee Robertson, Seattle, WA
`(US); Mary P. zrwins®
`Woodinville,WA (US); Kenneth P.
`Hinckley, GethellWA(US stem
`.
`Risden,
`Bothell,
`;
`Danie
`C. Robbins, Seattle, WA (US);
`Maarten R. van Dantzich, Seattle, WA
`(US)
`(73) Assignee: Microsoft Corporation, Redmond, WA
`(US)
`F thi
`h
`disclai
`bi
`this
`Subjectto any disclaimer, the term of
`patent is extended or adjusted under 35
`US.C, 154(b) by 0 days.
`(21) Appl. No.: 09/540,069
`oo °
`(22)
`Filed:
`Mar.31, 2000
`
`.
`‘
`(*) Notice:
`
`(60)
`
`Related U.S. Application Data
`Provisional application No. 60/128,003, filed on Apr. 6,
`aangProvisional application No. 60/127,997, filed on
`pr.
`6,
`.
`(SL) Ute C0 eeececccseeseeseeseereereeseeseeneesees G09G 5/00
`(52) U.S. Ch. ee eeeeeeereeeeeeeeeeeeenenees 345/782; 345/848
`(58) Field of Search o...0...cc eee 345/757, 752,
`345/848, 852, 850, 836
`
`(56)
`
`References Cited
`U.S. PATENT DOCUMENTS
`
`5,544,295 A
`veesessseesseeesessees 395/152
`8/1996 Capps:
`5,644,737 A
`7/1997 Tuniman etal. ............ 395/352
`5,724,492 A
`3/1998 Matthews, III et al.
`..... 395/119
`5,754,809 A *
`5/1998 Gandre oo... eee 345/782
`5,808,613 A *
`9/1998 Marrin et al. ow. 345/427
`
`
`5,835,692 A
`11/1998 Cragun et al... 395/119
`5,838,326 A * 11/1998 Card etal. we. 345/775
`
`1/1999 Strasnick et al.
`........... 345/355
`5,861,885 A
`
`2/1999 LaHood .........eeeeeeeeeee 345/339
`5,874,956 A
`
`
`3/1999 Southgate .......
`ee 345/340
`5,880,725 A
`3/1999 Horvitz et al. we. 345/427
`5,880,733 A *
`
`wo
`WO
`
`6,002,403 A * 12/1999 Sugiyama etal. .......... 345/717
`... 345/848
`6,088,032 A *
`7/2000 Mackinlay sesseeee
`
`
`... 345/350
`6,115,043 A
`9/2000 Levine et al.
`..
`... 345/782
`6,229,542 Bl *
`5/2001 Miller
`...........
`6,313,855 B1 * 11/2001 Shupingetal.
`.. 345/738
`2/2002 Matsuda ..........
`_.. 345/836
`6,346,956 B2 *
`.. 345/776
`6,486,895 B1
`11/2002 Robertsonetal.
`
`.......... 345/782
`6,590,593 Bl
`7/2003 Robertsonetal.
`9/2003 Fair v..ceseecesseeseeseee 345/763
`6,628,307 BL
`FOREIGN PATENT DOCUMENTS
`WO 97/41506
`11/1997
`WO 97/45782
`12/1997
`OTHER PUBLICATIONS
`How to user Microsoft Windows NT 4 Workstation, Copy-
`right 1996.*
`Y'g_ Appl. No. 09/541,133, filed Mar. 31, 2000, Robertson
`et al
`US. Appl. No. 09/540,744,filed Mar. 31, 2000, Robertson
`PP
`et al.
`(Continued)
`:
`:
`a
`oo,
`Primary Examiner—Kristine Kincaid
`Assistant Examiner—Thanh T. Vu
`(74) Attorney, Agent, or Firm—Theodore M. Magee;
`Westman, Champlin & Kelly, P.A.
`57
`ABSTRACT
`67)
`The present invention provides a three-dimensional user
`interface for a computer system that allows a user to
`combineandstore a group of windowsas a task. The image
`of each task can be positioned within a three-dimensional
`environment such that the user may utilize spatial memory
`in order to remember where a particular task is located. In
`further embodiments of the invention, the three-dimensional
`environment includes a stage, which is used to display the
`task with the current focus. When a user selects a new task
`in the gallery, the task is movedto the stage and given focus.
`If a previous task was on the stage, an image of the previous
`task is captured. This image is then moved into the task
`gallery away from the stage. This process allows users to
`switch between multiple window configurations with a
`simple action.
`
`4 Claims, 40 Drawing Sheets
`
`404
`
`
`
`
`
`217
`
`402
`
`408
`
`APPLE 1016
`
`APPLE 1016
`
`1
`
`
`
`US 6,909,443 B1
`Page 2
`
`OTHER PUBLICATIONS
`
`“Moving Objects in Space: Exploiting Proprioception in
`Virtual—EnvironmentInteraction,” Computer Graphics Pro-
`ceedings, Annual Conference Series, XP—000765798, pp.
`19-26 (1997).
`“Wayfinding Strategies and Behaviors in Large Virtual
`Worlds,” Conference on Human Factors in Computing Sys-
`tems, pp. 142-149 (1996).
`“Practical 3D User Interface Design: Siggraph °96,” Orga-
`nizer: Daniel C. Robbins, Microsoft Corporation, 30 pages.
`Summary of Video Entitled “CHIMP System,” by Mark
`Mine, University of North Carolina, 1 page (1996).
`Benjamin B. Bedersonet al., “Local Tools: An Alternative to
`Tool Palettes,” User Interface Software and Technology,pp.
`169-170 (1996).
`Mark Billinghurst et al., “3D Palette: A Virtual Reality
`Content Creation Tool,” Virtual Reality Software and Tech-
`nology, pp. 155-156 (1997).
`Jeff Butterworth et al., “3DM: A Three Dimensional Mod-
`eler Using a Head—Mounted Display,” Symposium onInter-
`active 3D Graphics, pp. 135-138 (1992).
`Brookshire D. Conneret al., “Three—Dimensional Widgets,”
`Symposium on Interactive 3D Graphics, pp. 183-188
`(1992).
`T. Todd Elvins et al., “3D Thumbnails for Wayfinding in
`Virtual Environments,” User Interface Software and Tech-
`nology, pp. 21-30 (1997).
`Ken Hinckleyet al. “Passive Real-World Interface Props for
`Neurosurgical Visualization,” Conference on Human Fac-
`tors in Computing Systems, pp. 452-458 (1994).
`Randy Pauschetal., “Navigation and Locomotion in Virtual
`Worlds via Flight Into Hand—Held Miniatures,” ACM SIG-
`GRAPH Conference Proceedings, pp. 399-400 (1995).
`Abigail J. Sellen et al., “The Role of Visual and Kinesthetic
`Feedback in the Prevention of Mode Errors,” INTERACT
`°90, pp. 667-673 (1990).
`
`Richard Stoakley et al., “Virtual Reality on a WIM: Inter-
`active Worlds in Miniature,” Conference on Human Factors
`in Computing Systems, pp. 265-272 (1995).
`Colin Ware et al., “Fish Tank Virtual Reality,” Conference
`on Human Factors in Computing Systems, pp. 37-42
`(1993).
`Bukowski, R., et al., “Object Associations: A Simple and
`Practical Approach to Virtual 3D Manipulation,” Proceed-
`ings of Symposium on Interactive 3D Graphics, pp. 131-138
`(1995).
`Czerwinski, M., et al., “The Contirubtion of Thumbnail
`Image, Mouse—Over Text and Spatial Location Memoryto
`Web Page Retrieval in 3D,” Proceedingsof Interact *99,pp.
`163-170. (1999).
`Kandogan E., et al., “Elastic Windows: Evaluation of Mul-
`ti-Window Operations,” CHI’97, ACM, pp. 250-257
`(1997).
`Morris, J., et al, “A Distributed Personal Computing Envi-
`ronment,” CACM, 29(3), pp. 184-201 (Mar. 1986).
`Robertson, G., et al., “Data Mountain: Using Spatial
`Memory for Document Management,” UIST °98, ACM,pp.
`153-162 (Nov. 1998).
`Feiner, S., et al., “Windows on the World: 2D Windowsfor
`3D Augmented Reality,” Proceedings of ACM UIST °93
`Symposium on UserInterface Software & Technology, pp.
`145-155 (Nov. 1993).
`Henderson, A., et al., “The Use of Mulitple Virtual Work-
`spaces to Reduce Space Contention in a Window-—Based
`Graphical User Interface,” ACM Transactions on Graphics
`5, 3, pp. 211-243 (1986).
`Robertson, G., et al., “Information Visualization Using 3D
`Interactive Animation,” CACM,36, 4, pp. 57-71 (1993).
`
`* cited by examiner
`
`2
`
`
`
`U.S. Patent
`
`Jun. 21, 2005
`
`Sheet 1 of 40
`
`US 6,909,443 B1
`
`
`
`YHOMLAN
`
`o€SAIN
`
`
` SP|0s|1YOdWIYASYSayNVads
`
`
`|AMOWSWNgeSWywHo0ud||geViVd
`YaLAdWOO09YSAINGSTWVY|92S018
`
`
`SLOWSY»39IAa0vzWOU
`PEAINI|cigWolldO[-apTeSIC
`HOLINOWnew
`
`
`HaLdvav4
`
`
`
`7SWAGON|rwa00K€FINOHdONOINONILNIOdGYVOSAay
`
`€¢G€SLNI|ycigoaNOWN
`ualdvayWOILdO
`
`
`WVE9ONd|Noivorddy
`Z€FINGOWG€WALSAS\b
`
`
`ean|aaa{Et
`ZpADIARAOrLc)|dj
`9ZSAING||
`
`0gYALNdINOD
`
`JISVAOWSY
`
`FOVYOLS
`
`62
`
`3
`
`
`
`
`
`
`
`U.S. Patent
`
`Jun. 21, 2005
`
`Sheet 2 of 40
`
`US 6,909,443 B1
`
`210
`
`212
`
`_— 216
`
`206
`
`202
`
`214
`
`FIG. 2
`
`4
`
`
`
`U.S. Patent
`
`Jun. 21, 2005
`
`Sheet 3 of 40
`
`US 6,909,443 B1
`
`FIG.3
`
`200 \ors
`
`202
`
`N
`
`217
`
`5
`
`
`
`U.S. Patent
`
`Jun. 21, 2005
`
`Sheet 4 of 40
`
`
`
`304
`
`
`
`302
`
`US 6,909,443 B1
`
`FIG. 6
`
`6
`
`
`
`U.S. Patent
`
`Jun. 21, 2005
`
`Sheet 5 of 40
`
`US 6,909,443 B1
`
`
`
`GALLERY 250
`
`
`ROOM 251
`
`
`
`STAGE 254
`TASK 268
`LOOSE STACK 270
`[ WINDOW 276|[WINDOW 278
`
`
`
`
`
`
`
`
`
`
`
`ORDEREDSTACK 272
`WINDOW]
`
`
`
`
`
`
`
`
`LEFT SIDE eaersc]
`
`
`
`
`
`
`
`280
`PRIMARY VIEW 274
`
`FOCUS WINDOW
`
`[WINDOW|
`
`
`
`RIGHT SIDE WALL =[TASK|ENDWALL 258 | FLOOR 258 -
`
`|CIELING 260)__
`[TASK]
`
`ROOM 252
`
`[TASK262][TASK262|[TASK2264]||[TASK |[ TASK |
`
`LEFT SIDE WALL __270__|
`
`[RIGHT SIDE WALL _272
`
`SEARa CIELING_276|TASK|[TASKol[TASK|+fsTASK |
`
`
`FIG.5
`
`7
`
`
`
`U.S. Patent
`
`Jun.21, 2005
`
`Sheet 6 of 40
`
`US 6,909,443 B1
`
`212
`
`FORW.
`
`8
`
`
`
`U.S. Patent
`
`Jun. 21, 2005
`
`Sheet 7 of 40
`
`US 6,909,443 B1
`
`
`
`354
`
`FIG. 10A
`
`FIG. 10B |
`
`9
`
`
`
`U.S. Patent
`
`Jun. 21, 2005
`
`Sheet 8 of 40
`
`US 6,909,443 B1
`
`
`
`FIG. 11A
`
`FIG. 11B
`
`10
`
`
`
`U.S. Patent
`
`Jun. 21, 2005
`
`Sheet 9 of 40
`
`US 6,909,443 B1
`
`
`
`
`
` FIG. 12F
`
`11
`
`
`
`U.S. Patent
`
`Jun. 21, 2005
`
`Sheet 10 of 40
`
`US 6,909,443 B1
`
`
`
`FIG.121
`
`12
`
`
`
`U.S. Patent
`
`Jun. 21, 2005
`
`Sheet 11 of 40
`
`US 6,909,443 B1
`
`
`
`
`
`
`
`13
`
`
`
`U.S. Patent
`
`Jun. 21, 2005
`
`Sheet 12 of 40
`
`US 6,909,443 B1
`
`FIG. 14B
`
`
`
`
`
`
`
`427a
`FIG.14C
`
`427
`
`FIG. 14D
`
`427
`
`FIG. 14F
`
`14
`
`14
`
`
`
`U.S. Patent
`
`Jun. 21, 2005
`
`Sheet 13 of 40
`
`US 6,909,443 B1
`
` FIG. 15B
` —/
`
`216
`
`FIG. 15C
`
`FIG. 15D
`
`15
`
`
`
`U.S. Patent
`
`Jun. 21, 2005
`
`Sheet 14 of 40
`
`US 6,909,443 B1
`
` FIG. 16
`
`16
`
`16
`
`
`
`U.S. Patent
`
`Jun. 21, 2005
`
`Sheet 15 of 40
`
`US 6,909,443 B1
`
`
`
` FIG. 17
`
`17
`
`
`
`U.S. Patent
`
`Jun. 21, 2005
`
`Sheet 16 of 40
`
`US 6,909,443 B1
`
`450
`
`S442 ale 442
`
`FIG. 18A
`20
`Aa2
`
`FIG. 18B
`442
`
`FIG. 18C
`
`FIG. 18D
`
`18
`
`
`
`U.S. Patent
`
`Jun. 21, 2005
`
`Sheet 17 of 40
`
`US 6,909,443 B1
`
`470
`—S&
`
`458
`
`406
`
`454
`
`456
`
`FIG.19A
`
`———
`
`
`
`
`462
`
`ay a FIG. 20A
`470
`
`468
`
`464
`
`aes
`
`FIG. 20B
`
`454
`
`460
`
`470
`
`a
`
`464
`
`468
`
`4
`
`
`
`FIG. 19C
`
`FIG. 20C
`
`19
`
`
`
`U.S. Patent
`
`Jun. 21, 2005
`
`Sheet 18 of 40
`
`US 6,909,443 B1
`
`
`474 472
`
`480
`
`FIG. 22B
`
` FIG. 21C
`
`484
`a“
`
`480
`
`FIG. 22C
`
`20
`
`20
`
`
`
`U.S. Patent
`
`Jun. 21, 2005
`
`Sheet 19 of 40
`
`US 6,909,443 B1
`
`490
`
`490
`
`FIG. 23A
`
`
`
`t FIG.23B
`SSOe .
`FIG.23C
`
`490
`
`21
`
`21
`
`
`
`U.S. Patent
`
`Jun. 21, 2005
`
`Sheet 20 of 40
`
`US 6,909,443 B1
`
`502
`
`590
`
`510
`
`506 oF
`
`508
`
`——1.s04
`
`FIG. 24A
`
`FIG. 25A
`
`
`
`FIG. 24B
`
`— FIG. 25B
`
`510 °°
`
`sog
`
`500
`
`= S
`
`_
`
`504
`
`FIG. 24C
`
`FIG. 25C
`
`22
`
`22
`
`
`
`U.S. Patent
`
`Jun.21, 2005
`
`Sheet 21 of 40
`
`US 6,909,443 B1
`
`FIG. 26B
`
`%
`
`/ /
`
`
`
`2
`
`518
`
`516
`
`520
`
`514
`
`
`
`518 520
`
`516
`
`ST 518
`
`FIG.26C
`
` 7, FIG.26E
`
`522
`
`518
`
`Ss 522
`
`FIG. 26D
`
`520
`
`23
`
`23
`
`
`
`U.S. Patent
`
`Jun. 21, 2005
`
`Sheet 22 of 40
`
`US 6,909,443 B1
`
`554
`
`552
`
`550 FIG. 28B
`
`556
`
`550
`
`-
`
`oT 550
`
`550
`
`FIG. 28C
`
`FIG. 28D
`
`524
`
`Ss
`
`POPOPGE
`
`/
`
`528
`
`536
`
`?
`
`532
`
`53
`
`526
`
`534
`
`537
`
`FIG. 27
`
`24
`
`24
`
`
`
`U.S. Patent
`
`Jun. 21, 2005
`
`Sheet 23 of 40
`
`US 6,909,443 B1
`
`
`562
`
`FIG. 29C
`
`FIG. 29B
`
`568
`
`
`
`FIG. 30C
`
`25
`
`25
`
`
`
`U.S. Patent
`
`Jun. 21, 2005
`
`Sheet 24 of 40
`
`US 6,909,443 B1
`
`LA© FIG. 32B
`
`
`
`FIG. 32A
`
`
`
`FIG.31C
`
`26
`
`FIG. 32C
`
`600
`
`26
`
`
`
`U.S. Patent
`
`Jun. 21, 2005
`
`Sheet 25 of 40
`
`US 6,909,443 B1
`
`FIG. 33B
`
`/
`
`606
`
`FIG. 33C
`
`27
`
`27
`
`
`
`U.S. Patent
`
`Jun. 21, 2005
`
`Sheet 26 of 40
`
`US 6,909,443 B1
`
`619
`
`621
`
`
`
` FIG. 35C
`
`= “Fe
`
`621
`
`FIG. 35B
`
`28
`
`28
`
`
`
`U.S. Patent
`
`US 6,909,443 B1
`
`628
`
`Jun. 21, 2005
`
`Sheet 27 of 40
`
`624
`
`FIG. 36B
`
`
`
`FIG. 36C
`
`29
`
`29
`
`
`
`U.S. Patent
`
`Jun.21, 2005
`
`Sheet 28 of 40
`
`US 6,909,443 B1
`
`
`
`
`@ @(
`
`@®
`
`x 6
`
`34
`
`FIG. 37D
`
`7
`
`630
`
`>
`®IOKO
`
`©
`
`32
`
`°
`
`634
`
`S
`
`as
`
`636
`
`FIG. 37A
`
` FIG.37C
`
`636
`
`636
`
`
`
`FIG.37E
`
`636
`
`634
`
`30
`
`30
`
`
`
`U.S. Patent
`
`Jun. 21, 2005
`
`Sheet 29 of 40
`
`US 6,909,443 B1
`
`>
`
`_—654
`
`654
`
`FIG. 38A
`
`FIG. 38B
`
`
`
`FIG. 38E
`
`FIG. 38F
`
`31
`
`31
`
`
`
`U.S. Patent
`
`Jun. 21, 2005
`
`Sheet 30 of 40
`
`US 6,909,443 B1
`
`
`660 FIG. 381
`
`FIG. 38]
`
`660
`
`656
`
`32
`
`32
`
`
`
`U.S. Patent
`
`Jun. 21, 2005
`
`Sheet 31 of 40
`
`US 6,909,443 B1
`
`
`
`404
`
`684
`
`682
`
`=
`
`680
`
`fF
`
`FIG. 39B
`
`FIG. 40B
`
`680
`
`684
`
`670
`
`688
`
`
`
`eee.
`
`FIG.39C
`
`)
`
`684
`
`692
`
`FIG. 40C
`
`33
`
`33
`
`
`
`U.S. Patent
`
`Jun. 21, 2005
`
`Sheet 32 of 40
`
`US 6,909,443 B1
`
`\
`
`216
`
`FIG. 441A
`
` FIG. 42B
`
`708
`
`FIG. 41C
`
`FIG. 42C
`
`34
`
`34
`
`
`
`U.S. Patent
`
`Jun.21, 2005
`
`Sheet 33 of 40
`
`US 6,909,443 B1
`
`
`
`FIG. 43A
`
`FIG. 44A
`
`
`
`
`FIG. 44B
`
`714
`
`FIG. 44C
`
`TOOLSPACE _
`- 712
`
`FIG. 43B
`
`Ss
`UP
`TOOLSPACE *
`
`FIG. 43C
`
`m8
`
`35
`
`35
`
`
`
`U.S. Patent
`
`Jun. 21, 2005
`
`Sheet 34 of 40
`
`US 6,909,443 B1
`
`
`
`
`
`——730
`
`730
`
`FIG. 45C
`
`FIG. 45D
`
`
`
`FIG. 45E
`
`36
`
`36
`
`
`
`U.S. Patent
`
`Jun. 21, 2005
`
`Sheet 35 of 40
`
`US 6,909,443 B1
`
`J |
`
`
`
`736
`
`—
`
`[8
`
`SSA
`— US
`
`FIG. 46B
`
`738
`
`— —_
`c
`A
`
`7
`
`736
`
`—
`
`736
`
`a
`LlSO
`
`SN
`
`FIG. 46C
`
`738
`
`FIG. 46D
`
`— 7
`| et
`
`738
`736
`
`FIG. 46E
`
`37
`
`37
`
`
`
`U.S. Patent
`
`Jun. 21, 2005
`
`Sheet 36 of 40
`
`US 6,909,443 B1
`
`
`
`FIG. 47A
`
`742
`
`49
`
`744
`
`FIG. 47B
`
`740
`
`38
`
`38
`
`
`
`U.S. Patent
`
`Jun. 21, 2005
`
`Sheet 37 of 40
`
`US 6,909,443 B1
`
`POINTING DEVICE }S~~ 774
`
`
`POINTING DEVICE
`
`
`USER.DLL
`
`776
`
`
`GDI.DLL
`756
`
`
`
`768
`
`
`DISPLAY
`
`DRIVER
`
`770 “2
`
`
`DISPLAY
`
`FIG. 48
`
`39
`
`
`
` DRAWING
`
`
`
`REDIRECT
`
`MEMORY
`
`CONTEXT
`760
`
`758
`
`
`3-D SHELL
`
`
`TEXTURE
`
`
`
`MEMORY
` 3-D
`
`
`762
`
`RENDERING
`
`BACK | FRONT
`
`764
`
`BUFF.|BUFF.
`
`
`DISPLAY
`MEMORY
`
`
`766
`767
`
`39
`
`
`
`U.S. Patent
`
`Jun. 21, 2005
`
`Sheet 38 of 40
`
`US 6,909,443 B1
`
`APPLICATION OR OS DETERMIENS
`WINDOW SHOULD BE REPAINTED
`
`800
`
`[ APPLICATIONREPAINTS THE WINDOW} $802
`
`|
`
`OS REDIRECTS PAINT DATA
`
`LS804
`
`OS NOTIFIES 3-D SHELL OF REPAINT AND
`
`GIVES POINTER TO REDIRECTED PAINT
`
`
`806
`
`
`
`DATA
`
`
`
`
`
`|
`
`3-D SHELL MARKS TEXTURE MAPAS
`_"DIRTY"
`
`~808
`
`
` -810
`
`
`STORE NEW POLYGONDEFINITIONS FOR
`
`WINDOWS IF NEEDED
`
`
`FIG. 49
`
`40
`
`40
`
`
`
`U.S. Patent
`
`Jun. 21, 2005
`
`Sheet 39 of 40
`
`US 6,909,443 B1
`
`
`
`
`
`DETERMINE LOCATION AND
`ORIENTATION OF VIRTUAL USER
`
`RENDER TASK GALLERY AND NON-
`__FOCUS TASKS
`
`<— 801
`
`— 803
`S
`
`— 805
`IDENTIFY WINDOWSIN THE FOCUS TASK|S
`THAT ARE VISIBLE
`
`
`
`
`
`
`
`
`
`
`810
`
`a T
`
`EXTURE MAP FOR
`VISIBLE WINDOW
`"DIRTY"?
`
`~
`
`611
`
`COPY REDIRECTED PAINT DATA INTO
`THE WINDOW'S TEXTURE MAP
`
`
`
`
`
`g12 —S
`
`RENDER
`WINDOWS
`
`
`
`
`FIG. 50
`
`41
`
`41
`
`
`
`U.S. Patent
`
`Jun. 21, 2005
`
`Sheet 40 of 40
`
`US 6,909,443 B1
`
`
`POINTING DEVICE DRIVER CREATES
`
`INPUT EVENT MESSAGE
`
`OS DETERMINES SCREEN COORDINATE
`FOR CURSOR DURING INPUT EVENT
`
`
`SCREEN COORDINATE OF CURSORIS
`PASSEDTO3-D SHELL
`
`
`
`
`
`FIG. 5 l
`
`820
`
`822
`
`824
`
`
`
`
`832
`
`
`
`
`
`NO| NO CHANGES
`
`
`IS CURSOR OVER WINDOW IN
`TO EVENT
`FOCUS TASK?
`MESSAGE
`
`
`
`
`DETERMINE2-D POSITION OF CURSOR
`WITHIN WINDOW
`
`
`838
`
`
`SET 2-D
`VIEWING AREA?
`
`NO|COORDINATES
`S WINDOW IN PREFERRED
`AND WINDOW
`
`
`HANDLE; SET
`AS CURSOR
`
`
`OVER
`
`
`IS CURSOR IN CLIENT AREA?
`MESSAGE
`
`
`844?
`
`USE EVENT
`
`
`SET 2-D COORDINATES OF CURSOR AND
`MESSAGEAS
` WINDOW HANDLE IN EVENT MESSAGE
`
`
`
`
`INPUT FOR
`
`
`
`CHANGETO 3-
`
`DENV.
`834
`
`42
`
`42
`
`
`
`US 6,909,443 B1
`
`1
`METHOD AND APPARATUS FOR
`PROVIDING A THREE-DIMENSIONAL TASK
`GALLERY COMPUTER INTERFACE
`
`REFERENCE TO RELATED APPLICATIONS
`
`The present application claims priority from U.S. Provi-
`sional Applications having Ser. Nos. 60/128,003 and 60/127,
`997, both filed on Apr. 6, 1999 and entitled METHOD AND
`APPARATUS FOR PROVIDING A THREE-
`DIMENSIONAL TASK GALLERY COMPUTER INTER-
`FACE and METHOD AND APPARATUSFOR PROVID-
`ING AND ACCESSING HIDDEN TOOL SPACES,
`respectively.
`The present applicationis also related to three US. patent
`application owned by a commonassignee with the present
`application and filed on even date herewith. The three
`applications are identified by Ser. Nos. 09/541,133, 09/539,
`817, and Ser. No. 09/540,744 and are entitled “METHOD
`AND APPARATUS FOR PROVIDING AND ACCESSING
`HIDDEN TOOLSPACES”, “METHOD AND APPARATUS
`FOR HANDLING DISMISSED DIALOGUE BOXES”,
`and “METHOD AND APPARATUS FOR SUPPORTING
`TWO-DIMENSIONAL WINDOWS IN A THREE-
`
`DIMENSIONAL ENVIRONMENT”, respectively.
`BACKGROUND OF THE INVENTION
`
`The present invention relates to computer interfaces. In
`particular, the present invention relates to three-dimensional
`computer interfaces.
`Many computer systems display images produced by
`different applications within different windows on a com-
`puter monitor. Examples of operating systems that generate
`such windowsinclude Windows 95®, Windows 98®, Win-
`dows NT®, and Windows® 2000 from Microsoft Corpora-
`tion. In such systems,usersare able to interact with multiple
`applications. For example, a user may have one window
`open for Word 97 from Microsoft Corporation and a second
`window open for Excel from Microsoft Corporation.
`It has been observed that computer users open different
`windowsfor different tasks and organize the display of their
`windowsdifferently for different tasks. For example, when
`a user performsthe task of writing a computer program, they
`may have two windowsopen in a split screen format, with
`one window containing a program editor and the other
`window containing the interface generated by the program.
`However, whenthe user is performing a task such as sending
`e-mails, the user may have the mail window opensothatit
`takes up most of the screen and a scheduling application
`open in a small part of the screen.
`Since each task may be associated with different windows
`in different layouts, one system of the prior art has allowed
`the user to associate the layout of particular windows with
`particular tasks. In this prior art system, such layouts were
`referred to as rooms, even though the layout provided a
`two-dimensional view of the windows as seen on most
`
`current two-dimensional displays. The user could select one
`of the rooms to work with by picking the room from a grid
`of icons representing each of the available rooms. In this
`prior art system, the rooms are placed in the grid by the
`system. This forces the user to scan the grid in orderto find
`the room that they wish to work with. Such a layout makes
`the use of the room system difficult for most users.
`SUMMARYOF THE INVENTION
`
`The present invention provides a three-dimensional user
`interface for a computer system that allows a user to
`
`2
`combine and store a group of windowsas a task. The image
`of each task can be positioned within a three-dimensional
`environment such that the user mayutilize spatial memory
`in order remember wherea particular task is located.
`In further embodiments of the invention,
`the three-
`dimensional environmentincludes a stage, which is used to
`display the task with the current focus. When a user selects
`a new task in the gallery, the task is moved to the stage and
`given focus. If a previous task was onthe stage, an image of
`the previous task is captured. This image is then movedinto
`the task gallery away from the stage. In many embodiments,
`the image of the previous task is sent to the location in the
`gallery where the previous task resided before it was
`selected to move to the stage.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`FIG. 1 is a plan view of a general computing environment
`in which embodiments of the invention may bepracticed.
`FIG. 2 is a top-back perspective view of a task gallery
`display of an embodimentof the present invention.
`FIG. 3 is a side perspective view of the task gallery of
`FIG. 2.
`
`FIG. 4 is a screen imageof a task gallery user interface
`generated under an embodimentof the present invention.
`FIG. 5 is a diagram of a container object hierarchy under
`an embodiment of the invention.
`
`FIG. 6 is a screen image of the task gallery of FIG. 4
`populated by tasks and windows.
`FIG. 7 is a diagram showing the relationship between
`mouse movement and object movement for objects associ-
`ated with different parts of the task gallery.
`FIGS. 8A-8B show selected frames from the animated
`movementof a task on the right side wall ofthe task gallery.
`FIGS. 9A-9B show selected frames from the animated
`
`movementof a task on the left side wall of the task gallery.
`FIGS. 10A—-10B show selected frames from the animated
`movementof a task on the floor of the task gallery.
`FIGS. 11A-11B show selected frames from the animated
`
`movementof a task on the ceiling of the task gallery.
`FIGS. 12A-12I1 showselected frames from the animated
`movementof a task as it is moved betweenthe walls, ceiling
`and floor of the task gallery.
`FIGS. 13A-13E show selected frames from the animated
`movementof tasks when focus is shifted to a new task.
`FIGS. 14A-14F show selected frames from the animated
`movementof the virtual user and tasks when focusis shifted
`to a new task using a menuselection.
`FIGS. 15A-15D show selected frames from the animated
`
`movement of a virtual user to the home viewing area.
`FIG. 16 shows a movementcontrol in a task gallery of one
`embodiment of the invention.
`
`FIG. 17 showsa focus task from the perspective of the
`home viewing area.
`FIGS. 18A-18D show selected frames from the animated
`
`movement of a window from the primary viewing location
`to the loose stack.
`FIGS. 19A-19C show selected frames from the animated
`movement of a window from the ordered stack to the loose
`stack.
`FIGS. 20A-20C showselected frames from the animated
`movement of a window from the ordered stack to the
`primary viewing location in place of an existing window in
`the primary viewing location.
`
`10
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`60
`
`65
`
`43
`
`43
`
`
`
`US 6,909,443 B1
`
`3
`FIGS. 21A—21C show selected frames from the animated
`movement of a window from the loose stack to the ordered
`stack.
`FIGS. 22A—22C show selected frames from the animated
`movement of a window from the loose stack to the primary
`viewing location in place of an existing window in the
`primary viewing location.
`FIGS. 23A—23C show selected frames from the animated
`
`movement of a window from the primary viewing location
`to the ordered stack.
`
`10
`
`FIGS. 24A-24C show selected frames from the dragging
`of a window within the loose stack.
`FIGS. 25A—25C show selected frames from the animated
`movement of a window within the loose stack.
`
`FIGS. 26A-26F show selected frames from the dragging
`and animated movement of a window within the ordered
`stack.
`
`FIG. 27 shows a set of iconic buttons for controlling
`movement of windows in an embodiment of the present
`invention.
`FIGS. 28A-28D show selected frames from the animated
`
`movement of a window from the primary viewing location
`to the loose stack using button icons.
`FIGS. 29A—29C show selected frames from the animated
`movement of a window from the ordered stack to the loose
`
`stack using button icons.
`FIGS. 30A-30C show selected frames from the animated
`movement of a window from the ordered stack to the
`
`primary viewing location in place of an existing window in
`the primary viewing location using button icons.
`FIGS. 31A-31C show selected frames from the animated
`
`movement of a window from the loose stack to the primary
`viewing location in place of an existing window in the
`primary viewing location using button icons.
`FIGS. 32A-—32C show selected frames from the animated
`movement of a window from the loose stack to the ordered
`stack using button icons.
`FIGS. 33A—33C show selected frames from the animated
`movement of a window from the primary viewing location
`to the ordered stack using button icons.
`FIGS. 34A-34C show selected frames from the animated
`movement of a window within the loose stack using button
`icons.
`FIGS. 35A-35C show selected frames from the animated
`movement of a window within the loose stack using a
`second embodimentof button icons.
`
`FIGS. 36A-36C show selected frames from the dragging
`of a window within the loose stack using button icons.
`FIGS. 37A-37F show selected frames from the dragging
`and animated movement of a window within the ordered
`stack using button icons.
`FIGS. 38A-38]J show selected frames from the animated
`
`movement associated with adding windowsto the primary
`viewing location.
`FIGS. 39A-39C show selected frames from an animated
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`movement associated with glancing at the left tool space.
`FIGS. 40A—40C show selected frames from an animated
`
`60
`
`movement associated with returning to a forward view after
`selecting an application from the left tool space.
`FIGS. 41A-41C show selected frames from an animated
`
`movement associated with glancing at the right tool space.
`FIGS. 42A-42C showselected frames from an animated
`movement associated with returning to a forward view after
`selecting an application from the right tool space.
`
`65
`
`4
`FIGS. 43A-43C show selected frames from an animated
`movement associated with glancing at the up tool space.
`FIGS. 44A4-44C show selected frames from an animated
`
`movement associated with glancing at the downtool space.
`FIGS. 45A-45E show selected frames from an animated
`movementof a dismissed dialog box as it moves toward the
`down tool space.
`FIGS. 46A-46E show selected frames from an animated
`movement of a window from onetask to another.
`FIGS. 47A-47B show selected frames from an animated
`
`movement of a window boundary during resizing.
`FIG. 48 is a block diagram of software and hardware
`elements of one embodiment of the present invention.
`FIG. 49 is a flow diagram for redirecting window display
`data generated by an application.
`FIG. 50 is a flow diagram of an animation loop for
`rendering redirected window data.
`FIG. 51 is a flow diagram for redirecting pointing device
`input messages.
`DETAILED DESCRIPTION OF ILLUSTRATIVE
`EMBODIMENTS
`
`FIG. 1 and the related discussion are intended to provide
`a brief, general description of a suitable computing envi-
`ronment
`in which the invention may be implemented.
`Although not required, the invention will be described, at
`least in part, in the general context of computer-executable
`instructions, such as program modules, being executed by a
`personal computer. Generally, program modules include
`routine programs, objects, components, data structures,etc.
`that perform particular tasks or implementparticularabstract
`data types. Moreover, those skilled in the art will appreciate
`that the invention may be practiced with other computer
`system configurations, including hand-held devices, multi-
`processor systems, microprocessor-based or programmable
`consumerelectronics, network PCs, minicomputers, main-
`frame computers, and the like. The invention may also be
`practiced in distributed computing environments where
`tasks are performed by remote processing devices that are
`linked through a communications network.In a distributed
`computing environment, program modules maybe located
`in both local and remote memorystorage devices.
`With reference to FIG. 1, an exemplary system for imple-
`menting the invention includes a general purpose computing
`device in the form of a conventional personal computer 20,
`including a processing unit (CPU) 21, a system memory 22,
`and a system bus 23 that couples various system components
`including the system memory 22 to the processing unit 21.
`The system bus 23 may be any of several types of bus
`structures including a memory bus or memory controller, a
`peripheral bus, and a local bus using any of a variety of bus
`architectures. The system memory 22 includes read only
`memory (ROM) 24 and random access memory (RAM)25.
`A basic input/output (BIOS) 26, containing the basic routine
`that helps to transfer information between elements within
`the personal computer 20, such as during start-up, is stored
`in ROM 24. The personal computer 20 further includes a
`hard disk drive 27 for reading from and writing to a hard disk
`(not shown), a magnetic disk drive 28 for reading from or
`writing to removable magnetic disk 29, and an optical disk
`drive 30 for reading from or writing to a removable optical
`disk 31 such as a CD ROM orother optical media. The hard
`disk drive 27, magnetic disk drive 28, and optical disk drive
`30 are connected to the system bus 23 by a hard disk drive
`interface 32, magnetic disk drive interface 33, and an optical
`
`44
`
`44
`
`
`
`US 6,909,443 B1
`
`5
`drive interface 34, respectively. The drives and the associ-
`ated computer-readable media provide nonvolatile storage
`of computer readable instructions, data structures, program
`modules and other data for the personal computer 20.
`Although the exemplary environment described herein
`employs the hard disk, the removable magnetic disk 29 and
`the removable optical disk 31, it should be appreciated by
`those skilled in the art that other types of computer readable
`media which can store data that is accessible by a computer,
`such as magnetic cassettes, flash memory cards, digital
`video disks, Bernoulli cartridges, random access memories
`(RAMs), read only memory (ROM), and the like, may also
`be used in the exemplary operating environment.
`Anumberof program modules may be stored on the hard
`disk, magnetic disk 29, optical disk 31, ROM 24 or RAM 25,
`including an operating system 35, one or more application
`programs 36, other program modules 37, and program data
`38. A user may enter commands and information into the
`personal computer 20 through local input devices such as a
`keyboard 40, pointing device 42 and a microphone 43. Other
`input devices (not shown) mayinclude a joystick, game pad,
`satellite dish, scanner, or the like. These and other input
`devices are often connected to the processing unit 21
`through a serial port interface 46 that is coupled to the
`system bus 23, but may be connected by other interfaces,
`such as a sound card, a parallel port, a game port or a
`universal serial bus (USB). A monitor 47 or other type of
`display device is also connected to the system bus 23 via an
`interface, such as a video adapter 4B. In addition to the
`monitor 47, personal computers may typically include other
`peripheral output devices, such as a speaker 45 and printers
`(not shown).
`The personal computer 20 may operate in a networked
`environment using logic connections to one or more remote
`computers, such as a remote computer 49. The remote
`computer 49 may be another personal computer, a hand-held
`device, a server, a router, a network PC, a peer device or
`other network node, and typically includes manyorall of the
`elements described above relative to the personal computer
`20, although only a memory storage device 50 has been
`illustrated in FIG. 1. The logic connections depicted in FIG.
`1 include a local area network (LAN) 51 and a wide area
`network (WAN) 52. Such networking environments are
`commonplacein offices, enterprise-wide computer network
`Intranets, and the Internet.
`When used in a LAN networking environment, the per-
`sonal computer 20 is connected to the local area network 51
`through a network interface or adapter 53. When used in a
`WAN networking environment, the personal computer 20
`typically includes a modem 54 or other meansfor establish-
`ing communications over the wide area network 52, such as
`the Internet. The modem 54, which may be internal or
`external, is connected to the system bus 23 viathe serial port
`interface 46. In a network environment, program modules
`depicted relative to the personal computer 20, or portions
`thereof, may be stored in the remote memory storage
`devices. It will be appreciated that the network connections
`shown are exemplary and other means of establishing a
`communications link between the computers may be used.
`For example, a wireless communication link may be estab-
`lished between one or more portions of the network.
`Under the present
`invention, a three-dimensional user
`interface is generated that allows a user to manipulate and
`use windows by associating the windows with separate
`tasks. In the description below,this three-dimensional user
`interface is referred to alternatively as a task gallery, a data
`
`10
`
`15
`
`20
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`6
`hallway, and a data mine. Generally, the three-dimensional
`user interface gives the user the perception that they are
`within a hallwayorgallery consisting of a numberof aligned
`hallway sections that end with a stage or display area at an
`end wall.
`
`Task Gallery Layout
`FIG. 2 provides a top back perspective view of a task
`gallery 200 of one embodimentof the present invention with
`the ceiling in the gallery. removed to expose the remainder
`of the gallery. Task gallery 200 includes rooms 202, 204, 206
`and 208 that each have walls forming a portion of side walls
`210 and 212, and floors that form a portion of gallery floor
`214. Room 208 also includes an end wall 216 and a stage
`217.
`
`FIG. 3 provides a perspective view from the side of task
`gallery 200 of FIG.2. In FIG. 3, ceiling 202 of task gallery
`200 is shown connecting side walls 210, and 212. Although
`only four rooms, 202, 204, 206 and 208 are shown in FIGS.
`2 and 3, many task galleries of the present invention are
`indefinitely extendable by the user. In one embodiment, the
`user interface automatically generates additional rooms as
`the user moves objects out of the last existing room or
`creates new objects that necessitate the creation of new
`rooms. In such embodiments,
`the interface also removes
`back-end roomsif they no longer contain objects. Thus, the
`task gallery may consist of as few as one room.
`Whenthe user is using task gallery 200 of FIG. 3, the
`three-dimensional image provided to the user is based upon
`the combination of the location of a virtual body, represent-
`ing the user’s bodyin the task gallery and the orientation of
`a virtual head or camera representing the user’s head in the
`task gallery. The user’s virtual head is able to rotate inde-
`pendently of the direction the virtual body is facing, so that
`the user can glance up and down and to the sides as
`discussed further below.
`
`FIG. 4 provides a screen image representing the view
`