throbber
United States Patent
`US 6,909,443 B1
`(10) Patent No.:
`(12)
`Robertson etal.
`(45) Date of Patent:
`Jun. 21, 2005
`
`
`US006909443B1
`
`(75)
`
`.
`
`(54) METHOD AND APPARATUS FOR
`PROVIDING A THREE-DIMENSIONAL TASK
`GALLERY COMPUTER INTERFACE
`Inventors: cee Robertson, Seattle, WA
`(US); Mary P. zrwins®
`Woodinville,WA (US); Kenneth P.
`Hinckley, GethellWA(US stem
`.
`Risden,
`Bothell,
`;
`Danie
`C. Robbins, Seattle, WA (US);
`Maarten R. van Dantzich, Seattle, WA
`(US)
`(73) Assignee: Microsoft Corporation, Redmond, WA
`(US)
`F thi
`h
`disclai
`bi
`this
`Subjectto any disclaimer, the term of
`patent is extended or adjusted under 35
`US.C, 154(b) by 0 days.
`(21) Appl. No.: 09/540,069
`oo °
`(22)
`Filed:
`Mar.31, 2000
`
`.
`‘
`(*) Notice:
`
`(60)
`
`Related U.S. Application Data
`Provisional application No. 60/128,003, filed on Apr. 6,
`aangProvisional application No. 60/127,997, filed on
`pr.
`6,
`.
`(SL) Ute C0 eeececccseeseeseeseereereeseeseeneesees G09G 5/00
`(52) U.S. Ch. ee eeeeeeereeeeeeeeeeeeenenees 345/782; 345/848
`(58) Field of Search o...0...cc eee 345/757, 752,
`345/848, 852, 850, 836
`
`(56)
`
`References Cited
`U.S. PATENT DOCUMENTS
`
`5,544,295 A
`veesessseesseeesessees 395/152
`8/1996 Capps:
`5,644,737 A
`7/1997 Tuniman etal. ............ 395/352
`5,724,492 A
`3/1998 Matthews, III et al.
`..... 395/119
`5,754,809 A *
`5/1998 Gandre oo... eee 345/782
`5,808,613 A *
`9/1998 Marrin et al. ow. 345/427
`
`
`5,835,692 A
`11/1998 Cragun et al... 395/119
`5,838,326 A * 11/1998 Card etal. we. 345/775
`
`1/1999 Strasnick et al.
`........... 345/355
`5,861,885 A
`
`2/1999 LaHood .........eeeeeeeeeee 345/339
`5,874,956 A
`
`
`3/1999 Southgate .......
`ee 345/340
`5,880,725 A
`3/1999 Horvitz et al. we. 345/427
`5,880,733 A *
`
`wo
`WO
`
`6,002,403 A * 12/1999 Sugiyama etal. .......... 345/717
`... 345/848
`6,088,032 A *
`7/2000 Mackinlay sesseeee
`
`
`... 345/350
`6,115,043 A
`9/2000 Levine et al.
`..
`... 345/782
`6,229,542 Bl *
`5/2001 Miller
`...........
`6,313,855 B1 * 11/2001 Shupingetal.
`.. 345/738
`2/2002 Matsuda ..........
`_.. 345/836
`6,346,956 B2 *
`.. 345/776
`6,486,895 B1
`11/2002 Robertsonetal.
`
`.......... 345/782
`6,590,593 Bl
`7/2003 Robertsonetal.
`9/2003 Fair v..ceseecesseeseeseee 345/763
`6,628,307 BL
`FOREIGN PATENT DOCUMENTS
`WO 97/41506
`11/1997
`WO 97/45782
`12/1997
`OTHER PUBLICATIONS
`How to user Microsoft Windows NT 4 Workstation, Copy-
`right 1996.*
`Y'g_ Appl. No. 09/541,133, filed Mar. 31, 2000, Robertson
`et al
`US. Appl. No. 09/540,744,filed Mar. 31, 2000, Robertson
`PP
`et al.
`(Continued)
`:
`:
`a
`oo,
`Primary Examiner—Kristine Kincaid
`Assistant Examiner—Thanh T. Vu
`(74) Attorney, Agent, or Firm—Theodore M. Magee;
`Westman, Champlin & Kelly, P.A.
`57
`ABSTRACT
`67)
`The present invention provides a three-dimensional user
`interface for a computer system that allows a user to
`combineandstore a group of windowsas a task. The image
`of each task can be positioned within a three-dimensional
`environment such that the user may utilize spatial memory
`in order to remember where a particular task is located. In
`further embodiments of the invention, the three-dimensional
`environment includes a stage, which is used to display the
`task with the current focus. When a user selects a new task
`in the gallery, the task is movedto the stage and given focus.
`If a previous task was on the stage, an image of the previous
`task is captured. This image is then moved into the task
`gallery away from the stage. This process allows users to
`switch between multiple window configurations with a
`simple action.
`
`4 Claims, 40 Drawing Sheets
`
`404
`
`
`
`
`
`217
`
`402
`
`408
`
`APPLE 1016
`
`APPLE 1016
`
`1
`
`

`

`US 6,909,443 B1
`Page 2
`
`OTHER PUBLICATIONS
`
`“Moving Objects in Space: Exploiting Proprioception in
`Virtual—EnvironmentInteraction,” Computer Graphics Pro-
`ceedings, Annual Conference Series, XP—000765798, pp.
`19-26 (1997).
`“Wayfinding Strategies and Behaviors in Large Virtual
`Worlds,” Conference on Human Factors in Computing Sys-
`tems, pp. 142-149 (1996).
`“Practical 3D User Interface Design: Siggraph °96,” Orga-
`nizer: Daniel C. Robbins, Microsoft Corporation, 30 pages.
`Summary of Video Entitled “CHIMP System,” by Mark
`Mine, University of North Carolina, 1 page (1996).
`Benjamin B. Bedersonet al., “Local Tools: An Alternative to
`Tool Palettes,” User Interface Software and Technology,pp.
`169-170 (1996).
`Mark Billinghurst et al., “3D Palette: A Virtual Reality
`Content Creation Tool,” Virtual Reality Software and Tech-
`nology, pp. 155-156 (1997).
`Jeff Butterworth et al., “3DM: A Three Dimensional Mod-
`eler Using a Head—Mounted Display,” Symposium onInter-
`active 3D Graphics, pp. 135-138 (1992).
`Brookshire D. Conneret al., “Three—Dimensional Widgets,”
`Symposium on Interactive 3D Graphics, pp. 183-188
`(1992).
`T. Todd Elvins et al., “3D Thumbnails for Wayfinding in
`Virtual Environments,” User Interface Software and Tech-
`nology, pp. 21-30 (1997).
`Ken Hinckleyet al. “Passive Real-World Interface Props for
`Neurosurgical Visualization,” Conference on Human Fac-
`tors in Computing Systems, pp. 452-458 (1994).
`Randy Pauschetal., “Navigation and Locomotion in Virtual
`Worlds via Flight Into Hand—Held Miniatures,” ACM SIG-
`GRAPH Conference Proceedings, pp. 399-400 (1995).
`Abigail J. Sellen et al., “The Role of Visual and Kinesthetic
`Feedback in the Prevention of Mode Errors,” INTERACT
`°90, pp. 667-673 (1990).
`
`Richard Stoakley et al., “Virtual Reality on a WIM: Inter-
`active Worlds in Miniature,” Conference on Human Factors
`in Computing Systems, pp. 265-272 (1995).
`Colin Ware et al., “Fish Tank Virtual Reality,” Conference
`on Human Factors in Computing Systems, pp. 37-42
`(1993).
`Bukowski, R., et al., “Object Associations: A Simple and
`Practical Approach to Virtual 3D Manipulation,” Proceed-
`ings of Symposium on Interactive 3D Graphics, pp. 131-138
`(1995).
`Czerwinski, M., et al., “The Contirubtion of Thumbnail
`Image, Mouse—Over Text and Spatial Location Memoryto
`Web Page Retrieval in 3D,” Proceedingsof Interact *99,pp.
`163-170. (1999).
`Kandogan E., et al., “Elastic Windows: Evaluation of Mul-
`ti-Window Operations,” CHI’97, ACM, pp. 250-257
`(1997).
`Morris, J., et al, “A Distributed Personal Computing Envi-
`ronment,” CACM, 29(3), pp. 184-201 (Mar. 1986).
`Robertson, G., et al., “Data Mountain: Using Spatial
`Memory for Document Management,” UIST °98, ACM,pp.
`153-162 (Nov. 1998).
`Feiner, S., et al., “Windows on the World: 2D Windowsfor
`3D Augmented Reality,” Proceedings of ACM UIST °93
`Symposium on UserInterface Software & Technology, pp.
`145-155 (Nov. 1993).
`Henderson, A., et al., “The Use of Mulitple Virtual Work-
`spaces to Reduce Space Contention in a Window-—Based
`Graphical User Interface,” ACM Transactions on Graphics
`5, 3, pp. 211-243 (1986).
`Robertson, G., et al., “Information Visualization Using 3D
`Interactive Animation,” CACM,36, 4, pp. 57-71 (1993).
`
`* cited by examiner
`
`2
`
`

`

`U.S. Patent
`
`Jun. 21, 2005
`
`Sheet 1 of 40
`
`US 6,909,443 B1
`
`
`
`YHOMLAN
`
`o€SAIN
`
`
` SP|0s|1YOdWIYASYSayNVads
`
`
`|AMOWSWNgeSWywHo0ud||geViVd
`YaLAdWOO09YSAINGSTWVY|92S018
`
`
`SLOWSY»39IAa0vzWOU
`PEAINI|cigWolldO[-apTeSIC
`HOLINOWnew
`
`
`HaLdvav4
`
`
`
`7SWAGON|rwa00K€FINOHdONOINONILNIOdGYVOSAay
`
`€¢G€SLNI|ycigoaNOWN
`ualdvayWOILdO
`
`
`WVE9ONd|Noivorddy
`Z€FINGOWG€WALSAS\b
`
`
`ean|aaa{Et
`ZpADIARAOrLc)|dj
`9ZSAING||
`
`0gYALNdINOD
`
`JISVAOWSY
`
`FOVYOLS
`
`62
`
`3
`
`
`
`
`
`

`

`U.S. Patent
`
`Jun. 21, 2005
`
`Sheet 2 of 40
`
`US 6,909,443 B1
`
`210
`
`212
`
`_— 216
`
`206
`
`202
`
`214
`
`FIG. 2
`
`4
`
`

`

`U.S. Patent
`
`Jun. 21, 2005
`
`Sheet 3 of 40
`
`US 6,909,443 B1
`
`FIG.3
`
`200 \ors
`
`202
`
`N
`
`217
`
`5
`
`

`

`U.S. Patent
`
`Jun. 21, 2005
`
`Sheet 4 of 40
`
`
`
`304
`
`
`
`302
`
`US 6,909,443 B1
`
`FIG. 6
`
`6
`
`

`

`U.S. Patent
`
`Jun. 21, 2005
`
`Sheet 5 of 40
`
`US 6,909,443 B1
`
`
`
`GALLERY 250
`
`
`ROOM 251
`
`
`
`STAGE 254
`TASK 268
`LOOSE STACK 270
`[ WINDOW 276|[WINDOW 278
`
`
`
`
`
`
`
`
`
`
`
`ORDEREDSTACK 272
`WINDOW]
`
`
`
`
`
`
`
`
`LEFT SIDE eaersc]
`
`
`
`
`
`
`
`280
`PRIMARY VIEW 274
`
`FOCUS WINDOW
`
`[WINDOW|
`
`
`
`RIGHT SIDE WALL =[TASK|ENDWALL 258 | FLOOR 258 -
`
`|CIELING 260)__
`[TASK]
`
`ROOM 252
`
`[TASK262][TASK262|[TASK2264]||[TASK |[ TASK |
`
`LEFT SIDE WALL __270__|
`
`[RIGHT SIDE WALL _272
`
`SEARa CIELING_276|TASK|[TASKol[TASK|+fsTASK |
`
`
`FIG.5
`
`7
`
`

`

`U.S. Patent
`
`Jun.21, 2005
`
`Sheet 6 of 40
`
`US 6,909,443 B1
`
`212
`
`FORW.
`
`8
`
`

`

`U.S. Patent
`
`Jun. 21, 2005
`
`Sheet 7 of 40
`
`US 6,909,443 B1
`
`
`
`354
`
`FIG. 10A
`
`FIG. 10B |
`
`9
`
`

`

`U.S. Patent
`
`Jun. 21, 2005
`
`Sheet 8 of 40
`
`US 6,909,443 B1
`
`
`
`FIG. 11A
`
`FIG. 11B
`
`10
`
`

`

`U.S. Patent
`
`Jun. 21, 2005
`
`Sheet 9 of 40
`
`US 6,909,443 B1
`
`
`
`
`
` FIG. 12F
`
`11
`
`

`

`U.S. Patent
`
`Jun. 21, 2005
`
`Sheet 10 of 40
`
`US 6,909,443 B1
`
`
`
`FIG.121
`
`12
`
`

`

`U.S. Patent
`
`Jun. 21, 2005
`
`Sheet 11 of 40
`
`US 6,909,443 B1
`
`
`
`
`
`
`
`13
`
`

`

`U.S. Patent
`
`Jun. 21, 2005
`
`Sheet 12 of 40
`
`US 6,909,443 B1
`
`FIG. 14B
`
`
`
`
`
`
`
`427a
`FIG.14C
`
`427
`
`FIG. 14D
`
`427
`
`FIG. 14F
`
`14
`
`14
`
`

`

`U.S. Patent
`
`Jun. 21, 2005
`
`Sheet 13 of 40
`
`US 6,909,443 B1
`
` FIG. 15B
` —/
`
`216
`
`FIG. 15C
`
`FIG. 15D
`
`15
`
`

`

`U.S. Patent
`
`Jun. 21, 2005
`
`Sheet 14 of 40
`
`US 6,909,443 B1
`
` FIG. 16
`
`16
`
`16
`
`

`

`U.S. Patent
`
`Jun. 21, 2005
`
`Sheet 15 of 40
`
`US 6,909,443 B1
`
`
`
` FIG. 17
`
`17
`
`

`

`U.S. Patent
`
`Jun. 21, 2005
`
`Sheet 16 of 40
`
`US 6,909,443 B1
`
`450
`
`S442 ale 442
`
`FIG. 18A
`20
`Aa2
`
`FIG. 18B
`442
`
`FIG. 18C
`
`FIG. 18D
`
`18
`
`

`

`U.S. Patent
`
`Jun. 21, 2005
`
`Sheet 17 of 40
`
`US 6,909,443 B1
`
`470
`—S&
`
`458
`
`406
`
`454
`
`456
`
`FIG.19A
`
`———
`
`
`
`
`462
`
`ay a FIG. 20A
`470
`
`468
`
`464
`
`aes
`
`FIG. 20B
`
`454
`
`460
`
`470
`
`a
`
`464
`
`468
`
`4
`
`
`
`FIG. 19C
`
`FIG. 20C
`
`19
`
`

`

`U.S. Patent
`
`Jun. 21, 2005
`
`Sheet 18 of 40
`
`US 6,909,443 B1
`
`
`474 472
`
`480
`
`FIG. 22B
`
` FIG. 21C
`
`484
`a“
`
`480
`
`FIG. 22C
`
`20
`
`20
`
`

`

`U.S. Patent
`
`Jun. 21, 2005
`
`Sheet 19 of 40
`
`US 6,909,443 B1
`
`490
`
`490
`
`FIG. 23A
`
`
`
`t FIG.23B
`SSOe .
`FIG.23C
`
`490
`
`21
`
`21
`
`

`

`U.S. Patent
`
`Jun. 21, 2005
`
`Sheet 20 of 40
`
`US 6,909,443 B1
`
`502
`
`590
`
`510
`
`506 oF
`
`508
`
`——1.s04
`
`FIG. 24A
`
`FIG. 25A
`
`
`
`FIG. 24B
`
`— FIG. 25B
`
`510 °°
`
`sog
`
`500
`
`= S
`
`_
`
`504
`
`FIG. 24C
`
`FIG. 25C
`
`22
`
`22
`
`

`

`U.S. Patent
`
`Jun.21, 2005
`
`Sheet 21 of 40
`
`US 6,909,443 B1
`
`FIG. 26B
`
`%
`
`/ /
`
`
`
`2
`
`518
`
`516
`
`520
`
`514
`
`
`
`518 520
`
`516
`
`ST 518
`
`FIG.26C
`
` 7, FIG.26E
`
`522
`
`518
`
`Ss 522
`
`FIG. 26D
`
`520
`
`23
`
`23
`
`

`

`U.S. Patent
`
`Jun. 21, 2005
`
`Sheet 22 of 40
`
`US 6,909,443 B1
`
`554
`
`552
`
`550 FIG. 28B
`
`556
`
`550
`
`-
`
`oT 550
`
`550
`
`FIG. 28C
`
`FIG. 28D
`
`524
`
`Ss
`
`POPOPGE
`
`/
`
`528
`
`536
`
`?
`
`532
`
`53
`
`526
`
`534
`
`537
`
`FIG. 27
`
`24
`
`24
`
`

`

`U.S. Patent
`
`Jun. 21, 2005
`
`Sheet 23 of 40
`
`US 6,909,443 B1
`
`
`562
`
`FIG. 29C
`
`FIG. 29B
`
`568
`
`
`
`FIG. 30C
`
`25
`
`25
`
`

`

`U.S. Patent
`
`Jun. 21, 2005
`
`Sheet 24 of 40
`
`US 6,909,443 B1
`
`LA© FIG. 32B
`
`
`
`FIG. 32A
`
`
`
`FIG.31C
`
`26
`
`FIG. 32C
`
`600
`
`26
`
`

`

`U.S. Patent
`
`Jun. 21, 2005
`
`Sheet 25 of 40
`
`US 6,909,443 B1
`
`FIG. 33B
`
`/
`
`606
`
`FIG. 33C
`
`27
`
`27
`
`

`

`U.S. Patent
`
`Jun. 21, 2005
`
`Sheet 26 of 40
`
`US 6,909,443 B1
`
`619
`
`621
`
`
`
` FIG. 35C
`
`= “Fe
`
`621
`
`FIG. 35B
`
`28
`
`28
`
`

`

`U.S. Patent
`
`US 6,909,443 B1
`
`628
`
`Jun. 21, 2005
`
`Sheet 27 of 40
`
`624
`
`FIG. 36B
`
`
`
`FIG. 36C
`
`29
`
`29
`
`

`

`U.S. Patent
`
`Jun.21, 2005
`
`Sheet 28 of 40
`
`US 6,909,443 B1
`
`
`
`
`@ @(
`
`@®
`
`x 6
`
`34
`
`FIG. 37D
`
`7
`
`630
`
`>
`®IOKO
`

`
`32
`

`
`634
`
`S
`
`as
`
`636
`
`FIG. 37A
`
` FIG.37C
`
`636
`
`636
`
`
`
`FIG.37E
`
`636
`
`634
`
`30
`
`30
`
`

`

`U.S. Patent
`
`Jun. 21, 2005
`
`Sheet 29 of 40
`
`US 6,909,443 B1
`
`>
`
`_—654
`
`654
`
`FIG. 38A
`
`FIG. 38B
`
`
`
`FIG. 38E
`
`FIG. 38F
`
`31
`
`31
`
`

`

`U.S. Patent
`
`Jun. 21, 2005
`
`Sheet 30 of 40
`
`US 6,909,443 B1
`
`
`660 FIG. 381
`
`FIG. 38]
`
`660
`
`656
`
`32
`
`32
`
`

`

`U.S. Patent
`
`Jun. 21, 2005
`
`Sheet 31 of 40
`
`US 6,909,443 B1
`
`
`
`404
`
`684
`
`682
`
`=
`
`680
`
`fF
`
`FIG. 39B
`
`FIG. 40B
`
`680
`
`684
`
`670
`
`688
`
`
`
`eee.
`
`FIG.39C
`
`)
`
`684
`
`692
`
`FIG. 40C
`
`33
`
`33
`
`

`

`U.S. Patent
`
`Jun. 21, 2005
`
`Sheet 32 of 40
`
`US 6,909,443 B1
`
`\
`
`216
`
`FIG. 441A
`
` FIG. 42B
`
`708
`
`FIG. 41C
`
`FIG. 42C
`
`34
`
`34
`
`

`

`U.S. Patent
`
`Jun.21, 2005
`
`Sheet 33 of 40
`
`US 6,909,443 B1
`
`
`
`FIG. 43A
`
`FIG. 44A
`
`
`
`
`FIG. 44B
`
`714
`
`FIG. 44C
`
`TOOLSPACE _
`- 712
`
`FIG. 43B
`
`Ss
`UP
`TOOLSPACE *
`
`FIG. 43C
`
`m8
`
`35
`
`35
`
`

`

`U.S. Patent
`
`Jun. 21, 2005
`
`Sheet 34 of 40
`
`US 6,909,443 B1
`
`
`
`
`
`——730
`
`730
`
`FIG. 45C
`
`FIG. 45D
`
`
`
`FIG. 45E
`
`36
`
`36
`
`

`

`U.S. Patent
`
`Jun. 21, 2005
`
`Sheet 35 of 40
`
`US 6,909,443 B1
`
`J |
`
`
`
`736
`
`—
`
`[8
`
`SSA
`— US
`
`FIG. 46B
`
`738
`
`— —_
`c
`A
`
`7
`
`736
`
`—
`
`736
`
`a
`LlSO
`
`SN
`
`FIG. 46C
`
`738
`
`FIG. 46D
`
`— 7
`| et
`
`738
`736
`
`FIG. 46E
`
`37
`
`37
`
`

`

`U.S. Patent
`
`Jun. 21, 2005
`
`Sheet 36 of 40
`
`US 6,909,443 B1
`
`
`
`FIG. 47A
`
`742
`
`49
`
`744
`
`FIG. 47B
`
`740
`
`38
`
`38
`
`

`

`U.S. Patent
`
`Jun. 21, 2005
`
`Sheet 37 of 40
`
`US 6,909,443 B1
`
`POINTING DEVICE }S~~ 774
`
`
`POINTING DEVICE
`
`
`USER.DLL
`
`776
`
`
`GDI.DLL
`756
`
`
`
`768
`
`
`DISPLAY
`
`DRIVER
`
`770 “2
`
`
`DISPLAY
`
`FIG. 48
`
`39
`
`
`
` DRAWING
`
`
`
`REDIRECT
`
`MEMORY
`
`CONTEXT
`760
`
`758
`
`
`3-D SHELL
`
`
`TEXTURE
`
`
`
`MEMORY
` 3-D
`
`
`762
`
`RENDERING
`
`BACK | FRONT
`
`764
`
`BUFF.|BUFF.
`
`
`DISPLAY
`MEMORY
`
`
`766
`767
`
`39
`
`

`

`U.S. Patent
`
`Jun. 21, 2005
`
`Sheet 38 of 40
`
`US 6,909,443 B1
`
`APPLICATION OR OS DETERMIENS
`WINDOW SHOULD BE REPAINTED
`
`800
`
`[ APPLICATIONREPAINTS THE WINDOW} $802
`
`|
`
`OS REDIRECTS PAINT DATA
`
`LS804
`
`OS NOTIFIES 3-D SHELL OF REPAINT AND
`
`GIVES POINTER TO REDIRECTED PAINT
`
`
`806
`
`
`
`DATA
`
`
`
`
`
`|
`
`3-D SHELL MARKS TEXTURE MAPAS
`_"DIRTY"
`
`~808
`
`
` -810
`
`
`STORE NEW POLYGONDEFINITIONS FOR
`
`WINDOWS IF NEEDED
`
`
`FIG. 49
`
`40
`
`40
`
`

`

`U.S. Patent
`
`Jun. 21, 2005
`
`Sheet 39 of 40
`
`US 6,909,443 B1
`
`
`
`
`
`DETERMINE LOCATION AND
`ORIENTATION OF VIRTUAL USER
`
`RENDER TASK GALLERY AND NON-
`__FOCUS TASKS
`
`<— 801
`
`— 803
`S
`
`— 805
`IDENTIFY WINDOWSIN THE FOCUS TASK|S
`THAT ARE VISIBLE
`
`
`
`
`
`
`
`
`
`
`810
`
`a T
`
`EXTURE MAP FOR
`VISIBLE WINDOW
`"DIRTY"?
`
`~
`
`611
`
`COPY REDIRECTED PAINT DATA INTO
`THE WINDOW'S TEXTURE MAP
`
`
`
`
`
`g12 —S
`
`RENDER
`WINDOWS
`
`
`
`
`FIG. 50
`
`41
`
`41
`
`

`

`U.S. Patent
`
`Jun. 21, 2005
`
`Sheet 40 of 40
`
`US 6,909,443 B1
`
`
`POINTING DEVICE DRIVER CREATES
`
`INPUT EVENT MESSAGE
`
`OS DETERMINES SCREEN COORDINATE
`FOR CURSOR DURING INPUT EVENT
`
`
`SCREEN COORDINATE OF CURSORIS
`PASSEDTO3-D SHELL
`
`
`
`
`
`FIG. 5 l
`
`820
`
`822
`
`824
`
`
`
`
`832
`
`
`
`
`
`NO| NO CHANGES
`
`
`IS CURSOR OVER WINDOW IN
`TO EVENT
`FOCUS TASK?
`MESSAGE
`
`
`
`
`DETERMINE2-D POSITION OF CURSOR
`WITHIN WINDOW
`
`
`838
`
`
`SET 2-D
`VIEWING AREA?
`
`NO|COORDINATES
`S WINDOW IN PREFERRED
`AND WINDOW
`
`
`HANDLE; SET
`AS CURSOR
`
`
`OVER
`
`
`IS CURSOR IN CLIENT AREA?
`MESSAGE
`
`
`844?
`
`USE EVENT
`
`
`SET 2-D COORDINATES OF CURSOR AND
`MESSAGEAS
` WINDOW HANDLE IN EVENT MESSAGE
`
`
`
`
`INPUT FOR
`
`
`
`CHANGETO 3-
`
`DENV.
`834
`
`42
`
`42
`
`

`

`US 6,909,443 B1
`
`1
`METHOD AND APPARATUS FOR
`PROVIDING A THREE-DIMENSIONAL TASK
`GALLERY COMPUTER INTERFACE
`
`REFERENCE TO RELATED APPLICATIONS
`
`The present application claims priority from U.S. Provi-
`sional Applications having Ser. Nos. 60/128,003 and 60/127,
`997, both filed on Apr. 6, 1999 and entitled METHOD AND
`APPARATUS FOR PROVIDING A THREE-
`DIMENSIONAL TASK GALLERY COMPUTER INTER-
`FACE and METHOD AND APPARATUSFOR PROVID-
`ING AND ACCESSING HIDDEN TOOL SPACES,
`respectively.
`The present applicationis also related to three US. patent
`application owned by a commonassignee with the present
`application and filed on even date herewith. The three
`applications are identified by Ser. Nos. 09/541,133, 09/539,
`817, and Ser. No. 09/540,744 and are entitled “METHOD
`AND APPARATUS FOR PROVIDING AND ACCESSING
`HIDDEN TOOLSPACES”, “METHOD AND APPARATUS
`FOR HANDLING DISMISSED DIALOGUE BOXES”,
`and “METHOD AND APPARATUS FOR SUPPORTING
`TWO-DIMENSIONAL WINDOWS IN A THREE-
`
`DIMENSIONAL ENVIRONMENT”, respectively.
`BACKGROUND OF THE INVENTION
`
`The present invention relates to computer interfaces. In
`particular, the present invention relates to three-dimensional
`computer interfaces.
`Many computer systems display images produced by
`different applications within different windows on a com-
`puter monitor. Examples of operating systems that generate
`such windowsinclude Windows 95®, Windows 98®, Win-
`dows NT®, and Windows® 2000 from Microsoft Corpora-
`tion. In such systems,usersare able to interact with multiple
`applications. For example, a user may have one window
`open for Word 97 from Microsoft Corporation and a second
`window open for Excel from Microsoft Corporation.
`It has been observed that computer users open different
`windowsfor different tasks and organize the display of their
`windowsdifferently for different tasks. For example, when
`a user performsthe task of writing a computer program, they
`may have two windowsopen in a split screen format, with
`one window containing a program editor and the other
`window containing the interface generated by the program.
`However, whenthe user is performing a task such as sending
`e-mails, the user may have the mail window opensothatit
`takes up most of the screen and a scheduling application
`open in a small part of the screen.
`Since each task may be associated with different windows
`in different layouts, one system of the prior art has allowed
`the user to associate the layout of particular windows with
`particular tasks. In this prior art system, such layouts were
`referred to as rooms, even though the layout provided a
`two-dimensional view of the windows as seen on most
`
`current two-dimensional displays. The user could select one
`of the rooms to work with by picking the room from a grid
`of icons representing each of the available rooms. In this
`prior art system, the rooms are placed in the grid by the
`system. This forces the user to scan the grid in orderto find
`the room that they wish to work with. Such a layout makes
`the use of the room system difficult for most users.
`SUMMARYOF THE INVENTION
`
`The present invention provides a three-dimensional user
`interface for a computer system that allows a user to
`
`2
`combine and store a group of windowsas a task. The image
`of each task can be positioned within a three-dimensional
`environment such that the user mayutilize spatial memory
`in order remember wherea particular task is located.
`In further embodiments of the invention,
`the three-
`dimensional environmentincludes a stage, which is used to
`display the task with the current focus. When a user selects
`a new task in the gallery, the task is moved to the stage and
`given focus. If a previous task was onthe stage, an image of
`the previous task is captured. This image is then movedinto
`the task gallery away from the stage. In many embodiments,
`the image of the previous task is sent to the location in the
`gallery where the previous task resided before it was
`selected to move to the stage.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`FIG. 1 is a plan view of a general computing environment
`in which embodiments of the invention may bepracticed.
`FIG. 2 is a top-back perspective view of a task gallery
`display of an embodimentof the present invention.
`FIG. 3 is a side perspective view of the task gallery of
`FIG. 2.
`
`FIG. 4 is a screen imageof a task gallery user interface
`generated under an embodimentof the present invention.
`FIG. 5 is a diagram of a container object hierarchy under
`an embodiment of the invention.
`
`FIG. 6 is a screen image of the task gallery of FIG. 4
`populated by tasks and windows.
`FIG. 7 is a diagram showing the relationship between
`mouse movement and object movement for objects associ-
`ated with different parts of the task gallery.
`FIGS. 8A-8B show selected frames from the animated
`movementof a task on the right side wall ofthe task gallery.
`FIGS. 9A-9B show selected frames from the animated
`
`movementof a task on the left side wall of the task gallery.
`FIGS. 10A—-10B show selected frames from the animated
`movementof a task on the floor of the task gallery.
`FIGS. 11A-11B show selected frames from the animated
`
`movementof a task on the ceiling of the task gallery.
`FIGS. 12A-12I1 showselected frames from the animated
`movementof a task as it is moved betweenthe walls, ceiling
`and floor of the task gallery.
`FIGS. 13A-13E show selected frames from the animated
`movementof tasks when focus is shifted to a new task.
`FIGS. 14A-14F show selected frames from the animated
`movementof the virtual user and tasks when focusis shifted
`to a new task using a menuselection.
`FIGS. 15A-15D show selected frames from the animated
`
`movement of a virtual user to the home viewing area.
`FIG. 16 shows a movementcontrol in a task gallery of one
`embodiment of the invention.
`
`FIG. 17 showsa focus task from the perspective of the
`home viewing area.
`FIGS. 18A-18D show selected frames from the animated
`
`movement of a window from the primary viewing location
`to the loose stack.
`FIGS. 19A-19C show selected frames from the animated
`movement of a window from the ordered stack to the loose
`stack.
`FIGS. 20A-20C showselected frames from the animated
`movement of a window from the ordered stack to the
`primary viewing location in place of an existing window in
`the primary viewing location.
`
`10
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`60
`
`65
`
`43
`
`43
`
`

`

`US 6,909,443 B1
`
`3
`FIGS. 21A—21C show selected frames from the animated
`movement of a window from the loose stack to the ordered
`stack.
`FIGS. 22A—22C show selected frames from the animated
`movement of a window from the loose stack to the primary
`viewing location in place of an existing window in the
`primary viewing location.
`FIGS. 23A—23C show selected frames from the animated
`
`movement of a window from the primary viewing location
`to the ordered stack.
`
`10
`
`FIGS. 24A-24C show selected frames from the dragging
`of a window within the loose stack.
`FIGS. 25A—25C show selected frames from the animated
`movement of a window within the loose stack.
`
`FIGS. 26A-26F show selected frames from the dragging
`and animated movement of a window within the ordered
`stack.
`
`FIG. 27 shows a set of iconic buttons for controlling
`movement of windows in an embodiment of the present
`invention.
`FIGS. 28A-28D show selected frames from the animated
`
`movement of a window from the primary viewing location
`to the loose stack using button icons.
`FIGS. 29A—29C show selected frames from the animated
`movement of a window from the ordered stack to the loose
`
`stack using button icons.
`FIGS. 30A-30C show selected frames from the animated
`movement of a window from the ordered stack to the
`
`primary viewing location in place of an existing window in
`the primary viewing location using button icons.
`FIGS. 31A-31C show selected frames from the animated
`
`movement of a window from the loose stack to the primary
`viewing location in place of an existing window in the
`primary viewing location using button icons.
`FIGS. 32A-—32C show selected frames from the animated
`movement of a window from the loose stack to the ordered
`stack using button icons.
`FIGS. 33A—33C show selected frames from the animated
`movement of a window from the primary viewing location
`to the ordered stack using button icons.
`FIGS. 34A-34C show selected frames from the animated
`movement of a window within the loose stack using button
`icons.
`FIGS. 35A-35C show selected frames from the animated
`movement of a window within the loose stack using a
`second embodimentof button icons.
`
`FIGS. 36A-36C show selected frames from the dragging
`of a window within the loose stack using button icons.
`FIGS. 37A-37F show selected frames from the dragging
`and animated movement of a window within the ordered
`stack using button icons.
`FIGS. 38A-38]J show selected frames from the animated
`
`movement associated with adding windowsto the primary
`viewing location.
`FIGS. 39A-39C show selected frames from an animated
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`movement associated with glancing at the left tool space.
`FIGS. 40A—40C show selected frames from an animated
`
`60
`
`movement associated with returning to a forward view after
`selecting an application from the left tool space.
`FIGS. 41A-41C show selected frames from an animated
`
`movement associated with glancing at the right tool space.
`FIGS. 42A-42C showselected frames from an animated
`movement associated with returning to a forward view after
`selecting an application from the right tool space.
`
`65
`
`4
`FIGS. 43A-43C show selected frames from an animated
`movement associated with glancing at the up tool space.
`FIGS. 44A4-44C show selected frames from an animated
`
`movement associated with glancing at the downtool space.
`FIGS. 45A-45E show selected frames from an animated
`movementof a dismissed dialog box as it moves toward the
`down tool space.
`FIGS. 46A-46E show selected frames from an animated
`movement of a window from onetask to another.
`FIGS. 47A-47B show selected frames from an animated
`
`movement of a window boundary during resizing.
`FIG. 48 is a block diagram of software and hardware
`elements of one embodiment of the present invention.
`FIG. 49 is a flow diagram for redirecting window display
`data generated by an application.
`FIG. 50 is a flow diagram of an animation loop for
`rendering redirected window data.
`FIG. 51 is a flow diagram for redirecting pointing device
`input messages.
`DETAILED DESCRIPTION OF ILLUSTRATIVE
`EMBODIMENTS
`
`FIG. 1 and the related discussion are intended to provide
`a brief, general description of a suitable computing envi-
`ronment
`in which the invention may be implemented.
`Although not required, the invention will be described, at
`least in part, in the general context of computer-executable
`instructions, such as program modules, being executed by a
`personal computer. Generally, program modules include
`routine programs, objects, components, data structures,etc.
`that perform particular tasks or implementparticularabstract
`data types. Moreover, those skilled in the art will appreciate
`that the invention may be practiced with other computer
`system configurations, including hand-held devices, multi-
`processor systems, microprocessor-based or programmable
`consumerelectronics, network PCs, minicomputers, main-
`frame computers, and the like. The invention may also be
`practiced in distributed computing environments where
`tasks are performed by remote processing devices that are
`linked through a communications network.In a distributed
`computing environment, program modules maybe located
`in both local and remote memorystorage devices.
`With reference to FIG. 1, an exemplary system for imple-
`menting the invention includes a general purpose computing
`device in the form of a conventional personal computer 20,
`including a processing unit (CPU) 21, a system memory 22,
`and a system bus 23 that couples various system components
`including the system memory 22 to the processing unit 21.
`The system bus 23 may be any of several types of bus
`structures including a memory bus or memory controller, a
`peripheral bus, and a local bus using any of a variety of bus
`architectures. The system memory 22 includes read only
`memory (ROM) 24 and random access memory (RAM)25.
`A basic input/output (BIOS) 26, containing the basic routine
`that helps to transfer information between elements within
`the personal computer 20, such as during start-up, is stored
`in ROM 24. The personal computer 20 further includes a
`hard disk drive 27 for reading from and writing to a hard disk
`(not shown), a magnetic disk drive 28 for reading from or
`writing to removable magnetic disk 29, and an optical disk
`drive 30 for reading from or writing to a removable optical
`disk 31 such as a CD ROM orother optical media. The hard
`disk drive 27, magnetic disk drive 28, and optical disk drive
`30 are connected to the system bus 23 by a hard disk drive
`interface 32, magnetic disk drive interface 33, and an optical
`
`44
`
`44
`
`

`

`US 6,909,443 B1
`
`5
`drive interface 34, respectively. The drives and the associ-
`ated computer-readable media provide nonvolatile storage
`of computer readable instructions, data structures, program
`modules and other data for the personal computer 20.
`Although the exemplary environment described herein
`employs the hard disk, the removable magnetic disk 29 and
`the removable optical disk 31, it should be appreciated by
`those skilled in the art that other types of computer readable
`media which can store data that is accessible by a computer,
`such as magnetic cassettes, flash memory cards, digital
`video disks, Bernoulli cartridges, random access memories
`(RAMs), read only memory (ROM), and the like, may also
`be used in the exemplary operating environment.
`Anumberof program modules may be stored on the hard
`disk, magnetic disk 29, optical disk 31, ROM 24 or RAM 25,
`including an operating system 35, one or more application
`programs 36, other program modules 37, and program data
`38. A user may enter commands and information into the
`personal computer 20 through local input devices such as a
`keyboard 40, pointing device 42 and a microphone 43. Other
`input devices (not shown) mayinclude a joystick, game pad,
`satellite dish, scanner, or the like. These and other input
`devices are often connected to the processing unit 21
`through a serial port interface 46 that is coupled to the
`system bus 23, but may be connected by other interfaces,
`such as a sound card, a parallel port, a game port or a
`universal serial bus (USB). A monitor 47 or other type of
`display device is also connected to the system bus 23 via an
`interface, such as a video adapter 4B. In addition to the
`monitor 47, personal computers may typically include other
`peripheral output devices, such as a speaker 45 and printers
`(not shown).
`The personal computer 20 may operate in a networked
`environment using logic connections to one or more remote
`computers, such as a remote computer 49. The remote
`computer 49 may be another personal computer, a hand-held
`device, a server, a router, a network PC, a peer device or
`other network node, and typically includes manyorall of the
`elements described above relative to the personal computer
`20, although only a memory storage device 50 has been
`illustrated in FIG. 1. The logic connections depicted in FIG.
`1 include a local area network (LAN) 51 and a wide area
`network (WAN) 52. Such networking environments are
`commonplacein offices, enterprise-wide computer network
`Intranets, and the Internet.
`When used in a LAN networking environment, the per-
`sonal computer 20 is connected to the local area network 51
`through a network interface or adapter 53. When used in a
`WAN networking environment, the personal computer 20
`typically includes a modem 54 or other meansfor establish-
`ing communications over the wide area network 52, such as
`the Internet. The modem 54, which may be internal or
`external, is connected to the system bus 23 viathe serial port
`interface 46. In a network environment, program modules
`depicted relative to the personal computer 20, or portions
`thereof, may be stored in the remote memory storage
`devices. It will be appreciated that the network connections
`shown are exemplary and other means of establishing a
`communications link between the computers may be used.
`For example, a wireless communication link may be estab-
`lished between one or more portions of the network.
`Under the present
`invention, a three-dimensional user
`interface is generated that allows a user to manipulate and
`use windows by associating the windows with separate
`tasks. In the description below,this three-dimensional user
`interface is referred to alternatively as a task gallery, a data
`
`10
`
`15
`
`20
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`6
`hallway, and a data mine. Generally, the three-dimensional
`user interface gives the user the perception that they are
`within a hallwayorgallery consisting of a numberof aligned
`hallway sections that end with a stage or display area at an
`end wall.
`
`Task Gallery Layout
`FIG. 2 provides a top back perspective view of a task
`gallery 200 of one embodimentof the present invention with
`the ceiling in the gallery. removed to expose the remainder
`of the gallery. Task gallery 200 includes rooms 202, 204, 206
`and 208 that each have walls forming a portion of side walls
`210 and 212, and floors that form a portion of gallery floor
`214. Room 208 also includes an end wall 216 and a stage
`217.
`
`FIG. 3 provides a perspective view from the side of task
`gallery 200 of FIG.2. In FIG. 3, ceiling 202 of task gallery
`200 is shown connecting side walls 210, and 212. Although
`only four rooms, 202, 204, 206 and 208 are shown in FIGS.
`2 and 3, many task galleries of the present invention are
`indefinitely extendable by the user. In one embodiment, the
`user interface automatically generates additional rooms as
`the user moves objects out of the last existing room or
`creates new objects that necessitate the creation of new
`rooms. In such embodiments,
`the interface also removes
`back-end roomsif they no longer contain objects. Thus, the
`task gallery may consist of as few as one room.
`Whenthe user is using task gallery 200 of FIG. 3, the
`three-dimensional image provided to the user is based upon
`the combination of the location of a virtual body, represent-
`ing the user’s bodyin the task gallery and the orientation of
`a virtual head or camera representing the user’s head in the
`task gallery. The user’s virtual head is able to rotate inde-
`pendently of the direction the virtual body is facing, so that
`the user can glance up and down and to the sides as
`discussed further below.
`
`FIG. 4 provides a screen image representing the view
`

This document is available on Docket Alarm but you must sign up to view it.


Or .

Accessing this document will incur an additional charge of $.

After purchase, you can access this document again without charge.

Accept $ Charge
throbber

Still Working On It

This document is taking longer than usual to download. This can happen if we need to contact the court directly to obtain the document and their servers are running slowly.

Give it another minute or two to complete, and then try the refresh button.

throbber

A few More Minutes ... Still Working

It can take up to 5 minutes for us to download a document if the court servers are running slowly.

Thank you for your continued patience.

This document could not be displayed.

We could not find this document within its docket. Please go back to the docket page and check the link. If that does not work, go back to the docket and refresh it to pull the newest information.

Your account does not support viewing this document.

You need a Paid Account to view this document. Click here to change your account type.

Your account does not support viewing this document.

Set your membership status to view this document.

With a Docket Alarm membership, you'll get a whole lot more, including:

  • Up-to-date information for this case.
  • Email alerts whenever there is an update.
  • Full text search for other cases.
  • Get email alerts whenever a new case matches your search.

Become a Member

One Moment Please

The filing “” is large (MB) and is being downloaded.

Please refresh this page in a few minutes to see if the filing has been downloaded. The filing will also be emailed to you when the download completes.

Your document is on its way!

If you do not receive the document in five minutes, contact support at support@docketalarm.com.

Sealed Document

We are unable to display this document, it may be under a court ordered seal.

If you have proper credentials to access the file, you may proceed directly to the court's system using your government issued username and password.


Access Government Site

We are redirecting you
to a mobile optimized page.





Document Unreadable or Corrupt

Refresh this Document
Go to the Docket

We are unable to display this document.

Refresh this Document
Go to the Docket