OMAR TACT AAAA
`
`US 20050057497A1
`
`as) United States
`a2) Patent Application Publication 0) Pub. No.: US 2005/0057497 Al
`
`
` Kawahara (43) Pub. Date: Mar. 17, 2005
`
`(54) METHOD AND APPARATUS FOR
`MANIPULATING. TWO-DIMENSIONAL
`WINDOWSWITHIN A
`THREE-DIMENSIONAL DISPLAY MODEL
`
`(76)
`
`Inventor: Hideya Kawahara, Mountain View, CA
`(US)
`
`Correspondence Address:
`c/o A. RICHARD PARK, REG. NO. 41241
`PARK, VAUGHAN & FLEMING LLP
`2820 FIFTH STREET
`DAVIS, CA 95616 (US)
`:
`Appl. No.:
`10/663,609
`Appl.
`No
`663,
`Filed:
`Sep. 15, 2003
`
`21)
`(21)
`(22)
`
`Publication Classification
`
`CSL) Tt C0 eecccccceeecccssssssnseeeccessnsnnnnsecceesnnnees G09G 5/08
`
`(52) U.S. Che veessssssststsssssstststssistststsensietet 345/157
`
`(57)
`
`ABSTRACT
`
`One embodimentof the present invention provides a system
`that facilitates manipulating a 2D window within a three-
`dimensional
`(3D) display model. During operation,
`the
`system receives an input from a 2D pointing device, wherein
`the input specifies a 2D offset within a 2D display, and
`wherein the 2D display provides a view into the 3D display
`model. Next, the system uses the 2D offset to move a cursor
`to a position in the 2D display, and then determinesif the
`cursor overlaps a window within the 3D display model. If so,
`the system determines a 2D position of the cursor with
`respect to a 2D coordinate system for the window, and
`communicates this 2D position to an application associated
`with the window. This enables a user of the 2D pointing
`device to interact with the application.
`
`3D-DISPLAY MODEL
`102
`
`RAY 107
`<{TTT
`
`__
`
`WINDOW 108
`
`VIEWPOINT
`106
`
`2D DISPLAY 104
`
` WINDOW 110
`e‘1.’.:1‘‘’‘.‘.''‘‘.’‘.11'‘..‘‘..‘',..,'*e‘1.‘‘‘..’
`122 TEXTURE
`132
`MOUSE 136 BACK BUFFER 128
`
`(Xq,¥2, 25)
`
`XSERVER
`(XWINDOW MGR)
`
`(X,Y)
`
`3D SCENE
`' MANAGER
`134
`
`1
`
`APPLE 1021
`
`APPLE 1021
`
`1
`
`

`

`OLLMOCNIM
`
`col
`
`%ZzAx)
`
`US 2005/0057497 Al
`
`ANSOSde
`
`YAOVNVW:
`
`
`
`gelASNOW
`
`SYNLXSL
`
`cel
`
`L‘Sls
`
`Patent Application Publication Mar. 17,2005 Sheet 1 of 9
`
`go.MOCNM~~))©—rs|dTEeHELNEee>LOLAWY
`
`JAGOWAV1dSIG-0€
`
`vOlLAWIdSICGz
`
`LNIOdMAIA
`
`901
`
`(HOWMOGNIMX)
`
`daAdaSX
`
`ezh
`
`
`
`
`
`ezlYasdNngdHOVE
`
`2
`
`
`
`
`

`

`Patent Application Publication Mar. 17, 2005
`
`Sheet 2 of 9
`
`US 2005/0057497 Al
`
`START
`
`
`
`
`RECEIVE INPUT FROM 2D POINTING
`DEVICE
`302
`
`
`ROTATE OBJECTS WITHIN 3D |
`DISPLAY MODEL AROUND
`VIEWPOINT
`304
`
`
`
`FIG. 3
`
`START
`
`
`
`
`RECEIVE INPUT SPECIFYING A 2D
`OFFSET
`202
`
`USE 2D OFFSET TO MOVE CURSOR
`TO A POSITIONIN 2D DISPLAY
`204
`
`OVERLAP WINDOW?
`
`
`
`
`
`
`DETERMINE 2D POSITION OF
`CURSOR WITH RESPECTTO 2D
`
`
`COORDINATE SYSTEM OF WINDOW
`208
`
`
`
`COMMUNICATE2D POSITION TO
`
`APPLICATION ASSOCIATED WITH
`
`WINDOW
`210
`
`
`
`FIG. 2
`
`3
`
`

`

`Patent Application Publication Mar. 17,2005 Sheet 3 of 9
`
`US 2005/0057497 Al
`
`WINDOW
`403
`
`ICON
`413
`
`ICON
`414
`
`FIG. 4A
`
`2D DISPLAY 104
`
`2D DISPLAY 104 WINDOW 401
`
`we401
`
`WINDOW
`
`Se
`
`Ww
`
`WINDOW
`402
`
`FIG. 4B
`
`4
`
`

`

`Patent Application Publication Mar. 17,2005 Sheet 4 of 9
`
`US 2005/0057497 Al
`
`START
`
`
`
`
`RECEIVE COMMAND TO ROTATE
`WINDOWS
`420
`
`ROTATE WINDOWSSO THAT
`WINDOWS CONTENTS REMAIN
`VISIBLE WHILE WINDOWS OCCUPY
`LESS SPACE
`
`422
`
`DRAW TITLES ON SPINES OF
`WINDOWS
`424
`
`
`
`
`
`
`
`RECEIVE USER SELECTION OF
`
`WINDOW
`426
`
`
`MOVE WINDOW TO FRONT AND
`UNROTATE TO FACE USER
`
`428
`
`
`MOVE OTHER WINDOWSBACK TO
`ORIGINAL POSITIONS
`
`430
`
`
`
`
`
`END
`
`FIG. 4C
`
`5
`
`

`

`Patent Application Publication Mar. 17,2005 Sheet 5 of 9
`
`US 2005/0057497 Al
`
`WINDOW 501
`
`WINDOW
`502
`
`START
`
`RECEIVE COMMAND TO
`MINIMIZE WINDOW
`510
`
`
`
`
`
`
`TILT WINDOW SO THAT
`
`
`WINDOW CONTENTS
`REMAIN VISIBLE AND SPINE
`BECOMES VISIBLE
`512
`
`
`
`MOVE MINIMIZED WINDOW
`
`TO EDGE OF DISPLAY
`514
`
`FIG. 5C
`
`2D DISPLAY 104
`
`
`2D DISPLAY 104
`
`FIG. 5A
`
`WINDOW
`502
`
`pecseceorcococe
`
`WINDOW
`
`FIG. 5B
`
`6
`
`

`

`Patent Application Publication Mar. 17,2005 Sheet 6 of 9
`
`US 2005/0057497 Al
`
`COMMAND MOVES WINDOW
`CLOSE TO EDGE OF
`DISPLAY
`602
`
`
`
`
`
`
`
`
`
`
`RECEIVE SELECTION OF
`WINDOW
`606
`
`
`
`
`IN RESPONSE TO
`
`
`SELECTION, UNTILT
`WINDOW
`608
`
`TILT WINDOW SO CONTENT
`OF WINDOW REMAINS
`VISIBLE WHILE WINDOW
`OCCUPIES LESS SPACE
`604
`
`
`
`FIG. 6E
`
`601
`
`104
`
`+<—
`
`FIG. 6A
`
`FIG. 6B
`
`mn
`DI
`
`
`FIG. 6C
`
`601
`
`104
`
`®B
`
`
`
`MOUSE
`CURSOR
`
`FIG. 6D
`
`7
`
`

`

`Patent Application Publication Mar. 17, 2005
`
`Sheet 7 of 9
`
`US 2005/0057497 Al
`
`START
`
`RECEIVE COMMAND TO
`ROTATE WINDOW
`704
`
`
`
`
`
`
`
`ROTATE WINDOW SO THAT
`APPLICATION INFORMATION
`
`
`ON BACK SIDE IS VISIBLE
`
`706
`
`FIG. 7C
`
`ROTATE
`COMMAND
`
`FRONTSIDE
`
`FIG. 7A
`
`
`
`
`BACK SIDE
`
`APPLICATION
`INFORMATION
`702
`
`FIG. 7B
`
`8
`
`

`

`Patent Application Publication Mar. 17,2005 Sheet 8 of 9
`
`US 2005/0057497 Al
`
`START
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`DETECT CURSOR CLOSE TO
`EDGE OF WINDOW
`812
`
`ROTATE WINDOW SO
`WINDOW CONTROLS ARE
`VISIBLE
`814
`
`RECEIVE COMMAND
`THROUGH WINDOW
`CONTROL
`
`ROTATE WINDOW BACK TO
`ORIGINAL ORIENTATION
`818
`
`
`
`816
`BUTTON 805
`
`FIG. 8A
`
`BUTTON802
`
`BUTTON 803
`
`FIG. 8C
`
`BUTTON 804
`
`FIG. 8B
`
`9
`
`

`

`Patent Application Publication Mar. 17,2005 Sheet 9 of 9
`
`US 2005/0057497 Al
`
`START
`
`
`
`RECEIVE GESTURE THROUGH
`POINTING DEVICE
`902
`
`MINIMIZE TOP-LEVEL WINDOW
`
`904
`
`START
`
`RECEIVE COMMAND FROM
`POINTING DEVICE TO THROW
`
`WINDOW
`
`4002
`
`
`
`
`THROW WINDOW
`
`1004
`
`
`
`
`
`
`
`
`
`RECEIVE CLICK ON ROOT WINDOW
`906
`
`FIG. 10
`
`RESTORE MINIMIZED WINDOW TO
`EXPANDED STATE
`908
`
`FIG. 9
`
`10
`
`10
`
`

`

`US 2005/0057497 Al
`
`Mar. 17, 2005
`
`METHOD AND APPARATUS FOR MANIPULATING
`TWO-DIMENSIONAL WINDOWSWITHIN A
`THREE-DIMENSIONAL DISPLAY MODEL
`
`RELATED APPLICATION
`
`[0001] The subject matter of this application is related to
`the subject matter in a co-pending non-provisional applica-
`tion entitled, “Enhancements for Manipulating Two-Dimen-
`sional Windows Within a Three-Dimensional Display
`Model,” by inventors Hideya Kawahara, Curtis J. Sasaki,
`Daniel J. Baigent and Yasuyo Okuda having Ser. No. TO BE
`ASSIGNED, andfiling date TO BE ASSIGNED(Attorney
`Docket No. SUN04-0196-JLM).
`BACKGROUND
`
`SUMMARY
`
`[0009] One embodimentof the present invention provides
`a system that facilitates manipulating a 2D window within a
`three-dimensional (3D) display model. During operation,
`the system receives an input from a 2D pointing device,
`wherein the input specifies a 2D offset within a 2D display,
`and wherein the 2D display provides a view into the 3D
`display model. Next, the system uses the 2D offset to move
`a cursor to a position in the 2D display, and then determines
`if the cursor overlaps a window within the 3D display
`model. If so, the system determines a 2D position of the
`cursor with respect
`to a 2D coordinate system for the
`window, and communicates this 2D position to an applica-
`tion associated with the window. This enables a user of the
`
`[0002]
`
`1. Field of the Invention
`
`2D pointing device to interact with the application.
`
`[0003] The present invention relates to user interfaces for
`computer systems. More specifically, the present invention
`relates to a method and an apparatusthat facilitates manipu-
`lating two-dimensional windows that are mapped into a
`three-dimensional display model.
`
`[0004]
`
`2. Related Art
`
`[0005] Today, most personal computers and other high-
`end devices support window-based graphical user interfaces
`(GUIs), which wereoriginally developed back in the 1970s.
`These window-based interfaces allow a user to manipulate
`windowsthrough a pointing device (such as a mouse), in
`much the same way that pages can be manipulated on a
`desktop. However, because of limitations on graphical pro-
`cessing power at the time windows were being developed,
`manyof the design decisions for windows were made with
`computational efficiency in mind. In particular, window-
`based systems providea very flat (two-dimensional) 2D user
`experience, and windows are typically manipulated using
`operations that keep modifications of display pixels to a
`minimum. Even
`today’s
`desktop
`environments
`like
`Microsoft Windows (distributed by the Microsoft Corpora-
`tion of Redmond, Wash.) include vestiges of design deci-
`sions made back then.
`
`In recent years, because of increasing computa-
`[0006]
`requirements of 3D applications, especially 3D
`tional
`games, the graphical processing power of personal comput-
`ers and other high-end devices has increased dramatically.
`For example,
`a middle range PC graphics card,
`the
`“GeForce2 GTS”distributed by the NVIDIA Corporation of
`Sunnyvale, Calif., provides a 3D rendering speed of 25
`million polygon-per-second, and Microsoft’s “Xbox” game
`console provides 125 million polygon-per-second. These
`numbers are significantly better than those of high-end
`graphics workstation in the early 1990’s, whichcosttens of
`thousands (and even hundreds of thousands) of dollars.
`
`[0007] As graphical processing power has increased in
`recent years, a number of 3D user interfaces have been
`developed. These 3D interfaces typically allow a user to
`navigate through and manipulate 3D objects. However,
`these 3D interfaces are mainly focused on exploiting 3D
`capabilities, while little attention has been given to support-
`ing existing, legacy window-based 2D applications within
`these 3D user interfaces.
`
`[0008] Hence, what needed is a method and an apparatus
`that supports legacy 2D window-based applications within a
`3D user interface.
`
`Ina variation on this embodiment, determining if
`[0010]
`the cursor overlaps a window within the 3D display model
`involves projecting a ray from a predefined viewpoint in the
`3D display model through the cursor, which is located in a
`rectangle representing the 2D display in the 3D display
`model,
`toward one or more windows in the 3D display
`model, and then determining if the ray intersects a window.
`
`Ina further variation, determining the 2D position
`(0011]
`of the cursor with respect to the 2D coordinate system of the
`window involvesfirst determining a 3D position where the
`ray intersects the window within the 3D display model, and
`then transforming the 3D position into a 2D position with
`respect to the 2D coordinate system for the window based
`uponthe size, position and orientation of the window within
`the 3D display model.
`
`Ina further variation, the size, position and orien-
`{0012]
`tation of the window within the 3D display model are
`specified by a numberofattributes of the window,including:
`a height, a width, an x-position, a y-position, a z-position, a
`first rotation around a vertical axis of the window, and a
`second rotation around a horizontal axis of the window.
`
`Ina variation on this embodiment, in response to
`{0013]
`another input from the 2D pointing device,
`the system
`changes a viewing angle for the 3D display model by
`rotating objects within the 3D display model around a
`predefined viewpoint.
`
`In a variation on this embodiment, if the cursor
`{0014]
`overlaps a given window,
`the given window becomes a
`selected window and appears opaque while other windows
`within the 3D display model appear translucent.
`
`Ina variation on this embodiment, if a commandis
`{0015]
`received to minimize a window, the window minimization
`operation is illustrated as an animation that moves the
`window toward a minimized position near a border of the 2D
`display while reducing the size of the window to its mini-
`mized size.
`
`Ina variation on this embodiment, if a commandis
`{0016]
`received to close a window, the window closing operation is
`illustrated as an animation that throws the window away by
`moving the window toward the background of the 3D
`display model and causing the window to fade away.
`
`Ina variation on this embodiment, if a commandis
`{0017]
`received to rotate all windows in the 3D display model, the
`system rotates all windowsin the 3D display model, so that
`windowsare viewed from an oblique angle through the 2D
`
`11
`
`11
`
`

`

`US 2005/0057497 Al
`
`Mar. 17, 2005
`
`display, whereby the contents of the windows remain vis-
`ible, while the windowsoccupyless space in the 2D display
`and are less likely to overlap each other.
`
`Ina further variation, when a window is rotated, a
`[0018]
`spine located on a side edge of the window becomesvisible,
`wherein the spine contains identification information for the
`window.
`
`Ina variation on this embodiment, upon receiving
`[0026]
`a predefined gesture through a pointing device, the system
`minimizes a top-level window in the 2D display, whereby
`repeating the predefined gesture causes subsequent top-level
`windows to be minimized.
`
`In a further variation, upon receiving a window
`[0027]
`restoration command, the system restores minimized win-
`dowsto their expandedstate.
`{0019] Ina further variation, whenauser selects one of the
`
`[0028]
`Ina variation on this embodiment,if the command
`rotated windows, the system movesthe selected window in
`is entered through a pointing device and the command
`front of the other windows. The system also unrotates the
`throws the window by moving the window quickly and
`selected window so it faces the user, and moves the other
`releasing it, the system “throws” the window by moving the
`windowsbackto their original positions and orientations.
`window in a continuous animated motion, which moves the
`window into the background of the 3D display model or
`minimizes the window.
`
`Ina variation on this embodiment, the 2D pointing
`[0020]
`device can include: a mouse, a track ball, a joystick, or a
`glide point.
`
`[0021] One embodimentof the present invention provides
`a system that facilitates manipulating a window within a
`three-dimensional (3D) display model, wherein the window
`provides a 2D user interface for a 2D application. During
`operation, the system displays a view into the 3D display
`model
`through a two-dimensional
`(2D) display. Upon
`receiving a command to manipulate the window within the
`3D display model,
`the system manipulates the window
`within the 3D display model so that the manipulation is
`visible within the 2D display.
`
`Ina variation on this embodiment,if the command
`[0022]
`movesthe window in close proximity to an edge of the 2D
`display, the system tilts the window so that the window
`appears at an oblique angle in the 2D display, whereby the
`contents of the window remain visible, while the window
`occupies less space in the 2D display andis less likely to
`overlap other windows.
`
`Ina variation on this embodiment, determining the
`[0023]
`2D position of the cursor with respect to the 2D coordinate
`system of the window involves determining a 3D position
`where the ray intersects the window within the 3D display
`model. It also involves transforming the 3D position in the
`3D display model into a corresponding 2D position with
`respect to the 2D coordinate system for the window based
`upon the size, position and orientation of the window within
`the 3D display model.
`
`Ina variation on this embodiment,if the command
`[0024]
`rotates the window so that the backside of the window is
`
`visible, the system displays information associated with the
`2D application on the backside of the window. This infor-
`mation can include: application version information, appli-
`cation settings, application parameters, application proper-
`ties, and notes associated with a file or a web pagethat is
`displayed in the window.In a further variation, the backside
`of the window can accept user input,
`including change
`settings, parameters, properties and/or notes.
`
`Ina variation on this embodiment,if the command
`[0025]
`is to minimize the window, manipulating the window
`involves: tilting the window so that a spine located on a side
`edge of the windowis visible and the contents of the window
`remains visible, wherein the spine contains identification
`information for the window. It also involves moving the
`minimized window to an edgeof the 2D display, wherein the
`operations of turning and moving the window are animated
`as a continuous motion.
`
`In a variation on this embodiment, receiving the
`[0029]
`command can involve: rotating the window so that window
`controls on the edge of the window become visible in
`response to a cursor moving close to an edge of a window;
`receiving the command through a window control; and then
`rotating the window backto its original orientation.
`
`BRIEF DESCRIPTION OF THE FIGURES
`
`[0030] FIG. 1 illustrates a 3D display model with sup-
`porting components in accordance with an embodiment of
`the present invention.
`
`[0031] FIG. 2 presents a flow chart illustrating how input
`from a pointing device is communicated to an application
`associated with a window in a 3D display model in accor-
`dance with an embodimentof the present invention.
`
`[0032] FIG. 3 presents a flow chart illustrating how input
`from a pointing device causes objects to rotate around a
`viewpoint in the 3D display model in accordance with an
`embodiment of the present invention.
`
`FIG.4Aillustrates an exemplary set of windowsin
`[0033]
`the 3D display model in accordance with an embodimentof
`the present invention.
`
`[0034] FIG. 4B illustrates how windowsare rotated in
`accordance with an embodimentof the present invention.
`
`[0035] FIG. 4C presents a flow chart of the process of
`rotating windowsin accordance with an embodimentof the
`present invention.
`
`[0036] FIG. 5A illustrates an exemplary window in the
`3D display model in accordance with an embodimentof the
`present invention.
`
`[0037] FIG. 5B illustrates how the exemplary window is
`minimized in accordance with an embodimentofthe present
`invention.
`
`[0038] FIG. 5C presents a flow chart of the process of
`minimizing a window in accordance with an embodiment of
`the present invention.
`
`[0039] FIG. 6A illustrates an exemplary window in the
`3D display model in accordance with an embodimentof the
`present invention.
`
`FIG.6B illustrates how a window is moved toward
`[0040]
`the edge of the display in accordance with an embodiment
`of the present invention.
`
`12
`
`12
`
`

`

`US 2005/0057497 Al
`
`Mar. 17, 2005
`
`[0041] FIG. 6C illustrates how a window is tilted in
`accordance with an embodimentof the present invention.
`
`[0042] FIG. 6D illustrates how a window is untilted in
`accordance with an embodimentof the present invention.
`
`[0043] FIG. 6E presents a flow chart of the process of
`minimizing windowsin accordance with an embodimentof
`the present invention.
`
`[0044] FIG. 7A illustrates an exemplary window in the
`3D display model in accordance with an embodimentof the
`present invention.
`
`[0045] FIG. 7B illustrates how the exemplary window is
`rotated to display application information on the backside of
`the window in accordance with an embodiment of the
`present invention.
`
`[0046] FIG. 7C presents a flow chart of the process of
`rotating a window in accordance with an embodimentof the
`present invention.
`
`[0047] FIG. 8A illustrates an exemplary window in the
`3D display model in accordance with an embodimentof the
`present invention.
`
`[0048] FIG. 8B illustrates how the exemplary window is
`rotated to reveal window controls on the edge of the window
`in accordance with an embodimentof the present invention.
`
`[0049] FIG. 8C presents a flow chart of the process of
`rotating a window to reveal window controls in accordance
`with an embodiment of the present invention.
`
`[0050] FIG. 9 presents a flow chart of the process of
`minimizing a top-level window in response to a gesture
`entered into a pointing device in accordance with an
`embodiment of the present invention.
`
`[0051] FIG. 10 presents a flow chart of the process of
`throwing a window in accordance with an embodiment of
`the present invention.
`
`DETAILED DESCRIPTION
`
`[0052] The following description is presented to enable
`any person skilled in the art to make and use the invention,
`and is provided in the context of a particular application and
`its requirements. Various modifications to the disclosed
`embodiments will be readily apparent to those skilled in the
`art, and the general principles defined herein may be applied
`to other embodiments and applications without departing
`from the spirit and scope of the present invention. Thus, the
`present inventionis not intendedto be limited to the embodi-
`ments shown, but
`is to be accorded the widest scope
`consistent with the principles and features disclosed herein.
`
`[0053] The data structures and code described in this
`detailed description are typically stored on a computer
`readable storage medium, which may be any device or
`medium that can store code and/or data for use by a
`computer system. This includes, but
`is not
`limited to,
`magnetic and optical storage devices such as disk drives,
`magnetic tape, CDs (compact discs) and DVDs (digital
`versatile discs or digital video discs), and computer instruc-
`tion signals embodied in a transmission medium (with or
`without a carrier wave upon which the signals are modu-
`lated). For example, the transmission medium may include
`a communications network, such as the Internet.
`
`[0054]
`
`3D Display Model
`
`[0055] FIG. 1 illustrates 3D display model 102 with
`supporting components in accordance with an embodiment
`of the present invention. Morespecifically, the top portion of
`FIG.3 illustrates 3D display model 102, which includes a
`number of 3D objects including window 110 and window
`112. Note that windows 108 and 110 are actually 3D objects
`which represent 2D windows. Hence, windows108 and 110
`can be moved and rotated within 3D display model 102,
`while they provide a 2D output and receive input for
`associated 2D applications. 3D display model 102 can
`additionally include a background (which is not shown).
`
`[0056] Windows 108 and 110 can be associated with a
`number of window attributes. For example, window 110 can
`include x, y, and z position attributes that specify the 3D
`position of the center of window 110 within 3D display
`model 102, as well as a rotation attributes that specify
`rotations of window 110 around horizontal and vertical axes.
`
`Window 110 can also be associated with scaling factor,
`translucency and shapeattributes.
`
`3D objects within 3D display model 102 are
`[0057]
`viewed from a viewpoint 106 through a 2D display 104,
`which is represented by a 2D rectangle within 3D display
`model 102. During the rendering process, various well-
`known techniques, such as ray tracing, are used to map
`objects from 3D display model 102 into corresponding
`locations in 2D display 104.
`
`[0058] The bottom portion of FIG. 1 illustrates some of
`the system components that make it possible to map 2D
`windowsinto 3D display model 102 in accordance with an
`embodimentof the present invention. Referring to FIG. 1,
`applications 114 and 116 are associated with windows 108
`and 110, respectively. A number of components are involved
`in facilitating this association. In particular, applications 114
`and 116 are associated with xclients 118 and 120, respec-
`tively. Xclients 118 and 120 in turn interact with xserver
`122, which includes an associated xwindow manager. These
`components worktogether to render output bitmaps 124 and
`126 for applications 114 and 116 to be displayed in windows
`108 and 110, respectively. These bitmaps 124 and 126 are
`maintained within back buffer 128.
`
`[0059] Code module 130 causes bitmaps 124 and 126 to
`be displayed on corresponding windows 108 and 110. More
`specifically, code module 130 retrieves bitmap 126 and
`coverts it into a texture 132, which is displayed on the front
`face of window 110. This is accomplished though interac-
`tions with 3D scene manager 134. Bitmap 124 is similarly
`mapped into window 108.
`
`3D scene manager 134 can also received input
`[0060]
`from a 2D pointing device, such as mouse 136, and can
`communicate this input to applications 114 and 116 in the
`following way. 3D scene manger 134first receives an input
`specifying a 2D offset from mouse 136 (step 202). Next, the
`system uses this 2D offset to move a cursor 109 to a new
`position (x,,y,) in 2D display 104 (step 204).
`
`[0061] The system then determines if cursor 109 overlaps
`a window in 3D display model 102 (step 206). This can be
`accomplished by projecting a ray 107 from viewpoint 106
`through cursor 109 and then determiningif the ray intersects
`a window.If there is no overlap, the process is complete.
`
`13
`
`13
`
`

`

`US 2005/0057497 Al
`
`Mar. 17, 2005
`
`[0062] Otherwise, if there is overlap, the system uses the
`3D position (X3,y,,z.) within display model 102 where ray
`107 intersects window 110, as well as attributes of window
`110, such as position and rotationattributes, to determine the
`2D position (x3,y) of this intersection with respect to a 2D
`coordinate system of window 110 (step 208). The system
`then communicates this 2D position (x;,y3) to application
`116, which is associated with window 110 (step 210).
`
`[0063] Various user inputs, for example through mouse
`136 or a keyboard, can be used to manipulate windows
`within 3D display model 102. Some of these manipulations
`are described below.
`
`[0064] Rotation Around Viewpoint
`
`[0065] FIG. 3 presents a flow chartillustrating how input
`from a pointing device causes objects to rotate around a
`viewpoint 106 in 3D display model 102 in accordance with
`an embodiment of the present invention. First, the system
`receives an input from a 2D pointing device indicating that
`a rotation is desired (step 302). For example, the system can
`receive a movement input from mouse 136. In response to
`this input,
`the system can rotate objects within the 3D
`display model around viewpoint 106, or alternatively around
`another point within 3D display model 102 (step 304). This
`rotational motion makes it easier for a user to identify
`window boundaries and also givesthe user a feeling of depth
`and space.
`
`[0066] Rotating Windows
`
`FIG.4Aillustrates an exemplary set of windowsin
`[0067]
`3D display model 102 in accordance with an embodimentof
`the present
`invention. This exemplary set of windows
`includes windows 401-404. In FIG. 4A, window 403 is
`partly obscured, and window 404 is completely obscured, by
`windows 401-402. Windows 401-404 are additionally asso-
`ciated with icons 411-414, respectively. However,
`icons
`411-412 are not visible in FIG. 4A because they are
`obscured by window 401.
`
`[0068] FIG. 4B illustrates how windows 401-404 are
`rotated in accordance with an embodimentof the present
`invention. In FIG. 4B, windows 401-404 are rotated so that
`they appear at an oblique angle, wherein the contents of the
`windows remain visible, while the windows occupy less
`space and are less likely to overlap each other. Note that
`windows 403 and 404 are now completely visible and icons
`411 and 412 are no longer obscured. Also note thattitles
`containing descriptive information appear on spines located
`on the edges of the windows 401-404.
`
`[0069] FIG. 4C presents a flow chart of the process of
`rotating windowsin accordance with an embodimentof the
`present invention. First, the system receives a pre-specified
`commandto rotate all of the windows. This command can be
`
`received from the pointing device, a keyboard, or some other
`input device (step 420). In response to this command, the
`system rotates windows 401-404 to an oblique angle so that
`the contents of the windows remain visible, while the
`windows occupy less space (step 422). The system also
`drawstitles on spines of the windows(step 424).
`
`[0070] Next, the system can receive a user selection of a
`window. For example, when the user moves cursor 109 over
`window 401, window 401is selected (step 426). In response
`to this user selection, the system movesthe selected window
`
`in front of all other windowsin 3D display model 102 and
`unrotates the selected window so that it faces the user (step
`428). The system also moves other windowsback to their
`original unrotated positions.
`In one embodiment of the
`present
`invention,
`the selected window appears opaque,
`while other windowsappear translucent.
`
`[0071] Minimizing Windows
`
`[0072] FIG. 5A illustrates exemplary windows 501-502 in
`the 3D display model 102, and FIG. 5B illustrates how
`window 501 is minimized in accordance with an embodi-
`
`mentof the present invention. Referring to the flow chart in
`FIG. 5C,the system first receives a command to minimize
`window 501 (step 510). For example, mouse 136 can be
`used to select a minimization button on window 501. In
`response to this minimization command, window 501 is
`tilted (and possibly reduced in size) so that the contents of
`window 501 remain visible, while window 501 occupiesless
`space (step 512). Tilting window 501 also causesa title on
`the spine of window 501 to become visible. At the same
`time, window 501 is moved toward an edge of the display
`(step 514).
`
`[0073] These operations take place through a continuous
`animation that starts with the original unminimized window
`and ends with the minimized window.This can be accom-
`
`plished by incrementally changing window parameters, such
`as position, rotation and scaling factor parameters. In this
`way,
`the user is better able to associate the minimized
`window with the original window.
`
`[0074] Once window 501 is minimized, another command
`from the user can cause the window to be maximizedsothat
`
`the window can be more easily viewed and so that the
`window can receive an input.
`
`[0075] Tilting Windows
`
`FIG.6Aillustrates an exemplary window in 601 in
`[0076]
`3D display model 102, and FIGS. 6B-6D illustrates how
`window 601 is tilted whenit is moved toward the edge of 2D
`display 104 in accordance with an embodiment of the
`present invention. Referring the flowchart in FIG. 6A, the
`system first receives a command to move the window to the
`edge of the display (step 602). For example, the user can use
`a pointing device to move window 601 so that it is near the
`edge of 2D display 104 (see FIG. 6B). When window 601
`is moved near the edge of 2D display 104, the system tilts
`window 601, so that the contents of window 601 remain
`visible, while window 601 occupies less space and is less
`likely to overlap other windows(step 604 see FIG. 6C).
`
`[0077] Next, the system can receive a selection of window
`601 by a user. For example, the user may move cursor 109
`near window 601 (step 606).
`In response to this user
`selection, the system can untilt the window 601 so that the
`user can see it better and can enter commands into the
`window(step 608, see FIG. 6D).
`
`[0078] Displaying Application Information on Back of
`Window
`
`[0079] FIG. 7A illustrates an exemplary window 701 in
`3D display model 102, and FIG.7B illustrates how window
`701 is rotated to display application information on the
`backside of window 701 in accordance with an embodiment
`of the present invention. Referring to the flow chart in FIG.
`7C, the system first receives a command (possibly through
`
`14
`
`14
`
`

`

`US 2005/0057497 Al
`
`Mar. 17, 2005
`
`a mouse or a keyboard) to rotate window 701 (step 704). In
`response to this command,the system rotates window 701 so
`that application information 702 on the backside of window
`701 becomesvisible (step 706). This application informa-
`tion can include application version information, application
`settings, application parameters, and application properties.
`It can also include notes associated with a file or a web page
`that is displayed in the window. In one embodimentof the
`present invention,
`the system allows the user to modify
`application information 702 on the backside of window 701.
`This enables the user to change application parameters, if
`necessary.
`
`[0080] Using Window Controls on Side of Window
`
`[0081] FIG. 8A illustrates an exemplary window 801 in
`3D display model 102, and FIG.8B illustrates how window
`801 is rotated to reveal window controls on the edge of the
`window in accordance with an embodimentof the present
`invention. Referring to the flow chart illustrated in FIG. 8C,
`the system first detects a cursor close to the edge of window
`801 (step 812). In response to detecting the cursor,
`the
`system rotates the window so that window controls on the
`edge of window 801are visible (step 814). For example, in
`FIG. 8B buttons 802-805 become visible. Note that
`in
`general other types of controls, such as pull-down menus,
`can be located on the edge of window 801. After the user
`enters a commandinto a window control(step 816), or after
`the user moves cursor 109 away from window 801,
`the
`system rotates window 801 back to its original orientation
`(step 818).
`
`[0082] Minimizing Top-Level Windows
`
`[0083] FIG. 9 presents a flow chartillustrating the process
`of minimizing a top-level window in response to a gesture
`inputted through a pointing device in accordance with an
`embodiment of the present
`invention. The system first
`receives a pre-defined gesture through a pointing device,
`such as mouse 136 (step 902). For example, the gesture can
`be a waving motion that causes cursor 109 to move in a
`specific pattern across 2D display 104. In response to this
`gesture, the system minimizes the top-level window (step
`904). As is indicated by the looping arrow in FIG. 9,
`repeating the predefined gesture causes subsequent top-level
`windowsto be minimized.
`
`[0084] Next, upon receiving a window restoration com-
`mand, such as a click on a special button on a root window
`(step 906), the system restores all minimized windowsto
`their expandedstate (step 908).
`
`[0085]