a2, United States Patent
`US 6,229,542 B1
`(10) Patent No.:
`Miller
`(45) Date of Patent:
`May8, 2001
`
`
`US006229542B1
`
`(54) METHOD AND APPARATUS FOR
`MANAGING WINDOWSIN THREE
`DIMENSIONSIN A TWO DIMENSIONAL
`WINDOWING SYSTEM
`;
`caar
`Inventor:
`John David Miller, Beaverton, OR
`(US)
`
`(75)
`
`(73) Assignee:
`
`Intel Corporation, Santa Clara, CA
`(US)
`
`(*) Notice:
`
`Subject to any disclaimer, the term ofthis
`patent is extended or adjusted under 35
`US.C. 154(b) by 0 days.
`
`(21) Appl. No.: 09/113,814
`
`(22)
`
`Filed:
`
`Jul. 10, 1998
`
`(SL) Ute C07 ciccccscsscssssssssnssnsssesnesnssnasnen GO6F 3/00
`(52) US. Ch. coscescssesecssssene 345/358; 345/355; 345/343
`
`(58) Field of Search 0... 345/339, 340,
`345/342, 343, 344, 345, 348, 355, 358,
`976, 977, 419, 425, 430, 433, 435, 438,
`439, 473
`
`(56)
`
`References Cited
`U.S. PATENT DOCUMENTS
`
`UAL9BS PIKE eceescccssssssssssssssseeeseeeeeee 345/344
`A,SS5,7TS—
`4/1994 Kreitmanetal.
`. 345/348
`5,303,388 *
`
`9/1996 Strasnick etal.
`.. 345/427
`5,555,354 *
`
`5,678,015 * 10/1997 Goh .........
`.. 345/340
`5,689,628 * 11/1997 Robertson......
`345/427
`
`3/1998 Matthews,III et al... 345/419
`5,724,492 *
`
`5,745,109 *
`4/1998 Nakano et al. wee 345/340
`5,754,809 *
`5/1998 Gandre veces 345/343
`
`5,774,125 *
`6/1998 Suzuoki et al.
`. 345/430
`5,835,094 * 11/1998 Ermel et al. wee 345/355
`5,838,326 * 11/1998 Cardet al. cece 345/355
`5,880,733 *
`3/1999 Horvitz etal.
`. 345/355
`
`5,990,900 * 11/1999 Seago cvcccsssssssssssssesseeeeen 345/427
`6,081,270 *
`6/2000 Berry et al. vvecsscsssssssssenen 345/419
`OTHER PUBLICATIONS
`
`Card, Stuart K.et al., “The WebBook and the Web Forager:
`An Information Workspace for the World—Wide Web”, CHI
`96 Conference on Human Factors in Computing Systems,
`Apr. 13-18, 1996, pp. 111-117.
`
`* cited by examiner
`
`Primary Examiner—Crescelle N. dela Torre
`(74) Attorney, Agent, or Firm—Steven P. Skabrat
`
`(57)
`
`ABSTRACT
`
`Managing windowsin a graphical user interface by receiv-
`ing a signal indicating a gesture from a user, capturing pixels
`of a window,applying the captured pixels as a texture to a
`display object in a three dimensional window,and animating
`the display objectto a first location in the three dimensional
`window corresponding to the window, when the gesture
`indicates deactivating the window. Further actions include
`moving the display object to a second location in the three
`dimensional window, displaying the window overthe dis-
`play object, and hiding the display object, when the gesture
`indicates activating the window.
`
`25 Claims, 8 Drawing Sheets
`
`1
`
`APPLE 1035
`
`APPLE 1035
`
`1
`
`

`

`U.S. Patent
`
`May8, 2001
`
`Sheet 1 of8
`
`US 6,229,542 Bl
`
`~ Gotens Welsh.
`
`Fans
`
`FIG. 1
`(Prior Art)
`
`2
`
`

`

`U.S. Patent
`
`May8, 2001
`
`Sheet 2 of8
`
`US 6,229,542 Bl
`
`12
`
`FIG. 2
`
`3
`
`

`

`U.S. Patent
`
`May8, 2001
`
`Sheet 3 of8
`
`US 6,229,542 Bl
`
`CAMERA
`
`3D
`VIRTUAL
`
`FAR
`PLANE
`
`FIG. 3
`
`4
`
`

`

`U.S. Patent
`
`May8, 2001
`
`Sheet 4 of 8
`
`US 6,229,542 BI
`
`
`
`5
`
`

`

`U.S. Patent
`
`May8, 2001
`
`Sheet 5 of 8
`
`US 6,229,542 Bl
`
`
`
`
`
`6
`
`

`

`U.S. Patent
`
`May8, 2001
`
`Sheet 6 of8
`
`US 6,229,542 Bl
`
`
`
`MICROPROCESSOR
`
`WINDOW
`ACTIVATOR
`
`
`
`— iO
`
`CACHE
`
`104
`
`PROCESSOR
`BUS
`/
`105
`
`
`
`HOST
`BRIDGE
`
`011
`
`//O BUS
`BRIDGE
`
`
`
`108
`
`HIGH PERFORMANCEI/O BUS
`
`112
`
`MAIN
`MEMORY
`
`
`
`114
`
`116
`
`VIDEO
`MEMORY
`
`VIDEO
`DISPLAY
`
`
`
`
`
`MEMORY
`
`
`
`
`
`
`
`
`
` 0
`12
`
`118
`
`STANDARD I/O BUS
`
`122
`
`MASS
`STORAGE
`
`KEYBOARD AND
`POINTING DEVICES
`
`FIG. 6
`
`7
`
`

`

`U.S. Patent
`
`May8, 2001
`
`Sheet 7 of8
`
`US 6,229,542 Bl
`
`USER GESTURES TO "PUSH
`
`200
`
`TAKE SNAPSHOT OF SELECTED WINDOW
`
`BACK" OR MINIMIZE AN ACTIVE WINDOW
`
`
`
`
`
`APPLY SNAPSHOT AS TEXTURE TO A
`DISPLAY OBJECT IN THE 3D DESKTOP WINDOW
`HAVING A SHAPE SIMILAR TO THE SELECTED WINDOW
`
`
`
`ANIMATE THE 3D DISPLAY OBJECT TO
`ITS NEW PLACE IN THE 3D DESKTOP WINDOW
`
`
`
`FIG. 7
`
`8
`
`

`

`U.S. Patent
`
`May8, 2001
`
`Sheet 8 of8
`
`US 6,229,542 Bl
`
`220
`
`
`
`USER GESTURES TO BRING A
`SELECTED APPLICATION WINDOW TO
`THE FOREGROUND OF 3D DESKTOP WINDOW
`
`
`
`
`
`
`
`
`
`MOVE AND ORIENT 3D DISPLAY OBJECT
`FOR THE SELECTED WINDOW
`TO THE FRONT OF THE 3D DESKTOP WINDOW
`
`224
`
`
`
`SHOW SELECTED 2D WINDOW DIRECTLY
`OVER 3D DISPLAY OBJECT FOR THE SELECTED WINDOW
`
`226
`
`HIDE 3D DISPLAY OBJECT FOR THE
`SELECTED WINDOW
`
`FIG. 8
`
`9
`
`

`

`US 6,229,542 B1
`
`1
`METHOD AND APPARATUS FOR
`MANAGING WINDOWSIN THREE
`DIMENSIONSIN A TWO DIMENSIONAL
`WINDOWING SYSTEM
`
`BACKGROUND
`
`1. Field
`
`The present invention relates generally to graphical user
`interfaces in computer systems and morespecifically to a
`generalized three dimensional graphical user interface.
`2. Prior Art
`
`Windowing systems are commonly used in graphical user
`interfaces of modern computer systems. The use of windows
`solves the problem of simultaneously representing more
`than one set of display output data from one or more
`application programs on a single computer monitor. The
`windowing solution typically allocates at least one display
`window on the computer screen for each executing appli-
`cation program and allowsthe user to resize and move the
`windowsas desired. The operating system (OS) software of
`the computer system managesthe resulting lists of possibly
`overlapping rectangular windows, thereby allowing appli-
`cation programs to draw on the portions of their windows
`that are visible (e.g., not obscured or clipped by the screen
`extents or other windows). The OStypically notifies appli-
`cation programs when portions of their windows become
`exposed and needto be repainted or redrawn. In somecases,
`application programs may request that a window manager
`program within the OS handle these events directly from an
`off-screen memory buffer either in whole (by having the
`application program draw directly to the off-screen memory
`buffer instead of the window,either for the whole windowor
`only for the obscured portion), or in part (as when the
`window manager “pops up” a menu—the contents underthe
`area to be destroyed are saved into off-screen memory so
`that it can be quickly restored after the menu is closed).
`The windowing system managesthe precious resource of
`screen space by manipulating a two and one half dimen-
`sional space. The space has dimensions of horizontal and
`vertical coordinates, as well as an order for overlapping
`windows. Windowsare objects in this space. The window-
`ing system maintains state information about these objects,
`including their position, extents, and sometimes even their
`contents.
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`2
`use. Interaction with many windowsin a single session of
`using the computer system may be cumbersome, confusing
`and slow. The well known metaphorof the desktop window
`and icons representing minimized application windows may
`be improved upon to provide a more elegant and efficient
`mechanism for interacting with a windowing system.
`
`SUMMARY
`
`An embodimentof the present invention is a method of
`managing windows. The method comprises receiving a
`signal indicating a gesture from a user, capturing pixels of a
`window, and applying the captured pixels as a texture to a
`display object locatedat a first location correspondingto the
`window in a three dimensional window, when the gesture
`indicates deactivating the window. The method further com-
`prises moving the display object to a second location in the
`three dimensional window and displaying the window over
`the display object when the gesture indicates activating the
`window.
`
`Another embodimentof the present invention is a method
`of displaying a representation of deactivating a window. The
`method comprises capturing pixels of the window and
`applying the captured pixels as a texture to a display object
`in a three dimensional window.
`
`Another embodimentof the present invention is a method
`of displaying a representation of activating a window. The
`method comprises moving and orienting a display object to
`a location in a three dimensional window,and displaying the
`window over the display object in the three dimensional
`window.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`The features and advantages of the present invention will
`become apparent from the following detailed description of
`the present invention in which:
`FIG. 1 is a diagram of an example display screen from a
`prior art windowing system;
`FIG. 2 is a diagram of a three dimensional (3D) graphics
`perspective projection frustrum with a 2D window aligned
`with the projection plane;
`FIG. 3 is a side view of a perspective viewing volume;
`FIG. 4 is a diagram of a first example of 2D windows
`texture-mapped onto window-shaped objects in a 3D space
`according to an embodimentof the present invention;
`FIG. 5 is a diagram of a second example of 2D windows
`texture-mapped onto window-shaped objects in a 3D space
`according to an embodimentof the present invention;
`FIG. 6 is a diagram illustrating a sample computer system
`suitable to be programmed according to an embodimentof
`a method for managing windows in 3D within a 2D win-
`dowing system in accordance with the present invention;
`FIG. 7 is a flow diagram of pushing a window back from
`2D space into a 3D space according to an embodimentof the
`present invention; and
`FIG. 8 is a flow diagram of bringing a window forward
`from the 3D space to 2D space according to an embodiment
`of the present invention.
`DETAILED DESCRIPTION
`
`10
`
`As the number of windowing application programsgrew,
`so did the number of windowslikely to be shown on a user’s
`screen at any given time. Icons, first developed as a way of
`launching applications, were later employed to represent
`running application program windows. These so-called
`“minimized” or “iconified” windows are temporarily
`replaced visually with very small (typically 32 by 32 pixels)
`images. FIG. 1 is an example diagram of a prior art win-
`dowing system showing the use of icons, a background or
`“desktop” window, and multiple overlapping windows. In
`current windowing systems such as WINDOWS95™, for
`example, commercially available from Microsoft
`Corporation, icons typically represent the application pro-
`gram rather than a documentorfile, but other systems may
`also use selected images generally as icons (called “picture
`icons” or “picons”).
`Useful as iconsare at simplifying the look of the computer
`In the following description, various aspects of the
`display,
`they have a disadvantage of being a discretely
`present
`invention will be described. For purposes of
`different representation of the windowsthey represent, with
`explanation, specific numbers, systems and configurations
`the cognitive mapping between the icons and the windows
`left to the user. In addition, when multiple windows are
`are set forth in order to provide a thorough understanding of
`
`shown onascreen, the user may have trouble managingtheir the present invention. However, it will also be apparent to
`
`50
`
`55
`
`60
`
`65
`
`10
`
`

`

`US 6,229,542 B1
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`4
`active. A bottom surface 58 may be represented in the 3D
`desktop window to provide a horizon line and to provide
`additional visual cues to the user for promoting the 3D
`illusion. The bottom surface optionally may include grid
`lines.
`
`3
`one skilled in the art that the present invention may be
`practiced without the specific details. In other instances, well
`known features are omitted or simplified in order not to
`obscure the present invention.
`An embodiment of the invention improves windowing
`FIG. 5 is a diagram of a second example of 2D windows
`systems by extending the representation of a window to
`include true depth (.e., three spatial dimensions as opposed
`texture-mapped onto window-shaped objects in a 3D space
`to two dimensions plus an overlapping order). This depth
`according to an embodimentof the present invention. From
`this vantage point,
`the illusion of a 3D image is more
`dimension addresses the window overcrowding problem by
`perceptible to the user. In one embodiment, false shadows
`allowing windowsto be positioned and arranged within a
`true three dimensional (3D) space. Because of “perspective
`60, 62 may be represented on the bottom surface 58 in the
`3D desktop window. This permits windowsto be selected by
`foreshadowing”, objects appear visually smaller when they
`their shadows even when the windows themselves may be
`are further away, and these smaller shapes occupy fewer
`obscured by other windows.
`pixels on the screen. In addition, 3D display objects can be
`rotated or otherwise viewed off-axis, which also occupyless
`Using three dimensions to manage 2D objects according
`space on the screen. FIG. 2 is a diagram of a 3D graphics
`to an embodimentof the present invention providesat least
`perspective projection frustrum with a 2D window aligned
`several advantages. The user is provided with more space on
`with the projection plane. When a window such as 2D
`his or her virtual desktop upon which to manipulate multiple
`window 10, for example, is shown in the foreground of the
`application program windows. Windows may be pushed
`3D space, it appears on projection plane 11. FIG. 3 is a side
`back into 3D space in full resolution when they are not
`view of a perspective viewing volume. When a window is
`currently being used. The user is provided with a better
`shown in the background of the 3D space, an embodiment
`context regarding inactive windows.Instead of displaying
`of the present invention provides the illusion to the user that
`small icons with a minimal amountof identifying informa-
`the window resides somewhere between the near plane 14
`tion or symbols as is currently done by graphical user
`and the far plane 16.
`interfaces such as is implemented in WINDOWS™, with
`embodiments of the present invention background applica-
`An embodiment of the present invention manages the
`transition between the 2D windowstate and the 3D window
`tion windowsretain their appearance in a 3D space. In an
`alternate embodimentof the present invention, the “pushed”
`state to exhibit a 3D representation of traditional 2D
`back windows may be implemented as being fully active
`windows, without modification of existing application pro-
`despite being in the background of the 3D desktop window.
`grams. More generally, an embodimentof the present inven-
`tion is a method for managing two dimensional (2D) win-
`Therefore, the ability to integrate true 3D objects into the
`dowsin a true three dimensional (3D) graphical space, all
`computer user’s workspaceis provided. In past systems, 3D
`objects were drawn in 2D windows. The present invention
`within a 2D windowing system. In an embodiment of the
`displays 2D objects in a 3D space, where other 3D objects
`present invention, the 3D space is maintained behind other
`may also reside. Anecdotal evidence suggests that users
`active 2D windows, much like a traditional 2D desktop
`prefer a 3D display environment over a complicated 2D
`image. In this way, 2D windows maybe “pushed back” by
`display environment having many overlapping windows.
`a user into the 3D desktop image by transforming the pixels
`The 3D desktop window ofthe present invention provides a
`of the 2D windowsonto similarly shaped 3D display objects
`more natural and intuitive graphical user interface than the
`in the 3D desktop image. When a 2D windowis brought to
`traditional 2D desktop interface.
`the foreground as a result of user interaction,
`the 2D
`window’s corresponding 3D display object in the 3D desk-
`FIG. 6 is a diagram illustrating a sample computer system
`top image may beoriented such that the actual 2D window
`suitable to be programmed according to an embodimentof
`is positioned over the top of the 3D display object, thereby
`a method for managing windows in 3D within a 2D win-
`giving the illusion that the 2D windowactually exists in the
`dowing system in accordance with the present invention.
`3D space. As used herein, the term “foreground” means a
`Sample system 100 may be used, for example, to execute the
`location appearing to the user to be in the front of the three
`processing for the methods described herein. Sample system
`dimensional viewing space onadisplay(that is, close to the
`100 is representative of computer systems based on the
`near plane), and the term “background” meansa location
`PENTIUM®, PENTIUM®Pro, and PENTIUM® II micro-
`appearing to the user to be towards the back of the three
`processors available from Intel Corporation, although other
`dimensional viewing space on the display (that is, farther
`systems (including personal computers (PCs) having other
`away from the near plane of the display in a direction away
`microprocessors, engineering workstations, set-top boxes
`from the user).
`and the like) may also be used. In one embodiment, sample
`FIG. 4 is a diagram of a first example of 2D windows
`system 100 may be executing a version of the WINDOWS™
`texture-mapped onto window-shaped objects in a 3D space
`operating system available from Microsoft Corporation,
`according to an embodimentof the present invention. Note
`although other operating systems and graphical user inter-
`that 3D graphics are rendered onto a 2D surface (i.e., a 3D
`faces may also be used. Sample system 100 includes micro-
`desktop window). This 2D surface may be used to replace
`processor 102 and cache memory 104 coupled to each other
`the traditional background desktop windowin graphical user
`through processor bus 105. Sample system 100 also includes
`interfaces (GUIs) such as is provided, for example, by the
`high performance I/O bus 108 and standard I/O bus 118.
`WINDOWS95™, WINDOWS98™, or WINDOWS NT®
`Processor bus 105 and high performance I/O bus 108 are
`operating systems commercially available from Microsoft
`bridged by host bridge 106, whereas high performance I/O
`Corporation. A window 50 is shown as being active by its
`bus 108 and standard I/O bus 118 are bridged by I/O bus
`representation in the foreground of the 3D desktop window
`bridge 110. Coupled to high performance I/O bus 108 are
`52. Other windows54, 56 are currently shownasinactive by
`main memory 112 and video memory 114. Coupled to video
`their representation in the background of the 3D desktop
`memory 114 is video display 116. Coupled to standard I/O
`window. These inactive windowsare represented by smaller
`bus 118 are mass storage 120, and keyboard and pointing
`images in the 3D desktop windowthan if they were currently
`devices 122. Mass storage 120 may comprise, for example,
`
`45
`
`50
`
`55
`
`60
`
`65
`
`11
`
`11
`
`

`

`US 6,229,542 B1
`
`5
`a hard disk drive, a floppy disk drive, a CD-ROM device, a
`flash memory device, or other storage device. Mass storage
`may comprise oneor a plurality of the described data storage
`devices.
`
`These elements perform their conventional functions
`well-knownin the art. In particular, mass storage 120 may
`be used to provide long-term storage for the executable
`instructions for embodiments of methods for managing
`windowsin accordance with the present invention, whereas
`main memory 112 is usedto store on a shorter term basis the
`executable instructions of embodiments of the methods for
`managing windowsin accordance with the present invention
`during execution by microprocessor 102. Window deactiva-
`tor 122 represents a set of instructions, which when executed
`by the microprocessor, deactivate a selected window. Win-
`dow activator 124 represents a set of instructions, which
`when executed by the microprocessor, activate a previously
`deactivated window.
`
`In conventional 2%D windowing systems having
`horizontal, vertical, and window overlapping parameters,
`the user is typically given a number of ways in which the
`user can direct the state transition of a window from “nor-
`mal” to “iconified” and back.
`In user-interface design
`parlance, these “gestures” might include double-clicking on
`an item, clicking an on-screen button,selecting an item from
`an on-screen menu, typing keystrokes, etc. Whatever the
`gesture, the window system (or the window manager, in
`more modular implementations such as the well known X
`Window System) may interpret
`this action as a state-
`transition event.
`
`In the “iconifying” case, the 2%D window system typi-
`cally “hides” the normal window (e.g., removes it in some
`way from the current display), optionally animates a shrink-
`ing outline of the window towards the icon position, and
`displays the icon on the “desktop” window. The desktop
`window is typically owned by the window system/window
`managerandexists specifically for this purpose. Application
`program icons are typically defined by the application
`program. The window system may also define icons. The
`window system also typically sends a message to the appli-
`cation program that owns the window notifying the appli-
`cation program of the state change.
`In the “restoring” case, the 2%.D window system typically
`hides the icon, optionally animates an expanding 2D outline
`of the window towards the window position, and displays
`the application program window. The window system also
`typically notifies the application program of the state
`change, and notifies the application program that its window
`needs to be redrawn.
`In some systems,
`the application
`program or window system may redraw the window con-
`tents from an off-screen bitmap image buffer that preserved
`the contents of the window before it was iconified or even
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`from an off-screen bitmap image buffer that allowed the
`application to draw to it even while the window was
`iconified.
`
`55
`
`In either the iconifying or restoring case, processing is as
`follows. The user or application program indicates its desire
`to changethe state of a window from “normal”to “iconified”
`or from “iconified” to “normal.” The window system (or, in
`some implementations,
`the window manager) effects this
`state change by hiding and showing the appropriate
`representations, optionally providing some cues (e.g.,
`animation) to visually link the two. The window system
`notifies the window owner(i.e., application program) of the
`state change. With most window systems,a transition from
`iconic to normal also brings about a notice to redraw the
`
`60
`
`65
`
`6
`window contents. One embodimentof the present invention
`is an extension to this processing model
`to provide 3D
`windows by implementation as an integral part of a window
`system and/or window manager, such as, for example, in a
`version of the WINDOWS™window system, although the
`invention is not limited in this respect.
`Traditional window systems maintain information for
`each window,including the window’s owner, position, size,
`style, and sometimes even the pixel contents. As mentioned
`above, top-level application windowsalso have an iconic
`alter ego that may also be a part of the window represen-
`tation. An embodimentof the present invention extendsthis
`state information for a window withat least five additional
`attributes. These attributes include a 3D shape or “geom-
`etry” model, the same or similar shape and relative size as
`the 2D window;the pixel contents of the 2D shape, applied
`to the 3D shape as a “texture”; a position in the 3D space;
`an orientation in the 3D space; and a transition state. In one
`embodiment,
`the transition state may be one of
`{inForeground,
`inBackground, movingToBackground,
`movingToForeground}. Like the relationship between icons
`and normal windows,the objects in an embodimentof the
`present invention are either in one state (e.g., 2D) or the
`other (e.g., 3D). The animating states “movingToBack-
`ground” and “movingToForeground” are transitory states
`that may be optional, but important, like the animating states
`for icons. When a window “isForeground”, it behaves as in
`traditional windowing systems. When a window
`“isBackground”, the window exists as an object in a 3D
`space, where it may be positioned, rotated, or otherwise
`manipulated, much like dragging icons arounda traditional
`2%D window system.
`As discussed above,traditional window systems are con-
`sidered to be 2% dimensional. That
`is, windows have
`horizontal and vertical dimensions, positions within a coor-
`dinate system, and a “stacking” order, which indicates their
`layering precedence. There are really two coordinate spaces
`available to the window owner, the screen space and the
`“client” space, whichis offset by the origin of the window
`within the screen space. In any case, 2%D windowsare like
`sheets of paper that are always parallel to the screen. In
`contrast, an embodiment of the present invention comple-
`ments the 24D space with a true, three-dimensional space
`that effectively replaces the flat 2D “desktop” window and
`2D icons. Whereas 2%D iconsare arranged on top ofa flat,
`2D desktop window,an embodimentof the present invention
`provides an environment where 3D windowrepresentations
`may be arranged within a 3D desktop space.
`When 3D windows according to an embodiment of the
`present invention are in the “foreground,” they obeyall the
`rules of the 24D window system, including position and
`stacking order. Windows in the “background”are true 3D
`objects and obeyall the rules of a 3D space, as practiced in
`the art of 3D graphics.
`Note that “true 3D space” is actually a mathematically
`true space. The present invention doesnot give real, physical
`depth to a screen of a computer monitor. In well known 3D
`computer graphics, 3D geometric models are projected onto
`the flat screen similar to the waythat light is projected onto
`camera film. This results in a two-dimensional image of a
`three-dimensional scene being represented on the monitor.
`An embodiment of the present invention provides flat 2D
`windows with 3D characteristics by creating 3D models out
`of 2D windows, and seamlessly joining the 2D and 3D
`environments to provide the illusion that the 2D window is
`really a 3D object in a 3D world.
`FIG. 7 is a flow diagram of pushing a window back from
`2D space into a 3D space according to an embodimentof the
`
`12
`
`12
`
`

`

`US 6,229,542 B1
`
`7
`present invention. At block 200, the user gestures to “push
`back”, deactivate, or minimize a currently active window by
`sending at
`least one input signal
`to the graphical user
`interface. This action may comprise one or more keyboard
`entries (such as an ALT-TAB key combination, for example),
`or one or more mouse clicks, or any other user actions
`recognized by the system, which are interpreted by the GUI
`as input signals to deactivate or close a selected window. At
`block 202, a “snapshot” of the selected window is taken.
`That is, the pixels making up the selected window to be
`pushed backare captured in a data structure by the graphical
`user interface. At block 204, the snapshot data as a texture
`is applied to a display object in the 3D desktop window
`having the same shapeas the selected window.At block 206,
`the 3D display object is animated to its new place in the 3D
`desktop window corresponding to the deactivated 2D win-
`dow. The new location in the 3D desktop window maybe in
`the background of the 3D space, thereby indicating that the
`selected window is inactive.
`
`FIG. 8 is a flow diagram of bringing a window forward
`from the 3D space to 2D space according to an embodiment
`of the present invention. At block 220, the user gestures to
`bring a selected windowto the foreground of the 3D desktop
`window,by sending at least one input signal to the graphical
`user interface. This action may comprise one or more
`keyboard entries (such as an ALT-TAB key combination, for
`example), or one or more mouse clicks, or any other user
`actions recognized by the system, which are interpreted by
`the GUI as input signals to activate or bring a selected
`window to the foreground of the 3D desktop window. At
`block 222, the 3D display object representing the selected
`window is moved and oriented to appear in the foreground
`or front of the 3D desktop window. At block 224, a 2D
`window representing the selected window is showndirectly
`over the 3D display object. At block 226, the 3D display
`object for the selected window is hidden from current
`display on the 3D desktop window.
`In one embodiment, the “push to back” operation may be
`defined as a “SendToBack”or “iconify” method comprising
`the following operations. If the 3D display object (ie.,
`geometry) for a selected 2D window does not currently
`exist, then a flat rectangular 3D object(e.g., a “face”) may
`be created in the same shape and relative size as the 2D
`window, and the background position and rotation may be
`initialized to nominal values. Next, the current pixel con-
`tents of the 2D window may be saved to an off-screen
`bitmap image buffer (known as a “bitblt” operation). The
`saved pixel contents may be saved as a texture onto the 3D
`geometry, optionally scaling the image down or otherwise
`creating alternate scale images (e.g. “mipmapping”) as is
`well-known to those skilled in the current 3D graphicsart.
`The current, “foreground”position of the 2D window in
`3D space may then be computed and stored. That is, the
`position and rotation of the alter ego 3D display object may
`be computed such that the 3D object is oriented parallel to
`the screen (and thus,
`the 2D window) and is positioned
`directly underneath the 2D window at a distance that pre-
`sents the 3D object at exactly the same size as the 2D
`window. This may be accomplished by the following opera-
`tions. The 2D screen-space coordinates (origin and extents)
`of the 2D window may be obtained. These coordinates may
`be translated into the client-space coordinates of the 3D
`window. The 3D object may be resized to match the aspect
`ratio (width/height) and relative size of the 2D window. The
`rotation of the 3D object may be set to be perpendicular to
`the 3D virtual camera’s line of sight, with the front of the
`object facing the virtual camera. The position of the 3D
`
`10
`
`15
`
`20
`
`25
`
`40
`
`45
`
`55
`
`60
`
`65
`
`8
`object may be suchthat its distance from the camera effects
`a visual shape that is the same size as the 2D window. This
`is solved using well-known trigonometry as practiced in the
`art of 3D graphics programming. The parameters of the 3D
`perspective frustum (as shown in FIG. 2) specify a viewing
`angle (called the fovy, the angle from the top of the frustum
`to the bottom) and aspect ratio (width/height). The frustum
`may be bisected top-to-bottom to produce right triangles
`with respect to the Y- and the Z-axis. Basic trigonometry
`enables one to compute the required distance of the 3D
`display object: taking the trivial case of the object centered
`around the line of sight, the tangent of “fovy=% object_
`height/distance, wherein tan(theta)=opposite/adjacent. The
`camera-relative XY placementof the 3D display object may
`be adjusted to affect
`the placement of the 2D window
`relative to the 2D screen.
`
`With the 3D display object now positioned, oriented, and
`sized in a similar manner to its 2D counterpart,
`the 2D
`window may now behidden from view, leaving a visually
`equivalent 3D representation where the 2D window used to
`be. In some embodiments, the 2D window may actually be
`“hidden” or otherwise removed from the list of actively
`displayed windows. In other embodiments, the 2D window
`may be hidden from view by allowing the 3D graphics
`window, usually sized to the full extent of the screen, to be
`placed before the 2D window in the stacking order, thus
`obscuring the 2D window.
`The 3D object may be animated to its 3D position and
`rotation using methods well known in the 3D graphics
`programmingart. Typically, this involves adjusting the posi-
`tion and rotation in small increments over time. In one
`embodiment of the present invention, a timer may be used
`to affect the animation changes at a rate of approximately
`10-30 frames per second, for example. At each frame, the
`current time may be examined to determine the percentage
`of time that has elapsed between the start of the animation
`and the end of animation. This percentage may be used to
`interpolate, either linearly or according to some non-linear
`function (e.g., slow-in/slow-out) the current, instantaneous
`position and rotation of the 3D display object. The resultant
`values may then be set on the 3D display object, according
`to methods well known in the art.
`
`In one embodiment, the restore to front operation may be
`implemented as a “BringToFront”or “restore” method. The
`3D display object may be animated to the foreground
`position. The 2D window may be shownatits previous size
`and location. In embodiments where the window wasactu-
`
`ally hidden whensent to the background, the window may
`nowberestored to its normal state. In one embodiment, the
`stacking order of the window may be adjusted to putit at the
`top, in front of the 3D desktop window. The 3D display
`object may then be hidden, according to methods well
`known in the 3D graphics programmingart.
`In some 2D window systems it may be possible and
`desirable to actually replace the conventional 2D desktop
`window with the 3D window of an embodiment of the
`
`present invention. In this case, application program windows
`will need to be “hidden”and “shown”, or otherwise removed
`from and addedto the operating system’s active windowlist.
`In an embodiment where the 3D desktop window behaves as
`other appl