United States Patent 55
`5,880,733
`[11] Patent Number:
`[45] Date of Patent:
`Mar. 9, 1999
`Horvitz et al.
`
`US005880733A
`
`[54] DISPLAY SYSTEM AND METHOD FOR
`DISPLAYING WINDOWSOF AN
`OPERATING SYSTEM TO PROVIDE A
`THREE-DIMENSIONAL WORKSPACE FOR A
`COMPUTER SYSTEM
`
`[75]
`
`Inventors: Eric J. Horvitz; Martin L. Sonntag,
`both of Kirkland; Michael E. Markley,
`Redmond, all of Wash.
`
`[73] Assignee: Microsoft Corporation, Redmond,
`Wash.
`
`[21] Appl. No.: 649,828
`
`[22]
`
`Filed:
`
`Apr. 30, 1996
`
`Int. Choeee G06F 3/00; GO6T 15/20
`[51]
`[52] U.S. Ch. ce ceeeeeeee 345/355; 345/340; 345/427;
`345/343; 345/978
`[58] Field of Search occ 345/355, 352,
`345/340, 351, 339, 326, 349, 341, 327,
`328, 329, 330, 331, 332, 333, 334, 335,
`336, 337, 338, 354, 353, 342, 343, 344,
`345, 346, 347, 348
`
`[56]
`
`References Cited
`U.S. PATENT DOCUMENTS
`
`5,670,984
`5,671,381
`5,724,492
`5,786,820
`5,790,713
`5,801,704
`
`9/1997 Robertson et al. oe eee 345/139
`.. 345/355
`9/1997 Strasnick et al.
`........
`
`3/1998 Matthews,II et al.
`......
`... 395/119
`
`7/1998 Robertson «0...
`w 345/357
`
`.. 382/285
`.
`8/1998 Kamada et al.
`9/1998 Oohara et al. oes 345/348
`
`OTHER PUBLICATIONS
`
`“The Perspective Wall: Detail and Context Smoothly Inte-
`grated”, Jock D. Mackinlay, George G. Robertson, and
`Stuart K. Card, Association for Computing Machinery,
`1991, pp. 173-179.
`“The Information Visualizer, An Information Workspace”,
`Stuart K. Card, George G. Robertson, and Jock D. Mack-
`inlay, Association for Computing Machinery, 1991, pp.
`181-188.
`
`“Cone Trees: Animated 3D Visualizations of Hierarchical
`
`Information”, George G. Robertson, Jock D. Mackinlay, and
`Stuart K. Card, Association for Computing Machinery,
`1991, pp. 189-194.
`
`“Information Visualization Using 3D Interactive Anima-
`tion”, George G. Robertson, Jock D. Mackinlay, and Stuart
`K. Card, Communications of the ACM, Apr. 1993, vol. 36,
`No. 4, pp. 57-71.
`
`“Rooms: The Use of Multiple Virtual Workspaces to Reduce
`Space Contention in a Window—Based Graphical User Inter-
`face”, D. Austin Henderson, Jr. and Stuart K. CardACM
`Transactions on Graphics, Jul. 1986, vol. 5, No. 3, pp.
`210-243.
`
`Primary Examiner—Raymond J. Bayerl
`Assistant Examiner—Steve Sax
`
`Attorney, Agent, or Firm—Jones & Askew, LLP
`
`[57]
`
`ABSTRACT
`
`invention provides a three-dimensional
`The present
`perspective, virtual workspace to window based display
`systems. The user may activate control buttons to transform
`a typical two-dimensional window to a three-dimensional
`perspective, virtual workspace. The transformation to the
`three-dimensional workspace provides a perspective or
`sense of depth to the display of the windows containing
`computer applications. In this manner, a simulated three-
`dimensional window display environmentfor the operating
`system is provided. The operating system utilized in con-
`nection with the present invention receives data for storage
`in a destination rectangle in a frame buffer. The data desig-
`nated for storage in a destination rectangle is generally given
`in pixel coordinates. The operating system scales the data’s
`dimensions to the provide a three-dimensional perspective
`appearanceon the display screen. Whena user indicates that
`a window should be displayed in a particular perspective or
`size, coordinates representing the window and data con-
`tained therein are multiplied by a transformation matrix to
`yield the specified perspective or size.
`
`42 Claims, 22 Drawing Sheets
`
`
`56
`
`
`
`
`
`
`
`ae?
`ebIG2
`
`eS126
`
`
`ayz)=65,37.33. +{
`438 17
`
`
`
`
`
`
`
`
`
`1
`
`APPLE 1014
`
`APPLE 1014
`
`1
`
`

`

`U.S. Patent
`
`Mar. 9, 1999
`
`Sheet 1 of 22
`
`5,880,733
`
`
`
`NNNN
`
`y_—(\DD
`
`
`
`FIG.1
`
`2
`
`

`

`U.S. Patent
`
`Mar. 9, 1999
`
`Sheet 2 of 22
`
`5,880,733
`
`
`
`CENTRAL
`PROCESSING
`
`UNIT
`
`
`PORTS
`
`INPUT / OUTPUT
`
`FIG.2
`
`3
`
`

`

`U.S. Patent
`
`Mar. 9, 1999
`
`Sheet 3 of 22
`
`5,880,733
`
`Ehjanu00
`
`errsubAK¢'68=[zal
`
`Ay}4feue'so=[24%a
`
`000
`
`oSSI
`
`86
`
`86Qe
`
`9g
`
`bodood
`
`JOMOIAO21TENslA
`
`EYyosoy
`
`SMOPUIM,
`
`suoneoiddy
`
`gaa
`
`ooo
`
`YOSOIIW
`
`oo]
`
`y
`
`50
`a
`a0
`
`“
`2a
`
`oqag
`oa
`
`gaa
`
`ooo
`
`SMODUIM,
`
`og
`aop
`NS
`ao]:
`=
`
`RscO
`
`2
`39
`ac 3
`a0] $
`
`suoneoddy
`
`85a So
`
`
`Jobeueyy|
`
`
`£°9ls
`
`89be
`
`
`
`4
`
`

`

`U.S. Patent
`
`Mar. 9, 1999
`
`Sheet 4 of 22
`
`5,880,733
`
` SY
`
`14
`
`SSs‘““
`
`15
`
`
`CZsSRERSSSSSASASASS
`
`
`SSSIAg00EESEASESESESESSSEESSSS
`
`
` REESEEEREAAESHUESSEASSSSUA
`
`
`
` WcxzzZLZ£Z2ZKEZEEE
`iy,TREN0r222ww|TdoQ_QAOAVA>—s_—qQa\Q<’/’_e
`
`LeLLLLhe
`
`CZs
`
`Lok.
`
`FIG.4b
`
`5
`
`

`

`U.S. Patent
`
`Mar. 9, 1999
`
`Sheet 5 of 22
`
`5,880,733
`
`14
`12
`OccLELLttdtcddlibdsidiiliddjsuisdilddiisigidisdidlisjgsllilsildss
`
`
`SEASSSNSSOSUNSSSRSSSNSSNSSSOTTTTTI
` Sy
`
` SSRRSSERENEOSAAAASUSSSASSSUSSSASSCLLAO
`
`
`15
`
`
`
`
`FIG.4c
`
`6
`
`

`

`U.S. Patent
`
`Mar. 9, 1999
`
`Sheet 6 of 22
`
`5,880,733
`
`
`
`
`
`
`
` GLCUMLIUTLLULAAMAIATALTLLULOLLYL
`SESSSESESESSSSESSROSESSSSESSESEASSCSSOSNS
`ESSSESELei
`
`SSSSASSEOSSSUSSSHASSSSSNSUUASESSSSASSNSSSNSSUASSSSSSSSNSSNSSNNSN
` SoSONARANANTH
`
`
`AHAAAAAAAATAAAAAAAAEEEE
`
`|
`rrr
`
`ALDIMUALYLLLOUDELGEDALTTULLLLDLLLLLULUYIDIULTLDUULLELLLILDLLLLLLLTUULLTLELITTLLLLLULL
`
`ULTLULIUhySS
`
`
`
`
`
`dddiiiddiigitidddiduipiillllcdelcdgldddsddicijddildédiliddlMllsdlddldsililg
`
`FIG.3a
`
`ViklliMlditillliLlMiiiMlligibIiliLiidiiiiiiisiiihiMiitiiitiditiigiiiiditiegg
`
`15
`
`WH
`
`
`
`
`ASSESSSSS
`
`FIG.7b
`
`7
`
`

`

`U.S. Patent
`
`Mar. 9, 1999
`
`Sheet 7 of 22
`
`5,880,733
`
`15
`
`YihLLLLLMbahedALLLedLLL
`
`
`
` SSSEEEESSERSEEESESSSNAEEEESSSSESSA
`
`
`eSAERSRSAMASSERENENSSONSLddddddlddddddlddlALLLLcLALLLLLLLLLLESS
`
`
`FIG.37c
`
`8
`
`

`

`U.S. Patent
`
`Mar. 9, 1999
`
`Sheet 8 of 22
`
`5,880,733
`
`15
`
`ldlLLddd
`LLLaLALLLLEILLL
`
`106
`
`FiG.6a
`
`15
`
`AMALLLcLLLLLLLLLLLLLLdllaldSSSSSsV
`
`yVA
`
`yyAZZyyyyUyyyyyyYY}yZyyyyyyy
`
`
`
`SSSsy REEAN
`ESSSSSSSS
`
`REELSDEELSASSUSSEAESOSSSASESSSSS LLLELLLEELLcelLLLLLLLbdddddddddlidLLLILLE
`
` AAANNMMAAA_-_|Q|—K
`
`)
`y)
`
`7yyyA
`
`]yyY2r
`
`
`
`
`
`xxxxnoLlcdddddddddddadidddddgiLillis
`
`0°
`
`FIG.6b
`
`9
`
`

`

`U.S. Patent
`
`Mar. 9, 1999
`
`Sheet 9 of 22
`
`5,880,733
`
`SS
`
`MLiiddlsiliit¢isisiaslsdda LLLLLLblsiilidiliidss ESSS
`
`E’&&MK&rsSSSSSSSSESSSSSSAG
`WSSREEEAEEESASSSHSASUASOO
`SSESESSSA
`RSSSESS
`
`
`
`
`LidddiddldiididilidisJiilidsdildesZL
`Middlibddidiiidiilisids
`LLLLLL
`
`15
`
`106
`
`FIG.6c
`
`10
`
`10
`
`

`

`Mar. 9, 1999
`
`Sheet 10 of 22
`
`5,880,733
`
`SSS
`Zy
`Zy)
`y)y)
`
`%%yy
`
`)gy}
`y}
`y}
`ZZ
`yy
`Uy}
`
`ZZy
`
`}ZZZZy
`
`U.S. Patent
`
`SSE
`
`SSS
`
`SSH
`SONNE LILLIE09
`
`
`}
`y}
`
`gyZgy)Uy}y%
`
`y}S797
`
`FIG.J7a
`
`PTTLELLIE:
`
`FIG.7b
`
`11
`
`11
`
`

`

`U.S. Patent
`
`Mar. 9, 1999
`
`Sheet 11 of 22
`
`EfV7OO
`
`5,880,733
`
`FIG.7d
`
`12
`
`12
`
`

`

`U.S. Patent
`
`Mar. 9, 1999
`
`Sheet 12 of 22
`
`5,880,733
`
`v
`
`LLLLLCLAgEELLLLLLLLhLLUldbMLL
`
`KgIII23
`7
`
`FIG.8b
`
`13
`
`13
`
`

`

`U.S. Patent
`
`Mar. 9, 1999
`
`Sheet 13 of 22
`
`5,880,733
`
`VV
`
`|pLALMAMLELEALLLLE
`
`FIG.8d
`
`14
`
`14
`
`

`

`OLcbdddddisdddETLLAMALLLLLLddLia:
`%
`yA
`
` SSSETSSEESVVyg
`
`LA
`
`yy
`yyy
`y
`
`VfLALLA
`
`U.S. Patent
`
`Mar. 9, 1999
`
`Sheet 14 of 22
`
`5,880,733
`
`
`
`GLLLLLLLadESSETbh
`
`15
`
`

`

`U.S. Patent
`
`Mar. 9, 1999
`
`Sheet 15 of 22
`
` (LLLadicdllccclaacacKCCZK
`
`
`38
`
`
`
`106
`
`FIG.10a
`
`
`
`LALLALEELALEALELLableildlALLALALLLLLLddlddddLedddddddadddicadddiiie)
`
`5,880,733
`
`SS SXSSSSS
`SESHSEGASASSER LLLETZTE:
`
` [imikimi
`
`SURESSOS
`SSS
`
`710
`
`LBMILLA
`
`
`EKdialiliimglitiitiEy,ididdddiiddidsiliiilileLidileicsiica
`
`06
`
`FIG.10b
`
`16
`
`16
`
`

`

`U.S. Patent
`
`Mar. 9, 1999
`
`Sheet 16 of 22
`
`5,880,733
`
`dLLLLLLLLLLdLLLlita
`SN
`
`
`
`
`Yj
`
`
`
`
`
`EOS
`
`
`
`
`
`120
`
`
`
`
`
`1110
`
`712
`
`56
`
`84
`
`
`
`50
`
`88
`
`FIG.11
`
`17
`
`17
`
`

`

`U.S. Patent
`
`Mar. 9, 1999
`
`Sheet 17 of 22
`
`FIG.12
`
`5,880,733
`
`18
`
`18
`
`

`

`U.S. Patent
`
`5,880,733
`
`<L°Ols SAOVdS
`OIWLAWOSIAILINWONISSAOOVGNVONINIVINIVA
`
`19
`
`

`

`
`
`jj
`
`PLO
`
`20
`
`
`

`

`e
`
`&
`
`to
`
`II. \ WE
`Hoe
`
`
`
`
`
`OQNIZISYOSHNOOINVNAG
`
`he
`
`(QOLSL
`
`BYMMMMYYJY
`
`
`LE:LESLWi
`
`a.
`GS—F
`
`4l°Old
`
`Y/GL
`fita7,
`
`(2)0LGL7
`PLExZe|OLE
`
`A
`
`21
`
`
`
`
`
`
`
`

`

`U.S. Patent
`
`Mar. 9, 1999
`
`Sheet 21 of 22
`
`5,880,733
`
`/ 1602
`
`POWER UP
`COMPUTER
`
`
`
`BOOT OPERATING
`
`ON FRONT PANEL
`DISPLAYING APPLICATION
`DATA
`
`
`1612
`
`AWAIT USER ACTION
`
`1604
`SYSTEM PROVIDE WINDOW
`
`
`
`1626
`
`
`
`UPDATE
`DISPLAY
`
`
`
`TRANSFORMATION
`CONTROL SELECTED
`
`?
`
` PERFORM APPROPRIATE
`
`
`
`
`WINDOW
`TRANSFORMATION
`
`FIG.16a
`
`22
`
`22
`
`

`

`U.S. Patent
`
`Mar. 9, 1999
`
`Sheet 22 of 22
`
`5,880,733
`
`1624
`
`
`
`PERFORM APPROPRIATE
`
`WINDOW
`
`PR
`
`PANELIZE LEFT, OR
`PANELIZE RIGHT
`?
`
`1712
`
`.
`
`1716
`
`IMPLEMENT LEFT
`TRANSFORMATION
`MATRIX
`
`
`
`IMPLEMENT RIGHT
`TRANSFORMATION
`MATRIX
`
`
`TRANSFORMATION
` ZOOM,
`PL
`
`
`
`
`
`IMPLEMENT ZOOM OUT
`OR ZOOM IN
`TRANSFORMATION MATRIX
`
`STORE TRANSFORMED
`DATA TO DISPLAY
`BUFFER
`
`
`
`
`FIG.1 6b
`
`23
`
`23
`
`

`

`5,880,733
`
`1
`DISPLAY SYSTEM AND METHOD FOR
`DISPLAYING WINDOWSOF AN
`OPERATING SYSTEM TO PROVIDE A
`THREE-DIMENSIONAL WORKSPACE FOR A
`COMPUTER SYSTEM
`
`FIELD OF THE INVENTION
`
`The present invention relates generally to a display sys-
`tem for managing and manipulating information on a com-
`puter display screen operating in a window display
`environment, and more particularly, relates to a display
`system for managing and manipulating the displayed win-
`dows of the operating system to provide a virtual three-
`dimensional workspace for the computer system.
`BACKGROUND OF THE INVENTION
`
`In a physical environment, such as on a deskortable, an
`individual may utilize certain areas for placement of docu-
`ments for convenient viewing. The surface area on the desk
`used to support the documents for viewing may be termed a
`workspace. Because the physical space on a desk for work-
`ing is limited multiple documents may be spread out or
`arranged in some systematic manner within the workspace.
`When multiple documents require more surface area than
`the available area of the workspace, documents may be
`stacked, within the workspace, with identifying portions
`being visible.
`In a computer system, the workspace for viewing docu-
`ments or applicationsis the display screen or monitor of the
`computer system. The operating system of the computer
`system generally manages or controls the documents or
`applications displayed on the monitor. Early computer sys-
`tems displayed only one document or application on the
`display screen at a time, therefore limiting the effective
`computer workspace. In order to switch among multiple
`applications or documents, the user had to close the appli-
`cation or document being displayed and open a different
`application or document. Such a computer system was
`cumbersome. With the introduction of multi-tasking and
`graphical interfaces,
`the display screen was divided into
`multiple portions generally referred to as windows. A win-
`dowis a portion of the display screen that may contain its
`own document or message.
`In window-based programs,the screen can be divided into
`several windows, each of which has its own boundaries and
`can contain a different document. The operating system
`controls these specially delineated window areas of the
`screen. Each window can act independently,as if it were a
`virtual display device. In window display environments,the
`windows may be resized and moved around onthe display
`screen. Thus, with the advent of window-based programs,
`the computer screen workspace became more flexible. The
`ability to resize and move windowsas well as to overlay or
`stack windowson top of each other essentially provided the
`same type of workspace available on a physical
`two-
`dimensional tabletop workspace except on a smaller scale.
`Present day computer systems often employ “object-
`oriented” displays with windows,icons, pictures, text, pop-
`up or drop-down menus, dialog boxes, and other graphical
`items or objects in what is known as a “Graphical User
`Interface” (“GUI”). In such systems, various combinations
`of graphical items or objects are utilized to convey infor-
`mation to users via the monitor or display. Users interact
`with the computer system by activating graphical items or
`objects on the display. The graphical items or objects are
`often activated by pointing at them with a cursor controlled
`by a mouse or pen and clicking a control button on the
`mouse or pen.
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`2
`An example of a computer graphical user interface that
`uses icons and windowsis the Microsoft “WINDOWS”™
`operating system manufactured and sold by the assignee of
`the present invention, Microsoft Corporation. Common user
`interfaces such as those found in the Microsoft “WIN-
`DOWS?”operating system and other GUI-based computer
`systemstypically rely on a keyboard and a pointing device
`with buttons such as a mouseor trackball for user interac-
`tion. The user points to a location on the screen where an
`action is desired, and invokes a command bypressing a
`button on the pointing device or entering a keyboard com-
`mand.
`
`Just as a physical desktop table can become cluttered
`when multiple documents are being viewed, a computer
`screen workspace can becomecluttered when multiple docu-
`ments are displayed in multiple windows. With limited
`working space or viewing area on a computer screen, the
`computer screen workspace can quickly becomecluttered if
`several applications or windows are opened at the same
`time. A numberof techniques have been used for overcom-
`ing the small-screen problem presented when using com-
`puter displays.
`Four general categories of computer screen workspace
`enhancement techniquesare: (1) alternating screen uses, (2)
`distorted views, (3) large virtual workspaces, and (4) mul-
`tiple virtual workspaces. Alternating-screen use is similar to
`windowsin that the user can switch or changethe allocation
`of screen space from one application to another. Some early
`alternating screens use techniques only allowed one appli-
`cation to be visible on a screen at a time. With distorted
`views, documents or applications appearing on the screen
`may be distorted to small
`icons. Here, applications are
`reduced to small pictures or icons that remind the userof the
`original window. Overlapping or stacked windows mayalso
`be considered a distorting technique. As noted above, when
`many windowsoverlap the appearance on a display screen
`may becomecluttered.
`the
`the set of objects,
`With large virtual workspaces,
`documents, or applications are organized as a large virtual
`workspace which cannotall be viewed on a display screen
`simultaneously.
`In one example application,
`the data is
`arranged in two dimensions, and the user has three screens,
`one for an overview of the whole space, one for a detailed
`view of someportion of the space, and one touch screen for
`control. Head-mounted displays, which monitor user head
`and body movements to give the user a complete simulated
`three-dimensional space view, are forms of large virtual
`workspaces. A problem with a single large workspaceis that
`only a limited number of things can be adjacent to any
`object. The space required for the objects and their shapes
`constrains the manner in which the space may be arranged
`and how densely packed documents or data can be pre-
`sented. Multiple virtual workspaces provide geometrically-
`oriented workspaceslinked together in which a project may
`contain a number of views and, when active, covers the
`entire screen. For example, when a user maneuvers close
`enough to a port, the user is swept into another workspace
`or subworkspace.
`The above techniques are examples of display systems
`which try to overcome the limited workspace inherent with
`computers that display multiple applications or documents
`on a two-dimensional display screen. However, these dis-
`play systems do not provide a method for displaying mul-
`tiple applications presented by an operating system in three
`dimensions. Thus, there is a need in the art for a display
`system which provides an enhanced system for displaying
`multiple applications or windows as simulated three-
`dimensional objects on a computer display screen.
`
`24
`
`24
`
`

`

`5,880,733
`
`3
`SUMMARYOF THE INVENTION
`
`invention provides a
`the present
`Generally described,
`three-dimensional virtual workspace for a window based
`display system. The display system of the present invention
`is an isometric display system for an operating system. The
`isometric display system provides a display with monocular
`depth cues by making automatic sizing and geometric trans-
`formations on two dimensional rectangles that define tradi-
`tional windows.
`
`The isometric display system performs geometric trans-
`formation operations on rectangular windowsto convey the
`impression to the user that the windowsare positioned in a
`three-dimensional space. The impression of depth and of
`working in a three-dimensional space are created by trans-
`forming two-dimensional windowsto appearas if the two-
`dimensional windows are embedded at orientations and
`
`positions in a three-dimensional isomeric space. The iso-
`metric display system respondsto user activation of control
`buttons by transforming the shape and orientation of win-
`dows consistent with the defined three-dimensional space.
`More particularly described, the present invention pro-
`vides a methodofutilizing left, right, front, back, upper, and
`lower planes. These planes define a polyhedron and these
`planes may bereferred to as “boundary planes”. The poly-
`hedron surrounds a three-dimensional space used by the
`isometric display system to manage the configuration and
`position of windowscontaining data from applications. The
`boundary planes of the virtual space are preferably orga-
`nized into a rectangular prism defining left, nght, front, and
`back walls, a ceiling, and a floor of a four-sided room as seen
`when viewing the room from the outside through a selected
`wall or user viewpoint.
`Boundary planes may be explicitly displayed with shaded
`solid surfaces or simply as lines defining the intersection of
`planes. However, the planes are preferably not visible. The
`front boundary plane is typically assumed to be the front
`surface of the display, closer to the user than the rear
`boundary plane. When the boundary planes are made
`visible, the left and right walls, and the ceiling and floor
`defined by the planes narrow in dimension as viewed from
`the front to back of the space defined on the display screen.
`Windows may beoriented and displayed in the boundary
`planes.
`Windowsoriented in the front boundary plane are dis-
`played without transformation, and appear as standard rect-
`angular window,asin traditional windowing systems. When
`a transformation command is entered by a user or from a
`program, a window is transformed and movedto the selected
`plane of the display system to create the perception of depth.
`A windowselected for display on the rear plane appears as
`a rectangular window that has been pushed back in the
`three-dimensional space. A window designated for another
`plane is rotated 90 degrees and appears pushedflat against
`the designated plane.
`For generating the illusion of moving the windows away
`from the user in the three-dimensional space, the display
`system decreases the size of the window and its contents
`equivalently in the x and y dimensions,as a function of its
`assigned distance from the user in the three-dimensional
`space. The display system moves the window towards the
`center of the space based on the relative distance between
`the front and the rear boundary planes. When a window on
`the front plane is moved toward the rear plane, the window
`can snap immediately to the rear plane and appear in a
`reduced size or the window may move incrementally to an
`intermediate plane or planes between the front and rear
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`4
`plane. Additionally, a window moved toward the rear plane
`may continuously decrease in size as the window
`approachesthe rear plane.
`Useror system accessible controls are used for displaying
`or hanging an untransformed window ona left, nght, rear,
`upper, or lower boundary plane. For example, when a user
`invokes the “perspective-transform right” command,
`the
`display system applies a perspective transform on the rect-
`angular window and its contents and redisplays the rectan-
`gular window as an appropriately sized and shaped trap-
`ezoid. The window as displayed appears to be rotated 90
`degrees form its initial position and appears to be reposi-
`tionedto fit within the right boundary plane. When a window
`is moved along or to a boundary plane, the window may be
`continuously resized based on its position in the three-
`dimensional space or the window may snap to predefined
`locations.
`Whenthe intended destination of a selected window is a
`
`boundary plane already containing a transformed window,
`the selected window is placed on an adjacent plane to the
`boundary plane that is closest to the boundary plane. Adja-
`cent planes refer to planes that are parallel (in the three-
`dimensional space) to the boundary plane but
`that are
`positioned at selected increments away from the boundary
`plane along the normal vector from the plane towards the
`center of the virtual space. Windowsselected for display on
`a designated plane are displayed on the closest available
`adjacent plane. This provides an ordering of displayed
`windows from the center of the three-dimensional space to
`the boundary plane for all transformed windows on the
`boundary plane. The contents of displayed windows maybe
`generally viewed in full when the windows are hung on
`different planes. Also, the windows may have focus or be
`operated on when displayed or hung on a plane.
`Three primary controls may be used to position a window
`hung on a plane: push back, perspective-transform right, and
`perspective-transform left. Push back causes a window to be
`transformed or pushed to the foreground or background of
`the display system. Perspective-transform right generally
`causes a window to be displayed on the right plane and
`perspective-transform left generally causes a window to be
`displayed on the left plane. Alternatively, a single transfor-
`mation control for transforming windows to the various
`planes may be implemented. Uponselection of the single
`transformation control, the display system provides a pointer
`to a selected plane. The pointer may be operative to point to
`a plurality of predefined planes. The pointer may be a
`direction arrow. Generally, the pointer is provided when the
`cursor control is activated or pulled in the direction of a
`predefined plane while the cursor is positioned on the
`transformation control. The user may release the cursor
`control
`to initiate the transfer or transformation of the
`window.
`
`The operating system utilized in connection with the
`present invention receives data from an operating system, an
`application or other source for display in a window. The data
`may bestored by the operating system in the same manner
`as in rectangular windows. Before displaying the data in a
`window on the display screen, the operating system scales
`the shape of the windowtofit the selected plane and the data
`presented in the window.The scaling of the window’s shape
`and the scaling of the data presented in pixel coordinates
`within the window creates a three-dimensional appearance
`on the display screen.
`Transformation matrices may be used in the operating
`system to transform or scale a windowto the selected plane.
`
`25
`
`25
`
`

`

`5,880,733
`
`5
`Whena user indicates that a window should be displayed on
`a particular plane, both the coordinates of the window and
`data contained in the window are multiplied by a transfor-
`mation matrix to fit the window in the area defined by the
`selected plane and transform the data content of the window.
`By transforming the window anddata, depth perception for
`the windowis created.
`
`The present invention may also provide a method of
`dynamically sizing the cursor as a function of the location of
`the cursor on the display screen. The present invention may
`determine the position of the cursor on the display screen,
`and resize the cursor based upon the location of the cursor
`on the display screen. The size of the cursor may change
`whenthe cursorcrosses over a border of a window. Thesize
`of the cursor may specifically change relative to the cursor
`location on a window to reflect the depth of that location in
`three-dimensional space defined by the transformed win-
`dows. This serves to give feedback to the user about the
`location of the cursor in three-dimensional space by gener-
`ating the perception that the cursor resides on or jumps to
`planes defined by the transformed windows.
`Thus,it is an object of the present invention to provide an
`enhanced display system for displaying applications and
`documents in a computer system.
`It is another object of the present invention to manipulate
`the displayed windowsto create a virtual three-dimensional
`workspace on the monitor of the computer system.
`It is another object of the present invention to provide a
`display system for simultaneously displaying multiple appli-
`cations and objects managed by the operating system as a
`consistent three-dimensional operating environment.
`It is another object of the present invention to transform
`the shapes and contents of a displayed window to create a
`perspective view of the window andits contents.
`It is another object of the present inventionto partition the
`display screen of a display monitor into multiple planes for
`displaying selected windowsin a preselected form.
`It is another object of the present invention to dynamically
`size a cursor relative to the location of the cursor on the
`
`display screen.
`These and other objects, features, and advantages of the
`present invention will become apparent from reading the
`following description in conjunction with the accompanying
`drawings.
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`FIG. 1 is a diagram showing the basic components of a
`computer system used in connection with the preferred
`embodiment of the present invention.
`FIG. 2 is a block diagram showing the basic components
`of the computer system used in connection with the pre-
`ferred embodiment of the present invention.
`FIG. 3 illustrates a detailed view of the display screen
`generated in accordance with the present invention.
`FIG. 4a, 4b, and 4c illustrate a manipulation of displayed
`windows in accordance with the present invention.
`FIG. 5a, 5b, and 5c illustrate another manipulation of
`displayed windows in accordance with the present inven-
`tion.
`
`FIG. 6a, 6b, and 6c illustrate another manipulation of
`displayed windows in accordance with the present inven-
`tion.
`
`FIG. 7a, 7b, 7c, and 7dillustrate an alternate embodiment
`of a control mechanism for manipulation of displayed win-
`dows in accordance with the present invention.
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`6
`FIG. 8a, 8b, 8c, and 8dillustrate an alternate embodiment
`of a control mechanism for manipulation of displayed win-
`dows in accordance with the present invention.
`FIG. 9a and 9b illustrate an alternate embodiment of a
`control mechanism for manipulation of displayed windows
`in accordance with the present invention.
`FIG. 10a and 10b illustrate alternate embodiment of a
`control mechanism for manipulation of displayed windows
`in accordance with the present invention.
`FIG. 11 illustrates lines demarcating areas for display of
`windows in accordance with the preferred embodiment of
`the present invention.
`FIG. 12 illustrates a non-planar bounding surface for
`defining a three-dimensional workspace.
`FIG. 13 illustrates multiple isometric spaces constructed
`according to an embodimentof the invention.
`FIG. 14 illustrates an alternate view of the multiple
`isometric spaces.
`FIG. 15 illustrates the method of dynamically sizing the
`cursor of the present invention.
`FIGS. 16a and 165 are flow diagrams showing the steps
`implemented in the preferred embodiment of the present
`invention.
`
`DETAILED DESCRIPTION
`
`Referring now to the drawings, in which like numerals
`illustrate like elements throughout the several views, FIG. 1
`illustrates the basic components of a computer system 10
`used in connection with the preferred embodiment of the
`present invention. FIG. 1 illustrates a graphical user inter-
`face 12 on a computer display screen 15 of a display monitor
`14 of the computer system 10. The graphical user interface
`of the display system is preferably implementedas part of
`the operating system to provide a three-dimensional envi-
`ronmentfor displaying and managing the windowobjects of
`the operating system. The present invention enables cutting
`and pasting, moving objects, and other functions as gener-
`ally performed by an operating system while providing
`visual depth cues displayed by the operating system consis-
`tent with the orientation and position of the object within
`predefined boundaries defining a space for providing the
`depth cues. Additionally, sounds such as beeps and warning
`signals, generally associated with the interaction of a user
`with a computer system, may appear, using stereo sound,to
`emanate from the window upon whicha user is operating in
`the three-dimensional space. By varying the level of the
`sound and/or direction from which the sounds emanates, the
`sound may appear to emanate from a selected window.
`Although the present invention is described in conjunction
`with a conventional desktop computer, it will be appreciated
`that the present invention may beutilized in other types of
`computer systems that use a window based display system.
`The graphical user interface system 12 is implemented with
`a computer 16 connected to the display monitor 14. The
`computer 16 receives input data from a conventional key-
`board 20 via an input line 18. Cursor keys on the keyboard
`20, a mouse 21, trackball, or other pointing device may be
`used to move a cursor 23 on the display screen 15 for
`selection of various options. It should be appreciated by
`those skilled in the art that text entry or commandinput also
`may be accomplished by using a touch-sensitive pad or
`handwriting recognition device in conjunction with software
`for recognizing the corresponding input signals as selection
`signals.
`the hardware components of the
`Referring to FIG. 2,
`computer 16 are illustrated. For simplicity of the drawings,
`
`26
`
`26
`
`

`

`5,880,733
`
`7
`many components of a standard computer system have not
`been illustrated such as address buffers, memory buffers and
`other standard control circuits because these elements are
`well known and illustrated in the prior art and are not
`necessary for the understanding of the present invention. A
`computer program used to implement the various steps of
`the present invention is generally located in the memory unit
`20, and the processes of the present invention are carried out
`throughthe use of a central processing unit (CPU) 22. Those
`skilled in the art will appreciate that the memory unit 20 is
`representative of both read-only memory and random access
`memory. The CPU 22 is typically implemented as a single-
`chip microprocessor, such as the models 80386 or 80486 that
`are available from Intel Corporation, Santa Clara, Calif. The
`CPU 22, in combination with computer software, such as an
`operating system 25 and application programs 27, controls
`the operations of the computer system 10. The operating
`system software 25, in conjunction with application pro-
`grams 27, controls the allocation and usage of hardware and
`software resources such as memory, CPU time, disk space,
`and peripheral devices. It should be appreciated that many
`window based operating systems exist such as Microsoft
`“WINDOWS”, a UNIX (e.g. X-windows) operating system,
`or a Macintosh operating system. The display routines are
`encoded in a graphical user interface display layer in the
`operating system. The system bus 24 supports communica-
`tions of control, address, and data signals between the CPU
`22 and the remaining components of the computer system
`10. The memory unit 20 and the CPU 22 are connected by
`a computer system bus 24 designed to provide an electrical
`interface between computer system components. The pro-
`cesses implemented by the CPU 22 may be communicated
`as electrical signals along the computer system bus 24 to an
`input/output device 26, such as display monitor 14. The
`display monitor 14 provides a visual display of computer
`generated graphics originating from the processes imple-
`mented by the CPU 22.
`In this detailed description, numerousdetails are provided
`such as computer display system elements, object
`definitions, display formats, sample data, etc., in order to
`provide an understanding of the invention. However, those
`skilled in the art will understand that the present invention
`may be practiced without the specific details. Well-known
`circuits, programming methodologies, and structures are
`utilized in the present invention but are not described in
`detail in order not to obscure the present invention.
`Certain of the descriptions which follow are presented in
`terms of display images, processes, and symbolic represen-
`tations of operations of data bits within the computer’s
`memory. As will be knownto those skilled in the program-
`ming arts,
`these process descriptions, class descriptions,
`messages, notification descriptions, and graphic displays are
`the means used by those skilled in the art of computer
`programming and computer constru