a2, United States Patent
`US 6,577,330 B1
`(10) Patent No.:
`
`(45) Date of Patent: Jun. 10, 2003
`Tsudaet al.
`
`US006577330B1
`
`(54) WINDOW DISPLAY DEVICE WITH A
`THREE-DIMENSIONAL ORIENTATION OF
`WINDOWS
`
`(56)
`
`References Cited
`U.S. PATENT DOCUMENTS
`
`(75)
`
`Inventors: Kenjirou Tsuda, Hirakata (JP); Satoru
`Inagaki, Suita (JP)
`
`(73) Assignee: Matsushita Electric Industrial Co.,
`Ltd., Osaka-fu (JP)
`:
`.
`Lo,
`Subject to any disclaimer, the term ofthis
`patent is extended or adjusted under 35
`US.C. 154(b) by 0 days.
`
`:
`(*) Notice:
`
`(21) Appl. No.:
`.
`(22) PCT Filed:
`(86) PCT 'No.:
`§ 371 (c)(1),
`(2), (4) Date:
`
`09/463,198
`
`Aug. 10, 1998
`PCT/JP98/03587
`
`Jan. 18, 2000
`
`(87) PCT Pub. No.: WO99/08178
`
`PCT Pub. Date: Feb. 18, 1999
`
`(30)
`
`Foreign Application Priority Data
`
`Aug. 12,1997
`Sep. 9,1997
`
`(TP)
`(TP)
`
`ceccsccsscessessesssesstsssesseeseesessseees 9-217848
`ceecesessesssessesssessssssessesssesssessees 9-243641
`
`(SV)
`
`Tint. C07 eee eeeseeceeeeeeeeeeeeeeeeeneees G09G 5/00
`
`(52) US. Ch. cece eceeeenstetereneneeneneeenes 345/782; 345/781
`
`(58) Field of Search oo... eee 345/781, 782,
`345/848, 849, 850, 851, 852, 427
`
`4/1997 O’Neill et al. 0... 345/839
`5,621,906 A *
`5,678,015 A * 10/1997 Goh wecceeeeeteeeees 345/839
`5,745,109 A *
`4/1998 Nakano et al.
`............. 345/823
`5,754,809 A *
`5/1998 Gandre 0... eee 345/782
`5,880,733 A *
`3/1999 Horvitz et al. we. 345/427
`5,977,973 A * 11/1999 Sobeski et al. ........ 345/798
`FOREIGN PATENT DOCUMENTS
`
`EP
`JP
`JP
`JP
`JP
`
`8/1998
`6/1993
`9/1993
`8/1994
`1/1997
`
`856786
`5135154
`5233199
`6222899
`9022340
`:
`:
`* cited by examiner
`Primary Examiner—John Cabeca
`Assistant Examiner—Brian J Detwiler
`(57)
`ABSTRACT
`
`A storage unit 5102 stores image data for a window,and a
`three-dimensional position calculating unit 5104 calculates a
`position for the window in a virtual three-dimensional space
`so that it appears to be at a position inclined in the depth
`direction. A perspective projection unit 5106 transforms
`image data so that a window positioned at the calculated
`coordinates is projected onto a screen in perspective, taking
`a point
`in front of the screen as a viewpoint. A frame
`memory unit 5107 holds image data for the window on
`which perspective projection has been performed, and a
`display unit 5108 displays the window on the screen.
`
`24 Claims, 64 Drawing Sheets
`
`
`
` EDIT DISPLAY HELP
`
`
`
`CONVENTIONAL WINDOWING SYSTEMS USED IN
`INFORMATION TERMINALS SUCH AS COMPUTERS
`GENERALLY USE A METHOD IN WHICH A PLURALITY
`OF APPLICATION WINDOWS OR DATA WINDOWS ARE
`DISPLAYED IN 2D ON THE SCREEN OF AN IMAGE
`DISPLAY APPARATUS SUCH ASA DISPLAY MONIT
`ORLCD DISPLAY, LIKE THE ONE INTHE DIAGRAM.
`
` ee ana
`
`
`
`
`
`[NIARcoacON
`
`@WINDOW 1
`
`FILE EDIT DISPLAY HI
`
`CONVENTIONAL WINDOWING SYSTEMS USED IN
`INFORMATION TERMINALS SUCH AS COMPUTERS
`GENERALLY USE A METHODIN WHICH A PLURALITY
`OF APPLICATION WINDOWS OR DATA WINDOWSARE
`DISPLAYED IN 2D ON THE SCREEN OF AN IMAGE
`DISPLAY APPARATUS SUCH AS A DISPLAY MONITOR
`OR LCD DISPLAY,LIKE THE ONE IN THE DIAGRAM.
`
`ITee7 A
`eeeora
`
`1
`
`APPLE 1008
`
`APPLE 1008
`
`1
`
`

`

`U.S. Patent
`
`Jun. 10, 2003
`
`Sheet 1 of 64
`
`US 6,577,330 B1
`
`FIG. 1
`
`PROGRAM
`EXECUTION
`UNIT
`
`[5101
`
`5102
`
`pTORAGE
`13
`
`INPUT
`UNIT
`
`5105
`TEXTURE
`MAPPING
`UNIT
`
`3D POSITION
`CALCULATING
`UNIT
`
`p2!04
`
`PERSPECTIVEP 18
`PROJECTION
`UNIT
`
`5107
`
`9108
`
`MEMORY
`
`DISPLAY
`UNIT
`
`2
`
`

`

`U.S. Patent
`
`Jun. 10, 2003
`
`Sheet 2 of 64
`
`US 6,577,330 B1
`
`FIG. 2A
`
`PERSPECTIVE-PROJECTED
`WINDOW
`
`
`
`
`
`
`WINDOW
`
`(PROJECTION
`BJECT)
`
`DISPLAY
`(PROJECTION SURFACE)
`
`USER'S VIEWPOINT
`(PROJECTION ORIGIN)
`
`FIG. 2B
`
` 4" \IANISHING POINT
`
`3
`
`

`

`U.S. Patent
`
`Jun. 10, 2003
`
`Sheet 3 of 64
`
`US 6,577,330 B1
`
`FIG. 3A
`
`USER'S
`VIEWPOINT |
`O
`
` O
`
`FROM
`ABOVE
`
`FROM
`THE SIDE
`
`
`
`FIG. 3B
`
`
`
`FROM
`
`OmaEN ABOVE
`O<< THE SIDE
`
`FROM
`
`DEPTH DIRECTION
`|
`DISPLAY SCREEN
`
`FIG, 3C PROJECTION
`
`SURFACE
`
`4
`
`

`

`U.S. Patent
`
`Jun. 10, 2003
`
`Sheet 4 of 64
`
`US 6,577,330 B1
`
`55121
`
`
`
`GENERATE WINDOW
`CODE DATA AND
`BIT-MAP IMAGE
`
`POSITION
`
` PERFORM TEXTURE
`
`MAPPING
`
`
`
` ACTIVATE
`APPLICATION
`PROGRAM
`
`
`
`
`
` CALCULATE 3D
`
`
`
` PERFORM PERSPECTIVE
`PROJECTION
`
`DISPLAY IMAGE
`
`
`
`5
`
`

`

`dIqHAVIdSICU04dk7MOCGNIM@®
`siSUALAUNODSVHONSEeeNOMLLNICASASALSASONIMOCNIATWNOLLNGANODSTVNINYELNOLLVIUOANI
`
`
`
`
`
`
`NSTHELISATALLLdTdHAVIdSIGLIGAaT
`
`
`OUGemma§~TMOCNIM®
`
`
`
`
`
`
`
`
`
`
`
`
`
`HOLINONAVIASICVSVHONSSALVEVddVAVISIC
`
`
`
`
`
`WVYOVICTHLNIANOIHLIMTAVISICCOTNO
`
`FOVININYdONAMIOSTALNOCeNICAVTdSIC
`
`Jun. 10, 2003
`
`Sheet 5 of 64
`
`US 6,577,330 B1
`
`U.S. Patent
`
`GOld
`
`
`
`
`
`
`
`NICASASWALSASONTIMOCNIMTYNOLINYANOO
`
`
`
`
`
`ALTTVaNTdVHOTAMNICOHLANVASAATTVINAD
`
`
`
`AAVSMOCNIMVIVCdOSMOUNIMNOLVOFTddVAO
`
`
`
`AOVININVdONYAYOSAHLNOCeNICYAVTdS1C
`
`
`
`
`
`YOUINOWAVIdSICVSVHONSSALVAVdd¥AVISIC
`
`
`
`
`
`SaadNOOSVHONSSTWNINGHLNOMLVAYOANI
`
`
`
`
`
`WVYOVIGTHLNIINOSHLATTAVTdSIGCOTaO
`
`
`
`ALIIVUCYIdVHOLMNICOHANWISIATIVYANAD
`
`
`
`TUVSMOCINIMVEVCIOSMOCNTMNOILVIIIdddO
`
`6
`
`

`

`U.S. Patent
`
`Jun. 10, 2003
`
`Sheet 6 of 64
`
`US 6,577,330 B1
`
`FIG. 6
`
`PROGRAM[ppl0l
`EXECUTION
`UNIT
`
`5102
`
`ORAGE 3D POSITION_5104
`
`CALCULATING
`=O
`UNIT
`
`INPUT
`UNIT
`
`520)
`
`WINDOW
`ANALYZING
`UNIT
`
`TEXTURE
`MAPPING UNIT
`
`5105
`
`5106
`
`PERSPECTIVE
`PROJECTION
`UNIT
`
`9107
`
`9108
`
`FRAME
`
`IMAGE
`DISPLAY
`
`7
`
`

`

`U.S. Patent
`
`Jun. 10, 2003
`
`Sheet 7 of 64
`
`US 6,577,330 B1
`
`FIG. 7
`
`FROM $5121
`
`59220
`
`READ WINDOW
`CODE DATA
`
`$5222
`55991
`Ves_|DECIDE ON UPPER
`EDGE AS ROTATION
`
`
`AXIS
`
`
`
`
`DECIDE ON LEFT
`WRITTEN
`EDGE AS ROTATION
`
`AXIS
`
`
`
`
`
`
`
`
`DECIDE ON RIGHT
`WRITTEN
`EDGE AS ROTATION
`
`
`VERTICALLY?
`AXIS
`
`
`
`
`
`
`
`DECIDE ON EDGE
`NEAREST IMAGE
`AS ROTATION AXIS
`
`DECIDE ON LEFT EDGE
`AS ROTATION AXIS
`
`FROM $5126
`
`8
`
`

`

`SNUIMOGNINYWORARS8
`
`(UUWORMORLShices
`WSSLND(2NIygFrSHOQDAOBOSiogcaNouRad
`
`CHELANSYHOMSTHNIRELNOLLWHEONE
`
`
`XfCUSSRELSISSATAOCHATCLAND
`
` prsseyounAeALNDCVC
`
`AUTIVAQYIdVHOUNICOHLANY2801ATIVAN)
`caySHOCALAW1¥CYOSOCALNOLY)
`
`OWNNYgONEEDSAaya
`
`SAELNGHODSVHONSSWNYVRALNORIO,
`
`AL/TPGITTdVHOTHMNICORLEYASARUSS
`
`SHBLNGNOOS¥HONSSTHNINYELNOLAN
`
`
`
`
`
`NUCCESASALSAONTACNIATHNOLLNSANOO)
`
`U.S. Patent
`
`Jun. 10, 2003
`
`Sheet 8 of 64
`
`US 6,577,330 B1
`
`
`
`
`
`
`
`AYaS)SAGLSAS
`
`8Old
`
`9
`
`

`

`U.S. Patent
`
`Sheet 9 of 64
`
`US 6,577,330 B1
`
`PROGRAM
`EXECUTION
`UNIT
`
`101
`
`9102
`
`STORAGE
`UNIT
`
`5103
`
`INPUT
`UNIT
`
`9105
`
`TEXTURE
`MAPPING
`UNIT
`
`9104
`
`
`
`3D POSITION
`CALCULATING
`5104a
`UNIT
`[AUTOMATIC
`ALIGNING UNIT
`
`PERSPECTIVEP??
`PROJECTION
`UNIT
`
`9107
`
`9108
`
`FRAME
`
`IMAGE
`
`UNIT
`
`10
`
`

`

`U.S. Patent
`
`Jun. 10, 2003
`
`Sheet 10 of 64
`
`US 6,577,330 B1
`
`FIG. 10
`
`FROM S5121
`
`$5230
`ACQUIRE ALIGNMENT METHOD
`AND ALIGNMENT OBJECT
`(NUMBER OF WINDOWSn)
`
`SECOND
`ALIGNMENT
`29231
`WHICH ALIGNMENTwpEHO?
`METHOD?
`
`FIRST
`ALIGNMENT
`METHOD
`
`59233
`
`
`
`
`
`
`ALIGN WITHOUT
`OVERLAPPING
`
`ALIGN WITH
`OVERLAPPING
`
`55234
`NOTIFY THE TEXTURE MAPPING
`UNIT 105 OF THE POSITIONS OF
`n WINDOWS SEQUENTIALLY
`
`FROM $5126
`
`11
`
`

`

`U.S. Patent
`
`Jun. 10, 2003
`
`Sheet 11 of 64
`
`US 6,577,330 B1
`
`oF
`
`FILE EDIT DISPLAY HELP
`
`ONERIMLSCasOE
`cuaONALWINDOWING§eeaeiN
`
`Nia riCHEEet
`i- AEDOME
`CSEHERESTELA
`
`ORLCPDISPLAY, LIKE THEONEIN THE DIAGRAM.
`| maAe
`
`FIG. 11A
`
`6
`
`INDO
`
`,
`
`ane mh
`ar
`aoeCU
`
`YSPLA VY
`
`sc
`APPARATUS SUCH AS A DISPLAN MONKOR
`
`FIG. 11B
`
`Oenmon| RaMRn
`
`12
`
`12
`
`

`

`U.S. Patent
`
`Jun. 10, 2003
`
`Sheet 12 of 64
`
`US 6,577,330 B1
`
`5240 9241 5242 5243 5244 FIG. 12A
`FIG. 12C
`
`vey CONVENTIONALWINDOWNGSYSTENSUSEDA.
`EA
`ErMNSHRSASERIAHINGICOGRRTECTT ot
`AMTUTUTY
`EORVIATION TERVINAISSUCHASCCNPLTERS
`EB
`
`FIG. 12B
`
`5240 5241 5242 5243 5244
`
`13
`
`13
`
`

`

`U.S. Patent
`
`Jun. 10, 2003
`
`Sheet 13 of 64
`
`US 6,577,330 B1
`
`PROGRAM
`EXECUTION
`UNIT
`
`9101
`
`9102
`
`STORAGE
`UNIT
`
`9103
`
`INPUT
`UNIT
`
`9301
`
`WINDOW
`NFORMATION
`IMAGE POSITION
`CHANGING UNIT
`
`CHANGING UNIT
`
`3D POSITION
`CALCULATING
`UNIT
`
`p2!04
`
`TEXTURE
`MAPPING UNIT
`
`5105
`
`PERSPECTIVEV 6
`PROJECTION
`UNIT
`
`5107
`
`9108
`
`FRAME
`MEMORY
`
`IMAGE
`DISPLAY
`UNIT
`
`— 4
`
`14
`
`

`

`U.S. Patent
`
`Jun. 10, 2003
`
`Sheet 14 of 64
`
`US 6,577,330 B1
`
`FIG. 14
`
` WINDOW AAORMALTON
`
`IMAGE DISPLAYED
`
` WINDOW INFORMATION
`IMAGE DISPLAYED
`FACING FRONT?
`
`
`
`SEPARATE WINDOW INFORMATION
`IMAGE AND MAIN WINDOW
`
`FROM $5122
`NEAR FRONT?
`
`
`INFORM
`
`MAGE IS FACING THE FRONT
`
`
`CUAPOSITIONS 50.Hand WINDOW
`
`
`AND MAINWINDOW IS iNPERSPECTIVE
`
`
`
`15
`
`15
`
`

`

`ANSLVS0SMOGNTMNOMVOITdF0ACACIA
`
`HOTHMNTCOHLSHWAS)A77PANED
`
`QOVWANY40NERSSHENOCNTCAVTS
`
`
`
`AMANOSYBOISSTPNINRLNOLAQHA?
`
`YACSSNAUMS
`
`YOUNONAVTGSICYSVHONSSALVddWAVTdT
`
`"WHMOWCSHENTINOHLDHTAVTSCC9790
`
`
`
`
`
`SATNGNBOTANYCOHANWAS.T7RMD
`
`SEAUNANODSYHONSSTNNOUPAGO
`
`TONYAONAGTHDNOCCNYCAVTaS7WSOC0SHOCMIMNOMPOITdd710
`
`
`
`YOUNONAVTGSICVSVHONSSALPaddVA7ST
`
`
`
`"WYROVIGSHENTINOFHLDITAVTdSIC@O7IO
`
`IMOCNIM@
`
`
`
`
`
`AVAHAVTdSICLICHDike)
`
`ONTOTINO?
`
`U.S. Patent
`
`Jun. 10, 2003
`
`Sheet 15 of 64
`
`US 6,577,330 B1
`
`
`
`
`
`AACASHRUGONMOCMATNO
`
`GT“Old
`
`16
`
`16
`
`

`

`U.S. Patent
`
`Jun. 10, 2003
`
`Sheet 16 of 64
`
`US 6,577,330 B1
`
`
`
`ALITYTdVHOLLNICOWLSAMS
`
`FAVSMOCNTMVVXOSHOTOKAN
`
`
`
`
`
`SUELAANODSVHONSSIWNIRALYOUSROW,
`
`
`
`COLINOKAVTASICVSYWORSERAO
`
`
`
`APOVICTHINNOWUWOO
`
`
`
`
`
`TOWRINY40NARSSANSCARO
`
`
`
`
`
`
`
`ATCAS1SNAISASONIMOONYWYONG
`
`WROUSERSOD
`
`NOUNS3ALTVdATdVHOTHMNIQOHLANYSSAKYWROD
`FEVSMOCNTAVIVA30SMOUNIA
`
`
`
`
`SAGLAANODSVHONSSTVNINSALNOUNTROMN,
`
`
`
`AfGIS)SHALSASONTMOCNIM
`
`
`
`
`
`FOVRINYdONATOSTHLNOCOMRNTSA
`
`
`
`GOLINONAVTASIC¥SYHOSSAINANWVQQ
`
`AVAOVICTHLNINOTHLE,WSO
`
`dIWHAVIdSIGUGHYITMOCNIM@®
`
`91Old
`
`17
`
`17
`
`

`

`U.S. Patent
`
`Jun. 10, 2003
`
`Sheet 17 of 64
`
`US 6,577,330 B1
`
`FIG. 17
`
`PROGRAM
`EXECUTION
`UNIT
`
`[5101
`
`5102
`
`
`
`PPRAGE 3D POSITION_p°!04
`
`CALCULATING
`
`INPUT
`UNIT
`
`5401
`
`
`
`5402
`SCROLL BAR
`
`ATTACHMENT
`
`TEXTURE
`MAPPING UNIT
`
`5105
`
`PERSPECTIVE}
`
`PROJECTION
`UNIT
`
`5403
`
`UNREADABLE AREA
`DETERMINING UNIT
`
`5107
`FRAME
`MEMORY
`
`5108
`IMAGE
`DISPLAY
`UNIT
`
`= ©
`
`18
`
`

`

`U.S. Patent
`
`Jun. 10, 2003
`
`Sheet 18 of 64
`
`US 6,577,330 B1
`
`FIG. 18
`
`
`
`FROM $5122
`
`
`
`$5420
`UNREADABLE
`AREA?
`
`N
`
`Yes
`
`
`
`DETERMINE POSITION
`OF SCROLL BUTTON
`
`ATTACH SCROLL BAR
`IMAGE
`
`
`
`
`
`
`19
`
`

`

`U.S. Patent
`
`Jun. 10, 2003
`
`Sheet 19 of 64
`
`US 6,577,330 B1
`
`FIG. 19A
`
`@WINDOW 1
`
`CONVENTIONAL WINDOWING SYSTEMS USED IN
`
`
`
`INFORMATION TERMINALS .CH AS COMPUTERS
`OFAPPLICATIONWINDOWS ORDATA WINDOWSARE
`
`GENERALLY USE A METHOD IN WHICH A PLURAL
`
`DISPLAYED IN 2D ON THE SCREEN OF AN IMAGE
`DISPLAY APPARATUS SUCH AS A DISPLAY MONITOR
`OR LCD DISPLAY, LIKE THE ONE IN THE DIAGRAM.
`FURTHERMORE, A MEMORYAREA LARGER THAN
`THAT REQUIREDFOR DISPLAYIS PROVIDEDIN THE
`DIAGRAN.
`THESCREENAREAIS EXPANDED
`
`VIRTUALLY. WHILE ONE PART OF THIS VIRTUAL
`SCREENIS USED FOR ACTUAL DISPLAY, A WAYOF
`ACCESSING AN UNDISPLAYEDAREA IS
`SIMULTANEOUSLYSHOWN._THIS ALLOWS THE
`WINDOWTO BE EXPANDED IN AN AREA LARGER
`THANTHE SCREEN OF THE DISPLAYDEVICE.
`
`---4
`
` CSNATORALTINDOWING SYSTEMS USEDIV
`
`
`YSRQRAANTOUTERMINALS SUCH AS COMPUTERS
`
`
`
`QSRRAAAUSEAMETHOD IN WHICHA PLURALITY
`
`SRARCATIONHINDOWS OR DATA HINDOWSARE
`WANTIVD NTHE SCREENOFAVMAG?
`NWSANAATRATUS SUCH AS4 DISPLAMOMTOP
`XRAY,LIKETHEONEINTHEA
`
`
`
`
`
`9432
`
`20
`
`20
`
`

`

`U.S. Patent
`
`Jun. 10, 2003
`
`Sheet 20 of 64
`
`US 6,577,330 B1
`
`FIG. 20
`
`PROGRAM
`EXECUTION
`UNIT
`
`[5101
`
`5102
`
`oTORAGE 3D POSITION_P5104
`
`CALCULATING
`03
`UNIT
`
`INPUT
`UNIT
`
`
`
`LAYER
`
`GENERATING
`UNIT
`
`TEXTURE
`MAPPING UNIT
`
`5105
`
`PERSPECTIVE) 18
`PROJECTION
`UNIT
`
`5107
`
`5108
`
`FRAME
`MEMORY
`
`IMAGE
`pr
`
`21
`
`21
`
`

`

`U.S. Patent
`
`Jun. 10, 2003
`
`Sheet 21 of 64
`
`US 6,577,330 B1
`
`FIG. 21
`
`FROM So121
`
`59920
`
`DIVIDE IMAGE DATA INTO
`LAYERS
`
`GENERATE WINDOW LAYER
`
`$9921
`
`$9922
`DISPLAY WINDOW LAYER AT
`A POSITION FLOATING OVER
`BASE LAYER
`
`FROM S5126
`
`TO $5122
`
`22
`
`22
`
`

`

`U.S. Patent
`
`Jun. 10,2003
`
`Sheet 22 of 64
`
`US 6,577,330 B1
`
`
`@WINDOW1
`
`FILE
`EDIT DISPLAY HELP
`CONVENTONALWINDOWING SYSTEMS USED IN
`
`INFORMA~ON TERMINALS SUCH AS COMPUTERS
`
`GENERALLYUSE A METHOD IN WHICH A PLURALITY
`OF APPLCATION WINDOWS
`OR DATAWADOWS ARE
`
`DISPLAY=) ‘N20 ON THE
`
`SCREEN OF AN IMAGE
`
`
`IMAGE
`
`
`ne
`
`9930
`
`
`
`
`
`
`FIG. 22A
`
`FIG. 22B
`
`@WINDOW|
`FILE
`EDIT DISPLAY HELP
`
`
`
`
`
`
`
`COMENONALTWINDOWING SYSTEMS USED IN
`NFORN'AON TERMINALS SUCH AS COMPUTERS
`GENERALSUSE A METHOD IN WHICH A PLURAUTY
`Or APP"CATION WINDOWS
`OR DATANDDWS ARE
`JISPLAY=9 IN 2D ON THE
`SCREEN 2 AN IMAGE
`
`Oo
`
`
`9931
`
`
`
`
`
`
`
`CREE\ OF AN IMAGE
`
`RE
`SYSTRISTSES
`CONTENTIONALWINDOWING
`INFORIAMIONTRRVINALSScenegSESE
`GENERALLY USEA METHOD IN WHICH A PLURALITY
`OFCeODATAWnota TN 2D ONTHE
`
`23
`
`23
`
`

`

`U.S. Patent
`
`Jun. 10, 2003
`
`Sheet 23 of 64
`
`US 6,577,330 B1
`
`PROGRAM
`EXECUTION
`UNIT
`
`[9101
`
`9102
`
`STORAGE
`UNIT
`
`5103
`
`INPUT
`UNIT
`
`2601
`
`KEYWORD
`SEARCH UNIT
`
`779603
`WINDOW LAYER
`GENERATING
`UNIT
`
`3D POSITION
`CALCULATING
`UNIT
`
`p2!04
`
`TEXTURE
`MAPPING UNIT
`
`5105
`
`PERSPECTIVE} 16
`PROJECTION
`UNIT
`
`9602
`
`SEARCH RESULT
`OUTPUT UNIT
`
`5107
`
`9108
`
`FRAME
`MEMORY
`
`IMAGE
`DISPLAY
`UNIT
`
`aN
`
`24
`
`

`

`U.S. Patent
`
`Jun. 10, 2003
`
`Sheet 24 of 64
`
`US 6,577,330 B1
`
`FIG. 24
`
`FROM $5121
`
`DETERMINE
`KEYWORD
`
`SEARCH FOR KEYWORD|$9621
`IN DOCUMENT AND
`INFORMATION
`
`55622
`
`KEYWORD FOUND?
`
`
`
`
`
`59623
`
`GENERATE WINDOW
`LAYER
`
`$5624
`DISPLAY WINDOW
`LAYER FLOATING
`OVER BASE LAYER
`
`FROM $5126
`
`TO $5122
`
`25
`
`25
`
`

`

`U.S. Patent
`
`Jun. 10, 2003
`
`Sheet 25 of 64
`
`US 6,577,330 B1
`
`
`
`
`
`PPA
`
`
`CONVENTIONAL WINDOWING SYSTEMS USED
`CeeaeustaHse
`ifAPPLICATIONWINDOWSORDATAWINDOWSARE
`
`DISPLAYEDINDonTHESCREENOFGr
`OeDSPLAT LIKETHEONEINTH
`E DIAGRAM.
`
`
`
`
`
`
`
`
`
`
`
`
`CONVENTIONAL WINDOWING SYSTEMS USE
`
`NFORMATIONTERMINALS SUCH AS COMPUTERS
`—_—
`
`
`
`OF APPLICAT!
`DoW
`
`N 2D ON THE SCREEN OF ANI
`SeAPPARATUSSUCHASAOSMONITOR
`a)
`ARLCDDISPLAY, LIKETHEONE
`
`
`
`
`
`26
`
`26
`
`

`

`U.S. Patent
`
`Jun. 10, 2003
`
`Sheet 26 of 64
`
`US 6,577,330 B1
`
`PROGRAM
`EXECUTION
`UNIT
`
`9101
`
`5102
`
`STORAGE
`UNIT
`
`INPUT
`UNIT
`
`9103
`
`9701
`
`WINDOW
`LINKING UNIT
`
`9702
` WINDOW
`POSITIONAL
`
`RELATIONSHIP
`STORAGE UNIT
`
`
`
`
`
`
`9703
` WINDOW
`
`POSITIONAL
`
`
`RELATIONSHIP
`CHANGING UNIT
`
`
`
`
`3D POSITION
`CALCULATING
`
`104
`
`5105
`
`CcCcZZrrj
`
`TEXTURE
`MAPPING UNIT
`
`PERSPECTIVEY 0°
`PROJECTION
`
`9107
`
`9108
`
`FRAME
`MEMORY
`
`IMAGE
`DISPLAY
`UNIT
`
`27
`
`

`

`U.S. Patent
`
`Jun. 10, 2003
`
`Sheet 27 of 64
`
`US 6,577,330 B1
`
`FIG. 27
`
`FROM $5121
`
`SPECIFY CONNECTED|©9729
`WINDOWS
`
`GENERATE AND STORE
`INITIAL LINK INFORMATION
`
`S972]
`
`S5722
`
`POSITION CHANGE?
`
`No
`
`CHANGE LINK INFORMATION TO
`FOLD UP SOME WINDOWS
`
`
`
`
`
`FROM $5126
`
`28
`
`28
`
`

`

`U.S. Patent
`
`Jun. 10, 2003
`
`Sheet 28 of 64
`
`US 6,577,330 B1
`
`FIG. 28A
`FIG. 28B
`
`| FOLD UP Page3
`
`29
`
`29
`
`

`

`U.S. Patent
`
`Jun. 10, 2003
`
`Sheet 29 of 64
`
`US 6,577,330 B1
`
`FIG. 29A
`
`FIG. 29B
`
`COPYORIGIN
`
`Page2
`
` Page4
`
`
`
`
`30
`
`30
`
`

`

`Jun. 10, 2003
`
`Sheet 30 of 64
`
`US 6,577,330 B1
`
`WYOASNVAL
`
`LINN)
`
`
`
`LIN‘NOILVANAOANI
`
`MOCNIM
`
`LINNNOLLVAMOANI|
`
`LINNOLLVAAOANI
`SNOUNVTTAOSIA|
`
`YALOVYVHO
`
`ASVaVIVd
`
`INO
`
`U.S. Patent
`
`HALLOYdSuad
`NOILVWAYOSNICe
`
`WYOASNVALL
`
`LIND)LNHNADVNVA
`
`Ocl
`
`0€“Old
`
`
`
`LINQONISSHOOdd
`
`NOILV.LOY
`
`QOT
`
`101
`
`SOI
`
`cOT
`
`31
`
`31
`
`
`
`
`
`
`
`
`
`

`

`U.S. Patent
`
`Jun. 10, 2003
`
`Sheet 31 of 64
`
`US 6,577,330 B1
`
`D(x1,y1,0)
` DISPLAY
`
`TEXTINA3DWINDOW
`
`DISPLAYTEXTINA2DWINDOW
`
`Y;lWILLLLLLLibrKishan7LLAo
`FIG.31
`
`
`
`Ve
`REA
`XQ
`
`
`qa
`
`32
`
`32
`
`

`

`U.S. Patent
`
`Jun. 10, 2003
`
`Sheet 32 of 64
`
`US 6,577,330 B1
`
`|0|0|0WTONV
`NOLLV.LOd
`
`SHLVNITHOOOMOCNIM
`(O'O'TX)|(O'0'O)|(O'r4‘0)
`Wadd|MSMOT|Maddn
`LHWLdainca
`SHLVNIGHOOD|YHTHILNYCI
`TVILVdS
`
`MOCNIMKage
`
`vOECOECOEL0¢
`
`ceOld
`
`
`
`
`
`(
`
`IZTAX
`
`33
`
`33
`
`
`
`

`

`vOP
`
`YHLOVAVHO
`
`SHLVNIGCHOOO
`
`ceOld TOV
`
`U.S. Patent
`
`Jun. 10, 2003
`
`Sheet 33 of 64
`
`US 6,577,330 B1
`
`QuATILXx
`
`Q‘A‘Zix
`
`Q‘ttA‘erx
`
`0)12A‘a2x
`
`()'teA‘1ex
`
`34
`
`34
`
`

`

`U.S. Patent
`
`Jun. 10, 2003
`
`Sheet 34 of 64
`
`US 6,577,330 B1
`
`(.ZtA'Lx) M
`
`I
`
`7
`
`35
`
`‘
`
`AAS
`imeLnaeWies
`
`veOld
`
`35
`
`
`

`

`U.S. Patent
`
`Jun. 10, 2003
`
`Sheet 35 of 64
`
`US 6,577,330 B1
`
`SIXV2
`
`
`
`AHTONVNOILV.LOd
`
`
`Sov»)GOTLAR|wali’|dat|sanvwcaoogfauna
` 0eoseees|(ooo|OKO|(ZA
`
`SHLVNIGNOODMOCNIMTVILVdS|MOCNIM
`
`
`SeOld
`
`36
`
`36
`
`
`
`
`
`

`

`U.S. Patent
`
`Jun. 10, 2003
`
`Sheet 36 of 64
`
`US 6,577,330 B1
`
`FOn------
`
`[SJocrereree
`
`[MJo-------
`
`IV|veeeeeee
`
`
`
`37
`
`37
`
`

`

`U.S. Patent
`
`Jun. 10, 2003
`
`Sheet 37 of 64
`
`US 6,577,330 B1
`
`FIG. 37A
`
`
`
`WINDOW COORDINATES
`
`
`
`UPPER
`LOWER|
`LOWER)
`UPPER)
`
`LEFT|LEFT RIGHT RIGHT
`
`
`
`
`
`
`FIG. 37B
`
`810 iy
`NG
`
`812
`
`DISPLAY
`Y
`POSITION 3D COORDINATES
`
`X1,Y11
`
`X1LYH,Z1
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`38
`
`ceeeeee) 'D)
`
`aesooe: Mi
`
`eeeee S|
`
`weoneon) rw,
`
`retcn rN
`
`38
`
`

`

`U.S. Patent
`
`Jun. 10, 2003
`
`Sheet 38 of 64
`
`US 6,577,330 B1
`
`FIG. 38
`
`900
`
`3D-Window
`
`DISPLAY TEXT IN A 3D WINDOW
`
`DISPLAY TEXT IN A 2D WINDOW
`
`FIG. 39
`
`
`
`OISPLayTEXTINA 3DB Win
`
`
`1000
`DISPLAYTextINAQDwives
`
`39
`
`39
`
`

`

`U.S. Patent
`
`Jun. 10, 2003
`
`Sheet 39 of 64
`
`US 6,577,330 B1
`
`FIG. 40
`
`FONT SCALE
`
`Z COORDINATE
`VALUE
`
`40
`
`40
`
`

`

`U.S. Patent
`
`Jun. 10, 2003
`
`Sheet 40 of 64
`
`US 6,577,330 B1
`
`FIG. 41
`
`ORIGINAL DISPLAY
`SIZE
`SIZE
`
`CHARACTER
`
`41
`
`41
`
`

`

`U.S. Patent
`
`Jun. 10, 2003
`
`Sheet 41 of 64
`
`US 6,577,330 B1
`
`FIG. 42
`
`RECEIVE WINDOW [91301
`ROTATION INPUT
`
`EXECUTE WINDOW
`ROTATION PROCESSING
`
`$1302
`
`$1303
`
`TRANSFORM POST-ROTATION WINDOW AND
`COORDINATES FOR CHARACTERSIN THE WINDOW
`TO 2D USING PERSPECTIVE TRANSFORM
`
`CALCULATE FONT TRANSFORM
`PARAMETERS FROM PERSPECTIVE
`TRANSFORM RESULT
`
`51304
`
`51308
` NO
`NPROCESSED
`CHARACTER IN THE
`WINDOW?
`
`
`
`
`
`J°!809
`
`YES
`READ INFORMATION FORA
`
`
`CHARACTER IN THE WINDOW
`
`READ FONT CHARACTER
`CORRESPONDING TO THE CHARACTER
`
`
`
`TRANSFORM THE FONT CHARACTER—J91307
`USING TRANSFORM PARAMETERS AND
`
`CREATE A CHARACTER IMAGE
`
`S1306
`
`NE A WINDOW THAT HAS UNDERGONEJf$1309
`TR
`SEORM PROCESSING WITH
`REAT
`TRANSFORMING A FONT CHARACTER
`
`42
`
`

`

`U.S. Patent
`
`Jun. 10, 2003
`
`Sheet 42 of 64
`
`US 6,577,330 B1
`
`
`
`OlI—~
`
`dSVaVLVd
`
`INOd
`
`
`
`
`UHLOVAVHO
`
`
`
`LINNNOILVAWYOANI
`
`cOl
`
`43
`
`
`
`SNOYENWTIAOSIN
`
`
`
`LINONOLLVANAYOANI
`
`cOl
`
`WHYOASNVaL
`
`LINOLNYNYOVNVA
`
`HAILOddSad00!
`
`NOLLVAYOANIde
`
`
`
`LINANOILVNYOANI
`
`TOT
`
`MOCNIM
`
`OcI
`
`ehOld
`
`LINONISSHOOUd
`
`NOLLV.LO&
`
`43
`
`
`
`
`
`
`
`
`

`

`U.S. Patent
`
`Jun. 10, 2003
`
`Sheet 43 of 64
`
`US 6,577,330 B1
`
`FIG, 44
`
`1500
`
`3D-Window
`
`DISPLAY TEXT IN A 3D WINDOW
`
`1501
`
`DISPLAY TEXT IN A 2D WINDOW
`
`BA
`
`1502
`
`FIG. 45
`
`1503
`
`
`
`DISPLayTEXTINafWINDow
`1600|DISPLAYTEXTINA 2p wixpow
`
`I1
`
`602
`
`
`
`44
`
`44
`
`

`

`U.S. Patent
`
`Jun. 10, 2003
`
`Sheet 44 of 64
`
`US 6,577,330 B1
`
`FIG. 46
`
`RECEIVE WINDOW|$1301
`ROTATION INPUT
`
`EXECUTE WINDOW
`ROTATION PROCESSING
`
`51302
`
`CHARACTER
`51701 WINDOW
`PROCESSING
`DIVIDE
`PROCESSING
`PROCESSING
`
`
`
`
`TRANSFORM
`
`POST-ROTATION WINDOW
`
`
`ND
`CHARACTER
`
`
`
`WINDOW TO 2D USING
`
`
`PERSPECTIVE PROJECTION
`
`
`
`READ INFORMATION
`
`FOR A CHARACTER IN
`
`
`THE WINDOW
`
`
`CALCULATE FONT
`CALE FROM Z
`Ww
`COORDINATE VALUE
`
`FOR THE CHARACTER
` READ CHARACTER|51705
`THE CHARACTER
`
`
`51706
`CHANGEFONTSIZE
`ACCORDING TO FONT
`
`SCALE AND GENERATE
`
`A CHARACTER IMAGE
`
`
`
`
`
`COMBINE A WINDOW THAT HAS UNDERGONE
`PERSPECTIVE TRANSFORM PROCESSING WITH
`
`
`
`A CHARACTER IMAGE CREATED BY
`TRANSFORMING A FONT CHARACTER
`
`CORRESPONDING TO
`
`END
`
`45
`
`45
`
`

`

`U.S. Patent
`
`Jun. 10, 2003
`
`Sheet 45 of 64
`
`US 6,577,330 B1
`
`
`
`OILVaLoadsVMOCNIM,
`
`HALLOYdSadd
`
`
`
`WHYOASNVaLNOILVANYOANICE
`
`
`
`LINNLINfLNFAYOVNVIA
`
`Ocl
`
`LyOld
`
`LINAONISSHOOdd
`
`NOILLV.LOY
`
`LINDONILWINOTVO
`
`LINNOILVNYOSNI
`TINANUYS20
`
`
`
`ONILVINOTVO
`
`
`
`HTVOSLNOA
`
`
`
`46
`
`46
`
`
`
`
`
`

`

`U.S. Patent
`
`Jun. 10, 2003
`
`Sheet 46 of 64
`
`FIG. 48
`
`3D-Window
`
`DISPLAY TEXTIN A 3D WINDOW
`
`DISPLAY TEXT IN A 2D WINDOW
`
` 2000|DISPLAYCTINA2DWINDowbs2003
`
`
`
`FIG. 49
`
`2002
`
`
`
`47
`
`47
`
`

`

`U.S. Patent
`
`Jun. 10, 2003
`
`Sheet 47 of 64
`
`US 6,577,330 B1
`
`O0S“O14
`
`d0S
`“Old
`
`VOS“Old
`
`ATVOS
`
`INO
`
`NOLLOSYId
`
`AIVO
`
`I
`
`J
`
`NOLLOHUICX
`
`ATVOS
`
`l
`
`0
`
`ALVNIGCHOOD4
`
`AVIVA
`
`NOLLV.LOY
`
`4TONV
`
`SIXV*
`
`NOILV.LO
`
`SIXV
`
`ATONV
`
`0
`
`48
`
`48
`
`

`

`U.S. Patent
`
`Jun. 10, 2003
`
`Sheet 48 of 64
`
`US 6,577,330 B1
`
`FIG. 51
`
`RECEIVE WINDOW|S1301
`ROTATION INPUT
`
`EXECUTE WINDOW
`ROTATION PROCESSING
`
`$1302
`
`CHARACTER
`PROCESSING
`
`S1701_ WINDOW
`PROCESSING
`
`91702
`
`DIVIDE
`PROCESSING
`S2201
`CALCULATE FONT ASPECT
`RATIO FROM WINDOW
`ROTATION ANGLE
`
`TRANSFORM
`
` WINDOW
`
`
`
`TO 2D USING
`
`
`PERSPECTIVE PROJECTION
`
`
`
`CALCULATE FONT
`CALE FROM Z
`
`COORDINATE VALUE
`FOR THE CHARACTER
`
`
`
`
`READ CHARACTER
`FONT CORRESPONDING
`TO THE CHARACTER
`
`S1705
`
`
`Ww
`
`
`
`
`TRANSFORM FONT [$2202
`CHARACTER BASED
`
`ON FONT SCALE
`AND ASPECT RATIO
`
`
`
`
`51309
`
`
`COMBINE A WINDOW THAT HAS UNDERGONE
`PERSPECTIVE TRANSFORM PROCESSING WITH
`
`
`
`A CHARACTER IMAGE CREATED BY
`TRANSFORMING A FONT CHARACTER
`
`
`
`END
`
`49
`
`49
`
`

`

`
`
`O1gZ—~
`
`ASVaVLVd
`
`INO
`
`U.S. Patent
`
`Jun. 10, 2003
`
`Sheet 49 of 64
`
`US 6,577,330 B1
`
`
`
`dALLOddSadd
`
`WHOASNVAL
`NOILVINUOANICe
`
`LIND
`
`
`
`LINf)LINYNYOVNVIN
`
`Ocl
`
`é$Old
`
`
`
`LINDONISSHO0O0Ud
`
`NOILV.LOY
`
`NOILVLOYLIN‘!ONIGVAa4TONV
`
`
`
`LINNNOLLVWYOANI
`
`dALOVAVHO
`
`MOCNIM
`
`ONILVTNOTVO
`
`
`
`HIVOSLNOA
`
`
`
`LINONOILVNYOANI
`
`50
`
`50
`
`
`
`
`
`
`

`

`U.S. Patent
`
`Jun. 10, 2003
`
`Sheet 50 of 64
`
`US 6,577,330 B1
`
`FIG. 53A
`
`FIG. 53B
`
`FIG, 53C
`
`FIG. 53D
`
`FIG. 53E
`
`FIG, 53F
`
`AAS
`
`51
`
`51
`
`

`

`U.S. Patent
`
`Jun. 10, 2003
`
`Sheet 51 of 64
`
`US 6,577,330 B1
`
`FIG. 54
`
`START
`
`RECEIVE WINDOW|S1301
`ROTATION INPUT
`
`EXECUTE WINDOW
`ROTATION PROCESSING
`
`$1302
`
`
`
`
`
`CALCULATE FONT
`SCALE FROM Z
`COORDINATE VALUE
`FOR THE CHARACTER
`
`
`1701 WINDOW
`CHARACTER
`PROCESSING
`DIVIDE
`PROCESSING
`PROCESSING
`
`$1702
` NPROCESSED
`
`
`TRANSFORM
`
`POST-ROTATION WINDOW
`AND CHARACTER
`
`
`COORDINATES IN THE
`WINDOW TO 2D USING
`
`PERSPECTIVE PROJECTION
`
`
`
`
` CHANGE FONT SIZE BASED
`51309
`ON FONT SCALE
`
`
`COMBINE A WINDOW THAT HAS UNDERGONE
`
`
`PERSPECTIVE TRANSFORM PROCESSING WITH
`
`
`A CHARACTER IMAGE CREATED BY
`TRANSFORMING A FONT CHARACTER
`
`
`
`END
`
`52
`
`52
`
`

`

`U.S. Patent
`
`Jun. 10, 2003
`
`Sheet 52 of 64
`
`US 6,577,330 B1
`
`I
`
`LNdN
`AVIdSIC
`
`TIN?
`
`LINN
`
`O61
`
`08I
`
`INOA
`
`ASVdVLIVd TINANOILVNYOSNI
`
`
`
`LINOLNAWEOVNVI
`
`NOILVWYOANICE
`
`LINNNOILVNYOANI
`
`MOCNIM
`
`LINANOILVNYOANI
`
`UALOVAVHO
`
`
`
`SNOANVTIHOSIN
`
`TOI
`
`cOT
`
`cOl
`
`53
`
`LINONISSHOOdd
`
`NOILV.LOY
`
`Ocl
`
`GS“Old
`
`53
`
`
`
`
`
`
`
`

`

`U.S. Patent
`
`Jun. 10, 2003
`
`Sheet 53 of 64
`
`US 6,577,330 B1
`
`
`
`3D-Window
`
`Cc)
`
`DISPLAY TEXT IN A 3D WINDOW
`
`
`
`
`
`
`
`
`DISPLAY TEXT IN A 2D WINDOW
`
`DISCARD UNREADABLE TEXT
`
`?
`2701
`
` S
`INA 3p WiNDee
`DISPLAYTEXT INa 2DWIN@ee
`
`
`DISCARD UNREADABLE @@ee
`
`
`
`2801
`
`
`
`
`54
`
`54
`
`

`

`U.S. Patent
`
`Jun. 10, 2003
`
`Sheet 54 of 64
`
`US 6,577,330 B1
`
`FIG. 58
`
`START
`
`RECEIVE WINDOW {81301
`ROTATION INPUT
`
`EXECUTE WINDOW
`ROTATION PROCESSING
`TRANSFORM POST-ROTATION WINDOW AND
`COORDINATES FOR CHARACTERS IN THE WINDOW
`TO 2D USING PERSPECTIVE TRANSFORM
`
`51303
`
`51302
`
`CALCULATE FONT TRANSFORM
`PARAMETERS FROM PERSPECTIVE
`TRANSFORM RESULT
`
`$1304
`
`
`51308
`
`NPROCESSED
`
`CHARACTER IN THE
`
`
`
`WINDOW?
`
`NO
`
`$1305
`PES
`READ INFORMATION FOR A
`CHARACTERIN THE WINDOW
`
`
`
`
`S290]
` OMPAR
`
`SIZE AND THRESHOLD
`
`VALUE FOR TRANSFORMED
`
`
`FONT CHARACTER
`
`
`
`READ FONT CHARACTER
`
`CORRESPONDING TO THE CHARACTER
`
`
`TRANSFORM THE FONT CHARACTER
`USING TRANSFORM PARAMETERS
`
`
`
`NE A WINDOW THAT HAS UNDERGONE]51309
`CTIVE TRANSFORM PROCESSING WITH
`
`55
`
`55
`
`

`

`U.S. Patent
`SAILLOAdSadd
`WYOASNVALLcOl
`
`WYOASNVaALL
`
`LIN{)LNHNAOVNVI
`
`LIND
`
`LINMNOILVANYOINI
`
`MOCNIM
`
`
`
`LINENOLLVIANAOANI
`
`
`
`LINNNOILVIANAOSNI
`
`
`
`SNOYUNVTTHOSIN
`
`YHLOVAVHO
`
`NOILVINAOANICE
`
`
`
`LINDONISSHOOUd
`
`NOILV.LOY
`
`OI
`
`6S“Old
`
`Jun. 10, 2003
`
`Sheet 55 of 64
`
`US 6,577,330 B1
`
`OLI—~
`
`ASVaVIVd
`
`INOd
`
`OOT
`
`TOT
`
`cOl
`
`56
`
`56
`
`
`
`
`
`
`
`

`

`U.S. Patent
`
`Jun. 10, 2003
`
`Sheet 56 of 64
`
`US 6,577,330 B1
`
`FIG. 60
`
`3100
`
`3D-Window
`
`DISPLAY TEXT IN A 3D WINDOW
`
`DISPLAY TEXT IN A 2D WINDOW
`
`WHEN FONT TYPE CANNOT BE
`
`DISTINGUISHED DISPLAY
`CHARACTER USING THE SAME FONT
`
`3101
`
`FIG. 61
`
`;
`
`OW
`
`WHEN FONT TYPE CANNOT BE
`NGUISHED DISPLAY
`
`3200
`
`
`DIsSPLAYTrypINA3Dwing
`DISPLAYTExINA2DWInpow
`3202
`
`
`
`
`
`ARATEUSINGTHESAMEFONT
`
`3201
`
`
`o7
`
`57
`
`

`

`U.S. Patent
`
`Jun. 10, 2003
`
`Sheet 57 of 64
`
`US 6,577,330 B1
`
`FIG. 62
`
`RECEIVE WINDOW
`ROTATION INPUT
`
`51301
`
`EXECUTE WINDOW
`ROTATION PROCESSING
`
`$1302
`
`$1303
`
`TRANSFORM POST-ROTATION WINDOW AND
`COORDINATES FOR CHARACTERS IN THE WINDOW
`TO 2D USING PERSPECTIVE TRANSFORM
`
`CALCULATE FONT TRANSFORM
`PARAMETERS FROM PERSPECTIVE
`TRANSFORM RESULT
`
`51304
`
`$1308
`
`UNPROCESSED
`CHARACTER IN THE
`
`NO
`
`WINDOW?
`
`READ INFORMATION FOR A
`CHARACTER IN THE WINDOW
`
`
`53301
`COMPARE
`
`SIZE AND THRESHOLD
`
`VALUE FOR TRANSFORMED
`
`
`53302
`FONT CHARACTER
`
`
`
`IGNORE FONT TYPE
`
`
`INFORMATION IN
`
`
`THE CHARACTER
`
`
`READ FONT CHARACTER
`
`INFORMATION AND
`CORRESPONDING TO THE CHARACTER
`
`
`
`DISPLAY CHARACTER
`
`
`USING A SPECIFIED
`TRANSFORM THE FONT CHARACTER
`
`USING TRANSFORM PARAMETERS AND
`
`CREATE A CHARACTER IMAGE
`
`}
`
`51305
`
` F
`
`$1309
`
`58
`
`

`

`U.S. Patent
`
`Jun. 10, 2003
`
`Sheet 58 of 64
`
`US 6,577,330 B1
`
`O61
`
`O81
`
`LOAdNI
`AVTdSIC
`
`LINA
`
`IINM)
`
` LIN‘)
`
`NOILVNYO-ANI
`
`YALOVAVHO
`
`59
`
`ALVNIGHOOO
`NOILVWAOANICe
`
`WALSAS
`
`
`
`LIN()LNANTOVNVIN
`
`ONILYHYANOO
`
`LIND
`
`LINMNOILVNYOANI
`
`MOCNIM
`
`Ocl
`
`69“Old
`
`
`
`LINNONISSHOOUd
`
`NOLLV.LOU
`
`59
`
`
`
`
`
`

`

`U.S. Patent
`
`Jun. 10, 2003
`
`Sheet 59 of 64
`
`US 6,577,330 B1
`
`v9Old
`
`
`
`d'TdHAVIdSIGLIGHT
`
`
`
`KOCNIA(&VNILXaLAVIdS1¢
`
`40LNOYAAHLYVANLXaL
`
`
`
`SUVAdd¥MOGNTMAHL
`
`SIGNYadavTOOL
`
`WaSOLLNT
`
`(v9Old
`
`MOCNIMGEVNILXALAVASO
`
`
`
`@7T4H_AVMSIGUGSWMA
`
`FASOLLIQ
`
`dT#HAVIdSICUGCAAVA
`
`dqTdH=ASILdaT
`KONI(&VYNYLXadAVTdS1C
`
`HOLNOWdAHLaVANLXaL
`MOCNIMCEVNILXALAVS.
`SUVadd¥YMOCNIKHHL
`
`ol(NVdOavTOOL
`WASOLLINOTAATC
`
`qdv9Dla
`
`FASOLLAAMNSMA
`
`Vv9Old
`
`60
`
`60
`
`
`
`
`

`

`U.S. Patent
`
`Jun. 10, 2003
`
`US 6,577,330 B1
`
`Sheet 60 of 64
`
`FIG.65CENFEaCO
`
`FIG.65D
`
`FIG.65A
`
`61
`
`61
`
`

`

`U.S. Patent
`
`Jun. 10, 2003
`
`Sheet 61 of 64
`
`US 6,577,330 B1
`
`FIG. 66A
`
`FIG. 66B
`
`
`
`
`TTTTTT
`274)me
`
`ge, uaeed
`
`FIG, 66D
`
`GRAY-SCALE FONT
`
`NO
`NORMAL
`
`FONT FONT
`
`DSPLAyTgTN
`DSHLAVTeennames
`DISCARD UNREADABLE TR@®
`
`62
`
`FIG. 66C
`
`PLAYTEX"oeseee
`DISCARD UNREAD@@e
`
`
`
`
`
`
`62
`
`

`

`U.S. Patent
`
`Jun. 10, 2003
`
`Sheet 62 of 64
`
`US 6,577,330 B1
`
`dl9Old
`
`
`
`63
`
`63
`
`
`

`

`U.S. Patent
`
`Jun. 10, 2003
`
`US 6,577,330 B1
`
`Sheet 63 of 64
`
`FIG. 68A
`
`64
`
`64
`
`

`

`U.S. Patent
`
`Jun. 10, 2003
`
`Sheet 64 of 64
`
`US 6,577,330 B1
`
`
`
`OOpeeredMOCINING®
`
`
`
`AHHAVUMSICLICHWd
`
`
`
`
`
`
`
`NICSSIVLLSASONINOGNIMTYNOWNAANOO
`
`OOpoe&MOCNIN®
`
`
`
`
`
`SRVSMOCNLMRLVYOSMOCNIMNOWVONTdd¥40AIPIVAY1dVHOUHANNICQHLIAVASATIMYANTD
`
`
`
`
`
`
`
`SHELAKOSWHOTSSTVNINAALNOLLVAYOAN
`
`OVINV10NATIONSAHLNOUeNISAVTdSIC
`
`
`
`
`
`
`
`MOLINOAVTSSICVSYHOTSSALVaVdd¥AVIdSIC
`
`NAaaOSAVIdSIC
`
`
`
`WVMOVIGAHLNINOSHEANTIAVTdSIGC7dO
`
`
`
`
`
`
`
`YOLINOWAVIdSIGVSVHONSSALVaVddVAVTdSId
`
`
`
`ALTVGNHVHOIHMNiCOHLANVSfATIVENAD
`
`
`
`
`
`aRTVSMOCNAMVVC8OSMOCNIMNOLLVOTIdd¥10
`
`OVNINVJONAHASAHLNOdéNCYAVIdSIC
`
`
`
`
`
`SUALAdNOOS¥HONSSTYNINGELNOLVINIOANI
`
`
`
`
`
`NIISASWHLSASONTIMOCNIATVNOLLNAANOO
`
`
`
`dIdHAVIdSICJIGHAd@MOCNIM®
`NVMOVIGSHINIYNOSHLAMTAVdSG07dO
`
`ALTTVaYTdVHOIHMNICOHANASOATIVEENAD
`
`
`
`dVSMOCNIMVIVAa0SMOCNIMNOWWOMTddVdO
`
`
`
`“POVNVlONAAaOSLENOdéNIGHAVIdSid
`
`
`
`
`
`SYaLAdINODSVHONSSTVNINGELNOLLVNYOHNI
`
`
`
`YOLINOWNAVTdSICYSVHONSSA.LVaVdd¥AVTdSId
`
`
`
`
`
`
`
`
`
`
`
`NI.GHSASWISASONTIMOCNIMTYNOLLNAANOO
`
`"WVYOVICTHLNISNOSHESMTAVIdSIGC91dO
`dTHHAVIdSIGUGHWild
`
`SOLNONAVISICYSVHONSSUVakaAVTdSIC
`
`AOVIIAYcONASDsSELNOGeNILAVTdS10
`AAUatBTAHTNCLICTk
`
`"AVONSHENTENOFHLTHITARTdSIOCOT80
`
`69“Old
`
`65
`
`65
`
`

`

`US 6,577,330 B1
`
`1
`WINDOWDISPLAY DEVICE WITH A
`THREE-DIMENSIONAL ORIENTATION OF
`WINDOWS
`
`INDUSTRIAL FIELD OF USE
`
`The present invention relates to a window display device
`for displaying windows on the screen of an information
`processing apparatus, and in particular to a device for
`displaying windows in a virtual
`three-dimensional (3D)
`space.
`
`BACKGROUND ART
`
`Multiple window display devices are used to increase the
`performance of the man-machine interface in computer
`information terminals and the like.
`
`technique is disclosed in
`One example of a prior art
`Japanese Laid-Open Patent No 6-222899. Here, a multiple
`window display device improves display efficiency by dis-
`playing as many windowsas possible in the limited screen
`space available.
`FIG. 69 showsa display screen for this prior art example.
`In the prior art, windowsare placed in a virtual 3D space,
`with windowspositioned deep within the 3D space reduced
`in size. As a result, more windowscanbe displayed, making
`the screen appear larger than it actually is.
`However, in this prior art example, a window placed deep
`within the 3D space is displayed reduced in size, making it
`impossible to distinguish any of the display content of the
`window.
`
`DISCLOSURE OF THE INVENTION
`
`The present invention, a device for displaying windows
`placed in a virtual 3D space, is designed to overcome the
`stated problem. The object of the invention is to provide a
`window display device that increases the efficiency with
`which windowscan be displayed, making the screen appear
`larger than it actually is, and moreover, preventing a situa-
`tion in which the entire display content of a window is
`unreadable.
`
`The present invention aiming to achieve this object is a
`window display device displaying a plurality of windows on
`a screen. The window display device includesthe following.
`A display data holding unit holds display data representing
`display content of the plurality of windows. A position
`determining unit determines a position for each window in
`a virtual three-dimensional (3D) space, the positions being
`such that the windows are not parallel to the screen. A
`perspective projection unit transforms the display data to
`project the windowsonto the screen in perspective based on
`the determined positions. A display unit then displays the
`transformed display data on the screen.
`This enables at least one windowto be displayed inclined
`in the depth direction of a virtual 3D space. Even if the
`windowis positioned deep within the 3D space, the problem
`apparent in the prior art, where reducing the size of the
`whole window by the same amount meantthat all of the
`window display content became unreadable, can be avoided.
`The screen display area can also be used moreeffectively,
`and the screen made to appear larger than it actually is.
`Furthermore, information in the inclined windowthatis near
`to the front of the 3D space remains easily recognizable, and
`less display area needs to be used for the part of the window
`deep in the 3D space. Thus a window display apparatus
`capable of using the screen display area effectively, while
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`2
`ensuring that at least one part of the window display content
`is readable can be realized.
`
`Here, the window display device further includes an input
`unit acquiring indication from a user, enabling the position
`determining unit to determine positions according to indi-
`cations acquired from the input unit. This enablesthe user to
`determine the size of the entire window and the degree at
`whichit slants in the depth direction.
`The position determining meansalso determines 3D coor-
`dinates for each vertex of a window asthe position for the
`window.If the window is a rectangular plane, this enables
`the position of the window in the 3D space to be determined
`simply by the coordinates for its four vertices.
`The window display device further includes a window
`analyzing unit detecting, as a key part for each window, a
`part of the display content fulfilling certain conditions.
`Detection is performed based on display data held by the
`display data holding unit. This enables the position deter-
`mining unit to determine the position of windowsso that the
`key parts detected by the window analyzing unit are dis-
`played near the front of the virtual 3D space. The window
`can thus be displayed inclined in the depth direction, so that
`the key part of the display contentis near the front of the 3D
`space. Therefore, even if a plurality of windowsare being
`displayed, the user can grasp the type and display content of
`each window at a glance.
`The window analyzing unit also determines whether a
`title is included in the display data for each window and,if
`a title is included, detects a part of a window in which the
`title is displayed as the key part. This enables the user to
`easily recognize window types.
`The window analyzing unit also determines whethera text
`and an indication for displaying the text horizontally are
`included in the display data for each window,andif the text
`and the indication are included, detects a left side of a
`window when viewed from the front as the key part. This
`enables the user to easily recognize the type and general
`content of a horizontally-written document displayed in a
`window.
`
`The window analyzing unit also determines whether a
`text, and an indication for displaying the text vertically are
`included in the display data for each window,andif the text
`and the indication are included, detects a right side of a
`window when viewed from the front as the key part. This
`enables the user to easily recognize the type and general
`content of a vertically-written document displayed in a
`window.
`
`includes an automatic
`The position determining unit
`alignment unit for determining the position of a plurality of
`windowsso that the windowsare placed in a predetermined
`arrangement, where at least one windowis notparallel with
`the screen. This enables a plurality of windows that were
`displayed scattered around the screen to be displayed in
`alignment, using the screen display area more effectively
`and allowing all of the windowscurrently displayed to be
`viewed at a glance.
`The position determining means further includes a unit
`determining a position according to an indication acquired
`from the input unit, so that a window displayed in the
`alignment position is newly displayed facing the front; and
`a unit determining a position according to an i