USOO9696868B2
`
`(12) United States Patent
`Bakhash
`
`(10) Patent No.:
`(45) Date of Patent:
`
`US 9,696,868 B2
`*Jul. 4, 2017
`
`(54) SYSTEM AND METHOD FOR PROVIDING
`THREE-DIMIENSIONAL GRAPHICAL USER
`INTERFACE
`
`(71) Applicant: Ezra Eddie Bakhash, Palo Alto, CA
`(US)
`
`(58) Field of Classification Search
`CPC ............. G06F 3/04815; G06F 3/04817: G06F
`3/0482; G06F 3/04845; G06F 17/30572;
`(Continued)
`References Cited
`
`(56)
`
`(*) Notice:
`
`(72) Inventor: Ezra Eddie Bakhash, Palo Alto, CA
`US
`(US)
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 356 days.
`Rplent is Subject to a terminal dis-
`
`(21) Appl. No.: 14/614,708
`(22) Filed:
`Feb. 5, 2015
`
`U.S. PATENT DOCUMENTS
`5,319,387 A * 6/1994 Yoshikawa ........... G06F 3/0304
`5.428.735 A *
`6, 1995 Kahl
`Gooris
`law i - - , K
`W k - - -
`kavku.k.
`TO7 (999-004
`(Continued)
`Primary Examiner — Ting Lee
`(74) Attorney, Agent, or Firm — Fitzsimmons IP Law
`(57)
`ABSTRACT
`Methods and systems are provided for an improved three
`dimensional graphical user interface. In one embodiment,
`the method generally comprises: receiving a plurality of
`inputs from a user and opening a plurality of applications in
`response thereto, wherein outputs from each application are
`displayed (e.g., individually) within a two-dimensional
`space. The method further comprises allowing a user to
`switch between the plurality of applications by generating
`images of each output, and displaying those images (e.g.,
`collectively) within a three-dimensional space. The user can
`then interact with a particular application by selecting a
`corresponding one of the images displayed within the three
`(2013.01)
`dimensional space. In one embodiment, the method further
`(2013.01)
`comprises replacing the output(s) in two-dimensional space
`Continued
`with the images in three-dimensional space, and replacing
`(Continued)
`(52) U.S. Cl.
`the images in three-dimensional space with the output from
`CPC ........ on, .t GF'io tt the selected application in two-dimensional space.
`(Continued)
`20 Claims, 30 Drawing Sheets
`
`(65)
`
`Prior Publication Data
`US 2015/O149965 A1
`May 28, 2015
`O
`O
`Related U.S. Application Data
`(63) Continuation of application No. 14/503,142, filed on
`Sep. 30, 2014, now Pat. No. 9,304,654, which is a
`(Continued)
`
`(51) Int. Cl.
`G06F 3/048
`G06F 3/0482
`
`
`
`http:ftwww.
`s
`ise
`
`assiss
`
`Rewind Play Forwart
`ticites,
`pig
`
`As s
`
`344- 34
`
`348
`
`Eisne
`
`APPLE 1001
`
`1
`
`

`

`US 9,696,868 B2
`Page 2
`
`Related U.S. Application Data
`continuation of application No. 127751,879, filed on
`Mar. 31, 2010, now Pat. No. 8,881,048, which is a
`continuation of application No. 1 1/531,676, filed on
`Sep. 13, 2006, now Pat. No. 7,735,018.
`Provisional application No. 60/717,019, filed on Sep.
`13, 2005.
`
`(2013.01)
`(2006.01)
`(2011.01)
`(2006.01)
`
`Int. C.
`G06F 3/0484
`G06F 7/30
`G06T I5/20
`GIB 27/34
`U.S. C.
`CPC. G06F 3/04845 (2013.01); G06F 17/30572
`(2013.01); G06F 17/30864 (2013.01); G06F
`17/30873 (2013.01); G06T 15/20 (2013.01);
`GIIB 27/34 (2013.01)
`Field of Classification Search
`CPC ......... G06F 17/30864; G06F 17/30873; G06T
`15/20: G 11B 27/34
`See application file for complete search history.
`
`(60)
`
`(51)
`
`(52)
`
`(58)
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`6,121,969
`
`A *
`
`9, 2000
`
`6,499,029
`
`B1* 12, 2002
`
`6,577,330
`
`B1* 6, 2003
`
`6,725,427
`
`B2 *
`
`4, 2004
`
`6,768,999
`
`B2 *
`
`7/2004
`
`2001/0050687
`
`A1* 12, 2001
`
`2002fOO33848
`
`A1* 3, 2002
`
`2003/O164827
`
`A1* 9, 2003
`
`2003/0179231
`
`A1* 9, 2003
`
`2004/02684.51
`
`A1* 12, 2004
`
`Jain ..................... G06F 3/04815
`382,305
`Kurapati ........... GO6F 17,30817
`707/750
`Tsuda .................. G06F 3/04815
`715,781
`Freeman ........... G06F 17/30011
`TO7/E17.008
`Prager ................... A63F 7,0676
`707,999.OO2
`Iida ....................... GO6T11 206
`345,581
`Sciammarella ....... G06F 3/0481
`T15,838
`Gottesman .............. GO6T 15.20
`345/419
`Kamiwada .......... G06F 3/04815
`715/757
`Robbin ................. G06F 9,4443
`715.738
`
`* cited by examiner
`
`2
`
`

`

`U.S. Patent
`
`Jul. 4, 2017
`
`Sheet 1 of 30
`
`US 9,696,868 B2
`
`10-N
`
`32
`
`Load Custom Script and its loon to Alter
`Application Logic of 3D Cartesian Space
`through Programmatic Access
`(External Proto Node) Represented by
`Interactive con on interface Panel
`
`
`
`User input
`
`Geometry in
`New Viewpoint
`or Perspective
`
`Redraw Scene
`
`Visit Viewpoint
`
`Record Viewpoint or
`x,y,z Coordinate
`Add Desktop
`Window Command
`
`Add Browser
`
`Add Document or
`Application Command
`
`3O
`s:Ey, I
`or trid User inpu
`on interface Panel to
`Execute Appropriate
`Helper Application
`
`Add ext
`Command
`Add Picture
`Command
`
`42
`
`50
`
`CE)
`
`(E)
`
`G
`
`62
`
`GE)
`
`72
`74 O
`
`Add Video Command
`
`Record Sound
`Command
`
`da Map Command
`
`Add Map Command
`Add 3D WRML or
`X3D File Command
`Save Space/Email
`Space Command
`Add Advertising to
`3D Scene
`
`88
`
`92
`
`94
`
`104
`
`F.G. 1A
`
`3
`
`

`

`U.S. Patent
`
`Jul. 4, 2017
`
`Sheet 2 of 30
`
`US 9,696,868 B2
`
`(P)
`
`(Q)
`
`46
`
`Draw interactive 3D
`lcon/Geometry in Scene
`to Represent Viewpoint
`with Viewpoint Name
`
`(A) 10
`36
`3.
`34
`- Road Program Prompts End
`End SEE.Oad User to Locate Script Fite
`(D Application) confrom froSier,
`Network, internet
`Input interface Panel
`or other location
`Load x,y,z Position or
`Prompt User to Choose Viewpoint Name
`from Compass or Click on icon of Previously-Viewpoint from Compass
`Recorded Viewpoint in 3D Cartesian Space
`File and Bind to it
`GE) 44
`Prompt User to
`Input Viewpoint
`Name(s) to be Added
`to Compass File
`
`
`
`Store x,y,z Position
`and Viewpoint Name
`within 3D Cartesian
`Space to Compass File
`
`
`
`Initiate Open File Class or other Web
`Service, AP, or Method for Loading
`Data from Local Computer,
`Network or the World Wide Web
`Or
`Allow Drag and Drop of File, Document
`or Application into 3D Cartesian Space
`from 2D Desktop
`
`76
`
`78
`
`
`
`
`
`
`
`
`
`82
`Draw Text, Map, Picture, Video,
`WRML, X3D or other Content
`into 3D Cartesian Space
`
`Draw interactive 3D confoeometry in
`Scene to Represent Sound or Music
`that if Clicked with Play Sound/Music
`8O
`96
`Prompt User to input Name of 3D
`Save Space on Local Computer
`Cartesian Space to Save or Email.
`and/or Our Company Server (FTP),
`Add Name to Compass File
`Server Files can be Public or Private
`Determine Subject of 3D Cartesian Space by Scanning all Words
`Associated with or within the Fife Names, Files, html Pages,
`Viewpoint Names, Hyperlinked Files or any other Available
`Information Associated with items within the 3D Cartesian Space
`or Words Entered into the input interface
`
`98
`
`
`
`GK)
`
`
`
`4
`
`

`

`U.S. Patent
`
`Jul. 4, 2017
`
`Sheet 3 of 30
`
`US 9,696,868 B2
`
`
`
`
`
`
`
`
`
`
`
`40
`End User is Prompted to Click
`interactive icon that Triggers Script is
`Dynamically Added to interface Panel to Hinteractive con on Interface Panel
`initiate Command Using XML. Encoding
`to initiate Program Altering Script
`There is no Control
`for this information
`
`Open Web Open File?
`Page
`Document
`
`Launch
`Application
`
`Window
`
`Execute Proper OS Control Specific to the
`Kind of information Requested which
`will Act as a Mirror to its 3D Representation
`
`Map Visual Output of OS Control, Bit Map
`or API onto Arbitrary 3D Geometry
`
`Request
`Periodically Capture the
`on Screen Output of
`Window for Said
`Program/information as
`a Bit Map image
`
`Store Bit Map in
`a Frame buffer
`
`Create Device
`input Event
`Handler in 3D
`
`Pass Mouse Clicks, Cursor Position,
`Keyboard input and Movements from
`Control Mig on 3D Geometry to the Mirror
`Control Running on 2D Environment
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`1 O8
`
`
`
`Sort Words by Frequency to
`Determine Subject of 3D
`Cartesian Space by identifying
`Most Frequently
`Appearing Words
`
`Generate Table of Contents Page in the Form of an html Page that is Stored on
`Company Server Next to Saved Space File that Stores Space Name, Hyperlinks
`in 3D Space, Description, Picture(s) of 3D Space with the Purpose of being
`an index for the 3D Space to be Crawled by the Major Search Engines as Part
`GD of their Web Crawling/indexing. Place Subject (See item "A") of 3D Space into
`Meta Tags of html Page. Embed Option in html Page to Download PROGRAM
`Active X Control within html Page to Auto install Saved Space from
`Company Server for which T.O.C. html Page was Generated and
`Execute on their Computer with the Click of One Button on html T.O.C. Page
`1 OO
`Email Table of Contents html Page to Recipient(s) and Embed
`Option for Recipient to Download PROGRAM ActiveX Control within
`102
`html Page to Auto install and Load Saved Space from Company
`Server for which T.O.C. html Page was Generated and Display
`on their Computer with the Click of One Button off html Page
`Return Appropriate text,
`Picture(s), Video(s), Sound(s),
`Music, Hyperlinks or other
`Advertisement Content from
`Advertising Server, Web
`Service or other Location
`Draw text, Picture(s), Video(s),
`Sounds(s), Music, Hyperlinks or
`other Ad Content from
`Advertising Server or other
`Location into 3D Cartesian Space
`
`Send Subject of 3D Cartesian
`Space via xmi File of Most
`Frequently Appearing Words to
`Advertising Server through
`internet Request
`
`
`
`FIG. 1 C
`
`
`
`5
`
`

`

`U.S. Patent
`
`Jul. 4, 2017
`
`Sheet 4 of 30
`
`US 9,696,868 B2
`
`20-N
`
`
`
`
`
`
`
`User input
`
`Recalculate Geometry
`in Scene Based on
`New Viewpoint
`or Perspective
`
`F.G. 2
`
`6
`
`

`

`U.S. Patent
`
`Jul. 4, 2017
`
`Sheet 5 of 30
`
`US 9,696,868 B2
`
`jOUODOUS!a3,
`
`byl
`
`cl
`
`jsonbayJospayeqiulpuetajsuesy
`
`
`
`
`
`ysonbayJasnsinideg
`
`
`
`ssaooyonewuebogySnoluy
`doysegdz0)a0edg-qewoy
`
`(uoNeziwo\snyWweibolgUopase)
`
`soedsGEUIyTIM
`
`9éL
`
`
`
`uonesiddyyoune7
`
`QESAHOBIa}UYBBS)
`
`
`
`adedsuelsayes
`
`col
`
`OcL
`
`vel
`
`
`
`
`
`
`
`
`
`
`
`UONBUJOJUTSIYIJO}
`
`paysanbayUOReLUOjUTJOpunyBuyBAIaNAY0}jO.NUODSOJado1qauUlUUa}IG
`
`J9yjngaweeuldeyy#@a0
`
`8yl
`
`
`
`AyawoegaeAesquyOUjdJOdeytg‘101U0DSOJOIndjn¢jensindey
`
`44)
`
`
`obew]deyjig&se
`
`UOHeUO}U]AUeIBOIdPIESJO}
`
`
`MODPUIAAJOINdINGUaaIISUO
`ayyaunjdeaAyeopouad
`
`
`
`JBUIOyeep
`
`
`
`
`
`ysanboyMOPUIM,yuawno0g
`
`O€L
`
`Ort
`
`
`
`uojejussadayGESI}!0}JOLY8SeJOY[TIMYOUM
`
`
`
`pajsanbayUOIEUWOsUyJOpulyBuy04DyloadsjosUODSOJedosyanNoSxXy
`
`
`
`
`
`
`
`‘UOIHSOdJOSING‘SyaIjQssnoyWssed
`
`€‘Sid
`
`
`
`juawuaNAUACzuoBuluunyjonuog
`
`
`
`JO.UIIN)SU}0}AjatuOasGEUOdey,joquoD
`
`WO}SJUBWIBAOWpuejndulpueoqhay
`
`
`
`GEUlJejpuReHyusAy
`
`
`indujaoiAegaye015
`
`
`
`OSt
`
`7
`
`
`
`
`
`
`
`
`
`
`
`
`

`

`U.S. Patent
`
`Jul. 4, 2017
`
`Sheet 6 of 30
`
`US 9,696,868 B2
`
`
`
`
`
`
`
`
`
`
`syINsayYosessjoulBUEWo4ysynseyYoreagBulsuedJO}aiqisuodsayjdiiogWo}sN5peor
`
`
`
`pjatyyxe_seBedqajys,uiGuoQyoessoul
`abe[O.UODJBSMOIgGAA,9}B919
`
`
`
`
`
`
`coalewfas|Nsay EsYoseagJoUIGUOJosjesOo}JESIdwWolY
`
`paysauueyWYNYyoesJoy
`
`
`
`
`
`899,4991-%991-S991|-Y99L-®991"991
`
`
`
`
`Jasp)woBuysynduyysenboyyoseessAlaooy
`yepueabeuosyuledAH
`ALILNYVNDMOVLSSBsiuj
`
`
`joAnuentsurwiajaq
`asojspuea6eyuo
`
`syulLedAyAjjuap
`
`
`
`
`
`weiGoldouleyeqyorees
`
`
`
`Buunjdey10)pouauluuajeq
`
`pol
`
`cg
`
`89L
`
`bog1
`
`081BL)
`
`OL|
`
`pL}cL}
`
`OLL
`
`
`
`weiBoiguojeoyddy
`
`Soepoyy]
`
`Buysjnduyjsenbayyoureeswoy
`
`
`
`jsenbayYyolieaspainydesJa;uq
`
`
`
`
`
`S,UIBUGYyoleasJo}YF48]US
`UusalDSHOUlAbedGap,
`
`ebegJasmolgaan
`
`abegJasmoigGana
`U8319SJOMeIq
`
`8st
`
`98t
`
`cB
`
`v8
`
`v61
`
`002
`
`
`
`abegGannasied
`
`
`
`yuipedALyoes0}
`
`c6b
`
`861
`
`VP‘Sis
`
`ATAVINVA
`
`Aeuyueul
`
`afeyJasmolgMONUl
`
`
`S}INSeYaAIaDay
`
`abewesSuyeinwa
`
`Aqyoueasyonpuo5
`
`winjay
`
`O61
`
`8
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`

`

`U.S. Patent
`U.S. Patent
`
`Jul. 4, 2017
`Jul. 4, 2017
`
`Sheet 7 of 30
`Sheet 7 of 30
`
`US 9,696,868 B2
`US 9,696,868 B2
`
`sueeddy}yeu)YONSSBedGaAAIKONJO)AsJoWOaS)a}EINo}eosy
`
`
`
`
`yiSuyeagAqyor;sulAemyJayying
`
`
`
`
`
`goedsq -€UlUONISOY$3}!Bugejsues,pueumoq
`
`
`9U}10}BdJONUODJESMOlgGAA}XONJOINd}Nojensi,MeIG
`
`
`$1Aeuyjopuzjunaoedgueisayeq-¢au)ulaBeysyinsoy
`
`
`
`
`yoJedsS,UIBOQYoeEgWOYyUIEdAHBy)0}pajsaUUODTWN
`
`payoray
`
`902702
`
`
`
`jou,sdepa}U|S,WeiG04UOYONGUBnoOIU)adedgURIsaLeEDGEUIUMeIGaq0}sabeyQaMjoyoR}S
`
`
`
`
`
`
`JensipJojnoAe7Ajpoyy0}weiBorg0}ajejdwiaywojsngpeoo}asfpuyMo}
`
`
`
`
`UeUlPayo[deedGan,Se]Ou}O}UITJey)SABEGearyBy)W]epuededUBISBHEDGESl)Ulpanoid
`
`
`
`
`
`wesbeigHe}SeQINESyUrT/Usur]Bag-BBegJONUODJasMolgqa;y}UALIND0}9edSJUSDelDY
`
`
`
`
`
`
`
`
`
`
`962dGaMJSE724}UOpuno,aleyey)SYUIPEdAH94)JoSabeyGaAAOY)IIEOld0}188A}PUYMOT|Y
`
`
`
`jdiwold Ole
`yORISeNsiAMANGumersAqaBeds}NSayYo1easjo19gIXENJO}Ss9d0lqjeaday0}Jaspy
`
`
`
`
`
`Bbc
`
`
`
`
`yOR}Ssensi,ulsebegjoAeuyposwesbeigaag-soeyayu)woysn5
`
`
`
`
`
`
`yBnowy)yoRsjensi,ulsabeJasmosgGayLOS0}JasnpuyMoly
`
`peyoeeysiAnny
`
`
`safedJasmosgGapJOHOEIS[ENSIASU}JOjxapUSeJOYOo}avedsUBISeLED
`
`
`
`
`GEulUMBIGSq0}Y9R]SJeNsIAeulBegysul4seabeysynsayMeIg
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`ZOZ
`COZ
`
`
`
`
`
`
`
`JasAgpaydwoseUO}B007JAYJOJOJOAJASqENA“OMEN‘VAINGWOD12907O}a}14BALES
`
`
`
`
`
`
`
`av‘Sls
`
`9
`
`
`
`
`
`
`
`

`

`U.S. Patent
`U.S. Patent
`
`Jul. 4, 2017
`Jul. 4, 2017
`
`Sheet 8 of 30
`Sheet 8 of 30
`
`US 9,696,868 B2
`US 9,696,868 B2
`
`F.G. 5A
`FIG. 5A
`
`
`
`
`
`
`F.G. 5B
`FIG. 5B
`
`10
`
`10
`
`

`

`U.S. Patent
`
`Jul. 4, 2017
`
`Sheet 9 Of 30
`
`US 9,696,868 B2
`
`a-te
`
`
`
`SEs
`
`718
`
`720,760
`
`FIG. 5C
`
`11
`
`

`

`U.S. Patent
`
`Jul. 4, 2017
`
`Sheet 10 of 30
`
`US 9,696,868 B2
`
`ove
`
`
`
`yaeISSNonatdulabejsityjoyuodmal,dnasoj9o}pug
`
`
`
`
`
`bye
`
`
`
`
`
`YORTSjensiaulabese]jojulodmaia,,dnasoiyojpuig
`
`Ob~
`
`
`
`
`
`yoRTSfensi,utebedXanJojulodmaiAdnasoyyo}puig
`
`
`
`J9piCpayloadsulyoR}9AeuyyOS
`
`
`
`orep22
`
`yor}oe
`
`
`
`rensieenginabeabeabegabey
`
`
`
`
`yogSNOIADeSe)}XONSNOIADIg1SJI-4
`
`
`
`ynduysaspuo
`
`8c9¢be?oC0&282922
`
`
`
`
`
`yOR}SIXANUlBegJsulyJoJulodMal,dnasojgo}pug
`
`12
`
`8h
`
`
`
`aoedsuRIseyeDGEUlB}eUIPI0NDZ'A'Xplod9y
`
`
`
`
`
`9°Sis
`
`LpSZyoole
`
`
`
`0szYORISlensi,ulaHegJSi1JJoUIOdMalAdnasojn0}pug
`
`
`
`
`
`ATA
`
`
`
`yOR}SJENSIAU!BBegSNOIAaL,JoJWIOdMAIAdnaso}D0}puig
`
`
`
`
`
`
`
`
`
`jaUegSOBpayU}UOjndulJesf)puzJO}
`
`
`
`JO|PUBYJUaATURIS
`
`0e2
`
`266
`
`
`
`peseguonoyayeudoiddyaynoexy
`
`
`
`
`
`12
`
`
`
`
`
`
`
`
`
`
`
`
`

`

`U.S. Patent
`
`Jul. 4, 2017
`
`Sheet 11 of 30
`
`US 9,696,868 B2
`
`260
`End User Clicks Close Button on any Texture Map
`of Web Browser Control Represented in Visual Stack
`
`Remove Web Browser Page
`from Visual Stack
`
`262
`
`264
`
`Animate Forward Page after
`Removed Page to Replace
`Position of Removed Page in Stack
`
`266
`Animate Page after Previously
`Animated Page to Original
`Position of Previously
`Animated Page Before it was Moved
`
`268
`Animate Forward Next Page in
`3D Visual Stack to Replace Position
`of Previously Animated Page
`
`27O
`
`Animate Next Page in
`Stack to Position
`of Previously Animated Page
`
`272
`
`
`
`274
`Exit loop if the End of Visual
`Stack or last Element in
`Array is Reached
`
`F.G. 7
`
`13
`
`

`

`U.S. Patent
`
`Jul. 4, 2017
`
`Sheet 12 of 30
`
`US 9,696,868 B2
`
`
`
`
`JualgAeAzWoy,JUBAZ[eNsi,,
`JayndwosJ39A19¢9Y}0}
`
`
`
`
`
`MONUdeSJO}sBueygpeziewnor
`
`YORJO}SqYoRRYSnd
`
`SUalIQUlaBueYD||Sjus}OulaBueYD|]syuayDuyeBueyD||syuaygulafueYD|]sjua_DuleBueyd|)sjuayoulaBueuD
`
`
`
`
`
`
`
`yBnoiy)payeoipuy||yBnoiuypayeojpuy||yBnosy}payeoipuy||yBnoujpayesipuy{|uBnowuypajeaipuy||yBnozuypayeoipuy
`
`
`
`
`
`
`
`
`JUBI|DJES]aidyINW||Al|DsasF]adgINW||JeIDJesnaidyinyy||JUaIDJespsidyiNy|{yUayDJas/aidniny
`
`
`s}goedsjenyi,s}adedsjenylAs|soedsJeni,siaoedsJeni,s!a0edsJeni,S|aoedsjenna
`
`
`
`
`
`
`
`
`
`
`JENUIAPEPeO|UMO||[ENUIAPapeo|UMO||[ENLIAPAPEO|UMad||/ENLIAPepeajuMmog||jeEnLIApapeojumoq
`
`
`
`
`
`
`
`
`
`USATJENSIA,,WJUSAF[ENSIA,,wWUBAZ|ENSIA,,nUBAZJENSIA,USAFJENSIA,,UA[ENSIA,
`
`
`
`
`
`
`
`
`Bursnsajndwo9BuisnsayndwoaBuisrsayndu045Buisnsayndwo4Buisnseyndwor
`Auejo8uDAuewjo8uQAueyJO9u0AueyJ0supAueyyJo2u0
`
`
`
`
`0}sabueyDpazijeusnorJoJajndwoyjusi|D0}aijjeulBuoysng
`
`
`B|l-4UMES}!ULJUBAfeNsiA,YoeswoseBueyDaziewnor
`
`
`
`
`payeuibucseGueyDaieymsayndwogBuipnjoxyayyjeulbuC
`
`J8AIBSUOSdlI4
`puealySaaigoayJanlag
`
`
`
`OsadedsjenuiApaves
`yissaooyuedauokuy
`
`Siy}0}TNseUsiiGnd
`
`
`PaysiiqndpuepayedStadedsjenyiAjeuIBUO
`
`PUZWO}JBALBSO}
`
`Jayndwo9sasr)
`
`v8¢C82
`
`8“Sid
`
`b262
`
`
`
`JUSIIDJeseidainyy
`
`AueyJosuo
`
`'o6z
`
`JENLIApapeojumog
`
`
`
`Buisnsayndwo
`
`aoeds
`
`14
`
`14
`
`
`
`
`
`
`
`
`

`

`U.S. Patent
`U.S. Patent
`
`Jul. 4, 2017
`Jul. 4, 2017
`
`Sheet 13 of 30
`Sheet 13 of 30
`
`US 9,696,868 B2
`US 9,696,868 B2
`
`
`
`6‘9l4
`
`
`
`areSore@
` prewosAelgpumeyaSowone_meh
`@
`
`
`
` elolave
`
`15
`
`15
`
`

`

`U.S. Patent
`U.S. Patent
`
`Jul. 4, 2017
`Jul. 4, 2017
`
`Sheet 14 of 30
`Sheet 14 of 30
`
`US 9,696,868 B2
`US 9,696,868 B2
`
`
`
`i:
`:
`A :
`
`©
`
`|
`
`|
`|
`
`s
`
`16
`
`16
`
`

`

`U.S. Patent
`U.S. Patent
`
`Jul. 4, 2017
`Jul. 4, 2017
`
`Sheet 15 Of 30
`Sheet 15 of 30
`
`US 9,696,868 B2
`US 9,696,868 B2
`
`
`
`bb“Std
`
`annydyy
`
`
`
`v g g g o
`
`|
`
`H}pastyMON-hege-utooAeqoawys-diy@aet
`
`WOONGW-woousumMuMrdny(m-.
`SaOARy|Melog
`
`Ory
`
`17
`
`17
`
`

`

`U.S. Patent
`U.S. Patent
`
`Jul. 4, 2017
`Jul. 4, 2017
`
`Sheet 16 of 30
`Sheet 16 of 30
`
`US 9,696,868 B2
`US 9,696,868 B2
`
`FIG.12
`
`File
`
`
`
`ViewFavorites
`
`E.
`
`(g
`Bi
`s
`[Jz
`Bi
`
`3.
`464
`
`18
`
`18
`
`

`

`U.S. Patent
`U.S. Patent
`
`Jul. 4, 2017
`Jul. 4, 2017
`
`Sheet 17 Of 30
`Sheet 17 of 30
`
`US 9,696,868 B2
`US 9,696,868 B2
`
`
`
`VEL‘Old
`
`* * * * * *
`
`b-SBES18
`
`
`
`
`
`SUNGHOSATINTYUNST1WOIdMOH
`
`sowoneyna,
`
`
`jeVeClalssaemeaERETes]ioSTs1O]-o[BarsOeeso
`Liraoeoa
`vogenesad(@©smoysAeydsipensosoumswwapsksGunezadoUREeynduicoApueuTDextoed
`
`
`
`
`
`
`Aum0S073pewon)anyyeWODENLpLEWEREmHJOSomoyUEeeddeOF
`
`1p0}vogoelgejouseyTREESeppgvoumoiyfudesausuopaasd
`
`
`
`
`Aang30‘sreuroyeuspareyaAut20juowncopjuayedeuorstacadsapJOuojonposdat
`
`
`
`
`
`
`
`
`
`
`tyBuédos0)pelgnsjeustewsureIODKH@WNIODJusEdjeuOISHostatSOONTHSRAIOS$
`
`
`
`TWNOIGN3NIKDSAUHLONIGIAOUdHOdGOHISNONVWALSAS
`zTREORagyesKeIsdo
`‘syuounoopsetg58tonswiding
`
`onoosetsqufksreSuessoswrreoIng‘KgurieDseKpO
`
`
`
`
`
`
`Soptuetadeverson“mauLApgLGZa&apis(2)
`yequry@Ind't-sBormeg”yeodBureeuryE71
`
`
`IpuruojensiBayeqopy[3]
`
`
`
`
`
`SOVIUALNINSSNTWOMdvud
`
`
`uny'suoenesoy-
`
`uacxog[6
`
`byyounsy-ueseds
`
`spd'ge60g1eoroau)EZ,
`
`oopieuyusiedGF
`
`ay>ssaig
`
`premiogAidpumayBe
`
`
`
`seoedgpanesAw[-)
`
`19
`
`19
`
`

`

`U.S. Patent
`
`Jul. 4, 2017
`
`Sheet 18 of 30
`
`US 9,696,868 B2
`
`
`
`:sseuppy
`
`20
`
`

`

`U.S. Patent
`U.S. Patent
`
`Jul. 4, 2017
`Jul. 4, 2017
`
`Sheet 19 Of 30
`Sheet 19 of 30
`
`US 9,696,868 B2
`US 9,696,868 B2
`
`
`
`vLSid
`
`oo0'OZE$818
`Jo}sbebuon,
`
`LUUOWINgS
`
`Saoneymarxayy
`[seen[vojeanpa|Aaveung|seynsnpul|suondg|spuog|$32.3|spunremnw|sxonig|seiswren|meningpep|ESreeeaaTeesFemgeS
`2“SEEIZ]aaySACSECBT[spe
`ATWadNosweg*
`
`ey]
`
`yareagsoueuig|dmpojoqus(Gg)
` HSV2ols+S30VHL rysOWaLMYfa]3304oSam
`
`
`frreovejsodjexdeyyooGuipry
`
`
`pOaqsatuoyoufryGmFoquiatdagBug
`L-ae
`[×]
`
`aridabueyy
`
`pur3
`
`Bea
`
`21
`
`21
`
`

`

`U.S. Patent
`U.S. Patent
`
`Jul. 4, 2017
`Jul. 4, 2017
`
`Sheet 20 of 30
`Sheet 20 of 30
`
`US 9,696,868 B2
`US 9,696,868 B2
`
`
`
`sewoaey=Met
`
`
`
`
`
`Bobouoyyays“2249UBynsYOON”
`
`S|
`ols
`
`CK)ae|
`
`22
`
`22
`
`

`

`U.S. Patent
`U.S. Patent
`
`Jul. 4, 2017
`Jul. 4, 2017
`
`Sheet 21 of 30
`Sheet 21 of 30
`
`US 9,696,868 B2
`US 9,696,868 B2
`
`CI)FIG.16A
`
`Eile
`
`od
`Be
`ai
`
`ViewFavorites
`
`i.
`
`
`
`
`
`
`
`v A C3
`
`3) O
`
`
`
`r O
`tr
`
`S.
`cy
`
`23
`
`23
`
`

`

`U.S. Patent
`U.S. Patent
`
`Jul. 4, 2017
`Jul. 4, 2017
`
`Sheet 22 of 30
`Sheet 22 of 30
`
`US 9,696,868 B2
`US 9,696,868 B2
`
`
`
`aol‘DI4
`
`[Gy SayLOAR=MOI,
`
`
`
`24
`
`24
`
`

`

`U.S. Patent
`U.S. Patent
`
`Jul. 4, 2017
`Jul. 4, 2017
`
`Sheet 23 of 30
`Sheet 23 of 30
`
`US 9,696,868 B2
`US 9,696,868 B2
`
`V2L‘SONAI
`
`
`
` sayorey=MOK
`
`25
`
`25
`
`

`

`U.S. Patent
`U.S. Patent
`
`Jul. 4, 2017
`Jul. 4, 2017
`
`Sheet 24 of 30
`Sheet 24 of 30
`
`US 9,696,868 B2
`US 9,696,868 B2
`
`
`
`aZt
`
`‘9ld S8|uoAR-yMOIR
`
`26
`
`26
`
`

`

`U.S. Patent
`U.S. Patent
`
`Jul. 4, 2017
`Jul. 4, 2017
`
`Sheet 25 of 30
`Sheet 25 of 30
`
`US 9,696,868 B2
`US 9,696,868 B2
`
`
`
`DLb
`
`Sid
`
`27
`
`27
`
`

`

`U.S. Patent
`U.S. Patent
`
`Jul. 4, 2017
`Jul. 4, 2017
`
`Sheet 26 of 30
`Sheet 26 of 30
`
`US 9,696,868 B2
`US 9,696,868 B2
`
`
`
`
`
`
`
`“PIEMIO.“Kel|puimay
`HOB
`seyioney=max
`
`Ove
`
`28
`
`28
`
`

`

`U.S. Patent
`U.S. Patent
`
`Jul. 4, 2017
`Jul. 4, 2017
`
`Sheet 27 Of 30
`Sheet 27 of 30
`
`US 9,696,868 B2
`US 9,696,868 B2
`
`
`
`6Old
`
`REQSUSOMAR
`
`
`yaleegSMAN]:49M8u}YoeagWOHgFOADGUMtocudHal
`
`paeunofsauaulHe189
`SOULYO,MONSUL
`
`
`yeSwO}]SMOUpUly
`
`
`“HUANSOTSSOULHOAMONOff
`““HAOK‘hapAreaeoes10)sia
`
`_CATEUTlopeUaypuesou
`
` HOSUOAB;USUAUIONelytePeaunoksmeuOU]12105)FULWOTMONaaLwoakeqown
`
`
`
`~"yoruy‘smouAeprty“sauByeryouqndGuay‘oquy
`
`
`
`TEPAORSOOMORRWoo
`AegaUdSMaN
`
`woo'AeqaMMM
`
`
`WO"SMQUDUIESYOOTSAAA,
`APMPOULIOH-HOMABSOF
`
` ‘smoHPSTUIESTOYCoyandWoo
`
`
`MO){S823SIPPYOL)Pua
`
`{OWS}WaySMOLGUI]}8E)
`
`
`““SUONETSOlpaESNOURA
`
`pods‘leyjeom
`sMeNpUESYOOIS
`
` BENEERRWEDDeseLUInAUAA,
`
`}PRIeHOyROU
`
`18SUI}SHOUPUY
`
`“HNO‘AepAlanaayesJ0jsuo}
`
`
`UOI{IILGJ8A0UIA,“SeduMO}
`
`
`
`29
`
`29
`
`

`

`U.S. Patent
`U.S. Patent
`
`Jul. 4, 2017
`Jul. 4, 2017
`
`Sheet 28 of 30
`Sheet 28 of 30
`
`US 9,696,868 B2
`US 9,696,868 B2
`
`02‘Did
`
`
`
`seytoaeymail
`
`GO
`
`oid
`
`
`
`
`
`[×] [37] [5]
`
`cSv
`
`OVP
`
`30
`
`30
`
`

`

`U.S. Patent
`U.S. Patent
`
`Jul. 4, 2017
`Jul. 4, 2017
`
`Sheet 29 of 30
`
`US 9,696,868 B2
`US 9,696,868 B2
`
`
`
`be‘Sid
`
`
`
`TavagONIZVAY
`
`
`
`sewoaeyMaik
`
`
`DIBMIOFeipumey
`ae|seep
`
`31
`
`31
`
`

`

`U.S. Patent
`
`Jul. 4, 2017
`
`Sheet 30 of 30
`
`US 9,696,868 B2
`
`
`
`
`
`[3] [3] [5]
`
`| y :sseupp
`
`32
`
`

`

`US 9,696,868 B2
`
`1.
`SYSTEMAND METHOD FOR PROVIDING
`THREE-DIMIENSIONAL GRAPHICAL USER
`INTERFACE
`
`2
`including metaphors that utilize graphic elements such as
`icons to make an operating systems input and output easier
`to manage. Most computer operating systems incorporate a
`GUI that utilizes two-dimensional graphics to capture, pro
`cess, and output all input from an end user in a 2D form—
`having height and width only.
`This output is usually confined within a window that is
`drawn on a finite-sized desktop, i.e., the working area of a
`computer, that has a given length and width. When the
`computer's output exceeds this finite working graphical
`area, elements of the GUI (the windows) are typically drawn
`on top of each other such that the GUI components overlap
`one another other. In some operating systems, a shadow is
`drawn beneath these overlapping windows on the desktop to
`make them appear as if they have depth. This technique
`allows an end user to identify the overlapping windows
`more easily.
`We live in a 3D (three-dimensional) world where we see
`that objects not only have a horizontal position (X) and
`vertical position (y) but also have depth (Z) that is also
`known as time, according to the three-dimensional coordi
`nate system of mathematics. This notion of expressing depth
`or time in a visual computer metaphor is important for the
`creation of a visual history of the end user's computing
`sessions. By plotting new output of the computer (instead of
`replacing) in a virtual space that does not overlap or Sub
`stitute what exists on the finite desktop, a new virtual space
`through depth and time is created. For example, if one were
`to pull up the webpage for the URL http://www.yahoo.com,
`and then click on a hyperlink (e.g., finance), the current
`webpage in its window would be replaced by the webpage
`for Yahoo! finance.
`3D has shown itself in computing primarily in the fol
`lowing areas: (1) games, (2) CAD/medical visualization, and
`(3) virtual worlds. A virtual world is a computer-simulated
`environment that its users can inhabit and interact with via
`avatars. This habitation usually is represented in the form of
`two- or three-dimensional graphical representations of
`humanoids (or other graphical or text-based avatars).
`The navigation window of many desktop operating sys
`tems use controls and buttons to allow end users to navigate
`to other folders and windows in the hierarchical structure of
`the file system. Often, in navigating to new windows, the
`new windows replace the display of the current window.
`Accordingly, it would be very desirable to provide an
`improved graphical user interface that allows the user to
`efficiently navigate though a virtual space wherein groups of
`windows can be easily organized, stored, and retrieved.
`
`SUMMARY OF THE INVENTION
`
`The present invention addresses the shortcomings of the
`prior-art systems and methods. In particular, the present
`invention is directed to a system and method for providing
`an improved 3D graphical user interface.
`In accordance with one aspect of the embodiments
`described herein, there is provided a graphical user interface
`that uses the two-dimensional (2D) display of a user's
`computer to display three-dimensional (3D) objects in a
`simulated real-time 3D immersive Cartesian space.
`In one embodiment, there is provided a system whereby
`new computing output occupies new virtual space near the
`original output, without losing the original output. When an
`end user clicks on a hyperlink on the webpage, there appears
`in the virtual space a new webpage that is linked to but does
`not replace the current webpage in its window; rather, the
`
`25
`
`30
`
`35
`
`CROSS-REFERENCE TO RELATED
`APPLICATION
`
`This application is a continuation of U.S. patent applica
`tion Ser. No. 14/503,142, filed Sep. 30, 2014, which is a
`continuation of U.S. patent application Ser. No. 127751,879,
`filed Mar. 31, 2010, now U.S. Pat. No. 8,881,048, which is
`a continuation of U.S. patent application Ser. No. 1 1/531,
`676, filed Sep. 13, 2006, now U.S. Pat. No. 7,735,018, which
`claims priority pursuant to 35 U.S.C. S 119(e) to U.S. Pro
`visional Application No. 60/717,019, filed Sep. 13, 2005,
`which applications are specifically incorporated herein, in
`their entirety, by reference.
`
`10
`
`15
`
`COPYRIGHT NOTICE
`
`This patent document contains material that is Subject to
`copyright protection. The copyright owner has no objection
`to the reproduction of this patent document or related
`materials as they appear in the files of the patent offices of
`the United States or other countries, but otherwise reserves
`all copyright rights whatsoever.
`
`BACKGROUND OF THE INVENTION
`
`1. Field of the Invention
`The present invention is directed toward graphical user
`interfaces for operating and accessing information on a
`computer, and more particularly, to a three-dimensional
`(3D) interactive computing interface and sorting interface
`comprising information from real-time and Static sources,
`including, but not limited to, meta search results from the
`Web; information from APIs, webservices, search engines,
`application programs, and networks; and files on the end
`user's desktop.
`2. Description of Related Art
`Currently, people use computers by inputting information
`into the computer to achieve a given output. Often this can
`be a series of tedious steps (mouse clicks and keyboard
`inputs) to run applications and documents or navigate to
`information. To get to new computing experiences, people
`often have to close their current applications and documents,
`hide them or overlap them on a finite desktop by drawing
`them on top of each other, and then mine through folders
`within folders to find them again at a later date. The user's
`desktop is finite, and one must redo the same tasks over and
`over again. This wastes time by (i) requiring many mouse
`clicks to open and close documents, (ii) requiring one to
`remember all the combinations of programs and documents
`one might need for a given purpose and (iii) requiring one
`to create elaborate hierarchical folder systems to aid in the
`process of storing and recalling applications and documents.
`This is primarily due to the limited space the end user has on
`their desktop.
`People currently compute within operating systems that
`present computer output, Such as documents, applications,
`and operating systems interface in a 2D (two-dimensional)
`visual display. After initially being loaded into the computer
`by the boot program, the operating system controls all the
`other programs in a computer. Typically, the component of
`the operating system that Summons the style in which this
`output is displayed is called the GUI or graphical user
`interface. A successful GUI will use screen presentations
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`33
`
`

`

`3
`new webpage is drawn in a new virtual space. This way, the
`end user can visit past visual computing moments in time.
`In accordance with another aspect of the embodiments
`described herein, there is provided a method for providing a
`three-dimensional graphical user interface, comprising
`receiving an input from an end user, capturing computing
`output from at least one computer source in response to the
`received end-user input, and presenting the computing out
`put as at least two objects within a three-dimensional virtual
`space displayed to the end user.
`In one embodiment, the method further comprises gen
`erating a timeline that includes an icon for each object
`presented within the virtual space, wherein the icons are
`organized in linear chronological order according to when
`the objects were presented within the virtual space and
`displaying the timeline within the virtual space. In another
`embodiment, the method further comprises providing a
`database module for storing and categorizing data regarding
`each object presented within the virtual space, providing a
`hyperlink within the database module to respective view
`point of each object presented within the virtual space, and
`displaying the data regarding one or more of the objects
`within the database module presented along with virtual
`Space.
`In accordance with another aspect of the embodiments
`described herein, there is provided a system for providing a
`three-dimensional graphical user interface, comprising a
`display screen, an input device for receiving an input from
`an end user, a processor module operatively coupled to the
`display screen and the user input device, and a memory
`module operatively coupled to the processor module. The
`memory module preferably comprises executable code for
`the processor to capture computing output from at least one
`computer Source in response to the received end-user input
`and present the computing output as at least two objects
`within a three-dimensional virtual space displayed on the
`display Screen.
`In one embodiment, the memory module further com
`prises executable code for the processor to generate a
`timeline that includes an icon for each object presented
`within the virtual space, wherein the icons are organized in
`linear chronological order according to when the objects
`were presented within the Cartesian space, and display the
`timeline within the virtual space. In another embodiment,
`the memory module further comprises executable code for
`the processor to provide a database module for storing and
`categorizing data regarding each object presented within the
`virtual space, provide a hyperlink within the database mod
`ule to respective viewpoint of each object presented within
`the virtual space, and display the data regarding one or more
`of the objects within the database module presented along
`with virtual space.
`In accordance with another aspect of the embodiments
`described herein, there is provided a system for providing a
`three-dimensional graphical user interface in a computer
`network, comprising a server connected to the computer
`network and a user-interface application executing in asso
`ciation with the server to provide the functions of receiving
`an input from an end user, capturing computing output from
`at least one computer source in response to the received
`end-user input, and presenting the computing output as at
`least two objects within a three-dimensional virtual space
`displayed to the end user.
`In one embodiment, the user-interface application execut
`ing in association with the server further provides the
`functions of generating a timeline that includes an icon for
`each object presented within the virtual space, wherein the
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`US 9,696,868 B2
`
`10
`
`15
`
`4
`icons are organized in linear chronological order according
`to when the objects were presented within the Cartesian
`space, and displaying the timeline within the virtual space.
`In one embodiment, the user-interface application executing
`in association with the server further provides the functions
`of providing a database module for storing and categorizing
`data regarding each object presented within the virtual
`space, providing a hyperlink within the database module to
`respective viewpoint of each object presented within the
`virtual space, and displaying the data regarding one or more
`of the objects within the database module presented along
`with virtual space.
`In accordance with another aspect of the embodiments
`described herein, there is provided a network system for
`providing a three-dimensional graphical user interface, com
`prising: a computer-server network comprising a plurality of
`servers in communication with each other; at least one
`display Screen operatively coupled to the computer-server
`network; at least one input device for receiving an input
`from an end user, the input device being operatively coupled
`to the computer server network; and a software module for
`providing a se