US008881048B2
`
`US 8,881,048 B2
`(10) Patent No.:
`a2) United States Patent
`Bakhash
`(45) Date of Patent:
`*Nov. 4, 2014
`
`
`(54) SYSTEM AND METHOD FOR PROVIDING
`THREE-DIMENSIONAL GRAPHICAL USER
`INTERFACE
`
`USPC. ow. 715/781, 848, 764, 766, 767, 782, 788;
`707/706
`See application file for complete search history.
`
`(76)
`
`Inventor: E. Eddie Bakhash, New York, NY (US)
`
`(56)
`
`References Cited
`
`(*) Notice:
`
`Subject to any disclaimer, the term ofthis
`patent is extended or adjusted under 35
`U.S.C, 1540) by 472 days.
`This patent is subject to a terminal dis-
`claimer
`,
`(21) Appl. No.: 12/751,879
`
`U.S. PATENT DOCUMENTS
`5319387 A
`6/1994 Yoshika
`oshikawa
`319,
`5.428.735 A
`6/1995. Kahl etal.
`6,121,969 A
`9/2000 Jain et al.
`6,499,029 Bl
`12/2002 Kurapatietal.
`6,577,330 Bl
`6/2003 Tsuda etal.
`6,725,427 B2
`4/2004 Freeman et al.
`(Continued)
`
`(65)
`
`Prior Publication Data
`US 2011/0029907 Al
`Feb. 3, 2011
`
`“Visualization Using Timelines”, Gerald M. Karam; TRIO Telecom-
`munications Software MethodsProject; PublishedAug. 17, 1994; pp.
`125-137; XP000476876.
`
`Related U.S. Application Data
`(63) Continuation of application No. 11/531,676, filed on
`Sep. 13, 2006, now Pat. No. 7,735,018.
`(60) Provisional application No. 60/717,019,filed on Sep.
`13, 2005.
`
`(Continued)
`
`Primary Examiner — Ting Lee
`(74) Attorney, Agent, or Firm — FitzsimmonsIP Law
`
`ABSTRACT
`(57)
`Methods and systemsare provided for providing an improved
`three-dimensional graphical user interface. In one embodi-
`ment, the method generally comprises: receiving an input
`from an end user, and capturing computing output from at
`least one computer source in response to the received end-
`user input. The computing output can be presented as two or
`more objects within a three-dimensional virtual space dis-
`played to the end user. In one embodiment, the methodfurther
`comprises generating a timelinethat includes an icon for each
`object presented within the virtual space. In another embodi-
`ment, the method further comprises providing a database for
`storing and categorizing data regarding each object presented
`within the virtual space.
`
`18 Claims, 30 Drawing Sheets
`
`(51)
`
`(2011.01)
`(2006.01)
`(2006.01)
`(2013.01)
`
`Int. Cl.
`GO6T 15/20
`GIB 27/34
`GO6F 17/30
`GO6F 3/0481
`(52) U.S. CL
`CPC veeseeeee G06T 15/20 (2013.01); GO6F 17/30873
`(2013.01); GO6F 3/04815 (2013.01); GO6F
`17/30864 (2013.01); G1IB 2734 (2013.01);
`G06F 17/30572 (2013.01)
`USPC. ow. 715/782; 715/764; 715/766; 715/767,
`715/781; 715/788
`
`(58) Field of Classification Search
`CPC wee GO6F 3/04815; GO6F 3/04817; GO6F
`17/30864
`
`
`
`
`
`
`
`
`APPLE 1001
`
`APPLE 1001
`
`1
`
`

`

`US 8,881,048 B2
`Page 2
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`6,768,999 B2
`2001/0050687 Al
`2002/0033848 Al
`2003/0164827 Al
`2003/0179231 Al
`2004/0268451 Al
`2007/0043700 Al*
`
`7/2004
`12/2001
`3/2002
`9/2003
`9/2003
`12/2004
`2/2007
`
`Prageret al.
`lida et al.
`Sciammarellaet al.
`Gottesman et al.
`Kamiwadaetal.
`Robbinetal.
`Dawsonet al. wees 707/3
`
`OTHER PUBLICATIONS
`
`“Execution Patterns in Object-Oriented Visualization”, Wim De
`Pauw et al.; Watson Research Center; Published Apr. 27, 1998; pp.
`219-234; XP002534202.
`“Dissatisfaction Sows Innovation”, Andrew Coulter Enright; The
`Treehouse + the Cave; Published Dec. 29, 2004; 6 Pages;
`XP-002466348.
`
`* cited by examiner
`
`2
`
`

`

`U.S. Patent
`
`Nov. 4, 2014
`
`Sheet 1 of 30
`
`US 8,881,048 B2
`
`10~.
`
`32
`
`
`
`Load Custom Script andits Icon to Alter
`
`
`Application Logic of 3D Cartesian Space
`
`
`
`through Programmatic Access
`(External Proto Node) Represented by
`
`
`Interactive Icon on interface Panel
`
`QCO@©&
`
`©)
`
`)
`
`©)
`
`®
`
`Ci)
`©
`
`4
`
`2
`
`5
`
`0
`
`6
`
`2
`
`72
`
`oA
`
`88
`
`Visit Viewpoint
`Command
`
`Record Viewpoint or
`x,y,z Coordinate
`
`Add Desktop
`Window Command
`
`Add Browser
`
`Add Document or
`Application Command
`
`Add Text
`
`Add Picture
`Command
`Add Music Command
`
`Add Video Command
`
`Record Sound
`Command
`
`Add Map Command
`
`Add 3D VRML. or
`X3D File Command
`
`Save Space/Email
`Space Command
`Add Advertising to
`3D Scene
`
`92
`
`9
`
`OQ®4
`
`104
`
`30
`
`
`Scan Event Handler
`
`for End User Input
`on Interface Panel to
`Execute Appropriate
`
`Helper Application
`
`
`
`
`User Input
`
`Recalculate
`Geometry in
`Scene Basedon
`New Viewpoint
`
`
`
`
`FIG. 1A
`
`3
`
`

`

`U.S. Patent
`
`Nov. 4, 2014
`
`Sheet 2 of 30
`
`US 8,881,048 B2
`
`36
`8)
`©
`O—4
`End User Selects Load]|,Program PromptsEnd
`
`Script (Helper
`Userto Locate Script File
`©) Application) Icon from
`from Local Computer,
`(Q)
`Network, Internet
`Input Interface Panel
`or other Location
`
`
`
`
`
`Load x,y,z Position or
`Prompt User to Choose Viewpoint Name
`from Compassor Click on icon of Previously—Viewpoint from Compass
`Recorded Viewpoint in 3D Cartesian Space
`File and Bind toit
`() 44
`
`
`Prompt Userto
`
`
`Input Viewpoint
`
`
`Name(s) to be Added
`
`
`to CompassFile
`
`
`
`
`
`
`
`Store x,y,z Position
`
`and Viewpoint Name
`within 3D Cartesian
`
`Space to CompassFile
`
`
`
`
`
`
`Draw Interactive 3D
`
`Icon/Geometry in Scene
`
`
`to Represent Viewpoint
`
`
`with Viewpoint Name
`
`Initiate Open File Class or other Web
`Service, API, or Method for Loading
`Data from Local Computer,
`Network or the World Wide Web
`
`Allow Drag and Drop of File, Document
`or Application into 3D Cartesian Space
`from 2D Desktop
`
`Draw Interactive 3D Icon/Geometry in
`Scene to Represent Sound or Music
`that if Clicked with Play Sound/Music
`
`Draw Text, Map, Picture, Video,
`VRML, X3D or other Content
`into 3D Cartesian Space
`
`
`
`Prompt User to Input Name of 3D
`Save Space on Local Computer
`Cartesian Space to Save or Email.
`and/or Our Company Server (FTP).
`Add Name to CompassFile
`Server Files can be Public or Private
`Determine Subject of 3D Cartesian Space by Scanning all Words
`(K)
`Associated with or within the File Names, Files, htm! Pages,
`
`
`Viewpoint Names, Hyperlinked Files or any other Available
`Information Associated with Items within the 3D Cartesian Space
`
`
`or Words Enteredinto the Input Interface
`
`
`
`O-—
`
`
`
`FIG. 1B
`
`—&)
`
`4
`
`

`

`U.S. Patent
`
`Nov. 4, 2014
`
`Sheet 3 of 30
`
`US 8,881,048 B2
`
`
`
`interactive Icon that Triggers Script is | End User is Prompted to Click
`
`
`Dynamically Added to Interface Panel to
`|}4Interactive Icon on Interface Panel
`
`
`Initiate Command Using XML Encoding
`to Initiate Program Altering Script
`
`
`
`Retreive the Kind of Information Requested
`
`Determine Proper OS Control to
`There is no Control
`
`
`for this Information
`
`
`
`
`Periodically Capture the
`
`
`on Screen Output of
`Execute Proper OS Control Specific to the
`Window for
`Said
`Kind of Information Requested which
`
`
`Program/Information as
`will Act as a Mirror to its 3D Representation
`
`
`
`a Bit Map Image
`
`
`Map Visual Output of OS Control, Bit Map
`or API onto Arbitrary 3D Geometry
`
`Store Bit Mapin
`
`a Frame buffer
`
`
`
`Pass Mouse Clicks, Cursor Position,
`
`
`
`Create Device
`Keyboard Input and Movements from
`
`Control Map on 3D Geometry to the Mirror
`Input Event
`
`
`
`Handler in 3D
`Control Running on 2D Environment
`
`
`
`
`Generate Table of Contents Page in the Form of an htmi Page that is Stored on
`
`Company Server Next to Saved Space File that Stores Space Name, Hyperlinks
`
`
`in 3D Space, Description, Picture(s) of 3D Space with the Purpose of being
`an Index for the 3D Space to be Crawled by the Major Search Engines as Part
`of their Web Crawling/Indexing. Place Subject (See Item "A") of 3D Spaceinto
`Meta Tags of html Page. Embed Option in htm! Page to Download PROGRAM
`Active X Control within html Page to Auto Install Saved Space from
`Company Server for which T.O.C. html Page was Generated and
`Execute on their Computer with the Click of One Button on html T.0.C. Page
`
`100
`
`Email Table of Contents html Page to Recipient(s) and Embed
`
`
`Option for Recipient to Download PROGRAMActive X Control within
`html Page to Auto Install and Load Saved Space from Company
`102
`
`
`
`Server for which T.O.C. html Page was Generated and Display
`
`
`on their Computer with the Click of One Button off html Page
`
`
`
`
`
`
`
`Return Appropriate text,
`Picture(s), Video(s), Sound(s),
`Sort Words by Frequencyto
`108~_|
`
`Determine Subject of 3D
`
`
`Music, Hyperlinks or other
`Cartesian Space byIdentifying
`
`
`
`Advertisement Content from
`Most Frequently
`
`Advertising Server, Web
`
`Appearing Words
`
`Service or other Location
`
`
`
`Send Subject of 3D Cartesian
`Draw text, Picture(s), Video(s),
`Space via xml File of Most
`Sounds(s), Music, Hyperlinks or
`
`
`
`other Ad Content from
`Frequently Appearing Wordsto
`
`
`Advertising Server or other
`Advertising Server through
`
`
`
`Location into 3D Cartesian Space
`Internet Request
`
`
`
`
`
`
`(T)
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`FIG. 1C
`
`5
`
`

`

`U.S. Patent
`
`Nov.4, 2014
`
`Sheet 4 of 30
`
`US 8,881,048 B2
`
`20,
`
`User Input
`
`
`
`
`
`
`
`Recalculate Geometry
`in Scene Based on
`New Viewpoint
`or Perspective
`
`FIG. 2
`
`6
`
`

`

`U.S. Patent
`
`Nov.4, 2014
`
`Sheet 5 of 30
`
`US 8,881,048 B2
`
`joujuoDOUSIsey,
`
`vr
`
`8cl
`
`ysanboyJasayeiiuypuesajsuedy
`
`
`ysanbayJasnainidey
`
`
`
`ssedoyonewueBoigyBnolu}
`dopseqdz0}soeds-qewo
`
`(uoHeziwoysnywei6oiguOpaseg)
`
`soedsqeuly
`
`9c}
`
`
`
`uogeoyddyyouney
`
`
`
`CEsajoelajuyayea19
`
`
`
`aoedsuejsape5
`
`coh
`
`Oct
`
`vel
`
`
`
`
`
`
`
`
`
`
`
`UONBWUOJUTSIU}JO}
`
`paysenbayUOeWOJU]JoPUlYBU}SAIaROY0}[0U0DSOJadolgaulWa}eq
`
`yeodoy
`
`awe,euldeyHgB0}S
`Jaying
`
`Srl
`
`
`
`deyIq‘Joqu0DSOJonding|ensi,dey
`
`AnswoagqeAleuiquyoo[dy10
`
`cr
`
`
`MopuiAA{0INdjNOuse1aSUO
`au)ainjdesgAyeoipousd
`
`
`
`
`
`
`
`
`
`uoneuuojul/UeJBoidgplesJO}
`
`OFL
`
`
`
`
`ysonbayMODUIAAyuawinoaqobey
`JaNIOsJeBID/\l4uedgemuado
`
`
`
`O€L
`
`ofew)deyiqese
`
`
`
`paysenboyUdieWJOJU]JOPUNYBU}O}YloadgjoNUODSOJedoigaNDaxy
`
`
`uoHeUssadayQESs}!0}JOU;@SeJOYjmYOTUM
`
`
`
`
`
`yuaWUOIIAUAGzUoBujuUNyjoAUOD
`
`
`
`JOU9U}0}AjatUOas)GEUOde|;josjuED
`
`WO}S}USLUSAOYYpueIndu]pueoghsy
`
`
`QEUlJajpueHJuaAg
`
`
`
`dusdAeqs}ealy
`
`
`
`
`
`‘UOMSO,JOSIND‘syaSssnoywssed
`
`
`
`
`
`OSt
`
`7
`
`
`
`
`
`
`
`
`
`
`
`

`

`U.S. Patent
`
`Nov.4, 2014
`
`Sheet 6 of 30
`
`US 8,881,048 B2
`
`v6L
`
`00¢
`
`
`
`
`
`
`
`BsIeLO}U]
`891
`98
`cOL
`JOOYeA|
`hggi
`Aeuiyueul
`winjoy
`
`
`
`Jas)wosBuysynduyysenbayyokessalsooy
`
`91
`synsayyoleasJouIBUGWoysynseyYolessBuissegJo}aiqisuodsayydswoysn5peo
`
`
`
`
`
`O8lBLL
`
`L
`
`881
`
`
`
`wesBoiguoqeaddy
`
`vlCLL
`
`OLL
`
`c8l
`
`pst
`
`
`
`[aoesosferfsSerBO}[HN
`
`8g9,49919991
`
`
`
`99]¥gq,-®g91“991
`
`
`
`
`
`syinsayYoIessjoUHCjOs}e¢g0}J8SPFJdWoIg
`
`
`
`
`
`BuunjdeyJ0)pourseulue}eq
`
`v9
`
`
`
`
`
`weJ6oldojuieyegyalees
`
`
`
`platxes,abedqayys,ulbucQyoueasojUl
`Buysinduyyenbayyoreaswo
`
`
`
`ysonbayyeaspainjdesJoyuy
`
`
`
`
`
`s,UIBUOYoIeSsSJO}FUNJe]uy
`UBdIISYOulabedGan,
`abedJasmoigday,
`
`
`|UBalDSJOMeld
`
`abeJasmoigGann
`
`
`
`yuipedAHyoes0}
`
`
`
`obeyGannasled
`
`c6L
`
`abe[OU0DJESMOIgGeya}eIID
`
`
`
`pajdauU0DTNYyoesJO}
`y8SpueabeguosyuiedAH
`ALLLNVWNDHOVLSsesiu)
`
`
`JoAyuenysulwiajeq
`ajo}§pueaBeduo
`
`syulvedayAyuepy
`
`861
`
`VP‘Sls
`
`FISVINVA
`
`abeJasmoigMONUl
`
`
`S}INSAYSAGDOy
`
`ebewesBuyeinwa
`
`Aqyoueasyonpuog
`
`O61
`
`8
`
`
`
`
`
`
`
`
`
`
`
`
`
`

`

`U.S. Patent
`
`Nov.4, 2014
`
`Sheet 7 of 30
`
`US 8,881,048 B2
`
`
`
`yeu)UoNsabeqeafqIXON10}Aljawiossayejnoje0ey
`
`
`aoedsG-€ulUONISO,Ss}!HBuyejsue,pueuMOG
`
`
`yiBuyeogAgyoe}sulAemyJaypingsieaddy}!
`
`
`
`ay}JO)BBEJOUODJESMOIGGAA,IXONJOINdjnoJensiAMeIq
`
`
`
`s|Aeuiyjopuljyunaoedsueisayeyq-¢au}ule6egsyinsey
`
`
`
`
`yoseass,ulbuCYoreesWoyyuILedAHay}0}pa}pauUODWYN
`
`peyoesy
`
`902vO?
`
`jensiajojnoAeyAypoyy0}wesHoi0}ayejduis,wWio}snDpeo]0}Jasqpugmony
`
`
`
`UeUlPayoidB6edGandJSE7SY}0}UITJeu)SeBeYGay,Oy}JepueadedsUeISAWeEDGEa}UIpanolg
`
`
`
`qayyou}{TeJO}0}JOSPUMoly jouegaoepayy|S,wes6olguouoyNgYBnosy)soedsUeIseyeDGEUIUMeIgaq0}sabeYgan,joyoRIS
`
`
`
`
`
`
`
`aedGan}Se7]Bu}UOpuNnoyaeyeu)syUIPadAHou}Josabe
`
`
`
`
`
`
`
`wesbeigyeySeQINOsuru]syur]aag-ee,joujuODJasmolgqanyJuauND0}soedsJuaoelpy
`
`
`
`
`
`
`sobbedJaSMOlgGapJOOBISJeENsI/Ou}JO}XepUTSejo0}aoedSUEISeeD
`
`
`
`
`
`
`
`
`
`CgulUMBIGaq0}YOR}SJensiAeBulBegjsil4seabegsynsaymeq
`
`c0¢
`
`
`
`yBnosy)yoR}sG38AAHOS0}J8SNPuyMOTjensi,ulseabedJaSMOJG
`
`
`
`
`
`
`
`
`
`
`
`yorlsjensi,ulsobegjoAenyosweibeiqaag-soepajyuywoysn5
`
`
`
`
`
`Jeadey0}Jas\dWOIg Ole
`YORISJeENsIAMONBuNeIsAqoesyinsayYoseasJOJesIXENJO}Ssed014
`
`
`
`42007]0}aIl4aves 8LC
`JasnAgpaydwosgUde907J8U}OJOJeAJaSqanA‘JOMJON‘AINdUWIOD
`
`
`
`
`
`avSls
`
`payoroysiAgu
`
`9
`
`
`
`
`
`
`

`

`U.S. Patent
`
`Nov. 4, 2014
`
`Sheet 8 of 30
`
`US 8,881,048 B2
`
`FIG. 5A
`
`
`732
`
`FIG. 5B
`
`10
`
`10
`
`

`

`U.S. Patent
`
`Nov. 4, 2014
`
`Sheet 9 of 30
`
`US 8,881,048 B2
`
`
`
`FIG. 5C
`
`11
`
`11
`
`

`

`U.S. Patent
`
`Nov.4, 2014
`
`Sheet 10 of 30
`
`US 8,881,048 B2
`
`
`
`YOR}SXENUlaBedJSJ1-4JojulodmaiAdnasoly0}puig
`Ovebe?
`
`YORISSNoIAaigulabegjsil4joyulodmaiAdnasojy0}puig
`
`
`
`
`
`OBISjensiauleedse]JoyuJOdmal,dnasojy0}puig
`
`yesjensi,ula6ed}XaVJoJulodmaiAdnasoly0}puig
`
`
`
`
`rensiprensinabedabeabeabe
`
`0Ssnoinalg|}3827SNOIADd||ISIS}XON
`
`0SzYOR}Sfensiulabe}Si14Jojulodmai,dnasojg0}puig
`
`
`
`BE?9ECvecCEC0€282%922
`
`
`goedsueblseLegQeuls}eUIP00DZ'A'xpioday
`
`
`yoeISJeNsiAUleedsnoiAaldJojWIodmai,dnasoj50}puig
`
`
`
`
`
`jaueddOB}H9}U]UOyNduyJasf)pUz0}
`
`
`
`Ja|puepyjuanguess
`
`Occ
`
`COC
`
`paseguonoyayeudoiddyaynoexy
`
`
`
`Japiopayjoedsulyoer}gAeuypos
`
`oe}oe}
`
`
`
`indujJesquo
`
`Gv
`
`vyc
`
`ORC
`
`8rZ
`
`12
`
`ASTA
`
`9°‘!
`
`pSZye
`
`
`
`
`
`
`
`12
`
`
`
`
`
`
`
`
`
`
`
`
`

`

`U.S. Patent
`
`Nov. 4, 2014
`
`Sheet 11 of 30
`
`US 8,881,048 B2
`
`260
`
`End User Clicks Close Button on any Texture Map
`of Web Browser Control Represented in Visual Stack
`
`262
`
`Remove Web Browser Page
`from Visual Stack
`
`264
`
`266
`
`Position of Removed Page in Stack
`Animated Page Before it was Moved
`
`Animate Forward Pageafter
`Removed Page to Replace
`
`Animate Page after Previously
`Animated Pageto Original
`Position of Previously
`
`268
`
`270
`
`of Previously Animated Page
`
`Animate Forward Next Page in
`3D Visual Stack to Replace Position
`of Previously Animated Page
`
`Animate Next Pagein
`Stack to Position
`
`272
`
`274
`
`Exit Loopif the End of Visual
`Stack or Last Elementin
`
`Array is Reached
`
`FIG. 7
`
`13
`
`13
`
`

`

`U.S. Patent
`
`Nov.4, 2014
`
`Sheet 12 of 30
`
`US 8,881,048 B2
`
`S
`
`
`
`yBnowyypayesipuy|)YyBnoiyypayeoipuy||yBnosuyypayeoipuy
`
`
`
`staoeds[enulAsiaoedsJeni,sisoedsjenulA
`jualjgulaBueyd||sjualgulaBueyo|}sjuayouleBueyo
`
`
`
`JUd!|DJASF)aICHINW|[JUSJes-yedyinw
`
`
`lenulApapeo|UMmoc]|;|ENIAPapeo|uMog
`
`
`
`
`
`wUSAF[ENsip,,WUBAZ|ENsiA,,wUBAZJENsiA,,
`
`
`Buisnsayndwo5Buisnsayndwo5
`Aue;jo9uOAueyJoauO
`
`
`MANYyoeyJO}sHueugpaezijeusnor
`
`JUal|QABAZWOY,JUBAZ[eNsiA,,
`
`Yyoes10}ajl4Yoesysnd
`
`JOINAWODJBAIBSBy}0}
`
`
`
`0}sabueyDpezijewnorJoJayndwWodywusl|D0}alyyeulBuOysng
`
`
`
`
`
`Oll{UMOS}IUl,JUBAZJeNsIA,yoeQwoyseBueuDazijewnor
`
`
`
`
`payeuibucsaBueysoieymJojndwoyBuipnjox3ayyjeurbuo
`
`
`
`IENUIAPEapEO|UMOG||JENLIAPepeo[uMmod||jeEnuIApapeojumoq
`
`
`
`
`JUa![DJasnajdyiny||JUalTDespaidninyy||yuayDsesqaidninyy
`
`
`
`
`
`BuispJayndwogBuisnsayndwoBuisnsayndwoy
`Auewjo3u0AuewJo3u0Auejo8uQ
`v8cC8C
`
`8‘Old
`
`$JUaIIDulebueYD
`yBnoiy)payeoipuyS|aoeds
`
`jenyiA
`,JUBAZ[ENSIA,
`
`SJUaIIDulaBueyD
`
`yBnosy}payeoipuy
`
`sia0edsJENA
`
`AUBAZ[ENSsIA,
`Squat|DulaBueyD
`
`yBnouypayeaipuy
`
`
`Ss]s9edsjenuiA
`
`UBSAZ[ENSIA,,
`
`b26z
`
`
`
`BuisnJajndwo
`
`
`
`[ENLIApapeojumog
`
`goeds
`
`JO8AI8SUOSail4
`PUBIl]SBAIGIOYJBAIOS
`
`
`osadedsjenulApaves
`yissaooyuedauoAuy
`
`SIU}0}THNSeysyqnd
`
`
`
`PaysiqndpuepajeagS|aoedsjenui,jeulbuC
`
`PuyWO}J8Auas0}
`
`Jayjndwio5sas
`
`
`
`AueyyJO9UQ
`
`bO6z
`
`
`
`
`
`JUaI|DJasnaydyinyy
`
`14
`
`14
`
`
`
`
`
`
`
`
`
`

`

`U.S. Patent
`
`Nov.4, 2014
`
`Sheet 13 of 30
`
`US 8,881,048 B2
`
`piesodAeld=purmeyG)CO&sewoaeyMaia,
`
`EOt“ay||_|dt
`SVE“9PE
`al
`be
`
`
`
`15
`
`15
`
`

`

`U.S. Patent
`
`Nov.4, 2014
`
`Sheet 14 of 30
`
`US 8,881,048 B2
`
`©
`
`Ob‘Sid
`Aadnf3oLv
`OLY&
`
`BO8e
`
`Ou
`
`Ve.
`
`16
`
`16
`
`
`
`
`
`

`

`U.S. Patent
`
`Nov.4, 2014
`
`Sheet 15 of 30
`
`US 8,881,048 B2
`
`Lb“Sls
`
`;i||
`fys1GBI@Or\
`
`iitSc
`
`
`itla
`[saintsofeuy
`[tase
`storesora
`
` SONJA
`
`
`
`
`e\BooBmMcIYBYFa;osn9MON-Aega-woodRgommmyrdny[BP&;16009-wos
`WOO'NSIWN-Woousurmawyrdiy(@1-.
`
`
`00uR,-Woo'coyehMMMyayy[a-ay--
`
`
`
`
`
`saoedspenes
`
`SAGAR
`
`17
`
`17
`
`
`
`

`

`U.S. Patent
`
`Nov.4, 2014
`
`Sheet 16 of 30
`
`US 8,881,048 B2
`
`
`
`cbOld(Cer
`
`[enaeemenereennnomen
`
`SoWoARy=Mei,
`
`18
`
`18
`
`

`

`U.S. Patent
`
`Nov.4, 2014
`
`Sheet 17 of 30
`
`US 8,881,048 B2
`
`Vel
`Sid
`
`V-S6ES19
`
`SONTHSAOS
`
`
`
`
`
`SUNGNODALINTSUNDJ1d03dMOH
`
`
`
`<v-e-OlelssigeeatPELyeSC8LolWerleUeeeso
` a5CeBeEEPenhETneEenooo
`
`
`
`
`
`
`Oz©smoysAejdsipjensasoymswaysésGuqezedouayUMayndwodAqueunoadoed
`
`
`
`Aigse‘speuayewpayee:Aue20JuowuTo0pjualedjevoisiox!6jovoqoNpasdal
`
`
`
`Su0}vogoelqayouseyTREGXEGoppzvoumoyyButdeooyvoqsojoxd
`
`
`
`Aue20severpayunainJoeou9yewoped|purewoIegaigJosetyax]ULeedde
`
`
`
`
`WyBuxdoooyyoolgrsjeuarewsueKoojWaunoOpwaledjeuorsiaoNdsit
`ea[eneavaenflameunmyaLwcrecratatomnieteeth
`
`
`
`
`
`
`
`SGTORLPSPOUTATLFoySIRS]WOTEARBUNeiodoPUESUOHEDIONSos
`‘TWNOIGNAWIT33YHLONIGIAOUdYOdCOHIAWONYWALSAS
`
`
`
`
`
`
`
`‘sTuswunoOsou]SeYonsIndinosaynduicoJouogeTUesadd(feuorsueUNDOn})
`z*7a
`
`
`
`
`soasosyeypasqyBugesexvseyastustpoing‘Agun0240
`
`
`UOYOR«a8TUREvrmamd”mst“peLgee”GA
`
` SEIS
`SHAG—BoyieEg
` SOM
`
`SopquejedjsuorsoidmeuLAPOLSLZ&oapia(J
`oises]>10alweual
`
`0}<YALNA>S88ig
`
`IpuruopensiBaseqopy{Qj
`
`
`
`
`
`
`
`yeyyouney-outteseds
`
`dopieuyusjed
`
`sowoneyMOA
`
`
`
`$651sopA]HTmatweedekPT97TF
`
`SBITOAE4
`
`SIGEMOIgGay
`
`
`
`
`
`ROVIUAININISNTWOINdvUd
`
`
`
`sevedgpengAwne
`
`syoogeAw(3
`
`soimydAWQorsnwyAWie
`
`
`
`19
`
`
`
`
`
`uny'suojenesey-yequry(@Jed't-sBormegpeedBuzewy(7)
`
`19
`
`

`

`U.S. Patent
`
`Nov.4, 2014
`
`Sheet 18 of 30
`
`US 8,881,048 B2
`
`
`
` gcL‘Siffr="=wuO]|rit(]3>JOaweUa!O}<YaINZ>Ssosq
`aOUNSHI]|dSeODOYINE}EU]—_OFEO/OyFOMUUAPAATTAG)|beeenTener"Leb
`
`
`
`
`
`sorsiudqBuoymoeOUSSKYUNADOODapm]|H$s.
`
`
`[oadisaTavingog(2b:4}suogdoyjsowo2epuos~|5~ABojoutha10dSHEN{3H-worwoesmu@a-ison]!iaa|SaaHa16009-wen
`
`
`
`
`
`“eB{fewerrx0us,0pu0osag0}exepre™,enc
`
`
`
`__memuramnrane)(3)ylsiansgoben
`
`fyBeotgroueDwrYomAendg~Bupyeaie-worsnansar-wosxnuseymaayccny(6)4“20‘SoMLEQ“SHONJey-war'06
`
`
`
`
`
`vappayepayCun!ag500~PSAON-WoD;YOOIYSN-wondepersrayedy(BR;
`
`
`
`
`
`aibonSsourdy(EPoa;
`
`fOISHIA7SpUNOS
`
`PuRSeYYeasOOUEA-SON-wenoyekYERcy[Awf0LSSin'salQ|
`Lo
`
`
`[sea[peraJsated]
`uumddReBuryeargwor'saauroymaura-[BPaNON-wor SMISNOBE-MOOGySHUNBDty-4‘TOboSetRpeaYseudolayy-worronuedsmauirdny(GYsy|“2A8SHON|SHOR
`
`
`
`
`
`
`
`“NSHWoRSaysAepo]-wootustursquswamardy[BD4ObedMold
`
`
`smounceydiy(FPmy:
`SHON
`Emapcs]
`
`
`
`“SHOU,O)\pRIgCfEC]
`
`
`
`SSEII-1S14S380aydnodBUC
`
`
`
`
`
`
`
`SAANLSALV1=
`
`S007‘Shendey40gweLLoat
`
`soworey=maiany
`
`bSP
`
`20
`
`20
`
`

`

`U.S. Patent
`
`Nov.4, 2014
`
`Sheet 19 of 30
`
`US 8,881,048 B2
`
`«Mark
`
`STepse
`
`
`
`
`
`
`
`
`
`
`pIEMIO.4Keldpuimay
`
`sajuoney
`
`vb“Sid
`
`
`
`AvWaYnofued
`
`000'0LL$232!
`10]BBBHOW,
`E4UOW/Q9S
`
`pennenepoaunasatoyeolonPEDNOUTEv929DNOU‘equardasos=
`
`21
`
`21
`
`

`

`U.S. Patent
`
`Nov.4, 2014
`
`Sheet 20 of 30
`
`US 8,881,048 B2
`
`SI
`Sid
`
`weTT)
`
`ang
`
`
`
`
`
`Bdfowoyy“ousargubpng
`
`22
`
`
`
`pIeMIO-|Keldpuimay
`sayoney=Mark
`
`22
`
`

`

`U.S. Patent
`
`Nov.4, 2014
`
`Sheet 21 of 30
`
`US 8,881,048 B2
`
`Ff
`
`VOLOld
`
`soyoneyMeth
`
`
`
`23
`
`23
`
`

`

`U.S. Patent
`
`Nov.4, 2014
`
`Sheet 22 of 30
`
`US 8,881,048 B2
`
`g9b“Old
`
`SaOAByMOT,
`
`
`
`vor
`
`24
`
`24
`
`

`

`U.S. Patent
`
`Nov.4, 2014
`
`Sheet 23 of 30
`
`US 8,881,048 B2
`
`Vv
`Lt
`
`saworey=MalR
`
`
`
`
`
`
`
`25
`
`BHO
`
`
`
`premio4Abipulay
`
`25
`
`

`

`U.S. Patent
`
`Nov.4, 2014
`
`Sheet 24 of 30
`
`US 8,881,048 B2
`
`az
`
`SOWOARGMOI
`Old(A
`
`
`
`
`
`premiog=Aeigpuimey
`QO
`
`26
`
`26
`
`

`

`U.S. Patent
`
`Nov.4, 2014
`
`Sheet 25 of 30
`
`US 8,881,048 B2
`
`OLbL
`Sid
`
`SeOAR=MOA,
`
`
`
`
`
`
`27
`
`27
`
`

`

`U.S. Patent
`
`Nov. 4, 2014
`
`Sheet 26 of 30
`
`US 8,881,048 B2
`
`0€9
`
`
`premiofeldpumey
`a|sseippy
`
`gL‘Sid
`
`sayoneymei
`
`Ove
`
`28
`
`28
`
`

`

`U.S. Patent
`
`Nov.4, 2014
`
`Sheet 27 of 30
`
`US 8,881,048 B2
`
`REISSTHAR
`
`
`
`
`““I,NOA‘epAJSA9ayesJO}Swe}!
`UOI!WGJBAOUIA“SBdtdMO}
`paeuNOKsmauautIle189)
`
`SOU]IO,MONSUL
`
`yeSWS}SMOUpul
`
`
`
`/“UONSE{OSOtLyL410)MONOY{UUMPOLLO]AB]SOF
`
`
`
`“NOK‘hepAreaosees10]suey
`(oSTeUyDAO|SLOATByUSUNLICOay]yoweulayypussmau
`
`
`
`UO}gJOA0UA,“SedudMO]
`
`
` [HeWOPAORISIOHUNTVERWOT
`
`‘ojuyaugnd~yonuysMOUAEYI“smeuBuySyary
`
`
`AbsUOSHAN]
`
`woo'AeqaMmm
`
`peounoksmouoyqe189SHUALHOLROST,
`
` LOTSMSUDUESYOOTSAK‘smonyBOMUITTOYOUTSgwos
`
`
`
`
`
`
`18sue]smouPuls Usou|S83SIPPIWSinUB1oBis]WONSMOUGUI}e5BSWSTRHwoo
`
`““SUORRISOIE)SNOUBA
`
`‘suods“rayjeom
`SMENNUESYIOIS
`seseuNAUAM
`}oRsHOYaMaL
`wooheqoMan
`
`
`
`6b“Slsbo
`
`29
`
`29
`
`

`

`U.S. Patent
`
`Nov.4, 2014
`
`Sheet 28 of 30
`
`US 8,881,048 B2
`
`
`
`PIEMIO4AidPUIMER
`sewoney=maih,ay
`
` S|
`
`CSV
`
`OVP
`
`30
`
`30
`
`

`

`U.S. Patent
`
`Nov.4, 2014
`
`Sheet 29 of 30
`
`US 8,881,048 B2
`
`Le‘Sis
`
`semanpetetana
`OALLYVN71OLEONIZVAY
`
`«Maik
`
`TaV3q
`
`
`
`
`ONIZVTY
`
`
`
`
`
`799SSS8002BisbyRe
`
`wdge's-g00z‘aisn6ny@)wa-{099uses-90008isntiny@a:
`
`
`‘gienBoy@a:~SiMGS'S9002‘gisnbay(-7p99~Tines-9002
`paewiojAigpuimey
`
`sawoaey
`
`31
`
`31
`
`

`

`U.S. Patent
`
`Nov.4, 2014
`
`Sheet 30 of 30
`
`US 8,881,048 B2
`
`ewelILI
`
`juesuyffIL| (up3a18us)sourowEfFIieuuey
`
`
`
`pleMo4APIpumeyHO
`
`
`
` ouibew|Geuy(IaACISNISALWHM
`
`32
`
`SeoAeyMOIR
`
`ajdoagsy,OL1emogEf)Ang)
`
`
`
`snojeartf
`
`
`
`32
`
`
`

`

`US 8,881,048 B2
`
`1
`SYSTEM AND METHOD FOR PROVIDING
`THREE-DIMENSIONAL GRAPHICAL USER
`INTERFACE
`
`CROSS-REFERENCE TO RELATED
`APPLICATION
`
`This application is a continuation of U.S. patent applica-
`tion Ser. No. 11/531,676, filed Sep. 13, 2006, now U.S. Pat.
`No. 7,735,018, which claimspriority pursuant to 35 U.S.C.
`§119(e) to U.S. Provisional Application No. 60/717,019, filed
`Sep. 13, 2005, which application is specifically incorporated
`herein,in its entirety, by reference.
`
`COPYRIGHT NOTICE
`
`This patent document contains material that is subject to
`copyright protection. The copyright ownerhas no objection to
`the reproduction of this patent documentorrelated materials
`as they appearin thefiles of the patent offices of the United
`States or other countries, but otherwise reserves all copyright
`rights whatsoever.
`
`BACKGROUND OF THE INVENTION
`
`1. Field of the Invention
`
`The present invention is directed toward graphical user
`interfaces for operating and accessing information on a com-
`puter, and more particularly, to a three-dimensional (“3D”)
`interactive computing interface and sorting interface com-
`prising information from real-time and static sources, includ-
`ing, but not limited to, meta search results from the Web;
`information from APIs, webservices, search engines, appli-
`cation programs, and networks; and files on the end user’s
`desktop.
`2. Description of Related Art
`Currently, people use computers by inputting information
`into the computerto achieve a given output. Often this can be
`aseries oftedious steps (mouse clicks and keyboard inputs) to
`run applications and documents or navigate to information.
`To get to new computing experiences, people often have to
`close their current applications and documents, hide them or
`overlap them on a finite desktop by drawing them on top of
`each other, and then mine through folders within folders to
`find them againat a later date. The user’s desktop is finite, and
`one mustredo the sametasks over and over again. This wastes
`time by (4) requiring many mouseclicks to open and close
`documents, (11) requiring one to rememberall the combina-
`tions of programs and documents one might need for a given
`purpose and(iii) requiring oneto create elaborate hierarchical
`folder systems to aid in the process of storing and recalling
`applications and documents. This is primarily due to the
`limited space the end user has on their desktop.
`People currently compute within operating systems that
`present computer output, such as documents, applications,
`and operating system’s interface in a 2D (two-dimensional)
`visual display. After initially being loaded into the computer
`by the boot program, the operating system controls all the
`other programs in a computer. Typically, the component of
`the operating system that summonsthe style in which this
`output is displayed is called the GUIor graphical user inter-
`face. A successful GUIwill use screen presentations includ-
`ing metaphors that utilize graphic elements such as icons to
`make an operating system’s input and output easier to man-
`age. Most computer operating systems incorporate a GUI that
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`2
`utilizes two-dimensional graphics to capture, process, and
`output all input from an end user in a 2D form—havingheight
`and width only.
`This output is usually confined within a windowthat is
`drawn ona finite-sized desktop, 1.e., the working area of a
`computer, that has a given length and width. When the com-
`puter’s output exceeds this finite working graphical area,
`elements ofthe GUI(the windows) are typically drawn on top
`of each other such that the GUI components overlap one
`another other. In some operating systems, a shadow is drawn
`beneath these overlapping windowsonthe desktop to make
`them appearas if they have depth. This technique allows an
`end userto identify the overlapping windows moreeasily.
`Welive in a 3D (three-dimensional) world where we see
`that objects not only have a horizontal position (x) and verti-
`cal position (y) but also have depth (z) that is also known as
`time, according to the three-dimensional coordinate system
`of mathematics. This notion of expressing depth or time in a
`visual computer metaphor is important for the creation of a
`visual history of the end user’s computing sessions. By plot-
`ting new output of the computer (instead of replacing) in a
`virtual space that does not overlap or substitute what exists on
`the finite desktop, a new virtual space through depth and time
`is created. For example, ifone were to pull up the webpage for
`the URL http://www.yahoo.com, and then click on a hyper-
`link (e.g., finance), the current webpagein its window would
`be replaced by the webpagefor Yahoo! finance.
`3D has shownitself in computing primarily in the follow-
`ing areas: (1) games, (2) CAD/medical visualization, and (3)
`virtual worlds. A virtual world is a computer-simulated envi-
`ronmentthat its users can inhabit andinteract with via avatars.
`
`This habitation usually is represented in the form of two- or
`three-dimensional graphical representations of humanoids
`(or other graphical or text-based avatars).
`The navigation window of many desktop operating sys-
`temsuse controls and buttonsto allow end users to navigate to
`other folders and windowsin the hierarchical structure of the
`file system. Often, in navigating to new windows, the new
`windowsreplace the display of the current window. Accord-
`ingly,
`it would be very desirable to provide an improved
`graphical user interface that allows the user to efficiently
`navigate though a virtual space wherein groups of windows
`can be easily organized, stored, andretrieved.
`
`SUMMARY OF THE INVENTION
`
`The present invention addresses the shortcomings of the
`prior-art systems and methods. In particular,
`the present
`inventionis directed to a system and methodfor providing an
`improved 3D graphicaluser interface.
`In accordance with one aspect of the embodiments
`described herein, there is provided a graphical user interface
`that uses the two-dimensional (“2D”) display of a user’s
`computer to display three-dimensional (“3D”) objects in a
`simulated real-time 3D immersive Cartesian space.
`In one embodiment, there is provided a system whereby
`new computing output occupies new virtual space near the
`original output, without losing the original output. When an
`end user clicks on a hyperlink on the webpage, there appears
`in the virtual space a new webpagethatis linked to but does
`not replace the current webpagein its window; rather, the new
`webpage is drawn in a new virtual space. This way, the end
`user can Visit past visual computing moments in time.
`In accordance with another aspect of the embodiments
`described herein, there is provided a methodfor providing a
`three-dimensional graphical user
`interface,
`comprising
`receiving an input from an end user, capturing computing
`
`33
`
`33
`
`

`

`US 8,881,048 B2
`
`3
`output from at least one computer source in response to the
`received end-user input, and presenting the computing output
`as at least two objects within a three-dimensional virtual
`space displayed to the end user.
`In one embodiment, the method further comprises gener-
`ating a timelinethat includes an icon for each object presented
`within the virtual space, wherein the icons are organized in
`linear chronological order according to when the objects were
`presented within the virtual space and displaying the timeline
`within the virtual space. In another embodiment, the method
`further comprises providing a database module for storing
`and categorizing data regarding each object presented within
`the virtual space, providing a hyperlink within the database
`module to respective viewpoint of each object presented
`within the virtual space, and displaying the data regarding one
`or more of the objects within the database module presented
`along with virtual space.
`In accordance with another aspect of the embodiments
`described herein, there is provided a system for providing a
`three-dimensional graphicaluser interface, comprising a dis-
`play screen, an input device for receiving an input from an end
`user, a processor module operatively coupled to the display
`screen and the user input device, and a memory module
`operatively coupled to the processor module. The memory
`module preferably comprises executable codefor the proces-
`sor to capture computing output from at least one computer
`source in responseto the received end-user input and present
`the computing output as at least two objects within a three-
`dimensionalvirtual space displayed on the display screen.
`In one embodiment, the memory module further comprises
`executable code for the processor to generate a timeline that
`includes an icon for each object presented within the virtual
`space, wherein the iconsare organized in linear chronological
`order according to whenthe objects were presented within the
`Cartesian space, and display the timeline within the virtual
`space. In another embodiment, the memory module further
`comprises executable code for the processor to provide a
`database module for storing and categorizing data regarding
`each object presented within the virtual space, provide a
`hyperlink within the database module to respective viewpoint
`of each object presented within the virtual space, and display
`the data regarding one or more of the objects within the
`database module presented along with virtual space.
`In accordance with another aspect of the embodiments
`described herein, there is provided a system for providing a
`three-dimensional graphical user interface in a computernet-
`work, comprising a server connectedto the computer network
`and a user-interface application executing in association with
`the server to provide the functions of receiving an input from
`an end user, capturing computing output from at least one
`computer source in responseto the received end-user input,
`and presenting the computing outputas at least two objects
`within a three-dimensionalvirtual space displayed to the end
`user.
`
`In one embodiment, the user-interface application execut-
`ing in association with the server further provides the func-
`tions of generating a timeline that includes an icon for each
`object presented within the virtual space, wherein the icons
`are organizedin linear chronological order according to when
`the objects were presented within the Cartesian space, and
`displaying the timeline within the virtual space. In one
`embodiment, the user-interface application executing in asso-
`ciation with the server further provides the functionsof: pro-
`viding a database module for storing and categorizing data
`regarding each object presented withinthe virtual space, pro-
`viding a hyperlink within the database module to respective
`viewpoint of each object presented within the virtual space,
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`4
`and displaying the data regarding one or moreof the objects
`within the database module presented along with virtual
`space.
`In accordance with another aspect of the embodiments
`described herein, there is provided a network system for
`providing a three-dimensional graphical user interface, com-
`prising: a computer-server network comprising a plurality of
`servers in communication with each other; at least one display
`screen operatively coupled to the computer-server network;
`at least one input device for receiving an input from an end
`user, the input device being operatively coupled to the com-
`puter server network; and a software module for providing a
`series of screen displays to the enduser, the software module
`being accessible by one or more ofthe servers of the com-
`puter-server network. The software module preferably com-
`prises instructions for directing the servers to capture com-
`puting output from at least one network sourcein response to
`the received end-user input and to present the computing
`output as at least two objects within a simulated three-dimen-
`sional Cartesian space displayed on the display screen.
`In one embodiment, the software module further com-
`prises instructions for directing the servers to generate a time-
`line that includes an icon for each object presented within the
`Cartesian space, wherein the icons are organized in linear
`chronological order according to when the objects were pre-
`sented within the Cartesian space and to display the timeline
`within the Cartesian space. In another embodiment, the soft-
`ware module further comprises instructions for directing the
`servers to provide a database module for storing and catego-
`rizing data regarding each object presented within the virtual
`space, to provide a hyperlink within the database module to
`respective viewpoint of eac