PTO/SB/O5 (4/98)
`Patent and Trademark Office: U.S. DEPARTMENT OF COMMERCE
`uired to respond to a collection of information unless it displays a valid OMB control number.
`
`Approved for usethrough 09/30/2000. OMB 0651-0032 +
`
`
`
`s 6
`
`
`
`Pleasetypea plus sign(+) inside this box —>
`Under the Paperwork Reduction Act of 1995, no persons are req
`
`
`l
`Attorney Docket No.) MS-53/4(116623.1)
`UTILITY
`
`AI
`matt
`
`
`First Inventor or Application Identifier| George G. ROBERTSON
`PATENT APPLICATION
`
`
`
`APPARA
`AND DA
`
`i
`a .
`SLOT
`TTY.
`:
`OPEC TA
`By
`N
`rn
`EA
`R
`N
`A
`3,
`cy
`
`
`
`TRANSMITTAL
`[eesMaLateno] BL15912951805
`‘Only for new nonprovisional applications under 37 C.F.R. § 1.53(b)
`
`
`Assistant Commissionerfor Patents
`
`
`|
`APPLICATION ELEMENTS
`
`
`ADDRESS TO: Box Patent Application
`Nachi
`0
`See MPEP chapter 600 concerning utilitypatent application contents.
`
`
`
`Microfiche Computer Program (Appendix)
`* Fee Transmittal Form fegsxREQSRey (3 pps., 5
`1.
`j
`
`
`(Submit an original and a duplicate for fee processing) w/o fee
`. Nucleotide and/or Amino Acid Sequence Submission
`2, [TotalPages|79|]Specification
`
`
`
` |
`(fapplicable, all necessary)
`(preferredarrangementset forth below)
`
`
`
`
`a.
`Computer Readable Copy
`- Descriptive title of the Invention
`b. [| Paper Copy(identical to computercopy)
`- Cross References to Related Applications
`
`- Statement Regarding Fed sponsored R & D
`
`c [| Statement verifying identity of above copies
`
`
`- Reference to Microfiche Appendix
`
`
`
`ACCOMPANYING APPLICATION PARTS
`- Backgroundof the Invention
`- Brief Summary of the Invention
`
`7.
`Assignment Papers (cover sheet & document(s))
`- Brief Description of the Drawings{/ffiled)
`
`
`[] 37 C.F.R.§3.73(b) Statement
`Powerof
`
`
`
`- Detailed Description
`8.
`(when there is an assignee)
`Attorney
`
`- Claim(s)
`of| English Translation Document(ffapplicable)
`- Abstract of the Disclosure
`
`HO. [| Information Disclosure
`Copies of IDS
`|
`[TotalSheets
`3. [xx] Drawing(s) (35 U.S.C. 113)
`
`
`.
`Statement (IDS)/PTO-1449
`Citations
`
`11 [7] Preliminary Amendment
`[Tota?rages|| J
`4. Oath or Declaration
`
`
`
`
`
`(2)
`io xx Return Receipt Postcard (MPEP 503)
`a. [] Newly executed (original or copy)
`
`
`(Should be specifically itemized)
`‘
`
`
`b. [| Copy from a prior application (37 C.F.R. § 1.63(d))
`
`* Smail Entity
`ey og
`
`
`(for continuation/divisional with Box 16 completed)
`43.
`Statement(s)
`Statementfiled in prior application,
`
`
`
`
`(PTO/SE/O9-12)
`Statusstill proper and desired
`
`
`
`[| DELETION OF INVENTOR(S
`
`
`Certified Copy of Priority Document(s)
`14.
`Signed statement attached deleting
`
`
`
`(if foreign priority is claimed)
`inventor(s) namedin the prior application,
`
`see 37 C.F.R. §§ 1.63(d)(2) and 1.33(b).
`
`
`
`,
`
`. .
`
`:
`
`
`
`
`
`16. Ifa CONTINUING APPLICATION,check appropriate box, and supply the requisite information below andin a preliminary amendment:
`Continuation C] Divisional [] Continuation-in-part (CIP)
`of prior application No:
`/
`
`Priorapplication information:—Examiner Group /Art Unit:
`Eor CONTINUATION or DIVISIONAL APPSonly: The entire disclosure of the prior application, from which an oath or declaration is supplied
`
`
`under Box 4b, is considered a part of the disclosure of the accompanying continuation or divisional application and is hereby incorporated by
`
`
`reference. The incorporation can only be relied upon whena portion has been Inadvertently omitted from the submitted application parts.
`17. CORRESPONDENCE ADDRESSee
`
`
`
`
`CustomerNumber orBar Code Label 3
`007265
`i
`or CI Correspondence address below
`
`
`
`
`Crer
`ware| C)ten(holt[om|le
`comments on the amountof time you are required to complete this form should be sent to the Chief Information Officer, Patent and Trademark Office,
`Washington, DC 20231. DO NOT SEND FEES OR COMPLETED FORMS TO THIS ADDRESS. SEND TO: Assistant Commissionerfor Patents,
`Box Patent Application, Washington, DC 20231.
`
`
`
`APPLE 1040
`
`1
`
`APPLE 1040
`
`

`

`d
`
`Peter L. Michaelson
`NJ, NY,PA Bars
`
`Robert M.: Wallace
`NY, CA Bars
`
`
`Edmond A. DeFrank
`CA Bar
`
`JohnC. Pokotylo
`NJ, NY Bars
`
`Michael P. Straub
`NJ, NY Bars
`
`MICHAELSON & WALLACE
`A Partnership ofIntellectual Property and Technology Lawyers
`
`Doc. No. MS-53/4(116623.1)
`
`September 14, 1998
`
`PATENT APPLICATION
`
`Assistant Commissioner of Patents
`Box Patent Application
`Washington, D. C.
`20231
`
`Christopher R. Baizan
`CA Bar
`
`Sir:
`
`Craig S. Fischer
`CA Bar
`
`
`Jeremiah G. Murray
`Se
`Patent Agent
`
`GfCounsel:
`Richard T. Lyon
`CABar
`
`Join T. Peoples
`NaBar
`Ronald L. Drumheller
`N¥ Bar
`
`Edvard M. Fink
`N&LWY, D.C. Bars
`Raabe
`
`
`
`Internet Address:
`mail@mandw.com
`
`Web Site:
`http:/Avww.mandw.com
`
`Please reply:
`
`Parkway 109 Office Ctr.
`328 NewmanSprings Rd.
`P.O. Box 8489
`Red Bank, NJ 07701
`Tel: 732-530-6671
`Fax: 732-530-6584 or
`732- 530- 6585
`Videoconference:
`732-224-0132 (ISDN)
`O 1001 Partridge Drive
`Suite 330
`Ventura, CA 93003
`Tel: 805-644-4035
`Fax: 805-644-8046
`
`Enclosed herewith for filing is the following
`
`utility patent application:
`
`Applicants: George G. ROBERTSON, Daniel C. ROBBINS,
`Maarten VAN DANTZICH
`
`Title of application: METHODS, APPARATUS AND DATA STRUCTURES
`FOR PROVIDING A USER INTERFACE, WHICH
`EXPLOITS SPATIAL MEMORY IN
`THREE-DIMENSIONS, TO OBJECTS AND WHICH
`VISUALLY PROXIMALLY LOCATED OBJECTS
`
`Total pages of specification: 79 (including 6 pages of
`claims [1-18] and i page
`of abstract)
`
`Sheets of drawing: 62 (FIGs. 1A-26)
`
`Docket No.: MS-53/4(116623.1)
`
`PATENT APPLICATION FILING FEE CALCULATION
`No. Filed
`Less
`Rate/Claim
`Fee
`
`Total
`Claims
`Independent
`Claims
`
`18
`
`_3
`
`-20
`
`-3
`
`0 x $22.00
`
`0 x $82.00
`
`Minimum Filing Fee
`
`$
`
`S
`
`00.00
`
`00.00
`
`$ 790.00
`
`***EXPRESS MAIL CERTIFICATE APPEARS ON LAST PAGE***
`
`-1-
`
`2
`
`

`

`Multiple Dependency Fee
`(if applicable - $270.00)
`
`$
`
`00.00
`
`50% Reduction for Small Entity
`(Independent Inventor, Non-profit
`Corporation, or Small Business
`Concern)
`- appropriate
`verified statement attached
`
`TOTAL FILING FEE
`TOTAL FILING FEE ENCLOSED
`
`$- 00.00
`
`5 790.00
`$
`00.00
`
`Also enclosed herewith for filing in connection
`
`with the enclosed application are:
`
`(2);
`
`Postcard Receipt
`Check;
`Utility Patent Application Transmittal;
`A Recordation Form Cover Sheet and an Assignment of
`the application to:
`;
`Oath;Declaration and Power of Attorney;
`
`
`
`LUTELLP
`
`Disclosure Statement;
`Letter referencing previously filed disclosure
`document; number
`filed
`Verified Statement Claiming Small Entity Status;
`Claim to priority (as noted below and included
`in declaration):
`Patent Application
`Filing date
`Serial Number
`Submission of Priority Document and a certified
`copy of a
`patent application or
`and
`inventor's certificate, filed
`serial no.
`, upon which a claim to
`priority is made; and
`
`;
`
`
`
`**k*EXPRESS MAIL CERTIFICATE APPEARS ON LAST PAGE***
`
`-?-
`
`3
`
`

`

`Other:
`
`
`Respectfully submitted,
`
`MICHAELSON & WALLACE
`
`aC Qk
`
`
`
`Pokotylo, Attorney
`John
`No. 36,242
`Reg.
`Customer No. 007265
`
`
`
`Iy
`
`
`
`
`***EXPRESS MAIL CERTIFICATION***
`
`"Express Mail" mailing label number: EL159129518US
`Date of deposit: September 14, 1998
`
`I hereby certify that this paper or fee is being
`deposited with the United States Postal Service "Express Mail
`Post Office to Addressee" service under 37 CFR 1.10 on the
`date indicated above and is addressed to the Assistant
`Commissioner of Patents, Box Patent Application, Washington,
`
`D.C. 20
`
`
` Signature
`
`f person making certification
`
`John C. Pokotylo
`Name of person making certification
`
`***EXPRESS MAIL CERTIFICATE APPEARS ON LAST PAGE***
`
`-3-
`
`4
`
`

`

`MS-53/4(116623.1)
`
`METHODS, APPARATUS AND DATA STRUCTURES FOR PROVIDING A
`USER INTERFACE, WHICH EXPLOITS SPATIAL MEMORY IN
`THREE-DIMENSIONS, TO OBJECTS AND WHICH VISUALLY GROUPS
`PROXIMALLY LOCATED OBJECTS
`
`§ 1.
`
`BACKGROUND OF THE INVENTION
`
`§ 1.1 Field of the Invention
`
` 10
`
`15
`
`The present invention concerns a user interface
`
`to objects, such as informational objects for example,
`
`which are stored on, or accessed via, a machine such as a
`
`computer for example.
`
`The present invention exploits the
`
`spatial memory of people.
`
`§ 1.2 Related Art
`
`A brief history of user interfaces is provided
`
`below. First, known user interfaces are introduced in
`
`§ 1.2.1.
`
`Then, user interfaces which facilitate
`
`information access, rather than information processing,
`
`are introduced in § 1.2.2. Finally, needs of users which
`
`have not yet been met by user interfaces are listed in
`
`§ 1.2.3.
`
`20
`
`§ 1.2.1 KNOWN USER INTERFACES
`
`The way in which people interact with computing
`
`machines has evolved over the last 50 or so years.
`
`25
`
`Initially,
`
`these machines were typically used as
`
`5
`
`

`

`~2-
`
`information processors, and in particular,
`
`for performing
`
`mathematical operations on numbers.
`
`People interacted
`
`with such early computing machines by punching and
`
`ordering cards to effect a sequence of commands.
`
`In
`
`later computing machines, also typically used to perform
`
`mathematical operations on numbers, people interacted
`
`with such computing machines by setting switches and
`
`viewing light emitting diodes to enter commands. With
`
`the advent of the desktop personal computer,
`
`10
`
`people-machine interaction evolved from the use a
`
`keyboard to enter lines of commands, discussed in
`
`§ 1.2.1.1 below,
`
`to the use of a keyboard and mouse to
`
`manipulate icon metaphors of the real world, discussed in
`
`§ 1.2.1.2 below.
`
`15
`
`§ 1.2.1.1 COMMAND LINES
`
`Early personal computers were also used to
`
`
`
`perform mathematical operations,
`
`from engineering
`
`20
`
`applications to accounting applications (e.g.,
`
`spreadsheets).
`
`In addition, such early personal
`
`computers were used to enter, store, and manipulate
`
`information, such as with word processing applications
`
`for example, and to effectively access stored
`
`25
`
`information, such as with relational database
`
`applications for example.
`
`People typically interacted
`
`with such computers by entering commands,
`
`in accordance
`
`with fairly rigid syntactical rules, or by entering data
`
`via a keyboard and viewing results via a video monitor.
`
`30
`
`Unfortunately, since the syntactical rules for
`
`
`
`
`
`6
`
`

`

`-3-
`
`interacting with a personal computer were typically not
`
`intuitive, people would have to invest a fairly
`
`substantial amount of time to master effective
`
`interaction with the computer. Until they mastered the
`
`syntactical rules, people would often become frustrated
`
`when using computers. Casual users would often conclude
`
`that learning to interact with computers would not be
`
`worth their time.
`
`Thus, computer use was usually limited
`
`to professionals (e.g., accountants, engineers, and
`
`10
`
`scientists) who needed the computational power of
`
`personal computers, and to hobbyists.
`
`§ 1.2.1.2 GRAPHICAL USER INTERFACES
`
`15
`
`20
`
`The advent of graphical user interfaces (or
`
`“GUIs") provided a more intuitive way for people to
`
`interact with computers.
`
`The casual user no longer
`
`needed to learn syntactical rules and enter sequences of
`
`commands. As personal computers penetrated business to
`
`an ever greater degree, probably due to popular and time
`
`Saving word processing and spreadsheet applications,
`
`operating systems began to employ user interfaces which
`
`used a office metaphor which included documents, folders,
`
`filing cabinets,
`
`trash cans,
`
`telephone directories, etc.
`
`25
`
`These so-called desktop GUIs have served their users well
`
`over the past decade or so. However,
`
`in addition to
`
`using computers for data entry, storage, and
`
`manipulation, people are using computers for access to
`
`information to an ever increasing degree. This recent
`
`30
`
`trend is discussed in § 1.2.2 below.
`
`
`
`7
`
`

`

`-~4-
`
`§ 1.2.2
`
`MIGRATION FROM INFORMATION PROCESSING
`AND DESKTOP METAPHOR TO INFORMATION
`ACCESS
`
`
`
` In recent decades, and in the past five (5)
`
`to
`
`ten (10) years in particular, computers have become
`
`interconnected by networks by an ever increasing extent;
`
`initially, via local area networks (or "LANs"), and more
`
`recently via LANs, private wide area networks (or "WANs")
`
`10
`
`and the Internet.
`
`The proliferation of networks,
`
`in
`
`conjunction with the increased availability of
`
`inexpensive data storage means, has afforded computer
`
`users unprecedented access to a wealth of content.
`
`Such
`
`content may be presented to a user (or "rendered")
`
`in the
`
`15
`
`form of text,
`
`images, audio, video, etc.
`
`The Internet is one means of inter-networking
`
`local area networks and individual computers.
`
`The
`
`popularity of the Internet has exploded in recent years.
`
`20
`
`Many feel that this explosive growth was fueled by the
`
`ability to link (e.g., via Hyper-text links) resources
`
`(e.g., World Wide Web pages)
`
`so that users could
`
`seamlessly transition from various resources, even when
`
`such resources were stored at geographically remote
`
`25
`
`resource servers. More specifically,
`
`the Hyper-text
`
`markup language (or "HTML") permits documents to include
`
`hyper-text links. These hyper-text links, which are
`
`typically rendered in a text file as text in a different
`
`font or color,
`
`include network address information to
`
`30
`
`related resources. More specifically,
`
`the hyper-text
`
`link has an associated uniform resource locator (or
`
`8
`
`

`

`-5-
`
`"URL") which is an Internet address at which the linked
`
`resource is located. When a user activates a hyper-text
`
`link, for example by clicking a mouse when a displayed
`
`cursor coincides with the text associated with the
`
`hyper-text link,
`
`the related resource is accessed,
`
`downloaded, and rendered to the user.
`
`The related
`
`resource may be accessed by the same resource server that
`
`provided the previously rendered resource, or may be
`
`accessed by a geographically remote resource server.
`
`Such transiting from resource to resource, by activating
`
`hyper-text links for example,
`
`is commonly referred to as
`
`"surfing".
`
`Although people continue to use computers to
`
`enter information, manipulate information, and store
`
`information,
`
`in view of the foregoing developments people
`
`are using computers to access information to an ever
`
`increasing extent.
`
`In a departure from the past,
`
`the
`
`information people want to access is often not created by
`
`them (which would typically reside on the person's
`
`desktop computer), or even by a company or group to which
`
`that person belongs (which would typically reside on a
`
`storage server, accessible via a local area network).
`
` 10
`
`20
`
`15
`
`Rather, given the world wide breadth of the Internet,
`
`the
`
`25
`
`information people want to access is usually created by
`
`unrelated third parties (or content providers).
`
`Unfortunately, GUIs using a desktop metaphor are not
`
`particularly well suited for such information access.
`
`9
`
`

`

`-6-
`
`New GUIs should therefore help people find
`
`information that they want, or that they might want.
`
`Unfortunately,
`
`the very vastness of available data can
`
`overwheim a user; desired data can become difficult to
`
`find and search heuristics employed to locate desired
`
`data often return unwanted data (also referred to as
`
`"noise™).
`
`Various concepts have been employed to heip
`
`users locate desired data.
`
`In the context of the
`
`Internet for example,
`
`some services have organized
`
`content based on a hierarchy of categories.
`
`A user may
`
`then navigate through a series of hierarchical menus to
`
`find content that may be of interest to them.
`
`An example
`
`of such a service is the YAHOO™ World Wide Web site on
`
`the Internet. Unfortunately, content,
`
`in the form of
`
`Internet "web sites" for example, must be organized by
`
`the service and users must navigate through a
`
`predetermined hierarchy of menus.
`
`If a user mistakenly
`
`believes that a category will be of interest or include
`
`what they were looking for, but the category turns out to
`
`be irrelevant,
`
`the user must backtrack through one or
`
`more hierarchical levels of categories.
`
` 10
`
`20
`
`15
`
`25
`
`30
`
`Again in the context of the Internet for
`
`example,
`
`some services provide "search engines" which
`
`search databased content or "web sites" pursuant to a
`
`user query.
`
`In response to a user's query, a rank
`
`ordered list, which includes brief descriptions of the
`
`uncovered content, as well as a hypertext links (text,
`
`10
`
`10
`
`

`

`~7~
`
`having associated Internet address information, which,
`
`when activated, commands a computer to retrieve content
`
`from the associated Internet address)
`
`to the uncovered
`
`content is returned.
`
`The rank ordering of the list is
`
`typically based on a match between words appearing in the
`
`query and words appearing in the content. Unfortunately,
`
`however, present limitations of search heuristics often
`
`cause irrelevant content
`
`(or "noise")
`
`to be returned in
`
`response to a query. Again, unfortunately,
`
`the very
`
`wealth of available content
`
`impairs the efficacy of these
`
`search engines since it is difficult to separate
`
`irrelevant content from relevant content.
`
`The foregoing means permit users to find
`
`content of interest to them, or to deliver content that
`
`may be of interest to a user. Unfortunately, neither of
`
`the foregoing means is particularly well suited when the
`
`user wants to "go back" to (or "relocate")
`
`information
`
`(or content),
`
`to revisit that information or to revisit a
`
`familiar content provider to view new information (or
`
`content). More specifically, a person typically does not
`
`want to go through steps of navigating through a
`
`hierarchy of menus, or entering a search query to get to
`
`favored content or favored content providers.
`
`§ 1.2.2.1 INFORMATION ACCESS
`USER INTERFACES AND THEIR
`LIMITATIONS
`
`In view of the shift towards using computers to
`
`access information, a number of user interfaces have been
`
`11
`
` 10
`
`15
`
`20
`
`25
`
`30
`
`11
`
`

`

`-8-
`
`developed to help people revisit or relocate information
`or content providers.
`A sample of these user interfaces,
`as well as perceived limitations of such user interfaces,
`
`is presented below.
`
`§ 1.2.2.1.1
`
`BOOKMARKS AND
`FAVORITES LISTS
`
`
`
`10
`
`15
`
`20
`
`25
`
`Some so-called "Internet browser" program
`services, such as Microsoft's Internet Explorer™ for
`example, permit people to create lists of favorite
`Internet locations (as located by a Uniform Resource
`
`Locator or "URL")
`
`represented by bookmarks.
`
`Unfortunately, as the number of bookmarks in a list
`increases, a person's ability to find a desired bookmark
`becomes more difficult. Although people can arrange and
`organize such lists of bookmarks into hierarchies,
`this
`requires some effort
`(mental effort is also referred to
`as "cognitive load")
`to organize and maintain the
`
`the use of hierarchies does not
`hierarchy. Moreover,
`fully exploit the spatial memory (This concept has also
`been referred to as "where it is is what it is".) of
`
`people.
`
`§ 1.2.2.1.2 INFORMATION VISUALIZER
`
`Recognizing the trend towards the use of
`
`computers for information access,
`
`the article:
`
`Robertson, et al.,
`
`"The Next Generation GUIs:
`
`30
`
`Information Visualization Using 3D Interactive
`
`Animation," Communications of the ACM, Vol. 35, No. 4,
`
`12
`
`12
`
`

`

`-9-
`
`(hereinafter referred to as "the
`pages 57-71 (April 1993)
` Information Visualizer article") discusses various
`
`proposals for an “Information Workspace". More
`three
`specifically, a three-dimensional rooms metaphor,
`dimensional hierarchical cone trees, and perspective
`
`walls are discussed as means to present information to
`
`users.
`
`Each of these techniques is introduced below.
`
`10
`
`Although the three-dimensional room metaphor
`exploits, at least to some degree, a person's spatial
`memory,
`the person has to manipulate objects or move to
`disambiguate images and reveal hidden information.
`
`Although the three-dimensional hierarchical
`
`15
`
`cone trees are useful for visualizing large amounts
`
`(e.g., 600 directories of 10,000 files) of hierarchical
`data,
`they lend themselves to predetermined hierarchies
`such as file directories, organizational structure, etc.
`
`That is, it is believed that the strength of
`
`20
`
`three-dimensional hierarchical cone trees is presenting
`
`information, not building or updating hierarchies of
`
`information.
`
`
`
`
`
`25
`
`The perspective wall permits information,
`to be
`having some sort of linear relationship or thread,
`presented in the relatively narrow aspect ratio of a
`typical video monitor. When an item is selected,
`the
`wall moves the item to a center portion of the wall as if
`
`it were a sheet in a player piano reel.
`
`Its intuitive
`
`30
`
`three-dimensional metaphor allows smooth transitions
`
`13
`
`13
`
`

`

`-10-
`
`thereby helping a user to perceive object
`among views,
`consistency. Files may be classified by their
`modification date. Although the perspective wall
`technique lends itself to information having a linear
`(e.g.,
`timeline)
`thread, it is less useful for other
`types of information, or for information in which a
`linear thread is unimportant.
`In fact,
`to have maximum
`impact,
`the perspective wall should be combined with an
`information retrieval technique for highlighting similar
`
`or related items to a selected item. Moreover, a user is
`
`confined to relating information to some type of a linear
`
`thread.
`
`Apart from the limitations of the
`three-dimensional hierarchical cone tree and perspective
`
`in these techniques,
`wall techniques introduced above,
`there is little user choice about where an informational
`
`object is located in the user interface -- it either has
`a spot in a hierarchy or a spot on a linear thread.
`Although it is true that a user can define a hierarchical
`structure, or determine what linear thread the
`
`informational objects are to be related, once the
`
`structure or thread is determined, user choice about
`
`where to locate the information object is removed.
`
`§ 1.2.2.1.3 WEB BOOK / WEB FORAGER
`
`Another article, Card, et al.,
`"The WebBook and
`Web Forager:
`An Information Workspace for the World-Wide
`Web," Proceedings of CHI
`'96, pp. 111-117 (April 13-18,
`
`14
`
` 10
`
`15
`
`20
`
`25
`
`30
`
`14
`
`

`

`~11-
`
`(hereafter referred to as "the WebBook article")
`1996)
`also recognized the trend towards the use of computers
`for information access.
`Focusing on the use of the
`Internet,
`the WebBook article noted, among other things,
`that web pages are often hard to find, users get lost and
`have difficulty relocating pages, and users have
`difficulty organizing found pages.
`The article then
`discusseda WebBook™ which simulates,
`in three
`dimensions, a physical book and includes pages, each of
`which correspond to a web page. Hypertext links are
`color coded to indicate whether the referenced web page
`is within the currently opened book or not.
`If the web
`page referenced by the link is in the book, activating
`the hypertext link will effect an animation of flipping
`pages to the desired web page.
`If the web page
`referenced by the link is not in the book, activating the
`hypertext link will close the currently opened WebBook
`and, if the web page is in another WebBook on a simulated
`pookshelf, will open that WebBook to the selected web
`page.
`The user can flip or ruffle through the pages of
`the WebBook using various input techniques. Portions of
`interest on a web page may be inspected with panning and
`zooming operations. The Document Lens feature described
`in the WebBook article is related to U.S. Patent
`
`
`
`
`10
`
`15
`
`20
`
`25
`
`No. 5,670,984 to Robertson (incorporated herein by
`
`reference).
`
`Although the WebBook uses screen space
`efficiently and uses a familiar metaphor, it has a number
`of limitations. First,
`the book metaphor limits the
`
`30
`
`15
`
`15
`
`

`

`-12-
`
`number of web pages that can be seen at one time -- most
`pages are occluded.
`Second,
`the book metaphor does not
`exploit spatial memory. That is, it is believed that
`user will only remember the ordered pages in a linear (or
`
`one-dimensional) manner.
`
`The WebBook article also discusses a Web
`
`Forager which embeds the WebBook and other objects ina
`hierarchical three-dimensional workspace.
`The workspace
`
`10
`
`includes four
`
`(4) hierarchical levels; namely a focus
`
`
`
`
`15
`
`20
`
`25
`
`place, an immediate memory space, a secondary storage
`area, and a tertiary storage area.
`In the focus place,
`objects are rendered in full size and can be acted upon
`by the user.
`The immediate memory space uses the
`metaphor of a desk top.
`The secondary storage area uses
`several tiers in Z space,
`in which objects can be moved.
`
`Finally the tertiary storage area uses a bookshelf
`metaphor.
`A book selected from the bookshelf will come
`up to the focus place and an object then in the focus
`place will be moved into the intermediate memory space.
`
`Although the Web Forager exploits, at least to
`
`some extent, spatial memory, particularly in the
`secondary storage area,
`the interaction between the focus
`
`immediate memory place, secondary storage area,
`place,
`and tertiary storage area may not be immediately obvious
`to new users. Further, if more than about 30 objects are
`
`placed in the secondary storage place, objects may become
`occluded. Finally,
`the use of tiers in the 4 dimension
`
`16
`
`16
`
`

`

`-13-
`
`of the secondary storage place limits the exploitation of
`
`a user's spatial memory.
`
`§ 1.2.2.1.4 WORKSCAPE
`
`Noting the concept of "what it is is where it
`the article, Ballay, “Designing Workscape™™:
`An
`is",
`Interdisciplinary Experience", Human Factors in Computing
`Systems, CHI
`'94, pp. 10-15 (April 1994)
`(hereafter
`referred to as "the Workscape article"), discusses a
`
`three-dimensional user interface for managing documents.
`
`As discussed at the web page on the Internet at
`http: //www.maya.com/Portfolio/workscape. html
`(downloaded
`July 7, 1998, revised 1998), Workscape permits users to
`drag documents in the X-Y plane, and also push and pull
`documents in the 2 dimension.
`
`Although Workscape exploits spatial memory, it
`is apparently not concerned with the problem of object
`occlusion.
`Furthermore, it is believed that separate
`
`in the X-Y
`user operations for (i) dragging an object
`plane, and (ii) pushing and pulling documents in the 4
`dimension, will be cumbersome for users in practice.
`
` Lo
`
`15
`
`20
`
`25
`
`30
`
`§ 1.2.2.1.5 MAPA
`
`The product MAPA, offered by Dynamic Diagrams
`of Providence, RI, uses a three-dimensional display to
`show the organization of web pages at a web site. As a
`web page is selected, its children in the web site
`
`17
`
`17
`
`

`

`-14-
`
`hierarchy are displayed.
`
`The pages are represented by
`
`sheets. As a user's cursor hovers over a page, a pop-up
`
`title is provided. Like the hierarchical cone trees
`discussed in § 1.2.2.1.2 above, although MAPA may be
`useful for visualizing large amounts of hierarchical
`data, it lends itself to a predetermined hierarchies
`(i.e., web pages of a web site). That is, it is believed
`that the strength of MAPA is presenting information, not
`building or updating hierarchies of information.
`
`§ 1.2.3 UNMET NEEDS
`
`As discussed above,
`
`there exists a need for a
`
`user interface, and in particular a graphical user
`
`Such a user
`to information or content.
`interface,
`interface should exploit spatial memory.
`For example,
`the user interface should simulate three dimensions, and
`should permit continuous movement
`in the simulated space,
`to exploit spatial memory to the fullest extent. Other
`means or cues for reinforcing the three-dimensional
`
`The user interface
`environment queues should be used.
`should also be intuitive to minimize the time needed for
`
`a user to become familiar with it.
`
`For example,
`
`the user
`
`interface should not be cumbersome or require too many
`
` 10
`
`15
`
`20
`
`25
`
`different types of inputs to manipulate objects.
`Finally,
`the user interface should provide intelligent
`
`help to the user.
`
`18
`
`18
`
`

`

`-15-
`
`§ 2.
`
`SUMMARY OF THE INVENTION
`
`The present invention provides a user
`interface, and in particular a graphical user interface,
`to organize and access information or content
`(also
`referred to as an "object").
`The present invention
`permits a user to view and organize all objects and to
`edit or otherwise work on a selected object by, for
`example, representing, graphically, objects or content
`with a low resolution image which can be added, moved, or
`deleted from a simulated three-dimensional environment on
`the user's video monitor.
`The present invention may use
`pop-up title or information bars for permitting a user to
`discern more about the object represented by a low
`resolution image.
`The present invention may use higher
`resolution image representations of objects, or "live"
`objects loaded within an associated application,
`for
`editing or otherwise working on a selected object. Other
`visual representations of objects may be used, such as
`schematic or iconic representations of the content of
`each object for example. Any and all visual
`representations of objects may be referred to as
`"thumbnails" in the following.
`
` 10
`
`15
`
`20
`
`25
`
`30
`
`The user interface of the present invention
`exploits spatial memory by, for example, simulating a
`plane located and oriented in three-dimensional space, or
`other three-dimensional landscape on which the object
`thumbnails may be manipulated.
`The plane or landscape
`may include visual landmarks for enhancing a user's
`
`19
`
`19
`
`

`

`-16-
`
`spatial memory. As the object thumbnails are moved about
`the landscape,
`the present
`invention may employ
`perspective views (perceived image scaling with
`distance), partial image occlusion,
`shadows, and/or
`spatialized audio to reinforce the simulated
`three-dimensional plane or landscape. Other audio cues
`may be used to indicate proximal relationships between
`object thumbnails, such as when an object thumbnail being
`"moved" is close to a pre-existing cluster of object
`thumbnails.
`An ancillary advantage of using a simulated
`three-dimensional landscape is that more objects can be
`represented, at one time, on a single display screen.
`
`The user interface of the present invention is
`
`intuitive and minimizes the time needed for a user to
`become familiar with it.
`The user interface of the
`present invention is neither cumbersome, nor does it
`require too many different types of inputs to manipulate
`the object "thumbnails".
`For example,
`the present
`invention may use inputs from a familiar input device
`such as a mouse or pointer to manipulate the object
`thumbnails.
`To minimize the number of different types of
`inputs required to manipulate the object thumbnails,
`the
`present invention may map two-dimensional inputs, such as
`moving a mouse on a mouse pad,
`to a three-dimensional
`The
`movement on the simulated three-dimensional display.
`location of the objects may be constrained to the plane
`or landscape.
`The present invention may also prevent one
`object thumbnail
`(or landscape feature)
`from totally
`occluding another object thumbnail so that it is not
`
`20
`
` 10
`
`15
`
`20
`
`25
`
`30
`
`20
`
`

`

`-17-
`
`To further reinforce the
`totally hidden from the user.
`simulated three-dimensional environment,
`the present
`invention may simulate head motion parallax.
`In this
`regard,
`the present invention may use a camera (or head
`or body mounted equipment)
`to detect the position or,
`alternatively,
`the orientation, of a user's head relative
`to a video monitor on which the user interface is
`
`rendered.
`
`the user interface of the present
`Finally,
`For
`invention can provide intelligent help to the user.
`example,
`the present invention may cluster, by rendering
`a visual boundary for example, object thumbnails which
`are arranged, by the user, relatively close to one
`another. Further,
`the present
`invention may employ a
`matching (e.g., correlation or similarity) algorithm to
`determine whether certain objects are related.
`Alternatively, objects may be related by an explicit
`selection or designation by a user or based on a property
`(e.g., age, storage location, etc.) of an object.
`The
`present invention may employ some type of visual
`indication, such as a colored halo around thumbnails of
`related objects for example, of related objects. This
`visual indication may be rendered continuously or,
`alternatively, upon an event, such as when an object
`thumbnail is made “active”. Audio indications may also
`be used to indicate when a thumbnail being manipulated by
`a user is in close proximity to a relevant pre-existing
`cluster. Other audio cues may also be used to indicate
`
` 10
`
`15
`
`20
`
`25
`
`30
`
`other relationships between thumbnails.
`
`21
`
`21
`
`

`

`-18-
`
`§ 3.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`The file of this patent contains at least one
`
`drawing executed in color. Copies of this patent with
`color drawing(s) will be provided by the Patent and
`
`Trademark Office upon request and payment of the
`
`necessary fee.
`
`Figure 1A is a personal computer on which the
`
`user interface of the present invention may be effected.
`
`Figure 1B is a machine on which the user interface of the
`
`present invention may be effected.
`
`Figure 2 is a diagram of processes and stored
`data which may be used to effect the user interface of
`
`the present invention.
`
`Figure 3 is an exemplary data structure of
`
`state data which may be temporarily stored and used when
`
`effecting the user interface of the present invention.
`
` 10
`
`15
`
`20
`
`Figure 4 is an exemplary data structure of a
`
`record of low resolution image data of an object which
`
`