(19) United States
`(12) Patent Application Publication (10) Pub. No.: US 2009/0113350 A1
`(43) Pub. Date:
`Apr. 30, 2009
`Hibino et al.
`
`US 200901 13350A1
`
`(54) SYSTEMAND METHOD FOR VISUALLY
`SUMMARIZING AND INTERACTIVELY
`BROWSNG HERARCHICALLY
`STRUCTURED DIGITAL OBJECTS
`
`Publication Classification
`
`(51) Int. Cl.
`G06F 3/048
`
`(2006.01)
`
`(76) Inventors:
`
`Stacie Lynn Hibino, San Jose, CA
`(US); Jiajian Chen, Atlanta, GA
`(US)
`Correspondence Address:
`Frank Pincelli, Patent Legal Staff
`Patent Legal Staff
`Eastman Kodak Company, 343 State Street
`Rochester, NY 14650-2201 (US)
`(21) Appl. No.:
`11/924,973
`
`(22) Filed:
`
`Oct. 26, 2007
`
`(52) U.S. Cl. ........................................................ 71.5/853
`
`ABSTRACT
`(57)
`A system and method of visually Summarizing a hierarchi
`cally structured set of digital objects and for facilitating effi
`cient access to Such objects through the selection of represen
`tative Summary objects. These digital objects are typically
`media objects such as digital image files, digital video clips,
`digital audio objects, such as “MP3 files, or other digital
`documents, such as text documents, that can be collected by
`a user and distributed over a variety of storage media and
`storage locations.
`
`HEIGHT 5 ?
`
`93
`
`ROOTNODE
`
`8O
`
`
`
`HEIGHT 4
`
`HEIGHT3
`
`HEIGHT2
`
`
`
`135
`
`3:) -833 -is:
`132/
`133/
`134
`
`
`
`142
`
`HEIGHTO
`7
`\
`7 7- -
`24 242
`22 222
`244. 245
`223
`243
`
`246
`
`7
`S.
`| N
`s
`231232 ( 234 235
`237 224, 225
`233
`236
`
`17
`It
`S.
`247 248 249 25 252
`254
`253
`255
`
`Petitioner Apple Inc. - Ex. 1041, p. 1
`
`

`

`Patent Application Publication
`
`Apr. 30, 2009 Sheet 1 of 18
`
`US 2009/0113350 A1
`
`10
`
`
`
`USB
`INTERFACE
`DEVICE
`32
`
`DVD
`DRIVE
`
`DIGITAL CAMERA
`6
`o
`
`40
`(o) DVD R-/R+
`
`30
`
`42
`
`CD-R/W
`DRIVE
`
`28
`
`DESKTOP
`INTERFACE
`
`12
`
`26
`AUDIO
`INTERFACE
`DEVICE
`
`
`
`
`
`DISPLAY
`INTERFACE
`DEVICE
`
`24
`
`CD-ROM OR CD-R/W
`
`44
`
`MOUSE
`
`KEYBOARD
`
`46
`
`O-48
`MICROPHONE
`
`G.
`
`SPEAKER
`
`14
`
`CPU
`
`
`
`16 N READ-ONLY
`MEMORY
`
`
`
`NETWORK
`ONNECTIO
`
`20 HARD DISK
`
`
`
`RANDOM
`22\,XSS
`MEMORY
`
`52
`
`VIDEO DISPLAY
`
`FIG. 1
`
`Petitioner Apple Inc. - Ex. 1041, p. 2
`
`

`

`Patent Application Publication
`
`Apr. 30, 2009 Sheet 2 of 18
`
`US 2009/0113350 A1
`
`DEPTH
`
`DEPTH 2
`
`DEPTH...
`
`90
`
`t
`
`
`
`
`
`
`
`DEPTH D
`
`-
`
`D-LARGEST DEPTH
`OF HERARCHY
`
`Nos
`
`a n
`
`s
`
`FIG 2
`
`Petitioner Apple Inc. - Ex. 1041, p. 3
`
`

`

`Patent Application Publication
`
`US 2009/0113350 A1
`
`
`
`
`
`992 #292 892 292 | 92
`
`
`
`
`
`Petitioner Apple Inc. - Ex. 1041, p. 4
`
`

`

`Patent Application Publication
`
`
`
`
`
`
`
`08EC]ON LOOB96 °C
`
`9 LH3)|EH
`
`Petitioner Apple Inc. - Ex. 1041, p. 5
`
`

`

`Patent Application Publication
`
`Apr. 30, 2009 Sheet 5 of 18
`
`US 2009/0113350 Al
`
`
`
`l2
`
`.
`
`gS‘Sid
`
`
`
`AARLYS3GONHANIVINOO
`
` 8229626cLez[[3%]|eee|EaEa|ona]aN|eee||eee.|Eazz
`‘C=H1d3dLYAHOYVHAIH4O1SA57
`‘SZN“b2N‘©2N22N“LZN
`962ezLezbetbe
`
`TSAq1LVSSQONJV31ONYYANIVLNOOTIVMOHS
`
`AINOMOHS‘@=HLid3dLYAHOWVHSIHAOTAAR7
`
`
`
`NINOSHCONYANIVLNODATEISSOdLSAMO1MOHS
`‘VeN‘oN‘CoN“LON‘LLN
`eoeLe?BreLeeLee
`‘SS31YO2@=H1Ld3dLVAHOYVHY3IH40TSAR
`9/2vedLyeLed
`8Le_)
`
`(L=HLd3d)AHOUVHSIHSOTSAST-dOL
`‘LN-cLN“LEN
`
`gS‘9l4
`
`S‘Dld
`
`022
`
`O22
`
`wa
`042
`
`O22J
`
`VS‘Sls
`
`ole
`
`OS‘ld
`
`“SON
`
`9€¢
`
`Petitioner Apple Inc. - Ex. 1041, p. 6
`
`Petitioner Apple Inc. - Ex. 1041, p. 6
`
`
`
`
`
`
`
`
`
`

`

`Patent Application Publication
`
`Apr. 30, 2009 Sheet 6 of 18
`
`US 2009/0113350 Al
`
`
`
`AINOMOHS‘2=LHDISHLYAHOYVHYSIH3OTSAR
`
`OL¢
`
`0L¢
`
`822
`
`O22
`
`9‘Did
`
`V9‘Sis
`
`cle~PSSLSeBed‘ECLN‘SEN“LON
`
`
`TAALYSACONYSNIVINOO
`946lveLoe:“Lou“hLU
`
`LSAMO1
`
`
`(L=LHDISH)AHOUVHAIH3OSHSNIVLNOOD13A37
`XINOSSQONYANIVLNODS1dIlSSOdLSSHOIHMOHS
`
`
`
`‘$S31YO2=LHOISHLVAHOUVHAIHSO13A37
`
`9°Dl4d99‘Hld
`
`Petitioner Apple Inc. - Ex. 1041, p. 7
`
`Petitioner Apple Inc. - Ex. 1041, p. 7
`
`
`
`

`

`Patent Application Publication
`
`Apr. 30, 2009 Sheet 7 of 18
`
`US 2009/0113350 A1
`
`HERARCHICAL DATA
`
`
`
`
`
`
`
`
`
`GE
`CONTANER
`NODES ONLY?
`
`300
`
`305
`
`NO
`
`302
`
`USES EED
`
`315
`
`
`
`ADD ALL CONTAINER yes
`NODESAT DEPTH-1
`TO LISTTOPROCESS
`
`GET LOWEST
`CONTAINER
`NODES UP TO 2
`
`GET FIRST NODE
`OF
`LISTTOPROCESS
`
`ADD ALL
`CONTANER
`NODESAT
`DEPTH-D
`TO NODESS
`
`
`
`
`
`
`
`ADD ALL NODES
`A DEPTHED
`TO NODESLIST
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`(NODE HAS
`CONTAINER
`NODES) AND
`(NODE DEPTH
`< D) 2
`
`ADD CHILDREN
`CONTAINER
`NODESO
`LISTTOPROCESS
`
`GE NEXT NODE
`FROM
`LIST TOPROCESS
`
`ADD NODE TO
`NODESLIST
`
`MORE NODES
`N
`STTOPROCE2
`
`
`
`RETURN
`NODESLIST
`
`
`
`33O
`
`335
`
`Petitioner Apple Inc. - Ex. 1041, p. 8
`
`

`

`Patent Application Publication
`
`Apr. 30, 2009 Sheet 8 of 18
`
`US 2009/0113350 A1
`
`HERARCHICAL DATA
`
`300
`
`345
`
`
`
`
`
`
`
`USER-SELECTED
`HEIGHT H
`
`GET HIGHEST
`CONTAINER
`NODES UP TO
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`ADD ALL
`CONTAINER
`NODESAT
`HEIGHT-1 TO
`LISTTOPROCESS
`
`GE FIRST NODE
`OF
`LISTTOPROCESS
`
`
`
`
`
`352 N
`(PARENT NODEP
`<> ROOTNODE)
`AND (HEIGHT(P)
`<= H) ?
`
`
`
`ADD ALL
`CONTANER
`NODESAT
`HEIGHT-H
`TO NODESLIST
`
`
`
`
`
`ADO NODE TO
`NODESLIST
`
`F P NOT IN
`LSTTOPROCE
`SS, ADD PTO
`LISTTOPROCE
`SS
`
`GET NEXT NODE
`FROM
`LISTOPROCESS
`
`MORE NODESN
`LSTTOPROCESS
`2
`
`
`
`RETURN
`NODESLIST
`
`358
`
`YES
`FIG. 8
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`Petitioner Apple Inc. - Ex. 1041, p. 9
`
`

`

`Patent Application Publication
`
`Apr. 30, 2009 Sheet 9 of 18
`
`US 2009/0113350 A1
`
`382
`
`GET NEXT NODE
`
`
`
`
`
`
`
`
`
`
`
`
`
`NODESLIST: LIST OF
`NODES TO DISPLAY
`
`375
`
`38O
`
`385
`
`390
`
`GET FIRST NODE
`
`GE SUMMARY
`VISUAL OBJECT FOR
`CURRENT NODE
`
`INSERT SUMMARY
`VISUAL OBJECT INTO
`LIST TODISPLAY,
`BASED ON
`SECRUENCE
`FUNCTION
`
`395
`
`MORE
`NODES
`
`NO
`
`DISPLAY
`LISTTODISPLAY
`
`398
`
`FIG. 9
`
`Petitioner Apple Inc. - Ex. 1041, p. 10
`
`

`

`Patent Application Publication
`
`Apr. 30, 2009 Sheet 10 of 18
`
`US 2009/0113350 A1
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`USER-SELECTED
`NODE FROM
`LISTTODISPLAY2
`
`
`
`
`
`
`
`NODE =
`CONTANER
`NODE2
`
`NO
`
`400
`
`YES
`
`- 404
`
`406
`N
`
`GET LIST OF CHILD
`ELEMENTS OF NODE
`
`GET FIRST CHILD
`ELEMENT
`
`41 O
`
`
`
`
`
`408 \
`
`GET NEXT CHLD
`ELEMENT
`
`
`
`CHILD
`ELEMENT =
`NODE WITH
`CHILDRENT
`
`44
`
`INSERTELEMENT
`NODE INTO
`LSTTODISPLAY2
`
`INSERT CHILD
`ELEMENT INTO
`LISTTODISPLAY2,
`BASED ON
`SECRUENCE
`FUNCTION
`
`
`
`
`
`
`
`
`
`418
`
`MORE CHILD
`ELEMENTS?
`
`NO
`
`RETURN
`LSTTODISPLAY2
`
`FIG 10
`
`Petitioner Apple Inc. - Ex. 1041, p. 11
`
`

`

`Patent Application Publication
`
`US 2009/0113350 A1
`
`
`
`Petitioner Apple Inc. - Ex. 1041, p. 12
`
`

`

`Apr. 30, 2009 Sheet 12 of 18
`
`US 2009/0113350 Al
`
`
`
`osPrgl2
`
`LSP
`
`ee]Fe)]Fe]
`
`89“DId4dOELNSAGONSONOILOSTASYASNAOSLINSAYSAILYSACONHSNIVLNODAINOMOHS
`
`
`
`
`‘C=LHDISHLVAHOUVHSIHAO13A37
`
`Patent Application Publication
`
`OSP
`
`
`
`82¢e122
`
`LSP
`
`dS“‘DIl4dJO122SGONJONOLLOSISSHSNJOSLINSAYT3A31LYSSCON431CNVHSNIVLNOOTVMOHS
`
`'2=H1LdadLYAHOHVHSIH401537ke
`
`VelSidneaS‘Old
`
`
`
`Gel‘Sld
`
`Zi‘DI
`
`Petitioner Apple Inc. - Ex. 1041, p. 13
`
`Petitioner Apple Inc. - Ex. 1041, p. 13
`
`
`
`

`

`Patent Application Publication
`
`Apr. 30, 2009 Sheet 13 of 18
`
`US 2009/0113350 A1
`
`
`
`1 t i 1 I ? ? ? ? ? ? ? ? ? ? i t i u
`
`Petitioner Apple Inc. - Ex. 1041, p. 14
`
`

`

`Patent Application Publication
`
`Apr. 30, 2009 Sheet 14 of 18
`
`US 2009/0113350 A1
`
`
`
`0
`4
`2
`
`Þl '91-'
`
`Pae
`
`! ! ! ! ! ! ! ! ! ! ) ? ? ? ? ? ? ? 1
`
`Petitioner Apple Inc. - Ex. 1041, p. 15
`
`

`

`Patent Application Publication
`
`Apr. 30, 2009 Sheet 15 of 18
`
`US 2009/0113350 A1
`
`
`
`Petitioner Apple Inc. - Ex. 1041, p. 16
`
`

`

`Patent Application Publication
`
`Apr. 30, 2009 Sheet 16 of 18
`
`US 2009/0113350 A1
`
`
`
`Caen ? O CD
`No.=B?HTETE
`
`C)
`
`
`
`suo?dO M3IAGO
`
`Petitioner Apple Inc. - Ex. 1041, p. 17
`
`

`

`Patent Application Publication
`
`Apr. 30, 2009 Sheet 17 of 18
`
`US 2009/0113350 A1
`
`
`
`
`
`900 9002700Z
`
`Petitioner Apple Inc. - Ex. 1041, p. 18
`
`

`

`Patent Application Publication
`
`Apr. 30, 2009 Sheet 18 of 18
`
`US 2009/0113350 A1
`
`
`
`
`
`| 19
`
`
`
`
`08EGION LOOBze /
`
`|-0 H.LdE1C]
`
`
`
`
`
`[][][][][][][][][][] [])', [][] [][][] [][][][][][][][][] º fila=0
`
`
`
`
`
`9 H.LdEG]
`
`Petitioner Apple Inc. - Ex. 1041, p. 19
`
`

`

`US 2009/01 13350 A1
`
`Apr. 30, 2009
`
`SYSTEMAND METHOD FOR VISUALLY
`SUMMARIZING AND INTERACTIVELY
`BROWSNG HERARCHICALLY
`STRUCTURED DIGITAL OBJECTS
`
`FIELD OF THE INVENTION
`0001. The present invention relates to providing a visual
`overview pertaining to hierarchically organized digital
`objects. In particular, the present invention pertains to the
`formation and display of ordered visual representations,
`where, each displayed visual representation symbolizes a
`different node, wherein all symbolized nodes belong to one
`level in the object hierarchy. The visual representation can be
`obtained from nodes descendant from the symbolized node,
`or generated separately.
`
`BACKGROUND OF THE INVENTION
`0002 Increased computer usage and creation of digital
`objects along with decreasing costs in removable memory,
`fixed hard disks, and network access have all contributed to
`the increase in the number of digital objects any individual
`person may own or want to access. Computer users have
`moved from tens or hundreds of files to thousands of files or
`more. Similarly, digital cameras have decreased in physical
`carrying size and increased in memory capacity, making it
`easy for users to carry cameras with them more often and
`capture more pictures and video clips per picture-taking
`opportunity. As personal digital media collections increase in
`magnitude, they are also becoming more and more cumber
`Some for users to manage and access.
`0003 Typically, as in a file system, users can easily view
`digital objects by browsing a hierarchical directory and file
`structure. Tree based representations are typically provided
`for users in a graphical user interface (GUI), making it simple
`for users to view any objects in any given directory. Unfortu
`nately, it is challenging for users to access object leaf nodes
`from more than one directory at once. In addition, users are
`not provided with any visual Summary information about
`what is contained within a directory, especially not for objects
`that are deeply nested below any given directory. Further
`more, a file system view does not provide flexibility for view
`ing other hierarchical organizations based on other data facets
`for the same set of digital objects. That is, if a user creates a
`folder and file structure based on temporal information such
`as year, then month, etc., then the user is limited to browsing
`only by that temporal information within the file browser.
`They cannot, for example, use the file browser to browse the
`same files based on a hierarchical content-based categoriza
`tion of the same data—not without first needing to duplicate
`files and create appropriate folders. Some modern digital
`object organization Software provide Support for viewing dif
`ferent data facets (e.g., for browsing by tags or date in a photo
`collection), but they do not provide a general approach for
`browsing any facet of hierarchical data. Accordingly, a need
`exists in the art for an improved way to review and organize
`digital objects.
`
`SUMMARY
`0004. The above-described problems are addressed and a
`technical solution is achieved in the art by a system and a
`method for providing a visual Summary pertaining to a col
`lection of digital assets, according to the present invention. In
`one embodiment of the present invention, information per
`
`taining to a collection of digital objects is provided by a
`computer implemented method including the steps of locat
`ing a collection of digital objects that are stored on a computer
`system and which are organized in a hierarchical fashion, and
`then simultaneously displaying visual representations of
`digital objects that belong to a selected level of the hierarchy.
`This is a useful method because computer users typically
`group and label text documents and image files in Such a
`hierarchical fashion, wherein nested folders, subfolders, and
`files are appropriately labeled. Digital objects of the present
`invention may also include video clips and presentation docu
`ments. The visual representations may include thumbnails
`versions of digital objects, a manually created or computer
`generated representation, an icon, and a montage.
`0005. Further embodiments of the present invention
`include recognizing depth and height levels of a hierarchy and
`their interrelationships, and selecting Such levels for generat
`ing visual representations to depict the levels. Other embodi
`ments of the present invention include recognition of leaf
`nodes and container nodes of the hierarchy, and their ancestry
`in the hierarchy. Another embodiment of the present inven
`tion is selection of a pattern to display the visual representa
`tions, such as in a spiral pattern. The present invention can be
`used to visually Summarize hierarchies based on time, events,
`people, image and document contents, and geographic loca
`tion.
`0006. Other embodiments of the present invention include
`a computer system suitably programmed to implement the
`methods of this invention. Such a computer system includes a
`processor and memory for organizing and storing a plurality
`of digital objects in tree hierarchy. The processor executes
`programs for visually Summarizing certain ones of the digital
`objects that belong to a selected level of the hierarchy. A
`display screen is coupled to the computer system for display
`ing visual representations comprising the visual Summary. In
`response to a user selecting a depth of the hierarchy to display,
`the system can then display a visual Summary of the corre
`sponding level of the hierarchy, and so on for additional lower
`depths. This is a top-down embodiment of the present inven
`tion (e.g. hierarchy depth). The system can also start with a
`visual Summary of a lowest level of the hierarchy and Succes
`sively display; higher levels, which is referred to as a bottom
`up embodiment of the present invention (e.g. hierarchy
`height).
`0007. Other embodiments that are contemplated by the
`present invention include computer readable media and pro
`gram storage devices tangibly embodying or carrying a pro
`gram of instructions readable by machine or a processor, for
`having the machine or computer processor execute instruc
`tions or data structures stored thereon. Such computer read
`able media can be any available media, which can be accessed
`by a general purpose or special purpose computer. Such com
`puter-readable media can comprise physical computer-read
`able media such as RAM, ROM, EEPROM, CD-ROM, DVD,
`or other optical disk storage, magnetic disk storage or other
`magnetic storage devices, for example. Any other media
`which can be used to carry or store software programs which
`can be accessed by a general purpose or special purpose
`computer are considered within the scope of the present
`invention.
`0008. These, and other aspects and objects of the present
`invention will be better appreciated and understood when
`considered in conjunction with the following description and
`the accompanying drawings. It should be understood, how
`
`Petitioner Apple Inc. - Ex. 1041, p. 20
`
`

`

`US 2009/01 13350 A1
`
`Apr. 30, 2009
`
`ever, that the following description, while indicating pre
`ferred embodiments of the present invention and numerous
`specific details thereof, is given by way of illustration and not
`of limitation. Many changes and modifications may be made
`within the scope of the present invention without departing
`from the spirit thereof, and the invention includes all such
`modifications. The figures below are not intended to be drawn
`to any precise scale with respect to size, angular relationship,
`or relative position.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`0009. The present invention will be more readily under
`stood from the detailed description of exemplary embodi
`ments presented below considered in conjunction with the
`attached drawings, of which:
`0010 FIG. 1 is a block diagram of a computer system
`capable of practicing various embodiments of the present
`invention;
`0011
`FIG. 2 is an example abstract tree representation of
`hierarchical data;
`0012 FIG. 3 is an example tree representation of hierar
`chical data, presenting nodes by tree depth;
`0013 FIG. 4 is an example tree representation of hierar
`chical data, presenting nodes by tree height;
`0014 FIG. 5 shows examples of nodes and data from FIG.
`3 to display, based on top-down browsing of the hierarchy:
`0015 FIG. 6 shows examples of nodes and data from FIG.
`4 to display, based on bottom-up browsing of the hierarchy;
`0016 FIG. 7 is a flow chart for generating the list of nodes
`to display for top-down browsing of the hierarchy:
`0017 FIG. 8 is a flow chart for generating the list of nodes
`to display for bottom-up browsing of the hierarchy:
`0018 FIG. 9 is a flow chart for generating a first ordered
`list of visual objects to display, based on list of nodes to
`display generated by FIG. 7 or FIG. 8:
`0019 FIG. 10 is a flow chart for generating a second
`ordered list of visual objects to display;
`0020 FIG. 11 shows examples of leaf nodes from FIG. 3
`to display, based on selection of different nodes depicted in
`FIG. 5;
`0021 FIG. 12 shows examples of leaf nodes from FIG. 3
`to display, based on selection of a node depicted in FIG. 5 or
`a node depicted in FIG. 6;
`0022 FIG. 13 shows a screen capture of a working proto
`type implementing an embodiment of the present invention,
`displaying thumbnail images of a first list in a grid layout:
`0023 FIG. 14 shows a screen capture of the same working
`prototype, displaying thumbnail images of a first list in a
`spiral layout;
`0024 FIG. 15 shows a screen capture of the same proto
`type, after a thumbnail from FIG. 14 has been selected;
`0025 FIG. 16 shows a screen capture of the same proto
`type, with an interactive bar plot filter for filtering the current
`data being displayed;
`0026 FIG. 17 shows the results of expanding all thumb
`nail images of FIG.16; and
`0027 FIG. 18 is an example tree representation of hierar
`chical data, presenting nodes by tree depth.
`
`0028. It should be understood that the attached figures are
`for purposes of illustrating the concepts of the invention and
`may not be drawn to scale.
`
`DETAILED DESCRIPTION
`0029 Embodiments of the present invention relate to an
`effective method of representing a hierarchically structured
`collection of digital objects, and for facilitating efficient
`access to Such objects through the selection of individual
`thumbnails or nodes in the hierarchy. These digital objects are
`typically media objects such as digital image files, digital
`video clips, digital audio objects, such as "MP3 files, or
`other digital documents that can be collected by a user and
`distributed over a variety of storage media and storage loca
`tions. However, these objects need not be limited to multime
`dia objects. Text files can also be represented according to the
`methods of the present invention.
`0030. When users only have a small number of digital
`objects, they can easily organize them hierarchically using a
`standard file and folder representation as typically provided
`by a modern computer operating system. For example, con
`sider a folder hierarchy for pictures viewed in the File
`Explorer application of Microsoft Windows XP. Using such a
`standard file and folder hierarchy, users can access items in a
`straightforward manner by clicking on individual folders.
`They can opt to view thumbnails of images or view them in a
`“filmstrip' format, enabling them to preview images without
`having to open them. Unfortunately, when users’ collections
`grow to thousands of objects, the simple file and folder access
`available via the operating system can become cumbersome.
`In order to access media deep in a file system structure, users
`have to expand each Subfolder along the way. They also
`cannot view the contents of two subfolders together unless
`they open two separate File Explorer windows and navigate to
`the appropriate place. Although modern file system views can
`provide a visual Summary of media files Such as showing
`thumbnails of underlying content on folder icons, such visual
`summaries are only provided for one level of depth in a
`hierarchy—i.e., for only the immediate “child' media of a
`given folder node. Two major limitations of using a current
`standard file and folder hierarchy organization for accessing
`hierarchically organized media today are thus: 1) visual rep
`resentations are not available for deeply nested media; and 2)
`users are limited to accessing only immediate child media at
`any given node in the hierarchy (i.e., they do not have easy
`access to all descendant media from any node at any level in
`the hierarchy).
`0031. In accordance with an embodiment of the present
`invention, a more compact and efficient method of browsing
`a hierarchically organized set of documents such as a digital
`image collection stored in folders and files is to provide a
`visual Summary of nodes of the set, based on user-selected
`level of the hierarchy to view. The level of hierarchy can be
`based on node depth, as described in more detail below in
`relation to FIG. 3, or node height, as explained in more detail
`below in relation to FIG. 4. This provides the user with a
`visual Summary of a horizontal slice through, or a level of the
`hierarchy. Furthermore, in accordance with another embodi
`ment of the present invention, given a visual Summary list of
`nodes as described above, users can easily access all descen
`dant leaf nodes of a selected Summary node, independent of
`the height of the leaves.
`0032 For example, consider the typical case where users
`organize their photo collection on their home computer in file
`
`Petitioner Apple Inc. - Ex. 1041, p. 21
`
`

`

`US 2009/01 13350 A1
`
`Apr. 30, 2009
`
`folders based on date and event. At the top level of their photo
`collection, they have file folders for each year. Within each
`year folder, they have folders organized by month and event
`information (e.g., events such as a birthday party, vacation,
`wedding, day hike, holiday, etc.). Within each of those fold
`ers, they may have additional Subfolders, usually depending
`on the number of pictures captured for a given event. For
`example, pictures from a vacation to Europe might be in a
`folder called “May 2007 trip to Europe” and then further
`divided into country visited folders and country folders could
`be further divided into major cities, activities or famous land
`marks. In accordance with an embodiment of the present
`invention, the user could choose to view the top level folder
`nodes which, in this example, would present the user with one
`representative visual Summary object per year. Such a visual
`Summary object could be a single picture from a particular
`year Such as the first picture of the year or a picture from a
`Subfolder containing the most pictures. The visual Summary
`object could also be a visual summary that is created by the
`System or the user, Such as a montage (e.g., a montage com
`posed of pictures from different subfolders). Once presented
`with the visual Summary of years, users can select a year node
`to view all images captured during that year, independent of
`how deeply nested those images might be within the year
`folder.
`0033. In the above example, year folders represent nodes
`at depth=1 in the hierarchy. Moving down one level in the
`hierarchy (i.e. in all year folders), folders within each year
`folder (month folders and event information folders) are at
`depth 2 and can represent user-specified events that took
`place during that year. Thus, in one embodiment of the
`present invention, users can choose to view folder nodes at
`depth-2 to get a visual overview of all major events of all
`years at once.
`0034. The present invention is not limited to browsing the
`hierarchy from the top-down by depth level, but also provides
`browsing the hierarchy from the bottom up by node height
`level. As a brief example, given a sample hierarchy as illus
`trated in FIG. 3, where nodes are presented by depth level,
`FIG. 4 shows how the same hierarchy can be depicted interms
`of node height. Folder n12112 shown in FIG.3 and in FIG. 4
`represents a node having a depth level=1 and a height level=4
`and which contains several depths of folders, similar to that
`described above for a vacation to Europe. Consider again the
`case of the example above of a user's collection organized by
`year folders and then by event folders and so on. As men
`tioned above, events may vary in size and subfolder depth,
`typically dependent on the number of pictures captured for an
`event. The bottom most subfolders of such a collection repre
`sentanatomic-level of events for users. In accordance with an
`embodiment of the present invention, bottom-up browsing of
`events provides users with easy access to such atomic-level
`events, independent of the depth of the corresponding nodes.
`Thus, this would allow a user to see the “Eiffel Tower' node
`of pictures right next to the “Roman Coliseum' node of
`pictures, even if they were at different depths in the hierarchy,
`as long as they were at the same height in the hierarchy.
`0035 FIG. 1 shows one type of system for practicing an
`embodiment of the present invention. In this example, the
`system includes a computer 10, which typically includes a
`keyboard 46 and mouse 44 as input devices communicatively
`connected to the computer's desktop interface device 28. The
`term “computer is intended to include any data processing
`device. Such as a desktop computer, a laptop computer, a
`
`mainframe computer, a personal digital assistant, a cell
`phone, a digital camera, a Smart phone device, and/or any
`other device for processing data, and/or managing data, and/
`or handling data, whether implemented with electrical and/or
`magnetic and/or optical and/or biological components, and/
`or otherwise. Any of the above-mentioned computers, need
`not include all the components illustrated in FIG. 1. The
`system illustrated in FIG. 1 is intended to cover many
`embodiments of a computer system. Therefore, taken alto
`gether, the system illustrated is more than sufficient for imple
`menting embodiments of the present invention. The phrase
`“communicatively connected' is intended to include any type
`of connection, whether wired, wireless, or both, between
`devices, and/or computers, and/or programs in which data
`may be communicated. Further, the phrase “communica
`tively connected” is intended to include a connection between
`devices and/or programs within a single computer, a connec
`tion between devices and/or programs located in different
`computers, and a connection between or within devices not
`located in computers.
`0036 Output from the computer 10 is typically presented
`on a video display 52, which may be communicatively con
`nected to the computer 10 via the display interface device 24.
`The display may also be disposed in the same housing as the
`computer 10. Internally, the computer 10 contains compo
`nents such as a CPU 14 communicatively connected to bus 12
`and computer-accessible memories, such as read-only
`memory 16, random access memory 22, and a hard disk drive
`20, all communicatively connected to the bus, and which may
`retain some or all of the digital objects referred to herein. The
`phrase “computer-accessible memory” is intended to include
`any computer-accessible data storage device, whether Vola
`tile or nonvolatile, electronic, magnetic, optical, or otherwise,
`including but not limited to, floppy disks, hard disks, compact
`discs, DVDs, flash memories, ROMs, and RAMs.
`0037. The CPU 14 communicates with other devices over
`data bus 12. The CPU 14 executes software that can be stored
`on the computer accessible memories. In addition to hard disk
`drive 20, the computer 10 may also contain other connectible
`computer-accessible memory drives for reading and writing
`data and software of various types. This may include a CD
`RW drive 30 for reading and writing various CD media 42 as
`well as a DVD drive 32 for reading and writing to various
`DVD media 40. Audio can be input to the computer 10
`through a microphone 48, or other storage device, for
`example, an MP3 player, communicatively connected to an
`audio interface device 26. Audio playback can be heard via a
`speaker 50, or other listening devices, also communicatively
`connected to an audio interface device 26. A digital camera 6.
`and most any of the devices discussed herein, or other image
`capture or storage device can be communicatively connected
`to the computer 10 through, for example, the USB interface
`device 34 to transfer digital objects from the camera 6 to the
`computer accessible memories and Vice-versa. Finally, the
`computer 10 can be communicatively connected to an exter
`nal network 60 via a network connection device 18, thus
`allowing the computer to access digital objects from other
`computers, devices, or data-storage systems communica
`tively connected to the network. A "data-storage system” may
`include one or more computer-accessible memories, and may
`be a distributed data-storage system including multiple com
`puter-accessible memories communicatively connected via a
`plurality of computers and/or devices, or over a network. On
`the other hand, a data storage system need not be a distributed
`
`Petitioner Apple Inc. - Ex. 1041, p. 22
`
`

`

`US 2009/01 13350 A1
`
`Apr. 30, 2009
`
`data-storage system and, consequently, may include one or
`more computer-accessible memories located within a single
`computer or device.
`0038 A collection of digital objects can reside exclusively
`on the hard disk drive 20, compact disc 42, DVD 40, or on
`remote data storage devices, such as a networked hard drive
`accessible via the network 60. A collection of digital objects
`can also be distributed across any or all of these storage
`locations.
`0039. A collection of digital objects may be represented
`by a database that uniquely identifies individual digital
`objects (e.g., Such as a digital image file) and their corre
`sponding location(s) in the computer memories. It will be
`understood that these digital objects can be media objects or
`non-media objects. Media objects can be digital still images,
`Such as those captured by digital cameras, audio data Such as
`digital music or Voice annotations, digital video clips with or
`without sound. Media objects could also include files pro
`duced by graphic or animation Software such as those pro
`duced by Adobe Photoshop or Adobe Flash. Non-media
`objects can be text documents such as those produced by
`word processing software or other office-related documents
`Such as spreadsheets or email. A database of digital objects
`can be comprised of only one type of object or any combina
`tion.
`0040. Once a collection of digital objects is associated,
`Such as in a database or by another mechanism of associating
`data, the objects can be hierarchically organized according to
`a user created hierarchy (e.g., as in a traditional folder (i.e.,
`directory) and file organization available at the operating
`system level). Alternatively, a hierarchical organization can
`be automatically derived based on information about the digi
`tal objects. For example, using the creation date of the digital
`object, a hierarchy of folders or container nodes can be
`formed based on year, month, and then day of month, only
`creating container nodes when necessary. That is, when a
`time-based hierarchy only includes years, months or days for
`which digital objects exist within the database. For the case of
`digital image files, the EXIF header, well known in the art, of
`an image captured from a digital camera can be used to
`provide the capture date for an image.
`0041
`FIG. 2 is an example abstract tree representation of
`hierarchical data presented in terms of node depth. There are
`three types of nodes: the root node 80, container nodes rep
`resented by folder icons 85, and digital object leaf nodes
`represented by document icons 90. The depth of a node in a
`hierarchical tree data structure is determined by the number
`of levels a node is located from the root node. The root node
`80 is at depth-092 and the depth of the hierarchical tree
`structure is equal to the largest depth-D95 in the set. FIG. 3
`provides a concrete example of a tree representation of hier
`archical data presented in terms of node depth. In this
`example, the depth of the tree is 5 and the number of container
`nodes at depth=1 is three. The node n11 111 contains three
`documents 221,