as) United States
`a2) Patent Application Publication 10) Pub. No.: US 2009/0113350 Al
`
`(43) Pub. Date: Apr. 30, 2009
`Hibinoet al.
`
`US 20090113350A1
`
`(54) SYSTEM AND METHODFOR VISUALLY
`SUMMARIZING AND INTERACTIVELY
`BROWSING HIERARCHICALLY
`STRUCTURED DIGITAL OBJECTS
`
`Publication Classification
`
`(51)
`
`Int. Cl.
`GO6F 3/048
`
`(2006.01)
`
`(76)
`
`Inventors:
`
`Stacie Lynn Hibino, San Jose, CA
`(US); Jiajian Chen, Atlanta, GA
`(US)
`
`Correspondence Address:
`Frank Pincelli, Patent Legal Staff
`Patent Legal Staff
`Eastman Kodak Company, 343 State Street
`Rochester, NY 14650-2201 (US)
`
`(21) Appl. No.:
`
`11/924,973
`
`(22)
`
`Filed:
`
`Oct. 26, 2007
`
`(52) US. CMe coeccccccssssssssssssssssvseseesessessssasisssssenes 715/853
`
`(57)
`
`ABSTRACT
`
`A system and method of visually summarizing a hierarchi-
`cally structured set of digital objects and forfacilitating effi-
`cient access to such objects through the selection ofrepresen-
`tative summary objects. These digital objects are typically
`media objects such as digital imagefiles, digital video clips,
`digital audio objects, such as “MP3”files, or other digital
`documents, such as text documents, that can be collected by
`a user and distributed over a variety of storage media and
`storage locations.
`
`HEIGHT 5 St
`
`93
`
`ROOT NODE
`
`80
`
`
`
`HEIGHT 4
`
`HEIGHT 3
`
`HEIGHT 2
`
`
`
`
`
`HEIGHT1
`111 /|
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`HEIGHT0
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`Dy ons\ody| 24) 242 (7 24a 285 | a 49 coi one | asa | 3o (ong od
`
`
`
`
`
`
`221 229|241 247 248249 9251 252|254242 244 245 231 232 234 235 237 224 6225
`
`
`236
`223
`243
`246
`253-255
`233
`
`SAMSUNG 1026
`
`SAMSUNG 1026
`
`1
`
`

`

`Patent Application Publication
`
`Apr. 30,2009 Sheet 1 of 18
`
`US 2009/0113350 Al
`
`10
`
`DEVICE 14
`CPU (0)DVDR-/R+DRIVE
`
`
`
`USB
`
`INTERFACE
`
`32
`
`DIGITAL CAMERA
`6
`
`men
`
`DVD
`
`.
`
`40
`
`30
`
`42
`
`16
`
`CD-R/W
`DRIVE
`
`
`(o)CD-ROM ORCD-R/W
`
`
`READ-ONLY
`MEMORY
`
`
`
`440)rouse
`
`8
`
`NETWORK
`
`2
`
`DESKTOP
`INTERFACE
`
`12
`
`26() 48
`
`DEVICE ©
`
`22
`
`
`
`DISPLAY
`INTERFACE
`DEVICE
`
`RANDOM
`ACCESS
`MEMORY
`
`_——__-___.
`
`KEYBOARD
`
`46
`
`| MICROPHONE
`‘a
`SPEAKER
`
`24
`
`52Y|CT
`
`VIDEO DISPLAY
`
`FIG. 1
`
`2
`
`

`

`Patent Application Publication
`
`Apr. 30,2009 Sheet 2 of 18
`
`US 2009/0113350 Al
`
`DEPTHO ~
`
`80
`
`ROOT
`NODE
`
`DEPTH 1
` DEPTH2 - we |
`
`i
`I td ene a
`
`90
`
`nee eee
`
`Len
`
`DEPTH ...
`
`
`
`DEPTH D
`D=LARGEST DEPTH
`OF HIERARCHY
`
`\ os
`
`
`
`
`
`
`
`FIG. 2
`
`3
`
`

`

`Patent Application Publication €Old
`
`Lez982SEZverEeZezLeeVV_—_———
`
`
`(|CILILILILIL)Eth.beste:‘el€H1dad
`0083GONLOOYfo
`
`LFLI
`ZHidad
`
`
`
`
`
`
`yHidad
`
`Apr. 30, 2009 Sheet 3 of 18
`
`US 2009/0113350 Al
`
`GScpSeE902G¢1S
`
`$Hid3d
`
`
`
`4
`
`
`
`

`

`G22peeLEZ\s¢2veeJZEZLezS72|ZS2SZereereZrZt\
`RR)TRAHEA)DS
`
`/Sel/a/VY
`
`
`Beanecol€LHOISH
`CL]OUILICELILILAR
`9b¢Creee
`Stevroyelpe|dcolec
`vuEEN’?+LHOISH
`
`eg
`§LHDISH
`
`08
`
`AGONLOO
`
`PvLHOISH
`
`&LHOISH
`
`Patent Application Publication
`
`Apr. 30, 2009 Sheet 4 of 18
`
`US 2009/0113350 Al
`
`b‘Dla
`
`€S¢
`
`\ S
`
`OS
`
`opt
`
`SEIbol
`
`5
`
`
`
`
`
`

`

`Patent Application Publication
`
`Apr. 30, 2009 Sheet 5 of 18
`
`US 2009/0113350 Al
`
`l2
`
`.
`
`
`
`AARLYS3GONHANIVINOO
`
`S‘Dld
`
`OS‘ld
`
`
` 82¢9626cLez[[3%]|eee|EaEa|ona]aN|eee||eee.|Eazz
`gS‘9l4
`‘SZN“b2N‘©2N22N“LZN
`962ezLezbetbe
`
`TSAq1LVSSQONJV31ONYYANIVLNOOTIVMOHS
`
`AINOMOHS‘@=HLid3dLYAHOWVHSIHAOTAAR7
`‘C=H1d3dLYAHOYVHAIH4O1SA57
`
`
`
`NINOSHCONYANIVLNODATEISSOdLSAMO1MOHS
`‘VeN‘oN‘CoN“LON‘LLN
`eoeLe?BreLeeLee
`‘SS31YO2@=H1Ld3dLVAHOYVHY3IH40TSAR
`9/2vedLyeLed
`8Le_)
`
`(L=HLd3d)AHOUVHSIHSOTSAST-dOL
`‘LN-cLN“LEN
`
`022
`
`O22
`
`wa
`042
`
`O22J
`
`gS‘Sid
`
`VS‘Sls
`
`ole
`
`“SON
`
`9€¢
`
`6
`
`
`
`
`
`
`
`
`

`

`Patent Application Publication
`
`Apr. 30, 2009 Sheet 6 of 18
`
`US 2009/0113350 Al
`
`
`
`AINOMOHS‘2=LHDISHLYAHOYVHYSIH3OTSAR
`
`‘$S31YO2=LHOISHLVAHOUVHAIHSO13A379‘DidV9‘Sis
`
`cle~PSSLSeBed‘ECLN‘SEN“LON
`odeIvelez
`
`2ez‘e4u
`0L¢
`
`XINOSSQONYANIVLNODS1dIlSSOdLSSHOIHMOHS
`
`
`
`
`OL¢
`
`O22
`
`9°Dl4d99‘Hld
`
`
`
`
`
`
`
`
`
`TAALYSACONYSNIVINOO(L=LHDISH)AHOUVHAIH3OSHSNIVLNOOD13A37LSAMO1
`
`seguffftiu
`
`7
`
`
`
`

`

`Patent Application Publication
`
`Apr. 30,2009 Sheet 7 of 18
`
`US 2009/0113350 Al
`
`300
`
`USER-SELECTED
`DEPTH =D
`
`
`
`
`
`
`
`
`
`GET LOWEST
`ADD ALL CONTAINER|yes
`CONTAINER
`
`NODES AT DEPTH=1
`TO LISTTOPROCESS
`NODES UP TO ?
`
`
`HIERARCHICAL DATA
` 302
`305
`
`GET
`NO
`
`CONTAINER
`NODES ONLY?
`
`
`
`
` ADD ALL
`
`GET FIRST NODE
`
`CONTAINER
`OF
`
`
`
`LISTTOPROCESS
`ADD ALL NODES
`NODES AT
`
`
`
`
`AT DEPTH=D
`DEPTH=D
`
`
`
` TO NODESLIST
`TO NODESLIST
`
`
`
`
`
`
`(NODE HAS
`
`CONTAINER
`
`
`NODES) AND
`(NODE DEPTH
`
`<D)?
`
`
`
`ADD NODE TO
`NODESLIST
`
`ADD CHILDREN
`CONTAINER
`NODES TO
`LISTTOPROCESS
`
` RETURN
`
`
`NODESLIST
`
`
`GET NEXT NODE
`FROM
`
`
`LISTTOPROCESS
`
`8
`
`

`

`Patent Application Publication
`
`Apr. 30,2009 Sheet 8 of 18
`
`US 2009/0113350 Al
`
`300
`
`HIERARCHICAL DATA
`
`345
`
`
`GET HIGHEST
`
`
`
`CONTAINER
`USER-SELECTED
`HEIGHT H
`NODES UP TO
`
`
`H?
`
`
`
`
`ADD ALL
`CONTAINER
`
`NODES AT
`HEIGHT=1 TO
`
`LISTTOPROCESS
`
`
`
`GET FIRST NODE
`
`
`ADD ALL
`OF
`
`LISTTOPROCESS
`CONTAINER
`
`NODES AT
`
`
`HEIGHT=H
`TO NODESLIST
`
`
`
`
`352 ~
`
`(PARENT NODE P
`<> ROOT NODE)
`
`AND (HEIGHT(P)
`
`<=)?
`
`
`
`NODESLIST
`
`ADD NODE TO
`
`IF P NOT IN
`
`
`
`LISTTOPROCE
`
`SS, ADD P TO
`LISTTOPROCE
`
`
`SS
`
`
`
`
`MORE NODESIN
`
`RETURN
`
`
`LISTTOPROCESS
`
`
`NODESLIST
`?
`
`
`GET NEXT NODE
`358
`
`FROM
`
`LISTTOPROCESS
`FIG. 8
`
`9
`
`

`

`Patent Application Publication
`
`Apr. 30,2009 Sheet 9 of 18
`
`US 2009/0113350 Al
`
`NODESLIST: LIST OF
`NODES TO DISPLAY
`
`,
`GET FIRST NODE
`
`382
`
`
`
`GET NEXT NODE
`
`
`
`GET SUMMARY
`
`
`
`VISUAL OBJECT FOR
`CURRENT NODE
`
`
`
`
`
`INSERT SUMMARY
`
`VISUAL OBJECT INTO
`
`LISTTODISPLAY,
`BASED ON
`SEQUENCE
`FUNCTION
`
`
`
`
`MORE
`NODES?
`
`375
`
`380
`
`385
`
`390
`
`395
`
`NO
`
`DISPLAY
`LISTTODISPLAY
`
`398
`
`FIG. 9
`
`10
`
`10
`
`

`

`Patent Application Publication
`
`Apr. 30,2009 Sheet 10 of 18
`
`US 2009/0113350 Al
`
`USER-SELECTED
`
`LISTTODISPLAY2
`
`NODE FROM
`
`400
`
`410
`
`NODE=
`CONTAINER
`NODE?
`
`ne
`
`GET LIST OF CHILD
`ELEMENTS OF NODE
`
`406
`
`GET FIRST CHILD
`ELEMENT
`
`
`
`
`
`
`YES a 404
`
`
`
`408 an
`
`
`CHILD
`
`
`
`ELEMENT=
`GET NEXT CHILD
`NODE WITH
`
`
`
`ELEMENT
`CHILDREN?
`
`
`LISTTODISPLAY2
`
`
`
`
`INSERT CHILD
`ELEMENT INTO
`LISTTODISPLAY2,
`BASED ON
`
`SEQUENCE
`FUNCTION
`
`414
`
`INSERT ELEMENT
`NODE INTO
`
`
`
`MORE CHILD
`ELEMENTS?
`
`NO
`
`418
`
`RETURN
`LISTTODISPLAY2
`
`FIG. 10
`
`11
`
`11
`
`

`

`Patent Application Publication
`
`Apr. 30, 2009 Sheet 11 of 18
`
`US 2009/0113350 Al
`
`OS8/2
`
`SoevSe
`
`Sp
`
`ttcLee=]=)=)]|)EE)
`
`LL‘Sld
`
`
`
`Git‘SlsoregS‘Old
`
`
`
`dS‘id402SNSGONSONOILOAIASHASNAOSLINSSY7SA31LYSSCONYANIVLNOOAINOMOHS
`
`
`
`
`
`
`
` OSPr VS‘DiddOLENAGONSONOILOATASHASNAOSLINSSY(L=HLd30)AHOUVYSIHJOTAAST-dOL
`
`
`
`
`
`
`
`aZaVil‘OldoteVS‘Old
`
`‘C=HidddLYAHOHVHSIHAOTSAR1
`
`12
`
`12
`
`
`

`

`Patent Application Publication
`
`Apr. 30, 2009 Sheet 12 of 18
`
`US 2009/0113350 Al
`
`OS?
`
`B22
`
`LES9ES
`
`LSP
`
`
`
`gzi‘DId
`
`Zi‘DI
`
`
` 9‘Old40€LNAGONSONOILOAIASYASNAOSLINSAY VelSid
`ee]Fe)]Fe]
` JAASLYSAGON
`oeaS‘Old
`
`YANIVLNOOAINOMOHS
`‘C=LHDIAHLVAHOYVHSIH4013A31
`
`OSP
`
`
`
`82¢e122
`
`LSP
`
`dS“‘DIl4dJO122SGONJONOLLOSISSHSNJOSLINSAY
`TAAS1LVSSQONAV]ONYHANIVLNOODTIVMOHS
`
`‘2=H1LdadLVAHOYVYSIH4013A41
`
`13
`
`13
`
`
`
`

`

`Patent Application Publication
`
`Apr.30, 2009 Sheet 13 of 18
`
`US 2009/0113350 Al
`
`ElDid
`
`cq Noe ee aeee ee ee wwe eee ee ro ee rere
`
`meal,Bulostuiuay!4epoy'
`
`
`[8Ae77-dolL©POSG——S]USAq|BA8"]-LUOWOgOS]UGAq
`
`
` COG——MBIA[ENGSA10}SIH©MBIAPLUS)©yoke]MIAC
`
`
`(Y)MalAJ9SeyI(W)IlyPUedXy
`
`
`punOTRSSE[Oy][Ssur]
`
`
`
`(-)g",3NO|(=)z,:ul]:wo007z
`
`
`apisinosebew!jus.ayYoO
`JejuaosebeuuijUsd98y©aul}AqWOS
`
`
`
`TEPUPbbeypapereetrs
`sandAN©Yoleespubloyld
`
`MAIABulosiulway
`BJOWIIAAES
`suodoMIAO
`
`
`
` aleyspueainjdea[AL__]:dnoigjeis0g
`
`MalAW607
`
`-BPIH/MOYUS
`
`14
`
`14
`
`

`

`MIIA@ MAlA
`
`MAABurosiulweay
`ZOS——MaIAfesldgMOISIH©MAAPUD©
`
`rw
`
`sellDee
`
`ajOWNAESO
`
`sainjoldAN©
`
`
`
`
`
`PUnGTBWSER][obo]SOU]
`
`aJeUSpuesunjden
`
`“@PIH/MOUS
`
`ri
`
`ts)
`
`rs}
`
`i
`
`
`
`
`
`(y)MaIAJeSeyll(y)IIlyPuedxy
`
`
`
`PEPEREDGStibetatied
`
`
`
`
`
`[A.___]:dnoigjeioog
`
`
`
`|@Ae77-dOL©bOG-——S}USAZ[9A97-WO}}0gO$]U8Aq
`
`
`
`15
`
`
`
`
`
`(-)G°sINOW|(=)Z.“Ul]:wo00Z
`
`
`
`MaiLUOOZ
`
`4Epoy
`
`
`
`BUIOsSIUIWUeyYcy
`
`jNoAe7]
`
`
`
`
`
`
`
`19}Ua0SHE!JUa00y@aul}AqPOS
`
`
`
`suondoMaIA©;
`
`
`
`
`
`apis}nosabeuuljuaIeYoO
`
`Patent Application Publication
`
`Apr. 30, 2009 Sheet 14 of 18
`
`US 2009/0113350 Al
`
`0ZC
`
`G b
`
`l‘Old
`
`15
`
`

`

`
`
`ico|e]:",MAIABUIDSIUILUDYca
`O22GL"K)ld
`
`
`
`
`
`
`
`
`
`
`16
`
`Patent Application Publication
`
`Apr. 30, 2009 Sheet 15 of 18
`
`US 2009/0113350 Al
`
`?
`
` ZOG——MBIA[BISALO]SIH©MalAPUDONOAeTMAA
`
`
`
`
`
`J9]U89Sofew!}UBIeN©aul}AqHOS
`
`SuodOMAA
`
`
`
`(admeraoseul|(v)ilypuedka
`
`MaiAWId07
`
`
`
`
`
`@pisinosebeulljueayYo
`
`peHORSJoneranonog°SUSRy
`
`aasnoSa
`
`16
`
`
`
`

`

` 9t‘Sls
`
`
`
`
`
`MAIA® MBIAJIGSAJO}SIH©MelABurosiuiuayMBIAPLDOyepoyiNOAe]]
`
`
`
`MaiBulosiuiwaeyco
`
`Patent Application Publication
`
`Apr.30, 2009 Sheet 16 of 18
`
`US 2009/0113350 Al
`
`G00Z|P00?|£O0~
`
`
`
`““YIIBISPJOMASY
`
`YoleaspueJ0}I1-4©SUOIdO
`MIIA®
`
`Jes
`
`17
`
`17
`
`
`

`

`Patent Application Publication
`
`Apr. 30, 2009 Sheet 17 of 18
`
`US 2009/0113350 Al
`
`0L2
`
`©eeoa
`
`pos/ 0€S
`
`MBIAWOO?
`
`OPIH/MOUS
`
`g
`
`18
`
`ZI
`Ys
`
`4epoy
`
`609
`
`Iv]
`
`MalABulosiuiwey
`SRSWSNNUSNSS_—a
`
`
`[amarreseal(W)ilvpuedka
`
`
`
`
` [punovByeeg]|O603]|Seu](-)G",‘NOI(=)Z~:Ul]:wo009Z
`
`
` SIeYSpuesinjdey[aT___|:dnoigjeioos
`
`
`apisinosebew!juaoayo
`““YyoIeagpiomAay
`
`J9jU99sabe!jUa98Y@au}AqWosS
`
`
`
`S}UGAF}AA87T-LUOHOGO
`
`$]U6AQ|eAe7-dOL©
`MOABupsiuWeyYco
`a}OwiAAeSO]
`SUOIAOMIAO
`
`}NOALTMIIA®
`
`on|900¢|S00z|vO0Z
`NoQ°o
`
`18
`
`
`

`

`Apr.30, 2009 Sheet 18 of 18
`
`Patent Application Publication
`
`1gTERYEAL19cy|Hidad
`
`:0Hidad08CONLOO25J
`
` LL9
`JUQUOGWOUGU.OUGOHgoOUOUOUR
`
`ERSALYeeuidaa
`
`19
`
`
`
`
`
`
`
`
`
`
`
`vezSezZ&?gee2%zzzjezgecVESezgezysezezlezES7°S%1gzSheayeLhgyeGpzPPSoyecheye
`
`US 2009/0113350 Al
`
`St‘Old
`
`S$Hidad
`
`19
`
`

`

`US 2009/0113350 Al
`
`Apr. 30, 2009
`
`SYSTEM AND METHOD FOR VISUALLY
`SUMMARIZING AND INTERACTIVELY
`BROWSING HIERARCHICALLY
`STRUCTURED DIGITAL OBJECTS
`
`FIELD OF THE INVENTION
`
`[0001] The present invention relates to providing a visual
`overview pertaining to hierarchically organized digital
`objects. In particular, the present invention pertains to the
`formation and display of ordered visual representations,
`where, each displayed visual representation symbolizes a
`different node, wherein all symbolized nodes belong to one
`level in the object hierarchy. The visual representation can be
`obtained from nodes descendant from the symbolized node,
`or generated separately.
`
`BACKGROUND OF THE INVENTION
`
`taining to a collection of digital objects is provided by a
`computer implemented method includingthe steps of locat-
`ing acollection ofdigital objects that are stored on a computer
`system and whichare organizedin a hierarchical fashion, and
`then simultaneously displaying visual representations of
`digital objects that belongto a selected level of the hierarchy.
`This is a useful method because computer users typically
`group and label text documents and image files in such a
`hierarchical fashion, wherein nested folders, subfolders, and
`files are appropriately labeled. Digital objects of the present
`invention mayalso include video clips and presentation docu-
`ments. The visual representations may include thumbnails
`versions of digital objects, a manually created or computer
`generated representation, an icon, and a montage.
`invention
`[0005]
`Further embodiments of the present
`include recognizing depth and height levels ofa hierarchy and
`their interrelationships, and selecting such levels for generat-
`ing visual representations to depict the levels. Other embodi-
`ments of the present invention include recognition of leaf
`nodes and container nodesofthe hierarchy, and their ancestry
`in the hierarchy. Another embodimentof the present inven-
`tion is selection of a pattern to display the visual representa-
`tions, such as in a spiral pattern. The present invention can be
`used to visually summarizehierarchies based on time, events,
`people, image and document contents, and geographic loca-
`tion.
`
`Increased computer usage and creation of digital
`[0002]
`objects along with decreasing costs in removable memory,
`fixed hard disks, and network access haveall contributed to
`the increase in the numberofdigital objects any individual
`person may own or want to access. Computer users have
`moved from tens or hundredsoffiles to thousandsoffiles or
`more. Similarly, digital cameras have decreased in physical
`carrying size and increased in memory capacity, making it
`easy for users to carry cameras with them more often and
`[0006] Other embodimentsofthe present invention include
`capture more pictures and video clips per picture-taking
`a computer system suitably programmed to implement the
`opportunity. As personal digital media collections increase in
`methodsofthis invention. Such a computer system includes a
`magnitude, they are also becoming more and more cumber-
`processor and memory for organizing and storing a plurality
`somefor users to manage andaccess.
`of digital objects in tree hierarchy. The processor executes
`[0003] Typically, as inafile system, users can easily view
`programsfor visually summarizing certain onesof the digital
`digital objects by browsing a hierarchical directory andfile
`objects that belong to a selected level of the hierarchy. A
`structure. Tree based representations are typically provided
`display screen is coupled to the computer system for display-
`for users in a graphical user interface (GUI), making it simple
`ing visual representations comprising the visual summary. In
`for users to view any objects in any given directory. Unfortu-
`response to a userselecting a depth ofthe hierarchyto display,
`nately, it is challenging for users to access object leaf nodes
`the system can then display a visual summary ofthe corre-
`from more than one directory at once. In addition, users are
`sponding level ofthe hierarchy, and so on for additional lower
`not provided with any visual summary information about
`depths. This is a top-down embodimentofthe present inven-
`whatis contained within a directory, especially not for objects
`tion (e.g. hierarchy depth). The system can also start with a
`that are deeply nested below any given directory. Further-
`visual summary of a lowestlevel of the hierarchy and succes-
`more,a file system view does notprovide flexibility for view-
`sively display; higher levels, which is referred to as a bottom-
`ing other hierarchical organizations based on other data facets
`up embodiment of the present
`invention (e.g. hierarchy
`for the sameset of digital objects. That is, if a user creates a
`height).
`folder andfile structure based on temporal information such
`[0007] Other embodiments that are contemplated by the
`as year, then month, etc., then the user is limited to browsing
`present invention include computer readable media and pro-
`only by that temporal information within the file browser.
`gram storage devices tangibly embodying or carrying a pro-
`They cannot, for example, use the file browser to browse the
`gram ofinstructions readable by machine or a processor, for
`samefiles based on a hierarchical content-based categoriza-
`having the machine or computer processor execute instruc-
`tion of the same data—notwithoutfirst needing to duplicate
`tions or data structures stored thereon. Such computer read-
`files and create appropriate folders. Some modern digital
`able media can be any available media, which can be accessed
`object organization software provide support for viewing dif-
`by a general purpose or special purpose computer. Such com-
`ferent data facets (e.g., for browsing by tags or date in a photo
`puter-readable media can comprise physical computer-read-
`collection), but they do not provide a general approach for
`able media such as RAM, ROM, EEPROM, CD-ROM, DVD,
`browsing any facet of hierarchical data. Accordingly, a need
`or other optical disk storage, magnetic disk storage or other
`exists in the art for an improved way to review and organize
`magnetic storage devices, for example. Any other media
`digital objects.
`which can be usedto carry or store software programs which
`can be accessed by a general purpose or special purpose
`computer are considered within the scope of the present
`invention.
`
`SUMMARY
`
`[0004] The above-described problemsare addressed and a
`technical solution is achieved in the art by a system and a
`method for providing a visual summary pertainingto a col-
`lection of digital assets, according to the present invention. In
`one embodimentof the present invention, information per-
`
`[0008] These, and other aspects and objects of the present
`invention will be better appreciated and understood when
`considered in conjunction with the following description and
`the accompanying drawings. It should be understood, how-
`
`20
`
`20
`
`

`

`US 2009/0113350 Al
`
`Apr. 30, 2009
`
`ever, that the following description, while indicating pre-
`ferred embodiments of the present invention and numerous
`specific details thereof, is given by wayofillustration and not
`of limitation. Many changes and modifications may be made
`within the scope of the present invention without departing
`from the spirit thereof, and the invention includes all such
`modifications. The figures below are not intendedto be drawn
`to any precise scale with respect to size, angular relationship,
`or relative position.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`[0009] The present invention will be more readily under-
`stood from the detailed description of exemplary embodi-
`ments presented below considered in conjunction with the
`attached drawings, of which:
`[0010]
`FIG. 1 is a block diagram of a computer system
`capable of practicing various embodiments of the present
`invention;
`[0011]
`FIG. 2 is an example abstract tree representation of
`hierarchicaldata;
`[0012]
`FIG. 3 is an example tree representation of hierar-
`chical data, presenting nodes by tree depth;
`[0013]
`FIG. 4 is an example tree representation of hierar-
`chical data, presenting nodesbytree height;
`[0014] FIG.5 shows examples of nodes anddata from FIG.
`3 to display, based on top-down browsingofthe hierarchy;
`[0015]
`FIG. 6 shows examples of nodes and data from FIG.
`4 to display, based on bottom-up browsing ofthe hierarchy;
`[0016]
`FIG. 7 isa flow chart for generating the list of nodes
`to display for top-down browsing ofthe hierarchy;
`[0017]
`FIG. 8 isa flow chart for generating the list of nodes
`to display for bottom-up browsing ofthe hierarchy;
`[0018]
`FIG. 9 is a flow chart for generating a first ordered
`list of visual objects to display, based on list of nodes to
`display generated by FIG.7 or FIG.8;
`[0019]
`FIG. 10 is a flow chart for generating a second
`orderedlist of visual objects to display;
`[0020]
`FIG. 11 shows examplesof leaf nodes from FIG. 3
`to display, based on selection of different nodes depicted in
`FIG.5;
`FIG. 12 shows examplesof leaf nodes from FIG. 3
`[0021]
`to display, based on selection of a node depicted in FIG. 5 or
`a node depicted in FIG.6;
`[0022]
`FIG. 13 showsa screen capture of a working proto-
`type implementing an embodimentof the present invention,
`displaying thumbnail imagesofa first list in a grid layout;
`[0023]
`FIG. 14 showsa screen capture ofthe same working
`prototype, displaying thumbnail images ofa first list in a
`spiral layout;
`[0024]
`FIG. 15 showsa screen capture of the same proto-
`type, after a thumbnail from FIG. 14 has beenselected;
`[0025]
`FIG. 16 showsa screen capture of the same proto-
`type, with an interactive barplotfilter forfiltering the current
`data being displayed;
`[0026]
`FIG. 17 showsthe results of expandingall thumb-
`nail images of FIG. 16; and
`[0027]
`FIG. 18 is an example tree representation of hierar-
`chical data, presenting nodesby tree depth.
`
`It should be understoodthat the attached figures are
`[0028]
`for purposesof illustrating the concepts of the invention and
`may not be drawnto scale.
`
`DETAILED DESCRIPTION
`
`[0029] Embodiments of the present invention relate to an
`effective method of representing a hierarchically structured
`collection of digital objects, and for facilitating efficient
`access to such objects through the selection of individual
`thumbnails or nodes in the hierarchy. These digital objects are
`typically media objects such as digital image files, digital
`video clips, digital audio objects, such as “MP3”files, or
`other digital documents that can be collected by a user and
`distributed over a variety of storage media and storage loca-
`tions. However, these objects need not be limited to multime-
`dia objects. Text files can also be represented accordingto the
`methods of the present invention.
`[0030] When users only have a small numberof digital
`objects, they can easily organize them hierarchically using a
`standardfile and folder representation as typically provided
`by a modern computer operating system. For example, con-
`sider a folder hierarchy for pictures viewed in the File
`Explorer application of Microsoft Windows XP. Using such a
`standardfile and folder hierarchy, users can access items in a
`straightforward manner by clicking on individual folders.
`They can opt to view thumbnails of images or view them ina
`“filmstrip” format, enabling them to preview images without
`having to open them. Unfortunately, when users’ collections
`grow to thousandsofobjects, the simple file and folder access
`available via the operating system can become cumbersome.
`In order to access media deep in a file system structure, users
`have to expand each subfolder along the way. They also
`cannot view the contents of two subfolders together unless
`they open twoseparate File Explorer windowsand navigate to
`the appropriate place. Although modern file system views can
`provide a visual summary of media files such as showing
`thumbnails ofunderlying content on folder icons, such visual
`summaries are only provided for one level of depth in a
`hierarchy—.e., for only the immediate “child” media of a
`given folder node. Two major limitations of using a current
`standard file and folder hierarchy organization for accessing
`hierarchically organized media today are thus: 1) visual rep-
`resentationsare not available for deeply nested media; and 2)
`users are limited to accessing only immediate child media at
`any given node in the hierarchy (i.e., they do not have easy
`access to all descendant media from any nodeat any level in
`the hierarchy).
`[0031]
`In accordance with an embodimentofthe present
`invention, a more compactandefficient method of browsing
`a hierarchically organized set of documents suchas a digital
`image collection stored in folders andfiles is to provide a
`visual summary of nodesof the set, based on user-selected
`level of the hierarchy to view. The level of hierarchy can be
`based on node depth, as described in more detail below in
`relation to FIG.3, or node height, as explained in more detail
`below in relation to FIG. 4. This provides the user with a
`visual summary ofa horizontalslice through,ora levelof, the
`hierarchy. Furthermore, in accordance with another embodi-
`mentof the present invention, given a visual summary list of
`nodesas described above, users can easily access all descen-
`dant leaf nodes of a selected summary node, independentof
`the height of the leaves.
`[0032]
`For example, consider the typical case where users
`organize their photo collection on their home computerin file
`
`21
`
`21
`
`

`

`US 2009/0113350 Al
`
`Apr. 30, 2009
`
`folders based on date and event. At the top level of their photo
`collection, they have file folders for each year. Within each
`year folder, they have folders organized by month and event
`information (e.g., events such as a birthday party, vacation,
`wedding, day hike, holiday, etc.). Within each of those fold-
`ers, they may have additional subfolders, usually depending
`on the number of pictures captured for a given event. For
`example, pictures from a vacation to Europe might be in a
`folder called “May 2007 trip to Europe” and then further
`divided into country visited folders and country folders could
`be further divided into majorcities, activities or famous land-
`marks. In accordance with an embodiment of the present
`invention, the user could choose to view the top level folder
`nodes which, in this example, would presentthe user with one
`representative visual summary object per year. Such a visual
`summary object could be a single picture from a particular
`year such asthefirst picture of the year or a picture from a
`subfolder containing the most pictures. The visual summary
`object could also be a visual summary thatis created by the
`system orthe user, such as a montage(e.g., a montage com-
`posedof pictures from different subfolders). Once presented
`with the visual summary ofyears, users can select a year node
`to view all images captured during that year, independent of
`how deeply nested those images might be within the year
`folder.
`
`In the above example, year folders represent nodes
`[0033]
`at depth=1 in the hierarchy. Moving down onelevel in the
`hierarchy (i.e. in all year folders), folders within each year
`folder (month folders and event information folders) are at
`depth=2 and can represent user-specified events that took
`place during that year. Thus,
`in one embodiment of the
`present invention, users can choose to view folder nodes at
`depth=2 to get a visual overview of all major events ofall
`years at once.
`[0034] The present invention is not limited to browsing the
`hierarchy from the top-downby depth level, but also provides
`browsing the hierarchy from the bottom up by node height
`level. As a brief example, given a sample hierarchyasillus-
`trated in FIG. 3, where nodes are presented by depth level,
`FIG.4 shows how the samehierarchy can be depicted in terms
`of node height. Folder n12 112 shownin FIG.3 and in FIG. 4
`represents a node having a depth level=1 and a height level=4
`and which contains several depths of folders, similar to that
`described above for a vacation to Europe. Consider again the
`case of the example aboveof a user’s collection organized by
`year folders and then by event folders and so on. As men-
`tioned above, events may vary in size and subfolder depth,
`typically dependent on the numberofpictures captured for an
`event. The bottommost subfolders of such a collection repre-
`sent an atomic-level of events for users. In accordance with an
`
`embodimentofthe present invention, bottom-up browsing of
`events provides users with easy access to such atomic-level
`events, independentof the depth of the corresponding nodes.
`Thus, this would allow a user to see the “Eiffel Tower” node
`of pictures right next to the “Roman Coliseum” node of
`pictures, even ifthey wereat different depths in the hierarchy,
`as long as they were at the same heightin the hierarchy.
`[0035]
`FIG. 1 showsone type of system for practicing an
`embodimentof the present invention. In this example, the
`system includes a computer 10, which typically includes a
`keyboard 46 and mouse44 as input devices communicatively
`connected to the computer’s desktop interface device 28. The
`term “computer” is intended to include any data processing
`device, such as a desktop computer, a laptop computer, a
`
`mainframe computer, a personal digital assistant, a cell
`phone, a digital camera, a smart phone device, and/or any
`other device for processing data, and/or managing data, and/
`or handling data, whether implemented with electrical and/or
`magnetic and/or optical and/or biological components, and/
`or otherwise. Any of the above-mentioned computers, need
`not include all the components illustrated in FIG. 1. The
`system illustrated in FIG. 1 is intended to cover many
`embodiments of a computer system. Therefore, taken alto-
`gether, the system illustrated is more than sufficient for imple-
`menting embodiments of the present invention. The phrase
`“communicatively connected”is intended to include any type
`of connection, whether wired, wireless, or both, between
`devices, and/or computers, and/or programs in which data
`may be communicated. Further, the phrase “communica-
`tively connected”is intended to include a connection between
`devices and/or programs within a single computer, a connec-
`tion between devices and/or programs located in different
`computers, and a connection between or within devices not
`located in computers.
`[0036] Output from the computer 10 is typically presented
`on a video display 52, which may be communicatively con-
`nected to the computer 10 via the display interface device 24.
`The display mayalso be disposed in the same housing as the
`computer 10. Internally, the computer 10 contains compo-
`nents such as a CPU 14 communicatively connected to bus 12
`and computer-accessible memories,
`such as
`read-only
`memory 16, random access memory 22, and a hard disk drive
`20, all communicatively connected to the bus, and which may
`retain someorall of the digital objects referred to herein. The
`phrase “computer-accessible memory”is intended to include
`any computer-accessible data storage device, whether vola-
`tile or nonvolatile, electronic, magnetic, optical, or otherwise,
`including butnotlimited to, floppy disks, hard disks, compact
`discs, DVDs, flash memories, ROMs, and RAMs.
`[0037] The CPU 14 communicates with other devices over
`data bus 12. The CPU 14 executes software that can be stored
`
`onthe computer accessible memories. In addition to hard disk
`drive 20, the computer 10 mayalso contain other connectible
`computer-accessible memory drives for reading and writing
`data and software of various types. This may include a CD-
`RWdrive 30 for reading and writing various CD media 42 as
`well as a DVD drive 32 for reading and writing to various
`DVD media 40. Audio can be input to the computer 10
`through a microphone 48, or other storage device, for
`example, an MP3 player, communicatively connected to an
`audio interface device 26. Audio playback can be heard via a
`speaker 50, or other listening devices, also communicatively
`connected to an audio interface device 26. A digital camera 6,
`and most any of the devices discussed herein, or other image
`capture or storage device can be communicatively connected
`to the computer 10 through, for example, the USB interface
`device 34 to transfer digital objects from the camera 6 to the
`computer accessible memories and vice-versa. Finally, the
`computer 10 can be communicatively connected to an exter-
`nal network 60 via a network connection device 18, thus
`allowing the computer to access digital objects from other
`computers, devices, or data-storage systems communica-
`tively connected to the network.A “data-storage system” may
`include one or more computer-accessible memories, and may
`bea distributed data-storage system including multiple com-
`puter-accessible memories communicatively connected via a
`plurality of computers and/or devices, or over a network. On
`the other hand,a data storage system need not be a distributed
`
`22
`
`22
`
`

`

`US 2009/0113350 Al
`
`Apr. 30, 2009
`
`data-storage system and, consequently, may include one or
`more computer-accessible memories located within a single
`computer or device.
`[0038] Avcollection of digital objects can reside exclusively
`on the hard disk drive 20, compact disc 42, DVD 40, or on
`remote data storage devices, such as a networked hard drive
`accessible via the network 60. A collection of digital objects
`can also be distributed across any or all of these storage
`locations.
`
`[0039] A collection of digital objects may be represented
`by a database that uniquely identifies individual digital
`objects (e.g., such as a digital image file) and their corre-
`sponding location(s) in the computer memories. It will be
`understoodthat these digital objects can be media objects or
`non-media objects. Media objects can be digitalstill images,
`such as those captured by digital cameras, audio data such as
`digital music or voice annotations, digital video clips with or
`without sound. Media objects could also includefiles pro-
`duced by graphic or animation software such as those pro-
`duced by Adobe Photoshop or Adobe Flash. Non-media
`objects can be text documents such as those produced by
`word processing software or other office-related documents
`such as spreadsheets or email. A database of digital objects
`can be comprised of only one type of object or any combina-
`tion.
`
`FIG. 5 shows examples ofnodes and data from FIG.
`[0042]
`3 presented on a display 270, based on top-down (depth)
`browsing ofthe hierarchy, whereas FIG. 6 shows examples of
`nodes and data from FIG.4 presented on display 270, based
`on bottom-up (height) browsing of the hierarchy. Keep in
`mindthat the hierarchical tree data illustrated in FIG. 3 and
`FIG. 4 contain the same data set. In both FIG. 5 and FIG.6,
`nodesare abstractly displayed in a grid layout in display area
`270. Container nodes are indicated by a double border 272,
`while digital objects are indicated by a single border 274.
`Container nodes have two values—the top value 276 repre-
`sents the node id ofa node from FIG.3 and FIG.4; the bottom
`value 278, indicates the id of a leaf digital object descendant
`from that nodethat could be used as a visual summary ofthat
`node. For example, ifFIG. 3 represented a user’s digital photo
`collection as saved on a home computer, then in accordance
`with an embodimentof the present i