as) United States
`a2) Patent Application Publication 0) Pub. No.: US 2002/0033848 Al
`
` Sciammarella et al. (43) Pub. Date: Mar. 21, 2002
`
`
`20020033848A
`
`(54) SYSTEM FOR MANAGING DATA OBJECTS
`
`(30)
`
`Foreign Application Priority Data
`
`(76)
`
`Inventors: Eduardo Agusto Sciammarella, Tokyo
`(JP); Junichi Rekimoto, Tokyo (JP);
`Haruo Oba, Kanagawa (JP); Makoto
`Imamura, Tokyo (JP); Hideyuki
`Agata, Tokyo (JP); Masakazu
`Hayashi, Chiba (JP); Masato
`Kuninori, Tokyo (JP)
`Correspondence Address:
`William S. FrommerEsq.
`FROMMER LAWRENCE & HAUG LLP
`745 Fifth Avenue
`New York, NY 10151 (US)
`
`(21) Appl. No.:
`
`09/838,484
`
`(22)
`
`Filed:
`
`Apr. 19, 2001
`
`Apr. 21, 2000
`
`(SP) cecscssssssssseesssscsssssneeeseesees 2000-121298
`
`Publication Classification
`
`Tint. C7 eeeeeeccceccceeeeeeseeenneeeeceesnnesecenneeeee GO06F 3/00
`(SV)
`(52) US. C1.
`ieee ceccceccccsesessseessseeessseeessseeeseeseseeseaes 345/838
`
`ABSTRACT
`67)
`A device for managing image and audio files displays
`thumbnails, which are representative of the files, on a screen
`in different arrangements, depending on the selected layout
`and order. Whenthe user enters input to indicates that he or
`she wishes to browse through the thumbnails, the thumb-
`nails move across the screen in sequence, following a path
`through a focus outline in the center of the screen. The path
`followed by the thumbnails depends on the selected layout.
`
`21
`
`20
`
`time 28
`
`I[MAGE60. JPG. jp
`
`IMAGET59. JP'
`
`IMAGE160. JPG
`
`IMAGE161. JPG
`
`IMAGE1 63. JPG
`
`type
`
`size
`
`27a
`
`27b 2%c
`
`27d
`
`28a" 28b
`
`28c 28d
`
`27
`
`28
`
`22
`
`SPACETIME3D
`EXHIBIT 2009 - PAGE 1
`
`SPACETIME3D
`EXHIBIT 2009 - PAGE 1
`
`

`

`Patent Application Publication Mar. 21,2002 Sheet 1 of 27
`
`US 2002/0033848 Al
`
`FIG.
`
`1
`
`21
`
`20
`
`[sie7AGESO. JPG. jp 31
`
`b
`
`
`<{ j
`ATNSZ.
`f
`pS
`YG}
`VS
`AA
`— of50. PS.jpg [0]
`is
`2000/01/27
`
`time
`
`c\
`
`__IMAGEGO. JPG. jp
`
`|MAGE159. JP
`
`IMAGE1 60. JPG
`
`IMAGE 161. JPG
`
`IMAGE 163. JPG
`
`type
`
`size
`
`28e
`2
`
`
`
`
`27a 27b 2%c©«27d 28a 28b 28 28d
`
`
`
`27
`
`28
`
`22
`
`SPACETIME3D
`EXHIBIT 2009 - PAGE 2
`
`SPACETIME3D
`EXHIBIT 2009 - PAGE 2
`
`

`

`Patent Application Publication Mar. 21,2002 Sheet 2 of 27
`
`US 2002/0033848 Al
`
`FIG. 2
`
`IMAGESO. JPG. ipa [O|
`167k
`2000/01/27
`
`IMAGE
`IMAGE30. JPG. jpg
`IMAGE20.
`JPG. jpg
`
`|
`
`G. jpg
`
`IMAGE159. JPG
`|MAGE160. JPG
`
`31
`
`IMAGE16
`
`IMA\
`
`Vip— fh name ay type
`
`size
`
`time
`
`22
`
`SPACETIME3D
`EXHIBIT 2009 - PAGE 3
`
`SPACETIME3D
`EXHIBIT 2009 - PAGE 3
`
`

`

`Patent Application Publication Mar. 21, 2002 Sheet 3 of 27
`
`US 2002/0033848 Al
`
`
`
`peN=IXSFESs
`
`
`
`
`31
`VAN
`
`21
`
`20
`
`IMAGEIQ. JPG. jog
`
` MUSICO1. MSA
`
`MUS1C02. MSA
`
`MUS ICQ3. MSA
`
`MUSICO4. MSA 93
`
` IMAGE20. JPG. jpg
`
`IMAGE30. JPG. jpg
`
` IMAGE] 67. JPG
`
`
`Ch >— BH name Sty type|size time
`io
`
`
`IMAGE168. JPG
`V
`
`ort
`
`22
`
`SPACETIME3D
`EXHIBIT 2009 - PAGE 4
`
`SPACETIME3D
`EXHIBIT 2009 - PAGE 4
`
`

`

`Patent Application Publication Mar. 21,2002 Sheet 4 of 27
`
`US 2002/0033848 Al
`
`FIG. 4
`
`time
`
`size
`
`23b
`
`22
`
`SPACETIME3D
`EXHIBIT 2009 - PAGE 5
`
`SPACETIME3D
`EXHIBIT 2009 - PAGE 5
`
`

`

`Patent Application Publication Mar. 21,2002 Sheet 5 of 27
`
`US 2002/0033848 Al
`
`FIG. 5
`
` 2
`
`
`
`MEMORY
`CARD
`INTERFACE
`
`MEMORY
`CARD
`
`
`
`
`
`INTERFACE7 |NTERFACE
`
`
`
`
`USER
`INTERFACE
`
`DISPLAY
`
`NETWORK
`
`DISPLAY
`
`SPACETIME3D
`EXHIBIT 2009 - PAGE 6
`
`SPACETIME3D
`EXHIBIT 2009 - PAGE 6
`
`

`

`Patent Application Publication Mar. 21,2002 Sheet 6 of 27
`
`US 2002/0033848 Al
`
`FIG. 6
`
`
`
`
`
`MEMORY
`FILE
`SEARCH
`CARD
`MANAGER
`
`MANAGER
`
`
`
`ORDER
`MANAGER
`
`
`
`DISPLAY
`
`ORDER
`TEMPLATE
`
`
`
`
`
`
`
`
` LAYOUT/
`
`PROGRAM
`MANAGER
`
` omeeeeee | |
`
` LAYOUT/
`
`
`
`|
`
`L
`
`SPACETIME3D
`EXHIBIT 2009 - PAGE 7
`
`SPACETIME3D
`EXHIBIT 2009 - PAGE 7
`
`

`

`Patent Application Publication Mar. 21,2002 Sheet 7 of 27
`
`US 2002/0033848 Al
`
`FIG. 7
`
`USER
`
`SYSTEM
`
`$1
`
`SELECT/ INSERT
`MEMORY STICK
`
`GO TO TO
`POINT B
`
`S&
` CHANGE
`
`YES
`LAYOUT OR
`
`
`ORDER ?
`
`
`NO
`S7
`LOOK
`
`CLOSER AT
`
`DATA OBJECT
`2
`
`YES GO TO TO
`
`POINT C
`
`NO
`
`38
`
`NO
`
` BROWSE
`THROUGH DATA
`
`OBJECTS ?
` END
`
`YES
`
`89
`
`USE INPUT METHOD
`TO BROWSE DATA
`
`OBJECTS
`
`$2
`
`ARE
`THERE DATA
`OBJECTS
`?
`
`NO END
`
`
`
`
`YES
`
`READ DATA OBJECTS
`INTO MEMORY
`
`
`
`APPLY LAYOUT AND
`ORDER TEMPLATE TO
`DATA OBJECTS
`
`$3
`
`S4
`
`SEND DATA VIEW TO
`OUTPUT DISPLAY
`
`$5
`
`$10
`
`ADJUST DATA VIEW AC
`CORDING TO USER'S
`INPUT
`
`
`
`DOES
`USER iNPUT
`CONT! NUE
`?
`
`NO
`
`GQ TO TO
`POINT A.
`
`SPACETIME3D
`EXHIBIT 2009 - PAGE 8
`
`SPACETIME3D
`EXHIBIT 2009 - PAGE 8
`
`

`

`Patent Application Publication Mar. 21,2002 Sheet 8 of 27
`
`US 2002/0033848 Al
`
`USER
`
`SYSTEM
`
`B. CHANGE LAYOUT OR ORDER
`
`USE THE LAYOUT OR
`ORDER INTERFACE TO
`MAKE NEW SELECTION
`
`
`
`
`
`LAYOUT OR ORDER
`
`
` DETERMINE NEW
`
`S21
`
`
`
`RULES
`
`
`
`APPLY LAYOUT AND
`
`ORDER TEMPLATE TO
`DATA OBJECTS
`
`
` $23
`
`S22
`
`SEND DATA VIEW TO
`OUTPUT DISPLAY
`
`
`
`THE DESIRED
`CHOICE ?
`
` IS THIS
`
`
`
`GO TO TO
`POINT A.
`
`SPACETIME3D
`EXHIBIT 2009 - PAGE 9
`
`SPACETIME3D
`EXHIBIT 2009 - PAGE 9
`
`

`

`Patent Application Publication Mar. 21, 2002 Sheet 9 of 27
`
`US 2002/0033848 Al
`
`USER
`
`SYSTEM
`
`C, LOOK CLOSER AT DATA OBJECT.
`
`S30
`
`HiT RETURN KEY OR
`OTHER EQUIVALENT
`INPUT TO SELECT THE
`
`DESIRED THUMBNAIL
`
`
`
`SCALE THE
`THUMBNAIL IN THE
`CURRENT WiNDOW
`
`$31
`
`DISABLE LAYOUT AND
`ORDER INTERFACE
`
`$32
`
`ENABLE ADDITIONAL
`COMMAND INTERFACES
`
`INCLUDING LAUNCHER
`
`$33
`
`SCALE THE THUMBNAIL
`BACK TO ORIGINAL
`
`SCALE
`
`S36
`
`SPACETIME3D
`EXHIBIT 2009 - PAGE 10
`
`GO TO TO
`POINT D.
`
`$35
`
`$34
`
` Do
`YES
`SOMETHING
`
`
`WITH THIS
`
`
`FILE ?
`
`
`NO
`
`HIT RETURN KEY OR
`OTHER EQU] VALENT
`INPUT TO SELECT TO
`RETURN TO
`
`THUMBNAIL VIEW
`
`GO TO TO POINT
`A OR C.
`
`SPACETIME3D
`EXHIBIT 2009 - PAGE 10
`
`

`

`Patent Application Publication Mar. 21, 2002 Sheet 10 of 27
`
`US 2002/0033848 Al
`
`FIG. 10
`
`USER
`
`SYSTEM
`
`D. DO SOMETHING WITH THIS FILE
`
`S40
`
`
`DECIDE
`
`WHAT TO DO
`WITH SELECTED
`DATA
`
`OBJECT
`LAUNCH
`
`APPLICATION
`
`EDIT
`
`USE EDIT
`INTERFACE
`
`BOOK
`MARK
`
`USE
`BOOKMARK
`INTERFACE
`
`USE INPUT METHOD TO
`SELECT DESIRED
`
`APPLICATION TO LAUNCH
`
`|.941
`
`$42
`
`ACCESS OS AND
`LAUNCH SELECTED
`
`APPLICATION
`
`$46
`? GO TO TO
`
`
`YES
`
`
`iS THIS
`DESIRED CHOICE
`
`POINT A.
`
`No
`
`USE NEW
`APPLICATION
`
`S47
`
`END
`
`$43
`
`544
`
`SEND DATA OBJECT
`TO LAUNCHED
`APPLICATION
`
`
`
`SCALE THUMBNAIL
`BACK TO ORIGINAL
`SCALE
`
`$45
`
`
`
`BRING LAUNCHED
`APPLICATION TO
`FRONT
`
`SPACETIME3D
`EXHIBIT 2009 - PAGE 11
`
`SPACETIME3D
`EXHIBIT 2009 - PAGE 11
`
`

`

`Patent Application Publication Mar. 21,2002 Sheet 11 of 27
`
`US 2002/0033848 Al
`
`FIG. 11
`
`23¢ P|
`™
`
`2]
`
`23d
`
`23e
`
`||
`
`SPACETIME3D
`EXHIBIT 2009 - PAGE 12
`
`SPACETIME3D
`EXHIBIT 2009 - PAGE 12
`
`

`

`Patent Application Publication Mar. 21,2002 Sheet 12 of 27.
`
`US 2002/0033848 A1
`
`FIG. 15
`
`
`
`SPACETIME3D
`EXHIBIT 2009 - PAGE 13
`
`SPACETIME3D
`EXHIBIT 2009 - PAGE 13
`
`

`

`Patent Application Publication Mar. 21,2002 Sheet 13 of 27
`
`US 2002/0033848 A1
`
`FIG. 16
`
`21
`
` Ci |__|ST
`
`aee|ee ee
`ee eeeeee
`ae
`EL
`
`SPACETIME3D
`EXHIBIT 2009 - PAGE 14
`
`SPACETIME3D
`EXHIBIT 2009 - PAGE 14
`
`

`

`
`
`SPACETIME3D
`2009 - PAGE 15
`
`SPACETIME3D
`EXHIBIT 2009 - PAGE 15
`
`

`

`Patent Application Publication Mar. 21,2002 Sheet 15 of 27
`
`US 2002/0033848 A1
`
`FIG. 18
`
`IMAGE1 0
`
`IMAGEGQ. JPG. jpg
`
`IMAGE159. JPG
`
`IMAGE1 60. JPG
`
`IMAGE1 61. JPG
`
`time
`
`IMAGE1 63. JPG
`
`type
`
`size
`
`28a
`
`28b
`
`SPACETIME3D
`EXHIBIT 2009 - PAGE 16
`
`SPACETIME3D
`EXHIBIT 2009 - PAGE 16
`
`

`

`Patent Application Publication Mar. 21,2002 Sheet 16 of 27
`
`US 2002/0033848 A1
`
`FIG. 19
`
`iMA
`
`a
`
`JPG
`
`IMAGE1 67.
`
`;
`
`MAGE] 77. JPG
`
`AGE166. JPG
`
`JPG
`
`IMAGE173. JPG [Ou]
`306k
`2000/01/25
`
`IMAGE1 74
`
`SPACETIME3D
`EXHIBIT 2009 - PAGE 17
`
`SPACETIME3D
`EXHIBIT 2009 - PAGE 17
`
`

`

`Patent Application Publication Mar. 21,2002 Sheet 17 of 27.
`
`US 2002/0033848 A1
`
`FIG. 20
`
`Co
`
`SPACETIME3D
`EXHIBIT 2009 - PAGE 18
`
`SPACETIME3D
`EXHIBIT 2009 - PAGE 18
`
`

`

`Patent Application Publication Mar. 21,2002 Sheet 18 of 27
`
`US 2002/0033848 A1
`
`FIG. 21
`
`THUMBNAIL RING
`
`SPACETIME3D
`EXHIBIT 2009 - PAGE 19
`
`SPACETIME3D
`EXHIBIT 2009 - PAGE 19
`
`

`

`Patent Application Publication Mar. 21,2002 Sheet 19 of 27.
`
`US 2002/0033848 A1
`
`FIG. 22
`
`Yi
`
`XI
`
`(Xcent’ Ycent)
`
`SPACETIME3D
`EXHIBIT 2009 - PAGE 20
`
`SPACETIME3D
`EXHIBIT 2009 - PAGE 20
`
`

`

`Patent Application Publication Mar. 21, 2002 Sheet 20 of 27
`
`US 2002/0033848 A1
`
`FIG. 23 (a)
`
`FIG. 23 (b)
`
`DESIRED TARGET
`
`ACTUAL DISPLAYED
`POSITIONS Si
`
`POSITIONS Di
`
`Qo
`
`Q
`
`SPACETIME3D
`EXHIBIT 2009 - PAGE 21
`
`SPACETIME3D
`EXHIBIT 2009 - PAGE 21
`
`

`

`Patent Application Publication Mar. 21,2002 Sheet 21 of 27
`
`US 2002/0033848 A1
`
`FIG. 24
`
`FOCUS THUMBNAIL CHANGED
`BY INPUT FROM USER
`
`952
`
`SET TARGET POSITION:
`BASED ON PRESENT LAYOUT
`
`AND NEW FOCUS THUMBNAIL
`
`
`
`CALCULATE NEW THUMBNAIL POSITION
`Sinew BASED ON THUMBNAIL'S PRESENT
`POSITION Sipresent AND TARGET
`POSITION Di
`
`S53
`
` $50
`
`
`
`
` i < TOTAL NUMBER
`OF THUMBNAILS ?
`
`FIG. 25
`
`Di
`
`
`
`Sipresent
`
`SPACETIME3D
`EXHIBIT 2009 - PAGE 22
`
`SPACETIME3D
`EXHIBIT 2009 - PAGE 22
`
`

`

`Patent Application Publication Mar. 21,2002 Sheet 22 of 27
`
`US 2002/0033848 A1
`
`FIG. 26
`
`MUSIC FILE
`
`
`
`~ FILE INFORMATION:
`SONG TITLE
`da
`ARTIST'S NAME
`
`PLAYING TIME
`
`
`~ MUSIC DATAisaileter]iewwa]
`
`
`
`54
`
`54b
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`SPACETIME3D
`EXHIBIT 2009 - PAGE 23
`
`SPACETIME3D
`EXHIBIT 2009 - PAGE 23
`
`

`

`
`
`©
`=
`Li
`
`
`
`Patent Application Publication Mar. 21,2002 Sheet 23 of 27.
`
`US 2002/0033848 A1
`
`aq
`ii
`
`>
`oo
`cAz
`3s]
`=
`
`w—O
`
`o |©L
`
`Ua
`
`m=—
`
`_
`=>
`
`SPACETIME3D
`EXHIBIT 2009 - PAGE 24
`
`—a
`
`=
`YH
`
`y:
`ttt
`£4
`
`
`aad
`N
`
`iu
`
`NI N|5
`a1
`a
`w|=
`=|
`-
`
`
`we]
`s
`ac
`LU
`
`SPACETIME3D
`EXHIBIT 2009 - PAGE 24
`
`

`

`Patent Application Publication Mar. 21,2002 Sheet 24 of 27
`
`US 2002/0033848 A1
`
`FIG. 28
`
`EXECUTABLE
`
`DATA
`FILES
`
`CD-ROM
`
`
`COMPUTER
`
`
`
`
`DISPLAY
`
`FIG. 29
`
`MEMORY CARD
`
`COMPUTER
`
`DATA
`FILES
`
`EXECUTABLE
`
`
`DISPLAY
`
`SPACETIME3D
`EXHIBIT 2009 - PAGE 25
`
`SPACETIME3D
`EXHIBIT 2009 - PAGE 25
`
`

`

`Patent Application Publication Mar. 21,2002 Sheet 25 of 27
`
`US 2002/0033848 A1
`
`FIG. 30
`
`PictureGear W
`
`Images
`
`Others
`
`CHIMAGEO1.jpg
`[1 99.9.27
`[1 680k
`
`Add to List ¥
`
`SPACETIME3D
`EXHIBIT 2009 - PAGE 26
`
`THUMBNAILS FOR
`
`|__+-——sWALLEST DATA FILES
`THUMBNAILS FOR||
`|of|] IEEE
`|ET
`
`| T
`
`HUMBNAIL FOR
`LARGEST DATA
`FILE
`
`SPACETIME3D
`EXHIBIT 2009 - PAGE 26
`
`

`

`Patent Application Publication Mar. 21, 2002 Sheet 26 of 27
`
`US 2002/0033848 A1
`
`FIG. 31
`
`37 items 100k remaining
`
`
`
`
`
`
`
`ProtureGear
`
`aw
`enith
`IMAGED1. jpg
`A
`99. 9. 27
`
`View [naaes]
`
`SPACETIME3D
`EXHIBIT 2009 - PAGE 27
`
`SPACETIME3D
`EXHIBIT 2009 - PAGE 27
`
`

`

`Patent Application Publication Mar. 21,2002 Sheet 27 of 27.
`
`US 2002/0033848 A1
`
`FIG. 32
`
`IMAGED01. jpg
`
`{MAGEOD02. jpg
`
`IMAGEOO3. jog
`
`IMAGEDO4. jpg
`
`PictureGear
`
`IMAGEQO5. jpg
`
`IMAGEG10. jpg
`
`IMAGEO11. jpg
`
`IMAGED12. jpg
`
`IMAGE013. 1p
`
`IMAGEO1 4. jpg
`
`IMAGEO15. 1p9
`
`View [Images|browse by open with
`
`A
`:
`4]
`—
`
`SPACETIME3D
`EXHIBIT 2009 - PAGE 28
`
`SPACETIME3D
`EXHIBIT 2009 - PAGE 28
`
`

`

`US 2002/0033848 Al
`
`Mar. 21, 2002
`
`SYSTEM FOR MANAGING DATA OBJECTS
`
`BACKGROUND OF THE INVENTION
`
`[0001]
`
`1. Field of the Invention
`
`invention relates to an on-screen
`[0002] The present
`graphical user interface, and database management.
`
`[0003]
`
`2. Related Art
`
`[0004] Digital cameras, mini disks, and other devices for
`recording and playing digital media are becoming quite
`popular. These devices store digital media as data objects in
`memories such as flash memories, hard disks, and on-line
`storage.
`
`[0005] There are many methods available for managing
`large numbersof data objects. For example, data objects are
`frequently managed using windows and folders. U.S. Pat.
`No. 5,917,488 describes a system for displaying and
`manipulating image data sets. First, a window including
`programmed groupsis displayed. When a userselects one of
`the programmed groups, then a second windowis displayed
`with thumbnails that represent data objects of the selected
`programmed group.
`
`selectively
`for
`is
`enable/disable means
`[0012] The
`enabling and disabling the moving means. When the enable/
`disable means disables the moving means, then the display-
`ing meansstatically displays a single thumbnail in the focus
`region. When the enable/disable means enables the moving
`means, then the displaying meansdisplays the thumbnails to
`move along the predetermined path through the focus
`region.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`[0013] The above and other objects, features and advan-
`tages of the invention will become more apparent from
`reading the following description of the embodiment taken
`in connection with the accompanying drawings in which:
`
`[0014] FIG. 1 is a schematic view showing a window
`display with thumbnails arranged in a line layout;
`
`[0015] FIG. 2 is a schematic view showing the window
`display with thumbnails arranged in a circle layout;
`
`[0016] FIG. 3 is a schematic view showing the window
`display with thumbnails arranged in a grid layout;
`
`[0017] FIG. 4 is a schematic view showing the window
`display with thumbnails arranged in a helix layout;
`
`[0006] Data objects are also handled in hierarchicallists,
`[0018] FIG.5is a block diagram showing components of
`using static light-box metaphors. A light-box metaphor
`a computer and peripheral equipment for displaying the
`displays images in the mannerof “slides”all the same size
`window display of FIGS. 1 to 4
`on a white background. The “slides” are arranged in a grid.
`
`[0019] FIG. 6 is a block diagram showing functional
`modules the computer of FIG.5;
`
`[0007] There are methods knownforselecting a format for
`displaying data objects. For example, U.S. Pat. No. 5,673,
`429 describes a system that displaysa list of different display
`formats. Whentheuserselects a desired display format from
`the list, then items that match the selected format are pulled
`from a database and displayed in a list that depends on the
`selected display format.
`
`[0008] Memory is becoming available in increasingly
`large capacities, which enable storage of increasingly large
`numbers of data objects. For example, memory cards are
`available that are capable of storing data for several hundred
`different digital photographs. However, conventional tech-
`FIG.11 is a schematic view showing movementof
`[0024]
`niques, such as windows,folders, and lists, have a problem
`a thumbnail representingafirst-loaded data object;
`in that they are insufficient for presenting such large num-
`bers of data objects in a manner that a user can easily and
`quickly understand.
`
`[0020] FIG. 7 is a flowchart representing overall pro-
`cesses for implementing the functional modules of FIG.6;
`
`[0021] FIG. 8 is a flowchart representing processes for
`changing layout or order of thumbnails in the window
`display;
`
`[0022] FIG. 9 is a flowchart representing processes for
`taking a closer look at a focus thumbnail;
`
`[0023] FIG. 10 is a flowchart representing processes for
`performing further process on the focus thumbnail;
`
`[0025] FIG. 12 is a schematic view showing movement of
`thumbnails as further data objects are loaded;
`
`SUMMARYOF THE INVENTION
`
`It is an objective of the present invention to solve
`[0009]
`the above-described problems, and to provide a method and
`device that allows a user to easily browse, view, manage,
`select, and command a large number of data objects.
`
`In order to achieve this objective, a device accord-
`[0010]
`ing to the present
`invention for managing data objects,
`includes displaying means, defining means, moving means,
`and enable/disable means.
`
`[0011] The displaying meansis for displaying thumbnails
`representative of data objects. The defining means is for
`defining a focus region that indicates a focus thumbnail
`subject to processes. The moving meansis for moving the
`displayed thumbnails along a predetermined path through
`the focus region.
`
`[0026] FIG. 13 is a schematic view showing movement of
`thumbnails as further data objects are loaded;
`
`[0027] FIG. 14 is a schematic view showing movement of
`thumbnails as further data objects are loaded;
`
`[0028] FIG. 15 is a schematic view graphically represent-
`ing pivoting movementof a thumbnailline in the line layout;
`
`[0029] FIG. 16 is a schematic view showing browsing
`motion of a thumbnail grid in the grid layout;
`
`[0030] FIG. 17 is a schematic view showing browsing
`motion of a thumbnail helix in the helix layout;
`
`[0031] FIG. 18 is a schematic view showing motion of a
`thumbnail when a new orderis selected;
`
`[0032] FIG. 19 is a schematic view showing the display
`window whena closer look is taken at the focus thumbnail;
`
`SPACETIME3D
`EXHIBIT 2009 - PAGE 29
`
`SPACETIME3D
`EXHIBIT 2009 - PAGE 29
`
`

`

`US 2002/0033848 Al
`
`Mar. 21, 2002
`
`[0033] FIG. 20 is a graphical view for explaining the
`pivoting movementof the thumbnailline;
`
`FIG.21 is a graphical view for explaining the form
`[0034]
`of the circle layout;
`
`[0035] FIG. 22 is a graphical view for explaining the form
`of the helix layout;
`
`[0036] FIG. 23(a) is a schematic view showing displayed
`position of thumbnails before, during, and after the focus
`thumbnail is changed;
`
`[0037] FIG. 23(5)is a schematic view showing timing of
`change of the focus thumbnail;
`
`[0038] FIG. 24 is a flowchart showing operations for
`implementing movement of the thumbnails shown in FIG.
`23(b);
`
`[0039] FIG. 25 is a view schematically showing determi-
`nation of a new position for the newly selected thumbnail;
`
`[0040] FIG. 26 is a schematic view representing processes
`for displaying audio files visually as thumbnails;
`
`[0041] FIG. 27 is a schematic view showing thumbnails
`divided into sections depending on the amount of data in
`corresponding audio files;
`
`[0042] FIG. 28 is a schematic view showing configuration
`of a CD-ROM;
`
`[0043] FIG. 29 is a schematic view showing configuration
`of a memory card;
`
`will be described in detail below, the layouts use the X and
`Y dimensionsandscale to layout thumbnails that represent
`various data objects.
`
`[0050] As shownin FIG.1, a focus outline 24 surrounds
`the center-most, or focus, thumbnail 25 in the thumbnail
`screen 21. The focus outline 24 indicates that the focus
`
`thumbnail 25 is currently the focus of any operationsiniti-
`ated by input from the user. For example, when input is
`received for browsing through the thumbnails 23,
`the
`thumbnails 23 are dynamically scrolled across the screen
`through the focus outline 24, following a path P according
`to the presently implemented layout as indicated by arrows
`Pin FIGS. 1, 2, 16, and 17. In this way, the user’s attention
`can be selectively drawnto particular ones of the thumbnails
`23 by dynamically moving the thumbnails 23 through the
`single screen 21, rather than by using multiple windows in
`the conventional manner. The single screen 21 is advanta-
`geous over multiple windows, because the single screen 21
`gives the user a primary focus and a single unified context
`as he or she manipulates the thumbnails 23, which represent
`data objects. As will be described in detail below,
`the
`thumbnails 23 are moved by panning,scrolling, and zoom-
`ing in the screen 21 to change the degrees of focus placed on
`the thumbnails 23.
`
`[0051] An information tab 26 is displayed within the focus
`outline 24 at a position below the focus thumbnail 25. The
`information tab 26 includes further information about the
`focus thumbnail 25 such as a file name, date, size, and file
`type of the data object that correspondsto the focus thumb-
`nail 25.
`
`[0044] FIG. 30 is a schematic view showing thumbnails
`displayed according to the size of the corresponding data
`files;
`
`[0045] FIG. 31 is a schematic view showing thumbnails
`displayed separated from each other depending on timing of
`the corresponding data files; and
`
`[0046] FIG. 32 is a schematic view showing thumbnails
`displayed in groups depending on the relationship of the
`corresponding datafiles.
`
`DETAILED DESCRIPTION OF THE
`EMBODIMENT
`
`[0052] The layout/order interface 22 is displayed at the
`bottom of the screen 21 below the display of the thumbnails
`23, and includes a layout interface 27 and an order interface
`28. The layout
`interface 27 includes icons 27a to 27b
`representing each of the four different layouts. The user can
`select a desired display layout by selecting the correspond-
`ing icon 27a to 27d, using a mouse and cursor for example.
`The icon 27a to 27d that represents the presently selected
`layout
`is displayed near the center of the layout/order
`interface 22 to indicate the presently selected layout. The
`selected icon can also be changed by clicking on the center
`icon, or by clicking on an icon 27e. With each click, the
`icons 27a to 27d switch places with the selected icon one
`after anotherin order. The graphical icons 27a to 27d allows
`[0047] Next, a system according to an embodimentof the
`a user to select a display layout with more confidence that
`present invention will be described while referring to the
`whenselecting by usingalist.
`accompanying drawings. The system is for managing data
`objects such as digital image files, digital audio files, text
`files, executable programs, program files, and moviefiles.
`
`[0048] As shown in FIGS. 1 to 4, the system displays a
`window 20 including a thumbnail screen 21 and a layout/
`order interface 22. Thumbnails 23, that represent a variety of
`data objects, are displayed in the screen 21. As shown in
`FIGS.1 and3, each of the thumbnails 23 includes an image
`that is the essence of the corresponding data object. The
`thumbnails shown in all other drawings include similar
`images, but the images have been omitted from the drawings
`to facilitate explanation.
`
`[0053] The order interface 28 is for selecting one of four
`different sequence orders that determined arrangementof the
`thumbnails 23 in the selected layout. The four orders
`include: nameof data object, type of data object, size of the
`data file, and time the file was stored. The order interface 28
`is displayed at the right bottom of the screen, and includes
`words 28a to 28d that represent each of the four different
`order types. The word, name 28a in this example,
`that
`represents the selected order is displayed near the center of
`the layout/order interface 22. The selected icon can be
`changed by either clicking on a desired word, or by clicking
`on an icon 28e.
`
`[0049] The thumbnails 23 can be displayed in any of four
`different graphical display layouts: a line layout as shown in
`FIG.1, a circle layout as shown in FIG.2, a grid layout as
`shownin FIG.3, and a helix layout as shownin FIG.4. As
`
`[0054] Here, a more detailed description of the four lay-
`outs will be provided. As shown in FIG.1, in the line layout,
`the thumbnails 23 are aligned in a slightly overlapping
`mannerin a single continuousangled line. File names 31 for
`
`SPACETIME3D
`EXHIBIT 2009 - PAGE 30
`
`SPACETIME3D
`EXHIBIT 2009 - PAGE 30
`
`

`

`US 2002/0033848 Al
`
`Mar. 21, 2002
`
`the data objects are displayed in a separate angled line that
`intersects the angled thumbnail line. The line layout has a
`clear beginning and end, so the user can easily grasp the
`number of data objects.
`
`[0055] As shown in FIGS. 2 and 16, the circle layout
`consists of slightly overlapping thumbnails 23 in an arc of a
`circular ring. Filenames 31 corresponding to the thumbnails
`23 appearin an arc of a secondring that is symmetric to, but
`inverse from the thumbnail arc. Although not shown in the
`drawings, when there are only a few thumbnailsin the circle
`layout, the entire circular ring of thumbnails is displayed in
`the screen 21. At this time, filenamesare displayed in a circle
`below the thumbnail circle. However, the ring enlarges as
`the number of thumbnails increases, until a portion of the
`thumbnail ring can not be displayed. Whenthe ring contains
`thumbnails up to a certain number, then the radius of the are
`increases as the size of the ring increases. However, once a
`certain numberof thumbnails 23 is exceeded, so that the ring
`reaches a certain size, then only an arc having a predeter-
`mined radius is displayed as shown in FIG.2. The radius of
`the arc is fixed regardless of how many thumbnails 23 are in
`the ring, because if the radius were varied according to the
`number of thumbnails 23 in the ring, then the displayed arc
`would appear linear when the ring contains a large number
`of thumbnails 23.
`
`[0056] As shownin FIG.4,in the grid layout, the thumb-
`nails 23 are aligned vertically and horizontally, that is, with
`fixed mutual position in horizontally-extending rows and
`vertically-extending columns. The number of thumbnails in
`the rows of the grid layout is determined by how many
`thumbnails 23 if aligned side by side, can be displayed
`completely on the screen. In the example shown in FIG.4,
`five thumbnails 23 are disposed side by side in each hori-
`zontal row, because in this example complete rows offive
`thumbnails 23 can be displayed in the screen 21 without the
`thumbnails in the rows overlapping. Any further thumbnails
`are added by increasing the number of rows, that is, by
`adding the thumbnails to elongate the columns.It should be
`noted that rows with moreor less than five thumbnails might
`be desirable, depending on the displaying conditions. The
`thumbnails 23 contact each other on lateral sides, but have
`a space between top and bottom sides. The file names 31
`appear below each corresponding thumbnail 23. The grid
`layout displays more complete rows of thumbnails 23,
`without the thumbnails overlapping, than any of the other
`layouts.
`
`[0057] As shownin FIG.4, the thumbnails 23 in the helix
`layout are displayed in the screen 21 in different scales to
`give the illusion of viewing a three-dimensional helix from
`the side. That is, thumbnails 23a at the far side of the helix
`are displayed in a smaller scale than thumbnails 235 at the
`near side of the helix. The helix shape includes rings 35a to
`35c of thumbnails 23. Each ring 35a, 35b, 35c includes
`about ten thumbnails 23. The helix layout shows a large
`number of thumbnails at the same time, while maintaining
`the benefits of a line configuration.
`
`[0058] Next, configuration ofthe file management system
`for displaying the window 20 will be explained, along with
`further explanation of various features of the file manage-
`ment system. As shown in FIG. 1,
`the file management
`system includes a computer 1, a memorycard 2, a display 3,
`and a user input unit 4.
`
`[0059] The memory card 2 can be any memory card, such
`as a Memory Stick™, a SmartMedia™ memorycard, or a
`Compact Flash™ memory card. The memory card 2 stores
`data files for a large number of photographs, songs, or any
`other type of data objects. The other modules of the system
`can write data objects into, or read data objects from, the
`memory card 2. The user input umit 4 can be any user
`interface for inputting commands and the like, such as a
`keyboard, a jog-dial, a game controller, or an on-screen unit
`such as a mouse or a pen.
`
`[0060] The computer 1 includes a slot 2a for insertion of
`the memory card 2, a memory card interface 10, a CPU 11,
`a ROM 12, a RAM 13,a hard disk 14, a user interface 15,
`a display interface 16, and a network interface 17, all
`connected together by a bus 16. The user interface 15 is
`connected to the user input unit 4. The display interface 16
`is connected to the display 3. The network interface 17 is
`connected to a network 5.
`
`[0061] FIG. 2 shows functional modules of the computer
`1. As shown the computer 1 includes a file manager 11, a
`layout/order manager 12, a search manager 13, a program
`manager 14, and a layout/order template stock 15. The
`functional modules are realized by application software
`stored in the hard disk 14, or retrieved over the network 5.
`
`[0062] The file manager 11 reads data objects from the
`memorycard 2 and stores them in memory, such as the RAM
`13.
`
`[0063] The layout/order template stock 15 is memory,
`such as the hard disk 14,that holds a layout/out template for
`each combination of the four layouts and the four orders.
`Each layout/order template contains two primary rule sets
`for the corresponding layout, one set of rules for setting
`relative positions of thumbnails 23 and another set of rules
`for movement of thumbnails 23.
`
`[0064] Althoughrules in the layout/order templates will be
`described later, with respect to relative position of thumb-
`nails, a layout/order template for the grid layout includes
`layout rules that indicate that each thumbnail should be
`positioned immediately to the right of the preceding thumb-
`nail, unless the thumbnail is sixth in a thumbnail sequence.
`If the thumbnail is sixth in the sequence, then the thumbnail
`should be positioned underneath the first
`image in the
`sequence in order to begin a new sequence, thatis, to start
`a new row in the grid. The templates for the line, circle, and
`helix layouts indicate that the thumbnails should be aligned
`in line, circle, and helix patterns, respectively.
`
`[0065] With respect to rule for movement of the thumb-
`nails, the grid template also includes rules defining that the
`thumbnails in the grid layout move as shown in FIG. 16
`when the user browses through the thumbnails in a manner
`to be described below. The templates for the line, circle, and
`helix layouts indicate that the thumbnails should be move
`following linear, circular, and spiraling paths, respectively.
`
`[0066] The program manager 14 reads a layout/order
`template from the layout/order template stock 15 for the
`presently selected layout and order. The program manager
`14 writes the template into the file manager 11 to apply the
`template to the data objects stored by the file manager 11.
`The program manager 14 reads templates from the layout/
`order template stock 15 and writes them in the file manager
`
`SPACETIME3D
`EXHIBIT 2009 - PAGE 31
`
`SPACETIME3D
`EXHIBIT 2009 - PAGE 31
`
`

`

`US 2002/0033848 Al
`
`Mar. 21, 2002
`
`11 each time the layout or order is changed, because only
`one set of display rules needs to be in active memory at any
`one time.
`
`[0067] The layout/order manager 12 reads information
`about the data objects in the file manager 11, and draws the
`corresponding thumbnails 23 accordingly in the presently
`selected layout. The file layout/order manager 12 controls
`the display 2 to display the thumbnails accordingly.
`
`[0068] The search engine 13 enables a user to search for
`data objects based on inputted search criteria. The search
`engine 13 incrementally writes the criteria in the file man-
`ager 11. The display 3 changes the screen 21 to display only
`thumbnails of data objects that satisfy the search criteria.
`The search engine 13 supports key words and file attributes,
`such as date, size, color.
`
`the computer 1 will be
`[0069] Here, operations of
`described while referring to the flowcharts in FIGS. 7 to 10.
`In S1, the operating system of the computer 1 recognizes the
`memory card 2 when the memorycard 2 is inserted into the
`slot 2a. As a result,
`the operating system launches an
`executable application for performing the functions of the
`file manager 11, the layout/order manager 12, and the other
`function modules in FIG. 6. The executable application can
`be eitherretrieved for this purpose from whereit is stored in
`the hard disk 14, or downloaded from over the network5, for
`example. In S2,
`the file manager 11 judges whether the
`memory card 2 stores any data objects. If not (S2:NO), then
`the application is ended here.
`
`[0070] On the other hand, if the memory card 2 stores data
`objects (S2:YES), then in S3 the file manager 11 loads the
`data objects from the memory card 2 one at a time into the
`RAM 13. In S4, the program manager 14 determines the
`presently selected layout and orderfor display of thumbnails
`to represent the data objects. In the present example, the line
`layoutis set as the default layout, and file nameis set as the
`default order. The program manager 14 then pulls the
`appropriate layout/order template from the layout/order tem-
`plate stock 15 and stores the layout/order template in the
`RAM 12. The program manager 14 writes the template on
`the file manager 11, thereby applying the template to each
`data object in the file manager 11.
`
`InS5, the file layout/order manager 12 drawsa data
`[0071]
`view for the thumbnails 23 according to the presently
`selected layout and order, and controls the display 2 to
`display the thumbnails accordingly.
`
`[0072] Each thumbnail 23 is displayed as soon asretrieved
`by the file manager 11, and during the loading process the
`thumbnails are dynamically arranged on the screen accord-
`ing to the presently selected layout. That is, as shown in
`FIG. 11, a thumbnail 23c of the first-loaded data object is
`displayed at the upper left corner of the screen 21 and then
`moved to the center of the screen 21. The focus outline 24
`is displayed surrounding the thumbnail 23c in a size that
`matches the dimensions of the thumbnail 23c.
`
`[0073] As shown in FIG. 12, a thumbnail 23d of the
`second-loaded data object is similarly displayed at the upper
`left