as) United States
`a2) Patent Application Publication 10) Pub. No.: US 2013/0326338 Al
`
` Secordetal. (43) Pub. Date: Dec. 5, 2013
`
`
`US 20130326338Al1
`
`(54) METHODS AND SYSTEMSFOR
`ORGANIZING CONTENTUSING TAGS AND
`FOR LAYING OUT IMAGES
`
`(75)
`
`Inventors: Adrian Secord, New York, NY (US);
`Hendrik Kueck, Vancouver (CA)
`
`(73) Assignee: Adobe Systems Incorporated
`
`(21) Appl. No.: 11/899,685
`
`(22)
`
`Filed:
`
`Sep. 7, 2007
`
`Publication Classification
`
`(52) U.S.CL
`USPC.
`ceecsesssssssssssssesessssssssessssessssesssesssesesees 715/243
`ABSTRACT
`(57)
`Methods and systems for placing images on a display such
`that imagesthat are closely related to each other are placed
`near each other, and images that are unrelated are placed
`further away. The invention encompasses various techniques
`that may be used to determine how related or similar images
`are to one another, including methodsthat take into account
`user-supplied tags or other tags that describe the image’s
`content and/or source. Color content, image content, time of
`image capture, and/or, other attributes may also be used in
`assessing image similarity. For example, one embodimentof
`the invention is a method for laying out images based on the
`similarity of the images with respect to color, tags, and/or
`other metadata. The layout of images may also involve
`
`(51)
`
`Int. Cl.
`GO6F 17/00
`
`(2006.01)
`
`removing overlap between images.
`
`
`
`400
`
`1
`
`SAMSUNG 1035
`
`SAMSUNG 1035
`
`1
`
`

`

`Patent Application Publication
`
`Dec. 5, 2013 Sheet 1 of 7
`
`US 2013/0326338 Al
`
`13 Memory
`
`14 Image Viewer
`
`14a Image a
`
`14b Image b
`
`14n Image n
`
`FIGURE1a
`
`50 Network
`
` 11 User Computer
`
`10
`
`
`
`21 User Computer
`
`31 User Computer
`
`we 33 Memory
`
`FIGURE 1b
`
`2
`
`

`

`Patent Application Publication
`
`Dec. 5, 2013 Sheet 2 of 7
`
`US 2013/0326338 Al
`
`200
`
`IDENTIFYING A COLLECTION OF IMAGES
`
`240
`
`DETERMINE A LAYOUTUSING THE SIMILARITY BETWEEN IMAGES
`230
`
`DETERMINING A SIMILARITY OF EACH IMAGE OF THE COLLECTION
`TO EVERY OTHER IMAGEOF THE COLLECTION USING TAGS
`
`220
`
`LAYING OUT THE IMAGES OF THE COLLECTION ACCORDING TO
`THE LAYOUT
`
`FIGURE2
`
`3
`
`

`

`Patent Application Publication
`
`Dec. 5, 2013 Sheet 3 of 7
`
`US 2013/0326338 Al
`
`at
`
`t
`
`FIGURE 3
`
`300
`
`4
`
`

`

`Patent Application Publication
`
`Dec. 5, 2013 Sheet 4 of 7
`
`US 2013/0326338 Al
`
`
`
`
`
`FIGURE 4
`
`400
`
`5
`
`

`

`Patent Application Publication
`
`Dec. 5, 2013 Sheet 5 of 7
`
`US 2013/0326338 Al
`
`
`
`500
`
`FIGURE 5
`
`6
`
`

`

`Patent Application Publication
`
`Dec. 5, 2013 Sheet 6 of 7
`
`US 2013/0326338 Al
`
`
`
`600
`
`FIGURE 6
`
`7
`
`

`

`Patent Application Publication
`
`Dec. 5, 2013 Sheet 7 of 7
`
`US 2013/0326338 Al
`
`700
`
`|
`705
`
`711
`
`712
`
`713
`
`721
`
`
`
`FIGURE 7
`
`8
`
`

`

`US 2013/0326338 Al
`
`Dec. 5, 2013
`
`METHODS AND SYSTEMS FOR
`ORGANIZING CONTENT USING TAGS AND
`FOR LAYING OUT IMAGES
`
`FIELD OF THE INVENTION
`
`[0001] The present invention relates to methods and sys-
`tems for laying out images, including methods and systems
`for laying out images based on imagesimilarities.
`
`BACKGROUND
`
`Personal image collections have grown larger and
`[0002]
`more extensive overtime. It is not uncommonfora collection
`
`to reach ten to twenty thousand images, often left unsorted or
`uncategorized in the digital version of a shoebox. The most
`commonorganization methodused,if one is used atall, is to
`sort the images into a hierarchical set of categories, for
`example people—family—mom.In addition, image tagging
`is supported in one form or another by most photo manage-
`ment programs on the market today. Such tagging allows
`imagesto be associated with one or moretagsthat typically
`describe the image in some way. For example, an image of a
`dog in a park might be tagged with “dog,” “park,” and
`“Sparky,” if that is the dog’s name,to describe the content of
`the image. Tags are a popular image metadata and are sup-
`ported by many Adobe® products such as Lightroom®, Pho-
`toshop®, and others, and can take other forms aside from
`keywords.
`[0003] Browsing though image collections becomes more
`difficult as the collection grows. Typically an application will
`provide a scrollable list of thumbnails and some method of
`filtering a collection. For example, the user might choose to
`view a specific folder of imagesor to search his/hercollection
`for a particular keyword or tag. The results are nearly always
`displayed as a long scrollable list of thumbnails. Lists of
`thumbnails givelittle or no information about how the items
`are related to each other. Image collectionsare relatively rich
`in information such as image capture time, file name, location
`on the hard drive, camera parameters (exposure, etc.), color
`information and tags. Current methodsof displaying lists of
`images do not take advantage of this extra information and
`fail to provide for the display of images in a way that shows
`the connectionsor similarities between images in an intuitive
`way.
`
`SUMMARY
`
`invention provide
`[0004] Embodiments of the present
`methods and systems for dynamically placing images on a
`screen such that imagesthat are closely related to each other
`are placed near each other, and imagesthat are unrelated are
`placed further away. The invention encompasses various
`techniquesthat may be usedto determine and/or quantify how
`related or similar imagesare to one another, including meth-
`ods that take into account user-suppliedtags or other tags that
`describe image content. Color content, image content, time of
`image capture, and/or other attributes may also be used in
`assessing image similarity. For example, one embodiment of
`the invention is a method for laying out images basedat least
`in part on the similarity of the images with respect to color,
`tags, and/or other metadata.
`invention is a
`[0005] One embodiment of the present
`methodof laying out images. In this method, a collection of
`images stored on at least one computing device is identified
`with at least one tag associated with at least some of the
`
`images of the collection. The similarity of each image to
`every other image is determined usingat least someofthe tags
`associated with at least some of the imagesofthe collection.
`A layout is determined using the similarity between at least
`someofthe images, the location of at least some ofthe images
`in the layout based at least in part on the similarity of the
`imageto at least one other image. The imagesare laid out or
`otherwise displayed or printed according to the determined
`layout.
`[0006] One embodimentofthe present invention comprises
`a method of laying out images that comprises identifying
`images, determining similarities, determining a layout, and
`laying out the images. This may comprise identifying a col-
`lection of images with at least one tag associated with at least
`some of the imagesof the collection. The method may com-
`prise determining a similarity of each imageofthe collection
`to other imagesofthe collection using at least some ofthe tags
`associated with at least some of the imagesofthe collection.
`Determining similarity may involve comparing every image
`to one another, recognizing that two imagesare tagged witha
`same tag, and/or using the frequency tags are used together
`with images. The method may comprise determining a layout
`using the similarity betweenat least some of the images. The
`location ofat least some imagesin the layout may be based at
`least in part on the similarity ofthe imageto at least one other
`image. Images maybepositioned in the layout and/or sized to
`avoid overlapping. And, the method may comprise laying out
`the imagesof the collection according to the layout.
`[0007] One embodimentofthe present invention comprises
`a method of laying out images that comprises identifying a
`collection of images, determining low-dimensionalpositions
`for the images based on image similarities, determining a
`layout using the low dimensionalpositions and laying out the
`images according to the layout. Determining low-dimen-
`sional positions for the images based on image similarities
`may comprise assessing a combination of similarities (for
`example relating to tag, time, or color similarity) and may
`comprise multidimensionalscaling or any other suitable tech-
`nique.
`computer-readable
`a
`embodiments,
`In other
`[0008]
`medium (such as, for example random access memory or a
`computer disk) comprises code for carrying out these meth-
`ods.
`[0009] These embodiments are mentioned not to limit or
`define the invention, but to provide examples of embodiments
`of the invention to aid understanding thereof. Embodiments
`are discussedin the Detailed Description, and further descrip-
`tion of the invention is provided there. Advantages offered by
`the various embodiments of the present invention may be
`further understood by examiningthis specification.
`
`BRIEF DESCRIPTION OF THE FIGURES
`
`[0010] These and other features, embodiments, advantages
`of the present invention are better understood when the fol-
`lowing Detailed Description is read with reference to the
`accompanying drawings, wherein:
`[0011]
`FIG. 1isasystem diagram illustrating anillustrative
`network environment according to one embodimentof the
`present invention;
`[0012]
`FIG. 2 is a flow chart illustrating one method of
`laying out a collection of images according to one embodi-
`mentof the present invention;
`[0013]
`FIG. 3 is an exemplary image layout according to
`one embodimentof the present invention;
`
`9
`
`

`

`US 2013/0326338 Al
`
`Dec. 5, 2013
`
`FIG. 4 is an exemplary image layout with gridded
`[0014]
`overlap according to one embodimentof the present inven-
`tion;
`FIG.5 illustrates a layout without overlap between
`[0015]
`images producedaccording to one embodimentofthe present
`invention;
`[0016]
`FIG. 6 illustrates another layout without overlap
`between images produced according to one embodiment of
`the present invention; and
`[0017]
`FIG. 7 illustrates another layout without overlap
`between images produced according to one embodiment of
`the present invention.
`[0018]
`Thefile of this patent contains at least one drawing
`executed in color. Copiesofthis patent with color drawing(s)
`will be provided by the Patent and Trademark Office upon
`request and paymentof the necessary fee.
`
`DETAILED DESCRIPTION
`
`Iilustrative Image Layout
`
`invention provide
`[0019] Embodiments of the present
`methods and systems for the layout of an image collection
`(such as a personal photographcollection) using the relation-
`ships amongstuser-defined tags. In one illustrative embodi-
`ment, a user downloadsa collection of images from a digital
`camera to a desktop computer where the imagesare stored as
`electronic files. The user then associates tags with several of
`the images. For example, the user tags a first image with
`“dog,” “cute,” and “park,” a second image with “birds” and
`“park,” a third image with “vacation,” “beach,” and “dog,” a
`fourth image with “baby”and “cute,”a fifth tag with “baby”
`and “eating,” and other images with various tags assigned or
`otherwise associated.
`
`[0020] The user executes an image organizing application.
`For example, the user may launch a file organizing applica-
`tion that displays icons (for example small versions of the
`images themselves) for the files in a specified folder. How-
`ever, the application need not be part of a file management
`system.In any case, the applicationidentifies the collection of
`images and displays a layout of the images. In determining
`whereto layout the imagesthe application takes into account
`the tags ofthe images. In the present example, the application
`recognizes that the first image and second imageare both
`tagged with a “park”tag, that the first image and third image
`are both tagged with a “dog”tag, andthatthefirst image and
`fourth image are both tagged with a “cute” tag. The layout
`determined for the images reflects these similarities between
`images by placing or otherwise positioning images sharing
`the sametags near one another.
`[0021] The application may also recognize relationships
`between tags within the collection or elsewhere. In the
`present example, the application may recognize that at least
`one imageis tagged with both “baby”and “cute.” The appli-
`cation may survey the entire collection (and/or other collec-
`tions or other sources) to determine how frequently “baby”
`and “cute” appear together as one way of determining
`whether those word tags are correlated. For example, a
`numerical value may be assigned or determined to represent
`the correlation between the usage of tags. As a simple
`example, in a collection of 1000 images, the tags “baby” and
`“cute” may tag the same image 20 times while the tags “baby”
`and “eating” may tag the same image only 8 times. Those
`occurrence values (e.g., 20 and 8) and/or other values that
`representthe correlation amongst tags may be usedto deter-
`
`minethe layoutoftiles. In the present example, the applica-
`tion recognizes that the first image andthe fifth image have
`some similarity even though they do not have any tags in
`common by recognizing that the fifth image’s “dog” tag is
`correlated with the “cute” tag and thusthat the fifth image has
`somesimilarity to images tagged with “cute” such as thefirst
`image.
`
`[0022] The layout application may also account for non-
`tag-based similarities amongst the images of a collection
`when determining the layout. For example, the application
`may analyze the actual content of the images and recognize
`common colors, common faces, common words, and/or any
`other suitable commonattributes. The application may also
`recognize other types of similarities amongst the images. For
`example, the application may examine metadata about the
`images such as date of image capture, date of download,etc.
`
`[0023] Each category or dimension of potential similarity
`(as examples, sharing a given tag, sharing a given tag asso-
`ciation, sharing a color, sharing a image-capture date, etc.)
`can be used to determine a layout. The application may use
`such similarities in any suitable way to determine an appro-
`priate layout including in ways that account for user prefer-
`ences and actions. The application may, for example, deter-
`mine a combinationof similarities (for example, one for each
`category/dimension associated with a given potential image
`similarity). The similarities represented may be converted to
`two dimensional space as part of the determination of posi-
`tions for the imagesof the layout. For example, multidimen-
`sional scaling or any other suitable technique may be used to
`compute low-dimensional layout positions for the images.
`Oncethe application has determined low-dimensional layout
`positions for the images, for example in two dimensional
`space, the image positions and the size of the images are
`adjusted so that the imagesto be displayed in the layout will
`not overlap.
`
`[0024] This illustrative example is given to introduce the
`reader to the general subject matter discussed herein. The
`invention is not limited to this example. The following sec-
`tions describe various additional embodiments and examples
`of methods and systems for laying out images.
`
`Illustrative Computing Environment
`
`FIGS. 1a-1are system diagramsillustrating image
`[0025]
`use and management environments according to several
`embodiments of the present invention. Other embodiments
`maybe utilized. The system 10 shown in FIG. 1a comprises
`a user computer 11 that comprises a computer-readable
`medium such as a random access memory (RAM) 13,
`coupledto a processor 12 that executes computer-executable
`program instructions stored in memory. Such a processor 12
`may comprise a microprocessor, an ASIC,a state machine, or
`other processor, and can be any of a number of computer
`processors, such as processors from Intel Corporation of
`Santa Clara, Calif. and Motorola Corporation of Schaum-
`burg, Ill. Such processor a comprise, or may be in communi-
`cation with, media, for example computer-readable media,
`whichstores instructions that, when executed by the proces-
`sor, cause the processorto perform the steps described herein.
`An alternative system 20 shown in FIG. 15 comprises a wired
`or wireless network 50 connecting user computers 21, 31 and
`a server 41. The devices 21, 31, 41 each may comprise a
`computer-readable medium such as a random access memory
`
`10
`
`10
`
`

`

`US 2013/0326338 Al
`
`Dec. 5, 2013
`
`or computer program. Other configurations may also involve
`server devices, mainframe computers, networked computers,
`and other system configurations appropriate for the particular
`context.
`
`[0030] Certain embodimentsofthe present inventionrelate
`to systems used for image management and viewing on a
`computing device. It will be recognized that this is merely one
`context for the methods and other features of the invention
`described herein. For example, certain embodiments will
`involve managementon a digital camera or video recorder
`and certain embodiments will involve other type of content,
`such as video, audio, animation, text documents, word pro-
`cessing documents, e-mail, generalfiles of any type, and any
`suitable category of documents. For example, various tech-
`niques of the invention for organizing files using tags are
`applicable in the imagefile context and in the more general
`context of any file or other object that can be tagged and
`organized using, at least in part, the tag. The techniques for
`organizing and displaying content and the other features
`described herein have uses in a variety of contexts, not to be
`limited by the specific illustrations provided herein. The sys-
`tems shown in FIGS. 1a-16 and methods described herein are
`merely illustrative and are not intended to recite any system
`componentorfeature as essential or necessary to any embodi-
`mentof the invention.
`
`Tilustrative Embodiment of Laying Out Images
`
`(RAM) 23, 33, 43, coupled to a processor 22, 32, 42 that
`executes computer-executable program instructionsstored in
`memory.
`[0026] Embodiments of computer-readable media com-
`prise, but are not limited to, an electronic, optical, magnetic,
`or other storage or transmission device capable of providing
`a processor with computer-readable instructions. Other
`examples of suitable media comprise, but are not limited to, a
`floppy disk, CD-ROM, DVD, magnetic disk, memory chip,
`ROM, RAM, an ASIC, a configured processor, all optical
`media, all magnetic tape or other magnetic media, or any
`other medium from which a computer processor can read
`instructions. Also, various other forms of computer-readable
`media may transmit or carry instructions to a computer,
`including a router, private or public network, or other trans-
`mission device or channel, both wired and wireless. The
`instructions may comprise code from any suitable computer-
`programming language, including, for example, C, C++, C#,
`Visual Basic, Java, Python, Perl, JavaScript, and Action-
`Script.
`the user computer 11 comprises an
`In FIG. 1a,
`[0027]
`image viewer application 14 that may reside in memory 13.
`The image viewer application 14, for example, may be an
`application that allows a user to organize, store,
`import,
`export, delete, tag, create, and/or view one or more images,
`amongother functions. The images themselves are typically
`graphical representations displayed on a computer screen of
`an imagefile 14a-n. An image viewing application may be a
`[0031] One embodimentofthe present invention involves a
`dedicated application designed to help a user manage and
`combination of using tags to determine how similar images
`view images, may be a file management application that
`are and using those determined similarities to derive the two
`allows a user to organizefiles and then launch or otherwise
`dimensional layout of images.
`display the contents ofthefile, whether in that application or
`[0032]
`FIG. 2 is a flow chart illustrating one method of
`through another application. An image viewing application
`laying out a collection of images according to one embodi-
`may display icons that are based or derived from the image
`ment of the present invention. For purposes of illustration
`content of an image file. For example, an image viewing
`only, the elements ofthis method are described with reference
`application maybeafile organizing application that displays
`to the system depicted in FIGS. 1a and 16. A variety of other
`imagefiles as icons of the images themselves.
`implementationsare also possible.
`[0028]
`Asillustrated in FIG. 14, methods according to the
`[0033]
`In the method 200 of laying out images shown in
`present invention need not operate within a single device. For
`FIG.2, an image viewerapplication 14 identifies a collection
`example, as illustrated in FIG. 15, users operating separate
`of images, as shownin block 210. The collection of images
`devices 21, 31 may view or receive images 44a-n stored ona
`may have been identified by the user, for example by the user
`separate device 31. A network 50, which may be any suitable
`manually selecting individual documents to be included in
`private network, public network, or the Internet, may be
`the collection, by the user selecting a folder or collection, by
`employed to facilitate the sharing of images and to allow
`the user launching an application, or in any other way. The
`different users and different locations to access, manage,
`user need not be involvedin the identification of images since
`view, tag, or otherwise use a collection of images, such as
`the viewing application may makethe identification with or
`without user interaction.
`images 44a-44n.
`[0029] Generally, the devices 11, 21,31, 41 of FIGS. la-15
`may also comprise a numberof external or internal devices
`such as a mouse, a CD-ROM, DVD,a keyboard, a display, or
`other input or output devices. Examples of such devices are
`personal computers, digital assistants, personal digital assis-
`tants, cellular phones, mobile phones, smart phones, pagers,
`digital tablets, laptop computers, Internet appliances, and
`other processor-based devices. In general, a device may
`involve any type ofprocessor-based platform that operates on
`any operating system, such as Microsoft® Windows® or
`Linux, capable of supporting one or more client application
`programs. Other applications can be contained in memory 13,
`23, 33, 43 and can comprise, for example, a word processing
`application, a spreadsheet application, an e-mail application,
`a media player application, an instant messengerapplication,
`a presentation application, an Internet browser application, a
`calendar/organizer application, and any suitable application
`
`[0034] There may be one or moretagsassociated with some
`or all of the images of the collection. Tags may have been
`automatically generated, provided by the user, or associated
`with an image in any other suitable manner. For example, if
`the identified collection is a collection of photographs, over
`time a user may have addedor associated tags with the pho-
`tographs. The user may have tagged various photographs with
`keywords, numbers, or other tags to, for example, describe
`each photo’s content and/or source. As a specific example, a
`user may have gone through a photo collection and labeledall
`pictures of his dog with a “dog”tag and a tag for the name of
`the dog—a“Fido”tag.
`[0035]
`In the method 200 of laying out images shown in
`FIG.2, the image viewer application 14 determinesa simi-
`larity of each imageof the collection to other images of the
`collection, as shown in block 220. These determinations use
`at least some of the tags associated with at least some of the
`
`11
`
`11
`
`

`

`US 2013/0326338 Al
`
`Dec. 5, 2013
`
`imagesof the collection. For example, a determination may
`involve assessing whether imagesshare tags. Such a determi-
`nation may examine two images, one tagged “dog” and
`“Fido” and the other tagged “dog” and “cat” and determine
`that the imagesare related because they are both associated
`with the “dog” tag as compared with another image with
`which neither image shares a tag, such as one tagged only
`with “water bottle.”
`
`Similarity may also be based on whether imagesuse
`[0036]
`tags that are related to one another evenif not identical, such
`as an image with a “dog” tag and an image with a “pet”tag.
`Such relatedness amongst tags may be known from the usage
`of tags together for a given image—tag co-occurrence. Tag
`relatedness may also be knownorpartially based on a data-
`base, thesaurus, and/or and other source. Generally, the simi-
`larity amongst tags may be usedto infer similarity amongst
`imagesthat use those tags. Whether based on shared tags, tag
`similarity, or otherwise, the image similarity may, but need
`not, be a semantic distance measure that quantifies the simi-
`larity amongst the images of a collection. Such distances may
`be used to create a matrix or table holding the relationships
`between every given pair of images in the collection.
`[0037]
`In the method 200 of laying out images shown in
`FIG.2, the image viewer application 14 determines a layout
`using the similarity between at least some of the images, as
`shownin block 230. The location of at least some images in
`the layout may be based atleast in part on the similarity of the
`imageto at least one other image. The layout will generally be
`either one, two, or three dimensional. Determining a layout
`may, but need not, utilize multi-dimensional scaling, for
`example to account for the similarity amongst tags and then
`use that to determine an appropriate layout. Multi-dimen-
`sional scaling (MDS) may be used on quantified similarities
`amongst the imagesof a collection, such as the values of the
`distance matrix describe above. These values represent
`semantic distance measures between any two imagesand thus
`provide parallel comparisons that need to be combined into
`one low dimensional space. MDS maythusbe used to convert
`or scale many parallel comparisonsto layout on a single one,
`two, or three low dimensional space.
`[0038] Asa specific example of using MDSto determine a
`layout, a set of 100x100 distances between the hundred
`images in a collection could be inputted to a MDSroutine.
`The MDSthen performs a numerical optimization to finda set
`of points (for example, in a two dimensional space) to assign
`to the images such that the distances are preservedas best as
`they can be. The set of points produced by an MDS may be
`scaled or otherwise adjusted to determine an image layout of
`an appropriate size for the circumstances, for example tofit
`the computer application window in whichthe images will be
`displayed. Additionally,
`the layout may be generated or
`adjusted to remove or minimize overlap between images.
`[0039] MDS may accountfor other thingsin addition to tag
`similarity. For example, MDS may also account for image
`color, metadata, GPS coordinates, camera settings, and other
`non-taggedattributes that can be relate or compared amongst
`images. For example, date of creation may be used and
`weighted above tag similarity or color.
`[0040]
`In the method 200 of laying out images shown in
`FIG.2, the image viewerapplication 14 lays out the images of
`the collection accordingto the layout determined, as shown in
`block 230. For example, the images maybe laid out within a
`window onthe user’s computer screen. As another example,
`the images maybeprinted on a printing device.
`
`Tag Co-Occurrence Determination
`
`[0041] An exemplary co-occurrence matrix C may be con-
`structed by counting the pair-wise co-occurrencesoftags ina
`set. The entry C,, contains the numberoftimesthe tags w, and
`w, appear together as annotations for the same image. A
`similarity measure for the bag of words model instatistical
`text processing is the cosine of the angle between the word
`count vectors:
`
`s(b;, bj) =
`
`(bj, bj)
`
`¥y(61, bi{bj, bj)
`
`Applied to tagged images, this measure corresponds to the
`count of shared tags between two images, normalized by the
`numberof tags in each image. By introducing the tag co-
`occurrence matrix C as a kernel matrix into the dot product, a
`simple similarity measure can be derived that takes the co-
`occurrencestatistics of tags into account.
`
`Ssem(Bj, bj) =
`
`b:Cb;
`VEC; xfb:Cb}
`
`[0042] Using this kernel, the similarity of different but
`commonly co-occurring tags is high, while it would be 0
`under the standard cosine measure. One improvementto this
`modelis to take indirect co-occurrencesinto account. Thatis,
`to treat two tags as somewhat moresimilar if both of them
`commonly co-occur with a third tag. This is particularly help-
`ful to semantically relate new tags introduced by the user to
`other pre-existing tags. For example ifthe user introduces the
`new tag “Spike” and uses it together with the “dog” tag,
`“Spike” will then be known to be somewhatsimilar to all
`things dog-related. This can be achieved by replacing the
`co-occurrence matrix C in the above equation with:
`
`where D is a diagonal matrix containing the row sums of C
`and A is a parameter controlling the trade off between the
`direct and indirect influences of tag co-occurrences. The nor-
`malization using D allowsit to choose A independently ofthe
`magnitude of the co-occurrence counts.
`
`Illustrative Embodiments of Image Overlap Removal
`and Image Scaling
`
`FIG. 3 is an exemplary image layout 300 according
`[0043]
`to one embodimentofthe present invention. In the example of
`the preceding section, an MDS may return locations for
`images in which images overlap such as in layout 300.
`Because the MDSessentially is returning a single x-y point,
`images may overlap even though differences exist.
`[0044]
`FIG. 41s an exemplary image layout without image
`overlap according to one embodimentof the present inven-
`tion. This example employs a gridding, overlap-resolution
`algorithm. In that case, imagesare alignedto a grid with grid
`elements the size of a thumbnail. Several images may occupy
`a single grid position but the grid format gives an appearance
`
`12
`
`12
`
`

`

`US 2013/0326338 Al
`
`Dec. 5, 2013
`
`of order and separation amongst the grid elements. Generally,
`thumbnail sharing a grid are related to each other because
`they were all put very, very close together by the MDSopera-
`tion. Userinterface behavior may take advantageofthis lay-
`out. For example, clicking in a single grid element mayselect
`all of the images within that grid element. Images in a single
`grid element could act as a single unit for other functions and
`purposes as may be suitable for a given application.
`[0045]
`FIGS. 5 and 6 illustrate additional layouts with
`eliminated or reduced overlap between images. FIG. 5 illus-
`trates the use of a simultaneous scaling overlap-resolution
`algorithm. Such an algorithm setsall the imagesizes to a large
`arbitrary number and then searchesfor the set of scales, one
`per image,that whenrescaled, those imageswill not overlap.
`The closer images are to other images the more scaling is
`likely to occur. With this type of algorithm,the center images
`tend to get scaled down more than those near the edges.
`[0046]
`FIG.6 illustrates the use of an ordered scaling over-
`lap-resolution algorithm or fluted scaling. Such an algorithm
`examines one imageat a time after ordering all the images in
`someorder. It takes the first image andsets that imageto be as
`big as possible without overlapping anyoneelse, but because
`it’s the first image you canset it to a preset maximum size.
`And then the algorithm takes the second imagein the order
`and scalesit to the largest size that does not cause overlapping
`and so on. The result is a kind ofa randomsizing ofthe images
`in the layout, so you get to see some of them bigger sizes and
`some smaller.
`
`FIG.7 illustrates another layout 700 without overlap
`[0047]
`between images produced according to one embodiment of
`the present invention. This example also further illustrates
`how images having similar content (shown as cross mark-
`ings) are positioned near one another while images having
`less similar content are positioned generally farther from one
`another. As shown, image 701 is nearest images 702, 703,
`704, 705 having the similar content (shown as cross marks
`going in the samedirection) but is farther from images 711,
`712, 713, 714, 721 having different content (illustrated by
`cross marks going the opposite direction or in both direc-
`tions).
`[0048] When positioned according to content (derived
`from tags, colors, metadata, or otherwise) images of a layout
`may appear grouped in waysthat facilitate use. In FIG. 7, a
`first set ofimages 701, 702, 703, 704, 705 with similar content
`(cross marks in one direction) are roughly grouped, a second
`set of images 711, 712, 713, 714 with similar content (cross
`marks in the other direction) are roughly grouped together,
`and another image 721 is somewhatseparated from both the
`first and secondset of images.
`the
`[0049] As with other examples described herein,
`images 701, 702, 703, 704, 705, 711, 712, 713, 714, 721 ofthe
`layout 700 of FIG. 7 are also positioned so that there is no
`overlap. As a specific example, certain images 701, 702, 703
`have similar or identical content but do not overlap.
`
`General
`
`[0050] The foregoing description of the embodiments