FIND THAT PHOTO!
`Interface Strategies to Annotate,
`Browse, and Share
`
`By Ben Shneiderman, Benjamin B. Bederson,
`and Steven M. Drucker
`
`A s digital photos become the
`
`standard media for personal photo taking,
`supporting users to explore those photos
`becomes a vital goal. Dominant strategies
`that have emerged involve innovative user
`interfaces that support annotation, browsing,
`and sharing that add up to rich support for
`exploratory search. Successful retrieval is
`based largely on attaching appropriate anno-
`tations to each image and collection since
`automated image content analysis is still lim-
`ited. Therefore, innovative techniques, novel
`hardware, and social strategies have been pro-
`posed. Interactive visualization to select and
`view dozens or hundreds of photos extracted
`from tens of thousands has become a popular
`strategy. And since the goal of photo search is
`to support sharing, storytelling, and remi-
`niscing, experiments with new collaborative
`strategies are being examined.
`While digital photographic databases and
`retrieval systems have been in use for many
`years, these systems were typically designed for
`
`professionals in museums, libraries, advertis-
`ing, and journalism, to name a few specíálitíes.
`Such systems employed a cadre of financially
`motivated individuals to hand-annotate the
`pictures with metadata such as keywords,
`dates, and locations, often using fixed vocabu-
`laries, to support traditional search techniques.
`By contrast, consumers typically put little
`effort into photo annotation; they are more
`focused on exploratory search and serendipi-
`tous discovery of photos with a stronger
`emphasis on entertainment. This leads to a
`very different set of requirements for personal
`photo use where ease of annotation, support
`for exploratory browsing, and convenient
`sharing is crucial.
`Annotate. In textual exploratory search,
`users can enter key phrases from a docu-
`ment to retrieve similar content. But for
`images, retrieval based on content
`through automated analysis is often
`limited to some forms of shape analysis
`(such as finding the presence of faces in
`an image) and color matching to find sunrises
`or determine whether an image was taken
`inside or outside.
`To support effective exploratory search on
`photos, appropriate annotations must be asso-
`
`COMMUNICATIONS OF THE ACM April 2006/Vol. 49, No. 4 69
`
`UNIFIED PATENTS EXHIBIT 1016
`Page 1 of 3
`
`

`

`well as making the future
`benefits more apparent.
`Automatic and manual
`annotations are valuable in
`supporting both searching
`and browsing.
`Browse. Users browse
`for fun and to find a spe-
`cific photograph. They
`may be looking for photos
`of their grandfather, their
`hike down
`the Grand
`Canyon, or a friend’s wed-
`ding. They also may be
`looking for a great photo to
`accompany a story of a sun-
`rise hike or memorable
`baseball game.
`Clearly, if the photo col-
`lection has been extensively
`annotated, techniques such
`as
`faceted
`search
`(see
`Hearst’s article in this sec-
`tion) can help users filter
`down a collection and show
`potential targets for brows-
`ing. User-controlled visualization of photos grouped
`by date, location, or annotation can greatly facilitate
`browsing and increase enjoyment [4]. Different lay-
`outs of photos can exploit this metadata to help peo-
`ple find desired photos and discover new ones. In
`particular, geo-tagging of photos and interfaces, like
`WWMX, allows people to find all those photographs
`associated with a particular area (see Figure 1).
`Chronological displays work well for dates as well,
`but large numbers of photos can be overwhelming, so
`groups of photos can be clustered by date and represen-
`tative photos can be manually or automatically chosen
`for each cluster [1, 2]. These representative photos
`again help to provide landmarks in order for users to
`locate photos from particular events. Interfaces such as
`PhotoMesa use powerful filtering tools, plus flexible
`grouping and rapid zooming, to enable users to explore
`thousands of photos fluidly (see Figure 2).
`
`Figure 1. The WorldWide Media
`Exchange (WWMX) interface
`showing map and calendar
`views along with images as
`published in ACM Multimedia
`2003; wwmx.org.
`
`ciated with the images either by the camera or by
`users of the images, such as the photographer or
`potentially a larger community of users. Cameras are
`increasingly recording information about the photo-
`graph including time and date stamps, tilt sensors for
`orientation, light levels, focal distances, and even
`global position. Barcodes, RFID tags, or other label-
`ing methods could enable a higher percentage of pho-
`
`tos to be annotated automatically.Many interfaces enable manual
`
`annotation of photographs by “painting” keywords
`[3] or dragging and dropping names onto images.
`Commercial tools such as Adobe PhotoShop Album
`make tags drag-able onto photo borders. Other tools
`perform temporal clustering to create a more man-
`ageable set of photo groups [1]. As with many tasks,
`manual annotation can be improved by designing
`interfaces that support faster and easier annotation as
`
`Consumers typically put little effort into
`photo annotation; they are more focused on
`exploratory search and serendipitous discovery
`of photos with a stronger emphasis on
`entertainment.
`
`70 April 2006/Vol. 49, No. 4 COMMUNICATIONS OF THE ACM
`
`UNIFIED PATENTS EXHIBIT 1016
`Page 2 of 3
`
`

`

`SUMMARY
`A combination of annota-
`tion, browsing, and sharing
`of photos can support the
`special exploratory search
`needs of personal digital
`photo users by getting
`around the fact that direct
`search of image content con-
`tinues to be beyond the capa-
`bilities of current systems.
`The special needs of
`amateur digital photogra-
`phers are pushing the photo
`industry to support users
`with their desired activities.
`Social networking, in com-
`bination with innovative
`user interfaces and visualiza-
`tion, is just beginning to
`support everyday photogra-
`phers. However, we see significant work remaining,
`especially in metadata standardization to help users
`cope with their rapidly growing and increasingly val-
`c
`ued collections.
`
`References
`1. Graham, A., Garcia-Molina, H., Paepcke, A., and Winograd, T. Time as
`essence for photo browsing through personal digital libraries. In Pro-
`ceedings of the 2nd ACM/IEEE-CS Joint Conference on Digital Libraries
`(2002). ACM Press, NY, 326–335.
`2. Huynh, D., Drucker, S., Baudisch, P., and Wong, C. Time Quilt: Scal-
`ing up zoomable photo browsers for large, unstructured photo collec-
`tions. In Proceedings of the SIGCHI Conference on Human Factors in
`Computing Systems (2005). ACM Press, NY, 1937–1940.
`3. Kuchinsky, A., Pering, C., Creech, M., Freeze, D., Serra, B., and Gwiz-
`dka, J. FotoFile: A consumer multimedia organization and retrieval sys-
`tem. In Proceedings of the SIGCHI Conference on Human Factors in
`Computing Systems (1999). ACM Press, NY, 496–503.
`4. Kustanowitz, J. and Shneiderman, B. Meaningful presentations of photo
`libraries: Rationale and applications of bi-level radial quantum layouts.
`In Proceedings of the 5th ACM/IEEE-CS Joint Conference on Digital
`Libraries (2005). ACM Press, NY, 188–196.
`5. van Ahn, L. and Dabbish, L. Labeling images with a computer game. In
`Proceedings of the SIGCHI Conference on Human Factors in Computing
`Systems (2004). ACM Press, NY, 319–326.
`
`Ben Shneiderman (ben@cs.umd.edu) is a professor and the
`founding director of the Human-Computer Interaction Lab,
`Computer Science Department, at the University of Maryland, College
`Park, MD.
`Benjamin B. Bederson (bederson@cs.umd.edu) is an associate
`professor and director of the Human-Computer Interaction Lab, Com-
`puter Science Department, at the University of Maryland, College
`Park, MD.
`Steven M. Drucker (sdrucker@microsoft.com) is lead researcher of
`the Next Media Research Group, Microsoft Research, Redmond, WA.
`
`© 2006 ACM 0001-0782/06/0400 $5.00
`
`COMMUNICATIONS OF THE ACM April 2006/Vol. 49, No. 4 71
`
`Figure 2. PhotoMesa showing 114
`photos in six groups in a single
`view with integrated annotation
`and search tools as published in
`ACM UIST 2001; (courtesy of
`www.photomesa.com).
`
`Share. Sharing photos
`by email, instant messag-
`ing, Web sites, and cell
`phones is a growing suc-
`cess story. When users
`select photos and make them available to others, they
`seem to be willing to invest more effort in annota-
`tion. Also by making them public, they invite others
`to comment and add annotations. More elaborate
`story-generating tools invite users to provide
`slideshow sequences with text captions and audio
`narration.
`Recent innovations in social experiences on the
`Web have sought to encourage annotation by increas-
`ing satisfaction and making the benefits immediately
`apparent. A game-like approach to image annotation
`gets players to cooperate with anonymous, remotely
`located partners in assigning keywords for pho-
`tographs [5]. This surprisingly addictive game has
`succeeded in labeling over 10 million images as of
`August 2005 (since its introduction in 2003). Other
`communities, such as Flickr, allow users to share and
`annotate images on a Web site using tags. These
`“folksonomies” have now gone past photos to Web
`pages and blogs as well (such as technorati and
`deli.cio.us).
`The trend toward annotating, browsing, and sharing
`your photos via Web sites such as Flickr, Ofoto, and
`Shutterfly is perhaps one of the biggest changes enabled
`by the transformation from analog to digital photogra-
`phy. Photos no longer sit unattended in shoeboxes
`stored in attics, but are available for ready viewing by
`friends and family distributed around the world.
`
`UNIFIED PATENTS EXHIBIT 1016
`Page 3 of 3
`
`