INTERNATIONAL JOURNAL OF HUMAN–COMPUTER INTERACTION, 23(3), 315–337
`Copyright © 2007, Lawrence Erlbaum Associates, Inc.
`
`
`
`
`
`HIHC1044-73181532-7590International journal of Human-Computer Interaction, Vol. 23, No. 3, Oct 2007: pp. 0–0International journal of Human–Computer Interaction
`
`
`
`
`
`Capture, Annotate, Browse, Find, Share:
`Novel Interfaces for Personal Photo Management
`
`
`
`Interfaces for Personal Photo ManagementKang, Bederson, Suh
`
`Hyunmo Kang
`Human-Computer Interaction Laboratory,
`University of Maryland Institute for Advanced Computer Studies, College Park, MD
`Benjamin B. Bederson
`Human-Computer Interaction Laboratory,
`Computer Science Department, University of Maryland
`Institute for Advanced Computer Studies, College Park, MD
`Bongwon Suh
`Palo Alto Research Center, Palo Alto,CA
`
`The vision of ubiquitous digital photos has arrived. Yet, despite their broad popular-
`ity, significant shortfalls remain in the tools used to manage them. We believe that
`with a bit more creativity and effort, the photo industry can solve many of these prob-
`lems, offering tools which better support accurate, rapid, and safe shared annotations
`with comfortable and efficient browsing and search. In this article, we review a num-
`ber of projects of ours and others on interfaces for photo management. We describe
`the problems that we see in existing tools and our vision for improving them.
`
`1. INTRODUCTION
`
`The days when people debated the relative merits of film versus digital imagery
`now seem almost quaint. And with hindsight, film seems destined to have been a
`chapter in history along with LP vinyl records—a temporary physical analog
`recording medium. Although some aficionados may still prefer some qualities of
`those dust-gathering mediums, the advantages of digital media have become
`clear. The rapid and inexpensive ability to edit, annotate, search, share, and access
`has brought digital media to ubiquity.
`And yet, with all this promise, shortfalls remain in the overall user experience.
`How many of us have replaced old unlooked-at shoeboxes of prints with unlooked-at
`digital archives of image files? How much has our ability to find a particular set of
`
`Color figures are available at http://www.cs.umd.edu/hcil/ben60
`Correspondence should be addressed to Hyunmo Kang, AVW 3211, University of Maryland
`Institute for Advanced Computer Studies, University of Maryland, College Park, MD 20742.
`E-mail: kang@cs.umd.edu
`
`UNIFIED PATENTS EXHIBIT 1009
`Page 1 of 68
`
`MemoryWeb Ex. 2002
`Apple v. MemoryWeb – IPR2022-00032
`1 of 68
`
`

`

`316
`
`Kang, Bederson, Suh
`
`photos really improved (i.e., can you find those photos of a visiting uncle with your
`sister when they were children?) How do we record stories from our parents describ-
`ing family photos? How do we make sure those stories stay with the photo in ques-
`tion and get distributed to all copies of that photo within the family? Perhaps of most
`concern, how do we ensure that these annotations stand the test of time and remain
`accessible as computers, file formats, recording mediums, and software change?
`These changes are all happening within the context of human behavior, which
`does not change so rapidly. People still like immediate gratification: take pictures
`rapidly, print them, and share in social settings. Some people spend a lot of effort
`creating photo albums or “scrapbooking.” Of course, many do not. Understand-
`ing which behaviors are fundamental and which are side effects of current tech-
`nology is crucial, because this understanding can and should influence where
`researchers spend their effort.
`We explore these issues and more with much of the intellectual motivation
`coming from Ben Shneiderman, our close colleague who has pushed for a deeper
`understanding of and better support of photo management for more than 10
`years. His personal photo archives document the field of human–computer inter-
`action (HCI) going back to its beginning. He regularly shares those photos with
`great enthusiasm to visitors, motivating and exciting all of us—largely because of
`the care and consistency he has applied to annotating and organizing his photos.
`He regularly pulls up old photos of lab visitors showing everyone what they
`worked on 5 or 15 years ago (and of course, showing what they looked like too!).
`His early exploration of tools to support photo management (with co-author
`Kang) led to PhotoFinder (Kang & Shneiderman, 2000), and the ensuing
`PhotoMesa tools (Bederson, 2001; Bederson, Shneiderman, & Wattenberg, 2002).
`His personal interest helped inspire the authors of this article as well as other lab
`members to pursue the development of approaches and software to improve all
`of our user experiences when managing photos.
`This, of course, all happened during a time of tremendous commercial activity
`in this area. There are wildly popular photo sharing Web sites, such as Flickr
`(http://www.flickr.com; Yahoo!), Picasa Web Albums (http://picasaweb.google.
`com; Google), Snapfish (http://www.snapfish.com; HP), Shutterfly (http://
`www.shutterfly.com; Shutterfly), and PhotoBucket (http://www.photobucket.
`com; Photobucket), as well as equally well-used desktop photo applications such
`as Google Picasa (http://www.picasa.com), Adobe PhotoShop Album (http://
`www.adobe.com/products/photoshopalbum/starter.html), and ACDSee (http:/
`/www.acdsee.com). The two approaches (desktop application and Web site) are
`interesting to look at because they each offer distinct advantages to users. For
`example, Web sites are available anywhere and facilitate sharing, whereas desk-
`top applications are faster, support higher resolution photos more easily, provide
`local ownership of photos, and offer richer interaction capabilities. It is interesting
`that each approach is gaining characteristics of the other. Web applications begin
`to offer dynamic, interactive content rather than static html pages through AJAX
`and Flash technologies. In addition, they often include plug-ins to ease uploading
`or improve performance, and some offer APIs to enable desktop applications to
`access their data directly. At the same time, many desktop applications are offer-
`ing Web capabilities such as sharing.
`
`UNIFIED PATENTS EXHIBIT 1009
`Page 2 of 68
`
`MemoryWeb Ex. 2002
`Apple v. MemoryWeb – IPR2022-00032
`2 of 68
`
`

`

`Interfaces for Personal Photo Management
`
`317
`
`Yet even with this commercial activity, the full potential of personal photo
`management has not been reached. There is the opportunity for richer annotation
`interfaces, automated content analysis, improved sharing, and more creative
`organizational strategies. Our hope is that more photos end up with better meta-
`data, enabling faster, easier, and more accurate and enjoyable retrieval and use.
`In this article, we look at some of the key activities and behavior patterns of
`personal photo users and examine how innovative user interfaces have the poten-
`tial to enhance users’ power, satisfaction, and control in managing and exploring
`their images. Starting with a close look at annotation, we examine how a combi-
`nation of manual and automated techniques can improve how people associate
`metadata with photos. We then look at how the resulting richer metadata can
`enable better interfaces for searching and browsing photos. Finally, we end with a
`discussion about the importance of sharing photos and how new interfaces enable
`that.
`
`2. GUIs FOR ANNOTATION
`
`An essential question is, How valuable is photo metadata? Our own assessment
`of user needs (Shneiderman & Kang, 2000) coupled with reports from other
`researchers (Chalfen, 1987; Naaman, 2005; Rodden & Wood, 2003), and our per-
`sonal experience come together on this. They indicate that the photo metadata
`such as dates, locations, and content of the photos (especially people’s names)
`play a crucial role in management and retrieval of personal photo collections.
`However, in spite of the high utility of the photo metadata, the usability of soft-
`ware for acquiring and managing the metadata has been poor. The manual photo
`annotation techniques typically supported by photo software are time-consuming,
`tedious, and error prone, and users typically put little effort into annotating their
`photos. In fact, the industry attitude tends to be that because users do not anno-
`tate their photos very much, it is not necessary to spend much energy adding
`good support for it. However the success of keyword labeling, so-called tagging,
`systems such as del.icio.us (http://www.del.icio.us ) and Flickr hints that users
`indeed want to make annotations when reasonable utility and usability are sup-
`ported. We believe that the photo annotation has enough utility for some users
`and it is the usability of software that needs to be improved.
`In this section, a few innovative approaches are presented to show how inter-
`action and graphical user interface (GUI) design can improve the usability of the
`photo annotation process when they are based on careful understanding of users’
`behavior and usage pattern. In addition, we explain how those designs have been
`evolving over time to support a broader range of annotation tasks by combining
`the accessible technologies with the analysis of users and their needs.
`
`2.1. Advanced Manual Annotation: Direct Annotation
`
`Because annotations based on automated image content analysis are still limited,
`we developed an advanced manual annotation mechanism that can significantly
`reduce users’ annotation workload under certain circumstances. From the
`
`UNIFIED PATENTS EXHIBIT 1009
`Page 3 of 68
`
`MemoryWeb Ex. 2002
`Apple v. MemoryWeb – IPR2022-00032
`3 of 68
`
`

`

`318
`
`Kang, Bederson, Suh
`
`observations of personal photo annotations (Shneiderman & Kang, 2000), we
`found that there were three interesting characteristics that could be useful for our
`interaction design.
`
`• Personal photo libraries often contain many images of the same people at
`different events. In the libraries we looked at, we typically found 100 to 200
`identifiable people in several thousand photos. Furthermore, the library has
`a highly skewed distribution with immediate family members and close
`friends appearing very frequently.
`• Textual search often doesn’t work reliably because of inconsistency in names
`with misspellings or variants (e.g., Bill, Billy, William).
`• Lists of names of people appearing in photos are often difficult to associate
`with individuals, especially in group shots. Textual captions often indicate
`left-to-right ordering in front and back rows, or give even more specific identi-
`fication of who is where. However this approach is tedious and error prone.
`
`Based on these observations, we collaborated with Ben Shneiderman to develop
`the concept of direct annotation (U.S. Patent #7010751), which is a technique using
`selectable, draggable labels that can be placed directly on the photo (Shneiderman
`& Kang, 2000). Similar interfaces have also appeared recently on Flickr and
`MySpace. Users can select from a scrolling or pop-up list and drag by mouse or
`touch screen. This applies direct manipulation principles (Shneiderman, 1983,
`2005) that avoid the use of a keyboard, except to enter a name the first time it
`appears. The name labels can be moved or hidden, and their presence is recorded
`in the database or in the header of an image file in a resolution-independent man-
`ner. The relative location of the target is stored based on an origin in the upper
`left-hand corner of the photo with the point in the range (0, 0) – (1.0, 1.0) corre-
`sponding to the full image. This approach not only associates a name with a posi-
`tion on the photo but also ensures that each name is always spelled the same way.
`This simple rapid process also enables users to annotate at any time. They can
`add annotations when they first see their photos on the screen, when they review
`them and make selections, or when they are showing them to others. This design,
`which supports continuous annotation encouraged users to do more annotation,
`especially in collaborative situations such as with PhotoFinder Kiosk (Kules,
`Kang, Plaisant, Rose, & Shneiderman, 2004; Shneiderman et al., 2002). In a public
`setting, visitors of PhotoFinder Kiosk added 1,335 name labels using direct anno-
`tation while adding 399 captions using traditional type-in method.
`The direct annotation mechanism was revised later so that it enables users to
`define their own categories such as activities, events, locations, objects in a photo
`in a hierarchical way (Figure 1) in addition to person names. A few alternative
`and complementary direct annotation mechanisms such as split menu annotation,
`hotkey annotation, and pop-up menu annotation have also been designed and
`developed to accelerate the annotation process. These are described in more detail
`in the following subsection.
`A pilot experiment was conducted to see if the direct annotation method
`improved the annotation process in terms of annotation time and users’ subjective
`preference compared with the traditional caption method or the click-and-type
`
`UNIFIED PATENTS EXHIBIT 1009
`Page 4 of 68
`
`MemoryWeb Ex. 2002
`Apple v. MemoryWeb – IPR2022-00032
`4 of 68
`
`

`

`Interfaces for Personal Photo Management
`
`319
`
`FIGURE 1 The revised direct annotation mechanisms as implemented in PhotoMesa,
`the successor to PhotoFinder. Note. Users can add a caption under a photo, or add a
`particular attribute such as favorite (a yellow star on the bottom left of photo) or hid-
`den. In addition, labels can be dragged from the list of people (on the top left) or from
`the user-defined category tree (on the bottom left). A label can be directly placed on the
`photo to represent where the individuals or objects are located within the photo.
`
`method (Goto, Jung, Ma, & McCaslin, 2000; Jung, 2000). Forty-eight volunteers
`participated in a within-subject design, whereby each participant attempted an
`annotation task (20 names in five photos) on each system. The direct annotation
`method was significantly preferred, but no significant difference was found in the
`mean annotation time. A more extensive user study might help identify under
`what circumstances (e.g., number of total labels, average number of people in a
`picture, pattern of people’s appearance in a personal photo collection, etc.) differ-
`ent annotation mechanisms work better than others in terms of completion time,
`error rate, users’ satisfaction, or confidence.
`
`2.2. Enhanced Direct Annotation: Bulk Annotation
`
`The direct annotation mechanism was enhanced through a series of design cycles
`to support more efficient and productive annotation. Perhaps the most notable
`
`UNIFIED PATENTS EXHIBIT 1009
`Page 5 of 68
`
`MemoryWeb Ex. 2002
`Apple v. MemoryWeb – IPR2022-00032
`5 of 68
`
`

`

`320
`
`Kang, Bederson, Suh
`
`one was applying the bulk annotation technique, which lets users annotate several
`photos with a single interaction (Kuchinsky et al., 1999). Bulk annotation is espe-
`cially useful when users have a set of photos with the same person or group of
`people involved in the same events. We thus designed and developed several
`variations of direct annotation and bulk annotation mechanisms to accelerate the
`annotation process.
`The split menu annotation mechanism (Figure 2a) was designed to minimize
`users’ time of scrolling the list to find correct labels to be used for photo annotation.
`The split menu featured a resizable splitter bar so that the number of the most fre-
`quent labels displayed in the top window (Sears & Shneiderman, 1994). The scrollbar
`was removed from the top window, whereas the bottom half retained its scrollbar.
`The split menu raises interesting questions about what kind of automatic algorithms
`should be used to predict users’ future access and facilitate rapid annotation.
`Because some annotations get used so much more frequently than others, we
`designed the interface so that each label (either person’s name or categories) can
`be associated with a hotkey (Figure 2b). After the key is assigned to a particular
`label, users can press that key whenever the mouse is over a photo. At that point,
`the spot that the cursor was over will be annotated with the specified label. When
`the expert user has a good idea about who or what is to be annotated, then the
`hotkeys can be put to work very efficiently. Instead of having to find the name
`desired and drag the name over to the position, the user can simply position the
`mouse and press the hotkey. This method is especially useful when a photo
`collection has only a few names that appear frequently.
`Pop-up menu annotation (Figure 2c) also aims at reducing mouse movement in
`selecting labels to be used for annotation. This method offers a menu when the
`right mouse button is pressed on a picture. The menu consists of the currently
`selected labels in the list so that users can annotate pictures with a label in the
`menu at the position the right mouse button was pressed. This mechanism was
`revised again later as Label Paint Annotation so that a photo can be annotated
`with the currently selected labels whenever users click on a photo without select-
`ing a pop-up menu.
`In addition to the proposed methods for improving the speed in annotating
`individual photos, we also designed two additional methods for improving the
`efficiency of bulk annotation as follows:
`
`• Checkbox Annotation: Select one or more photos and click the check box
`next to the label in the list (Figure 1).
`• Drag-Drop Annotation: Select one or more photos and drag and drop the
`selected labels onto the photo. All the photos that were selected get anno-
`tated with the labels.
`
`The best solution may be to include as many methods as possible while providing
`options for users to select and customize the methods. However it raises the ques-
`tion of how much the user can learn at first; if presented by too many options in
`the beginning, the user may become confused and frustrated. Therefore it may be
`optimal to use the multilayered approach (Kang, Plaisant, & Shneiderman, 2003,
`Shneiderman, 2003) when designing the initial interface.
`
`UNIFIED PATENTS EXHIBIT 1009
`Page 6 of 68
`
`MemoryWeb Ex. 2002
`Apple v. MemoryWeb – IPR2022-00032
`6 of 68
`
`

`

`Interfaces for Personal Photo Management
`
`321
`
`(a) Split menu annotation
`
`(b) Hotkey annotation
`
`(c) Pop-up menu annotation
`
`FIGURE 2 Enhanced direct annotation approaches within PhotoFinder.
`
`UNIFIED PATENTS EXHIBIT 1009
`Page 7 of 68
`
`MemoryWeb Ex. 2002
`Apple v. MemoryWeb – IPR2022-00032
`7 of 68
`
`

`

`322
`
`Kang, Bederson, Suh
`
`2.3. Semiautomatic Annotation
`
`The performance of direct annotation may be improved by supporting bulk anno-
`tation as described in the previous section. However this approach introduces
`two new challenges. First, unless photos to be annotated with the same labels are
`clustered together in some way so that they can be selected together, users need to
`manually select photos one by one before applying bulk annotation. Second,
`because multiple photos are annotated with a single interaction, the position of
`the labels in each photo cannot be explicitly specified by users.
`To cope with these challenges, we explored a semiautomatic annotation
`strategy that takes advantage of human and computer strengths (Suh &
`Bederson, 2007). The semiautomatic approach enables users to efficiently
`update automatically obtained metadata interactively and incrementally.
`Even though automatically identified metadata are compromised with inaccu-
`rate recognition errors, the process of correcting inaccurate information can be
`faster and easier than manually adding new metadata from scratch. To facili-
`tate efficient bulk annotation, two clustering algorithms were introduced to
`generate meaningful photo clusters (Suh & Bederson, 2007): hierarchical event
`clustering and clothing-based person recognition. The first method clusters photos
`based on their time stamps so that each group of photos corresponds to a
`meaningful event. The hierarchical event clustering identifies multiple event
`levels in personal photo collection and allows users to choose the right event
`granularity. The second method uses a clothing-based human model to group
`similar-looking people together. The clothing-based human model is based on
`the assumption that people who wear similar clothing and appear in photos
`taken within relatively short periods of time are very likely to be the same
`person.
`To explore our semiautomatic strategies, we designed and implemented a
`prototype called SAPHARI (Semi-Automatic PHoto Annotation and Recognition
`Interface). The prototype provides an annotation framework focusing on making
`bulk annotations on automatically identified photo groups. The two automatic
`clustering algorithms provide meaningful clusters to users and make annotation
`more usable than relying solely on vision techniques such as face recognition or
`human identification in photos. In addition, the clothing-based person recogni-
`tion automatically detects the positions of people in a photo, which can be used
`to position annotations. It is interesting that, although we found that absolute
`performance using this approach was not much better than manual annotation,
`the user study showed that the semiautomatic annotation was significantly pre-
`ferred over manual annotation (Suh & Bederson, 2007). As computer vision tech-
`niques improve, the relative advantage of semiautomated techniques can only
`improve.
`Figure 3 shows a prototype of SAPHARI, which shows the clusters of identi-
`fied people whose upper bodies are cropped from photos of the same event.
`Because the automatic clustering techniques can also have errors, we designed the
`GUI so that users can correct errors manually by dragging a photo into the correct
`group. After error correction, users can annotate multiple photos at once (also
`with position information) by dragging a label onto the group.
`
`UNIFIED PATENTS EXHIBIT 1009
`Page 8 of 68
`
`MemoryWeb Ex. 2002
`Apple v. MemoryWeb – IPR2022-00032
`8 of 68
`
`

`

`Interfaces for Personal Photo Management
`
`323
`
`FIGURE 3 Identified people who are cropped from photos are laid out on the
`screen in the SAPHARI prototype. Note. Similar looking people are clustered based
`on their clothing, facilitating bulk annotation.
`
`2.4. Image Annotation Discussion
`
`There has been significant research on how to acquire useful metadata for images.
`Many researchers have focused on automatic extraction of metadata by under-
`standing images. For example, researchers have focused on automatic object
`detection such as face recognition and content-based categorization and retrieval
`(Yang, Kriegman, & Ahuja, 2002; Yoshitaka & Ichikawa, 1999; Zhao, Chellappa,
`Philips, & Rosenfeld, 2003; Chellappa, Wilson & Sirohey, 1995). However, such
`automatic techniques have achieved limited success so far when applied to per-
`sonal photo management (Phillips et al., 2003; Suh & Bederson, 2007).
`Rather than pure image-based approaches, some researchers used related
`context information to identify relevant metadata. For example, Naaman, Yeh,
`Garcia-Molina, and Paepcke (2005) used time and location information to gener-
`ate label suggestions to identify the people in photos. Lieberman and Liu (2002)
`used relationships between text and photographs to semiautomatically annotate
`pictures.
`On the other hand, Web-based collaborative approaches to collecting metadata
`become popular. Web pages, online photographs, and Web links have been
`actively annotated with rich metadata through tagging (e.g., Flickr and
`Del.icio.us). Tagging is a simple kind of annotation where keywords can be
`
`UNIFIED PATENTS EXHIBIT 1009
`Page 9 of 68
`
`MemoryWeb Ex. 2002
`Apple v. MemoryWeb – IPR2022-00032
`9 of 68
`
`

`

`324
`
`Kang, Bederson, Suh
`
`associated with objects such as photos. However, the simple underlying approach
`belies the richness and power gained by broad communal access to these tags
`which enable cross-collection aggregation and searching along with innovative
`interfaces for creating and presenting those tags (i.e., tag clouds). With Flickr,
`users can share personal photos with friends, family, and colleagues, allowing the
`invited users to make annotations on shared photos. When users select photos
`and make them available to others, they seem to be willing to invest more effort in
`annotation (Shneiderman, Bederson, & Drucker, 2006). Also, by making them
`public, they invite others to comment and add annotations. We believe that this
`“folksonomy” based collaborative annotation has great potential for creating
`more useful metadata.
`On the other hand, folksonomy based tagging system has a set of limitations.
`Folksonomic tagging can be inconsistent because of its vocabulary problem
`(Furnas, Landauer, Gomez, & Dumais, 1987). In folksonomy-based systems, there
`is no bona fide standard for selecting a tag, and users can choose to use any word
`as a tag. For example, one can use the tag TV, whereas others choose to use the tag
`television. Furthermore, Furnas et al. showed that it is not always easy for users to
`come up with good descriptive keywords that can be shared. Tags, therefore, can
`be easily idiosyncratic and often causes meta-noise, which decreases the retrieval
`efficiency of systems. Going further with collaborative annotation approaches,
`Von Ahn addressed the challenge by adding gaming to the mix. The ESP Game
`(Von Ahn & Dabbish, 2004) combines a leisure game with practical goals. It lets
`people play an image guessing game that gathers image metadata as a side effect
`of their playing.
`Another ongoing challenge for metadata management is where to store the
`metadata. It is crucial that users own their metadata just as much as they own
`their photos. Yet some companies (such as Apple with their iPhoto software) cre-
`ate a custom database that separates metadata from the photos, leaving no easy
`way to get the annotations back for sharing or for use in other programs. Software
`developers tend to like centralized metadata stores because they make it easier to
`write software that perform fast searches (and because they “lock in” customers
`to their products). However this approach is not necessary. Instead, we suggest
`storing all annotations and metadata using the “IPTC” standard format (Interna-
`tional Press Telecommunications Council; http://www.iptc.org ) in the “EXIF”
`header (Japan Electronic and Information Technology Industries Association,
`2002) of JPEG images. Then the application can create a cached copy of the meta-
`data for efficient operations while leaving the “original” metadata with the photo.
`This is how we implemented PhotoMesa, and it appears to be the same approach
`taken by some other commercial software such as Picasa and ACDSee. However,
`even this approach has some challenges because of the following limitations:
`
`• Not every image format supports metadata fields in the file headers.
`• Metadata standards (i.e., EXIF/IPTC) do not have rich enough fields for
`objects such as people and hierarchical categories.
`• Some operating systems make it difficult and possibly dangerous to modify
`the image header without changing the file modification date (which users
`sometimes use to determine when a file was created).
`
`UNIFIED PATENTS EXHIBIT 1009
`Page 10 of 68
`
`MemoryWeb Ex. 2002
`Apple v. MemoryWeb – IPR2022-00032
`10 of 68
`
`

`

`Interfaces for Personal Photo Management
`
`325
`
`Although the issues and challenges relating to metadata standards are significant,
`they are beyond the scope of this article. We thus raised these few issues to point
`out the importance of getting it right, as it has such a dramatic affect on user expe-
`rience. We therefore call on the industry to collaborate more openly in developing
`consistent and rich photo metadata standards addressing at least the aforemen-
`tioned issues.
`Technological change in society is rapid. There are new opportunities that we
`do not explore in this article. For example, the growing number of cell phones and
`even dedicated cameras that offer voice recording and network connectivity offer
`new possibilities such as supporting annotation at the moment the photo is cap-
`tured. On the other hand, collaborative tagging has shown users are willing to
`make annotations when certain conditions are met. Further research is needed to
`investigate how the approaches we discuss in this article apply to those new
`settings.
`Even though the importance of photo metadata is well known, the develop-
`ment of systems to support annotation has not been so successful. That is partly
`because of the limitations of automatic content analysis approaches, and partly
`because software designers have not understood in which situations users will be
`ready to spend a substantial amount of energy annotating photos. In this section,
`we have illustrated how novel interface designs might facilitate photo annotation
`by individuals or communities of users.
`
`GUIs for using metadata in image management and exploration
`Once richer image metadata is available either by automatic image content
`analysis or by manual annotation, users can make use of it for various image
`management tasks including not only search but also organization, meaning
`extraction, navigation, and distribution.
`From our research experience with personal media management systems
`(Bederson, 2001; Bederson et al., 2002; Kang & Shneiderman, 2000, 2002, 2006;
`Khella & Bederson, 2004; Kules et al., 2004; Shneiderman et al., 2006; Shneiderman &
`Kang, 2000; Shneiderman et al., 2002; Suh & Bederson, 2007), we learned that
`managing personal media objects is a challenge for most users, who may struggle
`to understand, interpret, arrange, and use them. They wrestle with at least two
`major problems. First, most tools were designed and developed based on rigid
`and system-driven organizing metaphors (such as file-folder hierarchy). Second,
`those tools were not suitably designed to use metadata in exploring the media
`objects—beyond merely searching them. To address these challenges, we
`designed novel GUIs that hope to improve task performance as well as user satis-
`faction in exploring and managing personal media data.
`
`3.1. Organization
`
`With rich image metadata, even if users can find the images they need, they still
`frequently want to organize them for other reasons such as supporting future ser-
`endipitous browsing or to have the satisfaction of putting their images in order
`
`UNIFIED PATENTS EXHIBIT 1009
`Page 11 of 68
`
`MemoryWeb Ex. 2002
`Apple v. MemoryWeb – IPR2022-00032
`11 of 68
`
`

`

`326
`
`Kang, Bederson, Suh
`
`(Balabanovic, Chu & Wolff, 2000; Kuchinsky et al., 1999; Rodden & Wood, 2003).
`Hence, supporting image organization is important.
`One of the main challenges in designing a novel user interface for organizing
`images is how to let end-users represent and flexibly apply their conceptual mod-
`els to their image collections. Users typically understand their data by construct-
`ing conceptual models in their minds. There is no unique or right model. Rather,
`the models are personal, have meaning for the individual who creates them, and
`are tied to specific tasks. Even in a simple personal photo library, images can be
`organized by time lines, locations, events, people, or other attributes, depending
`on users’ conceptual models and specific tasks. Despite the diversity of users’
`conceptual models, the means available for users to organize and customize their
`information spaces are typically poor and driven mostly by storage and distribu-
`tion models, not by users’ needs.
`Ben Shneiderman and Hyunmo Kang tried to resolve this challenge by provid-
`ing users an environment to customize their information space appropriately for
`their conceptual models and specific tasks. We introduced a model called Seman-
`tic Regions (Kang & Shneiderman, 2006; see Figure 4), which are query regions
`drawn directly on a 2D information space. Users can specify the shapes, sizes, and
`positions of the regions and thus form the layout of the regions meaningful to
`them. Semantic Regions are spatially positioned and grouped on the 2D space
`based on personally defined clusters or well-known display representations such
`as a map, tree, time line, or organization chart. Once the regions are created and
`
`Graduate School Friends
`
`UMD Friends
`
`UMD CS
`Friends
`
`Highschool
`Friends
`
`College Friends
`
`FIGURE 4 A friend group conceptual model: Each region represents a person and
`contains all the photos annotated with the name defined in it. Note. The regions are
`grouped into five clusters to represent different friend groups (UMD friends—
`University of Maryland computer science students-, high school friends, graduate
`school friends, college friends, and UMCP friends–University of Maryland noncom-
`puter science students). Each group has its own color to represent the different group
`of friends.
`
`UNIFIED PATENTS EXHIBIT 1009
`Page 12