C O V E R F E A T U R E
`
`Expanding the
`Digital Camera’s
`Reach
`
`Chandra
`Narayanaswami
`M.T.
`Raghunath
`IBM T.J. Watson
`Research Center
`
`Today’s digital cameras and large-capacity portable storage devices could
`soon be integrated into compact cell phones that establish symbiotic
`relationships with stationary devices in the environment, providing users
`with the ability to view and share images in many new settings and
`enabling the creation of several novel applications.
`
`O ver the past decade, consumer digital cam-
`
`eras have evolved from expensive and
`bulky devices that provide low-resolution
`images to affordable compact units capa-
`ble of recording high-resolution pictures.
`Recently, cell phones that integrate digital cameras
`have far outsold regular digital cameras. We expect
`this trend to continue because this combination
`provides great value.
`First, the cell phone’s voice communication capa-
`bility makes it the most ubiquitous portable device.
`Second, people enjoy the convenience of capturing
`high-resolution digital images using a device they
`already carry. Third, this integration relieves people
`from having to make a conscious decision to take a
`camera in anticipation of taking pictures. Some dig-
`ital cameras even offer integrated Wi-Fi capabilities
`for direct image transfer. Recent trends in portable
`storage devices indicate that cellular camera phones
`can integrate several gigabytes of storage.
`More than a straightforward replacement for film
`cameras, digital cameras allow new and different
`uses. People tend to take more pictures in more sit-
`uations with digital cameras than they do with film
`cameras. For example, because digital images
`require little physical space, some users have con-
`verted their children’s space-consuming art projects
`into compact yet accessible digital albums. Others
`have recorded a skin rash’s progress or a dog bite so
`that they can show their doctor the digital image.
`Digital cameras have also become memory aids
`
`and transcription devices, creating a class of ephem-
`eral images with shorter lifetimes than images taken
`with film cameras. People take pictures of where they
`parked their car at the airport so that they needn’t
`spend time searching the lot when they return.
`Others have captured the license plate number of
`erratic drivers and reported them to authorities.
`Digital cameras have become input devices as well.1,2
`More applications like these will appear as high
`portability, minimal per-picture cost, and generous
`storage increase the digital camera’s popularity.
`Despite the increasing image resolution, storage
`capacity, and wireless connectivity of cellular cam-
`era phones, human preferences dictate their physi-
`cal size, which essentially limits the integrated
`display’s size. The advent of flexible displays could
`change this balance, but the size available in a truly
`portable form factor will limit the rolled-up display’s
`area. Another alternative, projection technologies,
`consume too much power at the present time to be
`practical for most portable devices. Therefore, for
`the next few years, the camera’s display size will
`remain constrained while the number of images that
`can be stored in the camera itself will incease dra-
`matically. Although immediately viewing pictures
`on the integrated display is a valuable feature, it can
`show only a limited amount of detail.
`To address this limitation, we recently described
`a futuristic scenario in which mobile computers
`establish symbiotic relationships with stationary
`devices in the environment to offer users a combi-
`
`0018-9162/04/$20.00 © 2004 IEEE
`
`P u b l i s h e d b y t h e I E E E C o m p u t e r S o c i e t y
`
`December 2004
`
`65
`
`IPR2017-02056, Petitioner Google Inc.
`Ex. 1013, p. 65
`
`

`

`Table 1. Typical display and maximum possible resolution according to human visual acuity limitations.
`
`Display type
`
`User distance
`(inches)
`
`Typical width
`(inches)
`
`Typical width
`(pixels)
`
`Typical
`resolution
`(dpi)
`
`Maximum
`resolution
`(dpi)
`
`Maximum
`width
`(pixels)
`
`Cell phone panel
`PDA display
`Laptop display
`Desktop monitor
`Meeting room screen
`
`10
`12
`16
`20
`230
`
`1.5
`2
`10
`20
`80
`
`150
`300
`1,200
`2,000
`1,200
`
`100
`150
`120
`100
`15
`
`350
`291
`218
`175
`15
`
`525
`582
`2,180
`3,500
`1,200
`
`Table 2. Some examples of camera specifications evolution, 1995-2003.
`
`Camera
`
`Ricoh RDC-1
`Kodak DC210
`Kodak DC290
`Sony DSC-P10
`Samsung
`VGA1000 cell phone
`
`Year
`
`1995
`1997
`1999
`2003
`2003
`
`Introductory
`retail
`price
`
`$1,800
`$600
`$800
`$600
`$299
`
`Image
`resolution
`
`Typical
`image size
`
`768 x 480
`1,152 x 864
`1,792 x 1,200
`2,592 x 1,944
`640 x 480
`
`100 Kbytes
`300 to 400 Kbytes
`600 to 800 Kbytes
`2 Mbytes
`100 Kbytes
`
`Display
`size
`(mm)
`
`51 x 31
`37 x 27
`41 x 28
`31 x 23
`34 x 42
`
`Pixels
`
`72 K
`
`123 K
`20.4 K
`
`Maximum
`memory
`available
`
`24 Mbytes
`64 Mbytes
`128 Mbytes
`2 Gbytes
`
`Image
`capacity
`
`240
`160
`160
`1,000
`
`nation of both systems’ best attributes: large, easy-
`to-read, high-quality displays and content person-
`alization through mobile computers.3
`In this scenario, environmental displays become
`intelligent network objects that offer their services
`the same way today’s network printers do. Mobile
`computers discover such displays, communicate
`with them to ascertain their characteristics, and
`securely transmit information.
`
`SYMBIOTIC DISPLAYS IN THE ENVIRONMENT
`Many researchers have explored the idea of using
`environmental displays to supplement portable
`devices’ display capabilities.3,4 They base this work
`on the key observation that every display has intrin-
`sic limitations that create usage barriers. For exam-
`ple, human visual acuity imposes an upper bound
`on display resolution. Even people with perfect
`vision cannot resolve details smaller than one
`minute of visual arc angle for prolonged durations
`at comfortable brightness levels. Thus, increasing
`display resolution beyond that point does not con-
`tribute significantly to improvements in the dis-
`played information’s readability.
`Table 13 shows the typical resolution of current
`displays and the maximum meaningful resolution
`as a function of display size. The table shows that
`technological advances will not improve portable
`displays enough to make viewing a large amount
`of detail easy. The improvements in picture quality
`resulting from better capture resolutions are sel-
`dom observable on a digital camera’s integrated
`display. To provide an extended set of applications,
`
`we expect future digital cameras to establish on-
`demand symbiotic relationships with large envi-
`ronmental displays.
`In addition to cameras, we expect several other
`mobile devices to leverage the services of network-
`attached displays to view documents and other
`types of content that cannot be viewed easily on
`small displays. We believe that a new class of intel-
`ligent network-connected displays will emerge to
`support such symbiotic relationships. These displays
`will have some or all of the following attributes:
`
`• direct connection to the network infrastruc-
`ture and network addressability;
`• support for content formats such as ASCII,
`HTML, PDF, PostScript, JPEG, GIF, MPEG,
`and Flash, plus the ability to negotiate with
`other network devices, much as browsers
`express accept tags to Web servers;
`• the ability to express their capabilities to
`mobile devices in terms of pixel resolution and
`dimensions;
`• support for direct user interaction via a key-
`board, mouse, touch-sensitive screen, or other
`forms of gesture recognition, so that users can
`perform simple operations such as scrolling
`and pausing video;
`• downloadable code support, such as Java-
`Script and Java applets, for richer user inter-
`activity;
`• support for a short-range wireless network
`interface to communicate with mobile devices
`and make it easy for mobile devices to discover
`
`66
`
`Computer
`
`IPR2017-02056, Petitioner Google Inc.
`Ex. 1013, p. 66
`
`

`

`just those displays in the immediate vicinity;
`and
`• support for one or more wired interfaces such
`as USB or FireWire to connect directly to
`mobile devices.
`
`Although network-attached displays have much
`in common with network printers, we believe these
`displays will be less expensive to operate because
`they use no consumables such as ink and paper.
`We already see trends in this direction with the
`addition of Ethernet and Wi-Fi interfaces to pro-
`jectors, some of which even come with software to
`display content from a remote PC. We expect to
`see network displays become standard appliances
`eventually, deployed in various environments.
`Over time, we anticipate that several different
`types of network displays will become available,
`all supporting standard software and hardware
`interfaces. The prices of such displays will likely
`drop as well, widening their deployment further.
`We have started prototyping such network-
`connected displays by enhancing standard projec-
`tors. Recently we demonstrated an application
`scenario in which our WatchPad wristwatch com-
`puter5 and the Everywhere Display research pro-
`totype establish a symbiotic session.6
`
`APPLICATION ENVIRONMENTS
`We expect digital cameras to be used symbioti-
`cally in several environments, including homes,
`offices, shops, cars, airplanes, and trains. This intro-
`duces several challenges because some of these envi-
`ronments are friendlier than others, some have
`better access to power, some have organization fire-
`walls, and some are more private. This variability
`affects both the camera’s design and required infra-
`structure.
`
`Camera image capacity
`Table 2 shows that the size of the images cap-
`tured by digital cameras has increased at a steady
`pace in the past few years, thanks to higher-reso-
`lution imaging chips. The number of images that
`could be stored in the camera did not increase sig-
`nificantly during these early years because increas-
`ing image resolution neutralized the increased
`storage capacity. Figure 1 shows that as technol-
`ogy improved, several cameras became physically
`smaller, shrinking display size accordingly.
`The human eye’s limitations make it unlikely that
`the integrated camera’s display size and resolution
`will continue increasing. While professional pho-
`tographers may want higher capture resolutions,
`
`Figure 1. Cameras from 1995 through 2003. As technology improved, cameras
`became physically smaller, shrinking display size accordingly.
`
`Figure 2. Portable-storage devices with capacities ranging from 512 Mbytes to
`40 Gbytes.
`
`many consumers find the current image capture res-
`olutions adequate because they can already print
`high-quality enlargements. Although larger cam-
`eras can include sophisticated optics, consumers
`often prefer the convenience of smaller devices.
`In contrast, we can expect camera storage capac-
`ities to continue doubling almost every year, effec-
`tively increasing the number of pictures the camera
`can store. IBM’s introduction of its MicroDrives,
`with a 340-Mbyte capacity in a compact flash form
`factor, began this trend. Atomic-force-microscope-
`based data storage technologies such as Millipede7
`can have a data storage density of 125 Gbytes per
`square inch—10 times higher than the densest mag-
`netic storage available today.
`This difference in the growth rates of image and
`storage size will eventually let users retain many
`images in their cameras. For example, the 40-Gbyte
`storage device shown in Figure 2 can hold 20,000
`5-megapixel images. A VGA resolution thumbnail
`takes about 100 Kbytes, so thumbnails for 10,000
`
`December 2004
`
`67
`
`IPR2017-02056, Petitioner Google Inc.
`Ex. 1013, p. 67
`
`

`

`Display
`
`User’s
`camera
`
`Short-range
`wireless
`link
`
`Figure 3. Image transfer via direct symbiosis. This approach transfers the image
`directly from the camera to the high-resolution display.
`
`Local
`network
`
`Internet
`
`Display
`
`Short-range
`wireless
`link
`
`Long-range
`wireless
`link
`
`User’s
`camera
`
`Web
`repository
`
`Figure 4. Image transfer via indirect symbiosis. When images are stored in a Web
`repository, if the bandwidth between the display and the Web server is greater
`than the direct link between the camera and display, letting the display fetch
`them directly can save the camera’s battery power.
`
`images will take only 1 Gbyte. Assuming that a user
`takes about 10 pictures a day, the storage device
`can hold five years’ worth of images. If the camera
`retains only thumbnails when the actual image is
`transferred from the camera, a 40-Gbyte storage
`device could hold 400,000 VGA-resolution thumb-
`nails.
`Soon then, some cameras will be able to retain
`images taken over several years, along with anno-
`tations. Others will be able to retain thumbnails
`and annotations for all pictures captured over sev-
`eral years. Users will likely replace the camera
`before they run out of space. The function provided
`by auxiliary portable devices that help offload
`images from digital cameras will be integrated into
`the cameras themselves. As users upgrade to newer
`cameras, they could copy the entire contents of
`their current camera to a new one.
`Videos captured by the camera will likely fill up
`even these large built-in storage capacities. Until
`storage capacities increase further, users will be
`forced to offload video content to auxiliary devices
`or storage media. This offloading of video content
`reserves ample storage capacity for still images.
`Regardless of camera capacity, we expect users
`will often copy their images to home computers or
`Web repositories, primarily to safeguard against
`the camera’s loss. Already, several providers—such
`
`68
`
`Computer
`
`as Ofoto, Yahoo, and Shutterfly—offer network-
`accessible storage and photo printing services. Such
`repositories also let users share their pictures with
`others and obtain prints when necessary.
`Just as MP3 players let music lovers carry huge
`collections of their favorite music, technology
`trends will let consumers personally carry a large
`image collection. To unleash the full potential of
`these collections, users must be able to view and
`share images easily.
`
`Direct symbiosis
`How users view their images on intelligent envi-
`ronmental displays will vary depending on where
`the image is stored. If the image resides on the cam-
`era itself, the simplest approach transfers it directly
`from the camera to the display, as Figure 3 shows.
`However, to retain privacy and security in less
`secluded settings, a user can preview images on the
`camera display before sending them to the larger
`display.
`To do this, once the camera discovers the display
`and its capabilities, the camera and display estab-
`lish a session, and, if required, the session completes
`a mutual authentication. With the built-in camera
`display, the user scrolls through the image thumb-
`nails and picks one to view on the environmental
`display. The camera sends the higher resolution
`image to the display. If the display does not sup-
`port the high-resolution image, the camera can
`resize the image to reduce the amount of data being
`transferred.
`The connection between the camera and the envi-
`ronmental display could be made over a short-
`range wireless link or through a wired connection
`such as USB. From a communication semantics per-
`spective, the camera is the master and the display
`the slave, even though physical cabling can make
`the camera a USB slave.
`
`Indirect symbiosis
`When images reside in a Web repository, letting
`the display fetch them directly from the repository
`can save the camera’s battery power. This approach,
`shown in Figure 4, may be faster if the bandwidth
`between the display and the Web server is greater
`than the direct link between the camera and display.
`We expect that the user will still view the thumb-
`nails privately on the camera display before decid-
`ing which images to view on the large display. Image
`selection will thus still be a problem when dealing
`with hundreds or thousands of thumbnails.
`Once the user selects an image, the camera can
`direct the display to fetch it from the repository. To
`
`IPR2017-02056, Petitioner Google Inc.
`Ex. 1013, p. 68
`
`

`

`use this approach, the camera must be able to
`authorize the image transfer. It can do so by access-
`ing the Web repository via its long-range wireless
`connection and requesting that the repository cre-
`ate a freshly computed ephemeral token. To pro-
`vide suitable security, the token could be derived
`based on the time, user’s password, location, name
`and address of the display, and sequence numbers
`sent by the camera. The camera then hands off the
`token to the environmental display over the short-
`range wireless interface. The environmental display
`obtains the image from the Web repository on the
`camera’s behalf, proving its authority by supplying
`the token. The repository rejects any requests with-
`out valid tokens.
`The entire mechanism can be simplified if the cam-
`era can send a URL to the display that includes both
`the image identifier and the authorization token. The
`Web repository usually cannot initiate the image
`transfer to the display because the display may reside
`behind a firewall. The camera’s short- and long-
`range wireless capabilities work in conjunction to
`download and view selected images without com-
`promising the user’s entire image collection.
`This approach can also be used to share images.
`For example, a user could receive a message from
`a friend who has uploaded an image collection he
`wishes to share. The message might include thumb-
`nails that arrive at the user’s mobile device. While
`scrolling through the message’s thumbnails, the user
`can instruct the mobile device to display selected
`thumbnails on the larger display. At this point, the
`user’s camera delegates its authority to the display
`so that it can fetch the images from the friend’s Web
`repository. Optionally, the display could transfer a
`copy of the image to the camera over the short-
`range link.
`
`CHALLENGES
`Digital camera designers must confront several
`challenges, including issues of security, usability,
`power management, wireless interface range, and
`storage type.
`
`Image management
`Cameras that store images must deal with the
`challenge of managing large amounts of data. This
`differs from managing thousands of songs stored
`on an MP3 player because songs have descriptive
`metadata such as names and artists, whereas
`images must be user annotated.
`Given that thousands of images can be stored in
`one place, easy and effective naming schemes must
`be developed to replace sequential numbering.
`
`Thousands of images
`can be stored in one
`place, so effective
`naming schemes
`must be developed
`to replace
`sequential
`numbering.
`
`Voice-based input could be used for naming
`picture groups. Although people typically
`avoid annotating images because doing so is
`cumbersome, several approaches can make
`this task simpler,8-11 such as allowing the addi-
`tion of voice labels at the time of capture that
`PCs and similar devices can later convert to
`searchable text. Automatically recording a
`picture’s context at the time of capture could
`be essential to building organized albums.
`Cameras can easily be enhanced to auto-
`matically record several parameters, such as
`location and the photographer’s identity.
`Researchers have studied the benefits of using
`location coordinates to tag and organize digital
`images.12,13 The camera could use such location tags
`to organize images into viewlists associated with
`specific locations. Or annotations could be built on
`another device and transferred to the camera.
`Many of today’s cameras have no notion of accept-
`ing image content or metadata from other devices,
`so this shortcoming must be addressed as well.
`Because users can keep copies of images on other
`devices and also edit image albums on them, the
`camera’s stored images may not be synchronized
`with those on the other device. Thus, standard syn-
`chronization techniques have to be adapted for dig-
`ital cameras. If images are deleted from the Web
`repository, for example, the corresponding thumb-
`nails must be deleted on the camera as well. Given
`that some images will be ephemeral in nature, a
`mechanism must be provided to specify when those
`images can be automatically deleted.
`Users often replace cameras every few years, and
`they will need to transfer images from the old cam-
`era to the new one just as we transfer data between
`computers today. Adjustments may have to be
`made to account for the differences in the cameras,
`such as image resolution and viewfinder size and
`resolution. Techniques to manage large collections
`of images on PCs14 can be applied here as well.
`
`Quick search and retrieval
`Clearly, as the amount of storage in cameras
`increases, quick image retrieval and image man-
`agement will become a problem. Several ap-
`proaches can facilitate image retrieval. First and
`preferred, the input controls and software on the
`environmental display can be used to compose a
`query and overcome the camera’s input limitations.
`The camera could send JavaScript code to the envi-
`ronmental display and present a form from which
`the user selects search criteria. The criteria would
`include all the metadata captured with the image,
`
`December 2004
`
`69
`
`IPR2017-02056, Petitioner Google Inc.
`Ex. 1013, p. 69
`
`

`

`For maximum
`flexibility, the
`camera must include
`wired, short-range
`wireless, and
`long-range wireless
`communication
`capabilities.
`
`such as ranges for date, time, location, anno-
`tation subtext, aperture, focus and flash set-
`tings, photographer identity, privacy levels,
`and retention settings. The environmental
`display would then send the encoded query
`back to the camera to complete the search.
`Other criteria, such as “Find images simi-
`lar to the one selected,” can be provided
`using research in image similarity.15 User
`feedback to query responses can be used to
`improve image annotations and future
`searches.8
`Another approach uses embedded voice
`recognition technology to specify the query.
`A limited vocabulary recognition engine can han-
`dle most parameters. If the camera incorporates a
`phone, it would be logical to use voice for interac-
`tions.
`The fallback option uses the camera input con-
`trols to search for the images. For example, the user
`could position the dials used to specify image cap-
`ture modes to filter the images to ones that match
`the settings. Additional controls could be used to
`specify date ranges.
`In our experience, the most difficult parameter
`to specify is location. For our purposes, a map
`interface with zoom capability, similar to the one
`available on world clocks to select time zones,
`works best to specify a location.
`
`Security and privacy issues
`Given that cameras can have thousands of per-
`sonal images, it may be necessary to encrypt the
`images or at least use passwords or biometric
`means to protect them from unauthorized access.
`Borrowing from USB storage devices, which
`include fingerprint sensors and partition the flash
`into public and private sections, we can divide the
`camera storage into public and private areas. The
`private section can be accessed only when the user
`provides the right biometric or password. It might
`be useful to indicate the privacy level associated
`with each image at the time of capture.
`Because the camera will interact with unknown
`displays, it must be able to execute challenge-and-
`response sequences to establish the display’s iden-
`tity and authenticity. The camera’s display serves
`two purposes:
`
`• to preview images privately before deciding to
`view them on the larger public display, and
`• to show information from the environmental
`display that verifies its credentials and trust-
`worthiness.
`
`Network-attached displays will also need to
`enforce access controls, verifying that only autho-
`rized users can access the display. If the display has
`a short-range wireless interface or a direct wired
`connection, anyone who can connect to the display
`could use it. If the display is on an access-controlled
`network, anyone who has permission to use the
`network could use its services.
`The displays will also need to support the notion
`of a user session in which one authorized user con-
`trols the display and anyone else wanting to use it
`must wait until the session ends. Users must have
`the same confidence that such displays will not
`retain their content after use that they have with
`printers and audio speakers today.
`
`Network and power issues
`For maximum flexibility, the camera must
`include wired, short-range wireless, and long-range
`wireless communication capabilities. The advan-
`tages of a wired connection include higher speed,
`lower power, and simplicity because device dis-
`covery is easier. However, wired connections are
`prone to hardware malfunction caused by incor-
`rect attachment or mechanical fatigue. Moreover,
`with some displays, such as ceiling-mounted pro-
`jectors, it may be difficult to make wired connec-
`tions.
`Short-range wireless is useful for establishing
`trust relationships and transferring images from
`the camera to the display. With a long-range wire-
`less channel, the camera can fetch images from
`other sources and send them to the display over
`the short-range wireless channel. It also helps the
`camera obtain a fresh ephemeral token that grants
`permission to fetch and display images from the
`user’s network-accessible image repository.
`Long-range wireless channels consume a fair
`amount of power for large images and are slow.
`The camera can use long- and short-range chan-
`nels to establish trust between it and the display
`even if the camera cannot implement high-strength
`security algorithms. The camera can use the long-
`range channel to fetch a response to a challenge by
`communicating with a trusted proxy.
`
`CAMERA ARCHITECTURE
`To meet the requirements for different settings,
`designers must augment current digital cameras
`with features from other mobile devices. Our hypo-
`thetical cell phone camera, designed for effective
`symbiosis, integrates
`
`• a high-resolution imager,
`
`70
`
`Computer
`
`IPR2017-02056, Petitioner Google Inc.
`Ex. 1013, p. 70
`
`

`

`• a phone with a wide-area 3G capability,
`• a short-range wireless interface,
`• low-power storage,
`• high-density storage, and
`• a location-tracking mechanism.
`
`The camera also includes a wired USB connector
`for fast data transfer while powered by a host
`device. Figure 5 shows the architecture that incor-
`porates these features.
`The storage segments have a public and a private
`area. The camera shares the contents of the private
`area only when a user provides suitable authenti-
`cation. In addition to optional passwords, a fin-
`gerprint sensor authenticates each user. This sensor
`also records the photographer’s identity. The loca-
`tion-sensing mechanism records where the camera
`captured the image. In addition, the image captures
`several parameters, such as the aperture, light con-
`dition, and flash status.9
`The camera includes a high-density drive with a
`capacity of more than 100 Gbytes and its own private
`and public area. This drive functions as a secondary
`storage device so that the camera can use the flash
`memory as the primary storage device. To save
`power, the drive may be turned off most of the time.
`When the camera is plugged into an external power
`source, images can be moved from the flash memory
`to the drive. Additional dials record an image’s level
`of privacy, which can determine the section where
`the camera stores the image. Possible choices for who
`can view these images include immediate family,
`friends and family, or everyone. Another dial on the
`camera indicates how long the user expects to keep
`the image. If necessary, the privacy and longevity val-
`ues can be changed after the picture is taken.
`Although designers ensure that the camera’s dis-
`play is as large as its form factor allows, the reso-
`lution will still be substantially smaller than the
`resolution of the captured images. The camera
`includes voice input for recording annotations and
`embedded voice recognition technology for access-
`ing images.
`One downside to an integrated device is that if the
`user turns it off, all its functions become unavailable.
`For example, when a cell phone must be turned off
`in an airplane, the camera function also is unavail-
`able. Providing independent power controls for the
`camera, phone, storage, and other subsystems could
`overcome this limitation and save energy.
`
`APPLICATIONS
`The overall architecture we have described can
`be applied in several ways.
`
`Phone subsystem
`
`Long-range wireless
`Short-range wireless
`I/O controls
`
`Camera
`subsystem
`Lens
`Shutter
`Flash
`CCD
`I/O controls
`
`Display
`
`Other functions
`
`MP3 player, PDA, Wired USB,
`Location (GPS)
`
`Fingerprint
`sensor
`
`Longevity
`Privacy
`dial
`dial
`Low-power storage
`Public
`Private
`area
`area
`
`High-density storage
`Public
`Private
`area
`area
`
`Figure 5. Hypothetical digital cell phone and camera components. Designed to
`function in different settings, this camera incorporates components such as
`a high-resolution imager, a phone with wide-area capability, and a location-
`tracking mechanism.
`
`Collaborative image viewing
`Consider, for example, sharing an image album
`with guests at a party. The camera serves as a
`remote control for displaying the images, which are
`stored on a home PC connected to a projector
`through a high-speed link. In this scenario, the cam-
`era must display the selected images without reveal-
`ing any pictures the host chooses not to show.
`An initial phase connects the camera, PC, and
`projector. This approach simplifies network bound-
`ary and authentication issues because the devices
`function in a friendly environment where they trust
`each other. In this situation, the camera has a
`thumbnail view of the images along with the anno-
`tations and links to the images on the home PC.
`Even though the camera may store the complete
`images, they can be fetched faster from the home
`PC, which helps sustain audience interest.
`The user can look at the thumbnails on the cam-
`era, decide if an image is suitable for public view-
`ing, then tap the thumbnail. The application then
`shows any notes the user might have associated
`with the image such as the date and circumstances
`when the image was taken. Private audio annota-
`tions can be delivered to the camera headphone
`directly. While the projector shows the image, the
`user can advance the camera’s viewer and queue up
`other images. The user can skip any images deemed
`unsuitable for this audience.
`To simplify the image selection process, the user
`can create viewlists in advance. In this case, the sys-
`tem prefetches the content, if possible, to reduce
`image loading delays. Based on comments received
`from the audience, the camera’s viewlist can be
`updated. If someone in the audience requests a copy
`of an image and the camera’s address book has an
`entry for that person, a simple action e-mails the
`image from the home PC.
`
`December 2004
`
`71
`
`IPR2017-02056, Petitioner Google Inc.
`Ex. 1013, p. 71
`
`

`

`Figure 6. Shopper
`assistance. Digital
`camera images can
`help shoppers
`purchase the
`correct parts for
`their home
`hardware, such
`as these furnace
`controls.
`
`In-vehicle symbiosis
`Those who travel by car, train, or plane could
`find new uses for digital cameras. Many cars today
`have navigation consoles that show maps and other
`information. Further, several high-end cars have
`rear-view cameras to assist drivers while reversing.
`Similarly, some cars have front-view cameras that
`monitor road markers and signal when the driver
`strays from the lane. Others have started including
`Bluetooth wireless communication within the car
`to facilitate hands-free cell phone operation, while
`still others have begun offering voice-based opera-
`tion of some subsystems. Digital camera users can
`exploit all these features.
`For example, in 1997, we built a prototype sys-
`tem that annotated pictures captured with a Ricoh-
`RDC1 digital camera with GPS location metadata
`to augment textual directions with turn-by-turn
`images.9 We sought to provide pictures of signifi-
`cant points along the route, such as important inter-
`sections, landmarks, exit signs, entry and exit
`ramps, and so on, along with text directions. To do
`so, we captured the images ahead of time, manu-
`ally. We then added these images to a database and
`annotated them with GPS coordinates and direc-
`tional information. When the system computed
`directions between two places, it fetched appro-
`priate images from the database and included them
`as hyperlinks.
`Given current automotive developments, our
`camera could store images of appropriate intersec-
`tions, signs, and other features before a trip, then
`supply them to the onboard display during the jour-
`ney. The short-range wireless interface transfers the
`images from the camera phone to the navigation
`console at the trip’s start. The navigation system
`then shows the pictures at appropriate junctures,
`as Figures 7a and 7b show. The limited amount of
`detail visible in Figures 7c and 7d make the point
`that we ne