In Presence: Teleoperators and Virtual Environments 6, 4 (August 1997), 355-385.
`
`A Survey of Augmented Reality
`
`Ronald T. Azuma
`Hughes Research Laboratories
`3011 Malibu Canyon Road, MS RL96
`Malibu, CA 90265
`azuma@isl.hrl.hac.com
`http://www.cs.unc.edu/~azuma
`W: (310) 317-5151
`Fax: (310) 317-5695
`
`Abstract
`
`This paper surveys the field of Augmented Reality, in which 3-D virtual
`objects are integrated into a 3-D real environment in real time. It describes the
`medical, manufacturing, visualization, path planning, entertainment and military
`applications that have been explored. This paper describes the characteristics of
`Augmented Reality systems, including a detailed discussion of the tradeoffs between
`optical and video blending approaches. Registration and sensing errors are two of the
`biggest problems in building effective Augmented Reality systems, so this paper
`summarizes current efforts to overcome these problems. Future directions and areas
`requiring further research are discussed. This survey provides a starting point for
`anyone interested in researching or using Augmented Reality.
`1.
`Introduction
`
`1.1 Goals
`
`This paper surveys the current state-of-the-art in Augmented Reality. It
`describes work performed at many different sites and explains the issues and
`problems encountered when building Augmented Reality systems. It summarizes the
`tradeoffs and approaches taken so far to overcome these problems and speculates on
`future directions that deserve exploration.
`
`A survey paper does not present new research results. The contribution comes
`from consolidating existing information from many sources and publishing an
`extensive bibliography of papers in this field. While several other introductory
`papers have been written on this subject [Barfield95] [Bowskill95] [Caudell94]
`[Drascic93b] [Feiner94a] [Feiner94b] [Milgram94b] [Rolland94], this survey is more
`comprehensive and up-to-date. This survey provides a good beginning point for
`anyone interested in starting research in this area.
`
`Section 1 describes what Augmented Reality is and the motivations for
`developing this technology. Six classes of potential applications that have been
`explored are described in Section 2. Then Section 3 discusses the issues involved in
`
`Falkbuilt Ex. 1024, Page 001
`
`

`

`building an Augmented Reality system. Currently, two of the biggest problems are in
`registration and sensing: the subjects of Sections 4 and 5. Finally, Section 6 describes
`some areas that require further work and research.
`
`1.2 Definition
`
`Augmented Reality (AR) is a variation of Virtual Environments (VE), or
`Virtual Reality as it is more commonly called. VE technologies completely immerse
`a user inside a synthetic environment. While immersed, the user cannot see the real
`world around him. In contrast, AR allows the user to see the real world, with virtual
`objects superimposed upon or composited with the real world. Therefore, AR
`supplements reality, rather than completely replacing it. Ideally, it would appear to
`the user that the virtual and real objects coexisted in the same space, similar to the
`effects achieved in the film "Who Framed Roger Rabbit?" Figure 1 shows an
`example of what this might look like. It shows a real desk with a real phone. Inside
`this room are also a virtual lamp and two virtual chairs. Note that the objects are
`combined in 3-D, so that the virtual lamp covers the real table, and the real table
`covers parts of the two virtual chairs. AR can be thought of as the "middle ground"
`between VE (completely synthetic) and telepresence (completely real) [Milgram94a]
`[Milgram94b].
`
`Figure 1: Real desk with virtual lamp and two virtual chairs. (Courtesy ECRC)
`
`Some researchers define AR in a way that requires the use of Head-Mounted
`Displays (HMDs). To avoid limiting AR to specific technologies, this survey defines
`AR as systems that have the following three characteristics:
`
`1) Combines real and virtual
`2) Interactive in real time
`3) Registered in 3-D
`
`This definition allows other technologies besides HMDs while retaining the
`essential components of AR. For example, it does not include film or 2-D overlays.
`Films like "Jurassic Park" feature photorealistic virtual objects seamlessly blended
`with a real environment in 3-D, but they are not interactive media. 2-D virtual
`overlays on top of live video can be done at interactive rates, but the overlays are not
`combined with the real world in 3-D. However, this definition does allow monitor-
`
`2
`
`Falkbuilt Ex. 1024, Page 002
`
`

`

`based interfaces, monocular systems, see-through HMDs, and various other
`combining technologies. Potential system configurations are discussed further in
`Section 3.
`
`1.3 Motivation
`
`Why is Augmented Reality an interesting topic? Why is combining real and
`virtual objects in 3-D useful? Augmented Reality enhances a user's perception of and
`interaction with the real world. The virtual objects display information that the user
`cannot directly detect with his own senses. The information conveyed by the virtual
`objects helps a user perform real-world tasks. AR is a specific example of what Fred
`Brooks calls Intelligence Amplification (IA): using the computer as a tool to make a
`task easier for a human to perform [Brooks96].
`
`At least six classes of potential AR applications have been explored: medical
`visualization, maintenance and repair, annotation, robot path planning, entertainment,
`and military aircraft navigation and targeting. The next section describes work that
`has been done in each area. While these do not cover every potential application area
`of this technology, they do cover the areas explored so far.
`
`2. Applications
`
`2.1 Medical
`
`Doctors could use Augmented Reality as a visualization and training aid for
`surgery. It may be possible to collect 3-D datasets of a patient in real time, using
`non-invasive sensors like Magnetic Resonance Imaging (MRI), Computed
`Tomography scans (CT), or ultrasound imaging. These datasets could then be
`rendered and combined in real time with a view of the real patient. In effect, this
`would give a doctor "X-ray vision" inside a patient. This would be very useful during
`minimally-invasive surgery, which reduces the trauma of an operation by using small
`incisions or no incisions at all. A problem with minimally-invasive techniques is that
`they reduce the doctor's ability to see inside the patient, making surgery more
`difficult. AR technology could provide an internal view without the need for larger
`incisions.
`
`AR might also be helpful for general medical visualization tasks in the
`surgical room. Surgeons can detect some features with the naked eye that they cannot
`see in MRI or CT scans, and vice-versa. AR would give surgeons access to both
`types of data simultaneously. This might also guide precision tasks, such as
`displaying where to drill a hole into the skull for brain surgery or where to perform a
`needle biopsy of a tiny tumor. The information from the non-invasive sensors would
`be directly displayed on the patient, showing exactly where to perform the operation.
`
`3
`
`Falkbuilt Ex. 1024, Page 003
`
`

`

`AR might also be useful for training purposes [Kancherla95]. Virtual
`instructions could remind a novice surgeon of the required steps, without the need to
`look away from a patient to consult a manual. Virtual objects could also identify
`organs and specify locations to avoid disturbing [Durlach95].
`
`Several projects are exploring this application area. At UNC Chapel Hill, a
`research group has conducted trial runs of scanning the womb of a pregnant woman
`with an ultrasound sensor, generating a 3-D representation of the fetus inside the
`womb and displaying that in a see-through HMD (Figure 2). The goal is to endow
`the doctor with the ability to see the moving, kicking fetus lying inside the womb,
`with the hope that this one day may become a "3-D stethoscope" [Bajura92]
`[State94]. More recent efforts have focused on a needle biopsy of a breast tumor.
`Figure 3 shows a mockup of a breast biopsy operation, where the virtual objects
`identify the location of the tumor and guide the needle to its target [State96b]. Other
`groups at the MIT AI Lab [Grimson94] [Grimson95] [Mellor95a] [Mellor95b],
`General Electric [Lorensen93], and elsewhere [Betting95] [Edwards95] [Taubes94]
`are investigating displaying MRI or CT data, directly registered onto the patient.
`
`Figure 2: Virtual fetus inside womb of pregnant patient. (Courtesy UNC Chapel
`Hill Dept. of Computer Science.)
`
`Figure 3: Mockup of breast tumor biopsy. 3-D graphics guide needle insertion.
`(Courtesy UNC Chapel Hill Dept. of Computer Science.)
`
`2.2 Manufacturing and repair
`
`4
`
`Falkbuilt Ex. 1024, Page 004
`
`

`

`Another category of Augmented Reality applications is the assembly,
`maintenance, and repair of complex machinery. Instructions might be easier to
`understand if they were available, not as manuals with text and pictures, but rather as
`3-D drawings superimposed upon the actual equipment, showing step-by-step the
`tasks that need to be done and how to do them. These superimposed 3-D drawings
`can be animated, making the directions even more explicit.
`
`Several research projects have demonstrated prototypes in this area. Steve
`Feiner's group at Columbia built a laser printer maintenance application [Feiner93a],
`shown in Figures 4 and 5. Figure 4 shows an external view, and Figure 5 shows the
`user's view, where the computer-generated wireframe is telling the user to remove the
`paper tray. A group at Boeing is developing AR technology to guide a technician in
`building a wiring harness that forms part of an airplane's electrical system. Storing
`these instructions in electronic form will save space and reduce costs. Currently,
`technicians use large physical layout boards to construct such harnesses, and Boeing
`requires several warehouses to store all these boards. Such space might be emptied
`for other use if this application proves successful [Caudell92] [Janin93] [Sims94].
`Boeing is using a Technology Reinvestment Program (TRP) grant to investigate
`putting this technology onto the factory floor [BoeingTRP94]. Figure 6 shows an
`external view of Adam Janin using a prototype AR system to build a wire bundle.
`Eventually, AR might be used for any complicated machinery, such as automobile
`engines [Tuceryan95].
`
`Figure 4: External view of Columbia printer maintenance application. Note that
`all objects must be tracked. (Courtesy Steve Feiner, Blair MacIntyre, and Dorée
`Seligmann, Columbia University.)
`
`Figure 5: Prototype laser printer maintenance application, displaying how to
`remove the paper tray. (Courtesy Steve Feiner, Blair MacIntyre, and Dorée
`Seligmann, Columbia University.)
`
`5
`
`Falkbuilt Ex. 1024, Page 005
`
`

`

`Figure 6: Adam Janin demonstrates Boeing's prototype wire bundle assembly
`application. (Courtesy David Mizell, Boeing)
`
`2.3 Annotation and visualization
`
`AR could be used to annotate objects and environments with public or private
`information. Applications using public information assume the availability of public
`databases to draw upon. For example, a hand-held display could provide information
`about the contents of library shelves as the user walks around the library
`[Fitzmaurice93] [Rekimoto95a] [Rekimoto95b]. At the European Computer-Industry
`Research Centre (ECRC), a user can point at parts of an engine model and the AR
`system displays the name of the part that is being pointed at [Rose95]. Figure 7
`shows this, where the user points at the exhaust manifold on an engine model and the
`label "exhaust manifold" appears.
`
`Figure 7: Engine model part labels appear as user points at them. (Courtesy
`ECRC)
`
`Alternately, these annotations might be private notes attached to specific
`objects. Researchers at Columbia demonstrated this with the notion of attaching
`windows from a standard user interface onto specific locations in the world, or
`attached to specific objects as reminders [Feiner93b]. Figure 8 shows a window
`superimposed as a label upon a student. He wears a tracking device, so the computer
`knows his location. As the student moves around, the label follows his location,
`providing the AR user with a reminder of what he needs to talk to the student about.
`
`6
`
`Falkbuilt Ex. 1024, Page 006
`
`

`

`Figure 8: Windows displayed on top of specific real-world objects. (Courtesy
`Steve Feiner, Blair MacIntyre, Marcus Haupt, and Eliot Solomon, Columbia
`University.)
`
`AR might aid general visualization tasks as well. An architect with a see-
`through HMD might be able to look out a window and see how a proposed new
`skyscraper would change her view. If a database containing information about a
`building's structure was available, AR might give architects "X-ray vision" inside a
`building, showing where the pipes, electric lines, and structural supports are inside
`the walls [Feiner95]. Researchers at the University of Toronto have built a system
`called Augmented Reality through Graphic Overlays on Stereovideo (ARGOS)
`[Milgram95], which among other things is used to make images easier to understand
`during difficult viewing conditions [Drascic93a]. Figure 9 shows wireframe lines
`drawn on top of a space shuttle bay interior, while in orbit. The lines make it easier to
`see the geometry of the shuttle bay. Similarly, virtual lines and objects could aid
`navigation and scene understanding during poor visibility conditions, such as
`underwater or in fog.
`
`Figure 9: Virtual lines help display geometry of shuttle bay, as seen in orbit.
`(Courtesy David Drascic and Paul Milgram, U. Toronto.)
`
`2.4 Robot path planning
`
`Teleoperation of a robot is often a difficult problem, especially when the robot
`is far away, with long delays in the communication link. Under this circumstance,
`
`7
`
`Falkbuilt Ex. 1024, Page 007
`
`

`

`instead of controlling the robot directly, it may be preferable to instead control a
`virtual version of the robot. The user plans and specifies the robot's actions by
`manipulating the local virtual version, in real time. The results are directly displayed
`on the real world. Once the plan is tested and determined, then user tells the real
`robot to execute the specified plan. This avoids pilot-induced oscillations caused by
`the lengthy delays. The virtual versions can also predict the effects of manipulating
`the environment, thus serving as a planning and previewing tool to aid the user in
`performing the desired task. The ARGOS system has demonstrated that stereoscopic
`AR is an easier and more accurate way of doing robot path planning than traditional
`monoscopic interfaces [Drascic93b] [Milgram93]. Others have also used registered
`overlays with telepresence systems [Kim93] [Kim96] [Oyama93] [Tharp94] [Yoo93].
`Figure 10 shows how a virtual outline can represent a future location of a robot arm.
`
`Figure 10: Virtual lines show a planned motion of a robot arm (Courtesy David
`Drascic and Paul Milgram, U. Toronto.)
`
`2.5 Entertainment
`
`At SIGGRAPH '95, several exhibitors showed "Virtual Sets" that merge real
`actors with virtual backgrounds, in real time and in 3-D. The actors stand in front of
`a large blue screen, while a computer-controlled motion camera records the scene.
`Since the camera's location is tracked, and the actor's motions are scripted, it is
`possible to digitally composite the actor into a 3-D virtual background. For example,
`the actor might appear to stand inside a large virtual spinning ring, where the front
`part of the ring covers the actor while the rear part of the ring is covered by the actor.
`The entertainment industry sees this as a way to reduce production costs: creating and
`storing sets virtually is potentially cheaper than constantly building new physical sets
`from scratch. The ALIVE project from the MIT Media Lab goes one step further by
`
`8
`
`Falkbuilt Ex. 1024, Page 008
`
`

`

`populating the environment with intelligent virtual creatures that respond to user
`actions [Maes95].
`
`2.6 Military aircraft
`
`For many years, military aircraft and helicopters have used Head-Up Displays
`(HUDs) and Helmet-Mounted Sights (HMS) to superimpose vector graphics upon the
`pilot's view of the real world. Besides providing basic navigation and flight
`information, these graphics are sometimes registered with targets in the environment,
`providing a way to aim the aircraft's weapons. For example, the chin turret in a
`helicopter gunship can be slaved to the pilot's HMS, so the pilot can aim the chin
`turret simply by looking at the target. Future generations of combat aircraft will be
`developed with an HMD built into the pilot's helmet [Wanstall89].
`
`3. Characteristics
`
`This section discusses the characteristics of AR systems and design issues
`encountered when building an AR system. Section 3.1 describes the basic
`characteristics of augmentation. There are two ways to accomplish this
`augmentation: optical or video technologies. Section 3.2 discusses their
`characteristics and relative strengths and weaknesses. Blending the real and virtual
`poses problems with focus and contrast (Section 3.3), and some applications require
`portable AR systems to be truly effective (Section 3.4). Finally, Section 3.5
`summarizes the characteristics by comparing the requirements of AR against those
`for Virtual Environments.
`
`3.1 Augmentation
`
`Besides adding objects to a real environment, Augmented Reality also has the
`potential to remove them. Current work has focused on adding virtual objects to a
`real environment. However, graphic overlays might also be used to remove or hide
`parts of the real environment from a user. For example, to remove a desk in the real
`environment, draw a representation of the real walls and floors behind the desk and
`"paint" that over the real desk, effectively removing it from the user's sight. This has
`been done in feature films. Doing this interactively in an AR system will be much
`harder, but this removal may not need to be photorealistic to be effective.
`
`Augmented Reality might apply to all senses, not just sight. So far,
`researchers have focused on blending real and virtual images and graphics. However,
`AR could be extended to include sound. The user would wear headphones equipped
`with microphones on the outside. The headphones would add synthetic, directional
`3–D sound, while the external microphones would detect incoming sounds from the
`environment. This would give the system a chance to mask or cover up selected real
`sounds from the environment by generating a masking signal that exactly canceled
`
`9
`
`Falkbuilt Ex. 1024, Page 009
`
`

`

`the incoming real sound [Durlach95]. While this would not be easy to do, it might be
`possible. Another example is haptics. Gloves with devices that provide tactile
`feedback might augment real forces in the environment. For example, a user might
`run his hand over the surface of a real desk. Simulating such a hard surface virtually
`is fairly difficult, but it is easy to do in reality. Then the tactile effectors in the glove
`can augment the feel of the desk, perhaps making it feel rough in certain spots. This
`capability might be useful in some applications, such as providing an additional cue
`that a virtual object is at a particular location on a real desk [Wellner93].
`
`3.2 Optical vs. video
`
`A basic design decision in building an AR system is how to accomplish the
`combining of real and virtual. Two basic choices are available: optical and video
`technologies. Each has particular advantages and disadvantages. This section
`compares the two and notes the tradeoffs. For additional discussion, see [Rolland94].
`
`A see-through HMD is one device used to combine real and virtual. Standard
`closed-view HMDs do not allow any direct view of the real world. In contrast, a see-
`through HMD lets the user see the real world, with virtual objects superimposed by
`optical or video technologies.
`
`Optical see-through HMDs work by placing optical combiners in front of the
`user's eyes. These combiners are partially transmissive, so that the user can look
`directly through them to see the real world. The combiners are also partially
`reflective, so that the user sees virtual images bounced off the combiners from head-
`mounted monitors. This approach is similar in nature to Head-Up Displays (HUDs)
`commonly used in military aircraft, except that the combiners are attached to the
`head. Thus, optical see-through HMDs have sometimes been described as a "HUD
`on a head" [Wanstall89]. Figure 11 shows a conceptual diagram of an optical see-
`through HMD. Figure 12 shows two optical see-through HMDs made by Hughes
`Electronics.
`
`The optical combiners usually reduce the amount of light that the user sees
`from the real world. Since the combiners act like half-silvered mirrors, they only let
`in some of the light from the real world, so that they can reflect some of the light
`from the monitors into the user's eyes. For example, the HMD described in
`[Holmgren92] transmits about 30% of the incoming light from the real world.
`Choosing the level of blending is a design problem. More sophisticated combiners
`might vary the level of contributions based upon the wavelength of light. For
`example, such a combiner might be set to reflect all light of a certain wavelength and
`none at any other wavelengths. This would be ideal with a monochrome monitor.
`Virtually all the light from the monitor would be reflected into the user's eyes, while
`almost all the light from the real world (except at the particular wavelength) would
`reach the user's eyes. However, most existing optical see-through HMDs do reduce
`the amount of light from the real world, so they act like a pair of sunglasses when the
`power is cut off.
`
`10
`
`Falkbuilt Ex. 1024, Page 010
`
`

`

`Head
`Tracker
`
`Head
`locations
`
`Monitors
`
`Graphic
`images
`
`Scene
`generator
`
`t
`
`Real
`world
`
`Optical
`combiners
`
`Figure 11:
`
`Optical see-through HMD conceptual diagram
`
`Figure 12:
`
`
`Two optical see-through HMDs, made by Hughes Electronics
`
`In contrast, video see-through HMDs work by combining a closed-view HMD
`with one or two head-mounted video cameras. The video cameras provide the user's
`view of the real world. Video from these cameras is combined with the graphic
`images created by the scene generator, blending the real and virtual. The result is
`sent to the monitors in front of the user's eyes in the closed-view HMD. Figure 13
`shows a conceptual diagram of a video see-through HMD. Figure 14 shows an actual
`video see-through HMD, with two video cameras mounted on top of a Flight Helmet.
`
`Video
`of
`real
`world
`
`Head
`locations
`
`Head
`Tracker
`
`Video cameras
`
`Scene
`generator
`
`Graphic
`images
`
`Real
`World
`
`Monitors
`
`Video compositor
`
`Combined video
`
`11
`
`Falkbuilt Ex. 1024, Page 011
`
`

`

`Figure 13:
`
`Video see-through HMD conceptual diagram
`
`An actual video see-through HMD. (Courtesy Jannick Rolland,
`Figure 14:
`Frank Biocca, and UNC Chapel Hill Dept. of Computer Science. Photo by Alex
`Treml.)
`
`Video composition can be done in more than one way. A simple way is to use
`chroma-keying: a technique used in many video special effects. The background of
`the computer graphic images is set to a specific color, say green, which none of the
`virtual objects use. Then the combining step replaces all green areas with the
`corresponding parts from the video of the real world. This has the effect of
`superimposing the virtual objects over the real world. A more sophisticated
`composition would use depth information. If the system had depth information at
`each pixel for the real world images, it could combine the real and virtual images by a
`pixel-by-pixel depth comparison. This would allow real objects to cover virtual
`objects and vice-versa.
`
`AR systems can also be built using monitor-based configurations, instead of
`see-through HMDs. Figure 15 shows how a monitor-based system might be built. In
`this case, one or two video cameras view the environment. The cameras may be
`static or mobile. In the mobile case, the cameras might move around by being
`attached to a robot, with their locations tracked. The video of the real world and the
`graphic images generated by a scene generator are combined, just as in the video see-
`through HMD case, and displayed in a monitor in front of the user. The user does not
`wear the display device. Optionally, the images may be displayed in stereo on the
`monitor, which then requires the user to wear a pair of stereo glasses. Figure 16
`shows an external view of the ARGOS system, which uses a monitor-based
`configuration.
`
`12
`
`Falkbuilt Ex. 1024, Page 012
`
`

`

`Monitor
`
`Stereo glasses
`(optional)
`
`Tracker
`
`Locations
`
`Video cameras
`
`Video of
`real world
`
`Scene
`generator
`
`Combiner
`
`Graphic
`images
`Figure 15: Monitor-based AR conceptual diagram
`
`External view of the ARGOS system, an example of monitor-based
`Figure 16:
`AR. (Courtesy David Drascic and Paul Milgram, U. Toronto.)
`
`Finally, a monitor-based optical configuration is also possible. This is similar
`to Figure 11 except that the user does not wear the monitors or combiners on her
`head. Instead, the monitors and combiners are fixed in space, and the user positions
`her head to look through the combiners. This is typical of Head-Up Displays on
`military aircraft, and at least one such configuration has been proposed for a medical
`application [Peuchot95].
`
`The rest of this section compares the relative advantages and disadvantages of
`optical and video approaches, starting with optical. An optical approach has the
`following advantages over a video approach:
`
`1) Simplicity: Optical blending is simpler and cheaper than video blending.
`Optical approaches have only one "stream" of video to worry about: the graphic
`images. The real world is seen directly through the combiners, and that time delay is
`generally a few nanoseconds. Video blending, on the other hand, must deal with
`separate video streams for the real and virtual images. Both streams have inherent
`
`13
`
`Falkbuilt Ex. 1024, Page 013
`
`

`

`delays in the tens of milliseconds. Digitizing video images usually adds at least one
`frame time of delay to the video stream, where a frame time is how long it takes to
`completely update an image. A monitor that completely refreshes the screen at 60 Hz
`has a frame time of 16.67 ms. The two streams of real and virtual images must be
`properly synchronized or temporal distortion results. Also, optical see-through
`HMDs with narrow field-of-view combiners offer views of the real world that have
`little distortion. Video cameras almost always have some amount of distortion that
`must be compensated for, along with any distortion from the optics in front of the
`display devices. Since video requires cameras and combiners that optical approaches
`do not need, video will probably be more expensive and complicated to build than
`optical-based systems.
`
`2) Resolution: Video blending limits the resolution of what the user sees, both
`real and virtual, to the resolution of the display devices. With current displays, this
`resolution is far less than the resolving power of the fovea. Optical see-through also
`shows the graphic images at the resolution of the display device, but the user's view
`of the real world is not degraded. Thus, video reduces the resolution of the real
`world, while optical see-through does not.
`
`3) Safety: Video see-through HMDs are essentially modified closed-view
`HMDs. If the power is cut off, the user is effectively blind. This is a safety concern
`in some applications. In contrast, when power is removed from an optical see-
`through HMD, the user still has a direct view of the real world. The HMD then
`becomes a pair of heavy sunglasses, but the user can still see.
`
`4) No eye offset: With video see-through, the user's view of the real world is
`provided by the video cameras. In essence, this puts his "eyes" where the video
`cameras are. In most configurations, the cameras are not located exactly where the
`user's eyes are, creating an offset between the cameras and the real eyes. The
`distance separating the cameras may also not be exactly the same as the user's
`interpupillary distance (IPD). This difference between camera locations and eye
`locations introduces displacements from what the user sees compared to what he
`expects to see. For example, if the cameras are above the user's eyes, he will see the
`world from a vantage point slightly taller than he is used to. Video see-through can
`avoid the eye offset problem through the use of mirrors to create another set of optical
`paths that mimic the paths directly into the user's eyes. Using those paths, the
`cameras will see what the user's eyes would normally see without the HMD.
`However, this adds complexity to the HMD design. Offset is generally not a difficult
`design problem for optical see-through displays. While the user's eye can rotate with
`respect to the position of the HMD, the resulting errors are tiny. Using the eye's
`center of rotation as the viewpoint in the computer graphics model should eliminate
`any need for eye tracking in an optical see-through HMD [Holloway95].
`
`Video blending offers the following advantages over optical blending:
`
`1) Flexibility in composition strategies: A basic problem with optical see-
`through is that the virtual objects do not completely obscure the real world objects,
`because the optical combiners allow light from both virtual and real sources.
`Building an optical see-through HMD that can selectively shut out the light from the
`real world is difficult. In a normal optical system, the objects are designed to be in
`
`14
`
`Falkbuilt Ex. 1024, Page 014
`
`

`

`focus at only one point in the optical path: the user's eye. Any filter that would
`selectively block out light must be placed in the optical path at a point where the
`image is in focus, which obviously cannot be the user's eye. Therefore, the optical
`system must have two places where the image is in focus: at the user's eye and the
`point of the hypothetical filter. This makes the optical design much more difficult
`and complex. No existing optical see-through HMD blocks incoming light in this
`fashion. Thus, the virtual objects appear ghost-like and semi-transparent. This
`damages the illusion of reality because occlusion is one of the strongest depth cues.
`In contrast, video see-through is far more flexible about how it merges the real and
`virtual images. Since both the real and virtual are available in digital form, video see-
`through compositors can, on a pixel-by-pixel basis, take the real, or the virtual, or
`some blend between the two to simulate transparency. Because of this flexibility,
`video see-through may ultimately produce more compelling environments than
`optical see-through approaches.
`
`2) Wide field-of-view: Distortions in optical systems are a function of the
`radial distance away from the optical axis. The further one looks away from the
`center of the view, the larger the distortions get. A digitized image taken through a
`distorted optical system can be undistorted by applying image processing techniques
`to unwarp the image, provided that the optical distortion is well characterized. This
`requires significant amounts of computation, but this constraint will be less important
`in the future as computers become faster. It is harder to build wide field-of-view
`displays with optical see-through techniques. Any distortions of the user's view of
`the real world must be corrected optically, rather than digitally, because the system
`has no digitized image of the real world to manipulate. Complex optics are expensive
`and add weight to the HMD. Wide field-of-view systems are an exception to the
`general trend of optical approaches being simpler and cheaper than video approaches.
`
`3) Real and virtual view delays can be matched: Video offers an approach for
`reducing or avoiding problems caused by temporal mismatches between the real and
`virtual images. Optical see-through HMDs offer an almost instantaneous view of the
`real world but a delayed view of the virtual. This temporal mismatch can cause
`problems. With video approaches, it is possible to delay the video of the real world to
`match the delay from the virtual image stream. For details, see Section 4.3.
`
`4) Additional registration strategies: In optical see-through, the only
`information the system has about the user's head location comes from the head
`tracker. Video blending provides another source of information: the digitized image
`of the real scene. This digitized image means that video approaches can employ
`additional registration strategies unavailable to optical approaches. Section 4.4
`describes these in more detail.
`
`5) Easier to match the brightness of real and virtual objects: This is disc