Augmented Reality: A class of displays on the reality-virtuality continuum
`Paul Milgram¥, Haruo Takemura¶, Akira Utsumi†, Fumio Kishino‡
`ATR Communication Systems Research Laboratoriesƒ
`2-2 Hikaridai, Seika-cho, Soraku-gun
`Kyoto 619-02, Japan
`
`ABSTRACT
`In this paper we discuss Augmented Reality (AR) displays in a general sense, within the context of a
`Reality-Virtuality (RV) continuum, encompassing a large class of "Mixed Reality" (MR) displays, which
`also includes Augmented Virtuality (AV). MR displays are defined by means of seven examples of existing
`display concepts in which real objects and virtual objects are juxtaposed. Essential factors which distinguish
`different Mixed Reality display systems from each other are presented, first by means of a table in which
`the nature of the underlying scene, how it is viewed, and the observer's reference to it are compared, and
`then by means of a three dimensional taxonomic framework, comprising: Extent of World Knowledge
`(EWK), Reproduction Fidelity (RF) and Extent of Presence Metaphor (EPM). A principal objective of the
`taxonomy is to clarify terminology issues and to provide a framework for classifying research across
`different disciplines.
`
`Keywords: Augmented reality, mixed reality, virtual reality, augmented virtuality, telerobotic control,
`virtual control, stereoscopic displays, taxonomy.
`
`1. INTRODUCTION
`Our objective in this paper is to review some implications of the term "Augmented Reality" (AR), classify
`the relationships between AR and a larger class of technologies which we refer to as "Mixed Reality" (MR),
`and propose a taxonomy of factors which are important for categorising various MR display systems. In
`the following section we present our view of how AR can be regarded in terms of a continuum relating
`purely virtual environments to purely real environments. In Section 3 we review the two principal
`manifestations of AR display systems: head-mounted see-through and monitor-based video AR displays. In
`Section 4 we extend the discussion to MR systems in general, and provide a list of seven classes of MR
`displays. We also provide a table highlighting basic differences between these. Finally, in Section 5 we
`propose a formal taxonomy of mixed real and virtual worlds. It is important to note that our discussion in
`this paper is limited strictly to visual displays.
`
`
`¥ This paper was initiated during the first author's 1993-94 research leave from the Industrial Engineering Department,
`University of Toronto, Canada. Email: milgram@ie.utoronto.ca . WWW: http://vered.rose.utoronto.ca
`¶ Now at Nara Institute of Science and Technology, Japan. Email: takemura@is.aist-nara.ac.jp
`† Email: utsumi@atr-sw.atr.co.jp
`‡ Email: kishino@atr-sw.atr.co.jp
`ƒ The authors gratefully acknowledge the generous support and contributions of Dr. K. Habara of ATR Institute and of Dr. N.
`Terashima of ATR Computer Systems Research Laboratories.
`
`282 / SPIE Vol. 2351, Telemanipulator and Telepresence Technologies (1994)
`
`Falkbuilt Ex. 1016, Page 001
`
`

`

`283 / SPIE Vol. 2351, Telemanipulator and Telepresence Technologies (1994)
`
`together within a single display, that is, anywhere between the extrema of the RV continuum.2
`generic Mixed Reality (MR) environment as one in which real world and virtual world objects are presented
`simulations, either monitor-based or immersive. Within this framework it is straightforward to define a
`consisting solely of virtual objects, examples of which would include conventional computer graphic
`some kind of a window, or via some sort of a (video) display. The case at the right defines environments
`includes whatever might be observed when viewing a real-world scene either directly in person, or through
`The case at the left of the continuum in Fig. 1 defines any environment consisting solely of real objects, and
`
`Figure 1: Simplified representation of a RV Continuum.
`
`Reality-Virtuality (RV) Continuum
`
`Environment
`
`Virtual
`
`♦
`
`Virtuality (AV)
`Augmented
`
`Reality (AR)
`Augmented
`
`t
`-
`
`Environment
`
`Real
`
`Mixed Reality (MR)
`
`•What is the relationship between Augmented Reality (AR) and Virtual Reality (VR)?
`somewhat different definitions bring to light two questions which we feel deserve consideration:
`head-mounted display is transparent, allowing a clear view of the real world" (italics added). These
`somewhat more restricted approach, by defining AR as "a form of virtual reality where the participant's
`is interesting to point out that the call for the associated special session on Augmented Reality took a
`was defined in a very broad sense as "augmenting natural feedback to the operator with simulated cues", it
`the present proceedings on Telemanipulator and Telepresence Technologies1, where Augmented Reality
`instance, although our own use of the term is in agreement with that employed in the call for participation in
`contend that this is occurring without what could reasonably be considered a consistent definition. For
`Although the term "Augmented Reality" has begun to appear in the literature with increasing frequency, we
`
`continuum. This concept is illustrated in Fig. 1 below.
`view them as lying at opposite ends of a continuum, which we refer to as the Reality-Virtuality (RV)
`physics. Rather than regarding the two concepts simply as antitheses, however, it is more convenient to
`longer hold. In contrast, a strictly real-world environment clearly must be constrained by the laws of
`reality by creating a world in which the physical laws governing gravity, time and material properties no
`a real-world environment, either existing or fictional, but which may also exceed the bounds of physical
`observer is totally immersed in a completely synthetic world, which may or may not mimic the properties of
`two concepts together. The commonly held view of a VR environment is one in which the participant-
`Perhaps surprisingly, we do in fact agree that AR and VR are related and that it is quite valid to consider the
`Should the term Augmented Reality be limited solely to transparent see-through head-mounted displays?
`
`•
`
`2. REALITY-VIRTUALITY CONTINUUM
`
`Falkbuilt Ex. 1016, Page 002
`
`

`

`3. TWO CATEGORIES OF AUGMENTED REALITY DISPLAYS
`Within the context of Fig. 1, the above-mentioned broad definition of Augmented Reality – "augmenting
`natural feedback to the operator with simulated cues" – is quite clear. Also noteworthy in this figure is the
`corresponding concept of Augmented Virtuality (AV), which results automatically, both conceptually and
`lexically, from the figure. Some examples of AV systems are given below, in Section 4. In the present
`section we first contrast two cases of AR, those based on head-mounted see-through displays and those
`which are monitor based, but both of which comply with the definition depicted in Fig. 1.
`3.1 "See-through" AR displays
`This class of displays is characterised by the ability to see through the display medium directly to the world
`surrounding the observer, thereby achieving both the maximal possible extent of presence and the ultimate
`degree of "realspace imaging".3 Most commonly display augmentation is achieved by using mirrors to
`superimpose computer generated graphics optically onto directly viewed real-world scenes. Such displays
`are already a mature technology in some (mostly military) aviation systems, as either panel-mounted or
`head-mounted displays (HMD's), but are currently finding new applications as a VR related technology,
`especially in manufacturing and maintenance environments.4,5,6 Of special interest are recent efforts to
`apply such AR displays to medical imaging, by superimposing data acquired via imaging techniques such
`as ultrasound, CT scanning, etc. conformally onto the actual patient.7,8
`Several research and development issues have accompanied the advent of optical see-through (ST) displays.
`These include the need for accurate and precise, low latency body and head tracking, accurate and precise
`calibration and viewpoint matching, adequate field of view, and the requirement for a snug (no-slip) but
`comfortable and preferably untethered head-mount.5,9 Other issues which present themselves are more
`perceptual in nature, including the conflicting effects of occlusion of apparently overlapping objects and
`other ambiguities introduced by a variety of factors which define the interactions between computer
`generated images and real object images.9 Perceptual issues become even more challenging when ST-AR
`systems are constructed to permit computer augmentation to be presented stereoscopically.10
`Some of these technological difficulties can be partially alleviated by replacing the optical ST with a
`conformal video-based HMD, thereby creating what is known as "video see-through". Such displays
`present certain advantages, both technological and perceptual9, even as new issues arise from the need to
`create a camera system whose effective viewpoint is identical to that of the observer's own eyes.11
`3.2 Monitor based AR displays
`We use the term monitor-based (non-immersive), or "window-on-the-world" (WoW), AR to refer to
`display systems where computer generated images are either analogically or digitally overlaid onto live or
`stored video images.12,13,14 Although the technology for achieving this has been well-known for some
`time, most notably by means of chroma-keying, a large number of useful applications present themselves
`when this concept is implemented stereoscopically.15,16,17
`In our own laboratory this class of monitor-based AR displays has been under development for some years,
`as part of the ARGOS (Augmented Reality through Graphic Overlays on Stereovideo) project.18 Several
`studies have been carried out to investigate the practical applicability of, among other things, overlaid
`stereographic virtual pointers and virtual tape measures19, virtual landmarks20, and virtual tethers21 for
`telerobotic control. Current efforts are focused on applying more advanced stereographic tools for achieving
`virtual control22,23,24 of telerobotic systems, through the use of overlaid virtual robot simulations, virtual
`encapsulators, and virtual bounding planes, etc.25
`
`284 / SPIE Vol. 2351, Telemanipulator and Telepresence Technologies (1994)
`
`Falkbuilt Ex. 1016, Page 003
`
`

`

`4 . THE GENERAL CASE: MIXED REALITY ENVIRONMENTS
`Thus far we have defined the concept of AR within the context of the reality-virtuality continuum, and
`illustrated two particular subclasses of AR displays. In this section we discuss how Augmented Reality
`relates to other classes of Mixed Reality displays.
`Through the brief discussion in Section 3, it should be evident that the key factors distinguishing see-
`through and monitor based AR systems go beyond simply whether the display is head mounted or monitor
`based, which governs the metaphor of whether one is expected to feel egocentrically immersed within one's
`world or whether is one is to feel that one is exocentrically looking in on that world from the outside. There
`is also the issue of how much one knows about the world being viewed, which is essential for the
`conformal mapping needed for useful see-through displays, but much less critical for WoW displays. In
`addition, there are other, largely perceptual, issues which are a function of the degree to which the fidelity
`of the 'substratal world' must be maintained. With optical see-through (ST) systems, one has very little
`latitude, beyond optical distortion, to change the reality of what one observes directly, whereas when video
`is used as the intermediate medium, the potential for altering that world is much larger.
`This leads us back to the concept of the RV continuum, and to the issue of defining the substratum:
`•
`Is the environment being observed principally real, with added computer generated enhancements? or
`•
`Is the surrounding environment principally virtual, but augmented through the use of real (i.e.
`unmodelled) imaging data?
`(Of course, as computer graphic and imaging technologies continue to advance, the day will certainly arrive
`in which it will not be immediately obvious whether the primary world is real or simulated, a situation
`corresponding to the centre of the RV continuum in Fig. 1).
`The case defined by the second question serves as our working definition of what we term "Augmented
`Virtuality" (AV), in reference to completely graphic display environments, either completely immersive,
`partially immersive, or otherwise, to which some amount of (video or texture mapped) 'reality' has been
`added.2,13 When this class of displays is extended to include situations in which real objects, such as a
`user's hand, can be introduced into the otherwise principally graphic world, in order to point at, grab, or
`somehow otherwise manipulate something in the virtual scene26,27, the perceptual issues which arise,
`especially for stereoscopic displays, become quite challenging.28,29
`In order further to distinguish essential differences and similarities between the various display concepts
`which we classify as Mixed Reality, it is helpful to make a formal list of these:
`1. Monitor-based (non-immersive) AR displays, upon which computer graphic (CG) images are overlaid.
`2. Same as 1, but using immersive HMD-based displays, rather than WoW monitors.§
`3. HMD-based AR systems, incorporating optical see-through (ST).
`4. HMD-based AR systems, incorporating video ST.
`5. Monitor-based AV systems, with CG world substratum, employing superimposed video reality.
`6.
`Immersive or partially immersive (e.g. large screen display) AV systems, with CG substratum,
`employing superimposed video or texture mapped reality.
`7. Partially immersive AV systems, which allow additional real-object interactions, such as 'reaching in'
`and 'grabbing' with one's own (real) hand.
` It is worth noting that additional classes could have been delineated; however, we are limiting ourselves
`here to the primary factors characterising the most prominent classes of MR displays. One important
`distinction which has been conspicuously left out of the above list, for example, is whether or not the
`systems listed are stereoscopic.
`
`§ This class refers, for example, to HMD-based AR-enhanced head-slaved video systems for telepresence in remote robotic
`control,30 where exact conformal mapping with the observer's surrounding physical environment is, however, not strictly
`necessary.
`
`285 / SPIE Vol. 2351, Telemanipulator and Telepresence Technologies (1994)
`
`Falkbuilt Ex. 1016, Page 004
`
`

`

`A summary of how some of the factors discussed thus far pertain to the seven classes of MR displays listed
`above is presented in Table 1. The first column encompasses the major distinction separating the left and
`right portions of Fig. 1, that is, respectively, whether the substratum defining the principal scene being
`presented derives from a real (R) or a computer generated (CG) world. It says nothing, however, about the
`hardware used to display that scene to the observer. That distinction is made in the second column, where
`we immediately note that there is no strict correspondence with column 1. A direct view here refers to the
`case in which the principal world is viewed directly, through either air or glass (otherwise known as
`"unmediated reality")3, whereas for the opposite case of non-direct viewing, the world must be scanned by
`some means, such as a video camera, laser or ultrasound scanner, etc., and then resynthesised, or
`reconstructed, by means of some medium such as a video or computer monitor.2
`The question addressed in the fourth column, of whether or not a strict conformal mapping is necessary, is
`closely related to the exocentric / egocentric distinction shown in the third column. However, whereas all
`systems in which conformal mapping is required must necessarily be egocentric, the converse is not the
`case for all egocentric systems.
`
`Real (R)
`or CG
`world?
`
`Direct (D) or
`Scanned (S)
`view of substrate ?
`
`Exocentric (EX) or
`Egocentric (EG)
`Reference?
`
`Conformal
`Mapping (1:1),
`or not (1:k) ?
`
`1:k
`
`1:k
`
`1:1
`
`1:1
`
`1:k
`
`1:k
`
`1:1
`
`EX
`
`EG
`
`EG
`
`EG
`
`EX
`
`EG
`
`EG
`
`Class of MR System
`1. Monitor-based video,
`with CG overlays
`2. HMD-based video,
`with CG overlays
`3. HMD-based optical
`ST, with CG overlays
`4. HMD-based video
`ST, with CG overlays
`5. Monitor/CG-world,
`with video overlays
`6. HMD/CG-world,
`with video overlays
`7. CG-based world,
`with real object
`intervention
`Table 1: Some major differences between classes of Mixed Reality (MR) displays.
`
`R
`
`R
`
`R
`
`R
`
`CG
`
`CG
`
`CG
`
`S
`
`S
`
`D
`
`S
`
`S
`
`S
`
`D, S
`
`Perhaps the most important message to derive from Table 1 is that no two rows are identical. Consequently,
`even though limited scope comparisons between pairs of Mixed Reality displays may yield simple
`distinctions, a global framework for categorising all possible MR displays is much more complex. This
`observation underscores the need for an efficient taxonomy of MR displays, both for identifying the key
`
`286 / SPIE Vol. 2351, Telemanipulator and Telepresence Technologies (1994)
`
`Falkbuilt Ex. 1016, Page 005
`
`

`

`dimensions that can be used parsimoniously to distinguish all candidate systems and for serving as a
`framework for identifying common research issues that span the breadth of such displays.
`
`5. A TAXONOMY FOR MIXING REAL AND VIRTUAL WORLDS
`The first question to be answered in setting up the taxonomy is why the continuum presented in Fig. 1 is
`not sufficient for our purposes as is, since it clearly defines the concept of AR displays and distinguishes
`these from the general class of AV displays, within the general framework of Mixed Reality. From the
`preceding section, however, it should be clear that, even though the RV continuum spans the space of MR
`options, its one dimensionality is too simple to highlight the various factors which distinguish one AR/AV
`system from another.
`What is needed, rather, is to create a taxonomy with which the principal environment, or substrate, of
`different AR/AV systems can be depicted in terms of a (minimal) multidimensional hyperspace. Three (but
`not the only three) important properties of this hyperspace are evident from the discussion in this paper:
`• Reality; that is, some environments are primarily virtual, in the sense that they have been created
`artificially, by computer, while others are primarily real.
`•
`Immersion; that is, virtual and real environments can each be displayed without the need for the
`observer to be completely immersed within them.
`• Directness; that is, whether primary world objects are viewed directly or by means of some electronic
`synthesis process.
`The three dimensional taxonomy which we propose for mixing real and virtual worlds is based on these
`three factors. A detailed discussion can be found elsewhere2; we limit ourselves here to a summary of the
`main points.
`5.1 Extent of World Knowledge
`Many of the discussions of reality and virtuality in the literature centre strictly on virtual environments,
`typically on the means by which one can depict virtual objects using graphic techniques which are of
`sufficiently high quality to make those virtual objects appear 'real'.3 Others deal with the distinction more
`multidimensionally, by focusing also on the factors which allow one to feel present within and influence a
`remote world.31,32,33 One important related consideration, which is of great practical importance for
`determining the operational capabilities of many display systems but is nevertheless often overlooked, is the
`Extent of World Knowledge (EWK). In simple terms, EWK defines how much we actually know about
`objects and the world in which they are displayed.
`The EWK dimension is depicted in Fig. 2. At one extreme, on the left, is the case in which nothing is
`known about the (remote) world being displayed. This end of the continuum characterises unmodelled data
`obtained from images of scenes that have been 'blindly' scanned and synthesised via non-direct viewing. It
`also pertains to directly viewed real objects in see-through displays. In the former instance, even though
`such images can be digitally enhanced, no information is contained within the knowledge base about the
`contents of those images. The unmodelled world extremum describes for example the class of video
`displays found in most current telemanipulation systems, especially those that must be operated in such
`unstructured environments as underwater exploration and military operations. The other end of the EWK
`dimension, the completely modelled world, defines the conditions necessary for displaying a totally virtual
`world, in the 'conventional' sense of VR, which can be created only when the computer has complete
`knowledge about each object in that world, its location within that world, the location and viewpoint of the
`observer within that world and, when relevant, the viewer's attempts to change that world by manipulating
`objects within it.
`
`287 / SPIE Vol. 2351, Telemanipulator and Telepresence Technologies (1994)
`
`Falkbuilt Ex. 1016, Page 006
`
`

`

`1 2 4 3
`
`Where / What
`
`Where
`+What
`
`1-----------1
`
`World
`Unmodelled
`
`World Partially Modelled
`
`5 7 6
`
`World
`Completely
`Modelled
`
`Extent of World Knowledge (EWK)
`Figure 2: Extent of World Knowledge (EWK) dimension
`
`Although both extrema occur frequently, the region covering all cases in between governs the extent to
`which real and virtual objects can be merged within the same display. In Figure 2, three types of subcases
`are shown: Where, What, and Where + What. Whereas in some instances we may know where an object is
`located, but not know what it is, in others we may know what object is in the scene, but not where it is.
`And in some cases we may have both 'where' and 'what' information about some objects, but not about
`others. This is to be contrasted with the completely unmodelled case, in which we have no 'where' or
`'what' information at all, as well as with the completely modelled case, in which we possess all 'where'
`and 'what' information.
`The practical importance of these considerations to Augmented Reality systems is great. Usually it is
`technically quite simple to superimpose an arbitrary graphic image onto a real-world scene, which is either
`directly (optical-ST) or non-directly (video-ST) viewed. However, for practical purposes, to make the
`graphical image appear in its proper place, for example as a wireframe outline superimposed on top of a
`corresponding real-world object, it is necessary to know exactly where that real-world object is (within an
`acceptable margin of error) and what its orientation and dimensions are. For stereoscopic systems, this
`constraint is even more critical. This is no simple matter, especially if we are dealing with unstructured, and
`completely unmodelled environments. Conversely, if we presume that we do know what and where all
`objects are in the displayed world, one must question whether an augmented reality display is really the
`most useful one, or whether a completely virtual environment might not be better.
`In our laboratory we view Extent of World Knowledge considerations not as a limitation of Augmented
`Reality technology, but in fact as one of its strengths. That is, rather than succumbing to the constraints of
`requiring exact information in order to place CG objects within an unmodelled (stereo)video scene, we
`instead use human perception to "close the loop" and exploit the virtual interactive tools provided by our
`ARGOS system, such as the virtual stereographic pointer19, to make quantitative measurements of the
`observed real world. With each measurement that is made, we are therefore effectively increasing our
`knowledge of that world, and thereby migrating away from the left hand side of the EWK axis, as we
`gradually build up a partial model of that world. In our prototype virtual control system24,25, we also create
`partial world models, by interactively teaching a telemanipulator important three dimensional information
`about volumetrically defined regions into which it must not stray, objects with which it must not collide,
`bounds which it is prohibited to exceed, etc.
`To illustrate how the EWK dimension might relate to the other classes of MR displays listed above, these
`have been indicated across the top of Fig. 2. Although the general grouping of Classes 1-4 towards the left
`and Classes 5-7 towards the right is reliable, it must be kept in mind that this ordering is very approximate,
`not only in an absolute sense, but also ordinally. That is, as discussed above, by using AR to interactively
`specify object locations, progressive increases in world knowledge can be obtained, so that a Class 1
`display might be moved significantly to the right in the figure. Similarly, in order for a Class 3 display to
`
`288 / SPIE Vol. 2351, Telemanipulator and Telepresence Technologies (1994)
`
`Falkbuilt Ex. 1016, Page 007
`
`

`

`provide effective conformal overlays, for example, a significant amount of world knowledge is necessary,
`which would also move Class 3 rightwards in the figure.
`5.2 Reproduction Fidelity
`The remaining two dimensions also deal with the issue of realism in MR displays, but in different ways: in
`terms of image quality and in terms of immersion, or presence, within the display. It is interesting to note
`that this approach is somewhat different from those taken by others. Both Sheridan's31 and Robinett's33
`taxonomies, for example, focus on the feeling of presence as the ultimate goal. This is consistent as well
`with the progression in "realspace imaging" technology outlined in Naimark's taxonomy3, towards
`increasingly more realistic displays which eventually make one feel that one is participating in "unmediated
`reality". In our taxonomy we purposely separate these two dimensions, however, in recognition of the
`practical usefulness of some high quality visual displays which nevertheless do not attempt to make one feel
`within the remote environment (e.g. Class 1), as well as some see-through display situations in which the
`viewer in fact is already physically immersed within the actual real environment but may be provided with
`only relatively low quality graphical aids (e.g. Classes 3 and 4).
`
`4 1 2 5 6 7
`Stereoscopic
`Colour
`Video
`Video
`Visible
`Surface
`Imaging
`
`High
`Definition
`Video
`Ray
`Tracing,
`Radiosity
`
`Conventional
`(Monoscopic)
`Video
`
`Simple
`Wireframes
`
`Shading,
`Texture,
`Transparency
`Reproduction Fidelity (RF)
`Figure 3: Reproduction Fidelity (RF) dimension.
`
`3
`
`3D HDTV
`
`Real-time,
`Hi-fidelity,
`3D Animation:
`Photorealism
`
`The elements of the Reproduction Fidelity (RF) dimension are illustrated in Fig. 3. The term "Reproduction
`Fidelity" refers to the relative quality with which the synthesising display is able to reproduce the actual or
`intended images of the objects being displayed. It is important to point out that this figure is actually a gross
`simplification of a complex topic, and in fact lumps together several classes of factors, such as display
`hardware, signal processing and graphic rendering techniques, etc., each of which could in turn be broken
`down into its own taxonomic elements.
`In terms of the present discussion, it is important to realise that the RF dimension pertains to reproduction
`fidelity of both real and virtual objects. The reason for this is not only because many of the hardware
`issues, such as display definition, are related. Even though the simplest graphic displays of virtual objects
`and the most basic video images of real objects are quite distinct, the converse is not true for the high
`fidelity extremum. In Fig. 3 the ordering above the axis is meant to show a rough progression, mainly in
`hardware, of video reproduction technology. Below the axis the progression is towards more and more
`sophisticated computer graphic modelling and rendering techniques. At the right hand side of the figure, the
`'ultimate' video display, denoted here as 3D HDTV, might be just as close in quality as the 'ultimate'
`graphic rendering, denoted here as "real-time, hi-fidelity 3D animation", both of which approach
`photorealism, or even direct viewing of the real world.
`The importance of Reproduction Fidelity for the MR taxonomy goes beyond having world knowledge for
`the purpose of superimposing modelled data onto unmodelled data images, or vice versa, as discussed
`
`289 / SPIE Vol. 2351, Telemanipulator and Telepresence Technologies (1994)
`
`Falkbuilt Ex. 1016, Page 008
`
`

`

`290 / SPIE Vol. 2351, Telemanipulator and Telepresence Technologies (1994)
`
`towards the right.
`multiscopic imaging), and Classes 6, 7, 2, 4 and 3, all of which are based on an egocentric metaphor,
`displays situated towards the left of the EPM axis, followed by Class 5 displays (which more readily permit
`indicated in Fig. 4, a generalised ordinal ranking of the display classes listed above might have Class 1
`conformality of the mapping of augmentation onto the background environment, as shown in Table 1. As
`exocentric vs egocentric differences between MR classes, while taking into account the need for strict
`The importance of the EPM dimension in our MR taxonomy is principally as a means of classifying
`metaphors depicted below the axis.
`in Fig. 4 is shown the progression of display media necessary for realising the corresponding presence
`which the observer's sensations are ideally no different from those of unmediated reality.3 Above the axis
`into the world from a single fixed monoscopic viewpoint, up to the metaphor of "realtime imaging", by
`EPM the axis spans a range of cases extending from the metaphor by which the observer peers from outside
`tends towards an extremum which ideally is indistinguishable from viewing reality directly. In the case of
`In some sense the EPM axis is not entirely orthogonal to the RF axis, since each dimension independently
`
`Figure 4: Extent of Presence Metaphor (EPM) dimension
`
`Extent of Presence Metaphor (EPM)
`
`•
`
`Imaging
`Realtime
`
`Surrogate
`
`Travel
`
`6 7 2 4 3
`
`l
`
`Imaging
`Panoramic
`
`J
`
`Imaging
`Multiscopic
`
`HMD's
`
`Screen
`Large
`
`1 5
`
`Imaging
`Monoscopic
`Based (WoW)
`
`Monitor
`
`and Class 5 AV-type displays.
`with a strong presence metaphor, such as Class 2, 3, 4 and 7 displays, to important exocentric Class 1 AR-
`this dimension we recognise the fact that Mixed Reality displays can range from immersive environments,
`that is, the extent to which the observer is intended to feel "present" within the displayed scene. In including
`The third dimension in our taxonomy, depicted in Fig. 4, is the Extent of Presence Metaphor (EPM) axis,
`5.3 Extent of Presence Metaphor
`
`represents the ultimate in fidelity: directly viewed reality.
`indicated on Fig. 3. Note that Class 3 has been placed far to the right in the figure, since optical see-through
`axis will depend on the particular technical implementation, a (very) approximate ordering is nevertheless
`distinguish the seven MR display classes as clearly as in Fig. 2, since the location of any one system on this
`directly viewed real objects, or with non-directly viewed images of real objects.) Although it is difficult to
`environment and of the overlaid objects. (It will also depend on whether CG images must be blended with
`real images while keeping them distinct, will depend greatly on the fidelity both of the principal
`above. The ultimate ability to blend CG images into real-world images or, alternatively, to overlap CG and
`
`Falkbuilt Ex. 1016, Page 009
`
`

`

`6. CONCLUSION
`In this paper we have discussed Augmented Reality (AR) displays in a general sense, within the context of
`a Reality-Virtuality (RV) continuum, which ranges from completely real environments to completely virtual
`environments, and encompasses a large class of displays which we term "Mixed Reality" (MR).
`Analogous, but antithetical, to AR within the class of MR displays are Augmented Virtuality (AV) displays,
`into whose properties we have not delved deeply in this paper.
`MR displays are defined primarily by means of seven (non-exhaustive) examples of existing display
`concepts in which real objects and virtual objects are displayed together. Rather than relying on the
`comparatively obvious distinctions between the terms "real" and "virtual", we have probed deeper, and
`posited some of the essential factors which distinguish different Mixed Reality display systems from each
`other: Extent of World Knowledge (EWK), Reproduction Fidelity (RF) and Extent of Presence Metaphor
`(EPM). One of our main objectives in presenting this taxonomy has been to clarify a number of terminology
`issues, in order that apparently unrelated developments being carried out by, among others, VR developers,
`computer scientists and (tele)robotics engineers can now be placed within a single frame