/Proceeding§
`1997§ymposium on
`Interactive 3D Graphics
`
`Providence, R.l., April 27 — 30, 1997
`
`Symposium Chair
`
`Andy van Dam, Brown University
`
`Program Co-Chairs
`
`Michael Cohen, Microsoft
`David Zeltzer, David Sarnoff Research Center
`
`Program Committee
`
`Kurt Akeley, Silicon Graphics
`Fred Brooks, Jr., University of North Carolina
`Ingrid Carlbom, Lucent Technologies
`Ed Catmull, Pixar
`Frank Crow, Interval Research
`Jessica Hodgins, Georgia Institute‘of Technology
`Fred Kitson, Hewlett Packard
`Marc Levoy, Stanford University
`Dan Ling, Microsoft
`Peter Schroder, California Institute of Technology
`Susumu Tachi, University of Tokyo
`Michael Zyda, Naval Postgraduate School
`
`Supporting Organizations
`
`Autodesk, Inc.
`Cyberware
`David Sarnoff Research Center
`Fraunhofer CRCG
`Hewlett-Packard
`Interval Research
`Microsoft
`Mitsubishi Electric Research Labs
`The National Science Foundation
`
`The NSF Science & Technology Center for Computer Graphics & Scientific Visualization
`Office of Naval Research
`Pixar Animation Studios
`
`Sense8 Corporation
`Silicon Graphics, Inc.
`Sun Microsystems
`US. Army Research Laboratory
`
`Proceedings Production Editor
`
`Stephen N. Spencer, The Ohio State University
`
`Sponsored by the Association for Computing Machinery’s
`Special Interest Group on Computer Graphics (SIGGRAPH)
`
`1
`
`MEDIATEK, EX. 1014, Page 1
`IPR2018-00101
`
`MEDIATEK, Ex. 1014, Page 1
`IPR2018-00101
`
`

`

`The Association for Computing Machinery, Inc.
`1515 Broadway
`New York, New York 10036
`
`Copyright © 1997 by the Association for Computing Machinery, Inc. (ACM). Copying without fee is permitted provided that copies are not
`made or distributed for prom or commercial advantage and credit to the source is given. Abstracting with credit is permitted. To copy oth—
`erwise, to republish, to post on servers or to distribute to lists,,requires prior specific permission and/or a fee. For other copying of articles
`that carry a code at the bottom of the first or last page, copying is permitted provided that the per-copy fee indicated in the code is paid
`through the Copyright Clearance Center, 222 Rosewood Drive, Danvers, MA 01923. Request permission to republish from: Publications
`Dept, ACM, InC., Fax +1-212-869-0481 or e-mail permissionseacm.org
`
`Orders from ACM Members:
`
`A limited number of copies are available from
`ACM Member Services. Send all inquiries, or
`order with payment in US. dollars to:
`
`USA. and Canada:
`ACM Order Department
`PO. Box 12114
`Church Street Station
`New York, NY 10257
`Telephone: +1-800-342-6626
`Telephone: +1-212-626-0500
`Fax: +1-212-944-1318
`E-mail: orderseacm.org
`URL: http: //www.acm. org/
`’
`
`All other countries:
`ACM European Service Center
`108 Cowley Road
`Oxford OX4 ”F
`United Kingdom
`Telephone: +44-l-865-382338
`Fax: +44-1-865-38l338
`E-mail: acm_europe@acm.org
`
`Please include the ACM order number in all
`inquiries.
`
`ACM Order Number: 429973
`ACM ISBN: 0-89791-884-3
`
`M.|.T. LIBRARIES
`
`1
`
`JUN 1 8 1997
`
`
`
`
`
`RECEIVEDW
`
`2
`
`MEDIATEK, EX. 1014, Page 2
`
`IPR2018-00101
`
`MEDIATEK, Ex. 1014, Page 2
`IPR2018-00101
`
`

`

`Preface ..................................................................................5
`
`Table of Contents
`
`Keynote Address
`Bill Buxton
`
`Image Based Rendering
`Chair: Kurt Akeley
`
`Post-Rendering 3D Warping ..................................................................7
`William R. Mark, Leonard McMillan, Gary Bisho
`Color Plate ........................... . .................................................. 180
`
`Time Critical Lumigraph Rendering ........................................................... 17
`Peter—Pike Sloan, Michael F. Cohen, Steven J. Gortler
`Color Plate ............................................................................. 181
`
`Navigating Static Environments Using Image-Space Simplification and Morphing ........................25
`Lucia Darsa, Bruno Costa Silva, Amitabh Varshney
`,
`Color Plate ............................................................................. 182
`
`Manipulation and Motion in VR
`Chair: Frederick Brooks, Jr.
`
`An Evaluation of Techniques for Grabbing and Manipulating Remote Objects in
`Immersive Virtual Environments ..............................................................35
`Doug A. Bowman, Larry F. Hodges
`Color Plate ............................................................................. 182
`
`Image Plane Interaction Techniques In 3D lmmersive Environments ...................................39
`Jeffrey S. Pierce, Andrew S. Forsberg, Matthew J. Conway, Seung Hang,
`Robert C. Zeleznik, Mark R. Mine
`Color Plate ............................................................................. 183
`
`Providing a Low Latency User Experience In A High Latency Application ..............................45
`Brook Conner, Loring Holden
`Color Plate ............................................................................. 184
`
`Managing Latency in Complex Augmented Reality Systems .........................................49
`Marco C. Jacobs, Mark A. Livingston, Andrei State
`Color Plate ............................................................................. 185
`
`Rendering Complex Models I
`Chair: Peter Schroder
`
`View—Dependent Culling of Dynamic Systems in Virtual Environments .................................55
`Stephen Chenney, David Forsyth
`
`Multi-Pass Pipeline Rendering: Realism For Dynamic Environments ...................................59
`Paul J. Diefenbach, Norman 1. Badler
`Color Plates ......................................................................... 186, 187
`
`Efficient Radiosity Rendering using Textures and Bicubic Reconstruction ...............................71
`Rui Bastos, Michael Goslin, Hansong Zhang
`Color Plate ............................................................................. 184
`
`Model Simplification Using Vertex-Clustering ....................................................75
`Kok-Lim Low, Tiow-Seng Tan
`Color Plate ............................................................................. 188
`
`3
`
`MEDIATEK, EX. 1014, Page 3
`IPR2018-00101
`
`MEDIATEK, Ex. 1014, Page 3
`IPR2018-00101
`
`

`

`Providing A Low Latency User Experience In A High
`Latency Application
`
`Brook Conner
`Sony Corporation of America
`550 Madison Avenue, 5th floor
`New York, New York, 10028
`brook_conner@sonyusa.com
`
`Loring Holden
`Brown University site of the
`NSF Science and Technology Center for
`Computer Graphics and Scientific Visualization
`Providence, RI 029 12
`lsh@cs.brown.edu
`
`ABSTRACT
`
`user’s actions sluggishly).
`
`Through the use of particular visual effects, we provide a low
`latency user experience, even when extremely large latencies
`occur in an application. We demonstrate these effects in a
`wide-area distributed virtual reality application. These effects
`include the use of motion blur, transparency, and defocusing.
`While the effects incur a performance penalty, the penalty
`is predictable, unlike the lag induced by network delays.
`Thus, we provide immediate feedback to each participant,
`even when the network prevents information more useful than
`the fact that delays are occurring. When updates are finally
`received, we use the same effects to provide coherent updates
`to the user’s information, without the jarring discontinuities
`that otherwise would confuse a participant’s understanding
`of the environment.
`
`1.3.6 [Computer
`CR Categories and Subject Descriptors:
`Graphics]: Methodology and Techniques - Interaction Tech-
`niques; 1.3.2 [Computer Graphics]: Graphics Systems - Dis-
`tributed/network graphics
`
`Additional Keywords:
`.
`tual envrronments
`
`motion blur, temporal aliasing, vir-
`
`Introduction
`
`Wide areanetworks such as the Internet provide no guarantees
`about the time it takes a packet of information to reach its goal.
`A packet might be delayed for any number of reasons, includ-
`ing high network traffic, slow hardware along the particular
`route chosen, load problems at either the source or the desti-
`nation machine, or simply an unnecessarily long route. While
`provrding dedicated resources can mitigate these effects, they
`cannot be eliminated without the use of a completely closed
`network.
`
`In contrast, virtual reality applications have very stringent
`latency requirements. High latency can induce a number of
`unpleasant effects, such as simulator sickness or a loss of
`feeling of control (as when an environment responds to a
`
`Pemtrssron to make digital/hard copies ofall or part ofthis material for
`personal or classroom use is granted without fee provided that the copies
`are not made or distributed for profit or commercial advantage, the copy-
`right notice, the title of the publication and its date appear, and notice is
`EVen tigzlrt :opyright is by permission ofthe ACM, Inc. To copy otherwise
`reurs,toos
`.'
`s
`'
`pemfissmn and/2r ftetem servers or to redistribute to lists, requrres specrfic
`1997 Symposium on Interactive 3D Graphics, Providence RI USA
`Copyright 1997 ACM 0-89791-884-3/97/04 ..$3.50
`
`,
`
`Increasingly virtual reality applications are becoming net-
`worked over general-purpose LANs [5, 7] making the con—
`flicting constraints of networks and virtual reality a growing
`problem.
`
`Prior Work
`
`Some researchers have attempted to address latency in dis-
`tributed VR at a system level [5, 7]. Others have attempted to
`increase a user’s understanding of a complex scene through
`the use of visual effects, such as motion blur and cartoon
`idioms such as squash and stretch [2, 3].
`In contrast, our
`work uses visual effects to increase the user’s understanding
`of latency as it occurs, thereby mitigating the effects of that
`latency. Before examining our own techniques, let’s consider
`both the system-based techniques addressing latency and the
`visual effects addressing complexity.
`
`Addressing Latency at a System Level
`In the past, distributed VR applications have used two com-
`plementary techniques to address problems of latency: dead
`reckoning and decoupling.
`
`Dead reckoning addresses late arrival of motion information.
`By using derivatives of earlier motion or derivative informa-
`tion provided by an object itself, the dead reckoning system
`calculates the position of an object locally, without needing
`to wait for the arrival of the actual information [5]. This
`can clearly introduce errors when the dead reckoning system
`uses a derivative whose own derivative is non-zero (such as
`the derivatives of objects under user control).
`
`More advanced dead reckoning systems use a Kalman filter
`[1] to attempt to predict where an object will be in the future.
`This technique also introduces errors, especially when the
`model of the object’s motion used by the Kalman filter is a
`poor one.
`
`Decoupling in VR systems is the process of making the sys-
`tem multi-threadcd, with certain threads assigned to tasks
`most in need of low latency [7]. Typically, these threads are
`Unix processes provided with dedicated hardware resources
`to guarantee a minimum level of performance. Non-critical
`tasks are serviced as resources become available. This ap-
`proach can guarantee, for example, that a new frame will be
`generated to track user head motion, but the contents of the
`environment (for example, balls moving under a kinematics
`simulation) may be changed at a slower rate. This technique
`
`
`
`45
`
`MEDIATEK, Ex. 1014, Page 4
`
`IPR2018-00101
`
`MEDIATEK, Ex. 1014, Page 4
`IPR2018-00101
`
`

`

`can result in jerky or unpredictable motion by the objects in
`the scene, making it harder for the user to comprehend what
`is happening.
`
`Visual Effects to Increase Comprehension of Complex
`Scenes
`
`We are using particular visual effects that have been used in
`the past to make complex scenes more understandable. By
`demonstrating visually that something complex is happening,
`a user can have a greater understanding of the scene.
`
`[4] focuses on this approach.
`The work of Brenda Laurel
`Other work, such as John Lasseter
`[3], demonstrates the
`use of traditional animation techniques in order to explain
`a scene as clearly as possible. This has been applied in a
`programming environment by the Self Project
`[2]. Objects
`in the Self environment draw attention to themselves with
`animated wiggles and expressive paths. When they move
`fast, they motion-blur to indicate a continuous motion, not a
`discrete change of location.
`
`Other work has developed techniques for producing the ef-
`fects of motion blur [11] and defocusing [6] in a fully three—
`dimensional, real-time environment, such as virtual reality.
`This work showed that rich visual effects could be achieved
`
`with relatively little cost in computation time or rendering
`performance.
`
`We use the effects and the concepts developed by these re-
`searchers to address a new problem, high latency. As ex-
`plained above, high latency can produce many effects that a
`user does not expect. With these techniques, we can show
`the user that latency-induced effects are occurring, and that
`as a result object behavior might be somewhat different from
`what was anticipated.
`
`Visual Effects to Reduce the Perception of Latency
`Let us now look at our particular effects and how they mitigate
`the user’s perception of lag. First, we discuss the actions that
`include large sources of lag. Second, we discuss the particular
`effects we use and in what situations we use them. Third,
`we briefly discuss how a range of possible effects might be
`applied in a consistent visual language of lag and its effects.
`
`Some of the User Actions Affected by Latency
`We focus on two aspects of interacting in a shared virtual
`world: actions initiated by a participant and actions observed
`by a participant.
`In both cases, we work with solely those
`cases where unbounded network lag can cause extremely
`undesirable effects, such as a frozen world. For example,
`navigating through a scene, while initiated by a participant, is
`generally not a source of unbounded network lag, as naviga-
`tion is typically handled locally. Thus, we make no attempt
`to mitigate the effects of lag on navigation, though there is
`lag present and our effects could be reformulated perhaps to
`address that lag.
`
`Typical actions initiated by a participant include grabbing an
`object and moving it through a scene. Grabbing a shared ob-
`ject is a source of network-based lag, as a shared object must
`be controlled by a mutually exclusive lock. Acquiring such
`a lock requires a network round trip (unless the lock coinci-
`dentally resides on the same machine as the participant who
`wishes to take the lock - a situation which is generally not
`true). A user interface that does not allow the user to manip-
`
`ulate an object until a lock is granted could have disturbing
`pauses. If the pauses were long enough, they would prevent
`the user from feeling in control of the object. Conversely, if
`the U1 lets the user manipulate the object before the lock is
`granted, the user will lose control once the system realizes
`that the user does not in fact have permission to modify the
`object.
`
`Some have suggested that social conventions can suffice in
`place of a lock, such as a convention that users not grab objects
`that others are already holding. However, in a collaborative
`environment these social conventions often break down. For
`
`example, in the midst of exciting collaboration, participants
`may lose track of social graces (as when two people in a
`physical room get excited about a project or get into a heated
`argument).
`
`The actions initiated by other participants are also a source of
`lag, since a remote participant’s actions necessarily require
`the potentially high-lag network transmission of information.
`An avatar might appear to move slowly or not at all, simply
`due to poor network performance. Objects being controlled
`by other participants suffer from similar problems.
`
`Transparency, Motion Blur, and Defocusing
`We address the problems of the users’ experience with real-
`time transparency, real-time motion blur, and real-time defo-
`cusing (e.g., per-object depth-of-field effects). Although all
`of these effects take more time to render, usually requiring
`multiple rendering passes,
`they can still be done by stan-
`dard 3D graphics libraries and accelerators quickly enough
`to maintain the frame rate required by a VR application.
`
`In the Macintosh Finder [10] the user clicks on an icon, and
`as she drags the icon across the screen, an outline of the icon
`(or a transparent copy in more recent versions) follows the
`pointer. When the user "drops" the icon, the transparent ver-
`sion becomes opaque and the old opaque version disappears.
`
`transparency can be used when manipulating a
`Similarly,
`shared object whose lock has not yet been acquired. Two
`copies of the object are used, one that follows the user’s
`manipulations and one that remains where the object will be
`if the lock fails (see color plate, left). Both objects are drawn
`transparent, indicating that both are tentative. If the lock is
`acquired, the copy the user is manipulating becomes solid
`and the other copy fades out (see color plate, bottom right).
`If the lock is not acquired, the copy the user is manipulating
`fades out while the other copy becomes solid (see color plate,
`top right). We call this technique "ghost locking".
`
`Ghost locking makes the user aware of many aspects of ma-
`nipulating a shared object in a networked environment. The
`user has an immediate response to her actions, affording a
`feeling of control over the environment. She also sees im-
`mediately that lag is occurring, because of the appearance
`of a "partial" or transparent object. Finally, fading out the
`copy that lost the lock gives the user time to understand the
`changed state of the manipulated object.
`
`This effect can be customized by varying levels of opacity in
`either the grabbed object or the original object, depending on
`outside knowledge about the state of the lock or the likelihood
`that the lock will be taken successfully. For example, a lock
`is more likely to fail for a highly contended object.
`In this
`
`46
`
`MEDIATEK, Ex. 1014, Page 5
`
`IPR2018-00101
`
`MEDIATEK, Ex. 1014, Page 5
`IPR2018-00101
`
`

`

`case, the grabbed object can be more transparent, while the
`remaining object is more opaque, showing the user the likely
`results of trying to grab a highly contended object.
`
`Ghost locking can further be customized by only having the
`"ghost" copy, with no copy at the original location. This
`may make sense when the expected lag is short and the user
`probably will not move the ghost far from the original location
`(or acquiring the lock is very likely). Ghost locking with no
`original copy lessens clutter, but removes the reference to the
`original location. Without this reference object, lock denial
`may be more disruptive.
`
`When an object is manipulated, motion blur can be used
`to show that the motion is a continuous path, even when
`network updates are late in arriving (in which situation many
`VR applications discard all but the latest update). Motion
`blur can similarly be used when correcting an object that has
`moved to an incorrect position because of dead reckoning.
`
`A blur effect using the real-time motion blur effect can be
`integrated with ghost locking. The grabbed object can be
`blurred back to its original (producing a taffy-like look), until
`the lock is acquired, when the original blurs to the manipu-
`lated position. If the lock is not acquired, the "taffy" snaps
`back into its original position.
`
`Defocusing can be applied in all situations when unknown
`delays are occurring. When a response is expected from
`an object (or an avatar) and none is forthcoming, the object
`that should be responding can be defocused. Note that this
`integrates nicely with earlier work, such as dead-reckoning -
`an object may be moving under dead-reckoning for too long,
`providing too much error. A blurry moving object tells the
`user that the object’s position may in fact not be correct.
`
`Objects can be defocused more as more delays occur. Even-
`tually, a non-responsive object can fade out entirely. When
`integrated with motion blurring and transparency manipula-
`tion, defocusing can inform a participant when unanticipated
`delays are occurring and how severe the delays are. This pro-
`vides a participant with information about when an avatar or
`object is non-responsive, as opposed to unchanging.
`
`Applying Visual Effects with a Consistent Visual Lan-
`guage
`
`These visual effects have many parameters. The level of
`transparency or blurring can be varied in all of them, and
`some have alternative visual representations. Exactly which
`parameter and style should be used depends on the exact situ-
`ation. For example, making grabbed objects transparent may
`not be appropriate if most objects in the scene are transparent
`- using wireframe w0uld be more suitable.
`
`While we have not performed user studies, we have found that
`a consistent choice of parameters and styles can effectively
`produce a visual language of the effects of lag in a system.
`Cloudiness as seen in blur effects visually tells the user that
`the situation is uncertain in ways that precise rendering styles
`like motion lines do not. We feel that using motion blur to
`Suggest continuous motion, the "taffy" effect for manipulating
`{ Objects, and defocusing when delays occur is a particularly
`'7»,
`good example of this kind of visual language.
`
` at
`‘
`
`Implementation Notes
`
`These effects were implemented using Open GL and Open
`Inventor
`[8] on both an SGI Reality Engine and an Ultra
`SPARC with a Creator 3D graphics accelerator. The effects
`are constructed as new Open Inventor node types
`[9] us-
`ing Open GL directly to render the effects. This lets these
`effects be used in a wide variety of applications, while pro-
`viding extremely good performance. We tested the nodes in
`an Inventor-based application called vrapp. This application
`supports wide-area distribution of immersive environments,
`including text, audio, and video communication, manipula-
`tion of shared objects, avatars, and support of specialized
`immersive hardware.
`
`All of the transparency or blur-based effects that we describe
`could be implemented in a straightforward way using multi-
`ple copies of the object and varying levels of transparency.
`Naively, this can be done by using Inventor’s support of mul-
`tiple instantiation of a scene graph, with varying transparency
`and transformation nodes interspersed between the instantia-
`tions. However, most of the visual techniques are amenable
`to more clever implementations that provide an improved ap-
`pearance and performance, as detailed in the papers that first
`described them.
`
`Future Work
`
`We currently have informal evidence that suggests that these
`effects make task performance in a wide area environment
`more productive. A formal user study that determines whether
`the added rendering cost of the visual effects provides a mea-
`surable user performance benefit should be performed.
`
`The exact styles and parameters that are most effective can
`also be determined through a suitable user study. It may be
`the case that some effects work best in concert with others,
`or with others but only within a certain range of parameters.
`Choosing particular values, e.g., for level of transparency, at
`this point is simply guesswork.
`
`It may be possible that the system—oriented methods for ad-
`dressing lag, such as dead—reckoning and decoupling, can be
`integrated more closely with the visual effects we describe
`here. If this is the case, it might be possible to provide the
`user with even more information about what is happening in
`the virtual world and why, without confusing her.
`
`Acknowledgements
`We would like to thank Andries van Dam and the Brown
`Graphics Group for their help, but most particularly Bob
`Zeleznik for help in porting his earlier work and in preparing
`the video and paper.
`
`We would also like to thank our sponsors: grants from NSF,
`NASA, Microsoft, Sun Microsystems and Taco; hardware
`from SGI, Hewlett-Packard, Sun.
`
`REFERENCES
`
`1. Ronald Azuma and Gary Bishop. A Frequency-Domain
`analysis of head-motion prediction.
`In Robert Cook,
`editor, SIGGRAPH 95 Conference Proceedings, Annual
`Conference Series, pages 401—408. ACM SIGGRAPH,
`Addison Wesley, August 1995. held in Los Angeles,
`California, 06-11 August 1995.
`
`
`
`47 MEDIATEK, EX. 1014, Page 6
`
`IPR2018-00101
`
`MEDIATEK, Ex. 1014, Page 6
`IPR2018-00101
`
`

`

`Bay-Wei Chang and David Ungar. Animation: From
`cartoons to the user interface. UIST Proceedings, pages
`45—55, November 1993.
`
`. John Lasseter. Principles of traditional animation ap-
`plied to 3d computer animation. Computer Graphics
`(SIGGRAPH ’87 Proceedings), 21(4):35—44, July 1987.
`
`Brenda Laurel. Computers as Theatre. Addison Wesley,
`1 991 .
`
`. M. R. Macedonia, D. P. Brutzmann, M. J. Zyda, D. R.
`Pratt, P. T. Barham, J. Falby, and J. Locke. NPSNET: A
`multi-player 3D virtual environment over the internet.
`In Pat Hanrahan and Jim Winget, editors, 1995 Sympo-
`sium on Interactive 3D Graphics, pages 93—94. ACM
`SIGGRAPH, April 1995. ISBN 0-89791-736-7.
`
`and
`Toshikazu Ohshima, Hiroyuki Yamamoto,
`Hideyuki Tamura. Gaze—directed adaptive rendering for
`interacting with virtual space. In Vinual Reality Annual
`International Symposium, pages 103—110, April 1996.
`
`. Chris Shaw, Jiandong Liang, Mark Green, and Yunqi
`Sun. The decoupled simulation model for virtual reality
`systems. Proceedings of CHI’92, pages 321—328, May
`1992.
`
`. P. Strauss and R. Carey. An object-oriented 3D graphics
`toolkit. Computer Graphics (SIGGRAPH ’92 Proceed-
`ings), 26(2):34l—349, July 1992.
`
`. Josie Wemecke. The Inventor Toolmaker. Addison-
`Wesley, 1994.
`
`10.
`
`11.
`
`G. Williams. The apple macintosh computer. Byte,
`9(2):709 — 718, 1984.
`
`Interactive
`Matthias Wloka and Robert C. Zeleznik.
`real-time motion blur. Visual Computer, pages 273 —
`295, 1996.
`
`48 MEDIATEK, Ex. 1014, Page 7
`IPR2018-00101
`
`48
`
`MEDIATEK, Ex. 1014, Page 7
`IPR2018-00101
`
`