Information Appliances
`
`Rock ’n’ Scroll Is
`Here to Stay
`
`Not missing a chance to show off the latest
`
`The Rock ’n’ Scroll input
`
`method lets users gesture to
`
`scroll, select, and command
`
`an application without
`
`resorting to buttons,
`
`touchscreens, spoken
`
`commands, or other input
`
`Joel F. Bartlett
`Compaq Computer Corporation
`
`Tilt to scroll
`For desktop devices, scrollbars have become the de
`facto control mechanism for paging through documents
`larger than the screen. When user interface designers
`use scrollbars on the smaller display of a handheld
`device, two problems arise: both horizontal and verti-
`cal scrollbars are often required, and the scrollbars occu-
`py a larger percentage of the screen. Designers can
`recover the display area used by scrollbars by allowing
`scrolling by dragging the document using a stylus or
`scrolling via a cursor key. However, all these techniques
`still require the use of both hands. One way to free a
`hand is to use the hand that holds the device to control
`scrolling. As users rotate the device in their hand about
`either or both axes away from a user-defined neutral ori-
`entation, the device senses the motion and the screen
`contents scroll appropriately, as shown in Figure 1.
`As described, scrolling is always on, just like dexteri-
`ty puzzles that you tilt to get the rolling ball in the cor-
`rect hole. This isn’t always the desired behavior, so other
`systems that have investigated “tilt to scroll” have pro-
`vided a “clutch” button to engage scrolling.3-6 When
`pressed, the device’s tilt scrolls the display; otherwise,
`scrolling is disabled.
`The clutch button seems a good solution to acciden-
`tal scrolling, but it comes at some cost. In some simple
`user tests that we conducted at the start of this investi-
`gation, users complained about having to tilt the device
`and press the button in a synchronized manner. For
`tasks like reading a column of text, users often want to
`continuously scroll and don’t want to keep their hand
`tensed to hold the button down the whole time.
`Finally, any time you provide a button, you make an
`assumption about how the user will hold the device and
`push the button. One solution to this problem is to add
`more buttons. A more elegant one is to make the whole
`device a button, where the user squeezes it to enable
`scrolling.4 Or, you can make the scrolling behavior modal.
`Rock ’n’ Scroll uses the last approach. We assume
`scrolling is the device’s normal mode and provide com-
`
`pictures of your children, you reach for
`your new photo album. As you remove it from your pock-
`et, it activates and you see a display of photograph
`thumbnails in the album. Tilting the album on either axis
`scrolls through the thumbnails until you find the pictures
`you want to show. A gentle fanning gesture zooms in on
`the first picture, then you hand the album to your friend.
`After admiring the picture, she gestures to step through
`the rest of the album. The pictures
`are in both landscape and portrait
`mode, so a simple gesture is all that’s
`required to reorient the album to
`best display them.
`Before putting it back in your
`pocket (where it will automatically
`shut down), you stop to admire the
`album itself. The album’s dimen-
`sions are that of the display with the
`addition of a thin, black border. In
`keeping with its spare, elegant
`design, it has no buttons or other vis-
`ible controls: all functions can be
`accessed by direct manipulation.
`While such an appliance has yet to
`reach the market, my colleagues and
`I have constructed a prototype that
`demonstrates the user interface we call Rock ’n’ Scroll.
`The photo album is an example of a device with an
`Embodied User Interface,1 where the control mechanism
`is directly integrated into the album’s display. The design
`of such a device draws on a long-standing interest in
`using motion sensing for user input and the realization
`that inertial sensing systems are the logical type of sys-
`tem to embed in small devices.2 While researchers have
`anticipated tilting a personal digital assistant (PDA) to
`navigate through a document, only in the last two to
`three years have the sensors become small, cheap, and
`sufficiently low power that they can be embedded in a
`handheld device and realize this vision.
`
`methods.
`
`40
`
`May/June 2000
`
`0272-1716/00/$10.00 © 2000 IEEE
`
`SCEA Ex. 1050 Page 1
`
`

`
`Roll: side to side rotation
`
`Pitch:
`fore and aft
`rotation
`
`1 As the left edge of the device is
`dipped, that is, rotated about the
`roll axis, the picture slides to the
`left of the display.
`
`2 The first frame of the sequence
`(left-to-right, top-to-bottom) shows
`the thumbnail photo. In the next
`three frames, the user fans the
`device. The fifth frame displays the
`full-size picture.
`
`mands to disable and enable scrolling and to set the
`device’s neutral orientation. However, for this behavior
`to be an improvement over a “clutch” button, the appli-
`cation designer must ensure that frequent mode switch-
`es aren’t required.
`
`Gesture to control
`The commands that control scrolling can be issued in
`any number of ways, including button presses or inter-
`action with a touchscreen. The same inertial sensors
`that control scrolling can also be used to recognize ges-
`tures as commands—this is the method we chose for
`Rock ’n’ Scroll. Levin and Yarin7 have also investigated
`this method to implement a tilt- and shake-sensitive
`interface for drawing.
`Other possible command gestures include snapping
`or shaking the device, tapping on it with the other hand,
`or fanning it. Unlike Levin and Yarin’s implementation,7
`we rejected snapping and shaking the device because of
`the strain it places on the user’s hand and wrist. The
`mass of a 4- to 6-ounce device proves sufficient to strain
`users when they repeatedly perform such a gesture. Tap-
`ping is attractive because it doesn’t require users to move
`the device, but it does have the disadvantage of requir-
`ing a second hand to operate.
`We then tried slower fanning gestures about either of
`the device axes. The gentler motion seems easier on the
`
`hand and wrist than a snap, and the longer duration of
`the gesture makes it easier to separate it from high-fre-
`quency noise like hand tremors and vehicle vibrations.
`The sequence of pictures in Figure 2 illustrates how to
`select a picture from the album with a downward fan-
`ning gesture about the roll axis. The first frame shows
`the thumbnail picture. Then the user smoothly dips and
`raises the left-hand edge of the device to display the
`desired picture.
`Three other similar gestures—upward about the roll
`axis, downward about the pitch axis, and upward about
`the pitch axis—can also be made. When combined with
`scrolling, they provide a vocabulary of commands more
`than sufficient to implement the rest of the album’s com-
`mands: step from a picture back to the menu of thumb-
`nails, step to the next picture, disable and re-enable
`scrolling, and reverse scrolling direction.
`The gestures don’t require a large amount of space to
`execute. For example, they can be performed with a user
`seated with an elbow on a chair’s armrest. When users
`hold the device as shown in Figure 2, the device motion
`is in front of them, and those seated on either side aren’t
`disturbed.
`
`Hold still to orient
`The final gesture used in the photo album is holding
`the device still for a couple of seconds, which serves two
`
`IEEE Computer Graphics and Applications
`
`41
`
`SCEA Ex. 1050 Page 2
`
`

`
`Information Appliances
`
`purposes. The first is that it reorients the device. In Fig-
`ure 3, the user moves the album from landscape to por-
`trait mode by positioning it vertically with the new top
`edge of the album up. The user holds the album still in
`that position for a couple of seconds, then the screen
`image rotates to the new orientation.
`The second purpose of the hold-still gesture is to
`change the album’s neutral orientation. To do this, the
`user first gestures to disable scrolling, then holds the
`device still in the new neutral orientation for a couple of
`
`3 As shown in
`the sequence,
`the user posi-
`tions the album
`in the desired
`orientation,
`then holds it for
`a couple of
`seconds to
`change the
`display from
`landscape to
`portrait mode.
`
`seconds, and gestures to enable scrolling. While this oper-
`ation takes three gestures to accomplish, it’s not a burden
`to users because they perform the task infrequently.
`
`Implementation
`Now that I’ve demonstrated the mechanics of control-
`ling the device, we can turn our attention towards its
`implementation. Verplaetse2 characterized the motion
`that should be expected as accelerations from 0.1 g to 1.0
`g and a frequency of motion of 8 to 12 Hz. The inertial
`detectors must also meet the needs of a personal device:
`rugged, self-contained, small, light, and low power. Final-
`ly, we need low-cost detectors to meet the manufactur-
`ing cost requirements for mass-market digital appliances.
`With these design constraints in mind we chose a two-
`axis, single-chip accelerometer—Analog Devices’
`ADXL202—to measure acceleration on each of the
`device’s axes (see Figure 1). With the addition of one
`resistor and three capacitors, we integrated it into Itsy,8
`Compaq Research’s experimental platform for “off the
`desktop” computing. Every 10 ms, the ADXL202 reports
`acceleration values in the range of - 2 g to +2 g for each
`axis. These measurements are averaged in groups of
`four, so the photo album application sees 25 sets of mea-
`surements per second. These values represent a combi-
`nation of gravitational acceleration, user motion, and
`noise. Initial noise filtering occurs by using a low-pass
`filter in the accelerometer set at 10 Hz. Exponentially
`averaging sequential results achieves additional filter-
`ing. When plotted over time, the values look something
`like Figure 4. At time t0 the user is scrolling the display
`by rocking it fore and aft. By time t1, the user is holding
`the device fairly still. At time t2, a gesture like that shown
`in Figure 2 starts and is completed at time t3.
`Figure 4 illustrates a problem with extracting both ges-
`ture and scrolling information from the same data stream.
`The user thinks the gesture starts at time t2, but the device
`doesn’t recognize it until t3. During this interval, scrolling
`continues on both axes, so the application may scroll from
`one item to the next in a menu and the gesture’s action is
`applied to the wrong object. This can be seen in Figure 2,
`where in the fourth frame, the thumbnail menu is
`scrolling because the gesture has not yet been recognized.
`To correctly associate the gesture with the right device
`state, the scrolling code is speculatively executed by sav-
`ing the time and current scrolling state before each
`change occurs. When the system recognizes a gesture at
`
`4 Acceleration versus time for each
`axis of motion is plotted here. Each
`signal is labeled and shown inde-
`pendently, with the acceleration
`value on the vertical axis and time
`on the horizontal axis. The vertical
`gray bars are at one-second inter-
`vals, and dotted vertical lines
`denote points of interest.
`
`Roll
`
`PItch
`
`42
`
`May/June 2000
`
`t0
`
`t1
`
`t2
`
`t3
`
`SCEA Ex. 1050 Page 3
`
`

`
`t3, the system restores itself to the
`state at t2 (shown in the first frame
`of Figure 2), then applies the ges-
`ture’s operation. The scrolling state
`need not be saved indefinitely, since
`a maximum length of a gesture can
`be defined.
`
`But will it work?
`Before implementing Rock ’n’
`Scroll for Itsy, we constructed a pro-
`totype system to test the ideas. The
`prototype system appears to the
`user as a handheld computer with a
`cable leading out of it. It consists of
`a small video liquid crystal display
`(LCD) with a two-axis accelerome-
`ter attached to the back. The
`accelerometer was made from two
`earlier generation (Analog Devices
`ADXL05) one-axis accelerometers
`and included a PIC16C73A micro-
`controller to convert the analog
`outputs from the ADXL05s into dig-
`ital values reported 25 times a sec-
`ond to a PC via an RS-232 serial
`line. The measurements drive a PC
`application whose output to the
`screen is converted to a National
`Television System Committee
`(NTSC) video signal via a scan con-
`verter, then displayed on the LCD.
`The prototype is decidedly inferior
`in handling to an Itsy, since it is larg-
`er and heavier, its screen has a nar-
`rower viewing angle, and it is
`tethered to a PC.
`Applications constructed include a viewer allowing
`users to scroll through a number of pictures and text
`images, a game where a cat catches mice, and an inter-
`active map of our building. The map application lets
`users tilt to browse a floor and gesture to change floors,
`set the device’s neutral orientation, or disable or enable
`scrolling. Fifteen members of Compaq’s Palo Alto
`research labs participated in a scripted trial that took
`about twenty minutes per person. During the trial the
`user stood and manipulated the test system under my
`instruction.
`The most important goal of the study was to confirm
`(or reject) the premise that the Rock ’n’ Scroll user inter-
`face can be rapidly mastered by a larger user commu-
`nity. Test users first familiarized themselves with the
`device by scrolling text and pictures. With little prac-
`tice, they were able to start and stop scrolling, find spe-
`cific articles in the text, and center selected items in a
`picture on the screen. Next, we demonstrated the map
`application, then the users tried it. A subjective mea-
`sure of their ability to operate the application was then
`made based on the three to five minutes that they used
`the application. Three quarters of the users found it
`easy to operate all features of the application. One quar-
`ter of the users had moderate difficulty making one or
`
`5 In the frame on the top, the
`picture is in the initial position.
`When the left edge of the device is
`dipped, seven of the fifteen subjects
`expected gravity to “slosh” the
`picture into the position it has in
`the center frame, and five of the
`fifteen subjects expected to see the
`picture like the bottom frame
`because they “pointed” with the
`left edge of the device to that
`corner of the picture.
`
`more of the types of gestures. With some coaching and
`adjusting of the gesture recognizer, all users were able
`to make all types of gestures.
`The more interesting result though, has to do with
`user expectations about how the device would scroll. As
`the prototype hardware and software were being con-
`structed, we observed that users have different (and
`often strong expectations) about which way the display
`should scroll when they tilted the device. For example,
`when the left edge of the device is dipped, should the
`picture on the device slide left or right? We designed the
`user study to examine this and found that while a
`default sensitivity and scrolling speed was acceptable, a
`default scrolling direction was not. For each user, the
`initial scrolling behavior was randomly selected. As the
`tests were run, the users were allowed to change
`scrolling behavior to a preferred setting, while the exper-
`imenter encouraged them to try the alternative as well.
`All but two test subjects immediately chose a preferred
`direction and maintained it across the entire test. The
`results, summarized in Figure 5, suggest that the world
`is divided into “sloshers” and “pointers” who expect the
`device to behave in opposite ways.
`However, when presented with a simple game, all sub-
`jects but one expressed a strong preference for the “slosh”
`
`IEEE Computer Graphics and Applications
`
`43
`
`SCEA Ex. 1050 Page 4
`
`

`
`Information Appliances
`
`6 Cat and
`mouse game.
`
`7 Doom on
`Itsy.
`
`8 Industrial
`glove with
`hands-free
`thermometer
`(mock-up).
`
`85°
`
`mode of operation. The game, shown in Figure 6, dis-
`played a cat on the screen. The user tilted the device to
`move the cat around to catch mice. Those who favored
`the “slosh” mode of operation in the earlier parts of the
`test explained that when they dipped the left edge of the
`device, they expected the cat to go left. Those who
`“point” explained that they were pointing to where they
`wanted the cat to go. Though the test subjects had dif-
`ferent explanations, the end result was that they expect-
`ed the device to behave in the same manner.
`We also observed this behavior with the game Doom
`that we ported to Itsy. In Figure 7, virtually all users
`
`44
`
`May/June 2000
`
`expected that when they tilted the top edge of the dis-
`play down, they’d move up the stairs. Whether they
`sloshed the gun up the stairs or pointed to where they
`wanted the gun to go, the action and expected result
`were identical.
`Directly manipulating a handheld computer to play
`games fascinated nearly all subjects. The simple cat and
`mouse game on the prototype was surprisingly com-
`pelling, and users would often continue to play with it
`for a few minutes after their test session ended. When
`watching users play Doom on an Itsy, it’s clear that they
`had significantly more kinesthetic satisfaction in tilting
`the device than in rocking a cursor key.
`
`A limit to embodied user interfaces
`An alternative way to play Doom is to maneuver using
`the diamond-shaped cursor key to the right of the
`screen. This offered an opportunity to determine user
`input preferences for this application. We encouraged
`users to play the game using both input methods and
`polled them later about their preference. The seven
`respondents to the poll were evenly split in their pref-
`erences: three preferred the cursor key and four pre-
`ferred tilt-to-scroll. However, when their user interface
`preference was correlated with their level of experience
`playing electronic games, all those with extensive expe-
`rience preferred to use the cursor key and the rest pre-
`ferred tilt-to-scroll. Experienced users felt they could
`maneuver with more precision with the cursor key and
`didn’t like the fact that an enthusiastic press of the fire
`button would move the player. Less experienced users
`preferred to tilt-to-scroll and described that method as
`more natural or intuitive than the cursor key.
`While most users had positive things to say about the
`Rock ’n’ Scroll interface, they also noted some problems
`with the display while playing the game. The Itsy dis-
`play is a reflective LCD, so ambient light conditions can
`cause significant changes in screen visibility as the
`device tilts. In addition, enthusiastic play often results
`in significant tilt that exceeds the viewing angle of the
`display. While LCD technology will improve and reduce
`these effects, some experienced users observed one dis-
`play difficulty inherent in embodied user interfaces: per-
`ceived motion on the screen is the sum of the motion of
`the embodied device and the changes made to the dis-
`play by the device. As you interact with the device by
`moving it, the orientation of the display to the user
`changes, then in response to that motion the display
`contents move on the display. For “twitch” games like
`Doom, where the user manipulates the device rapidly,
`tilt-to-scroll results in motion on the screen that may be
`harder to visually track than motion via the cursor key.
`On the other hand, for applications like the photo
`album, where the user is not rapidly manipulating the
`device while trying to track moving fine detail on the
`screen, this should not be an issue. The results from this
`small test suggest an area for further study.
`
`Additional ways to Rock ’n’ Scroll
`In the discussion to this point, users always held the
`device in either or both hands. However, since all inter-
`action occurs by manipulating the device, it could also
`
`SCEA Ex. 1050 Page 5
`
`

`
`be worn. One promising area for wearing a small ges-
`ture-controlled device is on the “snuff box” portion of
`either hand—that is, the area between the thumb and
`index finger on the back of the hand. This area is typi-
`cally visible when using your hands. Gestures can be
`made on either of the device’s axes by moving either
`your hand or forearm. Figure 8 shows how a ther-
`mometer that measures the temperature at the palm of
`the glove could be mounted. By gesturing, the user can
`change the display to show the minimum, maximum,
`or current temperature. Other gestures could be used
`to switch the scale between Fahrenheit and centigrade
`and clear the saved minimum and maximum values.
`While at first this might seem a contrived example, con-
`sider for a moment the difficulty of trying to push your
`watch’s buttons while you have gloves on.
`To date this work has focused on interacting with a
`handheld device with a display, but the same techniques
`could be used to control other personal electronics such
`as radios, MP3 players, or cellular telephones. Rather
`than being restricted to play, fast-forward, and reverse
`buttons on a music player, the user could tilt to have fine-
`grain control over playing speed and gesture to step to
`the next track. As fashion plays an increasing role in per-
`sonal electronics, invisible control interfaces could pro-
`vide desired product differentiation. Finally, the inertial
`sensor doesn’t have to be incorporated in the device, but
`could be in the form of a ring or bracelet that commu-
`nicates with the device via a personal wireless network.
`
`Conclusion
`The Rock ’n’ Scroll user interface shows how inertial
`sensors in handheld devices can provide additional func-
`tion beyond “tilt-to-scroll.” By also using them to rec-
`ognize gestures, a significantly richer vocabulary for
`controlling the device is available that implements an
`electronic photo album, pager, or other limited function
`digital appliance without any additional input methods.
`The examples in this article offer a glimpse at the free-
`dom for both the device designers and users inherent in
`devices that can be held in either hand, at any orienta-
`tion, operated with mittens on, or not in the hand at all.
`I hope I’ve communicated some of the excitement that
`surrounds inertial interfaces. What started out as a side
`investigation in our laboratory has captured sufficient
`interest that we’ve designed it into the next generation of
`Itsy. We expect others to adopt these interfaces as well
`and look forward to seeing the results of their efforts. n
`
`Acknowledgements
`Rock ’n’ Scroll was developed as part of the Itsy pock-
`et computer project at Compaq’s research labs in Palo
`Alto. The work relied on the interest and knowledge of
`other team members including William Hamburgen,
`Lawrence Brakmo, Keith Farkas, Dirk Grunwald, Tim
`Mann, Robert Mayo, Sharon Perl, Barton Sano, Marc
`
`Viredaz, Carl Waldspurger, and Deborah Wallach.
`Wayne Mack provided electronics assembly assistance.
`My colleagues at Compaq’s research labs in Palo Alto
`were willing and articulate test subjects. George Lange
`provided my author picture. Wendy Bartlett and CG&A’s
`guest editors and reviewers improved the presentation
`of the results. I thank you all.
`
`References
`1. K.P. Fishkin, T.P. Moran, and B.L. Harrison, “Embodied
`User Interfaces: Towards Invisible User Interfaces,” Proc.
`Engineering for Human-Computer Interaction (EHCI 98), S.
`Chatty and P. Dewan, eds., Kluwer Academic Publishers,
`Hingham, Mass., 1998, pp. 1-18.
`2. C. Verplaetse, “Inertial Propriceptive Devices: Self-Motion-
`Sensing Toys and Tools,” IBM Systems J., Vol. 35, No. 4/5,
`1996, pp. 639-650.
`3. G.W. Fitzmaurice, “Situated Information Spaces and Spa-
`tially Aware Palmtop Computers,” Comm. ACM, Vol. 36,
`No. 7, July 1993, pp. 38-49.
`4. B.L. Harrison et al., “Squeeze Me, Hold Me, Tilt Me! An
`Exploration of Manipulative User Interfaces,” Proc. Human
`Factors in Computing Systems (CHI 98), ACM, New York,
`1998, pp. 17-24.
`5. J. Rekimoto, “Tilting Operation for Small Screen Inter-
`faces,” Proc. ACM Symp. User Interface Software and Tech-
`nology (UIST 96), ACM, New York, 1996, pp. 167-168.
`6. D. Small and H. Ishii, “Design of Spatially Aware Graspable
`Displays,” Proc. CHI 97, ACM, New York, 1997, pp. 367-368.
`7. G. Levin and P. Yarin, “Bringing Sketching Tools to Keychain
`Computers with an Acceleration-Based Interface,” Proc. CHI
`98 Extended Abstracts, ACM, New York, 1998, pp. 268-269.
`8. C. Waldspurger, “The Itsy Pocket Computer,” invited talk,
`Int’l Symp. Wearable Computers 98, http://www.research.
`compaq.com/wrl/itsy/talk-iswc98/.
`
`Joel Bartlett is a Principal Mem-
`ber of the Technical Staff at Com-
`paq’s Western Research Laboratory
`in Palo Alto, California. His prima-
`ry research interest is in “off the desk-
`top” appliances and applications.
`Before joining the Western Research
`Laboratory, he was with Tandem Computers where he was
`one of the co-inventors of the Tandem NonStop system. He
`received a BS in statistics and an MS in computer science
`and computer engineering from Stanford University in
`1972. He is a member of the ACM.
`
`Readers may contact Bartlett at the Western Research
`Laboratory, Compaq Computer Corp., 250 University Ave.,
`Palo Alto, CA 94301, e-mail joel.bartlett@compaq.com.
`
`IEEE Computer Graphics and Applications
`
`45
`
`SCEA Ex. 1050 Page 6