See discussions, stats, and author profiles for this publication at: https://www.researchgate.net/publication/262164253
`
`Gesture output: Eyes-free output using a force
`feedback touch surface
`
`Conference Paper · April 2013
`
`DOI: 10.1145/2470654.2481352
`
`CITATIONS
`13
`
`6 authors, including:
`
`READS
`51
`
`Max Plauth
`Hasso Plattner Institute
`
`Pedro Lopes
`Hasso Plattner Institute
`
`13 PUBLICATIONS 17 CITATIONS
`
`31 PUBLICATIONS 169 CITATIONS
`
`SEE PROFILE
`
`SEE PROFILE
`
`Patrick Baudisch
`Hasso Plattner Institute
`
`177 PUBLICATIONS 5,148 CITATIONS
`
`SEE PROFILE
`
`Some of the authors of this publication are also working on these related projects:
`
`Digital Mechanical Metamaterials View project
`
`Metamaterial Mechanisms View project
`
`All content following this page was uploaded by Pedro Lopes on 01 April 2016.
`
`The user has requested enhancement of the downloaded file. All in-text references underlined in blue are added to the original document
`
`and are linked to publications on ResearchGate, letting you access and read them immediately.
`
`Immersion Ex 2007-1
`Apple v Immersion
`IPR2016-01381
`
`

`

`
`
`Gesture Output: Eyes-Free Output
`Using a Force Feedback Touch Surface
`Anne Roudaut, Andreas Rau, Christoph Sterz, Max Plauth, Pedro Lopes, Patrick Baudisch
`Hasso Plattner Institute, Potsdam, Germany
`roudauta@gmail.com, {andreas.rau, christoph.sterz, max.plauth}@student.hpi.uni-potsdam.de,
`{pedro.lopes, patrick.baudisch}@hpi.uni-potsdam.de
`
`
`Figure 1: With our proposed gesture output, the device outputs messages to users using the same gesture language used for
`input. (a) Here, the user draws an
` to check the house number of the upcoming meeting. (b) The device replies by trans-
`lating the user’s finger along the path of an
`. (c) The pocketOuija is one of the two force feedback touchscreen devices we
`built that support gesture output. It translates the user’s finger by means of a transparent plastic foil overlaid onto the
`screen actuated using motors located on the back of the device.
`ABSTRACT
`We propose using spatial gestures not only for input but
`also for output. Analogous to gesture input, the proposed
`gesture output moves the user’s finger in a gesture, which
`the user then recognizes. We use our concept in a mobile
`scenario where a motion path forming a “5” informs users
`about new emails, or a heart-shaped path serves as a mes-
`sage from a friend. We built two prototypes: (1) The long-
`RangeOuija is a stationary prototype that offers a motion
`range of up to 4cm; (2) The pocketOuija is self-contained
`mobile device based on an iPhone with up to 1cm motion
`range. Both devices actuate the user’s fingers by means of
`an actuated transparent foil overlaid onto a touchscreen.
`We conducted 3 studies on the longRangeOuija. Partici-
`pants recognized 2cm marks with 97% accuracy, Graffiti
`digits with 98.8%, pairs of Graffiti digits with 90.5%, and
`Graffiti letters with 93.4%. Participants previously unfamil-
`iar with Graffiti identified 96.2% of digits and 76.4% of
`letters, suggesting that properly designed gesture output is
`guessable. After the experiment, the same participants were
`able to enter 100% of Graffiti digits by heart and 92.2% of
`letters. This suggests that participants learned gesture input
`as a side effect of using gesture output on our prototypes.
`ACM Classification: H.5.2 [Information interfaces and
`presentation]: User Interfaces - Graphical user interfaces,
`Input devices and strategies, Haptic I/O.
`
`Keywords: Gestures; Eyes Free; Force feedback; Touch.
`INTRODUCTION
`Gesture input allows users to interact eyes-free (non-visual,
`non-auditory) with their mobile touch devices, using an
`expressive and mnemonic set of commands [1]. Saponas et
`al. found that this is even possible while walking, based on
`users’ sense of touch alone [22].
`In order to have a dialog with the device, users need not
`only eyes-free input, but also output. Unfortunately, audi-
`tory output is not always possible, and vibrotactile output
`[3], which is the predominant eyes-free non-auditory type
`of output, was found to offer limited expressiveness [13],
`low bandwidth [17] and is hard to learn because it lacks
`mnemonic properties [12]. As a result, mobile users typi-
`cally enter an expressive, mnemonic, easy-to-learn gesture
` to request “messages”), but
`as input (such as writing an
`the system’s response will be akin to Morse code [17]. This
`makes output the bottleneck of the system.
`A way to alleviate this bottleneck is to use an array of vi-
`brotactile cells, e.g. to render spatial strokes [10]. In this
`paper, however, we go one step further by enabling the
`system to actuate the finger in the form of a 2D gesture.
`GESTURE OUTPUT CONCEPT
`We propose the concept of Gesture output, a non-visual,
`non-auditory output technique that communicates 2D ges-
`tures to users by moving their finger along the path of a
`gesture. While this concept opens a new range of interac-
`tion possibilities, we think Gesture output is particularity
`interesting in a mobile scenario where visual and audio
`modalities are not always available. Fig.1 illustrates a sce-
`nario we envision. Without taking the device out of the bag,
`
`
`Permission to make digital or hard copies of all or part of this work for
`personal or classroom use is granted without fee provided that copies are
`not made or distributed for profit or commercial advantage and that copies
`bear this notice and the full citation on the first page. To copy otherwise,
`or republish, to post on servers or to redistribute to lists, requires prior
`specific permission and/or a fee.
`CHI 2013, April 27–May 2, 2013, Paris, France.
`Copyright © 2013 ACM 978-1-4503-1899-0/13/04...$15.00.
`
`
`
`Immersion Ex 2007-2
`Apple v Immersion
`IPR2016-01381
`
`

`

`
`the user touches the device and draws an
` to ask the sys-
`tem for the house number of a meeting. The device replies
`.
`by translating the user’s finger along the path of an
`We created two prototypes capable of performing gesture
`output. Fig.2 shows one of them up-close. The pocketOuija
`uses a set of motors and a pulley system to actuate a flexi-
`ble plastic foil on top of the screen of a touchscreen device,
`here an iPhone. We will present this device, as well as the
`desktop version using a PHANToM, in reproducible detail.
`
`
`
`Figure 2: Our pocketOuija, here two versions of it,
`actuate the user’s finger by moving a clear foil on top
`of the device’s touchscreen (here an iPhone).
`The language of gesture output
`Gesture output can be used with any gesture language made
`of single stroke characters. Conceptually, this allows defin-
`ing gesture output languages based on arbitrary strokes,
`which can be optimized for arbitrary objectives. Following
`the lead of the original unistroke input language, for exam-
`ple, we might pick strokes that are efficient to perform, so
`as to optimize for expert interaction performance (see Study
`1: recognizability of gesture output for a study on perform-
`ance with marks as an output language).
`However, we argue that the main opportunity in gesture
`output is learnability: Vibrotactile patterns are hard to
`memorize because there are few existing associations be-
`tween a vibrotactile pattern and the information it is encod-
`ing; thus users have to learn such associations to decode
`patterns before understanding their meaning for the system.
`Gestures in the 2D plane, in contrast, associate readily with
`a wealth of existing mnemonic associations, including
`doodling, scribbling, and handwriting. We exploit these by
`adopting a gesture alphabet built on such associations. Such
`languages are readily available, including Graffiti and
`EdgeWrite [35]. For the purpose of this paper, we adopt
` used in our introduc-
`Graffiti. Based on this, the shape
`tory example, naturally communicates the digit “8”, be-
`cause users have spent years building up this association.
`While gesture output is designed to simplify learning, in-
`terpreting a message requires cognitive focus. Although no
`visual focus is required, gesture output may require users to
`focus, making it difficult to perform other tasks in parallel.
`Single-character messages
`Single-character messages allow the system to notify the
`user or to answer a question.
`Notify: The system uses a vibrotactile buzz to get the atten-
`tion of the user. Then the user places the hand onto the
` for
`device, and the system delivers the message, such as
`“low battery warning”.
`
`
`
`Question: a user enters the word “messages” using Graffiti
` gesture for short) and the system might
`(or just a single
` for “5 unread messages”. The system may
`respond with
` or
` to binary questions from the user, or
`
`respond
`
`when asked what direction to go. A user can also enter
`to ask the system to repeat the message.
`Compound messages
` for “two new messages” or
`Compound messages, e.g.
` (T) for “turn right” require to add a delimiter to our
`language. For gesture input, the delimiter is implemented
`by users lifting the finger or stylus off the screen. This
`clarifies when a character ends and the next one begins. We
`could try to port this concept to gesture output, but we want
`to maintain contact between user and device at all times to
`make sure the user is not missing anything. We therefore
`use the vibrotactile buzzer as delimiter. Using this model,
` and the number thirty-six
`we output “Turn right” as
`
`
` with “” being the buzz delimiter.
`as the sequence
`We use the time span during which the delimiter is playing
`to move the finger to the beginning of the next gesture, e.g.
`
`the finger is translated diagonally between the
`for
` and the start of the
`, the digits being super-
`end of the
`imposed in space. This keeps gestures in a consistent spa-
`tial reference and prevents longer gesture output sequences
`from driving the finger out of the bounds of the device.
`Note that we can also extend this approach to more than
`two symbols. For instance, it can serve to spell out a con-
`tact name, words, and possibly even sentences. The ability
`of users to recognize gesture output composed of more than
`two symbols is, however, not addressed in this paper and
`requires further investigations.
`
`
`
`Figure 3: (a) A boyfriend sends a heart. (b) The girl-
`friend touches her device, the message is presented by
`moving her finger along the same heart.
`Between people
`We can use the same approach to enable the communica-
`tion between users (Fig.3). Since no automatic recognition
`engine is involved here, any gesture both users have agreed
`upon can be used for communication.
`STUDIES OVERVIEW
`The main benefit of gesture output is its learnability be-
`cause users are able to readily use a wealth of existing
`mnemonic associations. In this paper, we present three user
`studies that support this claim on the longRangeOuija.
`
`Immersion Ex 2007-3
`Apple v Immersion
`IPR2016-01381
`
`

`

`
`Study 1: recognizability of gesture output
`We wanted to verify the basic mechanics, i.e., if users were
`able to receive and recognize gestures. We therefore picked
`directional marks as a self-explaining gesture alphabet, and
`checked whether users were able to recognize their direc-
`tion. Results show that using 1cm marks allows participants
`to recognize the eight compass directions with 86.8% accu-
`racy, and marks longer than 2cm with 97% accuracy.
`Study 2: learnability of single-character messages
`The goal was to investigate if knowledge of input helps
`understand output (“transfer learning”). We picked the
`mnemonic alphanumeric Graffiti alphabet. We hypothe-
`sized that training in input would allow participants to
`successfully recognize output (and vice versa). Further-
`more, due to the design of Graffiti for guessability, we
`hypothesized that Graffiti output would also be guessable,
`so that participants without training should be able to de-
`code the gestures. Results show that users familiar with
`Graffiti input but with no training in Graffiti output recog-
`nized Graffiti output with 98.8% accuracy for digits and
`93.4% for letters, thus showing that transfer learning had
`occurred. Participants unfamiliar with Graffiti altogether
`correctly guessed 96.2% of digits and 76.4% of letters, thus
`showing that the alphabet is self-explanatory. Finally, the
`same participants correctly input 100% of digits and 92.2%
`of letters after the experiment, thus showing that reverse
`transfer learning had occurred as well.
`Study 3: learnability of bi-grams
`In this study, we go further in our investigation of learn-
`ability and explored compound messages. We picked a
`highly mnemonic gesture alphabet made of pairs of Graffiti
`digits. We hypothesized that training in gesture input would
`allow users to successfully recognize compound gesture
`output by transfer learning and by aggregation of input
`knowledge. Results show that participants familiar with
`Graffiti input but with no training in Graffiti output recog-
`nized compound Graffiti output with 90.5% accuracy, thus
`showing that our design works for two-digits sequences.
`Additional studies are required for longer gesture sequence.
`CONTRIBUTION
`Our main contribution is the concept of gesture output that
`creates symmetry between non-visual, non-auditory input
`and output. We also present two prototypes, a desktop force
`feedback touchscreen (longRangeOuija) and a pocketsize
`version (pocketOuija). We contribute three user studies on
`the longRangeOuija that support that the blending of input
`and output in gestures is learnable even without training.
`RELATED WORK
`Vibrotactile output
`Vibrotactile messages (Tactons [3]) allow communicating
`non-visual information using different rhythms and ampli-
`tude of vibration. For instance, Tan proposed associating
`vibration patterns with Morse code [26]. Another example
`is Shoogle that transforms the contents of the user’s inbox
`into virtual “message balls” [33]. A user shaking Shoogle
`hears and feels the impacts of the balls bouncing around.
`
`
`
`Implementing vibrotactile is comparably simple—it re-
`quires only an eccentric motor or voice coil—thus many of
`today’s mobile devices offer it [21]. However, vibrotactile
`lacks expressiveness [13] and bandwidth [17]. In particular,
`a single vibrotactile unit allows conveying binary informa-
`tion, such as “target hit”, but cannot directly encode loca-
`tions. Vibrotactile also requires long learning phases as it is
`perceptively and cognitively demanding [12]. For instance
`Geldard [8] reported that users required 65 hours of train-
`ing to recognize an encoding of the English alphabet.
`Several works extend the expressiveness of vibrotactile
`messages using arrays of vibrotactile cells (e.g., [24, 37]).
`For instance, Poupyrev proposed augmenting mobile de-
`vices with tactile arrays in order to guide the user’s finger
`and to create awareness interfaces [21]. In more recent
`work, Israr used a vibrotactile array mounted into a back-
`rest to provide gamers with directional feedback [10].
`Force feedback
`Unlike vibrotactile, force feedback mechanisms allow cre-
`ating a directional force. In their simplest form, force feed-
`back devices offer a single degree of freedom. For instance,
`Enriquez [6] proposed using an actuated 1DOF rotary knob
`for output of brief computer-generated signals (haptic
`icons). More complex devices include articulated arms (e.g.
`PHANToM or Falcon) that allow 3D force feedbacks
`through a pen or an intermediary object. For instance, with
`the Palmtop display [18], a mobile device is attached to the
`articulated arm. It enables users to manipulate a remote
`object as if they were holding it in their hands. A limitation
`of articulated arms is that they only create force feedback at
`a single point. In contrast, the SPIDAR system [23] offers
`multi-point controls: it uses motors and a pulley system to
`actuate each finger of the user independently in order to
`create a sensation of manipulating 3D objects in the air.
`Force feedback for communication between users
`Force feedback has been used to allow users to communi-
`cate over a distance. Each InTouch device, for example,
`consists of three cylindrical rollers mounted on a base [2].
`Each action done on one device is replicated on the other
`one creating the illusion of a single shared physical object.
`Telephonic Arm Wrestling [32] simulates the feeling of
`arm wresting over a telephone line. The Dents Dentata [9],
`device can squeeze a users hand while calling.
`Force feedback in training systems
`Much research has examined the use of force feedback to
`train users in performing tasks, such as surgery. Feygin [7]
`for instance introduced the term haptic guidance that con-
`sists in guiding users through an ideal motion, thus giving
`the user a kinesthetic understanding of what is required.
`Dang [5] also discusses a system that provides guidance to
`users performing surgery by restricting their movements
`from deviating from a path recorded previously by a real
`surgeon. Several researches have built on the same princi-
`ple of replaying expert gestures to train motor skills: [25]
`for handwriting, [27] for writing Chinese characters, [38]
`for training medical operations or [15] teach an abstract
`motor skill that requires recalling a sequence of forces.
`
`Immersion Ex 2007-4
`Apple v Immersion
`IPR2016-01381
`
`

`

`
`Actuated touchpads, tabletops, and touchscreens
`On tabletop systems, actuating systems were initially used
`to actuate tangibles. Actuated workbench [19] and Pico [20]
`were the first systems of this kind; they actuated tangible
`pucks using an array of electromagnets mounted below the
`table. Madgets [30] extend this approach by moving tangi-
`ble widgets consisting of multiple moving parts.
`Similar approaches have been used to actuate fingers. Fin-
`gerFlux, for example, combines the Madgets platform with
`finger-worn magnets to apply force feedback to that finger
`[31]. ShiverPad [4] combines a programmable friction
`device [34] with the slip stick effect, i.e., by alternating
`between low and high friction at the same frequency that
`the device is moving in the plane. At 60 milliNewtons, the
`device is not strong enough to move the finger, but it is
`able to actuate a little plastic ball.
`Other devices combine motors and pulley mechanisms.
`Wang introduced the Haptic Overlay Device [28, 29]: the
`user touches an overlay material connected to drive rollers
`that can translate. In ActivePad [16], the same mechanism
`is combined with a programmable friction surface. Fing
`Viewer [36], a 2D version of the SPIDAR system [23],
`actuates a ring the user is touching using four motors medi-
`ated by cables. Our prototype pocketOuija uses this same
`string-motor mechanism, but allows for a mobile form
`factor by using a different arrangement of motors.
`PROTOTYPE #1: THE LONG-RANGE-OUIJA
`The longRangeOuija is our first prototype design and it is
`optimized for providing us all the control we need to run a
`wide range of user studies, such as how scale of gesture
`affects comprehension (see User Study 1).
`As shown in Fig.4, the longRangeOuija transmits force to
`the user’s finger via a rigid transparent foil overlaying the
`actual touch surface (an iPad). The foil is actuated using a
`PHANToM force feedback device, a device normally de-
`signed for moving a stylus in 3D space.
`
`Mechanics
`Fig.5 illustrates the mechanics of the prototype. On the
`right, the foil is actuated by the PHANToM. On the left, the
`foil is guided by a groove that only permits left-right mo-
`tion. This mechanical design causes the foil to pivot around
`its left extremity labeled S in Fig.6. This creates a non-
`linear relationship between the motion of the PHANToM
`arm and coordinate system of the iPad and the user’s fin-
`ger. The system translates between both systems as fol-
`lows: Given F (finger start), A (arm start) and F’ (finger
`final), we search A’ (arm final):
`
`
`
`
`
`Figure 5: The longRangeOuija consists of a transpar-
`ent foil actuated by a PHANToM. Here the arm pulls
`the foil to the top right and away from the user, dis-
`placing the finger accordingly.
`
`
`
`Figure 4: The longRangeOuija translates the user’s
`finger via a clear foil actuated by a PHANToM arm.
`During gesture input the motors in the PHANToM are
`turned off and the foil drags with the user’s finger. Users
`can do so with reasonable resistance because the foil over-
`lay is designed for minimum weight (100g). During gesture
`output, the foil is actuated by the PHANToM. The PHAN-
`ToM delivers up to 3.3 Newtons.
`
`
`
`
`
`Figure 6: The longRangeOuija mechanics.
`Software
`The system senses the location of the finger via the iPad.
`The location of the foil is known via the PHANToM that is
`controlled using a computer running the OpenHaptics C++
`library. The interface on the iPad is done with HTML5 and
`
`Immersion Ex 2007-5
`Apple v Immersion
`IPR2016-01381
`
`

`

`
`JavaScript in the Safari browser. The browser sends the
`coordinates of the touch events over a wireless network
`using XMLHttpRequest. The computation required for
`rendering gesture output on the PHANToM is processed on
`the computer side, thus maintaining a low latency between
`the iPad touch events and the haptic stimuli. In the studies,
`we used a speed between 3 and 4 cm/s.
`PROTOTYPE #2: THE POCKET-OUIJA
`The pocketOuija enables to experience gesture output in a
`mobile scenario. Our prototype (two versions are shown in
`Fig.2) is mobile, battery-operated, and self-contained.
`The pocketOuija uses six motors to actuate a transparent
`sheet of plastic foil overlaid on the touchscreen of an iPh-
`one. The prototype delivers 4 Newtons of force, which it
`transmits to the foil via a system of nylon strings. The
`device receives instructions from the iPhone via the
`iPhone’s headphone-jack (we use frequency shift keying).
`The pocketOuija adds 30mm of thickness and 280g of
`weight to the iPhone. The smaller version we developed is
`limited in force and action radius but has a reduced weight
`of just 120g while measuring 17mm in depth.
`
`
`
`Figure 7: The pocketOuija mechanical design: (a) The
`casing contains 6 motors, 2 batteries and a Arduino
`Nano. (b) The casing implements a system of tubes
`that guide strings around the device.
`Mechanics
`Fig.7 illustrates the mechanical design of the pocketOuija,
`which consists of four parts:
`The clear foil measures 9.5x8cm, making it 3cm wider and
`higher than the iPhone screen. The foil is 32µm thick,
`which is thin enough to allow touch input to be picked up
`by the capacitive touchscreen.
`The string system is inspired by SPIDAR [23] and Fing
`Viewer [36]. In order to achieve the mobile form factor we
`modified the design as follows: (1) We built a system of
`tubes (Fig.7b) to guide the nylon strings around the device
`while minimizing friction. (2) We added two motors for an
`overall number of six. The additional motors, shown in the
`center of Fig.7b, provide additional motion range in that
`they allow pulling the foil all the way; at this point the
`strings of other two motors pull the foil into opposite direc-
`tions and thus lose their power. (3) To obtain torque with-
`out a gearbox we used the thin motor axels directly as
`winches to roll up the nylon string. The strings are made
`from 0.18mm Nylon. In order to attach them to the foil, we
`sandwiched plastic washers between the film and its folded
`border ears and knotted the strings trough both.
`
`
`
`The six motors are MABUCHI FK-280 DC-Motors turning
`at 7.4Volts with an approximate torque of 2Nm each.
`The 3D printed casing holds the motors in place and pro-
`vides the pipes which guide the pulley strings to the right
`location, while maximizing corner radii as shown in Fig.7b.
`Electronics
`Fig.8 illustrates the electronic design of the device. It con-
`sists in three components:
`An Arduino Nano is programmed via a mini-USB port.
`Soldering battery pins and controller wires directly onto the
`board allowed up to fit the Arduino and the batteries in the
`available space between the motors.
`A Lithium-ion polymer battery consists of 2 single cell S1
`LiPo modules that are specified for 3.7Volts each. It pro-
`vides the current of 1.3A required by the motors.
`Six transistors control the current of up to 400mA each.
`We used BD243C FET enhanced with a suppressor diode
`protecting the transistor from inductive flyback.
`
`
`
`Figure 8: The pocketOuija electrical design consists
`of a lithium battery, an Arduino as well as one FET
`and Diode per motor (here only one is shown).
`Software
`The software running on the Arduino receives messages to
`be performed from a program on the iPhone. To actuate the
`foil, the iPhone app calculates the sequence of motor volt-
`ages required to produce the respective motion path and
`sends it to the Arduino via the iPhone’s headphone jack (it
`encodes the information as a sequence of frequencies). As
`the device cannot detect the location of the foil itself, we
`can reset the sheet by pulling the foil alternatingly from all
`sides while decreasing force in each motor, which centers
`the foil in the middle of the screen. The motors are actuated
`at constant speed and allow translating the user’s finger at a
`speed between 3 and 4 cm/s. (3 to 4cm/s).
`Limitations and generalization of findings
`The pocketOuija is an early prototype of a force-feedback
`touchscreen and therefore portraits limitations. Most impor-
`tantly, it adds a plastic foil on top of the screen, affecting
`the user experience during regular interaction. It also adds
`120g and 17mm of thickness to the mobile phone for mo-
`tors and battery. Also, the motors generate noise equivalent
`to the vibrotactile motor already found on such devices.
`Our prototype also offer only 1cm range of motion: while
`this design had dedicated motors for North and South, it
`produced East motion by mixing force from the NE and SE
`motors. It is possible to improve this design by having a
`motor for all eight directions. Finally, we only explored the
`feasibility of gesture output on the longRangeOuija; other
`force feedback devices may produce different results.
`
`Immersion Ex 2007-6
`Apple v Immersion
`IPR2016-01381
`
`

`

`
`STUDY 1: RECOGNIZABILITY OF GESTURES
`While gesture output ultimately is about learnability (see
`Study 2 and 3), we first wanted to verify the basic mechan-
`ics, i.e., if users are able to receive and recognize gestures.
`We thus conducted a study to test only recognition without
`any learning part. To do this, we used a gesture alphabet
`that is even more self-explaining than Graffiti, namely
`simple directional strokes, also known as marks [11]. Par-
`ticipants’ task was thus to recognize their direction.
`We added stroke length as a second independent variable to
`determine the minimum distance required for reliable rec-
`ognition. This will inform the design of future implementa-
`tion and give us an idea of how small or large a device
`would have to be in order to offer reliable gesture output.
`As a side effect, this would also give us a slightly enlarged
`24 gestures repertoire of 8 directions  3 lengths, a lan-
`guage that presumable could support more elaborate appli-
`cations than the 8 simply directional strokes. As discussed
`earlier, we think of gesture alphabets as a matter of learn-
`ability – an aspect that is investigated in Study 2 and 3.
`Interface
`Participants were seated in front of the longRangeOuija
`prototype described earlier. Participant’s wrists were sup-
`ported by the armrest shown in Fig.9.
`
`3 Lengths  3 blocks = 72 trials per participant. Partici-
`pants performed 5min of training (30 trials) before the
`experiment. They had breaks (~1min) every 20 trials. All
`participants completed the study in 20min or less.
`Participants
`We recruited 12 right-handed participants (3 females) from
`our institution. They were between 21 and 25 years old.
`They received a small compensation for their time.
`Hypothesis
`The purpose of the study was to check if participants were
`able to recognize the direction of marks with different
`lengths. We thus picked a small length (1cm) as a baseline
`and hypothesized that stokes of this size would have a
`lower recognition rate than longer ones (2cm and 4cm).
`Results and discussion
`Fig.10 shows the results with a confusion matrix. Fig.11
`provides summary data for the recognition of direction.
`ANOVA were performed on the average recognition rate
`across the three blocks. As expected, an ANOVA showed
`that participants recognized longer strokes more reliably
`(F2,22=10.73, p<0.001). Post-hoc comparison tests (using a
`Tukey’s HSD test) indicated that users performed signifi-
`cantly better with 2cm and 4cm marks than with 1cm
`marks. The 2cm and 4cm strokes resulted in recognition
`rates of 97.6% and 96.5%, respectively.
`
`
`
`Figure 9: After participants had received the output
`gesture, they entered the answer on the iPad.
`Task
`For each trial, participants started by tapping the screen.
`They then followed four verbally given instructions. On
`“close”, participants closed their eyes. On “touch”, partici-
`pants pressed the screen and maintained contact with the
`screen. The system then performed one gesture from the
`repertoire of 24 different gestures, i.e., it actuated the par-
`ticipant’s finger along the shape of the respective stroke.
`On “up”, participants moved their hand up in the air above
`the hand rest, and the system moved the device back into its
`initial position. On “open”, finally, participants opened
`their eyes and selected the gesture they felt they had re-
`ceived in the interface shown in Fig.9. They completed the
`trial by tapping the “next” button on screen.
`Experimental design
`The study used an 833 within-subjects design, with two
`independent variables: Direction (the 8 compass directions)
`and Length (1, 2, and 4 cm). Direction and Length of the
`strokes were randomized for each of the 3 blocks. Each
`participant completed all conditions: 8 Directions 
`
`
`
`
`Figure 10: Confusion matrix for 8 directions & 3 lengths.
`The 2nd line reads as “19% of North-East strokes has
`been recognized as North strokes”.
`The study thus showed thata participants were able to rec-
`ognize the direction of simple directional strokes. Results
`suggest using strokes of 2cm or more to reach a recognition
`rate of 97% for the eight compass directions strokes.
`Preliminary results on the pocketOuija
`We replicated the 1cm condition from this study on the
`pocketOuija with 6 new participants. This time, they were
`holding the device in their dominant hand and operated
`using the thumb, i.e., single-handed use, as shown in the
`last scene of the video. We found a recognition rate of
`
`Immersion Ex 2007-7
`Apple v Immersion
`IPR2016-01381
`
`

`

`
`90.3% (compared to 86.8% reported with longRangeOuija).
`While these numbers are comparable to the numbers ob-
`tained with the longRangeOuija, additional testing is re-
`quired to validate that the two prototypes are comparable.
`
`
`Figure 11: Recognition rates of direction for 1cm, 2cm,
`and 4cm strokes (Bars are +/-95% confidence).
`STUDY 2: LEARNABILITY OF SINGLE-CHARACTER
`The purpose of this study was to validate our claims about
`the learnability of gesture output, in particular that users’
`knowledge of gesture input helps them understand gesture
`output (“transfer learning”). To tackle this, we picked a
`highly mnemonic gesture alphabet—the alphanumeric
`Graffiti alphabet. We hypothesized that training in gesture
`input would allow users to successfully recognize gesture
`output (and vice versa). Furthermore, due to the design of
`Graffiti for guessability, we hypothesized that Graffiti
`output would also be guessable, so that participants without
`training would be able to decode some of the gestures.
`Task and Interface
`The task and the interface were identical to the first study,
`except that the gesture alphabet was different. In the letters
`task, participants were presented with one of 26 output
`characters from the alphabetic portion of the Graffiti alpha-
`bet (Fig.12a). In the digit task, participants were presented
`with one of 10 output characters from the numeric portion
`of the Graffiti alphabet (Fig.12b).
`Each trial took place the same way as in Study 1. We re-
`moved the need to touch the screen between each trial to
`reduce the time of the experimentation.
`Second independent variable: Training vs. walk-up
`Participants were assigned to one of the two groups: Par-
`ticipants in the trained condition received 20min of training
`in Graffiti input prior to the study by playing the training
`game shown in Fig.13. The ga