Proceedings of the Second International Conference on Tangible and Embedded Interaction (TEI'08), Feb 18 20 2008, Bonn, Germany
`
`Gesture Recognition with a Wii Controller
`
`Thomas Schl¨omer,1 Benjamin Poppinga,1 Niels Henze,2 Susanne Boll1
`1University of Oldenburg
`2OFFIS Institute for Information Technology
`(cid:192)(cid:85)(cid:86)(cid:87)(cid:81)(cid:68)(cid:80)(cid:72)(cid:17)(cid:79)(cid:68)(cid:86)(cid:87)(cid:81)(cid:68)(cid:80)(cid:72)(cid:35)(cid:88)(cid:81)(cid:76)(cid:16)(cid:82)(cid:79)(cid:71)(cid:72)(cid:81)bur(cid:74)(cid:17)(cid:71)(cid:72)
`(cid:192)(cid:85)(cid:86)(cid:87)(cid:81)(cid:68)(cid:80)(cid:72)(cid:17)(cid:79)(cid:68)(cid:86)(cid:87)(cid:81)(cid:68)(cid:80)(cid:72)(cid:35)(cid:82)f(cid:192)(cid:86)(cid:17)(cid:71)(cid:72)
`
`ABSTRACT
`In many applications today user interaction is moving away
`from mouse and pens and is becoming pervasive and much
`more physical and tangible(cid:17) New emerging interaction (cid:87)(cid:72)(cid:70)(cid:75)(cid:16)
`nologies allow developing and experimenting with new (cid:76)(cid:81)(cid:16)
`teraction methods on the long way to providing intuitive (cid:75)(cid:88)(cid:16)
`man computer interaction(cid:17) In this paper, we aim at (cid:85)(cid:72)(cid:70)(cid:82)(cid:74)(cid:81)(cid:76)(cid:16)
`zing gestures to interact with an application and present the
`design and evaluation of our sensor(cid:16)(cid:69)(cid:68)(cid:86)(cid:72)d gesture (cid:85)(cid:72)(cid:70)(cid:82)(cid:74)(cid:81)(cid:76)(cid:87)(cid:76)(cid:16)
`on(cid:17) As input device we employ the W(cid:76)(cid:76)(cid:16)(cid:70)(cid:82)(cid:81)(cid:87)(cid:85)(cid:82)(cid:79)(cid:79)(cid:72)r (W(cid:76)(cid:76)(cid:80)(cid:82)(cid:16)
`te) which recently gained much attention world wide(cid:17) We
`use the Wiimote’s acceleration sensor independent of the g(cid:68)(cid:16)
`ming console for gesture recognition(cid:17) The system allows the
`training of arbitrary gestures by users which can then be (cid:85)(cid:72)(cid:16)
`called for interacting with systems like photo browsing on
`a home TV(cid:17) The developed library exploits W(cid:76)(cid:76)(cid:16)(cid:86)(cid:72)(cid:81)(cid:86)(cid:82)r data
`and employs a hidden Markov model for training and (cid:85)(cid:72)(cid:70)(cid:82)(cid:16)
`gnizing user(cid:16)(cid:70)(cid:75)(cid:82)(cid:86)(cid:72)n gestures(cid:17) Our evaluation shows that we
`can already recognize gestures with a small number of (cid:87)(cid:85)(cid:68)(cid:76)(cid:16)
`ning samples(cid:17) In addition to the gesture recognition we also
`present our experiences with the W(cid:76)(cid:76)(cid:16)(cid:70)(cid:82)(cid:81)(cid:87)(cid:85)(cid:82)(cid:79)(cid:79)(cid:72)r and the (cid:76)(cid:80)(cid:16)
`plementation of the gesture recognition(cid:17) The system forms
`the basis for our ongoing work on multimodal intuitive (cid:80)(cid:72)(cid:16)
`dia browsing and are available to other researchers in the
`(cid:192)(cid:72)(cid:79)(cid:71)(cid:17)
`
`Author Keywords
`tangible user interfaces, gesture recognition, Wiimote
`
`ACM (cid:38)(cid:79)(cid:68)(cid:86)(cid:86)(cid:76)(cid:192)(cid:70)(cid:68)(cid:87)(cid:76)(cid:82)n Keywords
`(cid:62)(cid:43)(cid:17)(cid:24)(cid:17)2 User Interfaces]: Haptic I/O
`
`INTRODUCTION
`In recent years, we (cid:192)(cid:81)d more and more affordable hardware
`that allows the development of multimodal user interf(cid:68)(cid:70)(cid:72)(cid:86)(cid:17)
`Recently one of these interfaces is the so called Wiimote [1],
`the device that serves as the wireless input for the Nintendo
`Wii gaming console(cid:17) The Wiimote can detect motion and (cid:85)(cid:82)(cid:16)
`tation in three dimensions through the use of accelerometer
`technology(cid:17) Separating the controller from the gaming (cid:70)(cid:82)(cid:81)(cid:16)
`sole, the accelerometer data can be used as input for gesture
`
`Permission to make digital or hard copies of all or part of this work for
`personal or classroom use is granted without fee provided that copies are
`not made or distributed for profit or commercial advantage and that copies
`bear this notice and the full citation on the first page. To copy otherwise, or
`republish, to post on servers or to redistribute to lists, requires prior specific
`permission and/or a fee.
`TEI 2008, February 18 20, 2008, Bonn, Germany.
`Copyright 2008 ACM 978 1 60558 004 3/08/02...$5.00.
`
`11
`
`Figure 1. The Wii Controller (Wiimote).
`
`recognition. In our work, we address the recognition of ge-
`stures for new multimodal user interfaces. We are interested
`in recognizing arbitrary gestures of users that are performed
`by one hand. We choose the Wiimote as our input device for
`its ease of use, the hardware price and the design.
`
`Accelerometer-based gesture recognition has been discussed
`in many publications, most prominently in those by Hof-
`mann et al. in [4] and most recently in those by M¨antyj¨arvi et
`al. in [6] and [7]. Like the commercial work by AiLive Inc.
`(cf. [2]) we aim for a system allowing the training and re-
`cognition of arbitrary gestures using an accelerometer-based
`controller. In doing so we have to deal with spatially as well
`as temporally variable patterns and thus need a theoretical
`backbone fulfilling these demands. We transfer the methods
`proposed in [6, 7] who are using special hardware for 2D
`gesture recognition to the consumer hardware of the Wii-
`mote and recognize 3D hand gestures. With the controller
`the user can make her own, closed gestures and our gesture-
`recognition aims at a Wii-optimized recognition. Our com-
`ponents as well as the filtering process is specifically targe-
`ted to the Wiimote. With this paper we also share our expe-
`riments and the resulting implementation with other resear-
`chers.
`
`CONCEPT
`In gesture recognition using an acceleration sensor, gestures
`are represented by characteristic patterns of incoming signal
`data, i.e. vectors representing the current acceleration of the
`controller in all three dimensions. Hence, we need a system
`pipeline preparing and analyzing this vector data in order
`to train as well as recognize patterns for distinct gestures.
`For this purpose we revert to the classic recognition pipeline
`shown in Figure 2. It consists of the three main components
`quantizer, model and classifier.
`
`Petitioner Samsung Ex-1018, 0001
`
`

`

`Proceedings of the Second International Conference on Tangible and Embedded Interaction (TEI'08), Feb 18 20 2008, Bonn, Germany
`
`Coane —HMM ese
`
`“idle state
`directorial equiv.
`
`Figure 2. Components of our recognition system. We use a total of two
`filters before following a traditional pipeline like [7]. The quantizer ap-
`plies a common k-mean algorithm to the incoming vector data, for the
`modela left-to-right hidden Markov modelis used andthe classifier is
`chosen to be a bayesian.
`
`Asan accelerometerconstantly producesvectordata wefirst
`need a quantizer clustering the gesture data. Here, a common
`k-meanalgorithm (cf. e.g. [5]) is applied. The model has be-
`en chosento be a discrete hidden Markov modelsince it of-
`fers a long history in the service of gesture recognition and
`promisesto deliverreliable results for patterns with spatial
`and temporal variation (cf. e.g. [4]). The remaining compo-
`nentis a classic Bayes-classifier. In addition to these main
`components we establish two filters for pre-processing the
`vector data, an “idle state” and a “directorial equivalence”
`filter. Both serve the purpose to reduce and simplify the in-
`comingacceleration data.
`
`As we want optimize the HMMforthe task of an accele-
`rometer based gesture recognition we select the reference
`gestures shown in Figure 3 during the following tests and
`evaluations. With regard to the components of the classic
`gesture recognition approach in Figure 2 we identify three
`components for analysis and improvement: vector quantiza-
`tion, the concrete hidden Markov modelandfilters.
`
` °
`
`i
`
`xX
`
`xX
`
`xX
`
`(ak =8
`
`(b)k=14
`
`()k =18
`
`Figure 4. Distribution of the cluster centres during quantization for k €
`{8, 14, 18}. We extrapolate from the two-dimensional case for k = 8
`with a simple circular distribution to a three-dimensional using two
`orthogonalcircles for k = 14 to another three-dimensional using three
`orthogonalcircles for k = 18 and evaluate which of them results in the
`mostreliable behavior.
`
`ced the spherical distribution to include another four centers
`on the XZ-plane and thus gain k = 18 cluster centres. The
`radius of each circle/sphere dynamically adapts itself to the
`incoming signal data.
`
`We conducted a small evaluation comparing the three set-
`tings shown in Figure 4 using the reference gestures from
`Figure 3. We found that for k = 8 the recognition process
`cannotclearly differentiate between the five reference gestu-
`res. Since the gestures explore all three dimensions, laying
`out the centres on a two dimensional plane is not sufficient.
`With k = 14 the probabilities for the respective gestures
`improve as expected and the modelcan clearly distinguish
`between the five gestures. Using k = 18 results in “over-
`trained” HMMs,do not improve the probabilities and slow
`downperformance. Consequently we choose k = 14 with
`the distribution shownin Figure 4(b).
`
`Vector quantization
`Like other acceleration-sensors the one integrated into the
`Wiimote delivers too much vectordata to be put into a single
`Hidden Markov Model
`HMM.In orderto cluster and abstract this data the common
`In our system a HMMisinitialized for every gesture and
`k-meanalgorithm is applied with k being the numberofclu-
`sters or codes in the so-called codebook. Since k must be
`then optimized by the Baum-Welch algorithm (cf. [3]). Ho-
`wever, there are two competing HMMinstances we might
`determined empirically we decided to conduct tests to find
`revert to: a left-to-right vs. an ergodic. While [4] claims that
`a codebooksize delivering satisfying results and as we are
`both approaches deliver comparable results, [9] states that a
`evaluating true 3D gestures we cannot rely on previous re-
`left-to-right modelis clearly to be preferred when the inco-
`sults by Mantyjarviet al. who empirically identified k = 8
`ming signals change over time. We implemented both mo-
`for gestures in a two-dimensional plane. However, we adopt
`dels and ran a test to determine which modelbetter suits our
`their idea of arranging the 8 cluster centres onacircle by
`needs. Table 1 showsthe results for both possible instances
`extending it to the 3D case. Instead of distributing the cen-
`and a varying numberofstates. Our results confirm the state-
`tres uniformly on a two-dimensional circle we put them on a
`mentby [4] that no instance is significantly better than the
`three-dimensional sphere, intersecting two circles orthogo-
`other as well as the statement by [8] that the influence of the
`nal to each other(cf. Figure 4). Consequently this leads to
`k = 8+ 6 = 14 centres. For comparison, we also enhan-
`
` roe 1.49- 10-7"7]1.59-10-™]46-10—
`
`
`Fs.3
`
`y
`
`.
`
`y \
`‘'
`fy
`
`(e) Tennis
`(d) Z
` (c) Roll
`(a) Square (b) Circle
`2263-10-77
`5.3-10-=
`1.02 - 10-75|7.55 - 10775|2.64- 10-7!|1.25- 10-7!|1.09 - 10-®°
`Figure 3. Reference Gestures. The gesture in (b) does not showastar-
`ting point because the gesture might start anywhere on thecircle. Ge-
`Table 1. Model probabilities for left-to-right and ergodic HMM with
`sture (c) describes a 90°-roll around the z-axis (forth and back) and
`varying numberof states. Our evaluation confirms the statement by
`gesture (e) symbolizes the serve of a regular tennis match: raising the
`[4] that neither the numberof states nor the concrete HMM instance
`influence the results all too much.
`controller and then rapidly loweringit in a bow-curved manner.
`
`12
`
`Petitioner Samsung Ex-1018, 0002
`
`Petitioner Samsung Ex-1018, 0002
`
`

`

`Proceedings of the Second International Conference on Tangible and Embedded Interaction (TEI'08), Feb 18 20 2008, Bonn, Germany
`
`IMPLEMENTATION
`In our prototype we use the Nintendo Wiimote Wireless
`Controller with an integrated three axis acceleration sensor
`(Analog Devices ADXL330). Since the Wiimote is designed
`for human interaction with the Wii-Console it provides the
`ability for basic in-game gesture recognition. Connected via
`the Bluetooth Human Interface Device (HID) protocol it is
`possible to readout its self-description data. The meaning of
`this communicated data has been reverse engineered by the
`open-source community.1 Based on these findings it is pos-
`sible to establish a basic communication with the Wiimote.
`
`We implemented the gesture recognition in Java using the
`standardization of Java APIs for Bluetooth Wireless Tech-
`nology (JABWT) defined by the JSR-82 specification. Using
`Java ensures platform independency, for developing and te-
`sting purposes we use the GNU/Linux platform with the
`Avetana Bluetooth implementation.2
`
`The recognition process is realized as a reusable and extensi-
`ble gesture recognition library based on an event-driven de-
`sign pattern. The library provides an interface for basic func-
`tions, e.g. acceleration readout with the WiiListener in-
`terface, as well as recognition functions using a Gesture-
`Listener interface. Through its modularity it is easy to
`adapt our prototype to other acceleration-based controllers.
`We intend to make the library available to other researchers
`in the field.
`
`EVALUATION
`In order to determine the performance of our system we con-
`ducted an evaluation. We collected quantitative data to de-
`termine the percentage of correctly recognized gestures for
`gestures trained by users themselves. In order to make the
`results comparable among the individual participants the fi-
`ve gestures described in Figure 3 were used by all partici-
`pants. The group consists of one woman and five men aged
`between 19 and 32 years. All participants had some minor
`experience with the Wiimote and none used the Wiimote re-
`gularly. None of the participants was physically disabled.
`
`Preparing the evaluation we set up our environment and the
`Bluetooth connection to the Wiimote. The participants got a
`brief explanation of the purpose of the system and how to in-
`teract with the Wiimote. Afterwards we introduced the five
`gestures using drawings of the five gestures (see Figure 3)
`and demonstrated the execution of the first gesture Square.
`Each participant was asked to perform each gesture fifteen ti-
`mes resulting in 75 gestures per participant. The participants
`had to push and hold the A-button on the Wiimote while
`performing gestures. After each completing of the respective
`fifteen gestures the user had to press the Wiimote’s HOME-
`button and the drawing of the next gesture was shown. Each
`session lasted for fifteen minutes on average and the parti-
`cipants received no feedback from the system. During the
`evaluation we stored the complete raw data transmitted by
`the Wiimote.
`1E.g., www.wiili.org
`2www.avetana-gmbh.de/avetana-gmbh/produkte/jsr82.eng.xml
`
`number of states is rather weak. In the end we chose our mo-
`del to be a left-to-right HMM with 8 states for convenience.
`
`Filtering
`Before the actual recognition process our system applies two
`filters to the vector data establishing a minimum representa-
`tion of a gesture before being forwarded to the HMM for
`training or recognition. The first filter is a simple threshold-
`filter eliminating all vectors which do not contribute to the
`characteristic of a gesture in a significant way, i.e. all (cid:2)a for
`which |(cid:2)a| < Δ. We call this filter the “idle state filter” and
`determined Δ to a value of Δ = 1.2g, g being the accelera-
`tion of gravity. The second filter is called “directorial equi-
`valence filter” and eliminates all vectors which are rough-
`ly equivalent to their predecessor and thus contribute to the
`characteristic of a gesture only weakly. Vectors are omitted
`if none of their components c ∈ {x, y, z} is all too different
`to the corresponding component of their predecessor, i.e. if
`c − (cid:2)a
`| ≤ for all c. was chosen to be 0.2 in the
`|(cid:2)a
`(n−1)
`(n)
`c
`case of the Wiimote.
`
`As Figure 5 shows, this filter would ideally lead to just four
`characteristic acceleration vectors in the case of the gesture
`“square”. In addition, Figure 6 demonstrates the reduction
`of the number of vectors for every reference gesture after
`applying both filters.
`
`(a) Before filtering
`
`(b) After filtering
`
`Figure 5. Effect of the directorial equivalence filter. Applying it would
`ideally lead to just four acceleration vectors for the gesture Square.
`
`Square Circle
`
`Roll
`
`Z
`
`Tennis
`
`Gesture
`
`140
`
`105
`
`70
`
`35
`
`0
`
`Averagenumberofvectors
`
`Figure 6. Reduction of vector data during filtering. The first bar for
`each gesture represents the average number of vectors after applying
`the first filter (“idle state”), the second bar the average number of vec-
`tors after applying the second, the “directorial equivalence” filter. As
`one can see the number of vectors are heavily reduced by this process
`which leads to more reliable as well as faster recognition results.
`
`13
`
`Petitioner Samsung Ex-1018, 0003
`
`

`

`Proceedings of the Second International Conference on Tangible and Embedded Interaction (TEI'08), Feb 18 20 2008, Bonn, Germany
`
`A
`
`B
`
`C
`
`D
`
`E
`
`F
`
`Participant
`
`100
`
`75
`
`50
`
`25
`
`0
`
`Averagerecognitionrate
`
`Figure 9. Average recognition rate of the four users. The results for the
`six participants were 84.0%, 87.8%, 87.8%, 92.0%, 93.4%, and 93.4%.
`
`on of the model and filters. We make the implementation of
`the gesture recognition library publicly available3 and as the
`Wiimote is a low-cost device we invite other researchers to
`extend and share their experiences.
`
`REFERENCES
`1. Nintendo. http://wii.nintendo.com
`2. LiveMove, AiLive Inc.
`http://www.ailive.net/liveMove.html.
`3. Baum, L.E. and Petrie, T. Statistical inference for
`probabilistic functions of finite state Markov chains.
`Annals of Mathematical Statistics, (1966), 1554-1563.
`4. Hofmann, F., Heyer, P. and Hommel, G. Velocity Profile
`Based Recognition of Dynamic Gestures with Discrete
`Hidden Markov Models. Proc. of the International
`Gesture Workshop on Gesture and Sign Language in
`Human-Computer Interaction, Springer London (2004),
`81–95.
`5. MacQueen, J. B. Some Methods for classification and
`Analysis of Multivariate Observations. Proc. of 5-th
`Berkeley Symposium on Mathematical Statistics and
`Probability, University of California Press 1967,
`281-297.
`6. M¨antyj¨arvi, J., Kela, J., Korpip¨a¨a, P. and Kallio S.
`Enabling fast and effortless customisation in
`accelerometer based gesture interaction. Proc. of the
`MUM ’04, ACM Press (2004), 25–31.
`7. M¨antyj¨arvi, J., Kela, J., Korpip¨a¨a, P., Kallio S., Savino,
`G., Jozzo L. and Marca, D. Accelerometer-based gesture
`control for a design environment. Personal Ubiquitous
`Computing, Springer London (2006), 285–299.
`8. M¨antyl¨a, V. M. Discrete hidden Markov models with
`application to isolated user-dependent hand gesture
`recognition. VTT Publications (2001).
`9. Rabiner, L.R. A Tutorial on Hidden Markov Models and
`Selected Applications in Speech Recognition. Proc. of
`the IEEE, IEEE (1989), 257–286.
`3http://wiigee.sourceforge.net
`
`Figure 7. Participant during the evaluation of the gesture recognition.
`
`To analyze the determined results we trained the gesture re-
`cognition system with the collected data. The system was
`trained using the leave-one-out method to make sure that the
`models were evaluated on sequences that were not used for
`training. That means for each participant fifteen training sets
`each containing the five gestures were computed. These trai-
`ning sets were used to recognize the remaining five gestu-
`res. The average rate of correctly recognized gestures was
`90 percent. The averaged recognition rate for each of the fi-
`ve gestures is shown in Figure 8. The averaged recognition
`rate for the six participants is shown in Figure 9.
`
`Square Circle
`
`Roll
`
`Z
`
`Tennis
`
`Gesture
`
`100
`
`75
`
`50
`
`25
`
`0
`
`Averagerecognitionrate
`
`Figure 8. Average recognition rate of the five gestures. The results for
`the five gestures were Square = 88.8%, Circle = 86.6%, Roll = 84.3%,
`Z = 94.3%, and Tennis = 94.5%.
`
`CONCLUSION
`Developing new intelligent user interfaces involves experi-
`mentation and testing of new devices for interaction tasks.
`In our research, we are working in the field of multimodal
`user interfaces including visual, acoustic and haptic I/O. Ba-
`sed on the Wiimote we developed a gesture recognition that
`employs state of the art recognition methodology such as
`HMM, filters and classifiers, and aim to optimize hand ge-
`sture recognition for the Wiimote. As the gestures can be
`user-chosen the system is not limited to predefined gestu-
`res but allows each user to train and use individual gestures
`for a personalized user interaction with gestures. To be ab-
`le to measure recognition results we trained and evaluated
`the system based on a set of reference gestures taken to be
`relevant for different task such as gaming, drawing or brow-
`sing. The recognition results vary between 85 to 95 percent,
`which is promising but leaves room for further optimizati-
`
`14
`
`Petitioner Samsung Ex-1018, 0004
`
`