`
`DECLARATION OF GORDON MACPHERSON
`
`I, Gordon MacPherson, am over twenty-one (21) years of age. I have never been
`convicted of a felony, and I am fully competent to make this declaration. I declare the following
`to be true to the best of my knowledge, information and belief:
`
`1.
`
`I am Director, Board Governance & Policy Development of The Institute of Electrical
`and Electronics Engineers, Incorporated
`
`2.
`
`IEEE is a neutral third party in this dispute.
`
`3.
`
`I am not being compensated for this declaration and IEEE is only being reimbursed
`for the cost of the article I am certifying.
`
`4. Among my responsibilities as Director, Board Governance & Policy Development, I
`act as a custodian of certain records for IEEE.
`
`5.
`
`I make this declaration based on my personal knowledge and information contained
`in the business records of IEEE.
`
`6. As part of its ordinary course of business, IEEE publishes and makes available
`technical articles and standards. These publications are made available for public
`download through the IEEE digital library, IEEE Xplore.
`
`7.
`
`It is the regular practice of IEEE to publish articles and other writings including
`article abstracts and make them available to the public through IEEE Xplore. IEEE
`maintains copies of publications in the ordinary course of its regularly conducted
`activities.
`
`8. The article below has been attached as Exhibit A to this declaration:
`
`A. Mark Joselli, et al.; gRmobile: A Framework for Touch and
`Accelerometer Gesture Recognition for Mobile Games , published in 2009
`VIII Brazilian Symposium on Games and Digital Entertainment, date of
`conference October 8-10, 2009.
`
`9.
`
`I obtained a copy of Exhibit A through IEEE Xplore, where it is maintained in the
`ordinary course of IE
`iness. Exhibit A is a true and correct copy of the
`Exhibit, as it existed on or about October 13, 2023.
`
`445 Hoes Lane Piscataway, NJ 08854
`
`Petitioner Samsung Ex-1046, 0001
`
`

`

`10. The article and abstract from IEEE Xplore shows the date of publication. IEEE
`Xplore populates this information using the metadata associated with the publication.
`
`
`
`11.
`
`rk for Touch and Accelerometer Gesture
`
`Games and Digital Entertainment, date of conference October 8-10, 2009. Copies of
`the conference proceedings were made available no later than the last day of the
`conference. The article is currently available for public download from the IEEE
`digital library, IEEE Xplore.
`
`12. I hereby declare that all statements made herein of my own knowledge are true and
`that all statements made on information and belief are believed to be true, and further
`that these statements were made with the knowledge that willful false statements and
`the like are punishable by fine or imprisonment, or both, under 18 U.S.C. § 1001.
`
`I declare under penalty of perjury that the foregoing statements are true and correct.
`
`
`
`
`
`
`Executed on:
`
`
`
`
`
`
`
`
`
`
`Petitioner Samsung Ex-1046, 0002
`
`

`

`DocuSign Envelope ID: 63A03AC4-9B55-40CC-A4E4-45F9777F6EE4
`
`
`
`
`
`
`
`
`
`
`
`
`
`EXHIBIT A
`EXHIBIT A
`
`
`
`Petitioner Samsung Ex-1046, 0003
`
`Petitioner Samsung Ex-1046, 0003
`
`

`

`(cid:14)(cid:25)(cid:19)(cid:27)(cid:18)(cid:22)(cid:21)(cid:24)(cid:1)(cid:15)(cid:24)(cid:28)(cid:20)(cid:23)(cid:25)(cid:26)(cid:20)(cid:1)(cid:17)(cid:14)(cid:10)(cid:1)(cid:7)(cid:4)(cid:11)(cid:3)(cid:4)(cid:11)(cid:13)(cid:5)(cid:2)(cid:9)(cid:12)(cid:6)(cid:6)(cid:2)(cid:5)(cid:3)(cid:13)(cid:13)(cid:2)(cid:11)(cid:5)(cid:15)(cid:5)(cid:2)(cid:5)(cid:6)(cid:16)(cid:9)(cid:8)(cid:8)(cid:8)(cid:16)(cid:7)(cid:15)(cid:15)(cid:5)
`
`
`
`2009 VIII Brazilian Symposium on Digital Games and EntertainmentVIII Brazilian Symposium on Games and Digital Entertainment
`
`gRmobile: A Framework for Touch and
`Accelerometer Gesture Recognition for Mobile
`Games
`
`Mark Joselli, Esteban Clua
`MediaLab, IC-UFF
`{mjoselli — esteban}@ic.uff.br
`
`Abstract—Mobile phone games are usually design to
`be able to play using the traditional number pads of the
`handsets. This is stressfully difficult for the user interaction
`and consequently for the game design. Because of that, one
`of the most desired features of a mobile games is the usage
`of few buttons as possible. Nowadays, with the evolution
`of the mobile phones, more types of user interaction are
`appearing, like touch and accelerometer input. With these
`features, game developers have new forms of exploring
`the user input, being necessary to adapt or create new
`kinds of game play. With mobile phones equipped with 3D
`accelerometers, developers can use the simple motion of
`the device to control the game or use complex accelerated
`gestures. And with mobile phones equipped with the touch
`feature, they can use a simple touch or a complex touch
`gesture recognitions. For the gesture to be recognized
`one can use different methods like simple brute force
`gestures, that only works well on simple gestures, or
`more complex pattern recognition techniques like hidden
`Markov fields, fuzzy logic and neural networks. This
`work presents a novel framework for touch/accelerometer
`gesture recognition that uses hidden Markov model for
`recognition of the gestures. This framework can also be
`used for the development of mobile application with the
`use of gestures.
`Index Terms—Mobile Games, Gesture Recognition, Mo-
`tion Sensors, Touch Phones,Tangible User Interfaces.
`
`I. INTRODUCTION
`Digital games are defined as real-time multime-
`dia applications that have time constraints to run
`their tasks. If the game is not able to execute its
`processing under some time threshold, it will fail
`[1]. Mobile games are also real-time multimedia
`application that runs on mobile phones that have
`time constraints and many others constraints [2],
`when compared to PC or console games,
`like:
`hardware constraints (processing power and screen
`
`size); user input, (buttons, voice, touch screen and
`accelerometers); and different operating systems,
`like Android, iPhone OS, Symbian and Windows
`Mobile. This makes streamilly difficult for the de-
`sign and development of mobile games.
`On the other hand, mobile games can have unique
`characteristics, making unique type of games: lo-
`cation based games [3], [4], voice based games
`[5], accelerometer based games [6], camera based
`games [7] and touch based games [8]. In order to
`develop good mobile games, they must be design to
`take advantages of such unique characteristics into
`gameplay [9].
`Mobile game phones are a growing market [10]
`and in 2010 the sales of smartphones is expected
`to suppress the laptops sales [11]. More than 10
`million of people worldwide play games on mobile
`phones and handheld devices [12] and the world-
`wide mobile gaming revenue is expected to reach
`$9.6 billion by 2011 [13]. These are important
`motivations for game developers and designer to
`create blockbusters games.
`One special characteristic of most mobile phones
`is that the user interaction is made mostly through
`number input [6], [14]. Because of that, the design
`of games must deal with this fact and design the
`game to use as few buttons as possible, like just
`one button games [15] and no-buttons at all games
`[14].
`The evolution of mobile phones increase the
`processing power of such devices and also new
`forms of input, like touch screen devices and de-
`vices equipped with accelerometers. With the de-
`velopment of touch phones, like Motorola a1200,
`Htc Diamond, Sony Ericson w960i, Samsung Ultra
`
`
`978-0-7695-3963-8/09 $26.00 © 2009 IEEE978-0-7695-3963-8/09 $26.00 © 2009 IEEE
`
`DOI 10.1109/SBGAMES.2009.24DOI 10.1109/SBGAMES.2009.24
`Authorized licensed use limited to: Bradley Berg. Downloaded on October 13,2023 at 15:12:45 UTC from IEEE Xplore. Restrictions apply.
`
`
`
`141141
`
`Petitioner Samsung Ex-1046, 0004
`
`

`

`(cid:14)(cid:25)(cid:19)(cid:27)(cid:18)(cid:22)(cid:21)(cid:24)(cid:1)(cid:15)(cid:24)(cid:28)(cid:20)(cid:23)(cid:25)(cid:26)(cid:20)(cid:1)(cid:17)(cid:14)(cid:10)(cid:1)(cid:7)(cid:4)(cid:11)(cid:3)(cid:4)(cid:11)(cid:13)(cid:5)(cid:2)(cid:9)(cid:12)(cid:6)(cid:6)(cid:2)(cid:5)(cid:3)(cid:13)(cid:13)(cid:2)(cid:11)(cid:5)(cid:15)(cid:5)(cid:2)(cid:5)(cid:6)(cid:16)(cid:9)(cid:8)(cid:8)(cid:8)(cid:16)(cid:7)(cid:15)(cid:15)(cid:5)
`
`Smart F520 and Nokia N810, new forms of user
`interaction has appeared through the use of the
`finger or pen. This innovation has led to change the
`way users interact with the operation system and
`with games.
`With the popularization of the use of accelerom-
`eter by the Nintendo Wiimote [16], the major mo-
`bile phone manufactures had also equipped their
`hardware with accelerometer, like Nokia N95, Sony
`ericson F305, Samsung Omnia and Motorola W7,
`among others. But this new form of user interaction
`has not led to major change on the interaction. This
`is mostly because programs/games only uses the
`accelerometer data as orientation.
`The iPhone was one of the first devices that
`is equipped with touch screen and accelerometer
`that has mostly of the user input made though
`touch or motion, soon others companies followed
`this tendency, like: RIM Blackberry Storm, Nokia
`5800, LG Arena, and many others. They basically
`use touch for user interaction, and the use of the
`accelerometer data is restricted for orientation, just
`like the others phones equipped with accelerometer.
`This paper tries to fulfill a gap on user interaction
`by providing a framework for gesture recognition
`though touch input or motion input, that can be used
`for games or programs.
`The gesture recognition is a type of pattern recog-
`nition and can be made by different ways like:
`brute force [17], fuzzy logic [18], Gabor wavelet
`transform [19], hidden Markov model [20], Support
`Vector Machine [21] and neural networks [22],
`[23]. This work has developed a framework that
`can be used for gesture recognition using hidden
`Makov model. In order to generate and recognize
`the gestures database the proposed framework is
`divided in two parts: one for database constriction
`and another for the gesture recognition.
`Summarizing, this work provides the following
`contributions:
`• A novel architecture for touch/motion gesture
`recognition on mobile phones;
`• Presentation of performance and tests of the
`framework showing that it can be used in real-
`time;
`• Recognition test showing a high accuracy rate;
`The paper is organized as follows: Section 2
`presents some related works on the mobile develop-
`
`ment and gesture recognition on devices equipped
`with touch screen and devices equipped with ac-
`celerometers. Section 3 presents and explain the
`gesture recognition framework. Section 4 present
`and discuss some results of the use of the frame-
`work. Section 5 presents the conclusions and future
`works.
`
`II. RELATED WORK
`Since the gRmobile has two kinds of gestures
`recognition (thought touch input or accelerometer
`motion input), this section is divided in two sub-
`sections: one for touch screen devices were user
`interaction and gesture recognition for this kind of
`device works; and another for accelerometer devices
`presenting user interaction works and gesture recog-
`nition approaches.
`This work does not cover the related work on
`mobile game development. For this purpose the
`authors suggest the works [24], [2] which covers
`state of the art for this topic.
`
`A. Touch Devices
`Nowadays, more and more devices are coming
`with touch screen, and most of this is because of the
`decreased in the respective price [25]. Touch screen
`phone devices has the characteristics of having very
`few buttons and most of its users input interfaces are
`made through touch by finger or pen. For example
`the Blackberry Storm has about only 8 buttons and
`almost all of its user interaction is made by touch.
`Most touch screen devices can have two kinds of
`input: dragged and pressed. The first is used when
`the user touches softly and can be used as a mouse
`being dragged. The second is when the user press
`hard on the device, and can be used as a mouse
`buttons pressed. Also modern devices has multi-
`touch screen devices like iPhone, Android T-G1 and
`Blackberry Storm, among others.
`Some of these devices have used some of this
`types of input as gestures to enable friendly user
`interaction:
`like dragging for changing the web
`page, zooming options on photo view and many
`others features. But in third-party mobile software
`and games this use is very restrict.
`Narayanaswamy et al [26] shows an implemen-
`tation of handwrite recognition on PDAs using
`Hidden Markov Model. Wei et al. [27] presents a
`
`Authorized licensed use limited to: Bradley Berg. Downloaded on October 13,2023 at 15:12:45 UTC from IEEE Xplore. Restrictions apply.
`
`142142
`
`Petitioner Samsung Ex-1046, 0005
`
`

`

`(cid:14)(cid:25)(cid:19)(cid:27)(cid:18)(cid:22)(cid:21)(cid:24)(cid:1)(cid:15)(cid:24)(cid:28)(cid:20)(cid:23)(cid:25)(cid:26)(cid:20)(cid:1)(cid:17)(cid:14)(cid:10)(cid:1)(cid:7)(cid:4)(cid:11)(cid:3)(cid:4)(cid:11)(cid:13)(cid:5)(cid:2)(cid:9)(cid:12)(cid:6)(cid:6)(cid:2)(cid:5)(cid:3)(cid:13)(cid:13)(cid:2)(cid:11)(cid:5)(cid:15)(cid:5)(cid:2)(cid:5)(cid:6)(cid:16)(cid:9)(cid:8)(cid:8)(cid:8)(cid:16)(cid:7)(cid:15)(cid:15)(cid:5)
`
`study about using pen gestures instead of buttons in
`a mobile FPS game. They showed that users have
`little preference in using buttons over gestures and
`sometimes prefer the use of gestures. The gRmobile
`also could be used for handwrite recognition and
`gesture for games but a gesture database must be
`constructed in order to have the this functionality in
`the framework (in the case of handwrite recognition
`all the alphabet must be in the database).
`Since the touch screens have similar behavior as
`the mouse, mouse gesture recognizer [28], [29], [30]
`could be adapted in order to be used by touch screen
`devices. But this could be very hard since they
`are not adapted for the low processing power and
`memory constraints of such devices. The gRmobile
`is very adapted to such devices having a very good
`performance as will be showed in the performance
`evaluation section.
`Even tough touch devices enables much more
`freedom when compared with buttons based phones,
`the input error by those devices are higher, as shows
`by Hoggan et al. [31] using keyboard input tests.
`
`B. Accelerometer Devices
`Accelerometer devices are getting more and more
`popular. This allows the interaction though the form
`of gestures recognition, being the Nintendo Wii
`game console the most prominent example of this
`new form of interaction. This approach allows users
`to become more engaged to video games [32],
`whose experience is not only affected by button
`pressing and timing but also by movement. Also
`Sony’s Playstation 3 has a controller that is equipped
`with accelerometers.
`Nowadays, most smartphones, and also some
`mobile phones, come equipped with accelerometer.
`This allows the use of motion and gestures as user
`input, but very little has been done in the field.
`Most mobile operating systems [11] only uses the
`motion to choose the screen orientation. And also
`most games only uses the “tilt” of the screen and
`not gestures as input like the works [14], [6], [33].
`There are many relevant work related to gestures
`recognition for accelerometer devices. With the us-
`age of the Wiimote can be highlighted the works
`[34], [35]. These works use Hidden Markov Models
`(HMM) as the recognition algorithm and the [34]
`presents a lower recognition of 66 % and [35] shows
`
`a lower recognition rate of 84 %. This work also
`uses HMM as the recognition algorithm, but it is
`adapted for the lower processing power of mobile
`devices.
`Accelerometer gesture recognition requires an
`intensive task to be achieved on a mobile phone.
`The work by Choi et al. [36] presents a Bayesian
`network algorithm with its computation performed
`in the PC to recognize numbers written on the air
`by accelerometer mobile phones with an average
`recognition rate of 97 %. Also [37] that shows a
`HMM recognition algorithm also implemented on
`the PC with the gestures done by mobile phones
`with a recognition rate of more that 99 %. There
`are also some work in development like [21], [38]
`shows two recognition algorithms, a HMM and a
`Support Vector Machine, which can have an average
`recognition rate of 96 %.
`the PC, MobiToss [39]
`Without
`the use of
`presents the use of mobile phone to use simple
`gestures to interact with large public displays with
`all the processing made on the phone.
`
`III. FRAMEWORK OVERVIEW
`
`The framework is build using Java language. The
`choice to use Java was to achieve a higher number of
`devices with the same framework, allowing its usage
`by any accelerated/touch device that can handle the
`acceleration data and/or touch data, and that has
`a Java virtual machine,
`like many devices from
`different manufacturers: Blackberries, Nokias, Sony
`Ericson, Motorola, Android, LG and Samsung. Also
`this framework can be used in a PC with a touch
`device like Microsoft surface or an accelerometer
`device like the Wiimote.
`Gesture recognition with touch/accelerated de-
`vices are represented by their patterns of the input
`data. The recognition is made by comparing the
`input pattern with the database pattern, checking if
`they match. In order to extract the pattern from the
`input data stream and comparing with the database
`pattern, this data must be prepared and analyzed.
`This work uses Hidden Markov Models in order
`to fulfill that need. In order to build a database, the
`framework needs to train and saves a set of gestures,
`which also used the mobile phone for this need.
`The proposed framework has the following steps:
`
`Authorized licensed use limited to: Bradley Berg. Downloaded on October 13,2023 at 15:12:45 UTC from IEEE Xplore. Restrictions apply.
`
`143143
`
`Petitioner Samsung Ex-1046, 0006
`
`

`

`(cid:14)(cid:25)(cid:19)(cid:27)(cid:18)(cid:22)(cid:21)(cid:24)(cid:1)(cid:15)(cid:24)(cid:28)(cid:20)(cid:23)(cid:25)(cid:26)(cid:20)(cid:1)(cid:17)(cid:14)(cid:10)(cid:1)(cid:7)(cid:4)(cid:11)(cid:3)(cid:4)(cid:11)(cid:13)(cid:5)(cid:2)(cid:9)(cid:12)(cid:6)(cid:6)(cid:2)(cid:5)(cid:3)(cid:13)(cid:13)(cid:2)(cid:11)(cid:5)(cid:15)(cid:5)(cid:2)(cid:5)(cid:6)(cid:16)(cid:9)(cid:8)(cid:8)(cid:8)(cid:16)(cid:7)(cid:15)(cid:15)(cid:5)
`
`• Segmentation: is used to determine when the
`gesture begins and when it ends;
`• Filtering: is used in order to eliminate some
`parts of the data stream that do not contribute
`to the gesture;
`• Quantitizer: is used to approximate the stream
`of input data into a smaller set of values;
`• Model: is used to compute likelihood of ana-
`lyzed gestures.
`• Classifier: is used in order to identify the input
`gesture accordingly to the database.
`All the steps from the mobile phone to the user
`feedback are illustrated in figure 1. It is possible to
`notice that the framework has two distinct modes:
`one for gesture training, i.e, build the database, and
`one for gesture recognition, i.e, comparing the input
`gesture with the database.
`A. Segmentation
`Segmentation is used mainly to automatic de-
`terminate the begin and end of the gesture. For
`identifying the begin and end of touch gestures is
`very easy since the begin of the gesture is when
`the user first touch the screen and it ends when
`the user release the screen. For the accelerometer
`gestures the segmentation is more difficult since
`the data is continues in a frequency normally that
`varies from 20 Hz to 80 Hz. The data that comes
`from the accelerometer is 3 floats representing the
`acceleration in three axis, as figure 2 illustrate, that
`ranges from -3G to 3G (G meaning the gravity).
`
`and the end of the gesture. This seems like an easy
`approach, since it avoids computations to determine
`the begin and the end of a gesture like automatic
`segmentation must have. But on the other hand, the
`interaction is worst than no-button segmentation.
`In a mobile phones sometimes the pressing of
`buttons in a game is normally hard since it was
`designed for number dialing. This comes as another
`reason why this work has decided to do a no-button
`segmentation. This work does a similar approach as
`the works [40], [21].
`In order to correct segmentate an accelerometer
`gesture, a definition of this kind of gesture is
`needed. This definition is made during the obser-
`vation of the accelerated data and the movement
`of user during the recognition of different gestures.
`Normally gestures begins with a fast acceleration, a
`continuous direction change during the gesture, and
`it ends with a stop of the movement. In this work,
`the authors have observed that normally, a good
`gesture needs a duration of more than 0.6 seconds
`and less than 2 seconds.
`To correct segmentate the accelerometer gesture
`based on the definitions some preprocessing of the
`accelerated data is needed. This work uses a simple
`mechanism. It checks the size of a vector made by
`the sums of the derivative, which is the difference
`between the axis float and the last axis float.
`(cid:2)
`
`(xk − xk−1)2 + (yk − yk−1)2 + (zk − zk−1)2
`(1)
`If the D value is bigger than 0.3, which was
`chosen during an extensively empirical study of
`gestures, the segmentation begins. And if the gesture
`is happening and this values drops to bellow 0.1 the
`gesture ends, i.e, it is assumed that the accelerom-
`eter device is in an idle position.
`
`D =
`
`Fig. 2. The axis accelerometer
`
`There are some accelerometer gesture recognition
`systems, like [35], that does the segmentation by
`using a button based segmentation,
`i.e,
`the user
`needs to press a button in order to sign the beginning
`
`B. Filtering
`This pass is used in order to eliminate some parts
`of the data stream that do not contribute to the
`gesture. This work uses two kinds of filters in order
`to eliminate noises and data that are very similar.
`When a gesture is made, the data stream of the
`gesture may contain errors that if are sent to the
`HMM, some errors on the recognition may occur.
`In order to avoid such errors, a low pass filter is
`
`Authorized licensed use limited to: Bradley Berg. Downloaded on October 13,2023 at 15:12:45 UTC from IEEE Xplore. Restrictions apply.
`
`144144
`
`Petitioner Samsung Ex-1046, 0007
`
`

`

`(cid:14)(cid:25)(cid:19)(cid:27)(cid:18)(cid:22)(cid:21)(cid:24)(cid:1)(cid:15)(cid:24)(cid:28)(cid:20)(cid:23)(cid:25)(cid:26)(cid:20)(cid:1)(cid:17)(cid:14)(cid:10)(cid:1)(cid:7)(cid:4)(cid:11)(cid:3)(cid:4)(cid:11)(cid:13)(cid:5)(cid:2)(cid:9)(cid:12)(cid:6)(cid:6)(cid:2)(cid:5)(cid:3)(cid:13)(cid:13)(cid:2)(cid:11)(cid:5)(cid:15)(cid:5)(cid:2)(cid:5)(cid:6)(cid:16)(cid:9)(cid:8)(cid:8)(cid:8)(cid:16)(cid:7)(cid:15)(cid:15)(cid:5)
`
`Touch
`
`Motion
`
`Segmentation
`
`Segmentation
`
`Filter
`
`Model
`
`Train
`
`Recognition
`
`Gesture
` Database
`
`Classifier
`
`Filter
`
`Quantitizer
`
`Model
`
`Train
`
`Recognition
`
`Gesture
` Database
`
`Classifier
`
`Fig. 1. System block diagram with major components of the framework.
`
`applied which is a very common filter used for noise
`remove.
`
`When the gesture is made, there are a lot of
`data on the data stream that does not contribute to
`the overall characteristic of the gesture. In order to
`diminished the data passed to the HMM, this work
`uses an idle threshold filter. This uses the same
`equation 1, and if D value is less than 0.2, it is
`not included in the gesture data stream.
`
`C. Quantitizer
`
`This step is only used for accelerated gestures.
`Because the accelerometer continues sends its data
`to the processor, the amount of data may be to big
`to for handling into a single HMM. Also, since
`the amount of RAM memory of mobile phones
`are not so big, the use of a quantitizer keeps less
`information in the gesture database. This work uses
`a k-mean algorithm which is a method of cluster an-
`alyzer. The algorithm aims to partition n observation
`
`Authorized licensed use limited to: Bradley Berg. Downloaded on October 13,2023 at 15:12:45 UTC from IEEE Xplore. Restrictions apply.
`
`145145
`
`Petitioner Samsung Ex-1046, 0008
`
`

`

`(cid:14)(cid:25)(cid:19)(cid:27)(cid:18)(cid:22)(cid:21)(cid:24)(cid:1)(cid:15)(cid:24)(cid:28)(cid:20)(cid:23)(cid:25)(cid:26)(cid:20)(cid:1)(cid:17)(cid:14)(cid:10)(cid:1)(cid:7)(cid:4)(cid:11)(cid:3)(cid:4)(cid:11)(cid:13)(cid:5)(cid:2)(cid:9)(cid:12)(cid:6)(cid:6)(cid:2)(cid:5)(cid:3)(cid:13)(cid:13)(cid:2)(cid:11)(cid:5)(cid:15)(cid:5)(cid:2)(cid:5)(cid:6)(cid:16)(cid:9)(cid:8)(cid:8)(cid:8)(cid:16)(cid:7)(cid:15)(cid:15)(cid:5)
`
`into k clusters in which each observation belongs to
`the cluster with the nearest mean. This is a similar
`approach as the work [35].
`For this approach, a k value must be chosen. For
`this work we selected k = 14 since experiments
`from [35] shows that this values is optimal for the
`Wiimote, which can be extended to the phone ac-
`celerometers, since they have a similar architecture.
`
`D. Hidden Markov Model
`A hidden Markov model (HMM) is a popular
`statistical tool for gesture/pattern recognition. This
`work has based its implementation in an open source
`HMM implementation [41] and in an open source
`HMM Wii gesture recognition [42].
`This work uses left-to-right HMM with 8 states
`for each gesture. The reason why this work choose
`this configuration is because it is a very efficient for
`accelerometer recognition, following the tests made
`in [35]. For the training process, the HMMs with the
`iterative Baum-Welch algorithm was used. And for
`recognition, the forward-backward algorithm was
`used. More information on this algorithms can be
`obtained in [43], [44], [45].
`
`E. Classifier
`The classifier is used in order to identify gesture
`selecting the gesture with more likehood between
`the input gesture and the database gesture. This
`work uses a naive Bayes classifier also called sim-
`ple Bayesian classifier [46]. Naive Bayes is sim-
`ple probabilistic classifier based on the so-called
`Bayesian theorem and it is a well known algorithm
`both in statistics and machine learning.
`Because the accelerometer data input has some
`noise movements, i.e., movements that are not ges-
`tures. Because of that, a probability threshold is
`included, so that any gesture probability below that
`threshold is considered as a noise. The value of this
`threshold was determined according to empirical
`values.
`
`IV. RESULTS EVALUATION
`All tests of this work were made in a BlackBerry
`Storm 9530 [47] which has a 528 MHz Qualcomm
`processor with 128 MB of RAM,
`touch screen
`and accelerometer. In order to evaluate properly
`the gRmobile framework an application to train,
`
`recognize and save gestures were developed for the
`BlackBerry Storm, a screenshot of the application
`can be seen on figure 3.
`
`Fig. 3. An Screenshot of the application.
`
`Two types of tests were made in order to validate
`the architecture performance, one for evaluating the
`impact that the architecture can have on the mobile
`phone, and another for recognition,
`in order to
`evaluate the accuracy of the gRmobile. These tests
`are presented in the next two subsections.
`
`A. Performance Evaluation
`There are some available frameworks for gesture
`recognition but most of them cannot be used in mo-
`bile phones since the processing power is very low
`
`Authorized licensed use limited to: Bradley Berg. Downloaded on October 13,2023 at 15:12:45 UTC from IEEE Xplore. Restrictions apply.
`
`146146
`
`Petitioner Samsung Ex-1046, 0009
`
`

`

`(cid:14)(cid:25)(cid:19)(cid:27)(cid:18)(cid:22)(cid:21)(cid:24)(cid:1)(cid:15)(cid:24)(cid:28)(cid:20)(cid:23)(cid:25)(cid:26)(cid:20)(cid:1)(cid:17)(cid:14)(cid:10)(cid:1)(cid:7)(cid:4)(cid:11)(cid:3)(cid:4)(cid:11)(cid:13)(cid:5)(cid:2)(cid:9)(cid:12)(cid:6)(cid:6)(cid:2)(cid:5)(cid:3)(cid:13)(cid:13)(cid:2)(cid:11)(cid:5)(cid:15)(cid:5)(cid:2)(cid:5)(cid:6)(cid:16)(cid:9)(cid:8)(cid:8)(cid:8)(cid:16)(cid:7)(cid:15)(cid:15)(cid:5)
`
`when compared to a PC. gRmobile was designed to
`be used with constraint hardware of mobile phones.
`The authors of this work have observed that the
`size of gesture database influenciates on the total
`time of the recognition. Another observation is that
`the time for touch gesture is lesser than motion
`gestures, since it has much less data. Table I shows
`the numerical results in average time with gesture
`different database sizes.
`The results shows that with a database with ten or
`less gestures, the gRmobile can be used in real-time
`applications, such as games, without major impact
`on the overall performance of the application. Since
`more than ten gestures becomes inpratical due to
`the fact that user must learn all different gestures,
`like [37] argue and the authors of this work agreed.
`These results shows that gRmobile provides a good
`rate for real-time gesture recognition.
`Also this architecture were tested in a PC with an
`3500+ Athlon64 processor with 2GB RAM memory
`with Wiimote as the accelerometer device and the
`mouse simulating the touch device. This tests shows
`that resource consumption is insignificant in a PC
`even with a gesture database of twenty gestures.
`B. Recognition Evaluation
`In order to test the accuracy of the recognition,
`a dataset of gestures have been created. These
`work have defined ten different gestures for motion
`gestures, and ten gestures for touch gestures similar
`to the motion gestures. These gestures can be seen
`on figure 4.
`
`C
`
`Circle
`Clockwise
`
`Circle
`Counter
`Clockwise
`
`Rool
`Left
`
`Rool
`Right
`
`examples overall. The group consists of three men
`users (participants A, B and D) and one women
`user (participant C) with age ranging between 21
`and 42. None of the participants was physically
`disabled. One participant have major experience
`with touch/accelerometer device (participant A),
`two have minor experience with such devices (par-
`ticipant B and C) and one have none experience with
`the touch and accelerometer mobile phone devices
`(participant D). The results can be seen on table II.
`These results show a high fidelity recognition
`rate of the gRmobile framework with an average
`recognition rate of 89% in motion gestures and 98
`% for touch gestures.
`Also the results shows that the specialist user
`obtained 99.9 % of recognition in motion gestures
`and 100 % in touch gestures. The low experience
`users have an 88.5 % in motion gestures and 98 %
`in touch gestures. The no experience user have 79
`% in motion gestures and 96 % in touch gestures.
`This results show that touch gestures are easily to
`be performed and recognized. The tests also shows
`that gestures can be used with both user with big
`expertise and no expertise on the subject.
`Figure 5 shows the average recognition in % rate
`of motion gestures. This results shows that the P
`gesture has the lowest recognition rate, an average
`rate of 75%, and RollLef t gesture has the highest
`recognition rate, an average rate of 97.5%.
`
` Square
`
`Z
`
`M
`
`U
`
`P
`
`Fig. 4. The gesture database used in tests.
`
`Fig. 5. Average Recognition Rate of the Motion Gestures.
`
`All gestures were repeated ten times by a group of
`four different users, which provided a four hundred
`
`Figure 6 shows the average recognition in %
`rate of touch gestures. This results shows that the
`
`Authorized licensed use limited to: Bradley Berg. Downloaded on October 13,2023 at 15:12:45 UTC from IEEE Xplore. Restrictions apply.
`
`147147
`
`Petitioner Samsung Ex-1046, 0010
`
`

`

`(cid:14)(cid:25)(cid:19)(cid:27)(cid:18)(cid:22)(cid:21)(cid:24)(cid:1)(cid:15)(cid:24)(cid:28)(cid:20)(cid:23)(cid:25)(cid:26)(cid:20)(cid:1)(cid:17)(cid:14)(cid:10)(cid:1)(cid:7)(cid:4)(cid:11)(cid:3)(cid:4)(cid:11)(cid:13)(cid:5)(cid:2)(cid:9)(cid:12)(cid:6)(cid:6)(cid:2)(cid:5)(cid:3)(cid:13)(cid:13)(cid:2)(cid:11)(cid:5)(cid:15)(cid:5)(cid:2)(cid:5)(cid:6)(cid:16)(cid:9)(cid:8)(cid:8)(cid:8)(cid:16)(cid:7)(cid:15)(cid:15)(cid:5)
`
`TABLE I
`NUMERICAL RESULTS FROM PERFORMANCE EVALUATION WITH DIFFERENT DATABASE SIZES IN AVERAGE TIME IN MILLISECONDS.
`
`# of gesture in the database
`5
`10
`15
`20
`
`time accelerometer (ms)
`52
`87
`125
`186
`
`time touch (ms)
`40
`71
`107
`154
`
`TABLE II
`RECOGNITION TESTS RESULTS IN % OF ACCURACY BETWEEN ALL PARTICIPANTS.
`
`Gesture
`C
`Circle Clockwise
`Circle counter Clockwise
`Roll Left
`Roll Right
`Square
`Z
`M
`U
`P
`
`User D
`User C
`User B
`User A
`Motion
`Touch Motion
`Touch Motion
`Touch Motion
`Touch
`100
`100
`90
`100
`100
`90
`90
`100
`100
`100
`100
`100
`80
`100
`80
`90
`100
`100
`90
`90
`100
`100
`80
`100
`100
`100
`100
`100
`100
`100
`90
`100
`100
`100
`100
`100
`90
`100
`100
`100
`100
`100
`80
`100
`90
`100
`70
`80
`100
`100
`80
`100
`90
`100
`70
`90
`100
`100
`90
`100
`80
`90
`70
`100
`100
`100
`80
`100
`80
`100
`80
`100
`90
`100
`70
`100
`80
`90
`60
`100
`
`Square gesture has the lowest recognition rate, with
`an average rate of 95% and RollLef t, RollRight
`and U gestures have the highest recognition rate,
`with an average rate of 100%.
`
`Fig. 6. Average Recognition Rate of the Touch Gestures.
`
`V. CONCLUSION
`The mobile gaming market is a growing market,
`motivating game developers to have more focus
`on mobile development. Also mobile phones now
`have much more processing power allowing mobile
`games to have more complexity.
`
`Even touch screens and accelerometers devices
`becoming available in mostly devices, current mo-
`bile games almost do not explore these features.
`This work has presented a novel framework, the
`gRmobile,
`that can recognize tou