TITLE OF THE INVENTION
`
`Gesture Recognition
`
`BACKGROUND OF THE INVENTION
`
`The invention relates to gesture recognition in particular gesture recognition by processing of
`
`time series of positional inputs received by a two-dimensional (2D) touch sensor, such as a
`
`capacitive or resistive touch sensor. The invention may also be applied to one-dimensional
`
`(1D) touch sensors, and the principles could also be applied to three-dimensional sensors. It
`
`may also be applied to proximity sensors, where no physical contact, i.e. touch, with a
`
`sensing surfaceis involved.
`
`The invention can be applied to sensing surfaces operable by a humanfinger, or a stylus.
`
`1D and 2D capacitive and resistive touch sensors have been in widespread use for many
`
`years. Examples include the screens of personaldigital assistants (PDAs), MP3 audio player
`
`controls, mobile phone keypads and/or displays, and multimedia devices. The touchpad in
`
`notebook computers provided in place of a mouseis another form of 2D capacitive touch
`
`sensor. 2D sensors are also provided in many white goods such as ovens and blenders.
`
`Detailed descriptions of 2D capacitive sensors have been given manytimes, for example in
`
`patents and patent applications with the inventor Harald Philipp such as
`
`US 2005/0041018 A1, US 2007/0247443 A1, US 2007/0257894 A1, and US 2007/0279395
`
`A1, the contents of which are incorporated herein in their entirety.
`
`Other prior art examples of touch screensare asfollows.
`
`US 3,593,115 shows a touch element having triangulated shapes for determining object
`
`position. However this scheme requires numerous secondary electrode connections as well
`
`as two or morelayers of construction, increasing construction costs and reducing
`
`transparency.
`
`US 5,650,597 shows a 2D sensing method whichin its active area requires only one layer
`
`but requires large numbersof electrode connections. Resistive strips resolve one axis of
`
`position, and the accuracy is dependent on the tolerance of large numbersofresistive strips.
`
`This method however does suppress hand shadoweffects.
`
`Petitioner Samsung Ex-1021, 0001
`
`Petitioner Samsung Ex-1021, 0001
`
`

`

`2
`
`US 6,297,811 describes a touch screen using triangulated wire outline electrode shapesto
`
`create field gradients. However this patent suffers from the problem thatit is difficult to scale
`
`up the screen size, as the numberof electrode connections to a sensing circuit is one per
`
`triangle. It is desirable to dramatically reduce the number of connections in order to reduce
`
`cost and simplify construction. Also it is desirable to use solid shapesrather than wire
`
`outlines which are more expensiveto construct. This method however does suppress hand
`
`shadoweffects.
`
`Gesture recognition has also been used for many years in such devices. An early example is
`
`character recognition in PDAs, such as the original machines from Palm. Tracking finger
`
`motion, and single and double taps on a notebook touchpad is another long used example.
`
`More recently, gesture recognition has been incorporated into handheld devices such as the
`
`Apple iPhone (RTM). Prior art patent publications on touch screensthat involve gesture
`
`recognition are also large in number, with significant numbers of publications from Synaptics,
`
`Inc. and also more recently Apple Computer, Inc, for example.
`
`US 2007/152984 A1 assigned to Apple Computer, Inc. discloses a portable communication
`
`device with multi-touch input which detects one or more multi-touch contacts and motions
`
`and performs one or more operations on an object based on the one or more multi-touch
`
`contacts and/or motions.
`
`US 2002/015024 A1 assigned to University of Delaware discloses simultaneously tracking
`
`multiple finger and palm contacts as hands approach, touch, and slide across a proximity-
`
`sensor. Segmentation processing extracts shape, position and surface proximity features for
`
`each contact and a persistent path tracker is used to detect individual contact touchdown
`
`and liftoff. Combinatorial optimization modules associate each contact's path with a
`
`particular fingertip, thumb, or palm of either hand on the basis of biomechanical constraints
`
`and contact features. Classification of intuitive hand configurations and motions enables
`
`unprecedented integration of typing, resting, pointing, scrolling, 3D manipulation, and
`
`handwriting into a versatile, ergonomic computer input device.
`
`US 6028271, US 6414671 and US 6750852are related patents assigned to Synaptics, Inc.
`
`which disclose gesture recognition of an object on a touch-sensor pad and for cursor motion.
`
`Tapping, drags, pushes, extended drags and variable drags gestures are recognized by
`
`analyzing the position, pressure, and movementof the conductive object on the sensor pad
`
`during the time of a suspected gesture, and signals are sent to a hostindicating the
`
`occurrenceof these gestures.
`
`Petitioner Samsung Ex-1021, 0002
`
`Petitioner Samsung Ex-1021, 0002
`
`

`

`3
`
`US2007/176906 A1 assigned to Synaptics, Inc. discloses a touch sensor having a signal
`
`processor adapted to distinguish between three gestures based on different finger motions
`
`on the sensing device by providing a workflow with anidle state and three gesture-specific
`
`states referred to as first, second and third result states, asillustrated in Figure 5 of
`
`US2007/176906 A1.
`
`Generally, the raw output from the 2D touch sensor will be a time series of x, y coordinates,
`
`which are then processed by software, or firmware generated from higher level software, to
`
`distinguish the nature of the gesture that has been input. Generally, the raw data is split into
`
`contiguous touch segments and then processed to determine whatif any gestures can be
`
`deduced. The processing of the raw data to identify the gestures may be carried out in the
`
`same chip as generates the raw data, or the raw data may be exported to an external chip,
`
`for example by transmission over a communication bus to the device's central processing
`
`unit (CPU). The former approachis preferred by Synaptics, the latter by Apple as
`
`exemplified by US 2006/0066582 A1.
`
`Mostof the patentliterature is unspecific about how the raw time series data are converted
`
`into gestures. The straightforward approachis to write appropriate high level code, for
`
`example in C or another suitable programming language, in which the interpretation of the
`
`time series data is analysed using conditional statements, such as /f.. then .. else.
`
`However, it is difficult to reliably and efficiently add code to identify a new gesture into an
`
`existing block of codefor distinguishing between a significant number of gestures, for
`
`example at least 3 or 4, perhaps 10 to 20. Testing of the codeis a particular difficulty. This is
`
`because in general at any intermediate point in a time series of x,y,t data the inout may
`
`relate to a plurality of possible gestures, thereby making the coding for recognising one
`
`gesture generally dependenton or linked to the coding for recognising another gesture.
`
`SUMMARYOF THE INVENTION
`
`The invention solvesthis problem by adopting a state machine approach to designing and
`
`writing the gesture recognition algorithm. In particular, the invention relates to a touch sensor
`
`device comprising an at least one-dimensional sensor arranged to output a sensesignal
`
`responsive to proximity of an object, a position processing unit for calculating a position of an
`
`interaction with the sensitive area from an analysis of the sense signals and output a time
`
`series of data indicative of interaction positions on the sensor, and a gesture processing unit
`
`Petitioner Samsung Ex-1021, 0003
`
`Petitioner Samsung Ex-1021, 0003
`
`

`

`4
`
`operable to analyse the time series data to distinguish one or more gesture inputs therefrom,
`
`wherein the gesture processing unit is coded with gesture recognition code comprising a
`
`plurality of linked state modules. The invention also relates to a corresponding signal
`
`processing method.
`
`The gesture recognition code can be written in a high level language such as C and then
`
`compiled and embeddedin a microcontroller chip, or CPU chip as desired. Preferably, the
`
`gesture recognition codeis loaded into the same chip that interprets the touch signals from
`
`the screen and generates the time series data, e.g. a microcontroller, or other programmable
`
`logic device such asa field programmable gate array (FPGA). This approach has been used
`
`to create reliable testable code both for single-touch data input screens and also multi-touch
`
`data input screens. A single-touch screen is one which assumesonly one simultaneous
`
`touch of the screen, and is designed to output only one x,y coordinate at any one time. A
`
`multi-touch screen is one that can sense multiple simultaneous touches, for example up to 2
`
`or 3 simultaneous touches.
`
`The state machine includes an idle state module whichis the start state, and also the state
`
`whichis returned to after a gesture interpretation state module has been exited.
`
`Responsive to a touch, the idle state passes control to a touch state. In a multi-touch
`
`environment, there are multiple touch states, one for a single touch, one for a double touch,
`
`one for a triple touch etc with control passing to the appropriate touch state based on the
`
`number of simultaneous touches defined by the time series data at the time. A touchis
`
`usually only output as a valid touch, if certain criteria are satisfied, typically that there are a
`
`succession of touch at a stable x,y location or x,y region over multiple time sample
`
`increments. If a touch of a duration longer than a threshold duration is sensedin the touch
`
`state, then control flow passesto a press state module, wherein the pressstate is for
`
`handling longer touches. The pressstate is preferably a superstate comprising multiple sub-
`
`states to distinguish between different durations of press and/or to allow a very long press to
`
`be interpreted as being repeat presses, which maybe useful for alphanumeric key entry
`
`applications for example.
`
`The state machine preferably also has a plurality of state modules for interpreting higher
`
`level gestures, such as one or morestates for interpreting double taps, flicks, drags and any
`
`other gestures. The gestures include those specifically described in this document as well as
`
`other gestures knownin the art, specifically all those disclosed in the above-referencedprior
`
`art documents.
`
`Petitioner Samsung Ex-1021, 0004
`
`Petitioner Samsung Ex-1021, 0004
`
`

`

`BRIEF DESCRIPTION OF THE DRAWINGS
`
`For a better understanding of the invention, and to show how the same maybecarried into
`
`effect, reference is now made by way of example to the accompanying drawings.
`
`Figure 1 Gesture recognition state machine.
`
`Figure 2 Gesture recognition state machine pressed superstate.
`
`Figure 3 Gesture recognition state machine for single touch gestures.
`
`DETAILED DESCRIPTION
`
`1.
`
`Introduction
`
`This documentdefines a state machine for recognising user gestures on a touch surface.
`
`Section 2 describes the user gestures recognised by the state machine.
`
`Section 3 describes the events generated by the state machine in responseto user actions.
`
`Section 4 describes the state machine itself.
`
`2 Gestures
`
`This section lists the gestures recognised by the state machine described in Section 4.
`
`2.1
`
`Tap
`
`A tap happens when the user quickly touches and releases the touch surface. No significant
`
`movement takes place while the user's finger is on the touch surface.
`
`It is characterised by
`
`a short touch duration. This could be used, for example, to activate a hyperlink on a
`
`displayed web page.
`
`2.2 Double Tap
`
`A double tap happens when the user quickly touches and releases the touch surface twice in
`
`quick succession. No significant movement takes place while the user's finger is on the
`
`touch surface, or between successive touches.
`
`It is characterised by short touch durations,
`
`and a short gap between the first release and the second press. This could be used, for
`
`example, to select a word in a displayed document.
`
`Petitioner Samsung Ex-1021, 0005
`
`Petitioner Samsung Ex-1021, 0005
`
`

`

`2.3
`
`Press
`
`A press happens when the usertouches and holds the touch surface. No significant
`
`movementtakes place while the user's finger is on the touch surface. This could be used,
`
`for example, to select a number from a displayed numeric keypad. The same mechanism
`
`could be used to auto-repeat the selected number if the user continues to press on the
`
`displayed number.
`
`2.4
`
`Flick
`
`A flick happens when the user quickly touches the touch surface, movesa short distance,
`
`and releases touch.
`
`It is characterised by a short touch duration. This could be used, for
`
`example, to display the next in a sequenceof images.
`
`2.5 Drag
`
`A drag happens when the user touchesthe touch surface, movestheir finger across the
`
`surface, and releases touch.
`
`It is characterised by a large movement across the touch
`
`surface. Depending on the application, multiple drag events may be generated as the user
`
`movestheir finger. This could be used, for example, to select a sentencein a displayed
`
`document.
`
`2.6
`
`Pinch
`
`A pinch happens when the user places two fingers on the touch surface, and then moves
`
`them towards each other. This could be used, for example, to zoom out on a displayed
`
`image.
`
`2.7~Stretch
`
`A stretch happens when the user places two fingers on the touch surface, and then moves
`
`them away from each other. This could be used, for example, to zoom in on a displayed
`
`image.
`
`2.8 Rotate
`
`A rotate operation happens when the user places two fingers on the touch surface, and then
`
`rotates them relative to each other. This could be used, for example, to rotate a displayed
`
`image.
`
`Petitioner Samsung Ex-1021, 0006
`
`Petitioner Samsung Ex-1021, 0006
`
`

`

`3
`
`Events
`
`This section lists the events generated by the state machine described in Section 4. Note
`
`that the state machine can generate multiple events at a time in responseto user actions.
`
`3.1.
`
`Tap
`
`A tap event is generated when the user makesa tap gesture.
`
`3.2 Double Tap
`
`A double tap event is generated when the user makes a double tap gesture.
`
`3.3
`
`Press
`
`A press event is generated when the user touches the touch surface and leavestheir finger
`
`stationary.
`
`3.4
`
`Short Press
`
`A short press eventis generated after a press event, when the user has continued to leave
`
`their finger stationary for a period.
`
`3.5
`
`Long Press
`
`A long press event is generated after a short press event, when the user has continued to
`
`leave their finger stationary for a period.
`
`3.6 Repeat Press
`
`A repeat press event is generated after a long press event, when the user has continued to
`
`leave their finger stationary for a period.
`
`If they continue to leave their finger stationary,
`
`repeat press events will be generated at regularintervals.
`
`3.7 Release
`
`A release event is generated when the user removestheir finger(s) from the touch surface.
`
`3.8
`
`Flick
`
`A flick event is generated when the user makesa flick gesture.
`
`3.9 Drag
`
`A drag eventis generated when the user makes a drag gesture.
`
`Petitioner Samsung Ex-1021, 0007
`
`Petitioner Samsung Ex-1021, 0007
`
`

`

`3.10 Multitouch (or Dual touch)
`
`A multitouch (or dual touch) event is generated when the user presses the touch surface
`
`with twofingers.
`
`3.11 Pinch
`
`A pinch eventis generated when the user makes a pinch gesture.
`
`3.12 Stretch
`
`A stretch event is generated when the user makesa stretch gesture.
`
`3.13 Rotate
`
`A rotate event is generated when the user makesa rotate gesture.
`
`4
`
`State Machine
`
`The following sections describe eachof the statesin the figures.
`
`4.1
`
`Idle
`
`4.1.1 Description
`
`The user is not touching the touch surface.
`
`4.1.2 Entry
`
`There are no actions on entering this state.
`
`4.1.3 Action
`
`There are no actions while in this state.
`
`4.1.4 Transitions
`
`If the user presses the touch surface with one finger, the FSM (Finite State Machine) enters
`
`the Touchedstate.
`
`If the user presses the touch surface with two fingers, the FSM generates a multitouch event
`
`and enters the Multitouch state.
`
`4.1.5 Parameters
`
`This state has no parameters.
`
`4.1.6 Timings
`
`There are no timing calculations associated with this state.
`
`Petitioner Samsung Ex-1021, 0008
`
`Petitioner Samsung Ex-1021, 0008
`
`

`

`4.2. Touched
`
`4.2.1 Description
`
`User has touched the touch surface.
`
`4.2.2 Entry
`
`A timeoutis started. This is used to distinguish between short duration gestures such as
`
`flicks and taps, and long duration gestures such as a press.
`
`Theinitial touch location is stored. This is used to decideif the user has moved their finger.
`
`4.2.3 Action
`
`The timeout expiry is checked.
`
`The distance moved from the initial touch location is calculated.
`
`4.2.4 Transitions
`
`If the timeout expires, the FSM generates a press event and enters the Pressedstate.
`
`If the user moves by more than a specified distance from the initial touch location, the FSM
`
`enters the Flick Pending state.
`
`If the user places a second finger on the touch surface, the FSM generates press and
`
`multitouch events and enters the Multitouch state.
`
`lf the user releases their touch, the FSM enters the Second Tap Pending state. Subsequent
`
`actions will determine if a tap, double tap, or other gestureis later generated.
`
`4.2.5 Parameters
`
`The timeout parameter in this state determines howlong the user has to removetheir finger,
`
`potentially generating a tap event.
`
`The movement parameter in this state determines by how much the user must movetheir
`
`finger to enter the Flick Pendingstate.
`
`4.2.6 Timings
`
`The maximum time from initial touch to generate an eventin this state is touched timeout.
`
`4.3. Second Tap Pending
`
`4.3.1 Description
`
`The user has tapped the touch surface.
`
`Petitioner Samsung Ex-1021, 0009
`
`Petitioner Samsung Ex-1021, 0009
`
`

`

`10
`
`4.3.2 Entry
`
`A timeoutis started. This is used to decideif the user has made a tap gesture.
`
`Theinitial touch location is stored. This is used to decideif the user is making a double tap
`
`gesture.
`
`4.3.3 Action
`
`The timeout expiry is checked.
`
`4.3.4 Transitions
`
`If the timeout expires, a tap event is generated and the FSM entersthe Idle state. This
`
`meansthat the user had tapped the surface once, but not twice.
`
`If the user presses the touch surface with two fingers, the FSM generates tap and multitouch
`
`events and enters the Multitouch state. This indicates that there was a single tap followed
`
`by a multitouch.
`
`If the user presses the touch surface with one finger, the distance of the second touch from
`
`the first is calculated.
`
`If it is less than a specified value, the FSM enters the Double Tap
`
`Pending state, as the user may be doing a double tap.
`
`If it more than the specified value,
`
`the FSM generates a tap event(for the previous tap gesture) and enters the Touchedstate.
`
`4.3.5 Parameters
`
`The timeout parameter in this state determines how long after the user releases touch a tap
`
`event is generated.
`
`If the user touches the surface again before the timeout expires, a
`
`double tap event could potentially be generated.
`
`4.3.6 Timings
`
`The maximum time from initial touch to generate an eventin this state is touched timeout +
`
`second tap pending timeout.
`
`4.4 Double Tap Pending
`
`4.4.1 Description
`
`The user has tapped the touch surface and then pressedit again.
`
`4.4.2 Entry
`
`A timeoutis started. This is used to decideif the user is making a double tap gesture.
`
`Theinitial touch location is stored. This is used to decideif the user has moved their finger.
`
`Petitioner Samsung Ex-1021, 0010
`
`Petitioner Samsung Ex-1021, 0010
`
`

`

`11
`
`4.4.3 Action
`
`The timeout expiry is checked.
`
`The distance moved from the initial touch location is calculated.
`
`4.4.4 Transitions
`
`If the timeout expires, tap and press events are generated and the FSM enters the Pressed
`
`state.
`
`If the user moves by more than a specified distance from the initial touch location, the FSM
`
`generates a tap event and enters the Flick Pending state. This is because the user has
`
`tapped once, and their potential second tap is now a potentialflick.
`
`If the user releases their touch, the FSM generates a double tap event and enters the Idle
`
`state. This is because the user completed both tapsinside the specified time and distance
`
`limits.
`
`If the user presses the touch surface with two fingers, the FSM generates tap, press, and
`
`multitouch events and enters the Multitouch state. This indicates that there was a single tap,
`
`a press, and then a multitouch.
`
`4.4.5 Parameters
`
`The timeout parameter in this state determines how long the user has to removetheir finger
`
`to generate a double tap event.
`
`If the timeout expires, it is assumed that the user tapped
`
`and then touched and held the surface.
`
`The movement parameter in this state determines by how much the user must movetheir
`
`finger to enter the Flick Pendingstate.
`
`4.4.6 Timings
`
`The maximum time from initial touch to generate an eventin this state is touched timeout +
`
`second tap pending timeout + double tap pending timeout.
`
`4.5
`
`Flick Pending
`
`4.5.1 Description
`
`The user has pressed the touch surface and movedtheir finger over the surface.
`
`4.5.2 Entry
`
`A timeout is started. This is used to decideif the user is makingaflick or a drag gesture.
`
`Petitioner Samsung Ex-1021, 0011
`
`Petitioner Samsung Ex-1021, 0011
`
`

`

`Theinitial touch location is stored. This will be passed on to the Drag Pending processingif
`
`12
`
`required.
`
`4.5.3 Action
`
`The timeout expiry is checked.
`
`4.5.4 Transitions
`
`If the timeout expires, a press event is generated and the FSM enters the Drag Pending
`
`state. This is because the user has not released touch within the time used to distinguish
`
`flick and drag operations.
`
`If the user releases their touch, the FSM generatesa flick event and enters the Idle state.
`
`This is because the user completed their movementinside the specified time limit.
`
`If the user presses the touch surface with two fingers, the FSM generatesflick and
`
`multitouch events and enters the Multitouch state. The multitouch is considered to have
`
`endedthe flick gesture.
`
`4.5.5 Parameters
`
`
`
`to generateaflick event. If the timeout expires, it is assumed that the user is performing a
`
`The timeout parameter in this state determines how long the user has to removetheir finger
`
`drag operation.
`
`4.5.6 Timings
`
`The maximum time from initial touch to generate an eventin this state is touched timeout +
`
`flick pending timeout.
`
`4.6 Drag Pending
`
`4.6.1 Description
`
`The user has pressed the touch surface and movedtheir finger over the surface. The
`
`movementhaslasted longer thana flick duration.
`
`4.6.2 Entry
`
`A timeout is started. This is used to decideif the user has become stationary.
`
`Theinitial touch location is stored. This is used to decideif the user has moved their finger.
`
`4.6.3 Action
`
`The timeout expiry is checked.
`
`Petitioner Samsung Ex-1021, 0012
`
`Petitioner Samsung Ex-1021, 0012
`
`

`

`13
`
`The distance moved from the initial touch location is calculated.
`
`4.6.4 Transitions
`
`If the timeout expires, the FSM enters the Pressed state. This is because the user has been
`
`stationary for too long.
`
`If the user moves by more than a specified distance from the initial touch location, the FSM
`
`generates a drag event.
`
`If the user presses the touch surface with two fingers, the FSM generates a multitouch event
`
`and enters the Multitouch state.
`
`If the user releases their touch, the FSM generates a release event and enters the Idle state.
`
`4.6.5 Parameters
`
`The timeout parameter in this state determines for how long the user's finger must be
`
`stationary before moving to the Pressedstate.
`
`The movement parameter in this state determines by how much the user must movetheir
`
`finger to generate a drag event.
`
`4.6.6 Timings
`
`The maximum time from initial touch to generate an eventin this state is touched timeout +
`
`flick pending timeout + drag pending timeout.
`
`4.7 Short Pending
`
`4.7.1 Description
`
`User has pressed the touch surface and been stationary for some time.
`
`4.7.2 Entry
`
`A timeout is started. This is used to determine when to generate a short press event.
`
`Theinitial touch location is stored. This is used to decideif the user has moved their finger.
`
`4.7.3 Action
`
`The timeout expiry is checked.
`
`The distance moved from the initial touch location is calculated.
`
`4.7.4 Transitions
`
`If the timeout expires, a short press event is generated and the FSM enters the Long
`
`Pending, Repeat Pending, Repeat, or Press Complete states.
`
`Petitioner Samsung Ex-1021, 0013
`
`Petitioner Samsung Ex-1021, 0013
`
`

`

`14
`
`If the user moves by more than a specified distance from the initial touch location, a drag
`
`event is generated and the FSM enters the Drag Pendingstate.
`
`If the user releases their touch, the FSM generates a release event and enters the Idle state.
`
`If the user presses the touch surface with two fingers, the FSM generates a multitouch event
`
`and enters the Multitouch state.
`
`4.7.5 Parameters
`
`The timeout parameter in this state determines for how long the user's finger must be
`
`stationary before moving to the Long Pending, Repeat Pending, or Press Complete states.
`
`The movement parameter in this state determines by how much the user must movetheir
`
`finger to generate a drag event and enter the Drag Pendingstate.
`
`4.7.6 Timings
`
`The maximum time from initial touch to generate an eventin this state is touched timeout +
`
`flick pending timeout + drag pending timeout + short pending timeout.
`
`4.8
`
`Long Pending
`
`4.8.1 Description
`
`User has pressed the touch surface and been stationary for some time.
`
`4.8.2 Entry
`
`A timeout is started. This is used to determine when to generate a long press event.
`
`Theinitial touch location is stored. This is used to decideif the user has moved their finger.
`
`4.8.3 Action
`
`The timeout expiry is checked.
`
`The distance moved from the initial touch location is calculated.
`
`4.8.4 Transitions
`
`If the timeout expires, a long press event is generated and the FSM enters the Repeat
`
`Pending or Press Completestates.
`
`If the user moves by more than a specified distance from the initial touch location, a drag
`
`event is generated and the FSM enters the Drag Pendingstate.
`
`If the user releases their touch, the FSM generates a release event and enters the Idle state.
`
`Petitioner Samsung Ex-1021, 0014
`
`Petitioner Samsung Ex-1021, 0014
`
`

`

`If the user presses the touch surface with two fingers, the FSM generates a multitouch event
`
`15
`
`and enters the Multitouch state.
`
`4.8.5 Parameters
`
`The timeout parameter in this state determines for how long the user's finger must be
`
`stationary before moving to the Repeat Pending or Press Complete states.
`
`The movement parameter in this state determines by how much the user must movetheir
`
`finger to generate a drag event and enter the Drag Pendingstate.
`
`4.8.6 Timings
`
`The maximum time from initial touch to generate an eventin this state is touched timeout +
`
`flick pending timeout + drag pending timeout + short pending timeout + long pending
`
`timeout.
`
`4.9 Repeat Pending
`
`4.9.1 Description
`
`User has pressed the touch surface and been stationary for some time.
`
`4.9.2 Entry
`
`A timeoutis started. This is used to determine when to generate a repeat press event.
`
`Theinitial touch location is stored. This is used to decideif the user has moved their finger.
`
`4.9.3 Action
`
`The timeout expiry is checked.
`
`The distance moved from the initial touch location is calculated.
`
`4.9.4 Transitions
`
`If the timeout expires, a repeat press event is generated and the FSM enters the Repeat
`
`state.
`
`If the user moves by more than a specified distance from the initial touch location, a drag
`
`event is generated and the FSM enters the Drag Pendingstate.
`
`If the user releases their touch, the FSM generates a release event and enters the Idle state.
`
`If the user presses the touch surface with two fingers, the FSM generates a multitouch event
`
`and enters the Multitouch state.
`
`Petitioner Samsung Ex-1021, 0015
`
`Petitioner Samsung Ex-1021, 0015
`
`

`

`16
`
`4.9.5 Parameters
`
`The timeout parameter in this state determines for how long the user's finger must be
`
`stationary before moving to the Repeatstate.
`
`The movement parameter in this state determines by how much the user must movetheir
`
`finger to generate a drag event and enter the Drag Pending state.
`
`4.9.6 Timings
`
`The maximum time from initial touch to generate an eventin this state is touched timeout +
`
`flick pending timeout + drag pending timeout + short pending timeout + long pending timeout
`
`+ repeat pending timeout.
`
`4.10 Repeat
`
`4.10.1 Description
`
`User has pressed the touch surface and been stationary for some time.
`
`4.10.2 Entry
`
`A timeoutis started. This is used to determine when to generate a repeat press event.
`
`Theinitial touch location is stored. This is used to decideif the user has moved their finger.
`
`4.10.3 Action
`
`The timeout expiry is checked.
`
`The distance moved from the initial touch location is calculated.
`
`4.10.4 Transitions
`
`If the timeout expires, a repeat press eventis generated.
`
`If the user moves by more than a specified distance from the initial touch location, a drag
`
`event is generated and the FSM enters the Drag Pendingstate.
`
`If the user releases their touch, the FSM generates a release event and enters the Idle state.
`
`If the user presses the touch surface with two fingers, the FSM generates a multitouch event
`
`and enters the Multitouch state.
`
`4.10.5 Parameters
`
`The timeout parameter in this state determines for how long the user's finger must be
`
`stationary before generating a repeat press event.
`
`Petitioner Samsung Ex-1021, 0016
`
`Petitioner Samsung Ex-1021, 0016
`
`

`

`17
`
`The movement parameter in this state determines by how much the user must movetheir
`
`finger to generate a drag event and enter the Drag Pendingstate.
`
`4.10.6 Timings
`
`The maximum time from initial touch to generate an eventin this state is touched timeout +
`
`flick pending timeout + drag pending timeout + short pending timeout + long pending timeout
`
`+ repeat pending timeout + repeat timeout.
`
`4.11 Press Complete
`
`4.11.1 Description
`
`User has pressed the touch surface and been stationary for some time.
`
`4.11.2 Entry
`
`Theinitial touch location is stored. This is used to decideif the user has moved their finger.
`
`4.11.3 Action
`
`The distance moved from the initial touch location is calculated.
`
`4.11.4 Transitions
`
`If the user moves by more than a specified distance from the initial touch location, a drag
`
`event is generated and the FSM enters the Drag Pendingstate.
`
`If the user releases their touch, the FSM generates a release event and enters the Idle state.
`
`If the user presses the touch surface with two fingers, the FSM generates a multitouch event
`
`and enters the Multitouch state.
`
`4.11.5 Parameters
`
`The movement parameter in this state determines by how much the user must movetheir
`
`finger to generate a drag event and enter the Drag Pendingstate.
`
`4.11.6 Timings
`
`Thereis no limit on how long it could take before an eventis generatedin this state.
`
`4.12 Multitouch (or Dual touch)
`
`4.12.1 Description
`
`The user has put two fingers on the touch surface.
`
`4.12.2 Entry
`
`The initial touch locations are stored.
`
`Petitioner Samsung Ex-1021, 0017
`
`Petitioner Samsung Ex-1021, 0017
`
`

`

`18
`
`The angle between the initial touch locations is stored.
`
`4.12.3 Action
`
`The separation between the two current touch locations is calculated.
`
`The angle between the two current touch locationsis calculated.
`
`4.12.4 Transitions
`
`If the user releases both fingers, the FSM generates a release event and enters the Idle
`
`state.
`
`If the user releases one finger, the FSM enters the Pressed state.
`
`If the touch separation increases by more than a specified amountfrom theinitial touch
`
`separation, a stretch eventis generated.
`
`If the touch separation decreases by more than a specified amountfrom the initial touch
`
`separation, a pinch eventis generated.
`
`If the angle between the touch locations changes by more than a specified amount from the
`
`initial angle, a rotate event is generated.
`
`Note: The FSM can generate multiple events in this state, namely, stretch and rotate, or
`
`pinch and rotate.
`
`4.12.5 Parameters
`
`The angle delta parameter in this state determines by how muchthe angle between the
`
`touch locations must change to generate a rotate event.
`
`The stretch threshold parameterin this state determines by how much the separation
`
`between the two touchlocations must increase to generate a stretch event.
`
`The pinch threshold parameter in this state determines by how muchthe separation between
`
`the two touch locations must decreaseto generate a pinch event.
`
`4.12.6 Timings
`
`Thereis no limit on how long it could take before an eventis generatedin this state.
`
`Petitioner Samsung Ex-1021, 0018
`
`Petitioner Samsung Ex-1021, 0018
`
`

`

`1/3
`
`
`
`multitouch/ multitouch
`