`US 20080036743Al
`
`c19) United States
`c12) Patent Application Publication
`Westerman et al.
`
`c10) Pub. No.: US 2008/0036743 Al
`Feb. 14, 2008
`(43) Pub. Date:
`
`(54) GESTURING WITH A MULTIPOINT
`SENSING DEVICE
`
`(75)
`
`Inventors: Wayne C. Westerman, San Francisco,
`CA (US); Steven P. Hotelling, San
`Jose, CA (US); Myra Haggerty, San
`Mateo, CA (US); Bas Ording, San
`Francisco, CA (US); Nima Parivar,
`South San Francisco, CA (US); Duncan
`Robert Kerr, San Francisco, CA (US)
`
`Correspondence Address:
`APPLE C/O MORRISON AND FOERSTER
`,LLP
`LOS ANGELES
`555 WEST FIFTH STREET SUITE 3500
`LOS ANGELES, CA 90013-1024 (US)
`
`(73) Assignee: Apple Computer, Inc.
`
`(21) Appl. No.:
`
`11/700,636
`
`(22) Filed:
`
`Jan.31, 2007
`
`Related U.S. Application Data
`
`(63) Continuation-in-part of application No. 10/903,964,
`filed on Jul. 30, 2004.
`Continuation-in-part of application No. 11/015,434,
`filed on Dec. 17, 2004, which is a continuation of
`application No. 09/236,513, filed on Jan. 25, 1999,
`now Pat. No. 6,323,846.
`
`(60) Provisional application No. 60/072,509, filed on Jan.
`26, 1998.
`
`Publication Classification
`
`(51)
`
`Int. Cl.
`G06F 3/041
`(2006.01)
`(52) U.S. Cl. .............................................................. 345/173
`
`ABSTRACT
`(57)
`Methods and systems for implementing gestures with sens(cid:173)
`ing devices are disclosed. More particularly, methods and
`systems related to gesturing with multipoint sensing devices
`are disclosed.
`
`START
`
`12
`
`PROVIDE MUL Tl POINT SENSING
`DEVICE
`
`14
`
`STANDBY
`
`DETERMINE CHORD ASSOCIATED
`WITH TOUCH
`
`YES
`
`DETERMINE GESTURE SET
`ASSOCIATING ONE OR MORE
`ACTIONS TO ONE OR MORE
`GESTURE EVENTS
`
`MONITOR TOUCH FOR GESTURE
`EVENT
`
`22
`
`PERFORM ACTION ASSOCIATED
`WITH GESTURE EVENT
`
`16
`
`18
`
`20
`
`24
`
`Petitioner Samsung Ex-1008, 0001
`
`
`
`Patent Application Publication Feb. 14, 2008 Sheet 1 of 49
`
`US 2008/0036743 Al
`
`START
`
`12
`
`PROVIDE MULTIPOINT SENSING
`DEVICE
`
`14
`
`DETERMINE CHORD ASSOCIATED
`WITH TOUCH
`
`YES
`
`DETERMINE GESTURE SET
`ASSOCIATING ONE OR MORE
`ACTIONS TO ONE OR MORE
`GESTURE EVENTS
`
`MONITOR TOUCH FOR GESTURE
`EVENT
`
`NO
`
`26
`
`22
`
`NO
`
`YES
`
`PERFORM ACTION ASSOCIATED
`WITH GESTURE EVENT
`
`FIG. 1
`
`18
`
`20
`
`24
`
`Petitioner Samsung Ex-1008, 0002
`
`
`
`Patent Application Publication Feb. 14, 2008 Sheet 2 of 49
`
`50,
`
`US 2008/0036743 Al
`
`START
`
`52
`
`DETERMINE GESTURE SET FOR TOUCH
`
`MONITOR TOUCH FOR GESTURE EVENT
`
`INITIATE COMMAND WHEN GESTURE EVENT
`IS PERFORMED
`
`FIG. 2
`
`so,
`
`RECOGNIZE ARRANGEMENT OF CONTACTS
`
`•
`+
`
`COMPARE ARRANGEMENT OF CONTACTS
`TO A STORED SET OF ARRANGEMENT OF
`CONTACTS
`
`LOAD GESTURE SET ASSIGNED TO INITIAL
`ARRANGEMENT OF CONTACTS
`
`54
`
`56
`
`58
`
`62
`
`j
`
`6 4
`_)
`
`6 6
`
`l)
`
`FIG. 3
`
`Petitioner Samsung Ex-1008, 0003
`
`
`
`Patent Application Publication Feb. 14, 2008 Sheet 3 of 49
`
`US 2008/0036743 Al
`
`70~
`
`DETECT TOUCH
`
`1 r
`
`DETERMINE LOCATION OF TOUCH
`
`72
`
`7 4
`
`DETERMINE ARRANGEMENT OF CONTACTS J
`
`,
`
`I
`
`7 6
`
`, r
`
`DETERMINE ACTIVE APPLICATION
`
`7 8
`
`J
`
`, r
`
`
`
`8 0
`
`DETERMINE STATE OF ACTIVE APPLICATION J
`
`, '
`SET GESTURE SET BASED ON THE ABOVE
`
`82
`l)
`
`FIG. 4
`
`Petitioner Samsung Ex-1008, 0004
`
`
`
`Patent Application Publication Feb. 14, 2008 Sheet 4 of 49
`
`US 2008/0036743 Al
`
`1 0 0~
`
`START
`
`102
`
`106---'-(cid:173)
`I
`I
`I
`I
`I
`I
`I
`L------- ----------------------------
`
`DETERMINE# OF
`CONTACTS
`
`----108
`
`I
`I
`
`110
`
`COMPARE RECOGNIZED ARRANGEMENT OF
`CONTACTS TO STORED ARRANGEMENT OF .__ ______ __,
`CONTACTS IN GESTURE MAP
`
`112
`
`116
`
`118
`
`LOAD GESTURE SET ASSOCIATED WITH
`INITIAL ARRANGEMENT OF CONTACTS
`
`MONITOR TOUCH FOR GESTURE EVENTS
`ASSOCIATED WITH THE GESTURE SET
`
`INITIATE COMMAND ASSOCIATED WITH
`GESTURE EVENT WHEN GESTURE EVENT IS
`PERFORMED.
`
`FIG. 5
`
`Petitioner Samsung Ex-1008, 0005
`
`
`
`Patent Application Publication Feb. 14, 2008 Sheet 5 of 49
`
`US 2008/0036743 Al
`
`1 2 0~
`
`DETECT TOUCH L/
`
`122
`
`, r
`
`1 24
`DETERMINE CHORD ASSOCIATED WITH __/
`TOUCH
`
`V
`
`1 26
`DETERMINE INPUT FUNCTIONALITY _/
`ASSOCIATED WITH CHORD
`
`LOAD GESTURE SET ASSOCIATED WITH _/
`
`, r
`
`1 28
`
`INPUT FUNCTIONALITY
`
`, r
`
`1 30
`
`INITIATE ACTIONS ASSOCIATED WITH
`GESTURE EVENTS WHEN GESTURE EVENTS
`ARE PERFORMED
`
`FIG. 6
`
`Petitioner Samsung Ex-1008, 0006
`
`
`
`Patent Application Publication Feb. 14, 2008 Sheet 6 of 49
`
`US 2008/0036743 Al
`
`144
`
`146
`
`148
`
`150
`
`152
`
`DETECT TOUCH
`
`RECOGNIZE ARRANGEMENT OF CONTACTS
`
`DETERMINE CHORD ASSOCIATED WITH
`ARRANGEMENT OF CONTACTS
`
`LOAD GESTURE SET ASSOCIATED WITH
`CHORD
`
`MONITOR TOUCH FOR GESTURE EVENTS
`PERFORMED BY ANY ARRANGEMENT OF
`CONTACTS DURING THE TOUCH
`
`151
`
`154
`
`INITIATE CONTROi/COMMAND ASSOCIATED
`WITH GESTURE EVENT
`
`FIG. 7
`
`Petitioner Samsung Ex-1008, 0007
`
`
`
`Patent Application Publication Feb. 14, 2008 Sheet 7 of 49
`
`US 2008/0036743 Al
`
`(160
`
`START
`
`164
`
`168
`
`172
`
`PERFORM POINTING
`IN ACCORDANCE
`WITH MOTION
`
`PERFORM
`DRAGGING IN
`ACCORDANCE WITH
`MOTION
`
`PERFORM
`SECONDARY
`DRAGGING IN
`ACCORDANCE WITH
`MOTION
`
`FIG. 8
`
`Petitioner Samsung Ex-1008, 0008
`
`
`
`Patent Application Publication Feb. 14, 2008 Sheet 8 of 49
`
`US 2008/0036743 Al
`
`180~
`
`182
`
`184
`
`DETERMINE BASE CHORD
`
`186
`
`DETECT MOTION
`
`DETECT NEW
`FIRST DIGIT
`
`DETECT NEW
`SECOND DIGIT
`
`190
`
`PERFORM
`POINTING
`
`PERFORM FIRST
`COMMAND EACH
`TIME DIGIT IS
`DETECTED
`
`PERFORM
`SECOND
`COMMAND EACH
`TIME DIGIT IS
`DETECTED
`
`.188
`
`194
`
`192
`
`FIG. 9
`
`Petitioner Samsung Ex-1008, 0009
`
`
`
`Patent Application Publication Feb. 14, 2008 Sheet 9 of 49
`
`US 2008/0036743 Al
`
`, - - - - - - - - - - - - - - 1~ DETECTTOUCH
`
`202
`
`204
`
`~ - - - - -
`
`RECOGNIZE ARRANGEMENT
`OF CONTACTS
`
`RECOGNIZE CHORD ASSOCIATED WITH
`ARRANGEMENT OF CONTACTS
`
`LOAD GESTURE SET ASSOCIATED WITH
`CHORD
`
`206
`
`208
`
`210
`
`MONITOR TOUCH FOR A GESTURE EVENT
`PERFORMED BY ANY ARRANGEMENT OF i----------,
`CONTACTS
`
`INITIATE
`CONTROL/
`COMMAND
`ASSOCIATED
`WITH GESTURE
`EVENT
`
`FIG. 10
`
`Petitioner Samsung Ex-1008, 0010
`
`
`
`Patent Application Publication Feb. 14, 2008 Sheet 10 of 49
`
`US 2008/0036743 Al
`
`(220
`
`START
`
`222
`
`226
`
`FIG. 11
`
`224
`
`228
`
`232
`
`236
`
`PERFORM POINTING
`IN ACCORDANCE
`WITH MOTION
`
`PERFORM
`DRAGGING IN
`ACCORDANCE WITH
`MOTION
`
`PERFORM
`SECONDARY
`DRAGGING IN
`ACCORDANCE WITH
`MOTION
`
`PERFORM
`SCROLLING IN
`ACCORDANCE WITH
`MOTION
`
`Petitioner Samsung Ex-1008, 0011
`
`
`
`Patent Application Publication Feb. 14, 2008 Sheet 11 of 49
`
`US 2008/0036743 Al
`
`240 \_
`
`START
`
`INITIATE POINTING
`- - - - - - - - UNTIL LIFT OFF OF
`ALL FINGERS
`
`254
`
`DROP
`
`FIG. 12
`
`244
`
`246
`
`250
`
`CLICK
`
`DRAG
`
`248
`
`252
`
`253
`
`Petitioner Samsung Ex-1008, 0012
`
`
`
`Patent Application Publication Feb. 14, 2008 Sheet 12 of 49
`
`US 2008/0036743 Al
`
`260\
`
`262
`
`DETECT TOUCH L)
`
`1,
`
`RECOGNIZE SIDE OF PINKY
`
`264
`
`L)
`
`11'
`
`DETECT SWIPE WITH SIDE OF PINKY
`
`'I
`
`PERFORM CONTROL/COMMAND
`ASSOCIATED WITH SWIPING PINKY
`
`2 66
`
`L)
`
`2 68
`
`FIG. 13
`
`Petitioner Samsung Ex-1008, 0013
`
`
`
`Patent Application Publication Feb. 14, 2008 Sheet 13 of 49
`
`US 2008/0036743 Al
`
`START
`
`PROVIDE GESTURE SET MAPPING UPWARD SLIDE TO
`EXPOSE AND DOWNWARD SLIDE TO DASHBOARD
`
`282
`
`RECOGNIZE BASE CHORD
`
`284
`
`288
`
`OPEN
`EXPOSE
`
`286
`
`296
`
`OPEN
`DASHBOARD
`
`298
`
`300
`
`294
`
`290
`
`301
`
`292
`
`CLOSE
`EXPOSE
`
`CLOSE
`DASHBOARD
`
`PERFORM POINTING
`WITH ALTERED
`CHORD
`
`302
`
`FIG. 14
`
`Petitioner Samsung Ex-1008, 0014
`
`
`
`Patent Application Publication Feb. 14, 2008 Sheet 14 of 49
`
`US 2008/0036743 Al
`
`3 1 0~
`
`DETECT TOUCH
`
`312
`L)
`
`,,
`14
`3
`DETERMINE ARRANGEMENT OF CONTACTS L)
`
`,r
`
`PERFORM ACTION BASED ON
`ARRANGEMENT OF CONTACTS
`
`31
`6
`L)
`
`FIG. 15
`
`320~
`
`MONITOR TOUCH MOTION
`
`322
`L)
`
`324
`, '
`DIFFERENTIATE MOTION BETWEEN A FLICK L)
`
`AND SWIPE
`
`, r
`
`326
`
`PERFORM FIRST l)
`
`ACTION IF MOTION
`IS A FLICK
`
`,,
`PERFORM FIRST l)
`
`3 28
`
`ACTION IF MOTION IS A
`SWIPE
`
`FIG. 16
`
`Petitioner Samsung Ex-1008, 0015
`
`
`
`Patent Application Publication Feb. 14, 2008 Sheet 15 of 49
`
`US 2008/0036743 Al
`
`330\
`
`DETECT TOUCH
`
`IDENTIFY TWO CONTACTS
`
`332
`
`3 34
`
`J
`J
`
`,,
`MONITORING THE TOUCH FOR A ROTATION
`GESTURE EVENT
`
`J
`
`3 36
`
`,,
`INITIATE ROTATE COMMAND WHEN A FIRST _/
`
`CONTACT IS STATIONARY AND A SECOND
`CONTACT IS ROTATED ABOUT THE FIRST
`CONTACT
`
`3 38
`
`FIG. 17
`
`Petitioner Samsung Ex-1008, 0016
`
`
`
`Patent Application Publication Feb. 14, 2008 Sheet 16 of 49
`
`US 2008/0036743 Al
`
`340~
`
`I
`
`PROVIDE MULTIPLE WINDOWS
`
`'
`DETECT POINTING MECHANISM OVER FIRST WINDOW
`
`342
`
`344
`
`346
`
`MONITOR TOUCH TO DETERMINE IF CHORD ASSOCIATED WITH~
`POINTING MECHANISM HAS BEEN PRESENTED
`
`INITIATE GESTURE SET ASSOCIATED WITH THE CHORD AND ;t
`35 0 u
`35 2 J
`
`FIRST WINDOW
`
`'
`PERFORM ACTIONS IN FIRST WINDOW WHEN GESTURE EVENTS
`ASSOCIATED WITH THE GESTURE SET ARE PERFORMED WHILE
`POINTING MECHANISM IS DETECTED OVER FIRST WINDOW
`
`'
`DETECT POINTING MECHANISM OVER SECOND WINDOW
`
`11'
`
`MONITOR TOUCH TO DETERMINE IF CHORD ASSOCIATED WITH
`POINTING MECHANISM HAS BEEN PRESENTED
`
`INITIATE SECOND GESTURE SET ASSOCIATED WITH THE CHORD
`AND SECOND WINDOW
`
`35 4
`
`l_/
`
`35 6
`
`~
`
`358
`
`PERFORM ACTIONS IN SECOND WINDOW WHEN GESTURE EVENTS
`ASSOCIATED WITH THE SECOND GESTURE SET ARE PERFORMED
`WHILE POINTING MECHANISM IS DETECTED OVER SECOND WINDOW
`
`FIG. 18
`
`Petitioner Samsung Ex-1008, 0017
`
`
`
`Patent Application Publication Feb. 14, 2008 Sheet 17 of 49
`
`US 2008/0036743 Al
`
`360\
`
`DETECT TOUCH
`
`362
`
`RECOGNIZE ARRANGEMENT OF CONTACTS ASSOCIATED WITH
`FIRST HAND
`
`LOAD GESTURE SET BASED ON ARRANGEMENT OF CONTACTS
`ASSOCIATED WITH FIRST HAND
`
`RECOGNIZE SECOND ARRANGEMENT OF CONTACTS
`ASSOCIATED WITH SECOND HAND
`
`CHANGE OR MODIFY GESTURE SET FOR FIRST ARRANGEMENT OF
`CONTACTS BASED ON SECOND ARRANGEMENT OF CONTACTS
`
`FIG. 19
`
`364
`
`366
`
`368
`
`370
`
`Petitioner Samsung Ex-1008, 0018
`
`
`
`Patent Application Publication Feb. 14, 2008 Sheet 18 of 49
`
`US 2008/0036743 Al
`
`380~
`
`PROVIDE MOUSE AND TOUCH SURFACE ARE PROVIDED
`
`MONITOR MOUSE FOR MOUSE EVENTS
`
`SIMULTANEOUSLY MONITOR TOUCH SURFACE FOR GESTURE
`EVENTS
`
`PERFORM MOUSE OPERATIONS IN ACCORDANCE WITH
`MOUSE EVENTS
`
`SIMULTANEOUSLY PERFORM GESTURE OPERATIONS IN
`ACCORDANCE WITH GESTURE EVENTS
`
`4 0 0~
`
`FIG. 20
`
`PROVIDE MOUSE AND TOUCH SURFACE ARE PROVIDED
`
`MONITOR MOUSE FOR MOUSE EVENTS
`
`SIMULTANEOUSLY MONITOR TOUCH SURFACE FOR
`ARRANGEMENT OF CONTACTS AND/OR GESTURE EVENTS
`
`CHANGE MOUSE MODE BASED ON ARRANGEMENT OF
`CONTACTS AND/OR GESTURE EVENTS
`
`FIG. 21
`
`382
`
`388
`
`390
`
`402
`
`404
`
`406
`
`408
`
`Petitioner Samsung Ex-1008, 0019
`
`
`
`Patent Application Publication Feb. 14, 2008 Sheet 19 of 49
`
`US 2008/0036743 Al
`
`4 1 0~
`
`DETECT FIRST TOUCH ON MUL Tl POINT SENSING SURFACE
`
`RECOGNIZE FIRST ARRANGEMENT OF CONTACTS
`(ASSOCIATED WITH FIRST HAND)
`
`SIMULTANEOUSLY DETECT FIRST TOUCH ON MULTIPOINT
`SENSING SURFACE
`
`SIMULTANEOUSLY RECOGNIZE SECOND ARRANGEMENT OF
`CONTACTS (ASSOCIATED WITH SECOND HAND)
`
`LOAD GESTURE SET ASSOCIATED WITH FIRST ARRANGEMENT
`OF CONTACTS
`
`SIMULTANEOUSLY LOAD GESTURE SET ASSOCIATED WITH
`SECOND ARRANGEMENT OF CONTACTS
`
`412
`
`414
`
`416
`
`418
`
`420
`
`422
`
`424
`
`MONITOR FIRST TOUCH FOR GESTURE EVENTS
`
`426
`
`SIMULTANEOUSLY MONITOR SECOND TOUCH FOR GESTURE
`EVENTS
`
`i - - - -~
`
`INITIATE ACTIONS ASSOCIATED WITH FIRST GESTURE EVENTS
`~ - - - WHEN WHEN FIRST GESTURE EVENTS ARE PERFORMED
`
`428
`
`430
`
`SIMULTANEOUSLY INITIATE ACTIONS ASSOCIATED WITH
`SECOND GESTURE EVENTS WHEN SECOND GESTURE EVENTS---~
`ARE PERFORMED
`.
`
`FIG. 22
`
`Petitioner Samsung Ex-1008, 0020
`
`
`
`Patent Application Publication Feb. 14, 2008 Sheet 20 of 49
`
`US 2008/0036743 Al
`
`440~
`
`DISPLAY GUI OBJECT
`
`DETECT MULTIPLE POINTERS AT DIFFERENT POINTS OF
`DISPLAYED OBJECT
`
`LOCK POINTERS TO DISPLAYED OBJECT
`
`MONITOR POSITION OF POINTERS RELATIVE TO THEIR
`LOCKED POSITION
`
`MODIFY DISPLAYED GUI OBJECT WHEN ONE OR BOTH OF THE
`POINTERS IS MOVED RELATIVE TO THEIR LOCKED POSITION
`
`4 5 0~
`
`FIG. 23
`
`DETECT TOUCH
`
`RECOGNIZE FIRST ARRANGEMENT OF CONTACTS
`
`DETECT SECOND TOUCH
`
`442
`
`445
`
`446
`
`448
`
`452
`
`SIMULTANEOUSLY RECOGNIZE SECOND ARRANGEMENT OF
`CONTACTS
`
`MONITOR FIRST TOUCH FOR GESTURE EVENTS
`
`SIMUL T. MONITOR SECOND TOUCH FOR GESTURE EVENTS
`
`~ - - - - - - - - - _ _ _ . . _ - - - - - - - - -~ 464
`PERFORM FIRST POINTING OPERATIONS WITH FIRST
`ARRANGEMENT OF CONTACTS
`
`466
`
`SIMULTANEOUSLY PERFORM SECOND POINTING OPERATIONS
`WITH SECOND ARRANGEMENT OF CONTACTS
`
`FIG. 24
`
`Petitioner Samsung Ex-1008, 0021
`
`
`
`Patent Application Publication Feb. 14, 2008 Sheet 21 of 49
`
`US 2008/0036743 Al
`
`4 7 0~
`
`DISPLAY GROUP OF SELECTABLE ITEMS
`
`DETECT FIRST POINTING MECHANISM OVER GROUP OF
`SELECT ABLE ITEMS
`
`ACTIVATE ITEM UNDERNEATH FIRST POINTING MECHANISM
`
`SIMULTANEOUSLY DETECT SECOND POINTING MECHANISM
`OVER WINDOW
`
`APPLY SELECTED ITEM TO SECOND POINTING MECHANISM
`
`5 0 0~
`
`FIG. 25
`
`DISPLAY GROUP OF MEDIA ITEMS
`
`DETECT POINTING MECHANISM OVER AN ITEM THEREBY
`HIGHLIGHTING THE THE ITEM
`
`DETECT TOUCH ON TOUCH SURFACE
`
`RECOGNIZE ARRANGEMENT OF CONTACTS ASSOCIATED WITH
`TOUCH
`
`INSERT METADATA TIED TO THE RECOGNIZED ARRANGEMENT
`OF CONTACTS INTO THE HIGHLIGHTED ITEM
`
`FIG. 26
`
`472
`
`474
`
`476
`
`478
`
`480
`
`502
`
`506
`
`508
`
`510
`
`Petitioner Samsung Ex-1008, 0022
`
`
`
`Patent Application Publication Feb. 14, 2008 Sheet 22 of 49
`
`US 2008/0036743 Al
`
`522
`
`524
`
`526
`
`DISPLAY CONTROL OBJECT
`
`DETECT TOUCH OVER CONTROL OBJECT
`
`RECOGNIZE ARRANGEMENT OF CONTACTS ASSOCIATED WITH
`TOUCH
`
`527
`
`528
`
`530
`
`NO
`
`YES
`
`IGNORE TOUCH
`
`MONITOR TOUCH FOR
`GESTURE EVENT
`ASSOCIATED WITH
`CONTROLLING THE
`CONTROL OBJECT
`
`FIG. 27
`
`532
`
`APPLY ACTION
`ASSOCIATED WITH ·
`GESTURE EVENT TO
`CONTROL OBJECT WHEN
`GESTURE EVENT IS
`PERFORMED
`
`Petitioner Samsung Ex-1008, 0023
`
`
`
`Patent Application Publication Feb. 14, 2008 Sheet 23 of 49
`
`US 2008/0036743 Al
`
`540\
`
`PROVIDE LIST OF INPUT FUNCTIONS
`
`ASSIGN INPUT FUNCTIONS TO CHORDS
`
`THEREAFTER LINK INPUT FUNCTION TO CHORD WHEN CHORD
`IS RECOGNIZED
`
`FIG. 28
`
`580\
`
`PROVIDE LIST OF INPUT FUNCTIONS
`
`PROVIDE COMMANDS ASSOCIATED WITH INPUT FUNCTIONS
`
`PROVIDE LIST OF CHORDS
`
`PROVIDE GESTURE EVENTS ASSOCIATED WITH CHORDS
`
`542
`
`544
`
`546
`
`582
`
`584
`
`586
`
`590
`
`ASSIGN INPUT FUNCTION TO CHORD
`
`ASSIGN COMMANDS TO GESTURE EVENTS
`
`FIG. 30
`
`Petitioner Samsung Ex-1008, 0024
`
`
`
`Standard Trackpad
`
`5
`
`<Custom Set>
`
`List of Chords
`
`l!I
`
`552
`
`558
`
`1 FINGER
`INPUT FUNCTION
`
`2 FINGERS
`INPUT FUNCTION
`
`I~~
`- -f~
`- -rr 2 N~;ctttt~~;tGERS
`~ 3 FINGERS
`- -rr
`
`INPUT FUNCTION
`
`4 FINGERS
`INPUT FUNCTION
`
`Gesture Control Panel
`
`Basic Multitouch
`
`Advanced Multitouch
`
`550
`
`(')
`
`~ .....
`
`""O
`~ .....
`('D = .....
`> "e
`"e -....
`.... 0 = ""O = O" -....
`.... 0 =
`?' ....
`~ ...
`
`(')
`
`~ .....
`
`"f'j
`('D
`
`N
`0
`0
`QO
`
`('D
`
`rJJ
`
`=-('D
`.....
`N ...
`0 .... ... 1,0
`
`554
`
`.... ...
`
`List of Input Functions
`Input Function 1
`
`Gesture Set
`
`gesture
`event
`
`gesture
`event
`
`gesture
`event
`
`command
`
`command
`
`command
`
`56
`
`gesture
`event
`
`gesture
`event
`
`gesture
`event
`
`command
`
`command
`
`command
`
`~
`
`gesture
`event
`
`gesture
`event
`
`gesture
`event
`
`command
`
`command
`
`command
`
`FIG. 29A
`
`c
`rJJ
`N
`0
`0
`
`0
`~
`O'I
`
`QO ---0
`-....J ... ~
`> ....
`
`Petitioner Samsung Ex-1008, 0025
`
`
`
`Gesture Contr
`
`Standard Trackpad
`
`Basic Multitouc
`
`< Custom Set>
`
`1~ I
`
`__Si
`552 = List of Chords
`558 ~
`~
`~
`
`1 FINGER
`PoinUclick
`
`2 FINGERS
`Scroll and Secondary click
`
`2 NON ADJACENT FINGERS
`none
`
`~ 3FINGERS
`~ /
`Primary Click and Drag
`
`~·. · .. E~~,~~~~]1~b6ard
`
`.... ...
`
`None
`Point
`PoinUClick
`Scroll
`Scroll & Secondary Click
`Primary Click & Drag
`Secondary Click and Drag
`Move Window
`Expose & Dashboard
`Web Browser Operations
`Editing Operations
`File Operations
`
`X
`
`554
`
`550 )
`
`i
`
`expose all
`
`56
`
`...
`expose
`applications
`
`...
`expose
`desktop
`
`dashboard
`
`!
`
`FIG. 298
`
`""O
`~ .....
`('D = .....
`
`(')
`
`~ .....
`
`(')
`
`~ .....
`
`t "e -....
`.... 0 =
`""O = O" -....
`.... 0 =
`?' ....
`~ ...
`
`"f'j
`('D
`
`N
`0
`0
`QO
`
`('D
`('D
`
`rJJ =(cid:173)
`.....
`N
`Ul
`
`0 .... ... 1,0
`
`c
`rJJ
`N
`0
`0
`
`0
`~
`O'I
`
`QO ---0
`-....J ... ~
`> ....
`
`Petitioner Samsung Ex-1008, 0026
`
`
`
`None
`Point
`Point/Click
`Scroll
`Scroll & Secondary Click
`gx,,,-;.'.:::'-~erimaryiClicf<t~&IDrag~~~;;\,:i~;~
`Secondary Click and Drag
`Move Window
`Expose & Dashboard
`Web Browser Operations
`Editing Operations
`File Operations
`
`554
`
`550 )
`
`DRAG
`
`DRAG
`
`~ i
`0
`
`56
`
`/
`
`DRAG
`
`--
`DRAG
`
`DRAG
`
`Gesture Contr
`
`Standard Trackpad
`
`Basic Multitouc
`
`_ )
`
`< Custom Set>
`
`1~ I
`
`552
`
`558
`
`List of Chords
`
`1 FINGER
`Point/click
`
`I~
`- -
`~ 2FINGERS
`Scroll and Secondary click
`
`- -J!'1 2 NON ADJACENT FINGERS
`
`none
`
`(cid:141)
`DRAG
`
`Expose and Dashboard
`
`....
`
`'T
`
`DRAG
`
`/
`
`FIG. 29C
`
`CLICK
`
`DRAG
`
`i ~
`
`""O
`~ .....
`('D = .....
`
`(')
`
`~ .....
`
`t "e -....
`.... 0 =
`""O = O" -....
`.... 0 =
`?' ....
`~ ...
`
`(')
`
`~ .....
`
`"f'j
`('D
`
`N
`0
`0
`QO
`
`('D
`('D
`
`rJJ =(cid:173)
`.....
`N
`O'I
`
`0 .... ... 1,0
`
`c
`rJJ
`N
`0
`0
`
`0
`~
`O'I
`
`QO ---0
`-....J ... ~
`> ....
`
`Petitioner Samsung Ex-1008, 0027
`
`
`
`Gesture Contr
`
`Standard Trackpad
`
`Basic Multitouc
`
`5523
`
`< Custom Set >
`
`List of Chords
`
`@]
`
`~ 2 NON ADJACENT FINGERS
`~ /
`none
`
`558
`
`~ 3FINGERS
`~ /
`Primary Click and Drag
`
`~ 4FINGERS
`~ /
`Expose and Dashboard
`
`None
`Point
`Point/Click
`Scroll
`Scroll & Secondary Click
`Primary Click & Drag
`Secondary Click and Drag
`Move Window
`Expose & Dashboard
`Web Browser Operations
`Editing Operations
`File Operations
`
`x.
`
`554
`
`550 )
`
`(
`
`open
`
`t
`'\,.._
`new
`
`/
`
`6
`
`· ; ; a _~ .~ · THUMB+: 1 FINGER
`
`~. tr -···. FHeQperations
`~ I
`
`-~ THUMB + 2 FINGERS
`none
`
`..... ...
`
`•
`/'-save
`
`FIG. 290
`
`}
`
`close
`
`""O
`~ ....
`('D = ....
`
`(')
`
`~ ....
`
`(')
`
`~ ....
`
`"f'j
`('D
`
`t "e -....
`.... 0 =
`""O = O" -....
`.... 0 =
`?' ....
`~ ...
`
`N
`0
`0
`QO
`
`('D
`
`rJJ =(cid:173)
`('D ....
`0 .... ... 1,0
`
`N
`-....J
`
`c
`rJJ
`N
`0
`0
`
`0
`~
`O'I
`
`QO ---0
`-....J ... ~
`> ....
`
`Petitioner Samsung Ex-1008, 0028
`
`
`
`Standard Trackpad
`
`_j
`552 = List of Chords
`
`< Custom Set>
`
`Gesture Contr
`
`Basic Multitouc
`
`I: I
`
`558
`
`~ 4FINGERS
`~ )
`Expose and Dashboard
`
`~ THUMB + 1 FINGER
`
`File Operations
`
`····~·••·
`
`-··-·-··· ..
`
`-····-··.
`
`·• .. ·.Editing,Gp~•=~:::~.;:•.:••·••·=:::c:.
`. -·······-····-······-·-·-·········-···- ........ , ... , ..... _ ..... ,, ......... , .. ,_,_ ........... , ___ _
`
`~ THUMB+ 3 FINGERS
`
`none
`
`-~ THUMB+ 4 FINGERS
`\
`)
`none
`
`....
`....
`
`None
`Point
`PoinUClick
`Scroll
`Scroll & Secondary Click
`Primary Click & Drag
`Secondary Click and Drag
`Move Window
`Expose & Dashboard
`Web Browser Operations
`Editing Operations·
`File Operations
`
`X
`
`r t
`
`cancel
`
`..
`
`undo
`
`0
`
`bktab
`
`t
`/ ' cut
`
`copy
`redo
`
`i
`
`FIG. 29E
`
`554
`
`550 )
`
`56
`
`/
`
`t
`'\,..
`paste
`..
`
`tab
`
`}
`
`select all
`
`""O
`~ .....
`('D = .....
`
`(')
`
`~ .....
`
`(')
`
`~ .....
`
`t "e -....
`.... 0 =
`""O = O" -....
`.... 0 =
`?' ....
`~ ...
`
`"f'j
`('D
`
`N
`0
`0
`QO
`
`('D
`('D
`
`rJJ =(cid:173)
`.....
`N
`QO
`
`0 .... ... 1,0
`
`c
`rJJ
`N
`0
`0
`
`0
`~
`O'I
`
`QO ---0
`-....J ... ~
`> ....
`
`Petitioner Samsung Ex-1008, 0029
`
`
`
`Gesture Contr
`
`Standard Trackpad
`
`Basic Multitouc
`
`552_5
`/f==-~
`I ~ ) Expose and Dashboard
`~ THUMB+ 1 FINGER
`
`< Custom Set >
`
`List of Chords
`
`4FINGERS
`
`File Operations
`
`5{8
`
`@]
`
`1-1
`
`_ ~ THUMB + 2 FINGERS
`~ )
`Editing Commands
`
`.~;L . . • .. THUM~
`. - ·····--··------
`
`: : FINGERS .
`0
`
`- ~ THUMB + 4 FINGERS
`~ )
`none
`
`~ ....
`~ ....
`
`None
`Point
`PoinUClick
`Scroll
`Scroll & Secondary Click
`Primary Click & Drag
`Secondary Click and Drag
`Move Window
`Expose & Dashbo~~d ... .....
`· Web Browser Operations
`Editing Operations
`File Operations
`
`_
`x
`
`554
`
`ll..,
`
`550 )
`
`home
`
`stop
`
`spot light
`
`6
`
`~ i /
`0
`..
`
`-
`
`back
`
`Cmd
`
`forward
`
`find
`
`/
`
`refresh
`
`Book mark
`
`i ~
`
`FIG. 29F
`
`""O
`~ ....
`('D = ....
`
`(')
`
`~ ....
`
`(')
`
`~ ....
`
`"f'j
`('D
`
`t "e -....
`.... 0 =
`""O = O" -....
`.... 0 =
`?' ....
`~ ...
`
`N
`0
`0
`QO
`
`('D
`
`rJJ =(cid:173)
`('D ....
`0 .... ... 1,0
`
`N
`1,0
`
`c
`rJJ
`N
`0
`0
`
`0
`~
`O'I
`
`QO ---0
`-....J ... ~
`> ....
`
`Petitioner Samsung Ex-1008, 0030
`
`
`
`Gesture Contr~
`
`X
`
`Standard Trackpad
`
`Basic Multitouc
`
`552_5
`
`< Custom Set >
`
`List of Chords
`
`~
`
`I
`
`554
`
`.. No~e
`Point.·
`Point/Click
`Scroll
`Scroll & Secondary Click
`Primary Click & Drag
`Secondary Click and Drag
`Move Window
`Expose & Dashboard
`Web Browser Operations
`Editing Operations
`File Operations
`
`550 )
`
`File Operations
`
`Editing Commands
`
`none
`
`~ t /
`
`point
`
`point
`
`point
`
`56
`
`...
`point
`
`point
`
`/
`
`...
`
`point
`
`point
`
`point
`
`i ~
`
`....
`....
`
`FIG. 29G
`
`Expose and Dashboard
`
`- -
`
`- -
`
`558 ~ 4 FINGERS
`~ THUMB + 1 FINGER
`~ THUMB + 2 FINGERS
`~ THUMB+ 3 FINGERS
`'-.: c~.·.·•••.••·••··.• ; THUMB-+A FINGERS
`'!;.: :} .
`none
`
`""O
`~ .....
`('D = .....
`
`(')
`
`~ .....
`
`t "e -....
`.... 0 =
`""O = O" -....
`.... 0 =
`?' ....
`~ ...
`
`(')
`
`~ .....
`
`"f'j
`('D
`
`N
`0
`0
`QO
`
`('D
`('D
`
`rJJ =(cid:173)
`.....
`~
`0
`
`0 .... ... 1,0
`
`c
`rJJ
`N
`0
`0
`
`0
`~
`O'I
`
`QO ---0
`-....J ... ~
`> ....
`
`Petitioner Samsung Ex-1008, 0031
`
`
`
`Patent Application Publication Feb. 14, 2008 Sheet 31 of 49
`
`US 2008/0036743 Al
`
`6 0 0 ,
`
`602
`
`PLACE SYSTEM IN GESTURE MAPPING MODE
`
`DETECT TOUCH
`
`STORE ARRANGEMENT OF CONTACTS ASSOCIATED WITH
`TOUCH
`
`MONITOR TOUCH FOR GESTURE EVENTS
`
`STORE GESTURE EVENT AND LINK TO ARRANGEMENT OF
`CONTACTS
`
`ASSIGN COMMAND TO THE ARRANGEMENT OF CONTACTS/
`GESTURE EVENT
`
`6 2 0 ,
`
`FIG. 31
`
`OPEN COMMAND MENU
`
`DISPLAY LIST OF COMMANDS
`
`HIGHLIGHT A COMMAND WITHIN COMMAND MENU
`
`DETECT TOUCH
`
`STORE ARRANGEMENT OF CONTACTS ASSOCIATED WITH
`TOUCH
`
`MONITOR TOUCH FQR GESTURE EVENT
`
`ASSIGN INITIAL ARRANGEMENT OF CONTACTS/GESTURE
`EVENTS TO HIGHLIGHTED COMMAND
`
`FIG. 32
`
`612
`
`622
`
`628
`
`Petitioner Samsung Ex-1008, 0032
`
`
`
`Patent Application Publication Feb. 14, 2008 Sheet 32 of 49
`
`US 2008/0036743 Al
`
`640~
`
`DETECT TOUCH
`
`RECOGNIZE ARRANGEMENT OF CONTACTS
`
`SET GESTURE SET BASED AT LEAST IN PART ON
`ARRANGEMENT OF CONTACTS
`
`DETECT PAUSE
`
`DISPLAY GESTURE SET SO THAT USER CAN VIEW COMMANDS
`AND GESTURE EVENTS ASSOCIATED THEREWITH
`
`REMOVE DISPLAYED GESTURE SET WHEN A GESTURE EVENT
`IS STARTED OR AFTER A PRESET TIME OUT
`
`642
`
`646
`
`648
`
`652
`
`FIG. 33
`
`664
`
`660~
`
`662
`
`667
`
`COMMAND 1
`
`(cid:141)
`
`COMMAND4
`
`•
`
`j(~
`COMMAND7
`
`• JI ':i4..
`
`COMMAND 3
`
`..
`
`COMMANDS
`
`__)
`
`COMMAND9
`
`COMMAND2
`
`COMMAND 5
`
`I
`0
`i
`
`COMMANDS
`
`FIG. 34
`
`Petitioner Samsung Ex-1008, 0033
`
`
`
`Patent Application Publication Feb. 14, 2008 Sheet 33 of 49
`
`US 2008/0036743 Al
`
`ACTIVATE TOUCH SCREEN KEYBOARD
`
`DURING TYPING OPERATION, MONITOR KEY ENTRIES OVER
`KEYBOARD
`
`DURING TYPING OPERATIONS, MONITOR SURFACE FOR
`FINGER SWIPES
`
`682
`
`686
`
`688
`
`692
`
`696
`
`700
`
`690
`
`SPACE
`
`694
`
`BACKSPACE
`
`698
`
`DOWN SWIPE?
`
`ENTER/RETURN
`
`702
`
`UP SWIPE?
`
`COMPLETE WORD
`
`FIG. 35
`
`Petitioner Samsung Ex-1008, 0034
`
`
`
`Patent Application Publication Feb. 14, 2008 Sheet 34 of 49
`
`US 2008/0036743 Al
`
`700\
`
`RATE EASIEST CHORD
`
`702
`
`__/
`
`__)
`GROUP OR MATCH EASIEST CHORDS WITH u
`
`7 04
`
`7 06
`
`'
`RATE MOST FREQUENT OPERATIONS
`
`MOST FREQUENT OPERATIONS AND GROUP
`OR MATCH HARDEST CHORDS WITH LESS
`FREQUENT OPERATIONS
`
`FINE TUNE EACH GROUP ~
`
`7 08
`
`FIG. 36
`
`Petitioner Samsung Ex-1008, 0035
`
`
`
`FINGER COMBO
`1 FINGER
`1 FINGER
`1 FINGER
`2 FINGER
`' 2 FINGER
`2 FINGER
`2 FINGER
`3 FINGER
`4 FINGER
`5 FINGER
`THUMB + 1 FINGER
`THUMB+ 2 FINGER
`THUMB+ 3 FINGER
`
`GESTURE
`SLIDE
`TAP
`TAP+ 1/2
`2 FING. DOWN
`SLIDE
`TAP
`SPREAD
`-
`-
`-
`-
`-
`-
`
`SPACE
`*
`
`*
`
`*
`
`**
`
`**
`
`*
`
`***
`
`***
`
`****
`*****
`****
`****
`*****
`
`SPEED
`5
`5
`3
`4
`5
`5
`3
`5
`4
`5
`4
`3
`3
`
`RELIABILITY COMFORT
`5
`5
`2
`5
`2
`3
`4
`3
`5
`5
`5
`5
`2
`3
`4
`4
`4
`3
`5
`5
`3
`3
`3
`3
`3
`3
`
`EASE
`5
`5
`2
`4
`4
`4
`3
`4
`4
`4
`3
`2
`2
`
`OVERALL
`5
`4.25
`2.5
`3.75
`4.75
`4.75
`2.75
`4.25
`3.75
`4.75
`3.25
`2.5
`2.5
`
`FIG. 37
`
`""O
`~
`
`~ = .....
`
`t "e -....
`
`(')
`~
`Q;
`
`0 =
`~
`O" = (')
`.... 0 =
`~ .....
`"f'j
`('D
`?'
`....
`J·
`N
`0
`0
`QO
`
`('D
`('D
`
`rJJ =(cid:173)
`.....
`(,H
`Ul
`
`0 ....
`
`.i;...
`1,0
`
`c
`rJJ
`N
`0
`0
`
`QO ---0
`
`0
`(,H
`O'I
`-....J
`.i;...
`(,H
`
`> ....
`
`Petitioner Samsung Ex-1008, 0036
`
`
`
`Patent Application Publication Feb. 14, 2008 Sheet 36 of 49
`
`US 2008/0036743 Al
`
`COMMAND/
`OPERATION
`
`POINT
`CLICK
`DRAG
`SCROLL
`BACK
`FORWARD
`CONTEXT MENU
`ZOOM
`CUT
`COPY
`PASTE
`SAVE
`EXPOSE
`DASHBOARD
`SPOTLIGHT
`OPEN
`CLOSE
`NEW
`SAVE
`EXIT
`FIND/REPLACE
`UNDO/REDO
`TAB
`ENTER
`
`FREQUENCY
`
`5
`4.5
`4
`3.5
`3
`2.5
`2.5
`2.5
`2
`2
`2
`2
`2
`1.5
`1
`1
`1
`1
`0.5
`0.5
`0.5
`0.5
`0.5
`0.5
`0.5
`
`FIG. 38
`
`Petitioner Samsung Ex-1008, 0037
`
`
`
`Patent Application Publication Feb. 14, 2008 Sheet 37 of 49
`
`US 2008/0036743 Al
`
`7 4 0~
`
`742
`
`)
`
`744
`
`~
`J
`
`74 6
`
`74 8
`
`i_)
`
`7 50
`
`__)
`
`PROVIDE BASE GESTURE MAP
`
`,,
`
`OPEN FIRST PROGRAM
`
`"
`
`ADJUST BASE GESTURE MAP IN ACCORDANCE WITH
`FIRST PROGRAM
`
`,,
`
`OPEN SECOND PROGRAM
`
`"
`ADJUST BASE GESTURE MAP IN ACCORDANCE WITH
`SECOND PROGRAM
`
`FIG. 39
`
`Petitioner Samsung Ex-1008, 0038
`
`
`
`Patent Application Publication Feb. 14, 2008 Sheet 38 of 49
`
`US 2008/0036743 Al
`
`7 6 0~
`
`START
`
`764
`
`762
`
`INITIATE EXPOSE
`FUNCTION 1
`
`768
`
`766
`
`INITIATE EXPOSE
`FUNCTION 2
`
`772
`
`770
`
`_)
`
`INITIATE EXPOSE
`FUNCTION 3
`
`774
`
`776
`
`DOWNWARD
`SWIPE
`
`- - - INITIATE DASHBOARD
`
`FIG. 40
`
`Petitioner Samsung Ex-1008, 0039
`
`
`
`Patent Application Publication Feb. 14, 2008 Sheet 39 of 49
`
`US 2008/0036743 Al
`
`7 8 0~
`
`782
`
`784
`
`786
`
`DETECT BASE CHORD
`
`DETECT PAUSE
`
`OPEN CONTEXTUAL
`MENU
`
`788
`
`_)
`
`790
`
`MOVE HIGHLIGHT BAR
`THROUGH
`CONTEXTUAL MENU
`
`792
`
`794
`
`FIG. 41
`
`IMPLEMENT
`FUNCTIONALITY OF
`HIGHLIGHTED ITEM
`
`Petitioner Samsung Ex-1008, 0040
`
`
`
`Patent Application Publication Feb. 14, 2008 Sheet 40 of 49
`
`US 2008/0036743 Al
`
`Hand
`primary
`
`# of fingers
`1
`
`primary
`
`2 adj.
`
`primary
`
`2 non adj.
`
`primary
`primary
`
`3
`3
`
`primary
`
`Thumb +1
`
`primary
`
`Thumb +2
`
`primary
`primary
`
`Thumb +3
`Thumb +4
`
`gesture event
`tap
`Position/motion
`tap
`Position/motion
`tap
`Position/motion
`Motion
`Slide right
`Slide left
`Slide up
`Slide down
`Slide right
`Slide left
`Slide up
`Slide down
`Rotate CCW
`Rotate CW
`spread
`contract
`tap
`Rotate CCW
`Rotate CW
`spread
`contact
`motion
`motion
`
`FIG. 42
`
`behavior
`click
`point
`click
`drag
`click
`Secondary drag
`Scroll/pan
`forward
`back
`stop
`reload
`tab
`bktab
`undo
`redo
`cancel
`Select all
`paste
`cut
`copy
`open
`close
`new
`save
`Move window
`point
`
`Petitioner Samsung Ex-1008, 0041
`
`
`
`Patent Application Publication Feb. 14, 2008 Sheet 41 of 49
`
`US 2008/0036743 Al
`
`Hand
`primary
`
`# of fingers
`1
`
`primary
`
`2
`
`primary
`primary
`primary
`primary
`
`3
`4
`5
`Thumb +1
`
`primary
`
`Thumb +2
`
`primary
`
`Thumb +3
`
`secondary
`
`secondary
`
`secondary
`
`secqndary
`
`1
`
`2
`
`3
`
`4
`
`secondary
`
`5
`
`behavior
`click
`point
`click
`drag
`Move window
`Scroll/pan
`point
`undo
`redo
`paste
`cut
`copy
`Expose 1
`Expose 2
`Expose 3
`dashboard
`replace
`find
`back
`forward
`open
`close
`new
`save
`
`Arrow up (text cursor)
`Arrow down
`Arrow right
`Arrow left
`Apply alUoption to other hand
`
`Select up (e.g., <shift> arrow)
`Select down
`Select right
`Select left
`Move/resize window in
`
`Conjunction with opposite hand
`Page up
`Page down
`begin
`endline
`Apply shift to opposite hand
`Zoom out screen
`Zoom in screen
`Rotate CCW
`Rotate CW
`
`gesture event
`tap
`Position/motion
`tap
`Position/motion
`motion
`motion
`motion
`Slide up
`Slide down
`spread
`contract
`tap
`Slide left
`Slide right
`Slide up
`Slide down
`spread
`contract
`Slide left
`Slide right
`Rotate CCW
`Rotate CW
`spread
`contact
`
`Slide up
`Slide down
`Slide right
`Slide left
`pause
`Slide up
`Slide down
`Slide right
`Slide left
`motion
`
`Slide up
`Slide down
`Slide right
`Slide left
`pause
`spread
`contact
`Rotate CCW
`Rotate CW
`
`FIG. 43
`
`Petitioner Samsung Ex-1008, 0042
`
`
`
`Patent Application Publication Feb. 14, 2008 Sheet 42 of 49
`
`US 2008/0036743 Al
`
`Hand
`primary
`
`# of fingers
`1
`
`primary
`
`primary
`primary
`
`primary
`primary
`
`primary
`
`primary
`
`2
`
`3
`4
`
`5
`Thumb+
`Middle+
`ring
`Thumb+
`index+
`pinky
`L3 + R3
`
`gesture event
`tap
`Position/motion
`tap
`spread
`contract
`slide
`rotate
`motion
`motion
`tap
`motion
`motion
`
`behavior
`click
`point
`Secondary click (right click)
`Enlarge object
`shrink object
`pan
`Rotate object
`Primary drag
`Scroll pan
`Secondary click (right click)
`point
`Secondary drag
`
`tap
`motion
`Slide up
`tap
`motion
`
`expose
`Window drag, resize
`Expose 3
`Maximize/restore
`
`Opposite comer frame drag/resize
`
`FIG. 44
`
`Petitioner Samsung Ex-1008, 0043
`
`
`
`Patent Application Publication Feb. 14, 2008 Sheet 43 of 49
`
`US 2008/0036743 Al
`
`FIG. 45
`
`FIG. 46
`
`FIG. 4 7
`
`FIG. 48
`
`Petitioner Samsung Ex-1008, 0044
`
`
`
`Patent Application Publication Feb. 14, 2008 Sheet 44 of 49
`
`US 2008/0036743 Al
`
`800~
`
`DETECT FIRST FINGER
`
`,,
`
`DETERMINE STATE OF FIRST FINGER
`
`,,
`
`DETECT ADDITIONAL FINGER(S)
`
`,,
`
`DETERMINE STATE OF ADDITIONAL
`FINGER(S)
`
`802
`l_/
`
`8 04
`l_/
`
`8 06
`l_/
`
`08
`8
`L)
`
`,,
`
`81 0
`
`IMPLEMENT DIFFERENT INPUT MODES l)
`
`BASED ON TIMING OF STATES OF FIRST AND
`ADDITIONAL FINGER(S) RELATIVE TO ONE
`ANOTHER
`
`FIG. 49
`
`Petitioner Samsung Ex-1008, 0045
`
`
`
`Patent Application Publication Feb. 14, 2008 Sheet 45 of 49
`
`US 2008/0036743 Al
`
`820~
`
`DETECT FIRST FINGER
`
`822
`
`J
`
`1r
`
`DETERMINE STATE OF FIRST FINGER J
`
`8 24
`
`H
`
`8 26
`IMPLEMENT FIRST MODE WITH FIRST _/
`FINGER WHEN FIRST FINGER IS IN FIRST
`STATE
`
`1'
`
`MONITOR FOR MODIFIER FINGER EVENT
`WHEN FIRST FINGER IN SECOND STATE
`
`28
`8
`I__)
`
`''
`IMPLEMENT SECOND MODE WITH FIRST _)
`
`FINGER WHEN FIRST FINGER IS IN SECOND
`STATE AND MODIFIER FINGER EVENT IS
`DETECTED
`
`8 30
`
`FIG. 50
`
`Petitioner Samsung Ex-1008, 0046
`
`
`
`Patent Application Publication Feb. 14, 2008 Sheet 46 of 49
`
`US 2008/0036743 Al
`
`840~
`
`INITIALLY DETECT FIRST FINGER
`
`842
`__/
`
`,r
`
`FINGER WHEN FIRST FINGER IS MOVED J
`
`IMPLEMENT FIRST MODE WITH FIRST
`
`8 44
`
`1'
`
`DETECT SECOND FINGER AFTER
`DETECTING FIRST FINGER
`
`8 46
`__/
`
`,,
`
`8 48
`
`SWITCH FROM FIRST MODE TO SECOND V
`
`MODE WHEN FIRST FINGER IS STATIONARY
`AND SECOND FINGER IS DETECTED
`
`1'
`
`8 50
`
`IMPLEMENT SECOND MODE WITH AT LEAST l)
`
`FIRST FINGER WHEN FIRST FINGER IS
`MOVED
`
`FIG. 51
`
`Petitioner Samsung Ex-1008, 0047
`
`
`
`Patent Application Publication Feb. 14, 2008 Sheet 47 of 49
`
`US 2008/0036743 Al
`
`860~
`
`START
`
`862
`
`NO
`
`866
`
`PERFORM POINTING
`OPERATION WITH
`FIRST FINGER
`MOVEMENT
`
`NO
`
`868
`
`NO
`
`870
`
`YES
`
`NO
`
`872
`
`YES
`
`PERFORM DRAGGING
`OPERATION WITH
`FIRST FINGER
`MOVEMENT
`
`FIG. 52
`
`Petitioner Samsung Ex-1008, 0048
`
`
`
`Patent Application Publication Feb. 14, 2008 Sheet 48 of 49
`
`US 2008/0036743 Al
`
`880~
`
`START
`
`NO
`>---~ STANDBY
`
`884
`
`IMPLEMENT
`POINTING MODE
`
`886
`
`STATIONARY
`
`892
`
`MOVING
`
`LOOK FOR MODIFIER
`EVENT
`
`IGNORE MODIFIER
`EVENT
`
`NO
`
`896
`
`888
`
`PERFORM POINTING
`OPERATIONS
`
`SWITCH FROM
`POINTING MODE TO 1 - -~ PERFORM DRAGGING
`DRAGGING MODE
`OPERATIONS
`
`898
`
`FIG. 53
`
`Petitioner Samsung Ex-1008, 0049
`
`
`
`Patent Application Publication Feb. 14, 2008 Sheet 49 of 49
`
`US 2008/0036743 Al
`
`00
`00
`0)
`
`>-0::
`0
`~
`UJ
`~
`
`0::
`0
`CJ')
`CJ')
`UJ
`<.)
`0
`0::
`CL
`
`UJ
`<.)
`
`> UJ
`0
`I-
`:::)
`CL z
`
`00
`LO
`0)
`
`;
`
`CL
`CJ')
`0
`
`co
`LO
`0)
`
`00 co
`
`0)
`
`0
`I"-
`0)
`
`UJ
`0::
`:::)
`LO ------------- 1-
`00
`Cl)
`UJ
`0)
`(.9
`
`0) co
`
`0)
`
`v
`LO .
`CJ
`LL
`
`Petitioner Samsung Ex-1008, 0050
`
`
`
`US 2008/0036743 Al
`
`Feb. 14,2008
`
`1
`
`GESTURING WITH A MULTIPOINT SENSING
`DEVICE
`
`CROSS-REFERENCE TO RELATED
`APPLICATIONS
`
`[0001] This application claims priority benefit of U.S.
`Provisional Patent Application No. 60/763,605, filed Jan. 30,
`2006, and entitled "GESTURING WITH A MULTIPOINT
`SENSING DEVICE," which is herein incorporated by ref(cid:173)
`erence.
`
`[0002] This application is a Continuation-in-Part of U.S.
`patent application Ser. No. 10/903,964, filed on Jul. 30,
`2004, entitled "GESTURES FOR TOUCH SENSITIVE
`INPUT DEVICES," and which is herein incorporated by
`reference.
`
`[0003] This application is a Continuation-in-Part of U.S.
`patent application Ser. No. 11/015,434, filed on Dec. 17,
`2004, entitled "METHOD AND APPARATUS FOR INTE(cid:173)
`GRATING MANUAL INPUT," which claims priority to
`U.S. Pat. No. 6,323,846, issued on Nov. 27, 2001, which
`claims priority to U.S. Provisional Patent Application No.
`60/072,509 filed Jan. 26, 1998, all of which are herein
`incorporated by reference.
`
`[0004] This application