as) United States
`a2) Patent Application Publication (10) Pub. No.: US 2002/0015024 Al
`(43) Pub. Date: Feb. 7, 2002
`
`Westermanetal.
`
`20020015024A
`
`METHOD AND APPARATUS FOR
`INTEGRATING MANUAL INPUT
`
`(57)
`
`ABSTRACT
`
`Apparatus and methods are disclosed for simultaneously
`tracking multiple finger and palm contacts as hands
`approach,touch, andslide across a proximity-sensing. com-
`pliant, and flexible multi-touch surface. The surface consists
`of compressible cushion, dielectric, electrode, and circuitry
`layers. A simple proximity transduction circuit is placed
`under each electrode to maximize signal-to-noise ratio and
`to reduce wiring complexity. Such distributed transduction
`circuitry is economical for large surfaces when implemented
`with thin-film transistor techniques. Scanning and signal
`offset removal on an electrode array produces low-noise
`proximity images. Segmentation processing of each prox-
`imity image constructs a group of electrodes corresponding
`to each distinguishable contact and extracts shape, position
`and surface proximity features for each group. Groups in
`successive images which correspond to the same hand
`contact are linked by a persistent path tracker which also
`detects individual contact touchdown and liftoff. Combina-
`torial optimization modules associate each contact’s path
`with a particular fingertip, thumb, or palm of either hand on
`the basis of biomechanical constraints and contact features.
`
`Classification of intuitive hand configurations and motions
`enables unprecedented integration of typing, resting, point-
`ing, scrolling, 3D manipulation, and handwriting into a
`versatile, ergonomic computer input device.
`
`(54)
`
`(75)
`
`Inventors: Wayne Westerman, Wellington, MO
`(US); John G. Elias, Townsend, DE
`(US)
`
`Correspondence Address:
`CONNOLLY BOVE LODGE & HUTZ LLP
`P.O. Box 2207
`
`Wilmington, DE 19899 (US)
`
`(73)
`
`Assignee: University of Delaware
`
`(21)
`
`Appl. No.:
`
`09/919,266
`
`(22)
`
`Filed:
`
`Jul. 31, 2001
`
`Related U.S. Application Data
`
`(60)
`
`Division of application No. 09/236,513, filed on Jan.
`25, 1999. Non-provisional of provisional application
`No. 60/072,509, filed on Jan. 26, 1998.
`
`Publication Classification
`
`(51)
`(52)
`
`Int. Cl.’
`US. Cl.
`
`sesseesceaeaeaessessessesscssesscescesnsensanssneets G09G 5/00
`sesseeseesesaeaeasesnsensessssseseesesnesnssnssneseeseeseea 345/173
`
`
`
`Petitioner Samsung Ex-1032, 0001
`
`Petitioner Samsung Ex-1032, 0001
`
`

`

`Patent Application Publication
`
`Feb. 7, 2002 Sheet 1 of 45
`
`US 2002/0015024 Al
`
`4
`
`ELECTRODE
`SCANNING
`HARDWARE
`
`6
`
`2
`
`CALIBRATION AND
`PROXIMITY IMAGE
`
`8
`
` 10 CONTACT
`
`TRACKING AND
`IDENTIFICATION
`
`12
`
`FORMATION
`
`
` 22
`
`
`
`HAND
`
`FINGER
`
`
`
`MOTION
`TYPING
`
`
`RECOGNIZER eeeOTOR| [COMPONENT
`
`
`
`EXTRACTION DETECTOR
`
`
`
`
`
`CHORD MOTION
`RECOGNIZER
`
`24
`
`DISPLAY
`
`HOST
`COMPUTER
`SYSTEM
`
`20
`
`HOST
`COMMUNICATION
`INTERFACE
`
`FIG. 1
`
`Petitioner Samsung Ex-1032, 0002
`
`Petitioner Samsung Ex-1032, 0002
`
`

`

`Patent Application Publication
`
`Feb. 7, 2002 Sheet 2 of 45
`
`US 2002/0015024 Al
`
`
`
`Petitioner Samsung Ex-1032, 0003
`
`Petitioner Samsung Ex-1032, 0003
`
`

`

`Patent Application Publication
`
`Feb. 7, 2002 Sheet 3 of 45
`
`US 2002/0015024 Al
`
`
`
`FIG. 3A
`
`
`
`FIG. 3B
`
`Petitioner Samsung Ex-1032, 0004
`
`Petitioner Samsung Ex-1032, 0004
`
`

`

`Patent Application Publication
`
`Feb. 7, 2002 Sheet 4 of 45
`
`US 2002/0015024 Al
`
`38
`
`ISEEEERRRERN
`
`estes|
`
`TIME
`
`FIG. 4B
`
`Petitioner Samsung Ex-1032, 0005
`
`Petitioner Samsung Ex-1032, 0005
`
`

`

`Patent Application Publication
`
`Feb. 7, 2002 Sheet 5 of 45
`
`US 2002/0015024 Al
`
` -
`
`7
`
`FIG. 5B
`
`Petitioner Samsung Ex-1032, 0006
`
`Petitioner Samsung Ex-1032, 0006
`
`

`

`Patent Application Publication
`
`Feb. 7, 2002 Sheet 6 of 45
`
`US 2002/0015024 Al
`
`9
`
`o>
`
`0W
`
`w
`
`a \
`
`2
`
`
`THU BUI yay
`
`TTIOOUUOTT
`
`opahaan
`atlattanlsrtststts
`
`SenetHet
`eee
`
`sarah Phat L syPLT|
`GSTS g
`sista)ststayateya
`
`&wt
`
`ana
`
`o
`
`+
`
`a
`
`st
`
`(>>
`
`alWw
`
`34
`
`Petitioner Samsung Ex-1032, 0007
`
`Petitioner Samsung Ex-1032, 0007
`
`

`

`Patent Application Publication
`
`Feb. 7, 2002 Sheet 7 of 45
`
`US 2002/0015024 Al
`
`
`
`FIG. 7A
`
`
`
`Petitioner Samsung Ex-1032, 0008
`
`Petitioner Samsung Ex-1032, 0008
`
`

`

`Patent Application Publication
`
`Feb. 7, 2002 Sheet 8 of 45
`
`US 2002/0015024 Al
`
`Cy
`
`
`
`
`
`FIG. 8
`
`Petitioner Samsung Ex-1032, 0009
`
`Petitioner Samsung Ex-1032, 0009
`
`

`

`Patent Application Publication
`
`Feb. 7, 2002 Sheet 9 of 45
`
`9/45
`
`US 2002/0015024 Al
`
`Petitioner Samsung Ex-1032, 0010
`
`Petitioner Samsung Ex-1032, 0010
`
`

`

`Patent Application Publication
`
`Feb. 7, 2002 Sheet 10 of 45
`
`US 2002/0015024 Al
`
`86
`
`85
`
`87
`
`Petitioner Samsung Ex-1032, 0011
`
`Petitioner Samsung Ex-1032, 0011
`
`

`

`Patent Application Publication
`
`Feb. 7, 2002 Sheet 11 of 45
`
`US 2002/0015024 A1
`
`85
`
`ee90
`
`Petitioner Samsung Ex-1032, 0012
`
`Petitioner Samsung Ex-1032, 0012
`
`

`

`Patent Application Publication
`
`Feb. 7, 2002 Sheet 12 of 45
`
`US 2002/0015024 Al
`
`Petitioner Samsung Ex-1032, 0013
`
`Petitioner Samsung Ex-1032, 0013
`
`
`

`

`Patent Application Publication
`
`Feb. 7, 2002 Sheet 13 of 45
`
`US 2002/0015024 Al
`
`"| — N
`N
`
`
`
`VERTICALPOSITIONONSURFACE(YAXIScm)
`
`— oO
`
`{
`
`0
`
`2
`
`16
`14
`12
`10
`8
`6
`4
`HORIZONTAL POSITION ON SURFACE (X AXIS cm)
`
`18
`
`FIG. 13
`
`Petitioner Samsung Ex-1032, 0014
`
`Petitioner Samsung Ex-1032, 0014
`
`

`

`Patent Application Publication
`
`Feb. 7, 2002 Sheet 14 of 45
`
`US 2002/0015024 Al
`
`ay
`
`
`
`201
`
`i
`
` 0
`
`2
`
`16
`14
`12
`10
`8
`6
`4
`HORIZONTAL POSITION ON SURFACE (X AXIS cm)
`
`18
`
`FIG. 14
`
`Petitioner Samsung Ex-1032, 0015
`
`oo
`
`— No
`
`— Oo
`
`
`
`VERTICALPOSITIONONSURFACE(YAXIScm) |
`
`4
`
`i)
`
`Petitioner Samsung Ex-1032, 0015
`
`

`

`Patent Application Publication
`
`Feb. 7, 2002 Sheet 15 of 45
`
`US 2002/0015024 A1
`
`14
`
`
`
`VERTICALPOSITIONONSURFACE(YAXIScm)
`
`206
`
`0
`
`2
`
`16
`14
`12
`10
`8
`6
`4
`HORIZONTAL POSITION ON SURFACE (X AXIS cm)
`
`18
`
`FIG. 15
`
`Petitioner Samsung Ex-1032, 0016
`
`Petitioner Samsung Ex-1032, 0016
`
`

`

`Patent Application Publication
`
`Feb. 7, 2002 Sheet 16 of 45
`
`US 2002/0015024 Al
`
`
` CURRENT
`
`
`PROXIMITY
`IMAGE
`
`240
`
`IMAGE
`SEGMENTATION
`
`
`
`
`
`PATHS FROM
`PARAMETERIZED
`
`
`
`LECTRODE GROUP
`
`CONTACT PATH
`TRACKING
`
`
`PREVIOUS IMAGES
`
`
`
`
`NEW PATHS &
`
`
`UPDATED PATH
`PARAMETERS
`
` 247
`
`
`
`
`
`
`
`HAND
`IDENTIFICATION
`
`FINGER & PALM
`IDENTIFICATION
`
`
`
`IDENTIFIED
`CONTACT PATHS
`
`HAND POSITION
`ESTIMATION
`
`
`
`ESTIMATED HAND &
`FINGER OFFSETS
`
`FIG. 16
`
`Petitioner Samsung Ex-1032, 0017
`
`Petitioner Samsung Ex-1032, 0017
`
`

`

`Patent Application Publication
`
`Feb. 7, 2002 Sheet 17 of 45
`
`US 2002/0015024 A1
`
`GET HAND'S
`IDENTIFIED PATHS
`
`250
`
`COMPUTE OFFSETS BETWEEN
`EACH FINGER'S MEASURED AND
`DEFAULT POSITIONS
`
`COMPUTE AVERAGE OF
`OFFSETS WEIGHTED BY
`
`CONTACT PROXIMITY
`CONTACT VELOCITIES
`
`ADJUST FILTER POLE TO
`CURRENTIDENTIFICATION
`CONFIDENCE
`
`COMPUTE WEIGHTED
`AVERAGE OF HAND
`
`254
`
`255
`
`256
`
`257
`
`
`
`AUTOREGRESSIVELY UPDATE HAND
`OFFSET ESTIMATES FROM MEASURED
`OFFSETS AND VELOCITIES
`
`258
`
`UPDATE FINGER
`OFFSET ESTIMATES
`
`259
`
`CONVERT ESTIMATED OFFSETS
`TO ABSOLUTE POSITIONS
`
`260
`
`END
`
`FIG. 17
`
`Petitioner Samsung Ex-1032, 0018
`
`Petitioner Samsung Ex-1032, 0018
`
`

`

`
`SMOOTHED
`
`PROXIMITY
`
`IMAGE
`
`
`
`
`
`
`
`
`SEARCH FOR
`SIGNIFICANT
`LOCAL MAXIMA
`
`LOCAL
`MAXIMUM
`PIXELS
`
`267
`
`
`FLATTENED
`FINGERTIP
`
`FEEDBACK
`
`252
`
`
`
`
`
`ESTIMATED HAND
`POSITION OFFSETS
`
`DEFINE SEGMENTATION
`STRICTNESS REGIONS
`
`\ 266
`
`Patent Application Publication
`
`Feb. 7, 2002 Sheet 18 of 45
`
`US 2002/0015024 Al
`
`240
`
` CURRENT
`PROXIMITY
`IMAGE
`
`DIFFUSE
`CURRENT IMAGE
`
`
`
`
`
`
`
`
`
`
`
`CONSTRUCT ELECTRODE
`GROUPS AROUND LOCAL
`MAXIMUM PIXELS
`
`COMBINE
`OVERLAPPING
`
`GROUPS
`GROUPS
`
`FIT ELLIPSES TO
`COMBINED
`
`270
`
`are
`
`
`
`
`PARAMETERIZED
`ELECTRODE GROUPS
`
`
`242
`
`FIG. 18
`
`Petitioner Samsung Ex-1032, 0019
`
`Petitioner Samsung Ex-1032, 0019
`
`

`

`Patent Application Publication
`
`Feb. 7, 2002 Sheet 19 of 45
`
`US 2002/0015024 Al
`
`281
`
`279
`
`278
`
`277
`
`281
`
`281
`
`FIG. 19
`
`Petitioner Samsung Ex-1032, 0020
`
`Petitioner Samsung Ex-1032, 0020
`
`

`

`Patent Application Publication
`
`Feb. 7, 2002 Sheet 20 of 45
`
`US 2002/0015024 Al
`
`15
`
`
`
`10
`VERTICALSURFACEPOSITION(cm)
`
`
`
`
`-20
`
`-15
`
`10
`5
`0
`-5
`-10
`HORIZONTAL SURFACE POSITION (cm)
`FIG. 20C
`
`15
`
`20
`
`Petitioner Samsung Ex-1032, 0021
`
`Petitioner Samsung Ex-1032, 0021
`
`

`

`Patent Application Publication
`
`Feb. 7, 2002 Sheet 21 of 45
`
`US 2002/0015024 A1
`
`290
`
`RAW PROXIMITY >
`BACKGROUND?
`
`
`Y
`
`N
`
`
`
`GET NEXT ELECTRODE IN
`DIRECTION OF SEARCH
`
`REACHED BACKGROUND
`LEVEL EDGE
`
`304
`
`
`
`
`SEGMENTATION
`HonontaL?
`
`
`
`
`
`
`
`
`REGION?
`
`Cy
`
`N
`
`
`
`SEARCHING
`HORIZONTAL ?
`
`
`
`312
`
`y
`
`310
`
`
`REACHED EDGE
`
`
`BETWEEN
`FINGERTIP AND
`
`THUMB OR PALM
`
`+.
`
`v
`
`296
`
`
`DIST. TO LOCAL MAX
`ORIZ
`
`OR DIAGONAL
`
`
`MINIMUM ?
`
`
`
`300
`
`298
`
`
`
`TALL
`
`HORIZONTAL 314_.|REACHED EDGE
`MINIMUM ?
`BETWEEN
`FINGERS
`
`REACHED EDGE
`BETWEEN PALM
`HEELS
`
`END
`
`END
`
`FIG. 21
`
`Petitioner Samsung Ex-1032, 0022
`
`Petitioner Samsung Ex-1032, 0022
`
`

`

`Patent Application Publication
`
`Feb. 7, 2002 Sheet 22 of 45
`
`US 2002/0015024 Al
`
`START
`
`PREDICT CURRENT POSITIONS
`OF EXISTING PATHS
`
`320
`
`FOR EACH GROUP
`FIND CLOSEST PATH
`
`322
`
`FOR EACH PATH, FIND
`CLOSEST GROUP WITHIN
`
`FORM GROUP-PATHPAIRSIF
`GROUP & ACTIVE PATH ARE
`
`TRACKING RADIUS
`CLOSEST TO ONE ANOTHER
`DEACTIVATED PATHS
`
`ATTEMPT TO PAIR REMAINING
`GROUPS WITH RECENTLY
`
`324
`
`326
`
`334
`
`ALLOCATE NEW PATHS FOR ANY
`REMAINING UNPAIRED GROUPS
`
`336
`
`DEACTIVATE ANY REMAINING
`UNPAIRED PATHS
`
`344
`
`UPDATE PATH
`PARAMETERS
`
`346
`
`FIG. 22
`
`Petitioner Samsung Ex-1032, 0023
`
`Petitioner Samsung Ex-1032, 0023
`
`

`

`Patent Application Publication
`
`Feb. 7, 2002 Sheet 23 of 45
`
`US 2002/0015024 Al
`
`DEFINE IDENTITY
`ATTRACTORS AT DEFAULT
`
`350
`
`TRANSLATE ATTRACTOR|59
`TEMPLATE BY ESTIMATED
`
`CONTACT POSITIONS
`HAND OFFSET
`TO EACH ATTRACTOR
`FEATURES OF EACH PATH
`
`
`
`
`COMPUTE MATRIX OF
`DISTANCES FROM EACH PATH
`
`COMPUTE ATTRACTOR
`WEIGHTING FACTORS FROM
`
`FIND ASSIGNMENT BETWEEN
`PATHS AND ATTRACTORS
`WHICH MINIMIZES SUM OF
`WEIGHTED DISTANCES
`
`
`
`354
`
`356
`
`358
`
`362
`
`
`
`# FINGER
`
`HAND
`
`
`ATTRACTORS
`
`
`ASSIGNMENTS
`
`
`
`ASSIGNED> 1
`
`
`ENTATIVE 2
`OR <5?
`
`VERIFY THUMB
`ASSIGNMENT
`
`368
`
` UPDATE FINGER
`
`
`
`
`COUNTS AND
`SUBSETS
`
`END
`
`FIG. 23
`
`Petitioner Samsung Ex-1032, 0024
`
`Petitioner Samsung Ex-1032, 0024
`
`

`

`Patent Application Publication
`
`Feb. 7, 2002 Sheet 24 of 45
`
`US 2002/0015024 Al
`
`12
`
`10
`
`(cm)
`
`
`VERTICALSURFACEPOSITION
`
`
`
`
`
`
`
`0 18=~«202 4 6 8 10 12 14 #16
`
`
`
`
`
`HORIZONTAL SURFACE POSITION (cm)
`
`
`
`FIG. 24
`
`Petitioner Samsung Ex-1032, 0025
`
`Petitioner Samsung Ex-1032, 0025
`
`

`

`Patent Application Publication
`
`Feb. 7, 2002 Sheet 25 of 45
`
`US 2002/0015024 Al
`
`150
`100
`50
`CONTACT ORIENTATION(degrees)
`FIG. 25A
`
`0
`
`{ 0
`
`RIGHT
`
`ORIENTATION
`
`FACTOR
`
`0
`
`5
`4
`3
`2
`1
`CONTACTSIZE (NORMALIZED TOTAL PROXIMITY)
`FIG. 25B
`
`ND
`
`G
`So
`=x
`ie |
`coal
`0
`
`or
`i O
`io
`Ig
`SuLL
`aN
`? 9
`
`0
`
`5
`4
`3
`2
`1
`TOTAL PROXIMITY DIVIDED BY ECCENTRICITY
`FIG. 25C
`
`6
`SEO 1
`ASF
`<eO
`ou x <x
`in LL
`
`0
`
`0
`
`6
`4
`2
`DISTANCE TO NEAREST NEIGHBOR CONTACT(cm)
`FIG. 25D
`
`Petitioner Samsung Ex-1032, 0026
`
`Petitioner Samsung Ex-1032, 0026
`
`

`

`Patent Application Publication
`
`Feb. 7, 2002 Sheet 26 of 45
`
`US 2002/0015024 Al
`
`COMPUTE INTER-PATH
`THUMB FACTORS
`
`COMBINE WITH THUMB SIZE
`& ORIENTATION FACTORS
`OF INNERMOST AND NEXT
`
`INNERMOST CONTACT
`
`400
`
`402
`
`
`
`
`Combined_thumb_fact >
`
`
`ASSIGNED TO
`is_thumb_thresh?
`
`THUMB?
`
`
`
`
`ATTRACTOR
`
`
`ASSIGNMENTS
`
`SHIFT INNERMOST
`PATH TO THUMB
`
`414
`
`413
`
`Combined_thumb_fact
`not_thumb_thresh?
`
`
`INNERMOST
`ASSIGNED TO
`
`THUMB?
`
`
`EXISTING
`
`OK
`
`SHIFT INNERMOST
`PATHS AWAY FROM
`THUMB ATTRACTOR
`
`4410
`
`
`
`END
`
`FIG. 26
`
`Petitioner Samsung Ex-1032, 0027
`
`Petitioner Samsung Ex-1032, 0027
`
`

`

`Patent Application Publication
`
`Feb. 7, 2002 Sheet 27 of 45
`
`US 2002/0015024 Al
`
`GET ALL PATHS ASSIGNED
`TO THE GIVEN HAND
`
`430
`
`432
`
`434
`
`COMPUTE DISTANCES
`FROM EACH PATH TO
`
`FIND SHORTEST RING
`CONNECTING ALL PATHS
`AND PASSING THROUGH
`
`OTHER PATHS
`EACH ONCE
`WEIGHTING FACTORS FOR EACH PATH
`
`COMPUTE THUMB & PALM|46
`
`PICK INNERMOST|,—
`PATH IN RING
`
`438
`
`442
`
`440
`
`
`
`
`INNERMOST
`
`
`INNERMOST
`
`PATH A PALM
`
`
`
`PATH THUMB ?
`
`HEEL?
`
`
`
`
`
`
`
`
`
`
`PATHS ABOVETHIS
`
`
`PATHS AT THIS
`PATHS AT THIS
`
`VERTICAL LEVEL ARE
`
`
`
`VERTICAL LEVEL
`VERTICAL LEVEL
`FINGERTIPS, BELOW
`
`
`
`
`ARE FINGERTIPS
`ARE PALMS
`ARE PALMS
`
`
`444
`
`END
`
`FIG. 27
`
`Petitioner Samsung Ex-1032, 0028
`
`Petitioner Samsung Ex-1032, 0028
`
`

`

`Patent Application Publication
`
`Feb. 7, 2002 Sheet 28 of 45
`
`US 2002/0015024 Al
`
`486
`
`
`
`omeTSTOLOMINc
`PREVIOUSLY
`
`
`
`
`
`PENGRIP
`
`N
`
`DETECTED ?
`
`
`GET ESTIMATED FINGER &|
`PALM POSITIONS FOR
`
`LIFTED HAND PARTS
`
`
`GET MEASURED POSITIONS &
`SIZES OF TOUCHING FINGERS
`& PALM HEELS
`
`488
`
`END
`
`90
`
`COMPUTE KNUCKLE FACTOR
`FOR OUTER FINGERS
`
`489
`
`COMPUTE INDEX JUTTING
`FACTOR FOR INNER FINGERS
`
`490
`
`COMBINE FACTORS& FILTER
`WITH OLD COMBINED FACTORS
`
`
`
`
`SEND PARAMETERS OF
`
`
`INNER FINGER PATHS TO
`HANDWRITING RECOGNIZER
`
`
`
`FILTERED
` INNER FINGERS
`
`FACTOR > PENGRIP
`TOUCHING ?
`
`
`THRESH ?
`
`
`
`
`492
`
`
`
`
`
`SEND STYLUS LIFT SIGNAL TO
`HANDWRITING RECOGNIZER &
`KNUCKLE/PALM MOTION TO CURSOR
`
`FIG. 28
`
`Petitioner Samsung Ex-1032, 0029
`
`Petitioner Samsung Ex-1032, 0029
`
`

`

`Patent Application Publication
`
`Feb. 7, 2002 Sheet 29 of 45
`
`US 2002/0015024 Al
`
`450
`
` CONTACT
`Y
`
`
`
`
`PROXIMITIES
`
`TABILIZED 2
`
`
`N
`
`PATH EXTENSION
`
`452
`
`RETAIN PREVIOUS
`IDENTIFICATIONS VIA
`
`DEFINE & TRANSLATE LEFT &
`RIGHT ATTRACTOR TEMPLATES
`
`453
`
`END
`
`GENERATE PARTITIONING
`CONTOURS
`
`454
`
`458
`
`460
`
`A462
`
`AGE
`RECORD PARTITION
`AS LOWEST COST
`
`END
`
`456
`
`PICK FIRST
`
`CONTOUR
`
`
`
`TENTATIVELY DIVIDE HAND
`IDENTITIES ACROSS CONTOUR
`
`TENTATIVELY ASSIGN FINGER
`IDENTITIES WITHIN EACH HAND
`
`EVALUATE BIOMECHANICAL
`COHERENCE OF PARTITION
`
`
`
`PICK NEXT
`CONTOUR
`
`464
`
`Y
`
`LOWEST SO FAR
`
`
`LAST
`
`CONTOUR
`?
`
`473
`
`Yr
`Cowes
`
`COST HAND
`
`PARTITION
`
`ATA
`ASSIGN FINAL
`
`CONTACTIDENTITIES
`WITHIN EACH HAND
`
`FIG. 29
`
`Petitioner Samsung Ex-1032, 0030
`
`Petitioner Samsung Ex-1032, 0030
`
`

`

`Patent Application Publication
`
`Feb. 7, 2002 Sheet 30 of 45
`
`US 2002/0015024 Al
`
` VERTICALSURFACEPOSITION(cm)
`
`
`
`
`
`
`
`
`
`-20
`
`-15
`
`10
`5
`0
`-5
`-10
`HORIZONTAL SURFACE POSITION (cm)
`FIG. 30C
`
`15
`
`20
`
`Petitioner Samsung Ex-1032, 0031
`
`Petitioner Samsung Ex-1032, 0031
`
`

`

`Patent Application Publication
`
`Feb. 7, 2002 Sheet 31 of 45
`
`US 2002/0015024 Al
`
`{ 0
`
`50
`0
`-—50
`HORIZONTAL VELOCITY OF RIGHT HAND CLUSTER (mm/s)
`FIG. 31A
`
`
`
`DIRECTIONFACTOR
`
`CLUTCHING
`
`2
`0
`-2
`—4
`-6
`-8
`-10
`VERTICAL POSITION OF OUTERMOST FINGER
`RELATIVE TO NEXT OUTERMOST(cm)
`FIG. 31B
`
`4
`
`wn
`wa
`Ww oc
`ar
`A ©
`Fi
`
`<r
`
`0
`-12
`
`1
`
`o
`
`5
`orHO
`teQo
`Og
`> LL
`—!
`
`xO
`
`L
`
`0
`
`20
`15
`10
`5
`HORIZONTAL SEPARATION BETWEEN PALM CONTACTS(cm)
`FIG. 31C
`
`Petitioner Samsung Ex-1032, 0032
`
`Petitioner Samsung Ex-1032, 0032
`
`

`

`Patent Application Publication
`
`Feb. 7, 2002 Sheet 32 of 45
`
`US 2002/0015024 Al
`
`-150
`
`100
`50
`0
`-50
`-100
`ANGLE BETWEEN INNERMOST AND
`NEXT INNERMOST FINGER CONTACTS(degrees)
`FIG. 32
`
`150
`
`-2
`
`6
`4
`2
`0
`ESTIMATED HORIZONTAL
`SEPARATION BETWEEN THUMBS(cm)
`FIG. 33
`
`8
`
`10
`
`3912
`
`-4
`
`0-
`
`Petitioner Samsung Ex-1032, 0033
`
`
`
`RIGHTINNER
`
`
`
`ANGLEFACTOR
`
`
`
`SEPARATIONFACTOR
`
`INTER-HAND
`
`Petitioner Samsung Ex-1032, 0033
`
`

`

`Patent Application Publication
`
`Feb. 7, 2002 Sheet 33 of 45
`
`US 2002/0015024 Al
`
`GET HAND'S CURRENT
`PATH PARAMETERS &
`ID'S
`
`SUPPRESSIVE
`FINGER VELOCITY
`FILTERING
`
`MEASURE HAND'S
`POLAR VELOCITY
`
`COMPONENTS
`COMPONENTS
`
`MEASURE HAND'S
`TRANSLATIONAL
`VELOCITY COMPONENTS
`
`MEASURE HAND'S
`DIFFERENTIAL TILT
`PRESSURE
`
`500
`
`502
`
`504
`
`506
`
`508
`
`DOWNSCALE
`WEAKER
`
`COMPONENTS
`FASTEST COMPONENT
`
`DEAD-ZONE FILTER ALL
`COMPONENTS
`BY FRACTION OF
`
`510
`
`512
`
`END
`
`FIG. 34
`
`Petitioner Samsung Ex-1032, 0034
`
`Petitioner Samsung Ex-1032, 0034
`
`

`

`Patent Application Publication
`
`Feb. 7, 2002 Sheet 34 of 45
`
`US 2002/0015024 Al
`
`204
`
`i(‘
`
`}
`
`/
`
`é
`
`mu
`
`/
`a °
`
`/
`
`/
`
`/
`
`‘
`
`;
`'
`
`/
`i
`
`205
`|
`
`be
`
`“
`
`ao
`
`“
`
`203
`
`(
`
`/
`
`i
`
`4
`
`é
`
`é1
`
`t
`\
`
`\
`
`\
`
`202
`
`0
`
`ft
`
`f
`
`|
`|
`
`1
`q
`
`i
`
`\
`
`Y a
`
`aad
`
`201-7
`}7
`
`2)
`
`206
`
`}
`
`FIG.
`
`35
`
`Petitioner Samsung Ex-1032, 0035
`
`Petitioner Samsung Ex-1032, 0035
`
`

`

`Patent Application Publication
`
`Feb. 7, 2002 Sheet 35 of 45
`
`US 2002/0015024 Al
`
`522
`
` AT LEAST
`2 FINGERS
`
`DOWN?
`
`
`524
`
`SET RADIAL AND
`ANGULAR VELOCITY
`
`TO ZERO
`
`GET CURRENT AND PREVIOUS
`POSITIONS OF INNERMOST AND
`
`OUTERMOST TOUCHING FINGERS
`INNERMOST AND OUTERMOST
`
`COMPUTE RADIAL VELOCITY FROM
`CHANGE IN SEPARATION BETWEEN
`
`
`
`COMPUTE ROTATIONAL VELOCITY
`
`
`FROM SEPARATION AND CHANGE IN
`ANGLE BETWEEN INNERMOST AND
`
`OUTERMOST
`BETWEEN THUMB AND OTHER FINGERS
`
`COMBINE WITH ROTATION AND
`SCALING ABOUTA FIXED POINT
`
`END
`
`926
`
`528
`
`530
`
`531
`
`934
`
`532
`
`CHECK FOR
`
`
`AVERAGE
`RADIAL OR
`
`
`PROXIMITY
`
`ROTATIONAL
`
`
`DROPPING ?
`DECELERATION
`
`
`
`
`
`
`
`END
`
`FIG. 36
`
`Petitioner Samsung Ex-1032, 0036
`
`Petitioner Samsung Ex-1032, 0036
`
`

`

`Patent Application Publication
`
`Feb. 7, 2002 Sheet 36 of 45
`
`US 2002/0015024 Al
`
`
`
`
`
`ACCEL RATIO =
`
`
`CURRENT SPEED/
`PAST AVERAGE
`
`
`
`552
`
`
`
` ACCEL RATIO
`
`> THRESH NEAR
`ONE ?
`
`START
`
`540
`
`INIT TRANSLATION
`WEIGHTINGS TO FINGER
`PROXIMITIES
`
`
`
`
`
`
`INCREASE
`
`544
`
`DECREASE TRANSLATION
`WEIGHTING OF
`RELATIVELY SLOW
`
`FINGERS
`546
`
`DECREASE TRANSLATION
`WEIGHTING OF CENTRAL
`FINGERS AS POLAR
`COMPONENT SPEEDS
`
`
`
`
`
`
`TRANSLATION
`
`DIRECTION CLOSE TO
`PAST AVERAGE ?
`
`
`
`
`
`548
`556
`
`
`
`COMPUTE TRANSLATION
`
`SET
`
`
`VELOCITY AS WEIGHTED
`
`AVERAGE OF FINGER
`TRANSLATION
`
`DECELERATION
`
`VELOCITIES
`“LAG
`
`
`
`CLEAR
`AVERAGE
`
`
`TRANSLATION
`PROXIMITY
`
`DECELERATION
`
`
`DROPPING ?
`FLAG
`
`
`
`
`
`UPDATE MOVING
`
`WINDOW AVERAGE OF
`TRANSLATION
`
`VELOCITIES
`
`560
`
`END
`
`FIG. 37
`
`Petitioner Samsung Ex-1032, 0037
`
`Petitioner Samsung Ex-1032, 0037
`
`

`

`Patent Application Publication
`
`Feb. 7, 2002 Sheet 37 of 45
`
`US 2002/0015024 A1
`
`SET TILT & ROLL
`COMPONENTS
`TO ZERO
`
`
`
`
`
`
`STORE CURRENT PATH
`
` PROXIMITIES
`PROXIMITIES AS
`
`TABILIZED 2
`
`REFERENCE PROXIMITIES
`
`Y
`
`COMPUTE UNWEIGHTED
`AVERAGEOF PATH POSITIONS
`
`570
`
`
`
`COMPUTE RATIOS OF CURRENT
`PROXIMITY TO REFERENCE
`PROXIMITY FOR EACH PATH
`
`572
`
`SET RATIOS LESS
`THAN ONE TO ONE
`
`074
`
`COMPUTE AVERAGE OF
`PATH POSITIONS
`WEIGHTED BY PROXIMITY
`
`RATIOS
`
`976
`
`
`
`AVERAGES
`
`COMPUTE TILT & ROLL
`COMPONENTS FROM
`DIFFERENCE VECTOR BETWEEN
`WEIGHTED AND UNWEIGHTED
`
`578
`
`END
`
`FIG. 38
`
`Petitioner Samsung Ex-1032, 0038
`
`Petitioner Samsung Ex-1032, 0038
`
`

`

`Patent Application Publication
`
`Feb. 7, 2002 Sheet 38 of 45
`
`US 2002/0015024 Al
`
`
`
`GET HAND'S CURRENT
`
`PATH PARAMETERS& ID'S
`DELETE
`
`
`
`ASSOCIATED
`
`
`KEYPRESS
`SEARCH FOR FINGER
`
`
`QUEUE
`SUBSETS PRESSED OR
`
`
`ELEMENTS
`RELEASED
`
`SIMULTANEOUSLY
`
`
`
`ANY
`SYNC
`
`
`# FINGER
`
`
`MARKER
`PRESSES
`
`
`
`RELEASES
`
`
`SYNCED
`PENDING
`
`
`YNCED > 22
`
`
`
`?
`?
`
`
`
`
`
`
`# FINGER
`
`
`YNCED > 2 2
`PRESSES
`
`
`Y
`DELETE ASSOC
`IATED
`KEY QUEUE ELEMS
`
`(A)
`
`PAUSE SENDING OF
`
`SYNC
`ASSOCIATED
`KEYPRESS QUEUE
`
`ELEMS MARKER
`
`
`
`DELETE
`
`OUCHING
`
`ASSOCIATED KEY
`
`
`
`OR HALTED TOO
`
`
`QUEUE
`
`
`
`LONG ?
`ELEMENTS
`
`FIG. 39A
`
`Petitioner Samsung Ex-1032, 0039
`
`Petitioner Samsung Ex-1032, 0039
`
`

`

`Patent Application Publication
`
`Feb. 7, 2002 Sheet 39 of 45
`
`US 2002/0015024 Al
`
`DELETE ASSOCIATED
`KEYPRESS QUEUE
`
`ELEMENTS
`
`620
`
`
`
`
`FINGERS DOWN
`
`BRIEFLY ?
`
`
`
`
` SIGNIFICANT
`
`
`LATERAL MOTION ?
`
`630
`
`LOOKUP GHORD
`FROM SYNCED
`FINGER ID'S
`
`
`
`
`
`
`
`APPEND CHORD
`TAP EVENTS TO
`COMM QUEUE
`
`FIG. 39B
`
`Petitioner Samsung Ex-1032, 0040
`
`634 632
`
`RESTING CHORD:
` HORD HAS TAP
`NO EVENTS
`GENERATED
`
`
`
`EVENTS ?
`
`
`Petitioner Samsung Ex-1032, 0040
`
`

`

`Patent Application Publication
`
`Feb. 7, 2002 Sheet 40 of 45
`
`US 2002/0015024 Al
`
`
`
`GET HAND'S
`EXTRACTED MOTION &
`
`IDENTIFIED PATHS
`
`
`
`
`CHORD
`
`
`# FINGERS
`
`SLIDE
`
`TOUCHING > 2 ?
`
`ONGOING ?
`
`
`
`
`
`SYNCED
`
`
`
`SUBSET OR
`ALL FINGERS
`
`658
`
`
`SELECT SLIDE CHORD
`
`
`FROM SYNCED SUBSET
`OR COMBINATION OF
`
`FINGERS TOUCHING
`
`
`664
`SLIDE MODE
`
` FINGER
`SUBSET
`
`LIFTED ?
`
`N
`LIFTOFF AVERAGE
`
`660
`
`DISABLE KEY &
`CHORD TAPS FOR
`THIS HAND
`
`666
`
`668
`
`DECELERATION
`
`FLAG SET ?
`
`N
`
`LEAVE
`CHORD
`
`667
`
`SET CURRENT VELOCITY
`COMPONENTSTO PRE-
`
`
`NEW
`
`SUBSET
`
`PRESSED IN
`SYNC ?
`
`
`
`
`HOLE SUBSE
`
`
`SLIDING ?
`
`673
`
`
`
`
`
`SELECT NEW SLIDE
`CHORD FROM NEW
`SYNCED SUBSET
`
`FIG. 40A
`
`674
`
`Petitioner Samsung Ex-1032, 0041
`
`Petitioner Samsung Ex-1032, 0041
`
`

`

`Patent Application Publication
`
`Feb. 7, 2002 Sheet 41 of 45
`
`US 2002/0015024 A1
`
`PICK FIRST SLICE
`DEFINED FOR SELECTED
`CHORD
`
`
`675
`
`
`APPLY SLICE'S VELOCITY
`
`GAIN FUNCTION TO
`MOTION COMPONENTS
`
`
`PROJECT
`
`
`
`LAST SLICE
`VELOCITY COMPONENTS
`FOR CHORD?
`
`
`INTO SLICE'S SPEED AND
`DIRECTION RANGE
`
`
`
`
`
`
`
`
`
`
`
`
`
`INTEGRATE PROJECTED
`VELOCITY COMPONENTS
`OVER TIME
`DISABLE FURTHER
`EVENTS FROM ONE-
`SHOTSLICE
`# UNITS OF
`
`
`
`
`
`LOOKUP SLICE'S
`
`
`RESET OTHER
`
`
`
`KEY/MOUSE/3D
`
`SLICES'
`
`EVENTS
`INTEGRATORS
`
`
`
`
`
`APPEND EVENTS W/
`REMOVE INTEGER #
`
`
`
`# MOTION UNITS
`MOTION UNITS FROM
`
`
`
`TO COMM QUEUE
`INTEGRATORS
`
`688
`
`686
`
`FIG. 40B
`
`Petitioner Samsung Ex-1032, 0042
`
`Petitioner Samsung Ex-1032, 0042
`
`

`

`Patent Application Publication
`
`Feb. 7, 2002 Sheet 42 of 45
`
`US 2002/0015024 Al
`
`RETRIEVE KEY LAYOUT
`REGIONS AND SYMBOLS
`
`700
`
`
`
`
`
`
`GET CURRENT IDENTIFIED
`PATHS FOR BOTH HANDS
`
`TRANSLATE HAND'S
`
`
`FINGERS
`KEY REGIONS BY
`
`
`
`ON SAME HAND
`MEASURED HAND
`
`
`
`
`PRESSED IN
`
`SYNC?
`OFFSETS
`
`
`
`
`FINGERS
`
`
`
`PARTIALLY
`NOTHING
`
`
`
`CLOSED ?
`TOUCHING
`SURFACE FOR
`
`AWHILE?
`
`
`
`
`
`
`
`
`
`UPDATE DISPLAYED
`
`POSITIONS OF KEY
`SYMBOLS
`PROCESS
`FINGER TAPS
`ON MORPHED KEY
`LAYOUT
`
`718
`
`
`
`ADJUST REGIONS IN
`RESET KEY
`
`
`LAYOUT
`EACH FINGER'S
`
`
`COLUMN BY FINGER
`OFFSETS TO
`
`
`ZERO
`OFFSETS
`
`FIG. 41
`
`Petitioner Samsung Ex-1032, 0043
`
`Petitioner Samsung Ex-1032, 0043
`
`

`

`Patent Application Publication
`
`Feb. 7, 2002 Sheet 43 of 45
`
`US 2002/0015024 Al
`
`
`
`
`
`GET ANY PATH RECENTLY
`CREATED BY HAND PART
`TOUCHDOWN
`
`750
`
`752
`
`PATH
`
`
`PROXIMITY
`JUST CROSSED
`KEYPRESS
`THRESH2
`
`
`
`
`754
`
`PATH
`
`
`IDENTIFIED
`AS FINGER NOT
`PALM?
`
`
`
`760
`
`FIND CLOSEST
`
`
`CREATE KEYPRESS
`
`
`APPEND KEYPRESS
`
`QUEUE ELEMENT
`
`
`
`QUEUE ELEMENT TO
`
`
`CONTAINING PATHID,
`
`
`TAIL OF FIFO
`CLOSEST KEY &
`
`
`
`
`KEYPRESS QUEUE
`PRESS TIMESTAMP
`
`
`FIG. 42
`
`Petitioner Samsung Ex-1032, 0044
`
`
`PATH'S
`HAND SLIDING/
`
`
`
`
`DEBOUNCE
`OK ?
`
`757
`
`WRITING ?
`
`
`KEY REGION
`
`Petitioner Samsung Ex-1032, 0044
`
`

`

`Patent Application Publication
`
`Feb. 7, 2002 Sheet 44 of 45
`
`US 2002/0015024 Al
`
`
`
` DELETE CURRENT
`ELEMENT FROM
`KEYPRESS QUEUE
`
`
`
`
`
`PATH INA
`
`
`
`FINGER SLID
`PATH STILL
`
`
`
`SYNCHRONIZED
`
`
`TOO FAR ?
`IDENTIFIED AS
`
`
`
`SUBSET ?
`
`
`
` TIME
`
`
`
`SINCE PRESS<
`TAP TIMEOUT.
`
`
`FINGER ?
`
`
`
`APPEND PRECEDING
`
`
`
`
`MODIFIERS &
`SKIP TO NEXT
`KEY REGION
`
`
`ELEMENT'S KEY
`
`ELEMENT IN
`
`
`
`A MODIFIER ?
`
`REGION SYMBOL TO
`
`
`HOST COMM QUEUE
`
` MOST
`
`
`
`FINGERS
`
`
`PROFILE
`
`TOUCHING
`
`
`
`IMPULSIVE
`
`
`?
`
`?
`
`
`
`FIG. 43A
`
`Petitioner Samsung Ex-1032, 0045
`
`Petitioner Samsung Ex-1032, 0045
`
`

`

`Patent Application Publication
`
`Feb. 7, 2002 Sheet 45 of 45
`
`US 2002/0015024 Al
`
`
`
`
`
`
`
`FINGER PRESS >
`TAP TIMEOUT,
`
`
`
`
`
`TYPEMATIC
`
`
`STARTED FOR
`
`
`ELEMENT?
`
`
`
`TOUCHDOWN
`MOST
`RECENT?
`
`
`
`
`ANOTHER
`FINGER
`HAND'S
`
`
`
`ASYNCHRONOUS
`
`DOWN> .5s AND
`OTHER FINGERS
`
`
`
`TAP ?
`<is?
`LIFTED > .5s ?
`
`
`
` COMPUTE REPEAT
`
`INTERVAL FROM
`
`
`CURRENT
`
`
`INITIALIZE
`FINGER PROXIMITY
`
`
`TYPEMATIC
`
`
`MODE FOR
`
`
`ELEMENT
`
`TIME
`
`
`SINCE LAST
`
`
`SEND > REPEAT
`
`INTERVAL
`
`
`?
`APPEND PRECEDING
`
`
`MODIFIERS & ELEMENT'S
`
`KEY REGION SYMBOL TO
`
`HOST COMM QUEUE
`
`UPDATE LAST TYPEMATIC
`SEND TIMESTAMP
`
`
`
`FIG. 43B
`
`Petitioner Samsung Ex-1032, 0046
`
`Petitioner Samsung Ex-1032, 0046
`
`

`

`US 2002/0015024 Al
`
`Feb. 7, 2002
`
`METHOD AND APPARATUS FOR INTEGRATING
`MANUALINPUT
`
`BACKGROUND OF THE INVENTION
`
`[0001] The present application is based upon provisional
`application Serial No. 60/072,509, filed Jan. 26, 1998.
`
`[0002] A. Field of the Invention
`
`[0003] The present invention relates generally to methods
`and apparatus for data input, and, more particularly,
`to a
`method and apparatus for integrating manual input.
`
`[0004] B. Description of the Related Art
`
`[0005] Many methods for manual input of data and com-
`mands to computers are in use today, but each is most
`efficient and easy to use for particular types of data input.
`For example, drawing tablets with pens or pucks excel at
`drafting, sketching, and quick command gestures. Handwrit-
`ing with a stylus is convenient for filling out forms which
`require signatures, special symbols, or small amounts of
`text, but handwriting is slow compared to typing and voice
`input for long documents. Mice, finger-sticks and touchpads
`excel at cursor pointing and graphical object manipulations
`such as drag and drop. Rollers, thumbwheels and trackballs
`excel at panning and scrolling. The diversity of tasks that
`many computer users encounter in a single day call for all of
`these techniques, but few users will pay for a multitude of
`input devices, and the separate devices are often incompat-
`ible in a usability and an ergonomic sense. For instance,
`drawing tablets are a must for graphics professionals, but
`switching between drawing and typing is inconvenient
`because the pen must be put down or held awkwardly
`between the fingers while typing. Thus, there is a long-felt
`needin the art for a manual input device which is cheap yet
`offers convenient integration of common manualinput tech-
`niques.
`
`[0006] Speech recognition is an exciting new technology
`which promisesto relieve some of the input burden on user
`hands. However, voice is not appropriate for inputting all
`types of data either. Currently, voice input is best-suited for
`dictation of long text documents. Until natural language
`recognition matures sufficiently that very high level voice
`commands can be understood by the computer, voice will
`have little advantage over keyboard hot-keys and mouse
`menus for command and control. Furthermore, precise
`pointing, drawing, and manipulation of graphical objects is
`difficult with voice commands, no matter how well speech is
`understood. Thus, there will always be a need in the art for
`multi-function manual
`input devices which supplement
`voice input.
`
`[0007] Ageneric manual input device which combines the
`typing, pointing, scrolling, and handwriting capabilities of
`the standard input dew ice collection must have ergonomic,
`economic, and productivity advantages which outweigh the
`unavoidable sacrifices of abandoning device specialization.
`The generic device must tightly integrate yet clearly distin-
`guish the different types of input. It should therefore appear
`modeless to the user in the sense that the user should not
`need to provide explicit mode switch signals such as but-
`tonpresses, arm relocations, or stylus pickups before switch-
`ing from one input activity to another. Epidemiological
`studies suggest that repetition and force multiply in causing
`repetitive strain injuries. Awkward postures, device activa-
`
`tion force, wasted motion, and repetition should be mini-
`mized to improve ergonomics. Furthermore, the workload
`should be spread evenly overall available muscle groupsto
`avoid repetitive strain.
`
`[0008] Repetition can be minimized by allocating to sev-
`eral graphical manipulation channels those tasks which
`require complex mouse pointer motion sequences. Common
`graphical user interface operations such as finding and
`manipulating a scroll bar or slider control are much less
`efficient than specialized finger motions which causescroll-
`ing directly, without the step of repositioning the cursor over
`an on-screen control. Preferably the graphical manipulation
`channels should be distributed amongst many finger and
`hand motion combinations to spread the workload. Touch-
`pads and mice with auxilliary scrolling controls such as the
`Cirque® Smartcat touchpad with edge scrolling, the IBM®
`ScrollPoint™ mouse with embeddedpointing stick, and the
`Roller Mouse described in U.S. Pat. No. 5,530,455 to Gillick
`et al. represent small improvements in thisarea, butstill do
`not provide enough direct manipulation channels to elimi-
`nate many often-used cursor motion sequences. Further-
`more, as S. Zhai et al. found in “Dual Stream Input for
`Pointing and Scrolling,”Proceedings of CHI ’97 Extended
`Abstracts (1997), manipulation of more than two degrees of
`freedom at a time is very difficult with these devices,
`preventing simultaneous panning, zooming and rotating.
`
`reducing excess
`[0009] Another common method for
`motion and repetition is to automatically continue pointing
`or scrolling movement signals once the user has stopped
`moving or lifts the finger. Related art methods can be
`distinguished by the conditions under which such motion
`continuation is enabled.
`In U.S. Pat. No. 4,734,685,
`Watanabe continues image panning when the distance and
`velocity of pointing device movement exceed thresholds.
`Automatic panning is stopped by moving the pointing
`device back in the opposite direction, so stopping requires
`additional precise movements. In U.S. Pat. No. 5,543,591 to
`Gillespie et al., motion continuation occurs whenthe finger
`enters an edge border region around a small
`touchpad.
`Continued motion speed is fixed and the direction corre-
`spondsto the direction from the center of the touchpadto the
`finger at the edge. Continuation mode ends whenthe finger
`leaves the border region or lifts off the pad. Disadvanta-
`geously, users sometimes pause at
`the edge of the pad
`without intending for cursor motion to continue, and the
`unexpected motion continuation becomes annoying. U.S.
`Pat. No. 5,327,161 to Logan et al. describes motion con-
`tinuation whenthe finger enters a border area as well, but in
`an alternative trackball emulation mode, motion continua-
`tion can be a function solely of lateral finger velocity and
`direction at liftoff. Motion continuation decays due to a
`friction factor or can be stopped by a subsequent touchdown
`on the surface. Disadvantageously, touch velocity atliftoff is
`not a reliable indicator of the user’s desire for motion
`
`continuation since when approaching a large target on a
`display at high speeds the user may not stop the pointer
`completely before liftoff. Thus it would be an advancein the
`art to provide a motion continuation method which does not
`become activated unexpectedly when the user
`really
`intended to stop pointer movementata target but happensto
`be on a border or happens to be movingat significant speed
`during liftoff.
`
`Petitioner Samsung Ex-1032, 0047
`
`Petitioner Samsung Ex-1032, 0047
`
`

`

`US 2002/0015024 Al
`
`Feb. 7, 2002
`
`[0010] Many attempts have been made to embed pointing
`devices in a keyboard so the hands don’t have to leave
`typing position to access the pointing device. These include
`the integrated pointing key described in U.S. Pat. No.
`5,189,403 to Franz et al.,
`the integrated pointing stick
`disclosed by J. Rutledge and T. Selker in “Force-to-Motion
`Functions for Pointing,”"Human-Computer Interaction -IN-
`TERACT ’90, pp. 701-06 (1990), and the position sensing
`keys described in U.S. Pat. No. 5,675,361 to Santilli. Nev-
`ertheless,
`the limited movement range and resolution of
`these devices leads to poorer pointing speed and accuracy
`than a mouse, and they add mechanical complexity to
`keyboard construction. Thusthere exists a need in the art for
`pointing methods with higher resolution, larger movement
`range, and more degrees of freedom yet which are easily
`accessible from typing hand positions.
`
`‘Touch screens and touchpads often distinguish
`[0011]
`pointing motions from emulated button clicks or keypresses
`by assuming very little lateral fingertip motion will occur
`during taps on the touch surface which are intended as
`clicks. Inherent in these methods is the assumption that
`tapping will usually be straight down from the suspended
`finger position, minimizing those components of finger
`motion tangential to the surface. This is a valid assumption
`if the surface is not finely divided into distinct key areas or
`if the user does a slow, “hunt and peck” visual search for
`each key before striking. For example,
`in U.S. Pat. No.
`5,543,591 to Gillespie et al., a touchpad sends all lateral
`motions to the host computer as cursor movements. How-
`ever, if the finger is lifted soon enough after touchdown to
`count as a tap and if the accumulated lateral motions are not
`excessive, any sent motions are undone and a mousebutton
`click is sent instead. This method only works for mouse
`commands such as pointing which can safely be undone, not
`for dragging or other manipulations. In U.S. Pat. No. 5,666,
`113 to Logan, taps with less than about “c lateral motion
`activate keys on a small keypad while lateral motion in
`excess of “1s activates cursor control mode. In both patents
`cursor modeis invoked by default whena finger stays on the
`surface a long time.
`
`[0012] However, fast touch typing on a surface divided
`into a large array of key regions tends to produce more
`tangential motions along the surface than related art filtering
`techniques can tolerate. Such an array contains keys in
`multiple rows and columns which may n