(19) United States
`(12) Patent Application Publication (10) Pub. No.: US 2002/0015024 A1
`Westerman et al. Feb. 7, 2002 (43) Pub. Date:
`
`
`
`US 20020015024A1
`
`(54) METHOD AND APPARATUS FOR
`INTEGRATING MANUAL INPUT
`
`(57)
`
`ABSTRACT
`
`(75)
`
`Inventors: Wayne Westerman, Wellington, MO
`(US); John G- E1135, Townsend, DE
`(US)
`
`Correspondence Address:
`CONNOLLY BOVE LODGE & HUTZ LLP
`P'(_)' BOX 2207
`Wilmington, DE 19899 (US)
`
`(73) Assignee: University of Delaware
`
`(21) Appl. No"
`(22)
`Filed:
`
`09/919’266
`Jul. 31 2001
`’
`Related US. Application Data
`
`(60) Division of application No. 09/236,513, filed on Jan.
`25, 1999. Non-provisional of provisional application
`No. 60/072,509, filed on Jan. 26, 1998.
`
`Publication Classification
`
`Int. Cl.7 ....................................................... G09G 5/00
`(51)
`(52) US. Cl.
`.............................................................. 345/173
`
`Apparatus and methods are disclosed for simultaneously
`tracking multiple finger and palm contacts as hands
`approach, touch, and slide across a proximity-sensing. com-
`pliant, and flexible multi-touch surface. The surface consists
`of compressible cushion, dielectric, electrode, and circuitry
`layers. A simple proximity transduction circuit is placed
`under each electrode to maximize signal-to-noise ratio and
`to reduce Wiring complexity. Such distributed transduction
`circuitry is economical for large surfaces When implemented
`with thin-film transistor techniques. Scanning and signal
`offset removal on an electrode array produces low-noise
`proximity images. Segmentation processing of each prox-
`imity image constructs a group of electrodes corresponding
`to each distinguishable contact and extracts shape, position
`and surface proximity features for each group. Groups in
`successive images which correspond to the same hand
`contact are linked by a persistent path tracker which also
`detects individual contact touchdown and liftoff. Combina-
`torial optimization modules associate each contact’s path
`with a particular fingertip, thumb, or palm of either hand on
`the basis of biomechanical constraints and contact features.
`
`Classification of intuitive hand configurations and motions
`enables unprecedented integration of typing, resting, point-
`ing, scrolling, 3D manipulation, and handwriting into a
`versatile, ergonomic computer input device.
`
`38
`
`
`
`Petitioner Samsung Ex-1032, 0001
`
`Petitioner Samsung Ex-1032, 0001
`
`

`

`Patent Application Publication
`
`Feb. 7, 2002 Sheet 1 of 45
`
`US 2002/0015024 A1
`
`ELECTRODE
`SCANMNG
`HARDWARE
`
`FORMAflON
`
`CAUBRKHONAND
`PROXMMTYIMAGE
`
`6
`
`8
`
` ‘IO
`CONTACT
`
`
`TRACKING AND
`IDENTIFICATION
`
`
`
`
`
`HAN D
`HNGER
`
`MOflON
`
`
`
`SYNSEfi§¥EfiJK”I
`COMPONENT
`
`
`
`EXTRACHON
`
`PENGRP
`DETECTOR
`
`12
`
`TYMNG
`
`RECOGNEER
`
`
`
`CHORD MOTION
`
`RECOGNIZER
`
`24
`
`DISPLAY
`
`22
`
`HOST
`COMPUTER
`
`SYSTEM
`
`20
`
`HOST
`COMMUNICATION
`INTERFACE
`
`
`I3K3.1
`
`Petitioner Samsung EX-1032, 0002
`
`Petitioner Samsung Ex-1032, 0002
`
`

`

`Patent Application Publication
`
`Feb. 7, 2002 Sheet 2 of 45
`
`US 2002/0015024 A1
`
`
`
`Petitioner Samsung EX-1032, 0003
`
`Petitioner Samsung Ex-1032, 0003
`
`

`

`Patent Application Publication
`
`Feb. 7, 2002 Sheet 3 of 45
`
`US 2002/0015024 A1
`
`
`
`FIG. 3A
`
`
`
`FIG. BB
`
`Petitioner Samsung EX-1032, 0004
`
`Petitioner Samsung Ex-1032, 0004
`
`

`

`Patent Application Publication
`
`Feb. 7, 2002 Sheet 4 of 45
`
`US 2002/0015024 A1
`
`38
`
`fo’o’v’v’v’v’v’v’o’fi
`iéottfvfivtvtiofioié
`
`
`
`TIME
`
`FIG. 4B
`
`Petitioner Samsung EX-1032, 0005
`
`Petitioner Samsung Ex-1032, 0005
`
`

`

`Patent Application Publication
`
`Feb. 7, 2002 Sheet 5 of 45
`
`US 2002/0015024 A1
`
`’
`
`' . ' ''Q C Q i
`
`moxtféoioitiéit
`
`
`
`46A
`
`444
`
`FIG. 5B
`
`Petitioner Samsung EX-1032, 0006
`
`Petitioner Samsung Ex-1032, 0006
`
`

`

`Patent Application Publication
`
`Feb. 7, 2002 Sheet 6 of 45
`
`US 2002/0015024 A1
`
`co
`LO
`
`m A0')
`
`l\(O
`
`(O
`(0
`
`m
`
`
`
`III-flWW.“
`VI 27!. VII fill ItrII 271'
`
`
`
`FIG.6
`
`G)
`CO
`CO
`0‘)
`V
`
`‘1'<1”‘1' <1'
`
`
`
`
`
`C3!LO
`
`34
`
`Petitioner Samsung EX-1032, 0007
`
`Petitioner Samsung Ex-1032, 0007
`
`

`

`Patent Application Publication
`
`Feb. 7, 2002 Sheet 7 of 45
`
`US 2002/0015024 A1
`
`
`
`FIG. 7A
`
`
`
`Petitioner Samsung EX-1032, 0008
`
`Petitioner Samsung Ex-1032, 0008
`
`

`

`Patent Application Publication
`
`Feb. 7, 2002 Sheet 8 of 45
`
`US 2002/0015024 A1
`
`E
`
`I\
`
`[\
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`J:
`Etc)
`:5
`
`
`°
`
`3
`II
`
`47
`
`zlnllllIlIlitrlmlzl
`lllllllllllllil
`i:1ill tr! III
`IIIIIIIIIIIIIII'
`II Illlllllllllllll
`ll--l-l-
`‘
`II-
`II-
`IIIIIIIIIIIIIII
`-- murmur-III:-
`Win
`lll--- -lll---l
`
`
`IIIIIlIlV
`IIIIII m I :1
`
`
`
`
`no
`[\
`
`9’
`
`fl.
`0’)
`
`C")
`<3.
`
`FIG. 8
`
`Petitioner Samsung EX-1032, 0009
`
`Petitioner Samsung Ex-1032, 0009
`
`

`

`Patent Application Publication
`
`Feb. 7, 2002 Sheet 9 of 45
`
`US 2002/0015024 A1
`
`9/45
`
`
`
`Petitioner Samsung EX-1032, 0010
`
`Petitioner Samsung Ex-1032, 0010
`
`

`

`Patent Application Publication
`
`Feb. 7, 2002 Sheet 10 of 45
`
`US 2002/0015024 A1
`
`85
`
`86
`
`87
`
`Petitioner Samsung Ex-1032, 001 1
`
`Petitioner Samsung Ex-1032, 0011
`
`

`

`Patent Application Publication
`
`Feb. 7, 2002 Sheet 11 of 45
`
`US 2002/0015024 A1
`
`85
`
`ll|||||||l|||||||||l||||||||||||||l||||||||||||||||IllllllllllI||||||lll|||||||||||||l|||||||||||||||
`
`90
`
`33
`
`Petitioner Samsung Ex-1032, 0012
`
`Petitioner Samsung Ex-1032, 0012
`
`

`

`Patent Application Publication
`
`Feb. 7, 2002 Sheet 12 of 45
`
`US 2002/0015024 A1
`
`Petitioner Samsung EX-1032, 0013
`
`Petitioner Samsung Ex-1032, 0013
`
`
`

`

`Patent Application Publication
`
`Feb. 7, 2002 Sheet 13 of 45
`
`US 2002/0015024 A1
`
`”I
`
`—L O
`
` —-L N
`J:-
`
`VERTICALPOSITIONONSURFACE(YAXIScm) INJ:-
`
`|
`
`|
`
`O)
`
`O
`
`2
`
`16
`14
`12
`1O
`8
`6
`4
`HORIZONTAL POSITION ON SURFACE (X AXIS cm)
`
`18
`
`FIG. 13
`
`Petitioner Samsung Ex-1032, 0014
`
`Petitioner Samsung Ex-1032, 0014
`
`

`

`Patent Application Publication
`
`Feb. 7, 2002 Sheet 14 of 45
`
`US 2002/0015024 A1
`
`14'-
`
`..L._|.O[\3
`
`CO
`
`.p. I
`
`VERTICALPOSITIONONSURFACE(YAXIScm) I
`
`m
`
`I
`
`O3
`
`0
`
`2
`
`4
`
`6
`
`8
`
`10
`
`12
`
`14
`
`16
`
`18
`
`HORIZONTAL POSITION ON SURFACE (X AXIS cm)
`
`FIG. 14
`
`Petitioner Samsung EX-1032, 0015
`
`Petitioner Samsung Ex-1032, 0015
`
`

`

`Patent Application Publication
`
`Feb. 7, 2002 Sheet 15 of 45
`
`US 2002/0015024 A1
`
`14
`
`.4._LON
`
`CD
`
`-I>~O)
`
`l\)
`
`I
`
`N
`
`I
`
`206
`
`
`VERTICALPOSITIONONSURFACE(YAXIScm) 0
`.p. |
`
`O)
`0
`2
`4
`6
`8
`10
`12
`14
`16
`18
`
`HORIZONTAL POSITION ON SURFACE (X AXIS cm)
`
`FIG. 15
`
`Petitioner Samsung Ex-1032, 0016
`
`Petitioner Samsung Ex-1032, 0016
`
`

`

`Patent Application Publication
`
`Feb. 7, 2002 Sheet 16 of 45
`
`US 2002/0015024 A1
`
`
` CURRENT
`PROXIMITY
`IMAGE
`
`240
`
`
`
`
`IMAGE
`SEGMENTATION
`
`
`
`PATHS FROM
`PARAM ETERIZED
`
`PREVIOUS IMAGES
`LECTRODE GROUP
`
`
`
`
`
`
`CONTACT PATH
`TRACKING
`
`
`
`
`NEW PATHS &
`UPDATED PATH
`PARAMETERS
`
`247
`
`HAND
`IDENTIFICATION
`
`
`FINGER & PALM
`
`IDENTIFICATION
`
`
`IDENTIFIED
`
`CONTACT PATHS
`
`
`
`
`HAND POSITION
`ESTIMATION
`
`
`
`ESTIMATED HAND &
`FINGER OFFSETS
`
`FIG. 16
`
`Petitioner Samsung EX-1032, 0017
`
`Petitioner Samsung Ex-1032, 0017
`
`

`

`Patent Application Publication
`
`Feb. 7, 2002 Sheet 17 of 45
`
`US 2002/0015024 A1
`
`GET HAND'S
`
`250
`
`IDENTIFIED PATHS
`
`COMPUTE OFFSETS BETWEEN
`EACH FINGER'S MEASURED AND
`
`DEFAULT POSITIONS
`
`COMPUTE AVERAGE OF
`OFFSETS WEIGHTED BY
`
`CONTACT PROXIMITY
`CONTACT VELOCITIES
`
`ADJUST FILTER POLE TO
`
`CURRENT IDENTIFICATION
`
`CONFIDENCE
`
`COMPUTE WEIGHTED
`AVERAGE OF HAND
`
`254
`
`255
`
`256
`
`257
`
`AUTOREGRESSIVELY UPDATE HAND
`OFFSET ESTIMATES FROM MEASURED
`
`
`
`OFFSETS AND VELOCITIES
`
`258
`
`UPDATE FINGER
`OFFSET ESTIMATES
`
`259
`
`CONVERT ESTIMATED OFFSETS
`TO ABSOLUTE POSITIONS
`
`260
`
`END
`
`FIG. 17
`
`Petitioner Samsung Ex-1032, 0018
`
`Petitioner Samsung Ex-1032, 0018
`
`

`

`Patent Application Publication
`
`Feb. 7, 2002 Sheet 18 of 45
`
`US 2002/0015024 A1
`
`240
`
` CURRENT
`
`PROXIMITY
`
`DIFFUSE
`
`CURRENT IMAGE
`
`IMAGE
`
`
`
`
`
`
`SMOOTHED
`PROXIMITY
`
`IMAGE
`
`
`
`
`
`
`
`LOCAL
`MAXI MUM
`
`PIXELS
`
`SEARCH FOR
`
`SIGNIFICANT
`
`LOCAL MAXIMA
`
`267
`
`FLATTENED
`
`
`FINGERTIP
`FEEDBACK
`
`
`252
`
`
`
`ESTIMATED HAND
`POSITION OFFSETS
`
`
`DEFINE SEGMENTATION
`
`STRICTNESS REGIONS
`
`\— 266
`
`
`
`
`
`CONSTRUCT ELECTRODE
`GROUPS AROUND LOCAL
`
`
`MAXIMUM PIXELS
`
`
`COMBINE
`
`OVERLAPPING
`
`GROUPS
`GROUPS
`
`FIT ELLIPSES TO
`COMBINED
`
`270
`
`272
`
`
`
`
`PARAM ETERIZED
`ELECTRODE GROUPS
`
`
`242
`
`FIG. 18
`
`Petitioner Samsung EX-1032, 0019
`
`Petitioner Samsung Ex-1032, 0019
`
`

`

`Patent Application Publication
`
`Feb. 7, 2002 Sheet 19 of 45
`
`US 2002/0015024 A1
`
`281
`
`278
`
`277
`
`279
`
`281
`
`281
`
`FIG. 19
`
`Petitioner Samsung EX-1032, 0020
`
`Petitioner Samsung Ex-1032, 0020
`
`

`

`Patent Application Publication
`
`Feb. 7, 2002 Sheet 20 of 45
`
`US 2002/0015024 A1
`
`
`
`
`
`
`
`VERTICALSURFACEPOSITION(cm)
`
`
`10 I
`
`
`,'\,O10
`
`l
`
`.—L m
`
`I
`
`—.L o
`
`I
`
`10
`
`15
`
`20
`
`—2o
`
`—15
`
`—10
`
`—5
`
`o
`
`5
`
`10
`
`15
`
`20
`
`HORIZONTAL SURFACE POSITION (cm)
`FIG. 20C
`
`Petitioner Samsung Ex-1032, 0021
`
`Petitioner Samsung Ex-1032, 0021
`
`

`

`Patent Application Publication
`
`Feb. 7, 2002 Sheet 21 of 45
`
`US 2002/0015024 A1
`
`290
`RAW PROXIMITY >
`
`BACKGROUND?
`
`
`N
`
`
`
`300
`
`GET NEXT ELECTRODE IN
`DIRECTION OF SEARCH
`
`REACHED BACKGROUND
`LEVEL EDGE
`
`304
`
`
`
`SEGMENTATION
`HOE/$53369
`
`
`
`
`
`REGION?
`-
`
`
`N
`
`0
`
`N
`
`
`
`SEARCHING
`HORIZONTAL '2
`
`
`
`296
`
`310
`
`Y
`
`312
`
`REACHED EDGE
`
`
`BETWEEN
`
`
`FINGERTIP AND
`THUMB OR PALM
`
`
`END
`
`
`
`DIST. TO LOCAL MAX
`HORIz.
`OR DIAGONAL
`
`MINIMUM '2
`
`
`
`Y
`
`
`
`300
`
`314
`
`REACHED EDGE
`BETWEEN
`FINGERS
`
`END
`
`298
`
`
`
`TALL
`HORIZONTAL
`MINIMUM 7
`
`REACHED EDGE
`BETWEEN PALM
`HEELS
`
`END
`
`FIG. 21
`
`Petitioner Samsung Ex-1032, 0022
`
`Petitioner Samsung Ex-1032, 0022
`
`

`

`Patent Application Publication
`
`Feb. 7, 2002 Sheet 22 of 45
`
`US 2002/0015024 A1
`
`START
`
`PREDICT CURRENT POSITIONS
`
`320
`
`OF EXISTING PATHS
`
`FOR EACH GROUP
`
`322
`
`FIND CLOSEST PATH
`
`FOR EACH PATH, FIND
`CLOSEST GROUP WITHIN
`
`TRACKING RADIUS
`
`324
`
`FORM GROUP—PATH PAIRS IF
`GROUP & ACTIVE PATH ARE
`
`CLOSEST TO ONE ANOTHER
`
`ATTEMPT TO PAIR REMAINING
`GROUPS WITH RECENTLY
`
`DEACTIVATED PATHS
`
`326
`
`334
`
`ALLOCATE NEW PATHS FOR ANY
`
`335
`
`REMAINING UNPAIRED GROUPS
`
`DEACTIVATE ANY REMAINING
`
`344
`
`UNPAIRED PATHS
`
`UPDATE PATH
`
`343
`
`PARAMETERS
`
`FIG. 22
`
`Petitioner Samsung Ex-1032, 0023
`
`Petitioner Samsung Ex-1032, 0023
`
`

`

`Patent Application Publication
`
`Feb. 7, 2002 Sheet 23 of 45
`
`US 2002/0015024 A1
`
`DEFINE IDENTITY
`ATTRACTORS AT DEFAULT
`
`TRANSLATE ATTRACTOR
`TEMPLATE BY ESTIMATED
`
`COMPUTE MATRIX OF
`DISTANCES FROM EACH PATH
`
`CONTACT POSITIONS
`HAND OFFSET
`TO EACH ATTRACTOR
`FEATURES OF EACH PATH
`
`COMPUTE ATTRACTOR
`WEIGHTING FACTORS FROM
`
`FIND ASSIGNMENT BETWEEN
`PATHS AND ATTRACTORS
`
`
`
`
`
`WHICH MINIMIZES SUM OF
`WEIGHTED DISTANCES
`
`
`350
`
`352
`
`354
`
`356
`
`358
`
`362
`
`
`
`#FWGER
`
`HAND
`
`ATTRACTORS
`
`ASSIGNMENTS
`
`
`ASEGNED>1
`
`ENTATIVE 7
`
`
`OR<5?
`
`VERIFY THUMB
`
`ASSIGNMENT
`
`368
`
`
`UPDATE FINGER
`COUNTS AND
`
`SUBSETS
`
`
`END
`
`FIG. 23
`
`Petitioner Samsung Ex-1032, 0024
`
`Petitioner Samsung Ex-1032, 0024
`
`

`

`Patent Application Publication
`
`Feb. 7, 2002 Sheet 24 of 45
`
`US 2002/0015024 A1
`
`12
`
`—L o
`
`«B0)OD
`
`(cm) N
`
`
`VERTICALSURFACEPOSITION
`
`o
`
`2
`
`4
`
`6
`
`8
`
`10
`
`12
`
`14
`
`16
`
`18
`
`20
`
`HORIZONTAL SURFACE POSITION (cm)
`
`FIG. 24
`
`Petitioner Samsung EX-1032, 0025
`
`Petitioner Samsung Ex-1032, 0025
`
`

`

`Patent Application Publication
`
`Feb. 7, 2002 Sheet 25 of 45
`
`US 2002/0015024 A1
`
`RIGHT
`
`ORIENTATION
`
`FACTORI
`
`0
`
`150
`100
`50
`CONTACT ORIENTATION (degrees)
`FIG. 25A
`
`SIZE FACTORI
`THUMB
`
`O
`
`5
`4
`3
`2
`‘I
`CONTACT SIZE (NORMALIZED TOTAL PROXIMITY)
`FIG. 25B
`
`6
`
`5
`4
`3
`2
`I
`TOTAL PROXIMITY DIVIDED BY ECCENTRICITY
`
`6
`
`FIG. 25C
`
`O
`
`1 0
`
`PALMHEEL
`
`
`
`SIZEFACTOR
`
`PALMSEPARATION
`
`FACTORI
`
`O
`
`6
`4
`2
`DISTANCE TO NEAREST NEIGHBOR CONTACT (Gm)
`FIG. 25D
`
`8
`
`Petitioner Samsung Ex-1032, 0026
`
`Petitioner Samsung Ex-1032, 0026
`
`

`

`Patent Application Publication
`
`Feb. 7, 2002 Sheet 26 of 45
`
`US 2002/0015024 A1
`
`COMPUTE INTER-PATH
`
`THUMB FACTORS
`
`400
`
`COMBINE WITH THUMB SIZE
`
`& ORIENTATION FACTORS
`
`OF INNERMOST AND NEXT
`
`INNERMOST CONTACT
`
`402
`
`
`
`
`
`
`combined_thumb_fact >
`
`ASSIGNED TO
`is_thumb_thresh?
`
`
`THUMB?
`
`
`
`
`ATTRACTOR
`
`
`
`THUMB?
`
`
`SHIFT INNERMOST
`PATHS AWAY FROM
`
`THUMB ATTRACTOR
`
`410
`
`EXISTING
`
`OK
`
`ASSIGNMENTS
`
`
`
`END
`
`FIG. 26
`
`Petitioner Samsung Ex-1032, 0027
`
`combined_thumb_fact
`not_thumb_thresh?
`
`
`
`INNERMOST
`
`
`
`ASSIGNED TO
`
`SHIFT INNERMOST
`
`PATH TO THUMB
`
`414
`
`413
`
`Petitioner Samsung Ex-1032, 0027
`
`

`

`Patent Application Publication
`
`Feb. 7, 2002 Sheet 27 of 45
`
`US 2002/0015024 A1
`
`GET ALL PATHS ASSIGNED
`TO THE GIVEN HAND
`
`430
`
`COMPUTE DISTANCES
`FROM EACH PATH TO
`
`432
`
`OTHER PATHS
`434
`
`FIND SHORTEST RING
`
`CONNECTING ALL PATHS
`AND PASSING THROUGH
`EACH ONCE
`
`COMPUTE THUMB & PALM
`WEIGHTING FACTORS FOR
`
`EACH PATH
`
`436
`
`PICK INNERMOST
`PATH IN RING
`
`438
`
`440
`
`
`
`INNERMOST
`
`
`
`INNERMOST
`
`PATH A PALM
`
`
`PATH THUMB ?
`
`
`HEEL?
`
`442
`
`
`
`
`
`PATHS ABOVE THIS
`
`
`PATHS AT THIS
`PATHS AT THIS
`
`VERTICAL LEVEL ARE
`
`VERTICAL LEVEL
`VERTICAL LEVEL
`
`
`
`
`FINGERTIPS, BELOW
`ARE FINGERTIPS
`ARE PALMS
`
`
`
`
`ARE PALMS
`
`444
`
`END
`
`FIG. 27
`
`Petitioner Samsung EX-1032, 0028
`
`Petitioner Samsung Ex-1032, 0028
`
`

`

`Patent Application Publication
`
`Feb. 7, 2002 Sheet 28 of 45
`
`US 2002/0015024 A1
`
`486
`
`PENGRIP
`
`& OUTER HAND
`
`
`PREVIOUSLY
`PARTS TOUCHING
`
`DETECTED ?
`
`
`
`GET ESTIMATED FINGER &
`
`PALM POSITIONS FOR
`LIFTED HAND PARTS
`
`SIZES OF TOUCHING FINGERS & PALM HEELS
`
`GET MEASURED POSITIONS &
`
`488
`
`END
`
`487
`
`COMPUTE KNUCKLE FACTOR
`FOR OUTER FINGERS
`
`COMPUTE INDEX JUTTING
`
`FACTOR FOR INNER FINGERS
`
`COMBINE FACTORS & FILTER
`
`489
`
`490
`
`
`
`
`
`SEND PARAMETERS OF
`INNER FINGER PATHS TO
`
`HANDWRITING RECOGNIZER
`
`
`
`WITH OLD COMBINED FACTORS
`
`491
`
`
`
`
`492
`
`493
`
`FILTERED
`
`INNER FINGERS
`
`FACTOR > PENGRIP
`
`TOUCHING ?
`
`THRESH ?
`
`
`
`
`
`SEND STYLUS LIFT SIGNAL TO
`HANDWRITING RECOGNIZER &
`KNUCKLE/PALM MOTION TO CURSOR
`
`FIG. 28
`
`Petitioner Samsung Ex-1032, 0029
`
`Petitioner Samsung Ex-1032, 0029
`
`

`

`Patent Application Publication
`
`Feb. 7, 2002 Sheet 29 of 45
`
`US 2002/0015024 A1
`
`450
`
`
` CONTACT
`
`Y
`PROXIMITIES
`
`
`TABILIZED 7
`
`
`N
`
`452
`
`RETAIN PREVIOUS
`IDENTIFICATIONS VIA
`
`PATH EXTENSION
`
`DEFINE & TRANSLATE LEFT &
`RIGHT ATTRACTOR TEMPLATES
`
`453
`
`END
`
`456
`
`PICK FIRST
`
`GENERATE PARTITIONING
`
`CONTOUR
`
`CONTOURS
`
`
`
`
`TENTATIVELY DIVIDE HAND
`IDENTITIES ACROSS CONTOUR
`
`
`
`IDENTITIES WITHIN EACH HAND
`
`
`
`EVALUATE BIOMECHANICAL
`
`
`TENTATIVELY ASSIGN FINGER
`
`COHERENCE OF PARTITION
`
`
`
`464
`
`
`
`LOWEST SO FAR
`
`Y
`
`
`
`PICK NEXT
`CONTOUR
`
`LAST
`
`CONTOUR
`?
`
`Y
`
`4
`
`73
`
`474
`
`CHOOSE
`LOWEST
`
`COST HAND
`
`PARTITION
`
`ASSIGN FINAL
`
`CONTACT IDENTITIES
`
`WITHIN EACH HAND
`
`454
`
`458
`
`460
`
`452
`
`466
`RECORD PARTITION
`AS LOWEST COST
`
`END
`
`FIG. 29
`
`Petitioner Samsung Ex-1032, 0030
`
`Petitioner Samsung Ex-1032, 0030
`
`

`

`Patent Application Publication
`
`Feb. 7, 2002 Sheet 30 of 45
`
`US 2002/0015024 A1
`
` 5
`
`15
`
`20
`
`1O
`
`
`
`
`
`
`
`VERTICALSURFACEPOSITION(cm)
`
`01
`
`....L O
`
`5
`
`1O
`
`15
`
`20
`
`
`
`—20
`
`~15
`
`1O
`5
`O
`—5
`—1O
`HORIZONTAL SURFACE POSITION (cm)
`FIG. 300
`
`15
`
`20
`
`Petitioner Samsung Ex-1032, 0031
`
`Petitioner Samsung Ex-1032, 0031
`
`

`

`Patent Application Publication
`
`Feb. 7, 2002 Sheet 31 of 45
`
`US 2002/0015024 A1
`
`1 O—
`
`50
`0
`50
`HORIZONTAL VELOCITY OF RIGHT HAND CLUSTER (mm/s)
`
`FIG. 31A
`
`CLUTCHING
`
`
`
`DIRECTIONFACTOR
`
`1
`
`co
`co
`% n:
`o E
`E O
`z E
`
`<I
`
`
`
`0
`—1 2
`
`2
`O
`—2
`—4
`—6
`—8
`—10
`VERTICAL POSITION OF OUTERMOST FINGER
`RELATIVE TO NEXT OUTERMOST (cm)
`
`FIG. 315
`
`1
`
`(22
`
`0')
`CC
`% O
`O B
`0 <
`2 L
`_I
`<12
`0.
`
`o
`
`20
`15
`10
`5
`0
`HORIZONTAL SEPARATION BETWEEN PALM CONTACTS (cm)
`FIG. 310
`
`Petitioner Samsung Ex-1032, 0032
`
`Petitioner Samsung Ex-1032, 0032
`
`

`

`Patent Application Publication
`
`Feb. 7, 2002 Sheet 32 of 45
`
`US 2002/0015024 A1
`
`
`
`RIGHTINNER
`
`
`
`ANGLEFACTOR
`
`INTER—HAND
`
`
`
`SEPARATIONFACTOR
`
`100
`50
`0
`—50
`—150 —100
`ANGLE BETWEEN INNERMOST AND
`
`150
`
`NEXT INNERMOST FINGER CONTACTS (degrees)
`FIG. 32
`
`-—6—4—2
`
`0
`
`2
`
`4
`
`6 81012
`
`ESTIMATED HORIZONTAL
`
`SEPARATION BETWEEN THUMBS (Om)
`FIG. 33
`
`Petitioner Samsung Ex-1032, 0033
`
`Petitioner Samsung Ex-1032, 0033
`
`

`

`Patent Application Publication
`
`Feb. 7, 2002 Sheet 33 0f 45
`
`US 2002/0015024 A1
`
`GET HAND'S CURRENT
`
`PATH PARAMETERS &
`
`ID'S
`
`SUPPRESSIVE
`
`FINGER VELOCITY
`
`FILTERING
`
`MEASURE HAND'S
`
`POLAR VELOCITY
`
`COMPONENTS
`
`
`MEASURE HAND'S
`
`TRANSLATIONAL
`
`VELOCITY COMPONENTS
`
`MEASURE HAND'S
`
`DIFFERENTIAL TILT
`
`PRESSURE
`
`COMPONENTS
`
`500
`
`502
`
`504
`
`506
`
`508
`
`DOWNSCALE
`
`COMPONENTS
`
`WEAKER
`
`DEAD—ZONE FILTER ALL
`
`FASTEST COMPONENT
`
`COMPONENTS
`
`BY FRACTION OF
`
`510
`
`512
`
`END
`
`FIG. 34
`
`Petitioner Samsung Ex-1032, 0034
`
`Petitioner Samsung Ex-1032, 0034
`
`

`

`Patent Application Publication
`
`Feb. 7, 2002 Sheet 34 of 45
`
`US 2002/0015024 A1
`
`203
`
`207
`
`206
`
`FIG. 35
`
`Petitioner Samsung EX-1032, 0035
`
`Petitioner Samsung Ex-1032, 0035
`
`

`

`Patent Application Publication
`
`Feb. 7, 2002 Sheet 35 of 45
`
`US 2002/0015024 A1
`
`522
`
`
` AT LEAST
`
`2 FINGERS
`
`DOWN ?
`
`
`
`N
`
`524
`
`SET RADIAL AND
`ANGULAR VELOCITY
`
`TO ZERO
`
`GET CURRENT AND PREVIOUS
`POSITIONS OF INNERMOST AND
`
`526
`
`OUTERMOST TOUCHING FINGERS
`
`END
`
`COMPUTE RADIAL VELOCITY FROM
`
`CHANGE IN SEPARATION BETWEEN
`
`528
`
`INNERMOST AND OUTERMOST
`
`
`
`COMPUTE ROTATIONAL VELOCITY
`FROM SEPARATION AND CHANGE IN
`
`
`ANGLE BETWEEN INNERMOST AND
`OUTERMOST
`
`
`530
`
`COMBINE WITH ROTATION AND
`
`SCALING ABOUT A FIXED POINT
`
`531
`
`BETWEEN THUMB AND OTHER FINGERS
`
`532
`
`
`CHECK FOR
`
`
`AVE RAGE
`PROXIMITY
`RADIAL OR
`
`
`
`DROPPING ,2
`ROTATIONAL
`
`
`'
`DECELERATION
`
`534
`
`
`
`
`
`
`
`END
`
`FIG. 36
`
`Petitioner Samsung Ex-1032, 0036
`
`Petitioner Samsung Ex-1032, 0036
`
`

`

`Patent Application Publication
`
`Feb. 7, 2002 Sheet 36 of 45
`
`US 2002/0015024 A1
`
`540
`
`INIT TRANSLATION
`
`WEIGHTINGS TO FINGER
`
`PROXIMITIES
`
`544
`
`DECREASE TRANSLATION
`WEIGHTING OF
`
`
`
`RELATIVELY SLOW
`FINGERS
`
`546
`
`
`
`
`
`
` ACCEL RATIO =
`CURRENT SPEED/
`
`
`PAST AVERAGE
`
`
`
`552
`
`
`
` ACCEL RATIO
`> THRESH NEAR
`
`ONE ?
`
`
`
`DECREASE TRANSLATION
`WEIGHTING OF CENTRAL
`
`INCREASE
`
`FINGERS AS POLAR
`COMPONENT SPEEDS
`
`
`
`
`TRANSLATION
`
`DIRECTION CLOSE TO
`
`'AST AVERAGE ?
`
`
`
`COMPUTE TRANSLATION
`SET
`VELOCITY AS WEIGHTED
`
`
`TRANSLATION
`
`AVERAGE OF FINGER
`DECELERATION
`
`VELOCITIES
`
`
`FLAG
`
` CLEAR
`
`AVERAGE
`TRANSLATION
`
`
`PROXIMITY
`DECELERATION
`
`DROPPING ?
`
`FLAG
`
`548
`
`556
`
`
`
`
`
`
`
`
`
`
`UPDATE MOVING
`WINDOW AVERAGE OF
`
`TRANSLATION
`VELOCITIES
`
`
`560
`
`END
`
`FIG. 37
`
`Petitioner Samsung EX-1032, 0037
`
`Petitioner Samsung Ex-1032, 0037
`
`

`

`Patent Application Publication
`
`Feb. 7, 2002 Sheet 37 of 45
`
`US 2002/0015024 A1
`
`END
`
`564
`
`562
`
`SET TILT & ROLL
`
`
`HAND
`
`
`FLATTENED ?
`
`
`TO ZERO
`
`
`COMPONENTS
`
`
`
`
`
`
`ALL PATH
`PROXIMITIES
`
`
`
`
`
`
`
`TABILIZED '7
`
`REFERENCE PROXIMITIES
`
`STORE CURRENT PATH
`
`PROXIMITIES AS
`
`
`Y
`
`COMPUTE UNWEIGHTED
`AVERAGE OF PATH POSITIONS
`
`570
`
`COMPUTE RATIOS OF CURRENT
`PROXIMITY TO REFERENCE
`
`
`
`PROXIMITY FOR EACH PATH
`
`SET RATIOS LESS
`THAN ONE TO ONE
`
`574
`
`
`
`COMPUTE AVERAGE OF
`
`PATH POSITIONS
`
`576
`
`
`
`WEIGHTED BY PROXIMITY
`RATIOS
`
`
`COMPUTE TILT & ROLL
`
`COMPONENTS FROM
`DIFFERENCE VECTOR BETWEEN
`
`WEIGHTED AND UNWEIGHTED
`
`AVERAGES
`
`572
`
`578
`
`END
`
`FIG. 38
`
`Petitioner Samsung Ex-1032, 0038
`
`Petitioner Samsung Ex-1032, 0038
`
`

`

`
`
`RELEASED
`SIMULTANEOUSLY
`
`Patent Application Publication
`
`Feb. 7, 2002 Sheet 38 of 45
`
`US 2002/0015024 A1
`
`
`
`SUBSETS PRESSED OR
`
`KEYPRESS
`QUEUE
`
`
`
`GET HAND‘S CURRENT
`PATH PARAMETERS & ID'S
`
`DELETE
`
`ASSOCIATED
`
`
`SEARCH FOR FINGER
`
`
`
`ELEMENTS
`
`
`
`
`
`
`
`ANY
`SYNC
`
`
`# FINGER
`MARKER
`PRESSES
`
`
`RELEASES
`
`
`
`SYNCED
`PENDING
`
`
`YNCED > 2’?
`
`
`?
`?
`
`
`
`
`
`# FINGER
`PRESSES
`
`
`YNCED > 2 7
`
`
`Y
`
`
`
`DELETE ASSOC
`'ATED
`KEY QUEUE ELEMS
`
`0
`
`PAUSE SENDING OF
`
`08$:le
`ASSOCIATED
`KEYPRESS QUEUE
`
`ELEMS MARKER
`
`DELETE
`
`
`
`OUCHING
`ASSOCIATED KEY
`
`
`OR HALTED TOO
`
`
`QUEUE
`
`
`LONG ?
`
`ELEMENTS
`
`F/G. 39A
`
`Petitioner Samsung Ex-1032, 0039
`
`Petitioner Samsung Ex-1032, 0039
`
`

`

`Patent Application Publication
`
`Feb. 7, 2002 Sheet 39 of 45
`
`US 2002/0015024 A1
`
`DELETE ASSOCIATED
`
`ELEMENTS
`
`KEYPRESS QUEUE
`
`620
`
`626
` SYNCED
`
`FINGERS DOWN
`
`
`BRIEFLY ?
`
`
` SIGNIFICANT
`
`
`LATERAL MOTION ?
`
`
`LOOKUP CHORD
`
`FROM SYNCED
`
`630
`
`FINGER ID'S
`
`634
`632
`
`
`RESTING CHORD:
` HORD HAS TA'
`
`NO EVENTS
`
`
`
`EVENTS ?
`
`
`
`
`GENERATED
`
`
`APPEND CHORD
`TAP EVENTS TO
`
`COMM QUEUE
`
`
`
`FIG. 398
`
`Petitioner Samsung Ex-1032, 0040
`
`Petitioner Samsung Ex-1032, 0040
`
`

`

`Patent Application Publication
`
`Feb. 7, 2002 Sheet 40 of 45
`
`US 2002/0015024 A1
`
`
`GET HAND‘S
`
`EXTRACTED MOTION &
`
`IDENTIFIED PATHS
`
`
`
`
`
`
`SLIDE
`
`
`CHORD
`
`
`# FINGERS
`
`TOUCHING > 2 ?
`
`
`ONGOING ?
`
`
`
`
`
`
`SYNCED
`SUBSET OR
`
`
`ALL FINGERS
`LIDING ?
`
`
`
`660
`
`DISABLE KEY 8:
`
`CHORD TAPS FOR
`THIS HAND
`
`
`
`
`
`
`558
`SELECT SLIDE CHORD
`
`FROM SYNCED SUBSET
`OR COMBINATION OF
`
`FINGERS TOUCHING
`
`
`
`
`
`
`
`FINGER
`SUBSET
`
`LIFTED ?
`
`
`N
`
`666
`
`
`
`
`PRE—LIFTOF
`DECELERATION
`
`
`
`FLAG SET ?
`
`
`N
`
`668
`
`LEAVE
`CHORD
`
`SLIDE MODE
`
`667
`
`SET CURRENT VELOCITY
`COMPONENTS TO PRE-
`
`LIFTOFF AVERAGE
`
`NEW
`
`
`SUBSET
`
`PRESSED IN
`
`SYNC ?
`
`
`
`
`
`673
`
`
`HOLE SUBSE
`SLIDING ?
`
`
`
`
`SELECT NEW SLIDE
`CHORD FROM NEW
`SYNCED SUBSET
`
`
`FIG. 40A
`
`674
`
`Petitioner Samsung Ex-1032, 0041
`
`Petitioner Samsung Ex-1032, 0041
`
`

`

`Patent Application Publication
`
`Feb. 7, 2002 Sheet 41 of 45
`
`US 2002/0015024 A1
`
`
`
`
`
`
`675
`
`
`
`PICK FIRST SLICE
`
`DEFINED FOR SELECTED
`CHORD
`
`
`
`APPLY SLICE'S VELOCITY
`GAIN FUNCTION TO
`
`MOTION COMPONENTS
`
`
`
`
`
`PROJECT
`
`
`
`
`LAST SLICE
`
`VELOCITY COMPONENTS
`FOR CHORD ?
`
`
`
`INTO SLICE'S SPEED AND
`DIRECTION RANGE
`
`
`
`INTEGRATE PROJECTED
`
`
`
`VELOCITY COMPONENTS
`OVER TIME
`DISABLE FURTHER
`EVENTS FROM ONE—
`SHOT SLICE
`# UNITS OF
`MOTION
`>= 1 ?
`
`
`
`
`
`
`
`LOOKUP SLICE'S
`KEY/MOUSE/3D
`
`
`
`
`EVENTS
`
`
`
`
`RESET OTHER
`
`
`SLICES'
`
`
`INTEGRATORS
`
`
`688
`
`
`
`
`
`APPEND EVENTS W/
`REMOVE INTEGER #
`# MOTION UNITS
`MOTION UNITS FROM
`
`
`
`TO COMM QUEUE
`INTEGRATORS
`
`
`
`
`686
`
`FIG. 403
`
`Petitioner Samsung Ex-1032, 0042
`
`Petitioner Samsung Ex-1032, 0042
`
`

`

`Patent Application Publication
`
`Feb. 7, 2002 Sheet 42 of 45
`
`US 2002/0015024 A1
`
`
`
`RETRIEVE KEY LAYOUT
`REGIONS AND SYMBOLS
`
`700
`
`
`
`
`
`GET CURRENT IDENTIFIED
`
`PATHS FOR BOTH HANDS
`
`
`
`TRANSLATE HAN D'S
`
`KEY REGIONS BY
`FINGERS
`
`
`ON SAME HAND
`MEASURED HAND
`
`
`
`PRESSEDIN
`
`
`
`SYNC?
`OFFSETS
`
`
`
`FINGERS
`
`PARTIALLY
`
`NOTHING
`
`CLOSED ?
`TOUCHING
`
`
`
`SURFACE FOR
`AWHILE?
`
`
`
`
`RESET KEY
`LAYOUT
`
`OFFSETS TO
`
`ZERO
`
`
`
`ADJUST REGIONS IN
`EACH FINGER‘S
`
`COLUMN BY FINGER
`
`OFFSETS
`
`
`
`
`
`
`
`UPDATE DISPLAYED
`POSITIONS OF KEY
`
`SYMBOLS
`PROCESS
`
`FINGER TAPS
`ON MORPHED KEY
`
`
`LAYOUT
`
`
`718
`
`FIG. 41
`
`Petitioner Samsung Ex-1032, 0043
`
`Petitioner Samsung Ex-1032, 0043
`
`

`

`Patent Application Publication
`
`Feb. 7, 2002 Sheet 43 of 45
`
`US 2002/0015024 A1
`
`
`
`GET ANY PATH RECENTLY
`CREATED BY HAND PART
`TOUCHDOWN
`
`
`
`
`
`752
`PATH
`
`
`PROXIMITY
`
`
`JUSTCROSSED
`
`750
`
`KEYPRESS
`THRESH?
`
`
`
`754
`
`
`PATH
`IDENTIFIED
`
`
`AS FINGER NOT
`PALM?
`
`
`
`760
`
`
`PATHS
`DEBOUNCE
`HNDCLOSEST
`
`
`HANDSUDWG/
`KEYREQON
`0K7
`
`WRWWG?
`‘
`
`
`
` REGION
`EARBY ?
`
`
`
`
`CREATE KEYPRESS
`
`APPEND KEYPRESS
`
`
`QUEUE ELEMENT
`
`
`QUEUE ELEMENT TO
`
`
`
`CONTAINING PATH ID,
`TAIL OF FIFO
`
`
`CLOSEST KEY &
`
`
`KEYPRESS QUEUE
`
`
`PRESS TIMESTAMP
`
`
`FIG. 42
`
`Petitioner Samsung Ex-1032, 0044
`
`Petitioner Samsung Ex-1032, 0044
`
`

`

`Patent Application Publication
`
`Feb. 7, 2002 Sheet 44 of 45
`
`US 2002/0015024 A1
`
`
`DELETE CURRENT
`
`ELEMENT FROM
`
`
`KEYPRESS QUEUE
`
`
`
`
`PICK ELEMENT AT HEAD
`
`OF KEYPRESS QUEUE
`
`
`
`
`
`PATH IN A
`
`
`
`FINGER SLID
`PATH STILL
`
`
`
`SYNCHRONIZED
`
`
`
`TOO FAR ?
`IDENTIFIED AS
`
`
`SUBSET ?
`
`
`
`FINGER ?
`
`
`
`
` TIME
`SINCE PRESS <
`
`TAP TIMEOUT
`
`
`
`APPEND PRECEDING
`
`
`
`MODIFIERS &
`
`
`SKIP TO NEXT
`KEY REGION
`
`ELEMENT'S KEY
`
`ELEMENT IN
`
`
`
`A MODIFIER ?
`
`REGION SYMBOL TO
`
`HOST COMM QUEUE
`
`
`
`
`PATH
`
`
`
`PRomeTY
`MOST
`
`
`HNGERS
`
`
`
`
`PROFKE
`TOUCHWG
`
`
`IMPULSVE ?
`
`
`:2
`
`
`
`FWC3.4Eb4
`
`Petitioner Samsung Ex-1032, 0045
`
`Petitioner Samsung Ex-1032, 0045
`
`

`

`Patent Application Publication
`
`Feb. 7, 2002 Sheet 45 of 45
`
`US 2002/0015024 A1
`
`
`
`
`TIME
`
`
`
`FINGER
`
`
`SINCE
`TYPEMATIC
`
`
`
`FINGER PRESS >
`STARTED FOR
`Toufigggw'“
`
`
`
`
`
`7
`
`
`TAP TIMEOUT
`ELEMENT .
`RECENT?
`
`
`
`
`
`ANOTHER
`FINGER
`
`
`DOWN > .53 AND
`ASYNCHRONOUS
`
`
`TAP ?
`< IS ?
`
`
`
`
`HAND‘S
`OTHER FINGERS
`
`LIFTED > .58 ?
`
`
`
`
`COMPUTE REPEAT
`INTERVAL FROM
`
`CURRENT
`
`INITIALIZE
`
`FINGER PROXIMITY
`
`
`TYPEMATIC
`MODE FOR
`
`ELEMENT
`
`
`
`TIME
`
`SINCE LAST
`
`SEND > REPEAT
`INTERVAL
`
`
`?
`
`APPEND PRECEDING
`
`MODIFIERS & ELEMENT'S
`KEY REGION SYMBOL TO
`
`HOST COMM QUEUE
`
`
`
`UPDATE LAST TYPEMATIC
`SEND TIMESTAMP
`
`FIG. 43B
`
`Petitioner Samsung Ex-1032, 0046
`
`Petitioner Samsung Ex-1032, 0046
`
`

`

`US 2002/0015024 A1
`
`Feb. 7, 2002
`
`METHOD AND APPARATUS FOR INTEGRATING
`MANUAL INPUT
`
`BACKGROUND OF THE INVENTION
`
`[0001] The present application is based upon provisional
`application Serial No. 60/072,509, filed Jan. 26, 1998.
`
`[0002] A. Field of the Invention
`
`[0003] The present invention relates generally to methods
`and apparatus for data input, and, more particularly,
`to a
`method and apparatus for integrating manual input.
`
`[0004] B. Description of the Related Art
`
`[0005] Many methods for manual input of data and com-
`mands to computers are in use today, but each is most
`efficient and easy to use for particular types of data input.
`For example, drawing tablets with pens or pucks excel at
`drafting, sketching, and quick command gestures. Handwrit-
`ing with a stylus is convenient for filling out forms which
`require signatures, special symbols, or small amounts of
`text, but handwriting is slow compared to typing and voice
`input for long documents. Mice, finger-sticks and touchpads
`excel at cursor pointing and graphical object manipulations
`such as drag and drop. Rollers, thumbwheels and trackballs
`excel at panning and scrolling. The diversity of tasks that
`many computer users encounter in a single day call for all of
`these techniques, but few users will pay for a multitude of
`input devices, and the separate devices are often incompat-
`ible in a usability and an ergonomic sense. For instance,
`drawing tablets are a must for graphics professionals, but
`switching between drawing and typing is inconvenient
`because the pen must be put down or held awkwardly
`between the fingers while typing. Thus, there is a long-felt
`need in the art for a manual input device which is cheap yet
`offers convenient integration of common manual input tech-
`niques.
`
`[0006] Speech recognition is an exciting new technology
`which promises to relieve some of the input burden on user
`hands. However, voice is not appropriate for inputting all
`types of data either. Currently, voice input is best-suited for
`dictation of long text documents. Until natural language
`recognition matures sufficiently that very high level voice
`commands can be understood by the computer, voice will
`have little advantage over keyboard hot-keys and mouse
`menus for command and control. Furthermore, precise
`pointing, drawing, and manipulation of graphical objects is
`difficult with voice commands, no matter how well speech is
`understood. Thus, there will always be a need in the art for
`multi-function manual
`input devices which supplement
`voice input.
`
`[0007] A generic manual input device which combines the
`typing, pointing, scrolling, and handwriting capabilities of
`the standard input dew ice collection must have ergonomic,
`economic, and productivity advantages which outweigh the
`unavoidable sacrifices of abandoning device specialization.
`The generic device must tightly integrate yet clearly distin-
`guish the different types of input. It should therefore appear
`modeless to the user in the sense that the user should not
`
`need to provide explicit mode switch signals such as but-
`tonpresses, arm relocations, or stylus pickups before switch-
`ing from one input activity to another. Epidemiological
`studies suggest that repetition and force multiply in causing
`repetitive strain injuries. Awkward postures, device activa-
`
`tion force, wasted motion, and repetition should be mini-
`mized to improve ergonomics. Furthermore, the workload
`should be spread evenly over all available muscle groups to
`avoid repetitive strain.
`
`[0008] Repetition can be minimized by allocating to sev-
`eral graphical manipulation channels those tasks which
`require complex mouse pointer motion sequences. Common
`graphical user interface operations such as finding and
`manipulating a scroll bar or slider control are much less
`efficient than specialized finger motions which cause scroll-
`ing directly, without the step of repositioning the cursor over
`an on-screen control. Preferably the graphical manipulation
`channels should be distributed amongst many finger and
`hand motion combinations to spread the workload. Touch-
`pads and mice with auxilliary scrolling controls such as the
`Cirque® Smartcat touchpad with edge scrolling, the IBM®
`ScrollPointTM mouse with embedded pointing stick, and the
`Roller Mouse described in US. Pat. No. 5,530,455 to Gillick
`et al. represent small improvements in this area, but still do
`not provide enough direct manipulation channels to elimi-
`nate many often-used cursor motion sequences. Further-
`more, as S. Zhai et al. found in “Dual Stream Input for
`Pointing and Scrolling,”Pr0ceedings 0f CHI ’97 Extended
`Abstracts (1997), manipulation of more than two degrees of
`freedom at a time is very difficult with these devices,
`preventing simultaneous panning, zooming and rotating.
`
`reducing excess
`[0009] Another common method for
`motion and repetition is to automatically continue pointing
`or scrolling movement signals once the user has stopped
`moving or lifts the finger. Related art methods can be
`distinguished by the conditions under which such motion
`continuation is enabled.
`In US. Pat. No. 4,734,685,
`Watanabe continues image panning when the distance and
`velocity of pointing device movement exceed thresholds.
`Automatic panning is stopped by moving the pointing
`device back in the opposite direction, so stopping requires
`additional precise movements. In US. Pat. No. 5,543,591 to
`Gillespie et al., motion continuation occurs when the finger
`enters an edge border region around a small
`touchpad.
`Continued motion speed is fixed and the direction corre-
`sponds to the direction from the center of the touchpad to the
`finger at the edge. Continuation mode ends when the finger
`leaves the border region or lifts off the pad. Disadvanta-
`geously, users sometimes pause at
`the edge of the pad
`without intending for cursor motion to continue, and the
`unexpected motion continuation becomes annoying. US.
`Pat. No. 5,327,161 to Logan et al. describes motion con-
`tinuation when the finger enters a border area as well, but in
`an alternative trackball emulation mode, motion continua-
`tion can be a function solely of lateral finger velocity and
`direction at liftoff. Motion continuation decays due to a
`friction factor or can be stopped by a subsequent touchdown
`on the surface. Disadvantageously, touch velocity at liftoff is
`not a reliable indicator of the user’s desire for motion
`
`continuation since when approaching a large target on a
`display at high speeds the user may not stop the pointer
`completely before liftoff. Thus it would be an advance in the
`art to provide a motion continuation method which does not
`become activated unexpectedly when the user
`really
`intended to stop pointer movement at a target but happens to
`be on a border or happens to be moving at significant speed
`during liftoff.
`
`Petitioner Samsung Ex-1032, 0047
`
`Petitioner Samsung Ex-1032, 0047
`
`

`

`US 2002/0015024 A1
`
`Feb. 7, 2002
`
`[0010] Many attempts have been made to embed pointing
`devices in a keyboard so the hands don’t have to leave
`typing position to access the pointing device. These include
`the integrated pointing key described in US. Pat. No.
`5,189,403 to Franz et al.,
`the integrated pointing stick
`disclosed by J. Rutledge and T. Selker in “Force-to-Motion
`Functions for Pointing,”Human-C0mputer Interaction -IN-
`TERACT ’90, pp. 701-06 (1990), and the position sensing
`keys described in US. Pat. No. 5,675,361 to Santilli. Nev-
`ertheless,
`the limited movement range and resolution of
`these devices leads to poorer pointing speed and accuracy
`than a mouse, and they add mechanical complexity to
`keyboard construction. Thus there exists a need in the art for
`pointing methods with higher resolution, larger movement
`range, and more degrees of freedom yet which are easily
`accessible from typing hand positions.
`
`[0011] Touch screens and touchpads often distinguish
`pointing motions from emulated button clicks or keypresses
`by assuming very little lateral fingertip motion will occur
`during taps on the touch surface which are intended as
`clicks. Inherent in these methods is the assumption that
`tapping will usually be straight down from the suspended
`finger position, minimizing those components of finger
`motion tangential to the surface. This is a valid assumption
`if the surface is not finely divided into distinct key areas or
`if the user does a slow, “hunt and peck” visual search for
`each key before striking. For example,
`in US. Pat. No.
`5,543,591 to Gillespie et al., a touchpad sends all lateral
`motions to the host computer as cursor movements. How-
`ever, if the finger is lift