PCT
`INTERNATIONAL APPLICATION PUBLISHED UNDER THE PATENT COOPERATION TREATY (PCT)
`WO 99/38149
`
`WORLD INTELLECTUAL PROPERTY ORGANIZATION
`International Bureau
`
`(51) International Patent Classification 6 :
`G09G 5/00
`
`(11) International Publication Number:
`
`Al
`
`(43) International Publication Date:
`
`29 July 1999 (29.07.99)
`
`(21) International Application Number:
`
`PCT/US99/01454
`
`(22) International Filing Date:
`
`25 January 1999 (25.01.99)
`
`(30) Priority Data:
`60/072,509
`09/236,513
`
`26 January 1998 (26.01.98)
`25 January 1999 (25.01.99)
`
`us
`us
`
`WESTERMAN, Wayne
`(71)(72) Applicants and Inventors:
`[US/US]; 715 Oak Street, P.O. Box 354, Wellington, MO
`64097 (US). ELIAS, John, G. [US/US); Huguenot Farm,
`798 Taylors Bridge Road, Townsend, DE 19734 (US).
`
`(74) Agent: OLSEN, James, M.; Connolly & Rutz, P.O. Box 2207,
`Wilmington, DE 19899 (US).
`
`(81) Designated States: AL, AM, AT, AU, AZ, BA, BB, BG, BR,
`BY, CA, CH, CN, CU, CZ, DE, DK, EE, ES, Fl, GB, GE,
`GH, GM, HU, ID, IL, IS, JP, KE, KG, KP, KR, KZ, LC,
`LK, LR, LS, LT, LU, LV, MD, MG, MK, MN, MW, MX,
`NO, NZ, PL, PT, RO, RU, SD, SE, SG, SI, SK, SL, TJ, TM,
`TR, TT, UA, UG, US, UZ, VN, YU, ZW, ARIPO patent
`(GH, GM, KE, LS, MW, SD, SZ, UG, ZW), Eurasian patent
`(AM, AZ, BY, KG, KZ, MD, RU, TJ, TM), European patent
`(AT, BE, CH, CY, DE, DK, ES, Fl, FR, GB, GR, IE, IT,
`LU, MC, NL, PT, SE), OAPI patent (BF, BJ, CF, CG, CI,
`CM, GA, GN, GW, ML, MR, NE, SN, TD, TG).
`
`Published
`With international search report.
`
`(54) Title: METHOD AND APPARATUS FOR INTEGRATING MANUAL INPUT
`
`(57) Abstract
`
`Apparatus and methods are disclosed for si(cid:173)
`multaneously tracking multiple finger (202-204) and
`palm (206, 207) contacts as hands approach, touch,
`and slide across a proximity-sensing, compliant, and
`flexible multi-touch surface (2). The surface con(cid:173)
`sists of compressible cushion (32), dielectric elec(cid:173)
`trode (33), and circuitry layers. A simple proxim(cid:173)
`ity transduction circuit is placed under each elec(cid:173)
`trode to maximize the signal-to-noise ratio and to
`reduce wiring complexity. Scanning and signal off(cid:173)
`set removal on electrode array produces low-noise
`proximity images. Segmentation processing of each
`proximity image constructs a group of electrodes cor(cid:173)
`responding to each distinguishable contacts and ex(cid:173)
`tracts shape, position and surface proximity features
`for each group. Groups in successive images which
`correspond to the same hand contact are linked by a
`persistent path tracker (245) which also detects indi(cid:173)
`vidual contact touchdown and liftoff. Classification
`of intuitive hand configurations and motions enables
`unprecedented integration of typing, resting, point(cid:173)
`ing, scrolling, 3D manipulation, and handwriting into
`a versatile, ergonomic computer input device.
`
`12
`
`ELECTRODE
`SCANNING
`HARDWARE
`
`CALIBRATION AND
`PROXIMITY IMAGE
`FORMATION
`
`CONTACT
`TRACKING AND
`IDENTIFICATION
`
`6
`
`8
`
`10
`
`DISPLAY
`
`HOST
`COMPUTER
`SYSTEM
`
`HOST
`COMMUNICATION
`INTERFACE
`
`20
`
`Microsoft Ex. 1007
`Microsoft v. Philips - IPR2018-00026
`Page 1 of 158
`
`

`

`FOR THE PURPOSES OF INFORMATION ONLY
`
`Codes used to identify States party to the PCT on the front pages of pamphlets publishing international applications under the PCT.
`
`AL
`AM
`AT
`AU
`AZ
`BA
`BB
`BE
`BF
`BG
`BJ
`BR
`BY
`CA
`CF
`CG
`CH
`CI
`CM
`CN
`cu
`CZ
`DE
`DK
`EE
`
`Albania
`Armenia
`Austria
`Australia
`Azerbaijan
`Bosnia and Herzegovina
`Barbados
`Belgium
`Burkina Faso
`Bulgaria
`Benin
`Brazil
`Belarus
`Canada
`Central African Republic
`Congo
`Switzerland
`Cllte d'Ivoire
`Can1eroon
`China
`Cuba
`Czech Republic
`Germany
`Denmark
`Estonia
`
`ES
`FI
`FR
`GA
`GB
`GE
`GH
`GN
`GR
`HU
`IE
`IL
`IS
`IT
`JP
`KE
`KG
`KP
`
`KR
`KZ
`LC
`LI
`LK
`LR
`
`Spain
`Finland
`France
`Gabon
`United Kingdom
`Georgia
`Ghana
`Guinea
`Greece
`Hungary
`Ireland
`Israel
`Iceland
`Italy
`Japan
`Kenya
`Kyrgyzstan
`Democratic People's
`Republic of Korea
`Republic of Korea
`Kazakstan
`Saint Lucia
`Liechtenstein
`Sri Lanka
`Liberia
`
`LS
`LT
`LU
`LV
`MC
`MD
`MG
`MK
`
`ML
`MN
`MR
`MW
`MX
`NE
`NL
`NO
`NZ
`PL
`PT
`RO
`RU
`SD
`SE
`SG
`
`Lesotho
`Lithuania
`Luxembourg
`Latvia
`Monaco
`Republic of Moldova
`Madagascar
`The former Yugoslav
`Republic of Macedonia
`Mali
`Mongolia
`Mauritania
`Malawi
`Mexico
`Niger
`Netherlands
`Norway
`New Zealand
`Poland
`Portugal
`Romania
`Russian Federation
`Sudan
`Sweden
`Singapore
`
`SI
`SK
`SN
`sz
`TD
`TG
`TJ
`TM
`TR
`TT
`UA
`UG
`us
`uz
`VN
`YU
`zw
`
`Slovenia
`Slovakia
`Senegal
`Swaziland
`Chad
`Togo
`Tajikistan
`Turkmenistan
`Turkey
`Trinidad and Tobago
`Ukraine
`Uganda
`United States of America
`Uzbekistan
`Viet Nam
`Yugoslavia
`Zimbabwe
`
`Microsoft Ex. 1007
`Microsoft v. Philips - IPR2018-00026
`Page 2 of 158
`
`

`

`WO 99/38149
`
`PCT/US99/01454
`
`METHOD AND APPARATUS FOR INTEGRATING MANUAL INPUT
`
`BACKGROUND OF THE INVENTION
`
`The present application is based upon U.S. provisional patent application Serial No.
`
`60/072,509, filed January 26, 1998, and the U.S. utility application Serial No. 09,236,513, filed
`
`January 25, 1999.
`
`A.
`
`Field of the Invention
`
`The present invention relates generally to methods and apparatus for data input, and, more
`
`particularly, to a method and apparatus for integrating manual input.
`
`B.
`
`Description of the Related Art
`
`Many methods for manual input of data and commands to computers are in use today, but
`
`each is most efficient and easy to use for particular types of data input. For example, drawing tablets
`
`with pens or pucks excel at drafting, sketching, and quick command gestures. Handwriting with a
`
`stylus is convenient for filling out forms which require signatures, special symbols, or small amounts
`
`of text, but handwriting is slow compared to typing and voice input for long documents. Mice,
`
`finger-sticks and touchpads excel at cursor pointing and graphical object manipulations such as drag
`
`and drop. Rollers, thumbwheels and trackballs excel at panning and scrolling. The diversity of tasks
`
`that many computer users encounter in a single day call for all of these techniques, but few users will
`
`pay for a multitude of input devices, and the separate devices are often incompatible in a usability
`
`and an ergonomic sense. For instance, drawing tablets are a must for graphics professionals, but
`
`switching between drawing and typing is inconvenient because the pen must be put down or held
`
`awkwardly between the fingers while typing. Thus, there is a long-felt need in the art for a manual
`
`input device which is cheap yet offers convenient integration of common manual input techniques.
`
`Speech recognition is an exciting new technology which promises to relieve some of the
`
`input burden on user hands. However, voice is not appropriate for inputting all types of data either.
`
`Currently, voice input is best-suited for dictation of long text documents. Until natural language
`
`recognition matures sufficiently that very high level voice commands can be understood by the
`
`computer, voice will have little advantage over keyboard hot-keys and mouse menus for command
`
`and control. Furthermore, precise pointing, drawing, and manipulation of graphical objects is
`
`difficult with voice commands, no matter how well speech is understood. Thus, there will always
`
`be a need in the art for multi-function manual input devices which supplement voice input.
`
`- 1 -
`
`SUBSTITUTE SHEET (RULE 26)
`
`Microsoft Ex. 1007
`Microsoft v. Philips - IPR2018-00026
`Page 3 of 158
`
`

`

`WO 99/38149
`
`PCT/US99/01454
`
`A generic manual input device which combines the typing, pointing, scrolling, and
`
`handwriting capabilities of the standard input device collection must have ergonomic, economic, and
`
`productivity advantages which outweigh the unavoidable sacrifices of abandoning device
`
`specialization. The generic device must tightly integrate yet clearly distinguish the different types
`
`of input. It should therefore appear modeless to the user in the sense that the user should not need
`
`to provide explicit mode switch signals such as buttonpresses, arm relocations, or stylus pickups
`
`before switching from one input activity to another. Epidemiological studies suggest that repetition
`
`and force multiply in causing repetitive strain injuries. Awkward postures, device activation force,
`
`wasted motion, and repetition should be minimized to improve ergonomics. Furthermore, the
`
`workload should be spread evenly over all available muscle groups to avoid repetitive strain.
`
`Repetition can be minimized by allocating to several graphical manipulation channels those
`
`tasks which require complex mouse pointer motion sequences. Common graphical user interface
`
`operations such as finding and manipulating a scroll bar or slider control are much less efficient than
`
`specialized finger motions which cause scrolling directly, without the step of repositioning the cursor
`
`over an on-screen control. Preferably the graphical manipulation channels should be distributed
`
`amongst many finger and hand motion combinations to spread the workload. Touchpads and mice
`
`with auxilliary scrolling controls such as the Cirque® Smartcat touchpad with edge scrolling, the
`
`IBM® ScrollPoint™ mouse with embedded pointing stick, and the Roller Mouse described in U.S.
`
`Patent No. 5,530,455 to Gillick et al. represent small improvements in this area, but still do not
`
`provide enough direct manipulation channels to eliminate many often-used cursor motion sequences.
`
`Furthermore, as S. Zhai et al. found in "Dual Stream Input for Pointing and Scrolling," Proceedings
`
`of CHI '97 Extended Abstracts (1997), manipulation of more than two degrees of freedom at a time
`
`is very difficult with these devices, preventing simultaneous panning, zooming and rotating.
`
`Another common method for reducing excess motion and repetition is to automatically
`
`continue pointing or scrolling movement signals once the user has stopped moving or lifts the finger.
`
`Related art methods can be distinguished by the conditions under which such motion continuation
`
`is enabled. In U.S. Patent No. 4,734,685, Watanabe continues image panning when the distance and
`
`velocity of pointing device movement exceed thresholds. Automatic panning is stopped by moving
`
`the pointing device back in the opposite direction, so stopping requires additional precise
`
`movements. In U.S. Patent No. 5,543,591 to Gillespie et al., motion continuation occurs when the
`
`finger enters an edge border region around a small touchpad. Continued motion speed is fixed and
`
`- 2 -
`
`SUBSTITUTE SHEET (RULE 26)
`
`Microsoft Ex. 1007
`Microsoft v. Philips - IPR2018-00026
`Page 4 of 158
`
`

`

`WO 99/38149
`
`PCT/US99/01454
`
`the direction corresponds to the direction from the center of the touchpad to the finger at the edge.
`
`Continuation mode ends when the finger leaves the border region or lifts off the pad.
`
`Disadvantageously, users sometimes pause at the edge of the pad without intending for cursor
`
`motion to continue, and the unexpected motion continuation becomes annoying. U.S. Patent No.
`
`5,327,161 to Logan et al. describes motion continuation when the finger enters a border area as well,
`
`but in an alternative trackball emulation mode, motion continuation can be a function solely of
`
`lateral finger velocity and direction at liftoff. Motion continuation decays due to a friction factor or
`
`can be stopped by a subsequent touchdown on the surface. Disadvantageously, touch velocity at
`
`liftoff is not a reliable indicator of the user's desire for motion continuation since when approaching
`
`a large target on a display at high speeds the user may not stop the pointer completely before liftoff.
`
`Thus it would be an advance in the art to provide a motion continuation method which does not
`
`become activated unexpectedly when the user really intended to stop pointer movement at a target
`
`but happens to be on a border or happens to be moving at significant speed during liftoff.
`
`Many attempts have been made to embed pointing devices in a keyboard so the hands don't
`
`have to leave typing position to access the pointing device. These include the integrated pointing
`
`key described in U.S. Patent No. 5,189,403 to Franz et al., the integrated pointing stick disclosed by
`
`J. Rutledge and T. Selker in "Force-to-Motion Functions for Pointing," Human-Computer Interaction
`
`- INTERACT '90, pp. 701-06 (1990), and the position sensing keys described in U.S. Patent No.
`
`5,675,361 to Santilli. Nevertheless, the limited movement range and resolution of these devices
`
`leads to poorer pointing speed and accuracy than a mouse , and they add mechanical complexity to
`
`keyboard construction. Thus there exists a need in the art for pointing methods with higher
`
`resolution, larger movement range, and more degrees of freedom yet which are easily accessible
`
`from typing hand positions.
`
`Touch screens and touchpads often distinguish pointing motions from emulated button clicks
`
`or keypresses by assuming very little lateral fingertip motion will occur during taps on the touch
`
`surface which are intended as clicks. Inherent in these methods is the assumption that tapping will
`
`usually be straight down from the suspended finger position, minimizing those components of finger
`
`motion tangential to the surface. This is a valid assumption if the surface is not finely divided into
`
`distinct key areas or if the user does a slow, "hunt and peck" visual search for each key before
`
`striking. For example, in U.S. No. Patent 5,543,591 to Gillespie et al., a touchpad sends all lateral
`
`motions to the host computer as cursor movements. However, if the finger is lifted soon enough
`
`- 3 -
`
`SUBSTITUTE SHEET (RULE 26)
`
`Microsoft Ex. 1007
`Microsoft v. Philips - IPR2018-00026
`Page 5 of 158
`
`

`

`WO 99/38149
`
`PCT/US99/01454
`
`after touchdown to count as a tap and if the accumulated lateral motions are not excessive, any sent
`
`motions are undone and a mouse button click is sent instead. This method only works for mouse
`
`commands such as pointing which can safely be undone, not for dragging or other manipulations.
`
`In U.S. Patent No. 5,666,113 to Logan, taps with less than about 1116" lateral motion activate keys
`
`on a small keypad while lateral motion in excess of 1/16" activates cursor control mode. In both
`
`patents cursor mode is invoked by default when a finger stays on the surface a long time.
`
`However, fast touch typing on a surface divided into a large array of key regions tends to
`
`produce more tangential motions along the surface than related art filtering techniques can tolerate.
`
`Such an array contains keys in multiple rows and columns which may not be directly under the
`
`fingers, so the user must reach with the hand or flex or extend fingers to touch many of the key
`
`regions. Quick reaching and extending imparts significant lateral finger motion while the finger is
`
`in the air which may still be present when the finger contacts the surface. Glancing taps with as
`
`much as 114" lateral motion measured at the surface can easily result. Attempting to filter or
`
`suppress this much motion would make the cursor seem sluggish and unresponsive. Furthermore,
`
`it may be desirable to enter a typematic or automatic key repeat mode instead of pointing mode when
`
`the finger is held in one place on the surface. Any lateral shifting by the fingertip during a prolonged
`
`finger press would also be picked up as cursor jitter without heavy filtering. Thus, there is a need
`
`in the art for a method to distinguish keying from pointing on the same surface via more robust hand
`
`configuration cues than lateral motion of a single finger.
`
`An ergonomic typing system should require minimal key tapping force, easily distinguish
`
`finger taps from resting hands, and cushion the fingers from the jarring force of surface impact.
`
`Mechanical and membrane keyboards rely on the spring force in the keyswitches to prevent
`
`activation when the hands are resting on the keys. This causes an irreconcilable tradeoffbetween
`
`the ergonomic desires to reduce the fatigue from key activating force and to relax the full weight of
`
`the hands onto the keys during rest periods. Force minimization on touch surfaces is possible with
`
`capacitive or active optical sensing, which do not rely on finger pressure, rather than resistive(cid:173)
`
`membrane or surface-acoustic-wave sensing techniques. The related art touch devices discussed
`
`below will become confused if a whole hand, including its four fingertips, a thumb and possibly
`
`palm heels, rests on the surface. Thus, there exists a long felt need in the art for a multi-touch
`
`surface typing system based on zero-force capacitive sensing which can tolerate resting hands and
`
`a surface cushion.
`
`-4-
`
`SUBSTITUTE SHEET (RULE 26)
`
`Microsoft Ex. 1007
`Microsoft v. Philips - IPR2018-00026
`Page 6 of 158
`
`

`

`WO 99/38149
`
`PCT/US99/01454
`
`An ergonomic typing system should also adapt to individual hand sizes, tolerate variations
`
`in typing style, and support a range of healthy hand postures. Though many ergonomic keyboards
`
`have been proposed, mechanical keyswitches can only be repositioned at great cost. For example,
`
`the keyboard with concave keywells described by Hargreaves et al. in U.S. Patent No. 5,689,253 fits
`
`most hands well but also tends to lock the arms in a single position. A touch surface key layout
`
`could easily be morphed, translated, or arbitrarily reconfigured as long as the changes didn't confuse
`
`the user. However, touch surfaces may not provide as much laterally orienting tactile feedback as
`
`the edges of mechanical keyswitches. Thus, there exists a need in the art for a surface typing
`
`recognizer which can adapt a key layout to fit individual hand postures and which can sustain typing
`
`accuracy if the hands drift due to limited tactile feedback.
`
`Handwriting on smooth touch surfaces using a stylus is well-known in the art, but it typically
`
`doesn't integrate well with typing and pointing because the stylus must be put down somewhere or
`
`held awkwardly during other input activities. Also, it may be difficult to distinguish the handwriting
`
`activity of the stylus from pointing motions of a fingertip. Thus there exists a need in the art for a
`
`method to capture coarse handwriting gestures without a stylus and without confusing them with
`
`pointing motions.
`
`Many of the input differentiation needs cited above could be met with a touch sensing
`
`technology which distinguishes a variety of hand configurations and motions such as sliding finger
`
`chords and grips. Many mechanical chord keyboards have been designed to detect simultaneous
`
`downward activity from multiple fingers, but they do not detect lateral finger motion over a large
`
`range. Related art shows several examples of capacitive touchpads which emulate a mouse or
`
`keyboard by tracking a single finger. These typically measure the capacitance of or between
`
`elongated wires which are laid out in rows and columns. A thin dielectric is interposed between the
`
`row and column layers. Presence of a finger perturbs the self or mutual capacitance for nearby
`
`electrodes. Since most of these technologies use projective row and column sensors which integrate
`
`on one electrode the proximity of all objects in a particular row or column, they cannot uniquely
`
`determine the positions of two or more objects, as discussed in S. Lee, "A Fast Multiple-Touch(cid:173)
`
`Sensitive Input Device," University of Toronto Masters Thesis (1984). The best they can do is count
`
`fingertips which happen to lie in a straight row, and even that will fail if a thumb or palm is
`
`introduced in the same column as a fingertip.
`
`In U.S. Patent Nos. 5,565,658 and 5,305,017, Gerpheide et al. measure the mutual
`
`- 5 -
`
`SUBSTITUTE SHEET (RULE 26)
`
`Microsoft Ex. 1007
`Microsoft v. Philips - IPR2018-00026
`Page 7 of 158
`
`

`

`WO 99/38149
`
`PCT/US99/01454
`
`capacitance between row and column electrodes by driving one set of electrodes at some clock
`
`frequency and sensing how much of that frequency is coupled onto a second electrode set. Such
`
`synchronous measurements are very prone to noise at the driving frequency, so to increase signal-to(cid:173)
`
`noise ratio they form virtual electrodes comprised of multiple rows or multiple columns, instead of
`
`a single row and column, and scan through electrode combinations until the various mutual
`
`capacitances are nulled or balanced. The coupled signal increases with the product of the rows and
`
`columns in each virtual electrodes, but the noise only increases with the sum, giving a net gain in
`
`signal-to-noise ratio for virtual electrodes consisting of more than two rows and two columns.
`
`However, to uniquely distinguish multiple objects, virtual electrode sizes would have to be reduced
`
`so the intersection of the row and column virtual electrodes would be no larger than a finger tip, i.e.
`
`about two rows and two columns, which will degrade the signal-to-noise ratio. Also, the signal-to(cid:173)
`
`noise ratio drops as row and column lengths increase to cover a large area.
`
`In U.S. Patent Nos. 5,543,591, 5,543,590, and 5,495,077, Gillespie et al measure the
`
`electrode-finger self-capacitance for row and column electrodes independently. Total electrode
`
`capacitance is estimated by measuring the electrode voltage change caused by injecting or removing
`
`a known amount of charge in a known time. All electrodes can be measured simultaneously if each
`
`electrode has its own drive/sense circuit. The centroid calculated from all row and column electrode
`
`signals establishes an interpolated vertical and horizontal position for a single object. This method
`
`may in general have higher signal-to-noise ratio than synchronous methods, but the signal-to-noise
`
`ratio is still degraded as row and column lengths increase. Signal-to-noise ratio is especially
`
`important for accurately locating objects which are floating a few millimeters above the pad.
`
`Though this method can detect such objects, it tends to report their position as being near the middle
`
`of the pad, or simply does not detect floating objects near the edges.
`
`Thus there exists a need in the art for a capacitance-sensing apparatus which does not suffer
`
`from poor signal-to-noise ratio and the multiple finger indistinguishability problems of touchpads
`
`with long row and column electrodes.
`
`U.S. Patent No. 5,463,388 to Boie et al. has a capacitive sensing system applicable to either
`
`keyboard or mouse input, but does not consider the problem of integrating both types of input
`
`simultaneously. Though they mention independent detection of arrayed unit-cell electrodes, their
`
`capacitance transduction circuitry appears too complex to be economically reproduced at each
`
`electrode. Thus the long lead wires connecting electrodes to remote signal conditioning circuitry
`
`- 6 -
`
`SUBSTITUTE SHEET (RULE 26)
`
`Microsoft Ex. 1007
`Microsoft v. Philips - IPR2018-00026
`Page 8 of 158
`
`

`

`WO 99/38149
`
`PCT/US99/01454
`
`can pickup noise and will have significant capacitance compared to the finger-electrode self(cid:173)
`
`capacitance, again limiting signal-to-noise ratio. Also, they do not recognize the importance of
`
`independent electrodes for multiple finger tracking, or mention how to track multiple fingers on an
`
`independent electrode array.
`
`Lee built an early multi-touch electrode array with 7 mm by 4 mm metal electrodes arranged
`
`in 32 rows and 64 columns. The "Fast Multiple-Touch-Sensitive Input Device (FMTSID)" total
`
`active area measured 12" by 16", with a .075 mm Mylar dielectric to insulate fingers from electrodes.
`
`Each electrode had one diode connected to a row charging line and a second diode connected to a
`
`column discharging line. Electrode capacitance changes were measured singly or in rectangular
`
`groups by raising the voltage on one or more row lines, selectively charging the electrodes in those
`
`rows, and then timing the discharge of selected columns to ground through a discharge resistor.
`
`Lee's design required only two diodes per electrode, but the principal disadvantage of Lee's design
`
`is that the column diode reverse bias capacitances allowed interference between electrodes in the
`
`same column.
`
`All of the related capacitance sensing art cited above utilize interpolation between electrodes
`
`to achieve high pointing resolution with economical electrode density. Both Boie et al. and Gillespie
`
`et al. discuss computation of a centroid from all row and column electrode readings. However, for
`
`multiple finger detection, centroid calculation must be carefully limited around local maxima to
`
`include only one finger at a time. Lee utilizes a bisective search technique to find local maxima and
`
`then interpolates only on the eight nearest neighbor electrodes of each local maximum electrode.
`
`This may work fine for small fingertips, but thumb and palm contacts may cover more than nine
`
`electrodes. Thus there exists a need in the art for improved means to group exactly those electrodes
`
`which are covered by each distinguishable hand contact and to compute a centroid from such
`
`potentially irregular groups.
`
`To take maximum advantage of multi-touch surface sensing, complex proximity image
`
`processing is necessary to track and identify the parts of the hand contacting the surface at any one
`
`time. Compared to passive optical images, proximity images provide clear indications of where the
`
`body contacts the surface, uncluttered by luminosity variation and extraneous objects in the
`
`background. Thus proximity image filtering and segmentation stages can be simpler and more
`
`reliable than in computer vision approaches to free-space hand tracking such as S. Ahmad, "A
`
`Usable Real-Time 3D Hand Tracker", Proceedin~s of the 281h Asilomar Conference on Si~nals.
`
`- 7 -
`
`SUBSTITUTE SHEET (RULE 26)
`
`Microsoft Ex. 1007
`Microsoft v. Philips - IPR2018-00026
`Page 9 of 158
`
`

`

`WO 99/38149
`
`PCT/US99/01454
`
`Systems. and Computers - Part 2, vol. 2, IEEE (1994) or Y. Cui and J. Wang, "Hand Segmentation
`
`Using Learning-Based Prediction and Verification for Hand Sign Recognition," Proceedings of the
`
`1996 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 88-93
`
`(1996). However, parts of the hand such as intermediate finger joints and the center of the palms
`
`do not show up in capacitive proximity images at all if the hand is not flattened on the surface.
`
`Without these intermediate linkages between fingertips and palms the overall hand structure can only
`
`be guessed at, making hand contact identification very difficult. Hence the optical flow and contour
`
`tracking techniques which have been applied to free-space hand sign language recognition as in F.
`
`Quek, "Unencumbered Gestural Interaction," IEEE Multimedia, vol. 3, pp. 36-47 (1996), do not
`
`address the special challenges of proximity image tracking.
`
`Synaptics Corp. has successfully fabricated their electrode array on flexible mylar film rather
`
`than stiff circuit board. This is suitable for conforming to the contours of special products, but does
`
`not provide significant finger cushioning for large surfaces. Even if a cushion was placed under the
`
`film, the lack of stretchability in the film, leads, and electrodes would limit the compliance afforded
`
`by the compressible material. Boie et al suggests that placing compressible insulators on top of the
`
`electrode array cushions finger impact. However, an insulator more than about one millimeter thick
`
`would seriously attenuate the measured finger-electrode capacitances. Thus there exists a need in
`
`the art for a method to transfer finger capacitance influences through an arbitrarily thick cushion.
`
`SUMMARY OF THE INVENTION
`
`It is a primary object of the present invention to provide a system and method for
`
`integrating different types of manual input such as typing, multiple degree-of-freedom
`
`manipulation, and handwriting on a multi-touch surface.
`
`It is also an object of the present invention to provide a system and method for
`
`distinguishing different types of manual input such as typing, multiple degree-of-freedom
`
`manipulation, and handwriting on a multi-touch surface, via different hand configurations which
`
`are easy for the user to learn and easy for the system to recognize.
`
`It is a further object of the present invention to provide an improved capacitance(cid:173)
`
`transducing apparatus that is cheaply implemented near each electrode so that two-dimensional
`
`sensor arrays of arbitrary size and resolution can be built without degradation in signal to noise.
`
`It is a further object of the present invention to provide an electronic system which
`
`SUBSTITUTE SHEET (RULE 26)
`
`Microsoft Ex. 1007
`Microsoft v. Philips - IPR2018-00026
`Page 10 of 158
`
`

`

`WO 99/38149
`
`PCT/US99/01454
`
`minimizes the number of sensing electrodes necessary to obtain proximity images with such
`
`resolution that a variety of hand configurations can be distinguished.
`
`Yet another object of the present invention is to provide a multi-touch surface apparatus
`
`which is compliant and contoured to be comfortable and ergonomic under extended use.
`
`Yet another object of the present invention is to provide tactile key or hand position
`
`feedback without impeding hand resting on the surface or smooth, accurate sliding across the
`
`surface.
`
`It is a further object of the present invention to provide an electronic system which can
`
`provide images of flesh proximity to an array of sensors with such resolution that a variety of hand
`
`configurations can be distinguished.
`
`It is another object of the present invention to provide an improved method for invoking
`
`cursor motion continuation only when the user wants it by not invoking it when significant
`
`deceleration is detected.
`
`Another object of the present invention is to identify different hand parts as they contact
`
`the surface so that a variety of hand configurations can be recognized and used to distinguish
`
`different kinds of input activity.
`
`Yet another object of the present invention is to reliably extract rotation and scaling as well
`
`as translation degrees of freedom from the motion of two or more hand contacts to aid in
`
`navigation and manipulation of two-dimensional electronic documents.
`
`It is a further object of the present invention to reliably extract tilt and roll degrees of
`
`freedom from hand pressure differences to aid in navigation and manipulation of three-dimensional
`
`environments.
`
`Additional objects and advantages of the invention will be set forth in part in the description
`
`which follows, and in part will be obvious from the description, or may be learned by practice of the
`
`invention. The objects and advantages of the invention will be realized and attained by means of the
`
`elements and combinations particularly pointed out in the appended claims.
`
`To achieve the objects and in accordance with the purpose of the invention, as embodied and
`
`broadly described herein, the invention comprises a sensing device that is sensitive to changes in
`
`self-capacitance brought about by changes in proximity of a touch device to the sensing device, the
`
`sensing device comprising: two electrical switching means connected together in series having a
`
`common node, an input node, and an output node; a dielectric-covered sensing electrode connected
`
`-9-
`
`SUBSTITUTE SHEET (RULE 26)
`
`Microsoft Ex. 1007
`Microsoft v. Philips - IPR2018-00026
`Page 11 of 158
`
`

`

`WO 99/38149
`
`PCT/US99/01454
`
`to the common node between the two switching means; a power supply providing an approximately
`
`constant voltage connected to the input node of the series-connected switching means; an integrating
`
`capacitor to accumulate charge transferred during multiple consecutive switchings of the series
`
`connected switching means; another switching means connected in parallel across the integrating
`
`capacitor to deplete its residual charge; and a voltage-to-voltage translation device connected to the
`
`output node of the series-connected switching means which produces a voltage representing the
`
`magnitude of the self-capacitance of the sensing device. Alternatively, the sensing device comprises:
`
`two electrical switching means connected together in series having a common node, an input node,
`
`and an output node; a dielectric-covered sensing electrode connected to the common node between
`
`the two switching means; a power supply providing an approximately constant voltage connected
`
`to the input node of the series-connected switching means; and