C H I
`
`' 8 5
`
`P R 0 C E E D I N G S
`
`A P RIL 1985
`
`A MULTI-TOUCHTHREE DIMENSIONALTOUCH-SENSITIVETABLET
`
`SK. Lee, W. Buxton, K.C. Smith
`Computer Systems ResearchInstitute
`University of Toronto
`Toronto,Ontario
`Canada, MSS 1A4
`
`(416)-978-8320
`
`ABSTRACT
`A prototype touch-sensillvetablet la presented. The tablet's main
`innovation is that It is capable of sensing rnore than one point of
`in additionto being able to provide position coor-
`contact at a time,
`dlnates, the tablet also gives a measure of degree of contact,
`In order to enable multi-
`independently for each polnt of contact.
`touch sensing, the tablet strface is divided into a grid of dlscrete
`points. The points are scanned using a rectrsivearea subdivision
`In order to minimize the resolution lost due to the
`algorithm.
`dlscrete nature of the grid, a novel interpolatlon scheme has been
`developed. Finally, the paper briefly discusses how multi-touch
`sensing, Interpolatlon,and degree of contact sensing can be com.
`bined to expand our vocabularyin hirnan-computerInteraction.
`
`1 INTRODUCTION
`Rapid advancement of computer technology has opened a varlety
`of new appllcations. New appilcationsand users mean demandsfor
`new modes of interaction. One consequence of this is a growing
`appreciationof the importance of using appropriate Input technolo-
`gles (Buxton, 1982). Positioning devices are seen to be essential to
`graphics applications,image transducers are required for pattern
`recognitIon in medical dlagnosis, touch screens are useful for the
`education of young chlldren, and the QWERTY keyboard remains
`the usual standard for text processing. However, the range of input
`devicesavallable is still qdte Ilmited, as is our understandingof how
`to use them in the most effective manner.
`The intent of the research presented in this paper la to Increase the
`vocabdarythat can be utillzed in human-computer Interaction. Our
`approachhas been to develop a new Input technology that enlarges
`the domain of human physical gestures that can be captured for
`In what follows, we will describe the technology,
`control purposes.
`what it evolved from, and some aspectsof how it can be used.
`
`2. OVERVIEW
`The transducerthat we have developed is a touch-sensitivetablet;
`that is, a flat surface that can sense where it is being touched by
`the operator'sfinger. This in itself la not new. Several such dev-
`Ices are commercially available from a number of manufacturers
`(see AppendlxA). What is unique about our tablet is that it com-
`
`Permission to copy without fee all or part of this materialis granted
`providedthat the copies are not made or distributedfor direct
`commercialadvantage, the ACM copyrightnotice and the title of the
`publicationand its date appear, and notice is given that copying is by
`permission of the Association for ComputingMachinery.To copy
`otherwise, or to republish,requires a fee and/or specific permission.
`
`©
`
`1985 ACM 0-89791-149-0/85/004/0021 $00.75
`
`blnes two additional features. First, it can sense the degree of con-
`tact in a continuous manner. Second, it can sense the amotrit and
`location of a number of simultaneous points of contact. These two
`features, when combined with touch sensing, are very importantin
`respect to the types of Interactionthat we can support. Some of
`these are dlscussed below, but see Buxton, Hlli, and Rowley (1985)
`and Brown, Buxton and Murtagh(1985)for more detail. The tablet
`which we present Is a contínuation of work done in our lab by
`Sasaki et al (1981)and Metha (1982).
`In the presentation which follows, we focus mainly on issues relat-
`Ing to the transducer'sImplementation. Two important contributions
`discussed are our method of scanningthe tablet surface, and our
`rnethod of maintalning high resolution despite the surface being
`partitioned into a discrete grid. Additional technical details can be
`found in Lee (1984).
`
`s. WHY MULTI-TOUCH7
`Touch sensinghas a number of Important characteristics. There is
`no physicalstylus or puck to get lost, broken, or vlbrate out of posi-
`tion. Touch tablets can be molded so as to make them easy to
`clean (therefore makingthem useful In clean environmentslike hos-
`plials, or dirty environments like factories). Since there is no
`mechanicalIntermedlarybetween hand and tablet, there is nothing
`to prevent multi-touch sensing. Templates can be placed over the
`tablet to define special regions and, since the hand is being used
`directly, these reglons can be manually sensed,thereby allowing the
`tralned user to effectively"touch type" on the tablet.
`Without pressure sensing, however, the utility of touch tablets is
`quite limited. One can move a tracking symbol around the screen,
`for example, but when the finger is over a light button, there is noth-
`Ing equivalent to the button on a mouse to push in order to make a
`selection. Yes, we could llft the finger off the tablet, but that would
`be more like pulling (rather than pushing) the button. And what if
`we wanted to drag an item being polnted at, or to Indlcate that we
`wanted to start Inking? Lifting our finger wodd leave otr finger off
`the tablet, just when we want 11In contact with it the most. There
`If, however,
`are ways around this problem, but they are Indirect.
`the tablet has pressure sensing, we can push a virtual button by
`giving an extra blt of pressureto signal a change in state.
`Pressure has other advantages. One example is to control line
`thickness In a paint program. But why do we want mdtlple point
`sensing? A simple example would be if we had a templateplaced
`over the tablet which dellmited three regions of 9 cm by 2 cm.
`Where we touch each region codd control the setting of a parame-
`If we wanted to simultaneously
`ter associated with each region.
`adjust all three parameters,then we would have to be able to sense
`three reglons. An even easier example is using the tablet to
`all
`emdate a piano keyboard that can play polyphonic music.
`
`SCEA Ex. 1031 Page 1
`
`

`

`C H I
`
`' 8 5
`
`P R 0 C E E D I N G S
`
`A P R I L
`
`1 9 8 5
`
`The software In the controlling CPU utlllzescommmicatlonwith the
`host computer to accommodate the interpolation scheme. The
`clock rate (10 MHz) allows about 10 counts to correspondto the
`sensor capacitancechangedue to a touch. But, of course, the capa-
`citance of all the circuitry attached to the column line during the
`discharging period is much larger than the sensor capacitance.
`Thus before scanning the tablet for a touch, it
`is scanned com-
`plately In all possible resolution modes when not touched. The
`values so obtained are stored as references. Touches are
`identified by the differencesbetween the reference values and the
`values measured during use.
`The capacitance change corresponding to the touch by more than
`one finger (or by the whole hand) is very large. Thus the number of
`bits in the counter should be enough to measure the maximum
`is ænecessary either to have sufficient
`capacitance. However it
`bits to measure the entire capacitance including the surrounding
`capacitances, or to store the corresponding "complete" counter
`is necessary only to have one more bli
`values as references. It
`than the number of bits required to count the value of change in
`the capacitance rather than the complete value in order to measure
`the differences of capacitance due to tot.ch. Thus only an 8 bit
`counter is implemented. The counter enablesthe measurement of a
`7 bit capacitance change regardless of the degree of overflow In
`the counter.
`A facllity is also provided for identlfying templates applied to the
`surface of the tablet.
`
`5. SCANNINGALGORITHM
`One Idea of some significance that can be introduced la to avold
`scanning of all the pixels In the tablet which contain no Information.
`For example, scanning all 2048 points of a tablet having a resolu-
`tion 64 by 32 for fewer than 10 points is really quite a ridlculous
`In fact, 11the number of points to be searched is comparably
`idea.
`small, then an improved algorithm,here called recursivearea subdl-
`implementation example is
`vision, can be used. A particular
`described as follows.
`Considera tablet with resolution 8 by 8 to be searchedfor a touch
`point as shown in Figtre 2. First, check the tablet for touch as a
`whole region as shown by the area ABCD in the figure. If touch is
`detected, divide the tablet into two equal regions shown by the line
`the two reglons ABEF and EFCD for
`EF and check each of
`touchedness. Select the touched region, region EFCD in this case,
`and divide this into two equal reglons as shown by the divisionline
`GH. Contlnue this process on the touched reglon until no further
`division is possible, that is, mill a unit sensor, designatedas the
`region PKMO In Figtre 2, is reached. The flgure also shows the
`sequence of subdivision in the recursive subdivision scheme.
`
`A
`
`B
`
`(4)
`
`N
`
`4. HARDWAREDESCRIPTION
`the hardware of
`the fast multiple-touch-
`A brief description of
`sensitive input device (FMTSID) is introducedhere. The design of
`the hardware is based on the reqdrements of the fast scanning
`algorithm and on tradeoffs between software and hardware. Many
`sensors have been examined for our particular application, however
`(Hurst, 1974; Hillis, 1982; TSD, 1982; TASA, 1980; JSRC, 1981;
`Metha, 1982)none seemed to have the properties that satisfy the
`requirements of a FMTSID. The hardware basically consists of a
`sensor matrix board, row and colurnn selection registers, A/D
`convertingcircults and a controllingCPU.
`The design of the sensor matrix is based on the technique of capa-
`citance measurement between a finger tip and a metal plate. To
`minimize hardware, the sensors are accessed by row and column
`selection. Row selection registers select one or more rows by set-
`ting the correspondingbits to a high state In order to charge up the
`sensors while the column selection registers select one or more
`columns by turning on correspondinganalog switches to discharge
`the sensorsthrough timing resistors. The Intersectingreglon of the
`selected rows and the selected columns represents the selected
`sensors as a group. A/D converting
`circuits rneasure the
`A Universityof
`discharging time Interval of the selectedsensors.
`Toronto 6809 board is used as a controlling CPU. The totch sur-
`face of the sensor board conslsis of ntsnber of small metal-coated
`ectangdar-shapedareas serving as sensor plate capacitors. T
`the metal plate area of a unit sensor depends on the
`design of
`rneasurable capacitance change that resdts when the area is
`covered by a finger tlp, and on the resolution that can be imple-
`rnented.
`
`Charging diode
`
`Dischargingdiode
`
`a row line
`
`SENSOR
`
`(C)
`
`C3
`
`? °°1°°°Al"°
`R
`
`a column selection
`switch
`
`Fig. 1 A model of a selectedsensor in the sensor matrix.
`
`In order to select a sensor by row and column access, two diodes
`are used with each sensor.One diode,connectedto the row line, la
`used to charge up the sensors in the row, it is referred to as the
`Charging Dlode (CD) as shown in Figure 1. The CD also serves to
`block the charge flowing back to the row line when the row line vol-
`tage is dropped to zero. The other diode called the Discharging
`Diode(DD),connected to the column line, enablesdischarging of the
`selected row sensors to a virtual ground. Also the DD blocks
`charge flow from the sensors in the selected row to the sensors in
`the unselected rows during the discharging period. The selectionof
`rows, by the row selection procedure, causes the sensors to be
`charged. The sensors in the column are then discharged through
`associated timing resistors connected to the column selection
`switches.
`The charges stored In the selected row(s) flow down through the
`selected switches to the virtual gromd of a fast operatlanal
`ampilHer. All the dischargingcurrents are correspondinglyadded to
`produce a signal from which the dischargingtime of all the selected
`sensors is found by comparisonwith a thresholdvoltage.
`Presswe sensitivity is incorporated by two measures:First there is
`the effect, here minor, of compressionof the overlaying insulator.
`Second there la the effect of intrinsic spreadingof the compressible
`finger tip as pressureis increased.
`
`22
`
`D
`
`?H
`(2)
`(n)-Sequence of subdivision in binary operation.
`Fig. 2 Rectrsivesubdivision operation for 8 by 8 tablet.
`
`L
`
`C
`
`SCEA Ex. 1031 Page 2
`
`

`

`C H I
`
`' 8 5
`
`P R 0 C E E D l N G S
`
`A P R I L 1985
`
`Using this algorlthm, a search for one polnt on a tablet having a
`resolution 64 by 32, requires 22 scanningtimes, that is
`
`2 * {log sub 2} (64 * 32) = 22
`
`If there is no overhead In the recursive subdivisionprocess and
`scanning begins at the "top of the tree" (that is, with a region in
`which all pixels are groupedtogether), then using this scheme, the
`number of touched points that can be identified in the time that it
`would take to detect one touch directly (that is, if all pixels are
`scanned one by one sequentially)is
`
`N = {{S4 ? 32} over 22} = 186.
`This shows immediately that the recursive subdivision scheme is
`much superior to sequentialscanning if the ntrnber of points to be
`scanned is fewer than 186.
`
`6.
`INTERPOt.ATION
`11may seem that the resolutionof the hardwareis too low for use in
`graphicsapplications. However touch intensity and multi-touchsen-
`sltivity can be used to enhance resolutlon. This is possible because
`the center of a touch can be most accuratelyestimated by an inter-
`potation utilizing the values of the adjacentsensor intensities.
`Direct Interpolation schemes for a few cases has been imple-
`rnented. One of Interest is to interpolatean array of 3 by 3 sensors
`in the center. Another is to Interpolate all
`using a touched point
`points on the tablet. The later one obviouslyprovides the highest
`resolution but as a result It slmply emulates a single touch tablet
`with very high resolution.
`
`7. PERFORMANCE
`
`7.1 Sensor
`An ideal sensormatrix for a FMTSIDwodd be one that has uniform
`and small reference values over a grotping level, a large variation
`of Intensity due to a touch, and fast measurementtime. The sensor
`matrix of the prototype, however, has a relatively wide range of
`reference values. However these values do not change very much
`over extended periods of tlme. The resuts show that doubling the
`number of sensors'ina groupin the column direction Increases the
`reference value by a factor of about 1.5. This corresponds well to
`theoretical estimates. As well the results show that Increasingthe
`number of sensors in a group in the row direction,in contrast, does
`not increase the reference value In general, even if the number of
`the sensors is dotbled in a group. The reference value ranges
`from 40 (for a single sensor in a group)to 580 (for the entire array
`of 64 by 32 sensorsconsideredas a group).
`In order to account for time and other varlations of the reference
`values, a thresholdis included which must be overcomein order for
`a touch to be detected. The threshold used ranges from 2 to 7
`comts depending on group size. Using these thresholdvalues the
`CPU does not report untouched points wrongly over intervals of at
`least 3 hotra in alther sequential or recursive subdivision modes.
`The recursive subdivlsion scheme uses 6 different thresholds,con-
`is very mllkely to report a wrong point whereas the
`sequently it
`linear scanning mode using only a single threshold Is likely to be
`more sensitive.
`The intenalty of a single touch for a single sensor group varies over
`the tablet but usually ranges above the thresholdvalue by as much
`as 15. For a single 64 by 32 sensor grotp,the intenaltyvaries from
`person to person but It ranges from the threshold to 124. This max-
`imurn is obtained when a palm rather than a finger touches the
`interesting feattre is that
`the response time
`tablet. Another
`becomes faster as the number of sensors in a group becomes
`larger. and furthermore that for the 64 by 32 sensor group, it is pos-
`sible to detect of a hand merely placed In the vicinityof the tablet.
`
`7.2 Spatial Resolution
`One possible and immediate interpolation scheme is to interpolate a
`"touched" point with all adjacent values which may not be large
`enough to be reported as touched. A local array of 3 by 3 points
`can be used for this Interpolatlon. Some examples drawn on a
`laser printer (consequently having no intensity scale) are shown in
`Figure 3. These pictures are produced without feedback, that is,
`drawn without the operator looking at the outputscreen. This does
`not allow the operator to compensate,that Is, to select pointswhere
`data are sparse In cornparisonwith the intended figure, but rather
`takes direct input from the location of the figure drawn on the input
`device. The first pleture (a) is drawn by moving a finger In a
`straight line (guided by a ruler) for various angles and the second
`one (b) ls drawn by movinga finger in a line gdde by a circle drawn
`Interpolation actually
`on a template. These tests show that
`increases the spatial resolutionas well as the locatability of a fine
`point on a screen.
`
`t
`
`j
`
`i
`
`-.
`
`_,,
`
`column
`scale
`
`row scale
`
`(a) Straight lines drawn by the tablet using 3 by 3
`sensor array interpolation.
`The scales shown represent the boundaries or the
`actual sensors.
`
`colu:nn
`scale
`
`row scale
`
`(b) A circle drawn by the tablet using 3 by 3
`sensor array interpolation.
`The scales shown represent the boundaries or tho
`actual eensors.
`
`Fig 3 Points drawn by the tablet using an Interpolatlonmethod.
`
`23
`
`SCEA Ex. 1031 Page 3
`
`

`

`C H I
`
`' 8 5
`
`P R 0 C E E D I N G S
`
`A P R I L 1985
`
`Since the spatial resolution in the local interpofation scheme is lirr>
`lted by the number of bits available from the Intensitlesof an array
`in this scheme,
`of 3 by 3 sensors, other scheme was considered,
`all the points from a complete scan of a tablet are Interpolated
`allowing the potential resolution to be almost infinite. Howeverthis
`process simply emulates a projective device and accordingly
`reports only single polnt, which is interpolatedfrom all the points on
`the tabiet. However with this scheme, there are a great many ways
`of pointing to a specificlocatlonon a display screen, a feature with
`some Intriguing appilcation possibilitles.
`7.3 Response Time Delay
`The responsetime delay is the time delay from the beginning of a
`touch to an output received elther by local terminal or by an output
`device attached to the host computer. For multlpie totches, this
`delay will increasewith the numberof touches. The prototype used
`with a 9600 baud-rate terminal to measure ilme delays. Actual
`response times were measured several times and averaged for
`variouscases and are tabulatedin Table 1.
`
`Case
`
`best
`
`typical worst
`
`(a) pts/sec
`rnsec/pt
`(b) pts/see
`rnsec/pl
`(c) pts/sec
`rnsec/pt
`
`17.6
`56.8
`
`19.2
`52.1
`
`24.0
`41.6
`
`15.2
`65.6
`
`17.2
`56.1
`
`22.0
`45.5
`
`12.8
`76.1
`
`16.0
`62.5
`
`10.0
`53.2
`
`TABLE 1. Actual Response Time Delays
`
`The cases in Table one are to be Interpretedas follows:
`a one sensor touchedcontinuously
`b two sensorstouched at the sametime continuously
`c four sensorstouched at the same time continuously
`
`8. CONCLUSIONS
`A prototype of a fast-scanning multiple-touch-sensitiveinput tablet
`havingboth the adaptabilityand flexlbilityfor a broad range of appil-
`catlons has been designed and Implemented. Capacitancemeas-
`trement of Individual sensor(s) which can be ælquely addressed
`using two diodes per sensor, makes it possible to sense both the
`positions and Intensities of one or more simdtaneous touches
`without amblgdty. The sensor matrix is controlled by Universityof
`Toronto 6809 board whose serial port is connected to one of the
`I/O ports of a host computer. Software that utillzes the recursive
`subdivlWonalgorithmfor fast scannlng an array of 64 by 32 sensors
`on the tablet, and that communicateswith the host computer, has
`been Implemented and tested.
`
`9. ACKNOWLEDGEMENTS
`The research described in this paper has been fmded by the
`Natural Sclences and Engineering Research Council of Canada.
`This support is gratefully acknowledged.
`
`10. REFERENCES
`
`Brown, E., Buxton, W. & Murtagh, K. (1985).Windows on Tablets as
`a Means of Achieving Virtual inpti Devices. Computer Sys-
`tems Research Institute, University of Toronto.
`
`Buxton, W. (1982). Lexical and Pragmatic Considerations of input
`Structures,ComputerGraphics,17 (1), 31 - 37.
`
`24
`
`Burton, W., Hill, R. & Rowley, P. (1985).Issues and Techniques in
`Touch-Sensitive Tablet input, Computer Systems Research
`Institute,University of Toronto.
`
`Hillis, W.D. (1982),A High ResolutionImaging Touch Sensor, Inter-
`national Journal of Robotics Research, 1 (2), 33 - 44.
`Hurst, G. (1974). Electrographic Sensor for Determining Planar
`Coordinates, United State Patent 3,798,370,March 19, 1974,
`Elographics,Incorporated.
`
`JSR (1981), Pressure-Sensitive Conductive Rubber Data Sheet,
`Japan Synthetle Rubber Co., New Product Development
`Department,JSR Bullding, 2-11-24 Tfukljl, Chuo-Ku, Tokyo 104,
`Japan.
`
`(1984), A Fast Multiple-Touch-Sensitive Input Device,
`Lee, S.
`M.A.Sc. Thesis, Department of Electrical Engineering, Univer-
`sity of Toronto.
`Metha, N. (1982), A Flexible Machine Interface, MA.Sc. Thesis,
`Department of Electrical Engineering,Universityof Toronto.
`
`Sasakl. L., Fedorkow, G., Buxton, W., Retterath, C., & Smith, K.C.
`(1981). A Touch-SensitiveInput Device. Proceedings of the
`Fifth International Conference on Computer Music, North
`TexasState University, Denton, Texas, November,1981.
`
`TASA (1980), Model: x-y 3600 and x-y controller, Model: FR-105
`Data Sheet, Touch Activated Switch Arrays Inc., 1270
`Lawrence Station Road., Suite G., S?nyvale,CA 94089.
`
`TSD (1982), Touch Screen Digitizer Data Sheet, TSD Display Pro.
`dtets Inc., 35 Orville Drive, Bohemla, NY 1T/16.
`
`11. APPENDIX A: TOUCHTABLET SOURCES
`Big Briar: 3 by 3 Inch contintous pressure sensingtouch tablet
`
`Big Brlar, Inc.
`LeIcester, NC
`28748
`
`Chalk Board Inc.: "Power Pad",
`computers
`
`large touch table for micro-
`
`Chalk Board inc.
`3772 Pleasanidale Rd.,
`Atlanta, GA 30340
`
`Elographics: various sizes of touch tablets, including pressure sens-
`Ing
`
`Elographics,Inc.
`1976 Oak Ridge Turnpike
`Oak Rldge, Tennessee
`37830
`
`KoalaPad Technologies:Approx. 5 by 7 inch touch tablet for micro-
`computers
`
`Koala Technologies
`3100 Patrick Henry Drive
`Santa Clara, California
`95050
`
`Spiral Systems:Trazor Touch Panel, 3 by 3 Inch touch tablet
`
`Spiral System Instruments, Inc.
`4853 Cordell Avenue, Stite A-10
`Bethesda,Maryland
`20814
`
`SCEA Ex. 1031 Page 4
`
`

`

`C H I
`
`' 8 5
`
`P R 0 C E E D I N G S
`
`A P R l L 19 85
`
`TASA:4 by 4 inch touch tablet (relative sensing only)
`Touch ActivatedSwitch Arrays Inc.
`1270 Lawrence Stn. Road, Stdte G
`Strinyvale,California
`94089
`
`25
`
`SCEA Ex. 1031 Page 5
`
`