US007973773B2
`
`(12) United States Patent
`Pryor
`
`(10) Patent N0.:
`(45) Date of Patent:
`
`US 7,973,773 B2
`Jul. 5, 2011
`
`(54) MULTIPOINT, VIRTUAL CONTROL, AND
`FORCE BASED TOUCH SCREEN
`APPLICATIONS
`
`(76) Inventor: Timothy R. Pryor, Tecumseh (CA)
`
`( * )
`
`Notice:
`
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 775 days.
`
`(21) Appl.No.: 11/9s0,701
`
`(22) Filed:
`
`Oct. 31, 2007
`
`(65)
`
`Prior Publication Data
`
`US 2008/0129704 A1
`
`Jun. 5, 2008
`
`Related US. Application Data
`
`(63) Continuation of application No. 11/495,666, ?led on
`Jul. 31, 2006, noW Pat. No. 7,714,849, Which is a
`continuation of application No. 09/435,854, ?led
`on
`Nov. 8, 1999, noW Pat. No. 7,098,891, Which is a
`continuation of application No. 08/496,908, ?led
`on
`Jun. 29, 1995, noW Pat. No. 5,982,352.
`
`(51) Int. Cl.
`(2006.01)
`G09G 5/00
`(52) US. Cl. ................................... .. 345/173; 178/18.01
`(58) Field of Classi?cation Search ................ .. 345/173,
`345/204, 156, 157; 178/18.01; 348/746,
`348/747; 341/31, 32, 33, 34; 340/407.1,
`340/407.2; 463/30
`See application ?le for complete search history.
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`3,825,730 A
`7/1974 Worthington, Jr. et al.
`4,014,000 A
`3/1977 Uno et a1.
`4,146,924 A
`3/1979 Birk et a1.
`4,219,847 A
`8/1980 Pinkney et a1.
`4,305,071 A 12/1981 Bell etal.
`
`8/1982 Mallos
`4,346,376 A
`8/1983 DiMatteo et a1.
`4,396,945 A
`3/1984 Sakow et a1.
`4,435,835 A
`4,475,122 A 10/1984 Green
`4,484,179 A 11/1984 Kasday
`4,613,942 A
`9/1986 Chen
`4,629,319 A 12/1986 Clarke et a1.
`4,631,676 A 12/1986 Pugh
`4,644,100 A
`2/1987 Brenner et a1.
`4,686,374 A
`8/1987 Liptay-Wagner et al.
`4,710,760 A 12/1987 Kasday
`4,746,770 A
`5/1988 McAvinney
`4,885,565 A 12/1989 Embach
`4,988,981 A
`1/1991 Zimmerman et al.
`5,045,843 A
`9/1991 Hansen
`5,072,294 A 12/1991 Engle
`5,153,829 A 10/1992 Furuya et al.
`5,162,618 A * 11/1992 Knowles .................. .. 178/1804
`5,168,531 A 12/1992 Sigel
`5,227,985 A
`7/1993 DeMenthon
`5,252,951 A 10/1993 Tannenbaum et al.
`5,412,189
`5/1995 Cragun
`A
`(Continued)
`
`DE
`
`FOREIGN PATENT DOCUMENTS
`4323863
`l/l995
`(Continued)
`
`Primary Examiner * Abbas Abdulselam
`(74) Attorney, Agent, or Firm * Warner Norcross & Judd
`LLP
`
`ABSTRACT
`(57)
`A method and apparatus are provided providing a touch
`screen having multipoint sensing and/or force based sensing
`and feedback capability useful in many applications Which
`extend beyond traditional computer applications. The touch
`screen can be located in many non-traditional locations as
`Well, such as desks, tables, Walls, vehicles, and the like. The
`apparatus may be used by a single user, or multiple users,
`employing ?ngers, hands, feet and other body portions, if
`desired or practical. Related applications for virtual image or
`physical control applications are also disclosed.
`
`42 Claims, 17 Drawing Sheets
`
`170
`
`INTENSITY
`0F LINE AA
`
`VTHRESHOLD
`
`SIGNIFYING TWO INPUTS
`AT POSITION x, y - ONE
`LARGER THAN THE OTHER
`Ie., Z1>Z2
`
`Samsung USP 7,973,773
` Exhibit 1001 Page 1
`
`

`
`US 7,973,773 B2
`Page 2
`
`US. PATENT DOCUMENTS
`5,459,793 A 10/1995 Naoietal.
`5,495,269 A
`2/1996 Elrod etal.
`5,559,301 A
`9/1996 Bryan, Jr. e161.
`5,835,080 A * 11/1998 Beeteson e161. ............ .. 345/173
`2009/0267921 Al* 10/2009 Pryor .......................... .. 345/177
`
`JP
`JP
`WO
`
`FOREIGN PATENT DOCUMENTS
`02-105919
`4/1990
`04-137419
`5/1992
`92/00559
`1/1992
`
`* cited by examiner
`
` Exhibit 1001 Page 2
`
`

`
`US. Patent
`
`Jul. 5, 2011
`
`Sheet 1 0117
`
`US 7,973,773 B2
`
` Exhibit 1001 Page 3
`
`

`
`US. Patent
`
`Jul. 5, 2011
`
`Sheet 2 0f 17
`
`US 7,973,773 B2
`
`$625,;
`
`<< wz: “6
`
`
`
`7777 .. :mémwwwwww .,
`
`2 I
`
`mm?
`
`ow?
`
`N 6E
`
` Exhibit 1001 Page 4
`
`

`
`US. Patent
`
`Jul. 5, 2011
`
`Sheet 3 0f 17
`
`US 7,973,773 B2
`
`_ _
`
`_ _ _
`
`qm 6E
`
`u /.
`
`m0 ZOFOwEO
`
`29.20206 X45
`
`mm 6E
`
`$13K
`
`0m GI
`
`m .QI
`
` Exhibit 1001 Page 5
`
`

`
`US. Patent
`
`Jul. 5, 2011
`
`Sheet 4 0f 17
`
`US 7,973,773 B2
`
`TOUCH. HIT. OR IMPACT
`SCREEN/PAD
`
`v
`DETERMINE LOCATION,
`MACNITUDE, (AND DIR
`ECTION. IF DESIRED) OF
`'NPUT
`
`I,
`EXECUTE PROGRAM IN
`RESPONSE To DATA
`
`D-SIGHT MEASUREMENT
`SEQUENCE
`ILLUMINATE REAR OF SCREEN
`(TYPICALLY FLAT AND
`REFLECTIVE)
`
`TV CAMERA IMAGES LIGHT
`RETURNED TO SCREEN BY
`RETROREFLECTOR
`
`PROCESS CAMERA IMAGE
`TO DETERMINE LOCATION
`AND MACNITUDE OF
`SCREEN DEFLECTIQN
`
`,,
`CORRECT WITH MULTIDIMEN
`SIONAL CORRECTION TABLE,
`IF DESIRED
`
`IF TOUCH PAD RATHER THAN TOUCH
`C SCREEN (OR IF vIDEO NOT VARIED)
`I
`:
`,
`,
`I
`I
`I
`I
`:
`I
`I
`I
`< --------- - -
`
`________________ .. .. ,
`
`CALL NEW vIDEO SEQUENCE
`FROM MASS STORAGE, IF
`DESIRED
`
`=
`
`‘
`
`FORMAT vIDEO AND DISPLAY
`RESULT TO USER
`
`II
`SOLICIT NEW USER INPUT IF
`DESIRED, AND REPEAT
`
` Exhibit 1001 Page 6
`
`

`
`U.S. Patent
`
`Jul. 5, 2011
`
`Sheet 5 of 17
`
`US 7,973,773 B2
`
`§\\\\\\\\\\\\\\\\\\§
`
`mGE
`
`.._Omwmmemom50o._.mgxmzozfimm
`
`
`an VOB_2o8mE
`
`zaozx15>wmomfim2:ha%-mmodagesmonsmmod
`
`>._mS__>__m@2586lomm§E<~mEo
`
`
`Bmsomm2858,:
`
`
`
`zoEzoomm._.m_0m<._.
`
`mommmuomn.
`
`Exhibit 1001 Page 7
`
` Exhibit 1001 Page 7
`
`
`
`
`
`

`
`US. Patent
`
`Jul. 5, 2011
`
`Sheet 6 0f 17
`
`US 7,973,773 B2
`
`601
`
`610x
`
`1
`
`//
`
`609
`
`TO GRID PATTERN IMAGE DEFORMA
`TION ANALYSIS COMPUTER
`
`650
`
`651
`
`FIG. 6
`
` Exhibit 1001 Page 8
`
`

`
`US. Patent
`
`Jul. 5,2011
`
`Sheet 7 0f 17
`
`US 7,973,773 B2
`
`v NNN
`
`\\ an
`
`N @E
`
` Exhibit 1001 Page 9
`
`

`
`U.S. Patent
`
`Jul. 5, 2011
`
`Sheet 8 of 17
`
`US 7,973,773 B2
`
`-n
`
`:--u————————unu-unn-
`
`--—--——
`unu-——an-—-——————na———————————-—-nu--u——————————-an--an-un-————————---u——nun»-——————————nuan-
`
`801
`
`Exhibit 1001 Page 10
`
` Exhibit 1001 Page 10
`
`

`
`US. Patent
`
`Jul. 5, 2011
`
`Sheet 9 0f 17
`
`US 7,973,773 B2
`
`6 3
`40/
`
`FIG. 9a
`(' 949
`
`920
`
`FIG. 9b
`
` Exhibit 1001 Page 11
`
`

`
`US. Patent
`
`Jul. 5, 2011
`
`Sheet 10 0f 17
`
`US 7,973,773 B2
`
`.22
`
`
`
`801 081
`
`30?
`m 82.
`
`2 .QE
`
`
`
`52 £22., mm
`
`Bop gone?
`
`mg? \ omow
`
`801
`
`:1?
`
`/
`
` Exhibit 1001 Page 12
`
`

`
`US. Patent
`
`Jul. 5, 2011
`
`Sheet 11 0117
`
`US 7,973,773 B2
`
`1320
`
`1100
`
`FIG. 11
`
` Exhibit 1001 Page 13
`
`

`
`US. Patent
`
`Jul. 5, 2011
`
`Sheet 12 0117
`
`US 7,973,773 B2
`
`1215
`
`FIG. 12
`
`1200\
`
`1201
`
` Exhibit 1001 Page 14
`
`

`
`US. Patent
`
`Jul. 5, 2011
`
`Sheet 13 0f 17
`
`US 7,973,773 B2
`
`/
`
`\
`
`1325
`
`1305 _ 1301
`
`\_ 7V
`
`1320
`
`1321
`1325
`
`W
`
`1305
`J
`
`1301
`
`J
`
`1310
`
`L
`
`FIG. 13
`
` Exhibit 1001 Page 15
`
`

`
`US. Patent
`
`Jul. 5, 2011
`
`Sheet 14 0f 17
`
`US 7,973,773 B2
`
`( 1400
`
`1410
`
`.4. lb A A
`
`)
`143s
`
`N1450
`
`1411
`
`FIG. 14
`
`
`
`,L‘iZB/ - 1400
`
` Exhibit 1001 Page 16
`
`

`
`US. Patent
`
`Jul. 5, 2011
`
`Sheet 15 0f 17
`
`US 7,973,773 B2
`
`1500\
`
`1525
`
`Ci: N520
`
`1510
`
`15g
`
`' \ 1511
`
`1540
`
`1531/
`
`///////////////////
`///////////////// /////
`
`FIG. 15
`
` Exhibit 1001 Page 17
`
`

`
`US. Patent
`
`Jul. 5, 2011
`
`Sheet 16 0117
`
`US 7,973,773 B2
`
`1660
`
`1661
`
`(-1670
`
`F////////// 51655
`p//////V/_
`
`FIG. 16
`
` Exhibit 1001 Page 18
`
`

`
`US. Patent
`
`Jul. 5, 2011
`
`Sheet 17 0117
`
`US 7,973,773 B2
`
`1710
`
`1700
`
`FIG. 17
`
` Exhibit 1001 Page 19
`
`

`
`US 7,973,773 B2
`
`1
`MULTIPOINT, VIRTUAL CONTROL, AND
`FORCE BASED TOUCH SCREEN
`APPLICATIONS
`
`CROSS REFERENCE TO RELATED
`APPLICATIONS
`
`This application is a continuation of application Ser. No.
`11/495,666 ?led Jul. 31, 2006, now US. Pat. No. 7,714,849;
`Which is a continuation of application Ser. No. 09/435,854
`?led Nov. 8, 1999, now US. Pat. No. 7,098,891; Which is a
`continuation of application Ser. No. 08/496,908 ?led Jun. 29,
`1995, now US. Pat. No. 5,982,352. All of these prior appli
`cations are hereby incorporated by reference.
`The disclosures of the following US. patents and patent
`applications are also incorporated herein by reference:
`US. Pat. No. 4,629,319 (“D-SIGHT”)
`US. Pat. No. 4,394,683 (“Circuits”)
`US. Pat. No. 4,373,804 (“Turbine”)
`U.S. Ser. No. 08/290,516 ?led Aug. 15, 1994 (“Man
`Machine Interface”)
`U.S. Ser. No. 08/203,603, ?led Feb. 28, 1994 (“Robot
`Vision Using Target Holes .
`.
`. ”)
`U.S. Ser. No. 07/664,574, ?led Mar. 6, 1991 (“Targets II’’)
`US. Ser. No. 07/875,282, ?led Apr. 29, 1992 (“Vision
`Target Based Assembly”)
`U.S. Ser. No. 08/161,304, ?led Dec. 2, 1993 (“Controlled
`Machining”)
`
`INTRODUCTION
`
`The invention disclosed herein is a neW type of data entry
`device for computers and other electronic equipment gener
`ally in the category of digitiZers and touch screens having
`several unique properties. It is based primarily on the electro
`optical determination of temporary surface distortion caused
`by the physical input signal, or force, creating the distortion
`(eg a ?nger “touch”). This
`is herein referred to as surface
`distortion imaging and depends on the ability to detect, and in
`some cases quantify, small distortions of a surface over a large
`(by comparison) area.
`distortion is that
`A preferred means of detecting surface
`given in reference 1, Which discloses illumination of a surface
`and subsequent retrore?ective re-illumination of the surface
`from Which an enhanced image of the distortion in such
`surface are created. This method (and the products based
`thereon sold under the trade name “D-SIGHTTM”), is at once,
`simple, fast, and capable of intelligibly measuring minute
`distortions over large surface areas. All of these are advan
`tages for the present disclosed invention, and D-SIGHTTM is
`the preferred method (but not the only method) for determin
`ing such distortions. Other optical techniques are grid and
`moire triangulation, also providing surface distortion data.
`Distortion in a material (rather than a surface thereof), can
`alternatively be used, detected by schlieren, transmissive
`D-SIGHT, and in photoelastic stress based techniques, rely
`ing on stress related differential refraction in the material,
`rather than de?ection of the surface. In both cases video
`cameras scanning the image of the area of the material or
`surface are the preferred transduction device
`Also disclosed herein are novel means to determine other
`events Which cooperatively or individually may a be imputed
`to a computer by means of the invention. These particularly
`concern electro-optically determinable datums on persons or
`other entry means.
`
`REVIEW OF THE PRIOR ART
`
`The typical data entry device for a computer to date, has
`been a keyboard. More recently the “mouse” and “joy stick”
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`2
`have been devised to alloW entry of data and particularly
`picture type data, onto a computer screen.
`Tracing tablets (digitiZers) have been derived using various
`technologies for indicating for example, the X, Y location of
`a pencil stylus, using ultrasonic, inductive or other means.
`In addition to the above, such data entry can be combined
`With a display in a product commonly knoWn as a “touch
`screen”. In this product, data is presented on a TV screen and
`the human can touch certain boxes typically Which have been
`encoded on the screen to register his input data choice.
`In regard to touch screens, these are generally categoriZed
`as being either of the non-contact beam-break type usually
`using multiple light sources or, a type employing multi layer
`overlays using optical, acoustic, or capacitive phenomenon to
`determine a location of the touching member.
`A brief revieW of the prior art relative to touch screen
`technology is given in US. Pat. No. 4,675,569 by BoWman.
`BoWman discloses a touch screen technology using a beZel
`With pieZo-electric elements at the four corners Which, upon
`being stressed by a touch on a glass faceplate for example,
`creates force signals Which can be used to decipher the X,Y
`location of the pressed point. Presumably this technology
`could also be used for determination of 3-D force variables as
`Well.
`Disadvantages of previous touch screens Which are pur
`ported to be overcome in part at least by the BoWman inven
`tion, are accuracy, shock, Wear, reliability and electro mag
`netic radiation.
`Other prior art technology (Touch screen or digitiZer)
`relates to capacitive devices in Which one plate is pressed
`closer to another at a different point and related by a grid scan
`mechanism and to scanned contact types Wherein a grid scan
`mechanism and to scanned contact types Wherein a grid of
`
`conductors (either ?ber optic or electrical), are
`caused to be
`contacted at one point Which again can be scanned out.
`Other touch screen technology (US. Pat. No. 4,700,176)
`uses surface Waves induced in a material Which are damped at
`a given location in space due to the touching arrangement.
`US. Pat. No. 4,740,781 describe conductive ?lm based touch
`screens. US. Pat. No. 4,710,758 addresses the problem of
`calibration of all touch screens, particularly a problem for
`those based on analog principles. Problems of electro mag
`netic shielding of touch screens Which can be a problem in
`secure environments are addressed, for example, in US. Pat.
`Nos. 4,692,809 and 4,591,710.
`Where one admits to a conductive stylus or other special
`Writing instrument, then higher resolution transmissive digi
`tiZing screens can be contemplated such as that of US. Pat.
`No. 4,639,720. Other “digitiZers” not incorporating a screen
`display are represented by US. Pat. No. 3,692,936 describing
`an acoustic digitiZer pad, US. Pat. No. 4,177,354 describing
`a digitiZer With a light responsive grid, and US. Pat. No.
`4,255,617 describing a digitiZer With a capacitive grid.
`US. Pat. No. 4,736,191 describes a digitiZer pad Which is
`a type of digitiZer capable of providing a X,Y
`and Z axis
`indication proportional to the area of touch, a third dimension
`of sorts.
`No knoWn prior art exists in the area of data entry devices
`based, like the instant invention, on optical surface distortion
`measurement.
`In general, it can be said that all of these prior art devices
`typically are one or tWo dimensional, that is, they either
`register a single command as in a typeWriter key or the XY
`location of a command as for example a light pen location on
`a screen or a stylus point, etc. It is therefore very desirable to
`have a three dimensional capability, capable of regi stering not
`only the X andY but also the Z value of a force or displace
`
` Exhibit 1001 Page 20
`
`

`
`US 7,973,773 B2
`
`3
`ment caused by a particular entry connnand. No known com-
`mercial devices can do this. and a limited technology set
`exists for this purpose—cspecially over large extensive
`screen or pad areas.
`In addition. conventional technologies typically limit the
`resolution or the size or both of the display to which entry
`could be made. For example, touch screen data entry is com-
`monly available over a standard let us say 12" to 19" computer
`terminal to the level of 1 part in 40 in both X and Y. While this
`suffices for many computer data entry purposes (e.g. selecting
`icons), it is certainly not suitable for l1igl1 resolution drawing
`on a screen or other such activities. Prior art is lacking which
`can accommodate high resolution “touch" or other inputs
`easily over large surfaces such as for example data displays in
`military war rooms and the like. In addition, high resolution
`seems possible in prior art digitizers only by moving special
`apparatus or using special writing instruments such as a con-
`ductive pen, and high resolution touch screens are difiicult
`with digital technologies such as the discrete grids. Such grids
`also min the risk of degrading the ligl1t transmission charac-
`teristics of the screen.
`Another drawback ofmost conventional data entry systems
`today is that they can only respond to one point at a time. In
`other words, a single finger on the screen, a single mouse
`location, a single joy stick entry, a single key on a keyboard.
`In many applications it would be desirable to be able to enter
`more than one point at a time or rock back and forth between
`two points or the like. Indeed it may be desirable to enter not
`just poi11ts but a complete “signature” as in a hand print or the
`equivalent. This is very useful for recognizing inputs from
`disable persons, or as a means of verifying authenticity.
`Accuracy is also a problem with most digiti7ers and touch
`screens, in particular those using analog principles (e.g. the
`Bowman reference above). Indeed for those digitizers and
`touch screens based for low cost or other reasons on analog
`transduction technologies, calibration is often required. One
`implication is that icon size on the screen must be large. if
`calibration can’t be maintained.
`There is virtually no evidence of 3-D capability in the
`finger touch devices. The closest art found is that ofcapacitive
`change in area due to contact with the touch panel.
`One priorart device (US. Pat. No. 4,639,720) describes an
`important capability of drawing directly on the screen with
`commonly used or available instrtuncnt (e.g. a pencil). This is
`a key item in a man-machine interface equation, getting away
`from the artifice of drawing pictures with a mouse let us say
`while looking at the screen (the technology disclosed herein
`also allows one to draw on the screen directly).
`AL)VAN'lAGES OF THE IN Vl-_<'N'l'l(,)N
`
`The disclosed invention at one and the same time obviates
`the difficulties above in a manner that is also cost effective. In
`addition, it contains several unique advantages not known to
`exist elsewhere, viz.;
`l, A potential “four” and “live dimensional” capability,
`wherein the force vector direction as well as the magnitude of
`force is measured.
`2, An ability to detect dynamic events over a very large
`area, also with temporary data storage.
`3. An ability to have a data storage of a complete signature
`at once, physically or in memory. The invention has a unique
`dynamic detection ability due to its image storage capability.
`No dynamic event detection is apparent in the prior art. and
`few prior art touch screens, even appear capable of transient
`or dynamic operation. The invention on the other hand can be
`used with strobed light sources which can be triggered to
`
`10
`
`tou.
`
`45
`
`S0
`
`91m
`
`60
`
`65
`
`4
`capture fast transient events. Even wl1en successive readings
`are required, several thousand readings a second can be
`obtained of the complete surface. Transient events can be
`stored by the image capture medium and in some cases can be
`actually stored by the sensing surface if it has a hysteresis
`“memory”. This allows it to be used for dynamic “hits” such
`as those of projectiles on the screen, not just continuous
`touches.
`4. An ability to have the surface distortion or touching input
`means of any material, completely removed from the actual
`sensing of the input. Specialized capacitive screens and the
`like are not required. However, an LCD display screen can for
`example, be used to form a portion of the surface.
`5. In addition, the invention is extremely cost competitive
`to other touch screen or data entry techniqucs—particularly
`for larger surfaces. (For example, one meter square and
`larger.) "he resolution obtainable in these larger surfaces is
`unmatched, being capable with today’s technology, ofreacl1—
`ing grea er than one part in 10,000 in each direction of the
`surface (100 million resolvable points on the surface).
`6. Un ike mos other types of displays, several embodi-
`ments ofthe disclosed invention give a desirable tactile feed-
`back since it is the actual physical deformation (and the
`amount hereof)
`l1at is responsive. Thus the feedback to a
`finger (or other member) ir1 terms ofresistive force is propor-
`tional to he desired input. This tactile feedback is particularly
`desirable in for example the automobile where one should not
`take one’s eyes 0 "the road.
`7. Another advantage of the disclosed invention is that it
`can crea e a touch screen data entry of very high resolution
`with suc 1 entry made directly on the screen (not on a special
`pad, as in most CAD systems) with the “beam” ofthe CRT or
`other type ofdisplay literally following the touch pointjust as
`a pencil line would follow a pencil. In this application the 3-D
`capability allows one to press harder and make a darker (or
`wider) line for example, just as one would do in normal
`practice.
`The capability of the invention to be ergonomically and
`“naturally" compatible with human data entry is a major
`feature of the invention.
`8. The reliability of some of the touch screen prior art is
`questionable. Capacitive devices in close contact are subject
`to wear, humidity, electric fields and other variables for
`example. In addition. many prior art touch screens are of
`speciali7ed construction and would be quite expensive to
`replace if they were broken, especially as the size increases.
`In the case of the invention, sensing of the screen is non
`contact, and the sensing screen can be as simple as a piece of
`plate glass, or a wall.
`Many touch screen designs appear to have problems con-
`nected with electro magnetic radiation and can pose a prob-
`len1 in high security areas. This problem does not exist with
`the disclosed invention.
`9. Multipoint Operation. Many ofthe existing touch screen
`prior art are capable only of measuring one touch point in X
`a11d Y at a time, While some other prior art designs would not
`appear to preclude n1ulti-point simultaneous measurement,
`none apparently is disclosed. The invention is easily capable
`of multi-point operation or even detection of complex area
`“signatures” notjust “points”.
`As an example of the n1ulti—poi11t difficulties with the prior
`art, the light curtain type non-contact touch screens clearly
`have an obscuration problem, as the finger indicating one
`point obscures the view of another.
`10. Reflection and Transmission. The invention, unlike
`most of the prior art, can be used botl1 in reflection and for
`transmission measurement of deformation. The camera sys-
`
`Exhibit 1001 Page 21
`
` Exhibit 1001 Page 21
`
`

`
`US 7,973,773 B2
`
`5
`tem used in the device can be used for otl1er purposes as when
`and indeed the touch screen or digitizing system disclosed
`can be used simultaneously with prior art systems for a com-
`bination eflect if desired.
`A further advantage oftlie inventions ability to detect n1ul-
`tiple input signatures, etc. at any point on its face, therefore a
`keyboard, a piano keyboard, a joy stick can be artificially
`created at any point under computer control or simply by
`random human command. This is a particular desirability in a
`car where you cannot necessarily keep your eye on the data
`entry device or for example for handicapped people who
`could not be expected to hit the right point on the device every
`time, but if they just hit the device anywhere, could make a
`move from that point in a mamier that would be intelligible to
`a computer for example.
`ll. Common Systems. In addition to the above advantages
`over the prior art, the invention also has an advantage that it
`employs essentially standzud hardware for any screen size.
`The same technology is applicable for a screen or “pad” ofsay
`3"><4" (8><l0 cm) such as might be in a car dashboard all the
`way to screens or digitizers; the size ofa wall. This allows the
`cost to be reduced as the technology can be shared.
`12. Variable and “Intelligent” orientation. It is also useful
`therefore to replace in many cases keyboards wl1icl1 have
`continuous arrays of keys, be they optical, mechanical, con-
`tact, electro mechanical or whatever. Unlike most keyboards
`the disclosed type can “float” (i.e. be at any zone on the
`surface) which is convenient for people W110 know how to
`type but cannot see the keys for example, while driving.
`l3. Tactile feedback, including progranmiable. The unique
`tactile feedback application aspect ofthe invention allows one
`to essentially use a deformable member as a sort of miniature
`joy stick for each finger or to rock back and forth between one
`or more fingers to create dilTerent signals. In addition, pro-
`grammable tactile feedback such as air blasts, vibration, etc..
`can also be added easily to the touch surface.
`14. Another advantage of the invention is that it can detect
`a force or displacement signature such as of an object that
`would be picked up by a robot hand. Of interest as well is the
`ability to sense the signature of someone. even one who
`would enter a signature of his pahn or what have you. This
`may be of considerable use to the blind a11d other physically
`disabled persons, allowing use of non-conventional inputs
`(e.g. elbows toes, etc) and the tactile feedback afforded is
`particularly helpful here.
`15. In a gaming and simulation context, the invention l1as
`the advantage that it is low in cost, and provides a method by
`which the game player can participate in a physical way, just
`as in real sports, and the like.
`l-'urther disclosed in this invention is a variation on the
`invention for use where inputs other than physical touching
`are used to temporarily deform the medium. This can be TV,
`thermal, air, vacuum, electro-magnetic—or any other physi-
`cal force which can temporarily either distort the surface in
`any direction.
`The ability ofthe invention to detect temporary distortions
`also leads it to be usable over broad areas for unique trans-
`duction applications such as weighing trucks on the highway
`“on the fly”. counting lumps ofcoal on a specialized conveyor
`belt, counting/weighing and other such applications, or in any
`other application where transient deflections are the desired
`signal.
`These and other advantages of the invention will become
`clear in the following disclosure which is depicted in the
`following figures.
`Summary of Advantages To conclude, the invention dis-
`closed herein has numerous advantages, for example:
`
`10
`
`tou.
`
`45
`
`S0
`
`U1m
`
`60
`
`65
`
`6
`1. 3-D MYX capability, plus an additional 2-D of vector
`input.
`2. Simplicity and Low cost, particularly for large area
`applications.
`3. Multi—point. and area signature capability, also with
`simultaneous detection
`4. Non-contact sensing of screen deformation due to
`“to uch”—wear free and reliable
`5. Excellent dynamic detection of both transients and
`repeating inputs
`6. High accuracy—all digital sensing, no analog calibra-
`tion required
`7. Achievable resolution 1/m,oon in both X &Y, via sub pixel
`digiti7ation 8. Screen/plate material
`independent. For
`example, can generate a surface response on program com-
`mand, or use transparent media such as rear projection
`screens.
`9. (‘,ommonality—all screen or pad si7es can use substan-
`tially the same hardware
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`The invention will further be described in the following
`figures:
`SIG. 1 is a prior art touch scree11 or digitizer having con-
`ducting bars in an orthogonal grid.
`SIG. 2 is a basic embodiment of the invention in touch
`screen fonn utilized with a projection TV (front or rear), and
`employing a D-SIGHT transduction of surface distortion.
`SIG. 3 illustrates sensing of force (displacement) vectors
`using the FIG. 2 device, and includes a hand print evaluation
`aspect ofuse to the handicapped.
`SIG. 4 illustrates a block diagram of one embodiment of
`the invention
`SIG. 5 illustrates a combined system employing a target
`tracking embodiment of the invention and touch screen
`embodiment similar to FIG. 2 for use with projectiles.
`SIG. 6 illustrates a multipoint, or signature version of the
`invention, in a virtual environment situation. Also illustrated
`is the use of grid projection triangulation for surface distor-
`tion analysis, and a multi-segment version of the invention
`SIG. 7 illustrates a “digitizer” pad embodiment of the
`invention used as an automotive dashboard input, further
`incorporating tactile feedback cues, both passive and active,
`via piezo vibrators, air blasts, sound waves. or the like.
`SIG. 8 illustrates a force field (electromagnetic, sound, or
`other) embodiment ofthe invention, using a moire grid trans-
`duction of surface distortion
`SIGS. 9a and 9b illustrate transmission and photoelastic
`variants of the invention. Also illustrated is a photoelastic
`based sensing of force location due to stress.
`SIG. 10 illustrates further the use of the invention for
`dynamic projectile detection, and methods thereof used in a
`single or multiperson sports gaming simulator embodiment
`designed for sports bars or home use.
`SIG. 11 is a D-SIGHT primary image embodiment of the
`invention
`SIG. 12 is a Multiple simulator application type panel.
`including programmable knob or other features, and overlays
`using the multi point and tactile advantage of the invention
`SIG. 13 is and additional overlay embodiment ofthe i11ven-
`lion
`SIG. 14 is an embodiment of the invention having deform-
`ing bladders for touch and feel.
`SIG. 15 illustrates a stereoscopic defiection determination,
`and correction matrix for the screen of the invention. Also
`illustrated is the case in which the screen is locally distorted
`
`Exhibit 1001 Page 22
`
` Exhibit 1001 Page 22
`
`

`
`US 7,973,773 B2
`
`7
`so as to impact tl1e ligl1t transmission or reflection from the
`screen from external sources not used in the TV display
`function.
`FIG. 1 6 is a screen location calibration system ofthe inven-
`tion
`FIG. 17 illustrates a non optical embodiment for screen
`deflection determination—in this case radar based, which is
`optionally or altematively capable of seeing an thrown object.
`DETAILED DESCRIPTION OF THE INVENTION
`
`10
`
`8
`nation source. The screen can be plain, or have writing or
`designs on it in certain areas where, for example, input of a
`certain type is desired.
`The preferred method of determining screen deformation
`shown hi this embodiment, is to use the D—SIGHTTM principle
`described in the aforementioned U.S. patent (ref 1). In this
`example light source 170 illuminates the rear smooth portion
`102 of the screen 101 and light from that smooth portion is
`reflected to retroreflector material 171 (typically Scotchlight
`7615 glass beaded material by 3M company) which then
`re-reflects it to the surface 102 and thence to TV camera 160.
`The light source 170 can be located offthe axis ofthe TV
`camera lens as shown, which creates a unique contour type
`“D—SIGHT image”, providing a shading effect of any distor-
`tion present which indicates the direction, magnitude and
`shape of the distorted portion of the surface. Alternatively
`however the light source can be located essentially on the axis
`of the camera often (through use of a beam splitter or otl1er—
`wise) and in wl1ich case the TV camera input detects a dark
`zone when the screen is pushed in by the action of the force
`(e.g. a finger “touch"). Detection of these types of images in
`the context of this application is further described below.
`For example, in pressing on back of painted steel 50 inches
`square and 0.030" thick secured on its edges, one can see
`effect of finger moving rapicly, and can see “Z” deformation
`local to ones finger by the increasing image darkness and spot
`size at the finger point The e ect seems reasonably localized
`size of indication about size 0.5 inch in diameter as seen or1
`TV screen display of the D-SIGHT image of the steel sheet.
`At edge of panel near point of support on edges panel stron-
`ger, didn’t deflect near as much.
`Rubber and latex provide extremely local surface defom1a—
`tion But latex as a writing pad is not be as nice as steel, plastic
`or glass with a hard surface. To improve screen or pad 1nate-
`rial properties, composite material combinations can be used.
`e.g. Rubber impregnated plastic, thin glass or plastic sheets
`with transparent stiffeners, etc.
`To operate the invention, the camera is typically scanned
`out at nonnal TV frame rates 30 times or 60 times a second,
`and the determination of the XY location of the force input
`determined from the centroid of the deflected disturbed light
`energy. This detection is accomplished ir1 either hardware or
`software. In the simplest “On-Axis” case (where the light
`source and camera are effectively co-axial’), the centroid “C
`in X and Y of dark energy, 200, shown in the representation
`201 of the o11—axis D—SIGHT image of stucface 102,
`is
`detected. The methods for accomplishing this detection and
`for providing, ifdesired, real time outputs, that is, every frame
`rate, are described in detail in U.S. patents of reference 2,
`which utilize the detection ofthe first or second derivative. the
`moment, or the average light intensity centroid for example.
`The XY coordinates are easily determined by finding the
`XY centroid location. This can typically be done to resolu-
`tions less than the actual pixel spacing on the camera array
`with 1/10 of a pixel being commonplace as described in the
`references. In this case we could see that for a 1000x1000
`array, the sensitivity in XY direction can be 1 part in 10,000 of
`the screen field—e.g. 0.1 mm (0.004":) on a 1 meter