Case 2:21-cv-00041-JRG Document 1-2 Filed 02/04/21 Page 1 of 37 PageID #: 61
`Case 2:21-cv-00041-JRG Document 1-2 Filed 02/04/21 Page 1 of 37 PageID #: 61
`
`
`EXHIBIT B
`
`
`EXHIBIT B
`
`
`
`

`

`USOO7933431B2
`
`(12) United States Patent
`Pryor
`
`(10) Patent No.:
`(45) Date of Patent:
`
`US 7,933,431 B2
`Apr. 26, 2011
`
`(54) CAMERA BASED SENSING IN HANDHELD,
`MOBILE, GAMING, OR OTHER DEVICES
`(76) Inventor: Timothy R. Pryor, Tecumseh (CA)
`(*) Notice:
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 0 days.
`
`(21) Appl. No.: 12/834,281
`
`(22) Filed:
`
`Jul. 12, 2010
`
`(65)
`
`Prior Publication Data
`US 2010/02774 12 A1
`Nov. 4, 2010
`
`Related U.S. Application Data
`(63) Continuation of application No. 1 1/980,710, filed on
`Oct. 31, 2007, now Pat. No. 7,756.297, which is a
`continuation of application No. 10/893.534, filed on
`Jul. 19, 2004, now Pat. No. 7,401,783, which is a
`continuation of application No. 09/612.225, filed on
`Jul. 7, 2000, now Pat. No. 6,766,036.
`(60) Provisional application No. 60/142,777, filed on Jul. 8,
`1999.
`(51) Int. Cl.
`G06K 9/00
`
`(2006.01)
`
`(52) U.S. Cl. ...................... 382/103: 382/104; 348/207.1
`(58) Field of Classification Search .................. 382/103,
`382/104; 348/207.1
`See application file for complete search history.
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`5,878,174 A * 3/1999 Stewart et al. ................ 382,293
`6,342,917 B1 *
`1/2002 Amenta .........
`... 348,207.1
`6,453,180 B1* 9/2002 Endoh et al. .................. 455,567
`6,597,817 B1 * 7/2003 Silverbrook ...
`... 382.289
`* cited by examiner
`
`Primary Examiner — Tom Y Lu
`(74) Attorney, Agent, or Firm — Warner Norcross & Judd
`LLP
`
`ABSTRACT
`(57)
`Method and apparatus are disclosed to enable rapid TV cam
`era and computer based sensing in many practical applica
`tions, including, but not limited to, handheld devices, cars,
`and video games. Several unique forms of social video games
`are disclosed.
`
`31 Claims, 22 Drawing Sheets
`
`816
`
`830
`
`(S 815
`s
`
`
`
`Case 2:21-cv-00041-JRG Document 1-2 Filed 02/04/21 Page 2 of 37 PageID #: 62
`
`803
`
`824
`
`853
`
`851
`
`852
`
`850
`
`849
`
`

`

`U.S. Patent
`
`Apr. 26, 2011
`
`Sheet 1 of 22
`
`US 7,933,431 B2
`
`
`
`138
`
`147
`
`148
`
`Nias
`
`160
`
`165
`
`164
`
`166
`
`\167
`
`Case 2:21-cv-00041-JRG Document 1-2 Filed 02/04/21 Page 3 of 37 PageID #: 63
`
`FIG. 1B
`
`181
`
`180
`
`183
`
`182
`
`

`

`U.S. Patent
`
`Apr. 26, 2011
`
`Sheet 2 of 22
`
`US 7.933,431 B2
`
`1000t
`
`y
`ELEMENTS
`
`
`
`
`
`
`
`
`
`205
`
`1000
`
`500
`X
`ELEMENTS
`
`COMPUTER
`
`240
`-1
`
`Case 2:21-cv-00041-JRG Document 1-2 Filed 02/04/21 Page 4 of 37 PageID #: 64
`
`251
`250
`
`

`

`U.S. Patent
`
`Apr. 26, 2011
`
`Sheet 3 of 22
`
`US 7,933,431 B2
`
`220
`
`COMPUTER
`
`275
`
`274
`
`Case 2:21-cv-00041-JRG Document 1-2 Filed 02/04/21 Page 5 of 37 PageID #: 65
`
`
`
`271
`
`FIG. 2C
`
`

`

`U.S. Patent
`
`Apr. 26, 2011
`
`Sheet 4 of 22
`
`US 7,933,431 B2
`
`
`
`2. N
`
`405
`
`406
`
`Case 2:21-cv-00041-JRG Document 1-2 Filed 02/04/21 Page 6 of 37 PageID #: 66
`
`FIG. 4A
`
`

`

`U.S. Patent
`
`Apr. 26, 2011
`
`Sheet 5 of 22
`
`US 7,933,431 B2
`
`- 320'
`
`302
`
`301
`
`
`
`Case 2:21-cv-00041-JRG Document 1-2 Filed 02/04/21 Page 7 of 37 PageID #: 67
`
`445
`
`

`

`U.S. Patent
`
`Apr. 26, 2011
`
`Sheet 6 of 22
`
`US 7.933,431 B2
`
`
`
`Case 2:21-cv-00041-JRG Document 1-2 Filed 02/04/21 Page 8 of 37 PageID #: 68
`
`

`

`U.S. Patent
`
`Apr. 26, 2011
`
`Sheet 7 of 22
`
`US 7,933,431 B2
`
`
`
`Case 2:21-cv-00041-JRG Document 1-2 Filed 02/04/21 Page 9 of 37 PageID #: 69
`
`530
`
`

`

`U.S. Patent
`
`Apr. 26, 2011
`
`Sheet 8 of 22
`
`US 7,933,431 B2
`
`
`
`COMPUTER
`
`Case 2:21-cv-00041-JRG Document 1-2 Filed 02/04/21 Page 10 of 37 PageID #: 70
`
`

`

`U.S. Patent
`
`Apr. 26, 2011
`
`Sheet 9 of 22
`
`US 7.933,431 B2
`
`730
`
`73r 700
`
`f 701
`
`m--
`
`733
`
`732
`
`705
`
`
`
`Case 2:21-cv-00041-JRG Document 1-2 Filed 02/04/21 Page 11 of 37 PageID #: 71
`
`905
`
`FIG. 9
`
`

`

`U.S. Patent
`
`Apr. 26, 2011
`
`Sheet 10 of 22
`
`US 7,933,431 B2
`
`816
`
`830
`
`&S computer
`
`803
`
`
`
`Case 2:21-cv-00041-JRG Document 1-2 Filed 02/04/21 Page 12 of 37 PageID #: 72
`
`

`

`U.S. Patent
`
`Apr. 26, 2011
`
`Sheet 11 of 22
`
`US 7.933,431 B2
`
`
`
`Case 2:21-cv-00041-JRG Document 1-2 Filed 02/04/21 Page 13 of 37 PageID #: 73
`
`990||
`
`990 ||
`
`

`

`U.S. Patent
`
`Apr. 26, 2011
`
`Sheet 12 of 22
`
`US 7,933,431 B2
`
`
`
`
`
`870/, ”5)||-||
`
`Case 2:21-cv-00041-JRG Document 1-2 Filed 02/04/21 Page 14 of 37 PageID #: 74
`
`

`

`U.S. Patent
`
`Apr. 26, 2011
`
`Sheet 13 of 22
`
`US 7,933,431 B2
`
`
`
`Case 2:21-cv-00041-JRG Document 1-2 Filed 02/04/21 Page 15 of 37 PageID #: 75
`
`FIG 11A
`
`

`

`U.S. Patent
`
`Apr. 26, 2011
`
`Sheet 14 of 22
`
`US 7,933,431 B2
`
`
`
`/9 || ||
`
`Case 2:21-cv-00041-JRG Document 1-2 Filed 02/04/21 Page 16 of 37 PageID #: 76
`
`

`

`U.S. Patent
`
`Apr. 26, 2011
`
`Sheet 15 of 22
`
`US 7,933,431 B2
`
`
`
`Case 2:21-cv-00041-JRG Document 1-2 Filed 02/04/21 Page 17 of 37 PageID #: 77
`
`O
`O
`N
`w
`
`

`

`U.S. Patent
`
`Apr. 26, 2011
`
`Sheet 16 of 22
`
`US 7,933,431 B2
`
`s
`
`v
`O
`cy
`ve
`
`
`
`Case 2:21-cv-00041-JRG Document 1-2 Filed 02/04/21 Page 18 of 37 PageID #: 78
`
`

`

`U.S. Patent
`
`Apr. 26, 2011
`
`Sheet 17 of 22
`
`US 7,933,431 B2
`
`
`
`Case 2:21-cv-00041-JRG Document 1-2 Filed 02/04/21 Page 19 of 37 PageID #: 79
`
`

`

`U.S. Patent
`
`Apr. 26, 2011
`
`Sheet 18 of 22
`
`US 7,933,431 B2
`
`
`
`Case 2:21-cv-00041-JRG Document 1-2 Filed 02/04/21 Page 20 of 37 PageID #: 80
`
`FIG. 14C
`
`

`

`U.S. Patent
`
`Apr. 26, 2011
`
`Sheet 19 of 22
`
`US 7,933,431 B2
`
`
`
`:
`
`O
`CY
`O
`
`V
`
`:
`
`Case 2:21-cv-00041-JRG Document 1-2 Filed 02/04/21 Page 21 of 37 PageID #: 81
`
`f
`
`

`

`U.S. Patent
`
`Apr. 26, 2011
`
`Sheet 20 of 22
`
`US 7.933,431 B2
`
`TEX|c}
`
`SILNE WETE
`
`?
`
`000}
`
`| /66
`
`
`
`9/, “50/-|
`
`
`
`-000||
`
`Case 2:21-cv-00041-JRG Document 1-2 Filed 02/04/21 Page 22 of 37 PageID #: 82
`
`TEX|c}
`
`
`
`SLNE WETE
`
`

`

`U.S. Patent
`
`Apr. 26, 2011
`
`Sheet 21 of 22
`
`US 7,933,431 B2
`
`99/ |
`
`00/ |
`
`09/ |
`
`?70/ |
`
`47
`
`Z01 ||
`
`?0/ |
`
`00/ |
`
`0€/ ),
`
`98/ |
`
`Case 2:21-cv-00041-JRG Document 1-2 Filed 02/04/21 Page 23 of 37 PageID #: 83
`
`

`

`U.S. Patent
`
`Apr. 26, 2011
`
`Sheet 22 of 22
`
`US 7,933,431 B2
`
`
`
`- O2
`Y
`h
`2 CD Q O
`
`Case 2:21-cv-00041-JRG Document 1-2 Filed 02/04/21 Page 24 of 37 PageID #: 84
`
`

`

`1.
`CAMERA BASED SENSING IN HANDHELD,
`MOBILE, GAMING, OR OTHER DEVICES
`
`CROSS REFERENCE TO RELATED
`APPLICATIONS
`
`US 7,933,431 B2
`
`2
`6. Apparel Manufacture and Distance Fashion Shopping
`in Both Present and Future': U.S. Applin. Ser. No.
`60/187,397, filed Mar. 7, 2000.
`
`FIELD OF THE INVENTION
`
`This application is a continuation of application Ser. No.
`11/980,710 filed Oct. 31, 2007, now U.S. Pat. No. 7,756,297;
`which is a continuation of application Ser. No. 10/893.534
`filed Jul. 19, 2004, now U.S. Pat. No. 7,401,783; which is a
`continuation of application Ser. No. 09/612.225 filed Jul. 7,
`2000, now U.S. Pat. No. 6,766,036; which claims the benefit
`of U.S. Provisional Application No. 60/142,777, filed Jul. 8,
`1999.
`Cross references to related co-pending US applications by
`the inventor having similar subject matter.
`1. Touch TV and other Man Machine Interfaces: Ser. No.
`09/.435,854 filed Nov. 8, 1999, now U.S. Pat. No. 7,098,
`891; which was a continuation of application Ser. No.
`07/946,908, now U.S. Pat. No. 5,982,352:
`2. More Useful Man Machine Interfaces and Applications:
`Ser. No. 09/433,297 filed Nov. 3, 1999, now U.S. Pat.
`No. 6,750,848;
`3. Useful Man Machine interfaces and applications: Ser.
`No. 09/138,339, Pub. Applin. 2002-0036617, now aban
`doned;
`4. Vision Target based assembly: Ser. No. 08/469,907 filed
`Jun. 6, 1995, now U.S. Pat. No. 6,301,783:
`5. Picture Taking method and apparatus: provisional appli
`cation 60/133,671, and regular application Ser. No.
`09/568,552 filed May 11, 2000, now U.S. Pat. No. 7,015,
`950;
`6. Methods and Apparatus for Man Machine Interfaces and
`Related Activity: Provisional Application: provisional
`application 60/133,673 filed May 11, 1999; and regular
`application Ser. No. 09/568,554 filed May 11, 2000, now
`U.S. Pat. No. 6,545,670;
`7. Tactile Touch Screens for Automobile Dashboards, Inte
`riors and Other Applications: provisional application
`Ser. No. 60/183,807; and regular application Ser. No.
`09/789,538, now U.S. Pat. No. 7,084,859; and
`8. Apparel Manufacture and Distance Fashion Shopping in
`Both Present and Future: provisional application
`60/187,397 filed Mar. 7, 2000.
`The disclosures of the following U.S. patents and co-pend
`ing patent applications by the inventor, or the inventor and his
`colleagues, are incorporated herein by reference:
`1. “Man machine Interfaces': U.S. application Ser. No.
`09/.435,854 and U.S. Pat. No. 5,982,352, and U.S. appli
`cation Ser. No. 08/290,516, filed Aug. 15, 1994, now
`U.S. Pat. No. 6,008,000, the disclosure of both of which
`is contained in that of Ser. No. 09/435,854;
`2. “Useful Man Machine Interfaces and Applications”:
`U.S. application Ser. No. 09/138,339, now Pub. Appln.
`2002-0036617;
`3. “More Useful Man Machine Interfaces and Applica
`tions”: U.S. application Ser. No. 09/433,297:
`4. "Methods and Apparatus for Man Machine Interfaces
`and Related Activity”: U.S. Applin. Ser. No. 60/133,673
`filed as regular application Ser. No. 09/568,554, now
`U.S. Pat. No. 6,545,670;
`5. “Tactile Touch Screens for Automobile Dashboards,
`Interiors and Other Applications: U.S. provisional
`Applin. Ser. No. 60/183,807, filed Feb. 22, 2000, now
`filed as reg. application Ser. No. 09/789,538; and
`
`10
`
`15
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`The invention relates to simple input devices for comput
`ers, particularly, but not necessarily, intended for use with 3-D
`graphically intensive activities, and operating by optically
`sensing a human input to a display Screen or other object
`and/or the sensing of human positions or orientations. The
`invention herein is a continuation in part of several inventions
`of mine, listed above.
`This continuation application seeks to provide further use
`ful embodiments for improving the sensing of objects. Also
`disclosed are new applications in a variety of fields such as
`computing, gaming, medicine, and education. Further dis
`closed are improved systems for display and control pur
`poses.
`The invention uses single or multiple TV cameras whose
`output is analyzed and used as input to a computer, Such as a
`home PC, to typically provide data concerning the location of
`parts of, or objects held by, a person or persons.
`
`DESCRIPTION OF RELATED ART
`
`The above mentioned co-pending applications incorpo
`rated by reference discuss many prior art references invarious
`pertinent fields, which form a background for this invention.
`Some more specific U.S. Patent references are for example:
`DeMenthon U.S. Pat. Nos. 5,388,059: 5,297,061; 5,227,
`985
`Cipola U.S. Pat. No. 5,581,276
`Pugh U.S. Pat. No. 4,631,676
`Pinckney U.S. Pat. No. 4,219,847
`
`DESCRIPTION OF FIGURES
`
`FIG.1 illustrates a basic computer terminal embodiment of
`the invention, similar to that disclosed in copending applica
`tions.
`FIG. 2 illustrates object tracking embodiments of the
`invention employing a pixel addressable camera.
`FIG. 3 illustrates tracking embodiments of the invention
`using intensity variation to identify and/or track object target
`datums.
`FIG. 4 illustrates tracking embodiments of the invention
`using variation in color to identify and/or track object target
`datums.
`FIG. 5 illustrates special camera designs for determining
`target position in addition to providing normal color images.
`FIG. 6 identification and tracking with stereo pairs.
`FIG. 7 illustrates use of an indicator or co-target.
`FIG. 8 illustrates control of functions with the invention,
`using a handheld device which itself has functions.
`FIG. 9 illustrates pointing at an object represented on a
`screen using a finger or laser pointer, and then manipulating
`the represented object using the invention.
`FIG.10 illustrates control of automobile or other functions
`with the invention, using detectedknob, Switch or slider posi
`tions.
`FIG. 11 illustrates a board game embodiment of the inven
`tion.
`FIG. 12 illustrates a generic game embodiment of the
`invention.
`FIG. 13 illustrates a game embodiment of the invention,
`Such as might be played in a bar.
`
`Case 2:21-cv-00041-JRG Document 1-2 Filed 02/04/21 Page 25 of 37 PageID #: 85
`
`

`

`US 7,933,431 B2
`
`3
`FIG. 14 illustrates a laser pointer or other spot designator
`embodiment of the invention.
`FIG. 15 illustrates a gesture based flirting game embodi
`ment of the invention.
`FIG.16 illustrates a version of the pixel addressing camera
`technique wherein two lines on either side of a 1000 element
`square array are designated as perimeter fence lines to initiate
`tracking or other action.
`FIG. 17 illustrates a 3-D acoustic imaging embodiment of
`the invention.
`
`THE INVENTION EMBODIMENTS
`
`4
`or other delineating geometric indicators of the position of the
`eyes and nose, (or Some other facial feature Such as the
`mouth), and the accuracy of the initial imputing of the spacing
`of the eyes and their respective spacing to the nose. Clearly if
`a standard human value is used (say for adult, or for a child or
`even by age) Some lessening of precision results, since these
`spacings are used in the calculation of distance and orienta
`tion of the face of human 167 from the camera 160.
`In another generally more photogrammetrically accurate
`case, one might choose to use four special targets (e.g., glass
`bead retro-reflectors, or orange dots) 180-183 on the object
`185 having known positional relationships relative to each
`other on the object Surface, such as one inch centers. This is
`shown in FIG. 1C, and may be used in conjunction with a
`pixel addressable camera such as described in FIG. 2 below,
`which allows one to rapidly determine the object position and
`orientation and track its movements in up to 6 degrees of
`freedom as disclosed by Pinkney U.S. Pat. No. 4,219,847 and
`technical papers referenced therein. For example, the system
`described above for FIGS. 1 and 2 involving the photogram
`metric resolution of the relative position of three or more
`known target points as viewed by a camera is known and is
`described in a paper entitled “A Single Camera Method for
`the 6-Degree of Freedom Sprung Mass Response of Vehicles
`Redirected by Cable Barriers' presented by M. C. van Wijk
`and H. F. L. Pinkney to The Society of Photo-optical Instru
`mentation Engineers.
`The stereo pair of cameras can also acquire a two view
`Stereo image of the scene as well, which can be displayed in
`3D using stereoscopic or auto-stereoscopic means, as well as
`transmitted or recorded as desired.
`In many applications of the foregoing invention it is desir
`able not just to use a large Screen but in fact one capable of
`displaying life size images. This particularly relates to human
`scaled images, giving a life-like presence to the data on the
`screen. In this way the natural response of the user with
`motions of hands, head, arms, etc., is scaled in “real' propor
`tion to the data being presented.
`FIG 2
`This embodiment and others discloses special types of
`cameras useful with the invention. In the first case, that of
`FIG. 2A, a pixel addressable camera such as the MAPP2200
`made by IVP corporation of Sweden is used, which allows
`one to do manythings useful for rapidly determining location
`of objects, their orientation and their motion.
`For example, as shown in FIG. 2A, an approximately cir
`cular image 201 of a target datum such as 180 on object 185
`of FIG.1C may be acquired by scanning the pixel elements on
`a matrix array 205 on which the image is formed. Such an
`array in the future will have for example 1000x1000 pixels, or
`more (today the largest IVP makes is 512x512. The IVP also
`is not believed to be completely randomly addressable, which
`some future arrays will be).
`As an illustration, computer 220 determines, after the array
`205 has been interrogated, that the centroid “x, y' of the pixel
`elements on which the target image lies is at pixel x=500,
`y=300 (including a sub-fraction thereof in many cases). The
`centroid location can be determined for example by the
`moment method disclosed in the Pinkney patent, referenced
`above.
`The target in this case is defined as a contrasting point on
`the object, and Such contrast can be in color as well as, or
`instead of intensity. Or with some added preprocessing, it can
`be a distinctive pattern on the object, Such as a checkerboard
`or herringbone.
`
`10
`
`15
`
`25
`
`30
`
`35
`
`FIG. 1
`The invention herein and disclosed in portions of other
`copending applications noted above, comprehends a combi
`nation of one or more TV cameras (or other suitable electro
`optical sensors) and a computer to provide various position
`and orientation related functions of use. It also comprehends
`the combination of these functions with the basic task of
`generating, storing and/or transmitting a TV image of the
`scene acquired—either in two or three dimensions.
`The embodiment depicted in FIG. 1A illustrates the basic
`embodiments of many of my co-pending applications above.
`A stereo pair of cameras 100 and 101 located on each side of
`the upper surface of monitor 102 (for example a rear projec
`tion TV of 60 inch diagonal screen size) with display screen
`103 facing the user, are connected to PC computer 106 (inte
`grated in this case into the monitor housing), for example a
`400 Mhz. Pentium II. For appearances and protection a single
`extensive cover window may be used to cover both cameras
`and their associated light sources 110 and 111, typically
`LEDS.
`The LEDs in this application are typically used to illumi
`nate targets associated with any of the fingers, hand, feet and
`head of the user, or objects such as 131 held by a user, 135
`with hands 136 and 137, and head 138. These targets, such as
`circular target 140 and band target 141 on object 131 are
`desirably, but not necessarily, retro-reflective, and may be
`constituted by the object features themselves (e.g., a finger
`tip, such as 145), or by features provided on clothing worn by
`the user (e.g., a shirt button 147 or polka dot 148, or by
`artificial targets other than retroreflectors.
`Alternatively, a three camera arrangement can be used, for
`example using additional camera 144, to provide added sen
`sitivity in certain angular and positional relationships. Still
`more cameras can be used to further improve matters, as
`desired. Alternatively, and or in addition, camera 144 can be
`used for other purposes, such as acquire images of objects
`Such as persons, for transmission, storage or retrieval inde
`pendent of the cameras used for datum and feature location
`determination.
`For many applications, a single camera can Suffice for
`measurement purposes as well, such as 160 shown in FIG. 1B
`for example, used for simple 2 dimensional (2D) measure
`ments in the Xy plane perpendicular to the camera axis (Z
`axis), or 3D (xyZ, roll pitch yaw) where a target grouping, for
`example of three targets is used such as the natural features
`formed by the two eyes 164, 165 and nose 166 of a human
`167. These features are roughly at known distances from each
`other, the data from which can be used to calculate the
`approximate position and orientation of the human face.
`Using for example the photogrammetric technique of
`Pinkney described below, the full 6 degree of freedom solu
`tion of the human face location and orientation can be
`achieved to an accuracy limited by the ability of the camera
`image processing software utilized to determine the centroids
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`Case 2:21-cv-00041-JRG Document 1-2 Filed 02/04/21 Page 26 of 37 PageID #: 86
`
`

`

`Subsequent Tracking
`To Subsequently track the movement of this target image, it
`is now only necessary to look in a small pixel window com
`posed of a small number of pixels around the target. For
`example the square 230 shown, as the new position x'y' of the
`target image cannot be further distant within a short period of
`time elapsed from the first scan, and in consideration of the
`Small required time to scan the window.
`For example, if the window is 100x100 pixels, this can be
`scanned in 1 millisecond or less with Such a pixel addressing
`camera, by interrogating only those pixels in the window,
`while still communicating with the camera over a relatively
`slow USB serial link of 12 mb transmission rate (representing
`12,000 pixel gray level values in one millisecond).
`One thus avoids the necessity to scan the whole field, once
`the starting target image position is identified. This can be
`known by an initial scan as mentioned, or can be known by
`having the user move an object with a target against a known
`location with respect to the camera such as a mechanical stop,
`and then indicate that tracking should start either by verbally
`saying so with Voice recognition, or by actuating a control key
`such as 238 or whatever.
`It is noted that if the tracking window is made large enough,
`then it can encompass a whole group of datums, such as
`180-183 on an object.
`FIG. 2B Reduction in Acquisition Time
`Another application of Such a pixel addressing camera is
`shown in FIG.2B. One can look at the whole field, xy of the
`camera, 240, but only address say every 10" pixel such as
`250, 251 and 252, in each direction, i.e., for a total 10,000
`pixels in a field of 1 million (1000x1000, say).
`In this case computer 220 simply queries this fraction of the
`pixels in the image, knowing apriorithat the target image Such
`as 260 will have an image size larger than 10x10 pixels, and
`must be detectable, if of sufficient contrast, by one of the
`queried pixels. (For Smaller or larger target images, the num
`ber and spacing of queried pixels can be adjusted accord
`ingly). This for example, allows one to find approximate
`location of targets with only /100 the pixel interrogation time
`otherwise needed, for example, plus any gain obtained as
`disclosed above, by knowing in what region of the image to
`look (for example during tracking, or given Some apriori
`knowledge of approximate location due to a particular aspect
`of the physical arrangement or the program in question).
`Once a target has been approximately found as just
`described, the addressing can be optimized for that region of
`the image only, as disclosed in Subsequent tracking section
`above.
`Given the invention, the potential for target acquisition in a
`millisecond or two thus is achievable with simple pixel
`addressable CMOS cameras coming on Stream now (today
`costing under S50), assuming the target points are easily
`identifiable from at least one of brightness (over a value),
`contrast (with respect to Surroundings), color, color contrast,
`and more difficult, shape or pattern (e.g., a plaid, or herring
`bone portion of a shirt). This has major ramifications for the
`robustness of control systems built on Such camera based
`acquisition, be they for controlling displays, or machines or
`whatever.
`Its noted that with new 2000x2000 cameras coming on
`stream, it may only be necessary to look at every 15" or 20"
`pixel in each direction to get an adequate feel for target
`location. This means every 200" to 400" pixel, not enough to
`cause image rendition difficulties even if totally darkgrey (as
`it might be in a normal white light image if set up for IR
`wavelengths only).
`
`US 7,933,431 B2
`
`5
`
`6
`
`FIG. 2C
`Another method for finding the target in the first place with
`limited pixel interrogation is to look at pixels near a home
`point where a person for example indicates that the target is.
`This could be for example, placing ones fingernail Such as
`270, whose natural or artificial (e.g., reflective nail polish)
`features are readily seen by the camera 275 and determined to
`be in the right corner of a pad 271 in FIG. 2C which approxi
`mately covers the field of view 274 of the camera 275. The
`computer 220 analyzes the pixels in the right corner 278 of the
`image field 279 representing the pad portion 271 with the
`camera 275, either continuously, or only when the finger for
`example hits a switch such as 280 at the edge of the pad, or on
`command (e.g., by the user pushing abutton or key, or a voice
`message inputted via microphone 285 for example). After
`Such acquisition, the target is then tracked to other locations in
`Xy space of the pad, for example as described above. Its noted
`that it helps to provide a beep or other sound or indication
`when acquisition has been made.
`Pick Windows in Real Time
`Another aspect of the invention is that one can also pick the
`area of the image to interrogate at any desired moment. This
`can be done by creating a window of pixels with in the field to
`generate information, for example as discussed relative to a
`specific car dashboard application of FIG. 10.
`FIG. 2D Scan Pattern
`A pixel addressing camera also allows a computer Such as
`220 to cause scans to be generated which are not typical raster
`scans. For example circular or radial, or even odd shapes as
`desired. This can be done by providing from the computer the
`sequential addresses of the Successive pixels on the camera
`chip whose detected voltages are to be queried.
`A circular scan of pixels addressed at high speed can be
`used to identify when and where a target enters a field
`enclosed by the circular pixel Scan. This is highly useful, and
`after that, the approximate location of the target can be deter
`mined by further scans of pixels in the target region.
`For example consider addressing the pixels c1 c2 c3 ... cn
`representing a circle 282 at the outer perimeter of the array,
`285, of 1000x1000 elements such as discussed above. The
`number of pixels in a full circle is approximately 1000 pi,
`which can be scanned even with USB (universal serial bus)
`limits at 300 times per second or better. For targets of /100
`field in width, this means that a target image entering the field
`Such as circular target image 289 (which is shown intersecting
`element cm and its neighbors) would have to travel /100 the
`field width in 0.0033 seconds to be totally missed in a worst
`case. If the image field corresponds to 20 inches in object field
`width this is 0.2 inchesX300/sec or 60 inches/second, very
`fast for human movement, and not likely to be exceeded even
`where Smaller targets are used.
`Alternative shapes to circular “trip wire” perimeters may
`be used. Such as Squares, Zig-Zag, or other layouts of pixels to
`determine target presence. Once determined, a group of pix
`els such as group 292 can be interrogated to get a better
`determination of target location.
`FIG. 3
`Since many applications of the invention concern, or at
`least have present a human caused motion, or motion of a part
`of a human, oran object moved by a human, the identification
`and tracking problem can be simplified if the features of
`interest, either natural or artificial of the object provide some
`kind of change in appearance during Such motion.
`FIG. 3 illustrates tracking embodiments of the invention
`using intensity variation to identify and/or track object target
`datums. In a simple case, a Subtraction of Successive images
`can aid in identifying Zones in an image having movement of
`
`5
`
`10
`
`15
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`Case 2:21-cv-00041-JRG Document 1-2 Filed 02/04/21 Page 27 of 37 PageID #: 87
`
`

`

`US 7,933,431 B2
`
`10
`
`15
`
`25
`
`30
`
`35
`
`40
`
`45
`
`7
`features as is well known. It is also useful to add pixel inten
`sities of successive images in computer 220 for example. This
`is particular true with bright targets (with respect to their
`usual surroundings) such as LEDs or retro-reflectors. If the
`pixels in use by the camera are able to gather light preferen
`tially at the same time a special illumination light is on, this
`will accentuate the target with respect to background. And if
`Successive frames are taken in this way, not only will a sta
`tionary image of the special target build up, but if movement
`takes place the target image then will blur in a particular
`direction which itself can become identify-able. And the blur
`direction indicates direction of motion as well, at least in the
`2-D plane of the pixel array used.
`Another form of movement can take place artificially,
`where the target is purposely moved to provide an indication
`of its presence. This movement can be done by a human easily
`by just dithering ones finger for example (if a portion of the
`finger Such as the tip is the target in question), or by vibrating
`an object having target features of interest on it, for example
`by moving the object up and down with ones hand.
`For example consider FIG.3A, where a human 301 moves
`his finger 302 in a rapid up and down motion, creating differ
`ent image positions sequentially in time of bright target ring
`320, 320' on his finger, as seen by camera 325. If the camera
`can read quickly enough each of these positions such as 326
`and 327 in image field 328 can be resolved, otherwise a blur
`image such as 330 is registered on the camera and recorded in
`the computer 335.
`Instead of using ones finger, it is also possible to create
`movement of a target for example with a tuning fork or other
`mechanism mechanically energizing the target movement, on
`what otherwise might be a static object say. And it is possible
`for the human, or a computer controlling the movement in
`question to create it in Such a manner that it aids identifica
`tion. For example, a certain number of moves of ones finger
`(e.g., 4), or 2 moves/sec of ones finger, or horizontal moves of
`ones finger etc., any or all of these could indicate to the
`computer upon analysis of the camera image, that a target was
`present.
`The invention comprehends this as a method for acquiring
`the datum to be tracked in the first place, and has provided a
`camera mechanism for tracking fast enough not to lose the
`data, assuming a sufficiently distinct feature. For example, it
`is desirable to not require Sophisticated image processing
`routines and the like if possible, to avoid the time it takes to
`execute same with affordable equipment. And yet in many
`scenes, finding a target cant be done easily today without
`Some aid, either a high contrast target (contrasting brightness
`or color or both, for example). Or the aid can be movement as
`noted, which allows the search for the target to be at least
`localized to a small region of the field of view, and thence take
`much less time to run, even if a Sophisticated algorithm is
`employed.
`FIG. 3B illustrates an embodiment wherein a target which
`blinks optically is used. The simplest case is a modulated
`LED target such 340 on object 341 shown. Successive frames
`taken with camera 345 looking at pixel window 346 at 300
`scans of the pixels within the window per second where the
`image 347 of the LED target is located, can determine, using
`computer 349 (which may be separate from, or incorporated
`with the image sensor), 5 complete blinks of target 340, if
`blinked at a 60hz rate. Both blink frequency, blink spacing,
`blink pulse length can all be determined if the scan rate is
`sufficiently faster than the blink rate, or pulse time.
`It should be noted that if the target 340 is a retro-reflectoras
`in FIG. 1, with an illumination source such as 355 near the
`
`Case 2:21-cv-00041-JRG Document 1-2 Filed 02/04/21 Page 28 of 37 PageID #: 88
`
`50
`
`55
`
`60
`
`65
`
`8
`axis of the camera, then the LEDs (or other sources) of the
`illuminator can be modulated, causing the same effect on the
`target.
`Somewhat more Sophisticated is the situation shown in
`FIG. 3C where a target 380 (on object 360) illuminated by a
`light source 365 provides a time variant intensity change in
`the camera image 368 obtained by camera 370 as the target
`moves its position and that of the image. This can be achieved
`naturally by certain patterns of material Such as herringbone,
`or by multifaceted reflectors such as cut diamonds (genuine or
`glass), which “twinkle” as the object moves. A relative high
`frequency “twinkle' in the image indicates then the presence
`of the target in that area of the image in which it is found.
`When analog sensors such as PSD (position sensing diode)
`sensor 369 described in a copending application is used in
`addition to, or instead of a matrix array in camera 370, the
`variation in light intensity or twinkle can be obtained