Case 2:21-cv-00040-JRG Document 70-13 Filed 09/02/21 Page 1 of 15 PageID #: 1502
`
`Exhibit R
`
`

`

`Case 2:21-cv-00040-JRG Document 70-13 Filed 09/02/21 Page 2 of 15 PageID #: 1503
`
`Case 2:21-cv-00040-JRG Document 70-13 AARTT Ti
`
`
`US006545670B1
`
`
`US 6,545,670 B1
`(10) Patent No:
`a2) United States Patent
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`Apr.8, 2003
`(45) Date of Patent:
`Pryor
`
`
`
`
`(76)
`
`
`
`(*) Notice:
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`(54) METHODS AND APPARATUS FOR MAN
`
`
`
`
`MACHINE INTERFACES AND RELATED
`¥
`7
`
`ACTIVITY
`Lom
`Inventor: panothy R. Pryor, ue ees
`
`
`
`Jecumsen,
`Untarlo
`ad,
`
`
`
`(CA)
`
`Subject to any disclaimer, the term ofthis
`
`
`
`
`
`
`
`
`
`
`patent is extended or adjusted under 35
`
`
`
`U.S.C. 154(b) by 0 days.
`(21) Appl. No.: 09/568,554
`
`
`
`
`.
`4
`
`:
`Filed:
`May 11,2000
`(22)
`
`
`
`
`
`
`
`Related U.S. Application Data
`(60)
`rewisional application No. 60/133,673, filed on May 11,
`
`
`
`
`
`
`
`
`(51) Unt, CUT oiccccccccccccccceccceesssseesseesereveevsevee G09G 5/00
`
`
`
`
`
`(52) US.Cl
`345/173: 345/156
`
`
`
`
`
`Fi ld f Sah
`345AB 179
`58
`~
`of
`"345/156:463/51,§4 36: 2737353
`(58)
`Field
`Seare
`
`
`
`
`
`ne 9
`,
`
`
`
`
`
`References Cited
`(56)
`U.S. PATENT DOCUMENTS
`
`
`3,748,751 A *
`7/1973 Breglia et al. 0.00.00. 463/51
`
`
`
`
`
`
`3,757,322 A *
`9/1973 Barkan etal.
`w 345/173
`...
`
`
`
`
`
`
`
`
`4,017,848 A *
`4/1977 Tannas, Jr.
`.......
`... 345/173
`
`
`
`
`
`
`4,772,028 A *
`9/1988 Rockholdet al.
`............. 463/36
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`8/1990 Hall oo eceeeeeeeeeeee 434/21
`4,948,371 A *
`
`
`
`
`
`
`
`..
`7/1994 Dart et al.
`273/358
`5,328,190 A *
`Ct al. occ eres
`>
`>
`TOd
`3
`
`
`
`
`
`
`2/1996 Elrod
`et al
`5,495,269 A *
`345/179
`3/1996 Vogeley etal.
`§,502,514 A
`
`
`
`
`5/1996 Tauck
`5,515,079 A
`
`
`
`3/1997 Ikematsu et al. wesc 463/52
`5,613,913 A *
`
`
`
`
`
`7/1997 Treat, Jr. et al...
`5,649,706 A *
`273/358
`
`
`
`
`
`
`
`5,982,352 A * 11/1999 Pryor cesses
`1. 345/156
`
`
`
`
`6,008,800 A * 12/1999 Pryor
`v. 345/173
`.........
`
`
`
`
`6,339,748 Bl *
`1/2002 Hiramatsu
`vs 345/156
`
`
`
`
`
`
`
`
`
`
`6,441,807 B1 *
`8/2002 Yamaguchi........
`... 345/173
`
`
`
`
`
`
`2001/0012001 Al *
`8/2001 Rekimotoet al.
`.......4.. 345/173
`kos
`.
`cited by examiner
`
`
`
`
`-
`
`
`
`
`Primary Examiner—Bipin Shalwala
`Assistant Examiner—David L. Lewis
`
`
`
`
`
`
`
`
`
`(74) Attorney, Agent, or Firm—Larson & Taylor PLC
`ABSTRACT
`(57)
`
`
`Disclosed herein are new forms computer monitors and
`
`
`
`
`
`
`
`
`displays, and preferred embodiments utilize electro-optical
`
`
`
`
`
`
`
`
`
`
`
`
`
`sensors, and particularly TV Cameras, providing optically
`inputted data from the display screen or from specialized
`
`
`
`
`
`
`
`
`datum’s on objects and/or natural features of objects, which
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`may be illuminated using specialized light sources, such as
`
`
`
`
`
`
`
`laser pointers. The invention is a continuation of earlier
`
`
`
`
`
`
`applications aimed at providing affordable methods and
`
`
`
`
`
`
`
`
`apparatus for data communication with respect to people and
`
`computers.
`
`
`
`
`17 Claims, 7 Drawing Sheets
`
`
`
`380
`
`
`
`Mf
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`370
`
`\.
`
`
`365
`
`
`
`
`
`DEFTS_00000023
`
`DEFTS_00000023
`
`

`

`Case 2:21-cv-00040-JRG Document 70-13 Filed 09/02/21 Page 3 of 15 PageID #: 1504
`Case 2:21-cv-00040-JRG Document 70-13 Filed 09/02/21 Page 3 of 15 PagelD #: 1504
`
`U.S. Patent
`
`
`
`
`Apr.8, 2003
`
`
`
`
`Sheet 1 of 7
`
`
`
`US 6,545,670 B1
`
`
`
`
`
`
`
`
`
`135
`
`
`
`
`102
`
`104
`
`
`
`DEFTS_00000024
`
`DEFTS_00000024
`
`

`

`Case 2:21-cv-00040-JRG Document 70-13 Filed 09/02/21 Page 4 of 15 PageID #: 1505
`Case 2:21-cv-00040-JRG Document 70-13 Filed 09/02/21 Page 4 of 15 PagelD #: 1505
`
`U.S. Patent
`
`
`
`
`Apr.8, 2003
`
`
`
`
`Sheet 2 of 7
`
`
`
`
`
`
`
`
`
`225
`
`US 6,545,670 B1
`
`
`
`
`
`DEFTS_00000025
`
`DEFTS_00000025
`
`

`

`Case 2:21-cv-00040-JRG Document 70-13 Filed 09/02/21 Page 5 of 15 PageID #: 1506
`Case 2:21-cv-00040-JRG Document 70-13 Filed 09/02/21 Page 5 of 15 PagelD #: 1506
`
`U.S. Patent
`
`
`
`
`
`Apr. 8, 2003
`
`
`
`
`Sheet 3 of 7
`
`
`
`US 6,545,670 B1
`
`380
`
`
`
`
`
`
`
`
`
`o>
`LLLLLLULLLLee
`
`
` [ZZZZZLZLILLOeTTEL
`
`PO
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`365
`
`
`
`375
`
`
`
`
`FIG. 3
`
`DEFTS_00000026
`
`DEFTS_00000026
`
`

`

`Case 2:21-cv-00040-JRG Document 70-13 Filed 09/02/21 Page 6 of 15 PageID #: 1507
`Case 2:21-cv-00040-JRG Document 70-13 Filed 09/02/21 Page 6 of 15 PagelD #: 1507
`
`U.S. Patent
`
`
`
`
`Apr.8, 2003
`
`
`
`
`Sheet 4 of 7
`
`
`US 6,545,670 B1
`
`
`
`
`
`430
`
`
`
`
`
`
`
`
`455
`
`
`
`
`FIG. 4a
`
`DEFTS_00000027
`
`DEFTS_00000027
`
`

`

`Case 2:21-cv-00040-JRG Document 70-13 Filed 09/02/21 Page 7 of 15 PageID #: 1508
`Case 2:21-cv-00040-JRG Document 70-13 Filed 09/02/21 Page 7 of 15 PagelD #: 1508
`
`U.S. Patent
`
`
`
`
`Apr.8, 2003
`
`
`
`
`Sheet 5 of 7
`
`
`
`
`
`
`US 6,545,670 B1
`
`
`
`502
`
`
`
`505
`
`
`
`901
`
`940
`
`
`
`
`
`
`
`DEFTS_00000028
`
`DEFTS_00000028
`
`

`

`Case 2:21-cv-00040-JRG Document 70-13 Filed 09/02/21 Page 8 of 15 PageID #: 1509
`Case 2:21-cv-00040-JRG Document 70-13 Filed 09/02/21 Page 8 of 15 PagelD #: 1509
`
`U.S. Patent
`
`
`
`
`Apr.8, 2003
`
`
`
`
`Sheet 6 of 7
`
`
`US 6,545,670 B1
`
`
`
`
`
`
`
`632
`
`
`FIG. 6
`
`DEFTS_00000029
`
`DEFTS_00000029
`
`

`

`Case 2:21-cv-00040-JRG Document 70-13 Filed 09/02/21 Page 9 of 15 PageID #: 1510
`Case 2:21-cv-00040-JRG Document 70-13 Filed 09/02/21 Page 9 of 15 PagelD #: 1510
`
`U.S. Patent
`
`
`
`
`Apr.8, 2003
`
`
`
`
`Sheet 7 of 7
`
`
`US 6,545,670 B1
`
`
`
`
`
`
`
`©w
`
`ei
`
`Oo
`N
`M—m
`
`
`
`
`
`
`
`Ww
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`WW
`OF
`
`
`
`=o
`
`
`
`@O
`Ww
`Bm
`
`
`
`
`
`
`
`
`FIG.7
`
`DEFTS_00000030
`
`
`
`
`
`
`
`
`
`
`
`
`COMPUTER
`
`DEFTS_00000030
`
`

`

`Case 2:21-cv-00040-JRG Document 70-13 Filed 09/02/21 Page 10 of 15 PageID #: 1511
`Case 2:21-cv-00040-JRG Document 70-13 Filed 09/02/21 Page 10 of 15 PagelD #: 1511
`
`
`
`US 6,545,670 B1
`
`
`1
`METHODS AND APPARATUS FOR MAN
`
`
`
`
`
`MACHINE INTERFACES AND RELATED
`
`
`
`
`ACTIVITY
`
`
`
`
`10
`
`
`
`
`
`
`
`15
`
`
`
`
`
`
`
`
`
`This application claims benefit of U.S. Provisional
`
`
`
`
`
`
`Application No. 60/133,673 filed May 11, 1999.
`CROSS REFERENCES TO RELATED CO-
`
`
`
`PENDING APPLICATIONS BY THE INVENTOR
`
`
`
`
`
`
`
`
`
`
`
`1. Touch TV and other Man Machine Interfaces, Ser. No.
`
`09/435,854.
`
`
`
`
`
`
`
`2. More Useful Man Machine Interfaces and applications
`Ser. No. 09/433,297.
`
`
`
`
`
`
`
`
`
`
`
`
`3. Target holes and corners U.S. Ser. Nos. 08/203,603, and
`
`08/468,358.
`
`
`
`
`
`
`
`4. Useful Man Machine interfaces and applications, Ser.
`No. 09/138,339.
`
`
`
`
`
`
`
`
`
`
`5. Vision Target based assembly, U.S. Ser. Nos. 08/469,
`
`
`
`429, 08/469,907, 08/470,325, 08/466,294.
`
`
`
`
`
`
`6. Camera based Applications of man machine interfaces
`
`
`
`
`
`(U.S. provisional application No. 60/142,777).
`
`
`
`
`
`
`7. Picture Taking method and apparatus(U.S. provisional
`
`
`
`
`
`
`application No. 60/133,671 filed May 11, 1999).
`8. Tactile Touch Screens for Automobile Dashboards,
`
`
`
`
`
`
`
`
`
`
`Interiors and Other Applications.
`
`
`
`
`
`
`9. Apparel Manufacture and Distance Fashion Shopping
`in Both Present and Future, filed March 2000.
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`The disclosures of the above referenced applications are
`
`
`
`incorporated herein by reference.
`FEDERALLY SPONSORED R AND D
`
`
`
`STATEMENT
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`not applicable
`
`
`MICROFICHE APPENDIX
`
`
`
`
`
`
`
`
`not applicable
`
`
`
`
`
`
`
`
`The disclosures of the following U.S. patents and
`
`
`
`
`
`
`co-pending, patent applications are incorporated herein by
`reference:
`
`
`
`
`
`
`
`
`1. U.S. Pat. No. 4,629,319 (Panel Surface Flaw
`
`
`
`
`
`inspection, which discloses a novel optical principle
`
`
`
`commonly called “D Sight”).
`2. U.S. Ser. No. 09/435,854 and U.S. Pat. No. 5,982,352,
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`and U.S. Ser. No. 08/290,516 (“Man Machine
`
`
`
`
`
`
`
`
`
`Interfaces”), filed Aug. 15, 1994, now US. Pat. No.
`6,008,000, the disclosure of both of which is contained
`
`
`
`
`
`
`
`
`
`
`
`in that of Ser. No. 09/435,854.
`
`
`
`
`
`
`
`3. U.S. application Ser. No. 09/138,339 Useful man
`
`
`
`
`machine interfaces and applications.
`
`
`
`
`
`
`
`
`4. US. application Ser. No. 09/433,297 More useful man
`
`
`
`
`machine interfaces and applications.
`PROVISIONAL PATENT APPLICATIONS
`
`
`
`
`
`
`
`
`5. Camera Based Applications of Man—Machine Inter-
`faces U.S. Ser. No. 60/142,777.
`
`
`
`
`
`
`
`
`
`
`
`
`
`6. Picture Taking method and apparatus U.S. No. 60/133,
`671.
`
`
`
`
`
`
`
`
`7. Methods and Apparatus for Man Machine Interfaces
`
`
`
`
`
`
`and Related Activily U.S. No. 60/133,673.
`8. Tactile Touch Screens for Automobile Dashboards,
`
`
`
`
`
`
`
`
`
`
`
`
`Interiors and Other Applications, Ser. No. 60/183,807,
`filed Feb. 22, 2000.
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`30
`
`
`
`
`
`40
`
`
`
`45
`
`
`
`50
`
`
`
`55
`
`
`
`60
`
`
`
`65
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`2
`
`
`
`
`
`
`9. Apparel Manufacture and Distance Fashion Shopping
`
`
`
`
`
`
`
`
`in Both Present and Future, Ser. No. 60/187,397,filed
`Mar. 7, 2000.
`
`
`BACKGROUND OF THE INVENTION
`
`
`1. Field of the invention
`
`
`
`
`
`
`
`
`
`
`
`
`The invention relates to simple input devices for
`
`
`
`
`
`
`
`
`computers, particularly, but not necessarily, intended for use
`
`
`
`
`
`
`
`with 3-D graphically intensive activities, and operating by
`
`
`
`
`
`
`
`optically sensing a human input to a display screen or other
`
`
`
`
`
`
`
`object and/or the sensing of humanpositionsororientations.
`
`
`
`
`
`
`
`The invention herein is a continuation in part of several
`inventions of mine, listed above.
`
`
`
`
`
`
`
`
`
`This continuation application seeks to provide further
`
`
`
`
`
`
`detail on useful embodiments for computing. One embodi-
`
`
`
`
`
`
`ment is a monitor housing for a computer that integrally
`
`
`
`
`
`incorporates digital TV camerasto look at points on the hand
`
`
`
`
`
`
`
`
`
`or the finger, or objects held in the hand of the user, which
`
`
`
`
`
`
`
`
`
`are used to input data to the computer.
`It may also or
`
`
`
`
`
`
`
`alternatively, look at the head of the user as well.
`
`
`
`
`
`
`
`
`Further disclosed are improved touchscreens, and further
`
`
`
`
`
`
`
`discloses camera based sensing of laser pointer indications.
`
`
`
`
`
`
`
`
`The invention in several other embodiments, uses real time
`
`
`
`
`
`
`
`stereo photogrammetry or other methods using single or
`
`
`
`
`
`
`
`multiple TV cameras whose output is analyzed and used as
`
`
`
`
`
`
`
`
`input to a personal computer, typically to gather data con-
`
`
`
`
`
`
`
`
`
`
`
`cerning the 3D location of parts of, or objects held by, a
`person or persons.
`
`
`
`
`
`
`2. Description of Related Art
`
`
`
`
`
`The above mentioned co-pending applications incorpo-
`
`
`
`
`
`
`
`
`rated by reference discuss many prior art references in
`
`
`
`
`
`
`
`various pertinent fields, which form a background for this
`invention.
`
`
`
`
`
`
`
`
`
`
`Regarding use of lascr pointers to signal or provide
`
`
`
`
`
`
`information to computers associated with TV displays, the
`
`
`
`
`
`
`
`
`closest reference I can find is U.S. Pat. No. 5,515,079 by
`
`
`
`
`
`
`
`
`Hauck “Computer input system and method of using same”
`
`
`
`
`
`
`
`(incorporated herein by reference). Hauck however, does not
`
`
`
`
`
`
`disclose function in a rear projection context as disclosed
`here.
`
`
`
`
`
`
`
`
`Another reference which does use rear projection is U.S.
`
`
`
`
`
`
`
`Pat. No. 5,502,514 by Vogeley, et al, entitled “Stylus posi-
`
`
`
`
`
`
`
`tion sensing and digital camera with a digital micromirror
`
`
`
`
`
`
`
`
`device”. This however can only be used with a DLP pro-
`
`
`
`
`
`
`
`
`
`jector comprising such a device, and does not use the simple
`
`
`
`
`
`
`
`‘l'V camera based sensing approachofthe instant invention.
`
`
`
`
`
`
`
`
`Noreference I have beenableto find discusses the unique
`
`
`
`
`
`
`
`aspects of the disclosed invention relative to human inter-
`
`
`
`
`
`
`
`
`
`action based information and the ability of the input data
`
`
`
`
`
`
`from the aiming pointer to be spatially encoded.
`DESCRIPTION OF FIGURES
`
`
`
`
`
`
`
`
`FIG. 1 illustrates a computer monitor of the invention
`
`
`
`
`
`
`
`integrally incorporating one or two cameras pointing out-
`
`
`
`
`
`
`
`
`
`ward from the surface of the Bezel facing the user to observe
`
`
`
`
`
`
`
`
`
`objects held by the user, or parts of the user such asfingers
`
`
`
`
`
`
`
`
`and hands, the bezel preferably including an opaque (to the
`
`
`
`
`
`
`
`
`
`
`user) plastic cover for both cameras and light sources. Also
`
`
`
`
`
`
`illustrated is an additional camera for viewing a user directly
`
`
`
`or other purposes.
`FIG. 2 illustrates a version of the invention wherein the
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`light illuminating the target datums of the object is itself
`
`
`
`
`
`
`
`
`generated by the monitor viewed by the user or users, said
`
`
`
`
`
`
`
`
`monitor being of the CRT, LED, Projected light, scanned
`
`
`
`
`
`laser spot or any other variety.
`
`DEFTS_00000031
`
`DEFTS_00000031
`
`

`

`Case 2:21-cv-00040-JRG Document 70-13 Filed 09/02/21 Page 11 of 15 PageID #: 1512
`Case 2:21-cv-00040-JRG Document 70-13 Filed 09/02/21 Page 11 of 15 PagelD #: 1512
`
`
`
`US 6,545,670 B1
`
`
`3
`FIG. 3 illustrates a touch screen of the invention of
`
`
`
`
`
`
`
`
`
`
`
`
`
`co-pending application 1
`referenced above, having
`
`
`
`
`
`
`
`improvedscreenrigidity. In this case, distortion of the screen
`
`
`
`
`
`
`
`
`occurs primarily in a zone that is able to distort, with the
`
`
`
`
`
`zone supported by a rigid backing member.
`FIG. 4 illustrates the use of a TV camera based transduc-
`
`
`
`
`
`
`
`
`
`
`
`
`
`tion using the camera for screen distortion determination
`similar to that of PIG. 3 with the camera used for the
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`determination of the position of a laser pointer indication
`
`
`
`
`
`
`such as a spot directed by a user on the screen, particularly
`
`
`
`
`
`
`in response to an image displayed on the screen.
`HIG. 5 illustrates a variation of FIG. 4 in which the laser
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`spot is a spatially encoded to carry informationthatis itself
`
`
`
`
`
`then sent to the camera system.
`
`
`
`
`
`
`FIG. 6 illustrates an embodimentusing laser pointers for
`
`
`
`
`
`
`
`acquaintance making purposes, including the use ofthe laser
`
`
`
`
`
`
`pointer to an image to designate an image ona television
`
`
`
`
`
`
`screen using the invention of FIG. 4 or 5 above.
`
`
`
`
`
`FIG. 7 illustrates Handwriting and signature recognition
`
`
`
`
`
`
`
`
`of sensed pencil position for internet commerce and other
`
`
`
`
`
`
`purpose, including a D-Sight technology based writing pad
`
`
`
`
`capable of distortion signature determination.
`THE INVENTION EMBODIMENTS
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`FIG. 1
`
`
`
`
`
`
`
`This embodiment illustrating a computer display with
`
`
`
`
`
`
`camera(s) and illumination system is an alternative or addi-
`
`
`
`
`
`
`
`
`tion to that of FIG. 1 of copending reference 3 above (Ser.
`
`
`No. 09/138,339).
`
`
`
`
`
`
`
`APC computer based embodimentis shownin FIG. La.
`
`
`
`
`
`
`
`
`
`
`In this case, a stereo pair of cameras 100 and 101 located on
`
`
`
`
`
`
`
`
`
`each side of the upper surface of monitor 102 (for example
`
`
`
`
`
`
`
`
`a rear projection TV having 80 inch diagonal screen 104)
`
`
`
`
`
`
`
`
`
`facing the user, desirably having one or more cover windows
`
`
`
`
`
`
`
`
`
`103. In this case a single extensive cover window 103 is
`
`
`
`
`
`
`
`
`
`covering both camerasandtheir associated light sources 110
`and 111, and mounted flush with the monitor front bezel
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`surface. The LED’s in this application are typically used to
`
`
`
`
`
`
`
`
`illuminate targets associated with the fingers, hand, head of
`
`
`
`
`
`
`
`
`
`
`the uscr, or objects held by the user, such as user 135 with
`
`
`
`
`
`
`
`
`
`
`hands 136 and 137, and head 138. These targets are
`
`
`
`
`
`
`
`desirably, but not necessarily, retro-reflective, and may be
`
`
`
`
`
`
`
`constituted by the object features themselves(e.g. a finger),
`
`
`
`
`
`
`
`or by features of clothing worn bythe user, or by artificial
`
`
`
`
`targets other than retroreflectors.
`
`
`
`
`
`
`
`The cameras are preferably pointed obliquely inward at
`
`
`
`
`
`
`
`
`angles theta, and downward,if desired, at further angles phi
`toward the center of the desired work volume 140in front of
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`the monitor, as required (angles depend on the computer
`monitor width, the distance of the work zone volume, 140,
`
`
`
`
`
`
`
`
`
`from the monitor etc.
`
`
`
`
`
`
`
`
`
`
`Alternatively, or in addition, a single camera can be used,
`
`
`
`
`
`
`
`
`for determining user or other object positions such as 120
`
`
`
`
`
`
`
`
`
`with light source 121, both also optionally located behind a
`
`
`
`
`cover window(such as 103).
`
`
`
`
`
`
`
`The cover window 103 is preferably black or dark plastic
`
`
`
`
`
`
`
`
`
`whichlets the LED light source wavelength pass easily, but
`
`
`
`
`
`
`
`attenuates sunlight or room lights, thus aiding camera signal
`
`
`
`
`
`
`
`
`to noise in many cases, and making the cameras and light
`
`
`
`
`
`
`
`sources substantially invisible to the user (especially if the
`
`
`
`
`
`
`
`
`
`light sourcesare in the near infrared) andthus pleasingto the
`
`
`
`
`eye and not distracting.
`
`
`
`
`
`
`
`Alternate camera locations may be used such as in the
`
`
`
`
`
`
`
`sides of the monitor bezel, or anywhere desired, for example
`
`
`
`
`
`
`
`
`as appendages to the monitor. They mayalternately or in
`
`
`
`
`
`
`
`
`
`
`addition, be at the rear of the keyboard in front of the
`monitor. In the case of cameras mounted at the rear of the
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`4
`
`
`
`
`
`
`
`
`keyboard (toward the display screen), these camerasare also
`
`
`
`
`
`
`
`inclined to point toward the user at an angle as well.
`
`
`
`
`
`
`
`
`
`It is noted that an additional camera for viewing a user
`
`
`
`
`
`
`
`directly or other purposes may be employed. For example if
`
`
`
`
`
`
`
`
`
`
`a stereo pair such as 100 and 101 is utilized for position
`
`
`
`
`
`
`
`determination, then a third camera such as 120 might be
`
`
`
`
`
`
`
`
`used just for imaging using ambient illumination such as
`
`
`
`
`
`
`
`
`
`
`room lights (i.e. LED source 121 is not needed, though could
`
`
`
`
`
`
`
`
`be provided if desired) When this camera is located far
`
`
`
`
`
`
`
`
`
`
`enough spaced from the other two, then any light sources
`
`
`
`
`
`
`
`
`
`
`
`such as 110 and 111 located near the optical axes of the other
`
`
`
`
`
`
`
`two, do not generally illuminate any retro-reflectors in such
`
`
`
`
`
`
`
`
`a wayas to register same on camera 120—dueto the limited
`
`
`
`
`angular return characteristic of retro-reflectors.
`FIG. 2
`
`FIG. 2 illustrates a version of the invention of the
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`co-pending applications wherein the light illuminating the
`
`
`
`
`
`
`
`target datumsof the objectis itself generated by the monitor
`
`
`
`
`
`
`
`
`viewed by the user or users, said monitor being of the CRT,
`
`
`
`
`
`
`
`
`
`LED, Projected light, scanned laser spot or any other variety.
`
`
`
`
`
`
`
`For example target 201 on finger 202 is illuminated by
`
`
`
`
`
`
`
`
`
`light 205 from zone 210 of screen 220 on which an image
`
`
`
`
`
`
`
`
`
`is projected by projection device 221 as shown. Light
`
`
`
`
`
`
`
`
`reflected by the target 201 is imaged by camera 230 such as
`
`
`
`
`
`
`
`
`that employed in the invention of FIG. 1 for example. The
`
`
`
`
`
`
`
`
`
`
`color of light 205, or the point (or points) 210 from which
`
`
`
`
`
`
`
`it emanates, may be varied by the program of computer 250
`
`
`
`
`
`
`
`
`controlling screen display driver 260. The control of point
`
`
`
`
`
`
`
`location and color allows selective illumination of targets or
`
`
`
`
`
`
`
`
`
`object features such asfinger tip 203, both by choice of color
`
`
`
`
`
`
`
`sensitivity response with respect to the target Uluminated,
`
`
`
`
`
`
`
`
`
`and if the target
`is retroreflective, by choice of screen
`
`
`
`
`
`
`
`
`location with in the viewing field of view 225 of TV camera
`
`
`
`
`
`
`
`
`
`230. This can be used to select, by choice of screen gener-
`
`
`
`
`
`
`
`
`
`ated light source location which camera, for example of two
`
`
`
`
`cameras sees the target.
`FIG. 3
`
`FIG. 3 illustrates a touch screen of the invention as in the
`
`
`
`
`
`
`
`
`
`
`
`co-pending invention 1 referenced above, having however,
`
`
`
`
`
`
`improved sercenrigidity. In this casc, distortion of the screen
`
`
`
`
`
`
`
`
`
`occurs primarily in a region that
`is able to distort or
`
`
`
`
`
`
`otherwise be changed, with the region supported bya rigid
`
`
`backing member.
`
`
`
`
`
`
`
`For example consider transparent screen member 305
`
`
`
`
`
`
`
`
`whose outer scattering surface 310 is touched, and locally
`
`
`
`
`
`
`
`distorted inward, by finger 315. This surface is separated
`
`
`
`
`
`
`
`
`from rigid (typically glass) backing member 320 by opti-
`
`
`
`
`
`
`cally transparent refractive medium 330 of thickness t,
`
`
`
`
`
`
`
`
`which is compressed by the force of finger 315. Alterna-
`
`
`
`
`
`
`
`
`tively the medium 330 can be a liquid, such as water or
`
`
`
`
`
`
`alcohol,that is either compressed, or displaced temporarily
`
`
`
`
`
`
`
`
`into a reservoir such as 340 (dotted lines) for example.
`
`
`
`
`
`
`
`
`
`
`Light from light source 350 from the rear passes through
`member 320 and is reflected off the back surface 311 of
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`screen member305. Desirably, but not necessarily, the index
`
`
`
`
`
`
`
`
`
`
`of refraction of member 320 and material 330 are closely
`
`
`
`
`
`
`
`
`
`matched such that little refraction occurs as light passes
`
`
`
`
`
`
`
`
`
`from 320 through 330 to surface 311 and back toward
`
`
`
`
`
`
`camera 370 (after retroreflection by expansiveretroreflector
`
`
`
`
`
`
`
`
`365 typically comprised of Scotchlight 7615 glass bead
`
`
`
`
`
`
`
`
`material) for example used to determine the distortion of 311
`
`
`
`
`
`
`
`
`due to the retroreflective based “D sight” image effect or
`
`
`
`
`
`
`another optical phenomenasuch as disclosed in copending
`reference 1.
`
`
`
`
`
`
`
`
`While primarily intended for rear projection application
`
`
`
`
`
`
`using a TV or computer display image projector such as a
`
`
`
`
`
`
`
`
`
`Sharp brand LCD basedprojector 375, there is also a front
`
`
`
`
`
`
`
`15
`
`
`
`
`
`30
`
`35
`
`
`
`40
`
`
`
`45
`
`
`
`50
`
`
`
`55
`
`
`
`60
`
`
`
`65
`
`
`
`DEFTS_00000032
`
`DEFTS_00000032
`
`

`

`
`
`6
`5
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`Computer 490 also controls the front projector 496, such
`projection version in which the distortion of surface 310 is
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`that detected laser spot indications from the digitized camera
`directly observed from the front side, for example by camera
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`image, can be used to modify software used to generate the
`380, also as taught in reference 1. (copending application
`
`
`
`
`
`
`
`
`
`
`
`
`
`Ser. No. 09/435,854).
`image on the screen in what ever manner desired,
`for
`
`
`
`
`
`
`
`In the embodimentof FIG. 3, I have foundit of use to coat
`
`
`
`
`
`
`
`
`example to make a car design larger in a certain area, or to
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`inject a new line on the drawing or whatever.
`the surface 311 with a partially reflecting coating as wasalso
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`A camera such as 485 to determine laser pointer indica-
`described in copending application Ser. No. 09/435,854. It is
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`tions on a front projection screen, can also be used to see
`also generally desirable that outer surface 310 of member
`
`
`
`
`
`
`
`
`
`
`
`
`
`datumson objects in front of the screen as well, as discussed
`315 be relatively hard, such as thin Plexiglas or Mylar.
`
`
`
`
`
`
`
`
`
`
`
`
`
`in FIG. 1 and referenced copending applications for
`As an alternative to providinga partially reflective coating
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`(such as commonly done with vacuum deposition of silver
`example. These can include natural features of the person
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`475, such as fingers, hands, or specialized datums such as
`or interference coatings—the latter useful if at the wave-
`
`
`
`
`
`
`
`
`
`
`
`
`retroreflectors generally located on apparel or extremities of
`length used for distortion determination, which additionally
`the user. Combined also can be a camera to determine laser
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`can be in the near infrared (e.g. 0.8 microns, where LED
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`sources and TV camera sensitivity is commonplace), it is
`pointer location and the screen deflection or other charac-
`teristic of touch or other contact with the screen. The camera
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`possible to provide on surface 311 a periodic array of zones
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`of reflective material (for example a reflective dot or stripe
`can also determine location of target datums on the object as
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`0.05 mm wide, every 1 mm), whose movementas a result of
`well as laser pointer indications, and other things as well.
`FIG. 5
`
`
`
`
`
`
`
`
`
`touch is detected using the electro-optical system of the
`FIG. 5 illustrates a variation of FIG. 4 in which the laser
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`invention. The projection device if desired can be pro-
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`spot
`is a spatially encoded to carry information. Such
`grammed to minimize the effect of such reflective or par-
`
`
`
`
`
`
`
`
`
`
`
`
`
`information can be in the form of a shape (such as a heart
`tially reflective zones in the displayed image.
`FIG. 4
`
`
`
`
`
`
`
`
`
`501 projected by laser pointer 502 whose beam is transmit-
`HIG. 4 illustrates the use of a TV camera based transduc-
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`ted through grating 505), an alphanumeric character, or
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`anything else desired. It is also foreseen that such informa-
`tion of finger
`touch using for example camera 410 to
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`tion can be easily changed by the user, either by changing
`determine screen distortion occurring on touch similar to
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`fixed lasers, selecting different lasers with different spatially
`that of FIG. 3. As disclosed herein, the camera 410 mayalso
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`encoded holographic gratings or by having a turret of such
`be used for the determination of the position of a laser or
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`gratings in front of a single laser. The color of the laser can
`other optical based pointer indication such as a spot 420
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`also be changed, with certain colors signifying desired
`from laser pointer 425 directed by a uscr 430 on the screen
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`440, particularly in response to an image (not shown for
`actions. Tunable wavelength lasers make this easier today.
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`The information can be projected directly on an object, or
`clarity) displayed on the screen, for example by image
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`on a front or rear projection screen displaying other infor-
`projector 455. For example if a pony image was displayed,
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`mation. The projected information can also be sensed as in
`a child could pointa pointer at the pony imageon the screen,
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`FIG. 4, using a TV camera such as 530, viewing an object
`and the program in system computer 460 could acknowledge
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`such as a chair 540, or alternatively a screen such as 481 on
`this with audio feedback from loudspeaker 465 to the child,
`
`
`
`
`
`
`
`
`
`
`
`
`
`which the information is projected.
`and if desired record in memory 470 that the child had
`FIG. 6
`
`
`
`
`
`
`
`
`
`
`correctly identified the pony, useful for tracking the child’s
`
`
`
`
`
`
`
`
`
`
`
`
`FIG. 6 illustrates embodiments using laser pointers for
`learning, or recording scores for game purposes.
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`acquaintance making purposes. This embodiment of the
`The optical pointer function is distinct from that of the
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`inventionis particularly illustrated here for social purposes,
`physical touch screen indication of FIG. 3 above. Either
`
`
`
`
`
`
`
`
`
`
`
`
`
`but any application to which it useful is contemplated.
`function can exist independently, or both together. A sepa-
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`In particular, it is adapted to signaling one’s indication for
`rate camera such as 411 can alternatively be used to deter-
`
`
`
`
`
`
`
`
`
`
`
`wishing to meet a person particularly of the opposite sex for
`mine laser pointer indication.
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`the purpose of dating, etc. It more particularly concerns the
`This method of laser pointer designation is highly
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`use in a bar, restaurant or other social scene and using a laser
`interesting, especially as one considers very large display
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`pointer or other means for indicating information on the
`screens. While one mightuse a finger pointing which is more
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`laser beam so pointed to point at the person at another table
`natural, let’s say, the actual holding ofthe laser pointeris not
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`or across the room in a way that would indicate an interest.
`much different and the use of this on the front projection
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`Acommonproblem in social interaction is to find some-
`screen shown in FIG. 4 ballows almost widespread use. A
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`one of the opposite sex to start a dialogue with. This is a
`person such as 475 can draw a drawing 487 with a laser
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`famous problem for persons typically males who need to
`pointer 480 for example by successively tracing it on display
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`approach females for this purpose. It is difficult because
`screen 481 where the successive laser pointer indications on
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`often the females are in a group and it makes it more
`the screen are digitized with the camera system 485 com-
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`embarrassing and in any case, there’s always the fear of
`prising one or more TV cameras connected via for example
`
`
`
`
`
`
`
`
`
`rejection.
`an IEEE 1394 firewire connection to computer 490 as
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`I am roughly aware of various types of radio transponders
`shown, equipped to digitize the camera inputs. This allows
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`and so forth for dealing with this issue that have been
`one to draw modifications for example sitting in a confer-
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`ence room on to a drawing and havethis digitized by the proposedin Japan, for example where a transponder onagirl
`camera.
`60
`
`
`
`
`
`
`
`
`would indicate her availability and it would match up with
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`The camera can optionally can be equipped with a laser
`the transponder signals of a guy.
`
`
`
`
`
`
`
`
`
`
`
`
`
`wavelength bandpassfilter such as 495 in front of it to make
`However, this particular invention is more direct because
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`it easy to detect the position laser spot anywhere on t he big
`it concernsthe actual pointing in an area at or near the person
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`screen, even in bright light surroundings. In a rear projection
`in question with a signal. Much as you might wave your
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`application such as depicted in FIG. 4a, the brightness of the
`hand or do somethingelse, butin this case it’s subtle and less
`
`
`
`
`
`
`
`
`
`
`
`
`
`laser alone is often significant enough to allow reliable
`embarrassing. For example, if one sits in a crowded restau-
`detection.
`
`
`
`
`
`
`
`
`
`rant and waves their hand in the air at somebody, everyone
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`30
`
`40
`
`
`
`45
`
`
`
`
`
`
`
`65
`
`
`
`Case 2:21-cv-00040-JRG Document 70-13 Filed 09/02/21 Page 12 of 15 PageID #: 1513
`Case 2