PTO/SB/14 (11-08)
`
`
`Approved for use through 09/30/2010. 0MB 0651-0032
`
`
`
`
`U.S. Patent and Trademark Office; U.S. DEPARTMENT OF COMMERCE
`
`
`
`Under the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless it contains a valid 0MB control number.
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`Application Data Sheet 37 CFR 1.76
`
`
`
`
`Attorney Docket Number 135873.new
`
`Application Number
`
`
`Title of Invention
`
`CAMERA BASED SENSING IN HANDHELD, MOBILE, GAMING OR OTHER DEVICES
`
`The application data sheet is part of the provisional or nonprovisional application for which it is being submitted. The following form contains the
`
`
`
`
`
`
`
`
`
`
`
`
`
`bibliographic data arranged in a format specified by the United States Patent and Trademark Office as outlined in 37 CFR 1.76.
`
`
`
`
`
`
`
`This document may be completed electronically and submitted to the Office in electronic format using the Electronic Filing System (EFS) or the
`
`
`
`document may be printed and included in a paper filed application.
`
`Secrecy Order 37 CFR 5.2
`to Data Sheet may fall under a Secrecy Order pursuant associated with this Application D Portions or all of the application
`
`
`
`
`
`
`
`37 CFR 5.2 (Paper filers only. Applications that fall under Secrecy Order may not be filed electronically.)
`
`A ,DD 1can r n orma 10n:
`t I f
`f
`Remove
`
`Aoolicant 1
`I
`I
`Applicant Authority (!)Inventor I QLegal Representative
`117 I QParty of Interest
`
`under 35 U.S.C.
`under 35 U.S.C. 118
`Prefix Given Name
`Middle Name
`Family Name
`Suffix
`Timothy
`R.
`Pryor
`
`
`Residence Information (Select One) (!) US Residency 0 Non US Residency O Active
`
`City Sylvania
`State/Province
`I OH I Country of Residence
`i I US
`
`Citizenship under 37 CFR 1.41(b)
`i us
`
`Mailing Address of Applicant:
`Address 1
`
`4148 Stonehenge Drive
`Address 2
`I State/Province
`City I Sylvania
`I OH
`I Countryi I us
`Postal Code
`43560
`
`
`
`
`All Inventors Must Be Listed Additional Inventor Information blocks may be
`Add
`the Add button.
`
`
`generated within this form by selecting
`I
`I
`
`US Military Service
`
`Enter either Customer Number or complete the Correspondence Information section below.
`
`of this application.□An Address is being provided for the correspondence Information
`
`Correspondence Information:
`
`
`
`
`
`
`
`For further information see 37 CFR 1.33(a).
`
`
`
`Customer Number 24335
`Email Address patents@wnj.com
`
`I
`Add Email
`!Remove Emaill
`I
`I
`
`I Sub Class (if any)I
`
`
`Application Information:
`Title of the Invention CAMERA BASED SENSING IN HANDHELD,
`MOBILE, GAMING OR OTHER DEVICES
`I Small Entity Status Claimed lg]
`
`
`Attorney Docket Number 135873.new
`
`Application Type Nonprovisional
`
`Subject Matter Utility
`
`Suggested Class (if any)
`
`
`Suggested Technology Center (if any)
`
`Total Number of Drawing Sheets (if any) 22
`
`
`
`
`
`I Suggested Figure for Publication (if any) I
`
`EFS Web 2.2.2
`
`

`

`PTO/SB/14 (11-08)
`
`
`Approved for use through 09/30/2010. 0MB 0651-0032
`
`
`U.S. Patent and Trademark Office; U.S. DEPARTMENT OF COMMERCE
`
`
`
`
`Under the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless it contains a valid 0MB control number.
`
`
`
`
`
`
`
`Docket Number 135873.new
`Attorney
`
`Data Sheet 37 CFR 1.76 Application Application Number
`
`CAMERA BASED SENSING IN HANDHELD, MOBILE, GAMING OR OTHER DEVICES
`Title of Invention
`
`Information:
`Publication
`
`
`at time of Request 37 CFR 1.219) D Request Early Publication (Fee required
`I hereby request that the attached application not be published under 35 U.S.
`
`
`
`
`Request Not to Publish.
`
`
`
`application disclosed in the attached D C. 122(b) and certify that the invention
`
`
`has not and will not be the subject of
`
`
`
`
`
`
`
`
`an application filed in another country, or under a multilateral international agreement, that requires publication at
`
`eighteen months after filing.
`
`
`
`Representative Information:
`
`Representative information should be provided for all practitioners having a power of attorney in the application. Providing
`
`
`
`
`
`
`
`
`
`
`
`this information in the Application Data Sheet does not constitute a power of attorney in the application (see 37 CFR 1.32).
`
`
`
`
`Enter either Customer Number or complete the Representative Name section below. If both sections
`
`
`
`
`
`are completed the Customer Number will be used for the Representative Information during processing.
`
`Please Select One: 0 Customer Number I O US Patent Practitioner
`
`
`
`Customer Number 24335
`
`(37 CFR 11.9) 10 Limited Recognition
`
`IRemove I
`
`Prior Application Number Filing Date (YYYY-MM-DD)
`
`12834281
`2010-07-12
`
`Domestic Benefit/National Stage Information:
`
`
`This section allows for the applicant to either claim benefit under 35 U.S.C. 119(e), 120, 121, or 365(c) or indicate National Stage
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`entry from a PCT application. Providing this information in the application data sheet constitutes the specific reference required by
`
`
`
`
`35 U.S.C. 119(e) or 120, and 37 CFR 1.78(a)(2) or CFR 1.78(a)(4), and need not otherwise be made part of the specification.
`
`Prior Application Status Pending
`Continuity Type
`
`Application Number
`
`
`Continuation of
`
`Prior Application Status Patented
`IRemove I
`Issue Date
`Prior Application Filing Date
`Application
`
`Number Continuity Type Number (YYYY-MM-DD) Patent Number
`(YYYY-MM-DD)
`
`12834281 Continuation of 11980710 2007-10-31 7756297 2010-07-13
`
`
`Prior Application Status Patented
`IRemove I
`Issue Date
`Prior Application Filing Date
`Application
`
`Number Continuity Type Number (YYYY-MM-DD) Patent Number
`(YYYY-MM-DD)
`11980710
`
`Continuation of 10893534
`
`2004-07-19 7401783 2008-07-22
`
`Prior Application Status Patented
`IRemove I
`Issue Date
`Prior Application Filing Date
`Application
`
`Number Continuity Type Number (YYYY-MM-DD) Patent Number
`(YYYY-MM-DD)
`
`10893534 Continuation of
`09612225
`
`2000-07-07 6766036 2004-07-20
`
`Prior Application Status Expired
`Continuity Type
`
`Application Number
`
`09612225
`
`non provisional of
`
`IRemove I
`
`Prior Application Number Filing Date (YYYY-MM-DD)
`
`60142777
`1999-07-08
`
`EFS Web 2.2.2
`
`

`

`PTO/SB/14 (11-08)
`
`
`Approved for use through 09/30/2010. 0MB 0651-0032
`
`
`U.S. Patent and Trademark Office; U.S. DEPARTMENT OF COMMERCE
`
`
`
`
`Under the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless it contains a valid 0MB control number.
`
`
`
`
`
`
`
`Attorney Docket Number 135873.new
`
`
`Data Sheet 37 CFR 1.76 Application Application Number
`
`MOBILE, GAMING OR OTHER DEVICES
`CAMERA BASED SENSING IN HANDHELD,
`Title of Invention
`
`Additional Domestic Benefit/National Stage Data may be generated
`
`
`
`
`within this form
`
`by selecting the Add button.
`
`Add
`
`Foreign Priority Information:
`
`
`of foreign priority and to identify any prior foreign application for which priority is
`
`
`
`
`
`
`
`
`
`This section allows for the applicant to claim benefit
`
`
`
`
`
`
`not claimed. Providing this information in the application data sheet constitutes the claim for priority as required by 35 U.S.C. 119(b)
`and 37 CFR 1.55(a).
`
`IR
`
`
`
`
`
`Application Number
`
`i
`Country
`
`
`
`
`
`
`
`
`Add button.
`
`Assignee Information:
`
`
`
`
`
`
`
`
`of the CFR to have an assignment recorded in the Office.
`
`
`
`Providing this information in the application data sheet does not substitute for compliance with any requirement of part 3 of Title 37
`
`
`
`emove
`
`I
`Suffix
`
`IR
`
`Assionee 1
`If the Assignee is an Organization check here.
`
`
`□
`Prefix
`Middle Name
`Given Name
`
`Family Name
`
`
`
`Mailing Address Information:
`
`Address 1
`
`Address 2
`City
`
`Country ii
`Phone Number
`Email Address
`
`
`button.
`
`
`
`Additional Assignee Data may be generated within this form by selecting the Add
`
`State/Province
`Postal Code
`Fax Number
`
`Add
`
`I I
`
`A signature of the applicant or representative is required in accordance with 37 CFR 1.33 and 10.18. Please see 37
`
`Signature:
`
`
`
`
`
`
`CFR 1.4(d) for the form of the signature.
`Signature Nito A. Ciaravino/
`I Last Name I Ciaravino
`First Name Vito A.
`
`2011-03-18
`Date (YYYY-MM-DD)
`
`Registration Number 62749
`
`EFS Web 2.2.2
`
`emove I
`
`Parent Filing Date (YYYY-MM-DD) Priority Claimed
`0 Yes ® No
`
`dd
`
`I
`
`IA
`
`
`
`Additional Foreign Priority Data may be generated within this form by selecting the
`
`

`

`PTO/SB/14 (11-08)
`
`
`Approved for use through 09/30/2010. 0MB 0651-0032
`
`
`
`U.S. Patent and Trademark Office; U.S. DEPARTMENT OF COMMERCE
`
`
`
`
`Under the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless it contains a valid 0MB control number.
`
`
`
`
`
`
`Application Data Sheet 37 CFR 1.76
`
`
`
`
`135873.new
`Attorney Docket Number
`
`Application Number
`
`Title of Invention CAMERA BASED SENSING IN HANDHELD, MOBILE, GAMING OR OTHER DEVICES
`
`This collection of information is required by 37 CFR 1.76. The information is required to obtain or retain a benefit by the public which
`
`
`
`
`
`
`
`
`
`
`
`
`is to file (and by the USPTO to process) an application. Confidentiality is governed by 35 U.S.C. 122 and 37 CFR 1.14. This
`
`
`
`
`
`
`
`
`
`
`collection is estimated to take 23 minutes to complete, including gathering, preparing, and submitting the completed application data
`
`
`
`
`sheet form to the USPTO. Time will vary depending upon the individual case. Any comments on the amount of lime you require to
`
`
`
`
`
`
`complete this form and/or suggestions for reducing this burden, should be sent to the Chief Information Officer, U.S. Patent and
`
`
`
`
`
`
`
`Trademark Office, U.S. Department of Commerce, P.O. Box 1450, Alexandria, VA 22313-1450. DO NOT SEND FEES OR
`
`
`
`COMPLETED FORMS TO THIS ADDRESS. SEND TO: Commissioner for Patents, P.O. Box 1450, Alexandria, VA 22313-1450.
`
`EFS Web 2.2.2
`
`

`

`
`
`Privacy Act Statement
`
`The Privacy Act of 1974 (P .L. 93-579) requires that you be given certain information in connection with your submission of the attached form related to
`
`
`
`
`
`
`
`
`
`pursuant
`
`
`
`
`to the requirements of the Act, please be advised that: (1) the general authority for the collection
`
`
`a patent application or patent. Accordingly,
`
`
`
`
`
`
`
`
`
`of this information is 35 U.S.C. 2(b)(2); (2) furnishing of the information solicited is voluntary; and (3) the principal purpose for which the information is
`
`
`
`
`
`
`
`used by the U.S. Patent and Trademark Office is to process and/or examine your submission related to a patent application or patent. If you do not
`
`
`
`
`
`
`
`
`furnish the requested information, the U.S. Patent and Trademark Office may not be able to process and/or examine your submission, which may
`
`
`
`
`
`
`result in termination of proceedings or abandonment of the application or expiration of the patent.
`
`
`
`The information provided by you in this form will be subject to the following routine uses:
`
`
`
`
`
`
`
`
`
`1. The information on this form will be treated confidentially to the extent allowed under the Freedom of Information Act (5 U.S.C. 552)
`
`
`
`
`
`
`
`
`
`
`and the Privacy Act (5 U.S.C. 552a). Records from this system of records may be disclosed to the Department of Justice to determine
`
`
`
`
`whether the Freedom of Information Act requires disclosure of these records.
`
`2. A record from this system of records may be disclosed, as a routine use, in the course of presenting evidence to a court, magistrate, or
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`administrative tribunal, including disclosures to opposing counsel in the course of settlement negotiations.
`
`3.
`
`A record in this system of records may be disclosed, as a routine use, to a Member of Congress submitting a request involving an
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`individual, to whom the record pertains, when the individual has requested assistance from the Member with respect to the subject matter of
`the record.
`
`4.
`
`A record in this system of records may be disclosed, as a routine use, to a contractor of the Agency having need for the information in
`
`
`
`
`
`
`
`
`
`
`
`
`order to perform a contract. Recipients of information shall be required to comply with the requirements of the Privacy Act of 1974, as
`
`
`amended, pursuant to 5 U.S.C. 552a(m).
`
`5. A record related to an International Application filed under the Patent Cooperation Treaty in this system of records may be disclosed,
`
`
`
`
`
`
`
`
`
`
`
`
`as a routine use, to the International Bureau of the World Intellectual Property Organization, pursuant to the Patent Cooperation Treaty.
`
`
`
`6. A record in this system of records may be disclosed, as a routine use, to another federal agency for purposes of National Security
`
`
`
`
`
`
`
`
`review (35 U.S.C. 181) and for review pursuant to the Atomic Energy Act (42 U.S.C. 218(c)).
`
`7.
`
`A record from this system of records may be disclosed, as a routine use, to the Administrator, General Services, or his/her designee,
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`during an inspection of records conducted by GSA as part of that agency's responsibility to recommend improvements in records
`
`
`
`
`
`
`
`management practices and programs, under authority of 44 U.S.C. 2904 and 2906. Such disclosure shall be made in accordance with the
`
`
`
`
`
`
`
`
`GSA regulations governing inspection of records for this purpose, and any other relevant (i.e., GSA or Commerce) directive. Such
`
`
`disclosure shall not be used to make determinations about individuals.
`
`A record from this system of records may be disclosed, as a routine use, to the public after either publication of the application pursuan
`
`
`
`
`
`
`
`
`8.
`
`
`
`
`
`
`to 35 U.S.C. 122(b) or issuance of a patent pursuant to 35 U.S.C. 151. Further, a record may be disclosed, subject to the limitations of 37
`
`
`
`
`
`CFR 1.14, as a routine use, to the public if the record was filed in an application which became abandoned or in which the proceedings were
`
`
`
`
`
`
`
`
`
`terminated and which application is referenced by either a published application, an application open to public inspections or an issued
`patent.
`
`
`
`
`
`9. A record from this system of records may be disclosed, as a routine use, to a Federal, State, or local law enforcement agency, if the
`
`
`
`
`
`
`
`
`
`
`
`USPTO becomes aware of a violation or potential violation of law or regulation.
`
`EFS Web 2.2.2
`
`

`

`HANDHELD, MOBILE, GAMING, OR OTHER DEVICES
`CAMERA BASED SENSING IN
`
`
`
`CROSS REFERENCE TO RELATED APPLICATIONS
`
`
`
`
`
`[0001] This application is a continuation of Application No. 12/834,281, filed July 12,
`
`
`
`2010 (now U.S. Patent _____ ), which is a continuation of Application No. 11/980,710,
`
`
`
`
`
`
`
`
`
`
`
`filed Oct. 31, 2007 (now U.S. Patent 7,756,297), which is a continuation of Application No.
`
`
`
`
`
`
`
`
`
`10/893,534, filed Jul. 19, 2004 (now U.S. Patent 7,401,783), which is a continuation of
`
`
`
`
`
`
`
`
`
`
`
`Application No. 09/612,225, filed Jul. 7, 2000 (now U.S. Patent 6,766,036), which claims the
`
`
`
`
`
`
`
`benefit of U.S. Provisional Application No. 60/142,777, filed Jul. 8, 1999.
`
`
`
`
`
`by the inventor having US applications [0002] Cross references to related co-pending
`
`
`
`
`
`
`
`
`
`similar subject matter.
`
`
`
`
`
`
`
`filed Nov. Interfaces: Ser. No. 09/435,854 [0003] 1. Touch TV and other Man Machine
`
`
`
`
`
`8, 1999, now U.S. Pat. No. 7,098,891; which was a continuation of application Ser. No.
`
`
`
`
`
`
`
`
`
`07/946,908, now U.S. Pat. No. 5,982,352;
`
`
`
`
`
`[0004] 2. More Useful Man Machine Interfaces and Applications: Ser. No. 09/433,297
`
`
`
`filed Nov. 3, 1999, now U.S. Pat. No. 6,750,848;
`
`
`
`Ser. No. 09/138,339, Pub. and applications: [0005] 3. Useful Man Machine interfaces
`
`
`
`
`
`Appln. 2002-0036617, now abandoned;
`
`
`
`
`
`filed Jun. 6, 1995, now Ser. No. 08/469,907 [0006] 4. Vision Target based assembly:
`
`
`
`
`
`
`
`
`
`U.S. Pat. No. 6,301,783;
`
`
`
`
`
`[0007] 5. Picture Taking method and apparatus: provisional application 60/133,671, and
`
`
`
`regular application Ser. No. 09/568,552 filed May 11, 2000, now U.S. Pat. No. 7,015,950;
`
`
`
`
`
`

`

`
`
`[0008] 6. Methods and Apparatus
`
`
`
`
`
`
`for Man Machine Interfaces and Related Activity:
`
`
`
`Provisional Application:
`
`provisional application 60/133,673 filed May 11, 1999; and regular
`
`
`
`
`
`
`
`application Ser. No. 09/568,554 filed May 11, 2000, now U.S. Pat. No. 6,545,670;
`
`
`
`and Other Dashboards, Interiors for Automobile [0009] 7. Tactile Touch Screens
`
`
`
`
`
`
`
`
`
`Applications: provisional application Ser. No. 60/183,807; and regular application Ser. No.
`
`
`
`
`
`09/789,538, now U.S. Pat. No. 7,084,859; and
`
`
`
`
`
`
`
`Shopping in Both Present and and Distance Fashion [0010] 8. Apparel Manufacture
`
`
`
`
`
`
`
`Future: provisional application 60/187,397 filed Mar. 7, 2000.
`
`
`
`
`
`
`
`[0011] The disclosures of the following U.S. patents
`
`
`
`and co-pending patent applications
`
`
`
`by the inventor, or the inventor and his colleagues, are incorporated herein by reference:
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`09/435,854 and U.S. Pat. U.S. application Ser. No. [0012] 1. "Man machine Interfaces":
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`No. 5,982,352, and U.S. application Ser. No. 08/290,516, filed Aug. 15, 1994, now U.S. Pat. No.
`
`
`
`
`
`6,008,000, the disclosure of both of which is contained in that of Ser. No. 09/435,854;
`
`
`
`
`
`
`
`
`
`Ser. No. [0013] 2. "Useful Man Machine Interfaces and Applications": U.S. application
`
`
`
`
`
`09/138,339, now Pub. Appln. 2002-0036617;
`
`
`
`U.S. application Ser. Interfaces and Applications": [0014] 3. "More Useful Man Machine
`
`
`
`
`
`
`
`No. 09/433,297;
`
`
`
`and Related Activity": for Man Machine Interfaces [0015] 4. "Methods and Apparatus
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`U.S. Appln. Ser. No. 60/133,673 filed as regular application Ser. No. 09/568,554, now U.S. Pat.
`
`
`
`No. 6,545,670;
`
`
`
`
`
`and Other Dashboards, Interiors for Automobile [0016] 5. "Tactile Touch Screens
`
`
`
`
`
`
`
`
`
`Applications": U.S. provisional Appln. Ser. No. 60/183,807, filed Feb. 22, 2000, now filed as
`
`
`
`
`
`
`
`
`
`reg. application Ser. No. 09/789,538; and
`
`-2-
`
`

`

`
`
`
`
`
`
`andin Both Present Shopping Fashion [0017] 6. "Apparel Manufacture and Distance
`
`
`
`
`
`
`
`Future":
`U.S. Appln. Ser. No.
`
`60/187,397, filed Mar. 7, 2000.
`
`FIELD OF THE INVENTION
`
`
`
`
`
`particularly, but not for computers, input devices [0018] The invention relates to simple
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`necessarily, intended for use with 3-D graphically intensive activities, and operating by optically
`
`
`
`
`
`sensing a human input to a display screen or other object and/or the sensing of human positions
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`or orientations. The invention herein is a continuation in part of several inventions of mine, listed
`
`
`
`
`
`
`
`
`
`above.
`
`
`
`for further useful embodiments [0019] This continuation application seeks to provide
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`improving the sensing of objects. Also disclosed are new applications in a variety of fields such
`
`
`
`as computing, gaming, medicine, and education. Further disclosed are improved systems for
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`display and control purposes.
`
`
`
`
`
`and is analyzed TV cameras whose output [0020] The invention uses single or multiple
`
`
`
`
`
`
`
`
`
`
`
`used as input to a computer, such as a home PC, to typically provide data concerning the location
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`of parts of, or objects held by, a person or persons.
`
`
`
`DESCRIPTION OF RELATED ART
`
`
`
`
`
`[0021] The above mentioned co-pending applications incorporated by reference discuss
`
`
`
`
`
`
`
`
`
`
`
`
`
`many prior art references in various pertinent fields, which form a background for this invention.
`
`
`
`
`
`Some more specific U.S. Patent references are for example:
`
`
`
`
`
`
`[0022]
`
`
`
`DeMenthon--U.S. Pat. Nos. 5,388,059; 5,297,061; 5,227,985
`
`
`[0023]
`
`Cipolla--U.S. Pat. No. 5,581,276
`
`-3-
`
`

`

`
`[0024]
`
`Pugh--U.S. Pat. No. 4,631,676
`
`
`[0025]
`
`Pinckney--U.S. Pat. No. 4,219,847
`
`
`
`DESCRIPTION OF FIGURES
`
`FIG. 1 illustrates
`
`[0026]
`
`a basic computer terminal embodiment of the invention, similar
`
`
`
`
`
`to that disclosed in copending applications.
`
`
`
`
`
`a pixel [0027] FIG. 2 illustrates object tracking embodiments of the invention employing
`
`
`
`
`
`
`
`addressable camera.
`
`
`
`
`
`using intensity variation [0028] FIG. 3 illustrates tracking embodiments of the invention
`
`
`
`
`
`
`
`
`
`to identify and/or track object target datums.
`
`
`
`
`
`in color variation [0029] FIG. 4 illustrates tracking embodiments of the invention using
`
`
`
`
`
`
`
`
`
`
`
`to identify and/or track object target datums.
`
`
`
`
`
`target position m [0030] FIG. 5 illustrates special camera designs for determining
`
`
`
`
`
`
`
`
`
`
`
`addition to providing normal color images.
`
`
`[0031]
`
`
`
`
`
`
`
`FIG. 6 identification and tracking with stereo pairs.
`
`
`[0032]
`
`
`
`FIG. 7 illustrates use of an indicator or co-target.
`
`
`[0033]
`
`
`
`
`
`
`
`FIG. 8 illustrates control of functions with the invention, using a handheld device
`
`
`
`which itself has functions.
`
`
`
`
`
`[0034] FIG. 9 illustrates pointing at an object represented on a screen using a finger or
`
`
`
`
`
`
`
`
`
`
`
`laser pointer, and then manipulating the represented object using the invention.
`
`
`
`
`
`
`
`[0035] FIG. 10 illustrates control of automobile or other functions with the invention,
`
`
`
`using detected knob, switch or slider positions.
`
`
`
`
`
`
`
`[0036] FIG. 11 illustrates a board game embodiment of the invention.
`
`-4-
`
`

`

`
`FIG. 12 illustrates a generic
`
`[0037]
`
`game embodiment of the invention.
`
`
`[0038]
`
`
`
`
`
`FIG. 13 illustrates a game embodiment of the invention, such as might be played
`
`
`
`in a bar.
`
`[0039]
`
`
`
`
`
`FIG. 14 illustrates a laser pointer or other spot designator embodiment of the
`
`
`
`
`
`invention.
`
`
`[0040]
`
`
`
`FIG. 15 illustrates a gesture based flirting game embodiment of the invention.
`
`
`
`
`
`
`[0041]
`
`
`
`
`
`
`
`
`
`FIG. 16 illustrates a version of the pixel addressing camera technique wherein two
`
`
`
`
`
`
`
`
`
`
`
`lines on either side of a 1000 element square array are designated as perimeter fence lines to
`
`
`
`
`
`initiate tracking or other action.
`
`
`[0042]
`
`
`
`
`
`FIG. 17 illustrates a 3-D acoustic 1magmg embodiment of the invention.
`
`
`
`THE INVENTION EMBODIMENTS
`
`FIG. 1
`
`
`[0043]
`
`
`
`The invention herein and disclosed in portions of other copending applications
`
`
`
`
`
`
`
`
`
`noted above, comprehends a combination of one or more TV cameras ( or other suitable electro­
`
`
`
`
`
`
`
`optical sensors) and a computer to provide various position and orientation related functions of
`
`
`
`
`
`
`
`
`
`
`
`
`
`use. It also comprehends the combination of these functions with the basic task of generating,
`
`
`
`
`
`
`
`storing and/or transmitting a TV image of the scene acquired--either in two or three dimensions.
`
`
`
`
`
`
`
`
`
`
`
`of many [0044] The embodiment depicted in FIG. lA illustrates the basic embodiments
`
`
`
`
`
`of my co-pending applications above. A stereo pair of cameras 100 and 101 located on each side
`
`
`
`
`
`
`
`
`
`
`
`
`
`of the upper surface of monitor 102 (for example a rear projection TV of 60 inch diagonal screen
`
`
`
`
`
`
`
`size) with display screen 103 facing the user, are connected to PC computer 106 (integrated in
`
`
`
`
`
`
`
`this case into the monitor housing), for example a 400 Mhz Pentium II. For appearances and
`
`
`
`-5-
`
`

`

`
`a single extensive
`
`protection
`
`cover window may be used to cover both cameras and their
`
`
`
`
`
`associated light sources 110 and 111, typically LEDs.
`
`
`
`
`
`
`
`
`
`to illuminate targets associated [0045] The LEDs in this application are typically used
`
`
`
`
`
`with any of the fingers, hand, feet and head of the user, or objects such as 131 held by a user, 135
`
`
`
`
`
`
`
`
`
`with hands 136 and 137, and head 138. These targets, such as circular target 140 and band target
`
`
`
`
`
`
`
`
`
`141 on object 131 are desirably, but not necessarily, retro-reflective, and may be constituted by
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`the object features themselves (e.g., a finger tip, such as 145), or by features provided on
`
`
`
`
`
`
`
`
`
`
`
`clothing worn by the user (e.g., a shirt button 147 or polka dot 148, or by artificial targets other
`
`
`
`
`
`
`
`
`
`than retroreflectors.
`
`
`
`can be used, for example usmg [0046] Alternatively, a three camera arrangement
`
`
`
`
`
`
`
`
`
`additional camera 144, to provide added sensitivity m certain angular and positional
`
`
`
`
`
`
`
`
`
`
`
`relationships. Still more cameras can be used to further improve matters, as desired.
`
`
`
`
`
`
`
`
`
`Alternatively, and or in addition, camera 144 can be used for other purposes, such as acquire
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`images of objects such as persons, for transmission, storage or retrieval independent of the
`
`
`
`
`
`
`
`
`
`
`
`cameras used for datum and feature location determination.
`
`
`
`
`
`
`
`as for measurement purposes a single camera can suffice [0047] For many applications,
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`well, such as 160 shown in FIG. lB for example, used for simple 2 dimensional (2D)
`
`
`
`measurements in the xy plane perpendicular to the camera axis (z axis), or 3D (xyz, roll pitch
`
`
`
`
`
`
`
`
`
`
`
`
`
`yaw) where a target grouping, for example of three targets is used such as the natural features
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`formed by the two eyes 164, 165 and nose 166 of a human 167. These features are roughly at
`
`
`
`
`
`
`
`
`
`
`
`known distances from each other, the data from which can be used to calculate the approximate
`
`
`
`
`
`
`
`
`
`position and orientation of the human face. Using for example the photogrammetric technique of
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`Pinkney described below, the full 6 degree of freedom solution of the human face location and
`
`
`
`
`
`-6-
`
`

`

`
`
`orientation
`
`
`
`can be achieved to an accuracy limited by the ability of the camera image processing
`
`
`
`software
`
`
`
`
`
`
`
`
`
`
`
`utilized to determine the centroids or other delineating geometric indicators of the
`
`
`
`
`
`
`
`position of the eyes and nose, ( or some other facial feature such as the mouth), and the accuracy
`
`
`
`
`
`
`
`
`
`of the initial imputing of the spacing of the eyes and their respective spacing to the nose. Clearly
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`if a standard human value is used (say for adult, or for a child or even by age) some lessening of
`
`
`
`
`
`
`
`
`
`
`
`precision results, since these spacings are used in the calculation of distance and orientation of
`
`
`
`the face of human 167 from the camera 160.
`
`
`
`accurate case, one might choose [0048] In another generally more photogrammetrically
`
`
`
`
`
`
`
`
`
`
`
`to use four special targets ( e.g., glass bead retro-reflectors, or orange dots) 180-183 on the object
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`185 having known positional relationships relative to each other on the object surface, such as
`
`
`
`
`
`one inch centers. This is shown in FIG. lC, and may be used in conjunction with a pixel
`
`
`
`
`
`
`
`addressable camera such as described in FIG. 2 below, which allows one to rapidly determine the
`
`
`
`
`
`
`
`
`
`
`
`
`
`object position and orientation and track its movements in up to 6 degrees of freedom as
`
`
`
`
`
`
`
`
`
`
`
`
`
`disclosed by Pinkney U.S. Pat. No. 4,219,847 and technical papers referenced therein. For
`
`
`
`
`
`
`
`
`
`
`
`example, the system described above for FIGS. 1 and 2 involving the photogrammetric
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`resolution of the relative position of three or more known target points as viewed by a camera is
`
`
`
`
`
`
`
`known and is described in a paper entitled "A Single Camera Method for the 6-Degree of
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`Freedom Sprung Mass Response of Vehicles Redirected by Cable Barriers" presented by M. C.
`
`
`
`
`
`
`
`van Wijk and H. F. L. Pinkney to The Society of Photo-optical Instrumentation Engineers.
`
`
`
`
`
`
`
`image of the scene a two view stereo can also acquire [0049] The stereo pair of cameras
`
`
`
`
`
`
`
`
`
`as well, which can be displayed in 3D using stereoscopic or auto-stereoscopic means, as well as
`
`
`
`
`
`
`
`
`
`transmitted or recorded as desired.
`
`-7-
`
`

`

`
`[0050] In many
`
`applications of the foregoing invention
`
`
`
`it is desirable not just to use a
`
`
`
`large screen but in fact one capable of displaying life size images. This particularly relates to
`
`
`
`
`
`
`
`
`
`
`
`
`
`human scaled images, giving a life-like presence to the data on the screen. In this way the natural
`
`
`
`
`
`
`
`
`
`
`
`response of the user with motions of hands, head, arms, etc., is scaled in "real" proportion to the
`
`
`
`data being presented.
`
`FIG. 2
`
`
`
`useful with the [0051] This embodiment and others discloses special types of cameras
`
`
`
`
`
`
`
`
`
`
`
`invention. In the first case, that of FIG. 2A, a pixel addressable camera such as the MAPP2200
`
`
`
`
`
`
`
`
`
`made by IVP corporation of Sweden is used, which allows one to do many things useful for
`
`
`
`
`
`
`
`
`
`rapidly determining location of objects, their orientation and their motion.
`
`
`
`
`
`
`
`
`
`
`
`201 of a 2A, an approximately circular image [0052] For example, as shown in FIG.
`
`
`
`
`
`
`
`target datum such as 180 on object 185 of FIG. lC may be acquired by scanning the pixel
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`elements on a matrix array 205 on which the image is formed. Such an array in the future will
`
`
`
`
`
`have for example 1000.times.1000 pixels, or more (today the largest IVP makes is
`
`
`
`
`
`512.times.512. The IVP also is not believed to be completely randomly addressable, which some
`
`
`
`
`
`
`
`
`
`future arrays will be).
`
`
`
`
`
`
`
`after the array 205 has been [0053] As an illustration, computer 220 determines,
`
`
`
`interrogated, that the centroid "x, y" of the pixel elements on which the target image lies is at
`
`
`
`
`
`pixel x=500, y=300 (including a sub-fraction thereof in many cases). The centroid location can
`
`
`
`
`
`
`
`
`
`
`
`
`
`be determined for example by the moment method disclosed in the Pinkney patent, referenced
`
`
`
`
`
`
`
`
`
`above.
`
`-8-
`
`

`

`
`[0054] The target
`
`
`
`and such in this case is defined as a contrasting point on the object,
`
`
`
`
`
`contrast can be in color as well as, or instead of, intensity. Or with some added preprocessing, it
`
`
`
`
`
`
`
`
`
`can be a distinctive pattern on the object, such as a checkerboard or herringbone.
`
`
`
`Subsequent Tracking
`
`
`
`
`
`
`
`
`
`target image, it is now only necessary [0055] To subsequently track the movement of this
`
`
`
`
`
`to look in a small pixel window composed of a small number of pixels around the target. For
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`example the square 230 shown, as the new position x'y' of the target image cannot be further
`
`
`
`
`
`
`
`distant within a short period of time elapsed from the first scan, and in consideration of the small
`
`
`
`
`
`
`
`
`
`required time to scan the window.
`
`
`
`
`
`[0056] For example, if the window is 100.times.100 pixels, this can be scanned in 1
`
`
`
`millisecond or less with such a pixel addressing camera, by interrogating only those pixels in the
`
`
`
`
`
`
`
`
`
`
`
`window, while still communicating with the camera over a relatively slow USB serial link of 12
`
`
`
`
`
`
`
`
`
`mb transmission rate (representing 12,000 pixel gray level values in one millisecond).
`
`
`
`
`
`
`
`
`
`field, once the starting target to scan the whole [0057] One thus avoids the necessity
`
`
`
`
`
`
`
`
`
`image position is identified. This can be known by an initial scan as mentioned, or can be known
`
`
`
`
`
`
`
`
`
`
`
`
`
`by having the user move an object with a target against a known location with respect to the
`
`
`
`
`
`camera such as a mechanical stop, and then indicate that tracking should start either by verbally
`
`
`
`
`
`
`
`saying so with voice recognition, or by actuating a control key such as 238 or whatever.
`
`
`
`
`
`
`
`[0058] It is noted that if the tracking window is made large enough, then it can
`
`
`
`
`
`
`
`encompass a whole group of datums, such as 180-183 on an object.
`
`
`
`
`
`FIG. 2B Reduction in Acquisition Time
`
`
`
`2B. One camera is shown in FIG. of such a pixel addressing [0059] Another application
`
`
`
`
`
`
`
`
`
`can look at the whole field, x y of the camera, 240, but only address say every IO.sup.th pixel
`
`
`
`
`
`
`
`
`
`-9-
`
`

`

`
`
`
`
`in a field of 1 million such as 250, 251 and 252, in each direction, i.e., for a total 10,000 pixels
`
`
`(1000.times.1000, say).
`
`
`
`
`
`
`
`fraction of the pixels in the image, 220 simply queries this [0060] In this case computer
`
`
`
`
`
`
`
`knowing apriori that the target image such as 260 will have an image size larger than 10.times.10
`
`
`
`
`
`
`
`
`
`
`
`pixels, and must be detectable, if of sufficient contrast, by one of the queried pixels. (For smaller
`
`
`
`
`
`
`
`or larger target images, the number and spacing of queried pixels can be adjusted accordingly).
`
`
`
`
`
`
`
`
`
`
`
`
`
`This for example, allows one to find approximate location of targets with only 1/100 the pixel
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`interrogation time otherwise needed, for example, plus any gain obtained as disclosed above, by
`
`
`
`
`
`
`
`knowing in what region of the image to look (for example during tracking, or given some apriori
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`knowledge of approximate location due to a particular aspect of the physical arrangement or the
`
`
`
`
`
`program in question).
`
`
`
`
`
`addressing can [0061] Once a target has been approximately found as just described, the
`
`
`
`
`
`
`
`
`
`
`
`
`
`be optimized for that region of the image only, as disclosed in subsequent tracking section above.
`
`
`
`
`
`in a millisecond or two acquisition for target [0062] Given the invention, the potential
`
`
`
`
`
`
`
`
`
`
`
`
`
`thus is achievable with simple pixel addressable CMOS cameras coming on stream now (today
`
`
`
`
`
`
`
`
`
`costing under $50), assuming the target points are easily identifiable from at least one of
`
`
`
`
`
`
`
`
`
`brightness (over a value), contrast (with respect to surroundings), color, color contrast, and more
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`difficult, shape or pattern (e.g., a plaid, or herringbone portion of a shirt). This has major
`
`
`
`
`
`
`
`
`
`
`
`
`
`ramifications for the robustness of control systems built on such camera based acquisiti