Case 2:21-cv-00040-JRG Document 70-11 Filed 09/02/21 Page 1 of 66 PageID #: 1431
`
`Exhibit P
`
`

`

`Case 2:21-cv-00040-JRG Document 70-11 Filed 09/02/21 Page 2 of 66 PageID #: 1432
`
`Case 2:21-cv-00040-JRG Document TMAIttDeiwei 1432
`
`
`
`
`
`
`
`
`
`
`
`
`a2) United States Patent
`US 8,405,604 B2
`(0) Patent No.:
`
`
`
`
`
`
`
`Mar.26, 2013
`(45) Date of Patent:
`Pryoretal.
`
`
`
`
`3/1979 Birk etal.
`4,146,924 A
`
`
`
`
`
`4/1980 Giguere
`4,199,137 A
`8/1980 Pinkneyetal.
`4,219,847 A
`
`
`
`
`
`
`
`12/1981 Best
`4,305,131 A
`
`
`
`
`3/1983 Thornton
`4,375,674 A
`
`
`
`
`
`
`
`
`8/1983 DiMatteoet al.
`4,396,945 A
`Inventors: Timothy R. Pryor, Tecumseh (CA);
`
`
`
`
`
`
`
`
`
`
`3/1984 Sakowet al.
`4,435,835 A
`Peter Smith, Ann Arbor, MI (US)
`
`
`
`
`10/1984 Green
`4,475,122 A
`
`
`
`
`
`
`
`
`
`
`
`11/1984 Kasday
`4,484,179 A
`Assignee: Motion Games, LLC, Austin, TX (US)
`
`
`
`
`
`9/1985 Alleset al.
`4,542,375 A
`7/1986 Maloomian
`4,602,280 A
`
`
`
`
`
`
`
`
`
`
`
`(*) Notice:|Subject to any disclaimer, the term ofthis
`4,613,942 A
`9/1986 Chen
`
`
`
`
`
`
`
`
`4,629,319 A
`12/1986 Clarke et al.
`patent is extended or adjusted under 35
`
`
`
`
`
`
`
`4,631,676 A
`12/1986 Pugh
`US.C. 154(b) by 1063 days.
`4,654,872 A
`3/1987 Hisanoetal.
`
`
`
`
`
`
`
`
`4,686,374 A
`8/1987 Liptay-Wagneret al.
`
`
`
`4,687,200 A
`8/1987 Shirai
`4,988,981 A *
`1/1991 Zimmerman etal.
`
`
`
`
`(Continued)
`
`
`FOREIGN PATENT DOCUMENTS
`
`
`
`63167923
`7/1988
`
`
`
`
`
`
`
`
`
`
`
`........ 345/158
`
`
`
`
`
`
`
`
`(54)
`
`
`
`(75)
`
`
`
`(73)
`
`ADVANCED VIDEO GAMING METHODS FOR
`
`
`
`
`
`
`
`
`
`EDUCATION AND PLAY USING CAMERA
`BASED INPUTS
`
`
`
`
`
`
`
`
`
`
`
`
`(21)
`
`(22)
`
`(65)
`
`
`
`Filed:
`
`
`
`
`
`
`
`Appl. No.: 11/980,716
`
`
`
`Oct. 31, 2007
`
`
`Prior Publication Data
`
`
`
`
`
`
`US 2008/0122786 Al
`May 29, 2008
`
`
`
`JP
`
`
`
`
`
`
`
`
`Related U.S. Application Data
`
`
`
`
`
`Continuation of application No. 11/429,305, filed on
`
`
`
`
`
`
`May8, 2006, which is a division of application No.
`
`
`
`
`
`
`
`
`
`10/622,812, filed on Jul. 21, 2003, now Pat. No.
`
`
`
`
`
`
`7,042,440, which is a division of application No.
`
`
`
`
`
`
`
`
`09/138,285, filed on Aug. 21, 1998, now Pat. No.
`6,720,949.
`
`
`
`
`
`
`
`
`Provisional application No. 60/056,639, filed on Aug.
`
`
`
`
`
`22, 1997, provisional application No. 60/059,561,
`
`
`
`
`filed on Sep. 19, 1997.
`
`
`Int. Cl.
`
`
`
`(2006.01)
`G09G 5/00
`
`
`
`
`
`ceecccccccccscccesteeerteeeteceneees 345/156; 345/173
`UWS. C1.
`
`
`
`
`
`
`Field of Classification Search ........... 345/156-158
`
`
`
`
`
`
`
`See application file for complete search history.
`
`
`References Cited
`
`
`
`
`(60)
`
`
`
`(60)
`
`
`
`
`
`
`
`
`
`(51)
`
`(52)
`(58)
`
`(56)
`
`U.S. PATENT DOCUMENTS
`
`
`
`
`
`11/1974 Mueller
`3,846,826 A
`4,014,000 A
`3/1977 Unoetal.
`
`
`
`
`
`
`
`
`
`
`
`OTHER PUBLICATIONS
`
`
`
`
`
`
`
`
`Baleset al., “Marking Parts to Aid Robot Vision”, NASA Technical
`
`
`
`
`Paper 1819, Apr, 1981.
`
`
`
`
`
`
`Primary Examiner — Kevin M Nguyen
`
`
`
`
`Assistant Examiner — Kemneth B Lee, Jr.
`
`
`
`
`
`
`
`(74) Attorney, Agent, or Firm — Warner Norcross & Judd
`LLP
`
`
`
`
`
`
`(57)
`
`
`ABSTRACT
`
`
`
`
`
`
`
`
`This application discloses active movement based and other
`
`
`
`
`
`
`
`
`
`video games which may be played overthe internet as well in
`
`
`
`
`
`
`
`
`
`the homeor elsewhere. One or two or more personscan play,
`
`
`
`
`
`
`
`
`generally using one or more camerasto determine therelation
`
`
`
`
`
`
`
`of an object or a part of a person to a video display on which
`
`
`
`
`
`
`
`gamerelated graphics are displayed. Determination of object
`
`
`
`
`
`
`
`
`position and orientation in up to 6 degrees of freedom is
`
`
`
`
`
`
`
`
`disclosed, at data rates sufficient to make game play reward-
`
`ing.
`
`
`
`
`14 Claims, 33 Drawing Sheets
`
`
`
`
`
`
`
`
`
`FAY
`
`|
`
`
`
`
`
`DEFTS_00000037
`
`DEFTS_00000037
`
`

`

`Case 2:21-cv-00040-JRG Document 70-11 Filed 09/02/21 Page 3 of 66 PageID #: 1433
`Case 2:21-cv-00040-JRG Document 70-11 Filed 09/02/21 Page 3 of 66 PagelD #: 1433
`
`
`
`US 8,405,604 B2
`
`
`Page 2
`
`
`
`
`U.S. PATENT DOCUMENTS
`
`
`
`
`
`
`5,297,061 A *
`3/1994 Dementhonetal........... 345/180
`
`
`
`
`
`5,325,472 A
`6/1994 Horiuchiet al.
`
`
`
`
`
`
`5,354,057 A * 10/1994 Pruittet al. voce 463/5
`
`
`
`
`5.521.616 A
`5/1996 Capperet al.
`
`
`
`5'566.283 A
`10/1996 Modegi et al.
`
`
`
`5,594,469 A
`1/1997 Freeman etal.
`
`
`
`
`5,616,078 A
`4/1997 Oh
`
`
`
`
`
`
`
`5,624,117 A *
`4/1997 Ohkuboetal... 273/148 B
`
`
`
`
`
`5,772,522 A
`6/1998 Nesbit et al.
`
`
`
`5,870,771 A
`2/1999 Oberg
`
`
`
`
`
`5,878,174 A
`3/1999 Stewart et al.
`
`
`
`
`§,889,505 A
`3/1999 Toyamaetal.
`
`
`
`
`
`5,890,963 A *
`4/1999 Ven cece 463/42
`
`
`
`
`5,913,727 A
`6/1999 Ahdoot
`
`
`
`
`
`5,926,168 A
`7/1999 Fan
`
`
`
`
`
`
`
`
`
`10/1999 Maedaetal.
`5,966,310 A
`9
`
`
`Cons’s0s A yoo eek
`
`
`
`
`et
`6.049.327 A
`4/2000 Walker
`al
`alxer et al.
`ete
`5
`
`
`
`6,057,856 A
`5/2000 Miyashita etal.
`
`
`
`
`6,084,979 A
`7/2000 Kanadeet al.
`
`
`
`
`6,097,369 A
`8/2000 Wambach
`
`
`
`
`
`
`6,166,746 A * 12/2000 Inadaetal. ou. 345/474
`
`
`
`
`
`6,198,487 Bl
`3/2001 Fortenbery etal.
`
`
`
`
`6,342,917 Bl
`1/2002 Amenta
`
`
`
`
`
`
`6,453,180 Bl
`9/2002 Endoh etal.
`
`
`
`
`6,597,817 Bl
`7/2003 Silverbrook
`
`
`
`
`
`6,727,887 Bl
`4/2004 Levineetal.
`
`
`
`
`6,954,906 B1
`10/2005 Kamachietal.
`/
`;
`
`
`* cited by examiner
`
`
`
`
`
`
`
`
`
`
`
`DEFTS_00000038
`
`DEFTS_00000038
`
`

`

`Case 2:21-cv-00040-JRG Document 70-11 Filed 09/02/21 Page 4 of 66 PageID #: 1434
`Case 2:21-cv-00040-JRG Document 70-11 Filed 09/02/21 Page 4 of 66 PagelD #: 1434
`
`
`U.S. Patent
`
`
`
`
`Mar. 26, 2013
`
`
`
`
`
`Sheet 1 of 33
`
`
`
`US 8,405,604 B2
`
`
`
`
`
`
`
`
`
`
`FIG. 1a
`
`DEFTS_00000039
`
`DEFTS_00000039
`
`

`

`Case 2:21-cv-00040-JRG Document 70-11 Filed 09/02/21 Page 5 of 66 PageID #: 1435
`Case 2:21-cv-00040-JRG Document 70-11 Filed 09/02/21 Page 5 of 66 PagelD #: 1435
`
`
`U.S. Patent
`
`
`
`
`Mar. 26, 2013
`
`
`
`
`Sheet 2 of 33
`
`
`
`US 8,405,604 B2
`
`
`
`
`
`41
`
`
`FIG. 1b
`
`DEFTS_00000040
`
`DEFTS_00000040
`
`

`

`Case 2:21-cv-00040-JRG Document 70-11 Filed 09/02/21 Page 6 of 66 PageID #: 1436
`Case 2:21-cv-00040-JRG Document 70-11 Filed 09/02/21 Page 6 of 66 PagelD #: 1436
`
`
`U.S. Patent
`
`
`
`
`Mar. 26, 2013
`
`
`
`
`Sheet 3 of 33
`
`
`
`US 8,405,604 B2
`
`
`
`
`
`
`
`
`
`83
`
`
`
`DEFTS_00000041
`
`DEFTS_00000041
`
`

`

`Case 2:21-cv-00040-JRG Document 70-11 Filed 09/02/21 Page 7 of 66 PageID #: 1437
`Case 2:21-cv-00040-JRG Document 70-11 Filed 09/02/21 Page 7 of 66 PagelD #: 1437
`
`
`U.S. Patent
`
`
`
`
`Mar. 26, 2013
`
`
`
`
`Sheet 4 of 33
`
`
`
`
`
`US 8,405,604 B2 OTHER INPUTS OPTIONAL
`
`
`
`
`SUCH AS VOICE
`
`
`
`
`
`
`
`DEFTS_00000042
`
`DEFTS_00000042
`
`

`

`Case 2:21-cv-00040-JRG Document 70-11 Filed 09/02/21 Page 8 of 66 PageID #: 1438
`Case 2:21-cv-00040-JRG Document 70-11 Filed 09/02/21 Page 8 of 66 PagelD #: 1438
`
`
`U.S. Patent
`
`
`
`
`Mar. 26, 2013
`
`
`
`
`Sheet 5 of 33
`
`
`
`US 8,405,604 B2
`
`
`
`
`
`
`
`MAPPING ONE RGB COMPONENT OF ACOLOR
`
`
`
`
`
`
`
`HERE, THE RED COMPONENTIS USED. THE SAME
`
`
`
`
`
`
`
`PROCESS CAN BE USED FOR THE GREEN AND BLUE
`
`
`COMPONENTSAS WELL.
`
`
`
`Ar
`
`
`
`
`
`
`
`
`- 1S THE RED COMPONENT OF THE AQUA COLOR
`
`
`
`
`
`
`
`
`
`
`Or - IS THE RED COMPONENT OF THE ORANGE COLOR
`
`
`
`
`
`
`
`
`
`
`Pr - 1S THE RED COMPONENTOF THE PIXEL COLOR
`
`
`
`
`
`
`
`
`
`
`Cr - IS THE RED COMPONENT OF THE COLOR ADJUSTED
`
`
`TO BE BETWEEN AAND O
`
`
`
`Cr = (Pr - Ar) f (Or - Ar)
`
`
`
`
`
`
`
`
`
`Cr = (Pr - Ar) / (Or - Ar)
`
`
`
`
`
`
`
`
`
`
`
`
`FIG. Te
`
`
`
`DEFTS_00000043
`
`DEFTS_00000043
`
`

`

`Case 2:21-cv-00040-JRG Document 70-11 Filed 09/02/21 Page 9 of 66 PageID #: 1439
`Case 2:21-cv-00040-JRG Document 70-11 Filed 09/02/21 Page 9 of 66 PagelD #: 1439
`
`
`U.S. Patent
`
`
`
`
`Mar. 26, 2013
`
`
`
`
`Sheet 6 of 33
`
`
`
`US 8,405,604 B2
`
`915
`
`
`
`206
`
`
`
`17
`
`
`
`216
`
`
`
`205
`
`
`
`220
`
`
`
`
`FIG. 2a
`
`DEFTS_00000044
`
`DEFTS_00000044
`
`

`

`Case 2:21-cv-00040-JRG Document 70-11 Filed 09/02/21 Page 10 of 66 PageID #: 1440
`Case 2:21-cv-00040-JRG Document 70-11 Filed 09/02/21 Page 10 of 66 PagelD #: 1440
`
`U.S. Patent
`
`Mar. 26, 2013
`
`US 8,405,604 B2
`
`Sheet 7 of 33
`
`DEFTS_00000045
`
`DEFTS_00000045
`
`

`

`Case 2:21-cv-00040-JRG Document 70-11 Filed 09/02/21 Page 11 of 66 PageID #: 1441
`Case 2:21-cv-00040-JRG Document 70-11 Filed 09/02/21 Page 11 of 66 PagelD #: 1441
`
`
`U.S. Patent
`
`
`
`
`Mar. 26, 2013
`
`
`
`
`Sheet 8 of 33
`
`
`
`
`
`US 8,405,604 B2
`
`
`
`FIG. 3a
`
`DEFTS_00000046
`
`DEFTS_00000046
`
`

`

`Case 2:21-cv-00040-JRG Document 70-11 Filed 09/02/21 Page 12 of 66 PageID #: 1442
`Case 2:21-cv-00040-JRG Document 70-11 Filed 09/02/21 Page 12 of 66 PagelD #: 1442
`
`
`U.S. Patent
`
`
`
`
`Mar. 26, 2013
`
`
`
`
`Sheet 9 of 33
`
`
`
`US 8,405,604 B2
`
`337
`
`
`
`POF
`
`336
`
`
`
`350
`
`
`
`342
`
`
`
`
`FIG. 3b
`
`DEFTS_00000047
`
`DEFTS_00000047
`
`

`

`Case 2:21-cv-00040-JRG Document 70-11 Filed 09/02/21 Page 13 of 66 PageID #: 1443
`Case 2:21-cv-00040-JRG Document 70-11 Filed 09/02/21 Page 13 of 66 PagelD #: 1443
`
`
`U.S. Patent
`
`
`
`Mar. 26, 2013
`
`
`
`
`Sheet 10 of 33
`
`
`
`
`
`
`US 8,405,604 B2
`
`
`
`
`FIG. 4
`
`DEFTS_00000048
`
`DEFTS_00000048
`
`

`

`Case 2:21-cv-00040-JRG Document 70-11 Filed 09/02/21 Page 14 of 66 PageID #: 1444
`Case 2:21-cv-00040-JRG Document 70-11 Filed 09/02/21 Page 14 of 66 PagelD #: 1444
`
`
`U.S. Patent
`
`
`
`
`Mar. 26, 2013
`
`
`
`
`Sheet 11 of 33
`
`
`
`US 8,405,604 B2
`
`
`
`305
`
`912
`
`/
`
`511
`
`DEFTS_00000049
`
`DEFTS_00000049
`
`

`

`
`SS
`
`554
`
`
`
`Case 2:21-cv-00040-JRG Document 70-11 Filed 09/02/21 Page 15 of 66 PageID #: 1445
`Case 2:21-cv-00040-JRG Document 70-11 Filed 09/02/21 Page 15 of 66 PagelD #: 1445
`
`
`U.S. Patent
`
`
`
`
`Mar. 26, 2013
`
`
`
`
`Sheet 12 of 33
`
`
`
`US 8,405,604 B2
`
`593
`
` PUPPETSLIPS
`
`
`
`
`
`540
`
`
`
`SAN FRANCISCO, CA
`
`
`BOSTON , MA
`
`545
`
`
`
`541
`
`
`
`999
`
`
`
`
`FIG. 5c
`
`DEFTS_00000050
`
`DEFTS_00000050
`
`

`

`Case 2:21-cv-00040-JRG Document 70-11 Filed 09/02/21 Page 16 of 66 PageID #: 1446
`Case 2:21-cv-00040-JRG Document 70-11 Filed 09/02/21 Page 16 of 66 PagelD #: 1446
`
`
`U.S. Patent
`
`
`
`
`Mar. 26, 2013
`
`
`
`
`Sheet 13 of 33
`
`
`
`US 8,405,604 B2
`
`
`
`
`
`
`
`630
`
`
`
`
`FIG. 6
`
`DEFTS_00000051
`
`DEFTS_00000051
`
`

`

`Case 2:21-cv-00040-JRG Document 70-11 Filed 09/02/21 Page 17 of 66 PageID #: 1447
`Case 2:21-cv-00040-JRG Document 70-11 Filed 09/02/21 Page 17 of 66 PagelD #: 1447
`
`
`U.S. Patent
`
`
`
`
`Mar. 26, 2013
`
`
`
`
`Sheet 14 of 33
`
`
`
`US 8,405,604 B2
`
`
`STEP |
`
`
`
`
`
`
`
`
`CAPTURE PIXEL FIELD OF EVERY CAMERA FOR NEWVIDEO FRAME. DISTINGUISH EVERY TARGET INEACH
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`STEP?
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`STEP 3
`
`
`
`
`
`
`
`
`
`
`
`
`
`DETERMINE THE (0 OF THE WORK TOOL FROM THE UNIQUE TARGET ANO SUBTARGET, COLOR, SHAPE, AND
`
`
`
`
`
`
`
`
`
`ORIENTATION INFORMATION. THIS 1S DONE BY FINDING THE BEST MATCH OF THIS TARGET INFORMATION TO BE
`
`
`
`
`
`
`
`
`
`
`STORED IN A TABLE. EACH PATTERN STORED IN THE TABLE HAS A CORRESPONDINGID NUMBER.
`
`
`STEP 4
`
`
`
`
`
`
`
`
`DETERMINE X,Y,2, AND ORIENTATION ANGLES A1, A2, 43 OF TARGETS USING PHOTOGRAPHIC OR LOOKUPTABLE
`
`
`PATTERN MATCHING TECHNIQUES.
`
`
`
`STEP §
`
`
`
`
`
`
`
`
`COMPUTE THE LOCATION AND ORIENTATION OF THE WORK TOOL'S “ACTION TIP” BY APPLYING AN OFFSET VECTOR
`
`
`
`
`
`
`
`
`
`
`
`
`TO THE TARGET POSITION AND ANOTHER VECTOR CALCULATION IS USED TO MODIFY THE ORIENTATION. THIS
`
`
`
`
`
`
`
`
`INFORMATION {S ALSO CONVERTED 10 THE PROPERAPPLICATION COORDINATES. STORE THIS INFORMATION,
`
`STEP 6
`
`
`
`
`
`
`
`
`
`
`
`
`
`CHECK 10 SEE IF ANY INTERRUPT BUTTON HAS BEEN CHANGED.IF NO, REPEAT FIRST STEPS, ELSE GO ON.
`
`STEP 7A
`
`
`
`
`
`
`
`
`
`
`DISTINGUISH UNIQUE COMBINATIONS OF INTERRUPT BUTTONS ANO TARGET PATH INFORMATION (LOCATION,
`
`
`ORIENTATIONANDTIMING PATTERNS}
`
`
`
`
`
`
`1. ONE UNIQUE BUTTON/PATH COMBINATION DETERMINES PATH LOCATION, ORIENTATION AND TIMING
`
`
`
`
`INFORMATION (REFERRED TO AS PATHDATA),
`
`
`
`
`
`
`
`
`
`
`2, ALL OTHER COMBINATIONSIDENTIFY INPUT PARAMETERS OR OBJECTS OR FUNCTIONS OR PROGRAM
`
`
`
`INSTRUCTIONS (REFERRED TO AS OBJECTDATA).
`
`
`STEP 7B
`
`
`
`
`
`
`
`
`IF THIS [S PATHDATA, CONVERTIT TO THE APPROPRIATE TASK PATH INFORMATION ANO USE THE EXTRA
`
`
`
`
`
`
`
`
`TARGET INFORMATION BEYOND THAT NEEDED FOR THE PATH TO DEFINE CONTROL PARAMETERS.
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`STEP 70
`
`
`
`
`
`
`
`
`
`
`IF THIS IS OBJECTDATA,FIND THE BEST MATCH OF A TARGETPATH OR SET OF TARGET PATHSINCLUDING
`
`
`
`
`
`THE SEVEN DIMENSIONAL DATA (LOCATION, ORIENTATION AND TIME) OF EACH TARGET AND THEIR
`
`
`
`
`CORRESPONDING WORK TOOLIDS. ASSOCIATE A UNIQUE NUMBER (REFER TO AS AQUANT) TO THE
`
`
`
`
`ABSOLUTE OR RELATIVE PATH INFORMATION.
`
`
`
`STEP §
`
`
`
`
`
`
`
`
`
`INPUT THE QUANTS TO THE CONVERSION TABLE AND OUTPUT CORRESPONDING PROGRAM MENU SELECTIONS,
`
`
`
`
`
`
`
`
`MACRO INITIATIONS, OBJECT SELECTIONS, FUNCTION CALLS, PARAMETER SELECTIONS, ETC.
`IN OTHER
`
`
`
`
`
`
`
`
`
`
`WORDS, THE CONVERSION TABLE OUTPUT CAN BE USED TO OEFINE PROGRAM COMMANDS, OR REDEFINE THE
`
`
`
`
`
`
`
`
`
`OBJECT THATIS ALIASED BY THE TARGETED TOOL OR ITS CORRESPONDING FUNCTIONS OR PARAMETERS.
`
`
`
`
`
`
`
`
`
`
`
`
`AUSER DEFINED CONVERSION TABLE ALLOWS USERS FLEXIBILITY SUCH AS TO DEFINE THE PATH PATTERN THA
`
`
`
`
`
`HESHE FINDS MOST COMFORTABLE.
`
`
`STEP 9
`
`
`
`
`
`
`APPLY CONVERSION TABLE OUTPUT TOA COMPUTER PROGRAM ENTHER DIRECTLY OR VIA LINK SUCH AS INTERNET,
`
`FIG. 7
`
`
`
`
`
`DEFTS_00000052
`
`DEFTS_00000052
`
`

`

`Case 2:21-cv-00040-JRG Document 70-11 Filed 09/02/21 Page 18 of 66 PageID #: 1448
`Case 2:21-cv-00040-JRG Document 70-11 Filed 09/02/21 Page 18 of 66 PagelD #: 1448
`
`
`Mar. 26, 2013
`
`
`
`
`Sheet 15 of 33
`
`
`
`US 8,405,604 B2
`
`
`U.S. Patent
`
`
`
`
`
`
`
`
`FIG. 8a
`
`DEFTS_00000053
`
`DEFTS_00000053
`
`

`

`Case 2:21-cv-00040-JRG Document 70-11 Filed 09/02/21 Page 19 of 66 PageID #: 1449
`Case 2:21-cv-00040-JRG Document 70-11 Filed 09/02/21 Page 19 of 66 PagelD #: 1449
`
`
`U.S. Patent
`
`
`
`
`Mar. 26, 2013
`
`
`
`
`Sheet 16 of 33
`
`
`
`US 8,405,604 B2
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`FIG. 8b
`
`DEFTS_00000054
`
`DEFTS_00000054
`
`

`

`Case 2:21-cv-00040-JRG Document 70-11 Filed 09/02/21 Page 20 of 66 PageID #: 1450
`Case 2:21-cv-00040-JRG Document 70-11 Filed 09/02/21 Page 20 of 66 PagelD #: 1450
`
`
`U.S. Patent
`
`
`
`
`Mar. 26, 2013
`
`
`
`
`Sheet 17 of 33
`
`
`
`
`
`US 8,405,604 B2
`
`
`
`
`FIG. 8c
`
`DEFTS_00000055
`
`DEFTS_00000055
`
`

`

`Case 2:21-cv-00040-JRG Document 70-11 Filed 09/02/21 Page 21 of 66 PageID #: 1451
`Case 2:21-cv-00040-JRG Document 70-11 Filed 09/02/21 Page 21 of 66 PagelD #: 1451
`
`
`U.S. Patent
`
`
`
`
`Mar.26, 2013
`
`
`
`
`Sheet 18 of 33
`
`
`
`US 8,405,604 B2
`
`
`
`
`
`PREREGISTER THE LOCATION AND ORIENTATION OF A
`
`
`
`
`SET OF OBJECTS WITH FREEFORM ATTACHMENTS
`
`
`
`
`
`
`
`
`
`
`OBJECTS
`
`REMAINING?
`
`
`
`YES
`
`NO
`
`
`
`
`
`
`
`
`
`
`
`ATTACH OBJECT TO OBJECT HOLDER
`
`
`
`
`ATTACH OBJECT HOLDER TO BASEBOARD
`
`
`
`
`
`
`ROTATE AND TRANSLATE OBJECT HOLDER
`
`
`TO DESIRED POSITION
`
`
`
`
`
`
`
`
`
`
`
`
`
`ATTACH APPROPRIATE TARGET TO OBJECT
`
`
`
`
`
`HOLDER SO THAT THE TARGETIS WITHIN THE
`
`
`
`
`
`TARGET VOLUME WITH AN ORIENTATION
`
`
`
`
`
`
`
`
`ROUGHLY FACING THE LENS PLANE OF THE
`
`CAMERA
`
`
`
`
`
`
`
`ENTER THE OBJECT ID, THE TARGETID, AND
`
`
`
`
`THE ID OF THE ATTACHMENT LOCATION OF
`
`
`
`
`
`
`THE OBJECT HOLDER WHERE THE TARGETIS
`
`ATTACHED
`
`
`
`
`
`
`
`
`
`
`
`
`
`SO THAT THE TARGET1S WITHIN THE TARGET
`
`
`
`
`VOLUME WITH AN ORIENTATION ROUGHLY
`
`
`
`
`
`FACING THE LENS PLANE OF THE CAMERA
`
`
`
`
`
`
`
`
`
`ENTER THE BASEBOARDID, THE TARGET1D.
`
`
`
`
`
`ANO THE ID OF THE TARGET ATTACHMENT
`LOCATION ON THE BASEBOARD
`
`
`
`
`
`
`
`
`
`
`START THE CAMERA SYSTEM, ACQUIRE THE
`
`
`
`
`TARGET, COMPUTE THE ORIENTATION AND
`
`
`
`
`
`LOCATION OF THE BASEBOARD, UPDATE THE
`
`
`COMPUTER DATABASE
`
` ATTACHED APPRORIATE TARGET TO BASEBOARD
`
`
`REMOVE TARGET
`
`
`
`COMPUTER DATABASE
`
`
`
`
`
`
`START THE CAMERA SYSTEM, ACQUIRE THE
`
`
`
`
`
`TARGET, COMPUTE THE ORIENTATION AND
`
`
`
`
`
`LOCATION OF THE OBJECT, UPDATE THE
`
`
`
`
`REMOVE TARGET
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`ADD TO A DATABASE DEFINING THIS OBJECT
`
`
`
`
`
`INCLUDING A) THE X,Y OF THE OBJECT CENTER.
`
`
`
`
`B) THE OBJECT PERIMETER PROJECTED ONTO
`
`
`
`
`
`
`THE BASEBOARD, C) THE GEOMETRIC
`
`
`
`
`
`
`DEFINITION OF THE OBJECT, D) MECHANICAL
`
`
`
`
`CHARACTERISTICS SUCH AS HOW A STEERING
`
`
`
`
`
`
`
`WHEEL WILL TURN, ORA SCALPEL WILL CUT
`
`
`
`
`INTO FLESH TILL IT HITS BONE,£} ASSOCIATED
`
`
`
`
`
`DATA SUCHAS QUICK LOOK UP OF DRUGS
`
`
`
`
`
`
`THAT CAN BE USED TO COMBAT A LEAKING
`
`
`
`
`
`COLON, F) ASSOCIATED COMPUTER PROGRAMS
`
`
`
`
`
`
`
`SUCH AS THOSE THAT COULD BE USED TO
`
`
`
`
`
`
`GUIDE A ROBOT CONTROLLED LAPROSCOPIC
`
`HEART SURGERY
`
`FIG. 8d
`
`
`
`DEFTS_00000056
`
`DEFTS_00000056
`
`

`

`Case 2:21-cv-00040-JRG Document 70-11 Filed 09/02/21 Page 22 of 66 PageID #: 1452
`Case 2:21-cv-00040-JRG Document 70-11 Filed 09/02/21 Page 22 of 66 PagelD #: 1452
`
`
`U.S. Patent
`
`
`
`
`Mar. 26, 2013
`
`
`
`
`Sheet 19 of 33
`
`
`
`US 8,405,604 B2
`
`
`
`
`
`PREREGISTER THE LOCATION AND ORIENTATION OF A
`
`
`
`
`SET OF OBJECTS WITH FIXED ATTACHMENTS
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`ATTACH OBJECT TO OBJECT HOLDER THAT
`
`
`
`APPROPRIATELY ORIENTS AND RAISES OBJECT
`
`
`
`
`
`ATTACH OBJECT HOLDER TO BASEBOARD
`
`
`AT APPROPRIATE LOCATION
`
`
`
`
`
`
`
`
`
`
`OBJECTS
`
`REMAINING?
`
`
`
`
`
`
`
`
`
`ATTACH APPRORIATE TARGET TO BASEBOARD
`
`
`
`
`
`
`SO THAT THE TARGET (S WITHIN THE TARGET
`
`
`
`
`
`VOLUME WITH AN ORIENTATION ROUGHLY
`
`
`
`
`
`
`
`
`
`FACING THE LENS PLANE OF THE CAMERA
`
`
`
`
`
`
`
`
`ENTER THE BASEBOARD ID, THE TARGETID,
`
`
`
`
`ANO THE ID OF THE TARGET ATTACHMENT
`
`
`
`LOCATION ON THE BASEBOARD
`
`
`
` START THE CAMERA SYSTEM, ACQUIRE THE
`
`
`
`
`
`
`
`
`
`TARGET, COMPUTE THE ORIENTATION AND
`
`
`
`
`
`LOCATION OF THE BASEBOARD, UPDATE THE
`
`
`
`COMPUTER DATABASE
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`REMOVE TARGET
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`ENTER THE OBJECT1D, THE OBJECT HOLDER
`
`
`
`
`
`
`1D, AND THE LOCATION AND ORIENTATION OF
`
`
`
`
`
`THE ATTACHMENT TO THE BASEBOARD
`
`
`
`
`
`
`
`
`
`THE ASSOCIATED COMPUTER PROGRAMWILL
`
`
`
`
`ADD TOA DATABASE DEFINING THIS OBJECT
`
`
`
`
`INCLUDING A) THE X,Y OF THE OBJECT
`
`
`
`
`
`CENTER, B) THE OBJECT PERIMETER
`
`
`
`
`
`
`PROJECTED ONTO THE BASEBOARD, C} THE
`
`
`
`
`
`GEOMETRIC DEFINITION OF THE OBJECT,
`
`
`
`
`D) MECHANICAL CHARACTERISTICS SUCH AS
`
`
`
`
`
`
`HOW ASTEERING WHEELWILL TURN, ORA
`
`
`
`
`
`
`SCALPEL WILL CUT INTO FLESHTILL IT HITS
`
`
`
`
`
`BONE, E) ASSOCIATED DATA SUCH AS QUICK
`
`
`
`
`
`
`
`LOOK UP OF DRUGS THAT CAN BE USED TO
`
`
`
`
`
`COMBAT ALEAKING COLON,F) ASSOCIATED
`
`
`
`
`
`
`COMPUTER PROGRAMS SUCH AS THOSE THAT
`
`
`
`
`
`COULD BE USED TO GUIDE A ROBOT
`
`
`
`
`CONTROLLED LAPROSCOPIC HEART SURGERY.
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`FIG. 8e
`
`DEFTS_00000057
`
`DEFTS_00000057
`
`

`

`Case 2:21-cv-00040-JRG Document 70-11 Filed 09/02/21 Page 23 of 66 PageID #: 1453
`Case 2:21-cv-00040-JRG Document 70-11 Filed 09/02/21 Page 23 of 66 PagelD #: 1453
`
`
`U.S. Patent
`
`
`
`
`Mar. 26, 2013
`
`
`
`
`Sheet 20 of 33
`
`
`
`US 8,405,604 B2
`
`
`
`
`
`
`UTILIZING TARGETED POINTERS WITHA SET OF PREREGISTERED OBJECTS
`
`
`
`
`
`
`
`
`
`
`CAPTURE ANY INPUT FROM MEMBER.
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`CAPTURE TARGET ATTACHED TO POINTER.
`
`
`
`
`
`
`DETERMINE POINTER TIP LOCATION(Xx,Y, Z
`
`
`
` DETERMINE NEAREST CENTER OF A PREREGISTERED
`
`
`
`
`
`
`
`
`
`OBJECT TO X,Y OF TIP. WE WILL CALL THIS OBJECT A
` IS TIP X.Y
`
`
`
`WITHIN
`
`
`
`PERIMETER OF
`
`OBJECTA?
` YES
`
`
`
`
`
`CHECK Z OF POINTERAGAINST 2 OF OBJECT'S
`
`
`UPPER SURFACEAT X,Y LOCATION
`
`
`
`
`
`
`
`
`
`(SZ OFTIP
`
`
`
`LESS THANE
`
`
`ABOVE Z OF
`
`
`OBJECT?
`
`
`
` IS TARGET
`
`
`
`ATTACHED
`
`
`
`TO AHAND?
`
` DATABASE DETERMINESIF FINGERTIPS’ LOCATION SHOULDBE
`
`
`
`
`
`
`
`
`
`
`INTERPRETED AS POINTING OR GRIPPING, BASED ON THE
`
`
`
`
`
`OBJECT TYPE AND CHARACTERISTICS
`
`
`
`
`
`
`
`
`APPLY MEMBER DATAAND OBJECT TO CURRENT COMPUTER
`
`
`
`
`
`
`PROGRAM WITH DATABASE, INFORMATION, FUNCTIONS, DATA,
`
`
`
`
`MECHANISMS DEPENDENTON (X.¥,2) OF POINTER TIP
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`FIG. 8f
`
`DEFTS_00000058
`
`DEFTS_00000058
`
`

`

`Case 2:21-cv-00040-JRG Document 70-11 Filed 09/02/21 Page 24 of 66 PageID #: 1454
`Case 2:21-cv-00040-JRG Document 70-11 Filed 09/02/21 Page 24 of 66 PagelD #: 1454
`
`
`U.S. Patent
`
`
`
`
`Mar. 26, 2013
`
`
`
`
`Sheet 21 of 33
`
`
`
`US 8,405,604 B2
`
`
`
`920
`
`910
`
`905
`
`FIG. 9a
`
`930
`
`931
`
`939
`
`935
`
`940
`
`945
`
`FIG. 9b
`
`DEFTS_00000059
`
`DEFTS_00000059
`
`

`

`Case 2:21-cv-00040-JRG Document 70-11 Filed 09/02/21 Page 25 of 66 PageID #: 1455
`Case 2:21-cv-00040-JRG Document 70-11 Filed 09/02/21 Page 25 of 66 PagelD #: 1455
`
`
`U.S. Patent
`
`
`
`
`Mar. 26, 2013
`
`
`
`
`Sheet 22 of 33
`
`
`
`US 8,405,604 B2
`
`
`
`970
`
`FIG. 9c
`
`DEFTS_00000060
`
`DEFTS_00000060
`
`

`

`Case 2:21-cv-00040-JRG Document 70-11 Filed 09/02/21 Page 26 of 66 PageID #: 1456
`Case 2:21-cv-00040-JRG Document 70-11 Filed 09/02/21 Page 26 of 66 PagelD #: 1456
`
`
`U.S. Patent
`
`
`
`
`Mar. 26, 2013
`
`
`
`
`Sheet 23 of 33
`
`
`
`US 8,405,604 B2
`
`
`
`
`1015
`
`
`1017
`
`1016
`
`
`
`1020
`
`
`
`
`
`1019
`
`4018
`
`
`
`1010
`
`1005
`
`1011
`
`1012
`
`FIG. 10
`
`DEFTS_00000061
`
`DEFTS_00000061
`
`

`

`Case 2:21-cv-00040-JRG Document 70-11 Filed 09/02/21 Page 27 of 66 PageID #: 1457
`Case 2:21-cv-00040-JRG Document 70-11 Filed 09/02/21 Page 27 of 66 PagelD #: 1457
`
`
`U.S. Patent
`
`
`
`
`Mar. 26, 2013
`
`
`
`
`Sheet 24 of 33
`
`
`
`US 8,405,604 B2
`
`1140
`
`
`
`1130
`
`
`
`
`
`1107
`
`
`
`
`1105
`
`1120
`
`
`
`
`FIG. 17
`
`DEFTS_00000062
`
`DEFTS_00000062
`
`

`

`Case 2:21-cv-00040-JRG Document 70-11 Filed 09/02/21 Page 28 of 66 PageID #: 1458
`Case 2:21-cv-00040-JRG Document 70-11 Filed 09/02/21 Page 28 of 66 PagelD #: 1458
`
`U.S. Patent
`
`
`
`
`
`
`Mar. 26, 2013
`
`
`
`
`Sheet 25 of 33
`
`
`
`US 8,405,604 B2
`
`
`
`1223
`
`1273
`
`FIG. 12
`
`DEFTS_00000063
`
`DEFTS_00000063
`
`

`

`Case 2:21-cv-00040-JRG Document 70-11 Filed 09/02/21 Page 29 of 66 PageID #: 1459
`Case 2:21-cv-00040-JRG Document 70-11 Filed 09/02/21 Page 29 of 66 PagelD #: 1459
`
`
`U.S. Patent
`
`
`
`
`Mar. 26, 2013
`
`
`
`
`Sheet 26 of 33
`
`
`
`
`
`US 8,405,604 B2
`
`
`
`FIG. 13
`
`DEFTS_00000064
`
`DEFTS_00000064
`
`

`

`Case 2:21-cv-00040-JRG Document 70-11 Filed 09/02/21 Page 30 of 66 PageID #: 1460
`Case 2:21-cv-00040-JRG Document 70-11 Filed 09/02/21 Page 30 of 66 PagelD #: 1460
`
`
`U.S. Patent
`
`
`
`
`Mar. 26, 2013
`
`
`
`
`Sheet 27 of 33
`
`
`
`US 8,405,604 B2
`
`
`
`
`
`DEFTS_00000065
`
`DEFTS_00000065
`
`

`

`Case 2:21-cv-00040-JRG Document 70-11 Filed 09/02/21 Page 31 of 66 PageID #: 1461
`Case 2:21-cv-00040-JRG Document 70-11 Filed 09/02/21 Page 31 of 66 PagelD #: 1461
`
`
`U.S. Patent
`
`
`
`
`Mar. 26, 2013
`
`
`
`
`Sheet 28 of 33
`
`
`
`US 8,405,604 B2
`
`
`
`
`
`
`
`FIG. 15a
`
`
`
`
`
`
`
`FIG. 15b 1585
`
`
`
`DEFTS_00000066
`
`DEFTS_00000066
`
`

`

`Case 2:21-cv-00040-JRG Document 70-11 Filed 09/02/21 Page 32 of 66 PageID #: 1462
`Case 2:21-cv-00040-JRG Document 70-11 Filed 09/02/21 Page 32 of 66 PagelD #: 1462
`
`
`U.S. Patent
`
`
`
`
`Mar. 26, 2013
`
`
`
`
`Sheet 29 of 33
`
`
`
`US 8,405,604 B2
`
`
`
`
`
`1615
`
`
`
`1605
`1703
`
`DEFTS_00000067
`
`DEFTS_00000067
`
`

`

`Case 2:21-cv-00040-JRG Document 70-11 Filed 09/02/21 Page 33 of 66 PageID #: 1463
`Case 2:21-cv-00040-JRG Document 70-11 Filed 09/02/21 Page 33 of 66 PagelD #: 1463
`
`
`U.S. Patent
`
`
`
`
`Mar. 26, 2013
`
`
`
`
`Sheet 30 of 33
`
`
`
`US 8,405,604 B2
`
`
`
`
`
`1860
`
`
`
`
`FIG. 18
`
`DEFTS_00000068
`
`DEFTS_00000068
`
`

`

`Case 2:21-cv-00040-JRG Document 70-11 Filed 09/02/21 Page 34 of 66 PageID #: 1464
`Case 2:21-cv-00040-JRG Document 70-11 Filed 09/02/21 Page 34 of 66 PagelD #: 1464
`
`
`U.S. Patent
`
`
`
`
`Mar. 26, 2013
`
`
`
`
`Sheet 31 of 33
`
`
`
`1912
`
`1913
`
`
`
`US 8,405,604 B2
`
`
`
`DEFTS_00000069
`
`DEFTS_00000069
`
`

`

`Case 2:21-cv-00040-JRG Document 70-11 Filed 09/02/21 Page 35 of 66 PageID #: 1465
`Case 2:21-cv-00040-JRG Document 70-11 Filed 09/02/21 Page 35 of 66 PagelD #: 1465
`
`
`U.S. Patent
`
`
`
`
`Mar. 26, 2013
`
`
`
`
`Sheet 32 of 33
`
`
`
`US 8,405,604 B2
`
`2015
`
`
`
`
`
`
`
`(
`
`
`
`2020
`
`
`
`2020
`
`
`
`2035
`
`
`FIG. 20
`
`DEFTS_00000070
`
`DEFTS_00000070
`
`

`

`Case 2:21-cv-00040-JRG Document 70-11 Filed 09/02/21 Page 36 of 66 PageID #: 1466
`Case 2:21-cv-00040-JRG Document 70-11 Filed 09/02/21 Page 36 of 66 PagelD #: 1466
`
`
`U.S. Patent
`
`
`
`
`Mar. 26, 2013
`
`
`
`
`Sheet 33 of 33
`
`
`
`US 8,405,604 B2
`
`
`
`
`
`
`
`2110
`
`
`
`
`
`
`
`
`
`2102
`
`
`FIG. 21
`
`DEFTS_00000071
`
`DEFTS_00000071
`
`

`

`Case 2:21-cv-00040-JRG Document 70-11 Filed 09/02/21 Page 37 of 66 PageID #: 1467
`Case 2:21-cv-00040-JRG Document 70-11 Filed 09/02/21 Page 37 of 66 PagelD #: 1467
`
`
`
`US 8,405,604 B2
`
`
`1
`ADVANCED VIDEO GAMING METHODS FOR
`
`
`
`
`EDUCATION AND PLAY USING CAMERA
`
`
`
`
`
`BASED INPUTS
`
`
`
`
`
`CROSS REFERENCE TO RELATED
`
`
`APPLICATIONS
`
`
`
`
`
`
`
`
`
`
`This application is a continuation of Ser. No. 11/429,305
`
`
`
`
`
`
`
`filed May 8, 2006; which is a divisional application of Ser.
`
`
`
`
`
`
`
`
`
`
`
`No. 10/622,812 filed Jul. 21, 2003 and now U.S. Pat. No.
`
`
`
`
`
`
`
`7,042,440; which is a divisional application of Ser. No.
`
`
`
`
`
`
`
`
`
`
`09/138,285, filed Aug. 21, 1998 and now U.S. Pat. No. 6,720,
`
`
`
`
`
`
`
`949; which application claims benefit of provisional applica-
`
`
`
`
`
`
`
`
`
`
`tion Ser. No. 60/056,639,filed Aug. 22, 1997, and provisional
`
`
`
`
`
`
`
`
`
`application Ser. No. 60/059,561, filed Sep. 19, 1997—(all of
`
`
`
`
`
`
`whichare hereby incorporated by reference). The disclosures
`
`
`
`
`
`
`
`of all of these applications are hereby incorporated by refer-
`ence.
`
`
`BACKGROUNDOF THE INVENTION
`
`
`
`
`
`
`
`
`
`15
`
`
`
`20
`
`
`2
`
`
`
`
`
`
`
`
`
`USS. Pat. No. 4,843,568: Real time perception of and
`
`
`
`
`
`responseto the actions of an unencumberedparticipant/
`
`
`user by Krueger, et al
`
`
`
`
`
`
`
`
`Tura and Sigel disclose meansfor using a video camera to
`
`
`
`
`
`
`
`
`look at a operators body orfinger and input control informa-
`
`
`
`
`
`
`
`tion to acomputer. Their disclosure is generally limited to two
`
`
`
`
`
`
`
`dimensionalinputs in an x-y plane, such as would be traveled
`
`
`
`
`by a mouse used conventionally.
`
`
`
`
`
`
`
`Dementhondiscloses the use objects equipped with 4
`
`
`
`
`
`
`
`LEDs detected with a single video camera to provide a 6
`
`
`
`
`
`
`
`degree offreedom solution ofobject position and orientation.
`
`
`
`
`
`
`
`
`
`He downplaysthe use ofretroreflector targets for this task.
`
`
`
`
`
`
`
`Cipolla et al discusses processing and recognition ofmove-
`
`
`
`
`
`
`
`
`ment sequence gesture inputs detected with a single video
`
`
`
`
`
`
`
`camera whereby objects or parts of humans equipped with
`
`
`
`
`
`
`
`
`
`
`four reflective targets or leds are moved thru space, and a
`
`
`
`
`
`
`
`sequence of images of the objects taken and processed. The
`
`
`
`
`
`targets can be coloredto aid discrimination.
`
`
`
`
`
`
`Pryor, one ofthe inventors, in several previous applications
`
`
`
`
`
`
`
`
`
`has described single and dual (stereo) camera systemsutiliz-
`
`
`
`
`
`
`
`
`ing natural features of objects or special targets including
`
`
`
`
`
`
`retroreflectors for determination of position and orientation
`
`
`
`
`
`
`
`
`of objects in real time suitable for computer input, in up to 6
`
`
`degrees of freedom.
`
`
`
`
`
`
`
`Pinckney has described a single camera method for using
`
`
`
`
`
`
`
`and detecting 4 reflective targets to determine position and
`
`
`
`
`
`orientation of an object in 6 degrees of freedom. A paper by
`
`
`
`
`
`
`Dr. H. F. L. Pinckney entitled Theory and Developmentof an
`
`
`
`
`
`
`
`on line 30 Hz video photogrammetry system for real-time 3
`
`
`
`
`
`
`dimensionalcontrol presented at the Symposium of Commis-
`
`
`
`
`
`
`sion V Photogrammetry for Industry, Stockholm, August
`
`
`
`
`
`
`
`1978, together with many of the references referred to therein
`
`
`
`
`
`
`gives many of the underlying cquationsof solution of photo-
`
`
`
`
`
`
`
`grammetry particularly with a single camera. Anotherrefer-
`
`
`
`
`
`
`
`ence relating to use of two or more cameras, is Development
`
`
`
`
`
`
`
`of Stereo Vision for Industrial Inspection, Dr. S. F. El-Hakim,
`
`
`
`
`
`
`
`
`Proceedings of the Instrument Society of America (ISA)
`
`
`
`
`
`
`
`
`
`
`Symposium, Calgary Alta, Apr. 3-5 1989. This paper too has
`
`
`
`
`
`
`several useful references to the photogrammetry art.
`
`
`
`
`
`
`
`
`Generally speaking, while several prior art references have
`
`
`
`
`
`
`
`providedpieces of the puzzle, none has disclosed a workable
`
`
`
`
`
`
`
`
`system capable of widespread use, the variety and scope of
`
`
`
`
`
`
`
`
`embodiments herein, nor the breath and novelty of applica-
`
`
`
`
`
`
`
`tions made possible with electro-optical determination of
`
`
`
`
`object position and/ororientation.
`
`
`
`
`
`
`
`
`In this invention, many embodiments may operate with
`
`
`
`
`
`
`
`natural features, colored targets, self-illuminated targets such
`
`
`
`
`
`
`
`
`as LEDS, or withretroreflective targets. Generally the latter
`
`
`
`
`
`
`
`
`
`
`
`twogive the best results from the point of viewof speed and
`
`
`
`
`
`reliability of detection—of major importance to widespread
`
`
`dissemination of the technology.
`
`
`
`
`
`
`
`
`However, of these two, only the retroreflector is both low
`
`
`
`
`
`
`
`
`
`cost, and totally unobtrusiveto the user. Despite certain prob-
`
`
`
`
`
`
`
`
`
`lems using same,it is the preferred type of target for general
`
`
`
`
`
`
`
`
`use, at least for detection in more than 3 degrees of freedom.
`
`
`
`
`
`
`
`
`
`Even in only two degrees, where standard “blob” type image
`
`
`
`
`
`
`
`
`processing might reasonably be used to find ones finger for
`
`
`
`
`
`
`
`
`
`example, (e.g., U.S. Pat. No. 5,168,531 by Sigel), use of
`
`
`
`
`
`
`
`
`
`simple glass bead based, or molded plastic corner cube based
`
`
`
`
`
`
`
`retroreflectors allows much higher frequency response (e.g.
`
`
`
`
`
`
`
`
`
`Hz, 60 Hz, or even higher detection rates) from the multiple
`
`
`
`
`
`
`
`incidence angles needed in normal environments, also with
`
`
`
`
`
`
`
`
`lower cost computers undera widervariety of conditions- and
`
`
`
`
`
`
`
`
`
`is more reliable as well.(at least with today’s PC processing
`
`power).
`
`SUMMARYOF THE INVENTION
`
`
`
`
`
`
`
`
`
`
`
`Numerous 3D input apparatus exist today. As direct com-
`
`
`
`
`
`
`
`
`puter input for screen manipulation, the most commonis the
`
`
`
`
`DEFTS_00000072
`
`1. Field of the Invention
`
`
`
`
`
`
`
`
`
`
`
`The invention relates to simple input devices for comput-
`
`
`
`
`
`
`
`
`
`
`ers, well suited for use with 3-D graphically intensive activi-
`
`
`
`
`
`
`
`
`ties, and operating by optically sensing object or human posi-
`
`
`
`
`
`
`
`tions and/or orientations. The invention in many preferred
`
`
`
`
`
`
`
`embodiments, uses real time stereo photogrammetry using
`
`
`
`
`
`
`
`single or multiple TV cameras whose output is analyzed and
`
`
`
`
`used as input to a personal computer.
`
`
`
`
`2. Description of Related Art
`
`
`
`
`
`
`
`The closest knownreferencesto the stereo photogrammet-
`
`
`
`
`
`
`
`
`ric imaging of datum’s employed by several preferred
`
`
`
`
`
`
`
`embodimentsofthe invention are thought to exist in the fields
`
`
`
`
`
`
`
`of flight simulation, robotics, animation and biomechanical
`
`
`
`
`
`
`
`
`
`studies. Someearly prior art references in these fields are
`
`
`
`
`
`Pugh U.S. Pat. No. 4,631,676;
`
`
`
`
`
`Birk U.S. Pat. No. 4,416,924;
`
`
`
`
`
`Pinckney U.S. Pat. No. 4,219,847;
`
`
`
`
`
`
`
`
`
`USS. Pat. No. 4,672,564 by Egli et al, filed Nov. 15, 1984;
`40
`
`
`
`
`
`
`
`
`Pryor U.S. Pat. No. 5,506,682 Robot Vision Using Targets;
`
`
`
`
`
`Pryor, Method for Automatically Handling, Assembling &
`
`
`
`
`
`
`
`Working on Objects, U.S. Pat. No. 4,654,949; and
`
`
`
`
`
`
`
`Pryor, U.S. Pat. No. 5,148,591, Vision Target Based
`
`Assembly.
`
`
`
`
`
`
`
`In whatis called “virtualreality”, a numberofother devices
`
`
`
`
`
`have appeared for humaninstruction to a computer. Examples
`
`
`
`
`
`
`
`
`are head trackers, magnetic pickups on the human and the
`
`
`
`
`
`
`
`
`like, which have their counterpart in the invention herein.
`
`
`
`
`
`
`
`References from this field having similar goals to some
`
`
`
`
`aspects of the invention hereinare:
`
`
`
`
`
`USS. Pat. No. 5,297,061 by Dementhonetal
`
`
`
`
`
`
`
`USS. Pat. No. 5,388,059 also by Dementhon, etal
`
`
`
`
`
`
`USS. Pat. No. 5,168,531: Real-time recognition of pointing
`
`
`
`
`information from video, by Sigel