Case 2:21-cv-00040-JRG Document 64-4 Filed 08/15/21 Page 1 of 18 PageID #: 1104
`Case 2:21-cv-00040-JRG Document 64-4 Filed 08/15/21 Page 1 of 18 PagelD #: 1104
`
`
`EXHIBIT D
`EXHIBIT D
`
`

`

`Case 2:21-cv-00040-JRG Document 64-4 Filed 08/15/21 Page 2 of 18 PageID #: 1105
`ceess TTTTTATA
`
`US008878949B2
`
`US 8,878,949 B2
`(10) Patent No.:
`a2) United States Patent
`Pryor
`(45) Date of Patent:
`*Nov. 4, 2014
`
`
`(54) CAMERA BASED INTERACTION AND
`INSTRUCTION
`
`(71) Applicant: Gesture Technology Partners, LLC,
`Sylvania, OH (US)
`
`(72)
`
`Inventor: Timothy R. Pryor, Sylvania, OH (US)
`
`(*) Notice:
`
`Subject to any disclaimer, the termofthis
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 0 days.
`
`This patent is subject to a terminal dis-
`claimer.
`
`:
`(21) Appl. No: 13/961,452
`vs
`.
`(22)
`Filed:
`Aug. 7, 2013
`oe
`.
`(65)
`Prior Publication Data
`US 2014/0028855 Al
`Jan. 30, 2014
`
`Related U.S. Application Data
`(63) Continuation of application No. 13/459,670, filed on
`Apr. 30, 2012, now Pat. No. 8,654,198, which is a
`continuation of application No. 12/891,480,filed on
`Sep. 27, 2010, now Pat. No. 8,189,053, which is a
`continuation of application No. 11/376,158,filed on
`Mar. 16, 2006, now Pat. No. 7,804,530, which is a
`continuation of application No. 09/568,552, filed on
`May 11, 2000, now Pat. No. 7,015,950.
`(60) Provisional application No. 60/133,671, filed on May
`11, 1999.
`,
`Int. Cl.
`HOAN 5/232
`GO6F 3/01
`GO6F 3/038
`HOAN 5/222
`(52) US.CI.
`CPC wee HO4N 5/23296 (2013.01); GO6F3/017
`
`(51)
`
`(2006.01)
`(2006.01)
`(2013.01)
`(2006.01)
`
`
`
`
`
`(2013.01); GO6F 3/0386 (2013.01); HO4N
`5/222 (2013.01); HO4N 5/232 (2013.01); HOAN
`5/23219 (2013.01)
`USPC icc ereeeetetteeenees 348/211.99; 348/211.4
`(58) Field of Classification Search
`CPC .. HO4N 5/23238; HO4N 5/247; HO4N 5/3415
`USPC w. 348/211.4, 211.5, 211.8, 211.9, 222.1,
`348/239
`See application file for complete searchhistory.
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`3,909,002 A
`4,219,847 A
`4330,708 A
`631,676
`1791589 A
`4,843,568 A
`
`9/1975 Levy
`8/1980 Pinkney et al.
`7/1982 Hedgeset al.
`2/1986
`12/1088 Blue etal
`6/1989 Kruegeret al.
`(Continued)
`Primary Examiner Tuan Ho
`(74) Attorney, Agent, or Firm --- Warner Norcross & Judd
`LLP
`ABSTRACT
`(57)
`.
`.
`.
`Disclosed are methods and apparatus for instructing persons
`using computer based programs and/or remote instructors.
`One or more video cameras obtain images of the student or
`other participant. In addition images are analyzed by a com-
`puter to determine the locations or motions of one or more
`points on the student. This location data is fed to computer
`program which compares the motions to known desired
`movements, or alternatively provides such movementdata to
`aninstructor, typically located remotely, who can aid in ana-
`lyzing student performance. The inventionpreferably is used
`with a substantially life-size display, such as a projection
`display can provide, in order to make the information dis-
`played a realistic partner or instructor for the student. In
`addition, other applications are disclosed to sports training,
`dance, and remote dating.
`
`18 Claims, 7 Drawing Sheets
`
`GTP_00000900
`
`

`

`Case 2:21-cv-00040-JRG Document 64-4 Filed 08/15/21 Page 3 of 18 PageID #: 1106
`Case 2:21-cv-00040-JRG Document 64-4 Filed 08/15/21 Page 3 of 18 PagelD #: 1106
`
`US 8,878,949 B2
`
`Page 2
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`4,908,704 A
`4,988,981 A
`5,008,946 A
`5,088,928 A
`5,227,986 A
`5,249,053 A
`5,297,061 A
`5,365,597 A
`5,376,796 A
`5,388,059 A
`5,454,043 A
`5,491,507 A
`5,534,921 A
`5,572,251 A
`5,581,276 A
`5,594,469 A
`5,616,078 A
`5,624,117 A
`5,781,647 A
`5,781,650 A
`5,828,770 A
`5,845,006 A
`5,853,327 A
`5,878,174 A
`5,904,484 A
`
`3/1990 Fujioka et al.
`1/1991 Zimmerman et al.
`4/1991 Ando
`2/1992 Chan
`9/1993 Yokota etal.
`9/1993 Jain
`3/1994 Dementhonetal.
`11/1994 Holeva
`12/1994 Chan etal.
`3/1995 DeMenthon
`9/1995 Freeman
`2/1996 Umezawaetal.
`4/1996 Sawanobori
`11/1996 Ogawa
`12/1996 Cipolla et al.
`1/1997 Freeman et al.
`4/1997 Oh
`4/1997 Ohkuboetal.
`7/1998 Fishbineetal.
`7/1998 Loboetal.
`10/1998 Leis et al.
`12/1998 Sumiet al.
`12/1998 Gilboa
`3/1999 Stewart etal.
`5/1999 Burns
`
`5,926,168 A
`5,940,126 A
`5,982,352 A
`5,999,840 A
`6,052,132 A
`6,098,458 A
`6,108,033 A
`6,148,100 A
`6,160,899 A
`6,204,852 Bl
`6,252,598 Bl
`6,342,917 Bl
`6,346,929 Bl
`6,359,647 Bl
`6,363,160 Bl
`6,373,472 Bl
`6,442,465 B2
`6,508,709 Bl
`6,529,617 Bl
`6,597,817 Bl
`6,663,491 B2
`6,750,848 Bl
`6,775,361 Bl
`6,788,336 Bl
`6,911,972 B2
`7A89,863 B2
`7,564,476 B1*
`.
`.
`* cited by examiner
`
`WA1999 Fan
`8/1999 Kimura
`11/1999 Pryor
`12/1999 Grimsonetal.
`4/2000 Christian etal.
`2000 Frenchet al.
`/2000 Ito et al.
`11/2000 Anderson etal.
`12/2000 Leeet al.
`3/2001 Kumar et al.
`6/2001 Segen
`1/2002, Amenta
`2/2002 Fukushimaet al.
`3/2002 Senguptaetal.
`3/2002 Bradski etal.
`#2002 Palalauetal.
`(2002 Breedet al.
`1/2003 Karmarkar
`(2003 Prokoski
`/2003 Silverbrook
`12/2003 Watabe etal.
`6/2004 Pryor
`/2004 Araietal.
`/2004 Silverbrook
`6/2005 Brinjes
`2/2009 Lee
`/2009 Coughlan et al.
`
`......... 348/14.08
`
`GTP_00000901
`
`

`

`Case 2:21-cv-00040-JRG Document 64-4 Filed 08/15/21 Page 4 of 18 PageID #: 1107
`Case 2:21-cv-00040-JRG Document 64-4 Filed 08/15/21 Page 4 of 18 PagelD #: 1107
`
`‘SN
`
`
`COMPUTER
`
`
`
`yuajed
`
`PLOT‘bh“AON
`LJOJ}99Y$
`
`7d6F6°8L8'8SA
`
`GTP_00000902
`
`

`

`Case 2:21-cv-00040-JRG Document 64-4 Filed 08/15/21 Page 5 of 18 PageID #: 1108
`Case 2:21-cv-00040-JRG Document 64-4 Filed 08/15/21 Page 5 of 18 PagelD #: 1108
`
`U.S. Patent
`
`Nov. 4, 2014
`
`US 8,878,949 B2
`
`Sheet 2 of 7
`
`GTP_00000903
`
`

`

`Case 2:21-cv-00040-JRG Document 64-4 Filed 08/15/21 Page 6 of 18 PageID #: 1109
`Case 2:21-cv-00040-JRG Document 64-4 Filed 08/15/21 Page 6 of 18 PagelD #: 1109
`
`U.S. Patent
`
`Nov. 4, 2014
`
`Sheet 3 of 7
`
`US 8,878,949 B2
`
`COMPUTER |_| POSEANALYSIS|_|
`
`CAMERA CONT.
`
`
`
`Li
`
`
`
`GTP_00000904
`
`

`

`Case 2:21-cv-00040-JRG Document 64-4 Filed 08/15/21 Page 7 of 18 PageID #: 1110
`Case 2:21-cv-00040-JRG Document 64-4 Filed 08/15/21 Page 7 of 18 PagelD #: 1110
`
`U.S. Patent
`
`Nov. 4, 2014
`
`Sheet 4 of 7
`
`US 8,878,949 B2
`
`430°
`
`450
`
`
`
`GTP_00000905
`
`

`

`Case 2:21-cv-00040-JRG Document 64-4 Filed 08/15/21 Page 8 of 18 PageID #: 1111
`Case 2:21-cv-00040-JRG Document 64-4 Filed 08/15/21 Page 8 of 18 PagelD #: 1111
`
`U.S. Patent
`
`Nov.4, 2014
`
`Sheet 5 of 7
`
`US 8,878,949 B2
`
`500
`
`
`[COMPUTER
`
`600
`
`660
`
`
`
`GTP_00000906
`
`

`

`Case 2:21-cv-00040-JRG Document 64-4 Filed 08/15/21 Page 9 of 18 PageID #: 1112
`Case 2:21-cv-00040-JRG Document 64-4 Filed 08/15/21 Page 9 of 18 PagelD #: 1112
`
`U.S. Patent
`
`Nov. 4, 2014
`
`Sheet 6 of 7
`
`US 8,878,949 B2
`
` /
`
`a
`NS
`
`\
`\
`
`At~
`bb
`
`!
`:
`705
`
`Fe FrHMR, | COMPUTER
`
`
`
`GTP_00000907
`
`

`

`Case 2:21-cv-00040-JRG Document 64-4 Filed 08/15/21 Page 10 of 18 PageID #: 1113
`Case 2:21-cv-00040-JRG Document 64-4 Filed 08/15/21 Page 10 of 18 PagelD #: 1113
`
`ene
`CAMERA
`.
`
`825
`
`
`
`DISPLAY
`
`
`INTERNET
`OR OTHER
`
`
`
`‘COMPUTER |
`
`
`
` OPTIONAL
`
`SCREEN
`
`810
`SCREEN OPTIONAL
`
`
`yuajed‘SN
`
`
`PLOT‘bh“AON
`LJ0L94S
`
`7d6F6°8L8'8SA
`
`GTP_00000908
`
`

`

`Case 2:21-cv-00040-JRG Document 64-4 Filed 08/15/21 Page 11 of 18 PageID #: 1114
`Case 2:21-cv-00040-JRG Document 64-4 Filed 08/15/21 Page 11 of 18 PagelD #: 1114
`
`US 8,878,949 B2
`
`1
`CAMERA BASED INTERACTION AND
`INSTRUCTION
`
`Method and apparatusare disclosed to enhancethe quality
`and usefulness of picture taking for pleasure, commercial, or
`other business purposes. In a preferred embodiment, stereo
`photogrammetry is combined with digital image acquisition
`to acquire or store scenes and poses ofinterest, and/or to
`interact with the subject in order to provide data to or from a
`computer. Other preferred embodiments illustrate applica-
`tions to control of display systems.
`
`BACKGROUND
`
`Representative ofUSA patents on Digital cameras are U.S.
`Pat. Nos. 5,534,921, 5,249,053 and many others which
`describe use of matrix array (CCD or otherwise) based cam-
`eras to take pictures of humansorother objects. The images
`taken are generally comprised of 400,000 or more pixels
`whichare often compressed to smaller record sizes for data
`storage,for later retrieval and display. Video cameras or Cam-
`corders are also increasingly able to take still photographs as
`well, and record or transmit them to computers.
`Aside from exposure control(to keep the light reaching the
`detector array within the dynamic range of same), and range
`finding (to effect the best lens focus given the object distance
`in question) there are few cases knownto the inventor where
`the camera taking the picture actually determines somevari-
`able in the picture and usesit for the process of obtaining the
`picture.
`One such example that does not take a picture of humans
`but rather of data, is exemplified by U.S. Pat. No. 4,791,589,
`where a certain wave form signature on an oscilloscope is
`searched for by processing the digital camera image, and
`whenit is seen, the image stored.
`More apropos the function of “Picture Taking” as the gen-
`eral public knowsit and ofinterest as the primary focus ofthe
`instant invention, is U.S. Pat. No. 5,781,650 by Lobo,et al
`whichdescribes analysis after the fact of recorded images to
`determine facial content and thusthe age of the subject. This
`disclosure also alludes to a potential point and shoot capabil-
`ity also based on the age classification of the individuals
`whose picture is desired.
`There is no knownpicture taking reference based on object
`position and orientation with respect to the camera, or other
`objects that I am awareof.
`
`SUMMARYOF THE INVENTION
`
`10
`
`15
`
`25
`
`30
`
`35
`
`40
`
`45
`
`High Resolution Digital still cameras employing matrix 5
`photodetector array chips to scan the image produced by the
`camera lens are now commonplace, and will be even more so
`in a few years as chips and memories becomevery inexpen-
`sive, and pixel density approaches 2000x2000pixels, rivaling
`photographic film. Even today Camcorders having 700x500
`pixel image chips are commonfor video based data andstills.
`This invention is aimed at improvements inutilization of
`these camerasand others which make use ofa computer based
`camera’s ability to analyze, in real timeif desired, the images
`obtained. Indeed a picture taking system may be composed of
`a combination of cameras, some used for purposes other than
`the recording ofthe picture proper.
`It is a goalof the invention to provide a methodfor taking
`pictures when certain poses of objects, sequences of poses,
`motions of objects, or any other states or relationships of
`objects are represented. It is also a goalto allow this to be done
`in a self timer like mode, when desired scene situations or
`
`2
`specific dates or other circumstances exist. In some cases,
`information as to what is desired may be entered remotely,
`even over the internet, or radio telephone.
`It is also a goal of the invention to provide a method for
`selecting from a digital or other picture memory, pictures
`obtained when certain pre programmed poses of objects,
`sequences of poses, or relationships of ebjects are repre-
`sented.
`It is a further goal of the invention to provide means by
`which users engaged in digital camera based activities, or
`other activities, using a computer can have their pictures
`taken.
`Itis a still further goal to provide all such functions ina 2D
`or 3D context, and using simple equipment capable of wide-
`spread use.
`It is another goal of the invention to feed back data to a
`subject or subjects havinghisor her, or their picture taken, in
`order that they assume another pose or engage in another
`activity, or juxtaposition of subject positions.
`While this inventionis primarily aimedat the general pic-
`ture taking public at large, it
`is realized that commercial
`photographers and cine-photographers, for example in the
`coming trend to digital “Hollywood” movie making, may
`benefit greatly from the invention herein, as it potentially
`allows more cost effective film production by giving the
`director the ability to expose the camera to the presence of
`masses ofdata, but only saving or taking that data which is
`useful, and if desired, to signal the creation offurther data
`based ondata obtained. All this with little or no humaninter-
`vention as desired, thus saving on the cost of direction, film
`crews, and other labor or venuerelated costs.
`
`DRAWINGS DEPICTING PREFERRED
`EMBODIMENTS OF THE INVENTION
`
`FIG.1 illustrates means by which users engagedin digital
`camera based activities, or other activities, using a computer
`can have their pictures taken.
`FIGS. 2A4-2Dillustrate a method for taking pictures when
`certain pre programmed poses of objects, sequences ofposes,
`or relationships of objects are represented.
`FIG.3 illustrates a self timer like mode, or when specific
`dates or other circumstances exist,
`including a system
`embodiment for taking pictures in shopping malls or other
`locales and providing instant print or other hardcopy capabil-
`ity (e.g. on a tee shirt).
`FIG.4 illustrates means to provide all such functions in a
`2D or 3D context, using simple equipment capable of wide-
`spread use. Various retroreflective artificialtarget configura-
`tions are also disclosed.
`FIG.§ illustrates a method to feed back data to a subject
`having his or her picture taken, in order that the subject
`assumes another pose or engage in anotheractivity.
`FIG. 6 illustrates a commercial version of the invention
`useful for police departments and real estate agents, among
`others.
`FIG. 7 illustrates an embodimentofthe invention used for
`photography of stage performances.
`FIG.8 illustrates an embodimentofthe invention used for
`ballet instruction and other teaching and interaction activities
`also with remotely located instructors or players.
`
`EMBODIMENTS OF THE INVENTION
`
`FIG. 1
`
`Illustrated in FIG. 1 of the invention is means by which
`users engaged in digital camera based activities, or other
`
`GTP_00000909
`
`

`

`Case 2:21-cv-00040-JRG Document 64-4 Filed 08/15/21 Page 12 of 18 PageID #: 1115
`Case 2:21-cv-00040-JRG Document 64-4 Filed 08/15/21 Page 12 of 18 PagelD #: 1115
`
`US 8,878,949 B2
`
`3
`activities, using a computer can havetheir pictures taken, and
`in this context, FIG. 1 resembles that of co-pending refer-
`enced application 9 above. A single camera, or a set, such as
`astereo pair are employedto see portions of an object, such as
`a person, a part of a person such as a hand,leg, foot, fingers,
`or head, and/or to view datums on an object, portion of an
`object, or an object held by the person or with which the
`personinteracts. In addition, multiple persons and objects can
`be seen.
`
`Where a single camera is employed, 2D measurements of
`object location relative to the camera (x and y perpendicular
`to the camera axis) areall that is possible, unless datums of
`known shapeor spacing are used onthe object viewed. Where
`a stereo pair or more of cameras are employed, 3D (xyz) data
`of a single point can be provided, for example retro-reflector
`50 on the head 52 ofperson 51. In both cases where 3 or more
`datumsare used onanobject, 6 Degree offreedom data can be
`obtained, allowing object orientation in 3 angular axes as well
`as range in 3 axes to be obtained. With two or more cameras,
`such 3D data may also be obtained using other features of
`objects such as edges of arms and the likely using known
`photogrammetric techniques.
`The cameras used may also be usedto take pictures of an
`object, or another specialized camera used for that purpose in
`conjunction with those used to determine the location of
`object features. Both examplesare illustrated in this applica-
`tion.
`
`10
`
`15
`
`25
`
`30
`
`35
`
`As showninthis figure, two cameras 101 and 102 are used
`as a stereo pair, with each camera located at opposite sides of
`a TV monitor 105, used for either computer or Television
`display or both. This is a desirable configuration commer-
`cially and discussed the co-pending application references
`above. In this particular case, an additional camera 110 is
`shown in the middle ofthe other two, said added camera used
`for picture taking, internet telephony and/or other purposes.
`Anoptional auxiliary LED light source 115 (or 116 or 117)
`for illuminating a user 60 or other object is also shown.
`All three cameras are connected to the computer 130 by
`means of a USB (Universal Serial Bus) daisy chain, or IEEE
`1394 firewire connections (faster). Each is accessed, as
`needed for position and orientation determination,or picture
`taking.
`Evenusing a single camera in two dimensions (as is normal
`today), some position and orientation data or sequences of
`same can be achieved using modern image processing tech-
`niques. (See for example the invention disclosed in US. Pat.
`No. 4,843,568 of Myron Krueger). However, accurate sens-
`ing and control of systems, such as camerashereinis difficult
`today with processors cost effective enough to be used by the 5
`public at large, and artificial target augmentation of image
`points is often desirable.
`It is thus possible using the invention to be taking pictures
`of users of interactive computer systems for whatever pur-
`pose. This allows one to automatically capture images of 5
`children at play, for example with a computer system such as
`a computer game.It also enables many other functions which
`are described below. Andit can be used in the field, where the
`computer, stereo position sensing and picture taking camera,
`may be co-located together in the same housing.
`It is noted that whereretro-reflectors are used, (as opposed
`to choosing for example less contrasting datums, for example
`natural object features such as edgesoffingers, or clothing
`features, or targets such as colored dots) then each of the two
`cameras for stereo location determination needs lights to
`illuminate retro-reflectors substantially co-located with the
`camera axes. These lights can alternatively provide general
`
`40
`
`45
`
`60
`
`65
`
`4
`lighting for any other camera or cameras to use in taking
`photographs or other purposes.
`It is noted that cameras 101 and 102 need not have the
`
`imageofthe retro-reflector or other discernable target be in
`precise focus, indeed it is often helpful to have a some blur
`due to defocusing so as to aid sub pixel position solution of
`datumlocation. If the LEDs or other light sources are in the
`near infrared, and the cameralenses are focusedin thevisible,
`this occurs naturally, unless the lens is also near infrared
`chromatic corrected.
`
`An optional laser pointer (or other suitable illumination
`source), comprised of diode laser and collimating optics 150
`is also usable with the invention to illuminate object portions
`from which 3D data is desired (such as the neck region of
`person 51 as shown), or in the simpler case to designate which
`areas of a picture are to be focused, or zoomed in on or
`transmitted or recorded—with or without consideration of
`
`3-D position data ofthe object. This can befixed as shown,or
`optionally hand held by the user, for example in left hand
`(dotted lines) and used by him or her to designate the pointto
`be measured in 3D location. (see also references above). In
`addition a person taking pictures, such as a photography can
`without looking through the viewfinder of the camera, point
`to appoint on the subject, which is then dealt with by camera
`typically by focusing the lens systemsuchthat the pointis in
`the desired state of focus (usually but not necessarily when
`the laser spot on the subject appears smallest in diameter
`and/or of highest contrast). Such as system is particularly
`useful for cameras with wide fields of view, or those mounted
`on pan tilt mechanisms, where the mechanism can also be
`activated to position the camera axis to take the picture with
`the laser spot for example centered in the camerafield.
`In the laser designated case, it is generally the laser spot or
`other indication on the surface that is imaged, (although one
`can also instruct, for example using voice recognition soft-
`ware in computer 130 inputted via voice activated micro-
`phone135, the camera processorto obtain and store if desired
`the imageofthe area aroundthe spot projected ontothe object
`as well oralternatively), and ifthe spot is desired, it is often
`useful that cameras 101 and 102 have band-passfilters which
`pass the laser wavelength, and any led illumination wave-
`lengths used for retro-reflector illumination for example, but
`block other wavelengths to the extent possible at low cost.It
`is noted that the discrimination in an image can also be made
`on color grounds---i.e. with red diode lasers and red LEDs,
`the system can analyze the image areas containing redsin the
`image, for example—with the knowledge that the answer
`can’t lie at any shorter wavelengths (e.g. green, yellow, blue).
`By using two cameras 101 and 102, a superior ranging
`system for the laser spot location on the subject results, since
`the baseline distance “BL” separating the camerasfor trian-
`gulation based ranging purposes can be sufficient to provide
`accurate measurementofdistance to the object.
`
`FIGS. 2A-2D
`
`Aswebegin to consider the apparatus of FIG.1, it is clear
`one could do much more to enhancepicture taking ability
`than hereto fore described and contained in the priorart. And
`it can be done with apparatus capableoffield use.
`FIGS. 2.4-2D for example, illustrates a method for taking
`pictures when certain pre programmed or otherwise desired
`poses of objects, sequences of poses, or relationships of
`objects are represented. No such ability is available to pho-
`tographers today.
`Consider still camera system 201, patterned after that of
`FIG.1 and comprising 3 cameras and associated image scan-
`
`GTP_00000910
`
`

`

`Case 2:21-cv-00040-JRG Document 64-4 Filed 08/15/21 Page 13 of 18 PageID #: 1116
`Case 2:21-cv-00040-JRG Document 64-4 Filed 08/15/21 Page 13 of 18 PagelD #: 1116
`
`US 8,878,949 B2
`
`6
`frames ofhigh resolution preferred data. (This doesn’t solve
`flicker problems, but it does mean that peor quality or extra-
`neous material isn’t sent!). Criteria such as degree of image
`motion blur or image focus can also be used in making trans-
`mission decisions.
`
`FIG.2B illustrates a block diagram showing a pose analy-
`sis software or hardware module 250 analyzing processed
`image data (for example utilizing camera image data pro-
`cessed by visionbloks software from Integral Vision Corp.)
`from the computer 220 (which may be the same physical
`microprocessor, such as a Intel Pentium 2 in a Dell inspiron
`3500 laptop computer, or different) and determining from
`same when a certain pose for example has been seen. When
`this occurs, a signal is sent to the camera control module 255
`to hold thelast frame taken by camera 202, and to display it to
`the photographer, digitally store it, or transmit it—to some-
`one else, or another data store or display. Such transmission
`can be by data link, internet, cell phone, or any other suitable
`means.
`
`15
`
`th>
`
`25
`
`35
`
`40
`
`45
`
`5
`ning chips. The central camera, 202,is for picture taking and
`has high resolution and color accuracy. The two cameras on
`either side, 210 and 211, may be lowerresolution (allowing
`lower cost, and higher framerate, as they haveless pixels to
`scan in a given frame time), with little or no accurate color
`capability, as they are used to simplysee object positions or
`special datum positions on objects (which may be distin-
`guished however by taught colors for example as taught in
`some of my co-pending inventions).
`Cost wise the distinction between cameras is important.
`Today low cost CMOSchips and lenses capable of the pro-
`viding stereo measurements as described above are $15 or
`less. High quality CCD color detector arrays and lenses for
`high quality photo images are over $100, and in many cases
`$1000 or more.
`Anoptical viewfinder 215 is one of many waysto indicate
`to the user what scene information is being gathered by the
`camera system. The user can in this invention specify with a
`viewfinder based readout, the area ofthefield that is desired.
`Another criteria could be that two or more preselected
`Use ofthe viewfinderin this manner, whether looked through
`poses wereseen oneafter the other, with a time delay between
`or displayed on a screen, is for example an alternative to
`them, also pre-selected if desired.
`designating an area on the actual object using a laser pointer
`FIG. 2C illustrates a specific case whereby a point on one
`for the purpose.
`person, say hand 260 of man 265 having head 271, is deter-
`The camera system 201 further contains a computer 220
`mined, and a picture is taken by camera system 201 ofthe
`which processes the data from cameras 210 and 211 to get
`invention whenthis point comes within a distance ofapproxi-
`various position and/or orientation data concerning a person
`mately 6 inches (or any other desired amount including con-
`(or other object, or personsplural, etc). Integral light sources
`tact—.e. zero distance) from another person or object, say
`as described in FIG. 1 above mayalso be provided such as
`
`the head 270 of woman 275. To obtain the data, one can look
`LED arrays 240 and 245 and xenonflash 246.
`for hand or head indications in the image using known
`In general, one can use the system to automatically “shoot”
`pictures for example, when anyorall of the following occur,
`machine vision techniques, and/or in a more simple case put
`a target marker such as colored triangle 285 or other type on
`as determined by the position and orientation determining
`the hand or heador both and look forit.
`system ofthe camera ofthe invention:
`The use ofthe natural features of the subjects heads, which
`1. Subject in a certain pose.
`are distinguishable by shape and size in a knownfield con-
`2. Subject in a sequence of poses.
`taining two persons, is now illustrated. For example, image
`3. Portion of Subject in a sequence ofposes(e.g. gestures).
`morphology or template matching in the image field of the
`4. Subject or portion(s) in a specific locationor orientation.
`solid stateTV camera 202 can be used todistinguish the head
`5. Subject in positionrelative to another object or person.
`shapes from background data and data concerningtherest of
`For example, this could be bride and groom kissing in a
`the features such as hands, etc. of subjects 265 and 275 (or
`wedding, boy with respect to cake on birthday, and sports
`conversely hand shapes if desired can be found and heads
`events sequences of every description (where the camera can
`excluded, or the hand ofthe right person, versus the head of
`eventrack the object datumsin the field and if desired adjust
`the left, and so forth).
`shutter speed based onrelative velocity of camera to subject).
`Asshown in FIG. 2D, when the imagefield 287 of camera
`6. Ditto all of above with respect to both personsin certain
`202 after processing contains the two head images, 290 and
`posesor gesturesituations.
`7. When a subject undertakes a particular signal compris- 291, spaced a distance “W”. WhenWis not within a tolerance
`
`ing a position or gesture—.e. a silent commandto take the
`D, the picture is not taken; whereas if the heads are close
`picture (this could be programmed, for example, to corre-
`enough, within D asillustrated in dotted lines, the picture is
`taken.
`spondto raising one’s right hand).
`the invention acts as a 5
`In addition it
`is noted that
`Criteria as mentioned caninclude proximity of other parts
`rangefinder, finding range to the subject, and even to other
`of the body, or objects associated with the subjects (which
`subjects around the subject, or to all parts of interest on an
`themselves can be objects). In addition, the motion orrelative
`extensive subject. This allows a desired lens focusto be set
`motion of objects can be the criteria. For example, one could
`based on any orall of this data, as desired. It also allows a
`take program the device to take the picture when on two
`successive frames the condition shown in FIG. 2D exists
`sequence of pictures to be taken of different objects or object
`portions, at different focal depths, or focus positions. The
`where the heads are apart in frame 1, but closer in frame 2
`same holds true for exposure ofthese locations as well.
`(probably corresponding to a movementsay of the boytokiss
`It is also possible to use the above criteria for other pur-
`the girl). Clearly other sequencesare possible as well, such as
`poses, such as determining whatto record (beyondthe record-
`movement taking place in several frames followed by a
`ing that is implicit in taking pictures), or in determining what
`sequence of frames in which no movement occurs. Other
`means to determine motion in front of the camera can also be
`to transmit. The latter is importantvis a vis internetactivity,
`where available internet communication bandwidth limits
`used in this context, such as ultrasonic sensors.
`Itis also noted that the actual position or movement desired
`what can be transmitted (at least today). In this case video
`telephony with the invention comprehends obtaining only
`can be “Taught” to the computer 220 of the picture taking
`those images you really care aboutin real time. So instead of
`system. For example, a boy and girl in a wedding could
`transmitting low resolution imagedata at 20 frames a second,
`approach each other and kiss beforehand. The sequence of
`you can transmit say 5 (albeit asynchronously gathered)
`frames of this activity (a “gesture” of sorts by both parties) is
`
`5
`
`GTP_0000091 1
`
`

`

`Case 2:21-cv-00040-JRG Document 64-4 Filed 08/15/21 Page 14 of 18 PageID #: 1117
`Case 2:21-cv-00040-JRG Document 64-4 Filed 08/15/21 Page 14 of 18 PagelD #: 1117
`
`US 8,878,949 B2
`
`7
`recorded, and the speed of approach, the head positions and
`any other pertinent data determined. Whenthe photographer
`thinksthe picture is right, the computer of the camera system
`is instructedto take the picture—for example it could beat the
`instant when after a suitable approach, two head images
`become joined into one——easily recognizable with machine
`vision processing software under uniform background con-
`ditions. Then in the future, when such a condition is reached
`in the camerafield of view, pictures are taken andstored, or
`transmitted. This allows a camera to free run whose image
`field for example takes in the head table at a wedding party,
`taking only the shots thoughtto be ofmost interest. Numerous
`conditions might be programmedin, or taught in---anotherat
`the same party, would be anyoneat the head table proposing
`a toast to the bride and groom, with arm andglass raised. If
`video is taken, it might be taken fromthe point at which the
`arm rises, until after it comes down. Or with suitable voice
`recognition, when certain toast type words are heard, for
`example.
`
`Application to “3-D”Pictures
`
`Whereit is desired to take “3-D”pictures, it can be appre-
`ciated that each camera, 210 and 211 can take images of the
`scene in place of camera 202, and that both cameras 210 and
`211 outputs can be stored for later presentation in a 3D view-
`ing context, using knowndisplay techniques with appropriate
`polarized glasses or switchable LCD goggles for example. In
`this case the camera outputs can serve double duty if desired,
`each both recording picture data, as well as determining posi-
`tion of one or more points on the object or objects desired.
`In addition,or alternatively, one can use in this 3Dpicture
`case, the camera 202 (or even a stereo camerapair in place of
`202) as a means for determining position and orientation
`independently from the stereo picture taking cameras.
`Ifnot used for immediate position information, camera 202
`does not have to be digital and could employ film or other
`media to record information.
`
`FIG. 3
`
`In a manner resembling that of FIGS. 2A-2D above, the
`invention can also serve to aid a person to take his or her own
`picture--a modern “Self timer” if you will. For example any
`or all ofthe criteria such as the items 1-7 above, can be used
`as criteria for the picture to be taken of oneself. This is in
`addition to other more normalthingslike taking picturesafter
`a certain time, or on a certain date or time interval, etc. This
`has particular appeal for taking pictures of one’s self, or in any
`other situation where the photographer is not present (e.g.
`unattended recording ofanimals, children, etc.). Similarly, a
`handsignalor other signal to the camera can be usedto trigger
`the picture to be taken, using the computer camera combina-
`tion to determine the hand position or movement. This can
`also be done by voice using microphone input and suitable
`voice recognition software in the computer.
`Today, in a conventional context, one can as a photogra-
`pher, choose to shoot a fashion model or other subject, and
`when yousee a pose you like record the picture. But as one’s
`own photographer, this is much more difficult, unless you
`stream in video and search through the poses after the fact.
`But even then, you don’t know that the poses were what was
`desired, as no feedback exists during the