I 1111111111111111 11111 111111111111111 11111 11111 11111 111111111111111 IIII IIII
`
`US008878949B2
`
`No.: US 8,878,949 B2
`
`(IO) Patent
`c12) United States Patent
`(45)Date of Patent:
`Pryor
`*Nov. 4, 2014
`
`(54)CAMERA BASE D INT ERACTION AND
`
`
`INSTRUCTION
`
`(71) Applicant: Gesture Technology Partners, LLC,
`
`
`
`
`
`(2013.01); G06F 3/0386 (2013.01); H04N
`
`
`
`
`51222 (2013.01); H04N 51232 (2013.01); H04N
`
`5123219 (2013.01)
`
`
`USPC ................................... 348/211.99; 348/211.4
`
`
`
`
`
`Sylvania, OH (US)
`
`
`(58)Field of Classification Search
`
`
`CPC .. H04N 5/23238; H04N 5/247; H04N 5/3415
`
`
`
`USPC .......... 348/211.4, 211.5, 211.8, 211.9, 222.1,
`348/239
`
`
`See application file for complete search history.
`
`
`
`
`
`(72) Inventor: Timothy R. Pryor, Sylvania, OH (US)
`
`
`
`( *) Notice: Subject to any disclaimer, the term ofthis
`
`
`
`
`
`patent is extended or adjusted under 35
`(56)
`
`
`U.S.C. 154(b) by O days.
`
`
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`This patent is subject to a terminal dis­
`
`
`
`
`claimer.
`
`
`
`(21) Appl. No.: 13/961,452
`
`
`
`(22)Filed:Aug. 7, 2013
`
`(65)
`
`
`
`Prior Publication Data
`
`
`
`US 2014/0028855 Al Jan.30,2014
`
`
`
`
`
`Related U.S. Application Data
`
`3,909,002 A 9/1975 Levy
`
`
`
`4,219,847 A 8/1980 Pinkney et al.
`
`
`4,339,798 A 7/1982 Hedges et al.
`
`
`4,631,676 A 12/1986 Pugh
`
`
`4,791,589 A 12/1988 Blazo et al.
`
`
`4,843,568 A 6/1989 Krueger et al.
`
`
`(Continued)
`
`Primary Examiner - Tuan Ho
`
`
`
`
`LLP
`
`(74)Attorney, Agent, or Firm - Warner Norcross & Judd
`
`(63)Continuation of application No. 13/459,670, filed on
`
`
`
`(57)
`
`
`Apr. 30, 2012, now Pat. No. 8,654,198, which is a
`Disclosed are methods and apparatus for instructing persons
`
`
`
`
`
`
`
`continuation of application No. 12/891,480, filed on
`
`
`
`using computer based programs and/or remote instructors.
`
`Sep. 27, 2010, now Pat. No. 8,189,053, which is a
`
`
`
`One or more video cameras obtain images of the student or
`
`
`
`continuation of application No. 11/376,158, filed on
`
`
`
`other participant. In addition images are analyzed by a com­
`
`Mar. 16, 2006, now Pat. No. 7,804,530, which is a
`
`
`
`puter to determine the locations or motions of one or more
`
`
`
`continuation of application No. 09/568,552, filed on
`
`
`
`points on the student. This location data is fed to computer
`May 11, 2000, now Pat. No. 7,015,950.
`
`
`
`program which compares the motions to known desired
`
`
`
`movements, or alternatively provides such movement data to
`
`
`
`
`an instructor, typically located remotely, who can aid in ana­
`
`
`
`
`lyzing student performance. The invention preferably is used
`
`
`
`with a substantially life-size display, such as a projection
`
`
`
`
`display can provide, in order to make the information dis­
`
`
`
`
`played a realistic partner or instructor for the student. In
`
`
`
`
`addition, other applications are disclosed to sports training,
`
`
`dance, and remote dating.
`
`(51)Int. Cl.
`H04N5/232
`
`G06F 3101
`
`G06F 3/038
`
`(2006.01)
`(2006.01)
`(2013.01)
`(2006.01)
`
`(60) Provisional application No. 60/133,671, filed on May
`
`
`
`
`
`11, 1999.
`
`H04N5/222
`(52)U.S. Cl.
`
`
`
`
`
`
`CPC ............ H04N 5123296 (2013.01); G06F 3/017
`
`ABSTRACT
`
`
`
`18 Claims, 7 Drawing Sheets
`
`565
`560 555
`
`PROC.
`
`500
`
`

`

`US 8,878,949 B2
`
`Page 2
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`5,926,168 A 7/1999 Fan
`
`
`5,940,126 A 8/1999
`
`Kimura
`
`5,982,352 A 11/1999 Pryor
`
`
`
`5,999,840 A 12/1999 Grimson et al.
`
`
`6,052,132 A 4/2000 Christian et al.
`4,908,704 A 3/1990 Fujioka et al.
`
`
`
`6,098,458 A 8/2000 French
`
`et al.
`
`
`4,988,981 A 1/1991 Zimmerman et al.
`
`6,108,033 A 8/2000
`
`Ito et al.
`
`5,008,946 A 4/1991 Ando
`
`
`6,148,100 A 11/2000 Anderson et al.
`
`5,088,928 A 2/1992
`
`Chan
`
`5,227,986 A 7/1993 Yokota et al. 6,160,899 A 12/2000 Lee et al.
`
`
`6,204,852 Bl 3/2001 Kumar et al.
`
`5,249,053 A 9/1993 Jain
`
`
`6,252,598 Bl 6/2001 Segen
`
`5,297,061 A 3/1994
`
`
`Dementhon et al.
`
`
`6,342,917 Bl 1/2002 Amenta
`
`
`5,365,597 A 11/1994 Holeva
`
`
`
`6,346,929 Bl 2/2002 Fukushima et al.
`
`5,376,796 A 12/1994 Chan et al.
`
`
`
`6,359,647 Bl 3/2002 Sengupta et al.
`
`5,388,059 A DeMenthon 2/1995
`
`
`
`
`6,363,160 Bl 3/2002 Bradski et al.
`
`5,454,043 A 9/1995 Freeman
`
`
`
`6,373,472 Bl 4/2002 Palalau et al.
`
`5,491,507 A 2/1996
`
`Umezawa et al.
`
`7/1996 Sawanobori 6,442,465 B2 8/2002 Breed et al.
`5,534,921 A
`
`
`6,508,709 Bl 1/2003 Karmarkar
`
`5,572,251 A 11/1996 Ogawa
`
`
`6,529,617 Bl 3/2003 Prokoski
`
`
`5,581,276 A 12/1996 Cipolla et al.
`
`
`6,597,817 Bl 7/2003 Silverbrook
`
`5,594,469 A 1/1997
`
`
`Freeman et al.
`
`
`6,663,491 B2 12/2003 Watabe et al.
`
`5,616,078 A 4/1997 Oh
`
`
`6,750,848 Bl 6/2004 Pryor
`
`5,624,117 A Ohkubo et al. 4/1997
`
`
`6,775,361 Bl 8/2004 Arai et al.
`
`5,781,647 A 7/1998 F
`
`
`ishbine et al.
`
`6,788,336 Bl 9/2004 Silverbrook
`
`5,781,650 A 7/1998
`
`Lobo et al.
`
`6,911,972 B2 6/2005 Brinjes
`
`
`5,828,770 A 10/1998 Leis et al.
`
`5,845,006 A 12/1998 Sumi et al.
`
`7,489,863 B2 2/2009 Lee
`
`5,853,327 A 12/1998 Gilboa
`
`
`
`
`7,564,476 Bl* 7/2009 Coughlan et al. .......... 348/14.08
`
`5,878,174 A 3/1999
`
`
`Stewart et al.
`
`5,904,484 A 5/1999 Burns
`
`* cited by examiner
`
`

`

`US8,878,949 B2
`U.S. Patent
`Nov.4, 2014
`Sheet 1 of 7
`US 8,878,949 B2
`U.S. Patent Nov. 4, 2014
`Sheet 1 of 7
`
`0 �
`�
`
`N f'-.. (/
`
`I
`
`
` “COMPUTER__&
`
`/
`
`O'.'.'.
`
`_ .. ------....... , ...
`. - ..
`",
`-----,, \
`
`...-
`
`Page 3 of 17
`
`

`

`Nov. 4, 2014 Sheet 2 of 7 US 8,878,949 B2
`U.S. Patent
`
`240
`
`246
`
`215
`
`I ill
`
`210
`
`211
`
`COMPUTER
`
`220
`
`FIG. 2A
`
`291
`
`',
`
`' ••• I
`i\\
`.' ..'
`
`'
`'
`'
`
`.
`
`�287
`
`290
`
`FIG. 2D
`
`

`

`U.S. Patent Nov. 4, 2014 Sheet 3 of 7 US 8,878,949 B2
`
`COMPUTER POSE ANALYSIS CAMERA CONT.
`
`1----<l-----il
`255
`250
`220
`
`FIG� 2B
`
`201
`
`271
`
`265
`
`

`

`Nov. 4, 2014 Sheet 4 of 7 US 8,878,949 B2
`U.S. Patent
`
`300
`
`315
`
`, /
`/
`
`320
`
`330
`
`310
`
`350
`
`FIG� 3
`
`422
`
`("r - ·-------·------- ----------------------- - --- ---- -
`_ _ ......,-- ------·------ - --------·----
`401
`
`421
`
`430
`
`415
`
`450
`
`FIG. 4
`
`

`

`U.S. Patent Nov. 4, 2014 Sheet 5 of 7 US 8,878,949 B2
`
`565
`560 555
`
`PROC.
`
`FIG. 5
`
`500
`
`640
`
`620
`
`630
`
`/ L
`L
`COMPUTER 600
`
`660
`
`FIG. 6
`
`

`

`U.S. Patent
`US 8,878,949 B2
`Nov. 4, 2014
`Sheet 6 of 7
`
`701
`
`711
`
`'-----1
`
`COMPUTER
`
`FIG. 7
`
`

`

`U.S. Patent
`US 8,878,949 B2
`Nov. 4, 2014
`Sheet 7 of 7
`Sheet 7 of 7 US 8,878,949 B2
`U.S. Patent
`Nov. 4, 2014
`
`LINUELNI
`
`YAHLOHO
`
`cc.:
`w
`
`AWIdSI0
`
`.....I
`0..
`(/)
`
`0
`
`
`
`�------!
`
`AL
`
`Page 9 of 17
`
`WNOLLdO
`
`NaSuOsS
`
`WNOLLdO
`
`NaaeoOs
`
`0
`('")
`OCl
`
`|oF|PHSLNcHACD|
`
`I-['.
`
`
`�I �co
`
`0
`0
`
`
`
`

`

`
`
`US 8,878,949 B2
`
`1
`
`2
`
`
`
`CAMERA BASED INTERACTION AND
`
`INSTRUCTION
`
`specific dates or other circumstances exist. In some cases,
`
`
`
`
`
`
`
`information as to what is desired may be entered remotely,
`
`
`even over the internet, or radio telephone.
`
`
`It is also a goal of the invention to provide a method for
`Method and apparatus are disclosed to enhance the quality
`
`
`
`
`
`
`memory, pictures 5 selecting from a digital or other picture
`
`
`
`
`and usefulness of picture taking for pleasure, commercial, or
`
`
`
`obtained when certain pre programmed poses of objects,
`
`
`
`
`other business purposes. In a preferred embodiment, stereo
`
`
`
`
`sequences of poses, or relationships of objects are repre­
`
`
`photogrammetry is combined with digital image acquisition
`sented.
`
`
`
`to acquire or store scenes and poses of interest, and/or to
`It is a further goal of the invention to provide means by
`
`
`
`
`
`
`interact with the subject in order to provide data to or from a
`
`
`
`
`
`which users engaged in digital camera based activities, or
`
`
`
`computer. Other preferred embodiments illustrate applica-
`10
`
`
`other activities, using a computer can have their pictures
`
`
`tions to control of display systems.
`taken.
`It is a still further goal to provide all such functions in a 2D
`
`
`
`
`
`
`
`
`
`or 3D context, and using simple equipment capable of wide­
`
`spread use.
`
`
`
`
`Representative of USA patents on Digital cameras are U.S. 15
`It is another goal of the invention to feed back data to a
`
`
`
`
`
`
`Pat. Nos. 5,534,921, 5,249,053 and many others which
`
`
`
`
`
`subject or subjects having his or her, or their picture taken, in
`
`describe use of matrix array (CCD or otherwise) based cam­
`
`
`
`order that they assume another pose or engage in another
`
`eras to take pictures of humans or other objects. The images
`
`
`
`
`activity, or juxtaposition of subject positions.
`taken are generally comprised of 400,000 or more pixels
`
`
`
`
`
`
`
`While this invention is primarily aimed at the general pic-
`which are often compressed to smaller record sizes for data
`
`
`
`
`
`
`
`
`20 ture taking public at large, it is realized that commercial
`
`
`
`
`
`storage, for later retrieval and display. Video cameras or Cam­
`
`
`
`photographers and cine-photographers, for example in the
`
`
`
`
`corders are also increasingly able to take still photographs as
`
`
`
`coming trend to digital "Hollywood" movie making, may
`
`
`well, and record or transmit them to computers.
`
`
`benefit greatly from the invention herein, as it potentially
`
`
`
`
`Aside from exposure control (to keep the light reaching the
`
`
`allows more cost effective film production by giving the
`detector array within the dynamic range of same), and range
`
`
`
`
`
`
`
`
`
`25 director the ability to expose the camera to the presence of
`
`
`
`
`
`finding (to effect the best lens focus given the object distance
`
`
`
`
`masses of data, but only saving or taking that data which is
`
`in question) there are few cases known to the inventor where
`
`
`
`
`further data useful, and if desired, to sign al the creation of
`
`
`
`
`the camera taking the picture actually determines some vari­
`
`
`based on data obtained. All this with little or no human inter-
`
`
`
`able in the picture and uses it for the process of obtaining the
`
`
`
`
`
`vention as desired, thus saving on the cost of direction, film
`picture.
`
`
`
`30 crews, and other labor or venue related costs.
`
`
`
`One such example that does not take a picture of humans
`DRAWINGS DEPICTING PREFERRED
`
`
`
`but rather of data, is exemplified by U.S. Pat. No. 4,791,589,
`
`EMBODIMENTS OF THE INVENTION
`
`
`
`is where a certain wave form sign ature on an oscilloscope
`
`
`
`
`searched for by processing the digital camera image, and
`FIG. 1 illustrates means by which users engaged in digital
`
`
`
`when it is seen, the image stored.
`35
`
`
`camera based activities, or other activities, using a computer
`
`
`
`
`
`More apropos the function of"Picture Taking" as the gen­
`
`can have their pictures taken.
`
`
`
`eral public knows it and of interest as the primary focus of the
`FIGS. 2A-2D illustrate a method for taking pictures when
`
`
`
`
`
`
`instant invention, is U.S. Pat. No. 5,781,650 by Lobo, et al
`
`
`
`
`certain pre progranmied poses of objects, sequences of poses,
`
`
`
`
`which describes analysis after the fact of recorded images to
`
`
`or relationships of objects are represented.
`
`
`
`
`
`determine facial content and thus the age of the subject. This 40
`
`
`FIG. 3 illustrates a self timer like mode, or when specific
`
`
`
`disclosure also alludes to a potential point and shoot capabil-
`
`
`
`dates or other circumstances exist, including a system
`
`ity also based on the age classification of the individuals
`
`
`
`
`embodiment for taking pictures in shopping malls or other
`
`whose picture is desired.
`
`
`
`
`
`locales and providing instant print or other hardcopy capabil­
`
`
`
`There is no known picture taking reference based on object
`
`ity ( e.g. on a tee shirt).
`
`
`
`
`
`position and orientation with respect to the camera, or other 45
`FIG. 4 illustrates means to provide all such functions in a
`
`
`
`
`objects that I am aware of.
`
`
`
`
`2D or 3D context, using simple equipment capable of wide­
`
`
`
`
`spread use. Various retroreflective artificialtarget configura­
`
`tions are also disclosed.
`
`
`
`
`
`High Resolution Digital still cameras employing matrix
`
`
`FIG. 5 illustrates a method to feed back data to a subject
`50
`
`
`photodetector array chips to scan the image produced by the
`
`
`having his or her picture taken, in order that the subject
`
`
`camera lens are now commonplace, and will be even more so
`
`
`
`assumes another pose or engage in another activity.
`
`in a few years as chips and memories become very inexpen­
`
`
`
`
`FIG. 6 illustrates a commercial version of the invention
`sive, and pixel density approaches 2000x2000 pixels, rivaling
`
`
`
`
`
`
`
`
`
`useful for police departments and real estate agents, among
`
`
`
`photographic film. Even today Camcorders having 700x500
`55 others.
`
`pixel image chips are common for video based data and stills.
`FIG. 7 illustrates an embodiment of the invention used for
`
`
`
`
`
`
`This invention is aimed at improvements in utilization of
`
`
`photography of stage performances.
`
`these cameras and others which make use of a computer based
`
`
`
`
`
`
`
`
`FIG. 8 illustrates an embodiment of the invention used for
`camera's ability to analyze, in real time if desired, the images
`
`
`
`
`
`
`
`
`
`ballet instruction and other teaching and interaction activities
`
`
`
`
`
`obtained. Indeed a picture taking system may be composed of
`
`
`
`60 also with remotely located instructors or players.
`
`
`
`a combination of cameras, some used for purposes other than
`
`
`the recording of the picture proper.
`
`
`
`
`It is a goal of the invention to provide a method for taking
`
`
`
`
`pictures when certain poses of objects, sequences of poses,
`
`
`
`
`motions of objects, or any other states or relationships of 65
`objects are represented. It is also a goal to allow this to be done Illustrated in FIG. 1 of the invention is means by which
`
`
`
`
`
`
`
`
`
`
`
`
`
`in a self timer like mode, when desired scene situations or users engaged in digital camera based activities, or other
`
`BACKGROUND
`
`SUMMARY OF THE INVENTION
`
`EMBODIMENTS OF THE INVENTION
`
`FIG.1
`
`

`

`
`
`US 8,878,949 B2
`
`4
`3
`activities, using a computer can have their pictures taken, and lighting for any other camera or cameras to use in taking
`
`
`
`
`
`
`
`
`
`
`
`
`
`in this context, FIG. 1 resembles that of co-pending refer­photographs or other purposes.
`
`
`
`
`enced application 9 above. A single camera, or a set, such as
`
`It is noted that cameras 101 and 102 need not have the
`
`
`
`image of the retro-reflector or other discernable target be in
`
`
`
`a stereo pair are employed to see portions of an object, such as
`
`
`
`precise focus, indeed it is often helpful to have a some blur
`
`
`a person, a part of a person such as a hand, leg, foot, fingers,
`5
`
`
`due to defocusing so as to aid sub pixel position solution of
`
`
`
`or head, and/or to view datums on an object, portion of an
`
`
`
`datum location. If the LEDs or other light sources are in the
`
`
`object, or an object held by the person or with which the
`
`
`
`
`
`near infrared, and the camera lenses are focused in the visible,
`
`
`
`
`
`
`person interacts. In addition, multiple persons and objects can
`
`
`
`this occurs naturally, unless the lens is also near infrared
`be seen.
`
`10 chromatic corrected.
`
`
`
`Where a single camera is employed, 2D measurements of
`An optional laser pointer ( or other suitable illumination
`
`
`
`
`
`
`object location relative to the camera (x and y perpendicular
`
`
`
`
`source), comprised of diode laser and collimating optics 150
`
`
`
`
`to the camera axis) are all that is possible, unless datums of
`
`
`
`
`
`is also usable with the invention to illuminate object portions
`
`
`
`known shape or spacing are used on the object viewed. Where
`
`
`
`from which 3D data is desired (such as the neck region of
`
`
`
`
`a stereo pair or more of cameras are employed, 3D (xyz) data
`to design ate which
`
`
`15 person 51 as shown), or in the simpler case
`
`
`
`of a single point can be provided, for example retro-reflector
`
`
`areas of a picture are to be focused, or zoomed in on or
`
`
`
`50 on the head 52 of person 51. In both cases where 3 or more
`
`
`
`transmitted or recorded-with or without consideration of
`
`
`
`datums are used on an object, 6 Degree of freedom data can be
`
`
`
`
`3-D position data of the object. This can be fixed as shown, or
`
`
`
`
`obtained, allowing object orientation in 3 angular axes as well
`
`
`
`
`optionally hand held by the user, for example in left hand
`
`as range in 3 axes to be obtained. With two or more cameras,
`
`
`
`( dotted lines) and used by him or her to designate the point to
`20
`
`
`such 3D data may also be obtained using other features of
`
`
`
`
`be measured in 3D location. (see also references above). In
`
`
`objects such as edges of arms and the likely using known
`
`
`
`
`
`addition a person taking pictures, such as a photography can
`
`photogrammetric techniques.
`
`
`
`
`
`without looking through the viewfinder of the camera, point
`
`
`The cameras used may also be used to take pictures of an
`
`
`
`
`to appoint on the subject, which is then dealt with by camera
`
`
`
`object, or another specialized camera used for that purpose in 25
`
`
`
`
`
`typically by focusing the lens system such that the point is in
`
`
`
`conjunction with those used to determine the location of
`
`
`
`
`the desired state of focus (usually but not necessarily when
`
`
`
`
`object features. Both examples are illustrated in this applica­
`
`
`
`
`the laser spot on the subject appears smallest in diameter
`tion.
`
`
`
`and/or of highest contrast). Such as system is particularly
`
`
`
`As shown in this figure, two cameras 101 and 102 are used
`
`
`
`
`useful for cameras with wide fields of view, or those mounted
`
`
`
`as a stereo pair, with each camera located at opposite sides of 30
`on pan tilt mechanisms, where the mechanism can also be
`
`
`
`a TV monitor 105, used for either computer or Television
`
`
`
`activated to position the camera axis to take the picture with
`
`
`
`display or both. This is a desirable configuration commer­
`
`
`
`the laser spot for example centered in the camera field.
`
`
`
`
`cially and discussed the co-pending application references
`
`
`
`
`In the laser designated case, it is generally the laser spot or
`
`
`above. In this particular case, an additional camera 110 is
`
`
`
`
`
`
`other indication on the surface that is imaged, (although one
`
`
`
`shown in the middle of the other two, said added camera used 35
`
`
`can also instruct, for example using voice recognition soft­
`
`
`
`
`for picture taking, internet telephony and/or other purposes.
`
`
`
`
`ware in computer 130 inputted via voice activated micro-
`
`
`
`An optional auxiliary LED light source 115 (or 116 or 117)
`
`
`phone 135, the camera processor to obtain and store if desired
`
`
`
`
`for illuminating a user 60 or other object is also shown.
`
`
`the image of the area around the spot projected onto the object
`
`
`
`All three cameras are connected to the computer 130 by
`
`
`
`
`as well or alternatively), and if the spot is desired, it is often
`
`
`
`means of a USB (Universal Serial Bus) daisy chain, or IEEE 40
`
`
`
`useful that cameras 101 and 102 have band-pass filters which
`
`
`
`1394 firewire connections (faster). Each is accessed, as
`
`
`
`pass the laser wavelength, and any led illumination wave-
`
`
`
`
`needed for position and orientation determination, or picture
`
`
`
`
`lengths used for retro-reflector illumination for example, but
`taking.
`
`
`
`block other wavelengths to the extent possible at low cost. It
`
`
`
`Even using a single camera in two dimensions ( as is normal
`
`
`is noted that the discrimination in an image can also be made
`
`
`
`
`today), some position and orientation data or sequences of 45
`
`
`on color grounds-i.e. with red diode lasers and red LEDs,
`
`
`same can be achieved using modern image processing tech­
`
`
`
`the system can analyze the image areas containing reds in the
`
`
`
`
`niques. (See for example the invention disclosed in U.S. Pat.
`
`
`
`image, for example-with the knowledge that the answer
`
`
`
`No. 4,843,568 of Myron Krueger). However, accurate sens­
`
`
`
`can't lie at any shorter wavelengths ( e.g. green, yellow, blue).
`
`
`
`ing and control of systems, such as cameras herein is difficult
`
`
`By using two cameras 101 and 102, a superior ranging
`
`
`today with processors cost effective enough to be used by the
`
`
`
`
`
`
`system for the laser spot location on the subject results, since
`50
`
`
`
`public at large, and artificial target augmentation of image
`
`
`
`
`the baseline distance "BL" separating the cameras for trian­
`
`points is often desirable.
`
`
`
`
`gulation based ranging purposes can be sufficient to provide
`
`
`
`
`It is thus possible using the invention to be taking pictures
`
`
`accurate measurement of distance to the object.
`
`
`
`of users of interactive computer systems for whatever pur­
`
`
`
`pose. This allows one to automatically capture images of 55
`
`
`
`
`children at play, for example with a computer system such as
`a computer game. It also enables many other functions which As we begin to consider the apparatus of FIG. 1, it is clear
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`are described below. And it can be used in the field, where the one could do much more to enhance picture taking ability
`
`
`
`
`
`
`
`
`
`
`computer, stereo position sensing and picture taking camera, than hereto fore described and contained in the prior art. And
`
`
`
`
`
`
`may be co-located together in the same housing. 60 it can be done with apparatus capable of field use.
`
`
`
`
`It is noted that where retro-reflectors are used, (as opposed FIGS. 2A-2D for example, illustrates a method for taking
`
`
`
`
`
`
`
`
`
`to choosing for example less contrasting datums, for example pictures when certain pre programmed or otherwise desired
`
`
`
`
`
`
`
`
`
`natural object features such as edges of fingers, or clothing poses of objects, sequences of poses, or relationships of
`
`
`
`
`
`
`
`
`
`features, or targets such as colored dots) then each of the two objects are represented. No such ability is available to pho­
`
`
`
`
`
`
`
`cameras for stereo location determination needs lights to 65tographers today.
`illuminate retro-reflectors substantially co-located with the Consider still camera system 201, patterned after that of
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`camera axes. These lights can alternatively provide general FIG. 1 and comprising 3 cameras and associated image scan-
`
`FIGS. 2A-2D
`
`

`

`
`
`US 8,878,949 B2
`
`5
`
`6
`
`ning chips. The central camera, 202, is for picture taking and
`
`
`
`
`
`
`frames of high resolution preferred data. (This doesn't solve
`
`
`
`
`
`
`
`
`has high resolution and color accuracy. The two cameras on
`
`
`
`flicker problems, but it does mean that poor quality or extra­
`
`
`either side, 210 and 211, may be lower resolution (allowing
`
`
`
`neous material isn't sent!). Criteria such as degree of image
`
`
`
`lower cost, and higher frame rate, as they have less pixels to
`
`
`motion blur or image focus can also be used in making trans-
`
`
`
`
`scan in a given frame time), with little or no accurate color
`
`5 mission decisions.
`
`
`
`capability, as they are used to simply see object positions or
`FIG. 2B illustrates a block diagram showing a pose analy­
`
`
`
`
`
`
`
`special datum positions on objects (which may be distin­
`
`
`sis software or hardware module 250 analyzing processed
`
`
`
`
`
`
`guished however by taught colors for example as taught in
`
`
`image data (for example utilizing camera image data pro­
`
`some ofmy co-pending inventions).
`
`
`
`
`cessed by visionbloks software from Integral Vision Corp.)
`
`
`
`Cost wise the distinction between cameras is important.
`
`10 from the computer 220 (which may be the same physical
`
`
`
`
`Today low cost CMOS chips and lenses capable of the pro­
`
`
`microprocessor, such as a Intel Pentium 2 in a Dell inspiron
`
`
`
`
`viding stereo measurements as described above are $15 or
`
`
`
`
`3500 laptop computer, or different) and determining from
`
`
`
`less. High quality CCD color detector arrays and lenses for
`
`
`
`same when a certain pose for example has been seen. When
`
`
`high quality photo images are over $100, and in many cases
`
`
`255 camera control module this occurs, a sign al is sent to the
`
`$ 1000 or more.
`
`15 to hold the last frame taken by camera 202, and to display it to
`
`
`
`
`An optical viewfinder 215 is one of many ways to indicate
`
`
`
`the photographer, digitally store it, or transmit it-to some­
`
`
`one else, or another data store or display. Such transmission
`
`
`
`to the user what scene information is being gathered by the
`
`
`
`can be by data link, internet, cell phone, or any other suitable
`camera system. The user can in this invention specify with a
`
`
`
`means.
`viewfinder based readout, the area of the field that is desired.
`
`
`
`
`Another criteria could be that two or more preselected
`
`
`
`Use of the viewfinder in this manner, whether looked through
`
`
`
`
`
`20
`
`
`poses were seen one after the other, with a time delay between
`
`
`
`
`or displayed on a screen, is for example an alternative to
`
`
`them, also pre-selected if desired.
`
`
`
`
`designating an area on the actual object using a laser pointer
`
`
`
`FIG. 2C illustrates a specific case whereby a point on one
`
`for the purpose.
`
`
`
`The camera system 201 further contains a computer 220
`
`
`person, say hand 260 of man 265 having head 271, is deter­
`which processes the data from cameras 210 and 211 to get 25
`
`
`
`mined, and a picture is taken by camera system 201 of the
`
`various position and/or orientation data concerning a person
`
`
`
`
`
`
`
`
`invention when this point comes within a distance of approxi-
`
`
`
`mately 6 inches ( or any other desired amount including con­
`( or other object, or persons plural, etc). Integral light sources
`
`
`
`
`
`
`
`
`
`tact-i.e. zero distance) from another person or object, say
`
`
`as described in FIG. 1 above may also be provided such as
`
`the head 270 of woman 275. To obtain the data, one can look
`LED arrays 240 and 245 and xenon flash 246.
`
`for hand or head indications in the image using known
`
`
`
`
`In general, one can use the system to automatically "shoot"
`30
`
`
`
`
`
`machine vision techniques, and/or in a more simple case put
`
`
`pictures for example, when any or all of the following occur,
`
`
`
`
`a target marker such as colored triangle 285 or other type on
`
`
`
`as determined by the position and orientation determining
`
`the hand or head or both and look for it.
`
`
`system of the camera of the invention:
`The use of the natural features of the subjects heads, which
`
`
`
`
`
`
`
`1. Subject in a certain pose.
`
`35 are distinguishable by shape and size in a known field con­
`
`
`2.Subject in a sequence of poses.
`
`
`
`
`taining two persons, is now illustrated. For example, image
`
`
`
`3.Portion of Subject in a sequence of poses (e.g. gestures).
`
`
`
`morphology or template matching in the image field of the
`
`
`
`4. Subject orportion(s) in a specific location or orientation.
`
`solid state TV camera 202 can be used to distinguish the head
`
`
`
`
`5.Subject in position relative to another object or person.
`
`
`
`shapes from background data and data concerning the rest of
`
`
`
`For example, this could be bride and groom kissing in a
`
`
`
`the features such as hands, etc. of subjects 265 and 275 (or
`
`
`wedding, boy with respect to cake on birthday, and sports
`40
`
`
`conversely hand shapes if desired can be found and heads
`
`
`
`
`events sequences of every description (where the camera can
`
`
`
`
`excluded, or the hand of the right person, versus the head of
`
`
`
`even track the object datums in the field and if desired adjust
`
`the left, and so forth).
`
`
`shutter speed based on relative velocity of camera to subject).
`As shown in FIG. 2D, when the image field in certain 287 of camera
`
`
`
`6. Ditto all of above with respect to both persons
`
`
`
`
`45 202 after processing contains the two head images, 290 and
`poses or gesture situations.
`
`
`
`
`
`
`
`
`7. When a subject undertakes a particular signal compris­291, spaced a distance "W". When W is not within a tolerance
`
`
`
`
`
`
`
`ing a position or gesture-i.e. a silent command to take the D, the picture is not taken; whereas if the heads are close
`
`
`
`
`
`
`
`
`
`
`picture (this could be programmed, for example, to corre­enough, within D as illustrated in dotted lines, the picture is
`
`spond to raising one's right hand).
`taken.
`In addition it is noted that the invention acts as a 50 Criteria as mentioned can include proximity of other parts
`
`
`
`
`
`
`
`
`
`
`of the body, or objects associated with the subjects (which
`
`
`
`
`rangefinder, finding range to the subject, and even to other
`
`
`subjects around the subject,
`motion or relative themselves can be objects). In addition, the or to all parts of interest on an
`
`
`
`
`
`motion of objects can be the criteria. For example, one could
`extensive subject. This allows a desired lens focus to be set
`
`
`
`
`
`
`
`
`
`when on two to take the picture take program the device allows
`
`
`
`based on any or all of this data, as desired. It also a
`
`
`
`sequence of pictures to be taken of different objects or object 55 successive frames the condition shown in FIG. 2D exists
`
`
`
`
`
`
`
`
`
`portions, at different focal depths, or focus positions. The
`
`
`where the heads are apart in frame 1, but closer in frame 2
`
`
`
`
`
`
`same holds true for exposure of these locations as well. (probably corresponding to a movement say of the boy to kiss
`
`
`
`
`It is also possible to use the above criteria for other pur­
`
`
`
`
`
`the girl). Clearly other sequences are possible as well, such as
`
`
`
`
`
`
`
`
`
`poses, such as determining what to record (beyond the record­movement taking place in several frames followed by a
`
`
`
`
`
`ing that is implicit in taking pictures), or in determining what
`
`
`
`60 sequence of frames in which no movement occurs. Other
`
`
`
`
`to transmit. The latter is important vis a vis internet
`
`
`means to determine activity, motion in front of the camera can also be
`
`
`
`where available internet communication bandwidth limits used in this context, such as ultrasonic sensors.
`
`
`
`
`
`
`
`
`
`
`what can be transmitted (at least today). In this case video It is also noted that the actual position or movement desired
`
`
`
`
`
`
`
`
`telephony with the invention comprehends obtaining only can be "Taught" to the computer 220 of the picture taking
`
`
`
`
`
`
`
`those images you really care about in real time. So instead of 65 system. For example, a boy and girl in a wedding could
`
`
`
`
`
`
`
`transmitting low resolution image data at 20 frames a second, approach each other and kiss beforehand. The sequence of
`
`
`
`
`
`
`
`you can transmit say 5 (albeit asynchronously gathered) frames of this activity ( a "gesture" of sorts by both parties) is
`
`

`

`
`
`US 8,878,949 B2
`
`8
`
`7
`recorded, and the speed of approach, the head positions and desired effect resulting in a picture). The effect desired can be
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`any other pertinent data determined. When the photographer changed in midstream to adjust for changing wants as well, by
`
`
`
`
`
`
`
`
`
`thinks the picture is right, the computer of the camera system changing the program of the computer (which could be done
`
`
`
`using hardware switches, inserting a disc, or otherwise
`
`
`
`is instructed to take the picture-for example it could be at the
`
`
`
`
`
`5 entered as a command). In addition, as mentioned above, the
`instant when after a suitable approach, two head images
`
`
`
`
`
`
`
`
`gesture or pose desired, can be taught to the system, by first
`
`
`become joined into one-easily recognizable with machine
`
`
`
`
`photographing a variety of acceptable positions or sequences,
`
`
`
`
`
`vision processing software under uniform background con­
`
`
`
`and putting bounds on how close to these will be accepted for
`
`
`
`ditions. Then in the future, when such a condition is reached
`photographing.
`
`
`
`in the camera field of view, pictures are taken and stored, or
`A specialized case is shown in FIG. 3, for self taking instant
`
`
`
`
`
`transmitted. This allows a camera to free run whose image 10
`
`
`
`
`picture or printout device for use in a shopping mall Kiosk or
`
`
`field for example takes in the head table at a wedding party,
`
`
`other venue. In this case two sweethearts 300 and 310 are on
`
`
`
`
`taking only the shots thought to be of most interest. Numerous
`
`
`
`a bench 315 in front of the digital or other camera 320. When
`
`
`
`
`conditions might be programmed in, or taught in-another at
`
`
`
`the computer 330 detects from processing the image (or
`
`the same party, would be anyone at the head table proposing
`
`
`
`
`15 images) of the invention that their faces are in close proximity
`
`
`If a toast to the bride and gr oom, with arm and glass raised.
`
`
`
`
`(for example using the centroid of mass of their head as the
`
`
`
`video is taken, it might be taken from the point at which the
`
`
`
`
`
`position indicator, or even facial features such as described in
`
`
`arm rises, until after it comes down. Or with suitable voice
`
`
`
`the Lobo et al patent reference), the computer then instructs
`
`
`
`
`
`recognition, when certain toast type words are heard, for
`
`
`
`the camera to record the picture. A push button or other
`example.
`
`
`
`
`
`
`20 selector on the device allows the subjects to select what cri­
`
`
`teria they want-for example when their heads are together
`
`
`
`
`for 5 seconds or more, or not together, or hands held, or
`
`
`
`
`Where it is desired to take "3-D" pictures,
`
`
`
`
`whatever. it can be appre­ Or when their faces are within a certain distance
`
`
`
`ciated that each camera, 210 and 211 can take images of the
`
`criteria, such as one inch.
`
`scene in place of camera 202, and that both cameras 210 and 25
`
`A