`Approved for use through 01/3112014. 0MB 0651-0032
`U.S. Paten! and Trademark Office; U.S. DEPARTMENT OF COMMERCE
`Under the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless it displays a valid 0MB control number.
`
`DECLARATION (37 CFR 1.63) FOR UTILITY OR DESIGN APPLICATION USING AN
`APPLICATION DATA SHEET (37 CFR 1.76)
`
`Title of
`Invention
`
`MORE USEFUL MAN MACHINE INTERFACES AND APPLICATIONS
`
`A:5 the below named inventor, I hereby declare that:
`
`This declaration
`is directed to:
`
`The attached application, or
`
`United States application or PCT international application number ________ _
`
`filed on _____________ _
`
`The above-identified application was made or authorized to be made by me.
`
`I believe that I am the original inventor or an original joint inventor of a claimed invention in the application.
`
`I hereby acknowledge that any willful false statement made in this declaration is punishable under 18 U.S.C. 1001
`by fine or imprisonment of not more than five (5) years, or both.
`
`WARNING:
`Petitioner/applicant is cautioned to avoid submitting personal information ,n documents filed 1n a patent appllcation that may
`contribute ta identity theft. Personal information such as social security numbers, bank account numbers, or credit card numbers
`(other than a check or credit card authorization form PTO-2038 submitted for payment purposes} is never required by the USPTO
`to support a petition or an application. lf this type of personal information is included in documents submitted to the US PTO,
`petitioners/applicants should consider redacting such personal information from the documents before submitting them to the
`USPTO. Petitioner/applicant is advised that the record of a patent application is available to the public after publication of the
`application (unless a non-publication request in compliance with 37 CFR 1.213(a) is made in the application) or issuance of a
`patent. Furthermore, the record from an abandoned application may also be available to the public lf the application is
`referenced in a published application or an issued patent (see 37 CFR 1, 14}. Checks and credit card authorization forms
`PTO-2038 submitted for payment purposes are not retained in the application file and therefore are not publicly available.
`
`LEGAL NAME OF INVENTOR
`
`Inventor: Timo~hy ir'Pryor
`~I ,r; 1 /
`/ IL..... I/~ ---,
`Signature:
`/
`/
`
`.,
`
`•
`
`Date(Optional)Ve c b u1·-z,
`
`r
`
`I
`
`Note: An application data sheet (PTO/S8114 or equivalent), including naming the en!ire inventive entity, must accompany this form or must have
`been previously filed Use an additional PTO/AJA/01 fonn for each additional inventor.
`
`This collect,on of mforma~on ,s required by 35 U.S.C. 115 and 37 CFR 1.63. The Informat,on Is required to obtain or retam a benefit by the public which is to file (and
`by the USPTO to proress) an application. Confidentiality is governed by 35 U.S C. 122 and 37 CFR 1.11 and 1.14. This collection is estimated lo lake 1 minute to
`complete,. indudlng gathering. preparing, and submitting the completed application form to the USPTO. Time will vary depending upon the individual case. Any
`comments on the amount of time you require to complete !his form and/or suggestions for reducing this burden, should be sent to the Chief Information Officer, U.S.
`Patent and Trademark Offire, U.S. Department of Commerce, P.O. Box 1450, Alexandria, VA 2231:J-1450. DO NOT SEND FEES OR COMPLETED FORM$ TO
`THlS ADDRESS. SEND TO: Commissioner for Patents, P.O. Box 1450, Alexandria, VA 22313-1450.
`If you need assistance in completing the form. call 1-B00PTCJ..9199 and select opo·on 2
`
`Page 1 of 195
`
`GOOGLE EXHIBIT 1002
`
`
`
`..... -
`
`--.J
`
`109
`
`138
`
`1/7
`
`107 / 170
`
`_,I
`211
`
`106
`
`
`
`FIG. 1
`
`100
`
`Page 2 of 195
`
`
`
`(\.) -- -....J
`
`109
`
`2/7
`
`/170
`
`107
`
`W 160
`
`~
`140
`
`
`
`F/G.2
`
`100
`
`'
`
`'.z_J cp
`
`Page 3 of 195
`
`
`
`-.....J
`.._
`vJ
`
`3/7
`
`138
`
`~~·04
`
`
`
`FIG. 3
`
`100
`
`Page 4 of 195
`
`
`
`FIG.4
`
`--..J
`~ ...._
`
`4/7
`
`430
`
`450
`
`vOld
`
`400
`
`411
`
`408
`
`CPU
`
`Page 5 of 195
`
`
`
`"=l
`(J\
`
`cP\J
`
`oes
`
`540
`
`535
`
`00g
`
`Ges
`
`(oh
`
`ovs
`
`5'\ '\
`
`09S
`
`505
`
`500
`
`FfG. 5
`
`560
`
`g‘Old
`
`. , .. ,
`.
`.......
`f ·r0·· ..•• f
`,t. ....
`.
`..
`.... ,
`,
`......... ~
`,all
`~
`'
`5 '\ 0
`···-...
`
`.
`
`,
`
`Page 6 of 195
`
`
`
`
`957
`
`--..J
`...._
`0)
`
`958
`
`o---,-990
`
`935
`
`/£
`
`/
`
`/
`
`/
`
`/
`
`/
`
`/
`
`,,,,✓
`/
`
`/
`
`/
`
`,,,, ,,,,
`
`955
`
`/
`
`930
`
`910
`
`956
`
`DISPLAY
`
`CPU
`
`951
`
`FIG. 6
`
`901
`
`CPU
`
`'--902a
`........... .
`...... ",: _______.
`
`,,,,✓✓/
`
`ciD
`
`906
`
`Page 7 of 195
`
`
`
`7/7
`
`1001
`
`1014
`
`..........
`,,, ...... =:: ..... ..
`,,..
`.. .... ;~.
`.............. ,/ ,,'
`·-... -.. /.:\. __ 1020
`
`,'
`
`1000
`
`1030
`
`C
`p
`u
`
`1002
`
`FIG. 7A
`
`1070
`
`1056
`
`1075
`
`~
`
`1060
`
`INPUT
`WOMAN'S
`MEASURE- ~~-~
`MENTS
`) ~ - - -~
`CPU
`
`INTERNET
`
`1055
`
`(
`INTERNET
`REMOTE
`
`1050
`
`FIG. 7B
`
`Page 8 of 195
`
`
`
`MORE USEFUL MAN MACHINE INTERFACES AND APPLICATIONS
`
`CROSS REFERENCE TO RELATED APPLICATIONS
`
`[0001]
`
`This application is a continuation of U.S. Patent Application No. 12/700,055, filed
`
`February 4, 2010 (now U.S. Patent _ _ _ _ ), which is a continuation of U.S. Patent
`
`Application No. 10/866,191, filed June 14, 2004, which is a continuation of U.S. Patent
`
`Application No. 09/433,297, filed November 3, 1999 (now U.S. Patent 6,750,848), which claims
`
`benefit of U.S. Provisional Application No. 60/107,652, filed November 9, 1998. These
`
`applications are hereby incorporated by reference.
`
`REFERENCES TO RELATED APPLICATIONS BY THE INVENTORS
`
`U.S. Patent Application No. 09/138,339, filed August 21, 1998.
`
`U.S. Provisional Application No. 60/056,639, filed August 22, 1997.
`
`U.S. Provisional Application No. 60/059,561, filed September 19, 1998.
`
`Man Machine Interfaces: SN 08/290,516, filed 8/15/1994, and now USP
`
`[0002]
`
`[0003]
`
`[0004]
`
`[0005]
`
`6,008,800.
`
`[0006]
`
`Touch TV and Other Man Machine Interfaces: SN 08/496,908, filed 6/29/1995,
`
`and now USP 5,982,352.
`
`[0007]
`
`Systems for Occupant Position Sensing: SN 08/968, 114, filed 11. /12/1997, now
`
`abandoned, which claims benefit of 60/031,256, filed 11/12/1996.
`
`[0008]
`
`Target holes and corners: USSN 08/203,603, filed 2/28/1994, and 08/468,358
`
`filed 6/6/1995, now USP 5,956,417 and USP 6,044, 183.
`
`[0009]
`
`Vision Target Based Assembly: USSN 08/469,429, filed 6/6/1995, now
`
`abandoned; 08/469,907, filed 6/6/1995, now USP 6,301,763; 08/470,325, filed 6/6/1995, now
`
`abandoned; and 08/466,294, filed 6/6/1995, now abandoned.
`
`-1-
`
`Page 9 of 195
`
`
`
`[00010]
`
`Picture Taking Method and Apparatus: Provisional Application no. 60/133,671,
`
`filed May 11, 1998.
`
`[00011]
`
`Methods and Apparatus for Man Machine Interfaces and Related Activity:
`
`Provisional Application no. 60/133,673 filed May 11, 1998.
`
`[00012]
`
`Camera Based Man-Machine Interfaces: Provisional Patent Application no.
`
`60/142,777, filed July 8, 1999.
`
`[00013]
`
`The copies of the disclosure of the above referenced applications are incorporated
`
`herein by reference.
`
`BACKGROUND OF THE INVENTION
`
`[00014]
`
`Field of the Invention
`
`[00015]
`
`The invention relates to simple input devices for computers, particularly, but not
`
`necessarily, intended for use with 3-D graphically intensive activities, and operating by optically
`
`sensing object or human positions and/or orientations. The invention in many preferred
`
`embodiments, uses real time stereo photogrammetry using single or multiple TV cameras whose
`
`output is analyzed and used as input to a personal computer, typically to gather data concerning
`
`the 3D location of parts of, or objects held by, a person or persons.
`
`[00016]
`
`This continuation application seeks
`
`to provide further detail on useful
`
`embodiments for computing. One embodiment is a keyboard for a laptop computer ( or stand
`
`alone keyboard for any computer) that incorporates digital TV cameras to look at points on,
`
`typically, the hand or the finger, or objects held in the hand of the user, which are used to input
`
`data to the computer. It may also or alternatively, look at the head of the user as well.
`
`[00017]
`
`Both hands or multiple fingers of each hand, or an object in one hand and fingers
`
`of the other can be simultaneously observed, as can alternate arrangements as desired.
`
`-2-
`
`Page 10 of 195
`
`
`
`DESCRIPTION OF RELATED ART
`
`[00018]
`
`My referenced co-pending applications incorporated herein by reference discuss
`
`many prior art references in various pertinent fields, which form a background for this invention.
`
`BRIEF DESCRIPTION OF FIGURES
`
`[00019]
`
`Fig. 1 illustrates a laptop or other computer keyboard with cameras according to
`
`the invention located on the keyboard surface to observe objects such as fingers and hands
`
`overhead of the keyboard.
`
`[00020]
`
`Fig. 2 illustrates another keyboard embodiment using special datums or light
`
`sources such as LEDs.
`
`[00021]
`
`Fig. 3 illustrates a further finger detection system for laptop or other computer
`
`input.
`
`[00022]
`
`Fig. 4 illustrates learning, amusement, monitoring, and diagnostic methods and
`
`devices for the crib, playpen and the like.
`
`[00023]
`
`Fig. 5 illustrates a puzzle toy for young children having cut out wood characters
`
`according to the invention.
`
`[00024]
`
`Fig. 6 illustrates an improved handheld computer embodiment of the invention, in
`
`which the camera or cameras may be used to look at objects, screens and the like as well as look
`
`at the user along the lines of Fig. 1.
`
`[00025]
`
`Fig. 7 illustrates new methods for internet commerce and other activities
`
`involving remote operation with 3D virtual objects display.
`
`-3-
`
`Page 11 of 195
`
`
`
`DESCRIPTION OF THE INVENTION
`
`Figure 1
`
`[00026]
`
`A laptop (or other) computer keyboard based embodiment is shown in Fig. 1. In
`
`this case, a stereo pair of cameras 100 and 101 located on each side of the keyboard are used,
`
`desirably having cover windows 103 and 104 mounted flush with the keyboard surface 102. The
`
`cameras are preferably pointed obliquely inward at angles <l> toward the center of the desired
`
`work volume 170 above the keyboard. In the case of cameras mounted at the rear of the
`
`keyboard (toward the display screen), these cameras are also inclined to point toward the user at
`
`an angle as well.
`
`[00027]
`
`Alternate camera locations may be used such as the positions of cameras 105 and
`
`106, on upper comers of screen housing 107 looking down at the top of the fingers ( or hands, or
`
`objects in hand or in front of the cameras), or of cameras 108 and 109 shown.
`
`[00028]
`
`One of the referenced embodiments of the invention is to determine the pointing
`
`direction vector 160 of the user's finger (for example pointing at an object displayed on screen
`
`107), or the position and orientation of an object held by the user. Alternatively, finger position
`
`data can be used to determine gestures such as pinch or grip, and other examples of relative
`
`juxtaposition of objects with respect to each other, as has been described in co-pending
`
`referenced applications. Positioning of an object or portions (such as hands or fingers of a doll) is
`
`also of use, though more for use with larger keyboards and displays.
`
`[00029]
`
`In one embodiment, shown in Fig. 2, cameras such as 100/101 are used to simply
`
`look at the tip of a finger 201 (or thumb) of the user, or an object such as a ring 208 on the
`
`finger. Light from below, such as provided by single central light 122 can be used to illuminate
`
`the finger that typically looks bright under such illumination.
`
`-4-
`
`Page 12 of 195
`
`
`
`[00030]
`
`It is also noted that the illumination is directed or concentrated in an area where
`
`the finger is typically located such as in work volume 170. If the light is of sufficient spectral
`
`content, the natural flesh tone of the finger can be observed - and recognized by use of the color
`
`TV cameras 100/101.
`
`[00031]
`
`As is typically the case, the region of the overlapping cameras viewing area is
`
`relatively isolated to the overlapping volumetric zone of their fields 170 shown due to focal
`
`lengths of their lenses and the angulation of the camera axes with respect to each other. This
`
`restricted overlap zone helps mitigate against unwanted matches in the two images due to
`
`information generated outside the zone of overlap. Thus there are no significant image matches
`
`found of other objects in the room, since the only flesh-toned object in the zone is typically the
`
`finger or fingers of the user. Or alternatively, for example, the user's hand or hands. Similarly
`
`objects or targets thereon can be distinguished by special colors or shapes.
`
`[00032]
`
`If desired, or required, motion of the fingers can be also used to further
`
`distinguish their presence vis-a-vis any static background. If for example, by subtraction of
`
`successive camera frames, the image of a particular object is determined to have moved it is
`
`determined that this is likely the object of potential interest which can be further analyzed
`
`directly to determine if is the object of interest.
`
`[00033]
`
`In case of obscuration of the fingers or objects in the hand, cameras in additional
`
`locations such as those mentioned above, can be used to solve for position if the view of one or
`
`more cameras is obscured.
`
`[00034]
`
`The use of cameras mounted on both the screen and the keyboard allows one to
`
`deal with obscurations that may occur and certain objects may or may not be advantageously
`
`delineated in one view or the other.
`
`-5-
`
`Page 13 of 195
`
`
`
`[00035]
`
`In addition, it may be in many cases desirable to have a datum on the top of the
`
`finger as opposed to the bottom because on the bottom, it can get in the way of certain activities.
`
`In this case the sensors are required on the screen looking downward or in some other location
`
`such as off the computer entirely and located overhead has been noted in previous application.
`
`[00036]
`
`To determine finger location, a front end processor like that described in the target
`
`holes and corners co-pending application reference incorporated USSN 08/203,603 and
`
`08/468,358 can be used to also allow the finger shape as well as color to be detected.
`
`[00037]
`
`Finger gestures comprising a sequence of finger movements can also be detected
`
`by analyzing sequential image sets such as the motion of the finger, or one finger with respect to
`
`another such as in pinching something can be determined. Cameras 100 and 101 have been
`
`shown at the rear of the keyboard near the screen or at the front. They may mount in the middle
`
`of the keyboard or any other advantageous location.
`
`[00038]
`
`The cameras can also see one's fingers directly, to allow typing as now, but
`
`without the physical keys. One can type in space above the plane of the keyboard (or in this case
`
`plane of the cameras). This is useful for those applications where the keyboard of conventional
`
`style is too big ( e.g., the hand held computer of Fig. 6).
`
`[00039]
`
`Figure 2
`
`[00040]
`
`It is also desirable for fast reliable operation to use retro-reflective materials and
`
`other materials to augment the contrast of objects used in the application. For example, a line
`
`target such as 200 can be worn on a finger 201, and advantageously can be located if desired
`
`between two joints of the finger as shown. This allows the tip of the finger to be used to type on
`
`the keyboard without feeling unusual - the case perhaps with target material on tip of the finger.
`
`-6-
`
`Page 14 of 195
`
`
`
`[00041]
`
`The line image detected by the camera can be provided also by a cylinder such as
`
`retroreflective cylinder 208 worn on the finger 201 which effectively becomes a line image in the
`
`field of view of each camera (assuming each camera is equipped with a sufficiently coaxial light
`
`source, typically one or more LEDs such as 210 and 211), can be used to solve easily using the
`
`line image pairs with the stereo cameras for the pointing direction of the finger that is often a
`
`desired result. The line, in the stereo pair of images provides the pointing direction of the finger,
`
`for example pointing at an object displayed on the screen 140 of the laptop computer 138.
`
`[00042]
`
`Figure 3
`
`[00043]
`
`It is also possible to have light sources on the finger that can be utilized such as
`
`the 2 LED light sources shown in Fig. 3. This can be used with either TV camera type sensors or
`
`with PSD type analog image position sensors as disclosed in references incorporated.
`
`[00044]
`
`In particular the ring mounted LED light sources 301 and 302 can be modulated at
`
`different frequencies that can be individually discerned by sensors imaging the sources on to a
`
`respective PSD detector. Alternatively, the sources can simply be turned on and off at different
`
`times such that the position of each point can be independently found allowing the pointing
`
`direction to be calculated from the LED point data gathered by the stereo pair of PSD based
`
`sensors.
`
`[00045]
`
`The "natural interface keyboard" here described can have cameras or other
`
`sensors located at the rear looking obliquely outward toward the front as well as inward so as to
`
`have their working volume overlap in the middle of the keyboard such as the nearly full volume
`
`over the keyboard area is accommodated.
`
`[00046]
`
`Clearly larger keyboards can have a larger working volume than one might have
`
`on a laptop. The pair of sensors used can be augmented with other sensors mounted on the screen
`
`-7-
`
`Page 15 of 195
`
`
`
`housing. It is noted that the linked dimension afforded for calibration between the sensors
`
`located on the screen and those on the keyboard is provided by the laptop unitary construction.
`
`[00047]
`
`One can use angle sensing means such as a rotary encoder for the laptop screen
`
`tilt. Alternatively, cameras located on the screen can be used to image reference points on the
`
`keyboard as reference points to achieve this. This allows the calibration of the sensors mounted
`
`fixedly with respect to the screen with respect to the sensors and keyboard space below. It also
`
`allows one to use stereo pairs of sensors that are not in the horizontal direction (such as 101/102)
`
`but could for example be a camera sensor such as 100 on the keyboard coupled with one on the
`
`screen, such as 106.
`
`[00048]
`
`Knowing the pointing angles of the two cameras with respect to one another
`
`allows one to solve for the 3D location of objects from the matching of the object image
`
`positions in the respective camera fields.
`
`[00049]
`
`As noted previously, it is also of interest to locate a line or cylinder type target on
`
`the finger between the first and second joints. This allows one to use the fingertip for the
`
`keyboard activity but by raising the finger up, it can be used as a line target capable of solving
`
`for the pointed direction for example.
`
`[00050]
`
`Alternatively one can use two point targets on the finger such as either
`
`retroreflective datums, colored datums such as rings or LED light sources that can also be used
`
`with PSD detectors which has also been noted in Fig. 2.
`
`[00051]
`
`When using the cameras located for the purpose of stereo determination of the
`
`position of the fingers from their flesh tone images it is useful to follow the preprocessing
`
`capable of processing data obtained from the cameras in order to look for the finger. This can be
`
`done on both color basis and on the basis of shape as well as motion.
`
`-8-
`
`Page 16 of 195
`
`
`
`[00052]
`
`In this invention, I have shown the use of not only cameras located on a screen
`
`looking downward or outward from the screen, but also cameras that can be used instead of or in
`
`combination with those on the screen placed essentially on the member on which the keyboard is
`
`incorporated. This allows essentially the keyboard to mounted cameras which are preferably
`
`mounted flush with the keyboard surface to be unobtrusive, and yet visually be able to see the
`
`users fingers, hands or objects held by the user and in some cases, the face of the user.
`
`[00053]
`
`This arrangement is also useful for 3D displays, for example where special
`
`synchronized glasses (e.g., the "Crystal Eyes" brand often used with Silicon Graphics work
`
`stations) are used to alternatively present right and left images to each eye. In this case the object
`
`may appear to be actually in the workspace 170 above the keyboard, and it may be manipulated
`
`by virtually grasping (pushing, pulling, etc.) it, as has been described in co-pending applications.
`
`[00054]
`
`Figure 4: Baby Learning and Monitoring System
`
`[00055]
`
`A baby's reaction to the mother (or father) and the mother's analysis of the baby's
`
`reaction 1s very important. There are many gestures of babies apparently indicated in child
`
`psychology as being quite indicative of various needs, wants, or feelings and emotions, etc.
`
`These gestures are typically made with the baby's hands.
`
`[00056]
`
`Today this is done and learned entirely by the mother being with the baby.
`
`However with an Electro-optical sensor based computer system, such as that described in co(cid:173)
`
`pending applications located proximate to or even in the crib (for example), one can have the
`
`child's reactions recorded, not just in the sense of a video tape which would be too long and
`
`involved for most to use, but also in terms of the actual motions which could be computer
`
`recorded and analyzed also with the help of the mother as to what the baby's responses were.
`
`-9-
`
`Page 17 of 195
`
`
`
`And such motions, combined with other audio and visual data can be very important to the
`
`baby's health, safety, and learning.
`
`[00057]
`
`Consider for example crib 400 with computer 408 having LCD monitor 410 and
`
`speaker 411 and camera system (single or stereo) 420 as shown, able to amuse or inform baby
`
`430, while at the same time recording (both visually, aurally, and in movement detected position
`
`data concerning parts of his body or objects such as rattles in his hand) his responses for any or
`
`all of the purposes of diagnosis of his state of being, remote transmission of his state, cues to
`
`various programs or images to display to him or broadcast to others, or the like.
`
`[00058]
`
`For one example, baby's motions could be used to signal a response from the TV
`
`either in the absence of the mother or with the mother watching on a remote channel. This can
`
`even be over the Internet if the mother is at work.
`
`[00059]
`
`For example, a comforting message could come up on the TV from the mother
`
`that could be prerecorded (or alternatively could actually be live with TV cameras in the
`
`mother's or father's workplace for example on a computer used by the parent) to tell the baby
`
`something reassuring or comfort the baby or whatever. Indeed the parent can be monitored using
`
`the invention and indicate something back or even control a teleoperater robotic device to give a
`
`small child something to eat or drink for example. The same applies to a disabled person.
`
`[00060]
`
`If the father or mother came up on the screen, the baby could wave at it, move its
`
`head or "talk" to it but the hand gestures may be the most important.
`
`[00061]
`
`If the mother knows what the baby is after, she can talk to baby or say something,
`
`or show something that the baby recognizes such as a doll. After a while, looking at this live one
`
`can then move to talking to the baby from some prerecorded data.
`
`-10-
`
`Page 18 of 195
`
`
`
`[00062]
`
`What other things might we suppose? The baby for example knows to puts its
`
`hand on the mother's cheek to cause the mother to turn to it. The baby also learns some other
`
`reflexes when it is very young that it forgets when it gets older. Many of these reflexes are hand
`
`movements, and are
`
`important in communicating with
`
`the remote TV based mother
`
`representation, whether real via telepresense or from CD Rom or DVD disk ( or other media,
`
`including information transmitted to the computer from afar) and for the learning of the baby's
`
`actions.
`
`[00063]
`
`Certainly just from the making the baby feel good point-of-view, it would seem
`
`like certain motherly ( or fatherly, etc.) responses to certain baby actions in the form of words and
`
`images would be useful. This stops short of physical holding of the baby which is often needed,
`
`but could act as a stop gap to allow the parents to get another hour's sleep for example.
`
`[00064]
`
`As far as the baby touching things, I've discussed in other applications methods
`
`for realistic touch combined with images. This leads to a new form of touching crib mobiles that
`
`could contain video imaged and or be imaged themselves - plus if desired - touched in ways that
`
`would be far beyond any response that you could get from a normal mobile.
`
`[00065]
`
`For example, let us say there is a targeted (or otherwise TV observable) mobile
`
`450 in the crib above the baby. Baby reaches up and touches a piece of the mobile which is
`
`sensed by the TV camera system (either from the baby's hand position, the mobile movement, or
`
`both, and a certain sound is called up by the computer, a musical note for example. Another
`
`piece of the mobile and another musical note. The mobile becomes a musical instrument for the
`
`baby that could play either notes or chords or complete passages, or any other desired
`
`programmed function.
`
`-11-
`
`Page 19 of 195
`
`
`
`[00066]
`
`The baby can also signal things. The baby can signal using agitated movements
`
`would often mean that it's unhappy. This could be interpreted using learned movement
`
`signatures and artificial intelligence as needed by the computer to call for mother even if the
`
`baby wasn't crying. If the baby cries, that can be picked up by microphone 440, recognized using
`
`a voice recognition system along the lines of that used in IBM Via Voice commercial product for
`
`example. And even the degree of crying can be analyzed to determine appropriate action.
`
`[00067]
`
`The computer could also be used to transmit information of this sort via the
`
`internet email to the mother who could even be at work. And until help arrives in the form of
`
`mother intervention or whatever, the computer could access a program that could display on a
`
`screen for the baby things that the baby likes and could try to soothe the baby through either
`
`images of familiar things, music or whatever. This could be useful at night when parents need
`
`sleep, and anything that would make the baby feel more comfortable would help the parents.
`
`[00068]
`
`It could also be used to allow the baby to input to the device. For example, if the
`
`baby was hungry, a picture of the bottle could be brought up on the screen. The baby then could
`
`yell for the bottle. Or if the baby needed his diaper changed, perhaps something reminiscent of
`
`that. If the baby reacts to such suggestions of his problem, this gives a lot more intelligence as to
`
`why he is crying and while mothers can generally tell right away, not everyone else can. In other
`
`words, this is pretty neat for babysitters and other members of the household so they can act
`
`more intelligently on the signals the baby is providing.
`
`[00069]
`
`Besides in the crib, the system as described can be used in conjunction with a
`
`playpen, hi-chair or other place of baby activity.
`
`[00070]
`
`As the child gets older, the invention can further be used also with more advanced
`
`activity with toys, and to take data from toy positions as well. For example, blocks, dolls, little
`
`-12-
`
`Page 20 of 195
`
`
`
`cars, and moving toys even such as trikes, scooters, drivable toy cars and bikes with training
`
`wheels.
`
`[00071]
`
`The following figure illustrates the ability of the invention to learn, and thus to
`
`assist in the creation of toys and other things.
`
`[00072]
`
`Figure 5: Leaming Puzzle Roy
`
`[00073]
`
`Disclosed in Fig. 5 is a puzzle toy 500 where woodcut animals such as bear 505
`
`and lion 510 are pulled out with handle such as 511. The child can show the animal to the camera
`
`and a computer 530 with TV camera (or cameras) 535 can recognize the shape as the animal, and
`
`provide a suitable image and sounds on screen 540.
`
`[00074]
`
`Alternatively, and more simply, a target, or targets on the back of the animal can
`
`be used such as triangle 550 on the back of lion 511. In either case the camera can solve for the
`
`3D, and even 5 or 6D position and orientation of the animal object, and cause it to move
`
`accordingly on the screen as the child maneuvers it. The child can hold two animals, one in each
`
`hand and they can each be detected, even with a single camera, and be programmed in software
`
`to interact as the child wishes ( or as he learns the program).
`
`[00075]
`
`This is clearly for very young children of two or three years of age. The toys have
`
`to be large so they can't be swallowed.
`
`[00076]
`
`With the invention in this manner, one can make a toy of virtually anything, for
`
`example a block. Just hold this block up, teach the computer/camera system the object and play
`
`using any program you might want to represent it and its actions. To make this block known to
`
`the system, the shape of the block, the color of the block or some code on the block can be
`
`determined. Any of those items could tell the camera which block it was, and most could give
`
`position and orientation if known.
`
`-13-
`
`Page 21 of 195
`
`
`
`[00077]
`
`At that point, an image is called up from the computer representing that particular
`
`animal or whatever else the block is supposed to represent. Of course this can be changed in the
`
`computer to be a variety of things if this is something that is acceptable to the child. It could
`
`certainly be changed in size such as a small lion could grow into a large lion. The child could
`
`probably absorb that more than a lion changing into a giraffe for example since the block
`
`wouldn't correspond to that. The child can program or teach the system any of his blocks to be
`
`the animal he wants and that might be fun.
`
`[00078]
`
`For example, he or the child's parent could program a square to be a giraffe where
`
`as a triangle would be a lion. Maybe this could be an interesting way to get the child to learn his
`
`geometric shapes!
`
`[00079]
`
`Now the basic block held up in front of the camera system could be looked at just
`
`for what it is. As the child may move the thing toward or away from the camera system, one may
`
`get a rough sense of depth from the change in shape of the object. However this is not so easy as
`
`the object changes in shape due to any sort of rotations.
`
`[00080]
`
`Particularly interesting then is to also sense the rotations if the object so that the
`
`animal can actually move realistically in 3 Dimensions on the screen. And perhaps having the
`
`de-tuning of the shape of the movement so that the child's relatively jerky movements would not
`
`appear jerky on the screen or would not look so accentuated. Conversely of course, you can go
`
`the other way and accentuate the motions.
`
`[00081]
`
`This can, for example, be done with a line target around the edge of the object is
`
`often useful for providing position or orientation information to the TV camera based analysis
`
`software, and in making the object easier to see in reflective illumination.
`
`-14-
`
`Page 22 of 195
`
`
`
`[00082]
`
`Aid to speech recognition
`
`[00083]
`
`The previous co-pending application entitled "Useful man machine interfaces and
`
`applications" referenced above, discussed the use of persons movements or positions to aid in
`
`recognizing the voice spoken by the person.
`
`[00084]
`
`In one instance, this can be achieved by simply using ones hand to indicate to the
`
`camera system of the computer that the voice recognition should start (or stop, or any other
`
`function, such as a paragraph or sentence end, etc.).
`
`[00085]
`
`Another example is to use the camera system of the invention to determine the
`
`location of the persons head (or other part), from which one can instruct a computer to
`
`preferentially evaluate the sound field in phase and amplitude of two or more spaced
`
`microphones to listen from that location - thus aiding the pickup of speech - which often times
`
`is not able to be heard well enough for computer based automatic speech recognition to occur.
`
`[00086]
`
`Digital interactive TV
`
`[00087]
`
`As you watch TV, data can be taken from the camera system of the invention and
`
`transmitted back to the source of programming. This could include voting on a given proposition
`
`by raising your hand for example, with your hand indication tran