I 1111111111111111 11111 111111111111111 11111 1111111111 111111111111111 11111111
`
`US006750848Bl
`
`(12)United States Patent
`
`
`Pryor
`
`(10)Patent No.:US 6,750,848 Bl
`
`Jun.15,2004
`(45)Date of Patent:
`
`(54)MORE USEFUL MAN MACHINE
`
`INTERFACES AND APPLICATIONS
`
`09/433,297
`(21)Appl. No.:
`
`
`
`(22)Filed: Nov. 3, 1999
`
`Related U.S. Application Data
`
`
`
`
`
`(60)Provisional application No. 60/107,652, filed on Nov. 9,
`1998.
`
`5,267,004 A 11/1993 Mills
`
`
`
`5,297,061 A 3/1994 Dementhon et al.
`
`5,388,059 A 2/1995 DeMenthon
`5,454,043 A * 9/1995 Freeman ..................... 382/168
`
`
`
`
`Timothy R. Pryor,
`(76)Inventor:
`516 Old Tecumseh
`
`5,478,423 A 12/1995 Sassa et al.
`
`Road, Tecumseh, Ontario (CA), N8N
`
`
`
`5,528,263 A * 6/1996 Platzker et al. ............. 345/156
`3S8
`
`5,534,062 A 7/1996 Dawson et al.
`
`
`
`5,581,276 A 12/1996 Cipolla et al.
`( *) Notice: Subject to any disclaimer, the term of this
`
`
`
`
`
`5,594,540 A 1/1997 Higaya et al.
`
`
`patent is extended or adjusted under 35
`
`5,616,078 A 4/1997 Oh
`U.S.C. 154(b) by O days.
`
`
`
`5,709,423 A 1/1998 Romero
`5,767,842 A * 6/1998 Korth ......................... 345/168
`
`
`
`
`
`5,808,672 A * 9/1998 Wakabayashi et al. ...... 348/552
`
`
`
`5,864,334 A * 1/1999 Sellers ....................... 345/156
`
`
`5,936,610 A * 8/1999 Endo .......................... 345/157
`
`
`
`5,936,615 A * 8/1999 Waters ....................... 345/156
`
`6,043,805 A * 3/2000 Hsieh ......................... 345/158
`
`
`
`6,160,899 A * 12/2000 Lee et al. ................... 345/863
`
`
`
`6,191,773 Bl * 2/2001 Maruno et al. ............. 345/158
`(51)Int. Cl.7 .................................................. G09G 5/00
`
`
`6,252,598 Bl * 6/2001 Segen ........................ 345/156
`
`
`6,265,993 Bl * 7/2001 Johnson ...................... 345/156
`
`(52)U.S. Cl. ........................ 345/168;
`
`345/719; 345/863
`
`
`
`
`6,346,929 Bl * 2/2002 Fukushima et al. ......... 345/156
`(58) Field of Search
`
`
`................................. 345/156, 157,
`
`345/158, 168, 700, 719, 863; 340/407.2;
`* cited by examiner
`
`341/22; 400/719
`Nguyen
`Primary Examiner----Chanh
`
`A. Bell
`
`Assistant Examiner-Paul
`
`
`
`& Harbison PLLC;(74)Attorney, Agent, or Firm-Stites
`Douglas E. Jackson
`
`5,617,312 A * 4/1997 Iura et al. ................... 345/157
`
`(56)
`
`
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`(57)
`
`ABSTRACT
`
`3,718,116 A 2/1973 Thettu
`
`
`
`
`3,831,553 A 8/1974 Thettu
`
`3,846,826 A 11/1974 Mueller
`The invention is a continuation of earlier applications aimed
`
`
`
`
`
`4,309,957 A 1/1982 Swift
`
`
`
`at providing affordable methods and apparatus for inputting
`4,484,179 A 11/1984 Kasday
`
`
`
`position, attitude( orientation) or other object characteristic
`
`
`4,542,375 A 9/1985 Alles et al.
`
`
`
`
`data to computers for the purpose of controlling the display
`
`
`
`4,629,319 A 12/1986 Clarke et al.
`
`
`
`
`thereof. Preferred embodiments utilize electro-optical sen­
`
`
`4,668,537 A 5/1987 Matsuyama et al.
`
`
`4,686,374 A 8/1987 Liptay-Wagner et al.
`
`
`
`sors such as TV cameras, to input data from objects and
`
`
`
`4,891,772 A 1/1990 Case et al.
`
`
`fingers, and/or other body parts of the user, to laptop, hand
`
`
`
`4,908,670 A 3/1990 Ndebi
`
`held and other computers used for Computer Aided learning,
`4,988,981 A
`
`1/1991 Zimmerman et al.
`
`Gaming, 3D graphics, internet commerce and other appli­
`
`5,045,843 A 9/1991 Hansen
`cations.
`
`
`
`5,168,531 A * 12/1992 Sigel .......................... 345/157
`
`
`
`5,227,985 A 7/1993 DeMenthon
`
`8/1993 5,232,499 A Kato et al.
`
`
`
`
`22 Claims, 7 Drawing Sheets
`
`IPR2021-00922
`Apple EX1003 Page 1
`
`

`

`U.S. Patent Jun.15,2004
`
`
`Sheet 1 of 7 US 6,750,848 Bl
`
`IPR2021-00922
`Apple EX1003 Page 2
`
`

`

`
`U.S. Patent Jun.15,2004
`Sheet 2 of 7
`
`US 6,750,848 Bl
`
`0
`0
`
`.....
`
`IPR2021-00922
`Apple EX1003 Page 3
`
`

`

`
`U.S. Patent Jun.15,2004
`
`Sheet 3 of 7 US 6,750,848 Bl
`
`IPR2021-00922
`Apple EX1003 Page 4
`
`

`

`U.S. Patent
`
`Jun.15,2004
`Sheet 4 of 7
`
`US 6,750,848 Bl
`
`0
`I!)
`-::I"
`
`0
`C")
`-::I"
`
`0
`0
`-::I"
`
`::J
`0..
`0
`
`co
`0
`-::I"
`
`IPR2021-00922
`Apple EX1003 Page 5
`
`

`

`
`U.S. Patent
`Jun.15,2004
`Sheet 5 of 7
`
`US 6,750,848 Bl
`
`0
`co LO
`
`0
`...... -.
`LO : ;-.
`' '•
`.... :' ,' .... ;
`, ,
`, ,
`
`,, ... ..,1'11( .. ,,, ,'
`,
`, .....
`,' .... ,
`
`.,._
`
`I
`
`,
`
`0
`0
`LO
`
`LO
`0
`LO
`
`IPR2021-00922
`Apple EX1003 Page 6
`
`

`

`
`U.S. Patent
`Jun.15,2004
`Sheet 6 of 7
`
`US 6,750,848 Bl
`
`co
`
`LC)
`0)
`
`I'--L{)
`
`
`0)
`
`0
`0)
`(J)
`
`�¾� \
`
`\
`\
`\
`
`co
`L{)
`0)
`
`� �_J:J
`
`LC) a.. a..
`O)U)(.)
`
`0
`
`\
`\
`\
`\
`\
`\
`\
`\
`\
`
`(0
`•
`(!)
`m Li:
`\� \
`0
`m'
`
`\
`\
`\
`
`0�
`
`m
`
`�
`
`0
`
`co
`0
`m
`
`IPR2021-00922
`Apple EX1003 Page 7
`
`

`

`
`U.S. Patent
`Jun.15,2004
`
`Sheet 7 of 7 US 6,750,848 Bl
`
`1001
`
`1014
`
`.. - .. ..
`,,, ...... : ......
`, .... , ,
`.... , ,
`
`,l.,
`
`I " ,.
`
`"r- .. ,
`
`I ,t
`
`1030
`
`C
`p
`
`.... )>\. __ 1020
`
`1000
`
`1002
`
`FIG. 7A
`
`1070
`
`1065
`
`1075
`
`1056
`
`1060
`
`INTERNET
`
`INPUT
`WOMAN'S
`MEASURE-����
`MENTS
`) ,,__
`(
`
`CPU 1055
`
`1050
`
`INTERNET
`REMOTE
`
`____
`
`FIG. 7B
`
`IPR2021-00922
`Apple EX1003 Page 8
`
`

`

`
`
`CROSS REFERENCES TO RELATED
`
`
`
`FIG. 1
`
`
`
`Microfiche Appendix
`
`not applicable
`
`
`
`
`
`US 6,750,848 Bl
`
`1
`
`MORE USEFUL MAN MACHINE
`
`keyboard surface to observe objects such as fingers and
`
`
`
`
`
`filed Nov. 9, 1998
`
`2
`
`
`
`
`
`
`hands overhead of the keyboard
`INTERFACES AND APPLICATIONS
`FIG. 2 illustrates another keyboard embodiment using
`
`
`
`
`This application Provisional Application No. 60/107,652
`
`
`
`special datums or light sources such as LEDs
`5
`
`
`
`FIG. 3 illustrates a further finger detection system for
`
`
`laptop or other computer input
`
`APPLICATIONS BY THE INVENTOR
`FIG. 4 illustrates learning, amusement, monitoring, and
`
`
`
`
`
`1.Man Machine Interfaces (Ser. No. 08/290,516)
`
`
`
`
`
`diagnostic methods and devices for the crib, playpen and the
`
`2.Touch TV and other Man Machine Interfaces (Ser. No.
`like
`10
`08/496,908)
`
`
`
`
`FIG. 5 illustrates a puzzle toy for young children having
`
`
`
`3.Systems for Occupant Position Sensing, Ser. No. 08/968,
`
`
`
`
`cut out wood characters according to the invention
`114
`
`
`
`FIG. 6 illustrates an improved handheld computer
`
`
`
`4.Target holes and corners U.S. Ser. Nos. 08/203,603, and
`
`
`embodiment of the invention, in which the camera or
`08/468,358
`
`
`
`15 cameras may be used to look at objects, screens and the like
`
`5.Useful Man Machine interfaces and applications, Prov.
`
`as well as look at the user along the lines of FIG. 1.
`
`
`Appl. filed Aug. 21, 1998
`
`
`
`
`FIG. 7 illustrates new methods for internet commerce and
`6. Vision Target based assembly, U.S. Ser. Nos. 08/469,429,
`
`
`
`
`
`
`other activities involving remote operation with 3D virtual
`
`08/469,907, 08/470,325, 08/466,294
`
`objects display.
`
`
`
`
`7.Stereo camera Based input to Computer systems (new
`20
`
`provisional application)
`8.Picture Taking method and apparatus(new provisional
`
`
`A laptop ( or other)computer keyboard based embodiment
`
`
`
`application)
`
`
`is shown in FIG. 1. In this case, a stereo pair of cameras 100
`
`
`9.Methods and Apparatus for Man Machine Interfaces and
`
`
`
`and 101 located on each side of the keyboard are used,
`
`
`
`Related Activity (new provisional application
`
`
`
`desirably having cover windows 103 and 104 mounted flush
`25
`
`
`10.Camera Based Man-Machine Interfaces new Provisional
`
`
`
`with the keyboard surface 102. The cameras are preferably
`
`
`Patent Application, filed July 1999
`
`
`
`
`
`pointed obliquely inward at angles cp toward the center of the
`
`The copies of the disclosure of the above referenced
`
`
`
`desired work volume 170 above the keyboard. In the case of
`applications are incorporated herein by reference.
`
`
`
`
`
`
`
`cameras mounted at the rear of the keyboard (toward the
`Federally Sponsored R and D Statement
`
`
`
`
`
`30 display screen), these cameras are also inclined to point
`
`toward the user at an angle as well.
`not applicable
`Alternate camera locations may be used such as the
`
`
`
`
`
`
`positions of cameras 105 and 106, on upper corners of
`
`
`
`
`
`screen housing 107 looking down at the top of the fingers ( or
`
`
`
`35 hands, or objects in hand or in front of the cameras), or of
`BACKGROUND OF THE INVENTION
`
`cameras 108 and 109 shown.
`1.Field of the Invention
`Alternate camera locations may be used such as positions
`
`
`
`The invention relates to simple input devices for
`
`
`
`
`
`105 and 106, on upper corners of screen housing 107
`
`
`
`
`computers, particularly, but not necessarily, intended for use
`
`
`
`looking down at the top of the fingers ( or hands, or objects
`
`
`
`
`with 3-D graphically intensive activities, and operating by
`40
`
`
`
`in hand or in front of the cameras), or 108 and 109 shown.
`
`
`
`
`optically sensing object or human positions and/or orienta­
`
`
`
`
`One of the referenced embodiments of the invention is to
`
`
`
`
`tions. The invention in many preferred embodiments, uses
`
`
`
`determine the pointing direction vector 160 of the users
`
`
`
`
`real time stereo photogrammetry using single or multiple TV
`
`
`
`finger (for example pointing at an object displayed on screen
`
`
`
`cameras whose output is analyzed and used as input to a
`
`
`
`107), or the position and orientation of an object held by the
`
`
`personal computer, typically to gather data concerning the
`45
`
`
`user. Alternatively, finger position data can be used to
`
`
`3D location of parts of, or objects held by, a person or
`
`
`
`
`determine gestures such as pinch or grip, and other examples
`persons.
`
`
`
`
`of relative juxtaposition of objects with respect to each other,
`
`
`This continuation application seeks to provide further
`as has been described in co-pending referenced applications.
`
`
`
`
`detail on useful embodiments for computing. One embodi­
`
`
`Positioning of an object or portions (such as hands or fingers
`
`
`
`ment is a keyboard for a laptop computer ( or stand alone
`50
`
`of a doll) is also of use, though more for use with larger
`TV
`
`
`
`keyboard for any computer) that incorporates digital
`keyboards and displays.
`
`
`
`cameras to look at points on, typically, the hand or the finger,
`
`
`In one embodiment, shown in FIG. 2, cameras such as
`
`
`
`or objects held in the hand of the user, which are used to
`
`
`
`100/101 are used to simply look at the tip of a finger 201 ( or
`
`
`
`
`input data to the computer. It may also or alternatively, look
`
`
`as a ring 208 on the 55 thumb) of the user, or an object such
`
`at the head of the user as well.
`
`
`
`
`finger. Light from below, such as provided by single central
`
`Both hands or multiple fingers of each hand, or an object
`
`light 122 can be used to illuminate the finger that typically
`
`in one hand and fingers of the other can be simultaneously
`
`looks bright under such illumination.
`
`
`
`observed, as can alternate arrangements as desired.
`
`
`It is also noted that the illumination is directed or con-
`
`
`2.Description of Related Art
`
`
`
`
`60 centrated in an area where the finger is typically located such
`
`
`My referenced copending applications incorporated
`
`as in work volume 170. If the light is of sufficient spectral
`
`
`
`herein by reference discuss many prior art references in
`
`
`
`content, the natural flesh tone of the finger can be
`
`
`
`various pertinent fields, which form a background for this
`
`
`observed-and recognized-by use of the color TV cameras
`invention.
`100/101.
`DESCRIPTION OF FIGURES
`
`65 As is typically the case, the region of the overlapping
`
`
`
`
`
`cameras viewing area is relatively isolated to the overlap­
`
`
`
`FIG. 1 illustrates a lap top or other computer keyboard
`
`
`ping volumetric zone of their fields 170 shown due to focal
`
`
`
`
`with cameras according to the invention located on the
`
`IPR2021-00922
`Apple EX1003 Page 9
`
`

`

`
`
`US 6,750,848 Bl
`
`FIG. 3
`
`4
`3
`lengths of their lenses and the angulation of the camera axes
`
`
`
`
`
`equipped with a sufficiently coaxial light source, typically
`
`
`
`
`
`
`
`
`with respect to each other. This restricted overlap zone, helps
`
`one or more LEDs such as 210 and 211), can be used to solve
`
`
`
`
`mitigate against unwanted matches in the two images due to
`
`
`
`
`
`easily using the line image pairs with the stereo cameras for
`
`
`
`
`information generated outside the zone of overlap. Thus
`
`
`
`the pointing direction of the finger that is often a desired
`
`
`
`there are no significant image matches found of other objects
`
`
`
`
`5 result. The line, in the stereo pair of images provides the 3D
`
`
`
`
`in the room, since the only flesh toned object in the zone is
`
`
`
`
`
`pointing direction of the finger, for example pointing at an
`
`
`
`
`typically the finger or fingers of the user. Or alternatively, for
`
`
`
`
`object displayed on the screen 140 of the laptop computer
`
`
`
`example, the users hand or hands. Similarly objects or
`138
`
`
`
`
`targets thereon can be distinguished by special colors or
`shapes.
`10
`It is also possible to have light sources on the finger that
`
`
`
`
`
`If desired, or required, Motion of the fingers can be also
`
`
`
`can be utilized such as the 2 LED light sources shown in
`
`
`
`
`used to further distinguish their presence vis a vis any static
`
`FIG. 3. This can be used with either TV camera type sensors
`
`
`
`background. If for example by subtraction of successive
`
`
`or with PSD type analog image position sensors as disclosed
`
`
`
`
`camera frames, the image of a particular object is deter­
`
`15 in references incorporated.
`
`
`
`mined to have moved it is determined that this is likely the
`In particular the ring mounted LED light sources 301 and
`
`
`
`
`
`
`
`
`object of potential interest which can be further analyzed
`
`
`302 can be modulated at different frequencies that can be
`
`
`
`
`directly to determine if is the object of interest.
`
`
`
`individually discerned by sensors imaging the sources on to
`
`
`In case of obscuration of the fingers or objects in the hand,
`
`
`
`
`a respective PSD detector. Alternatively, the sources can
`
`
`
`cameras in additional locations such as those mentioned
`20
`
`
`simply be turned on and off at different times such that the
`
`
`above, can be used to solve for position if the view of one
`
`
`position of each point can be independently found allowing
`
`or more cameras is obscured.
`
`
`
`the pointing direction to be calculated from the LED point
`
`
`
`
`
`The use of cameras mounted on both the screen and the
`
`
`
`
`data gathered by the stereo pair of PSD based sensors.
`
`
`keyboard allows one to deal with obscurations that may
`
`
`
`
`The "natural interface keyboard" here described can have
`
`
`
`occur and certain objects may or may not be advantageously
`
`
`
`
`camera or other sensors located at the rear looking obliquely
`25
`
`delineated in one view or the other.
`
`
`
`
`outward toward the front as well as inward so as to have their
`
`
`
`
`In addition, it may be, in many cases, desirable to have a
`
`
`
`
`working volume overlap in the middle of the keyboard such
`datum on the top of the finger as opposed to the bottom
`
`
`
`
`
`
`as the nearly full volume over the keyboard area is accom-
`
`because on the bottom, it can get in the way of certain
`
`
`
`
`modated.
`
`
`
`
`activities. In this case the sensors are required on the screen
`30
`Clearly larger keyboards can have a larger working vol­
`
`
`
`
`
`
`
`
`looking downward or in some other location such as off the
`
`
`ume than one might have on a laptop. The pair of sensors
`
`
`
`computer entirely and located overhead has been noted in
`
`
`
`used can be augmented with other sensors mounted on the
`
`previous application.
`
`
`
`screen housing. It is noted that the linked dimension afforded
`
`
`
`To determine finger location, a front end processor like
`
`
`
`
`
`
`for calibration between the sensors located on the screen and
`that described in the target holes and corners copending
`
`
`
`
`
`
`
`
`35 those on the keyboard is provided by the laptop unitary
`
`
`
`application reference incorporated U.S. Ser. Nos. 08/203,
`construction.
`
`
`603, and 08/468,358 can be used, to also allow the finger
`One can use angle sensing means such as a rotary encoder
`
`
`
`shape as well as color to be detected.
`
`
`
`
`for the lap top screen tilt. Alternatively, cameras located on
`
`
`
`
`Finger gestures comprising a sequence of finger move­
`
`
`40 the screen can be used to image reference points on the
`
`
`ments can also be detected, by analyzing sequential image
`
`
`
`
`
`keyboard as reference points to achieve this. This allows the
`
`
`sets such at the motion of the finger, or one finger with
`
`
`
`
`calibration of the sensors mounted fixedly with respect to the
`
`
`
`respect to another such as in pinching something can be
`
`
`
`
`screen with respect to the sensors and keyboard space below.
`
`
`
`determined. Cameras 100 and 101 have been shown at the
`
`
`
`It also allows one to use stereo pairs of sensors that are not
`
`
`
`
`
`rear of the keyboard near the screen or at the front. They may
`
`
`in the horizontal direction (such as 101/102) but could for
`
`
`
`mounted in the middle of the keyboard or any other advan­
`45
`
`
`example be a camera sensor such as 100 on the keyboard
`
`tageous location.
`
`
`
`coupled with one on the screen, such as 106
`
`
`The cameras can also see ones fingers directly, to allow
`Knowing the pointing angles of the two cameras with
`
`
`
`
`
`
`typing as now, but without the physical keys. One can type
`
`
`
`respect to one another allows one to solve for the 3 d location
`
`in space above the plane of the keyboard ( or in this case
`
`
`
`
`image positions of the object 50 of objects from the matching
`
`
`plane of the cameras), this is useful for those applications
`
`
`in the respective camera fields.
`
`
`where the keyboard of conventional style is too big (e.g. the
`As noted previously, it is also of interest to locate a line
`
`
`
`
`
`
`hand held computer of FIG. 6).
`
`
`
`or cylinder type target on the finger between the first and
`
`
`
`second joints. This allows one to use the fingertip for the
`
`
`55 keyboard activity but by raising the finger up, it can be used
`
`
`
`
`
`
`as a line target capable of solving for the pointed direction
`
`
`
`It is also desirable for fast reliable operation to use
`for example.
`
`
`
`
`retro-reflective materials and other materials to augment the
`
`
`
`contrast of objects used in the application. For example, A
`Alternatively one can use two point targets on the finger
`
`
`
`
`line target such as 200 can be worn on a finger 201, and
`
`
`
`
`such as either retro-reflective datums, colored datums such
`
`
`
`advantageously can be located if desired between two joints
`
`
`
`
`60 as rings or LED light sources that can also be used with PSD
`
`
`of the finger as shown. This allows the tip of the finger to be
`
`
`
`detectors which has also been noted in FIG. 2.
`
`
`
`
`used to type on the keyboard without feeling unusual-the
`
`
`
`
`When using the cameras located for the purpose of stereo
`
`
`
`case perhaps with target material on tip of the finger.
`
`
`determination of the position of the fingers from their flesh
`
`
`tone images it is useful to follow the preprocessing capable
`
`The line image detected by the camera can be provided
`
`
`
`65 of processing data obtained from the cameras in order to
`
`
`
`also by a cylinder such as retroreflective cylinder 208 worn
`
`
`
`look for the finger. This can be done on both color basis and
`
`
`
`
`on the finger 201 which effectively becomes a line image in
`
`on the basis of shape as well as motion.
`
`
`the field of view of each camera, ( assuming each camera is
`
`FIG. 2
`
`IPR2021-00922
`Apple EX1003 Page 10
`
`

`

`
`
`Baby Learning and Monitoring System
`
`
`
`
`
`US 6,750,848 Bl
`
`FIG. 4
`
`6
`5
`If the mother knows what the baby is after, she can talk
`
`
`
`In this invention, I have shown the use of not only
`
`
`
`
`to baby or say something, or show something that the baby
`
`cameras located on a screen looking downward or outward
`
`
`
`
`
`recognizes such as a doll. After a while, looking at this live
`
`from the screen, but also cameras that can be used instead of
`
`
`
`
`
`one can then move to talking to the baby from some
`
`or in combination with those on the screen placed essentially
`
`
`on the member on which the keyboard is incorporated. This
`
`
`5 prerecorded data.
`
`
`
`
`allows essentially the keyboard to mounted cameras which
`What other things might we suppose? The baby for
`
`
`
`
`
`
`are preferably mounted flush with the keyboard surface to be
`
`
`
`example knows to puts its hand on the mother's cheek to
`
`
`unobtrusive, and yet visually be able to see the users fingers,
`
`
`cause the mother to turn to it. The baby also learns some
`
`
`
`hands or objects held by the user and in some cases, the face
`
`
`
`other reflexes when it is very young that it forgets when it
`
`of the user.
`
`
`
`10 gets older. Many of these reflexes are hand movements, and
`
`
`This arrangement is also useful for 3D displays, for
`
`
`are important in communicating with the remote TV based
`
`
`
`
`
`example where special synchronized glasses ( eg the "Crystal
`
`
`
`
`mother representation, whether real via telepresense or from
`
`
`Eyes" brand often used with Silicon Graphics work stations)
`
`CD Rom or DVD disk (or other media, including informa­
`
`
`
`are used to alternatively present right and left images to each
`
`tion transmitted to the computer from afar) and for the
`
`15 learning of the baby's actions.
`
`
`
`eye. In this case the object may appear to be actually in the
`
`
`workspace 170 above the keyboard, and it may be manipu­
`Certainly just from the making the baby feel good point­
`
`
`
`
`
`
`
`lated by virtually grasping (pushing, pulling, etc) it, as has
`
`
`
`
`of-view, it would seem like certain motherly ( or fatherly, etc)
`
`
`been described in co-pending applications
`
`
`
`
`responses to certain baby actions i n the form of words and
`
`
`
`
`images would be useful. This stops short of physical holding
`
`
`
`20 of the baby which is often needed, but could act as a stop gap
`
`
`
`to allow the parents to get another hours sleep for example.
`
`
`
`As far as the baby touching things, ive discussed in other
`A baby's reaction to the mother (or father) and the
`
`
`
`
`
`
`
`applications methods for relalistic touch combined with
`
`
`
`
`mother's analysis of the baby's reaction is very important.
`
`
`
`images. This leads to a new form of touching crib mobiles
`
`
`
`There are many gestures of babies apparently indicated in
`25
`
`that could contain video imaged and or be imaged
`
`
`
`child psychology as being quite indicative of various needs,
`
`
`
`
`themselves-plus if desired, touched in ways that would be
`
`
`
`wants, or feelings and emotions, etc. These gestures are
`
`
`far beyond any response that you could get from a normal
`
`typically made with the baby's hands.
`mobile.
`
`
`
`Today this is done and learned entirely by the mother
`For example, let us say there is a targeted (or otherwise
`
`
`
`being with the baby. However with a Electro-optical sensor
`30
`
`
`
`TV observable) mobile 450 in the crib above the baby. Baby
`
`
`
`based computer system, such as that described in copending
`
`
`
`
`
`reaches up and touches a piece of the mobile which is sensed
`
`
`
`applications, located proximate to or even in the crib(for
`
`
`
`by the tv camera system (either from the baby's hand
`
`
`
`
`example), one can have the child's reactions recorded, not
`
`
`
`
`
`position, the mobile movement, or both, and a certain sound
`
`just in the sense of a video tape which would be too long and
`
`
`
`is called up by the computer, a musical note for example.
`
`involved for most to use, but also in terms of the actual
`35
`
`
`
`
`Another piece of the mobile and another musical note. The
`
`
`
`
`motions which could be computer recorded and analyzed
`
`
`
`
`mobile becomes a musical instrument for the baby that could
`
`
`
`also with the help of the mother as to what the baby's
`
`
`
`
`play either notes or chords or complete passages, or any
`
`
`
`responses were. And such motions, combined with other
`
`other desired programmed function.
`
`
`
`audio and visual data can be very important to the baby's
`
`
`health, safety, and learning.
`The baby can also signal things. The baby can signal using
`
`
`
`40
`
`
`
`Consider for example crib 400 with computer 408 having
`
`
`agitated movements would often mean that it's unhappy.
`
`
`LCD monitor 410 and speaker 411 and camera system
`
`This could be interpreted using learned movement signa­
`
`(single or stereo) 420 as shown, able to amuse or inform
`
`
`
`
`tures and artificial intelligence as needed by the computer to
`
`baby 430, while at the same time recording (both visually,
`
`call for mother even if the baby wasn't crying. If the baby
`
`
`aurally, and in movement detected position data concerning
`
`
`
`cries, that can be picked up by microphone 440, recognized
`45
`
`
`
`parts of his body or objects such as rattles in his hand) his
`
`
`using a voice recognition system along the lines of that used
`
`
`
`
`
`responses for any or all of the purposes of diagnosis of his
`
`
`in IBM Via Voice commercial product for example. And
`
`
`
`state of being, remote transmission of his state, cues to
`
`
`even the degree of crying can be analyzed to determine
`
`
`
`various programs or images to display to him or broadcast
`
`appropriate action.
`to others, or the like.
`The computer could also be used to transmit information
`
`
`50
`
`
`
`
`For one example, baby's motions could be used to signal
`
`
`
`of this sort via the internet email to the mother who could
`
`
`
`
`
`
`a response from the TV either in the absence of the mother
`
`even be at work. And until help arrives in the form of mother
`
`
`or with the mother watching on a remote channel. This can
`
`
`
`
`intervention or whatever, the computer could access a pro­
`
`
`even be over the Internet if the mother is at work.
`
`
`
`
`gram that could display on a screen for the baby things that
`
`
`55 the baby likes and could try to soothe the baby through either
`
`
`
`For example, a comforting message could come up on the
`
`
`
`
`
`TV from the mother that could be prerecorded (or alterna­images of familiar things, music or whatever. This could be
`
`
`
`
`
`
`
`
`tively could actually be live with TV cameras in the mothers useful at night when parents need sleep, and any thing that
`
`
`
`or fathers work place for example
`
`
`
`would make the baby on a computer used by the feel more comfortable would help the
`
`parent) to tell the baby something reassuring or comfort the parents.
`
`
`
`
`
`
`
`baby or whatever. Indeed the parent can be monitored using 60
`It could also be used to allow the baby to input to the
`
`
`
`
`
`
`the invention and indicate something back or even control a
`
`
`
`
`device. For example, if the baby was hungry, a picture of the
`
`
`
`
`teleoperater robotic device to give a small child something
`
`
`
`bottle could be brought up on the screen. The baby then
`
`
`
`to eat or drink for example. The same applies to a disabled
`
`
`
`could yell for the bottle. Or if the baby needed his diaper
`person.
`
`
`
`
`
`changed, perhaps something reminiscent of that. If the baby
`
`
`
`
`
`
`
`
`If the father or mother came up on the screen, the baby 65 reacts to such suggestions of his problem, This gives a lot
`
`
`
`
`could wave at it, move it's head or "talk" to it but the hand more intelligence as to why he is crying and while mothers
`
`gestures may be the most important.
`
`
`can generally tell right away, not every one else can. In other
`
`IPR2021-00922
`Apple EX1003 Page 11
`
`

`

`
`
`US 6,750,848 Bl
`
`8
`7
`
`
`
`
`
`
`
`words, this is pretty neat for babysitters and other members Particularly interesting then is to also sense the rotations
`
`
`
`
`
`
`
`
`of the household so they can act more intelligently on the if the object so that the animal can actually move realisti­
`
`signals the baby is providing.
`
`
`
`
`
`cally in 3 Dimensions on the screen. And perhaps having the
`
`
`
`de-tuning of the shape of the movement so that the child's
`
`
`Besides in the crib, the system as described can be used
`
`
`would not appear jerky on the 5 relatively jerky movements
`
`in conjunction with a playpen, hi chair or other place of baby
`
`
`
`
`
`
`
`screen or would not look so accentuated. Conversely of
`activity.
`
`
`
`course, you can go the other way and accentuate the
`
`
`
`
`As the child gets older, the invention can further be used
`motions.
`also with more advanced activity with toys, and to take data
`
`
`
`This can, for example, be done with a line target around
`
`from toy positions as well. For example, blocks, dolls, little
`
`
`
`cars, and moving toys even such as Trikes, Scooters, driv-10
`
`
`
`
`the edge of the object is often useful for providing position
`
`
`
`
`
`or orientation information to the tv camera based analysis
`
`
`able toy cars and bikes with training wheels
`
`
`
`software, and in making the object easier to see in reflective
`
`
`
`
`The following figure illustrates the ability of the invention
`illumination.
`
`
`
`to learn, and thus to assist in the creation of toys and other
`things.
`
`Aid to Speech Recognition
`
`FIG. 5
`
`15
`The previous copending application entitled "Useful man
`
`
`
`
`
`
`
`
`machine interfaces and applications" referenced above, dis­
`
`
`
`
`
`cussed the use of persons movements or positions to aid in
`Disclosed in FIG. 5 is a puzzle toy 500 where woodcut
`
`
`
`
`recognizing the voice spoken by the person.
`
`animals such as bear 505 and lion 510 are pulled out with 20
`
`
`
`
`
`In one instance, this can be achieved by simply using ones
`handle such as 511. The child can show the animal to the
`
`
`
`
`hand to indicate to the camera system of the computer that
`
`
`camera and a computer 530 with TV camera (or cameras)
`
`
`the voice recognition should start ( or stop, or any other
`
`
`535 can recognize the shape as the animal, and provide a
`
`
`
`function, such as a paragraph or sentance end etc).
`
`
`suitable image and sounds on screen 540.
`
`
`Another example is to use the camera system of the
`25
`
`
`
`
`
`Alternatively, and more simply, a target, or targets on the
`
`
`
`
`invention to determine the location of the persons head ( or
`
`back of the animal can be used such as triangle 550 on the
`
`
`
`
`
`other part), from which one can instruct a computer to
`back of lion 511. In either case the camera can solve for the
`
`
`
`
`preferentially evaluate the sound field in phase and ampli­
`
`3D, and even 5 or 6D position and orientation of the animal
`
`
`
`
`
`tude of two or more spaced microphones to listen from that
`
`object, and cause it to move accordingly on the screen, as the
`
`
`
`
`30 location-thus aiding the pickup of speech, which often
`
`child maneuvers it. The child can hold two animals, one in
`
`
`
`
`times is not able to be heard well enough for computer based
`
`
`each hand and they can each be detected, even with a single
`
`
`
`automatic speech recognition to occur.
`
`camera, and be programmed in software to interact as the
`
`
`
`
`child wishes.(or as he learns the program)
`
`
`This is clearly for very young children of two or three
`As you watch tv, data can be taken from the camera
`
`
`35
`
`years of age. The toys have to be large so they can't be
`
`
`
`system of the invention and transmitted back to the source
`swallowed.
`
`
`of programming. This could include voting on a given
`
`
`
`
`proposition by raising your hand for example, with your
`
`
`
`With the invention in this manner, one can make a toy of
`
`
`
`hand indication transmitted. Or you could hold up 3 fingers,
`
`virtually anything, for example a block. Just hold this block
`
`
`
`
`
`40 and the count of fingers transmitted. Or in a more extreme
`up, teach the computer/camera system the object and play
`
`
`
`
`
`
`case, your position, or the position of an object or portion
`
`
`using any program you might want to represent it and its
`
`
`
`thereof could be transmitted-for example you could buy a
`
`
`
`
`
`actions. To make this block known to the system, the shape
`
`
`
`coded object, whose code would be transmitted to indicate
`
`
`
`of the block, the color of the block or some code on the block
`
`
`
`
`that you personally (having been preregistered) had trans-
`
`
`can be determined. Any of those items could tell the camera
`
`
`
`mitted a certain packet of data.
`which block it was, and most could give position and
`45
`
`orientation if known.
`
`If the programming source can transmit individually to
`
`
`
`
`you (not possible today, but forecast for the future), then
`
`
`
`At that point, an image is called up from the computer
`
`
`
`much more is possible. The actual image and voice can
`
`
`
`representing that particular animal or whatever else the
`
`
`
`respond using the invention to positions and orientations of
`
`
`block is supposed to represent. Of course this can be
`
`
`
`
`persons or objects in the room-just as in the case of
`
`
`
`
`changed in the computer to be a variety of things if this is 50
`
`
`
`
`prerecorded data, or one to one internet connections. This
`
`
`
`
`something that is acceptable to the child. It could certainly
`
`allows group activity as well.
`
`
`be changed in size such as a small lion could grow into a
`
`
`
`large lion. The child could probably absorb that more than
`
`
`
`
`In the extreme case, full video is transmitted in both
`
`
`
`
`a lion changing into a giraffe for example since the block
`
`
`
`directions and total interaction of users and programming
`
`
`
`
`wouldn't correspond to that. the child can program or teach
`
`
`sources and each other becomes possible.
`55
`
`
`
`the system any of his blocks to be the animal h