United States Patent [19]
`Breed et al.
`
`US005 845000A
`Patent Number:
`Date of Patent:
`
`[11]
`[45]
`
`5,845,000
`*Dec. 1, 1998
`
`[54]
`
`[75]
`
`OPTICAL IDENTIFICATION AND
`MONITORING SYSTEM USING PATTERN
`RECOGNITION FOR USE WITH VEHICLES
`
`Inventors: David S. Breed, Boonton Township,
`N.J.; Wilbur E. DuVall, Kimberling
`City, Mo.; Wendell C. Johnson,
`Torrance, Calif.
`
`Assignee: Automotive Technologies
`International, Inc., Denville, NJ.
`
`Notice:
`
`The term of this patent shall not extend
`beyond the expiration date of Pat. No.
`5,835,613.
`
`Appl. No.: 474,786
`Filed:
`Jun. 7, 1995
`
`Related US. Application Data
`
`Continuation-in-part of Ser. No. 878,571, May 5, 1992,
`abandoned, Ser. No. 40,978, Mar. 31, 1993, abandoned, Ser.
`No. 247,760, May 23, 1994, and Ser. No. 239,978, May 9,
`1994, abandoned.
`
`Int. Cl.6 ..................................................... .. G06K 9/00
`
`US. Cl. ........................................... .. 382/100; 348/143
`
`Field of Search ........................... .. 340/436; 382/104,
`382/103, 291, 100; 280/735; 348/143, 148
`
`[56]
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`350/354
`1/1985 Shah ............ ..
`4,496,222
`..... .. 382/1
`4,625,329 11/1986 Ishikawa et al. .
`364/550
`4,648,052
`3/1987 Friedman et al.
`351/210
`4,720,189
`1/1988 Heynen et al.
`4,768,088
`8/1988 Ando ....................................... .. 358/93
`4,836,670
`6/1989 Hutchinson ........................... .. 351/210
`4,881,270 11/1989 Knecht et al.
`382/191
`4,906,940
`3/1990 Greene et al.
`382/100
`4,950,069
`8/1990 Hutchinson ..... ..
`351/210
`4,966,388 10/1990 Warner et al. ..................... .. 280/730.1
`5,003,166
`3/1991 Girod ................................. .. 250/201.4
`
`FOREIGN PATENT DOCUMENTS
`
`342337 2/1991 Japan.
`94/22692 10/1994 WIPO.
`
`OTHER PUBLICATIONS
`
`“Analysis of Hidden Units in a Layered Network Trained to
`Classify Sonar Targets”, R. Paul Gorman, et al., Neural
`Networks, vol. 1, pp.75—89, 1988.
`Learned Classi?cation of Sonar Targets Using a Massively
`Parallel Network, R. Paul Gorman et al., IEEE Transactions
`on Acoustics, Speech and Signal Processing, vol. 36, No. 7,
`Jul., 1988, pp. 1135—1140.
`“How Airbags Work”, David S. Breed, Presented at the
`Canadian Association of Road Safety Professional, Oct. 19,
`1992—Oct. 20, 1992.
`Derwent Abstract of German Patent Publication No. DE 42
`11 556, Oct. 7, 1993.
`Derwent Abstract of Japanese Patent Application No.
`02—051332, Nov. 13, 1991.
`
`Primary Examiner—Yon J. Couso
`
`[57]
`
`ABSTRACT
`
`Avehicle interior monitoring system to identify, locate and
`monitor occupants, including their parts, and other objects in
`the passenger compartment and objects outside of a motor
`vehicle, such as an automobile or truck, by illuminating the
`contents of the vehicle and objects outside of the vehicle
`with electromagnetic, and speci?cally infrared, radiation
`and using one or more lenses to focus images of the contents
`onto one or more arrays of charge coupled devices (CCD
`arrays). Outputs from the CCD arrays, are analyzed by
`appropriate computational means employing trained pattern
`recognition technologies, to classify, identify or locate the
`contents or external objects. In general, the information
`obtained by the identi?cation and monitoring system is used
`to affect the operation of some other system in the vehicle.
`When system is installed in the passenger compartment of
`an automotive vehicle equipped with an airbag, the system
`determines the position of the vehicle occupant relative to
`the airbag and disables deployment of the airbag if the
`occupant is positioned so that he/she is likely to be injured
`by the deployment of the airbag.
`
`(List continued on next page.)
`
`25 Claims, 12 Drawing Sheets
`
`1
`
`Mercedes-Benz USA, LLC, Petitioner - Ex. 1001
`
`

`

`5,845,000
`Page 2
`
`US. PATENT DOCUMENTS
`
`5,249,027
`
`9/1993 Mathur et a1. ............................ .. 356/1
`
`4/1991 Ando
`5 008 946
`5:026:53 6/1991 SuZukietaL __
`570647274 11/1991 Alten """"""""" ~~
`5,071,160 12/1991 White et a1,
`5,074,583 12/1991 Fujita et a1.
`5,118,134
`6/1992 Manes 9t a1~
`571627861 11/1992 Tambufim 6‘ a1~
`5,181,254
`1/1993 S0hWe1Zer et a1. ..
`5,185,667
`2/1993 Zlmmermann .
`5,193,124
`3/1993 Subbarao ..... ..
`
`3820
`_ 356/316
`" 359/604
`__ 280/735
`. 280/7301
`~- 280/735
`~~~~ ~~ 356/5
`.. 382/100
`.. 358/209
`.. 382/255
`
`9/1993 Taylor ................................... .. 340/435
`5,249,157
`3/1994 Che'n .................................. .. 250/203.4
`5,298,732
`4/1994 Fans .......................................... .. 345/7
`5,305,012
`5/1994 Kajiwara ............................... .. 348/148
`5,309,137
`7/1994 Slotkowski et a1.
`.. 315/159
`5,329,206
`7/1994 Gentry et a1. ......................... .. 280/735
`5,330,226
`8/1994 Abst et a1. ............................ .. 340/903
`5,339,075
`573557118 “V1994 Fukahara '
`“ 348/148
`5 390 136
`2/1995 Wan
`364/754
`’
`’
`_
`
`"" "
`
`"
`
`5,214,744
`5,227,784
`5,235,339
`
`395/11
`5/1993 SchWeiZer et a1.
`340/436
`7/1993 Masamori et al_
`8/1993 Morrison et a1, _____________________ __ 342/159
`
`- 126/661-09
`8/1995 Mlyallma
`5,441,052
`. 260/735
`5,454,591 10/1995 MaZur et a1. ..
`5,537,003
`7/1996 Bechtel et a1. ............................ .. 35/82
`
`2
`
`

`

`US. Patent
`U.S. Patent
`
`Dec. 1,1998
`Dec. 1, 1998
`
`Sheet 1 0f 12
`Sheet 1 0f 12
`
`5,845,000
`5,845,000
`
`w:
`
`me
`
`o:
`
`
`
`m2 /////
`
` 92
`
`I
`
`
`
`E n: FK
`
`m:
`
`2:
`
`N2
`
`3
`
`

`

`US. Patent
`U.S. Patent
`
`Dec. 1,1998
`Dec. 1, 1998
`
`Sheet 2 0f 12
`Sheet 2 0f 12
`
`5,845,000
`5,845,000
`
`:2
`
`m: .GE
`
`mow
`
`o:
`
` 03
`
`v:
`
`m:
`
`4
`
`

`

`US. Patent
`U.S. Patent
`
`Dec. 1,1998
`Dec. 1, 1998
`
`Sheet 3 0f 12
`Sheet 3 0f 12
`
`5,845,000
`5,845,000
`
`#2
`
`2:
`
`UM .UE
`
`m:
`
`5
`
`

`

`US. Patent
`U.S. Patent
`
`Dec. 1,1998
`Dec. 1, 1998
`
`Sheet 4 0f 12
`Sheet 4 0f 12
`
`5,845,000
`5,845,000
`
`
`
`A: .UE
`
`6
`
`

`

`U.S. Patent
`US. Patent
`
`Dec. 1,1998
`Dec. 1, 1998
`
`Sheet 5 0f 12
`Sheet 5 0f 12
`
`5,845,000
`5,845,000
`
`101
`
`FIG.1E
`m: .65
`
`105
`
`110
`
`100
`
`#2
`114
`
`mv—d
`F-It
`
`7
`
`

`

`U.S. Patent
`US. Patent
`
`Dec. 1,1998
`Dec. 1, 1998
`
`Sheet 6 0f 12
`Sheet 6 0f 12
`
`5,845,000
`5,845,000
`
`210
`
`FIG.2
`
`08
`
`8
`
`

`

`U.S. Patent
`
`Dec. 1,1998
`
`Sheet 7 0f 12
`
`5,845,000
`
`
`
`
`
`mMOmZHm ZOiHwOm HZ<HDUUO
`
`QZDOMU
`
`m .UE
`
`9
`
`

`

`US. Patent
`
`Dec. 1, 1998
`
`Sheet 8 0f 12
`
`5,845,000
`
`N523:
`
`waomMH80::va
`
`
`H0>CQHEEL3900
`
`now0029N52.31hocosdntmNEEwvE
`
`vowmowNov
`
`H0v
`
`N328.?a
`
`mow
`
`humus—00E
`
`SEEP3900
`
`
`
`mam090580
`
`
`
`o_.v
`
`an32::E
`
`00m3
`
`wow
`
`movSam—magnum5:5
`
`
`
`
`Namor"826DEE
`
`03000NS»N0800009
`
`v.GE
`
`“020a0.ma02
`
`030885<
`
`608026
`
`Sada—En.
`
`08mm‘EMon_
`
`EN2e
`
`10
`
`10
`
`
`
`
`
`
`
`
`

`

`US. Patent
`U.S. Patent
`
`Dec. 1,1998
`Dec. 1, 1998
`
`Sheet 9 0f 12
`Sheet 9 0f 12
`
`5,845,000
`5,845,000
`
`ow
`
`P
`
`m.URN
`
`11
`
`11
`
`

`

`U.S. Patent
`US. Patent
`
`Dec. 1,1998
`Dec. 1,1998
`
`Sheet 10 0f 12
`Sheet 10 0f 12
`
`5,845,000
`5,845,000
`
`
`
`m@
`
`O
`m@
`
`w Eng
`FIG.6
`
`12
`
`12
`
`

`

`U.S. Patent
`
`Dec. 1,1998
`1111111111
`
`Sheet 11 0f 12
`
`5,845,000
`5,845,000
`
`
`
`13
`
`

`

`U.S. Patent
`
`Dec. 1,1998
`
`Sheet 12 0f 12
`
`5,845,000
`
`120
`
`101
`
`FIG. 8
`
`102
`
`811
`
`810A
`
`14
`
`

`

`1
`OPTICAL IDENTIFICATION AND
`MONITORING SYSTEM USING PATTERN
`RECOGNITION FOR USE WITH VEHICLES
`
`CROSS REFERENCE TO RELATED
`APPLICATIONS
`
`This application is a continuation-in-part of application
`Ser. No. 07/878,571 ?led May 5, 1992, noW abandoned, of
`application Ser. No. 08/040,978 ?led Mar. 31, 1993 noW
`abandoned, of copending application Ser. No. 08/247,760
`?led May 23, 1994 and of application Ser. No. 08/239,978
`?led May 9, 1994 noW abandoned, the last three of Which are
`included herein by reference.
`
`10
`
`BACKGROUND OF THE INVENTION
`
`15
`
`1. Prior Art On Out Of Position Occupants And Rear
`Facing Child Seats
`Whereas thousands of lives have been saved by airbags,
`a large number of people have also been injured, some
`seriously, by the deploying airbag, and thus signi?cant
`improvements need to be made in this regard. As discussed
`in detail in copending patent applications Ser. Nos. 08/040,
`978 and 08/239,978 cross-referenced above, for a variety of
`reasons vehicle occupants may be too close to the airbag
`before it deploys and can be seriously injured or killed as a
`result of the deployment thereof. Also, a child in a rear
`facing child seat Which is placed on the right front passenger
`seat is in danger of being seriously injured if the passenger
`airbag deploys. For these reasons and, as ?rst publicly
`disclosed in Breed, D. S. “HoW Airbags Work” presented at
`the International Conference on Seatbelts and Airbags in
`1993, in Canada, occupant position sensing and rear facing
`child seat detection is required.
`Initially these systems Will solve the out-of-position occu
`pant and the rear facing child seat problems related to
`current airbag systems and prevent unneeded airbag deploy
`ments When a front seat is unoccupied. HoWever, airbags are
`noW under development to protect rear seat occupants in
`vehicle crashes and all occupants in side impacts. A system
`Will therefore be needed to detect the presence of occupants,
`determine if they are out-of-position and to identify the
`presence of a rear facing child seat in the rear seat. Future
`automobiles are expected to have eight or more airbags as
`protection is sought for rear seat occupants and from side
`impacts. In addition to eliminating the disturbance and
`possible harm of unnecessary airbag deployments, the cost
`of replacing these airbags Will be excessive if they all deploy
`in an accident needlessly.
`In?ators noW exist Which Will adjust the amount of gas
`?oWing to the airbag to account for the siZe and position of
`the occupant and for the severity of the accident. The vehicle
`identi?cation and monitoring system (VIMS) discussed in
`patent application Ser. No. 08/239,978 Will control such
`in?ators based on the presence and position of vehicle
`occupants or of a rear facing child seat. The instant invention
`is an improvement on that VIMS system and uses an
`advanced optical system comprising one or more CCD
`(charge coupled device) arrays and a source of illumination
`combined With a trained neural netWork pattern recognition
`system.
`The need for an occupant out-of-position sensor has been
`observed by others and several methods have been disclosed
`in US. patents for determining the position of an occupant
`of a motor vehicle. Each of these systems, hoWever, have
`signi?cant limitations. In White et al. (US. Pat. No. 5,071,
`160), for example, a single acoustic sensor and detector is
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`55
`
`60
`
`65
`
`5,845,000
`
`2
`disclosed and, as illustrated, is mounted loWer than the
`steering Wheel. White et al. correctly perceive that such a
`sensor could be defeated, and the airbag falsely deployed, by
`an occupant adjusting the control knobs on the radio and
`thus they suggest the use of a plurality of such sensors.
`Mattes et al. (US. Pat. No. 5,118,134) disclose a variety
`of methods of measuring the change in position of an
`occupant including ultrasonic, active or passive infrared and
`microWave radar sensors, and an electric eye. Their use of
`these sensors is to measure the change in position of an
`occupant during a crash and use that information to access
`the severity of the crash and thereby decide Whether or not
`to deploy the airbag. They are thus using the occupant
`motion as a crash sensor. No mention is made of determining
`the out-of-position status of the occupant or of any of the
`other features of occupant monitoring as disclosed in the
`above cross-referenced patent applications. It is interesting
`to note that noWhere does Mattes et al. discuss hoW to use
`active or passive infrared to determine the position of the
`occupant. As pointed out in the above cross-referenced
`patent applications, direct occupant position measurement
`based on passive infrared is probably not possible and, until
`very recently, Was very dif?cult and expensive With active
`infrared requiring the modulation of an expensive GaAs
`infrared laser. Since there is no mention of these problems,
`the method of use contemplated by Mattes et al. must be
`similar to the electric eye concept Where position is mea
`sured indirectly as the occupant passes by a plurality of
`longitudinally spaced-apart sensors.
`The object of an occupant out-of-position sensor is to
`determine the location of the head and/or chest of the vehicle
`occupant relative to the airbag since it is the impact of either
`the head or chest With the deploying airbag Which can result
`in serious injuries. Both White et al. and Mattes et al.
`disclose only loWer mounting locations of their sensors
`Which are mounted in front of the occupant such as on the
`dashboard or beloW the steering Wheel. Both such mounting
`locations are particularly prone to detection errors due to
`positioning of the occupant’s hands, arms and legs. This
`Would require at least three, and preferably more, such
`sensors and detectors and an appropriate logic circuitry
`Which ignores readings from some sensors if such readings
`are inconsistent With others, for the case, for example, Where
`the driver’s arms are the closest objects to tWo of the sensors.
`White et al. also disclose the use of error correction
`circuitry, Without de?ning or illustrating the circuitry, to
`differentiate betWeen the velocity of one of the occupant’s
`hands as in the case Where he/she is adjusting the knob on
`the radio and the remainder of the occupant. Three ultrasonic
`sensors of the type disclosed by White et al. might, in some
`cases, accomplish this differentiation if tWo of them indi
`cated that the occupant Was not moving While the third Was
`indicating that he or she Was. Such a combination, hoWever,
`Would not differentiate betWeen an occupant With both hands
`and arms in the path of the ultrasonic transmitter at such a
`location that they Were blocking a substantial vieW of the
`occupant’s head or chest. Since the siZes and driving posi
`tions of occupants are extremely varied, trained pattern
`recognition systems, such as neural netWorks, are required
`When a clear vieW of the occupant, unimpeded by his/her
`extremities, cannot be guaranteed.
`Fujita et al., in US. Pat. No. 5,074,583, illustrate another
`method of determining the position of the occupant but do
`not use this information to suppress deployment if the
`occupant is out-of-position. In fact, the closer that the
`occupant gets to the airbag the faster the in?ation rate of the
`airbag is according to the Fujita patent, Which thereby
`
`15
`
`

`

`3
`increases the possibility of injuring the occupant. Fujita et al.
`do not measure the occupant directly but instead determine
`his or her position indirectly from measurements of the seat
`position and the vertical siZe of the occupant relative to the
`seat. This occupant height is determined using an ultrasonic
`displacement sensor mounted directly above the occupant’s
`head.
`As discussed above, the optical systems described herein
`are also applicable for many other sensing applications both
`inside and outside of the vehicle compartment such as for
`sensing crashes before they occur as described in copending
`patent application Ser. No. 08/239,978 cross-referenced
`above, for a smart headlight adjustment system and for a
`blind spot monitor.
`2. De?nitions
`The use of pattern recognition is central to the instant
`invention as Well as those cross-referenced patent applica
`tions above. NoWhere in the prior art is pattern recognition
`Which is based on training, as exempli?ed through the use of
`neural networks, mentioned for use in monitoring the inte
`rior or exterior environments of the vehicle. “Pattern rec
`ognition” as used herein Will mean any system Which
`processes a signal that is generated by an object, or is
`modi?ed by interacting With an obj ect, in order to determine
`Which one of a set of classes that the object belongs to. Such
`a system might determine only that the object is or is not a
`member of one speci?ed class, or it might attempt to assign
`the object to one of a larger set of speci?ed classes, or ?nd
`that it is not a member of any of the classes in the set. The
`signals processed are generally electrical signals coming
`from transducers Which are sensitive to either acoustic or
`electromagnetic radiation and, if electromagnetic, they can
`be either visible light, infrared, ultraviolet or radar. A train
`able or a trained pattern recognition system as used herein
`means a pattern recognition system Which is taught various
`patterns by subjecting the system to a variety of examples.
`The most successful such system is the neural netWork.
`To “identify” as used herein Will mean to determine that
`the object belongs to a particular set or class. The class may
`be one containing, for example, all rear facing child seats,
`one containing all human occupants, or all human occupants
`not sitting in a rear facing child seat depending on the
`purpose of the system. In the case Where a particular person
`is to be recogniZed, the set or class Will contain only a single
`element, i.e., the person to be recogniZed.
`An “occupying item” of a seat may be a living occupant
`such as a human being or a dog, another living organism
`such as a plant, or an inanimate object such as a box or bag
`of groceries.
`In the description herein on anticipatory sensing, the term
`“approaching” When used in connection With the mention of
`an object or vehicle approaching another Will mean the
`relative motion of the object toWard the vehicle having the
`anticipatory sensor system. Thus, in a side impact With a
`tree, the tree Will be considered as approaching the side of
`the vehicle and impacting the vehicle. In other Words, the
`coordinate system used in general Will be a coordinate
`system residing in the target vehicle. The “target” vehicle is
`the vehicle Which is being impacted. This convention per
`mits a general description to cover all of the cases such as
`Where
`a moving vehicle impacts into the side of a
`stationary vehicle, (ii) Where both vehicles are moving When
`they impact, or (iii) Where a vehicle is moving sideWays into
`a stationary vehicle, tree or Wall.
`3. Pattern Recognition Prior Art
`Japanese patent 3-42337 (A) to Ueno discloses a device
`for detecting the driving condition of a vehicle driver
`
`10
`
`15
`
`25
`
`30
`
`40
`
`45
`
`55
`
`60
`
`65
`
`5,845,000
`
`4
`comprising a light emitter for irradiating the face of the
`driver and a means for picking up the image of the driver and
`storing it for later analysis. Means are provided for locating
`the eyes of the driver and then the irises of the eyes and then
`determining if the driver is looking to the side or sleeping.
`Ueno determines the state of the eyes of the occupant rather
`than determining the location of the eyes relative to the other
`parts of the vehicle passenger compartment. Such a system
`can be defeated if the driver is Wearing glasses, particularly
`sunglasses, or another optical device Which obstructs a clear
`vieW of his/her eyes. Pattern recognition technologies such
`as neural netWorks are not used.
`U.S. Pat. No. 5,008,946 to Ando uses a complicated set of
`rules to isolate the eyes and mouth of a driver and uses this
`information to permit the driver to control the radio, for
`example, or other systems Within the vehicle by moving his
`eyes and/or mouth. Ando uses natural light and illuminates
`only the head of the driver. He also makes no use of trainable
`pattern recognition systems such as neural netWorks, nor is
`there any attempt to identify the contents of the vehicle nor
`of their location relative to the vehicle passenger compart
`ment. Rather, Ando is limited to control of vehicle devices
`by responding to motion of the driver’s mouth and eyes.
`U.S. Pat. No. 5,298,732 to Chen also concentrates in
`locating the eyes of the driver so as to position a light ?lter
`betWeen a light source such as the sun or the lights of an
`oncoming vehicle, and the driver’s eyes. Chen does not
`explain in detail hoW the eyes are located but does supply a
`calibration system Whereby the driver can adjust the ?lter so
`that it is at the proper position relative to his or her eyes.
`Chen references the use of an automatic equipment for
`determining the location of the eyes but does not describe
`hoW this equipment Works. In any event, there is no mention
`of monitoring the position of the occupant, other that the
`eyes, of determining the position of the eyes relative to the
`passenger compartment, or of identifying any other object in
`the vehicle other than the driver’s eyes. Also, there is no
`mention of the use of a trainable pattern recognition system.
`U.S. Pat. No. 5,305,012 to Paris also describes a system
`for reducing the glare from the headlights of an oncoming
`vehicle. Faris locates the eyes of the occupant by the use of
`tWo spaced apart infrared cameras using passive infrared
`radiation from the eyes of the driver. Again, Paris is only
`interested in locating the driver’s eyes relative to the sun or
`oncoming headlights and does not identify or monitor the
`occupant or locate the occupant relative to the passenger
`compartment or the airbag. Also, Faris does not use trainable
`pattern recognition techniques such as neural netWorks.
`Paris, in fact, does not even say hoW the eyes of the occupant
`are located but refers the reader to a book entitled Robot
`Vision (1991) by Berthold Horn, published by MIT Press,
`Cambridge, Mass. Also, Faris uses the passive infrared
`radiation rather than illuminating the occupant With active
`infrared radiation or in general electromagnetic radiation as
`in the instant invention.
`The use of neural netWorks as the pattern recognition
`technology is central to this invention since it makes the
`monitoring system robust, reliable and practical. The result
`ing algorithm created by the neural netWork program is
`usually only a feW lines of code Written in the C computer
`language as opposed to typically hundreds of lines When the
`techniques of the above patents to Ando, Chen and Paris are
`implemented. As a result, the resulting systems are easy to
`implement at a loW cost making them practical for automo
`tive applications. The cost of the CCD arrays, for example,
`have been prohibitively expensive until very recently ren
`dering their use for VIMS impractical. Similarly, the imple
`
`16
`
`

`

`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`5
`mentation of the techniques of the above referenced patents
`requires expensive microprocessors While the implementa
`tion With neural netWorks and similar trainable pattern
`recognition technologies permits the use of loW cost micro
`processors typically costing less than $5.
`The present invention uses sophisticated trainable pattern
`recognition capabilities such as neural netWorks. Usually the
`data is preprocessed, as discussed beloW, using various
`feature extraction. An example of such a pattern recognition
`system using neural netWorks on sonar signals is discussed
`in tWo papers by Gorman, R. P. and SejnoWski, T. J.
`“Analysis of Hidden Units in a Layered Network Trained to
`Classify Sonar Targets”, Neural Networks, Vol. 1. pp. 75—89,
`1988, and “Learned Classi?cation of Sonar Targets Using a
`Massively Parallel Network”, IEEE Transactions on
`Acoustics, Speech, and Signal Processing, Vol. 36, No. 7,
`July 1988. Examples of feature extraction techniques can be
`found in US. Pat. No. 4,906,940 entitled “Process and
`Apparatus for the Automatic Detection and Extraction of
`Features in Images and Displays” to Green et al. Examples
`of other more advanced and efficient pattern recognition
`techniques can be found in US. Pat. No. 5,390,136 entitled
`“Arti?cial Neuron and Method of Using Same and US.
`patent application Ser. No. 08/076,601 entitled “Neural
`NetWork and Method of Using Same” to Wang, S. T. Other
`examples include US. Pat. Nos. 5,235,339 (Morrison et al.),
`5,214,744 (SchWeiZer et al), 5,181,254 (SchWeiZer et al),
`and 4,881,270 (Knecht et al). All of the above references are
`included herein by reference.
`4. Optics
`Optics can be used in several con?gurations for monitor
`ing the interior of a passenger compartment of an automo
`bile. In one knoWn method, a laser optical system uses a
`GaAs infrared laser beam to momentarily illuminate an
`object, occupant or child seat, in the manner as described
`and illustrated in FIG. 8 of the copending patent application
`Ser. No. 08/040,978 cross-referenced above. The receiver
`can be a charge coupled device or CCD, (a type of TV
`camera) to receive the re?ected light. The laser can either be
`used in a scanning mode, or, through the use of a lens, a cone
`of light can be created Which covers a large portion of the
`object. In these con?gurations, the light can be accurately
`controlled to only illuminate particular positions of interest
`Within the vehicle. In the scanning mode, the receiver need
`only comprise a single or a feW active elements While in the
`case of the cone of light, an array of active elements is
`needed. The laser system has one additional signi?cant
`advantage in that the distance to the illuminated object can
`be determined as disclosed in the 08/040,978 patent appli
`cation.
`In a simpler case, light generated by a non-coherent light
`emitting diode device is used to illuminate the desired area.
`In this case, the area covered is not as accurately controlled
`and a larger CCD array is required. Recently, hoWever, the
`cost of CCD arrays has dropped substantially With the result
`that this con?guration is noW the most cost effective system
`for monitoring the passenger compartment as long as the
`distance from the transmitter to the objects is not needed. If
`this distance is required, then either the laser system, a
`stereographic system, a focusing system, or a combined
`ultrasonic and optic system is required. Amechanical focus
`ing system, such as used on some camera systems can
`determine the initial position of an occupant but is too sloW
`to monitor his/her position during a crash. A distance mea
`suring system based of focusing is described in US. Pat. No.
`5,193,124 (Subbarao) Which can either be used With a
`mechanical focusing system or With tWo cameras, the latter
`
`55
`
`60
`
`65
`
`5,845,000
`
`6
`of Which Would be fast enough. Although the Subbarao
`patent provides a good discussion of the camera focusing art
`and is therefore included herein by reference, it is a more
`complicated system than is needed for the practicing the
`instant invention. In fact, a neural netWork can also be
`trained to perform the distance determination based on the
`tWo images taken With different camera settings or from tWo
`adjacent CCD’s and lens having different properties as the
`cameras disclosed in Subbarao making this technique prac
`tical for the purposes of this instant invention. Distance can
`also be determined by the system disclosed in US. Pat. No.
`5,003,166 (Girod) by the spreading or defocusing of a
`pattern of structured light projected onto the object of
`interest.
`In each of these cases, regardless of the distance mea
`surement system used, a trained pattern recognition system,
`as de?ned above, is used in the instant invention to identify
`and classify, and in some cases to locate, the illuminated
`object and its constituent parts.
`5. Optics And Acoustics
`The laser systems described above are expensive due to
`the requirement that they be modulated at a high frequency
`if the distance from the airbag to the occupant, for example,
`needs to be measured. Both laser and non-laser optical
`systems in general are good at determining the location of
`objects Within the tWo dimensional plane of the image and
`the modulated laser system in the scanning mode can
`determine the distance of each part of the image from the
`receiver. It is also possible to determine distance With the
`non-laser system by focusing as discussed above, or stereo
`graphically if tWo spaced apart receivers are used and, in
`some cases the mere location in the ?eld of vieW can be used
`to estimate the position relative to the airbag, for example.
`Finally, a recently developed pulsed quantum Well diode
`laser does provide inexpensive distance measurements as
`discussed beloW.
`Acoustic systems are also quite effective at distance
`measurements since the relatively loW speed of sound per
`mits simple electronic circuits to be designed and minimal
`microprocessor capability is required. If a coordinate system
`is used Where the Z axis is from the transducer to the
`occupant, acoustics are good at measuring Z dimensions
`While simple optical systems using a single CCD are good
`at measuring x and y dimensions. The combination of
`acoustics and optics, therefore, permits all three measure
`ments to be made With loW cost components.
`One example of a system using these ideas is an optical
`system Which ?oods the passenger seat With infrared light
`coupled With a lens and CCD array Which receives and
`displays the re?ected light and an analog to digital converter
`(ADC) Which digitiZes the output of the CCD and feeds it to
`an Arti?cial Neural NetWork (ANN) or other pattern recog
`nition system , for analysis. This system uses an ultrasonic
`transmitter and receiver for measuring the distances to the
`objects located in the passenger seat. The receiving trans
`ducer feeds its data into an ADC and from there into the
`ANN. The same ANN can be used for both systems thereby
`providing full three dimensional data for the ANN to ana
`lyZe. This system, using loW cost components, Will permit
`accurate identi?cation and distance measurements not pos
`sible by either system acting alone. If a phased array system
`is added to the acoustic part of the system as disclosed in
`copending patent application (ATI-102), the optical part can
`determine the location of the driver’s ears, for example, and
`the phased array can direct a narroW beam to the location
`and determine the distance to the occupant’s ears.
`
`17
`
`

`

`5,845,000
`
`7
`
`6. Applications
`The applications for this technology are numerous as
`described in the copending patent applications listed above.
`They include:
`the monitoring of the occupant for safety
`purposes to prevent airbag deployment induced injuries, (ii)
`the locating of the eyes of the occupant to permit automatic
`adjustment of the rear vieW mirror(s), (iii) the location of the
`seat to place the eyes at the proper position to eliminate the
`parallaX in a heads-up display in night vision systems, (iv)
`the location of the ears of the occupant for optimum adjust
`ment of the entertainment system, (v) the identi?cation of
`the occupant for security reasons, (vi) the determination of
`obstructions in the path of a closing door or WindoW, (vii) the
`determination of the position of the occupant’s shoulder so
`that the seat belt anchorage point can be adjusted for the best
`protection of the occupant, (viii) the determination of the
`position of the rear of the occupants head so that the headrest
`can be adjusted to minimiZe Whiplash injuries in rear
`impacts,
`anticipatory crash sensing,
`blind spot
`detection,
`smart headlight dimmers, and many others. In
`fact, over forty products alone have been identi?ed based on
`the ability to identify and monitor objects and parts thereof
`in the passenger compartment of an automobile or truck.
`
`10
`
`15
`
`20
`
`SUMMARY OF THE INVENTION
`
`25
`
`This invention is a system to identify, locate and monitor
`occupants, including their parts, and other objects in the
`passenger compartment and objects outside of a motor
`vehicle, such as an automobile or truck, by illuminating the
`contents of the vehicle and objects outside of the vehicle
`With electromagnetic, and speci?cally infrared, radiation
`and using one or more lenses to focus images of the contents
`onto one or more arrays of charge coupled devices (CCD
`arrays). Outputs from the CCD arrays, are analyZed by
`appropriate computational means employing trained pattern
`recognition technologies, to classify, identify or locate the
`contents or external objects. In general, the information
`obtained by the identi?cation and monitoring system is used
`to affect the operation of some other system in the vehicle.
`When the vehicle interior monitoring system of this
`invention is installed in the passenger compartment of an
`automotive vehicle equipped With a passenger protective
`device, such as an in?atable airbag, and the vehicle is
`subjected to a crash of sufficient severity that the crash
`sensor has determined that the protective device is to be
`deployed, the system in accordance With the invention
`determines the po