1
`
`Mercedes-Benz USA, LLC, Petitioner - Ex. 1011
`
`

`

`BAR CODE LABEL
`
`liIIIIlllllllllllllllllillmllilll
`
`Us. PATENT APPLICATION
`
`SERIAL NUMBER
`
`FILING DATE
`
`-
`
`GROUP AFIT UNIT
`
`08/474,786
`
`06/07/95
`
`2613
`
`DAVID S. BREED, BOONTON TOWNSHI, NJ; WILBUR E. DUVALL, KIMBERLING CITY,
`MO; WENDELL C.
`JOHNSON, TORRANCE, CA.
`
`**CONTINUING DATAi-tkt'ktki'iiink****i****
`
`VERIFIED
`
`THIS APPLN IS A CIP or
`AND A CIP or
`AND A CIP or
`AND A cxp or
`
`07/878,571 05/05/92 ABN
`08/040,978 03/31/93
`08/247,760 05/23/94
`08/239,978 05/09/94
`
`**FOREIGN/PCT APPLICATIONS**** *** * ****
`VERIFIED
`
`
`
`APPLICANT
`
`ADDRESS
`
`FOREIGN FILING LICENSE GRANTED 09/07/95
`
`***** SMALL ENTITY *****
`FILING FEE
`ATTORNEY DOCKET NO.
`RECEIVED
`
`$430.00
`
`ATI-95
`
`BRIAN ROFFE
`376 YALE AVENUE
`WOODMERE NY 11598-2051
`
`OPTICAL IDENTIFICATION AND MONITORING SYSTEM USING PATTERN
`RECOGNITION FOR USE WITH VEHICLES
`
`that annexed hereto is 8 true 00 y from the _records of the United States
`This is to certi
`Patent and Tra emark Office of the application w lch is identified above.
`By authority of the
`COMMISSIONER OF PATENTS AND TRADEMARKS
`
`Date
`
`Certifying Officer
`
`2
`
`

`

`
`
`I
`‘ mam APPLIcmmN //
`
`
`
`
`
`
`MW
`
`
`
`
`
`
`
`08474?86
`
`
`V’""g
`‘L
`x;
`
`53,i
`
`if
`
`; Q3¥ é!a'
`
`6‘
`
`g 3i
`
`.5I
`ft‘
`
`3
`
`€
`

`

`United States Patent
`Breed et al.
`
`[19]
`
`[11] Patent Number:
`
`5,845,000
`
`[45] Date of Patent:
`
`*Dec. 1, 1998
`
`U8005845000A
`
`[54]
`
`[75]
`
`Ol’I‘ICAL IDENTIFICATION AND
`MONITORING SYSTEM USING PATTERN
`RECOGNITION FOR USE WITH VEHICLES
`
`Inventors: David S. Breed, Boonton Township,
`NJ.; Wilbur E. DuVall, Kimberling
`City, Mo.; Wendell C. Johnson,
`Torrance, Calif.
`
`[73]
`
`Assignee: Automotive Technologies
`International, Inc., Denville, NJ.
`
`1‘]
`
`Notice:
`
`The term of this patent shall not extend
`beyond the expiration date of Pat. No.
`5,835,613.
`
`[211
`
`{221
`
`[63]
`
`[51]
`[52]
`[58]
`
`[56]
`
`Appl. No.: 474,786
`
`Filed:
`
`Jun. 7, 1995
`
`Related US. Application Data
`
`Continuation-impart at Ser. No. 878,571, May 5, 1992,
`abandoned, Ser. No. 40,978, Mar. 31, 1993, abandoned, Ser.
`No. 247,760, May 23, 1994, and Ser. No. 239,978, May 9,
`1994, abandoned.
`
`Int. Cl.“ ....................................................... G06K 9/00
`
`US. Cl. ..............
`382/100; 348/143
`Field of Search ............................. 340/436; 382/104,
`382/103, 291, 100; 280/735; 348/143, 148
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`1/1985 Shah ........................................ 350/354
`4,496,222
`4,625,329 11/1986 Ishikawa et al.
`.
`..... 382/1
`
`.
`. 364/550
`4,648,052
`3/1987 Friedman et al.
`4,720,189
`1/1988 Heynen et al.
`351/210
`
`8/1988 Ando .................
`4,768,088
`358/93
`.. 351/210
`4,836,670
`6/1989 Hutchinson
`
`..... 382/191
`4,881,270 11/1989 Knecht el al.
`.
`..... 382/100
`4,906,940
`3/1990 Greene et al.
`
`..... 351/210
`4,950,069
`8/1990 Hutchinson
`
`.
`.. 280/7301
`4,966,388 10/1990 Warner et al.
`3/1991 Girod ................................... 250/2014
`5,103,166
`
`FOREIGN PATENT DOCUMENTS
`
`Japan.
`2/1991
`342337
`94/22692 10/1994 WIPO.
`
`OTHER PUBLICATIONS
`
`"Analysis of Hidden Units in a Layered Network Trained to
`Classify Sonar Targets”, R. Paul Gorman, et al., Neural
`Networks, vol. 1, pp.75-89, 1988.
`Learned Clasification of Sonar Targets Using a Musively
`Parallel Network, R. Paul Gonnan et al., IEEE Transactions
`on Acoustics, Speech and Signal Processing, vol. 36, No. 7,
`Jul., 1988, pp. 1135—1140.
`“How Airbags Work”, David S. Breed, Presented at the
`Canadian Association of Road Safety Professional, Oct. 19,
`1992—Oct. 20, 1992.
`Derwent Abstract of German Patent Publication No. DB 42
`11 556, Oct. 7, 1993.
`Derwent Abstract of Japanese Patent Application No.
`02-051332, Nov. 13, 1991.
`
`Primary Examiner—Yon J. Couso
`
`[57]
`
`ABSTRACT
`
`A vehicle interior monitoring system to identify, locate and
`monitor occupants, including their parts, and other objects in
`the passenger compartment and objects outside of a motor
`vehicle, such as an automobile or truck, by illuminating the
`contents of the vehicle and objects outside of the vehicle
`with electromagnetic, and specifically infrared, radiation
`and using one or more lenses to focus images of the contents
`onto one or more arrays of charge coupled devices (CCD
`arrays). Outputs from the CCD arrays, are analyzed by
`appropriate computational means employing trained pattern
`recognition technologies, to classify, identify or locate the
`contents or external objects. In general.
`the information
`obtained by the identification and monitoring system is used
`to afiect the operation of some other system in the vehicle.
`When system is installed in the passenger compartment of
`an automotive vehicle equipped with an airbag, the system
`determines the position of the vehicle occupant relative to
`the airbag and disables deployment of the airbag if the
`occupant is positioned so that he/she is likely to be injured
`by the deployment of the airbag.
`
`(List continued on next page.)
`
`25 Claims, 12 Drawing Sheets
`
`
`
`4
`
`

`

`
`5,845,000
`Page 2
`
`
`
`
`
`US. PATENT DOCUMENTS
`
`
`382/2
`5,008,946
`4/1991 Ando
`
`
`
`
`
`5,026,153
`6/1991 Suzuki et a].
`.......... 356/316
`
`
`
`
`
`
`
`
`
`5,064,274 11/1991
`......... 359/604
`
`
`
`
`
`
`280/735
`5,071,160 12/1991
`
`
`
`
`
`
`
`
`5,074,583 12/1991
`. 280/7301
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`5,118,134
`6/1992 Mattes 9t a1.
`280/735
`
`
`
`
`
`
`
`11/1992 Tamburmo et a1.
`356/5
`5,162,861
`5,181,254
`1/1993 Schweizer et al.
`382/100
`
`
`
`
`
`
`
`
`5,185,667
`2/1993 Zimmermann
`. 358/209
`
`
`
`
`
`
`
`
`3/1993 Subbarao
`5,193,124
`382/255
`
`
`
`
`
`5,214,744
`5/1993 Schweizer ct a].
`395/11
`
`
`
`
`
`
`
`5,227,734
`7/1993 Masamofi et 31.
`340/436
`
`
`
`
`
`
`8/1993 Morrison et a1.
`5,235,339
`342/159
`
`
`
`
`
`
`
`
`
`
`
`
`5,249,027
`.............................. 356/1
`9/1993 Mathur et a1.
`
`
`
`
`
`
`
`
`
`
`..................................... 34
`
`
`
`
`
`5,292,252
`i122: gym
`25025::
`
`
`
`
`
`‘3“ ---------------------------------
`7
`,
`/
`~
`
`
`
`
`
`4/1994 Fans ............................................ 345/7
`5,305,012
`5/1994 Kajiwara ................................. 348/148
`5,309,137
`
`
`
`
`
`7/1994 Slotkowski at al.
`5,329,206
`.................... 315/159
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`7/1994 Gentry et al.
`........................... 280/735
`5,330,226
`
`8/1994 Abst et al.
`340/903
`5,339,075
`.
`
`
`
`
`
`
`
`
`
`
`
`. 348/148
`5,355,118 10/1994 Fukahara
`
`
`
`
`
`364/754
`2/1995 Wang
`5,390,136
`
`
`
`
`
`
`
`
`
`
`
`5:441:052
`8/1995 Miyajim
`126/661-09
`
`
`.................. 260/735
`10/1995 Mazur el al.
`5,454,591
`
`
`
`
`
`
`
`
`
`
`5,537,003
`7/1996 Bechtel ct a1. .......................... 35/82
`
`
`
`
`
`
`
`
`5
`
`

`

`US. Patent
`
`Dec 1, 1998
`
`Sheet 1 0f 12
`
`5,845,000
`
`w:
`
`5—
`
`mg
`
`C: 2:
`
`m:
`
`6
`
`

`

`US. Patent
`
`Dec. 1, 1998
`
`Sheet 2 of 12
`
`5,845,000
`
`:2
`
`mo—
`
`0: 02
`
`m:
`
`7
`
`

`

`US. Patent
`
`Dec. 1, 1998
`
`Sheet 3 0f 12
`
`5,845,000
`
`#2
`
`2:
`
`m3 on:
`
`m:
`
`
`
`8
`
`

`

`US. Patent
`
`Dec. 1, 1998
`
`Sheet 4 of 12
`
`5,845,000
`
`114
`
`112
`
`104
`
`120
`
`101
`
`FIG.ID
`
`105
`
`2- '.
`
`'-
`
`‘..
`,
`
`.
`.
`
`100 " 1;:
`
`1 .
`:5.
`.'.
`
`L ’13"
`x 34..
`1"1'3...
`'-
`
`..
`
`F"
`r-t
`
`9
`
`

`

`US. Patent
`
`Dec. 1,1998
`
`Sheet 5 of 12
`
`5,845,000
`
`114
`
`112
`
`104
`
`120
`
`110
`
`105
`
`100
`
`mPI!
`—-4
`
`101
`
`FIG.1E
`
`10
`
`10
`
`

`

`US. Patent
`
`Dec. 1, 1998
`
`Sheet 6 of 12
`
`5,845,000
`
`210
`
`
`
`FIG.2
`
`
`11
`
`11
`
`

`

`12 VOLTS
`
`l
`
`:
`l
`l’f
`
`/’
`
`I’
`1’
`
`I,
`
`
`L. _____ J
`ELECTRONIC
`CRASH SENSOR
`
`
`
`I
`.
`
`'— _ '—
`|
`I
`I
`I
`I
`I
`' ==6 :
`l v
`
`DIAGNOSTIC UNIT
`
`I
`
`FIG. 3
`
`
`
`
`
`
`OCCUPANT POSITION SENSORS
`
`INFLATABLE RESTRAINT SYSTEM
`
`GROUND
`
`mamaST]
`
`8661‘Ima
`
`Z1101.199118
`
`000‘9178‘9
`
`12
`
`12
`
`

`

`mamaso
`
`U
`3
`.:
`8
`
`co
`
`(I:
`Eg‘
`(D
`
`aG
`
`‘fi
`-
`
`“’
`
`
`
`401
`
`402
`
`403
`
`404
`
`f1 48 MHz
`Crystal
`Oscillator
`
`
`
`
`
`4
`
`407
`
`EB‘
`
`"
`
`f2 48.05 MHz
`C‘Ys‘a'
`Oscillator
`
`Frequency
`Triplet
`
`3f2
`L
`
`415
`
`Phase
`Detector
`
`144AMhz
`
`J4“)
`
`eceive
`
`
`
`0 to
`
`(
`
`2M 1so KH
`[x-
`
`'
`
`Filter
`
`.
`Automatxc
`
`Gain Control
`Amplifier
`
`gt-
`
`No Si :1
`.
`Detect .
`
`EQ .h
`
`.
`D = RX
`
`o _
`Velomty
`
`0 KHz
`[X
`
`000378?
`
`13
`
`13
`
`

`

`U.S. Patent
`
`Dec. 1, 1998
`
`Sheet 9 of 12
`
`5,845,000
`
`.DNL
`
`m.
`
`
`14
`
`14
`
`

`

`US. Patent
`
`Dec. 1, 1998
`
`Sheet 10 0f 12
`
`5,845,000
`
`
`
`FIG.6
`
`15
`
`15
`
`

`

`US. Patent
`
`Dec. 1, 1998
`
`Sheet 11 of 12
`
`5,845,000
`
`FIG.7
`
`
`
`16
`
`

`

`US. Patent
`
`Dec. 1, 1998
`
`Sheet 12 0f 12
`
`5,845,000
`
`ONu
`
`—d
`
`
`
`FIG.8
`
`17
`
`17
`
`

`

`
`5,845,000
`
`
`1
`OPTICAL IDENTIFICATION AND
`
`
`
`MONITORING SYSTEM USING PATTERN
`
`
`
`
`RECOGNITION FOR USE WITH VEHICLES
`
`
`
`
`CROSS REFERENCE TO RELATED
`
`
`
`
`APPLICATIONS
`
`
`
`
`This application is a continuation—in-part of application
`
`
`
`
`
`
`
`Ser. No. 07/878,571 filed May 5, 1992, now abandoned, of
`
`
`
`
`
`
`
`
`
`
`application Ser. No. 08/040,978 filed Mar. 31, 1993 now
`
`
`
`
`
`
`
`
`
`abandoned, of copending application Ser. No. 08/247,760
`
`
`
`
`
`
`
`filed May 23, 1994 and of application Ser. No. 08/239,978
`
`
`
`
`
`
`
`
`
`
`filed May 9, 1994 now abandoned, the last three of which are
`
`
`
`
`
`
`
`
`
`
`
`included herein by reference.
`
`
`
`
`BACKGROUND OF THE INVENTION
`
`
`
`
`1. Prior Art On Out Of Position Occupants And Rear
`
`
`
`
`
`
`
`
`
`
`Facing Child Seats
`
`
`
`Whereas thousands of lives have been saved by airbags,
`
`
`
`
`
`
`
`
`
`a large number of people have also been injured, some
`
`
`
`
`
`
`
`
`
`
`seriously, by the deploying airbag, and thus significant
`
`
`
`
`
`
`
`
`improvements need to be made in this regard. As discussed
`
`
`
`
`
`
`
`
`
`
`in detail in copending patent applications Ser. Nos. 08/040,
`
`
`
`
`
`
`
`
`
`978 and 08/239,978 cross-referenced above, for a variety of
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`reasons vehicle occupants may be too close to the airbag
`before it deploys and can be seriously injured or killed as a
`
`
`
`
`
`
`
`
`
`
`
`
`result of the deployment thereof. Also, a child in a rear
`
`
`
`
`
`
`
`
`
`
`
`facing child seat which is placed on the right front passenger
`
`
`
`
`
`
`
`
`
`
`
`seat is in danger of being seriously injured if the passenger
`
`
`
`
`
`
`
`
`
`
`
`airbag deploys. For these reasons and, as first publicly
`
`
`
`
`
`
`
`
`
`disclosed in Breed, D. S. “How Airbags Wor " presented at
`
`
`
`
`
`
`
`
`
`
`the International Conference on Seatbelts and Airbags in
`
`
`
`
`
`
`
`
`1993, in Canada, occupant position sensing and rear facing
`
`
`
`
`
`
`
`
`
`child seat detection is required.
`
`
`
`
`
`Initially these systems will solve the out-of-position occu~
`
`
`
`
`
`
`
`pant and the rear facing child seat problems related to
`
`
`
`
`
`
`
`
`
`
`current airbag systems and prevent unneeded airbag deploy-
`
`
`
`
`
`
`
`ments when a front seat is unoccupied. However, airbags are
`
`
`
`
`
`
`
`
`
`now under development to protect rear seat occupants in
`
`
`
`
`
`
`
`
`
`vehicle crashes and all occupants in side impacts. Asystern
`
`
`
`
`
`
`
`
`
`
`will therefore be needed to detect the presence of occupants,
`
`
`
`
`
`
`
`
`
`
`
`determine if they are out-of-position and to identify the
`
`
`
`
`
`
`
`
`presence of a rear facing child seat in the rear seat. Future
`
`
`
`
`
`
`
`
`
`
`
`automobiles are expected to have eight or more airbags as
`
`
`
`
`
`
`
`
`
`
`protection is sought for rear seat occupants and from side
`
`
`
`
`
`
`
`
`
`
`impacts. In addition to eliminating the disturbance and
`
`
`
`
`
`
`
`
`possrble harm of unnecessary airbag deployments, the cost
`
`
`
`
`
`
`
`
`of replacing these airbags will be excessive if they all deploy
`
`
`
`
`
`
`
`
`
`
`
`in an accident needlessly,
`
`
`
`
`Inflators now exist which will adjust the amount of gas
`
`
`
`
`
`
`
`
`
`
`flowing to the airbag to account for the size and position of
`
`
`
`
`
`
`
`
`
`
`
`
`the occupant and for the severity of the accident. The vehicle
`
`
`
`
`
`
`
`
`
`
`
`identification and monitoring system (VIMS) discussed in
`
`
`
`
`
`
`
`patent application Ser. No. 08/239,978 will control such
`
`
`
`
`
`
`
`
`inflators based on the presence and position of vehicle
`
`
`
`
`
`
`
`
`
`occupants or of a rear facing child seat. The instant invention
`
`
`
`
`
`
`
`
`
`
`is an improvement on that VIMS system and uses an
`
`
`
`
`
`
`
`
`
`
`advanced optical system comprising one or more CCD
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`(charge coupled device) arrays and a source of illumination
`combined with a trained neural network pattern recognition
`
`
`
`
`
`
`
`system.
`
`The need for an occupant out-of-position sensor has been
`
`
`
`
`
`
`
`
`
`observed by others and several methods have been disclosed
`
`
`
`
`
`
`
`
`
`in US. patents for determining the position of an occupant
`
`
`
`
`
`
`
`
`
`
`of a motor vehicle. Each of these systems, however, have
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`significant limitations. In White et al. (US. Pat. No. 5,071,
`
`
`
`
`
`
`
`
`
`
`160), for example, a single acoustic sensor and detector is
`
`
`
`
`2
`is mounted lower than the
`disclosed and, as illustrated,
`
`
`
`
`
`
`
`
`
`steering Wheel. White et a1. correctly perceive that such a
`
`
`
`
`
`
`
`
`
`
`sensor could be defeated, and the airbag falsely deployed, by
`
`
`
`
`
`
`
`
`
`
`an occupant adjusting the control knobs on the radio and
`
`
`
`
`
`
`
`
`
`
`thus they suggest the use of a plurality of such sensors.
`
`
`
`
`
`
`
`
`
`
`Mattes et al. (US. Pat. No. 5,118,134) disclose a variety
`
`
`
`
`
`
`
`
`
`
`of methods of measuring the change in position of an
`
`
`
`
`
`
`
`
`
`
`occupant including ultrasonic, active or passive infrared and
`
`
`
`
`
`
`
`
`microwave radar sensors, and an electric eye. Their use of
`
`
`
`
`
`
`
`
`
`
`these sensors is to measure the change in position of an
`
`
`
`
`
`
`
`
`
`
`
`occupant during a crash and use that information to access
`
`
`
`
`
`
`
`
`
`
`the severity of the crash and thereby decide whether or not
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`to deploy the airbag. They are thus using the occupant
`motion as a crash sensor. No mention is made of determining
`
`
`
`
`
`
`
`
`
`
`the out-of—position status of the occupant or of any of the
`
`
`
`
`
`
`
`
`
`
`
`other features of occupant monitoring as disclosed in the
`
`
`
`
`
`
`
`
`
`above cross-referenced patent applications. It is interesting
`
`
`
`
`
`
`
`to note that nowhere does Mattes et a1. discuss how to use
`
`
`
`
`
`
`
`
`
`
`
`
`active or passive infrared to determine the position of the
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`occupant. As pointed out
`in the above cross-referenced
`patent applications, direct occupant position measurement
`
`
`
`
`
`
`based on passive infrared is probably not possible and, until
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`very recently, was very diflicult and expensive with active
`infrared requiring the modulation of an expensive GaAs
`
`
`
`
`
`
`
`
`infrared laser. Since there is no mention of these problems,
`
`
`
`
`
`
`
`
`
`
`the method of use contemplated by Mattes et al. must be
`
`
`
`
`
`
`
`
`
`
`
`similar to the electric eye concept where position is mea-
`
`
`
`
`
`
`
`
`
`sured indirectly as the occupant passes by a plurality of
`
`
`
`
`
`
`
`
`
`longitudinally spaced-apart sensors.
`
`
`
`The object of an occupant out-of—position sensor is to
`
`
`
`
`
`
`
`
`
`determine the location of the head and/or chest of the vehicle
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`occupant relative to the airbag since it is the impact of either
`the head or chest with the deploying airbag which can result
`
`
`
`
`
`
`
`
`
`
`
`in serious injuries. Both White et a1. and Mattes et a1.
`
`
`
`
`
`
`
`
`
`
`
`disclose only lower mounting locations of their sensors
`
`
`
`
`
`
`
`
`which are mounted in front of the occupant such as on the
`
`
`
`
`
`
`
`
`
`
`
`
`dashboard or below the steering wheel. Both such mounting
`
`
`
`
`
`
`
`
`
`locations are particularly prone to detection errors due to
`
`
`
`
`
`
`
`
`
`positioning of the occupant’s hands, arms and legs. This
`
`
`
`
`
`
`
`
`
`would require at
`least three, and preferably more, such
`
`
`
`
`
`
`
`
`
`sensors and detectors and an appropriate logic circuitry
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`which ignores readings from some sensors if such readings
`are inconsistent with others, for the case, for example, where
`
`
`
`
`
`
`
`
`
`
`the driver’s arms are the closest objects to two of the sensors.
`
`
`
`
`
`
`
`
`
`
`
`
`White et al. also disclose the use of error correction
`
`
`
`
`
`
`
`
`
`
`circuitry, without defining or illustrating the circuitry,
`to
`
`
`
`
`
`
`
`
`differentiate between the velocity of one of the occupant’s
`
`
`
`
`
`
`
`
`
`hands as in the case where he/she is adjusting the knob on
`
`
`
`
`
`
`
`
`
`
`
`
`the radio and the remainder of the occupant. Three ultrasonic
`
`
`
`
`
`
`
`
`
`
`sensors of the type disclosed by White et a1. might, in some
`
`
`
`
`
`
`
`
`
`
`
`
`cases, accomplish this differentiation if two of them indi—
`
`
`
`
`
`
`
`
`cated that the occupant was not moving while the third was
`
`
`
`
`
`
`
`
`
`
`
`indicating that he or she was. Such a combination, however,
`
`
`
`
`
`
`
`
`
`would not differentiate between an occupant with both hands
`
`
`
`
`
`
`
`
`
`and arms in the path of the ultrasonic transmitter at such a
`
`
`
`
`
`
`
`
`
`
`
`
`location that they were blocking a substantial view of the
`
`
`
`
`
`
`
`
`
`
`occupant’s head or chest. Since the sizes and driving posi—
`
`
`
`
`
`
`
`
`
`tions of occupants are extremely varied,
`trained pattern
`
`
`
`
`
`
`
`
`recognition systems, such as neural networks, are required
`
`
`
`
`
`
`
`
`when a clear View of the occupant, unimpeded by his/her
`
`
`
`
`
`
`
`
`
`
`extremities, cannot be guaranteed.
`
`
`
`
`Fujita et al., in US. Pat. No. 5,074,583, illustrate another
`
`
`
`
`
`
`
`
`
`
`method of determining the position of the occupant but do
`
`
`
`
`
`
`
`
`
`
`not use this information to suppress deployment
`if the
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`occupant
`is out-of-position. In fact,
`the closer that
`the
`
`
`
`
`
`
`
`
`
`
`
`
`occupant gets to the airbag the faster the inflation rate of the
`
`
`
`
`
`
`
`
`
`airbag is according to the Fujita patent, which thereby
`
`
`
`10
`
`
`
`15
`
`
`20
`
`
`25
`
`
`30
`
`
`
`35
`
`
`40
`
`
`
`45
`
`
`50
`
`
`
`55
`
`
`60
`
`
`65
`
`
`
`
`18
`
`18
`
`

`

`
`5,845,000
`
`
`
`
`3
`increases the possibility of injuring the occupant. Fujita et al.
`
`
`
`
`
`
`
`
`
`
`do not measure the occupant directly but instead determine
`
`
`
`
`
`
`
`
`
`his or her position indirectly from measurements of the seat
`
`
`
`
`
`
`
`
`
`
`position and the vertical size of the occupant relative to the
`
`
`
`
`
`
`
`
`
`
`
`seat. This occupant height is determined using an ultrasonic
`
`
`
`
`
`
`
`
`
`displacement sensor mounted directly above the occupant’s
`
`
`
`
`
`
`
`head.
`
`As discussed above, the optical systems described herein
`
`
`
`
`
`
`
`
`are also applicable for many other sensing applications both
`
`
`
`
`
`
`
`
`
`inside and outside of the vehicle compartment such as for
`
`
`
`
`
`
`
`
`
`
`sensing crashes before they occur as described in copending
`
`
`
`
`
`
`
`
`
`patent application Ser. No. 08/239,978 cross-referenced
`
`
`
`
`
`
`above, for a smart headlight adjustment system and for a
`
`
`
`
`
`
`
`
`
`
`blind spot monitor.
`
`
`
`2. Definitions
`
`
`The use of pattern recognition is central to the instant
`
`
`
`
`
`
`
`
`
`invention as well as those cross—referenced patent applica-
`
`
`
`
`
`
`
`tions above. Nowhere in the prior art is pattern recognition
`
`
`
`
`
`
`
`
`
`
`which is based on training, as exemplified through the use of
`
`
`
`
`
`
`
`
`
`
`
`neural networks, mentioned for use in monitoring the inte-
`
`
`
`
`
`
`
`
`rior or exterior environments of the vehicle. “Pattern rec-
`
`
`
`
`
`
`
`
`ognition” as used herein will mean any system which
`
`
`
`
`
`
`
`
`
`processes a signal
`is generated by an object, or is
`that
`
`
`
`
`
`
`
`
`
`
`
`modified by interacting with an object, in order to determine
`
`
`
`
`
`
`
`
`
`
`which one of a set of classes that the object belongs to. Such
`
`
`
`
`
`
`
`
`
`
`
`
`a system might determine only that the object is or is not a
`
`
`
`
`
`
`
`
`
`
`
`
`
`member of one specified class, or it might attempt to assign
`
`
`
`
`
`
`
`
`
`
`
`the object to one of a larger set of specified classes, or find
`
`
`
`
`
`
`
`
`
`
`
`
`that it is not a member of any of the classes in the set. The
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`signals processed are generally electrical signals coming
`
`
`
`
`
`
`
`from transducers which are sensitive to either acoustic or
`
`
`
`
`
`
`
`
`
`electromagnetic radiation and, if electromagnetic, they can
`
`
`
`
`
`
`
`be either visible light, infrared, ultraviolet or radar. A train.-
`
`
`
`
`
`
`
`
`
`able or a trained pattern recognition system as used herein
`
`
`
`
`
`
`
`
`
`
`means a pattern recognition system which is taught various
`
`
`
`
`
`
`
`
`
`patterns by subjecting the system to a variety of examples.
`
`
`
`
`
`
`
`
`
`
`The most successful such system is the neural network.
`
`
`
`
`
`
`
`
`
`To “identify” as used herein will mean to determine that
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`the object belongs to a particular set or class. The class may
`be one containing, for example, all rear facing child seats,
`
`
`
`
`
`
`
`
`
`
`one containing all human occupants, or all human occupants
`
`
`
`
`
`
`
`
`
`not sitting in a rear facing child seat depending on the
`
`
`
`
`
`
`
`
`
`
`
`purpose of the system. In the case where a particular person
`
`
`
`
`
`
`
`
`
`
`is to be recognized, the set or class will contain only a single
`
`
`
`
`
`
`
`
`
`
`
`
`element, i.e., the person to be recognized.
`
`
`
`
`
`
`
`An “occupying item” of a seat may be a living occupant
`
`
`
`
`
`
`
`
`
`
`such as a human being or a dog, another living organism
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`such as a plant, or an inanimate object such as a box or bag
`of groceries.
`
`
`In the description herein on anticipatory sensing, the term
`
`
`
`
`
`
`
`
`
`“approaching” when used in connection with the mention of
`
`
`
`
`
`
`
`
`
`an object or vehicle approaching another will mean the
`
`
`
`
`
`
`
`
`
`relative motion of the object toward the vehicle having the
`
`
`
`
`
`
`
`
`
`
`anticipatory sensor system. Thus, in a side impact With a
`
`
`
`
`
`
`
`
`
`
`tree, the tree will be considered as approaching the side of
`
`
`
`
`
`
`
`
`
`
`
`the vehicle and impacting the vehicle. In other words, the
`
`
`
`
`
`
`
`
`
`
`coordinate system used in general will be a coordinate
`
`
`
`
`
`
`
`
`
`system residing in the target vehicle. The “target” vehicle is
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`the vehicle which is being impacted. This convention per-
`mits a general description to cover all of the cases such as
`
`
`
`
`
`
`
`
`
`
`
`where (i) a moving vehicle impacts into the side of a
`
`
`
`
`
`
`
`
`
`
`stationary vehicle, (ii) where both vehicles are moving when
`
`
`
`
`
`
`
`
`they impact, or (iii) where a vehicle is moving sideways into
`
`
`
`
`
`
`
`
`
`a stationary vehicle, tree or wall.
`
`
`
`
`
`
`3. Pattern Recognition Prior Art
`
`
`
`
`
`Japanese patent 342337 (A) to Ueno discloses a device
`
`
`
`
`
`
`
`
`for detecting the driving condition of a vehicle driver
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`10
`
`
`
`15
`
`
`20
`
`
`25
`
`
`
`30
`
`
`
`35
`
`
`
`40
`
`
`
`45
`
`
`
`50
`
`
`
`55
`
`
`
`60
`
`
`
`65
`
`
`
`4
`
`
`comprising a light emitter for irradiating the face of the
`
`
`
`
`
`
`
`
`
`
`driver and a means for picking up the image of the driver and
`
`
`
`
`
`
`
`
`
`
`
`
`storing it for later analysis. Means are provided for locating
`
`
`
`
`
`
`
`
`
`
`the eyes of the driver and then the irises of the eyes and then
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`determining if the driver is looking to the side or sleeping.
`
`
`
`
`
`
`
`
`
`
`
`Ueno determines the state of the eyes of the occupant rather
`
`
`
`
`
`
`
`
`
`
`
`than determining the location of the eyes relative to the other
`
`
`
`
`
`
`
`
`
`
`
`parts of the vehicle passenger compartment. Such a system
`
`
`
`
`
`
`
`
`
`can be defeated if the driver is wearing glasses, particularly
`
`
`
`
`
`
`
`
`
`
`sunglasses, or another optical device which obstructs a clear
`
`
`
`
`
`
`
`
`view of his/her eyes. Pattern recognition technologies such
`
`
`
`
`
`
`
`
`as neural networks are not used.
`
`
`
`
`
`
`US. Pat. No. 5,008,946 to Ando uses a complicated set of
`
`
`
`
`
`
`
`
`
`
`rules to isolate the eyes and mouth of a driver and uses this
`
`
`
`
`
`
`
`
`
`
`
`
`information to permit the driver to control the radio, for
`
`
`
`
`
`
`
`
`
`
`example, or other systems within the vehicle by moving his
`
`
`
`
`
`
`
`
`
`
`eyes and/or mouth. Ando uses natural light and illuminates
`
`
`
`
`
`
`
`
`
`only the head of the driver. He also makes no use of trainable
`
`
`
`
`
`
`
`
`
`
`
`
`
`pattern recognition systems such as neural networks, nor is
`
`
`
`
`
`
`
`
`
`there any attempt to identify the contents of the vehicle nor
`
`
`
`
`
`
`
`
`
`
`
`of their location relative to the vehicle passenger compart-
`
`
`
`
`
`
`
`
`ment. Rather, Ando is limited to control of vehicle devices
`
`
`
`
`
`
`
`
`
`by responding to motion of the driver’s mouth and eyes.
`
`
`
`
`
`
`
`
`
`
`US. Pat. No. 5,298,732 to Chen also concentrates in
`
`
`
`
`
`
`
`
`
`locating the eyes of the driver so as to position a light filter
`
`
`
`
`
`
`
`
`
`
`
`
`
`between a light source such as the sun or the lights of an
`
`
`
`
`
`
`
`
`
`
`
`
`
`oncoming vehicle, and the driver’s eyes. Chen does not
`
`
`
`
`
`
`
`
`
`explain in detail how the eyes are located but does supply a
`
`
`
`
`
`
`
`
`
`
`
`calibration system whereby the driver can adjust the filter so
`
`
`
`
`
`
`
`
`
`
`that it is at the proper position relative to his or her eyes.
`
`
`
`
`
`
`
`
`
`
`
`
`
`Chen references the use of an automatic equipment for
`
`
`
`
`
`
`
`
`
`determining the location of the eyes but does not describe
`
`
`
`
`
`
`
`
`
`
`how this equipment works. In any event, there is no mention
`
`
`
`
`
`
`
`
`
`
`
`of monitoring the position of the occupant, other that the
`
`
`
`
`
`
`
`
`
`
`eyes, of determining the position of the eyes relative to the
`
`
`
`
`
`
`
`
`
`
`
`passenger compartment, or of identifying any other object in
`
`
`
`
`
`
`
`
`
`the vehicle other than the driver’s eyes. Also, there is no
`
`
`
`
`
`
`
`
`
`
`
`mention of the use of a trainable pattern recognition system.
`
`
`
`
`
`
`
`
`
`US. Pat. No. 5,305,012 to Faris also describes a system
`
`
`
`
`
`
`
`
`
`
`for reducing the glare from the headlights of an oncoming
`
`
`
`
`
`
`
`
`
`
`vehicle. Faris locates the eyes of the occupant by the use of
`
`
`
`
`
`
`
`
`
`
`
`
`two spaced apart infrared cameras using passive infrared
`
`
`
`
`
`
`
`
`radiation from the eyes of the driver. Again, Paris is only
`
`
`
`
`
`
`
`
`
`
`
`interested in locating the driver’s eyes relative to the sun or
`
`
`
`
`
`
`
`
`
`
`
`oncoming headlights and does not identify or monitor the
`
`
`
`
`
`
`
`
`
`occupant or locate the occupant relative to the passenger
`
`
`
`
`
`
`
`
`
`compartment or the airbag. Also, Faris does not use trainable
`
`
`
`
`
`
`
`
`
`
`pattern recognition techniques such as neural networks.
`
`
`
`
`
`
`
`Faris, in fact, does not even say how the eyes of the occupant
`
`
`
`
`
`
`
`
`
`
`
`
`
`are located but refers the reader to a book entitled Robot
`
`
`
`
`
`
`
`
`
`
`
`Vision (1991) by Berthold Horn, published by MIT Press,
`
`
`
`
`
`
`
`
`
`Cambridge, Mass. Also, Faris uses the passive infrared
`
`
`
`
`
`
`
`
`radiation rather than illuminating the occupant with active
`
`
`
`
`
`
`
`
`infrared radiation or in general electromagnetic radiation as
`
`
`
`
`
`
`
`
`in the instant invention.
`
`
`
`
`The use of neural networks as the pattern recognition
`
`
`
`
`
`
`
`
`
`technology is central to this invention since it makes the
`
`
`
`
`
`
`
`
`
`
`monitoring system robust, reliable and practical. The result-
`
`
`
`
`
`
`
`ing algorithm created by the neural network program is
`
`
`
`
`
`
`
`
`
`usually only a few lines of code written in the C computer
`
`
`
`
`
`
`
`
`
`
`
`
`language as opposed to typically hundreds of lines when the
`
`
`
`
`
`
`
`
`
`
`techniques of the above patents to Ando, Chen and Paris are
`
`
`
`
`
`
`
`
`
`
`
`implemented. As a result, the resulting systems are easy to
`
`
`
`
`
`
`
`
`
`implement at a low cost making them practical for automo-
`
`
`
`
`
`
`
`
`
`tive applications. The cost of the CCD arrays, for example,
`
`
`
`
`
`
`
`
`
`have been prohibitively expensive until very recently ren-
`
`
`
`
`
`
`
`dering their use for VIMS impractical. Similarly, the imple-
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`19
`
`19
`
`

`

`
`5,845,000
`
`5
`
`mentation of the techniques of the above referenced patents
`
`
`
`
`
`
`
`
`
`requires expensive microprocessors while the implementa-
`
`
`
`
`
`
`tion with neural networks and similar trainable pattern
`
`
`
`
`
`
`
`
`recognition technologies permits the use of low cost micro--
`
`
`
`
`
`
`
`
`
`processors typically costing less than $5.
`
`
`
`
`
`
`The present invention uses sophisticated trainable pattern
`
`
`
`
`
`
`
`recognition capabilities such as neural networks. Usually the
`
`
`
`
`
`
`
`
`data is preprocessed, as discussed below, using various
`
`
`
`
`
`
`
`
`feature extraction. An example of such a pattern recognition
`
`
`
`
`
`
`
`
`
`system using neural networks on sonar signals is discussed
`
`
`
`
`
`
`
`
`
`in two papers by Gonnan, R. P. and Sejnowski, T. J.
`
`
`
`
`
`
`
`
`
`
`
`“Analysis of Hidden Units in a Layered Network Trained to
`
`
`
`
`
`
`
`
`
`
`Classify Sonar Targets”, Neural Networks, Vol. 1. pp. 75—89,
`
`
`
`
`
`
`
`
`
`1988, and “Learned Classification of Sonar Targets Using a
`
`
`
`
`
`
`
`
`
`Massively Parallel Network”,
`IEEE Transactions on
`
`
`
`
`
`
`Acoustics, Speech, and Signal Processing, Vol. 36, No. 7,
`
`
`
`
`
`
`
`
`
`July 1988. Examples of feature extraction techniques can be
`
`
`
`
`
`
`
`
`
`found in US. Pat. No. 4,906,940 entitled “Process and
`
`
`
`
`
`
`
`
`
`Apparatus for the Automatic Detection and Extraction of
`
`
`
`
`
`
`
`
`Features in Images and Displays” to Green et al. Examples
`
`
`
`
`
`
`
`
`
`
`of other more advanced and efficient pattern recognition
`
`
`
`
`
`
`
`
`techniques can be found in US. Pat. No. 5,390,136 entitled
`
`
`
`
`
`
`
`
`
`
`“Artificial Neuron and Method of Using Same and US.
`
`
`
`
`
`
`
`
`
`patent application Ser. No. 08/076,601 entitled “Neural
`
`
`
`
`
`
`
`Network and Method of Using Same” to Wang, S. T. Other
`
`
`
`
`
`
`
`
`
`
`
`examples include US. Pat. Nos. 5,235,339 (Morrison et a1),
`
`
`
`
`
`
`
`
`
`5,214,744 (Schweizer et a1), 5,181,254 (Schweizer et a1),
`
`
`
`
`
`
`
`
`and 4,881,270 (Knecht et a1). All of the above references are
`
`
`
`
`
`
`
`
`
`
`
`included herein by reference.
`
`
`
`
`4. Optics
`
`
`Optics can be used in several configurations for monitor»
`
`
`
`
`
`
`
`
`ing the interior of a passenger compartment of an automo-
`
`
`
`
`
`
`
`
`bile. In one known method, a laser optical system uses a
`
`
`
`
`
`
`
`
`
`
`
`GaAs infrared laser beam to momentarily illuminate an
`
`
`
`
`
`
`
`
`object, occupant or child seat, in the manner as described
`
`
`
`
`
`
`
`
`
`
`and illustrated in FIG. 8 of the copending patent application
`
`
`
`
`
`
`
`
`
`
`Ser. No. 08/040,978 cross-referenced above. The receiver
`
`
`
`
`
`
`
`can be a charge coupled device or CCD, (a type of TV
`
`
`
`
`
`
`
`
`
`
`
`
`camera) to receive the reflected light. The laser can either be
`
`
`
`
`
`
`
`
`
`
`
`used in a scanning mode, or, through the use of a lens, a cone
`
`
`
`
`
`
`
`
`
`
`
`
`of light can be created which covers a large portion of the
`
`
`
`
`
`
`
`
`
`
`
`object. In these configurations, the light can be accurately
`
`
`
`
`
`
`
`
`
`controlled to only illuminate particular positions of interest
`
`
`
`
`
`
`
`
`within the vehicle. In the scanning mode, the receiver need
`
`
`
`
`
`
`
`
`
`
`only comprise a single or a few active elements While in the
`
`
`
`
`
`
`
`
`
`
`case of the cone of light, an array of active elements is
`
`
`
`
`
`
`
`
`
`
`
`
`needed. The laser system has one additional significant
`
`
`
`
`
`
`
`
`advantage in that the distance to the illuminated object can
`
`
`
`
`
`
`
`
`
`
`be determined as disclosed in the 08/040,978 patent appli—
`
`
`
`
`
`
`
`
`cation.
`
`In a simpler case, light generated by a non-coherent light
`
`
`
`
`
`
`
`
`
`emitting diode device is used to illuminate the desired area.
`
`
`
`
`
`
`
`
`
`
`In this case, the area covered is not as accurately controlled
`
`
`
`
`
`
`
`
`
`
`
`and a larger CCD array is required. Recently, however, the
`
`
`
`
`
`
`
`
`
`
`cost of CCD arrays has dropped substantially with the result
`
`
`
`
`
`
`
`
`
`
`that this configuration is now the most cost effective system
`
`
`
`
`
`
`
`
`
`
`for monitoring the passenger compartment as long as the
`
`
`
`
`
`
`
`
`
`distance from the transmitter to the objects is not needed. If
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`this distance is required,
`then either the laser system, a
`stereographic system, a focusing system, or a combined
`
`
`
`
`
`
`
`
`ultrasonic and optic system is requ