(12)
`
`United States Patent
`Lemelson et al.
`
`(10) Patent N0.:
`(45) Date of Patent:
`
`US 6,553,130 B1
`Apr. 22, 2003
`
`US006553130B1
`
`(54) MOTOR VEHICLE WARNING AND
`CONTROL SYSTEM AND METHOD
`
`(76) Inventors: Jerome H. Lemelson, 868 Tyner Way,
`Incline Village, NV (US) 89450;
`3232:? tgzdfégénégjfss GlenEagle’
`’
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`U_S_C_ 154(k)) by 1543 days_
`
`( * ) Notice:
`
`(21) Appl' NO" 08/671’853
`(22) Filed:
`Jun. 28, 1996
`
`Related U-S- Application Data
`
`(63) Continuation of application No. 08/105,304, ?led on Aug.
`11, 1993, now abandoned.
`7
`Int. Cl. ................................................ ..
`(52) US. Cl. ..................... .. 382/104; 340/435; 340/436;
`340/903; 382/106
`Of Search ............................... ..
`348/115, 116, 118, 119, 135, 139, 140,
`142, 148, 149; 382/104, 215, 209, 106,
`156, 103, 217, 218; 340/907> 435, 903,
`901, 436; 180/168, 169, 167, 271, 274,
`275; 364/42402, 460, 461, 436—437, 425-04,
`42501, 42601, 443, 424032; 395/905,
`900, 913
`
`(56)
`
`References Cited
`Us PATENT DOCUMENTS
`
`3/1981 GOOdIICh ..................... ..
`4,257,703 A
`476267850 A 12/1986 Chey ~~~~~~~ ~~
`340/903
`4’825’211 A
`4/1989 Park "" "
`340/901
`4,872,051 A 10/1989 Dye ......................... .. 340/903
`(List Continued on next page)
`
`FOREIGN PATENT DOCUMENTS
`
`JP
`JP
`JP
`
`4-219900
`5424529
`5-143897
`
`8/1992
`5/1993
`6/1993
`
`............... .. 340/903
`
`............... .. 340/903
`
`OTHER PUBLICATIONS
`Abu—Mostafa, “Information Theory, Complexity, and Neu
`ral Networks,” IEEE Communications, pp. 25—28 (Nov.
`1989)‘
`Aggarwal et al., “On the Computation of Motion from
`Sequences of Images—A Review,” Proceedings of the
`IEEE, pp. 917—35 (Aug. 1988).
`(List continued on neXt page.)
`
`Primary Examiner—Timothy M. Johnson
`(74) Attorney, Agent, or Firm—Louis J. Hoffman
`
`(57)
`
`ABSTRACT
`
`Asystem and method assists the driver of a motor vehicle in
`preventing accidents or
`the effects of Same. In
`one form, a television Camera is mounted on a Vehicle and
`scans the roadway ahead of the vehicle as the vehicle travels.
`Continuously generated Video picture signals Output
`the
`camera are electronically processed and analyzed by an
`image analyzing computer, which generates codes that serve
`to identify obstacles. A decision computer mounted in the
`controlled vehicle receives such code signals along with
`code signals generated by the speedometer or one or more
`sensors sensing steering mechanism operation and generates
`control signals. Such code signals may be displayed, and a
`synthetic speech or special sound generating and warning
`means used, to warn the driver of the vehicle of approaching
`and eXisting'haZards. The system'may also use the control
`slgnals, partlcularly through appllcatlon of fuZZy loglc, to
`Control the Operation of the brakes and steering mechanism
`of the vehicle to avoid or lessen the effects of a collision. In
`a particular form, the decision computer may select the
`evasive action taken from a number of Choices, depending
`on whether and where the detection device senses other
`vehicles or obstacles.
`
`JP
`
`4-15799
`
`1/1992
`
`............... .. 340/905
`
`55 Claims, 13 Drawing Sheets
`
`HAZARD
`COLLISION
`VECTOR
`
`COLLISION
`VECTOR
`
`DISTANCE FUZZV
`MEMBERSHIP
`
`VELOCITV FUZZI
`MEMBERSHIP
`
`ACCELERATION
`FUZZY MEMBERSHIP
`
`ACCESS FUZZV
`ASSOCIATIVE
`MEMORY
`(FAM)
`
`I
`
`GENERATE OUTPUT
`EXPER'T CONTROL
`SIGNALS
`
`WARNING BRAKE STEERING
`
`1
`
`Mercedes-Benz USA, LLC, Petitioner - Ex. 1002
`
`

`

`US 6,553,130 B1
`Page 2
`
`US. PATENT DOCUMENTS
`
`4,901,362 A * 2/1990 Terzian ..................... .. 382/103
`4,926,170 A
`5/1990 Beggs et al. .... ..
`340/904
`4,931,937 A * 6/1990 Kakinami et al.
`180/169
`4,933,852 A
`6/1990 Lemelson ....... ..
`340/439
`4,969,038 A 11/1990 Lemelson
`358/107
`4,979,029 A 12/1990 Lemelson .................. .. 358/93
`5,018,689 A
`5/1991 Yasunobu et al. ........ .. 395/905
`5,039,217 A
`8/1991 Maekawa et al. ............ .. 356/1
`5,081,585 A
`1/1992 Kurami et al.
`.. 364/424.02
`
`5,091,726 A
`
`2/1992 Shyu . . . . . . . . . . . .
`
`. . . . . .. 340/904
`
`6/1992 Beggs et al. .............. .. 340/904
`5,122,796 A
`9/1992 Zechnall ................... .. 340/905
`5,146,219 A
`5,161,107 A 11/1992 Mayeaux et al. .
`340/937
`5,164,904 A 11/1992 Summer ......... ..
`340/905
`5,177,462 A
`1/1993 Kajiwara
`340/903
`5,179,377 A
`1/1993 Hancock
`340/961
`5,189,619 A
`2/1993 Adachi et al.
`395/905
`5,197,562 A * 3/1993 Kakinami et al.
`180/169
`5,230,400 A
`7/1993 Kakinami et al.
`180/169
`
`5,249,157 A
`
`9/1993 Taylor . . . . . . . . . . . . . .
`
`. . . . . .. 340/903
`
`.. 364/424.05
`1/1994 Bottesch ..... ..
`5,276,620 A
`..... .. 364/461
`1/1994 IiZuka et al.
`5,278,764 A
`340/468
`3/1994 Tsai ........... ..
`5,298,882 A
`340/903
`4/1994 Maekawa ..
`5,304,980 A
`180/167
`4/1994 Saneyoshi
`5,307,136 A
`180/169
`5/1994 Shaw et al.
`5,314,037 A
`340/459
`7/1994 Kohsaka ..... ..
`5,327,117 A
`180/169
`7/1994 Butsuen et al.
`5,332,057 A
`340/903
`8/1994 Abst et al. ...... ..
`5,339,075 A
`367/96
`8/1994 O’Brien et al.
`5,341,344 A
`180/169
`5,357,438 A 10/1994 Davidian .... ..
`..... .. 340/903
`5,369,591 A 11/1994 Broxmeyer
`5,545,960 A * 8/1996 Ishikawa ........... .. 364/424.032
`
`OTHER PUBLICATIONS
`
`Alspector, “Neural—Style Microsystems that Learn,” IEEE
`Communications, pp. 29—36 (Nov. 1989).
`Casasent, “Optics and Neural Nets,” Chapter 16 in Carpen
`ter et al., eds., Neural Networks for Vision and Image
`Processing, pp. 437—48, (MIT Press 1992).
`Cox, “Fuzzy Fundamentals,” IEEE Spectrum, pp. 58—61
`(Oct. 1992).
`Hammerstrom, “Neural Networks at Work,” IEEE Spec
`trum, pp. 26—32 (Jun. 1993).
`Hush et al., “Progress in Supervised Neural Networks,”
`IEEE Signal Processing, pp. 8—39 (Jan. 1993).
`Jurgen, “Smart Cars and Highways Go Global,” IEEE
`Spectrum, pp. 26—36 (May 1991).
`Kittler et al., eds., Image Processing System Architectures,
`Chapter 4, pp. 49—81, and Chapter 5, pp. 85—101 (John
`Wiley & Sons 1985).
`Kosko et al., “Fuzzy Logic,” Scienti?cAmerican, pp. 76—81
`(Jul. 1993).
`
`Lee, “Fuzzy Logic in Control Systems: Fuzzy Logic Con
`troller, Part II,” IEEE Transaction on Systems, Man, and
`Cybernetics, pp. 419—435 (vol. 20, No. 2, Mar./Apr. 1990).
`Lippmann, “An Introduction to Computing with Neural
`Nets,” IEEE ASSP, pp. 4—22 (Apr. 1987).
`Lippmann, “Pattern Classi?cation Using Neural Networks,”
`IEEE Communications, pp. 47—50, 59—64 (Nov. 1989).
`Lisboa, ed., Neural Networks—CurrentApplications, Chap
`ter 1, pp. 1—34, Chapter 2, pp. 35—48, Chapter 7, pp.
`123—147 (Chapman & Hall 1992).
`Lupo, “Defense Applications of Neural Networks,” IEEE
`Communications, pp. 82—88 (Nov. 1989).
`Maresca et al., “Parallel Architectures for Vision,” Proceed
`ings of the IEEE, pp. 970—981 (vol. 76, No. 8, Aug. 1988).
`Nijhuis et al., “Evaluation of Fuzzy and Neural Vehicle
`Control,” Institution of Electrical Engineers, pp. 447—452
`(1992).
`Pearson, ed.,Image Processing, Chapter 8, pp. 141—155, and
`Chapter 10, pp. 169—189 (McGraw—Hill 1991).
`Psaltis et al., “Optoelectronic Implementations of Neural
`Networks,” IEEE Communications, pp. 37—40, 71 (Nov.
`1989).
`Roth, “Neural Networks for Extraction of Weak Targets in
`High Clutter Environments,” IEEE Transactions on Sys
`tems, Man, and Cybernetics, pp. 1210—1217 (Sep./Oct.
`1989).
`Schwartz et al., “Fuzzy Logic Flowers in Japan,” IEEE
`Spectrum, pp. 32—35 (Jul. 1992).
`Soucek et al., Neural and Massively Parallel Computers,
`Chapter 12, pp. 245—276 (John Wiley & Sons 1988).
`Suaya et al., eds., VLSI and Parallel Computation, Chapter
`1, pp. 1—84, and Chapter 5, pp. 390—415 (Morgan Kaufmann
`1990).
`Teuber, Digital Image Processing, Chapter 1, pp. 1—30,
`Chapter 2, pp. 31—70, and Appendix D, pp. 254—255 (Pren
`tice Hall 1993).
`Wasserman, Neural Computing." Theory and Practice, Chap
`ter 1, pp. 11—26, Chapter 2, pp. 27—42, Chapter 3, pp. 43—59,
`and Chapter 9, pp. 151—166 (Van Nostrand Reinhold 1989).
`Yuhas et al., “Integration of Acoustic and Visual Speech
`Signals Using Neural Networks,” IEEE Spectrum, pp. 65—71
`(Nov. 1989).
`Shekhar et al., Design and Validation of Head up Displays
`for Navigation in IVHS, VNIS’QZ, Oct. 1991, pp. 537—542.
`“NHTSA IVHS Plan,” National Highway Safety Adminis
`tration US. Department of Transportation, Jun. 12, 1992.
`Bosacchi et al, Fuzzy Logic Technology & the Intelligent
`Highway System (IHS); IEEE, 1993, pp. 65—70.
`Rock et al., “Intelligent Road Transit: The Next Generation,”
`AI EXPERT, Apr. 1994, pp. 16—24 (not prior art).
`* cited by examiner
`
`2
`
`

`

`U.S. Patent
`
`Apr. 22, 2003
`
`Sheet 1 0f 13
`
`US 6,553,130 B1
`
`16'- 17w-E 18\
`TV CAMERA -> MD -
`
`14\
`1 2\ 1 3\
`RAM ROM ngoékgé??n
`LI
`II
`I I
`->
`
`19\
`
`1 5 /10
`MEM
`
`[20
`
`36\
`sTEERING i7
`SERVOISI is
`41 \
`HEAD LIGHT
`CONTROL
`42
`WAFRIING LIGHT
`coNTRoL
`‘
`
`40
`
`A/D ‘
`
`_
`
`D/A
`
`~
`
`J
`33\ 16
`35
`BRAKE sERvo —%]~
`34, MAN
`ovER
`
`51
`
`39
`
`I
`8
`8
`E
`o
`5
`g
`E
`E
`E
`O
`3
`43\
`2
`HORN A
`CONTROL @ E
`0
`44 \ w 0
`DIGITAL
`SPEEDOMETER
`46\
`45*
`AccELERoNIETER(s) I NO I
`
`:|_-L sI-IoRT WAVE __ ND _,
`RAE
`
`TRANsIvIITr R
`86
`
`FIG. 1
`
`_, IMAGE ANALYZING _,
`<- COMPUTER <- MEM
`I
`
`/22
`/ 21
`II
`_.
`RANGE
`_.
`‘- COMPUTER <- MEM
`
`,23 ,24
`_.
`_>
`DECISION
`<— COMPUTER
`<— MEM
`/25
`/26
`SPEECH
`—- REcoGNITIoN : MEM
`COMPUTER
`~25A
`<- Am‘
`
`25U
`1
`
`2s
`
`,27
`_ SYN SPEECH SIGNAL _
`— GEN COMPUTER - MEN‘
`/29
`/3()
`
`“’ 0”‘
`SPKH \
`,31
`DISPLAY DRNER
`“W
`MEM
`'" @l
`
`[32
`
`as
`
`82
`
`INTERFACE <— KEYBOARD
`
`3
`
`

`

`U.S. Patent
`
`Apr. 22, 2003
`
`Sheet 2 0f 13
`
`US 6,553,130 B1
`
`16\ H417 /18
`[:cAMEHA ND
`
`/19
`
`,52
`
`VIDEO
`PREPROCESSOR
`
`
`
`IMAGE MEMORY
`
`CO-PROCESSOR
`
`,55
`DISPLAY
`
`D/A
`
`IMAGE BUS
`
`K57
`RAM
`
`[56
`CONTROL
`PROCESSOR
`
`,58
`ROM
`
`,59
`BUS INTERFACE
`
`[11
`
`MICRO PROCESSOR
`CONTROLLER
`
`FIG. 2
`
`4
`
`

`

`U.S. Patent
`
`Apr. 22, 2003
`
`Sheet 3 0f 13
`
`US 6,553,130 B1
`
`INPUTS
`
`FIR
`
`IDDEN
`ER
`
`SECOND HIDDEN
`LAYER
`
`OUTPUT
`LAYER
`
`62
`
`I@ .
`
`64
`
`65
`
`FIG. 3
`
`1
`
`71
`WO=Q
`
`70
`
`Y
`
`r72
`__.
`U
`
`f
`
`FIG. 4
`
`68 \ 1
`X1 >w
`
`X
`
`><
`
`\W2
`/
`.
`
`.
`'
`|\Wn
`1/
`
`5
`
`

`

`U.S. Patent
`
`Apr. 22, 2003
`
`Sheet 4 0f 13
`
`US 6,553,130 B1
`
`IMAGE DATA
`
`1
`
`r74
`
`IMAGE
`PROCESSOR
`
`73\ ‘
`
`u
`
`l
`
`,
`
`IMAGE DATA BUS /75
`
`k
`
`,
`
`A
`
`"
`
`VIRTUAL
`PE
`
`VIRTUAL
`PE
`
`VIRTUAL
`PE
`
`VIRTUAL
`PE
`
`FIG. 5
`
`6
`
`

`

`U.S. Patent
`
`Apr. 22, 2003
`
`Sheet 5 0f 13
`
`US 6,553,130 B1
`
`DATA
`FILE
`
`WEATHER
`MOTION
`IMAGE
`RADAR
`VEHICLE OVERRIDE
`
`DETECTION SIGNAL ANALYSIS
`
`/_7 4
`
`7 V
`
`75
`
`N
`
`HAZARD
`
`‘
`
`Y
`
`[76
`
`FUZZY ASSOCIATIVE MEMORY (FAM) SELECTION
`
`, r77
`
`[78
`
`[79
`
`/so
`
`FAM-1
`
`FAM-2
`
`FAM-3 - o - FAM-N
`
`,
`
`/81
`
`7
`
`CONTROL SIGNAL GENERATOR
`DEFUZZIFIER
`
`STEER|NG\
`
`w’WARNlNGS IBRAKING
`
`V
`
`V
`
`—>STEERING SERVO
`MICROPROCESSOR CONTROLLER —>BRAK|NG SERVO
`—->DISPLAYS
`
`FIG. 6
`
`7
`
`

`

`U.S. Patent
`
`Apr. 22, 2003
`
`Sheet 6 0f 13
`
`US 6,553,130 B1
`
`II
`
`VERY
`CLOSE
`
`A
`
`CLOSE
`
`MEDIUM
`
`FAR
`
`VERY FAR
`
`DISTANCE
`
`VERY LOW LOW
`
`MEDIUM
`
`HIGH VERY HIGH
`
`VELOCITY
`
`HN
`
`MN
`
`LN
`
`n
`LP
`
`1
`
`MP
`
`HP
`
`MEMBERSHIP
`
`MEMBERSHIP
`
`MEMBERSHIP
`
`ACCELERATION
`
`FIG. 7
`
`8
`
`

`

`U.S. Patent
`
`Apr. 22, 2003
`
`Sheet 7 0f 13
`
`US 6,553,130 B1
`
`A
`
`&
`5 1
`E
`m
`2
`LIJ
`2
`
`A
`
`GREEN
`
`YELLOW
`
`RED
`
`WARNING LEVEL
`
`Low BRAKE
`
`MEDIUM BRAKE
`
`HIGH BRAKE
`,
`
`BRAKING PREssuRE
`
`p
`
`LOW G
`
`MEDIUM 6
`
`HIGH z
`
`B
`,1},
`E 1
`CD
`2
`l.Ll
`2
`
`A
`
`&
`55 1
`E
`III
`2
`LL]
`2
`
`STEERING ANGLE
`
`FIG. 8
`
`9
`
`

`

`U.S. Patent
`US. Patent
`
`22,
`m
`Apr. 22, 2003
`
`e
`S
`3
`Sheet 8 0f 13
`
`US 6,553,130 B1
`US 6,553,130 B1
`
`as?a...2.2.E2
`
`
`
`82.9..-."—GméIiG2é2$szqiGzéqcr0>
`
`
`
`
`
`Q2.m...>G2.m.._.>Q2.m2.0&Z.mz.mv832.00
`
`
`
`QwéiiGw.m...>82.93;sz2.>QZéz.>0
`
`1.
`
`2..u20.k<¢M4m00<A...u20F<mw4m00<
`
`
`
`
`
`9.3-:9.9;9.9;9.9;cm99;9-9-099-09.9-099-0:>9-9-:99.;9.9;99.599.6:>m9.9;E9.9....a9.9.39.3.:
`
`I>I
`
`_I
`
`
`III
`
`IIII2:
`
`
`
`
`
`
`
`
`82é2-082.mz.mv32.sz Gz.m2.6Qzéz.0Qz.m2.0azézéQZéZ-Gn.>EQz.m._.>
`
`
`
`GZ.MZ.G$2.szEZ-flZ.0832.0BzdziQ2.mz-0$32-6EzfiszGzé2-0n.> 82.m2$
`
`
`
`
`
`
`
`
`EQw-m2.>82-m2.>Q2.m2-082:02-0
`
`Z:nZO_._.<Iw._mOO<22nZO_._.<Ew4mOU<
`
`10
`
`
`
`
`
`
`
`
`
`
`
`mu.9.9-:99......99;9.9-:99-599-:99.:912.:9.9.:9.3;
`
`
`
`
`
`
`
`m.:2.zop<:m-_m_oo<n:.29:19:09M.9.3;9-3;9.9;9.9.09.9.0E9.3;9.3;9.9.09.9.o9.9.:>
`
`
`
`
`
`a
`
`
`
`99-:99;99;9.9.09.173I9.3;99;9&2-5Eu.
`
`
`@2.m2-mEHIEIEVA
`
`
`
`
`$1.93.:Efimbfii81.9.2".82.2.3:GméfiiQzfifiiEz.m...>IGIfiI-I
`
`
`
`GfiéxiGméxiQwéiiQzé+>Qzéz.
`
`
`
`
`82.01%.Q2.m:.mQmfiziGmégi834$O>
`
`
`
`
`
`Emir—iG2.mr-¢Q2-ms.-m$2.0...)82.n2.>E
`
`
`
`
`QméziGz-m2-mSz-m4.>Qz-m2.>azéz-
`
`Ig
`
`I:
`
`0
`
`a .UEA
`
`oHIHIHEIIIH
`
`
`
`
`
`
`
`
`
`
`
`
`89-:99.:9-9;9.9;99;0EE9.9.:9.9.:9.9-:oa.OHHH99-:9-9.:9.9-:99;99;o>9.11.:9.11.:9.11.:9-11.:9.9.:o>
`
`KSIIHII Zillflll EQIIHII
`
`10
`
`
`
`
`
`
`

`

`U.S. Patent
`
`Apr. 22, 2003
`
`Sheet 9 0f 13
`
`US 6,553,130 B1
`
`HF
`
`HB
`
`HL
`
`HH
`
`OF
`
`OB
`
`/82
`OL
`OR
`
`FIG. 1O
`
`D'STANCE
`
`RELATIVE
`VELOCITY
`
`RELATIVE
`ACCELERATION
`
`/83
`
`FIG. 11
`
`11
`
`

`

`U.S. Patent
`
`Apr. 22, 2003
`
`Sheet 10 0f 13
`
`US 6,553,130 B1
`
`F Q Q F F Q 8 Q Q Q Q F Q a Q Q F e F Q 3 mo .6 mo "6 m: ._I m: “=._ EoFow>
`
`
`F F Q F F Q 8 F Q Q F Q Q Q Q Q F Q F F Q .5 Q Q Q F Q Q Q N
`
`F F F Q Q Q F 2. F F Q Q F Q Q QF Q F F Q Q Q F FF. Q F Q Q F Q Q NF F o F o o o F 9‘. F c o o F o o FF
`F F Q Q Q Q F 3 F F F F Q Q Q Q Q F Q Q Q Q F 9 Q F F F Q Q Q Q F Q Q Q Q Q F NF‘ F Q F F Q Q Q F Q Q Q Q Q Q F FF. Q Q F F Q Q Q Q
`o o o o o F 3 o o o o F o 0 e
`
`F F F F o 3 F F o F o o o m C F F F 0 mm o F O F C O 0 v
`
`F F F O o F mm F F F o F c o F F Q F Q Q F Fm Q F F Q F Q Q QF F o F o o F on F o F o F c o m F o O F O O F 0.? O O F O F O O Q F
`
`F F F Q F Qm Q F Q Q Q F Q vw Q F F Q F 8 F Q Q Q Q F Q 8
`
`F F Q F F mm QQ
`
`F F Q F Q F B Q Q Q Q Q F Q “Q Q F Q F Q F 8 F F F F Q Q Fm F a o F o F mm o F F F o o om Q Q Q F Q F E F Q F F Q Q QF
`
`F F Q Q F F 8 Q F F Q Q F Q QN O F O O F F N0 F O F O O F 0 MN F o o o F F F0 0 o F o o F 0 mm Q Q Q Q F F 8 F F Q Q Q F Q 8
`
`N F M95
`
`Q F Q F F 3 F F F Q Q F Q QN
`
`F o F F F WM F O F o F 0 NM
`Q Q F F F 8 Q Q F Q F Q FQ
`
`F F F F F F o F 0 mm
`
`F mm 0 O F F O o m F
`
`FF. % F F Q e F Q 3
`
`1
`
`12
`
`

`

`U.S. Patent
`
`Apr. 22, 2003
`
`Sheet 11 0f 13
`
`US 6,553,130 B1
`
`1
`SCANNER INPUT
`DATA FILE
`
`74
`/
`
`>
`
`/99
`MODIFY HAZARD
`STATE VECTOR
`M
`/98
`RANK COLLISION
`HAZARDS
`
`1 00
`
`7 OF N
`
`Y 101
`OF=1
`
`102 ‘
`OB
`
`103
`
`Y
`
`05:1
`
`/82
`v
`[HFIHBIHLIHRIOFIOBIOLIORI
`
`FIG. 13
`
`13
`
`

`

`U.S. Patent
`
`Apr. 22, 2003
`
`Sheet 12 0f 13
`
`US 6,553,130 B1
`
`108\
`
`I
`
`INPUT DATA
`COLLISION
`STATE VECTOR / F|LE
`
`109\
`
`II
`
`110\
`
`FUZZY
`DISTANCE
`MEMBERSHIP
`
`II
`
`FUZZY
`VELOCITY
`MEMBERSHIP
`
`112\
`
`FUZZY
`ACCELERATION
`MEMBERSHIP
`
`II
`HAZARD/OBJECT
`STATE VECTOR
`ADDRESS
`TRANSLATION
`
`113\
`
`"
`
`FAM OUTPUT
`ADDRESS
`
`FIG. 14
`
`/76
`
`14
`
`

`

`U.S. Patent
`
`Apr. 22, 2003
`
`Sheet 13 0f 13
`
`US 6,553,130 B1
`
`INPUT DATA FILE
`
`HAZARD
`COLLISION
`VECTOR
`
`Y MULTIPLE
`
`OBJECT FRONT
`
`OBJECT BACK
`
`OBJECT LEFT
`
`OBJECT RIGHT
`
`1 0 0 O 1 0 O 1
`
`D V A
`
`HAZARD/OBJECT
`VECTOR
`
`V
`
`v
`
`ACCESS FUZZY
`ASSOCIATIVE
`MEMORY
`(FAM)
`
`'
`
`GENERATE OUTPUT
`EXPERT CONTR L
`SIGNALS o
`
`COLLISION
`VECTOR
`
`"
`
`DISTANCE FUZZY
`MEMBERSHIP
`
`V
`VELOCITY FUZZY
`MEMBERSHIP
`
`V
`
`ACCELERATION
`FUZZY MEMBERSHIP
`
`WARNING BRAKE STEERING
`
`FIG. 15
`
`15
`
`

`

`US 6,553,130 B1
`
`1
`MOTOR VEHICLE WARNING AND
`CONTROL SYSTEM AND METHOD
`
`CROSS-REFERENCE TO RELATED
`APPLICATION
`
`This application is a continuation of Ser. No. 08/105,304,
`?led Aug. 11, 1993, abandoned.
`
`FIELD OF THE INVENTION
`
`This invention relates to a system and method for oper
`ating a motor vehicle, such as an automobile, truck, aircraft
`or other vehicle, Wherein a computer or computerized sys
`tem is employed to assist and/or supplement the driver in the
`movement of the vehicle along a path of travel, such as a
`street or roadWay and may be used to avoid obstacles and
`accidents.
`
`15
`
`BACKGROUND OF THE INVENTION
`
`A major cause of human suffering is automobile acci
`dents. Approximately 49,000 people die in traf?c accidents
`each year in the United States, and another three million are
`injured. The costs of death and injury accidents are stagger
`ing. According to the United States National HighWay
`Traf?c Safety Administration, crash damage and medical
`bills total $137 billion a year.
`Automobile designers offer many safety features, includ
`ing passenger restraints, improved braking systems, and
`body designs, intended to better protect automobile crash
`victims. But very little has been done in the area of auto
`matic vehicle control systems based on modern electronics,
`computer systems, and advanced real-time softWare. This is
`true despite rapidly increasing capabilities in these technolo
`gies and pervasive application in many other areas
`including, for example the business, entertainment, and
`medical ?elds. Vehicle guidance and control technology has,
`of course, been applied With great success in military
`defense systems, avionics systems and space exploration
`systems. But, this technology is costly and has not been
`commercialiZed.
`The opportunity exists today to develop cost effective,
`commercial automated vehicle control systems. NeW
`advances in loW-cost hardWare and softWare technology
`make implementation feasible. High-speed, parallel com
`puter architectures, specialiZed image-processing
`equipment, and advanced special computers such as math
`coprocessors are available. Advanced expert system imple
`mentations based on concepts such as fuZZy logic and neural
`netWorks, and neW, improved scanning systems for sensing
`environments around moving vehicles make it very timely,
`indeed, to pursue neW approaches.
`Work on these problems has begun. Intelligent vehicle/
`highWay systems are being investigated With traffic control
`systems intended to minimiZe congestion. Vehicle location
`systems such as GPS (Global Positioning System) and route
`guidance systems are also being pursued. Certain systems
`for automated vehicle control have been proposed, including
`systems that scan the roadWay directly ahead of a vehicle
`using radar/lidar or television and attempt to Warn a driver
`of impending danger. FuZZy logic expert systems for con
`trolling vehicle speed (braking and throttle) based on scan
`ning the roadWay ahead of a vehicle have been described.
`Road tracking With electronic vehicle guidance is being
`pursued. FuZZy logic has been applied to braking systems in
`subWay and train systems.
`While these developments are important, they fail to
`protect vehicles from many types of collisions or minimiZe
`
`25
`
`35
`
`45
`
`55
`
`65
`
`2
`the damage therefrom. More particularly, such systems fail
`to exercise simultaneous, coordinated control over vehicle
`steering and speed, fail to take full advantage of identi?ca
`tion of different obstacle or haZard types using standard
`stored models of production vehicles and other commonly
`encountered roadWay objects, fail to deal effectively With
`objects and haZards located simultaneously on different
`sides of the vehicle, and fail to capitaliZe fully on modern
`expert system decision and control technology, such as
`represented by fuZZy logic and neural netWork methods, to
`deal With more complex haZardous situations.
`
`SUMMARY OF THE INVENTION
`In a preferred form of the invention, a video scanning
`system, such as a television camera and/or one or more laser
`scanners mounted on the vehicle scan the road in front of the
`vehicle and generate image information Which is computer
`analyZed per se or in combination With a range sensing
`system to Warn the driver of haZardous conditions during
`driving by operating a display, such as a heads-up display,
`and/or a synthetic speech generating means Which generates
`sounds or Words of speech to verbally indicate such road
`conditions ahead of the vehicle.
`The preferred form of the invention provides audible
`and/or visual display means to cooperate in indicating to the
`driver of a motor vehicle both normal and haZardous road
`conditions ahead as Well as driving variables such as dis
`tances to stationary objects, and other vehicles; the
`identi?cation, direction of travel and speed of such other
`vehicles, and the identi?cation of and distances to stationary
`or sloWly moving objects such as barriers, center islands,
`pedestrians, parked cars poles, sharp turns in the road and
`other conditions. In addition, the image analyZing computer
`of the vehicle may be operated to scan and decode coded
`and/or character containing signs or signals generated by
`indicia or code generating other devices Within or at the side
`of the road and indicating select road and driving conditions
`ahead.
`The computer is operable to analyZe video and/or other
`forms of image information generated as the vehicle travels
`to identify obstacles ahead of the vehicle and, in certain
`instances, quantify the distance betWeen the vehicle con
`taining same on the basis of the siZe of the identi?ed vehicle
`or object and/or by processing received pulse-echo signals.
`Using such identifying information and comparing it With
`information on the shapes and siZes of various objects such
`as rear and front pro?les of all production vehicles and the
`like and their relative siZes or select dimensions thereof,
`indications of distances to such objects may be computed
`and indicated as further codes. When the closing distance
`becomes haZardous, select vehicle subsystems may be auto
`matically controlled by the computer as it continues to
`analyZe image signals generated by the television camera. A
`?rst subsystem generates a ?rst select code or codes Which
`controls an electronic display, such as a heads-up display to
`cause it to display a Warning indication, such as one or more
`?ashing red light portions of the display or other lighted
`effect. For example, the display may project on the Wind
`shield or dashboard such information as images of the
`controlled vehicle and other vehicles in and adjacent its path
`of travel and relative distances thereto as Well as groups of
`characters de?ning same, colored and ?ashing Warning
`lights and the like for pre-Warning and Warning purposes. A
`second subsystem generates a code or series of codes Which
`control a sound generating means Which generates a select
`sound such as a horn, buZZing sound and/or select synthetic
`speech Warning of the haZardous condition detected and, in
`
`16
`
`

`

`US 6,553,130 B1
`
`3
`certain instances, generating sounds of select Words of
`speech Which may Warn of same and/or suggest corrective
`action (s) by the vehicle operator or driver to avoid an
`accident.
`A third subsystem comes on-line and generates one or
`more codes Which are applied to at least partly effect a
`corrective action such as by pulsing one or more motors or
`solenoids to apply the brakes of the vehicle to cause it to
`sloW doWn. If necessary to avoid or lessen the effects of an
`accident, the third subsystem stops the forWard travel of the
`vehicle in a controlled manner depending on the relative
`speeds of the tWo vehicles, and/or the controlled vehicle and
`a stationery object or structure and the distance therebe
`tWeen.
`A fourth subsystem, Which may be part of or separate
`from the third subsystem may generate one or more codes
`Which are applied to either effect partial and/or complete
`control of the steering mechanism for the vehicle to avoid an
`obstacle and/or lessen the effect of an accident. Either or
`both the third or fourth subsystem may also be operable to
`control one or more safety devices by controlling motors,
`solenoids or valves, to operate a restraining device or
`devices for the driver and passenger(s) of the vehicle, such
`as a safety belt tightening means, an air bag in?ation means
`or other device designed to protect human beings in the
`vehicle.
`The second, and/or third and fourth subsystems may also
`be operable to effect or control the operation of additional
`Warning means such as the horn, headlights and/or other
`Warning lights on the vehicle or other Warning means Which
`operates to alert, ?ag or Warn the driver of the approaching
`or approached vehicle or a pedestrian of the approaching
`haZardous condition. One or more of these subsystems may
`also be operable to generate and transmit one or more codes
`to be received and used by the approaching or approached
`vehicle or a roadside device to effect additional on-line
`Warning(s) of the haZardous condition, and/or may be
`recorded on a disc or RAM (random access memory) for
`future analysis, if necessary.
`In a modi?ed form of the invention, the vehicle Warning
`system may also include a short Wave receiving means to
`receive code signals from other vehicles and/or short Wave
`transmitters at the side of or Within the road for controlling
`the visual, audio and/or brake and steering means of the
`vehicle to avoid or lessen the effects of an accident and/or to
`maintain the vehicle in-lane and in proper operating condi
`tion as it travels.
`The systems and methods of this invention preferably
`employ computeriZed image analyZing techniques of the
`types disclosed and de?ned in such patents of mine as US.
`Pat. Nos. 4,969,038 and 4,979,029 and references cited in
`the ?le Wrappers thereof as Well as other more recent patents
`and include the use of knoWn arti?cial intelligence, neural
`networking and fuZZy logic computing electronic circuits.
`While the invention is described herein principally in
`connection With an automobile on a roadWay, it may be used
`in connection With controlling any poWered vehicle, includ
`ing a motor vehicle, a boat, a train, or an aircraft.
`Accordingly it is a primary object of this invention to
`provide a neW and improved system and method for con
`trolling the operation of a poWered vehicle.
`Another object is provide a system and method for
`assisting the driver of a poWered vehicle in controlling its
`operation to avoid an accident or haZardous driving condi
`tion.
`Another object is to provide a system and method
`employing computeriZed image analysis to control or assist
`
`15
`
`25
`
`35
`
`55
`
`65
`
`4
`the driver of a motor vehicle in controlling its operation to
`avoid haZardous conditions such as collisions With other
`vehicles, stationery objects or pedestrians.
`Another object is to provide a computeriZed system and
`method for controlling the speed of travel of a motor vehicle
`to lessen the chances of an accident While being driven by
`a person.
`Another object is to provide a system and method
`employing a television scanning camera mounted on a
`vehicle for scanning the ?eld ahead, such as the image of the
`road ahead of the vehicle and a computer for analyZing the
`image signals generated Wherein automatic image
`intensifying, or infra-red scanning and detection means is
`utiliZed to permit scanning operations to be effected during
`driving at night and in loW light, snoWing or fog conditions.
`Another object is to provide a system and method
`employing a television camera or other video scanning
`means mounted on a moving motor vehicle for scanning,
`detecting and identifying obstacles such as other vehicles
`ahead of such moving vehicle Wherein the video image
`signals are analyZed to determine distances to such objects.
`Another object is to provide a computer controlled safety
`system for a motor vehicle Which employs a television
`camera and an auXiliary scanning means to both identify
`obstacles in the path of the vehicle and determine distance
`therefrom on a real time and continuous basis for use in
`Warning the operator of same and/or in controlling the
`operation of the vehicle to avoid a collision.
`
`BRIEF DESCRIPTION OF DRAWINGS
`The various hardWare and softWare elements used to carry
`out the invention described herein are illustrated in the form
`of block diagrams, ?oW charts, and depictions of neural
`netWork and fuZZy logic algorithms and structures. The
`preferred embodiment is illustrated in the folloWing ?gures:
`FIG. 1 is a block diagram of the overall Motor Vehicle
`Warning and Control System illustrating system sensors,
`computers, displays, input/output devices and other key
`elements.
`FIG.2 is a block diagram of an image analysis computer
`19 of the type that can be used in the Vehicle HaZard
`Avoidance System herein of FIG. 1.
`FIG. 3 illustrates a neural netWork of the type useful in the
`image analysis computer of FIG. 4.
`FIG. 4 illustrates the structure of a Processing Element
`(PE) in the neural netWork of FIG. 3.
`FIG. 5 is an alternate embodiment of a neural netWork
`image processor useful in the system of FIG. 1.
`FIG. 6 is a How diagram illustrating the overall operation
`of the Motor Vehicle Warning and Control System of FIG.
`1.
`
`FIG. 7 illustrates typical input signal membership func
`tions for fuZZy logic algorithms useful in the Motor Vehicle
`Warning and Control System of FIG. 1.
`FIG. 8 illustrates typical output signal membership func
`tions for fuZZy logic algorithms useful in the Motor Vehicle
`Warning and Control System of FIG. 1.
`FIG. 9 illustrates typical FuZZy Associative Memory
`(FAM) maps for the fuZZy logic algorithms useful in the
`Motor Vehicle Warning and Control System of FIG. 1.
`FIG. 10 is a Hazard/Object state vector useful in imple
`menting the FuZZy Logic Vehicle Warning and Control
`System.
`FIG. 11 is a HaZard Collision Control vector useful in
`implementing the FuZZy Logic vehicle Warning and Control
`System.
`
`17
`
`

`

`US 6,553,130 B1
`
`5
`FIG. 12 is a table of HaZard/Object state vectors indicat
`ing possible combinations of hazards and objects useful in
`the FuZZy Associative Memory access system used herein.
`FIG. 13 is a more detailed logic ?oW diagram for the
`analysis of detection signals prior to accessing fuZZy logic
`control structures in the Motor Vehicle Warning and Control
`System.
`FIG. 14 is a more detailed Logic FloW Diagram for the
`FuZZy Associative Memory
`selection processing.
`FIG. 15 is an eXample system ?oW illustrating the opera
`tion of the Motor Vehicle Warning and Control System.
`
`DETAILED DESCRIPTION
`
`In FIG. 1 is shoWn a computeriZed control system 10 for
`controlling the operation of a motor vehicle to prevent or
`lessen the effects of accidents such as collisions With sta
`tionery and/or moving objects such as other vehicles. The
`system 10 employs a control computer or microprocessor 11
`mounted on the vehicle and operable to receive and gate
`digital signals, such as codes and control signals from
`various sensors, to one or more specialiZed computers and
`from such computers to a number of servos such as electric
`motors and lineal actuators or solenoids, sWitches and the
`like, speakers and display drivers to perform either or both
`the functions of audibly and/or visually informing or Warn
`ing the driver of the vehicle of a haZardous road condition
`ahead and/or to effect controlled braking and steering
`actions of the vehicle.
`ARAM 12 and ROM 13 are connected to processor 11 to
`effect and facilitate its operation. A television camera(s) 16
`having a Wide angle lens 16L is mounted at the front of the
`vehicle such as the front end of the roof, bumper or end of
`the hood to scan the road ahead of the vehicle at an angle
`encompassing the sides of the road and intersecting roads.
`The analog signal output of camera 16 is digitiZed in an A/D
`convertor 18 and passed directly to or through a video
`preprocessor 51 to microprocessor 11, to an image ?eld
`analyZing computer 19 Which is provided, implemented and
`programmed using neural netWorks and arti?cial intelli
`gence as Well as fuZZy logic algorithms to (a) identify
`objects on the road ahead such as other vehicles, pedestrians,
`barriers and dividers, turns in the road, signs and symbols,
`etc., and generate identi?cation codes, and (b) detect dis
`tances from such objects by their siZe (and shape) and
`provide codes indicating same for use by a decision
`computer, 23, Which generates coded control signals Which
`are applied through the computer 11 or are directly passed to
`various Warning and vehicle operating devices such as a
`braking computer or drive, 35, Which operates a brake servo
`33, a steering computer or drive(s) 39 and 40 Which operate
`steering servos 36; a synthetic speech signal generator 27
`Which sends trains of indicating and Warning digital speech
`signals to a digital-analog converter 29 connected to a
`speaker 30; a display driver 31 Which drives a (heads-up or
`dashboard) display 32; a head light controller 41 for ?ashing
`the head lights, a Warning light control 42 for ?ashing
`external and/or internal Warning lights; a horn control 43,
`etc.
`Adigital speedometer 44 and accelerometer(s) 45 provide
`information signals for use by the dec