(12) United States Patent
`Lemelson et al.
`
`(HI) Patent No.:
`(45) Date of Patent:
`
`US 6,553,130 Bl
`Apr. 22, 2003
`
`111111
`
`1111111111111111111111111111111111111111111111111111111111111
`US006553130B 1
`
`(54) MOTOR VEHICLE WARNING AND
`CONTROL SYSTEM AND METHOD
`
`(76)
`
`Inventors: Jerome H. Lemelson, 868 Tyner Way,
`Incline Village, NV (US) 89450;
`Robert Pedersen, 7808 GlenEagle,
`Dallas, TX (US) 75248
`
`( * ) Notice:
`
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 1543 days.
`
`(21) Appl. No.: 08/671,853
`
`(22) Filed:
`
`Jun. 28, 1996
`
`Related U.S. Application Data
`
`(58)
`
`(51)
`(52)
`
`(63) Continuation of application No. 08/105,304, filed on Aug.
`11, 1993, now abandoned.
`Int. Cl.7
`.................................................. G06K 9/00
`U.S. Cl. ....................... 382/104; 340/435; 340/436;
`340(903; 382/106
`Field of Search .......... ...... ........ ...... ... 348/113, 114,
`348/115, 116, 118, 119, 135, 139, 140,
`142, 148, 149; 382/104, 215, 209, 106,
`156, 103, 217, 218; 340/907, 435, 903,
`901, 436; 180/168, 169, 167, 271, 274,
`275; 364/424.02, 460, 461, 436-437, 425.04,
`425.01, 426.01, 443, 424.032; 395/905,
`900, 913
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`4,257,703 A
`4,626,850 A
`4.825.211 A
`4,S72,05l A
`
`3/1981 Goodrich ....................... 356/4
`12/1986 Chey .......................... 340/903
`4/1989 Park ........................... 340/901
`10/19S9 Dye ........................... 340/903
`
`(List continued on next page.)
`
`FOREIGN PATENT DOCUMENTS
`
`JP
`JP
`JP
`
`4-219900
`5-124529
`5-143897
`
`8/1992
`5/1993
`6/199 3
`
`................. 340/903
`
`................. 340(903
`
`OTHER PUBLICATIONS
`
`Abu-Mostafa, "Information Theory, Complexity, and Neu(cid:173)
`ral Networks," IEEE Communications, pp. 25-28 (Nov.
`1989).
`Aggarwal et al., "On the Computation of Motion from
`Sequences of Images-A Review," Proceedings of the
`IEEE, pp. 917-35 (Aug. 1988).
`
`(List continued on next page.)
`
`Primary Examiner-Timothy M. Johnson
`(74) Attorney, Agent, or Firm-Louis 1. Hoffman
`
`(57)
`
`ABSTRACT
`
`A system and method assists the driver of a motor vehicle in
`preventing accidents or minimizing the effects of same. In
`one form, a television camera is mounted on a vehicle and
`scans the roadway ahead of the vehicle as the vehicle travels.
`Continuously generated video picture signals output by the
`camera are electronically processed and analyzed by an
`image analyzing computer, which generates codes that serve
`to identify obstacles. A decision computer mounted in the
`controlled vehicle receives such code signals along with
`code signals generated by the speedometer or one or more
`sensors sensing steering mechanism operation and generates
`control signals. Such code signals may be displayed, and a
`synthetic speech or special sound generating and warning
`mt:ans ust:d, to warn tht: drivt:r of tht: vt:hick of approaching
`and existing hazards. The system may also use the control
`signals, particularly through application of fuzzy logic, to
`control the operation of the brakes and steering mechanism
`of the vehicle to avoid or lessen the effects of a collision. In
`a particular form, the decision computer may select the
`evasive action taken from a number of choices, depending
`on whether and where the detection device senses other
`vehicles or obstacles.
`
`JP
`
`4-15799
`
`1/1992
`
`................. 340/905
`
`55 Claims, 13 Drawing Sheets
`
`WARNING BRAKE STEERING
`
`IPR2015-00262 - Ex. 1102
`Toyota Motor Corp., Petitioner
`1
`
`

`

`US 6,553,130 Bl
`Page 2
`
`U.S. PATENT DOCUMENTS
`
`4,901,362 A *
`4,926,170 A
`4.931,937 A *
`4,933,852 A
`4,969,038 A
`4,979,029 A
`5,018,689 A
`5.039.217 A
`5,081,585 A
`5,091,726 A
`5,122,796 A
`5,146,219 A
`5.161.107 A
`5,164,904 A
`5,177,462 A
`5,179,377 A
`5.189,619 A
`5,197,562 A
`5,230,400 A
`5,249,157 A
`5,276,620 A
`5.278,764 A
`5,298,882 A
`5,304,980 A
`5,307,136 A
`5,314,037 A
`5.327.117 A
`5,332,057 A
`5,339,075 A
`5,341,344 A
`5,357,438 A
`5.369.591 A
`5,545,960 A
`
`*
`
`*
`
`2/1990
`5/1990
`6/1990
`6/1990
`11/1990
`12/1990
`5/1991
`8/1991
`1/1992
`2/1992
`6/1992
`9/1992
`11/1992
`11/1992
`1/1993
`1/1993
`2/1993
`3/1993
`7/1993
`9/1993
`1/1994
`1/1994
`3/1994
`4/1994
`4/1994
`5/1994
`7/1994
`7/1994
`8/1994
`8/1994
`10/1994
`11/1994
`8/1996
`
`Terzian ....................... 382/103
`Beggs et a!. .. . . . . . . . . .. . . . . 340/904
`Kakinami eta!. .......... 180/169
`Lemelson ................... 340/439
`Lemelson ................... 358/107
`Lemelson ... .. ... ... ... ...... 358/93
`Yasunobu et a!.
`.......... 395/905
`Maekawa et a!. .. ... ... ... ... 356/1
`Kurami et al. ..... ... . 364/424.02
`Shyu .......................... 340/904
`Beggs el a!. .. . . . . . . . . .. . .. . 340/904
`Zechnall ..................... 340/905
`. . . . . .. . .. . 340/937
`Mayeaux et a!.
`Summer ..................... 340/905
`Kajiwara .................... 340/903
`Hancock ..................... 340/961
`.............. 395/905
`Adachi et a!.
`Kakinami et al.
`.......... 180/169
`Kakinami et a!.
`.......... 180/169
`Taylor ........................ 340/903
`Bottesch . . ... .. ... ... ... 364/424.05
`Iizuka eta!. ................ 364/461
`Tsai ........................... 340/468
`Maekawa
`................... 340/903
`Saneyoshi ................... 180/167
`Shaw et a!. ................. 180/169
`Kohsaka ..................... 340/459
`Butsuen et al. ............. 180/169
`Abst et a!. . . .. . . . . . . . . .. . .. . 340/903
`O'Brien eta!. ............... 367/96
`Davidian .................... 180/169
`Broxmeyer ................. 340/903
`Ishikawa
`...... ...... . 364/424.032
`
`OTHER PUBLICATIONS
`
`Alspector, "Neural-Style Microsystems that Learn," IEEE
`Communications, pp. 29-36 (Nov. 1989).
`Casasent, "Optics and Neural Nets," Chapter 16 in Carpen(cid:173)
`ter et al., eds., Neural Networks for Vision and Image
`Processing, pp. 437-48, (MIT Press 1992).
`Cox, "Fuzzy Fundamentals," IEEE Spectrum, pp. 58-61
`(Oct. 1992).
`Hammerstrom, ''Neural Networks at Work," IEEE Spec(cid:173)
`trum, pp. 26-32 (Jun. 1993).
`Hush et al., "Progress in Supervised Neural Networks,"
`IEEE Signal Processing, pp. 8-39 (Jan. 1993).
`.Turgen, ''Smart Cars and Highways Go Glohal," IFFF:
`Spectrum, pp. 26-36 (May 1991).
`Kittler et al., eds., Image Processing System Architectures,
`Chapter 4, pp. 49-81, and Chapter 5, pp. 85-101 (John
`Wiley & Sons 1985).
`Kosko et al., "Fuzzy Logic," Scientific American, pp. 76-81
`(Jul. 1993).
`
`Lee, "Fuzzy Logic in Control Systems: Fuzzy Logic Con(cid:173)
`troller, Part II,'' IEEE Transaction on Systems, Man, and
`Cybernetics, pp. 419-435 (vol. 20, No.2, Mar./ Apr. 1990).
`Lippmann, ''An Introduction to Computing with Neural
`Nets," IEEE ASSP, pp. 4-22 (Apr. 1987).
`Lippmann, "Pattern Classification Using Neural Networks,"
`IEEE Communications, pp. 47-50, 59-64 (Nov. 1989).
`Lisboa, ed., Neural Networks-Current Applications, Chap(cid:173)
`ter 1, pp. 1-34, Chapter 2, pp. 35-48, Chapter 7, pp.
`123-147 (Chapman & Hall1992).
`Lupo, "Defense Applications of Neural Networks," IEEE
`Communications, pp. 82-88 (Nov. 1989).
`Maresca et al., "Parallel Architectures for Vision," Proceed(cid:173)
`ings of the Ililit., pp. 970-981 (vol. 76, No. 8, Aug. 1988).
`Nijhuis et al., ''Evaluation of Fuzzy and Neural Vehicle
`Control," Institution of Electrical Engineers, pp. 447-452
`(1992).
`Pearson, ed., Image Processing, Chapter 8, pp. 141-155, and
`Chapter 10, pp. 169-189 (McGraw-Hil11991).
`Psaltis et al., ''Optoelectronic Implementations of Neural
`Networks," IEEE Communications, pp. 37-40, 71 (Nov.
`1989).
`Roth, ''Neural Networks for Extraction of Weak Targets in
`High Clutter Environments," IEEE Transactions on Sys(cid:173)
`tems, Man, and Cybernetics, pp. 1210-1217 (Sep./Ocl.
`1989).
`Schwartz et al., "Fuzzy Logic Flowers in Japan," IEEE
`Spectrum, pp. 32-35 (Jul. 1992).
`Soucek et al., Neural and Massively Parallel Computers,
`Chapter 12, pp. 245-276 (John Wiley & Sons 1988).
`Suaya et al., eds., VLSI and Parallel Computation, Chapter
`1, pp. 1-84, and Chapter 5, pp. 390-415 (Morgan Kaufmann
`1990).
`Teuber, Digital Image Processing, Chapter 1, pp. 1-30,
`Chapter 2, pp. 31-70, and Appendix D, pp. 254-255 (Pren(cid:173)
`tice Hall 1993).
`Wasserman, Neural Computing: Theory and Practice, Chap(cid:173)
`ter 1, pp. 11-26, Chapter 2, pp. 27-42, Chapter 3, pp. 43-59,
`and Chapter 9, pp. 151-166 (Van Nostrand Reinhold 1989).
`Yuhas et al., ''Integration of Acoustic and Visual Speech
`Signals Using Neural Networks," IEEE Spectrum, pp. 65-71
`(Nov. 1989).
`Shekhar et al., Design and Validation of Head up Displays
`for Navigation in IVHS, VNIS'91, Oct. 1991, pp. 537-542.
`"NHTSA IVHS Plan," National Highway Safety Adminis(cid:173)
`tration U.S. Department of Transportation, Jun. 12, 1992.
`Bosacchi et a!, Fuzzy Logic Technology & the Intelligent
`Highway System (IHS); IEEE, 1993, pp. 65-70.
`Rocket al., "Intelligent Road Transit: The Next Generation,"
`AI EXPERT, Apr. 1994, pp. 16-24 (not prior art).
`* cited by examiner
`
`2
`
`

`

`U.S. Patent
`
`Apr. 22, 2003
`
`Sheet 1 of 13
`
`US 6,553,130 Bl
`
`RANGE
`COMPUTER
`
`DECISION
`COMPUTER
`
`FIG. 1
`
`3
`
`

`

`U.S. Patent
`
`Apr. 22, 2003
`
`Sheet 2 of 13
`
`US 6,553,130 Bl
`
`51
`
`57
`
`RAM
`
`56
`
`58
`
`CONTROL
`PROCESSOR
`
`ROM
`
`59
`
`BUS INTERFACE
`
`11
`
`MICRO PROCESSOR
`CONTROLLER
`
`FIG. 2
`
`4
`
`

`

`U.S. Patent
`
`Apr. 22, 2003
`
`Sheet 3 of 13
`
`US 6,553,130 Bl
`
`INPUTS
`
`FIRST HIDDEN
`LAYER
`
`SECOND HIDDEN
`LAYER
`
`OUTPUT
`LAYER
`
`63
`
`61
`
`68
`x1
`
`x2
`
`Xn
`
`•
`• •
`
`FIG. 3
`
`1
`
`W0 =0
`
`70
`
`y
`
`72
`
`fHL
`
`u
`
`FIG. 4
`
`5
`
`

`

`U.S. Patent
`
`Apr. 22, 2003
`
`Sheet 4 of 13
`
`US 6,553,130 Bl
`
`IMAGE DATA
`
`l
`
`IMAGE
`PROCESSOR
`
`{74
`
`IMAGE DATA BUS
`
`75
`/"
`
`VIRTUAL
`PE
`
`VIRTUAL
`PE
`
`VIRTUAL
`PE
`
`VIRTUAL
`PE
`
`FIG. 5
`
`6
`
`

`

`U.S. Patent
`
`Apr. 22, 2003
`
`Sheet 5 of 13
`
`US 6,553,130 Bl
`
`MOTION
`
`WEATHER
`
`IM1GE
`
`RAYAR
`
`VEHiCLE OVE,RIDE
`
`DETECTION SIGNAL ANALYSIS
`
`v-74
`
`DATA
`E!b£
`r-
`
`--
`
`75
`
`N
`
`HAZARD
`
`y
`
`f76
`
`FUZZV ASSOCIATIVE MEMORY (FAM) SELECTION
`
`f77
`
`f78
`
`FAM-1
`
`FAM-2
`
`/'79
`
`I" a 0
`FAM-3 • • • FAM-N
`
`-
`
`CONTROL SIGNAL GENERATOR
`DEFUZZIFIER
`
`STEERING-
`
`.-WARNINGS
`
`-BRAKING
`
`/ 81
`
`f--STEERING SERVO
`MICROPROCESSOR CONTROLLER f--BRAKING SERVO
`I--DISPLAYS
`
`\....· 11
`FIG. 6
`
`7
`
`

`

`U.S. Patent
`
`Apr. 22, 2003
`
`Sheet 6 of 13
`
`US 6,553,130 Bl
`
`CLOSE
`
`MEDIUM
`
`FAR
`
`VERY FAR
`
`VERY
`CLOSE
`
`o...
`a5 1
`a: w m
`~ w
`~
`
`DISTANCE
`
`MEDIUM
`
`HIGH
`
`VERY HIGH
`
`0...
`:c
`VERY LOW LOW
`rn 1~--o-~
`a:
`w
`m
`~ w
`~
`
`VELOCITY
`
`HN
`
`MN
`
`LN
`
`LP
`
`MP
`
`HP
`
`1
`
`0...
`
`J: rn
`a: w m
`~ w
`~
`
`ACCELERATION
`
`FIG. 7
`
`8
`
`

`

`U.S. Patent
`
`Apr. 22, 2003
`
`Sheet 7 of 13
`
`US 6,553,130 Bl
`
`a..
`GREEN
`J:
`~ 11 - - - - - (cid:173)
`w m
`~ w
`~
`
`a..
`LOWBRAKE
`J:
`~ 11 - - - - - -
`w
`m
`~ w
`~
`
`a..
`LOW0
`J:
`~ 11------(cid:173)
`w m
`~ w
`~
`
`YELLOW
`
`RED
`
`WARNING LEVEL
`
`MEDIUM BRAKE
`
`HIGH BRAKE
`
`BRAKING PRESSURE
`
`MEDIUM0
`
`HIGH0
`
`STEERING ANGLE
`
`FIG. 8
`
`9
`
`

`

`L
`
`~ VL
`vc Y·LB-N0 A·LB-S0 R-MB-S0 R-HB-M0 A·HB-M0
`c Y·NB·N0 Y·LB-N0 A·MB·S0 A·HB-S0 A·HB-M0
`
`M
`
`H
`
`VH
`
`M G-NB-N0 Y·NB-N0 Y-LB-N0 R-MB-N0 A·HB-S0
`
`~ VL
`vc R-MB-N0 R-MB-S0 R-HB-M0 R-HB-H0 R-HB-H0
`
`M
`
`H
`
`VH
`
`L
`
`c
`
`M
`
`Y-LB·N0 R·MB·N0 R·MB-S0 R·HB·M0 A·HB·H0
`
`Y-NB-N0 Y-LB-N0 R-MB-N0 R-HB-N0 R-HB-S0
`
`F G-NB-N0 G·NB·N0 G-NB-N0 Y·LB-N0 Y·LB-N0
`
`F G-NB-N0 G-NB-N0 Y·NB-N0 Y-LB-N0 R-LB-N0
`
`VF G-NB-N0 G·NB·N0 G-NB-N0 Y·LB-N0 Y·LB-N0
`
`VF G·NB-N0 G·NB·N0 Y-NB-N0 Y-LB-N0 Y-LB-N0
`
`ACCELERATION = LP
`
`ACCELERATION = MP
`
`~ VL
`L
`M
`H
`VH
`vc R-MB-N0 R-HB-N0 R-HB-S0 R-HB-H0 R·HB·H0
`c R·MB-N0 A·MB·N0 R-MB-S0 R·HB-M0 R·HB-H0
`
`VL
`
`VH
`L
`M
`H
`vc Y-LB-N0 I Y-LB-N0IR-MB-N01 R-HB-S0IR-HB-M0
`
`FIG. 9
`
`C
`
`I Y-NB-N01 Y·MB-N0IY-MB-N01 R-HB-N01 R-HB-S0
`
`Y-LB-N0 R-MB-N0 R-MB-80 R-HB-M0 R-HB-M0
`
`M
`
`I G-NB-N01 G·NB-N01 Y·SB-N0 I R-MB-S0IR-MB-S0
`
`M
`
`F
`
`Y-NB-N0 Y-NB-N0 Y-NB-N0 R-LB-N0 R·LB-N0
`
`VF G-NB-N0 G-NB-N0 Y-NB-N0 Y-NB·N0 R-MB-N0
`
`ACCELERATION = HP
`
`F
`IG·NB-N01 G·NB-N01G-NB-N01 Y·NB-N0IY-NB-N0
`VF I G-NB-N0l G-NB-N0IG-NB-N0I G·NB-N0I Y·NB-N0
`ACCELERATION = LN
`
`~ VL
`L
`M
`H
`VH
`vc Y-LB-N0 Y-LB-N0 Y-MB-N0 R-HB-80 R-HB-M0
`c
`
`Y-NB-N0 Y-NB-N0 Y-LB-N0 Y-LB-80 R-MB-S0
`
`L
`
`~ VL
`M
`H
`VH
`vc Y-NB-N0 Y-LB-N0 Y-LB-N0 Y-MB-S0 R-MB-M0
`c G-NB-N0 G-NB-N0 G-NB-N0 Y·LB-N0 Y·LB-N0
`
`M G-NB-N0 G·NB-N0 Y·MB·N0 Y-MB·S0 R-MB-S0
`
`M G-NB-N0 G·NB-N0 G·NB-N0 Y·LB-N0 Y-LB-N0
`
`F G-NB-N0 G-NB-N0 G-NB-N0 G-NB-N0 Y-NB-N0
`
`F G-NB-N0 G-NB-N0 G-NB-N0 G-NB-N0 G-NB-N0
`
`VF G-NB-N0 G-NB-N0 G-NB-N0 G-NB-N0 Y-NB-N0
`
`ACCELERATION = MN
`
`VF G-NB-N0 G-NB-N0 G-NB-N0 G-NB-N0 G-NB-N0 -
`ACCELERATION = HN
`
`e •
`
`00.
`•
`~
`~ ......
`~ = ......
`
`>
`'t:i :-;
`
`N
`~N
`N c
`8
`
`'Jl =(cid:173)~
`
`~ -00
`
`0 ......
`......
`~
`
`e
`
`rJJ.
`Q'\
`1J.
`
`(JJ w ~ 1-' w
`
`0
`~
`1-'
`
`10
`
`

`

`U.S. Patent
`
`Apr. 22, 2003
`
`Sheet 9 of 13
`
`US 6,553,130 Bl
`
`HF
`
`HB
`
`HL
`
`HR
`
`OF
`
`08 O~r OR
`
`FIG. 10
`
`DISTANCE
`
`RELATIVE
`VELOCITY
`
`f 83
`RELATIVE
`ACCELERATION
`
`FIG. 11
`
`11
`
`

`

`STATE
`VECTOR HF
`1
`0
`2
`0
`3
`0
`4
`0
`5
`0
`0
`6
`7
`0
`8
`0
`9
`0
`0
`10
`11
`0
`12
`0
`13
`0
`14
`0
`15
`0
`16
`0
`17
`0
`18
`0
`19
`0
`20
`0
`21
`0
`22
`0
`0
`23
`24
`0
`25
`0
`26
`0
`27
`0
`28
`0
`29
`0
`30
`31
`32
`33
`34
`
`0
`0
`0
`0
`
`\_
`
`HAZARD
`HL
`HB
`0
`0
`0
`0
`0
`0
`0
`0
`0
`0
`0
`0
`0
`0
`0
`0
`0
`0
`1
`0
`1
`0
`1
`0
`1
`0
`1
`0
`1
`0
`1
`0
`1
`0
`1
`0
`1
`0
`1
`0
`1
`0
`1
`0
`0
`1
`1
`0
`1
`0
`1
`0
`1
`0
`1
`0
`1
`0
`
`1
`1
`1
`1 ...
`
`0
`0
`0
`0
`
`OBSTACLE
`HR OF OB OL
`0
`1
`1
`1
`1
`1
`1
`1
`1
`0
`0
`0
`0
`0
`0
`0
`0
`1
`1
`1
`1
`0
`0
`0
`0
`0
`0
`0
`0
`
`0
`1
`0
`1
`0
`1
`0
`1
`
`0
`0
`1
`1
`0
`0
`1
`1
`
`0
`0
`1
`1
`0
`0
`1
`1
`0
`0
`1
`1
`0
`0
`1
`1
`0
`1
`0
`1
`
`0
`0
`0
`0
`1
`1
`1
`1
`0
`0
`0
`0
`1
`1
`1
`1
`0
`0
`1
`1
`0
`0
`0
`0
`1
`1
`1
`1
`
`1
`1
`1
`1
`
`0
`0
`1
`1
`
`0
`1
`0
`1
`
`OR
`
`0
`1
`0
`1
`0
`1
`0
`1
`
`0
`1
`0
`1
`0
`1
`0
`1
`
`35
`36
`37
`38
`39
`40
`41
`42
`43
`44
`45
`46
`47
`48
`49
`50
`51
`52
`53
`54
`55
`56
`57
`58
`59
`60
`61
`62
`63
`64
`65
`66
`67
`68
`
`0
`0
`0
`0
`0
`0
`1
`1
`1
`1
`1
`1
`1
`1
`1
`1
`1
`1
`
`1
`1
`1
`1
`1
`1
`1
`1
`1
`1
`1
`1
`1
`1
`1
`
`1 ..
`1
`1
`1
`1
`1
`0
`0
`0
`0
`0
`0
`0
`0
`0
`0
`0
`0
`
`0
`0
`0
`0
`0
`0
`1
`1
`1
`1
`1
`1
`1
`1
`1
`
`1
`1
`1
`1
`1
`1
`0
`0
`0
`0
`0
`0
`0
`0
`0
`0
`0
`0
`
`1
`1
`1
`1
`1
`1
`0
`0
`0
`0
`0
`0
`1
`1
`1
`
`0
`0
`0
`0
`1
`1
`0
`0
`0
`0
`0
`0
`0
`0
`1
`1
`1
`1
`
`0
`0
`0
`0
`1
`1
`0
`0
`0
`0
`1
`1
`0
`0
`1
`
`0
`0
`1
`1
`0
`1
`
`..
`
`0
`0
`0
`0
`
`1
`1
`1
`0
`0
`
`1
`1
`0
`0
`1
`1
`0
`1
`
`0
`1
`0
`1
`
`0
`1
`0
`1
`0
`1
`0
`1
`
`0
`1
`0
`1
`
`0
`1
`0
`1
`
`0
`1
`
`0
`0
`1
`1
`0
`0
`1
`1
`0
`1
`0
`1
`
`0
`0
`1
`1
`0
`1
`
`FIG. 12
`
`Cj
`•
`00
`•
`~
`~ ......
`~ = ......
`
`>
`":= :-;
`
`N
`~N
`N
`0
`
`8
`
`'Jl =(cid:173)~
`
`~ -......
`
`0
`0 .....
`......
`~
`
`e r;r;.
`="
`~
`Ul
`~
`1-o"
`~
`0
`~
`1-o"
`
`~
`
`12
`
`

`

`U.S. Patent
`
`Apr. 22, 2003
`
`Sheet 11 of 13
`
`US 6,553,130 Bl
`
`99
`MODIFY HAZARD
`STATE VECTOR
`
`98
`
`FIG. 13
`
`13
`
`

`

`U.S. Patent
`
`Apr. 22, 2003
`
`Sheet 12 of 13
`
`US 6,553,130 Bl
`
`108 \
`
`109
`~
`
`110 \
`
`111 \
`
`112 \
`
`113 \
`
`COLLISION
`STATE VECTOR
`
`INPUTDATA
`FILE
`
`/
`
`FUZZV
`DISTANCE
`MEMBERSHIP
`
`FUZZV
`VELOCITY
`MEMBERSHIP
`
`FUZZV
`ACCELERATION
`MEMBERSHIP
`
`HAZARD/OBJECT
`STATE VECTOR
`ADDRESS
`TRANSLATION
`
`FAMOUTPUT
`ADDRESS
`
`FIG. 14
`
`14
`
`

`

`U.S. Patent
`
`Apr. 22, 2003
`
`Sheet 13 of 13
`
`US 6,553,130 Bl
`
`HAZARD FRONT
`
`HAZARD BACK
`
`HAZARD RIGHT
`
`N
`
`OBJECT RIGHT
`
`y
`
`HAZARD/OBJECT
`VECTOR
`
`COLLISION
`VECTOR
`
`ACCESS FUZZV
`ASSOCIATIVE
`MEMORY
`(FAM)
`
`GENERATE OUTPUT
`EXPERT CONTROL
`SIGNALS
`
`DISTANCE FUZZV
`MEMBERSHIP
`
`VELOCITY FUZZV
`MEMBERSHIP
`
`ACCELERATION
`FUZZV MEMBERSHIP
`
`WARNING BRAKE STEERING
`
`FIG. 15
`
`15
`
`

`

`US 6,553,130 Bl
`
`1
`MOTOR VEHICLE WARNING AND
`CONTROL SYSTEM AND METHOD
`
`CROSS-REFERENCE TO RELATED
`APPLICATION
`
`This application is a continuation of Ser. No. 08/105,304,
`filed Aug. 11, 1993, abandoned.
`
`FIELD OF THE INVENTION
`
`2
`the damage therefrom. More particularly, such systems fail
`to exercise simultaneous, coordinated control over vehicle
`steering and speed, fail to take full advantage of idcntifica(cid:173)
`tion of different obstacle or hazard types using standard
`5 stored models of production vehicles and other commonly
`encountered roadway objects, fail to deal effectively with
`objects and hazards located simultaneously on different
`sides of the vehicle, and fail to capitalize fully on modern
`expert system decision and control technology, such as
`10 represented by fuzzy logic and neural network methods, to
`deal with more complex hazardous situations.
`
`This invention relates to a system and method for oper(cid:173)
`ating a motor vehicle, such as an automobile, truck, aircraft
`or other vehicle, wherein a computer or computerized sys(cid:173)
`tem is employed to assist and/or supplement the driver in the
`movement of the vehicle along a path of travel, such as a
`street or roadway and may he used to avoid obstacles and 15
`accidents.
`
`BACKGROUND OF THE INVENTION
`
`25
`
`A major cause of human suffering is automobile acci(cid:173)
`dents. Approximately 49,000 people die in traffic accidents
`each year in the United States, and another three million are
`injured. The costs of death and injury accidents are stagger(cid:173)
`ing. According to the United States National Highway
`Traffic Safety Administration, crash damage and medical
`bills total $137 billion a year.
`Automobile designers offer many safety features, includ(cid:173)
`ing passenger restraints, improved braking systems, and
`body designs, intended to better protect automobile crash
`victims. But very little has been done in the area of auto- 30
`malic vehicle control systems based on modern electronics,
`computer systems, and advanced real-time software. This is
`true despite rapidly increasing capabilities in these technolo(cid:173)
`gies and pervasive application in many other areas
`including, for example the business, entertainment, and 35
`medical fields. Vehicle guidance and control technology has,
`of course, been applied with great success in military
`defense systems, avionics systems and space exploration
`systems. But, this technology is costly and has not been
`commercialized.
`The opportunity exists today to develop cost effective,
`commercial automated vehicle control systems. New
`advances in low-cost hardware and software technology
`make implementation feasible. High-speed, parallel com(cid:173)
`puter architectures, specialized image-processing 45
`equipment, and advanced special computers such as math
`coprocessors are available. Advanced expert system imple(cid:173)
`mentations based on concepts such as fuzzy logic and neural
`networks, and new, improved scanning systems for sensing
`environments around moving vehicles make it very timely, 50
`indeed, to pursue new approaches.
`Work on these problems has begun. Intelligent vehicle/
`highway systems are being investigated with traffic control
`systems intended to minimize congestion. Vehicle location
`systems such as GPS (Global Positioning System) and route 55
`guidance systems arc also being pursued. Certain systems
`for automated vehicle control have been proposed, including
`systems that scan the roadway directly ahead of a vehicle
`using radar/lidar or television and attempt to warn a driver
`of impending danger. Fuzzy logic expert systems for con- 60
`trolling vehicle speed (braking and throttle) based on scan(cid:173)
`ning the roadway ahead of a vehicle have been described.
`Road tracking with electronic vehicle guidance is being
`pursued. Fuzzy logic has been applied to braking systems in
`subway and train systems.
`While these developments are important, they fail to
`protect vehicles from many types of collisions or minimize
`
`SUMMARY OF THE INVENTION
`In a preferred form of the invention, a video scanning
`system, such as a television camera and/or one or more laser
`scanners mounted on the vehicle scan the road in front of the
`vehicle and generate image information which is computer
`analyzed per se or in combination with a range sensing
`system to warn the driver of hazardous conditions during
`20 driving by operating a display, such as a heads-up display,
`and/or a synthetic speech generating means which generates
`sounds or words of speech to verbally indicate such road
`conditions ahead of the vehicle.
`The preferred form of the invention provides audible
`and/or visual display means to cooperate in indicating to the
`driver of a motor vehicle both normal and hazardous road
`conditions ahead as well as driving variables such as dis(cid:173)
`tances to stationary objects, and other vehicles; the
`identification, direction of travel and speed of such other
`vehicles, and the identification of and distances to stationary
`or slowly moving objects such as barriers, center islands,
`pedestrians, parked cars poles, sharp turns in the road and
`other conditions. In addition, the image analyzing computer
`of the vehicle may be operated to scan and decode coded
`and/or character containing signs or signals generated by
`indicia or code generating other devices within or at the side
`of the road and indicating select road and driving conditions
`ahead.
`The computer is operable to analyze video and/or other
`40 forms of image information generated as the vehicle travels
`to identify obstacles ahead of the vehicle and, in certain
`instances, quantify the distance between the vehicle con(cid:173)
`taining same on the basis of the size of the identified vehicle
`or object and/or by processing received pulse-echo signals.
`Using such identifying information and comparing it with
`information on the shapes and sizes of various objects such
`as rear and front profiles of all production vehicles and the
`like and their relative sizes or select dimensions thereof,
`indications of distances to such objects may be computed
`and indicated as further codes. When the closing distance
`becomes hazardous, select vehicle subsystems may be auto-
`matically controlled by the computer as it continues to
`analyze image signals generated by the television camera. A
`first subsystem generates a first select code or codes which
`controls an electronic display, such as a heads-up display to
`cause it to display a warning indication, such as one or more
`flashing reel light portions of the display or other lighted
`effect. For example, the display may project on the wind(cid:173)
`shield or dashboard such information as images of the
`controlled vehicle and other vehicles in and adjacent its path
`of travel and relative distances thereto as well as groups of
`characters defining same, colored and flashing warning
`lights and the like for pre-warning and warning purposes. A
`second subsystem generates a code or series of codes which
`65 control a sound generating means which generates a select
`sound such as a horn, buzzing sound and/or select synthetic
`speech warning of the hazardous condition detected and, in
`
`16
`
`

`

`3
`certain instances, generating sounds of select words of
`speech which may warn of same and/or suggest corrective
`action (s) by the vehicle operator or driver to avoid an
`accident.
`A third subsystem comes on-line and generates one or 5
`more codes which are applied to at least partly effect a
`corrective action such as by pulsing one or more motors or
`solenoids to apply the brakes of the vehicle to cause it to
`slow down. If necessary to avoid or lessen the effects of an
`accident, the third subsystem stops the forward travel of the 10
`vehicle in a controlled manner depending on the relative
`speeds of the two vehicles, and/or the controlled vehicle and
`a stationery object or structure and the distance therebe(cid:173)
`tween.
`A fourth subsystem, which may be part of or separate
`from the third subsystem may generate one or more codes
`which arc applied to either effect partial and/or complete
`control of the steering mechanism for the vehicle to avoid an
`obstacle and/or lessen the effect of an accident. Either or
`both the third or fourth subsystem may also be operable to
`control one or more safety devices by controlling motors, 20
`solenoids or valves, to operate a restraining device or
`devices for the driver and passenger(s) of the vehicle, such
`as a safety belt tightening means, an air bag inflation means
`or other device designed to protect human beings in the
`vehicle.
`'lbe second, and/or third and fourth subsystems may also
`be operable to effect or control the operation of additional
`warning means such as the horn, headlights and/or other
`warning lights on the vehicle or other warning means which
`operates to alert, flag or warn the driver of the approaching 30
`or approached vehicle or a pedestrian of the approaching
`hazardous condition. One or more of these subsystems may
`also be operable to generate and transmit one or more codes
`to be received and used by the approaching or approached
`vehicle or a roadside device to effect additional on-line 35
`warning(s) of the hazardous condition, and/or may be
`n:corded on a disc or RAM (random access memory) for
`future analysis, if necessary.
`In a modified form of the invention, the vehicle warning
`system may also include a short wave receiving means to
`receive code signals from other vehicles and/or short wave
`transmitters at the side of or within the road for controlling
`the visual, audio and/or brake and steering means of the
`vehicle to avoid or lessen the effects of an accident and/or to
`maintain the vehicle in-lane and in proper operating condi(cid:173)
`tion as it travels.
`The systems and methods of this invention preferably
`employ computerized image analyzing techniques of the
`types disclosed and defined in such patents of mine as U.S. 50
`Pat. Nos. 4,969,038 and 4,979,029 and references cited in
`the file wrappers thereof as well as other more recent patents
`and include the use of known artificial intelligence, neural
`networking and fuzzy logic computing electronic circuits.
`While the invention is described herein principally in 55
`connection with an automobile on a roadway, it may be used
`in connection with controlling any powered vehicle, includ(cid:173)
`ing a motor vehicle, a boat, a train, or an aircraft.
`Accordingly it is a primary object of this invention to
`provide a new and improved system and method for con- 60
`trolling the operation of a powered vehicle.
`Another object is provide a system and method for
`assisting the driver of a powered vehicle in controlling its
`operation to avoid an accident or hazardous driving condi(cid:173)
`tion.
`Another object is to provide a system and method
`employing computerized image analysis to control or assist
`
`40
`
`45
`
`65
`
`US 6,553,130 Bl
`
`4
`the driver of a motor vehicle in controlling its operation to
`avoid hazardous conditions such as collisions with other
`vehicles, stationery objects or pedestrians.
`Another object is to provide a computerized system and
`method for controlling the speed of travel of a motor vehicle
`to lessen the chances of an accident while being driven by
`a person.
`Another object is to provide a system and method
`employing a television scanning camera mounted on a
`vehicle for scanning the field ahead, such as the image of the
`road ahead of the vehicle and a computer for analyzing the
`image signals generated wherein automatic image
`intensifying, or infra-red scanning and detection means is
`utilized to permit scanning operations to be effected during
`driving at night and in low light, snowing or fog conditions.
`Another object is to provide a system and method
`employing a television camera or other video scanning
`means mounted on a moving motor vehicle for scanning,
`detecting and identifying obstacles such as other vehicles
`ahead of such moving vehicle wherein the video image
`signals are analyzed to determine distances to such objects.
`Another object is to provide a computer controlled safety
`system for a motor vehicle which employs a television
`camera and an auxiliary scanning means to both identify
`25 obstacles in the path of the vehicle and determine distance
`therefrom on a real time and continuous basis for use in
`warning the operator of same and/or in controlling the
`operation of the vehicle to avoid a collision.
`
`15
`
`BRIEF DESCRIPTION OF DRAWINGS
`The various hardware and software elements used to carry
`out the invention described herein are illustrated in the form
`of block diagrams, flow charts, and depictions of neural
`network and fuzzy logic algorithms and structures. The
`preferred embodiment is illustrated in the following figures:
`FIG. 1 is a block diagram of the overall Motor Vehicle
`Warning and Control System illustrating system sensors,
`computers, displays, input/output devices and other key
`elements.
`FIG.2 is a block diagram of an image analysis computer
`19 of the type that can be used in the Vehicle Hazard
`Avoidance System herein of FIG. 1.
`FIG. 3 illustrates a neural network of the type useful in the
`image analysis computer of FIG. 4.
`FIG. 4 illustrates the structure of a Processing Element
`(PE) in the neural network of FIG. 3.
`FIG. 5 is an alternate embodiment of a neural network
`image processor useful in the system of FIG. 1.
`FIG. 6 is a flow diagram illustrating the overall operation
`of the Motor Vehicle Warning and Control System of FIG.
`1.
`
`FIG. 7 illustrates typical input signal membership func(cid:173)
`tions for fuzzy logic algorithms useful in the Motor Vehicle
`Warning and Control System of FIG. 1.
`FIG. 8 illustrates typical output signal membership func(cid:173)
`tions for fuzzy logic algorithms useful in the Motor Vehicle
`Warning and Control System of FIG. 1.
`FIG. 9 illustrates typical Fuzzy Associative Memory
`(FAM) maps for the fuzzy logic algorithms useful in the
`Motor Vehicle Warning and Control System of FIG. 1.
`FIG. 10 is a Hazard/Object state vector useful in imple(cid:173)
`menting the Fuzzy Logic Vehicle Warning and Control
`System.
`FIG. 11 is a Hazard Collision Control vector useful in
`implementing the Fuzzy Logic vehicle Warning and Control
`System.
`
`17
`
`

`

`US 6,553,130 Bl
`
`5
`FIG. 12 is a table of Hazard/Object state vectors indicat(cid:173)
`ing possible combinations of hazards and objects useful in
`the Fuzzy Associative Memory access system used herein.
`FIG. 13 is a more detailed logic flow diagram for the
`analysis of detection signals prior to accessing fuzzy logic 5
`control structures in the Motor Vehicle Warning and Control
`System.
`FIG. 14 is a more detailed Logic Flow Diagram for the
`Fuzzy Associative Memory (FAM) selection processing.
`FIG. 15 is an example system flow illustrating the opera-
`tion of the Motor Vehicle Warning and Control System.
`
`DETAILED DESCRIPTION
`
`10
`
`6
`a range computer 21 which accepts digital code signals from
`a radar or lidar computer 14 which interprets radar and/or
`laser range signals from respective reflected radiation
`receiving means on the vehicle.
`In a modified form, video scanning and radar or lidar
`scanning may be jointly employed to identify and indicate
`distances between the controlled vehicle and objects ahead
`of, to the side(s) of, and to the rear of the controlled vehicle.
`The image analyzing computer 19 with associated
`memory 20 may be implemented in several different ways.
`Of particular concern is the requirement for high speed
`image processing with the capability to detect various haz(cid:173)
`ards in dynamic image fields with changing scenes, moving
`objects and multiple objects, more than one of which maybe
`15 a potential hazard. Requirements for wide angle vision and
`the ability to analyze both right and left side image fields
`also exists. The imaging system not only detects hazards, but
`also estimates distance based on image data for input to the
`range computer 21 implemented with the associated
`20 memory unit 22.
`High speed image processing can be implemented
`employing known special purpose computer architectures
`including various parallel system structures and systems
`based on neural networks. FIG. 2 shows a high speed
`parallel processor system embodiment with dedicated image
`processing hardware. The system of FIG. 2 has a dedicated
`image data bus 50 for high speed image data transfer. The
`video camera 16 transfers full-frame video picture signal/
`30 data to the image bus 50 via analog/digital converter 18 and
`video preprocessor 51. The video camera 16 is preferably a
`CCD array camera generating successive picture frames
`with individual pixels being digitized for processing by the
`video preprocessor 51. The video camera 16 may also be
`35 implemented with other technologies including known
`image intensifying electron gun and infrared imaging meth(cid:173)
`ods. Multipl