`Lemelson et ai.
`
`111111111111111111111111111111111111111111111111111111111111111111111111111
`US006553130Bl
`
`(10) Patent No.:
`(45) Date of Patent:
`
`US 6,553,130 Bl
`Apr. 22, 2003
`
`(54) MOTOR VEHICLE WARNING AND
`CONTROL SYSTEM AND METHOD
`
`JP
`JP
`JP
`
`4-219900
`5-124529
`5-143897
`
`8/1992
`5/1993
`6/1993
`
`................. 340/903
`
`................. 340/903
`
`(76)
`
`Inventors: Jerome H. Lemelson, 868 Tyner Way,
`Incline Village, NV (US) 89450;
`Robert Pedersen, 7808 GlenEagle,
`Dallas, TX (US) 75248
`
`( *) Notice:
`
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`U.S.c. 154(b) by 1543 days.
`
`(21) Appl. No.: 08/671,853
`
`(22) Filed:
`
`Jun. 28, 1996
`
`Related U.S. Application Data
`
`(63) Continuation of application No. 08/105,304, filed on Aug.
`11, 1993, now abandoned.
`
`(51)
`(52)
`
`(58)
`
`Int. CI? .................................................. G06K 9/00
`U.S. CI. ....................... 382/104; 340/435; 340/436;
`340/903; 382/106
`Field of Search .. ... ..... ... ... ... ..... ... ... ... 348/113, 114,
`348/115, 116, 118, 119, 135, 139, 140,
`142, 148, 149; 382/104, 215, 209, 106,
`156, 103, 217, 218; 340/907, 435, 903,
`901, 436; 180/168, 169, 167, 271, 274,
`275; 364/424.02, 460, 461, 436-437, 425.04,
`425.01, 426.01, 443, 424.032; 395/905,
`900,913
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`4,257,703 A
`4,626,850 A
`4,825,211 A
`4,872,051 A
`
`3/1981 Goodrich ... ... ... ... ... ..... ... 356/4
`12/1986 Chey .......................... 340/903
`4/1989 Park ........................... 340/901
`10/1989 Dye ........................... 340/903
`
`(List continued on next page.)
`
`FOREIGN PATENT DOCUMENTS
`
`OTHER PUBLICATIONS
`
`Abu-Mostafa, "Information Theory, Complexity, and Neu(cid:173)
`ral Networks," IEEE Communications, pp. 25-28 (Nov.
`1989).
`Aggarwal et aI., "On the Computation of Motion from
`Sequences of Images-A Review," Proceedings of the
`IEEE, pp. 917-35 (Aug. 1988).
`
`(List continued on next page.)
`
`Primary Examiner-Timothy M. Johnson
`(74) Attorney, Agent, or Firm-Louis J. Hoffman
`
`(57)
`
`ABSTRACT
`
`A system and method assists the driver of a motor vehicle in
`preventing accidents or minimizing the effects of same. In
`one form, a television camera is mounted on a vehicle and
`scans the roadway ahead of the vehicle as the vehicle travels.
`Continuously generated video picture signals output by the
`camera are electronically processed and analyzed by an
`image analyzing computer, which generates codes that serve
`to identify obstacles. A decision computer mounted in the
`controlled vehicle receives such code signals along with
`code signals generated by the speedometer or one or more
`sensors sensing steering mechanism operation and generates
`control signals. Such code signals may be displayed, and a
`synthetic speech or special sound generating and warning
`means used, to warn the driver of the vehicle of approaching
`and existing hazards. The system may also use the control
`signals, particularly through application of fuzzy logic, to
`control the operation of the brakes and steering mechanism
`of the vehicle to avoid or lessen the effects of a collision. In
`a particular form, the decision computer may select the
`evasive action taken from a number of choices, depending
`on whether and where the detection device senses other
`vehicles or obstacles.
`
`JP
`
`4-15799
`
`1/1992
`
`................. 340/905
`
`55 Claims, 13 Drawing Sheets
`
`WARNING BRAKE STEERING
`
`IPR2013-00419 - Ex. 1002
`Toyota Motor Corp., Petitioner
`
`1
`
`
`
`US 6,553,130 BI
`Page 2
`
`U.S. PATENT DOCUMENTS
`4,901,362 A *
`4,926,170 A
`4,931,937 A *
`4,933,852 A
`4,969,038 A
`4,979,029 A
`5,018,689 A
`5,039,217 A
`5,081,585 A
`5,091,726 A
`5,122,796 A
`5,146,219 A
`5,161,107 A
`5,164,904 A
`5,177,462 A
`5,179,377 A
`5,189,619 A
`5,197,562 A *
`5,230,400 A
`5,249,157 A
`5,276,620 A
`5,278,764 A
`5,298,882 A
`5,304,980 A
`5,307,136 A
`5,314,037 A
`5,327,117 A
`5,332,057 A
`5,339,075 A
`5,341,344 A
`5,357,438 A
`5,369,591 A
`5,545,960 A *
`
`Terzian ....................... 382/103
`Beggs et al. ................ 340/904
`Kakinami et al.
`.......... 180/169
`Lemelson ................... 340/439
`Lemelson ................... 358/107
`Lemelson ..... ... ... ... ...... 358/93
`.......... 395/905
`Yasunobu et al.
`Maekawa et al. .. ... ... ... ... 356/1
`Kurami et al. .. ... .... 364/424.02
`Shyu .......................... 340/904
`Beggs et al. ................ 340/904
`Zechnall ..................... 340/905
`Mayeaux et al.
`........... 340/937
`Summer ..................... 340/905
`Kajiwara .................... 340/903
`Hancock ..................... 340/961
`Adachi et al.
`.............. 395/905
`Kakinami et al.
`.......... 180/169
`Kakinami et al.
`.......... 180/169
`Taylor ........................ 340/903
`Bottesch .. ..... ... ... ... 364/424.05
`Iizuka et al. ................ 364/461
`Tsai ........................... 340/468
`Maekawa ................... 340/903
`Saneyoshi ................... 180/167
`Shaw et al. ................. 180/169
`Kohsaka ..................... 340/459
`Butsuen et al. ............. 180/169
`Abst et al. .................. 340/903
`O'Brien et al. ............... 367/96
`Davidian .................... 180/169
`Broxmeyer ................. 340/903
`Ishikawa ... ... ... .... 364/424.032
`
`2/1990
`5/1990
`6/1990
`6/1990
`11/1990
`12/1990
`5/1991
`8/1991
`1/1992
`2/1992
`6/1992
`9/1992
`11/1992
`11/1992
`1/1993
`1/1993
`2/1993
`3/1993
`7/1993
`9/1993
`1/1994
`1/1994
`3/1994
`4/1994
`4/1994
`5/1994
`7/1994
`7/1994
`8/1994
`8/1994
`10/1994
`11/1994
`8/1996
`
`OlliER PUBLICATIONS
`
`Alspector, "Neural-Style Microsystems that Learn," IEEE
`Communications, pp. 29-36 (Nov. 1989).
`Casasent, "Optics and Neural Nets," Chapter 16 in Carpen(cid:173)
`ter et aI., eds., Neural Networks for Vision and Image
`Processing, pp. 437-48, (MIT Press 1992).
`Cox, "Fuzzy Fundamentals," IEEE Spectrum, pp. 58-61
`(Oct. 1992).
`Hammerstrom, "Neural Networks at Work," IEEE Spec(cid:173)
`trum, pp. 26-32 (Jun. 1993).
`Hush et aI., "Progress in Supervised Neural Networks,"
`IEEE Signal Processing, pp. 8-39 (Jan. 1993).
`Jurgen, "Smart Cars and Highways Go Global," IEEE
`Spectrum, pp. 26-36 (May 1991).
`Kittler et aI., eds., Image Processing System Architectures,
`Chapter 4, pp. 49-81, and Chapter 5, pp. 85-101 (John
`Wiley & Sons 1985).
`Kosko et aI., "Fuzzy Logic," Scientific American, pp. 76-81
`(Jui. 1993).
`
`Lee, "Fuzzy Logic in Control Systems: Fuzzy Logic Con(cid:173)
`troller, Part II," IEEE Transaction on Systems, Man, and
`C~bernetics, pp. 419-435 (vol. 20, No.2, Mar./Apr. 1990).
`Lippmann, "An Introduction to Computing with Neural
`Nets," IEEE ASSP, pp. 4-22 (Apr. 1987).
`Lippmann, "Pattern Classification Using Neural Networks,"
`IE.EE Communications, pp. 47-50, 59-64 (Nov. 1989).
`LIsboa, ed., Neural Networks---Current Applications, Chap(cid:173)
`ter 1, pp. 1-34, Chapter 2, pp. 35-48, Chapter 7, pp.
`123-147 (Chapman & Hall 1992).
`Lupo, "Defense Applications of Neural Networks," IEEE
`Communications, pp. 82-88 (Nov. 1989).
`Maresca et aI., "Parallel Architectures for Vision," Proceed(cid:173)
`in?s or the IEEE;., pp. 97?-981 (vol. 76, No.8, Aug. 1988).
`NIJhms et aI., Evaluatlon of Fuzzy and Neural Vehicle
`Control," Institution of Electrical Engineers, pp. 447-452
`(1992).
`Pearson, ed., Image Processing, Chapter 8, pp. 141-155, and
`Chapter 10, pp. 169-189 (McGraw-Hill 1991).
`Psaltis et aI., "Optoelectronic Implementations of Neural
`Networks," IEEE Communications, pp. 37-40, 71 (Nov.
`1989).
`R?th, "Neural Net:vorks for Extraction of Weak Targets in
`HIgh Clutter EnvIronments," IEEE Transactions on Sys(cid:173)
`tems, Man, and Cybernetics, pp. 1210-1217 (Sep./Oct.
`1989).
`Schwartz et aI., "Fuzzy Logic Flowers in Japan," IEEE
`Spectrum, pp. 32-35 (Jui. 1992).
`Soucek et aI., Neural and Massively Parallel Computers,
`Chapter 12, pp. 245-276 (John Wiley & Sons 1988).
`Suaya et aI., eds., VLSI and Parallel Computation, Chapter
`1, pp. 1-84, and Chapter 5, pp. 390-415 (Morgan Kaufmann
`1990).
`Teuber, Digital Image Processing, Chapter 1, pp. 1-30,
`Chapter 2, pp. 31-70, and Appendix D, pp. 254-255 (Pren(cid:173)
`tice Hall 1993).
`Wasserman, Neural Computing: Theory and Practice, Chap(cid:173)
`ter 1, pp. 11-26, Chapter 2, pp. 27-42, Chapter 3, pp. 43-59,
`and Chapter 9, pp. 151-166 (Van Nostrand Reinhold 1989).
`Yuhas et aI., "Integration of Acoustic and Visual Speech
`Signals Using Neural Networks," IEEE Spectrum, pp. 65-71
`(Nov. 1989).
`Shekhar et aI., Design and Validation of Head up Displays
`for Navigation in IVHS, VNIS'91, Oct. 1991, pp. 537-542.
`"NHTSA IVHS Plan," National Highway Safety Adminis(cid:173)
`tration U.S. Department of Transportation, Jun. 12, 1992.
`Bosacchi et aI, Fuzzy Logic Technology & the Intelligent
`Highway System (IRS); IEEE, 1993, pp. 65-70.
`Rock et aI., "Intelligent Road Transit: The Next Generation,"
`AI EXPERT, Apr. 1994, pp. 16-24 (not prior art).
`* cited by examiner
`
`2
`
`
`
`u.s. Patent
`
`Apr. 22, 2003
`
`Sheet 1 of 13
`
`US 6,553,130 Bl
`
`STEERING ~
`SERVO(S) aB
`
`RANGE
`COMPUTER
`
`DECISION
`COMPUTER
`
`SPEECH
`RECOGNITION
`COMPUTER
`
`FIG. 1
`
`3
`
`
`
`u.s. Patent
`
`Apr. 22, 2003
`
`Sheet 2 of 13
`
`US 6,553,130 Bl
`
`51
`
`57
`
`RAM
`
`56
`
`58
`
`CONTROL
`PROCESSOR
`
`ROM
`
`59
`
`BUS INTERFACE
`
`11
`
`MICRO PROCESSOR
`CONTROLLER
`
`FIG. 2
`
`4
`
`
`
`u.s. Patent
`
`Apr. 22, 2003
`
`Sheet 3 of 13
`
`US 6,553,130 Bl
`
`INPUTS
`
`FIRST HIDDEN
`LAYER
`
`SECOND HIDDEN
`LAYER
`
`OUTPUT
`LAYER
`
`63
`
`61
`
`68
`Xl
`
`X2
`
`Xn
`
`•
`• •
`
`FIG. 3
`
`1
`
`Wo=0
`
`70
`
`Y
`
`72
`
`fHL
`
`U
`
`FIG. 4
`
`5
`
`
`
`u.s. Patent
`
`Apr. 22, 2003
`
`Sheet 4 of 13
`
`US 6,553,130 Bl
`
`IMAGE DATA
`
`1
`
`IMAGE
`PROCESSOR
`
`,74
`
`IMAGE DATA BUS f' 75
`
`VIRTUAL
`PE
`
`VIRTUAL
`PE
`
`VIRTUAL
`PE
`
`VIRTUAL
`PE
`
`FIG. 5
`
`6
`
`
`
`u.s. Patent
`
`Apr. 22, 2003
`
`Sheet 5 of 13
`
`US 6,553,130 Bl
`
`MOTION
`
`WEATHER
`
`VEHiCLE OVE1RIDE
`
`DETECTION SIGNAL ANALYSIS
`
`V- 74
`
`I -
`
`--
`
`75
`
`N HAZARD
`
`Y
`
`FUZZY ASSOCIATIVE MEMORY (FAM) SELECTION
`
`FAM-1
`
`FAM-2
`
`FAM-3 • • • FAM-N
`
`/"81
`~~----------~~----------~----------~~
`CONTROL SIGNAL GENERATOR
`DEFUZZIFIER
`
`STEERING-
`
`I-- WARNINGS ~ BRAKING
`
`-STEERING SERVO
`MICROPROCESSOR CONTROllER -BRAKING SERVO
`...... ________ ~-.... -DISPLAYS
`'-11
`FIG. 6
`
`7
`
`
`
`u.s. Patent
`
`Apr. 22, 2003
`
`Sheet 6 of 13
`
`US 6,553,130 Bl
`
`CLOSE
`
`MEDIUM
`
`FAR
`
`VERY FAR
`
`VERY
`Q.
`~ 1 CLOSE
`a:
`w m
`~ w
`~
`
`DISTANCE
`
`MEDIUM
`
`HIGH
`
`VERY HIGH
`
`Q.
`VERY LOW LOW
`J:
`rn 11---_
`a:
`w m
`~ w
`~
`
`VELOCITY
`
`HN
`
`MN
`
`LN
`
`LP
`
`MP
`
`HP
`
`1
`
`ACCELERATION
`
`FIG. 7
`
`8
`
`
`
`u.s. Patent
`
`Apr. 22, 2003
`
`Sheet 7 of 13
`
`US 6,553,130 Bl
`
`a..
`GREEN
`J:
`~ 11 - - - - - (cid:173)
`w m
`~ w
`~
`
`a..
`LOW BRAKE
`J:
`~ 11 - - - - - -
`w m
`~ w
`~
`
`a..
`LOW 0
`J:
`~ 11---------,.
`w m
`~ w
`~
`
`YELLOW
`
`RED
`
`WARNING LEVEL
`
`MEDIUM BRAKE
`
`HIGH BRAKE
`
`BRAKING PRESSURE
`
`MEDIUM 0
`
`HIGH 0
`
`STEERING ANGLE
`
`FIG. 8
`
`9
`
`
`
`rJl .
`d .
`
`ACCELERATION = HN
`
`ACCELERATION = MN
`
`VF G-NB-N0 G-NB-N0 G-NB-N0 G-NB-N0 G-NB-N0
`
`VF G-NB-N0 G-NB-N0 G-NB-N0 G-NB-N0 Y-NB-N0
`
`F G-NB-N0 G-NB-N0 G-NB-N0 G-NB-N0 G-NB-N0
`
`F G-NB-N0 G-NB-N0 G-NB-N0 G-NB-N0 V-NB-N0
`
`M G-NB-N0 G-NB-N0 G-NB-N0 V-LB-N0 V-LB-N0
`
`M G-NB-N0 G-NB-N0 V-MB-N0 V-MB-S0 R-MB-S0
`
`C G-NB-N0 G-NB-N0 G-NB-N0 V-LB-N0 V-LB-N0
`
`V-NB-N0 V-NB-N0 V-LB-N0 V-L8-S0 R-MB-S0
`
`C
`
`VC V-NB-N0 V-L8-N0 V-LB-N0 V-MB-S0 R-MB-M12l
`
`VC V-LB-N0 V-LB-N0 V-MB-N0 R-HB-S0 R-HB-M0
`
`VH
`
`H
`
`M
`
`L
`
`~ VL
`
`VH
`
`H
`
`M
`
`L
`
`~ VL
`
`ACCELERATION = LN
`
`ACCELERATION = HP
`
`VF G-NB-N0 G-NB-N0 G-NB-N0 G-NB-N0 V-NB-N0
`
`VF G-NB-N0 G-NB-N0 V-NB-N0 V-NB-N0 R-MB-N0
`
`F G-NB-N0 G-NB-N G-NB-N0 V-NB-N0 Y-NB-N0
`
`Y-NB-N0 Y-NB-N0 Y-NB-N0 R-LB-N0 R-LB-N0
`
`M G-NB-N0 G-NB-N0 V-SB-N0 R-MB-S0 R-MB-S0
`
`V-LB-N0 R-MB-N0 R-MB-S0 R-HB-M0 R-HB-M0
`
`F
`
`M
`
`FIG. 9
`
`V-NB-N0 V-MB-N0 V-MB-N0 R-HB-N0 R-HB-S0
`
`C
`
`C R-MB-N0 R-MB-N0 R-MB-S0 R-HB-M0 R-HB-H0
`
`VC V-LB-N0 V-LB-N0 R-MB-N0 R-HB-S0 R-HB-M0
`
`VC R-MB-N0 R-HB-N0 R-HB-S0 R-HB-H0 R-HB-H0
`
`VH
`
`H
`
`M
`
`L
`
`VL
`
`VH
`
`H
`
`M
`
`L
`
`~ VL
`
`ACCELERATION = MP
`
`ACCELERATION = LP
`
`VF G-NB-N0 G-NB-N0 V-NB-N0 V-L8-N0 V-LB-N0
`
`VF G-NB-N0 G-NB-N0 G-NB-N0 V-LB-N0 V-LB-N0
`
`F G-NB-N0 G-NB-N0 V-NB-N0 V-LB-N0 R-LB-N0
`
`F G-NB-N0 G-NB-N0 G-NB-N0 V-LB-N0 V-LB-N0
`
`V-NB-N0 V-LB-N0 R-MB-N0 R-HB-N0 R-HB-S0
`
`V-L8-N0 R-MB-N0 R-MB-S0 R-HB-M0 R-HB-H0
`
`M
`
`C
`
`M G-NB-N0 V-NB-N0 V-LB-N0 R-MB-N0 R-HB-S0
`
`C V-NB-N0 V-LB-N0 R-MB-S0 R-HB-S0 R-HB-M0
`
`VC R-MB-N0 R-MB-S0 R-HB-M0 R-HB-H0 R-HB-H0
`
`VC V-LB-N0 R-LB-S0 R-MB-S0 R-HB-M0 R-HB-M0
`
`VH
`
`H
`
`M
`
`L
`
`~ VL
`
`VH
`
`H
`
`M
`
`L
`
`!~ VL
`
`10
`
`
`
`u.s. Patent
`
`Apr. 22, 2003
`
`Sheet 9 of 13
`
`US 6,553,130 Bl
`
`HF
`
`HB
`
`HL
`
`HR
`
`OF
`
`OB
`
`O~r OR
`
`FIG. 10
`
`DISTANCE
`
`RELATIVE
`VELOCITY
`
`f 83
`RELATIVE
`ACCELERATION
`
`FIG. 11
`
`11
`
`
`
`rJl
`
`.
`d .
`
`1
`0
`
`1
`0
`1
`0
`
`1
`0
`1
`0
`
`1
`0
`1
`0
`1
`0
`1
`0
`
`1
`0
`1
`0
`
`1
`0
`1
`1
`0
`0
`
`1
`0
`1
`0
`1
`1
`0
`0
`1
`1
`0
`0
`
`1
`0
`1
`1
`0
`0
`1
`1
`
`0
`0
`1
`1
`1
`
`0
`0
`0
`0
`
`..
`
`A
`
`1
`0
`1
`1
`0
`0
`
`1
`0
`0
`1
`1
`0
`0
`0
`0
`1
`1
`0
`0
`0
`0
`
`1
`1
`1
`1
`0
`0
`0
`0
`0
`0
`0
`0
`1
`1
`0
`0
`0
`0
`
`1
`1
`1
`0
`0
`0
`0
`0
`0
`1
`1
`1
`1
`1
`1
`
`0
`0
`0
`0
`0
`0
`0
`0
`0
`0
`0
`0
`1
`1
`1
`1
`1
`1
`
`1
`1
`1
`1
`1
`1
`1
`1
`1
`0
`0
`0
`0
`0
`0
`
`1 ..
`
`A
`
`0
`0
`0
`0
`0
`0
`0
`0
`0
`0
`0
`0
`1
`1
`1
`1
`1
`
`1
`1
`1
`1
`1
`1
`1
`1
`1
`1
`1
`1
`1
`1
`1
`
`1
`1
`1
`1
`1
`1
`1
`1
`1
`1
`1
`1
`0
`0
`0
`0
`0
`0
`
`68
`67
`66
`65
`64
`63
`62
`61
`60
`59
`58
`57
`56
`55
`54
`53
`52
`51
`50
`49
`48
`47
`46
`45
`44
`43
`42
`41
`40
`39
`38
`37
`36
`35
`
`1
`0
`1
`0
`1
`0
`1
`0
`
`1
`0
`1
`0
`1
`0
`1
`0
`
`1
`0
`1
`0
`
`1
`1
`0
`0
`1
`1
`0
`0
`
`1
`0
`1
`0
`1
`0
`1
`0
`
`0
`0
`0
`0
`0
`0
`0
`0
`1
`1
`1
`1
`0
`0
`0
`0
`0
`0
`0
`0
`1
`1
`1
`1
`1
`1
`1
`1
`0
`HR OF OB OL OR
`
`1
`0
`1
`0
`1
`1
`0
`0
`1
`1
`0
`0
`1
`1
`0
`0
`1
`1
`0
`0
`
`1
`1
`1
`1
`0
`0
`0
`0
`1
`1
`0
`0
`1
`1
`1
`1
`0
`0
`0
`0
`1
`1
`1
`1
`0
`0
`0
`0
`
`OBSTACLE
`
`1
`0
`0
`1
`0
`1
`0
`1
`0
`1
`1
`0
`1
`0
`0
`1
`0
`1
`0
`1
`0
`1
`0
`1
`0
`1
`0
`1
`0
`1
`0
`1
`0
`1
`0
`1
`0
`1
`0
`1
`0
`0
`0
`0
`0
`0
`0
`0
`0
`0
`0
`0
`0
`0
`0
`0
`0
`0
`HB
`HL
`HAZARD
`
`\~ ____________________________________ ~ _____________________________________ --J)
`
`FIG. 12
`
`y-
`
`..
`
`1
`1
`0
`0
`
`1
`1
`1
`1
`
`0
`0
`0
`0
`
`y
`
`1 ..
`1
`1
`1
`
`0
`0
`0
`0
`0
`0
`0
`0
`0
`0
`0
`0
`0
`0
`0
`0
`0
`0
`0
`0
`0
`0
`0
`0
`0
`0
`0
`0
`0
`VECTOR HF
`STATE
`
`34
`33
`32
`31
`30
`29
`28
`27
`26
`25
`24
`23
`22
`21
`20
`19
`18
`17
`16
`15
`14
`13
`12
`11
`10
`9
`8
`7
`6
`5
`4
`3
`2
`1
`
`0
`0
`0
`0
`
`12
`
`
`
`u.s. Patent
`
`Apr. 22, 2003
`
`Sheet 11 of 13
`
`US 6,553,130 Bl
`
`SCANNER INPUT
`DATA FILE
`
`99
`MODIFY HAZARD
`STATE VECTOR
`
`98
`
`FIG. 13
`
`13
`
`
`
`u.s. Patent
`
`Apr. 22, 2003
`
`Sheet 12 of 13
`
`US 6,553,130 Bl
`
`108 '\
`
`109 '\
`
`110 '\
`
`111 '\
`
`112 '\
`
`113 '\
`
`COLLISION
`STATE VECTOR
`
`INPUTDATA
`FILE
`
`/
`
`FUZZY
`DISTANCE
`MEMBERSHIP
`
`FUZZY
`VELOCITY
`MEMBERSHIP
`
`FUZZY
`ACCELERATION
`MEMBERSHIP
`
`HAZARD/OBJECT
`STATE VECTOR
`ADDRESS
`TRANSLATION
`
`FAM OUTPUT
`ADDRESS
`
`FIG. 14
`
`14
`
`
`
`u.s. Patent
`
`Apr. 22, 2003
`
`Sheet 13 of 13
`
`US 6,553,130 Bl
`
`HAZARD FRONT
`
`HAZARD BACK
`
`HAZARD RIGHT
`
`N
`
`OBJECT RIGHT
`
`y
`
`HAZARD/OBJECT
`VECTOR
`
`COLLISION
`VECTOR
`
`ACCESS FUZZY
`ASSOCIATIVE
`MEMORY
`(FAM)
`
`GENERATE OUTPUT
`EXPERT CONTROL
`SIGNALS
`
`DISTANCE FUZZY
`MEMBERSHIP
`
`VELOCITY FUZZY
`MEMBERSHIP
`
`ACCELERATION
`FUZZY MEMBERSHIP
`
`WARNING BRAKE STEERING
`
`FIG. 15
`
`15
`
`
`
`US 6,553,130 Bl
`
`1
`MOTOR VEHICLE WARNING AND
`CONTROL SYSTEM AND METHOD
`
`CROSS-REFERENCE TO RELATED
`APPLICATION
`
`This application is a continuation of Ser. No. 08/105,304,
`filed Aug. 11, 1993, abandoned.
`
`FIELD OF THE INVENTION
`
`This invention relates to a system and method for oper(cid:173)
`ating a motor vehicle, such as an automobile, truck, aircraft
`or other vehicle, wherein a computer or computerized sys(cid:173)
`tem is employed to assist and/or supplement the driver in the
`movement of the vehicle along a path of travel, such as a
`street or roadway and may be used to avoid obstacles and
`accidents.
`
`BACKGROUND OF THE INVENTION
`
`A major cause of human suffering is automobile acci(cid:173)
`dents. Approximately 49,000 people die in traffic accidents
`each year in the United States, and another three million are
`injured. The costs of death and injury accidents are stagger(cid:173)
`ing. According to the United States National Highway
`Traffic Safety Administration, crash damage and medical 25
`bills total $137 billion a year.
`Automobile designers offer many safety features, includ(cid:173)
`ing passenger restraints, improved braking systems, and
`body designs, intended to better protect automobile crash
`victims. But very little has been done in the area of auto(cid:173)
`matic vehicle control systems based on modern electronics,
`computer systems, and advanced real-time software. This is
`true despite rapidly increasing capabilities in these technolo(cid:173)
`gies and pervasive application in many other areas
`including, for example the business, entertainment, and
`medical fields. Vehicle guidance and control technology has,
`of course, been applied with great success in military
`defense systems, avionics systems and space exploration
`systems. But, this technology is costly and has not been
`commercialized.
`The opportunity exists today to develop cost effective,
`commercial automated vehicle control systems. New
`advances in low-cost hardware and software technology
`make implementation feasible. High-speed, parallel com(cid:173)
`puter architectures, specialized image-processing 45
`equipment, and advanced special computers such as math
`coprocessors are available. Advanced expert system imple(cid:173)
`mentations based on concepts such as fuzzy logic and neural
`networks, and new, improved scanning systems for sensing
`environments around moving vehicles make it very timely, 50
`indeed, to pursue new approaches.
`Work on these problems has begun. Intelligent vehicle/
`highway systems are being investigated with traffic control
`systems intended to minimize congestion. Vehicle location
`systems such as GPS (Global Positioning System) and route 55
`guidance systems are also being pursued. Certain systems
`for automated vehicle control have been proposed, including
`systems that scan the roadway directly ahead of a vehicle
`using radar!1idar or television and attempt to warn a driver
`of impending danger. Fuzzy logic expert systems for con- 60
`trolling vehicle speed (braking and throttle) based on scan(cid:173)
`ning the roadway ahead of a vehicle have been described.
`Road tracking with electronic vehicle guidance is being
`pursued. Fuzzy logic has been applied to braking systems in
`subway and train systems.
`While these developments are important, they fail to
`protect vehicles from many types of collisions or minimize
`
`2
`the damage therefrom. More particularly, such systems fail
`to exercise simultaneous, coordinated control over vehicle
`steering and speed, fail to take full advantage of identifica(cid:173)
`tion of different obstacle or hazard types using standard
`stored models of production vehicles and other commonly
`encountered roadway objects, fail to deal effectively with
`objects and hazards located simultaneously on different
`sides of the vehicle, and fail to capitalize fully on modern
`expert system decision and control technology, such as
`10 represented by fuzzy logic and neural network methods, to
`deal with more complex hazardous situations.
`
`SUMMARY OF THE INVENTION
`In a preferred form of the invention, a video scanning
`15 system, such as a television camera and/or one or more laser
`scanners mounted on the vehicle scan the road in front of the
`vehicle and generate image information which is computer
`analyzed per se or in combination with a range sensing
`system to warn the driver of hazardous conditions during
`20 driving by operating a display, such as a heads-up display,
`and/or a synthetic speech generating means which generates
`sounds or words of speech to verbally indicate such road
`conditions ahead of the vehicle.
`The preferred form of the invention provides audible
`and/or visual display means to cooperate in indicating to the
`driver of a motor vehicle both normal and hazardous road
`conditions ahead as well as driving variables such as dis(cid:173)
`tances to stationary objects, and other vehicles; the
`identification, direction of travel and speed of such other
`30 vehicles, and the identification of and distances to stationary
`or slowly moving objects such as barriers, center islands,
`pedestrians, parked cars poles, sharp turns in the road and
`other conditions. In addition, the image analyzing computer
`of the vehicle may be operated to scan and decode coded
`35 and/or character containing signs or signals generated by
`indicia or code generating other devices within or at the side
`of the road and indicating select road and driving conditions
`ahead.
`The computer is operable to analyze video and/or other
`40 forms of image information generated as the vehicle travels
`to identify obstacles ahead of the vehicle and, in certain
`instances, quantify the distance between the vehicle con(cid:173)
`taining same on the basis of the size of the identified vehicle
`or object and/or by processing received pulse-echo signals.
`Using such identifying information and comparing it with
`information on the shapes and sizes of various objects such
`as rear and front profiles of all production vehicles and the
`like and their relative sizes or select dimensions thereof,
`indications of distances to such objects may be computed
`and indicated as further codes. When the closing distance
`becomes hazardous, select vehicle subsystems may be auto-
`matically controlled by the computer as it continues to
`analyze image signals generated by the television camera. A
`first subsystem generates a first select code or codes which
`controls an electronic display, such as a heads-up display to
`cause it to display a warning indication, such as one or more
`flashing red light portions of the display or other lighted
`effect. For example, the display may project on the wind(cid:173)
`shield or dashboard such information as images of the
`controlled vehicle and other vehicles in and adjacent its path
`of travel and relative distances thereto as well as groups of
`characters defining same, colored and flashing warning
`lights and the like for pre-warning and warning purposes. A
`second subsystem generates a code or series of codes which
`65 control a sound generating means which generates a select
`sound such as a horn, buzzing sound and/or select synthetic
`speech warning of the hazardous condition detected and, in
`
`16
`
`
`
`US 6,553,130 Bl
`
`4
`the driver of a motor vehicle in controlling its operation to
`avoid hazardous conditions such as collisions with other
`vehicles, stationery objects or pedestrians.
`Another object is to provide a computerized system and
`method for controlling the speed of travel of a motor vehicle
`to lessen the chances of an accident while being driven by
`a person.
`Another object is to provide a system and method
`employing a television scanning camera mounted on a
`vehicle for scanning the field ahead, such as the image of the
`road ahead of the vehicle and a computer for analyzing the
`image signals generated wherein automatic image
`intensifying, or infra-red scanning and detection means is
`utilized to permit scanning operations to be effected during
`driving at night and in low light, snowing or fog conditions.
`Another object is to provide a system and method
`employing a television camera or other video scanning
`means mounted on a moving motor vehicle for scanning,
`detecting and identifying obstacles such as other vehicles
`ahead of such moving vehicle wherein the video image
`signals are analyzed to determine distances to such objects.
`Another object is to provide a computer controlled safety
`system for a motor vehicle which employs a television
`camera and an auxiliary scanning means to both identify
`25 obstacles in the path of the vehicle and determine distance
`therefrom on a real time and continuous basis for use in
`warning the operator of same and/or in controlling the
`operation of the vehicle to avoid a collision.
`
`3
`certain instances, generating sounds of select words of
`speech which may warn of same and/or suggest corrective
`action (s) by the vehicle operator or driver to avoid an
`accident.
`A third subsystem comes on-line and generates one or
`more codes which are applied to at least partly effect a
`corrective action such as by pulsing one or more motors or
`solenoids to apply the brakes of the vehicle to cause it to
`slow down. If necessary to avoid or lessen the effects of an
`accident, the third subsystem stops the forward travel of the 10
`vehicle in a controlled manner depending on the relative
`speeds of the two vehicles, and/or the controlled vehicle and
`a stationery object or structure and the distance therebe(cid:173)
`tween.
`A fourth subsystem, which may be part of or separate 15
`from the third subsystem may generate one or more codes
`which are applied to either effect partial and/or complete
`control of the steering mechanism for the vehicle to avoid an
`obstacle and/or lessen the effect of an accident. Either or
`both the third or fourth subsystem may also be operable to
`control one or more safety devices by controlling motors, 20
`solenoids or valves, to operate a restraining device or
`devices for the driver and passenger(s) of the vehicle, such
`as a safety belt tightening means, an air bag inflation means
`or other device designed to protect human beings in the
`vehicle.
`The second, and/or third and fourth subsystems may also
`be operable to effect or control the operation of additional
`warning means such as the horn, headlights and/or other
`warning lights on the vehicle or other warning means which
`operates to alert, flag or warn the driver of the approaching 30
`or approached vehicle or a pedestrian of the approaching
`hazardous condition. One or more of these subsystems may
`also be operable to generate and transmit one or more codes
`to be received and used by the approaching or approached
`vehicle or a roadside device to effect additional on-line 35
`warning(s) of the hazardous condition, and/or may be
`recorded on a disc or RAM (random access memory) for
`future analysis, if necessary.
`In a modified form of the invention, the vehicle warning 40
`system may also include a short wave receiving means to
`receive code signals from other vehicles and/or short wave
`transmitters at the side of or within the road for controlling
`the visual, audio and/or brake and steering means of the
`vehicle to avoid or lessen the effects of an accident and/or to 45
`maintain the vehicle in-lane and in proper operating condi(cid:173)
`tion as it travels.
`The systems and methods of this invention preferably
`employ computerized image analyzing techniques of the
`types disclosed and defined in such patents of mine as U.S. 50
`Pat. Nos. 4,969,038 and 4,979,029 and references cited in
`the file wrappers thereof as well as other more recent patents
`and include the use of known artificial intelligence, neural
`networking and fuzzy logic computing electronic circuits.
`While the invention is described herein principally in 55
`connection with an automobile on a roadway, it may be used
`in connection with controlling any powered vehicle, includ(cid:173)
`ing a motor vehicle, a boat, a train, or an aircraft.
`Accordingly it is a primary object of this invention to
`provide a new and improved system and method for con- 60
`trolling the operation of a powered vehicle.
`Another object is provide a system and method for
`assisting the driver of a powered vehicle in controlling its
`operation to avoid an accident or hazardous driving condi(cid:173)
`tion.
`Another object is to provide a system and method
`employing computerized image analysis to control or assist
`
`65
`
`BRIEF DESCRIPTION OF DRAWINGS
`The various hardware and software elements used to carry
`out the invention described herein are illustrated in the form
`of block diagrams, flow charts, and depictions of neural
`network and fuzzy logic algorithms and structures. The
`preferred embodiment is illustrated in the following figures:
`FIG. 1 is a block diagram of the overall Motor Vehicle
`Warning and Control System illustrating system sensors,
`computers, displays, input/output devices and other key
`elements.
`FIG.2 is a block diagram of an image analysis computer
`19 of the type that can be used in the Vehicle Hazard
`Avoidance System herein of FIG. 1.
`FIG. 3 illustrates a neural network of the type useful in the
`image analysis computer of FIG. 4.
`FIG. 4 illustrates the structure of a Processing Element
`(PE) in the neural network of FIG. 3.
`FIG. 5 is an alternate embodiment of a neural network
`image processor useful in the system of FIG. 1.
`FIG. 6 is a flow diagram illustrating the overall operation
`of the Motor Vehicle Warning and Control System of FIG.
`1.
`
`FIG. 7 illustrates typical input signal membership func(cid:173)
`tions for fuzzy logic algorithms useful in the Motor Vehicle
`Warning and Control System of FIG. 1.
`FIG. 8 illustrates typical output signal membership func(cid:173)
`tions for fuzzy logic algorithms useful in the Motor Vehicle
`Warning and Control System of FIG. 1.
`FIG. 9 illustrates typical Fuzzy Associative Memory
`(FAM) maps for the fuzzy logic algorithms useful in the
`Motor Vehicle Warning and Control System of FIG. 1.
`FIG. 10 is a Hazard/Object state vector useful in imple(cid:173)
`menting the Fuzzy Logic Vehicle Warning and Control
`System.
`FIG. 11 is a Hazard Collision Control vector useful in
`implementing the Fuzzy Logic vehicle Warning and Control
`System.
`
`17
`
`
`
`US 6,553,130 Bl
`
`5
`FIG. 12 is a table of Hazard/Object state vectors indicat(cid:173)
`ing possible combinations of hazards and objects useful in
`the Fuzzy Associative Memory access system used herein.
`FIG. 13 is a more detailed logic flow diagram for the
`analysis of detection signals prior to accessing fuzzy logic
`control structures in the Motor Vehicle Warning and Control
`System.
`FIG. 14 is a more detailed Logic Flow Diagram for the
`Fuzzy Associative Memory (FAM) selection processing.
`FIG. 15 is an example system flow illustrating the opera-
`tion of the Motor Vehicle Warning and Control System.
`
`DETAILED DESCRIPTION
`In FIG. 1 is shown a computerized control system 10 for
`controlling the operation of a motor vehicle to prevent or
`lessen the effects of accidents such as collisions with sta(cid:173)
`tionery and/or moving objects such as other vehicles. The
`system 10 employs a control computer or microprocessor 11
`mounted on the vehicle and operable to receive and gate
`digital signals, such as codes and control signals from
`various sensors, to one or more specialized computers and
`from such computers to a number of servos such as electric
`motors and lineal actuators or solenoids, switches and the
`like, speakers and display drivers to perform either or both 25
`the functions of audibly and/or visually informing or warn(cid:173)
`ing the driver of the vehicle of a hazardous road condition
`ahead and/or to effect controlled braking and steering
`actions of the vehicle.
`A RAM 12 and ROM 13 are connected to processor 11 to 30
`effect and facilitate its operation. A television camera(s) 16
`having a wide angle lens 16L is mounted at the front of the
`vehicle such as the front end of the roof, bumper or end of
`the hood to scan the road ahead of the vehicle at an angle
`encompassing the sides of the road and intersecting roads. 35
`The analog signal output of camera 16 is digitized in an AID
`convertor 18 and passed directly to or through a video
`preprocessor 51 to microprocessor 11, to an image field
`analyzing computer 19 which is provided, implemented and
`programmed using neural networks and artificial intelli- 40
`gence as well as fuzzy logic algorithms to (a) identify
`objects on the road ahead such as other vehicles, pedestrians,
`barriers and dividers, turns in the road, signs and symbols,
`etc., and generate identification codes, and (b) detect dis(cid:173)
`tances from such objects by their size ( and shape) and 45
`provide codes indicating same for use by a decision
`computer, 23, which generates coded control signals which
`are applied through the computer 11 or are directly passed to
`various warning and vehicle operating devices such as a
`braking computer or drive, 35, which operates a brake servo 50
`33, a steering computer or drive(s) 39 and 40 which operate
`steering servos 36; a synthetic speech signal