(12) United States Patent
`Lemelson et al.
`
`(10) Patent N0.:
`(45) Date of Patent:
`
`US 6,553,130 B1
`Apr. 22, 2003
`
`US006553130B1
`
`(54) MOTOR VEHICLE WARNING AND
`CONTROL SYSTEM AND METHOD
`
`JP
`JP
`JP
`
`4—219900
`5'124529
`5—143897
`
`8/1992
`5/1993
`6/1993
`
`............... .. 340/903
`
`............... .. 340/903
`
`(76)
`
`Inventors: Jerome H. Lemelson, 868 Tyner Way,
`Incline Village, NV (US) 89450;
`gglliit_lI.)§d(e5:§n%5728f88 G1enEag1e’
`’
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`Utstct 154(b) by 1543 days.
`
`( * ) Notice:
`
`(21) App1' N03 08/671353
`(22)
`Filed:
`Jun. 28, 1996
`
`Related U-S- APP11e3t1011 Data
`
`(63) Continuation of application No. 08/105,304, filed on Aug.
`11, 1993, now abandoned.
`Int. C17 .................................................. coax 9/00
`(51)
`(52) US. Cl.
`..................... .. 382/104; 340/435; 340/436;
`340/903; 382/106
`Of Search ............................... ..
`348/115; 116; 118; 119; 135; 139; 140;
`142; 148; 149; 382/104; 215; 209; 106;
`156; 103; 217; 218; 340/907; 435; 903;
`901; 436; 180/168; 169; 167; 271; 274;
`275; 364/424-02; 460; 461; 436—437; 425-04;
`42501; 42601; 443; 424032; 395/905;
`900; 913
`
`(56)
`
`References Cited
`Us. PATENT DOCUMENTS
`
`3/1981 G00dfiCh ~~~~~~~~~~~~~~~~~~~~~ ~~ 356/4
`4;257;703 A
`12/1986 Che)’ ~ - - - - - -
`- - - - -~ 340/903
`4;626;850 A
`
`4/1989 Park
`““ " 340/901
`498259211 A
`10/1989 Dye ......................... .. 340/903
`4,872,051 A
`(List continued on next page.)
`FOREIGN PATENT DOCUMENTS
`
`OTHER PUBLICATIONS
`Abu—Mostafa, “Information Theory, Complexity, and Neu-
`ral Networks,” IEEE Communications, pp. 25-28 (Nov.
`1989)’
`“
`.
`.
`Aggarwal et al., On the Computation of Motion from
`Sequences of Images—A Review,” Proceedings of the
`IEEE, pp. 917-35 (Aug. 1988).
`
`(List continued on next page.)
`
`Primary Examiner—Timothy M. Johnson
`(74) Attorney, Agent, or Firm—Louis J. Hoffman
`
`i
`
`ABSTRACT
`h d
`i
`t
`t
`i
`
`(57)
`h 1
`f
`h d
`d
`t
`i
`A t
`t
`eaf§f;’S°§fV:aIfe9
`prjzzjfflgnacfljenfs
`one form’ a television Camera is mounted on a Vehicle and
`scans the roadway ahead of the vehicle as the vehicle travels.
`Cgntinugusly generated Videg picture Signals Output
`the
`camera are electronically processed and analyzed by an
`image analyzing computer, which generates codes that serve
`to identify obstacles. A decision computer mounted in the
`controlled vehicle receives such code signals along with
`code signals generated by the speedometer or one or more
`sensors sensing steering mechanism operation and generates
`control signals. Such code signals may be displayed, and a
`s nthetic s eech or s ecial sound eneratin and warnin
`nieans used: to warn tllie driver of thegvehicle if approachin:
`and existing.hazards. The system.may also use the control
`signals, particularly through application of fuzzy logic, to
`control the operation of the brakes and steering mechanism
`of the vehicle to avoid or lessen the effects of a collision. In
`a particular form,
`the decision computer may select the
`evasive aenen taken from a number of eneieee’ depending
`on whether and where the detection device senses other
`vehicles or obstacles.
`
`JP
`
`4-15799
`
`1/1992
`
`............... .. 340/905
`
`55 Claims, 13 Drawing Sheets
`
`INPUT DATA FILE
`
`
`
`COLLISION
`
`HAZARD
`
`
`
`HAZARDIOBJECT
`VECTOR
`
`
`
`ACCESS FUZZV
`ASSOCIATIVE
`MEMORY
`(FAM)
`
`GENERATE OUTPUT
`EXPERT CONTROL
`SIGNALS
`
`
`WARNING BRAKE STEERING
`
`ACCELERATION
`FUZZV MEMBERSHIP
`
`Valeo Exhibit 1005_001
`
`
`
`
`
`Valeo Exhibit 1005_001
`
`

`
`US 6,553,130 B1
`Pane 2
`
`U.S. PATENT DOCUMENTS
`4,901,362 A * 211990 Terzian ....................... 3821103
`511990 Beggs et al. ................ 3401904
`4,926,170 A
`4,931,937 A * 611990 Kakinami et al. .......... 1801169
`611990 Lemelson ................... 3401439
`4,933,852 A
`1111990 Lemelson ................... 3581107
`4,969,038 A
`1211990 Lemelson .................... 358193
`4,979,029 A
`511991 Yasunobu et al. .......... 3951905
`5,018,689 A
`811991 Maekawa et al. .............. 35611
`5,039,217 A
`111992 Kurami et al. ......... 3641424.02
`5,081,585 A
`211992 Shyu .......................... 3401904
`5,091,726 A
`611992 Beggs et al. ................ 3401904
`5,122,796 A
`911992 Zechnall ..................... 3401905
`5,146,219 A
`1111992 Mayeaux et al. ........... 3401937
`5,161,107 A
`1111992 Summer ..................... 3401905
`5,164,904 A
`111993 Kajiwara .................... 3401903
`5,177,462 A
`111993 Hancock ..................... 3401961
`5,179,377 A
`211993 Adachi et al. .............. 3951905
`5,189,619 A
`5,197,562 A * 311993 Kakinami et al. .......... 1801169
`711993 Kakinami et al. .......... 1801169
`5,230,400 A
`911993 Taylor ........................ 3401903
`5,249,157 A
`111994 Bottesch ................ 3641424.05
`5,276,620 A
`111994 Iizuka et al. ................ 3641461
`5,278,764 A
`311994 Tsai ........................... 3401468
`5,298,882 A
`411994 Maekawa ................... 3401903
`5,304,980 A
`411994 Saneyoshi ................... 1801167
`5,307,136 A
`511994 Shaw et al. ................. 1801169
`5,314,037 A
`711994 Kohsaka ..................... 3401459
`5,327,117 A
`711994 Butsuen et al. ............. 1801169
`5,332,057 A
`811994 Abst et al. .................. 3401903
`5,339,075 A
`811994 O'Brien et al. ............... 367196
`5,341,344 A
`1011994 Davidian .................... 1801169
`5,357,438 A
`1111994 Broxmeyer ................. 3401903
`5,369,591 A
`5,545,960 A * 811996 Ishikawa ............. 3641424.032
`OTHER PUBLICATIONS
`
`Alspector, "Neural-Style Microsystems that Learn," ZEEE
`Communications, pp. 29-36 (Nov. 1989).
`Casasent, "Optics and Neural Nets," Chapter 16 in Carpen-
`ter et al., eds., Neural Networks for Vision and Zrnage
`Processing, pp. 437-48, (MIT Press 1992).
`Cox, "Fuzzy Fundamentals," ZEEE Spectrum, pp. 58-61
`(Oct. 1992).
`Hammerstrom, "Neural Networks at Work," ZEEE Spec-
`trum, pp. 26-32 (Jun. 1993).
`Hush et al., "Progress in Supervised Neural Networks,"
`ZEEE Signal Processing, pp. 8-39 (Jan. 1993).
`Jurgen, "Smart Cars and Highways Go Global," ZEEE
`Spectrum, pp. 26-36 (May 1991).
`Kittler et al., eds., Zrnage Processing System Architectures,
`Chapter 4, pp. 49-81, and Chapter 5, pp. 85-101 (John
`Wiley & Sons 1985).
`Kosko et al., "Fuzzy Logic," ScientificAmerican, pp. 76-81
`(Jul. 1993).
`
`\
`
`,
`
`
`
`Lee, "Fuzzy Logic in Control Systems: Fuzzy Logic Con-
`troller, Part 11," ZEEE Tramaction on Systems, Man, and
`Cybernetics, pp. 419435 (vol. 20, No. 2, Mar./Apr. 1990).
`Lippmann, "An Introduction to Computing with Neural
`Nets," ZEEE ASSP, pp. 4-22 (Apr. 1987).
`Lippmann, "Pattern Classification Using Neural Networks,"
`ZEEE Communicatiom, pp. 47-50, 59-64 (Nov. 1989).
`Lisboa, ed., Neural Networks4urrent Applications, Chap-
`ter 1, pp. 1-34, Chapter 2, pp. 35-48, Chapter 7, pp.
`123-147 (Chapman & Hall 1992).
`Lupo, "Defense Applications of Neural Networks," ZEEE
`Communications, pp. 82-88 (Nov. 1989).
`Maresca et al., "Parallel Architectures for Vision," Proceed-
`ings of the ZEEE, pp. 970-981 (vol. 76, No. 8, Aug. 1988).
`Nijhuis et al., "Evaluation of Fuzzy and Neural Vehicle
`cdntrol," Institution of Electrical ~ h ~ i n e e r s , pp. 447-452
`(1992).
`Pearson, ed., Zrnage Processing, Chapter 8, pp. 141-155, and
`Chapter 10, pp. 169-189 (McGraw-Hill 1991).
`Psaltis et al., "Optoelectronic Implementations of Neural
`Networks," ZEEE Communications, pp. 37-40, 71 (Nov.
`1989).
`Roth, "Neural Networks for Extraction of Weak Targets in
`High Clutter Environments," ZEEE Tramactiom on Sys-
`tems, Man, and Cybernetics, pp. 1210-1217 (Sep./Oct.
`1989).
`Schwartz et al., "Fuzzy Logic Flowers in Japan," ZEEE
`Spectrum, pp. 32-35 (Jul. 1992).
`Soucek et al., Neural and Massively Parallel Computers,
`Chapter 12, pp. 245-276 (John Wiley & Sons 1988).
`Suaya et al., eds., VLSZ and Parallel Computation, Chapter
`1, pp. 1-84, and Chapter 5, pp. 390-415 (Morgan Kaufmann
`1990).
`Teuber, Digital Zrnage Processing, Chapter 1, pp. 1-30,
`Chapter 2, pp. 31-70, and Appendix D, pp. 254-255 (Pren-
`tice Hall 1993).
`Wasserman, Neural Computing: Theory and Practice, Chap-
`ter 1, pp. 11-26, Chapter 2, pp. 2 7 4 2 , Chapter 3, pp. 43-59,
`and Chapter 9, pp. 151-166 (Van Nostrand Reinhold 1989).
`Yuhas et al., "Integration of Acoustic and Visual Speech
`Signals Using Neural Networks," ZEEE Spectrum, pp. 65-71
`(Nov. 1989).
`Shekhar et al., Design and Validation of Head up Displays
`for Navigation in IVHS, VNZS'91, Oct. 1991, pp. 537-542.
`"NHTSA IVHS Plan," National Highway Safety Adminis-
`tration U.S. Department of Transportation, Jun. 12, 1992.
`Bosacchi et al, Fuzzy Logic Technology & the Intelligent
`Highway System (IHS); ZEEE, 1993, pp. 65-70.
`Rocket al., "Intelligent Road Transit: The Next Generation,"
`AZ EXPERT, Apr. 1994, pp. 16-24 (not prior art).
`* cited by examiner
`
`Valeo Exhibit 1005_002
`
`

`
`U.S. Patent
`
`Apr. 22,2003
`
`Sheet 1 of 13
`
`OR COMP
`
`p H q / l o
`pTGyim-yjq
`COMPUTER p & & e I
`
`COMPUTER
`
`DECISION
`COMPUTER
`
`SPEECH
`RECOGNITION
`COMPUTER
`
`pmtjMEM1
`
`GEN COMPUTER
`
`HEAD LIGHT
`*
`CONTROL
`
`n
`
`CONTROL
`
`f7LtFd
`
`SENSOR
`
`SPEEDOMETER
`
`w
`
`DISPLAY DRIVER
`
`INTERFACE
`
`lolspLAvl ~TH-I
`KEYBOARD 0
`
`FIG. 1
`
`Valeo Exhibit 1005_003
`
`

`
`U.S. Patent
`
`Apr. 22,2003
`
`Sheet 2 of 13
`
`16\ +(I7 (18
`- AID
`CAMERA
`I
`
`51 .,
`
`VIDEO
`PREPROCESSOR .
`I
`
`DISPLAY
`
`IMAGE BUS
`'50
`
`(56
`CONTROL
`PROCESSOR
`
`BUS INTERFACE 9
`
`MICRO PROCESSOR
`CONTROLLER
`
`FIG. 2
`
`Valeo Exhibit 1005_004
`
`

`
`U.S. Patent
`
`Apr. 22,2003
`
`Sheet 3 of 13
`
`US 6,553,130 BI
`
`INPUTS
`
`FIRST HIDDEN
`LAYER
`
`SECOND HIDDEN OUTPUT
`LAYER
`LAYER
`
`6 1
`
`xx FIG. 4
`
`Valeo Exhibit 1005_005
`
`

`
`U.S. Patent
`
`Apr. 22,2003
`
`Sheet 4 of 13
`
`IMAGE DATA
`
`I
`
`IMAGE
`PROCESSOR
`
`I
`
`VIRTUAL D
`
`FIG. 5
`
`Valeo Exhibit 1005_006
`
`

`
`U.S. Patent
`
`Apr. 22,2003
`
`Sheet 5 of 13
`
`DATA
`%
`
`-
`-
`
`MOTION
`WEATHER
`I
`IMAGE
`RADAR
`VEHICLE OVERRIDE
`
`DETECTION SIGNAL ANALYSIS
`
`r 7 4
`
`75
`
`1
`
`I
`
`FUZZY ASSOCIATIVE MEMORY (FAM) SELECTION
`
`76
`
`I
`
`+ f 7 7
`
`r
`
`/ 7 8
`
`r
`
`/ 7 9
`
`r
`
`/-80
`
`FAM-1
`
`FAMP
`
`FAM-3
`
`FAM-N
`
`v
`
`-
`
`STEERING
`
`r
`
`7
`
`r
`
`/81
`
`CONTROL SIGNAL GENERATOR
`DEFUZZlFlER
`
`STEERING SERVO
`
`FIG. 6
`
`Valeo Exhibit 1005_007
`
`

`
`U.S. Patent
`
`Apr. 22,2003
`
`Sheet 6 of 13
`
`4
`VERY
`
`A
`
`FAR
`
`VERY FAR
`
`DISTANCE
`
`VERY LOW LOW
`
`MEDIUM
`
`HIGH
`
`VERY HIGH
`
`+
`
`b
`
`VELOCITY
`
`ACCELERATION
`
`FIG. 7
`
`Valeo Exhibit 1005_008
`
`

`
`U.S. Patent
`
`Apr. 22,2003
`
`Sheet 7 of 13
`
`A
`
`a -
`I
`P 1
`W m
`5
`W
`5
`
`A
`
`GREEN
`
`YELLOW
`
`RED
`
`WARNING LEVEL
`
`LOW BRAKE
`
`MEDIUM BRAKE
`
`HIGH BRAKE
`
`BRAKING PRESSURE
`
`LOW 0
`
`MEDIUM 0
`
`HIGH 0
`
`Q - I
`V) a 1
`W m
`5 I
`
`n -
`I
`V) r r 1
`W m
`I
`W z
`
`b
`
`b
`
`b
`
`STEERING ANGLE
`
`FIG. 8
`
`Valeo Exhibit 1005_009
`
`

`
`U.S. Patent
`
`Apr. 22,2003
`
`Sheet 8 of 13
`
`Valeo Exhibit 1005_010
`
`

`
`U.S. Patent
`
`Apr. 22,2003
`
`Sheet 9 of 13
`
`FIG. 10
`
`DISTANCE
`
`RELATIVE
`VELOCITY
`
`RELATIVE
`ACCELERATION
`
`FIG. 11
`
`Valeo Exhibit 1005_011
`
`

`
`U.S. Patent
`
`Apr. 22,2003
`
`Sheet 10 of 13
`
`O r O r O r O r O r O r
`
`O r O r O r O r O r
`
`O O r - O 0 - - 0 ~ = 0 -
`
`0 0 r r 0 r
`
`q b
`
`0 0 0 0 r - r O O
`
`r r O O r r O r
`
`0 0 r r O r
`
`0 0 0 0 ~ - 0 0 0 0 0 0 0 0 ~ ~ - - 0 0 0 0 r - 0 0 0 0 r r 0 0 r
`
`~ ~ r r ~ - 0 0 0 0 0 0 0 0 0 0 0 0
`r r r r r r 0 0 0 0 0 0 r r r
`
`r r r r r r 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 r r r r r r r r r
`
`0 0 0 0 0 0 r r r r r r r r r r r r r r r r r - r r r r r r - - -
`
`3%%%39q2g333S338G%3S%3%3385%88%8%%
`
`a
`0
`
`36 s
`g m
`0 0
`
`O r O r O r O r
`
`O r O r O r O r
`
`o r o r o r o r
`
`0 0 r r O o r r O r o r
`
`o o r r o o r r o o r r o o r r o r o r
`
`+
`
`o o o o r r r r 0 0 0 0 r r r . - O O r r O O O O r r r r 0 0 r r
`
`! $ o r r r r r r r r 0 0 0 0 0 0 0 0 ~ r r r O O O O O O O O r - r r
`
`~ ~ 0 0 0 0 0 0 0 0 0 r r r r r r r r r r r r O O O O O O O O 0 0 0 0
`
`L
`
`~ 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
`
`Valeo Exhibit 1005_012
`
`

`
`U.S. Patent
`
`Apr. 22,2003
`
`Sheet 11 of 13
`
`SCANNER INPUT
`
`b
`
`-
`/ 99
`MODIFY HAZARD -
`STATE VECTOR
`A
`
`FIG. 13
`
`Valeo Exhibit 1005_013
`
`

`
`U.S. Patent
`
`Apr. 22,2003
`
`Sheet 12 of 13
`
`COLLISION
`
`STATE VECTOR / FILE
`INPUT DATA
`
`I
`
`FUZZY
`DISTANCE
`MEMBERSHIP
`
`FUZZY
`VELOCITY
`MEMBERSHIP
`
`FUZZY
`ACCELERATION
`MEMBERSHIP
`
`HAZARDIOBJECT
`STATE VECTOR
`ADDRESS
`TRANSLATION
`
`FAM OUTPUT
`ADDRESS
`
`FIG. 14
`
`112,
`
`Valeo Exhibit 1005_014
`
`

`
`U.S. Patent
`
`Apr. 22,2003
`
`Sheet 13 of 13
`
`INPUT DATA FILE
`
`OBJECT FRONT
`
`COLLISION -
`
`HAZARD
`
`VECTOR
`
`Y
`
`v
`1 0 0 0 1 0 0
`
`T
`
`HAZARDIOBJ ECT
`VECTOR
`
`v
`ACCESS FUZZY
`ASSOCIATIVE
`MEMORY
`(FAMI
`
`L
`
`GENERATE OUTPUT
`EXPERT CONTROL
`
`COLLISION
`VECTOR
`
`t
`DISTANCE FUZZY
`MEMBERSHIP
`
`I
`
`VELOCITY FUZZY
`MEMBERSHIP
`
`ACCELERATION
`FUZZY MEMBERSHIP
`
`WARNING BRAKE STEERING
`
`FIG. 15
`
`Valeo Exhibit 1005_015
`
`

`
`US 6,553,130 B1
`
`1
`MOTOR VEHICLE WARNING AND
`CONTROL SYSTEM AND METHOD
`
`CROSS-REFERENCE TO RELATED
`APPLICATION
`m i s application is a continuation of ser, N ~ ,
` 081105,304,
`filed Aug. 11, 1993, abandoned.
`
`BACKGROUND OF THE INVENTION
`
`FIELD OF THE INVENTION
`This invention relates to a system and method for oper-
`ating a motor vehicle, such as an automobile, truck, aircraft
`or other vehicle, wherein a computer or computerized sys-
`tem is employed to assist andlor supplement the driver in the
`such as a
`a path of
`movement of the
`and
`street Or roadway and may be wed to avoid
`accidents.
`
`2
`the damage therefrom. More particularly, such systems fail
`to exercise simultaneous, coordinated control over vehicle
`steering and speed, fail to take full advantage of identifica-
`tion of different obstacle or hazard types using standard
`s stored models of production vehicles and other commonly
`encountered roadway objects, fail to deal effectively with
`objects and hazards located simultaneously on different
`sides of the vehicle, and fail to capitalize fully on modern
`expert system decision and control technology, such as
`10 represented by fuzzy logic and neural networkhethods, to
`deal with more complex hazardous situations.
`SUMMARY OF THE INVENTION
`In a preferred form of the invention, a video scanning
`system, such as a television camera andlor one or more laser
`scanners mounted on the vehicle scan the road in front of the
`vehicle and generate image information which is computer
`analyzed per se or in combination with a range sensing
`system to warn the driver of hazardous conditions during
`A major cause of human suffering is automobile acci- 20 driving by operating a display, such as a heads-up display,
`dents. Approximately 49,000 people die in traffic accidents
`and/or a synthetic speech generating means which generates
`each year in the United States, and another three million are
`sounds or words of speech to verbally indicate such road
`injured. The costs of death and injury accidents are stagger-
`conditions ahead of the vehicle,
`ing.
`the United States
`The preferred form of the invention provides audible
`Administration, crash damage and
`Traffic
`2s and/or visual display means to cooperate in indicating to the
`bills total $137 billion a year.
`driver of a motor vehicle both normal and hazardous road
`Automobile designers offer many safety features, includ-
`conditions ahead as well as driving variables such as dis-
`ing Passenger restraints, improved braking systems, and
`tances to stationary objects, and other vehicles; the
`body designs, intended to better protect automobile crash
`identification, direction of travel and speed of such other
`victims. But very little has been done in the area of auto- 30 vehicles, and the identification of and distances to stationary
`matic vehicle control systems based on modern electronics,
`or slowly moving objects such as barriers, center islands,
`computer systems, and advanced real-time software. This is
`pedestrians, parked cars poles, sharp turns in the road and
`other conditions, In addition, the image analyzing computer
`true despite rapidly increasing capabilities in these technolo-
`gies and pervasive application in many other areas
`of the vehicle may be operated to scan and decode coded
`including, for example the business, entertainment, and 35 and/or character containing signs or signals generated by
`indicia or code generating other devices within or at the side
`medical fields. Vehicle guidance and control technology has,
`of course, been applied with great success in military
`of the road and indicating select road and driving conditions
`defense systems, avionics systems and space exploration
`ahead,
`systems. But, this
`is
`and has not been
`The computer is operable to analyze video and/or other
`commercialized.
`40 forms of image information generated as the vehicle travels
`The opportunity exists today to develop cost effective,
`to identify obstacles ahead of the vehicle and, in certain
`commercial automated vehicle control systems. New
`instances, quantify the distance between the vehicle con-
`advances in low-cost hardware and software technology
`taining same on the basis of the size of the identified vehicle
`make implementation feasible. High-speed, parallel corn-
`or object andlor by processing received pulse-echo signals.
`puter architectures, specialized image-processing 45 Using such identifying information and comparing it with
`equipment, and dvanced special computers such as math
`information on the shapes and sizes of various objects such
`coprocessors are available. Advanced expert system imple-
`as rear and front profiles of all production vehicles and the
`mentations based on concepts such as fuzzy logic and neural
`like and their relative sizes or select dimensions thereof,
`networks, and new, improved scanning systems for sensing
`indications of distances to such objects may be computed
`environments around moving vehicles make it very timely, so and indicated as further codes. When the closing distance
`indeed, to pursue new approaches.
`becomes hazardous, select vehicle subsystems may be auto-
`Work on these problems has begun. Intelligent vehicle1
`matically controlled by the computer as it continues to
`highway systems are being investigated with traffic control
`analyze image signals generated by the television camera. A
`systems intended to minimize congestion. Vehicle location
`first subsystem generates a first select code or codes which
`systems such as GPS (Global Positioning System) and route ss controls an electronic display, such as a heads-up display to
`guidance systems are also being pursued. Certain systems
`cause it to display a warning indication, such as one or more
`for automated vehicle control have been proposed, including
`flashing red light portions of the display or other lighted
`systems that scan the roadway directly ahead of a vehicle
`effect. For example, the display may project on the wind-
`using radarilidar or television and attempt to warn a driver
`shield or dashboard such information as images of the
`of impending danger. Fuzzy logic expert systems for con- 60 controlled vehicle and other vehicles in and adjacent its path
`trolling vehicle speed (braking and throttle) based on scan-
`of travel and relative distances thereto as well as groups of
`ning the roadway ahead of a vehicle have been described.
`characters defining same, colored and flashing warning
`Road tracking with electronic vehicle guidance is being
`lights and the like for pre-warning and warning purposes. A
`pursued. Fuzzy logic has been applied to braking systems in
`second subsystem generates a code or series of codes which
`subway and train systems.
`65 control a sound generating means which generates a select
`sound such as a horn, buzzing sound andlor select synthetic
`While these developments are important, they fail to
`protect vehicles from many types of collisions or minimize
`speech warning of the hazardous condition detected and, in
`
`Valeo Exhibit 1005_016
`
`

`
`US 6,553,130 B1
`
`3
`4
`the driver of a motor vehicle in controlling its operation to
`certain instances, generating sounds of select words of
`avoid hazardous conditions such as collisions with other
`speech which may warn of same andlor suggest corrective
`vehicles, stationery objects or pedestrians.
`action (s) by the vehicle operator or driver to avoid an
`accident.
`Another object is to provide a computerized system and
`A third subsystem comes on-line and generates one or 5 method for controlling the speed of travel of a motor vehicle
`more codes which are applied to at least partly effect a
`to lessen the chances of an accident while being driven by
`corrective action such as by pulsing one or more motors or
`a person.
`solenoids to apply the brakes of the vehicle to cause it to
`~object is to provide a system and method t h ~ ~
`
`
`
`
`
`
`~
`~
`slow down. If necessary to avoid or lessen the effects of an
`employing a television scanning camera mounted on a
`accident, the third
`the forward
`the 10 vehicle for scanning the field ahead, such as the image of the
`vehicle in a controlled manner depending on the relative
`road ahead of the vehicle and a computer for analyzing the
`speeds of the two vehicles, andlor the controlled vehicle and
`image signals generated wherein automatic image
`a stationery object or structure and the distance therebe-
`intensifying, or infra-red scanning and detection means is
`tween.
`utilized to permit scanning operations to be effected during
`A fourth subsystem, which may be part of or separate
`IS driving at night and in low light, snowing or fog conditions.
`from the third subsystem may generate one or more codes
`Another object is to provide a 'ystem and method
`which are applied to either effect partial andlor complete
`control of the steering mechanism for the vehicle to avoid an
`camera Or
`scanning
`a
`obstacle andlor lessen the effect of an accident. Either or
`for scanning,
`means
`On a
`both the third or fourth subsystem may also be operable to
`detecting and identifying obstacles such as other vehicles
`control one or more safety devices by controlling motors, 20 ahead of such moving vehicle wherein the video image
`signals are analyzed to determine distances to such objects.
`solenoids or valves, to operate a restraining device or
`devices for the driver and passenger(s) of the vehicle, such
`Another object is to provide a computer controlled safety
`as a safety belt tightening means, an air bag inflation means
`system for a motor vehicle which employs a television
`camera and an auxiliary scanning means to both identify
`or other device designed to protect human beings in the
`25 obstacles in the path of the vehicle and determine distance
`vehicle.
`me second,
`therefrom on a real time and continuous basis for use in
`third and fourth subsystems may also
`warning the operator of same andlor in controlling the
`be operable to effect or control the operation of additional
`the
`a
`warning means such as the horn, headlights andlor other
`warning lights on the vehicle or other warning means which
`BRIEF DESCRIPTION OF DRAWINGS
`operates to alert, flag or warn the driver of the approaching 30
`The various hardware and software elements used to carry
`or approached vehicle or a pedestrian of the approaching
`out the invention described herein are illustrated in the form
`hazardous condition. One or more of these subsystems may
`of block diagrams, flow charts, and depictions of neural
`also be operable to generate and transmit one or more codes
`network and fuzzy logic algorithms and structures. The
`to be received and used by the approaching or approached
`35 preferred embodiment is illustrated in the following figures:
`vehicle or a roadside device to effect additional on-line
`FIG. 1 is a block diagram of the overall Motor Vehicle
`warning(s) of the hazardous condition, andlor may be
`recorded on a disc or RAM (random access memory) for Warning and
`system
`computers, displays, inputloutput devices and other key
`future analysis, if necessary.
`elements.
`In a modified form of the invention, the vehicle warning
`is a
`an image
`diagram
`system may also include a short wave receiving means to 40
`the type that can be wed in the
`receive code signals from other vehicles andlor short wave
`'ystem herein of
`transmitters at the side of or within the road for controlling
`FIG. 3 illustrates a neural network of the type useful in the
`the visual, audio
`brake and steering means of the
`4.
`image
`vehicle to avoid or lessen the effects of an accident andlor to
`FIG. 4 illustrates the structure of a Processing Element
`maintain the vehicle in-lane and in proper operating condi- 45
`(PE) in the neural network of FIG. 3.
`tion as it travels.
`me systems and methods of this invention preferably
`FIG. 5 is an alternate embodiment of a neural network
`image Processor useful in the system of FIG. 1.
`employ computerized image analyzing techniques of the
`FIG. 6 is a flow diagram illustrating the overall operation
`types disclosed and defined in such patents of mine as U.S.
`of the Motor Vehicle Warning and Control System of FIG.
`Pat. Nos. 4,969,038 and 4,979,029 and references cited in
`1.
`the file wrappers thereof as well as other more recent patents
`FIG. 7 illustrates typical input signal membership func-
`and include the use of known artificial intelligence, neural
`networking and fuzzy logic computing electronic circuits.
`tions for fuzzy logic algorithms useful in the Motor Vehicle
`While the invention is described herein principally in 55 Warning and Control System of FIG. 1.
`FIG. 8 illustrates typical output signal membership func-
`connection with an automobile on a roadway, it may be used
`in connection with controlling any powered vehicle, includ-
`tions for fuzzy logic algorithms useful in the Motor Vehicle
`Warning and Control System of FIG. 1.
`ing a motor vehicle, a boat, a train, or an aircraft.
`FIG. 9 illustrates typical Fuzzy Associative Memory
`Accordingly it is a primary object of this invention to
`provide a new and improved system and method for con- 60 (FAM) maps for the fuzzy logic algorithms useful in the
`Motor Vehicle Warning and Control System of FIG. 1.
`trolling the operation of a powered vehicle.
`FIG. 10 is a HazardIObject state vector useful in imple-
`Another object is provide a system and method for
`assisting the driver of a powered vehicle in controlling its
`menting the Fuzzy Logic Vehicle Warning and Control
`operation to avoid an accident or hazardous driving condi-
`System.
`tion.
`FIG. 11 is a Hazard Collision Control vector useful in
`Another object is to provide a system and method
`implementing the Fuzzy Logic vehicle Warning and Control
`employing computerized image analysis to control or assist
`System.
`
`Hazard
`
`l9
`
`65
`
`Valeo Exhibit 1005_017
`
`

`
`US 6,553,130 B1
`
`5
`FIG. 12 is a table of HazardIObject state vectors indicat-
`ing possible combinations of hazards and objects useful in
`the Fuzzy Associative Memory access system used herein.
`FIG. 13 is a more detailed logic flow diagram for the
`analysis of detection signals prior to accessing fuzzy logic 5
`control structures in the Motor Vehicle Warning and Control
`System.
`FIG. 14 is a more detailed Logic Flow Diagram for the
`Fuzzy Associative Memory (FAM) selection processing.
`FIG. 15 is an example system flow illustrating the opera-
`tion of the Motor Vehicle Warning and Control System.
`
`6
`a range computer 21 which accepts digital code signals from
`a radar or lidar computer 14 which interprets radar and/or
`laser range signals from respective reflected radiation
`receiving means on the vehicle.
`In a modified form, video scanning and radar or lidar
`scanning may be jointly employed to identify and indicate
`distances between the controlled vehicle and objects ahead
`of, to the side(s) of, and to the rear of the controlled vehicle.
`The image analyzing computer 19 with associated
`memory 20 may be implemented in several different ways.
`Of particular concern is the requirement for high speed
`image processing with the capability to detect various haz-
`ards in dynamic image fields with changing scenes, moving
`and
`than one of which maybe
`a potential hazard. Requirements for wide angle vision and
`the
`right and left side image fields
`exists. The imaging system
`detects hazards, but
`estimates distance based 0'
`image data for input to the
`'1'0
`21
`range
`with the
`
`DETAILED DESCRIPTION
`In FIG. 1 is shown a computerized control system 10 for 15
`controlling the operation of a motor vehicle to prevent or
`lessen the effects of accidents such as collisions with sta-
`tionery and/or moving objects such as other vehicles. The
`system 10 employs a control computer or microprocessor 11
`mounted on the vehicle and operable to receive and gate
`unit 22.
`digital signals, such as codes and control signals from 20
`High speed image processing can be implemented
`various sensors, to one or more specialized computers and
`employing known special purpose computer architectures
`from such computers to a number of servos such as electric
`including various parallel system structures and systems
`motors and lineal actuators or solenoids, switches and the
`like, speakers and display drivers to perform either or both 25 based on neural networks. FIG. 2 shows a high speed
`the functions of audibly and/or visually informing or warn-
`parallel processor system embodiment with dedicated image
`processing hardware. The system of FIG. 2 has a dedicated
`ing the driver of the vehicle of a hazardous road condition
`image data bus 50 for high speed image data transfer. The
`ahead andlor to effect controlled braking and steering
`video camera 16 transfers full-frame video picture signall
`actions of the vehicle.
`ARM 12 and ROM 13 are connected to processor 11 to 30 data to the image bus 50 via analog/digital converter 18 and
`Preprocessor 51. The
`camera 16 is preferably a
`effect and facilitate its operation. A television camera(s) 16
`CCD
`camera generating successive picture frames
`having a wide angle lens 16L is mounted at the front of the
`with individual pixels being digitized for processing by the
`vehicle such as the front end of the roof, bumper or end of
`camera 16 may also be
`preprocessor 51. The
`the hood to scan the road ahead of the vehicle at an angle
`encompassing the sides of the road and intersecting roads. 35 implemented with other technologies including known
`~h~ analog signal output of camera 16 is digitized in an A/D
`image intensifying electron gun and infrared imaging meth-
`convertor 18 and passed directly to or through a video
`O ~ S . Multiple cameras may be used for front, side and rear
`preprocessor 51 to microprocessor 11, to an image field
`viewing and for stereo imaging capabilities suitable for
`analyzing computer 19 which is provided, implemented and
`generation of three dimensional image information includ-
`programmed using neural networks and artificial intelli- 40 ing capabilities for depth perception and placing multiple
`gence as well as fuzzy logic algorithms to (a) identify
`objects in three dimensional image fields to further improve
`objects on the road ahead such as other vehicles, pedestrians,
`hazard
`As shown in PIG. 2, the video preprocessor 51 performs
`barriers and dividers, turns in the road, signs and symbols,
`necessary video image frame management and data manipu-
`etc., and generate identification codes, and (b) detect dis-
`tances from such objects by their size (and shape) and 45 lation in preparation for image analysis. The preprocessor 51
`provide codes indicating same for use by a decision
`may also be used in some embodiments for digital prefil-
`computer, 23, which generates coded control signals which
`tering and image enhancement. Actual image data can be
`are applied through the computer 11 or are directly passed to
`displayed in real time using video display 55 via analog-to-
`digital converter 54. The image display may include high-
`various warning and vehicle operating devices such as a
`braking computer or drive, 35, which operates a brake servo so lighting of hazards, special warning images such as flashing
`33, a steering computer or drive(s) 39 and 40 which operate
`lights, alpha-numeric messages, distance values, speed indi-
`steering servos 36; a synthetic speech signal generator 27
`cators and other hazard and safety related messages. Simu-
`lated displays of symbols representing the hazard objects as
`which sends trains of indicating and warning digital speech
`signals to a digital-analog converter 29 connected to a
`well as actual video displays may also be used to enhance
`speaker 30; a display driver 31 which drives a (heads-up or 55 driver recognition of dangerous situations. The image analy-
`dashboard