`
`
`ISSUECLASSIF‘CAILQN
`I .l
`
`, UTI' TY
`assiitamssm
`
`9:66p ART UNtT
`
`EXAMINER
`
`
`
`‘
`“+3.13?
`’4“
`243:
`
`
`L”
`
`T,
`‘1: ‘
`
`
`
`its"
`
`It
`
`"7T.
`
`;
`
`
`
`o
`
`,
`PATENT DATE
`
`.
`
`*
`
`pATE'NT "in
`
`3
`
`s '
`FILING DATE cuss
`
`an 34‘s’
`3;
`
`suacuss _
`'
`)4‘]
`
`'
`
`swim. numean
`-:3,/1tjr5§;.,jga;14
`
`FH:
`
`_
`--'
`
`7;”:
`
`r
`
`_
`
`
`
`art-4.31:"! 2—1,:
`
`.{z 1;; :2.
`
`:1: 1|:
`
`I
`
`::_:
`
`:4:
`
`:t; :i: '9'
`
`:1:
`
`:;~
`
`..j
`
`:1:
`
`t»
`
`E:
`
`:‘_: .t: 't:
`
`£2
`
`'
`
`g FflfifilfiwaCT éPPLifiéTiflNS$$***$W***$i
`:VERIFISE
`~
`
`.
`
`\
`
`
`
`
`
`'
`
`.3. m5:
`
`'
`
`'
`
`1!; :r: :g: :r: :1".
`
`I yes
`F-‘iiulgn priority claimed
`‘ {35158139 condmbhifmn D you
`
`I no
`[3 no
`
`g;qgg4__g_ gmq‘g‘tV int-4‘; 15::ts'
`
`‘ NEV'S
`
`ATT'
`
`'.n
`Monce'or ALLOWANCE MAILED \u
`
`? Van-rs or; AP?LICATION
`FILEB SEEAMTELY
`
`.
`:3
`f »
`
`. ISSUE FEE
`
`.
`
`“ Amount Due
`
`Date Paid
`
`,
`
`1 Assistant Examiner
`
`.
`
`-'
`:5;
`1};
`i»
`L"I
`
`Primary Examiner
`PREPARED FOR ISSUE
`
`WARNING: The information disclosed herein may be restricted. Unauthorized disclosure may be prohibited :
`by the United States Code Tltie 35. Sections 122, 181 and 368. Possession outside the US.
`i
`Patent & Trademark Office is restricted to authorized errployees and contractors only.
`
`33 t" arm PTO-436A
`Rev. 3/92)
`
`(FACE)
`
`|PR2015-00261 - EX. 1103
`
`Toyota Motor Corp., Petitioner
`1
`
`IPR2015-00261 - Ex. 1103
`Toyota Motor Corp., Petitioner
`1
`
`
`
`08 105304
`
`
`
`PATENT APPLICATION SERIAL NO.
`
`
`
`
`U.S. DEPARTMENT OF COMMERCE
`
`
`
`
`PATENT AND TRADEMARK OFFICE
`
`
`
`
`FEE RECORD SHEET
`
`
`
`
`
`
`
`
`140 KM 09/02/93 08105304
`
`1 101
`
`
`
`
`11520.00 CK
`
`
`
`PTO-1556
`(5/87)
`
`
`
`
`
`
`
`2
`
`
`
`BAR CODE LABEL
`
`
`
`
`‘
`
`II
`
`IllllllllllllIIIlllllllllllllllIllllllllllll
`
`
`
`us. PATENT APPLICATION
`
`
`
`
`
`SERIAL NUMBER
`
`
`
`I
`
`FILING DATE
`
`
`
`08/105,304
`
`
`
`08/11/93
`
`
`
`
`
`
`
`GROUP ART UNIT
`
`
`
`
`2617
`
`
`
`
`
`APPLICANT
`
`
`
`
`
`
`
`
`
`
`JEROME H. LEMELSON,
`
`
`
`
`INCLINE VILLAGE, NV; ROBERT D. PEDERSEN, DALLAS, TX.
`
`
`
`
`
`
`
`
`
`
`**CONTINUING DATA*************~k******~k
`
`VERIFIED
`
`
`
`
`
`
`**FOREIGN/PCT APPLICATIONS************
`VERIFIED
`
`
`
`
`
`
`
`
`
`FOREIGN FILING LICENSE GRANTED 09/14/93
`STATE OR
`SHEETS
`INDEPENDENT
`
`
`
`
`COUNTRY
`DRAWING
`CLAIMS
`
`
`
`
`
`
`
`
`
`
`
`
`***** SMALL ENTITY *HH
`ATTORNEY DOCKET NO.
`FILING FEE
`
`
`
`
`
`RECEIVED
`
`
`NV
`
`
`13
`
`
`
`
`
`11
`
`
`
`$1,520.00
`
`
`
`JEROME H. LEMELSON
`
`
`
`SUITE 286
`
`
`930 TAHOE BLVD.
`
`
`
`UNIT 802
`
`
`INCLINE VILLAGE, NV 89451-9436
`
`
`
`
`
`
`
`
`ADDRESS
`
`
`
`
`
`
`MOTOR VEHICLE WARNING AND CONTROL SYSTEM AND METHOD
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`that annexed hereto is a trge copy from_ the records of the United States
`This is to certif
`
`
`
`
`
`
`
`
`
`
`
`Patent and Tra emark Office of the application whlch is Identified above.
`By authority-of the
`
`
`
`
`COMMISSIONER OF PATENTS AND TRADEMARKS
`
`
`
`
`
`
`
`
`Date
`
`
`Certifying Officer
`
`
`
`
`
`
`
`3
`
`
`
`«I.
`2
`
`w..-
`
`IN THE UNITED STATES PATENT AND TRADEMARK OFFICQH 105304
`
`icant:
`
`Jerome H.Lemelson
`
`Application Branch
`
`August 11,1993
`
` -
`
`Filed:
`
`August 1121993
`
`For:
`
`MOTOR VEHICLE WARNING AND CONTROL
`SYSTEM AND METHOD
`
`APPLICATION TRANSMI‘ICTL‘AL
`
`COmmissioner of
`Patents and Trademarks
`
`Washington2D.C. 20231
`
`Sir:
`
`I hereby certify that
`
`the above identified patent
`
`application enclosed herewith is being deposited in the U.s.
`
`Postal Servive as Express Mail No:
`
`BB 242939400
`
`on AUGUSt 11/1993
`
`addressed to the Commissioner of Patents
`
`and Trademarks, Washington2D.C. 20231.
`
`A check covering the filing of 99 claims including
`
`
`8 independent claims over three (total of ll independent claims)
`
`is enclosed herewith together with a Small Entity Declaration;
`
`a Declaration and Power of Attorney; a 31 page specification
`
`and claims 1—99 and 13 pages showing FIGS.
`
`1—15 of drawings.
`
`Calculation of Total Filing Fee amount:
`
`1. Basic Filing Fee .................................
`2. Claims in Excess of 20 = 79....( 79 x Sll=
`3.
`Independent Claims (in excesss of 3)
`8x $37) =
`
`$ 355.00
`869.00
`296.00
`
`Total Filing Fee=51520.00
`
`A check in the amount of $1522 is enclosed as is a postcard
`listing the above enclosed items to be mailed to applicant.
`
`spectfully s bm'
`
`2
`
`erome H.Leme son
`
`
`
`
`Suite 286 — Unit 802
`930 Tahoe Blvd.
`
`Incline Village/NV. 89451—9436
`
`
`
`
`4
`
`
`
`
`
`l I520 —/ Di A"
`
`Jem'é'h'sirgglm
`930 Tahoe Blvd. Unit 802
`lncllneVillage, NV 89451-9436
`
`08 105304
`
`MOTOR VEHICLE WARNING AND CONTROL SYSTEM AND METHOD
`
`Wm
`
`" ‘
`
`A system and method £9; assistsxig the driver of a motor vehicle in preventing accidents or
`' minimizing the effects of same.
`In one form, a television camera is mounted on a vehicle and
`
`scans the roadway ahead of the vehicle as the vehicle travels. Continuously generated video
`
`picture signals output by the camera are electronically processed and analyzed by an image
`flab
`analyzing computer,which generates codes‘Qvl-tieh serve to identify obstacles-en—the-reaé—aW
`
`J
`
`
`
`receives such code signals along with code signals generated by the speedometer or one or more
`
`k...
`
`epoed—anddir-ectien sensors sensing steering mechanism operation and generates control signalsifen
`
` \
`
`‘
`
`_n.n 9".---‘
`".:..Vl.‘
`-
`__a--
`:<"'
`A
`.
`.
`_
`or '
`.
`dis 1qu2Ond
`Rdaspfiyfs—as—wel-l—assynthetic speechka-ndi‘or specral sound generating and warning meansmay—be~
`gemfilgyed-ato warn the driver of the vehicle of approaching and existing hazardsmdrm—eertaflr
`and
`detected-instances; to control the operation of the brakesmdtlersteering mechanism of the vehicle
`”A:
`N 59:19 to avoid or lessen the effects of a collision.
`
`_.
`..
`1N3 n>
`
`us
`
`ln—a—pmmelar—fom—t-he—telemoa—eamera—aaé‘er
`’73-;
`
`
`5'!
`
`fl
`
`1.}
`
`l
`
`1.1
`
`l'
`
`'l
`
`l'
`
`.
`
`l
`
`'l‘E
`
`.
`
`infiM—Whmdmeemr'tbmmmm
`
`
`
`
`5
`
`
`
`..-
`..
`immune-z
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`6
`
`
`
`
`tried
`
`
`
`
`
`
`
`
`,Wh
`
`
`
`
`
`INS (5}
`
`T1
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`This invention relates to a systemand method for operating a motor vehicle, such as an
`
`
`
`
`
`
`
`
`
`
`
`
`
`' automobile, truck, aircraft or other vehicle, wherein a computer or computerized system is
`
`
`
`employed to assist and/or supplement the driver in the movement of the vehicle along a path of
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`travel, such as a street or roadway and may be used to avoid Obstacles and accidentstn aBl
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`preferred form of the invention, a video scanning system, such as a televisionmcamera and/or one or
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`more laser scanners mounted on the vehicle scan. the road in front of the vehicle and generate image
`information which is computer analyzed per se or in combination with a range sensing system to
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`warn the driver of hazardous conditions during driving by operating a display, such as a heads—up"
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`display, and/or a synthetic speech generating means which generates sounds or words of speech to
`verbally indicate such road conditions ahead of the vehicle.
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`The preferred form of the invention provides audible and/or visual display means to
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`cooperate in indicating to the driver of a motor vehicle both normal and hazardous road conditions
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`ahead as well as driving variables such as distances to stationary objects, and other vehicles; the
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`identification, direction of travel and speed of such other vehicles, and the identification of and
`
`
`
`
`
`
`
`
`
`
`
`
`
`distances to stationary or slowly moving objects such as barriers, center islands, pedestrians,
`0MB
`
`
`
`
`
`
`
`
`
`
`parked cars poles, sharp turns in the road ganother conditions.
`
`
`
`
`
`
`
`In addition, the image analyzing
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`computer of the vehicle may be operated to scan and decode coded and/or character containing
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`signs or signals generated by indicia or code generating other devices within or at the side of the
`
`
`
`
`
`
`
`
`
`
`road and indicating select road and driving conditions ahead.
`
`
`
`
`
`
`
`
`
`
`
`
`The computer is operable to analyze video and/or other formsts;~ of image information
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`generated as the vehicle travels to identify obstacles ahead of the vehicle and, in certain instances,
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`quantify the distance between the vehicle containing same on the basis of the size of the identified
`
`
`
`
`
`
`
`
`7
`
`
`
`~
`vehicle or object and/or by processing received pulse-echo signals.J'"'~hen the closing distance
`
`becomes hazardous, select vehicle subsystems may be automatically controlled by the computer as
`
`it continues to analyze image signals generated by the television camera. A first subsystem
`
`generates a first select code or codes which controls an electronic display, such as a heads-up
`
`\I~ ti1\
`
`display to cause it to display a warning indication, such as one or more flashing red light portions
`of the display or other lighted effect.~ second subsystem generates a code or series of codes
`1 t\IJ-
`which control a sound gene1:ating means which generates a select sound such as a horn, buzzing
`
`sound and/or select synthetic speech warning of the hazardous condition detected and, in certain
`
`instances, generating sounds of select words of speech which may warn of same and/or suggest
`
`corrective action~ by the vehicle operator or driver to avoid an accident.
`
`A third subsystem comes on-line and generates one or more codes which are applied to at
`
`least partly effect a cmTective action such as by pulsing one or more motors or solenoids to apply
`
`the brakes of the vehicle to cause it to slow down. If necessary to avoid or lessen the effects of an
`
`accident, the third subsystem stops the forward travel of the vehicle in a controlled manner
`
`depending on the relative speeds of the two vehicles, and/or the controlled vehicle and a stationery
`
`object or structure and the distance therebetween.
`
`A fourth subsystem, which may be part of or separate from the third subsystem may
`
`generate one or more codes which are applied to either effect partial and/or complete control of the
`
`steeling mechanism for the vehicle to avoid an obstacle and/or lessen the effect of an accident.
`
`Either or both the third or fourth subsystem may also be operable to control one or more safety
`
`devices by controlling motors, solenoids or valves, to operate a restraining device or devices for
`
`the driver and passeng~r(s) of the vehicle, such as a safety belt tightening means, an air bag
`
`inflation means or other device designed to protect human beings in the vehicle.
`"
`
`4
`
`
`
`
`8
`
`
`
`The second, and/or third and fourth subsystems may also be operable to effect or control
`
`the operations~ of additional warning 1p.eans such as the horn,headlights and/or other warning
`4\
`lights on the vehicle or other wa.ming means which operates to alett, flag or warn the driver of the
`
`approaching or approached vehicle or a pedestrian of the approaching hazardous condition. One or
`
`more of these subsystems may also be operable to generate and transmit one or more codes to be
`
`received and used by the approaching or approached vehicle or a roadside device to effect
`
`additional on-line warning(s) of the hazardous condition, and/or may be recorded on a disc or
`
`RAM (random access memory) for future analysis, if necessary.
`
`In a modified form of the invention, the vehicle waming system may also include a short
`
`wave receiving means to receive code signals from other vehicles and/or short wave transmitters at
`
`the side of or within the road for controlling the visual, audio and/or brake and steering means of
`the vehicle to avoid or~~:'the effects of an accident and/or to maintain the vehicle in-lane and in
`proper operating condition as it travels.
`
`The systems and methods of this invention preferably employ computerized image
`
`;x
`
`I
`
`analyzing techniques of the types disclosed and defined in such patents of mine as 4,969,038 and
`
`4,979,029 and references cited in the file wrappers thereof as well as other more recent patents
`
`and include the use of known artificial intelligence, neural networking and fuzzy logic computing
`
`electronic circuits.
`·Ns~
`
`'>
`
`Accordingly it is a primary object of this invention to provide a new and improved system
`'+'t~were.a \1~,~
`and method for controlling the operation of a..Hiotor vehiele, ~(')at, tra:in or a:irerM't. •
`1\
`Another object is provide a system and method for assisting the driver of ~ :nrotor vehicle~
`rp oi.PQ.~e-d \l.e.\i~~
`-train, boat or ahcrafH.n controlling its operation to avoid an accident or hazardous driving
`f\
`-
`condition.
`
`5
`
`
`
`
`9
`
`
`
`-- --~~ ...........
`Another object is to provide a s?'stem and method ~111ploying _computerized image analySis)
`
`to control or assist the driver of a moto: vehicle in controlling its operation to avoid hazardous
`
`conditions such as collisions with other vehicles, stationery objects or pedestrians.
`
`Another object is to provide a computerized system and method for controlling the speed of
`
`travel of a motor vehicle to lessen the chances of an accident while being driven by a person.
`
`Another object is to provide a system and method employing a television scanning camera
`
`mounted on a vehicle for scanning the field ahead, such as the image of the road ahead of the
`
`vehicle and a computer for analyzing the image signals generated wherein automatic image
`
`intensifying, or infra-red scanning and detection means is utilized to permit scanning operations to
`
`be effected duli:p:g di1ving at night and in low light, snowing or fog conditioris>,
`
`\
`
`Another object is to provide a system and method employing a television camera or other
`
`video scanning means mounted on a moving motor vehicle for scanning, detecting and identifying
`
`obstacles such as other vehicles ahead of such moving vehicle wherein the video image signals are
`
`analyzed to determine distances to such objects.
`
`Another object is to provide a computer controlled safety system for a motor vehicle which
`
`employs a television camera and an auxiliary scanning means to both identify obstacles in the path
`
`of the vehicle and dete1mine distance therefrom on a real time and continuous basis for use in
`
`warning the operator of same and/or in controlling the operation of the vehicle to avoid a collision.
`
`BRIEF DESCRIPTION OF DRAWINGS
`
`The vruious harqware and software elements used to carry out the invention described
`
`herein are illustrated in the f01m of block diagrams, flow chrui:S, and depictions of neural network
`
`6
`
`
`
`
`10
`
`
`
`and fuzzy logic algorithms and structures. The preferred embodiment is illustrated in the following
`
`figures:
`
`Fig. 1 is a block diagram of the overall Motor Vehicle Warning and Control System
`
`·illustrating system sensors, computers, displays, input! output devices and other key elements.
`IL
`•,
`<1..1
`~ is a block diagram of an ~mage ~alysis ~omputer 19 of the type that can be used in
`
`the Vehicle Hazard A voidance System herein of Fig. 1.
`•
`~
`\
`OJ
`Fig. 3 illustrates a neural network of the type useful in the l.lnage ~nalysis ~omputer of
`ll
`1\
`(\
`
`Fig. 4.
`
`3.
`
`E:ig._1 illustrates the structure of a Processing Element (PE) in the neural network of Fig.
`
`Fig. 5 is an alternate embodiment of a neural network image processor useful in the system
`
`of Fig. 1.
`
`d
`-F
`Eig._Q is a ~low 't)iagram illustrating the overall operation of the Motor Vehicle W aming
`"
`1'-
`and Control System of Fig. 1.
`
`1
`~
`~
`E.igJ illustrates typical input signal membership functions for):uzzy \ogic 'iigorithms
`~
`~
`~\.
`
`useful in the Motor Vehicle Warning and Control System of Figure 1.
`
`1
`Q.
`~
`Fig. 8 illustrates typical output signal membership functions for ~uzzy \ogic )\lgorithms
`" "
`"
`
`useful in the Motor Vehicle Warning and Control System of Fig. 1.
`
`~illustrates typical Fuzzy Associative Memory (FAM) maps for the Fuzzy Logic
`
`Algorithms useful in the Motor Vehicle W aming and Control System of Fig. 1.
`.
`v
`$
`Fig. 10 is a Hazard/Object ~tate Xector useful in implementing the Fuzzy Logic Vehicle
`"
`...
`
`W aming and Control Sy~tem.
`
`'
`v
`Fig. 11 is a Hazard Collision Cont:rol'\cector useful in implementing the Fuzzy Logic
`,..
`
`Vehicle Warning and Control System.
`
`7
`
`-
`
`
`
`
`11
`
`
`
`fi).!. 12 is a table of Hazard/Object state vectors indicating possible combinations of hazards
`
`and objects useful in the Fuzzy Associative Memory access system used herein.
`t
`~-
`~
`Fig. 13 is a more detailed \ogic 'ilow 'f<iagram for the analysis of detection signals prior to
`I
`,_
`"
`1\
`~
`.
`accessing 1\uzzy \ogic control structures in the Motor Vehicle Warning and Control System.
`-f o
`"
`"
`I
`Fig. 14 is a more detailed\ogic ~ow ~iagrant for the Fuzzy Associative Memory (PAM)
`"
`..
`'(\..
`
`selection processing.
`
`Fi~. 15 is an example system flow illustrating the operation of the Motor Vehicle Warning
`
`and Control System.
`
`, r::De \a.\ \-eci. "]) esc:r \.fC~ on
`·SYSTEM DESCRIPTION
`
`In Fig. 1 is shown a computelized control system 10 for controlling the operation of a
`
`motor vehicle to prevent or lessen the effects of accidents such as collisions with stationery and/or
`
`moving objects such as other vehicles. The system 10 employs a control computer or
`
`microprocessor 11 mounted on the vehicle and operable to receive and gate digital signals, such as
`
`codes and control signals from various sensors, to one or more specialized computers and from
`
`such computers to a number of servos such as electric motors and lineal actuators or solenoids,
`
`switches and the like, speakers and display drivers to perform either or both the functions of
`
`audibly ancVor visually informing or wm·ning the driver of the vehicle of a hazardous road
`
`condition ahead and/or to effect controlled braking and steering actions of the vehicle.
`
`A RAM 12 and ROM 13 are connected to processor 11 to effect and facilitate its operation.
`
`A television camera(s) 1_6 having a wide angle lens 16L is mounted at the front of the vehicle such
`
`as the front end of the roof, bumper or end of the hood to scan the road ahead of the vehicle at an
`
`angle encompassing the sides of the road and intersecting roads. The analog signal output of
`
`8
`
`
`
`
`12
`
`
`
`camera 16 is digitized in an A/D convertor 18 and passed directly to or through a video
`
`preprocessor 51 to microprocessor 11, to _an image field analyzing computer 19 which is provided,
`
`implemented and programmed using neural networks and artificial intelligence as well as fuzzy
`
`logic algorithms to (a) identify objects on the road ahead such as other vehicles, pedestrians,
`
`~
`
`baniers and dividers, turns in the road, signs and symbols, et~._and generate identification codes,
`A
`and (b) detect distances from such objects by their size (and shape) and provide codes indicating
`
`same for use by a decision computer, 23, which generates coded control signals which are applied
`
`through the computer 11 or are directly passed to various warning and vehicle operating devices
`
`such as a braking computer or drive, 35, which operates a brake servo 33, a steering computer or
`
`drive(s) 39 and 40 which operate ·steering servos 36; a synthetic speech signal generator 27 which
`
`sends trains of indicating and warning digital speech signals to a digital-analog converter 29
`
`connected to a speaker 30; a display driver 31 which drives a (heads-up or dashboard) display 32;
`
`_ _.
`
`a head light controller 41 for flashing the head lights, a warning light control 42 for flashing
`
`external and/or internal warning lights; a hom control43, etc.
`
`A digital speedometer 44 and accelerometer(s) 45 provide information sign.als for use by
`
`the decision computer, 23, in issuing its commands. Accelerometer(s) 45 are connected to control
`
`computer microprocessor 11 through analog-to-digital converter 46. The accelerometer(s) 45 may
`
`pass data continuously to control computer microprocessor 11, or, alternatively, respond to query
`
`signals from said control computer 11. An auxiliary range detection means comprises a range
`
`computer 21 which accepts digital code signals from a radar or lidar computer 14 which interprets
`
`radar and/or laser range signals from respective reflected radiation receiving means on the vehicle.
`
`The image analyzing computer 19 with associated memory 20 may be implemented in
`
`several different ways. Of particular concern is the requirement for high speed image processing
`
`with the capability to detect various hazards in dynamic image fields with changing scenes, moving
`
`9
`
`
`
`
`13
`
`
`
`objects and multiple objects, more than one of which maybe a potential hazard. Requirements for
`
`wide angle vision and the ability to analyze both right and left side image fields also ~xis~. The
`
`imaging system not only detects hazards, but also estimates distance based on image data for input
`
`to the range computer 21 implemented with the associated memory .unit 22.
`
`High speed image processing can be implemented employing known special purpose
`
`computer architectures including various parallel system structures and systems based on neural
`~~-
`networks. ~ 2 shows a high speed parallel processor system embodiment with dedicated
`r~-
`,.
`image processing hardware. The system of~ 2 has a dedicated image data bus 50 for high
`speed image data transfer. The video camera 16 transfers full-frame video picture signal!data to the
`
`.
`
`image bus 50 via analog/digital converter 18 and video preprocessor 51. The video camera 16 is
`
`preferably a CCD array camera generating successive picture frames with individual pixels being
`
`digitized for processing by the video preprocessor 51. The video camera 16 may also be
`
`implemented with other technologies including known image intensifying electron gun and infra(cid:173)
`
`red imaging methods. Multiple cameras may be ~r f~~-nt, side and rear viewing and for
`
`stereo imaging capabilities suitable for generation o:R;f.Dimel)sional image information including
`"
`capabilities for depth perception and placing multiple objects in three dimensional image fields to
`
`further improve hazard detection capabilities.
`
`As shown in Fig. 2, the video preprocessor 51 performs necessary video image frame
`
`management and data manipulation in preparation for image analysis. The preprocessor 51 may
`
`also be used in some embodiments for digital prefiltering and in1age enhancement. Actual image
`
`data can be displayed in real time using video display 55 via analog-to-digital converter 54. The
`
`image display may inclupe highlighting of hazards, special warning linages such as flashing lights,
`
`alpha-numeric messages, distance values, speed indicators and other hazard and safety related
`
`10
`
`
`
`
`14
`
`
`
`l-..
`
`._,
`
`messages. Simulated.as well as actual video displays may also be used to enhance driver
`"
`'Ci3
`recognition of dangerous situations .
`•
`The ~age~alysis ~omputer 19 operates under the control of control processor 56 with
`\"4:\tl.
`fl
`"
`"
`· random-access-memory (RAM) 57 and program and reference data stored in~-only memory
`
`(ROM) 58. The control processor 56 communicates with the motor vehicle warning and control
`
`system micro-processor controller ll through the Bus Interrace Unit 59. Results of the image
`
`analysis are passed in real-time to microprocessor controller 11 for integration with other sensory,
`
`computing, warning and control signals as depicted in Figure 1.
`i
`·i=" ~ •
`c:u
`c....
`The image !\nalysis 'l0omputer 19 of Figure 2 uses high speed dedicated co-processor 53
`k\
`~
`~
`for actual linage analysis under control of the control processor 56. Typical operations performed
`
`using co-processors 53 include multidimensional filtering for operations such as feature extraction
`
`and motion detection. The co-processors 53 are used for multidimensional discrete transforms and
`
`other digital flltering operations used in image analysis. Multiple image memorie~52 with parallel
`
`access to successive image data frames via image bus 50 permit~ concurrent processing with high
`
`speed data access by respective co-processing elements 53. The co-processor elements 53 may be
`
`high speed programmable processors or special purpose hardware processors specifically
`
`constructed for image analysis operations. SIMD (single. instruction, multiple data) architectures
`
`provide high speed operation with multiple identical processing elements under control of a control
`
`unit that broadcasts instructions to all processing elements. The same instruction is executed
`
`simultaneously on different data elements making this approach particularly well suited for matrix
`
`and vector operations commonly employed in image analysis operations. Parallel operations of ·
`
`this type are particularly important with high pixel counts. A 1000 x 1000 pixel image has one
`
`million data points. Tightly coupled Multiple Instruction, Multiple Data (MIMD) architectures also
`
`are used in image processing applications. MIMD systems execute independent but related
`
`11
`
`
`
`
`15
`
`
`
`programs concunently on multiple processing elements. Va1ious anay processor and massively
`
`parallel architectures known to those s]<illed in the art may also be used for real-time image
`
`analysis.
`
`The calculation of the distance of certain recognizable objects from the vehicle is facilitated
`
`by having standard images stored in memory and recalling and comparing such image data with
`
`image data representing the object detected by the vehicle scanning mechanisms. For example,
`
`virtually all automobiles, trucks, and other standard vehicles have known widths. It f2!!?_~-~-~~t
`
`\I -~e dista?ce to;:~ther vehicle can b~--~~-~~-~e.d by _ca1c.ulatin_¥ i~ vj_d_th ~:~~--~~~-~-~-!mage. If a
`CCD camera is used, for example, the width can ascertained in pixels in the image field. The
`
`distance to the vehicle can then be easily calculated using a simple relationship wherein the distance
`
`will be directly proportional to the object image width in pixels. The relative velocities and
`
`accelerations can also be easily calculated from respective first and second derivatives of the image ·
`
`width with respect to time. These image measurements and calculations can be used in addition to
`
`radar/lidar signal measurements or they may be used alone depending on system requirements.
`
`In another embodiment, the image analyzing computer 19 is implemented as a neural
`
`-····
`
`computing network with networked processingel~ments perfonning S\}CCessive computations on
`
`input image structure as shown in~:l~Fe 3 where signal inp-u~ 61 ~e connected to multiple
`processing elements 63, 65 and 67 through the network connections 62, 64 and 66. The
`
`processing elements (PE's) 63, 65 and 67 map input signal vectors to the output decision layer,
`
`performing such tasks as image recognition and image parameter analysis.
`
`A typical neural network processing element known to those skilled in the art is shown in
`
`o.__.
`
`Fig. 4 where input yectors~ (Xl, X2 .... Xn) are connected via weighing elements (Wl,
`
`W2 ..... Wn) to a summing node 70. The output of node 70 is passed through a nonlinear
`
`processing element 72 to produce an output signal, U. Offset or bias inputs can be added to the
`
`12
`
`
`
`
`16
`
`
`
`inputs through weighing circuit Wo. The output signal from summing node 70 is passed through
`
`the nonlinear element 72. The nonlinear function is preferably a continuous, differentiable
`
`function such as a sigmoid which is typically used in neural network processing element nodes.
`
`Neural networks used in the vehicle warning system are trained to recognize roadway hazards
`
`which the vehicle is approaching including automobiles, trucks, and pedestrians. Training
`
`involves providing known inputs to the network resulting in desired output responses.. The
`
`weights are automatically adjusted based on error signal measurements until the desired outputs are
`
`generated. Various leruning algorithms may be applied. Adaptive operation is also possible with
`
`on-line adjustment of network weights to meet imaging requirements. The neural network
`
`embodiment of the image analysis computer 19 provides a highly parallel image processing
`
`structure with rapid, real-time image recognition necessary for the Motor Vehicle Wruning and
`
`Control System. Very Large Scale Integrated (VLSI) Circuit implementation of the neural
`
`processing elements permits low-cost, low-weight implementation. Also, a neural network has
`
`certain reliability advantages important in a safety warning system. Loss of one processing
`
`element does not necessru·ily result in a processing system failure.
`In a alternate embodiment, the neural network computing network of 'Fl~tu's- 3 can be
`"
`implemented using multiple vi.Itual processing elements 73 interconnected via an image data bus 75
`.v~
`,..
`with an image processor 74 as shown in~ 5. Image data presented to the Image Processor 74
`is routed to selected virtual processing elements 73 which implement the neural network computing
`
`.Q•
`
`. .....
`
`functions. The virtual PE' s may be pipelined processors to increase speed and computational
`
`efficiency.
`
`The ~ecision tomputer 23 of-~]'i~re 1 integrates the inputs from the image analysis
`
`.
`
`~
`~
`computer 19, range computer 21, digital accelerometer 45, and the radar or lidar computer 14 to
`'
`generate output warning and contml signals. W ru·ning signals alert the driver of impending
`
`I
`
`13
`
`
`
`
`17
`
`
`
`hazards and, depending on the situation, actual vehicle control signals may be generated to operate
`
`the vehicle in a manner that will avoiq the hazard or minimize the danger to the vehicle and
`
`'
`
`passengers. ~;ntrol signals,iwHl be generated to operate brake servos 33 and steering servos 36.
`--- ...... .
`--
`Manual overrides are provided to ensure driver vehicle control if necessary.
`t
`{
`'··
`...
`..
`- ·--
`A particularly attractive embodiment of the decision computer 23 makes use of fuzzy '&ogic
`ov
`;.
`-
`I\
`Algorithmic structures to implement the automated control and warning signal generation. Fuzzy
`
`..
`
`·-· -----·--------·-·-----------------~---------
`
`'\ogic is particularly well suited to the vehicle control problem wherein it is necessary to deal with a
`
`"' multiplicity of image, motion, and environmental parameters, each of which may extend over
`
`ranges of values and in different combinations which require different responses.
`
`·~~re 6 illustrates a\low ~iagram for implementing a Fuzzy Logic Vehicle Control and
`"
`~
`Warning signal generation system suitable for the decision computer 23. The system of Fig. 6
`receives inputs via the ~ontrol ~omputer~croprocessor 11 of·~~ffi:e 1. Inputs include image
`
`analysis outputs, motion sensor outputs, distance measurements from radar/lidar systems, and
`
`-
`
`environmental parameters which may indicate adverse driving condition.s including rain or ice. The
`
`input signals are analyzed in a preprocessing step for hazardous conditions in the processing block
`
`_.
`
`74. When a hazard is detected, the Fuzzy Associative Memory (FAM) Seetimt block 76 described
`
`in more detail below is activated via decision element 75.
`
`If no hazard is present, the system
`
`_.
`
`continues to analyze scanning signals until a hazardous situation is encountered.
`b
`The Fuzzy Associative Memory (FAM) )}lock 76 also receives a parameter input file from
`k.
`t..:
`the Detection Signal Analysis ~lock 7 4. This file contains necessary information to make control
`'
`. -
`1\
`decision including, for example, hazard location (front, back, left side, right sidy-), hazard distance,
`
`relative velocity, steeriJ:!g angle, braking pressure, weather data, and the presence or a,bsence of
`
`obstructions or objects to the front, rear,\~or to either side of the vehicle.
`
`14
`
`
`
`
`18
`
`
`
`Control signals are derived using FAM' ~ 77, 78, 79 and 80. In practice , a large number
`
`ofF AM's may be used to reflect different possible driving conditions and hazard scenarios. Each
`
`Fuzzy Associative Memory maps input control parameter combinations to appropriate output
`
`· control signals. The output signals are defuzzified in the control signal generator 81 for input to
`t-\•
`the microprocessor controller 11 or Fii:ne 1. This controller in tum generates control signals for
`...
`