PATENT DATE
`
`SUBCLA$5
`I 4j
`
`2.!::31!~
`2..0 ~)
`
`: . . ;-~~' ..
`
`·.,·
`
`,-.·-;.
`
`·.:
`
`... .... ~ ::
`
`. : ·,. ~
`
`·.
`
`.: .. __ : .
`
`APPLICATIONS************
`
`l
`\
`\.
`
`. U.S. bEPr. of COMM.-Pat. •TM OffD..,.. Pf0.43.&L ,,_.,. ill'**ft)
`:. ----------------·-~·····-········-----··._. __
`
`' .. · ... ---~···,;_ ······
`
`Amourit Due
`
`Date Paid
`
`Sheets Drwg.
`
`Print Fig.
`
`ISSUE
`BATCH
`Primary Examiner · NUMBER
`
`WARNING: The information disclosed herein may be restricted. Unauthorized disclosure may be prohibited
`by the United States Code Title 35, Sections 122, 181 and 368. Possession outside the U.S.
`Patent & Trademark Office is restricted to authorized employees and contractors only.
`
`Label
`Area
`
`PT0·4.36A
`8/92)
`
`(FACE)
`
`IPR2013-00424 - Ex. 1003
`Toyota Motor Corp., Petitioner
`1
`
`

`

`DB 105304
`
`PATENT APPLICATION SERIAL NO~.------
`
`U.S. DEPARTMENT OF COMMERCE
`PATENT AND TRADEMARK OFFICE
`FEE RECORD SHEET
`
`140 KM 09/02/93 08105304
`
`1 101
`
`1,520.00 CK
`
`PT0-1556
`(5/87)
`
`2
`
`

`

`BAR CODE LABEL
`
`11111111111111111111111111111111111111111111111111
`
`U.S. PATENT APPLICATION
`
`SERIAL, NUMBER
`
`FILING DATE
`
`CLASS
`
`GROUP ART UNIT
`
`08/105,304
`
`08/11/93
`
`340
`
`2617
`
`JEROME H. LEMELSON,
`
`INCLINE VILLAGE, NV; ROBERT D. PEDERSEN, DALLAS, TX.
`
`1-z
`5
`
`:J
`a..
`a..
`q:
`
`**CONTINUING DATA*********************
`VERIFIED
`
`**FOREIGN/PCT APPLICATIONS************
`VERIFIED
`
`FOREIGN FILING LICENSE GRANTED 09/14/93
`
`***** SMALL ENTITY *****
`
`STATE OR
`COUNTRY
`
`SHEETS
`DRAWING
`
`TOTAL
`CLAIMS
`
`INDEPENDENT
`CLAIMS
`
`FILING FEE
`RECEIVED
`
`ATTORNEY DOCKET NO.
`
`NV
`
`13
`
`99
`
`11
`
`$1,520.00
`
`JEROME H. LEMELSON
`SUITE 286
`930 TAHOE BLVD.
`UNIT 802
`INCLINE VILLAGE, NV 89451-9436
`
`MOTOR VEHICLE WARNING AND CONTROL SYSTEM AND METHOD
`
`(/)
`
`(/) w
`0:
`0
`0 q:
`
`w
`...J
`1-
`i=
`
`This is to certify that annexed hereto is a true copy from the records of the United States
`Patent and Trademark Office of the application wh1ch is identified above.
`By authority of the
`COMMISSIONER OF PATENTS AND TRADEMARKS
`
`Date
`
`Certifying Officer
`
`3
`
`

`

`IN
`
`'l'UE UNITED S'l'l\'l'ES PA'l'EN'l' AND
`
`'l'RADEMAEK OPFICQB 105304
`
`i can t :
`
`J e rome I-I • L em e 1 son
`
`Application Branch
`
`August 11,1993
`
`Filed:
`
`For:
`
`No:
`
`August 11,1993
`
`MOTOR VEHICLE WARNING AND CONTROL
`SYSTEM AND METHOD
`
`APPLICATION TRANSMITTAL
`
`CoJJIIiJis:s_i_oncc of
`P a L: e n L: s a n d
`'1' L" u d c Ill u c k s
`\·v a .s !1 i n y L: o n , D . C . 2 0 2 3 1
`
`sir-:
`
`I hereby cei:l:ify thal: the ubove identified potent
`
`application enclosed hecewith is being deposited in tl1e u.s.
`
`HB 242939400
`Posl:al Servive as Express Mail No:
`-------------------------------
`addressed to the Commissioner of Patents
`
`on August 1Ll993
`
`and Trademarks, Washington,D.C. 20231.
`
`A check covering the filing of 99 claims including
`
`8
`
`ir1depenrlent claims over three (total of 11 independent claims)
`
`is enclosed herewith together with a Small Entity Declaration~
`
`a Declaration and Power of Attorney; a 31 page specification
`
`and claims 1-99 and 13 pages showing FIGS. 1-15 of drawings.
`
`Calculation of Total Filing Fee amount:
`
`1. Basic F i 1 ing Fee . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
`2. Claims in Excess of 20 = 79 .... ( 79 x $11=
`3. Independent Claims (in excesss of 3) = 8x $37)
`
`$ 355.00
`869.00
`296.00
`
`Total Filing Fee=$1520.00
`
`A check in the amount of $1522 is enclosed as is a postcard
`listing the above enclosed items to be mailed to applicant.
`_...--R spectfu~b~
`erome H.Lem~~~
`Suite 286 - Unit 802
`930 Tahoe Blvd.
`Incline Village,NV. 89451-9436
`
`4
`
`

`

`l r'?'2.o -t D l 4-
`08 1 0530~
`Jero~~t;~~itson
`930 Tahoe Blvd., Unit 802
`lnclineVillage, NV 89451-9436
`
`MOTOR VEHICLE WARNING AND CONTROL SYSTEM AND METHOD
`
`Abstract of the Disclosure
`A system and method fa¥ assis~ the driver of a motor vehicle in preventing accidents or
`minimizing the effects of same. In one form, a television camera is mounted on a vehicle and
`
`scans the roadway ahead of the vehicle .as the vehicle travels. Continuously generated video
`
`picture signals output by the camera are electronically processed and analyzed by an image
`~
`analyzing computer,which generates codes~ serve to identify obstacles'ffil th~ road ahead of..
`7'
`-tHe vehicle. B~iHg such id€mtifyiog
`
`infmmation aad eenrparing it v1ith infofffl:atioa on-the-strapes
`
`.&Hd sizes of various objects such a~ rear and front profiles of all production vehicles tm.d the like·
`
`and their mlative sires er select dnnensions thereof, indieatiolls of di£ta.llces to snch objects may be
`
`computed and indicated as further cod~ A decision computer mounted in the controlled vehicle.
`
`receives such code signals along with code signals generated by the speedometer or one or more
`
`speed and directioR sensors sensing steering mechanism operation and generates control signals~~
`
`dRving a display, such as a heads up diSfl"l:tty on the windshield and/or dashoofl:f'd to display sncn
`
`~
`
`i:ffferma:tieft as images of the coatrolled vehicle and other vehicles in and adjacent its path. of travel •
`and relative distances thereto as weJl as groups of characters defioiog sa:FB:e, eolmed and flasl~ \:v
`• 1.\'arHing lights and the like for pte- warnia.g and maming pl:li'pesss. Such code signa~s ~~
`I'-
`•
`I
`.
`d. \.S p LO..~-ed..) 0.'1"\C
`.
`0'1""
`.
`~ispfay(s) as '.-veil a.s-.synthetic speec~ttfttlfor special sound generating and warning meansffia~ be-
`\JSC?£1
`_
`i-.. e:mp±oyed tbto warn the driver of the vehicle of approaching and existing haza.rdskafttl, in eerta:ift •
`Qna
`~
`~
`~N .J A'( -tieteeteei--ffi.·stanc-es, to control the operation of the brakes "Mdlol" steering mechanism of the vehicle
`~~,
`~
`:"\1 s .§ to avoid or lessen the effects of a collision. In a particular form, the television camera and/or
`!A.;L
`auxiliary radiation scanning mecms may sean a 'vvide angle in froa.t of the vehicle and areas to the
`
`sides of the vehicle to sense other vehicles uaveliug in t11e same direction a11:d provide information
`
`in--the form of cedes to the deci-s-ieH-eomputer tb permit it to a11:a:lyze sa:rne m effectmg autonrat:ie
`
`1
`
`5
`
`

`

`1 contrel of the :vehicle to avoid a collislclhJJ,.vith a vehicle clirectly aheacl Me to €lither or both siS€lp
`.aa-6 the rear ofthe controlled velrisJ?. _Wl:llle._manual override .means is provided, the decision
`
`cGmpurer may oven ide 01 pre vefl:t the operatiofl: of same if it detenuines that a col:l:ision may occur
`to he operated. --In a moeified :£Gnn, video scanning ana t:ad.ar
`QJ.:.lidar scanning may be jointly employed to identify and indicate distances between the coRtroilcxl
`
`if-such 0 v'Crrids means is psnnitt€d
`
`vehicle and objects ahead to the side(s) and I ear of the eont:F~.
`
`2
`
`6
`
`

`

`\="\e.\4
`Summary of the Invention
`I'
`.
`This invention relates to a system and method for operating a motor vehicle, such as an
`
`automobile, truck, aircraft or other vehicle, wherein a computer or computerized system is
`
`employed to assist and/or supplement the driver in the movement of the vehicle along a path of
`
`travel, such as a street or roadway and may be used to avoid obstacles and accidents.Jn a
`JBl
`preferred form of the invention, a video scanning system, such as a televisiot.'l. camera and/or one or
`
`more laser scanners mounted on the vehicle scan the road in front of the vehicle and generate image
`
`information which is computer analyzed per se or in combination with a range sensing system to
`
`warn the driver of hazardous conditions during driving by operating a display, such as a heads-l!:p_
`
`r - - - - - -
`
`display, and/or a synthetic speech generating means which generates sounds or words of speech to
`
`verbally indicate such road conditions ahead of the vehicle.
`
`The preferred form of the invention provides audible and/or visual display means to
`
`cooperate in indicating to the driver of u motor vehicle both normal and hazardous road conditions
`
`ahead as well as driving variables such as distances to stationary objects, and other vehicles; the
`
`identification, direction of travel and speed of such other vehicles,. and the identification of and
`
`distances to stationary or slowly moving objects such as barriers, center islands, pedestrians,
`~0
`parked cars poles, sharp turns in the road~ other conditions. In addition, the image analyzing
`
`~
`
`computer of the vehicle may be operated to scan and decode coded and/or character containing
`
`signs or signals generated by indicia or code generating other devices within or at the side of the
`
`road and indicating select road and driving conditions ahead.
`
`The computer i~ operable to analyze video and/or other forms~ of image information
`
`generated as the vehicle travels to identify obstacles ahead of the vehicle and, in certain instances,
`
`'
`quantify the distance between the vehicle containing same on the basis of the size of the identified
`
`3
`
`7
`
`

`

`~
`vehicle or object and/or by processing received pulse-echo signals.J'"'~hen the closing distance
`
`becomes hazardous, select vehicle subsystems may be automatically controlled by the computer as
`
`it continues to analyze image signals generated by the television camera. A first subsystem
`
`generates a first select code or codes which controls an electronic display, such as a heads-up
`
`\I~ ti1\
`
`display to cause it to display a warning indication, such as one or more flashing red light portions
`of the display or other lighted effect.~ second subsystem generates a code or series of codes
`1 t\IJ-
`which control a sound gene1:ating means which generates a select sound such as a horn, buzzing
`
`sound and/or select synthetic speech warning of the hazardous condition detected and, in certain
`
`instances, generating sounds of select words of speech which may warn of same and/or suggest
`
`corrective action~ by the vehicle operator or driver to avoid an accident.
`
`A third subsystem comes on-line and generates one or more codes which are applied to at
`
`least partly effect a cmTective action such as by pulsing one or more motors or solenoids to apply
`
`the brakes of the vehicle to cause it to slow down. If necessary to avoid or lessen the effects of an
`
`accident, the third subsystem stops the forward travel of the vehicle in a controlled manner
`
`depending on the relative speeds of the two vehicles, and/or the controlled vehicle and a stationery
`
`object or structure and the distance therebetween.
`
`A fourth subsystem, which may be part of or separate from the third subsystem may
`
`generate one or more codes which are applied to either effect partial and/or complete control of the
`
`steeling mechanism for the vehicle to avoid an obstacle and/or lessen the effect of an accident.
`
`Either or both the third or fourth subsystem may also be operable to control one or more safety
`
`devices by controlling motors, solenoids or valves, to operate a restraining device or devices for
`
`the driver and passeng~r(s) of the vehicle, such as a safety belt tightening means, an air bag
`
`inflation means or other device designed to protect human beings in the vehicle.
`"
`
`4
`
`8
`
`

`

`The second, and/or third and fourth subsystems may also be operable to effect or control
`
`the operations~ of additional warning 1p.eans such as the horn,headlights and/or other warning
`4\
`lights on the vehicle or other wa.ming means which operates to alett, flag or warn the driver of the
`
`approaching or approached vehicle or a pedestrian of the approaching hazardous condition. One or
`
`more of these subsystems may also be operable to generate and transmit one or more codes to be
`
`received and used by the approaching or approached vehicle or a roadside device to effect
`
`additional on-line warning(s) of the hazardous condition, and/or may be recorded on a disc or
`
`RAM (random access memory) for future analysis, if necessary.
`
`In a modified form of the invention, the vehicle waming system may also include a short
`
`wave receiving means to receive code signals from other vehicles and/or short wave transmitters at
`
`the side of or within the road for controlling the visual, audio and/or brake and steering means of
`the vehicle to avoid or~~:'the effects of an accident and/or to maintain the vehicle in-lane and in
`proper operating condition as it travels.
`
`The systems and methods of this invention preferably employ computerized image
`
`;x
`
`I
`
`analyzing techniques of the types disclosed and defined in such patents of mine as 4,969,038 and
`
`4,979,029 and references cited in the file wrappers thereof as well as other more recent patents
`
`and include the use of known artificial intelligence, neural networking and fuzzy logic computing
`
`electronic circuits.
`·Ns~
`
`'>
`
`Accordingly it is a primary object of this invention to provide a new and improved system
`'+'t~were.a \1~,~
`and method for controlling the operation of a..Hiotor vehiele, ~(')at, tra:in or a:irerM't. •
`1\
`Another object is provide a system and method for assisting the driver of ~ :nrotor vehicle~
`rp oi.PQ.~e-d \l.e.\i~~
`-train, boat or ahcrafH.n controlling its operation to avoid an accident or hazardous driving
`f\
`-
`condition.
`
`5
`
`9
`
`

`

`-- --~~ ...........
`Another object is to provide a s?'stem and method ~111ploying _computerized image analySis)
`
`to control or assist the driver of a moto: vehicle in controlling its operation to avoid hazardous
`
`conditions such as collisions with other vehicles, stationery objects or pedestrians.
`
`Another object is to provide a computerized system and method for controlling the speed of
`
`travel of a motor vehicle to lessen the chances of an accident while being driven by a person.
`
`Another object is to provide a system and method employing a television scanning camera
`
`mounted on a vehicle for scanning the field ahead, such as the image of the road ahead of the
`
`vehicle and a computer for analyzing the image signals generated wherein automatic image
`
`intensifying, or infra-red scanning and detection means is utilized to permit scanning operations to
`
`be effected duli:p:g di1ving at night and in low light, snowing or fog conditioris>,
`
`\
`
`Another object is to provide a system and method employing a television camera or other
`
`video scanning means mounted on a moving motor vehicle for scanning, detecting and identifying
`
`obstacles such as other vehicles ahead of such moving vehicle wherein the video image signals are
`
`analyzed to determine distances to such objects.
`
`Another object is to provide a computer controlled safety system for a motor vehicle which
`
`employs a television camera and an auxiliary scanning means to both identify obstacles in the path
`
`of the vehicle and dete1mine distance therefrom on a real time and continuous basis for use in
`
`warning the operator of same and/or in controlling the operation of the vehicle to avoid a collision.
`
`BRIEF DESCRIPTION OF DRAWINGS
`
`The vruious harqware and software elements used to carry out the invention described
`
`herein are illustrated in the f01m of block diagrams, flow chrui:S, and depictions of neural network
`
`6
`
`10
`
`

`

`and fuzzy logic algorithms and structures. The preferred embodiment is illustrated in the following
`
`figures:
`
`Fig. 1 is a block diagram of the overall Motor Vehicle Warning and Control System
`
`·illustrating system sensors, computers, displays, input! output devices and other key elements.
`IL
`•,
`<1..1
`~ is a block diagram of an ~mage ~alysis ~omputer 19 of the type that can be used in
`
`the Vehicle Hazard A voidance System herein of Fig. 1.
`•
`~
`\
`OJ
`Fig. 3 illustrates a neural network of the type useful in the l.lnage ~nalysis ~omputer of
`ll
`1\
`(\
`
`Fig. 4.
`
`3.
`
`E:ig._1 illustrates the structure of a Processing Element (PE) in the neural network of Fig.
`
`Fig. 5 is an alternate embodiment of a neural network image processor useful in the system
`
`of Fig. 1.
`
`d
`-F
`Eig._Q is a ~low 't)iagram illustrating the overall operation of the Motor Vehicle W aming
`"
`1'-
`and Control System of Fig. 1.
`
`1
`~
`~
`E.igJ illustrates typical input signal membership functions for):uzzy \ogic 'iigorithms
`~
`~
`~\.
`
`useful in the Motor Vehicle Warning and Control System of Figure 1.
`
`1
`Q.
`~
`Fig. 8 illustrates typical output signal membership functions for ~uzzy \ogic )\lgorithms
`" "
`"
`
`useful in the Motor Vehicle Warning and Control System of Fig. 1.
`
`~illustrates typical Fuzzy Associative Memory (FAM) maps for the Fuzzy Logic
`
`Algorithms useful in the Motor Vehicle W aming and Control System of Fig. 1.
`.
`v
`$
`Fig. 10 is a Hazard/Object ~tate Xector useful in implementing the Fuzzy Logic Vehicle
`"
`...
`
`W aming and Control Sy~tem.
`
`'
`v
`Fig. 11 is a Hazard Collision Cont:rol'\cector useful in implementing the Fuzzy Logic
`,..
`
`Vehicle Warning and Control System.
`
`7
`
`-
`
`11
`
`

`

`fi).!. 12 is a table of Hazard/Object state vectors indicating possible combinations of hazards
`
`and objects useful in the Fuzzy Associative Memory access system used herein.
`t
`~-
`~
`Fig. 13 is a more detailed \ogic 'ilow 'f<iagram for the analysis of detection signals prior to
`I
`,_
`"
`1\
`~
`.
`accessing 1\uzzy \ogic control structures in the Motor Vehicle Warning and Control System.
`-f o
`"
`"
`I
`Fig. 14 is a more detailed\ogic ~ow ~iagrant for the Fuzzy Associative Memory (PAM)
`"
`..
`'(\..
`
`selection processing.
`
`Fi~. 15 is an example system flow illustrating the operation of the Motor Vehicle Warning
`
`and Control System.
`
`, r::De \a.\ \-eci. "]) esc:r \.fC~ on
`·SYSTEM DESCRIPTION
`
`In Fig. 1 is shown a computelized control system 10 for controlling the operation of a
`
`motor vehicle to prevent or lessen the effects of accidents such as collisions with stationery and/or
`
`moving objects such as other vehicles. The system 10 employs a control computer or
`
`microprocessor 11 mounted on the vehicle and operable to receive and gate digital signals, such as
`
`codes and control signals from various sensors, to one or more specialized computers and from
`
`such computers to a number of servos such as electric motors and lineal actuators or solenoids,
`
`switches and the like, speakers and display drivers to perform either or both the functions of
`
`audibly ancVor visually informing or wm·ning the driver of the vehicle of a hazardous road
`
`condition ahead and/or to effect controlled braking and steering actions of the vehicle.
`
`A RAM 12 and ROM 13 are connected to processor 11 to effect and facilitate its operation.
`
`A television camera(s) 1_6 having a wide angle lens 16L is mounted at the front of the vehicle such
`
`as the front end of the roof, bumper or end of the hood to scan the road ahead of the vehicle at an
`
`angle encompassing the sides of the road and intersecting roads. The analog signal output of
`
`8
`
`12
`
`

`

`camera 16 is digitized in an A/D convertor 18 and passed directly to or through a video
`
`preprocessor 51 to microprocessor 11, to _an image field analyzing computer 19 which is provided,
`
`implemented and programmed using neural networks and artificial intelligence as well as fuzzy
`
`logic algorithms to (a) identify objects on the road ahead such as other vehicles, pedestrians,
`
`~
`
`baniers and dividers, turns in the road, signs and symbols, et~._and generate identification codes,
`A
`and (b) detect distances from such objects by their size (and shape) and provide codes indicating
`
`same for use by a decision computer, 23, which generates coded control signals which are applied
`
`through the computer 11 or are directly passed to various warning and vehicle operating devices
`
`such as a braking computer or drive, 35, which operates a brake servo 33, a steering computer or
`
`drive(s) 39 and 40 which operate ·steering servos 36; a synthetic speech signal generator 27 which
`
`sends trains of indicating and warning digital speech signals to a digital-analog converter 29
`
`connected to a speaker 30; a display driver 31 which drives a (heads-up or dashboard) display 32;
`
`_ _.
`
`a head light controller 41 for flashing the head lights, a warning light control 42 for flashing
`
`external and/or internal warning lights; a hom control43, etc.
`
`A digital speedometer 44 and accelerometer(s) 45 provide information sign.als for use by
`
`the decision computer, 23, in issuing its commands. Accelerometer(s) 45 are connected to control
`
`computer microprocessor 11 through analog-to-digital converter 46. The accelerometer(s) 45 may
`
`pass data continuously to control computer microprocessor 11, or, alternatively, respond to query
`
`signals from said control computer 11. An auxiliary range detection means comprises a range
`
`computer 21 which accepts digital code signals from a radar or lidar computer 14 which interprets
`
`radar and/or laser range signals from respective reflected radiation receiving means on the vehicle.
`
`The image analyzing computer 19 with associated memory 20 may be implemented in
`
`several different ways. Of particular concern is the requirement for high speed image processing
`
`with the capability to detect various hazards in dynamic image fields with changing scenes, moving
`
`9
`
`13
`
`

`

`objects and multiple objects, more than one of which maybe a potential hazard. Requirements for
`
`wide angle vision and the ability to analyze both right and left side image fields also ~xis~. The
`
`imaging system not only detects hazards, but also estimates distance based on image data for input
`
`to the range computer 21 implemented with the associated memory .unit 22.
`
`High speed image processing can be implemented employing known special purpose
`
`computer architectures including various parallel system structures and systems based on neural
`~~-
`networks. ~ 2 shows a high speed parallel processor system embodiment with dedicated
`r~-
`,.
`image processing hardware. The system of~ 2 has a dedicated image data bus 50 for high
`speed image data transfer. The video camera 16 transfers full-frame video picture signal!data to the
`
`.
`
`image bus 50 via analog/digital converter 18 and video preprocessor 51. The video camera 16 is
`
`preferably a CCD array camera generating successive picture frames with individual pixels being
`
`digitized for processing by the video preprocessor 51. The video camera 16 may also be
`
`implemented with other technologies including known image intensifying electron gun and infra(cid:173)
`
`red imaging methods. Multiple cameras may be ~r f~~-nt, side and rear viewing and for
`
`stereo imaging capabilities suitable for generation o:R;f.Dimel)sional image information including
`"
`capabilities for depth perception and placing multiple objects in three dimensional image fields to
`
`further improve hazard detection capabilities.
`
`As shown in Fig. 2, the video preprocessor 51 performs necessary video image frame
`
`management and data manipulation in preparation for image analysis. The preprocessor 51 may
`
`also be used in some embodiments for digital prefiltering and in1age enhancement. Actual image
`
`data can be displayed in real time using video display 55 via analog-to-digital converter 54. The
`
`image display may inclupe highlighting of hazards, special warning linages such as flashing lights,
`
`alpha-numeric messages, distance values, speed indicators and other hazard and safety related
`
`10
`
`14
`
`

`

`l-..
`
`._,
`
`messages. Simulated.as well as actual video displays may also be used to enhance driver
`"
`'Ci3
`recognition of dangerous situations .
`•
`The ~age~alysis ~omputer 19 operates under the control of control processor 56 with
`\"4:\tl.
`fl
`"
`"
`· random-access-memory (RAM) 57 and program and reference data stored in~-only memory
`
`(ROM) 58. The control processor 56 communicates with the motor vehicle warning and control
`
`system micro-processor controller ll through the Bus Interrace Unit 59. Results of the image
`
`analysis are passed in real-time to microprocessor controller 11 for integration with other sensory,
`
`computing, warning and control signals as depicted in Figure 1.
`i
`·i=" ~ •
`c:u
`c....
`The image !\nalysis 'l0omputer 19 of Figure 2 uses high speed dedicated co-processor 53
`k\
`~
`~
`for actual linage analysis under control of the control processor 56. Typical operations performed
`
`using co-processors 53 include multidimensional filtering for operations such as feature extraction
`
`and motion detection. The co-processors 53 are used for multidimensional discrete transforms and
`
`other digital flltering operations used in image analysis. Multiple image memorie~52 with parallel
`
`access to successive image data frames via image bus 50 permit~ concurrent processing with high
`
`speed data access by respective co-processing elements 53. The co-processor elements 53 may be
`
`high speed programmable processors or special purpose hardware processors specifically
`
`constructed for image analysis operations. SIMD (single. instruction, multiple data) architectures
`
`provide high speed operation with multiple identical processing elements under control of a control
`
`unit that broadcasts instructions to all processing elements. The same instruction is executed
`
`simultaneously on different data elements making this approach particularly well suited for matrix
`
`and vector operations commonly employed in image analysis operations. Parallel operations of ·
`
`this type are particularly important with high pixel counts. A 1000 x 1000 pixel image has one
`
`million data points. Tightly coupled Multiple Instruction, Multiple Data (MIMD) architectures also
`
`are used in image processing applications. MIMD systems execute independent but related
`
`11
`
`15
`
`

`

`programs concunently on multiple processing elements. Va1ious anay processor and massively
`
`parallel architectures known to those s]<illed in the art may also be used for real-time image
`
`analysis.
`
`The calculation of the distance of certain recognizable objects from the vehicle is facilitated
`
`by having standard images stored in memory and recalling and comparing such image data with
`
`image data representing the object detected by the vehicle scanning mechanisms. For example,
`
`virtually all automobiles, trucks, and other standard vehicles have known widths. It f2!!?_~-~-~~t
`
`\I -~e dista?ce to;:~ther vehicle can b~--~~-~~-~e.d by _ca1c.ulatin_¥ i~ vj_d_th ~:~~--~~~-~-~-!mage. If a
`CCD camera is used, for example, the width can ascertained in pixels in the image field. The
`
`distance to the vehicle can then be easily calculated using a simple relationship wherein the distance
`
`will be directly proportional to the object image width in pixels. The relative velocities and
`
`accelerations can also be easily calculated from respective first and second derivatives of the image ·
`
`width with respect to time. These image measurements and calculations can be used in addition to
`
`radar/lidar signal measurements or they may be used alone depending on system requirements.
`
`In another embodiment, the image analyzing computer 19 is implemented as a neural
`
`-····
`
`computing network with networked processingel~ments perfonning S\}CCessive computations on
`
`input image structure as shown in~:l~Fe 3 where signal inp-u~ 61 ~e connected to multiple
`processing elements 63, 65 and 67 through the network connections 62, 64 and 66. The
`
`processing elements (PE's) 63, 65 and 67 map input signal vectors to the output decision layer,
`
`performing such tasks as image recognition and image parameter analysis.
`
`A typical neural network processing element known to those skilled in the art is shown in
`
`o.__.
`
`Fig. 4 where input yectors~ (Xl, X2 .... Xn) are connected via weighing elements (Wl,
`
`W2 ..... Wn) to a summing node 70. The output of node 70 is passed through a nonlinear
`
`processing element 72 to produce an output signal, U. Offset or bias inputs can be added to the
`
`12
`
`16
`
`

`

`inputs through weighing circuit Wo. The output signal from summing node 70 is passed through
`
`the nonlinear element 72. The nonlinear function is preferably a continuous, differentiable
`
`function such as a sigmoid which is typically used in neural network processing element nodes.
`
`Neural networks used in the vehicle warning system are trained to recognize roadway hazards
`
`which the vehicle is approaching including automobiles, trucks, and pedestrians. Training
`
`involves providing known inputs to the network resulting in desired output responses.. The
`
`weights are automatically adjusted based on error signal measurements until the desired outputs are
`
`generated. Various leruning algorithms may be applied. Adaptive operation is also possible with
`
`on-line adjustment of network weights to meet imaging requirements. The neural network
`
`embodiment of the image analysis computer 19 provides a highly parallel image processing
`
`structure with rapid, real-time image recognition necessary for the Motor Vehicle Wruning and
`
`Control System. Very Large Scale Integrated (VLSI) Circuit implementation of the neural
`
`processing elements permits low-cost, low-weight implementation. Also, a neural network has
`
`certain reliability advantages important in a safety warning system. Loss of one processing
`
`element does not necessru·ily result in a processing system failure.
`In a alternate embodiment, the neural network computing network of 'Fl~tu's- 3 can be
`"
`implemented using multiple vi.Itual processing elements 73 interconnected via an image data bus 75
`.v~
`,..
`with an image processor 74 as shown in~ 5. Image data presented to the Image Processor 74
`is routed to selected virtual processing elements 73 which implement the neural network computing
`
`.Q•
`
`. .....
`
`functions. The virtual PE' s may be pipelined processors to increase speed and computational
`
`efficiency.
`
`The ~ecision tomputer 23 of-~]'i~re 1 integrates the inputs from the image analysis
`
`.
`
`~
`~
`computer 19, range computer 21, digital accelerometer 45, and the radar or lidar computer 14 to
`'
`generate output warning and contml signals. W ru·ning signals alert the driver of impending
`
`I
`
`13
`
`17
`
`

`

`hazards and, depending on the situation, actual vehicle control signals may be generated to operate
`
`the vehicle in a manner that will avoiq the hazard or minimize the danger to the vehicle and
`
`'
`
`passengers. ~;ntrol signals,iwHl be generated to operate brake servos 33 and steering servos 36.
`--- ...... .
`--
`Manual overrides are provided to ensure driver vehicle control if necessary.
`t
`{
`'··
`...
`..
`- ·--
`A particularly attractive embodiment of the decision computer 23 makes use of fuzzy '&ogic
`ov
`;.
`-
`I\
`Algorithmic structures to implement the automated control and warning signal generation. Fuzzy
`
`..
`
`·-· -----·--------·-·-----------------~---------
`
`'\ogic is particularly well suited to the vehicle control problem wherein it is necessary to deal with a
`
`"' multiplicity of image, motion, and environmental parameters, each of which may extend over
`
`ranges of values and in different combinations which require different responses.
`
`·~~re 6 illustrates a\low ~iagram for implementing a Fuzzy Logic Vehicle Control and
`"
`~
`Warning signal generation system suitable for the decision computer 23. The system of Fig. 6
`receives inputs via the ~ontrol ~omputer~croprocessor 11 of·~~ffi:e 1. Inputs include image
`
`analysis outputs, motion sensor outputs, distance measurements from radar/lidar systems, and
`
`-
`
`environmental parameters which may indicate adverse driving condition.s including rain or ice. The
`
`input signals are analyzed in a preprocessing step for hazardous conditions in the processing block
`
`_.
`
`74. When a hazard is detected, the Fuzzy Associative Memory (FAM) Seetimt block 76 described
`
`in more detail below is activated via decision element 75.
`
`If no hazard is present, the system
`
`_.
`
`continues to analyze scanning signals until a hazardous situation is encountered.
`b
`The Fuzzy Associative Memory (FAM) )}lock 76 also receives a parameter input file from
`k.
`t..:
`the Detection Signal Analysis ~lock 7 4. This file contains necessary information to make control
`'
`. -
`1\
`decision including, for example, hazard location (front, back, left side, right sidy-), hazard distance,
`
`relative velocity, steeriJ:!g angle, braking pressure, weather data, and the presence or a,bsence of
`
`obstructions or objects to the front, rear,\~or to either side of the vehicle.
`
`14
`
`18
`
`

`

`Control signals are derived using FAM' ~ 77, 78, 79 and 80. In practice , a large number
`
`ofF AM's may be used to reflect different possible driving conditions and hazard scenarios. Each
`
`Fuzzy Associative Memory maps input control parameter combinations to appropriate output
`
`· control signals. The output si