`
`SUBCLA$S
`
`141
`
`2.!::31i~
`2.·0 ~)
`
`: .. ;-~~, ..
`
`.... , .... ;.
`
`".:
`
`': .... ~::
`
`.:.,' ~
`
`".
`
`.:"...:'
`
`jVEHIFIEJ.)
`
`APPLICATIONS************
`
`I.
`
`I
`\
`\.
`
`. u.s. barr. of COMM .. Pat. .TMOff .. ..,.. PfO.43.&L ,,..,. iIi,**ft)
`:. ----.~.--.---~--.~~ .. -.. -- .. -.... ---.-.. --:--
`...... -.-~ ... .:. -.....
`
`,
`
`.. ",,,,,,,,,,,,nr Examiner _.
`
`Amourit Due
`
`Date Paid
`
`Sheets Drwg.
`
`Print Fig.
`
`ISSUE
`BATCH
`Primary Examiner . NUMBER
`
`WARNING: The information disclosed herein may be restricted. Unauthorized disclosure may be prohibited
`by the United States Code Title 35, Sections 122, 181 and 368. Possession outside the U.S.
`Patent & Trademark Office is restricted to authorized employees and contractors only.
`
`Label
`Area
`
`PTO·4.36A
`8/92)
`
`(FACE)
`
`IPR2013-00419 - Ex. 1003
`Toyota Motor Corp., Petitioner
`1
`
`
`
`08 105304
`
`PATENT APPLICATION SERIAL NO~. __ ___ _
`
`u.S. DEPARTMENT OF COMMERCE
`PATENT AND TRADEMARK OFFICE
`FEE RECORD SHEET
`
`140 KM 09/02/93 08105304
`
`1 101
`
`1,520.00 CK
`
`PTO-lSS6
`(5/87)
`
`2
`
`
`
`BAR CODE LABEL
`
`11111111111111111111111111111111111111111111111111
`
`U.S. PATENT APPLICATION
`
`SERIAl., NUMBER
`
`FILING DATE
`
`CLASS
`
`GROUP ART UNIT
`
`08/105,304
`
`08/11/93
`
`340
`
`2617
`
`JEROME H. LEMELSON,
`
`INCLINE VILLAGE, NV; ROBERT D. PEDERSEN, DALLAS, TX.
`
`**CONTINUING DATA*********************
`VERIFIED
`
`**FOREIGN/PCT APPLICATIONS************
`VERIFIED
`
`FOREIGN FILING LICENSE GRANTED 09/14/93
`
`***** SMALL ENTITY *****
`
`STATE OR
`COUNTRY
`
`SHEETS
`DRAWING
`
`TOTAL
`CLAIMS
`
`INDEPENDENT
`CLAIMS
`
`FILING FEE
`RECEIVED
`
`ATTORNEY DOCKET NO.
`
`NV
`
`13
`
`99
`
`11
`
`$1,520.00
`
`JEROME H. LEMELSON
`SUITE 286
`930 TAHOE BLVD.
`UNIT 802
`INCLINE VILLAGE, NV 89451-9436
`
`MOTOR VEHICLE WARNING AND CONTROL SYSTEM AND METHOD
`
`(/)
`
`(/) w
`c: o
`o
`<{
`
`W
`...J
`l(cid:173)
`F
`
`This is to certify that annexed hereto is a true copy from the records of the United States
`Patent and Trademark Office of the application whIch is identified above.
`By authority of the
`COMMISSIONER OF PATENTS AND TRADEMARKS
`
`Date
`
`Certifying Officer
`
`3
`
`
`
`IN
`
`'I'[]G UNITED S'l'l\'l'ES PA'l'EN'l' AND
`
`'l'RADEMAEK OPFICQ8 105304
`
`i can t :
`
`J e r Q III e
`
`[-1. L e l\1 e 1 son
`
`Application Branch
`
`August 11,1993
`
`Filed:
`
`For:
`
`No:
`
`August 11,1993
`
`MOTOR VEHICLE WARNING AND CONTROL
`SYSTEM AND METHOD
`
`APPLICATION TRANSMITTAL
`
`COJlllilis:sj_oncc of
`Pal: e n L: G
`'1' l" u deli I Cl c k s
`Ll n d
`\,'1 a .s 11 i 11 <j L: 0 n , D . C. 2 0 2 3 1
`
`Sir-:
`
`I hereby cCI:L:ify thaL: the ubove identified potent
`
`application enclosed hecewith is being deposited in tile U.S.
`
`HB 242939400
`PosL:al Servive as Express MClil No:
`-------------------------------
`addressed to the Commissioner of Patents
`
`on August lL1993
`
`and Trademarks, Washington,D.C. 20231.
`
`A check covering the filing of 99 claims including
`
`8 irldepenrient claims over three (total of 11 independent claims)
`
`is enclosed herewith together with a Small Entity Declaration~
`
`a Declaration and Power of Attorney; a 31 page specification
`
`and claims 1-99 and 13 pages showing FIGS. 1-15 of drawings.
`Calculation of Total Filing Fee amount:
`
`1. Bas i c F i 1 ing Fee ................................ .
`2. Claims in Excess of 20 = 79 .... ( 79 x $11=
`3. Independent Claims (in excesss of 3) = 8x $37)
`
`$ 355.00
`869.00
`296.00
`
`Total Filing Fee=$1520.00
`
`A check in the amount of $1522 is enclosed as is a postcard
`listing the above enclosed items to be mailed to applicant.
`~ speCtfU~b~
`erome H.Lem~~~
`Suite 286 - unit 802
`930 Tahoe Blvd.
`Incline Village,NV. 89451-9436
`
`4
`
`
`
`l (5'2.0 -/ Dc 4-
`08 1 0530~
`Jero~~t;~~:lson
`930 Tahoe Blvd., Unit 802
`InclineVillage, NV 89451-9436
`
`MOTOR VEHICLE WARNING AND CONTROL SYSTEM AND METHOD
`
`Abstract of the Disclosure
`A system and method fe¥ assis~ the driver of a motor vehicle in preventing accidents or
`
`minimizing the effects of same. In one form, a television camera is mounted on a vehicle and
`
`scans the roadway ahead of the vehicle .as the vehicle travels. Continuously generated video
`
`picture signals output by the camera are electronically processed and analyzed by an image
`~
`analyzing computer,which generates codes~ serve to identify obstacles'ffil th~ road ahead of..
`7'
`4e vehicle. B~iHg such id€mtifyiIJg
`
`jnfmJUation aRd eelnrparing it v/ith infofffl:atioR on-the-strapes
`
`.&fld sizes of various objeets such a~ rear and front profiles of all production vehicles tmd the like"
`
`and their mlative sires elf select dnnensiolls thereof, indieatiolls of di£tallces to sllch objects may be
`
`computed and indicated as further cod~ A decision computer mounted in the controlled vehicle"
`
`receives such code signals along with code signals generated by the speedometer or one or more
`
`spees and directioR sensors sensing steering mechanism operation and generates control signals~~
`
`dRving a display, such as a heads up diSfl"ltry on the windshield and/or dashoofl:f'd to display stlcn
`
`~
`
`1fffermatielft as images of the cORtrolled vehicle and other vehicles in and adjacent its path. of travel '
`and relative distances thereto as we]J as groups of characters defiIJiIJg safB:e, edOled and flaSl~ \:v
`.1,1,'arning lights and the like for PIe- warning and warning pBi'pelS8S. Such code signa~s ~~
`'"")
`,
`1'-
`d. \S P Lo..~-ed.J QV'\C:
`.
`0'1""
`"
`~ispray(s) as '.-",e11 a.s-.synthetic speec~ttfttlfflr special sound generating and warning meansffia~ b@-
`\JSe£1
`_
`i-.. e:mplOyed tbto warn the driver of the vehicle of approaching and existing hazardsk!tfttl, in eerta:iH +
`Qna
`~
`~
`~N.) A I( -tieteeteei--ffi.~flC-es, to control the operation of the brakes 'Mdiol" steering mechanism of the vehicle
`~~J
`~
`:"\I s.§ to avoid or lessen the effects of a collision. In a particular fonn, the television camera and/or
`IA.;L
`auxiliary radiatioll scanning lIleetn3 may sean a 'vvide angle in front Qf the vehicle and areas to the
`
`sides of the vehicle to sense other vehicles uaveliIlg ill !lIe same direction aftd provide infonn8:tion
`
`in-the Form of cedes to the deci-s-ieH-eompmer to permit it to aftalyze SaIne ill effectmg autonrat:Ie
`
`1
`
`5
`
`
`
`I contrel of the :vehicle to avoid a collisicjfi1l,..."ith a vehicle clirectly aheacl Me to €lither or both siS€lp
`a.aJ the rear of-the controlled ve1risJ?
`.Whlle...manual override means is provided, the decision
`
`cGmpurer may ovenide 01 pre veftt the operatioft of same if jt detenuines that a COl:1:iSiOlI may occor
`
`if-sllch 0 v'Crrids means is psnnitt€d
`
`to be operated. --In 8: ffioaifieo :£gnu, video scanning ana t:ad.ar
`
`QJ.:.lidar scanning may be jointly employed to ideatify and indicate distances between the cOftffoilcxi
`
`vehicle and objccts ahead to the side(s) and lear of the eont:F~.
`
`2
`
`6
`
`
`
`\="\e.\4
`"
`Summary of the Inyention
`.
`This invention relates to a system and method for operating a motor vehicle, such as an
`
`automobile, truck, aircraft or other vehicle, wherein a computer or computerized system is
`
`employed to assist and/or supplement the driver in the movement of the vehicle along a path of
`
`travel, such as a street or roadway and may be used to avoid obstacles and accidents.un a
`)Sl
`preferred form of the invention, a video scanning system, such as a televisiot.'l. camera and/or one or
`
`more laser scanners mounted on the vehicle scan the road in front of the vehicle and generate image
`
`information which is computer analyzed per se or in combination with a range sensing system to
`
`warn the driver of hazardous conditions during driving by operating a display, such as a heads-l!:t
`
`r - - - - - -
`
`display, and/or a synthetic speech generating means which generates sounds or words of speech to
`
`verbally indicate such road conditions ahead of the vehicle.
`
`The preferred form of the invention provides audible and/or visual display means to
`
`cooperate ill illdicatillg to the driver of H motor vehicle both normal and hazardous road conditions
`
`ahead as well as driving variables such as distances to stationary objects, and other vehicles; the
`
`identification, direction of travel and speed of such other vehicles~ and the identification of and
`
`distances to stationary or slowly moving objects such as barriers, center islands, pedestrians,
`~o
`parked cars poles, sharp turns in the road~ other conditions. In addition, the image analyzing
`
`>'-
`
`computer of the vehicle may be operated to scan and decode coded and/or character containing
`
`signs or signals generated by indicia or code generating other devices within or at the side of the
`
`road and indicating select road and driving conditions ahead.
`
`The computer i~ operable to analyze video and/or other forms~ of image information
`
`generated as the vehicle travels to identify obstacles ahead of the vehicle and, in certain instances,
`,
`quantify the distance between the vehicle containing same on the basis of the size of the identified
`
`3
`
`7
`
`
`
`~
`vehicle or object and/or by processing received pulse-echo signals'J'"'~hen the closing distance
`
`becomes hazardous, select vehicle subsystems may be automatically controlled by the computer as
`
`it continues to analyze image signals generated by the television camera. A first subsystem
`
`generates a first select code or codes which controls an electronic display, such as a heads-up
`
`\\~ ti1\
`
`display to cause it to display a warning indication, such as one or more flashing red light portions
`of the display or other lighted effect. ~ second subsystem generates a code or series of codes
`1 t\1J-
`which control a sound genel:ating means which generates a select sound such as a horn, buzzing
`
`sound and/or select synthetic speech warning of the hazardous condition detected and, in certain
`
`instances, generating sounds of select words of speech which may warn of same and/or suggest
`
`corrective action ~ by the vehicle operator or driver to avoid an accident.
`
`A third subsystem comes on-line and generates one or more codes which are applied to at
`
`least partly effect a cOlTective action such as by pulsing one or more motors or solenoids to apply
`
`the brakes of the vehicle to cause it to slow down. If necessary to avoid or lessen the effects of an
`
`accident, the third subsystem stops the forward travel of the vehicle in a controlled manner
`
`depending on the relative speeds of the two vehicles, and/or the controlled vehicle and a stationery
`
`object or structure and the distance therebetween.
`
`A fourth subsystem, which may be part of or separate from the third subsystem may
`
`generate one or more codes which are applied to either effect partial and/or complete control of the
`
`steeling mechanism for the vehicle to avoid an obstacle and/or lessen the effect of an accident.
`
`Either or both the third or fourth subsystem may also be operable to control one or more safety
`
`devices by controlling motors, solenoids or valves, to operate a restraining device or devices for
`
`the driver and passeng~r(s) of the vehicle, such as a safety belt tightening means, an air bag
`
`inflation means or other device designed to protect human beings in the vehicle.
`"
`
`4
`
`8
`
`
`
`The second, and/or third and fourth subsystems may also be operable to effect or control
`
`the operations~ of additional warning 1p.eans such as the horn)headlights and/or other warning
`4\
`lights on the vehicle or other waming means which operates to alelt, flag or warn the driver of the
`
`approaching or approached vehicle or a pedestrian of the approaching hazardous condition. One or
`
`more of these subsystems may also be operable to generate and transmit one or more codes to be
`
`received and used by the approaching or approached vehicle or a roadside device to effect
`
`additional on-line warning(s) of the hazardous condition, and/or may be recorded on a disc or
`
`RAM (random access memory) for future analysis, if necessary.
`
`In a modified form of the invention, the vehicle waming system may also include a short
`
`wave receiving means to receive code signals from other vehicles and/or short wave transmitters at
`
`the side of or within the road for controlling the visual, audio and/or brake and steering means of
`the vehicle to avoid or~~:'the effects of an accident and/or to maintain the vehicle in-lane and in
`proper operating condition as it travels.
`
`The systems and methods of this invention preferably employ computerized image
`
`;x
`
`\
`
`analyzing techniques of the types disclosed and defined in such patents of mine as 4,969,038 and
`
`4,979,029 and references cited in the file wrappers thereof as well as other more recent patents
`
`and include the use of known artificial intelligence, neural networking and fuzzy logic computing
`
`electronic circuits.
`'NS~
`
`'>
`
`Accordingly it is a primary object of this invention to provide a new and improved system
`,+,bwere.a \I~,,~
`and method for controlling the operation of a..H3:otof 'reb-isle, ~(')at, train or airerM't.·
`1\
`Another object is provide a system and method for assisting the driver of ~ firotor vehicle~
`rp OWQ.~e-d \l.e.\i~~
`-train, boat or ahcrafH.n controlling its operation to avoid an accident or hazardous driving
`f\
`-
`condition.
`
`5
`
`9
`
`
`
`-- ... ~~, ..........
`Another object is to provide a s?,stem and method ~l1lploying .computerized image analySis)
`
`to control or assist the driver of a moto! vehicle in controlling its operation to avoid hazardous
`
`conditions such as collisions with other vehicles, stationery objects or pedestrians.
`
`Another object is to provide a computerized system and method for controlling the speed of
`
`travel of a motor vehicle to lessen the chances of an accident while being driven by a person.
`
`Another object is to provide a system and method employing a television scanning camera
`
`mounted on a vehicle for scanning the field ahead, such as the image of the road ahead of the
`
`vehicle and a computer for analyzing the image signals generated wherein automatic image
`
`intensifying, or infra-red scanning and detection means is utilized to pennit scanning operations to
`
`be effected dud:p:g clflving at night and in low light, snowing or fog conditiorls>,
`
`\
`
`Another object is to provide a system and method employing a television camera or other
`
`video scanning means mounted on a moving motor vehicle for scanning, detecting and identifying
`
`obstacles such as other vehicles ahead of such moving vehicle wherein the video image signals are
`
`analyzed to determine distances to such objects.
`
`Another object is to provide a computer controlled safety system for a motor vehicle which
`
`employs a television camera and an auxiliary scanning means to both identify obstacles in the path
`
`of the vehicle and detelmine distance therefrom on a real time and continuous basis for use in
`
`warning the operator of same and/or in controlling the operation of the vehicle to avoid a collision.
`
`BRIEF DESCRIPTION OF DRAWINGS
`
`The vruious harqware and software elements used to carry out the invention described
`
`herein are illustrated in the fOlm of block diagrams, flow chruiS, and depictions of neural network
`
`6
`
`10
`
`
`
`and fuzzy logic algorithms and structures. The preferred embodiment is illustrated in the following
`
`figures:
`
`Fig. 1 is a block diagram of the overall Motor Vehicle Warning and Control System
`
`. illustrating system sensors, computers, displays, inputJoutput devices and other key elements.
`IL
`0......
`'1
`~ is a block diagram of an ~mage ~alysis ~omputer 19 of the type that can be used in
`
`the Vehicle Hazard Avoidance System herein of Fig. 1.
`•
`~
`\
`OJ
`Fig. 3 illustrates a neural network of the type useful in the l.luage ~nalysis ~omputer of
`II.
`1\
`(\
`
`Fig. 4.
`
`3.
`
`E:ig:.A illustrates the structure of a Processing Element (PE) in the neural network of Fig.
`
`Fig. 5 is an alternate embodiment of a neural network image processor useful in the system
`
`of Fig. 1.
`
`d
`-F
`Eig...Q is a ~low 'E)iagram illustrating the overall operation of the Motor Vehicle Waming
`t--
`"
`and Control System of Fig. 1.
`
`1
`~
`~
`E.igJ illustrates typical input signal membership functions for):uzzy \ogic 'ilgorithms
`~
`~
`~\.
`
`useful in the Motor Vehicle Warning and Control System of Figure 1.
`
`1
`Q.
`~
`Fig. 8 illustrates typical output signal membership functions for ~uzzy \ogic \lgorithms
`" "
`"
`
`useful in the Motor Vehicle Warning and Control System of Fig. 1.
`
`~ illustrates typical Fuzzy Associative Memory (FAM) maps for the Fuzzy Logic
`
`Algorithms useful in the Motor Vehicle Waming and Control System of Fig. 1.
`.
`v
`$
`Fig. 10 is a Hazard/Object ~tate Xector useful in implementing the Fuzzy Logic Vehicle
`"
`...
`
`Waming and Control Sy~tem.
`
`'
`V
`Fig. 11 is a Hazard Collision ControI'\cector useful in implementing the Fuzzy Logic
`,..
`
`Vehicle Warning and Control System.
`
`7
`
`-
`
`11
`
`
`
`Pi).!. 12 is a table of Hazard/Object state vectors indicating possible combinations of hazards
`
`and objects useful in the Fuzzy Associative Memory access system used herein.
`t
`~.
`~
`Fig. 13 is a more detailed \ogic 'ilow 'Riagram for the analysis of detection signals prior to
`,.""
`\
`~
`accessing 1\uzzy \ogic control structures in the Motor Vehicle Warning and Control System.
`-f
`I
`0
`" I \ ,
`Fig. 14 is a more detailed\ogic ~ow ~iagran1 for the Fuzzy Associative Memory (PAM)
`'f\. .
`..
`f\
`
`'
`
`selection processing.
`
`Fi~. 15 is an example system flow illustrating the operation of the Motor Vehicle Warning
`
`and Control System.
`
`, r::De\o,\ \-eci. 'J) e6c"('\.fC~on
`.SYSTEM DESCRIPTION
`
`In Fig. 1 is shown a computerized control system 10 for controlling the operation of a
`
`motor vehicle to prevent or lessen the effects of accidents such as collisions with stationery and/or
`
`moving objects such as other vehicles. The system 10 employs a control computer or
`
`microprocessor 11 mounted on the vehicle and operable to receive and gate digital signals, such as
`
`codes and control signals from various sensors, to one or more specialized computers and from
`
`such computers to a number of servos such as electric motors and lineal actuators or solenoids,
`
`switches and the like, speakers and display drivers to periorm either or both the functions of
`
`audibly ancVor visually informing or wm'ning the driver of the vehicle of a hazardous road
`
`condition ahead and/or to effect controlled braking and steering actions of the vehicle.
`
`A RAM 12 and ROM 13 are connected to processor 11 to effect and facilitate its operation.
`
`A television camera(s) 1,6 having a wide angle lens 16L is mounted at the front of the vehicle such
`
`as the front end of the roof, bumper or end of the hood to scan theroad ahead of the vehicle at an
`
`angle encompassing the sides of the road and intersecting roads. The analog signal output of
`
`8
`
`12
`
`
`
`camera 16 is digitized in an AJD convertor 18 and passed directly to or through a video
`
`preprocessor 51 to microprocessor 11, to ,an image field analyzing computer 19 which is provided,
`
`implemented and programmed using neural networks and artificial intelligence as well as fuzzy
`
`logic algorithms to (a) identify objects on the road ahead such as other vehicles, pedestrians,
`
`~
`
`baniers and dividers, turns in the road, signs and symbols, et~.,and generate identification codes,
`A
`and (b) detect distances from such objects by their size (and shape) and provide codes indicating
`
`same for use by a decision computer, 23, which generates coded control signals which are applied
`
`through the computer 11 or are directly passed to various warning and vehicle operating devices
`
`such as a braking computer or drive, 35, which operates a brake servo 33, a steering computer or
`
`drive(s) 39 and 40 which operate 'steering servos 36; a synthetic speech signal generator 27 which
`
`sends trains of indicating and warning digital speech signals to a digital-analog converter 29
`
`connected to a speaker 30; a display driver 31 which drives a (heads-up or dashboard) display 32;
`
`---
`
`a head light controller 41 for flashing the head lights, a warning light control 42 for flashing
`
`external anellor internal warning lights; a hom control 43, etc.
`
`A digital speedometer 44 and accelerometer(s) 45 provide information sign,als for use by
`
`the decision computer, 23, in issuing its commands. Accelerometer(s) 45 are connected to control
`
`computer microprocessor 11 through analog-to-digital converter 46. The accelerometer(s) 45 may
`
`pass data continuously to control computer microprocessor 11, or, alternatively, respond to query
`
`signals from said control computer 11. An auxiliary range detection means comprises a range
`
`computer 21 which accepts digital code signals from a radar or lidar computer 14 which interprets
`
`radar and/or laser range signals from respective reflected radiation receiving means on the vehicle.
`
`The image analyzing computer 19 with associated memory 20 may be implemented in
`
`several different ways. Of particular concern is the requirement for high speed image processing
`
`with the capability to detect various hazards in dynamic image fields with changing scenes, moving
`
`9
`
`13
`
`
`
`objects and multiple objects, more than one of which maybe a potential hazard. Requirements for
`
`wide angle vision and the ability to analyze both right and left side image fields also~xis~. The
`
`imaging system not only detects hazards, but also estimates distance based on image data for input
`
`to the range computer 21 implemented with the associated memory .unit 22.
`
`High speed image processing can be implemented employing known special purpose
`
`.
`
`computer architectures including various parallel system structures and systems based on neural
`~~,
`networks. ~ 2 shows a high speed parallel processor system embodiment with dedicated
`r~,
`,.
`image processing hardware. The system of~ 2 has a dedicated image data bus 50 for high
`speed image data transfer. The video camera 16 transfers full-frame video picture signalJdata to the
`
`image bus 50 via analog/digital converter 18 and video preprocessor 51. The video camera 16 is
`
`preferably a CCD array camera generating successive picture frames with individual pixels being
`
`digitized for processing by the video preprocessor 51. The video camera 16 may also be
`
`implemented with other technologies including known image intensifying electron gun and infra(cid:173)
`
`red imaging methods. Multiple cameras may be ~r f~~,nt, side and rear viewing and for
`
`stereo imaging capabilities suitable for generation oRkDimel)sional image information including
`"
`capabilities for depth perception and placing multiple objects in three dimensional image fields to
`
`further improve hazard detection capabilities.
`
`As shown in Fig. 2, the video preprocessor 51 performs necessary video image frame
`
`management and data manipulation in preparation for image analysis. The preprocessor 51 may
`
`also be used in some embodiments for digital prefIltering and inlage enhancement. Actual image
`
`data can be displayed in real time using video display 55 via analog-to-digital converter 54. The
`
`image display may inclupe highlighting of hazards, special warning linages such as flashing lights,
`
`alpha-numeric messages, distance values, speed indicators and other hazard and safety related
`
`10
`
`14
`
`
`
`)...,
`
`.....
`
`messages. Simulated.as well as actual video displays may also be used to enhance driver
`f(i3
`.
`recognition of dangerous situations .
`•
`The ~age~alysis ~omputer 19 operates under the control of control processor 56 with
`\"4;q,
`"
`1\
`'1\
`, random-access-memory (RAM) 57 and program and reference data stored in~-only memory
`
`(ROM) 58. The control processor 56 communicates with the motor vehicle warning and control
`
`system micro-processor controller 11 through the Bus Interface Unit 59. Results of the image
`
`analysis are passed in real-time to microprocessor controller 11 for integration with other sensory,
`
`computing, warning and control signals as depicted in Figure 1.
`i
`c:u
`'i="~.
`c....
`The image !\nalysis 'l0omputer 19 of Figure 2 uses high speed dedicated co-processor 53
`k\
`~
`~
`for actual unage analysis under control of the control processor 56. Typical operations performed
`
`using co-processors 53 include multidimensional filtering for operations such as feature extraction
`
`and motion detection. The co-processors 53 are used for multidimensional discrete transforms and
`
`other digital flltering operations used in image analysis. Multiple image memorie~52 with parallel
`
`access to successive image data frames via image bus 50 permit~ concurrent processing with high
`
`speed data access by respective co-processing elements 53. The co-processor elements 53 may be
`
`high speed programmable processors or special purpose hardware processors specifically
`
`constructed for image analysis operations. SIMD (single. instruction, multiple data) architectures
`
`provide high speed operation with multiple identical processing elements under control of a control
`
`unit that broadcasts instructions to all processing elements. The same instruction is executed
`
`simultaneously on different data elements making this approach particularly well suited for matrix
`
`and vector operations commonly employed in image analysis operations. Parallel operations of .
`
`this type are particularly important with high pixel counts. A 1000 x 1000 pixel image has one
`
`million data points. Tightly coupled Multiple Instruction, Multiple Data (MIMD) architectures also
`
`are used in image processing applications. MIMD systems execute independent but related
`
`11
`
`15
`
`
`
`programs conculTently on multiple processing elements. Va110us anay processor and massively
`
`parallel architectures known to those sl<illed in the art may also be used for real-time image
`
`analysis.
`
`The calculation of the distance of certain recognizable objects from the vehicle is facilitated
`
`by having standard images stored in memory and recalling and comparing such image data with
`
`image data representing the object detected by the vehicle scanning mechanisms. For example,
`
`virtually all automobiles, trucks, and other standard vehicles have known widths. It tQ!!?_~_~_~~t
`
`\\ _~e dista?ce tor:~thef vehicle can b~ __ ~~_~~_~e.d by_calc.ulatin_¥i~ vj.d.th~:~~ .. ~~~.~_~_!mage. If a
`
`CCD camera is used, for example, the width can ascertained in pixels in the image field. The
`
`distance to the vehicle can then be easily calculated using a simple relationship wherein the distance
`
`will be directly proportional to the object image widtll in pixels. The relative velocities and
`
`accelerations can also be easily calculated from respective first and second derivatives of the image·
`
`width with respect to time. These image measurements and calculations can be used in addition to
`
`radarllidar signal measurements or they may be used alone depending on system requirements.
`
`In another embodiment, the image analyzing computer 19 is implemented as a neural
`
`-....
`
`computing network with networked processingel~mentsperfonning s\1ccessive computations on
`input image structure as shown in~;;~Fe 3 where signal inp-u~ 61 ~e connected to multiple
`processing elements 63, 65 and 67 through the network connections 62, 64 and 66. The
`
`processing elements (PE's) 63, 65 and 67 map input signal vectors to the output decision layer,
`
`performing such tasks as image recognition and image parameter analysis.
`
`A typical neural network processing element known to those skilled in the art is shown in
`
`0..-
`
`Fig. 4 where input 'lectors~ (Xl, X2 .... Xn) are connected via weighing elements (WI,
`
`W2 ..... Wn) to a summing node 70. The output of node 70 is passed through a nonlinear
`
`processing element 72 to produce an output signal, U. Offset or bias inputs can be added to the
`
`12
`
`16
`
`
`
`inputs through weighing circuit Woo The output signal from summing node 70 is passed through
`
`the nonlinear element 72. The nonlinear function is preferably a continuous, differentiable
`
`function such as a sigmoid which is typically used in neural network processing element nodes.
`
`Neural networks used in the vehicle warning system are trained to recognize roadway hazards
`
`which the vehicle is approaching including automobiles, trucks, and pedestrians. Training
`
`involves providing known inputs to the network resulting in desired output responses.· The
`
`weights are automatically adjusted based on error signal measurements until the desired outputs are
`
`generated. Various leruning algorithms may be applied. Adaptive operation is also possible with
`
`on-line adjustment of network weights to meet imaging requirements. The neural network
`
`embodiment of the image analysis computer 19 provides a highly parallel image processing
`
`structure with rapid, real-time image recognition necessary for the Motor Vehicle Wruning and
`
`Control System. Very Large Scale Integrated (VLSI) Circuit implementation of the neural
`
`processing elements pennits low-cost, low-weight implementation. Also, a neural network has
`
`certain reliability advantages important in a safety warning system. Loss of one processing
`
`element does not necessru'ily result in a processing system failure.
`"""
`In a alternate embodiment, the neural network computing network of 'Fi~tlf:s- 3 can be
`"
`implemented using multiple viItual processing elements 73 interconnected via an image data bus 75
`.f~
`,..
`with an image processor 74 as shown in~ 5. Image data presented to the Image Processor 74
`is routed to selected virtual processing elements 73 which implement the neural network computing
`
`. .....
`
`functions. The virtual PE's may be pipelined processors to increase speed and computational
`
`efficiency.
`
`The ~ecision tomputer 23 of-~;'i~re 1 integrates the inputs from the image analysis
`
`.
`
`~
`~
`computer 19, range computer 21, digital accelerometer 45, and the radar or lidar computer 14 to
`,
`generate output warning and contl'Ol signals. W ru'ning signals alert the driver of impending
`
`I
`
`13
`
`17
`
`
`
`hazards and, depending on the situation, actual vehicle control signals may be generated to operate
`
`the vehicle in a manner that will avoiq the hazard or minimize the danger to the vehicle and
`
`passengers. ~;ntrol signals,lwHl be generated to operate brake servos 33 and steering servos 36.
`
`-'~"-""
`
`,
`
`-.
`
`. .. _. -'---"--~---"~."---'-'-'-----'-~---"'-'-
`Manual overrides are provided to ensure driver vehicle control if necessary.
`{ t
`'-.
`...
`..
`-'--
`A particularly attractive embodiment of the decision computer 23 makes use of fuzzy ,&ogic
`OV
`; ._
`1\
`Algorithmic structures to implement the automated control and warning signal generation. Fuzzy
`
`'\ogic is particularly well suited to the vehicle control problem wherein it is necessary to deal with a
`"-
`mUltiplicity of image, motion, and environmental parameters, each of which may extend over
`
`ranges of values and in different combinations which require different responses .
`. ~~!e 6 illustrates a\low ~iagram for implementing a Fuzzy Logic Vehicle Control and
`"
`~
`Warning signal generation system suitable for the decision computer 23. The system of Fig. 6
`receives inputs via the ~ontrol ~omputer~croprocessor 11 Of.~~ffi:e 1. Inputs include image
`analysis outputs, motion sensor outputs, distance measurements from radarllidar systems, and
`
`environmental parameters which may indicate adverse driving condition,s including rain or ice. The
`
`input signals are analyzed in a preprocessing step for hazardous conditions in the processing block
`
`-
`
`-"
`
`74. When a hazard is detected, the Fuzzy Associative Memory (PAM) SeetiOft block 76 described
`
`-'
`
`in more detail below is activated via decision element 75.
`
`If no hazard is present, the system
`
`continues to analyze scanning signals until a hazardous situation is encountered.
`b
`The Fuzzy Associative Memory (FAM) )}lock 76 also receives a parameter input fIle from
`k
`t..:
`the Detection Signal Analysis ~lock 74. This file contains necessary information to make control
`,
`. -
`1\
`decision including, for example, hazard location (front, back, left side, right sidy)' hazard distance,
`relative velocity, steeriI!g angle, braking pressure, weather data, and the presence or a,bsence of
`
`obstructions or objects to the front, rear,\~or to either side of the vehicle.
`
`14
`
`18
`
`
`
`Control signals are derived using FAM' ~ 77, 78, 79 and 80. In practice, a large number
`
`of F AM's may be used to reflect different possible driving conditions and hazard scenarios. Each
`
`Fuzzy Associative Memory maps input control parameter combinations to appropriate output
`
`. control signals. The output sig