throbber
Mercedes-Benz USA, LLC, Petitioner - Ex. 1003
`
`

`
`00 005304
`
`
`
`PATENT APPLICATION SERIAL NO.
`
`
`
`
`U.S. DEPARTMENT OF COMMERCE
`
`
`
`
`PATENT AND TRADEMARK OFFICE
`
`
`
`
`FEE RECORD SHEET
`
`
`
`
`
`
`
`
`140 KM 09/02/93 09105304
`
`1 101
`
`
`
`
`1,52o_oo CK
`
`
`
`PTO—1556
`(5/87)
`
`
`
`
`2
`
`

`
`BAR ODE LABEL
`
`
`
`
`
`
`II
`
`IlillllllllllIllllllllllllllllllIIIIIIIIIIII
`
`
`
`Us PATENT APPLICATION
`
`
`
`
`
`SERIAL NUMBER
`
`
`
`I
`
`FILING DATE
`
`
`
`oe/105,304
`
`
`
`08/11/93
`
`
`
`
`
`
`
`GROUP ART UNIT
`
`
`
`
`2617
`
`
`
`JEROME H. LEMELSON,
`
`
`
`
`INCLINE VILLAGE, NV; ROBERT D. PEDERSEN, DALLAS, TX.
`
`
`
`
`
`
`
`
`
`
`**coNTINUIN(; DAT‘A*~k:\'**:\'*~k**'k*:\'~k******'k
`
`VERIFIED
`
`
`
`
`
`**FOREIGN/PCT APPLICATIONS************
`VERIFIED
`
`
`
`
`
`
`APPLICANT
`
`
`
`
`
`
`
`
`
`*+«*** SMALL ENTITY *****
`ATTORNEY DOCKET NO.
`FILING FEE
`
`
`
`
`
`RECEIVED
`
`
`
`
`
`
`
`FOREIGN FILING LICENSE GRANTED 09/14/93
`STATE OR
`SHEETS
`INDEPENDENT
`
`
`
`COUNTRY
`DRAWING
`CLAIMS
`
`
`
`
`
`
`
`
`
`NV
`
`
`13
`
`
`
`
`
`11
`
`
`
`$1,520.00
`
`
`
`JEROME H. LEMELSON
`
`
`
`SUITE 286
`
`
`930 TAHOE BLVD.
`
`
`
`UNIT 802
`
`
`INCLINE VILLAGE, NV 89451-9436
`
`
`
`
`
`
`
`
`
`
`ADDRESS
`
`
`
`MOTOR VEHICLE WARNING AND CONTROL SYSTEM AND METHOD
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`that annexed hereto is 531 true copy from_ the records of the United States
`This is to certif
`
`
`
`
`
`
`
`
`
`
`
`Patent and Tra emark Office of the application which is identified above.
`By authority-of the
`
`
`
`
`COMMISSIONER OF PATENTS AND TRADEMARKS
`
`
`
`
`
`
`
`
`Date
`
`
`Certifying Officer
`
`
`
`
`3
`
`

`
`.._u...
`
`__“
`
`...
`
`
`
`
`
`

`
`2.
`
`....5
`
`
`
`
`

`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`6
`
`

`
`INS &}
`
`
`
`
`_
`field
`
`
`
`- of the Inyention
`F‘
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`This invention relates to a system.and method for operating a motor vehicle, such as an
`
`
`
`
`
`
`
`
`
`
`
`
`
`' automobile, truck, aircraft or other vehicle, wherein a computer or computerized system is
`
`
`
`employed to assist and/or supplement the driver in the movement of the vehicle along a path of
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`travel, such as a street or roadway and may be used to avoid obstacles and accidentsfjn a
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`preferred form of the invention, a video scanning system, such as a telev_i>‘sg_ioi_i__"_c_Aarr_iera and/or one or
`
`B1
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`more laser scanners mounted on the vehicle scan the road in front of the vehicle and generate image
`information which is computer analyzed per se or in combination with a range sensing system to
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`warn the driver of hazardous conditions during driving by operating a display, such as a headsjupri
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`display, and/or a synthetic speech generating means which generates sounds or words of speech to
`verbally indicate such road conditions ahead of the vehicle.
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`The preferred form of the invention provides audible and/or visual display means to
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`cooperate in indicating to the driver of ii motor vehicle both normal and hazardous road conditions
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`ahead as well as driving variables such as distances to stationary objects, and other vehicles; the
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`identification, direction of travel and speed of such other vehicles, and the identification of and
`
`
`
`
`
`
`
`
`
`
`
`
`
`distances to stationary or slowly moving objects such as barriers, center islands, pedestrians,
`Omd
`
`
`
`
`
`
`
`
`
`
`parked cars poles, sharp turns in the road ;a~n\other conditions.
`
`
`
`
`
`
`
`In addition, the image analyzing
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`computer of the vehicle may be operated to scan and decode coded and/or character containing
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`signs or signals generated by indicia or code generating other devices within or at the side of the
`
`
`
`
`
`
`
`
`
`
`road and indicating select road and driving conditions ahead.
`
`
`
`
`
`
`
`
`
`
`
`
`The computer is operable to analyze video and/or other forrnsb; of image information
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`generated as the vehicle travels to identify obstacles ahead of the vehicle and, in certain instances,
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`quantify the distance between the vehicle containing same on the basis of the size of the identified
`
`
`
`
`
`7
`
`

`
`~
`vehicle or object and/or by processing received pulse-echo signals.J'"'~hen the closing distance
`
`becomes hazardous, select vehicle subsystems may be automatically controlled by the computer as
`
`it continues to analyze image signals generated by the television camera. A first subsystem
`
`generates a first select code or codes which controls an electronic display, such as a heads-up
`
`\I~ ti1\
`
`display to cause it to display a warning indication, such as one or more flashing red light portions
`of the display or other lighted effect.~ second subsystem generates a code or series of codes
`1 t\IJ-
`which control a sound gene1:ating means which generates a select sound such as a horn, buzzing
`
`sound and/or select synthetic speech warning of the hazardous condition detected and, in certain
`
`instances, generating sounds of select words of speech which may warn of same and/or suggest
`
`corrective action~ by the vehicle operator or driver to avoid an accident.
`
`A third subsystem comes on-line and generates one or more codes which are applied to at
`
`least partly effect a cmTective action such as by pulsing one or more motors or solenoids to apply
`
`the brakes of the vehicle to cause it to slow down. If necessary to avoid or lessen the effects of an
`
`accident, the third subsystem stops the forward travel of the vehicle in a controlled manner
`
`depending on the relative speeds of the two vehicles, and/or the controlled vehicle and a stationery
`
`object or structure and the distance therebetween.
`
`A fourth subsystem, which may be part of or separate from the third subsystem may
`
`generate one or more codes which are applied to either effect partial and/or complete control of the
`
`steeling mechanism for the vehicle to avoid an obstacle and/or lessen the effect of an accident.
`
`Either or both the third or fourth subsystem may also be operable to control one or more safety
`
`devices by controlling motors, solenoids or valves, to operate a restraining device or devices for
`
`the driver and passeng~r(s) of the vehicle, such as a safety belt tightening means, an air bag
`
`inflation means or other device designed to protect human beings in the vehicle.
`"
`
`4
`
`8
`
`

`
`The second, and/or third and fourth subsystems may also be operable to effect or control
`
`the operations~ of additional warning 1p.eans such as the horn,headlights and/or other warning
`4\
`lights on the vehicle or other wa.ming means which operates to alett, flag or warn the driver of the
`
`approaching or approached vehicle or a pedestrian of the approaching hazardous condition. One or
`
`more of these subsystems may also be operable to generate and transmit one or more codes to be
`
`received and used by the approaching or approached vehicle or a roadside device to effect
`
`additional on-line warning(s) of the hazardous condition, and/or may be recorded on a disc or
`
`RAM (random access memory) for future analysis, if necessary.
`
`In a modified form of the invention, the vehicle waming system may also include a short
`
`wave receiving means to receive code signals from other vehicles and/or short wave transmitters at
`
`the side of or within the road for controlling the visual, audio and/or brake and steering means of
`the vehicle to avoid or~~:'the effects of an accident and/or to maintain the vehicle in-lane and in
`proper operating condition as it travels.
`
`The systems and methods of this invention preferably employ computerized image
`
`;x
`
`I
`
`analyzing techniques of the types disclosed and defined in such patents of mine as 4,969,038 and
`
`4,979,029 and references cited in the file wrappers thereof as well as other more recent patents
`
`and include the use of known artificial intelligence, neural networking and fuzzy logic computing
`
`electronic circuits.
`·Ns~
`
`'>
`
`Accordingly it is a primary object of this invention to provide a new and improved system
`'+'t~were.a \1~,~
`and method for controlling the operation of a..Hiotor vehiele, ~(')at, tra:in or a:irerM't. •
`1\
`Another object is provide a system and method for assisting the driver of ~ :nrotor vehicle~
`rp oi.PQ.~e-d \l.e.\i~~
`-train, boat or ahcrafH.n controlling its operation to avoid an accident or hazardous driving
`f\
`-
`condition.
`
`5
`
`9
`
`

`
`-- --~~ ...........
`Another object is to provide a s?'stem and method ~111ploying _computerized image analySis)
`
`to control or assist the driver of a moto: vehicle in controlling its operation to avoid hazardous
`
`conditions such as collisions with other vehicles, stationery objects or pedestrians.
`
`Another object is to provide a computerized system and method for controlling the speed of
`
`travel of a motor vehicle to lessen the chances of an accident while being driven by a person.
`
`Another object is to provide a system and method employing a television scanning camera
`
`mounted on a vehicle for scanning the field ahead, such as the image of the road ahead of the
`
`vehicle and a computer for analyzing the image signals generated wherein automatic image
`
`intensifying, or infra-red scanning and detection means is utilized to permit scanning operations to
`
`be effected duli:p:g di1ving at night and in low light, snowing or fog conditioris>,
`
`\
`
`Another object is to provide a system and method employing a television camera or other
`
`video scanning means mounted on a moving motor vehicle for scanning, detecting and identifying
`
`obstacles such as other vehicles ahead of such moving vehicle wherein the video image signals are
`
`analyzed to determine distances to such objects.
`
`Another object is to provide a computer controlled safety system for a motor vehicle which
`
`employs a television camera and an auxiliary scanning means to both identify obstacles in the path
`
`of the vehicle and dete1mine distance therefrom on a real time and continuous basis for use in
`
`warning the operator of same and/or in controlling the operation of the vehicle to avoid a collision.
`
`BRIEF DESCRIPTION OF DRAWINGS
`
`The vruious harqware and software elements used to carry out the invention described
`
`herein are illustrated in the f01m of block diagrams, flow chrui:S, and depictions of neural network
`
`6
`
`10
`
`

`
`and fuzzy logic algorithms and structures. The preferred embodiment is illustrated in the following
`
`figures:
`
`Fig. 1 is a block diagram of the overall Motor Vehicle Warning and Control System
`
`·illustrating system sensors, computers, displays, input! output devices and other key elements.
`IL
`•,
`<1..1
`~ is a block diagram of an ~mage ~alysis ~omputer 19 of the type that can be used in
`
`the Vehicle Hazard A voidance System herein of Fig. 1.
`•
`~
`\
`OJ
`Fig. 3 illustrates a neural network of the type useful in the l.lnage ~nalysis ~omputer of
`ll
`1\
`(\
`
`Fig. 4.
`
`3.
`
`E:ig._1 illustrates the structure of a Processing Element (PE) in the neural network of Fig.
`
`Fig. 5 is an alternate embodiment of a neural network image processor useful in the system
`
`of Fig. 1.
`
`d
`-F
`Eig._Q is a ~low 't)iagram illustrating the overall operation of the Motor Vehicle W aming
`"
`1'-
`and Control System of Fig. 1.
`
`1
`~
`~
`E.igJ illustrates typical input signal membership functions for):uzzy \ogic 'iigorithms
`~
`~
`~\.
`
`useful in the Motor Vehicle Warning and Control System of Figure 1.
`
`1
`Q.
`~
`Fig. 8 illustrates typical output signal membership functions for ~uzzy \ogic )\lgorithms
`" "
`"
`
`useful in the Motor Vehicle Warning and Control System of Fig. 1.
`
`~illustrates typical Fuzzy Associative Memory (FAM) maps for the Fuzzy Logic
`
`Algorithms useful in the Motor Vehicle W aming and Control System of Fig. 1.
`.
`v
`$
`Fig. 10 is a Hazard/Object ~tate Xector useful in implementing the Fuzzy Logic Vehicle
`"
`...
`
`W aming and Control Sy~tem.
`
`'
`v
`Fig. 11 is a Hazard Collision Cont:rol'\cector useful in implementing the Fuzzy Logic
`,..
`
`Vehicle Warning and Control System.
`
`7
`
`-
`
`11
`
`

`
`fi).!. 12 is a table of Hazard/Object state vectors indicating possible combinations of hazards
`
`and objects useful in the Fuzzy Associative Memory access system used herein.
`t
`~-
`~
`Fig. 13 is a more detailed \ogic 'ilow 'f<iagram for the analysis of detection signals prior to
`I
`,_
`"
`1\
`~
`.
`accessing 1\uzzy \ogic control structures in the Motor Vehicle Warning and Control System.
`-f o
`"
`"
`I
`Fig. 14 is a more detailed\ogic ~ow ~iagrant for the Fuzzy Associative Memory (PAM)
`"
`..
`'(\..
`
`selection processing.
`
`Fi~. 15 is an example system flow illustrating the operation of the Motor Vehicle Warning
`
`and Control System.
`
`, r::De \a.\ \-eci. "]) esc:r \.fC~ on
`·SYSTEM DESCRIPTION
`
`In Fig. 1 is shown a computelized control system 10 for controlling the operation of a
`
`motor vehicle to prevent or lessen the effects of accidents such as collisions with stationery and/or
`
`moving objects such as other vehicles. The system 10 employs a control computer or
`
`microprocessor 11 mounted on the vehicle and operable to receive and gate digital signals, such as
`
`codes and control signals from various sensors, to one or more specialized computers and from
`
`such computers to a number of servos such as electric motors and lineal actuators or solenoids,
`
`switches and the like, speakers and display drivers to perform either or both the functions of
`
`audibly ancVor visually informing or wm·ning the driver of the vehicle of a hazardous road
`
`condition ahead and/or to effect controlled braking and steering actions of the vehicle.
`
`A RAM 12 and ROM 13 are connected to processor 11 to effect and facilitate its operation.
`
`A television camera(s) 1_6 having a wide angle lens 16L is mounted at the front of the vehicle such
`
`as the front end of the roof, bumper or end of the hood to scan the road ahead of the vehicle at an
`
`angle encompassing the sides of the road and intersecting roads. The analog signal output of
`
`8
`
`12
`
`

`
`camera 16 is digitized in an A/D convertor 18 and passed directly to or through a video
`
`preprocessor 51 to microprocessor 11, to _an image field analyzing computer 19 which is provided,
`
`implemented and programmed using neural networks and artificial intelligence as well as fuzzy
`
`logic algorithms to (a) identify objects on the road ahead such as other vehicles, pedestrians,
`
`~
`
`baniers and dividers, turns in the road, signs and symbols, et~._and generate identification codes,
`A
`and (b) detect distances from such objects by their size (and shape) and provide codes indicating
`
`same for use by a decision computer, 23, which generates coded control signals which are applied
`
`through the computer 11 or are directly passed to various warning and vehicle operating devices
`
`such as a braking computer or drive, 35, which operates a brake servo 33, a steering computer or
`
`drive(s) 39 and 40 which operate ·steering servos 36; a synthetic speech signal generator 27 which
`
`sends trains of indicating and warning digital speech signals to a digital-analog converter 29
`
`connected to a speaker 30; a display driver 31 which drives a (heads-up or dashboard) display 32;
`
`_ _.
`
`a head light controller 41 for flashing the head lights, a warning light control 42 for flashing
`
`external and/or internal warning lights; a hom control43, etc.
`
`A digital speedometer 44 and accelerometer(s) 45 provide information sign.als for use by
`
`the decision computer, 23, in issuing its commands. Accelerometer(s) 45 are connected to control
`
`computer microprocessor 11 through analog-to-digital converter 46. The accelerometer(s) 45 may
`
`pass data continuously to control computer microprocessor 11, or, alternatively, respond to query
`
`signals from said control computer 11. An auxiliary range detection means comprises a range
`
`computer 21 which accepts digital code signals from a radar or lidar computer 14 which interprets
`
`radar and/or laser range signals from respective reflected radiation receiving means on the vehicle.
`
`The image analyzing computer 19 with associated memory 20 may be implemented in
`
`several different ways. Of particular concern is the requirement for high speed image processing
`
`with the capability to detect various hazards in dynamic image fields with changing scenes, moving
`
`9
`
`13
`
`

`
`objects and multiple objects, more than one of which maybe a potential hazard. Requirements for
`
`wide angle vision and the ability to analyze both right and left side image fields also ~xis~. The
`
`imaging system not only detects hazards, but also estimates distance based on image data for input
`
`to the range computer 21 implemented with the associated memory .unit 22.
`
`High speed image processing can be implemented employing known special purpose
`
`computer architectures including various parallel system structures and systems based on neural
`~~-
`networks. ~ 2 shows a high speed parallel processor system embodiment with dedicated
`r~-
`,.
`image processing hardware. The system of~ 2 has a dedicated image data bus 50 for high
`speed image data transfer. The video camera 16 transfers full-frame video picture signal!data to the
`
`.
`
`image bus 50 via analog/digital converter 18 and video preprocessor 51. The video camera 16 is
`
`preferably a CCD array camera generating successive picture frames with individual pixels being
`
`digitized for processing by the video preprocessor 51. The video camera 16 may also be
`
`implemented with other technologies including known image intensifying electron gun and infra(cid:173)
`
`red imaging methods. Multiple cameras may be ~r f~~-nt, side and rear viewing and for
`
`stereo imaging capabilities suitable for generation o:R;f.Dimel)sional image information including
`"
`capabilities for depth perception and placing multiple objects in three dimensional image fields to
`
`further improve hazard detection capabilities.
`
`As shown in Fig. 2, the video preprocessor 51 performs necessary video image frame
`
`management and data manipulation in preparation for image analysis. The preprocessor 51 may
`
`also be used in some embodiments for digital prefiltering and in1age enhancement. Actual image
`
`data can be displayed in real time using video display 55 via analog-to-digital converter 54. The
`
`image display may inclupe highlighting of hazards, special warning linages such as flashing lights,
`
`alpha-numeric messages, distance values, speed indicators and other hazard and safety related
`
`10
`
`14
`
`

`
`l-..
`
`._,
`
`messages. Simulated.as well as actual video displays may also be used to enhance driver
`"
`'Ci3
`recognition of dangerous situations .
`•
`The ~age~alysis ~omputer 19 operates under the control of control processor 56 with
`\"4:\tl.
`fl
`"
`"
`· random-access-memory (RAM) 57 and program and reference data stored in~-only memory
`
`(ROM) 58. The control processor 56 communicates with the motor vehicle warning and control
`
`system micro-processor controller ll through the Bus Interrace Unit 59. Results of the image
`
`analysis are passed in real-time to microprocessor controller 11 for integration with other sensory,
`
`computing, warning and control signals as depicted in Figure 1.
`i
`·i=" ~ •
`c:u
`c....
`The image !\nalysis 'l0omputer 19 of Figure 2 uses high speed dedicated co-processor 53
`k\
`~
`~
`for actual linage analysis under control of the control processor 56. Typical operations performed
`
`using co-processors 53 include multidimensional filtering for operations such as feature extraction
`
`and motion detection. The co-processors 53 are used for multidimensional discrete transforms and
`
`other digital flltering operations used in image analysis. Multiple image memorie~52 with parallel
`
`access to successive image data frames via image bus 50 permit~ concurrent processing with high
`
`speed data access by respective co-processing elements 53. The co-processor elements 53 may be
`
`high speed programmable processors or special purpose hardware processors specifically
`
`constructed for image analysis operations. SIMD (single. instruction, multiple data) architectures
`
`provide high speed operation with multiple identical processing elements under control of a control
`
`unit that broadcasts instructions to all processing elements. The same instruction is executed
`
`simultaneously on different data elements making this approach particularly well suited for matrix
`
`and vector operations commonly employed in image analysis operations. Parallel operations of ·
`
`this type are particularly important with high pixel counts. A 1000 x 1000 pixel image has one
`
`million data points. Tightly coupled Multiple Instruction, Multiple Data (MIMD) architectures also
`
`are used in image processing applications. MIMD systems execute independent but related
`
`11
`
`15
`
`

`
`programs concunently on multiple processing elements. Va1ious anay processor and massively
`
`parallel architectures known to those s]<illed in the art may also be used for real-time image
`
`analysis.
`
`The calculation of the distance of certain recognizable objects from the vehicle is facilitated
`
`by having standard images stored in memory and recalling and comparing such image data with
`
`image data representing the object detected by the vehicle scanning mechanisms. For example,
`
`virtually all automobiles, trucks, and other standard vehicles have known widths. It f2!!?_~-~-~~t
`
`\I -~e dista?ce to;:~ther vehicle can b~--~~-~~-~e.d by _ca1c.ulatin_¥ i~ vj_d_th ~:~~--~~~-~-~-!mage. If a
`CCD camera is used, for example, the width can ascertained in pixels in the image field. The
`
`distance to the vehicle can then be easily calculated using a simple relationship wherein the distance
`
`will be directly proportional to the object image width in pixels. The relative velocities and
`
`accelerations can also be easily calculated from respective first and second derivatives of the image ·
`
`width with respect to time. These image measurements and calculations can be used in addition to
`
`radar/lidar signal measurements or they may be used alone depending on system requirements.
`
`In another embodiment, the image analyzing computer 19 is implemented as a neural
`
`-····
`
`computing network with networked processingel~ments perfonning S\}CCessive computations on
`
`input image structure as shown in~:l~Fe 3 where signal inp-u~ 61 ~e connected to multiple
`processing elements 63, 65 and 67 through the network connections 62, 64 and 66. The
`
`processing elements (PE's) 63, 65 and 67 map input signal vectors to the output decision layer,
`
`performing such tasks as image recognition and image parameter analysis.
`
`A typical neural network processing element known to those skilled in the art is shown in
`
`o.__.
`
`Fig. 4 where input yectors~ (Xl, X2 .... Xn) are connected via weighing elements (Wl,
`
`W2 ..... Wn) to a summing node 70. The output of node 70 is passed through a nonlinear
`
`processing element 72 to produce an output signal, U. Offset or bias inputs can be added to the
`
`12
`
`16
`
`

`
`inputs through weighing circuit Wo. The output signal from summing node 70 is passed through
`
`the nonlinear element 72. The nonlinear function is preferably a continuous, differentiable
`
`function such as a sigmoid which is typically used in neural network processing element nodes.
`
`Neural networks used in the vehicle warning system are trained to recognize roadway hazards
`
`which the vehicle is approaching including automobiles, trucks, and pedestrians. Training
`
`involves providing known inputs to the network resulting in desired output responses.. The
`
`weights are automatically adjusted based on error signal measurements until the desired outputs are
`
`generated. Various leruning algorithms may be applied. Adaptive operation is also possible with
`
`on-line adjustment of network weights to meet imaging requirements. The neural network
`
`embodiment of the image analysis computer 19 provides a highly parallel image processing
`
`structure with rapid, real-time image recognition necessary for the Motor Vehicle Wruning and
`
`Control System. Very Large Scale Integrated (VLSI) Circuit implementation of the neural
`
`processing elements permits low-cost, low-weight implementation. Also, a neural network has
`
`certain reliability advantages important in a safety warning system. Loss of one processing
`
`element does not necessru·ily result in a processing system failure.
`In a alternate embodiment, the neural network computing network of 'Fl~tu's- 3 can be
`"
`implemented using multiple vi.Itual processing elements 73 interconnected via an image data bus 75
`.v~
`,..
`with an image processor 74 as shown in~ 5. Image data presented to the Image Processor 74
`is routed to selected virtual processing elements 73 which implement the neural network computing
`
`.Q•
`
`. .....
`
`functions. The virtual PE' s may be pipelined processors to increase speed and computational
`
`efficiency.
`
`The ~ecision tomputer 23 of-~]'i~re 1 integrates the inputs from the image analysis
`
`.
`
`~
`~
`computer 19, range computer 21, digital accelerometer 45, and the radar or lidar computer 14 to
`'
`generate output warning and contml signals. W ru·ning signals alert the driver of impending
`
`I
`
`13
`
`17
`
`

`
`hazards and, depending on the situation, actual vehicle control signals may be generated to operate
`
`the vehicle in a manner that will avoiq the hazard or minimize the danger to the vehicle and
`
`'
`
`passengers. ~;ntrol signals,iwHl be generated to operate brake servos 33 and steering servos 36.
`--- ...... .
`--
`Manual overrides are provided to ensure driver vehicle control if necessary.
`t
`{
`'··
`...
`..
`- ·--
`A particularly attractive embodiment of the decision computer 23 makes use of fuzzy '&ogic
`ov
`;.
`-
`I\
`Algorithmic structures to implement the automated control and warning signal generation. Fuzzy
`
`..
`
`·-· -----·--------·-·-----------------~---------
`
`'\ogic is particularly well suited to the vehicle control problem wherein it is necessary to deal with a
`
`"' multiplicity of image, motion, and environmental parameters, each of which may extend over
`
`ranges of values and in different combinations which require different responses.
`
`·~~re 6 illustrates a\low ~iagram for implementing a Fuzzy Logic Vehicle Control and
`"
`~
`Warning signal generation system suitable for the decision computer 23. The system of Fig. 6
`receives inputs via the ~ontrol ~omputer~croprocessor 11 of·~~ffi:e 1. Inputs include image
`
`analysis outputs, motion sensor outputs, distance measurements from radar/lidar systems, and
`
`-
`
`environmental parameters which may indicate adverse driving condition.s including rain or ice. The
`
`input signals are analyzed in a preprocessing step for hazardous conditions in the processing block
`
`_.
`
`74. When a hazard is detected, the Fuzzy Associative Memory (FAM) Seetimt block 76 described
`
`in more detail below is activated via decision element 75.
`
`If no hazard is present, the system
`
`_.
`
`continues to analyze scanning signals until a hazardous situation is encountered.
`b
`The Fuzzy Associative Memory (FAM) )}lock 76 also receives a parameter input file from
`k.
`t..:
`the Detection Signal Analysis ~lock 7 4. This file contains necessary information to make control
`'
`. -
`1\
`decision including, for example, hazard location (front, back, left side, right sidy-), hazard distance,
`
`relative velocity, steeriJ:!g angle, braking pressure, weather data, and the presence or a,bsence of
`
`obstructions or objects to the front, rear,\~or to either side of the vehicle.
`
`14
`
`18
`
`

`
`Control signals are derived using FAM' ~ 77, 78, 79 and 80. In practice , a large number
`
`ofF AM's may be used to reflect different possible driving conditions and hazard scenarios. Each
`
`Fuzzy Associative Memory maps input control parameter combinations to appropriate output
`
`· control signals. The output signals are defuzzified in the control signal generator 81 for input to
`t-\•
`the microprocessor controller 11 or Fii:ne 1. This controller in tum generates control signals for
`...
`steering servos, braking servos, and display and warning signals.
`
`The FAM's operate with input signals measuring, for example, distance to the hazard,
`
`relative velocity of the vehicle relative to the hazard and relative acceleration between the vehicle
`£.;~.
`and the hazard. Membership functions for these three variables ~re shown in ~ 7. The
`
`distance variable is classified as being Very Close (VC), Close (C), Mediwn (M), Far (F) or Very
`
`Far (VF). Ov~rlap between membership in the various grades is indicated by the overlapping
`+"18.
`trapezoids of ,..Fi~ttre 7. Certain distances are in more than one membership grade, being, for
`
`example, on the high end of being very close and the low end of being close.
`
`Similarly, the membership functions for relative velocity grades inputs as Very Low (VL),
`
`Low (L), Medium (M), High (H) and Very High (VH) with overlap of membership grades
`
`indicated by the intersection of membership grade trapezoids. Relative acceleration is graded as
`
`being either positive or negative. Deceleration of the vehicle's velocity relative to the hazard is
`
`classified as negative acceleration. Bother positive and negative acceleration are classified as being ·
`
`Low (L), Medium (M) or High (H). Overlapping "fuzzy" membership is indicated with the
`
`overlapping trapezoids, permitting possible membership in multiple grades. For example, a
`
`particular velocity might have a degree of membership in grade "Low" of 0.2 and a degree of
`
`membership in grade "Medium" of 0.6.
`
`Three outputs are generated from the Fuzzy Associative Memory or FAM bank:
`~
`Warning Level; (2) Braking Pressure and (3) Steering Angle. The xuzzy output membership
`"
`
`;
`
`(1)
`
`15
`
`19
`
`

`
`-
`
`~\~
`functions for these signals are shown in._Figtne 8. Three trapezoidal membership functions used
`
`for Braking Pressure: (1) Low Brake (LB), (2) Medium Brake (MB), and (3) High Brake (HB).
`
`Similru:ly, the Steering Angle is graded as Low Angle (L0), Medium Angle (M0), or High Angle
`
`(H0). Steering will be right or left depending on side obstructions, vehicles, or other conditions
`
`as indicated by the detection signal analysis bloc~ 74,__of Fig. 6. The warning level is indicated as
`
`being green, yellow, or red, depending on the danger level presented by the detected hazard.
`
`Continuous or discrete warnings can be generated on the output. Possibilities include visual light
`
`indicators of different intensity, continuously variable audible alarms, continuously variable color
`
`indicators, or other arrangements with possible combinations of visib~~-~d ~udible al?.::ffi1S.
`
`-
`
`Warning indicators can be combined with actual video displays of vehicle situations including .
`
`hazards and nearby objects. The synthetic speech signal generator.., 27-._.of Fig. 1 may be used to
`
`generate synthetic speech signals defining spoken alru-m wrunings.
`fi=', •
`~ 9 depicts typical PAM's for generating the output control signals from the input
`signals. Each F AM is segmented in six sections depending on the membership grade of the
`
`acceleration variable. Interpretation of the FAM logic rules is straight forward. For example, if the
`
`relative acceleration is High Positive (HP), the distance is Close (C), and the relative velocity is
`
`Medium (M), then the rule stated in the FAM requires grading the warning as Red (R), the Brakes
`
`as Medium (MB), and the steering as Small ~,11-~~ef As a logic statement or premise, this
`
`J\
`
`becomes:
`
`If Acceleration is High Positive (HP), Distance is Close (C), and Velocity is Medium (M),
`then Wru·ning equals Red (R), Braking equals Medium (M) and Steering Angle equals
`Smnll Anglo (S0).
`
`As another example:
`
`If Acceleration is Low Negative (LN), Distance is Medium (M) and Velocity is Very High
`(VH), then Warning equals Red, Braking equals Medium (MB), and Steering Angle equals
`Small Angle (S0).
`
`16
`
`20
`
`

`
`Each premise has multiple control variables, each with possibly different degrees of
`
`membership. Using}uzzy }ogic principl~s, the minimum of the truth expressio

This document is available on Docket Alarm but you must sign up to view it.


Or .

Accessing this document will incur an additional charge of $.

After purchase, you can access this document again without charge.

Accept $ Charge
throbber

Still Working On It

This document is taking longer than usual to download. This can happen if we need to contact the court directly to obtain the document and their servers are running slowly.

Give it another minute or two to complete, and then try the refresh button.

throbber

A few More Minutes ... Still Working

It can take up to 5 minutes for us to download a document if the court servers are running slowly.

Thank you for your continued patience.

This document could not be displayed.

We could not find this document within its docket. Please go back to the docket page and check the link. If that does not work, go back to the docket and refresh it to pull the newest information.

Your account does not support viewing this document.

You need a Paid Account to view this document. Click here to change your account type.

Your account does not support viewing this document.

Set your membership status to view this document.

With a Docket Alarm membership, you'll get a whole lot more, including:

  • Up-to-date information for this case.
  • Email alerts whenever there is an update.
  • Full text search for other cases.
  • Get email alerts whenever a new case matches your search.

Become a Member

One Moment Please

The filing “” is large (MB) and is being downloaded.

Please refresh this page in a few minutes to see if the filing has been downloaded. The filing will also be emailed to you when the download completes.

Your document is on its way!

If you do not receive the document in five minutes, contact support at support@docketalarm.com.

Sealed Document

We are unable to display this document, it may be under a court ordered seal.

If you have proper credentials to access the file, you may proceed directly to the court's system using your government issued username and password.


Access Government Site

We are redirecting you
to a mobile optimized page.





Document Unreadable or Corrupt

Refresh this Document
Go to the Docket

We are unable to display this document.

Refresh this Document
Go to the Docket