VALEO EXHIBIT 1006
`Valeo v. Magna
`IPR2015-____
`
`(cid:57)(cid:36)(cid:47)(cid:40)(cid:50)(cid:3)(cid:40)(cid:59)(cid:17)(cid:3)(cid:20)(cid:19)(cid:19)(cid:25)(cid:66)(cid:19)(cid:19)(cid:20)
`
`

`
`US 6,553,130 B1
`Page 2
`
`U.S. PATENT DOCUMENTS
`
`Lee, “Fuzzy Logic in Control Systems: Fuzzy Logic Con-
`troller, Part II,” IEEE Transaction on Systems, Man, and
`Cybernetics, pp. 419-435 (vol. 20, No. 2, Mar./Apr. 1990).
`Lippmann, “An Introduction to Computing with Neural
`Nets,” IEEE ASSP, pp. 4-22 (Apr. 1987).
`Lippmann, “Pattern Classification Using Neural Networks,”
`IEEE Communications, pp. 47-50, 59-64 (Nov. 1989).
`Lisboa, ed., Neural Networks—CurrentApplications, Chap-
`ter 1, pp. 1-34, Chapter 2, pp. 35-48, Chapter 7, pp.
`123-147 (Chapman & Hall 1992).
`Lupo, “Defense Applications of Neural Networks,” IEEE
`Communications, pp. 82-88 (Nov. 1989).
`Maresca et al., “Parallel Architectures for Vision,” Proceed-
`ings of the IEEE, pp. 970-981 (vol. 76, No. 8, Aug. 1988).
`Nijhuis et al., “Evaluation of Fuzzy and Neural Vehicle
`Control,” Institution of Electrical Engineers, pp. 447-452
`(1992).
`Pearson, ed.,Image Processing, Chapter 8, pp. 141-155, and
`Chapter 10, pp. 169-189 (McGraw-Hill 1991).
`Psaltis et al., “Optoelectronic Implementations of Neural
`Networks,” IEEE Communications, pp. 37-40, 71 (Nov.
`1989).
`Roth, “Neural Networks for Extraction of Weak Targets in
`High Clutter Environments,” IEEE Transactions on Sys-
`tems, Man, and Cybernetics, pp. 1210-1217 (Sep./Oct.
`1989).
`Schwartz et al., “Fuzzy Logic Flowers in Japan,” IEEE
`Spectrum, pp. 32-35 (Jul. 1992).
`Soucek et al., Neural and Massively Parallel Computers,
`Chapter 12, pp. 245-276 (John Wiley & Sons 1988).
`Suaya et al., eds., VLSI and Parallel Computation, Chapter
`1, pp. 1-84, and Chapter 5, pp. 390-415 (Morgan Kaufmann
`1990).
`Teuber, Digital Image Processing, Chapter 1, pp. 1-30,
`Chapter 2, pp. 31-70, and Appendix D, pp. 254-255 (Pren-
`tice Hall 1993).
`Wasserman, Neural Computing: Theory and Practice, Chap-
`ter 1, pp. 11-26, Chapter 2, pp. 27-42, Chapter 3, pp. 43-59,
`and Chapter 9, pp. 151-166 (Van Nostrand Reinhold 1989).
`Yuhas et al., “Integration of Acoustic and Visual Speech
`Signals Using Neural Networks,” IEEE Spectrum, pp. 65-71
`(Nov. 1989).
`Shekhar et al., Design and Validation of Head up Displays
`for Navigation in IVHS, VNIS’9Z, Oct. 1991, pp. 537-542.
`“NHTSA IVHS Plan,” National Highway Safety Adminis-
`tration U.S. Department of Transportation, Jun. 12, 1992.
`Bosacchi et al, Fuzzy Logic Technology & the Intelligent
`Highway System (IHS); IEEE, 1993, pp. 65-70.
`Rock et al., “Intelligent Road Transit: The Next Generation,”
`AI EXPERT, Apr. 1994, pp. 16-24 (not prior art).
`
`* cited by examiner
`
`(cid:57)(cid:36)(cid:47)(cid:40)(cid:50)(cid:3)(cid:40)(cid:59)(cid:17)(cid:3)(cid:20)(cid:19)(cid:19)(cid:25)(cid:66)(cid:19)(cid:19)(cid:21)
`VALEO EX. 1006_002
`
`2/1990 Terzian ..................... .. 382/103
`4,901,362 A *
`5/1990 Beggs et al.
`..... ..
`.. 340/904
`4,926,170 A
`6/1990 Kakinami et al.
`........ .. 180/169
`4,931,937 A *
`6/1990 Lemelson ........ ..
`340/439
`4,933,852 A
`11/1990 Lemelson
`358/107
`4,969,038 A
`12/1990 Lemelson ........ ..
`358/93
`4,979,029 A
`5/1991 Yasunobu et al.
`.. 395/905
`5,018,689 A
`8/1991 Maekawa et al.
`............ .. 356/1
`5,039,217 A
`1/1992 Kurami et al.
`....... .. 364/424.02
`5,081,585 A
`2/1992 Shyu . . . . . . . . . . . . .
`. . . . .. 340/904
`5,091,726 A
`6/1992 Beggs et al.
`.. 340/904
`5,122,796 A
`9/1992 Zechnall
`.......... ..
`340/905
`5,146,219 A
`11/1992 Mayeaux et al.
`.. 340/937
`5,161,107 A
`11/1992 Summer .......... ..
`340/905
`5,164,904 A
`1/1993 Kajiwara
`340/903
`5,177,462 A
`1/1993 Hancock ...... ..
`340/961
`5,179,377 A
`2/1993 Adachi et al.
`............ .. 395/905
`5,189,619 A
`3/1993 Kakinami et al.
`........ .. 180/169
`5,197,562 A *
`7/1993 Kakinami et al.
`.. 180/169
`5,230,400 A
`9/1993 Taylor ...................... .. 340/903
`5,249,157 A
`1/1994 Bottesch
`. 364/424.05
`5,276,620 A
`
`.............. .. 364/461
`1/1994 Iizuka et al.
`5,278,764 A
`3/1994 Tsai
`......................... .. 340/468
`5,298,882 A
`4/1994 Maekawa
`340/903
`5,304,980 A
`4/1994 Saneyoshi
`180/167
`5,307,136 A
`5/1994 Shaw et al.
`.. 180/169
`5,314,037 A
`7/1994 Kohsaka
`340/459
`5,327,117 A
`.
`7/1994 Butsuen et al.
`.. 180/169
`5,332,057 A
`8/1994 Abst et al.
`....... ..
`.. 340/903
`5,339,075 A
`8/1994 O’Brien et al.
`.
`367/96
`5,341,344 A
`10/1994 Davidian ..... ..
`180/169
`5,357,438 A
`11/1994 Broxmeyer
`.... .. 340/903
`5,369,591 A
`8/1996 Ishikawa ........... .. 364/424.032
`5,545,960 A *
`OTHER PUBLICATIONS
`
`
`
`.
`
`
`
`
`
`Alspector, “Neural-Style Microsystems that Learn,” IEEE
`Communications, pp. 29-36 (Nov. 1989).
`Casasent, “Optics and Neural Nets,” Chapter 16 in Carpen-
`ter et al., eds., Neural Networks for Vision and Image
`Processing, pp. 437-48, (MIT Press 1992).
`Cox, “Fuzzy Fundamentals,” IEEE Spectrum, pp. 58-61
`(Oct. 1992).
`Hammerstrom, “Neural Networks at Work,” IEEE Spec-
`trum, pp. 26-32 (Jun. 1993).
`Hush et al., “Progress in Supervised Neural Networks,”
`IEEE Signal Processing, pp. 8-39 (Jan. 1993).
`Jurgen, “Smart Cars and Highways Go Global,” IEEE
`Spectrum, pp. 26-36 (May 1991).
`Kittler et al., eds., Image Processing System Architectures,
`Chapter 4, pp. 49-81, and Chapter 5, pp. 85-101 (John
`Wiley & Sons 1985).
`Kosko et al., “Fuzzy Logic,” ScientificAmerican, pp. 76-81
`(Jul. 1993).
`
`

`
`U.S. Patent
`
`Apr. 22, 2003
`
`Sheet 1 of 13
`
`US 6,553,130 B1
`
`12
`
`15 /10
`14
`13
`M -M
`3
`g
`
`oncomp
`
`1
`
`15L 17 _ 18
`F M I
`16
`51
`
`33
`
`35
`Q
`
`19
`IMAGE ANALYZING
`" COMPUTER
`
`"'
`
`20
`
`DECISION
`
`31
`WSPLAY DRNER
`
`32
`
`83
`
`82
`
`'“*E"F*‘°E
`
`(cid:57)(cid:36)(cid:47)(cid:40)(cid:50)(cid:3)(cid:40)(cid:59)(cid:17)(cid:3)(cid:20)(cid:19)(cid:19)(cid:25)(cid:66)(cid:19)(cid:19)(cid:22)
`VALEO EX. 1006_003
`
`
`
`34
`
`3
`
`MAN
`OVER
`
`STEERING 17
`SERVO(S) E {Q
`41
`40
`HEAD LIGHT
`°°”TR°'-
`42
`WARNING LIGHT
`CONTROL
`
`39
`
`mpg
`
`_
`
`21
`22
`RANGE M
`E -"
`8
`24
`23
`3
`E --M
`8
`25
`26
`Q
`SPEECH
`E
`_ RECOGNITION _
`E
`COMPUTER
`3
`25A
`25”
`5
`MICROPHONE [
`3
`27
`28
`43
`8 --w
`c«'>*»%“a“i»
`4 Pg
`GEN COMPUTER
`O
`29
`30
`44 4
`DIGITAL
`SPEEDOMETER
`45
`ACCELEROMETER(S) II ND :1
`
`46
`
`SHORT WAVE M
`
`RADIO
`
`34
`
`35
`
`11
`
`TRANSMITT H
`
`86
`
`FIG. 1
`
`

`
`U.S. Patent
`
`Apr. 22, 2003
`
`Sheet 2 of 13
`
`US 6,553,130 B1
`
`17
`
`13
`
`1e _
`'
`
`/19
`
`51
`
`52
`
`53
`
`54
`
`55
`
`°°"’”°°ESS°“
`
`IMAGE BUS
`
`50
`
`57
`
`56
`
`58
`
`RAM
`
`CONTROL
`
`PROCESSOR W
`
`59
`
`BUS INTERFACE
`
`CONTROLLER
`
`MICRO PROCESSOR
`
`FIG. 2
`
`(cid:57)(cid:36)(cid:47)(cid:40)(cid:50)(cid:3)(cid:40)(cid:59)(cid:17)(cid:3)(cid:20)(cid:19)(cid:19)(cid:25)(cid:66)(cid:19)(cid:19)(cid:23)
`VALEO EX. 1006_004
`
`

`
`U.S. Patent
`
`Apr. 22, 2003
`
`Sheet 3 of 13
`
`US 6,553,130 B1
`
`INPUTS
`
`FIRST HIDDEN
`LAYER
`
`SECOND HIDDEN
`LAYER
`
`OUTPUT
`LAYER
`
`
`:»e« Aw“;
`V,«.‘%@£,“«Agr
`
`
`
`;\ 1
`9/
`
`V K
`2
`
`
`
`
`
`
`
`(cid:57)(cid:36)(cid:47)(cid:40)(cid:50)(cid:3)(cid:40)(cid:59)(cid:17)(cid:3)(cid:20)(cid:19)(cid:19)(cid:25)(cid:66)(cid:19)(cid:19)(cid:24)
`VALEO EX. 1006_005
`
`

`
`U.S. Patent
`
`Apr. 22, 2003
`
`Sheet 4 of 13
`
`US 6,553,130 B1
`
`IMAGE DATA
`
`74
`
`
`IMAGE
`PROCESSOR
`
`
`
`
`73
`
`
`VIRTUAL
`PE
`
`VIRTUAL
`PE
`
`VIRTUAL
`PE
`
`IMAGE DATA BUS
`
`75
`
`VIRTUAL
`PE
`
`
`
`FIG. 5
`
`(cid:57)(cid:36)(cid:47)(cid:40)(cid:50)(cid:3)(cid:40)(cid:59)(cid:17)(cid:3)(cid:20)(cid:19)(cid:19)(cid:25)(cid:66)(cid:19)(cid:19)(cid:25)
`VALEO EX. 1006_006
`
`

`
`U.S. Patent
`
`Apr. 22, 2003
`
`Sheet 5 of 13
`
`US 6,553,130 B1
`
`MOTION
`
`WEATHER
`
`OVERRIDE
`
`VEHICLE
`
`74
`
`DETECTION SIGNAL ANALYSIS
`
`CONTROL SIGNAL GENERATOR
`DEFUZZIFIER
`
`BRAKING
`
`STEERING SERVO
`
`
`
`
`MICROPROCESSOR CONTROLLER
`
`BRAKING SERVO
`
`DISPLAYS
`
`
`
`FIG. 6
`
`(cid:57)(cid:36)(cid:47)(cid:40)(cid:50)(cid:3)(cid:40)(cid:59)(cid:17)(cid:3)(cid:20)(cid:19)(cid:19)(cid:25)(cid:66)(cid:19)(cid:19)(cid:26)
`VALEO EX. 1006_007
`
`

`
`U.S. Patent
`
`Apr. 22, 2003
`
`Sheet 6 of 13
`
`US 6,553,130 B1
`
`VERY
`&
`c251CLOSE
`
`CLOSE
`
`MEDIUM
`
`FAR
`
`VERY FAR
`
`
`
`DISTANCE
`
`VERYLOW LOW
`
`MEDIUM
`
`HIGH
`
`VERY HIGH
`
`5c
`
`a 2L
`
`IJ
`2
`
`2%
`
`Il
`
`.IJ
`an
`
`VELOCITY
`
`2|
`
`.lJ
`2
`
`
`2
`
`EI(
`
`DEU
`
`Jm 2L
`
`IJ
`
`ACCELERATION
`
`FIG. 7
`
`(cid:57)(cid:36)(cid:47)(cid:40)(cid:50)(cid:3)(cid:40)(cid:59)(cid:17)(cid:3)(cid:20)(cid:19)(cid:19)(cid:25)(cid:66)(cid:19)(cid:19)(cid:27)
`VALEO EX. 1006_008
`
`

`
`U.S. Patent
`
`Apr. 22, 2003
`
`Sheet 7 of 13
`
`US 6,553,130 B1
`
`D.
`
`5 1
`
`GREEN
`
`YELLOW
`
`RED
`
`
`
`WARNING LEVEL
`
`LOW BRAKE
`
`MEDIUM BRAKE
`
`HIGH BRAKE
`
`
`
`BRAKING PRESSURE
`
`LOW G
`
`MEDIUM 9
`
`HIGH 9
`
`
`
`STEERING ANGLE
`
`FIG. 8
`
`(cid:57)(cid:36)(cid:47)(cid:40)(cid:50)(cid:3)(cid:40)(cid:59)(cid:17)(cid:3)(cid:20)(cid:19)(cid:19)(cid:25)(cid:66)(cid:19)(cid:19)(cid:28)
`VALEO EX. 1006_009
`
`EI
`
`n 2U
`
`J 2
`
`E 9
`
`,3,
`
`EI
`
`n 2U
`
`J 2
`
`E 5
`
`,5,
`
`EI
`
`n 2L
`
`u 2
`
`

`
`U.S. Patent
`
`2
`
`
`
`
`
`
`
`.4.as.uzo_»<mm._m_oo<Suzo:<...m_.m_oo<M.az.m._.>az-B.>EEEu.>az.m._.>az.m...>§az.mz.oE&z.m:.>
`
`3$331.:2,mEIIIHW.El!
`
`324u2o:<mm:moo<.5uzo:<mw._moo<
`
`mEEEa,..._.:ElWg3,e:_EE§
`
`E§¢_.,_EE@EEEaz.§-._EI
`
`as:EBEE
`
`
`
`«W,EfifiazézaEl
`
`M2:uzoE_Em._moo<22uzo:<mm._moo<mEEassfigfi
`
`(cid:57)(cid:36)(cid:47)(cid:40)(cid:50)(cid:3)(cid:40)(cid:59)(cid:17)(cid:3)(cid:20)(cid:19)(cid:19)(cid:25)(cid:66)(cid:19)(cid:20)(cid:19)
`
`VALEO EX. 1006 010
`
`§._.m_._.mas_-m.._-m&m-m:-maz-m_2-m&z-m:.>
`
`
`
`
`n.m-m_._...._az.m:-mQz-m2-maz.m4.>&z.mz.>
`
`
`
`
`a_._.m.+m&:-m_._-mEam¢s_-..._&z,ms.-m0>
`
`
`
`az.m._-mQZ.m...>az.mz.>&z-mz-uez.mz-ou_
`
`
`
`
`>HIE
`&s_.m:.m.u.2.m.._.m &_2.m:.mem.m:.mam.ms_.:&m.ms_.mIllll
`
`
`
`
`
`
`
`am.m:-m&z-m_2-m.uz.3.>
`ez.m._.>ez.mz-......
`
`az-mz.> QZ.m._.> 3.2EAH
`
`IIIIII
`HI
`II
`
`0
`
`&z-mz-u.
`mz.u.>
`.>O>
`zsllall an
`
`
`
`

`
`U.S. Patent
`
`Apr. 22, 2003
`
`Sheet 9 of 13
`
`US 6,553,130 B1
`
`
`
`
`
`82
`
`
`
`83
`
`
`
`FIG. 10
`
`FIG. 11
`
`(cid:57)(cid:36)(cid:47)(cid:40)(cid:50)(cid:3)(cid:40)(cid:59)(cid:17)(cid:3)(cid:20)(cid:19)(cid:19)(cid:25)(cid:66)(cid:19)(cid:20)(cid:20)
`VALEO EX. 1006_011
`
`

`
`U.S. Patent
`
`Apr. 22, 2003
`
`Sheet 10 of 13
`
`US 6,553,130 B1
`
`Ov-Q1-O1-CF
`
`O01--I-CO1-FOv-O1-
`
`OOOO -r-v-POO I-r-O01--I-CF
`
`FIG.12
`
`0000'-v-OOOOOOOOv-v-u—w O<3OOv-1-OOOOV-1-GOP
`
`1-v--I-1-v-v-OOOOOOOOOOOO I-1-V-v-v-FOOOOOOPPP
`
`v-v-1-1-v-1-OCOCOCCQOOOO OOCCCCV-V-V-v-V-1-1-1-F
`
`©QQC$C1'-1-1-<P-1-\-v-v-v-v-v-v-
`
`v-1-1--1-v-1-v-1--v-v—1-1-1-1-1--
`
`$$R$$8%86%8838$8
`
`O1-Ov-O--Or
`
`Or-O1-Ow-OF
`
`OOv-v-O07-V'-
`
`©3331-w-v-1-333$!-v-1-F©®1-v-QOCO1--I-1-F
`
`O1-v-1-I-V-v-V-1-'OOOOOOOOV-v-v-v-OOOOOOOO
`
`HROFOBOLOR
`
`OQOOOOQOQ1-1-I-v-v-v-V-v-V-F1-FCOCCCOQC
`
`HBHL OOOOOOOOOOOOOOOOOOOOO1-1-v-FF‘I-F-F
`
`IJJ
`_J
`
`2‘
`
`:3(D
`O
`
`D E
`
`DOOOOOOOOOOOOODOOOOOOOOOOCOOO
`
`“NC0‘?'0<°'*°°<7’3599393'l539-’8K:a'8§38N33¢9:59':8§
`
`HF
`
`»“—‘
`
`5U) VECTOR
`
`(cid:57)(cid:36)(cid:47)(cid:40)(cid:50)(cid:3)(cid:40)(cid:59)(cid:17)(cid:3)(cid:20)(cid:19)(cid:19)(cid:25)(cid:66)(cid:19)(cid:20)(cid:21)
`VALEO EX. 1006_012
`
`

`
`U.S. Patent
`
`Apr. 22, 2003
`
`Sheet 11 of 13
`
`US 6,553,130 B1
`
`SCANNER INPUT
`
`DATA FILE
`
`74
`
`/
`
`84
`
`37
`
`“ <9 85
`Y
`H =1
`
`86
`
`N ® 88
`
`v
`
`39
`
`-113-
`
`1Noo
`
`6
`
`man
`
`ea‘
`
`KEEI
`
`MODIFY HAZARD
`STATE VE°T°R
`
`104
`
`90
`
`coon
`
`9
`
`coto
`
`Il'z<9
`
`H=1
`
`COLLISION VECTOR
`
`N <9 9‘
`
`Y
`
`95
`
`IIEEI
`
`96
`
`
`I'-6i
`
`
` MULTIPLE
`HAZARDS
`”
`
`mag-
`
`82
`
`ucanmmamanm
`
`FIG. 13
`
`(cid:57)(cid:36)(cid:47)(cid:40)(cid:50)(cid:3)(cid:40)(cid:59)(cid:17)(cid:3)(cid:20)(cid:19)(cid:19)(cid:25)(cid:66)(cid:19)(cid:20)(cid:22)
`VALEO EX. 1006_013
`
`

`
`U.S. Patent
`
`Apr. 22, 2003
`
`Sheet 12 of 13
`
`US 6,553,130 B1
`
`108
`
`109
`
`110
`
`111
`
`112
`
`113
`
`COLLISION
`
`STATE VECTOR
`
`INPUT DATA
`
`/ HLE
`
`FUZZY
`
`DISTANCE
`
`MEMBERSHIP
`
`FUZZY
`
`VELOCITY
`
`MEMBERSHIP
`
`FUZZY
`
`ACCELERATION
`
`MEMBERSHIP
`
`HAZARD/OBJECT
`
`STATE VECTOR
`
`ADDRESS
`
`TRANSLATION
`
`FAM OUTPUT
`
`ADDRESS
`
`FIG. 14
`
`/76
`
`(cid:57)(cid:36)(cid:47)(cid:40)(cid:50)(cid:3)(cid:40)(cid:59)(cid:17)(cid:3)(cid:20)(cid:19)(cid:19)(cid:25)(cid:66)(cid:19)(cid:20)(cid:23)
`VALEO EX. 1006_014
`
`

`
`U.S. Patent
`
`Apr. 22, 2003
`
`Sheet 13 of 13
`
`US 6,553,130 B1
`
`
`
`@ OBJECT FRONT
`
`Y
`
`HAZARD
`
`vEcToR
`
`Y MULTIPLE
`HAZARD
`
`@ OBJECT RIGHT
`
`INPUT DATA FILE
`
`HAZARD FRONT @ N
`
`Y
`
`N
`
`HAZARD LEFT @
`
`N
`
`HAZARD RIGHT ®
`
`N
`
`Y
`
`Y
`
`Y
`
`
`
`HAZARD/OBJECT
`VECTOR
`
`an
`COLLISION
`VECTOR
`
`DISTANCE FUZZY
`MEMBERSHIP
`
`VELOCITY FUZZY
`MEMBERSHIP
`
`ACCELERATION
`FUZY MEMBERSHIP
`
`
`ACCESS FUZZY
`ASSOCIATIVE
`
`MEMORY
`
`
`
`(FAM)
`
`
`
`GENERATE OUTPUT
`EXPERT CONTROL
`
`SIGNALS
`
`WARNING BRAKE STEERING
`
`FIG. 15
`
`(cid:57)(cid:36)(cid:47)(cid:40)(cid:50)(cid:3)(cid:40)(cid:59)(cid:17)(cid:3)(cid:20)(cid:19)(cid:19)(cid:25)(cid:66)(cid:19)(cid:20)(cid:24)
`VALEO EX. 1006_015
`
`

`
`US 6,553,130 B1
`
`1
`MOTOR VEHICLE WARNING AND
`CONTROL SYSTEM AND METHOD
`
`CROSS-REFERENCE TO RELATED
`APPLICATION
`
`This application is a continuation of Ser. No. 08/105,304,
`filed Aug. 11, 1993, abandoned.
`FIELD OF THE INVENTION
`
`This invention relates to a system and method for oper-
`ating a motor vehicle, such as an automobile, truck, aircraft
`or other vehicle, wherein a computer or computerized sys-
`tem is employed to assist and/or supplement the driver in the
`movement of the vehicle along a path of travel, such as a
`street or roadway and may be used to avoid obstacles and
`accidents.
`
`BACKGROUND OF THE INVENTION
`
`A major cause of human suffering is automobile acci-
`dents. Approximately 49,000 people die in traffic accidents
`each year in the United States, and another three million are
`injured. The costs of death and injury accidents are stagger-
`ing. According to the United States National Highway
`Traffic Safety Administration, crash damage and medical
`bills total $137 billion a year.
`Automobile designers offer many safety features, includ-
`ing passenger restraints,
`improved braking systems, and
`body designs, intended to better protect automobile crash
`victims. But very little has been done in the area of auto-
`matic vehicle control systems based on modern electronics,
`computer systems, and advanced real-time software. This is
`true despite rapidly increasing capabilities in these technolo-
`gies and pervasive application in many other areas
`including,
`for example the business, entertainment, and
`medical fields. Vehicle guidance and control technology has,
`of course, been applied with great success in military
`defense systems, avionics systems and space exploration
`systems. But, this technology is costly and has not been
`commercialized.
`
`The opportunity exists today to develop cost effective,
`commercial automated vehicle control systems. New
`advances in low-cost hardware and software technology
`make implementation feasible. High-speed, parallel com-
`puter architectures, specialized image-processing
`equipment, and advanced special computers such as math
`coprocessors are available. Advanced expert system imple-
`mentations based on concepts such as fuzzy logic and neural
`networks, and new, improved scanning systems for sensing
`environments around moving vehicles make it very timely,
`indeed, to pursue new approaches.
`Work on these problems has begun. Intelligent vehicle/
`highway systems are being investigated with traffic control
`systems intended to minimize congestion. Vehicle location
`systems such as GPS (Global Positioning System) and route
`guidance systems are also being pursued. Certain systems
`for automated vehicle control have been proposed, including
`systems that scan the roadway directly ahead of a vehicle
`using radar/lidar or television and attempt to warn a driver
`of impending danger. Fuzzy logic expert systems for con-
`trolling vehicle speed (braking and throttle) based on scan-
`ning the roadway ahead of a vehicle have been described.
`Road tracking with electronic vehicle guidance is being
`pursued. Fuzzy logic has been applied to braking systems in
`subway and train systems.
`to
`they fail
`While these developments are important,
`protect vehicles from many types of collisions or minimize
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`2
`
`the damage therefrom. More particularly, such systems fail
`to exercise simultaneous, coordinated control over vehicle
`steering and speed, fail to take full advantage of identifica-
`tion of different obstacle or hazard types using standard
`stored models of production vehicles and other commonly
`encountered roadway objects, fail to deal effectively with
`objects and hazards located simultaneously on different
`sides of the vehicle, and fail to capitalize fully on modern
`expert system decision and control
`technology, such as
`represented by fuzzy logic and neural network methods, to
`deal with more complex hazardous situations.
`SUMMARY OF THE INVENTION
`
`In a preferred form of the invention, a video scanning
`system, such as a television camera and/or one or more laser
`scanners mounted on the vehicle scan the road in front of the
`vehicle and generate image information which is computer
`analyzed per se or in combination with a range sensing
`system to warn the driver of hazardous conditions during
`driving by operating a display, such as a heads—up display,
`and/or a synthetic speech generating means which generates
`sounds or words of speech to verbally indicate such road
`conditions ahead of the vehicle.
`
`The preferred form of the invention provides audible
`and/or visual display means to cooperate in indicating to the
`driver of a motor vehicle both normal and hazardous road
`conditions ahead as well as driving variables such as dis-
`tances to stationary objects, and other vehicles;
`the
`identification, direction of travel and speed of such other
`vehicles, and the identification of and distances to stationary
`or slowly moving objects such as barriers, center islands,
`pedestrians, parked cars poles, sharp turns in the road and
`other conditions. In addition, the image analyzing computer
`of the vehicle may be operated to scan and decode coded
`and/or character containing signs or signals generated by
`indicia or code generating other devices within or at the side
`of the road and indicating select road and driving conditions
`ahead.
`
`The computer is operable to analyze video and/or other
`forms of image information generated as the vehicle travels
`to identify obstacles ahead of the vehicle and, in certain
`instances, quantify the distance between the vehicle con-
`taining same on the basis of the size of the identified vehicle
`or object and/or by processing received pulse-echo signals.
`Using such identifying information and comparing it with
`information on the shapes and sizes of various objects such
`as rear and front profiles of all production vehicles and the
`like and their relative sizes or select dimensions thereof,
`indications of distances to such objects may be computed
`and indicated as further codes. When the closing distance
`becomes hazardous, select vehicle subsystems may be auto-
`matically controlled by the computer as it continues to
`analyze image signals generated by the television camera. A
`first subsystem generates a first select code or codes which
`controls an electronic display, such as a heads-up display to
`cause it to display a warning indication, such as one or more
`flashing red light portions of the display or other lighted
`effect. For example, the display may project on the wind-
`shield or dashboard such information as images of the
`controlled vehicle and other vehicles in and adjacent its path
`of travel and relative distances thereto as well as groups of
`characters defining same, colored and flashing warning
`lights and the like for pre-warning and warning purposes. A
`second subsystem generates a code or series of codes which
`control a sound generating means which generates a select
`sound such as a horn, buzzing sound and/or select synthetic
`speech warning of the hazardous condition detected and, in
`
`(cid:57)(cid:36)(cid:47)(cid:40)(cid:50)(cid:3)(cid:40)(cid:59)(cid:17)(cid:3)(cid:20)(cid:19)(cid:19)(cid:25)(cid:66)(cid:19)(cid:20)(cid:25)
`VALEO EX. 1006_016
`
`

`
`US 6,553,130 B1
`
`3
`certain instances, generating sounds of select words of
`speech which may warn of same and/or suggest corrective
`action (s) by the vehicle operator or driver to avoid an
`accident.
`
`A third subsystem comes on-line and generates one or
`more codes which are applied to at least partly effect a
`corrective action such as by pulsing one or more motors or
`solenoids to apply the brakes of the vehicle to cause it to
`slow down. If necessary to avoid or lessen the effects of an
`accident, the third subsystem stops the forward travel of the
`vehicle in a controlled manner depending on the relative
`speeds of the two vehicles, and/or the controlled vehicle and
`a stationery object or structure and the distance therebe-
`tween.
`
`A fourth subsystem, which may be part of or separate
`from the third subsystem may generate one or more codes
`which are applied to either effect partial and/or complete
`control of the steering mechanism for the vehicle to avoid an
`obstacle and/or lessen the effect of an accident. Either or
`both the third or fourth subsystem may also be operable to
`control one or more safety devices by controlling motors,
`solenoids or valves,
`to operate a restraining device or
`devices for the driver and passenger(s) of the vehicle, such
`as a safety belt tightening means, an air bag inflation means
`or other device designed to protect human beings in the
`vehicle.
`
`The second, and/or third and fourth subsystems may also
`be operable to effect or control the operation of additional
`warning means such as the horn, headlights and/or other
`warning lights on the vehicle or other warning means which
`operates to alert, flag or warn the driver of the approaching
`or approached vehicle or a pedestrian of the approaching
`hazardous condition. One or more of these subsystems may
`also be operable to generate and transmit one or more codes
`to be received and used by the approaching or approached
`vehicle or a roadside device to effect additional on-line
`
`warning(s) of the hazardous condition, and/or may be
`recorded on a disc or RAM (random access memory) for
`future analysis, if necessary.
`In a modified form of the invention, the vehicle warning
`system may also include a short wave receiving means to
`receive code signals from other vehicles and/or short wave
`transmitters at the side of or within the road for controlling
`the visual, audio and/or brake and steering means of the
`vehicle to avoid or lessen the effects of an accident and/or to
`maintain the vehicle in-lane and in proper operating condi-
`tion as it travels.
`
`The systems and methods of this invention preferably
`employ computerized image analyzing techniques of the
`types disclosed and defined in such patents of mine as U.S.
`Pat. Nos. 4,969,038 and 4,979,029 and references cited in
`the file wrappers thereof as well as other more recent patents
`and include the use of known artificial intelligence, neural
`networking and fuzzy logic computing electronic circuits.
`While the invention is described herein principally in
`connection with an automobile on a roadway, it may be used
`in connection with controlling any powered vehicle, includ-
`ing a motor vehicle, a boat, a train, or an aircraft.
`Accordingly it is a primary object of this invention to
`provide a new and improved system and method for con-
`trolling the operation of a powered vehicle.
`Another object
`is provide a system and method for
`assisting the driver of a powered vehicle in controlling its
`operation to avoid an accident or hazardous driving condi-
`tion.
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`4
`the driver of a motor vehicle in controlling its operation to
`avoid hazardous conditions such as collisions with other
`vehicles, stationery objects or pedestrians.
`Another object is to provide a computerized system and
`method for controlling the speed of travel of a motor vehicle
`to lessen the chances of an accident while being driven by
`a person.
`is to provide a system and method
`Another object
`employing a television scanning camera mounted on a
`vehicle for scanning the field ahead, such as the image of the
`road ahead of the vehicle and a computer for analyzing the
`image signals generated wherein automatic image
`intensifying, or infra-red scanning and detection means is
`utilized to permit scanning operations to be effected during
`driving at night and in low light, snowing or fog conditions.
`Another object
`is to provide a system and method
`employing a television camera or other video scanning
`means mounted on a moving motor vehicle for scanning,
`detecting and identifying obstacles such as other vehicles
`ahead of such moving vehicle wherein the video image
`signals are analyzed to determine distances to such objects.
`Another object is to provide a computer controlled safety
`system for a motor vehicle which employs a television
`camera and an auxiliary scanning means to both identify
`obstacles in the path of the vehicle and determine distance
`therefrom on a real time and continuous basis for use in
`
`warning the operator of same and/or in controlling the
`operation of the vehicle to avoid a collision.
`BRIEF DESCRIPTION OF DRAWINGS
`
`The various hardware and software elements used to carry
`out the invention described herein are illustrated in the form
`
`of block diagrams, flow charts, and depictions of neural
`network and fuzzy logic algorithms and structures. The
`preferred embodiment is illustrated in the following figures:
`FIG. 1 is a block diagram of the overall Motor Vehicle
`Warning and Control System illustrating system sensors,
`computers, displays,
`input/output devices and other key
`elements.
`
`FIG.2 is a block diagram of an image analysis computer
`19 of the type that can be used in the Vehicle Hazard
`Avoidance System herein of FIG. 1.
`FIG. 3 illustrates a neural network of the type useful in the
`image analysis computer of FIG. 4.
`FIG. 4 illustrates the structure of a Processing Element
`(PE) in the neural network of FIG. 3.
`FIG. 5 is an alternate embodiment of a neural network
`
`image processor useful in the system of FIG. 1.
`FIG. 6 is a flow diagram illustrating the overall operation
`of the Motor Vehicle Warning and Control System of FIG.
`1.
`
`FIG. 7 illustrates typical input signal membership func-
`tions for fuzzy logic algorithms useful in the Motor Vehicle
`Warning and Control System of FIG. 1.
`FIG. 8 illustrates typical output signal membership func-
`tions for fuzzy logic algorithms useful in the Motor Vehicle
`Warning and Control System of FIG. 1.
`FIG. 9 illustrates typical Fuzzy Associative Memory
`(FAM) maps for the fuzzy logic algorithms useful in the
`Motor Vehicle Warning and Control System of FIG. 1.
`FIG. 10 is a Hazard/Object state vector useful in imple-
`menting the Fuzzy Logic Vehicle Warning and Control
`System.
`FIG. 11 is a Hazard Collision Control vector useful in
`
`is to provide a system and method
`Another object
`employing computerized image analysis to control or assist
`
`implementing the Fuzzy Logic vehicle Warning and Control
`System.
`
`(cid:57)(cid:36)(cid:47)(cid:40)(cid:50)(cid:3)(cid:40)(cid:59)(cid:17)(cid:3)(cid:20)(cid:19)(cid:19)(cid:25)(cid:66)(cid:19)(cid:20)(cid:26)
`VALEO EX. 1006_017
`
`

`
`US 6,553,130 B1
`
`5
`FIG. 12 is a table of Hazard/Object state vectors indicat-
`ing possible combinations of hazards and objects useful in
`the Fuzzy Associative Memory access system used herein.
`FIG. 13 is a more detailed logic flow diagram for the
`analysis of detection signals prior to accessing fuzzy logic
`control structures in the Motor Vehicle Warning and Control
`System.
`FIG. 14 is a more detailed Logic Flow Diagram for the
`Fuzzy Associative Memory
`selection processing.
`FIG. 15 is an example systcm flow illustrating the opera-
`tion of the Motor Vehicle Warning and Control System.
`DETAILED DESCRIPTION
`
`In FIG. 1 is shown a computerized control system 10 for
`controlling the operation of a motor vehicle to prevent or
`lessen the effects of accidents such as collisions with sta-
`tionery and/or moving objects such as other vehicles. The
`system 10 employs a control computer or microprocessor 11
`mounted on the vehicle and operable to receive and gate
`digital signals, such as codes and control signals from
`various sensors, to one or more specialized computers and
`from such computers to a number of servos such as electric
`motors and lineal actuators or solenoids, switches and the
`like, speakers and display drivers to perform either or both
`the functions of audibly and/or visually informing or warn-
`ing the driver of the vehicle of a hazardous road condition
`ahead and/or to effect controlled braking and steering
`actions of the vehicle.
`
`ARAM 12 and ROM 13 are connected to processor 11 to
`effect and facilitate its operation. A television camera(s) 16
`having a wide angle lens 16L is mounted at the front of the
`vehicle such as the front end of the roof, bumper or end of
`the hood to scan the road ahead of the vehicle at an angle
`encompassing the sides of the road and intcrsccting roads.
`The analog signal output of camera 16 is digitized in an A/D
`convertor 18 and passed directly to or through a video
`preprocessor 51 to microprocessor 11,
`to an image field
`analyzing computer 19 which is provided, implemented and
`programmed using neural networks and artificial
`intelli-
`gence as well as fuzzy logic algorithms to (a) identify
`objects on the road ahead such as other vehicles, pedestrians,
`barriers and dividers, turns in the road, signs and symbols,
`etc., and generate identification codes, and (b) detect dis-
`tances from such objects by their size and shape) and
`provide codes indicating same for use by a decision
`computer, 23, which generates coded control signals which
`are applied through the computer 11 or are directly passed to
`various warning and vehicle operating devices such as a
`braking computer or drive, 35, which operates a brake servo
`33, a steering computer or drive(s) 39 and 40 which operate
`steering servos 36; a synthetic speech signal generator 27
`which sends trains of indicating and warning digital speech
`signals to a digital-analog converter 29 connected to a
`speaker 30; a display driver 31 which drives a (heads-up or
`dashboard) display 32; a head light controller 41 for flashing
`the head lights, a warning light control 42 for flashing
`external and/or internal warning lights; a horn control 43,
`etc.
`
`Adigital speedometer 44 and accelerometer(s) 45 provide
`information signals for use by the decision computer, 23, in
`issuing its commands. Accelerometer(s) 45 are connected to
`control computer microprocessor 11 through analog-to-
`digital convector 46. The accelerometer(s) 45 may pass data
`continuously to control computer microprocessor 11, or,
`alternatively, respond to query signals from said control
`computer 11. An auxiliary range detection means comprises
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`6
`
`a range computer 21 which accepts digital code signals from
`a radar or lidar computer 14 which interprets radar and/or
`laser range signals from respective reflected radiation
`receiving means on the vehicle.
`In a modified form, video scanning and radar or lidar
`scanning may be jointly employed to identify and indicate
`distances between the controlled vehicle and objects ahead
`of, to the side(s) of, and to the rear of the controlled vehicle.
`The image analyzing computer 19 with associated
`memory 20 may be implemented in several different ways.
`Of particular concern is the requirement for high speed
`image processing with the capability to detect various haz-
`ards in dynamic image fields with changing scenes, moving
`objects and multiple objects, more than one of which maybe
`a potential hazard. Requirements for wide angle vision and
`the ability to analyze both right and left side image fields
`also exists. The imaging system not only detects hazards, but
`also estimates distance based on image data for input to the
`range computer 21 implemented with the associated
`memory unit 22.
`High speed image processing can be implemented
`employing known special purpose computer architectures
`including various parallel system structures and systems
`based on neural networks. FIG. 2 shows a high speed
`parallel processor system embodiment with dedicated image
`processing hardware. The system of FIG. 2 has a dedicated
`image data bus 50 for high speed image data transfer. The
`video camera 16 transfers full-frame video picture signal/
`data to the image bus 50 via analog/digital converter 18 and
`video preprocessor 51. The video camera 16 is preferably a
`CCD array camera generating successive picture frames
`with individual pixels being digitized for processing by the
`video preprocessor 51. The video camera 16 may also be
`implemented with other
`technologies including known
`image intensifying electron gun and infrared imaging meth-
`ods. Multiple cameras may be used for front, side and rear
`viewing and for stereo imaging capabilities suitable for
`generation of three dimensional image information includ-
`ing capabilities for depth perception and placing multiple
`objects in three dimensional image fields to further improve
`hazard detection capabilities.
`As shown in FIG. 2, the video preprocessor 51 performs
`necessary video image frame management and data manipu-
`lation in preparation for image analysis. The preprocessor 51
`may also be used in some embodiments for digital prefil-
`tering and image enhancement. Actual image data can be
`displayed in real time using video display 55 via analog-to-
`digital c