(12) United States Patent
`Schofield et al.
`
`(10) Patent NO.:
`(45) Date of Patent:
`
`US 8,643,724 B2
`Feb. 4,2014
`
`MULTI-CAMERA VISION SYSTEM FOR A
`VEHICLE
`
`Applicant: Donnelly Corporation, Holland, MI
`(US)
`
`Inventors: Kenneth Schofield, Holland, MI (US);
`Mark L. Larson, Grand Haven, MI
`(US); Keith J. Vadas, Coopersville, MI
`(US)
`
`Assignee: Magna Electronics Inc., Auburn Hills,
`MI (US)
`
`Notice:
`
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 0 days.
`
`Appl. No.: 131800,691
`
`Filed:
`
`Mar. 13.2013
`
`Prior Publication Data
`US 201310194426 A1
`Aug. 1,2013
`
`Related U.S. Application Data
`
`Continuation of application No. 121688,146, filed on
`Jan. 15, 2010, which is a continuation of application
`No. 121496,357, filed on Jul. 1, 2009, now Pat. No.
`8,462,204, which is a continuation of application No.
`111122,880, filed on May 5, 2005, now Pat. No.
`7,561,181, which is a continuation of application No.
`101324,679, filed on Dec. 20, 2002, now Pat. No.
`6,891,563, which is a continuation of application No.
`081952,026, filedas applicationNo. PCT/US96107382
`on May 22, 1996, now Pat. No. 6,498,620.
`
`(2006.01)
`
`Int. C1.
`H04N 7/18
`U.S. C1.
`USPC ............................ 3481148; 3481113; 34811 15
`Field of Classification Search
`USPC ........................................ 34811 11-1 15, 148
`See application file for complete search history.
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`3,985,424 A
`4,200,361 A
`
`1011976 Steinacher
`411980 Malvano
`(Continued)
`
`FOREIGN PATENT DOCUMENTS
`
`(Continued)
`
`OTHER PUBLICATIONS
`
`G. Wang, D. Renshaw, P.B. Denyer and M. Lu, CMOS Video Cam-
`eras, article, 1991, 4 pages, University of Edinburgh, UK.
`
`(Continued)
`
`Primary Examiner A n d y Rao
`(74) Attornq, Agent, or Firm - Gardner, Linn, Burkhart &
`Flory, LLP
`
`(57)
`
`ABSTRACT
`A multi-camera vision system for a vehicle includes first,
`second and third image capture devices disposed at respective
`vehicle portions. The first image capture device field of view
`overlaps with the third image capture device field of view
`defining a first overlap zone, and the second image capture
`device field of view overlaps with the third image capture
`device field of view defining a second overlap zone. Respon-
`sive to processing by an image processor of received image
`data, a synthesized image is generated without duplication of
`objects present in the first overlap zone and in the second
`overlap zone. The synthesized image approximates a view as
`would be seen by a virtual camera at a single location exterior
`of the vehicle, and is displayed by a single display screen of a
`reconfigurable display device that is viewable by a driver of
`the vehicle when normally operating the vehicle.
`
`86 Claims, 13 Drawing Sheets
`
`VALEO EX. 1001_001
`
`

`

`References Cited
`
`U.S.
`
`PATENT DOCUMENTS
`
`711980 Myers
`811980 Bart et al.
`1111980 Rosenblum
`111981 Gabel et al.
`21198 1 Chilvers
`51198 1 Wainwright
`711981 Robison
`811981 Ochiai
`911981 Talley et al.
`1011982 Noack
`1111982 Massoni et al.
`511983 Momiyama
`1211983 Felix
`211984 Lodetti
`411984 Bauer
`711984 Oettinger et al.
`1 111984 Watanabe et al.
`111985 Tong-Shen
`411985 Ballmer
`711985 Ballmer
`711985 Ballmer
`1011985 Franks
`1011985 Kamejima et al.
`211986 Downs
`211986 Reininger
`411986 Bechtel
`711986 Caine
`811986 Kato
`911986 Hyatt
`1011986 McCumber et al.
`1111986 Itoh
`1211986 Chey
`1211986 Ellis
`1211986 Barton
`1211986 Ohmi
`111987 Umebayashi et a1
`311987 Miiller
`311987 Fukuhara
`611987 Itoh
`611987 Itoh
`611987 Fukada
`611987 Hyatt
`611987 Itoh
`911987 Jacob
`911987 Seko et al.
`1011987 Suzuki
`1011987 Jacob
`1211987 Nishimura et al.
`111988 Botts
`211988 Smith
`311988 Hayashi et al.
`511988 Miyagi
`811988 Kretschmer et al.
`911988 Tuck
`1211988 Peterson
`1211988 Gahan
`411989 Simonelli
`411989 Hong
`411989 Howdle
`611989 Stewart
`711989 Michalopoulos el
`811989 Narendra et al.
`811989 Berman et al.
`811989 Farber et al.
`911989 Fujii et al.
`1011989 O'Farrell et al.
`1011989 Dye
`1111989 Shiraishi et al.
`1111989 Gallmeyer
`1211989 Molyneux
`111990 Matsumoto et al.
`111990 Rachael, I11
`111990 Swanson et al.
`111990 Miyaji
`211990 Berman
`
`Bmcker
`Petrossian et al.
`Schierbeek
`Bechtel et al.
`Tendler
`Van Lente et al.
`Schierbeek et al.
`Wood et al.
`Seko
`Kenue
`Lynas
`Tsai
`Masaki
`Berman et al.
`Walters
`Burley et al.
`Wilhelm
`Watanabe
`Baude et al.
`Torbert
`Petrossian et al.
`Chen
`Berman
`French
`Teder
`Alten
`Chen
`Lawler
`Kakinami et al.
`Lynas
`Choi
`Michaels et al.
`Toyama et al.
`Lynam
`Black
`Shimohigashi et al.
`Wilm et al.
`Davis et al.
`Slotkowski et al.
`Langlais et al.
`Hong
`Lipton et al.
`Schofield
`Bechtel
`Maeda
`Asayama
`Borcherts et al.
`O'Farrell
`Levers
`Larson et al.
`Brillard et al.
`Secor
`Faris
`Saneyoshi
`Kajiwara
`Vachss
`Pakett
`Jewel1 et al.
`Slotkowski et al.
`Kudoh
`Levers
`Nakayama
`Mathur et al.
`Fukuhara
`Parkes
`Asayama
`Kao
`Wilson et al.
`Trissel et al.
`Saneyoshi et al.
`Stanton
`Kishi et al.
`Larson et al.
`Hegyi
`Morinaga
`Asayama
`Kobayashi et al.
`Nelson
`Bauer et al.
`
`VALEO EX. 1001_002
`
`

`

`References Cited
`
`U.S. PATENT DOCUMENTS
`
`811995 Hegg et al.
`811995 Lelong et al.
`911995 Bechtel et al.
`1011995 Leddy et al.
`1011995 Yoshioka et al.
`1011995 Moore
`1111995 Suman et al.
`1111995 Fossum et al.
`1211995 Nishida et al.
`111996 Nakano et al.
`311996 Bendicks et al.
`311996 Stonecypher
`411996 Iino
`511996 Nishitani
`511996 Nakajima et al.
`611996 Kamei et al.
`611996 Shaw et al.
`611996 Larson et al.
`611996 Tsuchiya et al.
`711996 Alves et al.
`711996 Bechtel et al.
`711996 Asanuma et al.
`711996 Nishio
`811996 Schofield et al.
`911996 Shima et al.
`911996 Sato et al.
`1011996 Teder
`1111996 Hsieh
`1211996 Woll et al.
`111997 Caldwell
`311997 Mullins
`411997 Guinosso
`611997 Iwama
`611997 Hardin et al.
`711997 Uzawa
`711997 Kise
`811997 Mori et al.
`811997 Teder
`911997 Bechtel et al.
`911997 Varaprasad et al.
`911997 Schofield et al.
`1011997 Pomerleau
`1011997 Kingdon et al.
`1211997 Van Lente et al.
`211998 Schierbeek et al.
`311998 Varaprasad et al.
`311998 Bmnts
`411998 Olson et al.
`511998 Kinoshita et al.
`611998 Nayar
`611998 Cortes
`611998 Saburi et al.
`611998 Schofield et al.
`611998 Olson et al.
`611998 Wilson-Jones et 2
`711998 Wiemer et al.
`711998 Schofield et al.
`811998 Nakayama
`811998 Blaker et al.
`811998 Rosinski et al.
`811998 Schmidt
`811998 Schofield et al.
`811998 O'Farrell et al.
`1111998 Miles
`1111998 Stam et al.
`1211998 Van Ryzin
`1211998 Kiyomoto et al.
`1211998 Breed et al.
`1211998 Breed et al.
`1211998 Kinoshita et al.
`1211998 Takano et al.
`211999 Onda
`311999 Kowalick
`311999 Schofield et al.
`311999 Olson
`
`Ashihara et al.
`Lion
`Onoda
`Mori et al.
`Chan
`Bos
`Stam et al.
`Schofield et al.
`Frankhouse et al.
`Schofield et al.
`Lin
`O'Farrell et al.
`Fumta
`Banitt
`Alland et al.
`O'Farrell et al.
`Miles
`Bechtel et al.
`Nagao et al.
`Varaprasad et al.
`Harris et al.
`Buschur
`Stam et al.
`Ponziana
`Coulling et al.
`DeLine et al.
`Schofield et al.
`Stam et al.
`Hoek
`Marcus et al.
`DeLine et al.
`Bos et al.
`Tenenbaum et al.
`DeLine et al.
`O'Farrell et al.
`Kendrick
`Schofield et al.
`Bos
`Schofield et al.
`DeLine et al.
`DeLine et al.
`Lynam
`Duroux
`Yonezawa et al.
`Laumeyer et al.
`Shimoura et al.
`Marcus et al.
`Schofield et al.
`Turnbull et al.
`Schofield et al.
`Caldwell
`Bos et al.
`Lee
`Schofield et al.
`Caldwell
`Heslin et al.
`Skiver et al.
`Mazzilli
`Lynam
`Schofield et al.
`DeLine et al.
`Teuchert
`Bos et al.
`Bloomfield et al.
`Franke et al.
`DeLine et al.
`Gutta et al.
`Hutzel et al.
`Naoi et al.
`DeLine et al.
`Guerra
`Breed et al.
`McCarthy et al.
`Duroux et al.
`Dassanayake et a1
`Schofield et al.
`Schierbeek et al.
`Lynam
`Schofield et al.
`
`VALEO EX. 1001_003
`
`

`

`812007 McCabe et al.
`1212007 Schofield et al.
`212008 Schofield et al.
`212008 Schofield et al.
`Lynam
`Schofield et a1
`Schofield et al.
`Uken et al.
`DeWind et al.
`Bamji
`Schofield et al.
`Schofield et al.
`Schofield et al.
`Schofield et al.
`Bechtel et al.
`Schofield et al.
`Schofield et al.
`Schofield et al.
`Salmeen et al.
`Schofield et al.
`Stam et al.
`Schofield et al.
`Lynam et al.
`Dunsmoir et al.
`Katoh
`Dhua et al.
`Higgins-Luthman
`Schofield et al.
`Lafon
`Yung et al.
`Schofield et al.
`Camilleri et al.
`Higgins-Luthman et
`Hosagrahara et al.
`Lu et al.
`Taylor et al.
`Schofield et al.
`Saito
`Williams
`Lewellen
`Hamdan et al.
`Stam et al.
`Kosaka et al.
`Stam et al.
`Stam et al.
`Hutzel et al.
`Stam et al.
`Tseng
`Wildmann et al.
`512007 Yasutomi et al.
`512007 Schofield et al.
`Schofield et al.
`Bradley
`Howard et al.
`Bechtel et al.
`Tseng et al.
`Bechtel et al.
`Bechtel et al.
`Bechtel et al.
`Lundblad et al.
`
`200710104476 A1
`200710109406 A1
`
`FOREIGN PATENT DOCUMENTS
`
`References Cited
`
`U.S.
`
`PATENT DOCUMENTS
`
`7,255,451 B2
`7,3 11,406 B2
`7,325,934 B2
`7.325.935 B2
`
`312003 Marcus et al.
`312003 Turnbull
`412003 DeVries, Jr. et al.
`412003 Lemelson et al.
`512003 Schofield et al.
`612003 Chui et al.
`612003 Ebersole et al.
`712003 Stam et al.
`712003 Kothari et al.
`712003 Heslin et al.
`712003 Ogura et al.
`812003 Schofield et al.
`812003 Stam et al.
`912003 Getz et al.
`1012003 Suzuki et al.
`1012003 Stmmolo
`1112003 Hutzel et al.
`1112003 DeLine et al.
`1112003 Miles
`112004 Schnell et al.
`112004 Miles
`112004 Downs
`112004 McCarthy et al.
`112004 Miles
`212004 Schofield et al.
`312004 Toyoda et al.
`312004 Steffel
`312004 Stein et al.
`312004 Miles et al.
`312004 Treyz et al.
`312004 Lewis et al.
`412004 Bos et al.
`512004 Breed et al.
`512004 Miles
`612004 Sjonell
`612004 Bos
`712004 Lippert et al.
`912004 Miles
`912004 Urey
`1012004 Schofield et a1
`1012004 Bos et al.
`1112004 Bos et al.
`1112004 Shirato et al.
`1112004 Schofield et al.
`1212004 Schofield et al.
`112005 Burgner
`412005 Schofield
`512005 Winner et al.
`512005 Schofield et al.
`612005 Meehan et al.
`912005 Schofield
`1012005 Schofield et al.
`1112005 Lynam
`1212005 Rykowski et al.
`212006 Weller et al.
`212006 Schofield
`212006 McMahon et al.
`512006 Pawlicki et al.
`512006 Burgner
`612006 Kim
`612006 Moisel et al.
`812006 Breed et al.
`812006 Laumeyer et al.
`1012006 Winter et al.
`1012006 Schofield
`1112006 Hatae et al.
`1212006 Stam et al.
`112007 Taylor et al.
`312007 Lynam et al.
`412007 Breed
`512007 Quist et al.
`612007 Bos et al.
`612007 Hull et al.
`712007 Kulas et al.
`812007 Lindahl et al.
`
`VALEO EX. 1001_004
`
`

`

`US 8,643,724 B2
`Page 5
`
`(56)
`
`References Cited
`
`FOREIGN PATENT DOCUMENTS
`
`OTHER PUBLICATIONS
`
`Tokimam et al., "CMOS Rear-View TV System with CCD Camera",
`National Technical Report vol. 34, No. 3, pp. 329-336, Jun. 1988
`(Japan).
`Reexamination Control No. 901007,519, dated Jun. 9, 2005, Reex-
`amination of U.S. Patent No. 6,222,447, issued to Schofield et al.
`
`Reexamination Control No. 901007,520, dated Jun. 9, 2005, Reex-
`amination of U.S. Patent No. 5,949,331, issued to Schofield et al.
`Reexamination Control No. 90101 1,478, dated Mar. 28,201 1, Reex-
`amination of U.S. Patent No. 6,222,447, issued to Schofield et al.
`Reexamination Control No. 90101 1,477, dated Mar. 14,201 1, Reex-
`amination of U.S. Patent No. 5,949,331, issued to Schofield et al.
`J. Borenstein et al., "Where am I? Sensors and Method for Mobile
`Robot Positioning", University of Michigan, Apr. 1996, pp. 2, 125-
`128.
`Bow, Sing T., "Pattern Recognition and Image Preprocessing (Signal
`Processing and Communications)", CRC Press, Jan. 15, 2002, pp.
`557-559.
`Vlacic et al., (Eds), "Intelligent Vehicle Tecnologies, Theoly and
`Applications", Society of Automotive Engineers Inc., edited by SAE
`International, 2001.
`Van Leuven et al., "Real-Time Vehicle Tracking in Image
`Sequences", IEEE, US, vol. 3, May 21, 2001, pp. 2049-2054,
`XP010547308.
`Van Leeuwen et al., "Requirements for Motion Estimation in Image
`Sequences for Traffic Applications", IEEE, US, vol. 1, May 24, 1999,
`pp. 145-150, XP010340272.
`Van Leeuwen et al., "Motion Estimation with a Mobile Camera for
`Traffic Applications", IEEE, US, vol. 1, Oct. 3, 2000, pp. 58-63.
`Van Leeuwen et al., "Motion Interpretation for In-Car Vision Sys-
`tems", IEEE, US, vol. 1, Sep. 30, 2002, p. 135-140.
`Pratt, "Digital Image Processing, Passage-ED.?",
`Sons, US, Jan. 1,2001, pp. 657-659, XP002529771.
`Donnelly Panoramic VisionTM on Renault Talisman Concept Car at
`Frankfort Motor Show, PR Newswire, Frankfort, Germany Sep. 10,
`200 1.
`
`John Wiley &
`
`VALEO EX. 1001_005
`
`

`

`U.S. Patent
`
`Feb. 4,2014
`
`Sheet 1 of 13
`
`VALEO EX. 1001_006
`
`

`

`U.S. Patent
`
`Feb. 4,2014
`
`Sheet 2 of 13
`
`FIG. 2
`
`FIG. 3
`
`VALEO EX. 1001_007
`
`

`

`U.S. Patent
`
`Feb. 4,2014
`
`Sheet 3 of 13
`
`VALEO EX. 1001_008
`
`

`

`U.S. Patent
`
`Feb. 4,2014
`
`Sheet 4 of 13
`
`VALEO EX. 1001_009
`
`

`

`U.S. Patent
`
`Feb. 4,2014
`
`Sheet 5 of 13
`
`VALEO EX. 1001_010
`
`

`

`U.S. Patent
`
`Feb. 4,2014
`
`Sheet 6 of 13
`
`VALEO EX. 1001_011
`
`

`

`U.S. Patent
`
`Feb. 4,2014
`
`Sheet 7 of 13
`
`FIG. I I
`
`VALEO EX. 1001_012
`
`

`

`U.S. Patent
`
`Feb. 4,2014
`
`Sheet 8 of 13
`
`FIG. 13
`
`VALEO EX. 1001_013
`
`

`

`U.S. Patent
`
`Feb. 4,2014
`
`Sheet 9 of 13
`
`AMBIENT
`
`FIG. 14
`
`FIG. 15
`
`FIG. 16
`
`VALEO EX. 1001_014
`
`

`

`U.S. Patent
`
`Feb. 4,2014
`
`Sheet 10 of 13
`
`VALEO EX. 1001_015
`
`

`

`U.S. Patent
`
`Feb. 4,2014
`
`Sheet 11 of 13
`
`ELECTRO-OPT IC
`M I RROR
`
`I
`
`1 MAG€
`PROCESSOR
`
`100
`I
`
`,
`
`
`
`1
`
`0 \SPLAY
`
`r
`
`16 ,, CENTER
`
`IMAGE
`CAPTURE
`
`CAPTURE
`
`ELECTRO-OPTIC
`WINDOW
`
`FIG. 18
`
`FIG. 19
`
`FIG. 20
`
`VALEO EX. 1001_016
`
`

`

`U.S. Patent
`
`Feb. 4,2014
`
`Sheet 12 of 13
`
`14
`
`16
`
`14
`
`FIG. 22
`
`AND ROAD
`EDGES
`
`FIG. 23
`
`VALEO EX. 1001_017
`
`

`

`U.S. Patent
`
`Feb. 4,2014
`
`Sheet 13 of 13
`
`CAPTURE
`
`18
`
`144-1
`IMAGE
`I
`I CAPTURE
`COMMUNICATION
`U DATA
`
`1
`
`FIG. 24
`
`FIG. 25
`
`198
`
`VALEO EX. 1001_018
`
`

`

`US 8,643,724 B2
`
`CROSS-REFERENCE TO RELATED
`APPLICATIONS
`
`5
`
`1
`MULTI-CAMERA VISION SYSTEM FOR A
`VEHICLE
`
`2
`is usually captured by more than one image capture device at
`a time and displayed in multiple images. This may confuse
`the driver as to whether more than one object is present. When
`multiple image capture devices are positioned at different
`longitudinal locations on the vehicle, objects behind the
`vehicle are at different distances from the image capture
`devices. This results in different image sizes for the same
`This application is a continuation of U.S. patent applica-
`object. This effect is especially noticeable for laterally
`tion Ser. No. 121688,146, filed Jan. 15, 2010, which is a
`extending images, such as a bridge, highway crosswalkmark-
`continuation of U.S. patent application Ser. No. 121496,357,
`filed Jul. 1, 2009, now U.S. Pat. No. 8,462,204, which is a 10 ings, the earth's horizon, and the like. Such images are at
`continuation of U.S. patent application Ser. No. 111122,880,
`different vertical angles with respect to the image capture
`filed May 5, 2005, now U.S. Pat. No. 7,561,181, which is a
`devices. This results in different vertical positions on the
`continuation of U.S. patent application Ser. No. 101324,679,
`display causing the elongated image to appear disjointed.
`filedonDec. 20,2002, now U.S. Pat. No. 6,891,563, whichis
`A camera system provides a monocular view of the scene,
`a continuation 0fU.S. patent application Ser. No. 081952,026, 15 compared to the binocular stereoscopic view obtained when
`filedunder35 U.S.C. g371, filedNov. 19,1997, now U.S. Pat.
`the scene is viewedthrough arearview mirror. This makes the
`No. 6,498,620, which is based on International Patent Appli-
`ability to judge distances in a camera system a problem. This
`cation No. PCTiUS96107382, filed May 22, 1996.
`effect is most noticeable at distances close to the vehicle
`where stereoscopic imaging is relied upon extensively by the
`20 driver in judging relative locations of objects. Therefore,
`known camera systems fail to provide to the driver important
`This invention relates generally to vision systems for
`information where that information is most needed at small
`vehicles and, more particularly, to rearview vision systems
`separation distances from surrounding objects.
`which provide the vehicle operator with scenic information in
`Another difficulty with camera systems is that, in order to
`the direction rearward of the vehicle. More particularly, the 25 provide a sufficient amount of information, the camera sys-
`tem typically presents the driver with a greatly increased field
`invention relates to a rearview vision system utilizing image
`capture devices, such as CMOS imaging arrays and the like.
`of view. This improves performance by further reducing blind
`A long-felt need in the art of vehicle rearview vision sys-
`spots at the side and rear of the vehicle. However, an increased
`tems is to reduce the amount of time spent gathering infor-
`field of view is often obtained by utilizing a wide-angle lens
`mation of the condition around the vehicle in order to safely 30 which introduces distortion of the scene and further impairs
`carry out a vehicle maneuver, such as a turn or a lane change.
`the ability of the driver to judge distances of objects dis-
`It has been determined that up to about 50 percent of maneu-
`played. The problem with such distortion of the scene is that
`ver time is spent gathering information with conventional
`the driver must concentrate more on the display and take a
`rearview mirrors. This typically requires viewing one or more
`longer time to interpret and extract the necessary information.
`mirrors, turning the head and body to check blind spots, and 35 This further distracts the driver from the primary visual task
`double-checking the mirrors prior to executing the maneuver.
`of maintaining awareness of vehicles and other objects in the
`Some improvement has been obtained by adjusting mirror
`vicinity of the driven vehicle.
`optics, for example, to increase the observed field of view
`Yet an additional difficulty with camera systems is that flat
`rearward of the vehicle. However, this is achieved with an
`panel displays present the image captured by the rearward-
`increase in image distortion whichmakes driver awareness of 40 facing image capture device, or devices, at a focal length that
`approximates the arm length of the vehicle driver. In order to
`conditions to the side and rear of the vehicle even more
`difficult.
`observe the condition of the vehicle utilizing the rearview
`Another long-felt need in the art of vehicle rearview vision
`vision system, the driver must change his or her gaze from the
`systems has been to eliminate exterior rearview mirrors by
`forward field of view of the vehicle to the display. Because the
`utilizing image capture devices, such as cameras, in combi- 45 forward field of view of the vehicle is at a focal length that is
`nation with dashboard displays. This would be beneficial
`much greater than the focal length of the displayed image, the
`because it would reduce wind drag on the vehicle, wind noise
`eyes of the driver must refocus upon changing gaze. This
`and vehicle weight. Furthermore, rearview mirrors protrude a
`refocusing further increases the amount of time for the driver
`substantial distance from the side of the vehicle, which makes
`to assimilate the information in the displayed image. Further-
`maneuvering in tight spaces more difficult. Image capture 50 more, when the gaze of the driver returns to the forward field
`devices are capable of positioning in a greater variety of
`of view, the eyes must, again, refocus to the greatly longer
`locations on the vehicle, providing more flexibility of vehicle
`distance.
`styling. It is further expected that camera systems would
`Yet an additional difficulty with camera systems is that of
`greatly reduce the blind spots to the sides and rear of the
`finding adequate space in the crowded area of the vehicle's
`vehicle common with vehicles equipped with conventional 55 dashboard for the components making up the display.
`rearview mirror systems. The driver cannot perceive vehicles,
`objects, or otherroadusers in such blind spots without turning
`his or her body, which interferes with forward-looking visual
`The present invention is directed towards enhancing the
`activities.
`Camera-based rearview vision systems for vehicles have 60 interpretation of visual information in a rearview vision sys-
`not obtained commercial acceptance. One difficulty with pro-
`tem by presenting information in a manner which does not
`posed systems has been that they present a large amount of
`require significant concentration of the driver or present dis-
`visual information in a manner which is difficult to compre-
`tractions to the driver. This is accomplished according to an
`aspect of the invention in a rearview vision system having at
`hend. This difficulty arises from many factors. In order to
`significantly reduce blind spots, multiple image capture 65 least two image capture devices positioned on the vehicle and
`devices are typically positioned at various locations on the
`directed generally rearwardly with respect to the direction of
`travel of the vehicle. A display is provided for images cap-
`vehicle. The image of an object behind the equipped vehicle
`
`BACKGROUND OF THE INVENTION
`
`SUMMARY OF THE INVENTION
`
`VALEO EX. 1001_019
`
`

`

`US 8,643,724 B2
`
`3
`4
`required for the driver to gaze at the displayed image and
`tured by the image capture devices. The display displays an
`image synthesized from outputs of the image capture devices
`interpret objects displayed in the image. Furthermore, the
`which approximates a rearward-facing view from a single
`reduction in the repeated refocusing of the driver's eyes
`location. In order to obtain all of the necessary information of
`reduces driver fatigue. If there are any near field objects in the
`activity, not only behind but also along side of the vehicle, the 5 periphery of the
`forward field of view, such as wind-
`virtual camera should be positioned forward of the driver. The
`shield wipers,
`frame, dashboard, and the like, the
`image synthesized from the multiple image capture devices
`display system is preferably positioned in a manner which
`may have a dead 'pace which corresponds with the area
`blocks the view of such near field objects. In this manner, the
`This dead space is useable
`the
`the
`driver's gaze may shift between the forward field of view and
`driver's sense of perspective in judging the location of l o the long focal length display system without being refocused
`vehicles behind and along side of the equipped vehicle.
`on the near field objects. This is based upon a recognition that
`The present invention provides techniques for synthesizing
`the driver's eyes will tend to refocus on the near field object
`images captured by individual, spatially separated, image
`the gaze is being redirected
`even
`capture devices into such ideal image, displayed on the dis-
`view and the
`system.
`play device, This may be accomplished, according to an 15 between the forward
`According to yet an additional aspect of the invention, a
`aspect of the invention, by providing at least three image
`rearview vision system for a vehicle includes at least one
`capture devices,
`least two ofthe image capture devices are
`image capture device positioned on the vehicle and directed
`side image capture devices mounted on opposite sides ofthe
`generally reamardly with respect to the direction of travel of
`vehicle. At least one of the image capture devices is a center
`image capture device mounted laterally between the side 20 the vehicle. A display system is provided for displaying a rear
`image capture devices. A display system displays an image
`image captured by the image capture device. The displayed
`synthesized from outputs of the image capture devices. The
`image is a unitary image having an aspect ratio that is between
`approximately 4: 1 andapproximately 2: 1. Ina most preferred
`displayed image includes an image portion from each of the
`embodiment, the image has an aspect ratio that is approxi-
`image capture devices. The image portion from the center
`image capture device is vertically compressed.
`25 mately 8:3. The aspect ratio, according to this aspect of the
`It has been discovered that such vertical compression sub-
`invention, is especially useful where the unitary image is
`stantially eliminates distortion resulting from the spatial
`synthesized from a plurality of images which are captured by
`separation between the cameras and can be readily accom-
`a plurality of image captured devices and are tiled by the
`plished. In an illustrated embodiment, the image compression
`display device.
`is carried out by removing selective ones of the scan lines 30 According to yet an additional aspect of the invention, a
`making up the image portion. A greater number of lines are
`rearview vision system for a vehicle includes a plurality of
`removed further away from the vertical center of the image.
`image capture devices positioned on the vehicle and directed
`The compression of the central image portion produces a
`generally rearwardly with respect to the direction of travel of
`dead space in the displayed image which may be made to
`the vehicle. A display system which includes at least one
`correspond with the area that would be occupied by the 35 image generator and an optical correction system is provided
`vehicle in the view from the single virtual camera. Preferably,
`which displays an image synthesized from outputs of the
`perspective lines are included at lateral edges of the dead
`image capture devices as a unitary image. Alternatively, the
`space which are aligned with the direction of travel of the
`display system may include a plurality of image generators,
`vehicle and, therefore, appear in parallel with lane markings.
`each associated with one or more of the image capture devices
`This provides visual clues to the driver's sense of perspective 40 and an optical correction system which amplifies images
`in order to assist in judging distances of objects around the
`generated by the image generators and merges them into a
`vehicle.
`unitary image. The optical correction system additionally
`According to another aspect of the invention, image
`increases the focal length, or lengths, of the image, or images,
`enhancement means are provided for enhancing the displayed
`generated by the image generator, or generators. The display
`image. Such means may be in the form of graphic overlays 45 system may be an opaque projection display which is posi-
`superimposed on the displayed image. Such graphic overlap
`tioned approximately at the driver's arm length in front of the
`may include indicia of the anticipated path of travel of the
`driver. Alternatively, the display system may be a view-
`vehicle which is useful in assisting the driver in guiding the
`through heads-up display which projects the unitary image
`onto a combiner in order to combine the unitary image with
`vehicle in reverse directions. Such graphic overlay may
`include a distance grid indicating distances behind the vehicle 50 the forward field of view of the driver.
`of objects juxtaposed with the grid.
`These and other objects, advantages, and features of this
`According to yet an additional aspect of the invention, a
`invention will become apparent by review of the following
`rearview vision system for a vehicle includes at least one
`specification in conjunction with the drawings.
`image capture device positioned on the vehicle and directed
`generally rearwardly with respect to the direction of travel of 55
`the vehicle. A display system is provided which displays a
`FIG. 1 is a top plan view of a vehicle having a rearview
`rear image synthesized from an output of the image capture
`device. The rear image is substantially contiguous with the
`vision system according to the invention;
`FIG. 2 is a side elevation of the vehicle in FIG. 1;
`forward field of view of the vehicle driver and at a focal length
`FIG. 3 is a front elevation of a display according to the
`that is forward of the vehicle passenger compartment and 60
`invention;
`preferably within the depth of field of a vehicle driver viewing
`FIG. 4 is the same view as FIG. 1 illustrating an alternative
`a distant object. Because the image has a focal length that
`embodiment of the invention;
`more closely matches that of the forward field of view
`FIG. 5 is a block diagram of an electronic system according
`observed by the driver, the need for the driver's eyes to refo-
`cus from the forward field of view to a much shorter focus 65 to the invention;
`FIG. 6 is the same view as FIG. 3 illustrating an alternate
`distance each time the gaze of the driver is directed at the
`display system is minimized. This reduces the amount of time
`mode of operation of the system;
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`VALEO EX. 1001_020
`
`

`

`US 8,643,724 B2
`
`25
`
`6
`5
`FIG. 7 is the same view as FIG. 2 illustrating an alternative
`rearwardly of the vehicle, with the vehicle being transparent
`to the view of the virtual image capture device. Vision system
`embodiment of the invention;
`12 provides a substantially seamless panoramic view rear-
`FIG. 8 is the same view as FIG. 3 illustrating an alternative
`wardly of the vehicle without duplicate or redundant images
`embodiment of the invention;
`FIG. 9 is the same view as FIGS. 1 and 4 illustrating an 5 of objects. Furthermore, elongated, laterally-extending,
`objects, such as the earth's horizon, appear uniform and
`alternative embodiment of the invention;
`straight across the entire displayed image. The displayed
`FIG. 10 is the same view as FIGS. 3 and 8 illustrating an
`image provides a sense of perspective, which enhances the
`alternative embodiment of the invention;
`ability of the driver to judge location and speed of adjacent
`FIG. 11 is a chart illustrating the horizontal row of pixels
`(nl, n2) on which an object will be imaged from two longi- 10 trailing vehicles.
`Each of side image capture devices 14 has a field of view 22
`tudinally separated image capture devices as that object is
`and is aimed rearwardly with respect to the vehicle about an
`spaced at different longitudinal distances from the image
`axis 24 which is at an angle, with respect to the vehicle, that
`capture devices;
`is half of the horizontal field of view of the image capture
`FIG. 12 is a forward elevation of a vehicle passenger corn-
`15 device. In this manner, each of the image capture devices 14
`partment as viewed by a vehicle driver;
`covers an area bounded by the side of the vehicle and extend-
`FIG. 13 is a sectional view t&en along the lines XIII-XIII
`ing outwardly at an angle defined by the horizontal field of
`in FIG. 12;
`view of the respective side image capture device. Center
`FIG. 14 is a sectional view t&en along the lines XIV-XIV
`image capture device 16 has a horizontal field of view 26,
`in FIG. 12;
`FIG. 15 is the same view as FIG. 14 of an alternative 20 which is symmetrical about the longitudinal axis of the
`vehicle. The field of view of each side image capture device
`embodiment;
`14 intersect the field of view of center image capture device
`FIG. 16 is the same view as FIG. 14 of another alternative
`16 at a point P which is located a distance Q behind vehicle
`embodiment;
`10.
`FIG. 17 is an enlargedview ofthe display system inFIG. 14
`Rear blind zones 30 are located symmetrically behind
`illustrating details thereof;
`vehicle 10 extending from the rear of the vehicle to point P.
`FIG. 18 is a block diagram similar to FIG. 5 of an alterna-
`Side blind zones 25 located laterally on respective sides of the
`tive embodiment of the invention;
`vehicle extend rearwardly of the forward field of view 36 of
`FIG. 19 is a side elevation similar to FIG. 2 of an alternative
`the driver to the field of view 22 of the respective side image
`embodiment of the invention;
`FIG. 20 is an enlarged side elevation of an image capture 30 capture device 14. An object will not be captured by side
`image capture devices 14 or center image capture device 16 if
`device with portions of the housing removed in order to reveal
`the object is entirely within one of the blind zones 25,30. In
`internal structure thereof;
`order for an object, such as another vehicle V or other road
`FIG. 21 is a block diagram similar to FIG. 5 of another
`user travelling to the side of vehicle 10, to be observed by an
`alternative embodiment of the invention;
`FIG. 22 is a block diagram similar to FIG. 5 of another 35 operator of vehicle 10, the object must be either at least
`partially within the forward field of view 36 of the driver or be
`alternativ