United States Patent
`
`[19]
`
`[11]
`
`Patent Number:
`
`5,670,935
`
`Schofield et al.
`
`[45] Date of Patent:
`
`Sep. 23, 1997
`
`US005670935A
`
`REARVIEW VISION SYSTEM FOR VEHICLE
`INCLUDING PANORAMIC VIEW
`
`4,987 ,357
`5,001,558
`
`1/1991 Masaki.
`3/1991 Buriey et al. .
`
`(List continued on next page.)
`FOREIGN PATENT DOCUMENTS
`
`.
`
`11/1992
`0513476A1
`European Pat. Ofi.
`France .
`2/1987
`2585991
`France .
`8/1992
`2672857
`France .
`9/1992
`2673499
`7/1984
`324851 1A1
`Germany .
`3/1980
`55-039843
`Japan .
`12/1983
`58-209635
`Japan .
`60261275 of 1985
`Japan .
`62- 122844
`6/1987
`Japan .
`62122487
`6/1987
`Japan .
`30061192
`3/1991
`Japan .
`40245886
`9/1992
`Japan .
`50000638
`1/1993
`Japan .
`934037
`8/1963
`United Kingdom .
`2137573
`10/1984
`United Kingdom .
`2244187
`11/1991 United Kingdom .
`2255539
`1 1/1992 United Kingdom .
`
`OTHER PUBLICATIONS
`
`Kobe. Gerry. “Hypnotic Wizardry! (interior e1ectronics).”
`Automotive Industries. vol. 169. No. 5. p. 60. published
`May. 1989. Relevant section is entitled Instrumentation.
`
`Primary Examiner—Jel‘1‘ery Hofsass
`Assistant Examiner—'I‘irnothy Edwards. Jr.
`Attomey, Agent, or Firm——Van Dyke. Gardner. Linn &
`Burkhalt. LLP
`
`[57]
`
`ABSTRACT
`
`A rearview vision system for a vehicle includes at least one
`image capture device directed rearwardly with respect to the
`direction of travel of the vehicle. A display system displays
`an image synthesized from output of the image captive
`device. The display system is preferably contiguous with the
`forward field of view of the vehicle driver at a focal length
`that is forward of the vehicle passenger compartment. A
`plurality of image capture devices my be provided and the
`display system displays a unitary image synthesized from
`outputs of the image captive devices which approximates a
`rearward-facing view from a single location. such as forward
`of the vehicle.
`
`54 Claims, 7 Drawing Sheets
`
`[54]
`
`[75]
`
`[73]
`
`[21]
`
`[22]
`
`[63]
`
`[5 1]
`[52]
`
`[5 8]
`
`[56]
`
`Inventors :
`
`Kenneth Schofield. Holland; Mark L.
`Larson. Grand Haven; Keith J. Vadas.
`Coopersville. all of Mich.
`
`Assignec: Donnelly Corporation. Holland. Mich.
`
`Appl. No.: 445,527
`
`Filed:
`
`May 22, 1995
`
`Related U.S. Application Data
`
`Continuation-in-part of Ser. No. 23,918, Feb. 26, 1993, Pat.
`No. 5,550,677.
`
`Int. CL6 ................................. B60Q 1/00; G09F 9/00
`US. Cl. .......................... 340/461; 340/435; 340/903;
`348/113; 348/148
`Field of Search ..................................... 340/435. 436.
`340/901. 903. 461; 348/118, 148
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`.
`
`.
`
`7/1964 Flatt.
`3,141,393
`9/1972 Rosenfield et al.
`3,689,695
`4,052,712 10/1977 Ohama et al.
`.
`4,093,364
`6/1978 Miller.
`4,214,266
`7/1980 Myers.
`4,247,870
`1/1981 Gabe] et al. .
`4,277,804
`7/1981 Robison .
`4,288,814
`9/1981 Talley et al. .
`4,420,238 12/1983 Felix .
`4,626,850 12/1986 Chey.
`4,630,109 12/1986 Barton.
`4,713,685 12/1987 Nishimura et al.
`4,731,669
`3/1988 Hayashi et al.
`.
`4,789,904 12/1988 Peterson .
`4,825,232
`4/1989 Howdle .
`4,847,772
`7/1989 Michalopoulos et al.
`4,872,051
`10/1989 Dye.
`4,892,345
`1/1990 Rachael, III .
`4,910,591
`3/1990 Petmssian et al. .
`4,937,796
`6/1990 Tendler.
`4,970,653
`11/1990 Kenue ..................................... 364/461
`
`.
`
`
`
`Valeo Exhibit 1007_001
`
`Valeo Exhibit 1007_001
`
`

`
`5,670,935
`Page 2
`
`U.S. PATENT DOCUMENTS
`6/1991 Petrossian et 2]..
`3/1992 Kakinami etal..
`6/1992 Choi ........................................ 343/143
`a
`‘
`2/1994 Secor ...................................... 359/896
`
`.. 340/705
`4/1994 Paris .... ..
`4/1994 Saneyoshi
`............................... 343/116
`5/1994 pakgu,
`6/1994 Jewell eta]. .............................. 372/50
`7/1994 Kudoh.
`
`5,027,200
`5,096,237
`5,121,200
`
`5,239,321
`5,305,012
`5,307,136
`5,325,096
`5,325,386
`5,331,312
`
`5,341,437
`3/1994 Nakayama .............................. 343/113
`535
`3
`ukuham.
`' in 10/1994 F
`5’33°’235
`"1995 A“Y“““"
`5*4°6=395
`‘"1995 W“-°’°" “F 31-
`.....................
`Cl
`5/1995 K15“ °‘ 31-
`----------------------------- 343/113
`7/1995 Nelson-
`--------------------------------- 340/903
`5/1996 Nishitani
`6/1996 Shaw et a1.
`........................... .. 340/903
`7/1996 Nishio
`. 340/435
`
`11/1996 Hsieh ...................................... 340/436
`
`5414,45‘
`5»430«431
`5515.448
`5,529,138
`5.541.590
`5,574,443
`
`-
`
`Valeo Exhibit 1007_002
`
`Valeo Exhibit 1007_002
`
`

`
`5,670,935
`
`Valeo Exhibi0000000 03
`
`Valeo Exhibit 1007_003
`
`

`
`U.S. Patent
`
`Sep. 23, 1997
`
`Sheet 2 of 7
`
`5,670,935
`
`FIG. 3
`
`Valeo Exhibit 1007_004
`
`Valeo Exhibit 1007_004
`
`

`
`U.S. Patent
`
`Sep. 23, 1997
`
`Sheet 3 of 7
`
`5,670,935
`
`72“.
`
`Va_leo Exhibit 1007 005
`
`Valeo Exhibit 1007_005
`
`

`
`U.S. Patent
`
`Sep. 23, 1997
`
`Sheet 4 of 7
`
`5,670,935
`
`m.o:.
`
`><._._ma
`
`mommmooxm
`
`uo<:_
`
`mupzuu
`
`M042.
`
`mm:»a<o
`
`.mmzpmqo
`
`uw<:_
`
`maaw
`
`umapmqo
`
`uo<2_
`
`menI
`
`Valeo Exhibit 1007 006
`
`Valeo Exhibit 1007_006
`
`
`
`
`
`
`

`
` U.S.Patent
`
`Sep. 23, 1997
`
`Sheet 5 of 7
`
`5,670,935
`
`Valeo Exhibit 1007 007
`
`Valeo Exhibit 1007_007
`
`

`
`U.S. Patent
`
`Sep. 23, 1997
`
`Sheet 6 of 7
`
`5,670,935
`
`Valeo Exhibit 1007_008
`
`Valeo Exhibit 1007_008
`
`

`
`U.S. Patent
`
`Sep. 23, 1997
`
`Sheet 7 of 7
`
`5,670,935
`
`29
`
`%
`
`49
`
`90
`
`m.
`
`2
`
`92
`
`N I
`
`9.4
`2747.7
`
`..7.
`
`jifi
`IKE
`jfll
`II-I]
`
`I777.N9HH
`
`I..I67878
`
`N
`
`I7777Wu.7
`
`FIG. I
`
`I
`
`70I.5
`
`I74
`
`Valeo Exhibit 1007 009
`
`Valeo Exhibit 1007_009
`
`
`
`
`
`

`
`5 ,670,935
`
`1
`REARVIEW VISION SYSTEM FOR VEHICLE
`INCLUDING PANORAMIC VIEW
`
`CROSS-REFERENCE TO RELATED
`APPLICATION
`
`This application is a continuation-in-part of patent appli-
`cation Ser. No. 08/023.918 filed Feb. 26. 1993. now U.S.
`Pat. No. 5.550.677. by Kenneth Schofleld and Mark Larson.
`BACKGROUND OF THE INVENTION
`
`This invention relates generally to vision systems for
`vehicles and. more particularly. to rearview vision systems
`which provide the vehicle operator with scenic information
`in the direction rearward of the vehicle. More particularly.
`the invention relates to a rearview vision system utilizing
`image capture devices. such as CMOS imaging arrays and
`the like.
`
`A long-felt need in the art of vehicle rearview vision
`systems has been to eliminate exterior rearview mirrors by
`utilizing image capture devices. such as cameras. in com-
`bination with dashboard displays. This would be beneficial
`because it would reduce wind drag on the vehicle. wind
`noise and vehicle weight. Furthermore. rearview mirrors
`protrude a substantial distance from the side of the vehicle,
`which makes maneuvering in tight spaces more diflicult.
`Image capture devices are capable of positioning in a greater
`variety of locations on the vehicle. providing more flexibil-
`ity of vehicle styling. It is further expected that camera
`systems would greatly reduce the blind spots to the sides and
`rear of the vehicle common with vehicles equipped with
`conventional rearview mirror systems. The driver cannot
`perceive vehicles. objects. or other road users in such blind
`spots without turning his or her body. which interferes with
`forward-looking visual activities.
`Camera-based rearview vision systems for vehicles have
`not obtained commercial acceptance. One difliculty with
`proposed systems has been that they present a large amount
`of visual
`information in a manner which is diflicult to
`comprehend. This difliculty arises from many factors. In
`order to significantly reduce blind spots. multiple image
`capture devices are typically positioned at various locations
`on the vehicle. The image of an object behind the equipped
`vehicle is usually captured by more than one image capture
`device at a time and displayed in multiple images. This may
`confuse the driver as to whether more than one object is
`present. When multiple image capture devices are posi-
`tioned at different longitudinal locations on the vehicle.
`objects behind the vehicle are at different distances from the
`image capture devices. This results in diflerent image sizes
`for the same object. This eifect is especially noticeable for
`laterally extending images. such as a bridge. highway cross-
`walk marldngs.
`the earth’s horizon. and the like. Such
`images are at difierent vertical angles with respect to the
`image capture devices. This results in different vertical
`positions on the display causing the elongated image to
`appear disjointed.
`A camera system provides a monocular view of the scene.
`compared to the binocular. or stereoscopic. View obtained
`when the scene is viewed through a rearview mirror. This
`makes the ability to judge distances in a camera system a
`problem. This effect is most noticeable at distances close to
`the vehicle whae stereoscopic imaging is relied upon exten-
`sively by the driver in judging relative locations of objects.
`Therefore. known camera systems fail to provide to the
`driver important information where that information is most
`needed—at small separation distances from surrounding
`objects.
`
`2
`Another difliculty with camera systems is that. in order to
`provide a sufficient amount of information.
`the camera
`system typically presents the driver with a greatly increased
`field of view. This improves performance by further reduc-
`ing blind spots at the side and rear of the vehicle. However.
`an increased field of view is often obtained by utilizing a
`wide-angle lens which introduces distortion of the scene and
`further impairs the ability of the driver to judge distances of
`objects displayed. The problem with such distortion of the
`scene is that the driver must concentrate more on the display
`and take a longer time to interpret and extract the necessary
`information. This further distracts the driver from the pri-
`mary visual task of maintaining awareness of vehicles and
`other objects in the vicinity of the driven vehicle.
`
`SUMMARY OF THE INVENTION
`
`20
`
`25
`
`30
`
`35
`
`45
`
`50
`
`55
`
`65
`
`The present invention is directed towards enhancing the
`interpretation of visual information in a rearview vision
`system by presenting information in a manner which does
`not require significant concentration of the driver or present
`distractions to the driver. This is accomplished according to
`the invention in a rearview vision system having at least two
`image capture devices positioned on the vehicle and directed
`rearwardly with respect to the direction of travel of the
`vehicle. A display is provided for images captured by the
`image capture devices. The display combines the captured
`images into an image that would be achieved by a single
`rearward-looking camera having a view unobstructed by the
`vehicle. In order to obtain all of the necessary information
`of activity. not only behind but also along side of the vehicle.
`the virtual camera should be positioned forward of the
`driver. The image synthesized from the multiple image
`capture devices may have a dead space which corresponds
`with the area occupied by the vehicle. This dead space is
`useable by the driver’s sense of perspective in judging the
`location of vehicles behind and along side of the equipped
`vehicle.
`
`The present invention provides techniques for synthesiz-
`ing images captured by individual. spatially separated.
`image capture devices into such ideal image. displayed on
`the display device. This may be accomplished according to
`an aspect of the invention by providing at least three image
`capture devices. At least two of the image capture devices
`are side image capture devices mounted on opposite sides of
`the vehicle. At least one of the image capture devices is a
`center image capture device mounted laterally between the
`side image capture devices. A display system displays an
`image synthesized from outputs of the image capture
`devices. The displayed image includes an image portion
`from each of the image capture devices. The image portion
`from the center image capture device is vertically com-
`pressed.
`It has been discovered that such vertical compression
`substantially eliminates distortion resulting from the spatial
`separation between the cameras and can be readily accom-
`plished. In an illustrated embodiment. the image compres-
`sion is carried out by removing selective ones of the scan
`lines making up the image portion. A greater number of lines
`are removed further away from the vertical center of the
`image.
`The compression of the central image portion produces a
`dead space in the displayed image which may be made to
`correspond with the area that would be occupied by the
`vehicle in the view from the single virtual camera.
`Preferably. perspective lines are included at lateral edges of
`the dead space which are aligned with the direction of travel
`
`Valeo Exhibit 1007_010
`
`Valeo Exhibit 1007_010
`
`

`
`5,670,935
`
`3
`of the vehicle and. therefore. appear in parallel with lane
`markings. This provides visual clues to the driver’s sense of
`perspective in order to assist in judging distances of objects
`around the vehicle.
`
`image
`According to another aspect of the invention.
`enhancement means are provided for enhancing the dis-
`played image. Such means may be in the form of graphic
`overlays superimposed on the displayed image. Such
`graphic overlap may include indicia of the anticipated path
`of travel of the vehicle which is useful in assisting the driver
`in guiding the vehicle in reverse directions. Such graphic
`overlay may include a distance grid indicating distances
`behind the vehicle of objects juxtaposed with the grid.
`These and other objects. advantages. and features of this
`invention will become apparent by review of the following
`specification in conjunction with the drawings.
`
`10
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`FIG. 1 a top plan view of a vehicle having a rearview
`vision system according to the invention;
`FIG. 2 is a side elevation of the vehicle in FIG. 1;
`
`FIG. 3 is a front elevation of a display according to the
`invention;
`
`FIG. 4 the same view as FIG. 1 illustrating an alternative
`embodiment of the invention;
`FIG. 5 is a block diagram of an electronic system accord-
`ing to the invention;
`FIG. 6 is the same view as FIG. 3 illustrating an alternate
`mode of operation of the system;
`FIG. 7 is the same view as FIG. 2 illustrating an alterna-
`tive embodirnent of the invention;
`
`30
`
`FIG. 8 is the same view as FIG. 3 illustrating an alterna-
`tive embodiment of the invention;
`
`FIG. 9 is the same view as FIGS. 1 and 4 illustrating an
`alternative embodiment of the invention;
`FIG. 10 is the same view as FIGS. 3 and 8 illustrating an
`alternative embodiment of the invention; and
`FIG. 11 is a chart illustrating the horizontal row of pixels
`(n1. n2) on which an object will be imaged from two
`longitudinally separated image capture devices as that object
`is spaced at different longitudinal distances from the image
`capture devices.
`
`DESCRIPTION OF THE PREFERRED
`EMBODIMENT
`
`Referring now specifically to the drawings, and the illus-
`trative embodiments depicted therein. a vehicle 10. which
`may be an automobile. a light truck. a sport uti.lity vehicle.
`a van. a bus. a large truck, or the like includes a rearview
`vision system. generally illustrated at 12. for providing a
`driver of the vehicle with a view rearwardly of the vehicle
`with respect to the direction of travel D of the vehicle (FIG.
`1). Vision system 12 includes at least two side image capture
`devices 14 positioned. respectively. on opposite sides of
`vehicle 10 and a center image capture device 16 positioned
`on the lateral centerline of the vehicle. All of the image
`capture devices are directed generally rearwardly of the
`vehicle. Rearview vision system 12 additionally includes an
`image processor 18 for receiving data signals from image
`capture devices 14. 16 and synthesizing. from the data
`signals. a composite image 42 which is displayed on a
`display 20.
`As will be set forth in more detail below. the images
`captured by image capture devices 14. 16 are juxtaposed on
`
`45
`
`50
`
`55
`
`65
`
`4
`
`display 20 by image processor 18 in a manner which
`approximates the view from a single virtual image capture
`device positioned forwardly of the vehicle at a location C
`and facing rearwardly of the vehicle. with the vehicle being
`transparent to the view of the virtual image capture device.
`Vision system 12 provides a substantially seamless pan-
`oramic View rearwardly of the vehicle without duplicate or
`redundant
`images of objects. Furthermore. elongated.
`laterally-extending. objects. such as the earth’s horizon.
`appear uniform and straight across the entire displayed
`image. The displayed image provides a sense of perspective.
`which enhances the ability of the driver to judge location and
`speed of adjacent trailing vehicles.
`Each of side image capture devices 14 has a field of view
`22 and is aimed rearwardly with respect to the vehicle about
`an axis 24 which is at an angle. with respect to the vehicle.
`that is half of the horizontal field of view of the image
`capture device. In this manner. each of the image capture
`devices 14 covers an area bounded by the side of the vehicle
`and extending outwardly at an angle defined by the hori-
`zontal field of view of the respective side image capture
`device. Center image capture device 16 has a horizontal field
`of view 26. which is symmetrical about the longitudinal axis
`of the vehicle. The field of view of each side image capture
`device 14 intersect the field of view of center image capture
`device 16 at a point P which is located a distance Q behind
`vehicle 10.
`
`Rear blind zones 30 are located symmetrically behind
`vehicle 10 extending from the rear of the vehicle to point P.
`Side blind zones 25 located laterally on respecfive sides of
`the vehicle extend rearwardly of the forward field of View 36
`of the driver to the field of view 22 of the respective side
`image capture device 14. An object will not be captured by
`side image capture devices 14 or center image capture
`devices 16 if the object is entirely within one of the blind
`zones 25. 30. In order for an object, such as another vehicle
`V or other road user travelling to the side of vehicle 10. to
`be observed by an operator of vehicle 10. the object must be
`either at least partially within the forward field of view 36 of
`the driver or be captured by image capture devices 14. 16
`and displayed on display 20. FIG. 4 illustrates vehicle 10
`travelling on a three-lane highway having lanes L1. L2. and
`L3 with the vehicle in lane 12. Another vehicle V is shown
`positioned mostly within one of the blind zones 25. but with
`the rearmost portion of the vehicle V extending into field of
`view 22 where the vehicle image will be captured by one of
`side image capture devices 14.
`In the illustrated
`embodiment, vehicle V is a motorcycle travelling in the
`center of lanes L1 or L3 and represents a worst case for
`observing a vehicle travelling at least partially within one of
`the blind zones 25. In order for a portion of vehicle V to be
`extending either forwardly or rearwardly of the respective
`blind zone 25. where the vehicle V may be observed by
`either the forward field of view 36 of the driver or by the
`rearview vision system 12. the field of view 22 of side image
`capture devices 14 must be sufliciently wide to capture a
`portion of vehicle V as illustrated in FIG. 4. Preferably. the
`horizontal field of view 22 of side image capture devices 14
`is no greater than that required to provide suflicient coverage
`which would be in the range of between approximately 55
`degrees and approximately 70 degrees. In the illustrated
`embodiment. the horizontal field of View 22 is 61 degrees.
`In order for a portion of vehicle V to be within a vertical field
`of view 40 of one of side image capture devices 14. the field
`of view should extend to the pavement at a plane M which
`intersects vehicle V (FIG. 2). Preferably. vertical field of
`view 40 is between approximately 60 degrees and approxi-
`
`Valeo Exhibit 1007_011
`
`Valeo Exhibit 1007_011
`
`

`
`5 ,670,935
`
`5
`
`mately 75 degrees. In the illustrated embodiment. vertical
`field of view 40 is 66 degrees.
`A left overlap zone 32 and a right overlap zone 34 extend
`rearward from respective points Pwhere the horizontal fields
`of view of the side image capture devices intersect the field
`of view of center image capture device 16. Overlap zones
`32. 34 define areas within which an object will be captured
`both by center image capture device 16 and one of the side
`image capture devices 14. An object in an overlap zone 32.
`34 will appear on display 20 in multiple image portions in
`a redundant or duplicative fashion. In order to avoid the
`presentation of redundant information to the driver. and
`thereby avoid confusion and simplify the task of extracting
`information from the multiple images or combined images
`on display 20. the object should avoid overlapping zones 32.
`34. In practice. this may be accomplished to a satisfactory
`extent by moving points P away from the vehicle and
`thereby increasing distance Q. It is desirable to increase
`distance Q to a length that will exclude vehicles travelling at
`a typical separation distance behind vehicle 10 from over-
`lapping zones 32. 34. This separation distance is usually a
`function of the speed at which the vehicles on the highway
`are travelling. The faster the vehicles are travelling. the
`further Q should be moved behind vehicle 10 to keep
`overlap zones 32 and 34 outside of the recommended
`vehicle spacing. If. however. the vehicles are travelling at a
`slower speed. then the generally accepted recommendation
`for vehicle spacing decreases and it is more likely that a
`vehicle will be within overlap zone 32. 34. Therefore. the
`distance Q may be selected to accommodate expected
`vehicle spacing for an average driving speed of vehicle 10.
`Distance Q is a function Of the effective horizontal field
`of View 26 of center image capture device 16. As field of
`View 26 decreases. points P move further rearward of the
`vehicle from a distance Q1.
`to a distance Q2. as best
`illustrated in FIG. 4. In order to increase distance Q to
`eliminate redundant and duplicative information displayed
`on display 20 for most driving conditions of vehicle 10. field
`of View 26 is preferably less than 12 degrees. In the
`illustrated embodiment, field of view 26 is between 6 and 8
`degrees. Alternatively. distance Q may be dynamically
`adjusted according to some parameter. such as the speed of
`vehicle 10. This would allow Q to be greater when the
`vehicle is travelling at a faster speed. where vehicle sepa-
`ration tends to be larger. and vice versa. Field of view 26
`may be adjusted by utilizing a selective presentation of
`pixels of the captured image in the displayed image.
`Referring to FIG. 3. image display device 20 displays a
`composite image 42 made up of a left image portion 44. a
`right image portion 46. and a center image portion 48. Each
`image portion 44-48 is reversed from the image as captured
`by the respective image capture device 14. 16 utilizing
`conventional techniques. These techniques include reading
`the image in reverse with the image capture device. writing
`the image in reverse to display device 20. or reversing the
`image in image processor 18. Left image portion 44 isjoined
`with central image portion 48 at a boundary 50. Central
`image portion 48 is joined with right image portion 46 at a
`boundary 52. As may best be seen in FIG. 3. the image
`portions at boundaries 50 and 52 are continuous whereby
`composite image 42 is a seamless panoramic view rear-
`wardly of the vehicle. As also is apparent from FIG. 3.
`central image portion 48 is narrower than either left image
`portion 44 or right image portion 46. This is a result of
`reducing the horizontal field of View 26 of center image
`capture device 16 sufficiently to move points P. and thus
`overlap zones 32 and 34. a sufficient distance behind vehicle
`
`6
`10 to reduce redundant and duplicative images between
`image portions 44-48. Composite image 42 provides a clear
`image. which avoids confusion and simplifies the task of
`extracting information from the multiple image portions
`44-48. As also may be seen by reference to FIG. 3. display
`20 may additionally include indicia such as the readout of a
`compass 54. vehicle speed 56. turn signals 58. and the like
`as well as other graphical or video displays. such as a
`navigation display. a map display. and a forward-facing
`vision system. In this manner. rearview vision system 12
`may be a compass vision system or an information vision
`system.
`In the embodiment of rearview vision system 12 having a
`dynamically adjusted value of distance Q.
`the spacing
`between boundaries 50 and 52 will dynamically adjust in
`sequence with the adjustment of distance Q. Thus. as overlap
`zones 32. 34 move further away from the vehicle; for
`example. in response to an increase in speed of the vehicle.
`boundary lines 50 and 52 will move closer together and vice
`versa.
`In this manner. composite image 42 is dynamic.
`having image portions of dynamically adaptive sizes.
`Display 20 is of a size to be as natural as possible to the
`driver. This is a function of the size of the display and the
`distance between the display and the driver. Preferably. the
`displayed image simulates an image reflected by a rearview
`mirror. As such. the size of display 20 is approximately the
`combined areas of the three rearview mirrors (one interior
`mirror and two exterior mirrors) conventionally used with
`vehicles. As best seen by reference to FIG. 2. display 20 is
`preferably positioned within the driver’s physiological field
`of view without obstructing the view through the wind-
`shield. It is known that the driver’s field of view. with the
`
`head and eyes fixed forward. extends further in a downward
`direction than in an upward direction. Display 20 could be
`located above the vertical view through the windshield
`wherein the display may be observed at the upward portion
`of the driver’s field of view. However. the position for the
`display illustrated in FIG. 2 is preferred wherein the display
`is within the lower portion of the driver’s field of view.
`Display 20. in the illustrated embodiment. is a flat panel
`display. such as a back-lit liquid crystal display. a plasma
`display. a field emission display. or a cathode ray tube.
`However. the synthesized image could be displayed using
`other display techniques such as to provide a projected or
`virtual image. One such virtual display is a heads-up display.
`The display may be mounted/attached to the dashboard.
`facia or header. or to the windshield at a position conven-
`tionally occupied by an interior rearview mirror.
`Although various camera devices may be utilized for
`image capture devices 14. 16. an electro-optic. pixelated
`imaging array.
`located in the focal plane of an optical
`system. is preferred. Such imaging array allows the number
`of pixels to be selected to meet the requirements of rearview
`vision system 12. The pixel requirements are related to the
`imaging aspect ratio of the respective image capture devices.
`which. in turn. are a function of the ratio of the vertical-to-
`horizontal field of view of the devices. as is well known in
`the art. In the illustrated embodiment. the imaging aspect
`ratio of side image capmre devices 14 is 2:1 and the image
`aspect ratio of central image capture device 16 is variable
`down to 0.l:l. Such aspect ratio will produce images which
`will not typically match that of commercially available
`displays. A commercially available display may be used.
`however. by leaving a horizontal band of the display for
`displaying alpha-numeric data. such as portions of an instru-
`ment cluster. compass display. or the like. as illustrated in
`FIG. 3.
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`65
`
`Valeo Exhibit 1007_012
`
`Valeo Exhibit 1007_012
`
`

`
`5,670,935
`
`7
`
`In the illustrated embodiment. image capture devices 14.
`16 are CMOS imaging arrays of the type manufactured by
`VLSI Vision Ltd. of Edinburgh. Scotland. which are
`described in more detail in co-pending U.S. patent applica-
`tion Ser. No. 08/023.918 filed Feb. 26. 1993. by Kenneth
`Schofield and Mark Larson for an AUTOMATIC REAR-
`VIEW MIRROR SYSTEM USING A PHOTOSENSOR
`ARRAY. the disclosure of which is hereby incorporated
`herein by reference. However. other pixelated focal plane
`image-array devices. which are sensitive to visible or invis-
`ible electromagnetic radiation. could be used. The devices
`could be sensitive to either color or monochromatic visible
`radiation or near or far infrared radiation of the type used in
`night-vision systems. Each image capture device could be a
`combination of diflerent
`types of devices. such as one
`sensitive to visible radiation combined with one sensitive to
`
`infrared radiation. Examples of other devices known in the
`art include charge couple devices and the like.
`Preferably.
`image capture devices 14 and 16 are all
`mounted at the same vertical height on vehicle 10. although
`compromise may be required in order to accommodate
`styling features of the vehicle. The horizontal aim of image
`capture devices 14 and 16 is preferably horizontal. However.
`the portion of the image displayed is preferably biased
`toward the downward portion of the captured image because
`significantly less useful information is obtained above the
`horizontal position of the image capture devices.
`Each image-capturing device 14. 16 is controlled by
`appropriate supporting electronics (not shown) located in the
`vicinity of the imaging array such that. when operating
`power is supplied. either an analog or a digital data stream
`is generated on an output signal line supplied to image
`processor 18. The support electronics may be provided
`partially on the image chip and partially on associated
`electronic devices. For each exposure period. a value indica-
`tive of the quantity of light incident on each pixel of the
`imaging array during the exposure period is sequentially
`outputted in a predetermined sequence. typically row-by-
`row. The sequence may conform to video signal standards
`which support a direct view such that. when a scene is
`viewed by an image-capturing device. the image presented
`on a display represents directly the scene viewed by the
`image-capturing devices. However. when looldng forward
`and observing a displayed image of a rearward scene. the
`driver will interpret the image as if it wae a reflection of the
`scene as viewed through a mirror. Objects to the left and
`rearward of the vehicle. as viewed by the rearward-looking
`camera. are presented on the left-hand side of the display
`and vice versa. If this reversal is effected in image processor
`18. it may be by the use of a data storage device. or buifer.
`capable of storing all of the pixel values from one exposure
`period. The data is read out of the data storage device in a
`reversed row sequence. Alternatively.
`the imaging array
`electronics could be constructed to provide the above-
`described reversal at the image-capturing device or at the
`display.
`Data transmission between image capture devices 14. 16
`and image processor 18 and/or between image processor 18
`and display 20 may be by electrically conductive leads or
`fiber-optic cable. It is possible. for particular applications. to
`eliminate image processor 18 and direct drive display 20
`from image capture devices 14. 16 at the pixel level.
`The data streams from image-capturing devices 14. 16 are
`combined in image processor 18 and directly mapped to the
`pixel array of display 20. This process is repeated preferably
`at a rate of at least 30 times per second in order to present
`an essentially real time video image. The image captured by
`
`8
`side image capture device 14 on the right side of the vehicle
`is presented in right image portion 46 and the image from
`side image capture device 14 on the left side of the vehicle
`is displayed on left image portion 44. The image from center
`image capture device 16 is displayed on central image
`portion 48. The three image portions 44-48 are presented in
`horizontal alignment and adjacent to each other: However.
`the composite image may be positioned at any desired
`vertical position in the display 20. It is also possible to
`display image portions 44-48 on separate image devices
`which are adjacent each other.
`In vision system 12. side image capture devices 14 are
`positioned preferably at a forward longitudinal position on
`vehicle 10 and center image capture device 16 is positioned
`at a rearward longitudinal position on the vehicle. As best
`seen by reference to FIG. 7.
`this positioning creates a
`dilference in the vertical angle between each side image
`capture device 14 and center image capture device 16 with
`respect to a fixed location P1 that is a distance D1 behind the
`vehicle. This diflerence in sensing angle will cause each side
`image capture device 14 to image an object located at P. on
`a horizontal row of pixels that is different from the horizon-
`tal row of pixels that center image capture device 16 will
`image the same object. If the image is below the horizontal
`centerline of the image capture device. it will be imaged on
`a lower row of pixels by center image capture device 16 than
`the row of pixels it will be imaged by the side image capture
`devices 14. as illus