United States Patent
`
`[19]
`
`[11] Patent Number:
`
`5,670,935
`
`Schofield et aL
`
`[45] Date of Patent:
`
`Sep. 23, 1997
`
`US00567093SA
`
`[54]
`
`REARVIEW VISION SYSTEM FOR VEHICLE
`INCLUDING PANORAMIC VIEW
`
`4,987,357
`5,001,558
`
`111991 Mfiaki.
`311991 Burleyetal.
`
`175]
`
`Inventors :
`
`Kenneth Schofield. Holland; Mark L.
`Larson. Grand Haven; Keith J. Vedas.
`Coopersville. all of Mich.
`
`[73]
`
`12 1]
`
`[22]
`
`[63]
`
`[5 1]
`[521
`
`[5 3]
`
`[S6]
`
`Assignee: Donnelly CDl'p0l'Ili0II.. Holland. Mich.
`
`App]. No.: 445,527
`
`Filed:
`
`May 22, 1995
`
`Related US. Application Data
`
`Continuation-in-part of Ser. No. 23,918. Feb. 26. 1993, Pat.
`No. 5.550.671
`
`Int. Cl.‘
`U.S. C1.
`
`Field of Search
`
`B601) 1190; G09F 9100
`3401461; 34131435; 3401903;
`3481118; 3481148
`3401435. 436.
`3401901. 903. 461: 348.1118. 148
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`(List continued 011 next page.)
`FOREIGN PKFENT DOCUMENTS
`
`.
`
`1111992 European Pat. Off.
`05134'16Al
`211937
`2585991
`France .
`811992
`261285?
`Fnmoe .
`91992
`2613499
`711984
`3211-8511A1
`Germany .
`311980
`55039843
`Japan .
`1211983
`58-209635
`Japan .
`617261275 of 1985
`Japan .
`62422844
`611987
`Japan .
`6212248’?
`61198?
`Japan .
`30061192
`311991
`Japan .
`40245886
`911992
`Japan .
`50000638
`111993
`Japan .
`934037
`811963
`United Kingdom .
`2l3T5?3 1011984
`United Kingdom .
`224418?
`1111991 United Kingdom .
`9255539
`1111992 United Kingdom.
`OTHER PUBLICKIIONS
`
`Kobe. Geny. “Hypnotic Wiza.rd.ry! (interior clectronics)."'
`Automotive Industries. vol. 169. No. 5. p. 60. published
`May. 1989. Relevant section is entitled Instrumentation.
`
`Primary E.ranr1'm.-1~——Jefl’ery I-lofsass
`Assistant E.x:a:m‘ner—TiJnofl1y Edwards. Jr.
`Attorney, Agent, or F1’:-m—Van Dyke. Gardner. Li.nn &
`Burldiafl. LLP
`
`[571
`
`ABSTRACT
`
`A rearview vision system for a vehicle includes at least one
`image capture device directed rearwardly with respect to the
`direction of travel of the vehicle. A display system displays
`an image synthesized from output of the image captive
`device. The display system is preferably contiguous with the
`forward field of View of the vehicle drive: at a focal length
`that is forward of the vehicle passenger oompa11InenL A
`plurality of image eapune devices my be provided and the
`display system displays a unitary image synthesized from
`outputs of the image captive devices which approximates a
`rearward-facing view fro1'11a single location. such asforward
`of the vehicle.
`
`54Cl:I1l:ns,7DnwingSheets
`
`.
`
`.
`
`Plait .
`Roseniidd et al. .
`Ohama et a1.
`.
`Miller .
`Myers .
`Gabd er. a1.
`Robison _
`Talley et a1.
`Felix .
`Grey .
`Barton .
`Nishimura et a1.
`1-Iayashi et a1.
`.
`Peterson .
`H0wd1e .
`Michaiopoulos er. al. .
`
`.
`
`3,141,393
`3,689,695
`4.052.712
`4,093,364
`4214266
`4,247,370
`4.277.304
`4.233.314
`4.420.233
`4.626.850
`4.630.109
`4,713,635
`4,731,669
`4.789.904
`4.825232
`4.947.772
`4.372.051
`4.392.345
`4,910,591
`4.937.796
`4.970.653
`
`111964
`911972
`101197?
`611978
`711930
`111981
`711981
`911981
`1211933
`1211986
`1211986
`121198?
`311988
`1211988
`411939
`1111989
`1011989
`111990
`311990
`611990
`1111990
`
`
`
`Magna 2002
`Magna 2002
`Valeo v. Magna
`Valeo V. Magna
`IPR2015-01410
`IPR2015-01410
`
`

`
`
`
`3431113
`
`3431116
`-- 3‘‘3*'”3
`
`340/903
`3401903
`. 3401435
`.. 3401436
`
`5,670,935
`Page 2
`
`.. 3481143
`
`.. 3591896
`3401105
`__ 343,-115
`
`.. 372150
`
`5,341,437
`311994 Nakayatna
`5 55
`Fnkuhara.
`sissfiéi “I23
`5406.395
`411995
`'
`5.410.346
`411995
`5-‘“"»“‘5‘
`5”995
`«
`5339-431
`731995
`5_“115,4-48
`SII996 Nishitani
`5.529.135
`611996 Shaweta].
`5,541,590
`711996
`5.574.443
`1111996
`
`U.s. PATENT DOCUMENTS
`519200 ~99
`5'°95’m Mm
`5.121200
`611992
`5,245,122
`9,193
`5239321
`211994
`5,305,012
`411994
`5Jg1_135
`#1994
`5.325.096
`611994
`5,325,386
`611994
`5331.312
`711994
`
`
`
`

`
`5,670,935
`
`FIG.I
`
`

`
` U.S.Patent
`
`Sep. 23, 1997
`
`Sheet 2 of 7
`
`5,670,935
`
`FIG. 3
`
`

`
`U.S. Patent
`
`Sep. 23, 1997
`
`Sheet 3 of 7
`
`5,670,935
`
`F|G.4
`
`

`
`U.S. Patent
`
`Sep. 23, 1997
`
`Sheet 4 of 7
`
`5,670,935
`
`20
`
`FIG.5
`
` CAPTURE SEE IMAGE
`
`
`
`IMAGE
`
` PROCESSOR
`
` $06IMAGECAPTURE CENTER
`CAPTURE
`
`IMABE
`
`E
`
`*0
`
`E
`
`

`
`

`
`U.S. Patent
`
`Sep. 23, 1997
`
`Sheet 6 of 7
`
`5,670,935
`
`

`
`U.S. Patent
`
`Sep. 23, 1997
`
`Sheet 7 of 7
`
`5,670,935
`
`IT....7.7.9In
`
`92
`
`N I
`
`27.7...T
`
`M
`
`92
`
`2N
`
`T}
`
`.
`
`__.T
`
`I.I_.7.1.
`
`x:..B
`
`FiG. ll
`
`
`
`
`
`
`

`
`5.670.935
`
`1
`REARVIEW VISION SYSTEM FOR VEHICLE
`INCLUDING PANORAMIC VIEW
`
`CROSS-REFERENCE TO RELATE.D
`APPLICKFICIN
`
`This application is a continuation-in-part of patent appli-
`cation Ser. No. 08lO23.9l8 filed Feb. 26. 1993. now U.S.
`Pat. No. 5.550.677. by Kenneth Schofield and Mark Larson.
`BACKGROUND OF THE INVENTION
`
`This invention relates generally to vision systems for
`vehicles and. more particularly. to rearview vision systems
`which provide the vehicle opaator with scenic inftnnation
`in the direction rearward of the vehicle. More particularly.
`the invention relates to a rearview vision system utilizing
`image capture devices. such as CMOS imaging arrays and
`the like.
`
`A long-felt need in the art of vehicle rearview vision
`systems has been to eliminate exterior rearview mirrors by
`utilizing image capture devices. such as cameras. in com-
`bination with dashboard displays. This would be beneficial
`because it would reduce wind drag on the vehicle. wind
`noise and vehicle weight. Furthermore. rcarview mirrors
`protrude a substantial distance from the side of the vehicle.
`which makes maneuvedng in tight spaces more diflicult.
`Image capture devices are capable of positioning in a greater
`variety of locations on the vehicle. providing more flexibil-
`ity of vehicle styling. It is further expected tha.t camera
`systems would greatly reduce the blind spots to the sides and
`rear of the vehicle common with vehicles equipped with
`conventional rearview mirror systems. The driver cannot
`perceive vehicles. objects. or other road users in such blind
`spots without turning his or her body. which inttrferes with
`forward-looking visual activities.
`Camera-based reatview vision systems for vehicles have
`not obtained commercial acceptance. One dificulty with
`proposed systems has been that they jreseut a large amount
`of visual information in a manner which is diflicult to
`comprehend. This difliculty arises from many factors. In
`order to significantly reduce blind spots. multiple image
`captttrc devices are typically positioned at various locations
`on the vehicle. The image of an object behind the equipped
`vehicle is usually captured by more than one image capture
`device at a time and displayed in multiple images. This may
`confuse the driver as to whether more than one object is
`present. When multiple image capture devices are posi-
`tioned nt diflercnt longitudinal locations on the vehicle.
`objects behind the vehicle are at different distances from the
`image capture devices. This results in d.i.fl°e:rent image sizes
`for the same object. This effect is especially noticeable for
`laterally extending images. such as a bridge. highway cross-
`walk marlcings. the earth’s horizon. and the like. Such
`images are at dilferent verliml angles with respect to the
`image capture devices. This results in difierent vertical
`positions on the display causing the elongated image to
`appear disjointed.
`A camera system provides a monocular view of the scene.
`compared to the binocular. or stereoscopic. view obtained
`when the scene is viewed through a rearview mirror. This
`makes the ability to judge distances in a camera system a
`problern. This efiect is most noticeable at distances close to
`the vehicle where stereoscopic imaging is relied upon exten-
`sively by the driver in judging relative locations of objects.
`Therefore. known camera systems fail to provide to the
`driver important information where that information is most
`needed—at small separation distances from stnrounding
`objects.
`
`5
`
`ll)
`
`2
`Another dilficulty with camera systems is that. in order to
`provide a suflicient amount of information.
`the camera
`system typically presents the driver with a greatly increased
`field of view. This improves performance by further reduc-
`ing blind spots at the side and rear of the vehicle. However.
`an increased field of view is often obtained by utilizing a
`wide-angle lens which introduces distortion of the scene and
`further impairs the ability of the driver to judge distances of
`objects displayed. The problem with such distortion of the
`scene is that the driver must concentrate more on the display
`and take a longer time to interpret and extract the necessary
`information. This further distracts the driver from the pri-
`mary visual task of maintaining awareness of vehicles and
`other objects in the vicinity of the driven vehicle.
`
`SUMMARY OF TI-[E INVENTION
`
`The present invention is directed towards enhancing the
`interpretation of visual information in a rearview vision
`system by presenting information in a manner which does
`not require significant concentration of the driver or present
`distractions to the driver. This is accomplished according to
`the invention in arearview vision system having at least two
`image capture devices positioned on the vehicle and directed
`rcarwardly with respect to the direction of travel of the
`vehicle. A display is provided for images captured by the
`image capture devices. The display combines the captured
`images into an image that would be achieved by a single
`rearward-looking camera having a view unobstructed by the
`vehicle. In order to obtain all of the necessary inforrnation
`of activity. not only behind but also along side of the vehicle.
`the virtual camera should be positioned forward of the
`driver. The image synthesized from the multiple image
`capture devices may have a dead space which corresponds
`with the area occupied by the vehicle. This dead space is
`useable by the driver’s sense of perspective in judging the
`location of vehicles behind and along side of the equipped
`vehicle.
`
`The present invention provides techniques for synthesiz-
`ing images captured by individual. spatially separated.
`image capture devices into such ideal image. displayed on
`the display device. This may be accomplished according to
`an aspect of the invention by providing at least three image
`capture devices. At least two of the image capture devices
`are side image capture devices mounted on opposite sides of
`the vehicle. At least one of the image capture devices is a
`center image capture device mounted laterally between the
`side image capture devices. A display system displays an
`image synthesized from outputs of the image capture
`devices. The displayed image includes an image portion
`from each of the image capture devices. The image portion
`from the center image capture device is vertically com-
`pressed.
`It has been discovered that such vertical compression
`substantially eliminates distortion resulting from the spatial
`separation between the cameras and can be readily accom-
`plished. In an illustrated embodiment. the image compres-
`sion is carried out by removing selective ones of the scan
`lines making up the image portion. A greater number of lines
`are removed further away from the vertical center of the
`image.
`‘The compression of the central image pcrtion produces a
`dead space in the displayed image which may be made to
`correspond with the area that would be occupied by the
`vehicle in the view from the single virtual camera.
`Preferably. perspective lines are included at lateral edges of
`the dead space which are aligned with the direction of travel
`
`35
`
`45
`
`55
`
`

`
`5 _.670.935
`
`3
`of the vehicle and. therefore. appear in parallel with lane
`markings. This provides visual clues to the driver's sense of
`perspective in order to assist in judging distances of objects
`around the vehicle.
`
`According to another aspect of the invention. image
`enhancement means are provided for enhancing the dis-
`played image. Such means may he in the form of graphic
`overlays superimposed on the displayed image. Such
`graphic overlap may include indicia of the anticipated path
`of travel of the vehicle which is useful in assisting the driver
`in guiding the vehicle in reverse directions. Such graphic
`overlay may include a distance grid indicating distances
`behind the vehicle of objects juxtaposed with the grid.
`These and other objects. advantages. and features of this
`invention will become apparent by review of the following
`specification in conjunction with the drawings.
`
`BRIEF DESCR.l1’I'lON OF THE DRAWINGS
`
`FIG. 1 a top plan view of a vehicle having a re-arview
`Vision system according to the invention;
`FIG. 2 is a side elevation of the vehicle in FIG. 1;
`
`FIG. 3 is a front elevation of a display according to the
`invention:
`
`FIG. 4 the same view as FIG. 1 illustrating an alternative
`embodiment of the invention;
`FIG. 5 is a block diagram of an electronic system accord-
`ing to the invention;
`FIG. 6 is the same view as FIG. 3 illustrating an alternate
`mode of operation of the system;
`FIG. 7 is the same view as FIG. 2 illustrating an alterna-
`tive embodiment of the invention;
`FIG. 8 is the same view as FIG. 3 illustrating an alterna-
`tive embodiment of the invention;
`FIG. 9 is the same view as FIGS. 1 and 4 illustrating an
`alternative embodiment of the invention:
`
`FIG. 10 is the same view as FIGS. 3 and 8 illustrating an
`alternative embodiment of the invention; and
`FIG. 11 is a chart illustrating the horizontal row of pixels
`(n1. n2) on which an object will be imaged from two
`longitudinally separated image capture devices as that object
`is spaced at d.i.fi’erent longitudinal distances from the image
`capture devices.
`
`DESCRIPTION OF THE PREFERRED
`EMBODIMENT
`
`Referring now specifically to the drawings. and the illus-
`trative embodiments depicted therein. a vehicle 10. which
`may be an automobile. a light truck. a sport utility vehicle.
`avan. a bus.a large truck. ortlie lilreincludes arearview
`vision system generally illustrated at 12. for providing a
`driver of the vehicle with a view rearwanzlly of the vehicle
`with respect to the direction 0! travel D of the vehicle (FIG.
`1}. Vision system 12 includes at least two side image capture
`devices 14 positioned. respectively. on opposite sides of
`vehicle 10 and a center image capture device 16 positioned
`on the lateral centerline of the vehicle. All of the image
`capture devices are directed generally rearwardly of the
`vehicle. Rearview vision system 12 additionally includes an
`image processor 18 for receiving data signals from image
`capture devices 14. 16 and synthesizing. from the data
`signals. a composite image 42 which is displayed on a
`display 20.
`As will be set forth in more detail below. the images
`captI.tred by image capture devices 14. 16 are juxtaposed on
`
`35
`
`45
`
`65
`
`4
`display 20 by image processor 18 in a manner which
`approximates the view from a single virtual image capture
`device positioned forwardly of the vehicle at a location C
`and facing rearwardly of the vehicle. with the vehicle being
`transparent to the view of the virtual image capttlre device.
`Vision system 12 provides a substantially seamless pan-
`oramic view rearwardly of the vehicle without duplicate or
`redundant
`images of objects. Furthermore. elongated.
`laterally-extending. objects. such as the earth’s horizon.
`appear uniform and straight across the entire displayed
`image. ‘Ute displayed image provides a sense of pa-spective.
`which enhances the ability of the driver tojudge location and
`speed of adjacent trailing vehicles.
`Each of side image capture devices 14 has a field of view
`22 and is aimed rearwardly with respect to the vehicle about
`an axis 24 which is at an angle. with respect to the vehicle.
`that is half of the horizontal field of view of the image
`capture device. In this manner. each of the image capture
`devices 14 covers an area bounded by the side of the vehicle
`and extending outwardly at an angle defined by the hori-
`zontal field of view of the respective side image capture
`device. Center image capture device 16 has a horizontal field
`of view 26. which is symmetrical about the longitudinal axis
`of the vehicle. The field of view of each side image capture
`device 14 intersect the field of view of center image capture
`device 16 at a point P which is located a distance Q behind
`vehicle 10.
`Rear blind zones 30 are located symmetrically behind
`vehicle 10 extending from the rear of the vehicle to point B
`Side blind zones 25 located laterally on respective sides of
`the vehicle extend retnvardly of the forward field of view 36
`of the driver to the field of view 22 of the respective side
`image capture device 14. An object will not be captured by
`side image capture devices 14 or center i.I'nage (rapture
`devices 16 if the object is entirely within one of the blind
`zones 25. 30. In order for an object. such as another vehicle
`V or other road user travelling to the side of vehicle 10. to
`be observed by an operator of vehicle 10. the object must be
`either at least partially within the forward field of view 3-6 of
`the driver or be captured by image capture devices 14. 16
`and displayed on display 20. FIG. 4 illustrates vehicle 10
`travelling on a three-lane highway having lanes L1. L2. and
`L3 with the vehicle in lane L2. Another vehicle V is shown
`positioned mostly witl1.i.n one of the blind zones 25. but with
`the rearmost portion of the vehicle V extending into field of
`view 22 where the vehicle image will be captured by one of
`side image capture devices 14.
`In the illustrated
`embodiment. vehicle V is a motorcycle travelling in the
`center of lanes L1 or 1.3 and represents a worst case for
`observing a vehicle travelling at least partially within one of
`the blind zones 25. In orda for a portion of vehicle V to be
`extending either forwardly or rearwardly of the respective
`blind zone 25. where the vehicle V may be observed by
`either the fcrward field of view 36 of the driver or by the
`rearview vision system 12. the field of view 22 of side image
`capture devices 14 must be suiliciently wide to capture a
`portion of vehicle V as illustrated in FIG. 4. Preferably. the
`horizontal field of view 22 of side image capture devices 14
`is no greater than that required to provide sufficient coverage
`which would be in the range of between approximately 55
`degrees and approximately 70 degrees. In the illustrated
`embodiment. the horizontal field of view 22 is 61 degrees.
`In order for a portion of vehicle V to be within a vertical field
`of view 40 of one of side image capttne devices 14. the field
`of view should extend to the pavement at a plane M which
`intersects vehicle V (FIG. 2). Preferably. vertical field of
`view 40 is between approximately 60 degrees and approxi-
`
`

`
`5,670,935
`
`5
`
`mately 75 degrees. In the illustrated embodiment. vertical
`field of view 40 is 66 degrees.
`A left overlap zone 32 and a right overlap zone 34 extend
`rearward from respective points Pwhere the horizontal fields
`of view of the side image capture devices intersect the field
`of view of center image capttne device 16. Overlap zones
`32. 34 define areas within which an object will be eaptllred
`both by center image capture device 16 and one of the side
`image capture devices 14. An object in an overlap zone 32.
`34 will appear on display 20 in multiple image portions in
`a redundant or duplicative fashion. In order to avoid the
`presentation of redundant information to the driver. and
`thereby avoid confusion and simplify the task of extracting
`information from the multiple irn.ages or combined images
`on display 20. the object should avoid overlapping zones 32.
`34. In practice. this may be accomplished to a satisfactory
`extent by moving points P away from the vehicle and
`thereby increasing distance Q. It is desirable to increase
`distance Q to a length that will exclude vehicles travelling at
`a typical separation distance behind vehicle 10 from over-
`lapping zones 32. 3-4. This separation distance is usually a
`function of the speed at which the vehicles on the highway
`are travelling. The faster the vehicles are travelling.
`the
`further Q should be moved behind vehicle 10 to keep
`overlap zones 32 and 34 outside of the recommended
`vehicle spacing. If. however. the vehicles are travelling at a
`slower speed. then the generally accepted recommendation
`for vehicle spacing decreases and it is more likely that a
`vehicle will be within overlap zone 32. 34. Therefore. the
`distance Q may be selected to accommodate expected
`vehicle spacing for an average driving speed of vehicle 10.
`Distance Q is a function Of the etfective horizontal field
`of view 26 of center image capture device 16. As field of
`view 26 decreases. points P move further rearward of the
`vehicle from a distance Q1.
`to a distance Q, as best
`i.Ilustrat.ed in FIG. 4. In order to increase distance Q to
`eliminate redundant and duplicative information displayed
`on display 20 for most driving conditions of vehicle 10. field
`of view 26 is preferably less than 12 degrees. In the
`illustrated embodiment. field of view 26 is between 6 and 8
`degrees. Alternatively. distance Q may be dynamically
`adjusted according to some parameter. such as the speed of
`vehicle 10. This would allow Q to be greater when the
`vehicle is travelling at a faster speed. where vehicle sepa-
`ration tends to be larger. and vice versa. Field of view 26
`may be adjusted by utilizing a selective presentation of
`pixels of the captured image in the displayed image.
`Referring to FIG. 3. image display device 20 displays a
`composite image 42 made up of a left image portion 44. a
`right image portion 46. and a center image portion 43. Each
`image portion 44-48 is revased from the image as captured
`by the respective image capture device 14. 16 utilizing
`conventional techniques. These techniques include reading
`the image in reverse with the image capture device. writing
`the image in reverse to display device 20. or reversing the
`image in image processor 18. Left image portion44is joined
`with ce.ntral image portion 48 at a boundary 50. Central
`image portion 48 is joined with right image portion 46 at a
`boundary 52.. As may best be seen in FIG. 3. the image
`portions at boundaries 50 and 52 are continuous whereby
`composite image 42 is a seamless panoramic view rear-
`wardly of the vehicle. As also is apparent from FIG. 3.
`central image portion 48 is narrower than either left image
`portion 44 or right image portion 46. This is a result of
`reducing the horizontal field of view 26 of center image
`capture device 16 suificziently to move points P. and thus
`overlap zones 32 and 34. a sufficient distance behind vehicle
`
`35
`
`55
`
`65
`
`6
`10 to reduce redundant and duplicative images between
`image portions 44-48. Composite image 42 provides a clear
`image. which avoids confusion and simplifies the task of
`extracting infcmnation from the multiple image portions
`44-48. As also may be seen by referent: to FIG. 3. display
`20 may additionally include indicia such as the readout of a
`compass S4. vehicle speed 56. turn signals 58. and the like
`as well as other graphical or video displays. such as a
`navigation display. a map display. and a forward-facing
`vision system. In this manner. rearview vision system 12
`may be a compass vision system or an information vision
`system.
`In the embodiment of rearview vision system 12 having a
`dynamically adjusted value of distance Q.
`the spacing
`between boundaries 50 and 52 will dynamically adjust in
`sequence with the adjustment of distance Q. Thus. as overlap
`zones 32. 34 move further away from the vehicle; for
`example. in response to an increase in speed of the vehicle.
`boundary lines 50 and 52 will move closer together and vice
`versa. In this manner. composite image 42 is dynamic.
`having image portions of dynamically adaptive sizes.
`Display 20 is of a size to be as natural as possible to the
`driver. This is a function of the size of me display and the
`distance between the display and the driver. Preferably. the
`displayed image simulates an image refiected by a rearview
`mirror. As such. the size of display 20 is approximately the
`combined areas of the three rearview mirrors (one interior
`mirror and two exterior mirrors) conventionally used with
`vehicles. As best seen by reference to FIG. 2. display 20 is
`preferably positioned within the driver's physiological held
`of view without obstructing the view through the wind-
`shield. It is known that the d.river’s field of view. with the
`head and eyes fixed forward. extends further in a downward
`direction than in an upward direction. Display 20 could be
`located above the vertiaal view through the windshield
`wherein the display may be observed at the upward portion
`of the driver's field of view. However. the position for the
`display illustrated in FIG. 2 is preferred wherein the display
`is within the lower portion of the driver's field of view.
`Display 20. in the illustrated embodiment. is a liar panel
`display. such as a back-lit liquid crystal display. a plasma
`display. a field emission display. or a cathode ray tube.
`However. the synthesized image could be displayed using
`other display techniques such as to provide a projected or
`virtual image. One such virtual display is a heads-up display.
`The display may be mountedfattached to the dashboard.
`facia or header. or to the windshield at a position conven-
`tionally occupied by an interior rearview rnincr.
`Although various camera devices may be utilized for
`image capture devices 14. 16. an electro-optic. pixelated
`imaging array.
`located in the focal plane of an optical
`system. is preferred. Such imaging array allows the number
`of pixels to be selected to meet the requirements of rearview
`vision system 12. The pixel requirements are related to the
`imaging aspect ratio of the respective image captttre devices.
`which. in turn. are a function of the ratio of the vertical-to~
`horizontal field of view of the devices. as is well known in
`the art. In the illustrated embodiment. the imaging aspect
`ratio of side image capture devices 14 is 2:1 and the image
`aspect ratio of central image capture device 16 is variable
`down to 0.121. Such aspect ratio will produce images which
`will not typically match that of commercially available
`displays. A commercially available display may be used.
`however. by leaving a horizontal band of the display for
`displaying alpha-numeric data. such as portions of an instru-
`ment cluster. compass display. or the like. as illustrated in
`FIG. 3.
`
`

`
`5 _.670_.935
`
`7
`In the illustrated embodiment. image capture devices 14.
`16 are CMOS imaging arrays of the type manufactured by
`VLSI Vision Ltd. of Edinburgh. Scotland. which are
`described in more detail in co-pending U.S. patent applica-
`tion Ser. No. 08/DQ3.9l8 filed Feb. 26. 1993. by Kenneth
`Schofield and Mark Larson for an Al.l'IDMA'I‘IC REAR-
`VIEW MIRROR SYSTEM USING A PHOFOSENSOR
`ARRAY. the disclosure of whidt is hereby incorporated
`herein by reference. However. other pixelated focal plane
`image-array devices. which are sensitive to visible or invis-
`ible electromagnetic radiation. could be used. The devices
`could be sensitive to either color or monocltrornatic visible
`radiation or near or far infrared radiation of the type used in
`night-vision systems. Each image capture device could be a
`combination of different types of devices. such as one
`sensitive to visible radiation combined with one sensitive to
`infrared radiation. Examples of other devices known in the
`an include charge couple devices and the like.
`Preferably. image capture devices 14 and 16 are all
`mounted at the same vertical height on vehicle 10. although
`compromise may be required in order to accommodate
`styling features of the vehicle. The horizontal aim of image
`capture devices 14 and 16 is preferably horizontal. However.
`the portion of the image displayed is preferably biased
`toward the downward portion of the captured image because
`significantly less useful information is obtained above the
`horizontal position of the image capture devices.
`Each i.mage~capturi.ng device 14. 16 is controlled by
`appropriate supporting electronics (not shown) located in the
`vicinity of the imaging array such that. when opuating
`power is supplied. either an analog or a digital data stream
`is generated on an output signal line supplied to image
`processor 18. The support electronics may be provided
`partially on the image chip and partially on associated
`electronic devices. For each exposure period. a value indica-
`tive of the quantity of light incident on each pixel of the
`imaging array during the cxpostne period is sequentially
`outpntted in a predetermined sequence. typically row-by-
`row. The sequence may conform to video signal standards
`which support a direct view such that. when a scene is
`viewed by an image-capturing device. the image presented
`on a display represents directly the scene viewed by the
`image-capturing devices. However. when looking forward
`and observing a displayed image of a rearward scene. the
`driver will interpret the image as if it were a reflection of the
`scene as viewed through a mirror. Objects to the left and
`rearward of the vehicle. as viewed by the rearward-looking
`camera. are presented on the left-hand side of the display
`and vice versa. If this reversal is effected in image processor
`18. it may be by the use ofa data storage device. or bufier.
`capable of storing all of the pixel values from one exposure
`period. The data is read out of the data storage device in a
`reversed row sequence. Alternatively. the imaging array
`electronics could be constructed to provide the above-
`describcd reversal at the image-capturing device or at we
`display.
`Data transmission between image capture devices 14. I6
`and image processor 18 andfor between image processor 18
`and display 20 may be by electrically conductive leads or
`fiber-optic cable. It is possible. for pmicular applications. to
`eliminate image processor 18 and direct drive display 20
`from image capture devices 14. 16 at the pixel level.
`The data streams from image-capturing devices 14. 16 are
`combined in image processor 18 and directly mapped to the
`pixel array of display 20. ‘This process is repeated preferably
`at a rate of at least 30 times per second in order to present
`an essentially real time video image. The image captured by
`
`10
`
`15
`
`45
`
`55
`
`8
`side image capture device 14 on the right side of the vehicle
`is presented in right image portion 46 and the image from
`side image capture device 14 on the lefi side of the vehicle
`is displayed on left image portion 44. The image from center
`image capture device 16 is displayed on central image
`portion 48. The three image portions 44-48 are presented in
`horizontal alignment and adjacent to each other. However.
`the composite image may be positioned at any desired
`vertical position in the display 20. It is also possible to
`display image ponions 44-48 on separate image devices
`which are adjacent each other.
`In vision system 12. side image capture devices 14 are
`positioned preferably at a forward longitudinal position on
`vehicle 10 and center image mpmre device 16 is positioned
`at a rearward longitudinal position on the vehicle. As best
`seen by reference to FIG.
`'7. this positioning creates a
`dilference in the vertical angle between each side image
`capture device 14 and center image capture device 16 with
`respect to a fixed location P. that is adistance D, behindlhe
`vehicle. This dilference in sensing angle will cause each side
`image capture device 14 to image an object located at P1 on
`a horizontal row of pixels that is dilferent from the horizon-
`tal row of pixels that center image capture device 16 will
`image the same object. If the image is below the horizontal
`cent-saline of the image capture device. it will be imaged on
`a lowerrow ofpixels by center image capture device 16 than
`the row of pixels it will be imaged by the side image capture
`devices 14. as illustrated in FIG. 7. This mismatch between
`horizontal pixel rows of the captured image is finfirerrnore
`a function of the distance of the captured image from the rear
`of the vehicle. This can be understood by reference to FIG.
`11 which presents a chart 90 having a first column 92 of
`pixel lines nl. measured from the array centerline. at which
`an object will be imaged by side image capture device 14
`and a second column 94 of pixel lines n2. measured from the
`array vertical centerline. at which the same object will be
`imaged by center image capture device 16. The result is that
`an object. which is captured by both side and center image
`capttue devices 14. 16. will be vertically disjointed at the
`boundary of the displayed image. if the object is captured by
`more than one image capture device. The amount of dis-
`jointment will be greater closer to the vehicle and less at
`further distances. If the object is elongated in the horizontal
`direction. such as earth's horizon. bridges. or cross-
`marltings on highways. then the object will appear to be
`either broken or crooked.
`
`In order to provide uniform display of laterally elongated
`images. a rearview vision system 12' is provided having a
`central image pcltion 48' which is processed