`
`Buckinghamshire, United Kingdom, hereby declares that, to the best of its knowledge and
`
`belief, the following document, prepared by one of its translators competent in the art and
`
`conversant with the English and Japanese languages, is a true and correct translation of the
`
`accompanying Japanese Patent Application No. H3-99952 filed on 25 April 1991.
`
`Signed this 10th day of October 2013
`
`
`
`
`
`
`
`C. E. SITCH
`
`Managing Director – UK Translation Division
`
`For and on behalf of RWS Group Ltd
`
`
`
`
`
`
`
`
`
`
`
`
`
`VALEO EX. 1004_001
`
`
`
`(19) Japanese Patent Office
`(JP)
`
`
`
`(11) Patent application
`publication number: H3-99952
`(12) Official Gazette for Unexamined Patents (A)
`(43) Application
`publication date: April
`25, 1991
`Internal File Nos.
`7812-3D
`7626-3D
`7033-5C
`
`(51) Int. Cl.5
`B 60 R 1/00
`21/00
`H 04 N 7/18
`
`Identification
`Nos.
`C
`J
`
`Number of Claims: 1
`
`(Total 5 Japanese
`pages)
`
`Vehicle Surroundings Monitor
`
`Request for
`Examination:
`Not filed
`(54) Title of
`the Invention
`
`
`
`
`(72) Inventor
`
`(21) Application
`No.
`(22) Application
`Date
`Kazunori Noso
`
`Patent Application No.
`H1-234612
`September 12, 1989
`
`Nissan Motor Co., Ltd.
`2, Takara-cho, Kanagawa-
`ku, Yokohama, Kanagawa
`Prefecture
`2, Takara-cho, Kanagawa-
`ku, Yokohama, Kanagawa
`Prefecture
`Hidekazu Miyoshi, Patent Attorney, and one
`other
`
`(71) Applicant Nissan Motor Co.,
`Ltd.
`
`(74) Agent
`
`VALEO EX. 1004_002
`
`
`
`
`
`
`
`
`
`
`Specifications
`1. Title of the Invention
`Vehicle Surroundings Monitor
`2. Patent Claim
`A vehicle surroundings monitor characterized in having
`one or a plurality of cameras installed in a vehicle,
`means for converting images input by said cameras to
`other coordinates by a perspective conversion, means
`for combining said converted images into one image
`related to an image of the vehicle itself, and a
`display for displaying said image to the vehicle
`occupants.
`3. Brief Description of the Invention
`Objectives of the Invention
`Industrial Field of the Invention
`The present invention relates to a device for
`displaying the positions of other vehicles, the states
`of obstructions, and the center line, etc. as
`environment information of the vehicle surroundings to
`the driver by using television (TV) cameras (if an
`object that acquires images is simply referred to as a
`camera, then the type of camera does not matter in
`these Specifications).
`Prior Art
`A conventional vehicle monitor is as shown, for
`example, in Figure 5. This system has a camera 21 set
`up in the back of the vehicle directed towards the rear
`(Figure 5(a)), reverses the obtained image for left-
`right conversion of the image in an image signal
`processor 22 (Figure 5(b)), displays the image on a
`monitor TV 23, and obtains the so-called rearview image
`as in Figure 5(c) for monitoring. The driver can
`determine how to back up the vehicle by looking at this
`image.
`Problems to Be Solved by the Invention
`However, this type of conventional vehicle monitor is a
`device for simply displaying the rearview image as
`shown in Figure 5(c). One camera is inadequate because
`it is difficult to get a feel of the distance between
`
`5
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`VALEO EX. 1004_003
`
`
`
`
`
`
`
`- 3 -
`
`the vehicle and the obstructions and to understand the
`environment surrounding the vehicle. Even if the number
`of cameras is increased, and the number of monitors is
`increased, if there is no awareness about the relative
`positions of the cameras, it is difficult to
`sufficiently understand the environment. Therefore, a
`problem was that suitable driving measures could not be
`obtained. The present invention has the objective of
`providing a vehicle surroundings monitor that enables
`adequate awareness of the relationship between the
`vehicle and the surrounding environment and taking
`suitable driving measures to solve said problems.
`Structure of the Invention
`Means for Solving the Problems
`To achieve the objective, the present invention has one
`or a plurality of cameras installed in a vehicle, means
`for converting images input by said cameras to other
`coordinates by a perspective conversion, means for
`combining said converted images into one image related
`to an image of the vehicle itself, and a display for
`displaying said image to the occupants.
`Operation
`For example, a camera image is converted to an image,
`in plane coordinates and is displayed so that the
`vehicle is at the origin in these coordinates. The
`driver of the vehicle understand the relationship
`between the traveling direction of the vehicle and the
`state of surroundings by looking at the display, and
`can take appropriate measures.
`Working Examples
`A working example of the present invention is described
`below with reference to Figures 1 to 4.
`
`Figure 1 is a plane view of a vehicle equipped with
`cameras of the present invention. First, the structure
`is explained. A plurality of cameras 1-6 is set up so
`that the respective imaging ranges (field of view) 1a-
`6a cover the environment surrounding the vehicle as
`
`5
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`VALEO EX. 1004_004
`
`
`
`
`
`
`
`- 4 -
`
`much as possible. Specifically, two cameras are
`embedded in each of the front and rear bumpers, and one
`in each front directional indicator. Alternatively, the
`cameras may be set in the rear combination lamp instead
`of the rear bumper. If a mirror is placed in front of
`the lens in the camera and a reversed image is input,
`the camera becomes easy to install, and the degrees of
`freedom in modeling of the image increases. The number
`of cameras may be one or any number. Alternately, the
`cameras are positioned to give importance to the road
`surface and are oriented downward somewhat.
`
`Figure 2 is a block diagram showing the structure of a
`processor of the images captured by the cameras in
`Figure 1. In an image converter 7, the images from
`cameras 1 to N are input and converted to other
`coordinates by a perspective conversion, are combined
`into one image by an image display unit 8, then
`displayed on a TV monitor 9 positioned at the driver’s
`seat. Preferably, the image converter 7 has an internal
`high-speed processor capable of parallel processing of
`the images of only N cameras. In the image display unit
`8, the vehicle’s position is simultaneously drawn in a
`diagram. The displayed position of the vehicle is
`offset from the center of the image screen depending on
`signals corresponding to the gear position, the vehicle
`speed, and the operation of the directional indicator.
`The region of the vehicle environment to be viewed is
`widened. Figure 3 is an example of display results of
`these screens, and each is displayed in plane (road
`surface) coordinates. Figure 3(a) shows backing up
`(direction of the arrow) and is a fixed display with
`the vehicle 10 in the upper part in the center. The
`other neighboring vehicle 11 is displayed in a side
`view. Figure 3(b) is the operation of the right
`directional indicator. A lane change (arrow) to the
`right is assumed, and another vehicle 11 behind in the
`same direction can be displayed.
`
`5
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`VALEO EX. 1004_005
`
`
`
`
`
`- 5 -
`
`
`
`
`
`Figure 3(c) is an example when stopped before an
`intersection. At the intersection, there is a wall 12
`and the view is obstructed, therefore, the vehicle
`should stop because there is another vehicle 11.
`Another vehicle 11’ that is behind is displayed.
`
`Figure 4 is a flow chart of the process of the image
`converter 7 in Figure 2. The process of this converter
`is described.
`
`The images of cameras 1-N are input simultaneously to
`the image converter 7, and the images are set to
`Ai(x,y), where i indicates the camera number. Next, the
`perspective conversions (coordinate conversion) of the
`N images are performed in parallel. If the images after
`conversion are set to Bi(x,y) and are represented by
`
`
`where ai, bi, ci, di, ei, fi, gi, hi, and ki are the
`parameters determined by the focal lengths of the
`cameras and the set-up positions of the cameras (angle
`with respect to the vehicle, angle with the road
`surface, etc.). Once the parameters are set, they are
`completely determined and can be determined in advance
`and stored. The perspective conversion converts to the
`road surface (plane) coordinates (center of vehicle is
`the origin, left and right sides of the traveling
`vehicle are along the X axis, the traveling direction
`of the vehicle is the Y axis) from the screen
`coordinates of the camera.
`
`Next, in the image display unit 8, each image converted
`to the coordinates described above is constructed.
`Thus, it is determined in what environment to display
`the vehicle in response to the directional indicator
`
`5
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`VALEO EX. 1004_006
`
`
`
`
`
`
`
`- 6 -
`
`operation, the gear position, and the vehicle speed.
`That coordinate is (Xo, Yo). Next, the monitor image
`C(X,Y) is cleared, and a figure of the vehicle is drawn
`with (Xo,Yo) at the center. Next, a camera conversion
`image Bi(X,Y) (i – 1 to N: number of cameras) is sent
`to C(X,Y). In this region overlapping the field of
`view, the average value of the density is set.
`Specifically, the following is acceptable.
`C(X,Y) =
`
`Bi(X-Xo, Y-Yo)...when C(X,Y) = 0
`
`C(X,Y)...when B(X-Xo, Y-Yo) = 0
`
`1/2(C(X,Y) + Bi(X-Xo, Y-Yo))... otherwise
`where i = 1 – N
`X = 1 – Mx (number of pixels in the horizontal
`direction of the display image)
`Y = 1 – My (number of pixels in the vertical direction
`of the display image)
`
`From the above, the converted image of each camera in
`image C(X,Y) and the drawn vehicle figure can be
`displayed together.
`
`By looking at the display device described above, the
`driver (and other occupants) can behave appropriately
`in the future when backing up as in Figure 3(a), or
`moving to the right lane as in (b), or when passing
`through an intersection with an obstructed view as in
`(c). The conversion coordinates may have an inclined
`axis on the road surface as the reference. The
`coordinates are not limited to the road surface and may
`be, for example, on a plane at some height above the
`road surface.
`
`Furthermore, depending on the gear position or the
`operating state of the directional indicator, the
`region of the surrounding state desired for viewing can
`be widened and displayed by offsetting the display
`position of the vehicle on the TV monitor from the
`
`5
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`VALEO EX. 1004_007
`
`
`
`
`
`
`
`- 7 -
`
`center. In addition, at fast speeds, the image on the
`TV monitor is compressed and displayed, and the display
`region is widened and displayed, thus, more effective
`images can be obtained.
`
`In addition, if a color TV is used to display in color,
`objects can be easily recognized.
`
`In addition, if infrared cameras are used at night, the
`recognition of obstructions becomes easier even if the
`illumination is inadequate.
`Effects of the Invention
`As in the above description, according to the present
`invention, this structure has one or a plurality of
`cameras installed in a vehicle, means for converting
`the images input by the cameras to other coordinates by
`a perspective conversion, means for combining the
`converted images into one image in relation to an image
`of the vehicle, and a display for displaying the image
`to the occupants. Therefore, the relative positions of
`the obstructions, the other vehicles surrounding the
`vehicle, and the center line to the vehicle itself can
`be easily recognized. Thus, the effect obtained is that
`the driver can take appropriate measures.
`
`4. Brief Descriptions of the Drawings
`Figure 1 is a plane view of a vehicle showing an
`example of the camera installation of the present
`invention. Figure 2 is a structural block diagram of
`the device of the present invention. Figure 3 is an
`example of the display results of the TV monitor of the
`present invention. (a) shows the image when backing up;
`(b), when operating the right directional indicator;
`and (c), when stopping. Figure 4 is a flow chart of the
`image processing of the block diagram in Figure 2.
`Figure 5 is a drawing of a conventional example, where
`(a) is an example of an installation diagram of the TV
`
`5
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`VALEO EX. 1004_008
`
`
`
`
`
`
`
`- 8 -
`
`camera; (b), an example of the process structure
`diagram; and (c), an example of an image.
`
`cameras
`1,2,3,4,5,6
`7
`image converter
`8
`image display
`9
`TV monitor
`
`Hidekazu Miyoshi, Patent Attorney
`
`5
`
`10
`
`VALEO EX. 1004_009
`
`
`
`
`
`Figure 1
`
`- 9 -
`
`
`
`
`AA Forward
`1,2,3,4,5,6
`
`Figure 2
`
`5
`
`
`
`cameras
`
`
`
`Camera
`1,2,N
`7
`Image converter
`8
`Image display unit
`9
`TV monitor
`AA
`Gear position
`BB
`Speed
`CC
`Directional indicator operation
`DD
`Vehicle
`
`10
`
`15
`
`VALEO EX. 1004_010
`
`
`
`
`
`Figure 3(a)
`
`- 10 -
`
`
`
`
`Figure 3(b)
`
`
`
`5
`
`Figure 3(c)
`
`
`
`
`
`VALEO EX. 1004_011
`
`
`
`
`
`Figure 4
`
`- 11 -
`
`
`
`
`5
`
`10
`
`15
`
`20
`
`
`
`Input image -> Ai(x,y)
`AA
`Input image -> Ai+1(x,y)
`BB
`Coordinate conversion
`CC
`Image synthesis
`DD
`Determine Xo and Yo from directional indicator and
`EE
`gear
`FF
`Clear the image screen C(X,Y), and draw the
`vehicle at (Xo,Yo)
`GG
`i = 1-N (number of cameras) loop
`HH
`Image scanning x = 1 – Mx } 2 overlapping loops
`
`
`
` y = 1 – My
`II
`C(X,Y) =
`
`Bi(X-Xo, Y-Yo)...when C(X,Y) = 0
`
`C(X,Y)...when Bi(X-Xo, Y-Yo) = 0
`
`(C(X,Y) + Bi(X-Xo, Y-Yo))... otherwise
`JJ
`Does x = Mx and y = My?
`KK
`Is i = N?
`LL
`Display the image C(X,Y).
`
`Figure 5(a)
`
`(Driver’s seat)
`22,23
`AA
`Forward
`
`25
`
`
`
`VALEO EX. 1004_012
`
`
`
`
`
`Figure 5(b)
`
`- 12 -
`
`
`
`
`Camera
`21
`Image signal processor
`22
`Monitor TV
`23
`Figure 5(c)
`
`5
`
`
`
`
`
`
`
`VALEO EX. 1004_013