1
`
`Mercedes-Benz USA, LLC, Petitioner - Ex. 1006
`
`

`
`5,245,422
`
`2
`
`

`
`U.S. Pateht
`
`
`
`
`491D..eS
`
`
`3W1.
`
`
`
`
`Sheet 2 of 16
`
`5,245,422
`
`
`
`
`
`3
`
`

`
`U.S. Patent
`
`
`
`
`Sep. 14, 1993
`
`
`
`
`
`Sheet 3 of 16
`
`
`5,245,422
`
`
`
`
`
`WHEEL ANGLE
`
`
`AND DRIVER
`
`
`
`STEER SENSOR
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`14
`
`
`
`
`
`
`IDENTIHCATION
`
`
`
`SEARCH
`
`AREA
`
`PREDICTOR
`
`
`
`
`
`STEERING
`
`ACTUATOR
`
`
`
`
`
`
`
`
`
`
`
`
`
`srsenwc;
`comnousn
`
`
`54
`
`
`
`42
`
`
`
`
`2 44
`
`
`
`
`' 56
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`AUDIO AND VIDEO
`
`
`
`
`PEDAL
`
`posrnons
`
`
`
`
`
`
`
`ACTIVATION
`
`
`
`
`CRUISE
`
`conmon.
`
`
`
`
`
`
`38
`
`
`
`
`
`
`
`
`INFORMATION *AUTO STEER
`I swncn E swrrcu E
`
`
`DRIVER INPUT
`
`
`
`
`
`
`
`
`
`
`
`
`
`Figure 3. A
`
`4
`
`

`
`
`U.S. Patent
`
`
`
`
`Sep. 14, 1993
`
`
`Sheet 4 of 16
`
`5,245,422
`
`
`
`
`
`
`Figure 2B.
`
`
`
`5
`
`

`
`U.S. Patent
`
`
`
`Sep. 14, 1993
`
`
`
`
`
`Sheet 5 of 15
`
`5,245,422
`
`
`
`
`A.4eFl.UM.F
`
`
`
`6
`
`

`
`U.S. Patent
`
`
`
`Sep. 14, 1993
`
`
`
`
`
`Sheet 6 of 16
`
`
`5,245,422
`
`
`
`
`
`
`
`7
`
`

`
`U.S. Patent
`
`
`
`
`Sep. 14, 1993
`
`
`
`
`Sheet 7 of 16
`
`
`5,245,422
`
`
`
`Figure 4C. 2
`
`
`
`8
`
`

`
`U.S. Patent
`
`
`
`
`Sep. 14, 1993
`
`
`
`
`Sheet 8 of 16
`
`
`5,245,422
`
`
`
`Drlvnr Inlflatu
`
`
`Crubo Comrol
`
`
`
`SCS switch
`
`
`b turned on
`
`
`
`j
`
`
`
`
`
`
`spood
`and smnng
`cub. cgrm-0|
`
`
`Mode
`
`
`
`
`
`
`
`
`Spud
`Cruise Confrol
`
`M060
`
`
`
`
`
`
`
`
`
`
`scs
`
`Audio &\/Idoo
`
`
`
`
`
`
`
`
`
`
`
`Braking or
`Cruise Confrol
`
`
`switch Dbabtod
`
`
`
`
`Broking or
`Cruise Comrol
`
`
`switch Disabled
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`scs
`
`Inmaltzaflon
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`9
`
`

`
`
`
`U.S. Patent
`
`Sep. 14, 1993
`
`
`
`
`
`
`5,245,422
`
`
`
`
`
`
`
`10
`
`

`
`US. Patent
`
`
`
`
`Sep. 14, 1993
`
`
`
`
`Sheet 10 of 15
`
`
`5,245,422
`
`
`
`<f
`t~
`
`
`
`9:
`
`1
`.9
`u.
`
`11
`
`

`
`US. Patent
`
`
`
`
`Sep. 14, 1993
`
`
`
`Sheet 11 of 16
`
`
`5,245,422
`
`
`
`:15
`rs
`
`
`
`9:
`
`1
`.9
`u.
`
`12
`
`

`
`
`
`Uoso Patent
`
`M
`
`
`5017- 14, 1993
`
`
`
`
`Sheet 12 of 16
`
`
`5,245,422
`
`
`
`
`
`(5
`I~
`
`
`
`9: E
`
`’
`u.
`
`13
`
`

`
`U.S. Patent
`
`
`
`Sep. 14, 1993
`
`
`
`
`
`Sheet 13 of 16
`
`
`5,245,422
`
`
`
`14
`
`

`
`U.S. Pateltt
`
`
`
`Sep. 14,1993
`
`
`
`
`Sheet 14 of 16
`
`
`5,245,422
`
`
`
`
`
`
`
`
`
`
`
`
`Compute Gradient Magnitude
`
`
`
`cnd Gracfient Direction
`
`
`
`
`
`
`
`
`
`
`Hypothesize the intersection Point
`
`
`
`Based on the Secrch Area
`
`
`
`
`
`
`
`
`
`Collect Support for Each
`
`
`H - thesized intersection Point
`
`
`
`
`
`
`
`
`Select the Right cnd Left
`
`
`
`
`Intersection Point cnd Conver ent i. Ines
`
`
`
`
`
`
`U - de the secrch Area
`
`
`
`
`
`15
`
`

`
`
`
`
`
`Sep. 14, 1993
`
`
`
`
`Sheet 15 of 16
`
`5,245,422
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`Figure 10.
`
`
`
`16
`
`

`
`
`U.S. Patent
`
`
`
`
`
`Sep. 14, 1993
`
`
`
`
`Sheet 16 of 16
`
`5,245,422
`
`
`
`
`
`
`
`
`
`
`Gtcxient Diredion d pixd locdion (I. I)
`
`
`
`
`Consmunt wilh the mole:
`
`
`
`
`
`
`
`
`Figure 11.
`
`
`
`17
`
`

`
`1
`
`
`SYSTEM AND METHOD FOR AUTOMATICALLY
`
`
`
`
`
`STEERING A VEHICLE WITHIN A LANE IN A
`
`
`
`
`ROAD
`
`
`5,245,422
`
`
`
`30
`
`
`
`BACKGROUND OF THE INVENTION
`
`
`
`Technical Field
`
`
`image processing
`This invention relates to digital
`
`
`
`
`
`
`
`systems, and, more particularly, to systems for automat-
`
`
`
`
`
`
`
`ically controlling the steering of a vehicle.
`
`
`
`
`
`Discussion
`
`The technical literature suggests the desirability of a
`
`
`
`
`
`
`control system for automatically controlling the steer-
`
`
`
`
`
`
`
`ing of a vehicle. Representative examples of some
`
`
`
`
`
`
`
`known approaches are disclosed in European Patent
`
`
`
`
`
`
`Application Nos. EP 0 354 56 A2 filed Aug. 9, 1989 and
`
`
`
`
`
`
`
`EP 0 361 914 A2 filed Sep. 28, 1989, both assigned to
`
`
`
`
`
`
`
`
`
`Honda Giken Kogyo Kabushiki Kaisha, Japanese Ap-
`
`
`
`
`
`
`plication No. 62-97935 and European Patent Applica-
`
`
`
`
`
`
`
`tion No. EP 0 304 042 A2 filed Aug. 17, 1988 assigned
`
`
`
`
`
`
`
`
`
`to Kabushiki Kaisha Toshiba. Briefly, these documents
`
`
`
`
`
`
`
`disclose the general concept of using a video input de-
`
`
`
`
`
`
`
`vice, such as a camera, that is mounted to the vehicle
`
`
`
`
`
`
`
`and a computer processor for processing the image data
`
`
`
`
`
`
`
`
`and providing control signals to mechanisms for con-
`
`
`
`
`
`
`
`
`trolling the steering of the vehicle.
`'
`
`
`
`
`
`
`Generally, the prior art approaches do not appear to
`
`
`
`
`
`
`
`be cost effective. As a result, their implementation in a
`
`
`
`
`
`
`vehicle affordable by the ordinary consumer is not very
`
`
`
`
`
`
`
`practical. One reason for the expense is that most of
`
`
`
`
`
`
`
`
`these techniques process the video input data in a very
`
`
`
`
`
`
`
`
`complex manner. For example, the EP ’9l4 application
`
`
`
`
`
`
`
`utilizes a Hough transform to analyze the image data.
`
`
`
`
`
`
`
`The use of transforms of these types are relatively so-
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`phisticated and difficult to analyze thereby requiring
`expensive computer equipment to perform the analysis
`
`
`
`
`
`
`since an exceedingly large amount of data is required in
`
`
`
`
`
`
`order to perform these transforms.
`
`
`
`
`Most of the known systems continuously analyze all
`
`
`
`
`
`
`
`of the video input data and the majority of their algo-
`
`
`
`
`
`
`
`
`
`
`rithm parameters are either fixed or predetermined. As
`
`
`
`
`
`
`a result, the processor is given the enormous task of
`
`
`
`
`
`
`
`
`
`isolating those smaller areas of interest that contain
`
`
`
`
`
`
`
`
`45
`meaningful image data points. The prior art systems also
`
`
`
`
`
`
`
`
`
`generally require an extensive manual tuning effort for
`
`
`
`
`
`
`
`each specific traffic scene and condition. Even so, there
`
`
`
`
`
`
`
`
`is no high degree of probability that the processor has
`
`
`
`
`
`
`
`
`
`correctly detected the actual lane boundary lines that
`
`
`
`
`
`
`
`
`are often used as criteria for controlling the vehicle
`
`
`
`
`
`
`
`
`steering. This is because there is no good preset criteria
`
`
`
`
`
`
`
`for initiating the processing of the image data associated
`
`
`
`
`
`
`
`
`only with relevant road features. As a result, the proces-
`
`
`
`
`
`
`
`
`sor’s power and resources are often wasted in process-
`
`
`
`
`
`
`
`
`ing image data from scenes which do not actually con-
`
`
`
`
`
`
`
`
`
`tain the lane boundary lines. In addition, the prior art
`
`
`
`
`
`
`
`
`
`
`approaches do not generally embody any mechanisms
`
`
`
`
`
`
`which allow the driver of the vehicle to operate the
`
`
`
`
`
`
`
`
`
`
`automatic steering control system only when traffic
`
`
`
`
`
`
`
`conditions are proper and safe.
`
`
`
`
`
`SUMMARY OF THE INVENTION
`
`
`
`In accordance with the preferred embodiment of the
`
`
`
`
`
`
`
`present invention, a system is provided for automati-
`
`
`
`
`
`
`cally steering a vehicle. Included is a sensor which is
`
`
`
`
`
`
`mounted to the vehicle and generates position informa-
`
`
`
`
`
`
`
`tion about the road in front of the vehicle. The vehicle
`
`
`
`
`
`
`
`
`
`contains a cruise control system that has a switch for
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`2
`
`initiating vehicle speed control. The invention advanta-
`
`
`
`
`
`
`
`geously utilizes the actuation of the cruise control
`
`
`
`
`
`
`
`
`switch to initiate the processing of the sensor informa-
`
`
`
`
`
`
`
`tion and to provide automatic steering control of the
`
`
`
`
`
`
`
`
`vehicle under safe traffic and road conditions. A pro-
`
`
`
`
`
`
`
`
`
`
`
`
`
`grammable processor provides signal processing and
`analyzes the information, while a steering controller
`
`
`
`
`
`
`controls the steering of the vehicle as a function of the
`
`
`
`
`
`
`
`analysis by the processor.
`
`
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`
`
`
`The various advantages of the present invention will
`
`
`
`
`
`
`become apparent to those skilled in the art by reading
`
`
`
`
`
`
`
`the following specification and by reference to the
`
`
`
`
`
`
`
`drawings in which:
`
`
`FIG. 1 is a schematic diagram of a vehicle equipped
`
`
`
`
`
`with an automatic vehicle steering system in accor-
`
`
`
`
`
`
`
`dance with the present invention;
`
`
`
`
`
`FIGS. 2A-2B are schematic diagrams which illus-
`
`
`
`
`
`
`
`trate detection of the lane in the road in front of the
`
`
`
`
`
`
`
`
`
`
`vehicle;
`FIG. 3 is a block diagram which illustrates the system
`
`
`
`
`
`
`
`configuration in accordance with the present invention;
`
`
`
`
`
`
`FIGS. 4A—4C are pictures which illustrate the opera-
`
`
`
`
`
`
`
`tion of the present invention;
`
`
`
`
`FIG. 5 is a llow diagram which illustrates the pro-
`
`
`
`
`
`
`
`cessing steps;
`
`
`FIG. 6 is a schematic diagram illustrating the detec-
`
`
`
`
`
`
`tion and prediction of the lane boundaries in the road in
`
`
`
`
`
`
`
`
`front of the vehicle;
`
`
`
`FIGS. 7A—7C are continued schematic diagrams
`
`
`
`
`
`
`illustrating the detection and prediction of the lane in
`
`
`
`
`
`
`
`
`the road;
`
`
`FIG. 8 is a continued schematic diagram illustrating
`
`
`
`
`the detection and prediction of the lane in the road;
`
`
`
`
`
`
`
`
`FIG. 9 is a flow chart diagram which illustrates the
`
`
`
`
`
`
`
`lane detection algorithm in accordance with the present
`
`
`
`
`
`
`
`invention;
`
`FIG. 10 is a flow chart diagram which illustrates the
`
`
`
`
`
`
`operation of the lane detection algorithm; and
`
`
`
`
`
`
`FIG. 11 is a flow chart diagram which further illus-
`
`
`
`
`
`
`
`trates the operation of the lane detection algorithm.
`
`
`
`
`
`
`
`DETAILED DESCRIPTION OF THE
`
`
`PREFERRED EMBODIMENT
`
`
`Turning now to FIG. I, a vehicle 10 is shown therein
`
`
`
`
`
`
`which illustrates the essential components of the auto-
`
`
`
`
`
`
`
`matic vehicle steering system in accordance with the
`
`
`
`
`
`
`
`present invention. An image input device 12 is mounted-
`
`
`
`
`
`
`to the front portion of the vehicle 10 at a location near
`
`
`
`
`
`
`
`
`the rear view mirror assembly. Such a device may be a
`
`
`
`
`
`
`
`
`video camera of the conventional or infrared kind and is
`
`
`
`
`
`
`
`used to monitor the road geometry and traffic condition
`
`
`
`
`
`
`
`
`in front of the vehicle 10 by providing a plurality of
`
`
`
`
`
`
`
`
`
`
`frames of video images of the road. Image input device
`
`
`
`
`
`
`
`
`12 may be mounted in combination with the rear view
`
`
`
`
`
`
`
`
`mirror assembly or separate therefrom or at any other
`
`
`
`
`
`
`location which adequately monitors the road in front of
`
`
`
`
`
`
`
`the vehicle 10.
`
`
`
`An image digitization electronics and processing unit
`
`
`
`
`
`
`
`14 is shown mounted under the hood of the vehicle 10.
`
`
`
`
`
`
`
`
`
`The processing unit 14 may be one of several standard
`
`
`
`
`
`
`
`
`off the shelf programmable processors capable of pro-
`
`
`
`
`
`
`
`viding image processing. Image digitization and elec-
`
`
`
`
`
`
`
`tronics and processing unit 14 is made up of both hard-
`
`
`
`
`
`
`
`ware and software. The hardware is connected to image
`
`
`
`
`
`
`
`input device 12 and contains all the signal conditioning
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`18
`
`

`
`5
`
`I0
`
`3
`
`electronics. Included in the hardware are image digitiz-
`
`
`
`
`
`
`
`ing frame grabbers for converting each frame of the
`
`
`
`
`
`
`
`
`analog video images to digital signals or pulses, and
`
`
`
`
`
`
`
`
`computer processors for providing digital image pro-
`
`
`
`
`
`
`
`cessing. The software provides control for the image
`
`
`
`
`
`
`
`
`input device 12, image processing for lane detection and
`
`
`
`
`
`
`
`
`
`a predictor for improving the efficiency of the image
`
`
`
`
`
`
`
`
`
`processing function by providing for the necessary
`
`
`
`
`
`
`
`search area.
`
`
`A steering control actuator 16 is mounted on the
`
`
`
`
`
`
`
`vehicle 10. Steering control actuator 16 may be either
`
`
`
`
`
`
`
`hydraulic or electric and controls the steering angle of
`
`
`
`
`
`
`
`the wheels, subject to the manual steering override by
`
`
`
`
`
`
`
`the driver, so that the vehicle is at the desired position
`
`
`
`
`
`
`
`
`
`within the lane in the road when the automatic vehicle
`
`
`
`
`
`
`
`
`steering system is engaged.
`
`
`
`Steering actuator electronics and control unit 18 is
`
`
`
`
`
`
`
`also mounted to the vehicle 10. Steering actuator con-
`
`
`
`
`
`
`
`
`trol unit 18 drives the steering control actuator 16 so
`
`
`
`
`
`
`
`
`
`that the vehicle motion follows the desired path pro-
`
`
`
`
`
`
`
`
`
`vided from the output of the image digitization elec-
`
`
`
`
`
`
`
`
`tronics and processing unit 14.
`
`
`
`
`
`Wheel angle and driver steer sensors 20 are mounted
`
`
`
`
`
`
`
`
`to the vehicle 10. The wheel angle sensor measures the
`
`
`
`
`
`
`
`
`
`
`steering wheel angle. The driver steer sensor measures
`
`
`
`
`
`
`
`
`the driver force applied to the steering wheel to detect
`
`
`
`
`
`
`
`
`driver effort
`in controlling the steering wheel. The
`
`
`
`
`
`
`
`
`detection of a significant driver steer will temporarily
`
`
`
`
`
`
`disengage the steering control actuator 16 so that the
`
`
`
`
`
`
`
`
`automatic vehicle steering function is overridden by
`
`
`
`
`
`
`conventional driver steering.
`
`
`
`A conventional cruise control system 22 is employed
`
`
`
`
`
`
`to provide automatic vehicle speed control of the vehi-
`
`
`
`
`
`
`
`
`cle 10. A manually actuable cruise control switch 26 is
`
`
`
`
`
`
`
`mounted inside the vehicle 10 for engaging the cruise
`
`
`
`
`
`
`
`
`
`control system 22. It is generally assumed that the cruise
`
`
`
`
`
`
`
`
`control system 22 is engaged when the vehicle is under
`
`
`
`
`
`
`proper and safe traffic and road conditions.
`
`
`
`
`
`
`
`An automatic steering switch 24 is also mounted to
`
`
`
`
`
`
`the interior of the vehicle 10. Automatic steering switch
`
`
`
`
`
`
`
`
`24 allows the driver to engage the automatic vehicle
`
`
`
`
`
`
`
`
`
`steering system. In order to engage the automatic vehi-
`
`
`
`
`
`
`
`cle steering system to steer the vehicle 10, the system
`
`
`
`
`
`
`
`
`
`requires that both the cruise control switch 26 and auto-
`
`
`
`
`
`
`
`
`
`matic steering switch 24 be engaged. Cruise control
`
`
`
`
`
`
`
`
`switch 26 and automatic steering switch 24 can also be
`
`
`
`
`
`
`
`configured such that with the cruise control system 22
`
`
`
`
`
`
`
`
`disengaged, engagement of the automatic steering
`
`
`
`
`
`
`switch 24 will also simultaneously engage the cruise
`
`
`
`
`
`
`
`
`control switch 22 which also engages the cruise control
`
`
`
`
`
`
`
`
`system 22, thereby providing engagement of the auto-
`
`
`
`
`
`
`
`matic vehicle steering system. On the other hand, when
`
`
`
`
`
`
`
`the cruise control system 22 or switch 26 is disengaged,
`
`
`
`
`
`
`the automatic steering switch M and the automatic
`
`
`
`
`
`
`
`
`steering control function are both disengaged.
`
`
`
`
`
`
`Two additional system components are included,
`
`
`
`
`
`
`whose location in the vehicle 10 are irrelevant. The first
`
`
`
`
`
`
`
`
`being a sensor and vehicle system interface 64 which
`
`
`
`
`
`
`
`
`includes a standard vehicle speed sensor added to the
`
`
`
`
`
`
`
`standard vehicle equipment, a vehicle power supply
`
`
`
`
`
`
`interface and a standard vehicle cruise system interface.
`
`
`
`
`
`
`
`The vehicle speed sensor may be used for steering con-
`
`
`
`
`
`
`
`
`
`trol purposes to modify controller
`response time
`
`
`
`
`
`
`
`thereby enhancing the operation of the automatic vehi-
`
`
`
`
`
`
`
`cle steering system. The vehicle power supply and
`
`
`
`
`
`
`
`
`cruise control interface may be necessary to connect the
`
`
`
`
`
`
`
`video cruising system to the standard vehicle equipment
`
`
`
`
`
`
`
`to ensure that both systems operate properly.
`
`
`
`
`
`
`
`
`
`4
`
`The second is a driver interface and warning inforrna-
`
`
`
`
`
`
`
`tion center 54 which may consist of audio, visual and
`
`
`
`
`
`
`
`
`
`other sensory interactions. Such devices may inform the
`
`
`
`
`
`
`
`
`driver about performance of the automatic vehicle
`
`
`
`
`
`
`
`steering system to enable the driver to make proper
`
`
`
`
`
`
`
`
`judgment on’ the safety of the driving situation.
`
`
`
`
`
`
`In operation, the driver, while driving the vehicle 10
`
`
`
`
`
`
`
`
`on a road having lanes such as a freeway, may engage
`
`
`
`
`
`
`
`
`the automatic vehicle steering system. During normal
`
`
`
`
`
`
`
`weather and driving conditions, the driver is required to
`
`
`
`
`
`
`
`have engaged both the cruise control switch 26 and
`
`
`
`
`
`
`
`
`
`automatic steering switch 24. With the cruise control
`
`
`
`
`
`
`
`
`system 22 engaged, the driver may engage the auto-
`
`
`
`
`
`
`
`
`
`matic steering switch 24 to engage the automatic vehi-
`
`
`
`
`
`
`
`cle steering system. With the cruise control system 22
`
`
`
`
`
`
`
`
`disengaged, the system may be configured so that en-
`
`
`
`
`
`
`
`gagement of the automatic steering switch will further
`
`
`
`
`
`
`
`cause engagement of the cruise control switch 26 to
`
`
`
`
`
`
`
`thereby allow engagement of the automatic steering
`
`
`
`
`
`
`system. By requiring engagement of the cruise control
`
`
`
`
`
`
`system 22, the system may assume that the vehicle is
`
`
`
`
`
`
`
`
`
`under proper and safe traffic road conditions.
`
`
`
`
`
`
`
`Engagement of the automatic vehicle steering system
`
`
`
`
`
`
`initiates the video input device 12. Video input device
`
`
`
`
`
`
`
`
`
`12 generates a continuous plurality of frames of video
`
`
`
`
`
`
`
`images of the road in front of the vehicle 10. The image
`
`
`
`
`
`
`
`
`
`digitization electronics and processing unit 14 receives
`
`
`
`
`
`
`and analyzes the frames of the video images. In doing
`
`
`
`
`
`
`
`
`so, processing unit 14 converts the analog inputs from
`
`
`
`
`
`
`
`
`each frame to a plurality of digital signals. Processing
`
`
`
`
`
`
`
`unit 14 then analyzes the digital signals and attempts to
`
`
`
`
`
`
`
`
`detect the lane boundaries on both sides of the vehicle
`
`
`
`
`
`
`
`
`10. Furthermore, processing unit 14 analyzes the path
`
`
`
`
`
`
`
`
`and determines the proper directional response needed
`
`
`
`
`
`
`
`to maintain the vehicle 10 in the desired position within
`
`
`
`
`
`
`
`
`the lane.
`
`
`The automatic vehicle steering system utilizes the
`
`
`
`
`
`
`
`processed data to lock on to the lane and steer the vehi-
`
`
`
`
`
`
`
`
`
`cle 10 in a desired position therein. In doing so, the
`
`
`
`
`
`
`
`
`
`processing unit 14 provides a directional control re-
`
`
`
`
`
`
`
`sponse to steering actuator control unit 18 which in turn
`
`
`
`
`
`
`
`directs steering control actuator 16 to steer the vehicle
`
`
`
`
`
`
`
`in the desired direction. Wheel angle and driver steer
`
`
`
`
`
`
`
`
`
`sensors 20 measure the steering wheel angle and fur-
`
`
`
`
`
`
`
`
`
`thermore measure and detect driver effort to override
`
`
`
`
`
`
`
`
`the automatic vehicle steering system. The detection of
`
`
`
`
`
`
`
`a significant driver steer by the driver steer sensor will
`
`
`
`
`
`
`
`
`
`result in temporary disengagement of the steering con-
`
`
`
`
`
`
`trol actuator 16 thereby temporarily disengage the auto-
`
`
`
`
`
`
`
`matic vehicle steering system. This may occur, for ex-
`
`
`
`
`
`
`
`
`ample, when the driver of the vehicle 10 changes lanes.
`
`
`
`
`
`
`
`Once in the new lane the automatic vehicle steering
`
`
`
`
`
`
`
`
`system will be re-engaged to provide steering within the
`
`
`
`
`
`
`
`new lane provided the driver is no longer manually
`
`
`
`
`
`
`
`
`overriding the automatic steering of the vehicle 10.
`
`
`
`
`
`
`
`FIGS. 2A and 2B illustrate the basic geometry in-
`
`
`
`
`
`
`volved for providing images of the road for the auto-
`
`
`
`
`
`
`
`
`
`matic vehicle steering system. Vehicle 10 is shown
`
`
`
`
`
`
`
`within the lane of a road 28 having a left lane boundary
`
`
`
`
`
`
`
`
`34 and a right lane boundary 36. Image input device 12
`
`
`
`
`
`
`
`
`monitors the road geometry and provides a plurality of
`
`
`
`
`
`
`
`frames of video images of the road in front of the vehi-
`
`
`
`
`
`
`
`
`cle 10 such as frame 66.
`
`
`
`
`
`FIG. 3 illustrates the system configuration for the
`
`
`
`
`
`
`
`automatic vehicle steering system. Video input device
`
`
`
`
`
`
`
`12 provides continuous frames of the road in front of the
`
`
`
`
`
`
`
`
`vehicle 10 to image processor 14. Image processor 14
`
`
`
`
`
`
`
`
`performs lane identification 42 within the area specified
`
`
`
`
`
`
`
`
`5,245,422
`
`
`
`
`
`
`
`
`
`
`
`
`
`19
`
`

`
`
`
`
`
`
`
`
`
`35
`
`
`
`5,245,422‘
`5
`
`by the search area predictor 40 and furthermore, a lane
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`centering algorithm 44. Search area predictor 40 pro-
`vides the necessary search area in an efficient manner.
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`The response signal from lane centering algorithm 44 is
`provided to steering controller 18, which in turn con-
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`trols steering actuator 16. Steering actuator 16 adjusts
`the angle of the wheels 60 of vehicle 10 to direct the
`
`
`
`
`
`
`
`
`vehicle 10 in the desired direction.
`
`
`
`
`Wheel angle and driver steer sensors 20 measure the
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`wheel angle and detect conventional driver steering.
`Wheel angle and driver steer sensors 20 are adapted to
`
`
`
`
`
`
`
`
`provide a signal to search area predictor 40. The image
`
`
`
`
`
`
`
`
`processor 14 receives this signal and uses the wheel
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`angle signal to check for a consistent steering angle
`sufficient to allow for the initiation of the system. The
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`wheel angle signal further provides the image processor
`14 with vehicle turning information. As such, the pro-
`
`
`
`
`
`
`
`
`cessor 14 is able to use this information to provide for a
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`better prediction of the lane position. The wheel angle
`and driver steer sensors 20 are further adapted to pro-
`
`
`
`
`
`
`
`
`vide a driver steer signal to steering controller 18 to
`
`
`
`
`
`
`
`
`disengage steering actuator 16 when the driver manu-
`
`
`
`
`
`
`
`ally operates the steering wheel 32 while the automatic
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`vehicle steering system is engaged. A wheel angle sig-
`
`
`
`
`
`
`
`nal is also provided to steering controller 18. Steering
`controller 18 is further adapted to receive inputs from
`
`
`
`
`
`
`steering wheel 32 and steering actuator 16. Further-
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`more, steering controller 18 is adapted to provide sig-
`nals to a warning system 54.
`
`
`
`
`Cruise control switch 26 engages the cruise control
`
`
`
`
`
`
`
`system 22 which is adapted to control vehicle speed 38
`
`
`
`
`
`
`by controlling throttle control 58 which in turn controls
`
`
`
`
`
`
`
`the throttle 60. The cruise control switch 26, vehicle
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`speed 38, automatic steering switch 24 and steering
`wheel 32 are adapted to receive driver inputs 46. Auto-
`
`
`
`
`
`
`
`
`matic steering switch 24 is further adapted to receive
`
`
`
`
`
`
`cruise control inputs from cruise control switch 26.
`
`
`
`
`
`
`
`
`Automatic steering switch 24 in turn communicates
`
`
`
`
`
`
`with steering wheel 32. Cruise control switch 26 further
`
`
`
`
`
`
`
`communicates with pedal positions 56 which in turn 40
`
`
`
`
`
`
`
`controls throttle control 58.
`
`
`
`
`FIGS. 4A~4C are photographs which illustrate the
`
`
`
`
`
`
`operation of the automatic vehicle steering system.
`
`
`
`
`
`
`
`FIGS. 4A—4C illustrate operation of the vehicle 10
`
`
`
`
`
`
`
`
`45
`within the lane boundaries of the road. The automatic
`
`
`
`
`
`
`
`
`steering system maintains the vehicle 10 at the desired
`
`
`
`
`
`
`
`location within the lane, under normal traffic condi-
`
`
`
`
`
`
`
`
`tions. FIG. 4C illustrates the vehicle 10 changing lanes,
`
`
`
`
`
`
`
`whereby the automatic vehicle steering system is tem-
`
`
`
`
`
`
`
`porarily disengaged as long as the driver manually oper-
`
`
`
`
`
`
`
`ates the steering. Once in the desired position of the new
`
`
`
`
`
`
`
`
`
`lane the driver may discontinue manual steering which
`
`
`
`
`
`
`
`
`re-engages the automatic vehicle steering system.
`
`
`
`
`
`
`The flow chart in FIG. 5 illustrates the processing
`
`
`
`
`
`
`
`steps performed by the automatic vehicle steering sys-
`
`
`
`
`
`
`
`tem. The driver of the vehicle 10 initially turns on the
`
`
`
`
`
`
`
`
`cruise control switch 26 to engage the cruise control
`
`
`
`
`
`
`
`system 22 or turns the automatic steering switch 24 to
`
`
`
`
`
`
`engage both the cruise control system 22 and automatic
`
`
`
`
`
`
`
`
`vehicle steering system. With the cruise control system
`
`
`
`
`
`
`
`
`22 engaged and the automatic vehicle steering disen-
`
`
`
`
`
`
`
`
`gaged or not ready to operate, the vehicle maintains
`
`
`
`
`
`
`
`speed control in the cruise control mode unless the
`
`
`
`
`
`
`
`
`
`cruise control system 22 is disengaged. Cruise control
`
`
`
`
`
`
`system 22 may be disengaged by conventional tech-
`
`
`
`
`
`
`
`niques such as applying the brakes or disengaging the
`
`
`
`
`
`
`
`cruise control switch 26 or may be temporarily disen-
`
`
`
`
`
`
`gaged while manually depressing the throttle control
`
`
`
`
`
`
`
`
`6
`
`58. With the cruise control system 22 and the automatic
`
`
`
`
`
`
`
`
`
`vehicle steering switch 24 both engaged, the vehicle 10
`
`
`
`
`
`
`
`locks on to the lane and operates in the speed and steer-
`
`
`
`
`
`
`
`
`
`ing cruise control mode until being disengaged.
`
`
`
`
`
`
`
`The automatic vehicle steering system may be disen-
`
`
`
`
`
`
`
`gaged in several ways. The driver may disengage the
`
`
`
`
`
`
`
`
`vehicle steering system by turning off either the cruise
`
`
`
`
`
`
`
`
`control switch 26 or the automatic steering switch 24.
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`Depressing the brake pedal will further disengage the
`system. Temporary disengagement will result from
`
`
`
`
`
`
`manual driver steer. When the driver depresses the
`
`
`
`
`
`
`
`
`throttle control 58 the cruise control system 22 will be
`
`
`
`
`
`
`
`temporarily overridden, however, the automatic vehi-
`
`
`
`
`
`
`cle steering system will continue to steer the vehicle.
`
`
`
`
`
`
`
`
`When the driver engages the automatic vehicle steer-
`
`
`
`
`
`
`
`ing system, the system initially undergoes an initializa-
`
`
`
`
`
`
`
`tion process. Audio and video information is provided
`
`
`
`
`
`
`
`to the driver of the vehicle 10 which indicates whether
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`the system is ready. During automatic vehicle steering
`system initialization, all that is required of the driver is
`
`
`
`
`
`
`
`that he maintain the vehicle in the desired position be-
`
`
`
`
`
`
`
`tween the lane boundaries of the road.
`
`
`
`
`
`
`FIGS. 6-11 illustrate how processing unit 14 operates
`
`
`
`
`
`
`
`to analyze the frames of road images and predict the
`
`
`
`
`
`
`
`
`
`path of the lane in the road in front of the vehicle 10.
`
`
`
`
`
`
`
`
`
`Processing unit 14 receives a continuous series of frames
`
`
`
`
`
`
`of the road in front of the vehicle 10 from image input
`
`
`
`
`
`
`
`
`
`device 12. Image input device 12 provides frames of
`
`
`
`
`
`
`
`
`images at a rate of thirty frames per second, capable of
`
`
`
`
`
`
`
`providing an adequate response for vehicles travelling
`
`
`
`
`
`
`at normal highway speeds. For higher speeds, the sys-
`
`
`
`
`
`
`
`
`
`tem may require a higher rate of frame speed.
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`The processing unit 14 includes image digitizing
`frame grabbers for receiving each analog input frame
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`from image input device 12 and converting each frame
`to a plurality of digital signals. Processing unit 14 in-
`
`
`
`
`
`
`
`
`cludes computer processors for providing digital pro-
`
`
`
`
`
`
`
`cessing to analyze the digital information provided by
`
`
`
`
`
`
`the image digitizing frame grabbers. Processing unit 14
`
`
`
`
`
`
`
`is further equipped with software for controlling the
`
`
`
`
`
`
`
`
`image input device, image processing for lane detection
`
`
`
`
`
`
`
`
`and a predictor to improve the efficiency of the image
`
`
`
`
`
`
`
`processing function.
`
`
`In order to locate the lane boundaries in the image of
`
`
`
`
`
`
`
`
`a road scene, the processing unit 14 first detects all edge
`
`
`
`
`
`
`
`
`
`points in the image. In doing so, there are certain as-
`
`
`
`
`
`
`
`
`
`sumptions that are made in order to simplify the prob-
`
`
`
`
`
`
`
`
`lem. For an automatic vehicle steering system we first
`
`
`
`
`
`
`
`assume low curvature lane boundaries. In addition, we
`
`
`
`
`
`
`assume that in most situations a pair of boundaries exist.
`
`
`
`
`
`
`
`Finally, it is assumed that the ground is locally level and
`
`
`
`
`
`
`
`
`the images are taken while the car is in the lane of the
`
`
`
`
`
`
`
`
`
`
`
`road. This letter assumption is usually correct because
`
`
`
`
`
`
`
`the driver is likely to engage the cruise control switch
`
`
`
`
`
`
`
`
`22 and/or steering control switch only when the car is
`
`
`
`
`
`
`
`
`travelling between one lane boundaries and the car is
`
`
`
`
`
`
`
`
`usually travelling in a straight line. Under these assump-
`
`
`
`
`
`
`
`tions, the location of the lane in the image can be pre-
`
`
`
`
`
`
`
`
`
`dicted by the predictor 40 based on lane curvature,
`
`
`
`
`
`
`
`
`
`vehicle dynamics and steering inputs.
`
`
`
`
`
`Two main lane boundaries are modeled close to the
`
`
`
`
`
`
`
`
`vehicle using two parallel line segments. The first line
`
`
`
`
`
`
`
`
`
`segment being the tangent
`to the current
`left
`lane
`
`
`
`
`
`
`
`
`
`boundary 34 and the second being tangent to the cur-
`
`
`
`
`
`
`
`
`rent right lane boundary 36. Due to the projective ge-
`
`
`
`
`
`
`
`
`ometry of the image, these two convergent lines must
`
`
`
`
`
`
`
`
`converge at a point in the image called a vanishing point
`
`
`
`
`
`
`
`84.
`
`
`
`50
`
`
`
`60
`
`65
`
`
`
`20
`
`

`
`7
`The best two convergent lines are essentially chosen
`from a set of candidates. Here, however, we will use
`two intersection points 86 and 88. that is, where the left
`convergent line 78 and the right convergent line 80 each
`cross the chosen search area 82 as shown in FIG. 7. The
`use of two intersection points rather than one vanishing
`point allows for the ability to follow the lane in situa-
`tions where one side of a lane boundary is less promi-
`nent than the other or is completely missing.
`Since the location of the intersection points does not
`change much between two continuous frames, an as-
`sumption is made that its location in the current frame
`will be close to that in the previous frame. This fact
`allows for combining road edge detection and intersec-
`tion point determination in one step.
`To select the two best intersection points, the algo-
`rith.m collects evidence supporting each candidate from
`the image. The supporting evidence, coming from the
`pixel level local computation, includes the strength and
`direction of edge points and length of line segments.
`Functions are provided to measure the support of each
`of the evidence and combine them in the performance
`measure that gives confidence in an intersection point.
`The intersection point having the highest confidence is
`selected and the corresponding convergent line is con-
`sidered as the image of the lane boundary. FIG. 8 illus-
`trates the characteristics of such an image. Shown
`therein are edge responses and associated orientation of
`several line samples. It is desirable to obtain the data
`that provides a strong edge response in addition to a
`consistent orientation such as line 90. The overall re-
`sponse is then used to calculate the intersection point
`for that boundary line within a chosen search area 82.
`FIG. 6 illustrates a left convergent line 78 and a right
`convergent line 80 as both pass through the chosen
`search area 82 to obtain the left convergent line inter-
`section point 86 and the right convergent line intersec-
`tion point 88. Left and right convergent lines 78 and 80
`cross at the point known as the vanishing point 84. It is
`most desirable to obtain the intersection of intersection
`points 86 and 88 or vanishing point 84 within the search
`area 82. In order to do so, the system employ a predic-
`tor to continuously adjust the search area as shown in
`FIG. 7. The predictor determines the area in which to
`search. Upon system initialization, the predictor initially
`searches a large area. As the predictor locates the inter-
`section points it is able to adjust to that location and
`search a smaller area, thereby enabling the system to
`operate faster and more efficiently. Upon initialization
`the predictor could be adjusted to monitor a narrower
`area based on various assumpti