(12) Umted States Patent
`(16) Patent N0.:
`US 6,631,316 B2
`
`Stam et al.
`(45) Date of Patent:
`Oct. 7, 2003
`
`U5006631316B2
`
`(54)
`
`(75)
`
`IMAGE PROCESSING SYSTEM TO
`CONTROL VEHICLE HEADLAMPS OR
`OTHER VEHICLE EQUIPMENT
`
`Inventors: Joseph S. Stam, Holland, MI (US);
`Mark W. Pierce, Grandville, MI (US);
`Harold C. Ockerse, Holland, MI (US)
`
`FR
`JP
`W0
`WO
`
`$8
`WO
`
`2726144
`8466221
`8605147
`9735743
`
`3323332
`0022881
`
`4/1996
`6/1996
`9/1986
`10/1997
`
`lgfiggg
`4/2000
`
`(73) Assignee: Gentex Corporation, Zeeland, MI (US)
`
`OTHER PUBLICATIONS
`
`(*) Notice:
`
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`U.S.C.154b b 0d
`.
`( )
`y
`ays
`
`(21) Appl. No.2 09/799,310
`(22)
`Filed'
`Mar 5 2001
`'
`'
`’
`D t
`t'
`65
`Pr'
`P bl'
`m u ”a "m a a
`US 2002/0156559 A1 Oct. 24, 2002
`
`(
`
`)
`
`51
`
`I t. C].7 ............................ G06F 7 00‘ H01L 27 00
`n
`/
`’
`/
`)
`(
`(52) us. Cl.
`....................................... 701/36; 250/2081
`(58) Field of Search ............................ 701/36; 359/111;
`250/2081
`
`(56)
`
`DE
`FR
`
`References CitEd
`U S PATENT DOCUMENTS
`.
`.
`3/1953 RabinOW
`2,632,040 A
`3/1958 RabinOW
`29827594 A
`9/1987 Seko et al'
`476927798 A
`1
`:13;133(9) if?“ ettal’ t
`:Jggi’ggg 2
`6/1990 $013153: :teala.
`4’930’742 A *
`10/1990 Seko
`' """"
`4,967,319 A
`7/1991 Macks
`5,036,437 A
`6/1992 Michaels er a1.
`5,124,549 A
`8/1993 Hegyi
`5,235,178 A
`7/1994 Slotkowski et al.
`5,329,206 A
`t'
`d
`t
`.
`L' t
`IS con 1nue on nex page )
`(
`FOREIGN PATENT DOCUMENTS
`2946561
`5/1981
`2641237
`7/1990
`
`248/475 1
`'
`
`1
`
`1
`
`0
`
`(mega Sensor
`
`Christopher M. Kormanyos, SAE Paper No. 980003, pp.
`13—18. No date.
`Franz—Josef Kalze, SAE Paper No. 980005, pp. 23—26. No
`date'
`J.P. Lowenau et al., SAE Paper No. 980007, pp. 33—38. No
`date.
`
`Tohru Simizu et al., SAE Paper No. 980322, pp. 113—117.
`No date.
`
`Primary Examiner—Yonel Beaulieu
`$43,152:??? Age”) 0r Firm—Price> HeneVeld> COOP“
`6
`1
`1 0“
`(57)
`
`ABSTRACT
`
`An imaging system of the invention includes an image array
`sensor including a plurality of pixels. Each of the pixels
`generate a signal indicative of the amount of light received
`on the pixel. The imaging system further includes an analog
`to digital converter for quantizing the signals from the pixels
`into a digital value. The system further includes a memory
`including a plurality of allocated storage locations for stor-
`ing the digital values from the analog to digital converter.
`The number of allocated storage locations in the memory is
`less than the number of pixels in the image array sensor.
`According to another embodiment, an imaging device
`includes an image sensor having a plurality pixels arranged
`in an array; and a multi-layer interference filter disposed
`over said pixel array, said multi-layer interference filter
`being patterned so as to provide filters of different colors to
`neighboring pixels or groups of pixels.
`
`45 Claims, 7 Drawing Sheets
`
`
`
`A103
`
`
`
`ms\
`
`
`
`\1/\ms
`
`
`
`
`
`Mme
`
`l/\
`To Vemcle Eqmpmens
`
`VALEO EX. 1004
`
`VALEO EX. 1004
`
`

`

`U.S. PATENT DOCUMENTS
`
`5,416,318 A
`5,426,294 A
`5,451,822 A
`5,471,515 A
`5,508,592 A
`5,537,003 A
`5,550,677 A
`5,614,788 A
`5,660,454 A
`5,666,028 A
`5,715,093 A
`
`511995 Hegyi
`611995 Kobayashi et al.
`911995 Bechtel et al.
`1111995 Fossum et al.
`411996 Lapatovich et al.
`711996 Bechtel et al.
`811996 Schofield et al.
`311997 Mullins et al.
`811997 Mori et al.
`911997 Bechtel et al.
`211998 Schierbeek et al.
`
`811998 Schofield et al.
`5,796,094 A
`1111998 Stam et al.
`5,837,994 A
`5,841,126 A
`1111998 Fossum et al.
`711999 Stam et al.
`5,923,027 A
`811999 Piscart
`5,942,853 A
`1111999 Bechtel et al.
`5,990,469 A
`1211999 Stam et al.
`6,008,486 A
`6,097,023 A
`812000 Schofield et al.
`6,102,546 A * 812000 Carter ........................ 3591604
`1112000 Beam .......................... 315182
`6,144,158 A
`
`* cited by examiner
`
`

`

`U.S. Patent
`
`Oct. 7,2003
`
`Sheet 1 of 7
`
`US 6,631,316 B2
`
`101
`
`Figure 1
`
`Image Sensor
`
`1
`
`Processor
`
`To Vehicle Equipment
`
`

`

`U.S. Patent
`
`Oct. 7,2003
`
`Sheet 2 of 7
`
`US 6,631,316 B2
`
`Figure 2
`
`

`

`Acquire Image and
`store as rough list of
`segments.
`
`Clean up list of
`segments.
`
`I
`
`Create lists of lights
`from segment list.
`
`Clean up and sort light
`lists, calculating center
`
`Select and correlate
`
`Continue with further
`classification.
`
`Segment List
`Row, Min XI Max X,
`Totar Value,
`Saturated Flag,
`Merge Index
`
`Center X and Y,
`Corresponding Val,
`(X Sum, Y Sum)
`Next lndex
`
`White Light List
`Min X and Y,
`Max X and Y,
`Size, Type,
`Saturated Flag,
`Grand Value,
`Center X and Y,
`Corresponding Val,
`(X Sum, Y Sum)
`Next lndex
`
`

`

`U.S. Patent
`
`O C ~ . 7,2003
`
`Sheet 4 of 7
`
`IMAGE READ
`
`302
`
`INITIALIZE DATA
`
`INCLUDING
`SEGMENT LIST
`AND START
`IMAGE
`GATHERING
`
`I
`
`306
`
`STORE ROW #
`IN CURRENT
`SEGMENT
`
`I
`
`31 2
`
`STORE FIRST
`
`- 1
`
`I
`I
`I
`I
`I
`I
`I
`I
`
`- - - - - - - - - - I
`
`NO
`
`STORE FIRST COLUMN #
`INCREMENT SEGMENT
`INDEX IF NOT A LIMIT
`SET STATE TO FLASE
`
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`L - - - - - - - - - - - - -
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`
`YES b
`
`ADD VALUE TO SUM
`SET FLAG IF SATURATED
`SET STATE TO TRUE
`
`I Figure 4A
`
`v
`STORE LAST COLUMN #
`INCREMENT SEGMENT
`INDEX IF NOT A LIMIT
`SET STATE TO FLASE
`
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`L - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - J
`
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`
`

`

`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`
`I
`
`U.S. Patent
`
`Oct. 7,2003
`
`Sheet 5 of 7
`
`I _ - _ _ _ - _ _ _ _ _ - - - - _ _ _ - - _ - _ _ _ - - - - - - - _ _ - - - - - _ _ _ _ _ - - -
`
`WHITE
`LOOP
`
`330 7
`ADD VALUE TO SUM
`SET FLAG IF SATURATED
`SET STATE TO TRUE
`
`YES
`
`NO
`
`NO
`
`334
`
`v
`STORE LAST COLUMN #
`INCREMENT SEGMENT
`INDEX IF NOT A LIMIT
`SET STATE TO FLASE
`
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I _ _ _ _ _ _ _ _ _ _ _ - _ _ _ _ _ - - - - _ - - - - - -
`
`STORE FIRST
`COLUMN #
`
`STORE FIRST COLUMN #
`INCREMENT SEGMENT
`INDEX IF NOT A LIMIT
`SET STATE TO F U S E
`
`I ' I
`
`I
`
`I I
`I
`I
`I
`I
`
`Figure 4 8
`
`NOT
`DONE
`
`THROUGH ALL
`
`

`

`U.S. Patent
`
`O C ~ . 7,2003
`
`Sheet 6 of 7
`
`/ RECEIVE
`
`/ ZF~YZJ L 4 0 0
`
`Figure 5A
`
`LEFT SOURCE
`INDEX TO 0
`
`NO
`
`ELEMENT TO
`0
`
`414-
`
`CREATE NEW
`SOURCE AND
`ADD CURRENT
`PIXEL
`INFORMATION
`
`415x
`
`SET LEFT
`SOURCE INDEX
`TO NEW SOURCE
`INDEX
`
`430
`
`v
`ADD CURRENT
`PIXEL
`INFORMATION
`TO SOURCE AT
`LEFT SOURCE
`INDEX
`
`432
`
`SET ROW
`ELEMENT TO
`LEFT SOURCE
`
`DONE WITH
`PIXEL
`
`

`

`U.S. Patent
`
`Oct. 7,2003
`
`Sheet 7 of 7
`
`US 6,631,316 B2
`
`Figure 5B
`
`NO
`
`I
`
`ADDCURRENT
`PIXEL
`INFORMATION
`TO SOURCE
`ABOVE SOURCE
`INDEX
`
`422
`
`SET LEFT
`SOURCE INDEX
`TO ABOVE
`SOURCE INDEX
`
`.r424
`
`SET MERGE
`INDEX OF
`SOURCE
`REFERENCEDBY
`MERGER INDEX
`OF ABOVE
`SOURCE
`
`I
`
`425
`RGE -/
`INDEX OF
`SOURCE
`REFERENCED BY
`ABOVE INDEX
`
`PIXEL
`INFORMATION
`TO SOURCE A
`LEFT SOURCE
`INDEX
`
`7
`A
`SET ROW
`ELEMENT TO
`LEFT SOURCE
`INDEX
`I
`
`

`

`US 6,631,316 B2
`
`1
`IMAGE PROCESSING SYSTEM TO
`CONTROL VEHICLE HEADLAMPS OR
`OTHER VEHICLE EQUIPMENT
`
`2
`images without first storing them to memory, thus reducing
`the total amount of memory required in the system.
`
`5
`
`SUMMARY OF THE PRESENT INVENTION
`The present invention solves the problem of the prior art
`BACKGROUND OF THE INVENTION
`by providing an image processing system, which does not
`require a memory with enough storage hcations to store the
`The present invention generally relates to image process-
`digital grayscale value of every pixel in the image. The
`ing systems, and more particularly, to image processing
`System contains an image sensor array containing a plurality
`systems used to control vehicle equipment, such as vehicle
`10 of pixels, each of which is configured to provide an output
`headlamps, windshield wipers, etc.
`indicative of the amount of light received by the pixel over
`Recently, many vehicular control accessories utilizing
`the exposure time. The 'ystem
`provides an
`image processing systems have been proposed, For example,
`digital (AD) converter to quantize the signal from the pixel
`U.S. Pat. No. 5,837,994 entitled "Control System to Auto-
`into a digital grayscale value. The system also provides a
`matically Dim Vehicle Head Lamps,n commonly assigned
`with the present invention and hereby incorporated by 15 processor in communication with the image sensor and A/D
`the images acquired by the image
`Converter to
`reference, discloses a control system which utilizes an image
`Processor and
`based up0n the
`sensor and a processor to detect the headlamps of other
`processing the images.
`the system
`vehicles at night and automatically control the state of the
`a memory for
`the grayscale
`of some of the
`high beam headlamps of a vehicle, Another example of an
`the processor.
`data used
`image processing system to control vehicle equipment is 20 pixels and for
`providing an image processing system with less
`given in U,S, Pat, No, 5,923,027 entitled "Moisture Sensor
`memory than is needed to store all pixels, the cost and the
`and Windshield Fog Detector,n also commonly assigned
`the image processing system can be reduced.
`with the present invention and hereby incorporated by
`According to one aspect of the present invention, the
`reference. This patent discloses an image processing system,
`which acquires images of a vehicle windshield in order to 25 amount of memory required is reduced by acquiring and
`analyzing only a portion of the image at a given time. This
`detect the presence of rain or fog.
`may be
`acquiring a sing1e
`pixels at
`In each of these systems, as well as several other disclosed
`a time or by acquiring a subwidow of pixels, which is a
`automotive image processing systems (see, for example,
`the pixels.
`U.S. Pat. Nos. 5,765,116,5,675,489, and 5,660,454 and PCT 30 Subset of the window containing
` wo 00/53465), images
`In another aspect of the present invention, a digital image
`published patent ~
`~
`~
`l
`i
`~
`~
`~
`i
`~
`~
`
`N ~ ,
`processing filter is implemented by only storing a most
`are acquired by an image sensor and stored into a memory
`recent group of pixels and performing the filter algorithm on
`in their entirety for subsequent processing. While techni-
`this group of pixels. The most recent group of pixels may be,
`cally very convenient, the use of this amount of image
`memory presents problems when adapting the system for 35 for example, the last few rows of the image received. The
`commercial use, especially in the highly cost-sensitive auto-
`System performs the filter algorithm on the last few acquired
`motive market, ~~~t low cost microcontro~~ers or digital
`rows and discards the oldest row as a new row is acquired.
`signal processors (DSPs) suitable for these types of appli-
`In another aspect of the invention, some of the pixel
`cations are equipped with only a few hundred bytes to a few
`values received by the processor are discarded as they are
`kilobytes of random access memory (RAM) into which the 40 received and only a subset of the values is stored in the
`images can be stored. The processing core of many micro-
`memory. Pixels may be discarded at a uniform rate through-
`controllers is typically available with a variety of RAM
`out the image or at a non-uniform rate. For example, pixels
`may be discarded at a high rate near the periphery of the
`configurations, with the price increasing as the amount of
`image and at a low or zero rate near the center of the image
`memory increases. Therefore, it is advantageous to use a
`microcontroller with the least amount of RAM possible. The 45 to provide a greater resolution near the center of the image.
`use of a small amount of RAM in prior art systems limits the
`In another aspect of the invention, the values of adjacent
`size of the image, which can be stored and thus greatly limits
`pixels may be averaged with one another and a single value
`the resolution.
`stored in memory representing the average of several adja-
`Acommon off-the-shelf image sensor may have 352x288
`cent pixels. The number of pixels averaged with each other
`pixels, known as the CIF format. Storing an entire image so may be uniform throughout the image or may be non-
`uniform. Many pixels may be averaged with each other near
`from this sensor requires approximately 100 kilobytes of
`the periphery of the image and few or zero pixels may be
`RAM-far more than is typically available on a low cost
`microcontroller. Some microcontrollers have the ability to
`averaged with each other near the center of the image to
`provide a greater resolution near the center of the image.
`increase the amount of RAM available by the addition of an
`external memory chip. These microcontrollers must have an 5s
`In another aspect of the invention, objects identified in the
`external memory bus, increasing the pin count and thus the
`image may be extracted from the image and stored as the
`cost and complexity of the microcontroller and the circuit
`image is being acquired. For example, the present invention
`board to which it is attached. The cost of the external
`may be used to identify headlamps of oncoming vehicles in
`memory itself must also be considered, and despite rapid
`order to determine the high beam state of the headlamps of
`memory price declines, this cost is anticipated to remain 60 the controlled vehicle. The present invention can identify the
`significant for some time to come. Finally, if an image must
`presence of an oncoming headlamp in the image as it is
`be transferred to memory before it can be processed, the
`being acquired and store various information about the
`headlamp in an object list.
`total time required to acquire and analyze an image will be
`greater than if the analysis could occur simultaneously with
`In another aspect of the present invention, the image data
`the acquisition.
`65 may be compressed as it is being acquired. This compression
`What is needed is a low cost image processing system to
`is accomplished by many ways including reducing the
`control automotive equipment which is capable of analyzing
`number of quantization levels (bit depth) of a pixel value as
`
`

`

`US 6,631,316 B2
`
`3
`4
`it is being received. Reducing bit depth of the image may be
`lamps of a vehicle. The control system comprises: an image
`array sensor including a plurality of pixels, each of the pixels
`performed either uniformly or non-uniformly across the
`is operable to generate a signal indicative of the amount of
`image and by requantizing either linearly or non-linearly.
`light received on the pixel; an optical system configured to
`This aspect of the present invention is particularly useful for
`image the scene forward of the controlled vehicle onto the
`reducing data rate when the camera is positioned away from 5
`image array sensor; and a control circuit for processing the
`the processor.
`image of the scene obtained from the imaging system and for
`In another aspect of the present invention, image data is
`controlling the brightness of the headlamps in response to
`reduced by storing segments of connected pixels, which
`objects detected in the processed scene, wherein the control
`together make up an object of interest. For example, a string lo circuit generates a segment list identifying segments of
`of connected pixels in one row of an image, all with
`adjacent ones of the pixels that generate a signal having a
`grayscale values above a threshold, may be stored as one
`grayscale value above a threshold as the signals are received
`object. Rather than storing the grayscale values of each
`from the pixels.
`pixel, only the starting and ending pixels and the cumulative
`According
`to another embodiment of the present
`grayscale value of all the pixels are stored. In a color
`invention, an inside rearview mirror assembly for a vehicle
`implementation, the average color of the segment may be
`comprises: a mirror mount adapted to be mounted inside the
`stored, Finally, two dimensional groups of objects may also
`vehicle in a location proximate to or on the front windshield
`be stored, This aspect is particularly useful for greatly
`of the vehicle; a mirror bezel coupled to the mirror mount;
`reducing the memory requirements and transmitting image
`an imaging system
`in the
`a mirror
`data with the least possible overhead, The processing to
`reduce the image to a list of segments may be contained in 20 mounted to the mirror mount and configured to image the
`scene forward of the vehicle; and a control circuit electri-
`the image sensor itself, in a
`processor (ASIC,
`cally
`the imaging system for processing the
`microcontroller, DSP or the like) near the image sensor, or
`image of the scene obtained from the image array sensor and
`in the main processor.
`for performing predetermined function in response to
`image
`the present
`In
`aspect
`objects detected in the processed scene, wherein at least a
`a
`sensor is used to detect color information about an object in " portion of the control circuit is mounted to the mirror mount,
`the scene. In this embodiment, multi-layer thin film inter-
`According to still another embodiment of the present
`ference filters are placed over each pixel in a checkerboard
`invention, an imaging system for a vehicle is provided that
`or mosaic pattern such that adjacent pixels are exposed to
`comprises: an image array sensor; an optical system con-
`different 'pectral bands of light' The
`of multi-la~er thin 30 figured to image the scene forward of the controlled vehicle
`film interference filters allows the implementation of a color
`onto the image array sensor; and a control circuit coupled to
`imaging system in an automotive environment. Typical
`the image array sensor for processing the image of the scene
`polymer color filters would be degraded by direct focusing
`obtained from the image array sensor to control the vehicle
`of the sun onto the array which will occur when the vehicle
`headlamps in response to objects detected in the processed
`is traveling or is parked such that the camera is in direct view
`35 scene, the control circuit further processes the scene
`of the sun.
`obtained from the image array sensor to perform at least one
`To achieve these and other aspects and advantages, the
`of the following functions: (a) to generate a collision avoid-
`imaging system
`the present
`an image
`ance warning; (b) to control the speed of the vehicle; and (c)
`array sensor including a plurality of pixels, each of the pixels
`to generate a lane departure indication signal,
`is 'perable to generate a
`indicative of the amount of 40
`These and other features, advantages, and objects of the
`light received On the pixel; an
`digita1
`for
`present invention will be further understood and appreciated
`quantizing the signals from the pixels into a digital value;
`by those skilled in the art by reference to the following
`and a memory including a plurality of allocated storage
`specification, claims, and appended drawings,
`locations for storing the digital values from the analog to
`BRIEF DESCRIPTION OF THE DRAWINGS
`digital converter, wherein the number of storage locations in 45
`the allocated memory is less than the number of pixels in the
`In the drawings:
`image array sensor.
`FIG. 1 is an electrical circuit diagram in block form
`According to another embodiment of the present
`showing the image processing system 100 of the present
`invention, a control system is provided to control the head-
`invention;
`lamps of a vehicle. The control system comprises: an image 50
`FIG. 2 is a cross-sectiona~ view of a rearview mirror
`array sensor including a plurality of pixels, each of the pixels
`assembly in which the image processing system of the
`is operable to generate a signal indicative of the amount of
`present invention may be implemented;
`light received on the pixel; an optical system configured to
`FIG, 3 is a flow chart illustrating the image reading and
`image the scene forward of the controlled vehicle onto the
`processing portion of the first embodiment that develops the
`image array sensor; an analog to digital converter for 55 segment list;
`quantizing the signals from the pixels into a digital value;
`FIGS, 4A and 4B are a flow chart illustrating the image
`and a control circuit for processing the image of the scene
`reception and processing phase by which an object list is
`obtained from the imaging 'ystem and for
`the
`created in accordance with a first embodiment of the present
`brightness of the headlamps in response to objects detected
`invention; and
`in the processed scene. The control circuit including a 60
`FIGS. 5A and 5B are a block diagram illustrating the flow
`memory including a plurality of allocated storage locations
`of data in accordance with an image processing routine
`for storing the digital values from the analog to digital
`according to a second embodiment of the invention,
`converter, wherein the number of allocated storage locations
`DETAILED DESCRIPTION OF THE PRESENT
`in the memory is less than the number of pixels in the image
`INVENTION
`array sensor.
`Referring to FIG. 1, the image processing system 100 of
`According to yet another embodiment of the present
`invention, a control system is provided to control the head-
`the present invention includes at least one image array
`
`65
`
`

`

`US 6,631,316 B2
`
`6
`5
`The image processing system 100 of the present invention
`sensor 101. The image array sensor may be of a variety of
`types such as a CMOS active pixel sensor, a Charge Coupled
`can be advantageously integrated into a rearview mirror
`Device (CCD), or any other type of electronic image sensor
`assembly 200 as illustrated in FIG. 2, wherein the image
`capable of sensing light in the visible, UV, near-IR or far-IR
`sensor 101 is integrated into an automatic dimming electro-
`spectral bands. The image sensor 101 contains an array of s chromic (EC) mirror subassembly 205, or other variable
`pixels 102 onto which an image of a scene is formed by a
`reflectance mirror assembly. This location provides an unob-
`lens. The pixels are exposed to light over for a re deter-
`structed forward view of a scene forward of the vehicle
`mined exposure time after which a voltage signal is present
`through a region of the windshield 220 of the vehicle that is
`on the pixel indicative of the amount of light received. This
`typically cleaned by the vehicle,s windshield wipers (not
`voltage is then quantitized by the A/D converter 103 result- lo shown), Additionally, mounting the image sensor in the
`ing in a digital grayscale value indicative of the amount of
`mirror
`such as the
`permits sharing of
`light received on the pixel, A/D converters for this purpose
`microcontroller, and light sensors. More
`power
`typically quantize the pixel signals into 256 levels or 8 bits,
`the same ambient light
`may be
`However, different bit depths are frequently used and appli-
`the
`provide an ambient light measurement for
`cable to the present invention. Timing and control circuitry
`104 provides all the signaling and timing necessary to IS dimming mirror function and the headlamp control function.
`operate the image sensor and transmit grayscale values to
`This embodiment is useful for automatic control of
`processor 105. An example of timing and control circuitry
`vehicle headlights, for imaging a region of a windshield 220
`suitable for the present invention is given in U.S. Pat. NO.
`to detect rain, or any other application where a forward view
`5,990,469 entitled "Control Circuit for Image Array Sen-
`of a vehicle is useful to control vehicle equipment,
`sors" to Bechtel et al., commonly assigned with the present 20
`Referring to FIG, 2, image sensor 101 is mounted within
`invention and hereby incorporated by reference.
`a rearview mirror mount 207, which is mounted to the
`Images acquired by image sensor 101 are transmitted to
`vehicle
`220, ~h~ rearview mirror mount 207
`the processor lo5 Over bus lo7. The processor lo5 may be
`provides an opaque enclosure for image sensor 101, An
`of a variety of types such as a microcontroller, digital signal
`infrared filter 206 may be mounted over a hole 208 in the
`processor (DSP), field programmable gate array (FPGA), 25
`rearview mirror mount 207, as is shown. Light from the
`application specific integrated circuit (ASIC), or the like, An
`208 and infrared
`appropriate processor for the implementation of several
`be imaged passes
`Scene
`206 and impinges up0n a lens 201. Lens 201 forms an
`aspects of the present invention is a microcontroller of the
`image of the forward scene onto image array 101. Image
`MCS912 family available from Motorola. The bus 107 can
`also be of a variety of types, for example bus 107 may be a 30 Sensor 101 is mounted onto a camera circuit board 202.
`Camera circuit board 202 is mounted to rearview mirror
`three wire serial interface which communicates image expo-
`mount 207 using mounting brackets 212. The mounting
`sure instructions from processor 105 to the timing and
`control circuit 104 and communicates image data from
`brackets may be implemented using any suitable
`image sensor 101 to processor 105, such as is described in
`construction, such as metal brackets; plastic brackets, which
`the '469 patent. Alternatively, bus 107 may be a parallel bus
`can be formed either integrally with housing 207 or as
`for communicating image data with image control param- 35 separate components; mechanical fasteners that engage
`camera circuit board 202; or the like. Separate brackets can
`communicated on the same lines or separate signal
`lines. Bus 107 could also carry analog signals such as the
`be attached using an adhesive, metal fasteners, or other
`common NTSC video signal. Bus 107 could be implemented
`mechanical fastening means, I~~~~ sensor 101 is thus
`by using a wireless network, by using a MOST bus, using an
`attached to, and held stationary by, rearview mirror mount
`1EEE-1394 bus, a CAN bus, Or
`using a bus conforming 40 207, which is securely attached to the vehicle windshield
`to the AMI-C specification. In this case, the conversion from
`220 or the vehicle roof by conventional means,
`analog to digital signals would occur on the processor side
`As also shown in FIG. 2, a mirror circuit board 203 may
`of bus 107.
`be provided in mirror housing body 210 (i.e., the mirror
`prior art systems, images received by processor 105
`On which processor circuit lo5 may be
`would be stored to memory 106 in their entirety. In the 45
`lo5 and image
`lol are
`present invention, image data may be processed as the image
`by a bus 107, which is attached to camera circuit board 202
`data is being received by processor 105 and the results of
`214.
`this processing may be stored in memory 106. Memory 106
`by
`of a
`Bus 107 may be constructed as a multi-wire cable, which
`may be integral to processor 105 or external to the processor.
`The present invention provides the advantage of performing so provides bus signals as well as power, ground, and clock
`signals to camera circuit board 202. This cable may be
`many of the analysis tasks necessary to control vehicle
`formed from conventional multi-conductor wire, shielded
`equipment utilizing the amounts of memory which are
`cable, or as a flex circuit; the last of which may be especially
`typically integrated with many common microcontrollers
`advantageous if bus 107 is a parallel bus requiring multiple
`and DSPs.
`I~~~~ sensor 101, AID converter 103, and timing and 55 connections. Processor 105 may alternatively be positioned
`0" Camera circuit board 202 (or integrated into the image
`control circuitry 104 are preferably made integral on the
`sensor) and another cable could be connected to mirror
`same monolithic device. Optionally, processor 105 andlor
`circuit board 203 if mirror assembly 200 contains additional
`memory 106 may also be made integral with image sensor
`electronic functions, such as electrochromic auto dimming
`101, A/D converter 103, and timing and control circuit 104,
`Alternatively, any or all of the above components may be 60 mirror, a compass, etc.
`made as separate devices or any of the above components
`Another method to reduce the complexity of the bus cable
`107, particularly if the bus 107 is a parallel bus, is to position
`may be made integral with any other component as is
`another processor 105' or logic circuit on camera circuit
`convenient for the specific implementation of the invention.
`board 202 while main processor 105 is remotely located
`Finally, components other than those listed or multiple
`instances of the components listed either may be present as 65 such as in the mirror body or elsewhere in the vehicle.
`Communication between the image sensor, second
`discrete components or combined integrally with other com-
`ponents.
`processor, and main processor may be serial or parallel. In
`
`

`

`US 6,631,316 B2
`
`7
`a preferred embodiment, communication between the image
`sensor and the second processor 105' is parallel and com-
`munication between the two processors is serial. Optionally,
`second processor 105' may perform some of the processing
`functions described hereinafter. The use of a second proces-
`sor may facilitate the use of an off-the-shelf image sensor
`which otherwise may have too many control and data signals
`to conveniently communicate over a cable between the body
`210 and mount 207 of a mirror assembly 200.
`
`Headlamp control and rain sensing may also be accom-
`plished using two separate cameras in the mirror mount but
`using only one processor. In this case, both cameras could
`share the same bus to communicate with the main processor.
`This allows substantially different optics optimized for each
`application to be used with each image sensor.
`
`A pixel grayscale value is represented by Val(x,y), where
`x and y are the coordinates of the current pixel. The value of
`the pixel in a new image created by applying the filter kernel
`to an existing image is:
`
`A*Val(x-l,y-1)+B*Val(x,y-1)+C*Val(x+l,y-1)+D*Val(x-l,y)+
`E*Val(x,y)+F*Val(x+l,y)+G*Val(x-l,y+l)+H*Val(x,y+l)+
`I*val(x+l,y+l)
`
`In an embodiment of the present invention, the required
`allocated memory necessary to process images from image
`sensor 101 is reduced by analyzing only particular regions or
`windows of the image at a time. By acquiring only a
`particular subwindow at a given time, only enough allocated
`memory to store that particular subwindow is required. Each
`subwindow is acquired and processed individually until the
`entire image area, or whatever subset of the entire area is
`desired, is processed. The subwindow may be of a variety of 40
`sizes depending on the application. The subwindow may be
`a single or row, a single column, or any rectangular or square
`region of pixels.
`
`35
`
`As an example, a high pass filter can be implemented by
`setting the coefficients A, C, G & I to 0; D, B, F & H to -1;
`Information determined by processor 105 or decisions
`and E to 4. Other filters, which can be implemented with
`made by processor lo5 may be communicated with
`20 kernels using more complex procedures, include edge filters
`wiring
`wiring harness
`Over a
`such as the Sobel or Roberts edge filters. Finally, the use of
`harness 108 may be implemented in a variety of ways
`a 3x3 kernel is only exemplary. Larger kernels or non-square
`including a dedicated point-to-point signal to the controlled
`kernels are also applicable.
`In many applications of the present invention, it is not
`vehicle equipment or by the use of a multiplexed vehicle bus
`such as the CAN bus or J18.50 bus. Such a vehicle wiring 25 actually necessary to create a new image by using the filter
`harness 108 may be used as a power and communication link
`but rather to quantify the total high or low frequency
`with vehicle components such as headlamps, windshield
`comPo"ents of an image or subwindow of an image. Alter-
`natively it may be
`detect edges in an image and
`wipers, and other displays or warning or control subsystems
`store only their location or the number of edges rather than
`within the vehicle.
`30 to create a new image. In this case, it is possible to reduce
`the memory required to process images by performing the
`filtering as the image is received. Memory is needed only to
`store one or a few rows of the image at a time. If a 3x3 kernel
`is used, three rows worth of memory can be used to store
`pixel values while the kernel is being executed. After the
`kernel has been executed across the second row of the
`image, a fourth row can be received and the first row can be
`discarded. The result of the kernel being applied to the
`second row can be tallied in