EXHIBIT 1204
`
`VELLACOTT, OLIVER, “CMOS IN CAMERA,” IEE REVIEW PP. 111-114
`
`(MAY 1994)
`
`("VELLACOTT")
`
`TRW Automotive U.S. LLC: EXHIBIT 1204
`PETITION FOR INTER PARTES REVIEW
`OF U.S. PATENT NUMBER 8,599,001
`
`

`

`CMOS
`
`Oliver Vellacott
`explains how a device
`small enough to fit
`inside a car's rear-view
`mirror can be
`programmed to see
`
`single-channel MOS
`in
`technology
`which only doping levels are altered to
`optimise optical performance measures
`such as anti-blooming (preventing the
`charge from optically saturated pixels
`from spilling over to their neighbours).
`Because of
`the specialised process
`needed to implement the sensor array
`digital functions cannot be implemented
`on the same chip. This means that (for
`instance) timing signals for controlling
`exposure and readout of the sensor array
`must be generated by a separate CMOS
`chip.
`WL's approach has been to combine
`
`lk
`
`image sensing with control functions on
`a single CMOS chip. As in previous
`attempts to realise CMOS sensors, each
`pixel
`is formed by extending and
`exposing the source region of a standard
`MOS transistor to make a photodiode
`(Fig. 1). This can be reset and then isolated
`within the array under the control of a
`MOS transistor gate. All pixels in a row
`are reset together.
`Once reset, the reverse-biased photo-
`diode converts incident light to a small
`current, which gradually discharges the
`gate capacitance. The pixel is then read
`by opening the gate, thus connecting the
`photodiode to the MOS transistor drain.
`In each column of the array, the transistor
`drains are connected in common and
`thus only one row of pixels is read at a
`time.
`This structure had been used in
`previous designs, but without success.
`This was because all pixel outuuts were
`gaied
`tkough
`a
`single charge-sense
`ampllher,
`placing
`huge demands on its
`operation: this sense
`amplifier had to give
`a wide dynamic
`range
`from pixel
`charge packets as low
`as 1 fC, yet operating
`at a 6 MHz read
`frequency.
`The Edinburgh
`approach
`gets
`around this problem
`by using a separate
`charge-sense ampli-
`fier at the head of
`each
`column
`of
`pixels. This means
`that the amplifiers
`operate at line rate
`rather than at pixel
`rate.
`
`...................
`...................
`................
`................
`................
`................
`..................
`..................
`.....................
`.....................
`
`Vertical shift register
`
`I
`
`i
`
`I
`
`n 1988, a team of researchers at
`Edinburgh University developed a
`hgerprint matching system. This
`compared a hgerprint captured by a
`charge-coupled device (CCD) image
`sensor with a description held on a smart
`card and gave a padfail result within a
`second. The matching of the live finger-
`print with the reference was performed
`with a hghly parallel application-specific
`integrated circuit, performing some 3
`billion operations per second.
`Despite the technical success of the
`project, in concluding the work, team
`leader Prof. Denyer noted that by far the
`most expensive component of the system
`was the CCD image sensor. In the light of
`this, the team turned its attention to mak-
`ing image-sensing technology cheaper.
`Instead of fabricating sensors using
`the MOSvarient process found in CCD
`sensors, the team tried to realise an image
`sensor using a commercial CMOS pro-
`cess. When"the res-
`ults were presented at
`CICC in 1990, they
`met with near incred-
`ulity; several research
`groups had tried the
`same thing and con-
`cluded that it was not
`technically
`feasible.
`Since then, Denyer
`formed VLSI
`has
`Vision Ltd in Edin-
`burgh to exploit the
`technology; this com-
`pany has grown to
`some 40 people.
`Over the previous
`20 years, CCD tech-
`nology had been
`highly
`refined
`to
`allow quality image
`capture. Some CCD
`sensor manufacturers
`use a variant of
`
`VVL's CMOS imagesensor archltecture
`
`IEE REVIEW MAY 1994
`0 IE€ 1994
`
`111
`
`1204-001
`
`

`

`2 The WL Peach
`camera: 12 V in,
`video out
`
`compensate for these effects. These
`cancel out the process variations and
`thus allow sensitivity down to 0.5 lux,
`matching CCDs. According to WL, there
`is no intrinsic reason why CMOS sensors
`should not be able to perform just as well
`as CCD sensors.
`If this is an achievable aim, single-chip
`CMOS sensors could eventually displace
`the multi-chip CCDs that are the current
`standard. This would result in smaller,
`cheaper, less power-hungry cameras.
`W s Peach camera, which is compar-
`able with low-end CCDs in performance,
`measures only 35 mm across and is quite
`literally '12 V in, video out' (Fig. 2).
`The full potential of this technology
`becomes more apparent when we turn to
`the hgerprint system and similar
`machine-vision tasks. Having developed
`a cheap image sensor, Denyer and his
`team immediately applied the tech-
`nology to their fingerprint system by
`combining everything on one chip
`(Fig. 3), integrating all the functions
`needed
`
`258 x 258 x 8-bit pixel array
`ADC
`to digitise analogue pixel
`outputs
`preprocessing and quantisation to
`form normahsed binary image
`64cell correlator array p e r f o e g
`3 billion operations per second
`
`I
`I
`
`at the CUR speed of 50 Hz.
`The result is the single-chip image
`sensor, which, because of the low power
`consumption of CMOS, consumes just
`200 mW, compared with the 1 W typical
`of CCDs. WL's first CMOS cameras
`could not match the performance of
`CCDs in terms of noise and sensitivity
`m d y because of fixed-pattern noise
`effects arising from process variations
`inherent in unmodified (digital) CMOS
`processes.
`To overcome this, VVL has devised
`several novel techniques that actively
`
`3 Singlechip CMOS fingerprint acqulsition and matching system
`
`IEE REVIEW MAY 1994
`
`Outputs from the sense amplifiers
`are sampled and stored on a row of
`capacitors,
`then multiplexed out
`through an on-chip charge integrator,
`including a sample-and-hold stage. By
`using an analogue multiplexer to switch
`in blanking and synchronisation levels
`at the appropriate times, it is then
`relatively easy to produce the 1 V peak-
`to-peak composite video waveform
`required by the CCIR (International
`Consultative Committee for Radio) and
`EIA (Electronic Industries Association)
`standards.
`The readout of the sensor array is
`controlled by vertical and horizontal
`digital shift registers placed along the
`edges of the array The vertical register
`activates each row, while the horizontal
`regster reads out the pixels within each
`row. Unlike CCDs, the performance of
`the array is insensitive to these control
`signals, which is a significant advantage.
`Exposure control is also implemented
`on-chip, by monitoring the output wave-
`form to determine the appropriate expos-
`ure setting. The length of exposure is
`controlled by varying the pixel reset time
`via the vertical shift register; this allows
`the exposure period to be set in multiples
`of the line readout time.
`By gating this readout signal with a
`pulse that is a multiple of the pixel read-
`out time, it is possible to decrease the
`exposure even further, down to 500 ns.
`This gives a total exposure range of
`40 0001, since the maximum exposure
`time is 20 ms - the time to read out a field
`
`112
`
`1204-002
`
`

`

`4 The imputer,
`a complete
`standalone
`machinevision
`system
`
`post-correlation decision hardware
`16 kbyte R A M cache
`16 kbyte ROM look-up table
`
`This approach could allow single-chip
`implementations of smart cameras at low
`cost, which is not possible using CCD
`technology
`The imputer
`This was all very grahfymg for the
`cause of UK research, but the technology
`remained inaccessible to applications
`developers because of the high enpeer-
`ing costs involved in toohg an ASIC.
`Accordingly VVL then set out to produce
`a completely programmable machine-
`vision system. The result was
`the
`‘imputer’, launched a year ago (Fig. 4)
`Similar in concept to the configuration
`of PCs, the imputer contains a mother-
`card, into which expansion cards can be
`plugged as needed (Fig.5). The mother-
`card contains all the necessary compon-
`ents for most machine-vision applic-
`ations: image sensor, frame grabber,
`microprocessor, framestore and external
`VO. This is implemented on a board little
`larger than a m&t card - 100 x 50 mm -
`and smaller than many CCD cameras
`alone, even though it forms a complete
`machine-vision system.
`One of the limitations of the device is
`its processing power, which consists of
`an 8-bit Intel 8032 microcontroller. How-
`ever, many machine-vision applications
`consist of very simple techniques such as
`line gauges.
`Line gauge techniques treat lines of
`pixels as if they were physical gauges on
`the object being measured and take
`readings accordingly;
`the
`imputer
`mothercard has enough processing
`for
`power
`these applications. For
`example, to measure the height of
`mercury in a thermometer, the imputer
`would measure at which pixel the line
`moves over a greyscale threshold and
`correlate this to a temperature.
`Another apparent limitation of the
`imputer is the sensor resolution, which is
`restricted to 256 x 256 pixels. However, if
`one doubles the resolution in each
`dimension, to 512 x 512 pixels, the
`amount of image processing is quad-
`rupled; this creates a shpng incentive to
`solve applications using the lower resolu-
`tion. To meet those applications that
`
`LEE REVIEW MAY 1994
`
`really cannot be solved at lower resolu-
`tion, W L is now working on a 512 x 512
`pixel imputer.
`The processor is programmed in C
`using IDS, the imputer development
`system, a Windows s o h package. A
`
`full library of machine-vision functions is
`provided
`includmg morphological
`(shape) filters, transforms, correlators,
`convolvers,
`image
`segmentation,
`frequency fltering rotation, reflection
`and logical operators. For high process-
`
`5 More cards may be added as needed for more complex applications
`
`113
`
`1204-003
`
`

`

`Component inspection by imputer
`
`for
`Renishaw plc is a leading supplier of inspect
`co-ordinate measuring machines (CMMs)
`ine
`tools. Touch-trigger tools remain the standard inspection
`technique in these markets. However, for applications
`where it is necessary to probe deformable or twodimen-
`sional components, noncontact measurement is prefer-
`able.
`
`a probe for
`Y, whe
`ignal t
`probe.
`isting
`
`The edgedetection algorithm developed by Renishaw
`samples a number of pixels around the centre of the
`imputer sensor to check whether the probe has cros
`
`an edge; if so, it outputs a trigger signal to the CMM. By
`sampling a collection of pixels, contributions from stray
`and background light are minimised.
`Performance of the edgedetection probe is limited by
`the frameupdate rate of the video sensor (50 Hz) and the
`pixel resolution (50 pm). For example, at a probing speed
`of 10 mm/s and a 50 mm field of view, the theoretical
`error would be 200 pm. A similar performance can be
`amestore card and PC
`however, the imputer
`and cheaper way to
`. 'The investigation of an
`ection on a cmrdinate
`: 'Wealth creation from
`
`technology', DTI/JflT/SERC
`
`ing power, there is an optional plug-in
`coprocessor (based on a Motorola 56002
`DSP), giving a 3000-fold speed improve-
`ment.
`The CMOS image sensor generates its
`own pixel clock, making pixel digitisation
`accurate and allowing an exact correl-
`ation between the physical photosen-
`sitive silicon area and its digtal value in
`memory. This is important for accuracy
`in measurement applications.
`The sensor can be reset to the start of
`the frame by an external source, so that it
`can be synchronised to fast-moving
`objects - a common requirement in
`production-line inspection, where the
`
`analysis of an image can be greatly
`simplihed by catching it at the optimum
`time. For this to be effective, the scene
`must also be strobed at the start of the
`frame.
`Once the image has been captured
`and analysed, the imputer can interact
`with its environment using an RS232
`interface and eight binary I/Os. A binary
`I/O might be used, for instance, as a
`padfail signal for
`the item under
`inspection.
`is only needed during
`The PC
`application development; thereafter, the
`imputer units are left to work unsuper-
`vised and will communicate directly
`
`6 lmputer application: a rear-view-mirror controller ASIC
`
`114
`
`with other machinery to provide the
`required levels of verhcation. This is a
`break from the tradition of machine
`vision tied to I'C or workstation plat-
`forms. An imputer replaces a camera,
`frme grabber, processing board and
`PC/workstation with a single integrated
`arcfutecture, optimised for machine
`vision.
`Field trials
`One of WL's customers is US auto-
`motive
`components manufacturer
`Donnelly Corp. Donnelly has used the
`imputer to develop electro-chromic rear-
`view mirrors, which automatically
`reduce headlamp glare from behind. The
`imputer was housed inside the rear-view
`mirror and positioned to look out the
`rear and sides of the car in a 90"arc, using
`a chip-mounted microlens (Fig. 6).
`The imputer was programmed to
`analyse this image to recognise when
`and where headlamps are present in the
`field of view Based on this information,
`the imputer then dims the rear-view and
`wing mirrors automatically to reduce
`glare to the driver. The dimming is
`controlled by an analogue voltage from
`the imputer, which directly sets the
`chrominance of the mirror.
`Donnelly's system is now undergoing
`field
`trials with car manufacturers.
`Following this, it can be migrated to a
`single ASIC costing less than $10.
`Because of the engineering costs involved
`in tooling an ASIC, h s is obviously only
`viable for volume applications. However,
`it does open up a large market that
`would remain nascent without CMOS
`imaging technolog.
`Acknowledgment
`Research and development of the imputer
`was supported by the DTI and SERC.
`
`IEE REVIEW MAY 1994
`
`1204-004
`
`