EXHIBIT 1053
`
`P.B. DENYER, D. RENSHAW, W. GOUYU, L.M.YING, and S. ANDERSON,
`
`“On-Chip CMOS sensors for VLSI Imaging Systems,”
`
`IFIP Transactions VLSI 91 Halaas & Denyer eds. pp. 157-166 (1992) by Peter
`
`Denyer et al.
`
`TRW Automotive U.S. LLC: EXHIBIT 1053
`PETITION FOR INTER PARTES REVIEW
`OF U.S. PATENT NUMBER 8,599,001
`IPR2015-00436
`
`

`
`IFIP Transactions
`
`A-1
`
`VLSI 91
`
`A. HALAAS
`P. B. DENYER
`Editors
`
`1053-001
`
`

`
`
`VLSI 91
`
`1053-002
`
`1053-002
`
`

`
`
`
`IFIP Transactions A:
`Computer Science and Technology
`
`
`‘.
`{A
`
`International Federation for Information Processing
`
`Technical Committees:
`
`SOFTWARE: THEORY AND PRACTICE (TC2)
`EDUCATION (TC3)
`SYSTEM MODELLING AND OPTIMIZATION (TC7)
`INFORMATION SYSTEMS (TC8)
`I
`RELATICNSHIP BETWEEN COMPUTERS AND SOCIETY (TC9)
`COMPUTER SYSTEMS TECHNOLOGY (TC10)
`SECURITY AND PRoTECTIoN IN INFORMATION PROCESSING SYSTEMS (TC11)
`ARTIFICIAL INTELLIGENCE (TC12)
`HUMAN-COMPUTER INTERAcTIoN (TC13)
`
`FOUNDATIONS OF COMPUTER SCIENCE (SG14)
`
`1053-003
`
`1053-003
`
`

`
`A-1
`
`VLSI 91
`
`
`
`Proceedings of the IFIP TC10/\NG 10.5 International Conference on
`Very Large Scale Integration
`Edinburgh, Scotland, 20-22 August, 1991
`
`Edited by
`
`A. HALAAS
`
`University of Trondheim
`
`Trondheim, Norway
`
`P. B. DENYER
`
`University of Edinburgh
`
`,-'Edinburgh/, Scotland
`
`
`
`2%,asemlifi
`
`19
`
`NORTH-HOLLAND
`
`AMSTERDAM ° LONDON ' NEW YORK 0 TOKYO
`
`1053-OO4
`
`1053-004
`
`

`
`ELSEVIER SCIENCE PUBLISHERS B.V.
`Sara Burgerhartstraat 25
`P.O. Box 211, 1000 AE Amsterdam, The Netherlands
`
`Distributors for the United States and Canada:
`ELSEVIER SCIENCE PUBLISHING COMPANY INC.
`655 Avenue of the Americas
`
`New York, N.Y. 10010, U.S.A.
`
`¢;;~..c£.,
`
`C0/03/M_({)
`T K
`W374
`« I 5 2 S
`
`l“’l°ll
`
`ISBN: 0 444 89019 X
`
`ISSN: 0926-5473
`
`© 1992 IFIP. All rights reserved.
`No part of this publication may be reproduced, stored in a retrieval system or transmitted in any form or by any
`means, electronic, mechanical, photocopying, recording or otherwise, without the prior written permission of
`the publisher, Elsevier Science Publishers B.V., Copyright & Permissions Department, P.0. Box 521, 1000 AM,
`Amsterdam, The Netherlands.
`
`Special regulations for readers in the U.S.A. - This publication has been registered with the Copyright Clearance
`Center Inc. (CCC), Salem, Massachusetts. Information can be obtained from the CCC about conditions under
`which photocopies of parts ofthis publication may be made in the U.S.A. All othercopyright questions, including
`photocopying outside of the U.S.A., should be referred to the publisher, Elsevier Science Publishers B.V., unless
`otherwise specified.
`
`No responsibility is assumed by the publisher or by IFIP for any injury and/or damage to persons or properly
`as a matter of products liability, negligence or otherwise, orfrom any use or operation of any methods, products,
`instructions or ideas contained in the material herein.
`
`pp. 127-136, 177-186, 367-386: Copyright not transferred.
`_
`_
`Printed In The Netherlands
`
`1o53_oo5
`
`‘
`,3
`-..,
`\.?«,_
`
`S:
`l\)
`. pg
`:fl
`
`1053-005
`
`

`
`vii
`
`PREFACE
`
`Ten years ago, IFIP (International Federation of Information Processing) asked
`Professor Sidney Michaelson to establish a working group on "VLSI". The aim of
`this group should be to initiate conferences and workshops all over the world to pro-
`mote the development of VLSI. This has been achieved with great success and the
`biannual series of conferences VLSI 81 through to VLSI 89 has been recognised for
`its high quality. VLSI 91 comes full circle, returning the conference to Edinburgh,
`the venue for the original VLSI 81.
`
`In 1981 Sidney said that "The major problem in VLSI is really the control of com-
`plexity. The hardest part is the control of autonomous yet interacting processes.
`We do not yet have satisfactory techniques for handling that sort of thing, but I
`think the techniques we need to develop are independent of whether you are pro-
`gramming or designing a chip".
`
`This statement is still valid. Even if a decade has passed where the focus in the
`VLSI arena has been on tools of all kinds to assist our imperfect minds and hands,
`we certainly still have a long way to go before we have become Masters of Complex-
`ity.
`
`The papers chosen for this conference by the hard working International Pro-
`gramme Committee reflect the continuing interest in improving design tools and
`the wide range of engineering concerns which surround the effective exploitation
`of VLSI.
`
`Professor Michaelson was the initiator of VLSI 91, a conference that for us also
`would be an opportunity to show him our deep respect in his year of retirement. It
`was a sad moment during our preparations for VLSI 91 in February that we heard
`that Sidney had died. His firm voice ofjustice, and his particular care for the young
`and less privileged will for many of us be strongly linked to his memory.
`These proceedings take a few steps further in our common understanding.
`Professor Arne Halaas,
`Programme Chairman
`
`1053-006
`
`1053-006
`
`

`
`ORGANIZATION
`
`ix
`
`Programme Chairman
`
`_
`A. Halaas
`University of Trondheim
`
`Programme Committee
`
`G. Birtwistle
`University of Calgary
`
`P. B. Denyer (Deputy Chairman)
`University of Edinburgh
`
`M. Fourman
`
`University of Edinburgh
`
`C. E. Goutis
`
`University of Patras
`
`J. P. Gray
`Algotronix
`
`P. Michel
`Siemens
`
`S. Murai
`
`Mitsubishi
`
`T. Nishimukai
`Hitachi
`
`G. Saucier
`INPG
`
`Grenoble
`
`C. H. Sequin
`University of Berkeley
`
`J. Staunstrup
`DTH
`
`T. Yanagawa
`NEC Corporation
`
`General Chairman
`
`1. Barron
`Division Ltd
`
`Organising Committee
`
`Chairman and Treasurer
`D. J. Rees
`
`University of Edinburgh
`
`Tutorials
`M. Fourman
`
`University of Edinburgh
`
`Proceedings
`P. B. Denyer
`University of Edinburgh
`
`Exhibitions Coordinator
`R. A. McKenzie
`
`University of Edinburgh
`
`Demonstrations Coordinator
`
`D. Rogers
`University of Edinburgh
`
`Publicity
`A. D. Milne
`
`Wolfson Microelectronics Ltd
`
`Sponsorship
`J. P. Gray
`Algotronix Ltd
`
`Conference Secretariat
`
`CEP Consultants Ltd
`26-28 Albany Street
`Edinburgh
`EH1 3QH
`
`1053-007
`
`1053-007
`
`

`
`vLsI91
`d_
`B D
`A. Halaas and 13,
`_ em, (E S)
`science Publishers B.V. (North-Holland)
`é"fJ$‘§':np. All rights reserved.
`
`157
`
`()N-CHIP CMOS SENSORS FOR VLSI IMAGING SYSTEMS
`
`Peter B. Denyer, David Renshaw, Wang Gouyu, Lu Ming Ying,
`Stuart Anderson
`
`Department of Electrical Engineering, University of Edinburgh,
`Mayfield Road, Edinburgh, EH9 3JL, Scotland
`
`Abstract
`
`We present techniques for the integration of vision sensors and processors on
`common CMOS substrates. This establishes the feasibility of single-chip vision
`systems which offer reductions in size, power and cost over contemporary
`techniques,
`
`1-
`
`INTRODUCTION
`
`Potential electronic vision applications are widespread; examples include:-
`
`. bar-code and text readers
`
`. security cameras
`
`. image capture for DTP
`
`. biometric verification; fingerprints, faces, etc.
`
`. fax
`
`. production line inspection
`
`. video telephones
`
`. vision subsystems in robots and autonomous vehicles
`
`. consumer camcorders and still cameras
`
`. automotive applications
`
`Virtually all of these applications, and a host of others, are sensitive to cost, size
`and power consumption. This applies not only to the camera function but also to
`
`1053-008
`
`1053-008
`
`

`
`158
`
`the subsequent image processing functions, which are invariably computationally
`demanding.
`Commonly such systems are constructed from commodity camera modules, frame
`grabbers and PC’s, or dedicated image processing hardware. These systems can
`hardly be described as miniature, and their cost and power consumption severely
`limit their application. Only where production volumes are very large, as in
`consumer camcorders, do all parameters fall to attractive levels.
`technology
`Fortunately the advancing capability of VLSI, especially CMOS,
`permits the integration of video~rate A/D conversion and the implementation of
`powerful
`custom image
`processing
`architectures. Challenging
`imaging
`applications, such as fingerprint verification, have become possible within a few
`tens of cubic inches, consuming a few watts of power. This is still far from ideal and
`the dominant limiting factor is usually the camera module itself.
`It is well known that silicon can act as an excellent photoreceptor over the visible
`spectrum. The majority of solid-state cameras today use CCD technology (a variant
`of MOS) which, over two decades, has been highly refined to optimise this function
`Some sensor manufacturers use a variant of single-channel MOS technology in
`which only doping levels are altered to optimise optical performance parameters
`such as anti-blooming.
`Other workers [1,2] have recognised the attraction of implementing sensors in an
`unmodified CMOS process, permitting the inclusion of the sensor with other
`control and processing functions on the same chip. Despite encouraging results this
`technique has never been developed to the point at which the sensor performance
`matches that of CCD cameras. The purpose of our work is to realise, in unmodified
`CMOS technology, array image sensors which match the performance of CCD
`cameras. We have succeeded in this aim and report here the circuit techniques and
`results that we have achieved, including demonstrator single-chip vision systems,
`
`2. TECHNIQUES
`
`In common with others [3] we use a photodiode sensor comprising an array of MOS
`transistors, one per pixel, Figure 1. The photodiode is implemented by extending
`the source region of the transistor. This may be reset and then isolated under
`control of the MOS transistor gate. All of the gates in each row are driven in
`common.
`Once reset, the reverse-biased (photo)diode converts incident light into a small
`photocurrent which gradually discharges the photodiode capacitance. The pixel is
`read by opening the gate, connecting the photodiode to the MOS transistor drain.
`All of the drain in each column are connected in common and only one row is read
`at any time.
`Commonly, the column lines are gated through an analogue multiplexer to a single
`external charge sense amplifier. The requirements of this amplifier are daunting
`considering that high-speed and wide dynamic range must be achieved from 3
`
`E.C/JQnJ:‘QU?o-J!
`
`1053-009
`
`1053-009
`
`

`
`159
`
`charge packet in the pixel which may be of the order of lit). Accordingly we have
`replaced this scheme by providing charge sense amplifiers at the top of each
`column. These need not work so quickly, since their activation frequency is equal
`'
`te rather than the pixel rate, and they are situated as close as possible
`to the pixel array. Their sole constraints are the need to achieve a good dynamic
`range and to be realised within the pixel pitch (of the order of 10—20um). We use a
`single-ended differential charge integrator which gives a low impedance 1v
`analogue representation of the pixel charge with a theoretical dynamic range of
`70dB. The read time is approximately 500nsec.
`
`OUTPUT
`HORIZONTAL
`SHIFT REGISTER AMPLIFIER
`
`COLUMN SENSE AMPLIFIER
`
`
`
` VERTICAL
`
`SHIFT REGISTER
`
`a--—...
`
`WV‘\u
`
`Figure 1. Architecture of a CMOS image sensor with
`column charge sense amplifiers.
`
`The voltage representation at the output of these sense amplifiers is sampled and
`stored on a row of capacitors and the information on these is multiplexed out in the
`conventional manner, with the exception that we implement the output charge
`integrator on-chip, including a sample-and-hold stage. For applications requiring
`a composite video waveform it is relatively easy at this stage to include an analogue
`multiplexer to switch in blanking and sync. levels at appropriate times.
`Serially-scanned operation is achieved by adding vertical and horizontal digital
`shift registers at the periphery of the array and these also must match the pixel
`pitch. The vertical register successively activates the row lines, whilst the
`
`1053-010
`
`m
`
`1053-010
`
`

`
`
`
`160
`
`1 read-out within each line. The
`horizontal register controls the sequential pixe
`'
`f the array is quite insensitive to these control waveforms and
`'
`s, in contrastto CCD, and this IS a distinctive advantage ofthe approach_
`performance 0
`_
`_
`_
`_
`.
`Prototype CMOS arrays usingthis architecture give remarkably good results. They
`de margins of temperature and supply voltage. The Single
`operate over very Wi _
`‘
`parameter ofconcernis fixed pattern noise from two sources; threshold vamationg
`in the MOS pixel access tran
`'
`'
`'
`rtical stripes. Without compensation these
`
`effects have an rms v
`eliminated these effects by:—
`(i) reducing the applied pixel reset voltage to make the actual reset value in-
`dependent of the gate potential and gate threshold.
`an offset compensating phase in the common sense ampfi.
`(ii) implementing
`hronisation.
`fiers during idle periods, such as line and frame sync
`These circuit
`techniques successfully eliminate the fixed pattern noise and
`overcome a traditional objection to the potential ofthis approach.
`
`
`
`Figure 2. A demonstration CMOS CCTV camera.
`Figure 2 shows a completed CCTV camera chip fabricated on a standard 2 11110110“
`CMOS ASIC line. The chip contains an array of array of 312 x 287 pixels With
`timing and automatic exposure control on«chip.
`
`l
`.1
`,1
`
`i
`
`1053-011
`
`1053-011
`
`

`
`161
`
`
`
`(a)
`
`(b)
`
`Figure 3. Comparison of CCD and CMOS camera performance
`under identical conditions:- (a) CCD (b) CMOS
`
`Figure 3 compares the picture output ofthis device with an existing CCD camera
`module. The results are subjectively indistinguishable.
`
`By electronically controlling the integration period we can proportionately
`decrease the sensitivity ofthe array. We can achieve this through the vertical shift
`registerby controlling the duration, in cycles, ofa ‘reset’ pulse entered at the top of
`this register. This varies the integration time in steps equal to the line period. We
`further gate this signal with one of short duration to reduce the exposure time in
`Steps equal to the pixel period, down to a minimum time constrained by the read
`time of the column sense amplifiers. This is approximately 500nsec, giving a total
`exposure range of 40,000:1.
`Ifwe now alter the exposure time in response to the monitored video output we can
`implement fully automatic electronic exposure control and avoid the need for
`mechanical iris control on the lens.
`
`1053-012
`
`'
`
`1053-012
`
`

`
` _
`
`162
`
`Figure 4 Shows a simple scheme for such a control algorithm. The video stream 15
`d in three bins:- very white, average and very black. The
`exposure is increased or decreased according to whether the image content is
`judged to be too bright or too dark.
`
`% VERY
`BLACK
`
` "%/////////
`
`Figure 4. Automatic exposure control decision diagram
`
`We have implemented th.is scheme (which costs approximately 1000 gates) on
`several CMOS sensor arrays and obtained satisfactory performance.
`
`4. RESULTS
`
`Over a series of prototypes we have i
`arrays to match or out—perform typical monochrome CCD performance in most
`respects. A detailed companson is given in Table 1. The CM
`from the CCTV camera demonstrator device shown in Figure 3. The CCD data is
`compiled from manufacturers data sheets.
`
`1053-013
`
`1053-013
`
`

`
`163
`
`
`
`
`
`
`Integrated
`
`Edinburgh
`
`CMOS Camera
`
`
`
`
`
`CCD Camera
`
`Module
`
`output
`
`
`
`1.5um process
`sawraaonlevel mam
`
`52dB 3%
`composite video
`composite video
`1v p-p
`1v p-p
`
`
`
`integration
`time range
`
`dark current
`
`
`
`300:1
`
`40,000:1
`
`0.0004
`
`
`
`
`
`
`
`
`
`as fraction of saturation
`
`at room temperature,
`20msec integration time
`a““"‘°°“’i“g fa°"°r
`
`Table 1. Comparison of CCD and CMOS sensor performance
`
`5. Application Examples
`
`We report here two examples which illustrate the potential of this technology for
`integrated vision applications.
`Our first example is a low-resolution camera for use in intruder-alarm verification.
`This device is installed with a Passive Infrared detector and, upon detecting an
`alarm, transmits a short sequence ofvideo to a control unit which compresses and
`transmits this data to a remote monitoring station. Within a few seconds of an
`alarm event a remote observer can deduce its cause and take appropriate action.
`As greater than 95% ofsuch alarms are false, the provision ofvideo verification will
`eliminate much unnecessary police action.
`Passive alarm units are very low cost items and cost is a primary constraint on this
`device.
`
`1053-014
`
`rnis
`
`_t is
`
`OI1
`
`sor
`°st
`'"e_d
`1 15
`
`1053-014
`
`

`
`I
`
`164
`
`Figure 5 shows the complete intruder~alarm camera module. This includes a 1.53pm
`CMOS camera chip measuring 5.8mm x 4.0mm combining a 156x100 pixel sensor
`array with all timing and control electronics on—chip. The device is customised for
`this application, and this includes drivers which automatically trigger lamp or
`flash devices to assist in dark conditions. A further novel feature of this device is
`the use ofa miniature glass lens which is bonded directly to the sensor chip surface_
`This enables a 90° field ofview and assists in improving th
`
`this small unit.
`
`Figure 5. Complete alarm verification camera module using a
`custom CMOS sensor with a miniature chip-mounted lens
`
`modem unit.
`
`The assembly shown is on a ceramic hybrid substrate and includes a 5v regulator.
`a clock crystal and approximately one dozen passive components. A simple bipolar
`stage provides suflicient output impedance to drive 20OPt of co—axial cable to the
`Our second example shown in Figure 6 is a very substantial single-chip ViSi°“
`processing system. This includes:-
`. 258 X 258 pixel array.
`. Image preprocessing and quantisation to form a normalised
`
`binary image.
`
`1053-015
`
`1053-015
`
`

`
`165
`
`. 64-cell 2000 Mops/sec correlator array.
`
`. Post-correlation decision hardware.
`
`. 16k bits RAM cache.
`
`. 16k bits ROM look—up table.
`
`two external devices (one 64kbit RAM and one 8051
`with the aid of
`microcontroller) this device performs all of the image sensing and processing
`functions necessary to capture and verify a fingerprint against a stored reference
`print within one second.
`
`.
`
`..v-
`
`II
`
`o
`
`<14"
`
`....
`I
`
`
`:“éqjriI *1.‘-I.
`
`!..
`‘
`-.
`--
`-
`..g-..
`__I
`‘ 250-
`...;-
`I."
`
`.
`-J
`.
`-
`,
`l___ 3
`‘ D
`--
`.: i
`__
`:_-
`._ ._
`._
`FII --
`— .|
`';,.',.
`--I
`1'2 s
`:
`- -
`.
`_-___ ! _________
`-r-“H -
`_"*
`l-ran-1-u
`'.--
`”--Q
`--'-i-..‘_.".
`._
`_
`'
`-I‘.
`
`.
`
`_
`
`-u'
`
`‘nu.-
`’
`---
`I
`‘.nI‘.
`.'
`-
`3 “.-
`].- -
`‘. .1.‘
`‘ ‘E -J
`
`_
`
`_
`
`»
`
`~
`
`-
`
`_
`
`-no --
`I
`I _'
`
`__
`
`...~
`\
`~- ‘tr
`.
`p.
`"‘-. ‘-
`'
`‘_i.;I.‘...
`-'_
`‘:1:
`1
`.t.?
`".
`'u-_:_-
`Ll
`_
`"IT
`1:-...-.
`.
`;.*-'.:'_
`“*'*'
`4!:
`I
`.
`._ ‘.
`‘._l_.
`1
`I
`
`--
`
`I
`
`"-
`
`.13:-Tfi_'yi?H1i‘.‘-1*"!-IT
`
`Figure 6. An integrated vision system 258 x 258 sensor array and
`image processor for fingerprint capture and verification
`This is perhaps the best example of our goals; the integration of a sensor and
`powerful image processor on one substrate.
`We have demonstrated several other devices in both 2 micron and 1.5 micron ASIC
`CMOS technologies and claim that these techniques are portable to any commodity
`CMOS process.
`
`1053-016
`
`
`
`-
`.
`<_
`__
`.2 E"
`;:
`‘ ei
`%2C
`-1-
`L1‘
`aw
`‘
`_"‘~“-
`-_
`..
`.
`‘Ir
`Jun:
`-. ~=
`hf.‘ '*-"
`_ __I
`I
`
`.
`
`U '
`""
`
`..
`
`'
`
`'
`
`
`
`1053-016
`
`

`
`E‘.
`
`166
`
`6. CONCLUSIONS
`
`The aim of ‘smart’ vision devices, incorporating image sensors and processors in
`one chip has been substantiated. These integrated devices can be implemented
`today in unmodified commodity CMOS technology. Vision systems implemented in
`this way enjoy unprecedented reductions in size, cost and power consumption.
`
`7. ACKNOWLEDGEMEN'I‘S
`
`This work was supported by SERC award GR/F 36538, Automated Security
`Holdings P1c., De La Rue and VLSI Vision Ltd.
`
`8. REFERENCES
`
`1. Lyon, R.F., “The Optical Mouse and an Architectural Methodology for Smart
`Digital Sensors”, VLSI Systems and Computations, 1981, Springer-Verlag,
`ISBN 3-540-11251-0, pp. 1-19.
`2. Tanner, J.E. and Mead, C., “A Correlating Optical Motion Detector”, 1984, Con-
`ference on Advanced Research in VLSI, MIT, pp. 57-64.
`3. Renshaw, D. and Denyer, P.B., “ASIC Vision”, CICC 1990, §7.3.
`4. Anderson, S. et al., “A Single Chip Sensor and Image Processor for Fingerprint
`identification”, CICC 1991, §12.1.
`
`1053-017
`
`1053-017
`
`

`
`nWW
`
`...
`
`u.E;
`
`ETA'
`
`0
`
`Tn.P.m
`
`wA
`
`
`
`1053-018
`
`1053-018