Machine Vision and Applications (1995) 8:101-109
`
`Machine Vision and
`Applications
`© Springer-Verlag 1995
`
`A miniaturized space-variant active vision system: Cortex-I
`
`Benjamin B. Bederson1,*, Richard S. Wallace2,**, Eric Schwartz3,***
`
`1 Computer Science Department, University of New Mexico, Albuquerque, NM 87131, USA
`2 Courant Institute of Mathematical Sciences, New York University, 251 Mercer Street, New York, NY 10012, USA
`3 Department of Cognitive and Neural Systems, Boston University, 111 Cummington Street, Boston, MA 02146, USA
`
`Abstract. We have developed a prototype for a miniatur(cid:173)
`ized, active vision system with a sensor architecture based
`on a logarithmically structured, space-variant, pixel geome(cid:173)
`try. The central part of the image has a high resolution, and
`the periphery has a a smoothly falling resolution. The human
`visual system uses a similar image architecture. Our sys(cid:173)
`tem integrates a miniature CCD-based camera, a novel pan(cid:173)
`tilt actuator/controller, general purpose processors, a video(cid:173)
`telephone modem and a display. Due to the ability of space(cid:173)
`variant sensors to cover large work spaces, yet provide high
`acuity with an extremely small number of pixels, architec(cid:173)
`tures with space-variant, active vision systems provide a po(cid:173)
`tential for reductions in system size and cost of several or(cid:173)
`ders of magnitude. Cortex-I takes up less than a third of a
`cubic foot, including camera, actuators, control, computers,
`and power supply, and was built for a (one-off) parts cost of
`roughly US $2000. In this paper, we describe several appli(cid:173)
`cations that we have developed for Cortex-I such as tracking
`moving objects, visual attention, pattern recognition (license
`plate reading), and video-telephone communcications (tele(cid:173)
`operation). We report here on the design of the camera and
`optics (8 x 8 x 8 mm), a method to convert the uniform im(cid:173)
`age to a space-variant image, and a new miniature pan-tilt
`actuator, the spherical pointing motor (SPM), (4 x 5 x 6cm).
`Finally, we discuss applications for motion tracking and li(cid:173)
`cense plate reading. Potential application domains for sys(cid:173)
`tems of this type include vision systems for mobile robots
`and robot manipulators, traffic monitoring systems, security
`and surveillance, telerobotics, and consumer video commu(cid:173)
`nications. The long-range goal of this project is to demon(cid:173)
`strate that major new applications of robotics will become
`feasible when small, low-cost, machine-vision systems can
`be mass produced. We use the term "commodity robotics" to
`express the expected impact of the possibilities for opening
`up new application niches in robotics and machine vision,
`for what has until now been an expensive, and therefore
`limited, technology.
`
`* e-mail: bederson@cs.unm.edu
`** e-mail: rsw@cs.nyu.edu
`*** e-mail: eric@thing4.bu.edu
`Correspondence to: B .B. Bederson
`
`Key words: Cortex-I - Active vision system, miniature -
`Space-variant sensors - Spherical pointing motor
`
`1 Introduction
`
`Computer vision is a field that typically requires fast and
`expensive machines. One reason for this is the need to pro(cid:173)
`cess images of size O(N2), for N = 250 -+ 1000 at frame
`rates of 30-60 Hz. By using novel image architecture, image
`processing, and actuation, we have been able to construct a
`system that performs competitively with existing systems
`that are orders of magnitude larger and more costly to build.
`In other work, a detailed examination of the design consid(cid:173)
`erations and advantages of space-variant sensing is provided
`[20], and it is shown that the human visual system, which
`uses a similar space-variant sensor design, achieves up to
`four orders of magnitude of image compression from this
`strategy.
`Any system that uses a space-variant sensor must aim
`the sensor properly. Since space-variant sensors only have
`high resolution in the fovea, the current region of interest
`must be continuously tracked or foveated. Such a sensor
`must be mounted in a device that can aim it with precision.
`A system with this capability is an example of an active
`or attentive vision system [4, 5]. In other words, the ad(cid:173)
`vantages of space-variant sensing are provided at the cost
`of increased algorithmic and robotic complexity: high speed
`and accurate actuators and control strategies, attentional al(cid:173)
`gorithms, novel image processing, and pattern recognition
`must be developed.
`In this paper, we describe Cortex-I, a prototype miniatur(cid:173)
`ized active vision system with a space-variant image archi(cid:173)
`tecture. This system integrates a CCD sensor, a miniature
`pan-tilt actuator, a controller, a general purpose processor
`and a display (Fig. 1). The system can be connected to a
`host processor for applications development, integrated as
`the visual module of a larger robotic system, or operated in
`a stand-alone configuration.
`As an architecture for image processing and computer
`vision, our prototype represents a significant departure from
`the types of large-scale parallel vision systems we and others
`
`Petitioner Samsung Ex-1051, 0001
`
`

`

`102
`
`Ftg. 1. Cortex-I: a prototyPe of a miniaturized, active vision system
`
`have previously investigated [14, 19, 28, 3 la]. Our design
`is a small, loosely coupled network of embedded microcom(cid:173)
`puter modules, integrated directly with a sensor and robotic
`pan-tilt actuator. Moreover, our architecture has benefited
`from research into the space-variant nature of the human
`visual system (23, 25- 27, 32). We use a logmap sensor, a
`useful type of space-variant sensor that is modeled on the hu(cid:173)
`man visual system. The space complexity aspect of logrnap
`sensors is particularly attractive and has been analyzed in
`detail [20].
`Other researchers have developed a variety of active vi(cid:173)
`sion systems that mimic some biological functions such as
`pan, tilt, vergence and focus control, attentional algorithms,
`and real-time tracking (1- 3, 6, 11, 13-16). Through imple(cid:173)
`menting image processing at the low data rate of a logmap
`sensor, and exploiting our novel pan-tilt mechanism, our pro(cid:173)
`totype demonstrates a significant subset of these capabilities
`with only modest processing power and inexpensive com(cid:173)
`ponents. The two principles of embedded system design and
`space-variant active sensing, taken together, enable us to de(cid:173)
`velop new applications flexibly at a very low cost.
`
`2 Space-variant images
`
`A space-variant image sensor is characterized by an irreg(cid:173)
`ular pixel geometry. We have experimented with several
`sensor designs, including custom VLSI (in collaboration
`with Synaptics, Inc.). Their common features are large-scale
`changes between the smallest and largest pixels, a wide field
`of view, and a pixel population far smaller than that of a
`conventional uniform image sensor. One such space-variant
`geometry is logarithmic [20, 21), in which the pixel pattern
`approximates the sensor geometry of the human eye. In this
`section, we review space-variant images. See (30] for more
`detail on this subject.
`The mapping defined by the logarithmic geometry is il(cid:173)
`lustrated in Fig. 2. It is convenient to represent the point in
`the domain, P 1
`, as
`, as rei8 and the point in the range, P2
`the complex number x+iy. If we talce the log of Pi, we get:
`
`log(rei9) = logr + i0
`Substituting
`x = log r and y = 0,
`we get
`1og(ri9) = x + iy
`or,
`log(P1) = Pi
`so that the (complex) logarithm of a point in polar coordi(cid:173)
`nates is transformed to a point in rectangular coordinates.
`This mapping has some interesting geometric proper(cid:173)
`.ties. Specifically, fovea-centered circles are mapped to ver(cid:173)
`tical lines, and radial lines are mapped to horizontal lines.
`This has the effect of transforming the scale and rotation in
`fovea-centered retinal images to translation in cortical im(cid:173)
`ages. Computationally, translation is easier to handle than
`scaling or rotation, so this provides another justification for
`a complex log sensor geometry, which has attracted consid(cid:173)
`erable interest during the past 15 years. In the present work,
`we do not attempt to use this feature of complex log map(cid:173)
`ping; we merely see it as a convenient representation for
`space-variant sensing, and one which is motivated by the
`similar structure of the human visual system.
`One problem in using the analytic space-variant map(cid:173)
`ping, log(P), is the existence of a singularity at the origin
`of the coordinate system. Figure 2b shows this singularity.
`The left points of each triangle in the range all represent the
`same point in the domain. Our solution to thjs problem is
`the introduction of a small real constant, a, into the mapping
`and the use of a map function of the form log(z + a). We
`call this mapping the "logmap". The constant a is used in
`an analogous fashion in models of the human visual system:
`it is a measure of the size of the central linear representation
`of the human fovea (25].
`As can be seen in Fig. 2d, the pixels on the edge are cut
`off in different places yielding 3, 4, and sometimes 5-sided
`pixels. This greatly complicates image processing. Even sim(cid:173)
`ple operators, such as convolution, are difficult to implement.
`We have found a general solution to this problem, which in
`effect, generalizes image processing to sensors of arbitrary
`neighborhood relations or topology. This work is described
`in detail in Sect. 4.2.
`We have followed two approaches to acquiring logmap
`images in real time. The first involves the design and fabrica(cid:173)
`tion of a custom VLSI sensor, in collaboration with Synap(cid:173)
`tics. This work is in an early stage, and we do not describe
`it further here. The second approach is to use an imbed(cid:173)
`ded processor, together with a conventional CCD imag(cid:173)
`ing chip, to produce the logmap transformation by a real(cid:173)
`time, lookup-table technique. Thus we can formally defi ne
`the logmap as a mapping from a TV image, I(i,j), where
`i E {0, . . . , NROWS - 1} and j E {0, ... , NCOLS -
`l}.
`let L(u, v) be the logmap, with u E {0, .. . , NS POKES - 1}
`and v E {0, ... , NRI NGS - 1 }.
`The forward mapping from TV image space to logmap
`space is specified by the spoke and ring lookup tables, S(i, j)
`and R(i,j) (Fig. 3), where again i E {O, ... , NROWS -
`I}
`l}. Let a(u, v) be the area (in
`and j E {0, ... , NCOLS -
`TV pixels) of a Jogmap pixel (u, v).
`
`Petitioner Samsung Ex-1051, 0002
`
`

`

`log(z)
`-------
`
`2b
`
`e--+-t-+fl:ffift++---t---7 log{: +a) "
`
`2d
`
`2c
`
`3a
`
`0,170,1B 0,19
`
`1.141,15 1,16 1,171,1B 1,19
`
`2,122,132.14 2,15 2,1B 2,172,1B 2,19
`
`3,11 3,123,133,14 3,153,163,173.18 3.19
`
`4,041
`
`4,2
`
`4,3
`
`4,4
`
`4,5
`
`4,6 4,7!4,6
`
`4,9 4,104,11 4,1214.134,14 4,154,164,174.18 4,19
`
`s,2
`
`s,a
`
`s.4
`
`s,5
`
`s.s
`
`s,1
`
`s,s
`
`5,9 s,1os,11 s,12js,1als.1, s,1s s.1s1s.11s,1ss.19
`
`6,2
`
`6,3
`
`6,4
`
`6,5
`
`6,6
`
`6,7iB,B
`
`6,,1 6,106,11 6,126,13\6,14 6,155,1Sl6,176,1Bl6,19
`
`7,0
`
`7,1
`
`7,2
`
`7,3
`
`7,4
`
`7,5
`
`7,6
`
`7,7
`
`7,8
`
`7,9 7,10 7,11 7,12 7,13 7,14 7,15 7,16 7,17 7,18 7,19
`
`s,2
`
`s,3 B,4
`
`e.5
`
`s,s
`
`a,7
`
`a.a
`
`e.s la,10 a,11 e.12 a,13 B,14 a,1s B,18 B,17 a.1a B,19
`
`9,2
`
`9,3
`
`9,4
`
`8,5
`
`9,6
`
`9,7
`
`9,8
`
`9,9 9,109,119.12 9,13 9.149,159,16 9,179,189,191
`
`10,0 10.1 102 10,3 10,4 10,5 10,6 10,7 10,B
`
`10,1110,1210,1310,1410,1510,1610,1710,1810,19
`
`11.0 11.111,211,3 11,411,S 11,,; 11,7
`
`11,1211,1311,1411,1511,1611,1711,1811,19
`
`3b
`
`12,012.1 12,2 12,3 12.412,S
`
`13,013,1 13,2
`
`Fig. 2a-d. This shows the complex logmap: a,b log(z) c,d its relative, the
`mapping, log(z + a). It can be seen that the log(z + a) map results from
`cutting the hatched region out of the log(z) map. Here, P1 is mapped to
`P2 . The log(z) map has a singularity in the center while the log(z+a) map
`has a branch-cut along the vertical meridian
`Fig. 3. a The lookup tables S(i, j) and R(i, j) combined into one figure; b
`the forward map space. The correspondence between inverse and forward
`map pixels is shown
`
`a(u,v) =LI I S(i,j) = u and R(i,j) = v.
`i,j
`
`103
`
`(1)
`
`L(u, v) = -(-- L I(i, j) I S(i, j) = u and R(i, j) = v. (2)
`
`The logmap (or forward map) image (Fig. 2c) is defined by
`1
`a u,v) ..
`i,J
`The inverse map, illustrated in Fig. 2b, is
`L- 1(i,j)=L(S(i,j),R(i,j)).
`
`(3)
`
`The relationship between a space-variant image and the
`lookup tables S(i,j) and R(i,j) is illustrated by comparing
`Fig. 3 with Fig. 4. The values in the lookup tables depict
`the row and column addresses of pixels in the space-variant
`image array. We observe that if n is the number pixels for
`which a(u, v) > 0, then n ::;: NSPOKES x NRINGS, and
`we define NPIXELS = n.
`
`3 System description
`
`Cortex-I consists of the ·emulated logmap sensor, a mmia(cid:173)
`ture pan-tilt actuator, a controller, a general purpose pro(cid:173)
`cessor, a display, and an optional video telephone interface.
`The controller consists of a camera driver, a 2-MIPS pro(cid:173)
`grammable microcontroller (Motorola MC68332), a video
`display driver, and three 12-MIPS digital signal processors
`(Analog Devices AD2101). The actuator and camera are
`mounted to the electronics chassis ( 14 x 22 x 22 cm) and con(cid:173)
`nected by twisted-pair cables. The system is powered from
`a standard 110-volt AC line, but uses less than 25 watts and
`could be battery powered.
`The camera consists of a miniature, commercially avail(cid:173)
`able, CCD image sensor and a custom Jens assembly (fab(cid:173)
`ricated in collaboration with Barry Levin) mounted in the
`actuator. The camera image is mapped to a logmap image
`with a fast lookup-table algorithm, as we have already out(cid:173)
`lined. For the mapping that we have used extensively, the
`maximum resolution is 0.175° /pixel and the horizontal field
`of view measures 33°. The sensor has a fixed focus 4-mm
`lens with a set of fixed, manual apertures (three sizes). Since
`objects that are more than 10 focal lengths from the camera
`essentially have an infinite depth of field, we avoid the need
`of focus for working distances larger than about 40 mm. The
`system outputs up to 30 frarnes/s and measures 256 gray lev(cid:173)
`els/pixel. The camera head (CCD and lens assembly) mea(cid:173)
`sures only 8 x 8 x 10 mm, and is controlled by a lab-built
`driver board that provides timing signals to the sensor and
`converts analog sensor data to 8-bit digital data for the pro(cid:173)
`cessors.
`The spherical pointing motor (SPM) is a novel pan-tilt
`actuator using three orthogonal motor windings to achieve
`open-loop pan-tilt actuation of the camera sensor in a small,
`low-power package. The SPM can orient the sensor through
`approximately 60° pan and tilt, at speeds of several hundred
`degrees per second. It measures 4 x 5 x 6 cm and weighs
`170 g. It is actuated by currents of roughly 50 mA in each
`of the three coils.
`The MC68332 controls the SPM and runs the application
`software. The MC68332 can be connected to a host proces(cid:173)
`sor for applications development, to upload sensor data, or
`
`Petitioner Samsung Ex-1051, 0003
`
`

`

`104
`
`a
`
`C
`
`Fig. 4. a A TV image, l(i ,j); b the inverse logmap image L - 1(i,j); c the forward logmap image L(u, v)
`
`to receive pan-tilt commands. The MC68332 is a 32-bit 16-
`MHz microconcroller integrating peripheral controls, such
`as programmable digital control lines and timing signals, di(cid:173)
`rectly on chip. We connected a 192-kB RAM and a 128-kB
`EPROM to the microcontroller. The logmap data is output
`via a high-speed serial interface, either to an external device
`such as a host computer, to the internal display driver, or to
`the video-telephone transmitter.
`Several image processing demonstrations have been im(cid:173)
`plemented in the microcontroller ROM. These include a sim(cid:173)
`ple motion-tracking program that turns the camera to center
`the observed motion field, at 16 frames/s. The ROM also
`contains a test pattern, an image-binarization program (22
`frames/s), and a motor motion demo. Other demonstrations
`illustrate the connectivity graph (Sect. 4.2) to implement im(cid:173)
`age smoothing (3 frames/s), edge detection (2 frames/s) and
`connected components (1 frame/s), illustrating both the gen(cid:173)
`erality of the programming model and the limitations of the
`relatively low power microcontroller used in the current im(cid:173)
`plementation of Cortex-I.
`The Analog Devices 2101 digital signal processor (DSP)
`provides additional real-time, image-processing capabilities.
`The DSP controls the CCD readout, computes the logmap,
`and sends the logmap image to the MC68332 via a high(cid:173)
`speed serial connection. The DSP board combines an AD2 l 01
`12MHz DSP having two 2-MHz serial ports, an 8-kB in(cid:173)
`ternal RAM, an 80-kB external RAM, and a 64-kB boot
`EPROM. A second identical DSP board functions as a video
`display driver, generating an RS-170 video output suitable
`for display on a standard TV monitor. An optional third
`DSP board is used as a video transmitter that can send four
`logmap images ( uncompressed) at 4 frames/s over a standard
`voice-grade telephone line. To achieve this, the DSP imple(cid:173)
`ments a custom-designed protocol for an analog modem that
`encodes the logmap frames.
`
`3.1 Camera sensor
`
`We have pursued two routes to real-time acquisition of
`logmap images. The first is to develop a custom VLSI sensor
`chip with a logmap geometry that consists of pixels vary(cid:173)
`ing in size and arranged in a logmap geometry. The second
`is to use a commercially available sensor chip that returns
`conventional rectangular images and maps them into logrnap
`images.
`The main advantages of a custom logmap sensor chip are
`the high frame rate and small resultant system size. This is
`
`because the geometry of the chip allows intrinsic mapping.
`The sensor layouts that we have designed consist of several
`thousand pixels on chip, and this is a very small number of
`pixels to digitize. It is likely that a frame rate of thousands
`of frames per second could be supported, contingent on the
`ambient light intensity. However, at the present time, we do
`not have a fully working space-variant chip.
`The main advantages of an off-the-shelf conventional
`(uniform) sensor chip are low price, high image quality, and
`immediate availability. Because uniform sensors have been
`refined for many years and are produced in great quantities,
`they have extremely high quality, which is difficult to match
`with custom sensor chips. The image from a uniform sensor,
`however, must be mapped to a logmap image and this is
`computationally expensive, since the final logmap image size
`of thousands of pixels must be supplied through a read-out
`bottleneck of hundreds of thousands of pixels on the chip.
`However, with careful design of the readout and logmap
`emulation algorithms, it is possible to achieve rates as high
`as 100 frames/s, without resorting to excessively complex
`readout electronics.
`
`3.2 Spherical pointing motor (SPM)
`
`A pan-tilt mechanism is a computer-controlled actuator de(cid:173)
`signed to point an object such as a camera sensor. For appli(cid:173)
`cations in active vision, we prefer a pan-tilt mechanism to be
`accurate, fast, small, low-power, and inexpensive. We have
`constructed two actuators meeting these requirements: one
`based on a parallel linkage actuated by two identical, small,
`stepper motors [7, 9]; the other incorporates both pan and tilt
`into a single two-axis actuator (Fig. 5). The SPM [protected
`by the U.S. patent no. 5204573 entitled "Spherical Pointing
`Motor" (1993)) is described in detail in (8, 10, 31b, 31c], but
`is summarized here. The SPM consists of three orthogonal
`motor windings in a permanent magnetic field, configured
`to move a small camera attached to a gimbal.
`The simplest and most obvious direct drive pan-tilt
`mechanism is the two-stage motor-on-motor (MOM) design.
`The first motor turns the mechanism through one degree
`of freedom, usually pan, and the second through the other
`degree of freedom usually tilt. The second motor must be
`powerful enough to move the camera sensor. The first must
`move both the camera and the second motor. The MOM de(cid:173)
`sign therefore usually consists of one larger motor and one
`smaller one. Because of the large accelerations involved in
`starting and stopping a high-speed device accurately, the ex-
`
`Petitioner Samsung Ex-1051, 0004
`
`

`

`105
`
`net rotating on the inside, or vice versa. Figure 6 depicts an
`example of the former.
`
`3.3 Microcontroller board
`
`A Motorola MC68332 microcontroller controls the SPM and
`runs some applications software. The MC68332 can be con(cid:173)
`nected to a host processor for applications development, or
`to upload sensor data and to receive pan-tilt commands. The
`MC68332 is a 32-bit 16-MHz microcontroller integrating pe(cid:173)
`ripheral controls, such as programmable digital control lines
`and timing signals, directly on chip. The logmap data is out(cid:173)
`put via a high-speed serial interface, either to an external
`device such as a host computer or to the internal display
`driver.
`We have implemented image processing demonstrations
`in the microcontroller ROM. A simple motion tracking pro(cid:173)
`gram, which turns the camera to center the observed motion
`field, runs at 16 frames per second (16 frames/s). The ROM
`also contains a test pattern, an image binarization program
`(22 frames/s) and a motor scan-sequence demo. We also
`use the connectivity graph (Sect 4.2) to smooth images (3
`frames/s), detect edges (2 frames/s) and connect components
`(1 frame/s), illustrating both the generality of the program(cid:173)
`ming model and the limitations of the microcontroller in the
`current prototype.
`
`3.4 Digital signal processors (DSPs)
`
`The Analog Devices AD-2101 DSP provides additional real(cid:173)
`time image processing capabilities on logmap images. The
`DSP reads the camera board, computes the logmap and sends
`the resulting image to the M68332 via a high-speed serial
`connection. The DSP board combines an AD 2101 12 MHz
`DSP having two 2-MHz serial ports, an 8-kB internal RAM,
`an 80-kB external RAM, and a 64-kB boot EPROM. An sec(cid:173)
`ond identical DSP board functions as a video display driver,
`generating an RS-170 video output suitable for display on a
`standard TV monitor. An optional third DSP board is used
`as a video transmitter that can send Iogmap images over a
`standard voice-grade telephone line at 4 frame.sis.
`
`4 Programming and development model
`
`4.1 Embedded system model
`
`Each of the three microprocessors in our prototype is a gen(cid:173)
`eral purpose computer, so the programming model bears
`great resemblance to that for embedded control systems on
`general purpose processors. One of the key elements in this
`model is a distinction between a development system and
`a target system. We initially developed our applications on
`a Sun workstation-based system. This development system
`simulates the target system closely: it combines a robotic
`pan-tilt actuator made by Klinger Scientific, a Sony XC-
`77RR camera and Analogic video digitizer, with software
`tools to simulate a variety of sensor geometries.
`There are four stages to developing an application.
`
`5
`
`amera and
`glmbal fit
`Inside coil
`cylinder
`
`Permanent
`magnet
`
`Coil A
`(Home Position)
`6
`
`Coil B
`(Up/Down)
`
`Coil C
`(Left/Right)
`
`Fig. S. The spherical pointing motor. At the center is a miniature camera
`consisting of a single CCD sensor chip and a lens assembly that fits on the
`rotor of this motor
`Fig. 6. Illustration of the external sphe,rical pointing motor shown in its
`home position with labels for the three coils
`
`tra inertial load presented by the MOM design requires very
`large torques in the primary motor.
`An alternative to the direct drive MOM design requires
`some form of linkage to couple the camera to the motor.
`We have built a stepper-motor actuated, modified Stewart(cid:173)
`platform device, but it is our impression that direct drive DC
`motors are advantageous, except for the limitations (already
`noted) of the MOM design. It is clear that some form of
`direct drive design, in which the MOM design is avoided,
`but in which no linkages are required, would be ideal. We
`have designed such an actuator, which provides a spheri(cid:173)
`cally symmetric, direct drive, DC motor actuator, namely
`the SPM.
`The SPM is an absolute positioning device, designed to
`orient a small camera sensor in two degrees of rotational
`freedom. The basic principle is to orient a permanent mag(cid:173)
`net in three orthogonal motor windings by applying the ap(cid:173)
`propriate ratio of currents to the three coils. The SPM can
`have the coils either on the outside with the permanent mag-
`
`Petitioner Samsung Ex-1051, 0005
`
`

`

`106
`
`l. We design a prototype program on the development host.
`At this stage, there are few memory, timing, or numeric
`precision restrictions. The commercial camera and actu(cid:173)
`ator provide a high degree of reliability.
`2. Development continues on the host, but we use the cam(cid:173)
`era and actuator of Cortex-I. The host controls the SPM
`by sending commands over an RS-232 serial line, and
`receives the logmap image from the miniature camera,
`digitizing the video signal output. This allows the same
`software to be tested with the new hardware.
`3. The application is debugged on the target hardware us(cid:173)
`ing the host as a terminal. Since C language cross(cid:173)
`development environments are available for both the
`DSPs and the microcontroller used in this system, we
`develop C code on the Sun host, then download it to the
`target system. The code must account for the more strin(cid:173)
`gent hardware restrictions such as integer arithmetic, less
`memory, and less processing power. Such restrictions are
`a result of the inherent tradeoff between a highly flexible,
`programmable development system and an inexpensive
`target system.
`4. We burn the application into the ROM of the target sys(cid:173)
`tem, at which point we can take the target system "into
`the field."
`Some applications, like an attentional OCR program for
`reading license plates [17, 18] are at stage two, while
`many simpler image processing algorithms, such as mo(cid:173)
`tion tracking, are at stage four.
`
`4.2 Space-variant image processing
`
`As mentioned earlier, the complex logmapping has a singu(cid:173)
`lar point at the origin, and a branch cut associated with the
`"phase-wrapping" of the angles O and 21T. In addition, the
`local pixel connectivity does not generally possess the sim(cid:173)
`ple 4-connectivity or 8-connectivity familiar in conventional
`computer graphics and image processing. The local topology
`of the complex log sensor is variable and complicated, and
`this is a feature that might be shared by other novel sensor
`architectures. To deal with image processing on an arbitrary
`sensor topology. we developed an approach to image pro(cid:173)
`cessing based on the use of a connectivity graph. We define
`the connectivity graph, G = (V, E), as the graph whose ver-
`. tices, V, stand for sensor pixels and the edges, E, represent
`the adjacency relations between pixels. Associated with a
`vertex pis a pixel address (u, v). Thus we write (u(p), v(p))
`for a pixel coordinate identified by its graph vertex.
`Each graph node, p, in the connectivity graph is rep(cid:173)
`resented by a constant size data structure in memory. The
`graph edges {(p,q)} are represented by one field contai~(cid:173)
`ing a list of pointers. The structure for p may also contam
`fields such as the pixel array coordinates (u(p), v(p)), the
`pixel centroid µ(p), the pixel area a(p) =_a(u(p),v(p)), and
`the number of neighbors 1,./V(p)! . The pixel centroid µ(p)
`represents the centroid of the TV pixels that map to p and
`is given by
`
`Fig. 7. a The image from a sensor having a random pixel geometry; b the
`connectivity graph for this sensor
`
`We define the set of neighbors of p to be denoted by
`
`,A/'(p) = {q I (p,q) EE}
`
`(5)
`
`We can define a space-variant sensor geometry that is not
`a complex logarithm, but is the result of a stochastic process.
`We define this map by placing seed points randomly within
`the image space with a density fu nction defined so that more
`points will be in the center of the image. The map is then
`created by growing each seed point until its neighbor is met
`via a simple relaxation procedure. The map is called the
`random map and is illustrated in Fig. 7.
`One application for the random map would be to use
`it with large area CCD image sensors with defective pix(cid:173)
`els. Large CCD sensors (1 kxl k or 2kx2k pixels) are ex(cid:173)
`tremely expensive because of their low production yield.
`These sensors are much cheaper when they have defective
`pixels. A random map with the associated connectivity graph
`could be created for each of these sensors, skipping the faulty
`pixels. This would allow for a very high-resolution camera
`that would not be prohibitively expensive.
`A variety of space-variant sensors have been designed
`and fabricated [ 12, 20-22, 24, 27]. The connectivity graph
`can be defined for any of these mappings, and then used to
`develop a library of image processing routines that can run
`on data from any of these sensors, and therefore we provide
`a generic approach to image processing on an unconstrained
`sensor geometry [29].
`Using the connectivity graph, we define a variety of im(cid:173)
`age processing operations. For example, if p represents a
`pixel, L(p) its gray value, and j//'(p) its neighbors, then we
`can define a simple edge operator e(p) as
`
`e(p) = IA;(p)I L_ (L(p) - L(q))2
`qE./f/ (p)
`
`(6)
`
`Note that the defi nition of e(p) contains no special cases
`for pixels having differing numbers of neighbors_, and thus
`applies equally to pixels at the boundary of the image and
`to those in the interior.
`Another simple example is the Laplacian, in sensor co(cid:173)
`ordinates, f(p), given by
`
`µ(p) =
`
`(1 ) L(i,j)T I
`ap . .
`i,J
`(S(i, j) = u(p) and R(i, j) = v(p)) .
`
`e(p) = IJ V(p)I L(p) E L(q)
`
`qE./V°(p)
`
`(4)
`
`(7)
`
`Petitioner Samsung Ex-1051, 0006
`
`

`

`107
`
`Using such definitions, we can build a library of image
`processing routines that is independent of the sensor geome(cid:173)
`try. We report the details of the connectivity graph in another
`paper [29], where we show how to develop programs for
`template matching, connected components analysis, pan-tilt
`calibration, and pyramid operations.
`
`camera scans the license plate from left to right with five
`frames. The program analyzes each frame to segment the
`characters and recognize them optically. It then merges the
`results from the five frames to yield the string of characters
`on the license plate.
`
`5 Applications
`
`5.1 Motion tracking
`
`We have implemented a motion-tracking application on
`Cortex-I in ROM that integrates all the capabilities of the
`system. It is based on a simple frame-differencing algo(cid:173)
`rithm and is able to track moving objects at approximately
`5 frames/s.
`The algorithm subtracts successive pairs of frames, com(cid:173)
`putes the centroid of motion, and moves the motor to point
`the camera at the computed centroid. Because this algorithm
`needs two frames to compute the centroid of motion, and
`then misses a frame while the motor is moving, it can only
`make one motor movement every three frames. Therefore,
`although it can compute the frame difference and centroid
`at 16 Hz, the motor is only moved about 5 times/s.
`
`5.2 License plate tracking and reading
`
`We solved a difficult tracking and pattern recognition prob(cid:173)
`lem using a combination of Cortex-I and a Sun Sparcstation,
`bringing it to the second stage of development (as described
`in Sect. 4.1). This application is described in detail in [17,
`18], but is summarized here. The task is to find the license
`plate on a moving car, track it as the car moves towards the
`camera, and finally read the characters on the license plate
`when it gets close enough to the camera.
`The experimental setup consists of a real license plate
`attached to a toy truck that is pulled by a string wound
`around a motor-driven pulley. The truck moves at approx(cid:173)
`imately 5 emfs. It is tracked for about 165 cm before the
`license plate is read.
`Cortex-I is set up to allow commands sent via an RS-232
`serial port to control the SPM position. The commands spec(cid:173)
`ify the pan and tilt angles of the motor in tenths of a