(12) United States Patent
`Hung et al.
`
`(16) Patent N6.=
`(45) Date of Patent:
`
`US 6,749,120 B2
`Jun. 15, 2004
`
`US006749120B2
`
`open aver e a .
`
`..... ..
`
`,
`
`,
`
`5,367,596 A * 11/1994 Chow ....................... .. 385/116
`5,399,852 A * 3/1995 Zheng et al. ............. .. 250/225
`5,408,084 A * 4/1995 Brandorff et al. ...... .. 250/208.1
`i
`glenghetaL
`~~~~~ ~~ 253045237;
`5,567,934 A * 10/1996 Zheng et al. ......... .. 250/237 R
`5,616,908 A * 4/1997 WilZ et al. .... ..
`235/46208
`5,702,059 A * 12/1997 Chu et al.
`235/472.01
`5,920,061 A * 7/1999 Feng ......... ..
`235/47201
`5,939,697 A * 8/1999 Spitz ..... ..
`235/46215
`5,984,186 A * 11/1999 Tafoya
`235/46201
`6,062,475 A * 5/2000 Feng _____ __
`235/462,21
`6,065,678 A * 5/2000 Li et al. .............. .. 235/462.46
`6,177,683 B1 * 1/2001 Kolesar et al. . . . . .
`. . . .. 250/566
`
`(54) METHOD AND APPARATUS FOR SCANNING
`ELECTRONIC BARCODES
`
`(75) Inventors: Patrick Siu-ying Hung, Saratoga, CA
`(52)’ James C‘ Dunphy’ San Jose’ CA
`(
`)
`
`_
`
`,
`.
`(73) AsslgneeZ CPO Technologles Corp-2 Sunnyvale’
`CA (Us)
`
`(*) Notice:
`
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`U.S.C. 154(1)) by 68 days.
`
`(21) AppL NO‘: 09/736,731
`
`(22) Filed:
`
`Dec. 11,2000
`
`(65)
`
`Prior Publication Data
`
`US 2002/0070278 A1 Jun. 13, 2002
`
`(51) Int. Cl.7 ................................................ .. G06K 7/10
`(52) US. Cl. ............................... .. 235/47201
`(58) Field of Search
`235/472 01 462 25
`46.2 472 469’
`494 491. 356/73 343. 348/272 304 308
`’
`’
`’
`’
`’
`’
`References Cited
`
`(56)
`
`Us PATENT DOCUMENTS
`
`6,347,163 B2 * 2/2002 Roustaei . . . . . . . . .
`
`. . . .. 382/324
`
`6,385,352 B1 * 5/2002 Roustaei ................... .. 382/324
`
`* ‘med by exammer
`
`Primary Examiner—Thien M. Le
`Assistant Examzner—EdWyn LabaZe
`(57)
`ABSTRACT
`
`A Scanner is provided that can read two-dimensional bar
`codes from re?ective or emissive electronic displays.
`Illumination, exposure, and contrast can be automatically
`adjusted according to the type of display and variations in
`brightness or re?ectivity. The scanner includes an imaging
`camera With high spectral response for reading from color
`displays. The scanner can also adapt to read barcodes from
`displays With non-square pixels.
`
`4,488,679 A * 12/1984 Bockholt et al. ......... .. 235/469
`5,317,331 A * 5/1994 Patty et al. ................. .. 345/16
`
`32 Claims, 6 Drawing Sheets
`
`16 14
`
`Page 1 of 15
`
`ROTHSCHILD EXHIBIT 1008
`
`

`
`U.S. Patent
`
`Jun. 15,2004
`
`Sheet 1 0f 6
`
`US 6,749,120 B2
`
`FIG. 1B
`
`Page 2 of 15
`
`ROTHSCHILD EXHIBIT 1008
`
`

`
`U.S. Patent
`
`Jun. 15,2004
`
`Sheet 2 0f 6
`
`US 6,749,120 B2
`
`10
`
`32
`
`54
`
`I
`
`44/’
`
`40
`
`5
`
`46
`
`/52
`48
`/
`1/ /
`
`4
`
`~
`
`/42
`
`FIG. 1C
`
`FIG. 2A
`
`Page 3 of 15
`
`ROTHSCHILD EXHIBIT 1008
`
`

`
`U.S. Patent
`
`Jun. 15,2004
`
`Sheet 3 0f 6
`
`US 6,749,120 B2
`
`PLACE SCANNER
`AGAINST THE
`DlSPLAY (LAMP OFF)
`
`MEASURE LIGHT
`FROM DISPLAY
`SCREEN
`
`SCAN BARCODE
`
`FIG.
`
`TURN ON
`ILLUMINATlON
`
`f31o
`
`Page 4 of 15
`
`ROTHSCHILD EXHIBIT 1008
`
`

`
`U.S. Patent
`
`Jun. 15,2004
`
`Sheet 4 0f 6
`
`US 6,749,120 B2
`
`320
`j
`
`322
`CAPTURE IMAGE /
`324
`CORRECT
`
`EXPOQSURE
`
`NO
`ADJUST ExPosuRE /325
`PARAMETER
`326
`L
`CAPTURE IMAGE /
`
`327
`
`SUCCESSF L
`DECODEU
`?
`
`328
`
`FLICKER
`
`DETEoCTED
`
`329
`INDICATE /
`SUCCESSFUL 2
`SCAN
`
`INDICATE / 330
`FAILED SCAN
`
`YES
`DETERMINE FLICKER /332
`PERIOD AND PHASE
`l
`SET EXPOSURE TIME
`
`/-334
`
`336
`CAPTURE IMAGE /
`
`V
`
`338
`
`SUCCESSFUL
`DECQODE
`
`YES
`
`FIG. 3B
`
`Page 5 of 15
`
`ROTHSCHILD EXHIBIT 1008
`
`

`
`U.S. Patent
`
`Jun. 15,2004
`
`Sheet 5 0f6
`
`US 6,749,120 B2
`
`PLACE SCANNER
`AGAINST
`DISPLAY SCREEN
`
`34o
`342
`/ j
`
`LAMP ON
`
`344
`/
`
`CAPTURE IMAGE / 346
`
`348
`
`CORRECT
`
`EXPOfzSURE
`
`ADJUST EXPOSURE
`PARAMETER(S)
`
`CAPTURE IMAGE
`
`SUCCESSFUL
`DEC‘7ODE
`YES
`
`INDICATE
`FAILED SCAN
`
`NO
`
`/ 352
`INDICATE SUCCESSFUL SCAN
`
`FIG. 3C
`
`CAPTURE IMAGE
`OF BARCODE
`
`faez
`
`360
`
`PROVIDE DIGITIZED
`IMAGE TO
`PROCESSOR
`
`f364
`
`EVALUATE
`LIGHTIDARK ELEMENTS
`
`/366
`
`SCALE IMAGE,
`IF NECESSARY
`
`PROCESS SCALED
`VIRTUAL IMAGE TO
`EXTRACT BARCODE
`
`/ 370
`
`FIG. 30
`
`Page 6 of 15
`
`ROTHSCHILD EXHIBIT 1008
`
`

`
`U.S. Patent
`
`Jun. 15,2004
`
`Sheet 6 6f 6
`
`US 6,749,120 B2
`
`PLACE SCANNER / 381
`AGA'NST SCREEN
`
`38°
`
`SEéRR'SQRTI?
`
`“0
`
`T TURN ON tLLUMlNATtON
`
`3B3
`
`391
`CAPTURE #MAGE /
`
`l
`CAPTURE IMAGE
`3B5
`
`f 384
`
`CORRECT
`EXPOSURE
`7
`
`SUCCESSFUL
`
`DECOODE
`
`' NO
`
`?
`NO
`386
`ADJUST EXPOSURE ff
`1
`387
`CAPTURE IMAGE
`
`393
`_2 ADJUST ElXPOSURE / "T
`394
`CAPTURE IMAGE /
`
`390
`
`SUCCESSFUL
`DECODE
`?
`No
`FAIL
`\ INDICATOR
`
`3B9
`
`‘ /
`PASS
`moacATo
`
`I
`
`A
`
`k
`
`N0
`
`400
`
`SUCCESSFUL
`DEC'PDE
`
`395
`
`SUCCESSFUL
`
`DECOODE
`
`YES
`
`FUCKER
`7
`
`MEASURE FUCKER
`398
`1
`SET EXPOSURE TIME /
`&
`CAPTURE IMAGE
`
`FIG. 35
`
`Page 7 of 15
`
`ROTHSCHILD EXHIBIT 1008
`
`

`
`US 6,749,120 B2
`
`1
`METHOD AND APPARATUS FOR SCANNING
`ELECTRONIC BARCODES
`
`CROSS REFERENCE TO RELATED
`APPLICATIONS
`
`Not Applicable.
`
`STATEMENT AS TO THE RIGHTS TO
`INVENTION MADE UNDER FEDERALLY
`SPONSORED RESEARCH AND
`DEVELOPMENT
`
`Not Applicable.
`
`FIELD OF THE INVENTION
`
`This invention relates to computer-readable codes such as
`barcodes, and more particularly to methods and apparatus
`for scanning codes from an electronic display or screen.
`
`10
`
`15
`
`BACKGROUND
`
`Barcodes provide a quick and convenient method of
`reliably entering information into a computer or computer
`system by scanning the barcode. They have become very
`common on a Wide variety of consumer goods and services.
`An eXample of When a barcode is used With a service is When
`a barcode is printed on a membership card. Some barcodes
`are printed on packaging, such as barcodes indicating the
`universal product code (“UPC”) of an item, and others are
`unique, such as a barcode printed on a parking garage ticket
`that indicates the date and time of When the ticket Was taken.
`The data in the barcode is generally represented as a pattern
`of tWo levels of surface re?ectivity, such as black and White
`stripes for a one-dimensional barcode. TWo-dimensional
`barcodes also use a pattern of alternating re?ectivities, but in
`a pattern of squares or other shapes. In either event, the data
`contained in the barcode is coded according to a binary
`coding system and the light and dark areas are interpreted as
`1’s and Us or vice versa.
`Barcodes are typically scanned by a scanner or reader
`(hereinafter “reader”). One type of reader has a lamp that
`uniformly illuminates the barcode, and an imaging sensor,
`such as a charge-coupled device (“CCD”) array, CMOS
`detector, or other electronic imaging method, that detects the
`scattered light that is re?ected off the barcode. Dark ink on
`light paper is a typical medium for presenting barcodes. The
`dark ink or paint provides loW-re?ectivity regions, and the
`light colored or matte (light scattering) paper provides
`high-re?ectivity regions. Of course, one may use light paint
`on a dark background.
`Another type of reader builds a map of surface re?ectivity
`by rastering a laser beam over the surface. A single photo
`diode detects the light re?ected from each position of the
`laser spot on the surface. This type of barcode reader is
`called a laser barcode scanner.
`Unfortunately, neither approach is currently Well-adapted
`to reading barcode images displayed on information display
`screens, such as the display screen of a Wireless telephone
`(“cell phone”) or personal digital assistant (“PDA”). Ren
`dering a barcode on a display of a cell phone or PDA Would
`be desirable because it Would relieve the consumer from
`needing to carry the bits of paper usually associated With
`tickets and coupons, for eXample.
`Display screens can be divided into tWo classes, re?ective
`and emissive. Re?ective screens are broadly de?ned as
`screens that alter their re?ectivity of ambient light to form an
`
`25
`
`35
`
`45
`
`55
`
`65
`
`2
`image, typically from light and dark piXels, such as passive
`black and White liquid crystal displays (“LCDs”). Emissive
`screens, such as backlit LCDs and cathode ray tubes
`(CRTs”) internally generate the light emitted from their
`surface. Conventional laser barcode scanners do not Work
`Well With emissive display screens because the emissive
`displays do not necessarily change their surface re?ectivity
`With image content. Imaging scanners may be able to read
`both types of screens, but might have dif?culty With re?ec
`tions off the surface of the display and insufficient contrast
`betWeen the light and dark portions of the scannable code.
`One reason re?ective display screens do not Work Well
`With conventional barcode readers is that the re?ective
`display screens are typically designed With the assumption
`that little light Will be incident from the direction of the
`vieWer. In normal use the vieWer’s head Will block light
`from this direction. Therefore, screens are often built With
`out antire?ection coatings on the front surface or at internal
`optical interfaces.
`HoWever, the light source built into laser barcode scan
`ners typically projects bright light from the direction of the
`camera or photodiode(s). This causes re?ections in the
`image observed by the camera that can interfere With the
`displayed image. In addition, the overall light levels of
`barcodes displayed on re?ective screens are different than
`for normal barcodes. For eXample, LCDs have internal
`polariZers that re?ect a much smaller fraction of incident
`light than a painted or White paper surface. To the scanner,
`both light and dark piXels may appear dark.
`Emissive screens may be easier for barcode scanners to
`read; hoWever, they too may have compatibility problems.
`Emissive screens often have a diffusing front coating or
`surface treatment of the glass to reduce specular re?ection of
`ambient light. Illuminating emissive screens With the lamps
`on a barcode scanner adds a scattered light background
`component to the light emitted from the screen. This back
`ground of scattered light reduces the contrast betWeen light
`and dark pixels. Additionally, specular re?ections of the
`light from the barcode scanner lamp to the camera or
`photodiode(s) can interfere With scanning as With re?ective
`display screens.
`If a barcode reader can’t scan the barcode the ?rst time,
`it typically tries again and again. Often, the scanning fails
`and the clerk must take additional time to enter the data that
`Would have otherWise been scanned into the computer.
`Therefore, a barcode reader that can quickly and reliably
`scan codes rendered on an electronic display screen is
`desirable. It is further desirable that the barcode scanner be
`able to read codes from emissive or re?ective screens.
`
`SUMMARY
`
`A barcode reader uses an imaging technique to quickly
`and accurately scan barcodes rendered on electronic dis
`plays. A shade on the barcode reader blocks ambient light
`from the CCD imager (“camera”) and con?gures the bar
`code reader illumination to be off-normal from the surface of
`the display. Avariety of CCD cameras are available from a
`variety of vendors, such as cameras sold under the trade
`name MEGAFILLTM by KODAK. The light detector array
`is typically integrated With on-chip electronics for interfac
`ing the camera to an instrument. In one embodiment, a CCD
`array With a 640 by 480 piXel detector array Was used With
`a ?ll factor greater than 80%; hoWever, it is possible to use
`cameras With loWer ?ll factors and feWer elements to read
`tWo-dimensional barcodes, especially if softWare ?ltering
`methods are used to enhance the image. The barcode reader
`
`Page 8 of 15
`
`ROTHSCHILD EXHIBIT 1008
`
`

`
`US 6,749,120 B2
`
`3
`can detect light emitted from the display to turn off scanner
`illumination, thus avoiding interference betWeen the emitted
`light and the scanner light and specular re?ections of the
`scanner light into the camera.
`
`In a further embodiment, the exposure time of the CCD
`camera is set to betWeen 2—20 times the vertical refresh
`period of a display. In a yet further embodiment, the refresh
`rate of the display is detected by the barcode reader and the
`exposure time of the CCD camera is adjusted accordingly to
`betWeen 2—20 times the measured refresh rate. This
`enhances accuracy of the scan by reducing errors caused by
`the display ?icker.
`
`The CCD imager uses automatic exposure or gain to
`optimiZe image contrast. In a particular embodiment,
`decoder softWare is used to set the detection threshold of the
`CCD camera. In an alternative or further embodiment, the
`scanner illumination is adjusted to achieve the desired
`contrast. In another embodiment, a CCD array With a Wide
`spectral response is used to reduce or avoid Moire patterns
`from color pixels.
`In another embodiment, the barcode scanner adjusts scal
`ing in response to displays With non-square pixels. In a
`particular embodiment an initial scan determines if a tWo
`dimensional barcode is rendered as the standard pattern of
`squares, or as a pattern of rectangles arising from rectangular
`display pixels. The scanner alters the horiZontal-vertical
`scaling to interpret the displayed rectangles as appropriate
`barcode elements.
`
`10
`
`15
`
`25
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`FIG. 1A is a simpli?ed cross section of a barcode reader
`according to an embodiment of the present invention applied
`to a display screen.
`FIG. 1B is a simpli?ed diagram illustrating the relation
`ship betWeen the illumination source, camera, and display
`screen.
`
`35
`
`FIG. 1C is a simpli?ed diagram of a barcode scanning
`system according to an embodiment of the present inven
`tion.
`
`FIG. 2A is a simpli?ed representation of a black and White
`pixel.
`FIG. 2B is a simpli?ed representation of a color pixel.
`
`45
`
`FIG. 2C is a simpli?ed representation of an image of the
`color pixel of FIG. 2B With a photopic camera.
`
`FIG. 3A is a simpli?ed ?oW chart of a process for
`detecting the type of display screen and setting a barcode
`scanner according to an embodiment of the present inven
`tion.
`
`FIG. 3B is a simpli?ed ?oW chart of a process for setting
`exposure time according to display refresh rate according to
`an embodiment of the present invention.
`
`55
`
`FIG. 3C is a simpli?ed ?oW chart of a process for
`optimiZing image contrast according to an embodiment of
`the present invention.
`
`FIG. 3D is a simpli?ed ?oW chart of a process for scaling
`non-square scanned barcode elements according to an
`embodiment of the present invention.
`
`FIG. 3E is a simpli?ed ?oW chart of a process for
`operating a barcode scanning system according to an
`embodiment of the present invention.
`
`65
`
`4
`DETAILED DESCRIPTION OF PREFERRED
`EMBODIMENTS
`
`1. Introduction
`The present invention provides a scanner for scanning
`computer-readable codes from an electronic display, such as
`tWo-dimensional barcodes. Scanners according to the
`present invention can incorporate both physical features,
`such alignment shrouds, illumination lamps, a high ?ll factor
`CCD camera With multiple pixel blocks for each barcode
`element, and a CCD camera With a Wide spectral response
`for use With color displays. Scanning systems according to
`the present invention can also include a number of operating
`features, such as dynamically adjusting the exposure time to
`overcome display ?icker or variations in brightness or
`re?ectivity of the display screens, automatically re-scaling
`the scanning system to adapt to displays With non-square
`pixels, and identifying display type and con?guring the
`scanner accordingly.
`2. An Exemplary Scanner
`FIG. 1A is a simpli?ed cross section of a barcode reader
`(“scanner”) 10 according to an embodiment of the present
`invention. The scanner is held against a display screen 12
`that is illustrated as being ?at, but could be curved. The
`display includes layers of glass or plastic 14, 16, Which may
`be coated. For example, if the display is an LCD display, one
`of the layers Would have a polariZation coating. The scanner
`is sectioned along section line 17 to facilitate illustration of
`the interior 15 of the scanner.
`A CCD camera 18 is mounted in the scanner, Which in this
`case includes a handle 20 With a trigger 21 that activates the
`scanner When depressed. A separate (from the imager)
`photodiode 13 is included to measure the emissivity of the
`screen, and the refresh rate in some embodiments. An
`illumination lamp 22 is mounted such that light emitted by
`the lamp is not directly re?ected off of the layers of the
`display back into the CCD camera. In a particular
`embodiment, the lamp is placed so that the light from the
`lamp, represented by the arroWs 26 is re?ected off the outer
`layer 14 of the display at an oblique angle to avoid direct
`re?ection into the CCD camera, as represented by the arroWs
`30. Generally, the lamp is placed on the center of the lateral
`axis above the CCD camera, but could be placed in a
`sideWays oblique orientation. A cable 32 connects the scan
`ner to a poWer supply and camera control and imaging
`electronics. A ?at display is shoWn for simplicity of illus
`tration. If a display is curved, it is typically only slightly
`curved, and not sufficiently to alter operation of the scanner.
`Some conventional scanners have a built-in laser that
`generates a crosshair pattern over the imaged region to assist
`the user in aiming the scanner at the barcode. The image is
`intended to re?ect off of a matte surface and does not shoW
`up Well When projected onto some types of display screens,
`Which can be both transparent and smooth. For reading
`displays, a mechanical alignment system, such as a Wire
`frame attached to the scanner, should be used instead of the
`optical alignment marker. Alternatively, the CCD camera
`can be used to shoW the imaged region on a second computer
`display screen vieWed by the sales clerk, so that the clerk
`sees What the scanner is imaging and adjust the position of
`the scanner accordingly.
`Scanners according to one embodiment of the present
`invention use an alignment shroud 34 that serves at least tWo
`functions. First, the sides of the shroud block ambient light
`(such as room lights) from entering the scanner and causing
`an error in reading the light level or causing bright specular
`re?ections off the screen in the camera ?eld of vieW. Second,
`
`Page 9 of 15
`
`ROTHSCHILD EXHIBIT 1008
`
`

`
`5
`the shroud aligns the CCD camera 18 and illumination lamp
`22 relative to the surface 36 of the display screen 12 so that
`the bright light from the illumination lamp is not directly
`re?ected into the CCD camera. Thus, the illumination lamp
`illuminates the display screen Within the aperture of the
`CCD camera, but does not directly re?ect into the camera.
`OtherWise, the result Would be similar to pointing a video
`camera at a bright light. The specularly re?ected light Would
`saturate a portion of the camera ?eld of vieW, preventing
`imaging of the barcode. The light from the illumination lamp
`should be incident to the surface of the display at an angle
`greater than the maximum angle measured from the edge of
`the imaged area nearest the illumination lamp. For example,
`if the camera vieWing angle is betWeen 10—13 degrees from
`normal to the screen, the light source should be more than
`13 degrees from normal to any point on the screen.
`The CCD camera is preferably a high ?ll factor CCD
`array With a ?ll factor of at least 80%. It is further preferable
`that the CCD camera has a Wide spectral response, generally
`from about 400 nm to 700 nm in order to avoid Moiré
`patterns that might otherWise form When scanning a color
`display screen that is generally designed to be vieWed by a
`person. Moiré patterns and similar optical artifacts can
`combine With the intended barcode pattern to cause scanning
`errors, and even a complete inability to recogniZe the
`barcode.
`It is particularly desirable that the sensitivity in the range
`of 400—700 nm not fall beloW 10% of the peak sensitivity
`Within this range. Ideally, the detector Would image the red,
`green, and blue pixels of a White display screen With equal
`sensitivity. This reduces the brightness variation across the
`complete pixel as measured by the camera. (The actual
`sensitivity to the 3 subpixel colors may vary slightly due to
`variations in subpixel spectra among different display
`types.) In a particular embodiment, a color ?lter betWeen the
`detector array and the display screen compensates for
`Wavelength-dependent sensitivity of the detector.
`Speci?cally, the ?lter preferentially absorbs the middle
`(green portion) of the range compared to the ends (red and
`blue) since a photopic camera is more sensitive to the light
`emitted by the green subpixels of a typical display. It Was
`found that near infra-red detectors used on conventional
`linear scanners did not Work Well because the display screen
`absorbed most of the light at the operating Wavelength of the
`scanner.
`It is generally desirable that tWo-dimensional barcodes
`rendered to the electronic display use only square-format
`elements, such as tWo-dimensional barcodes according to
`the DATAMATRIXTM format or according to the AZTECTM
`format. Other barcode formats, such as PDF417TM, Which is
`a stacked linear (non-square) format, can also be used. For
`example, a tWo-dimensional barcode according to the
`AZtecTM format can be 15 by 15 up to 151 by 151 display
`elements (squares).
`When rendering a tWo-dimensional barcode to a display,
`it is generally desirable to display each barcode element as
`an N><M block of pixels to reduce the effect of Moiré
`patterns, particularly on display screens With loW ?ll factors.
`N and M are integers, and typically N=M; hoWever, in some
`instance, such as With non-square display screen pixels, N
`might not equal M. Examples of portable electronic devices
`With displays having non-square pixels include portable
`high-de?nition television (“HDTV”) and digital video disk
`(“DVD”) players. Each element of the barcode is repre
`sented by one pixel, or by 2x2, 3x3 or similar groups of
`pixels. Barcode standards typically require that all elements
`of the barcode be of equivalent siZe so arbitrary non-integer
`
`35
`
`45
`
`55
`
`65
`
`US 6,749,120 B2
`
`15
`
`25
`
`6
`scaling of the barcode onto the display screen is not possible.
`HoWever, the present invention overcomes this limitation by
`scaling the high-resolution image obtained by the CCD
`camera using post-imaging softWare.
`By making the barcode elements larger than one pixel, the
`feature of the pixels are made smaller relative to the barcode
`elements. This does not actually suppress interference, but
`makes the effects of interference easier to remove using
`?ltering softWare, or to separate the imaged interference
`from the barcode features. For example, a “Gaussian blurr”
`or a tWo-dimensional Fourier Transform ?lter may be used
`to remove high spatial frequencies, such as Would lie above
`the image artifacts associated With the pixels, While retain
`ing the details of the barcode symbol. The ?ltering tech
`niques may be adaptive, adjusting the ?ltering parameters to
`the interference pattern present in the image data.
`FIG. 1B is a simpli?ed diagram illustrating the relation
`ship betWeen the illumination source 18, camera 22, and
`display screen 12. For simplicity of illustration, it is assumed
`that the camera lens is relatively small so that it can be
`represented as a point, and that the screen is ?at and
`perpendicular to the center axis 19 of the camera. Those
`skilled in the art Will appreciate that this illustrative example
`can be adapted to screens that are slightly curved and to
`cameras With typical lens diameters, and to situations Where
`the center axis of the camera is not precisely perpendicular
`to the screen.
`Abarcode image 21 displayed on the screen is represented
`in side vieW as a solid dark region. The camera has a vieWing
`angle (26) that is sufficient to image the entire barcode
`image. Ideally the vieWing angle is slightly larger than the
`barcode image to facilitate alignment of the camera, as
`shoWn by the solid lines 27, 29. At a minimum it is the same
`as the siZe of the barcode as shoWn by the dashed lines 23,
`25. Theta is typically betWeen 10—13 degrees for a distance
`d betWeen the screen 12 and the camera lens of about 15—20
`cm and a Width (2s) of the imaged region being about 14 cm.
`The general case is that the illumination source should be
`above a limit line 31 starting at the edge of the imaged region
`33 and extending aWay from the screen at an angle from
`normal of inverse tangent (s/2d). This con?guration avoids
`specular re?ections into the camera.
`3. An Exemplary Scanning System
`FIG. 1C is a simpli?ed diagram of a barcode scanning
`system 40 according to an embodiment of the present
`invention. A scanner 10, such as is described above in
`reference to FIG. 1A, is coupled to a computer 42 or
`computer netWork. The computer includes a processor 44
`and memory 46, such as random-access memory (“RAM”),
`read-only memory (“ROM”), compact-disk ROM, magnetic
`hard drive, magnetic ?oppy drive, and other types of
`memory and combinations thereof as are generally knoWn in
`the art. A computer-readable program 48 stored in the
`computer memory 46 con?gures the system to execute
`various tasks under the control of the processor 44. Addi
`tional input to the computer may optionally be provided With
`a user input device 50, such as a keypad or mouse used in
`conjunction With a screen, as is Well knoWn in the art.
`The scanning system is adapted in a number of fashions
`to scan barcodes from display screens. In one embodiment
`of the present invention, the CCD camera in the scanner 10
`detects Whether any light is being emitted by the display, ie
`whether the display is an emissive display. This is done by
`starting With the illumination lamp off. If no emitted light is
`detected, then the system presumes that the display is a
`re?ective display and the lamp is turned on.
`
`Page 10 of 15
`
`ROTHSCHILD EXHIBIT 1008
`
`

`
`US 6,749,120 B2
`
`7
`In another embodiment the system dynamically compen
`sates for brightness variations across the display screen. For
`example, emissive screens often exhibit a brightness varia
`tion of up to 30% across the screen. This variation does not
`affect vieWing by a typical user, as the human eye generally
`compensates for the variation. HoWever, this variation can
`cause the light signals from light elements of one region of
`the barcode and dark elements of another region to be
`overlap in magnitude. This makes determination of light and
`dark elements by comparison With a threshold signal level
`during a scan inaccurate.
`The CCD camera is coupled through a cable 32 to control
`electronics 52 that can be adjusted by the computer accord
`ing to instructions stored in memory and/or by dedicated
`memory, such as ROM in association With local processor,
`such as an application-speci?c integrated circuit. The con
`trol electronics can be integrated With the computer system,
`or coupled to the computer With a cable 54, as shoWn. The
`amplitude of the signal from detectors or detector blocks can
`be monitored to automatically adjust the gain in the control
`electronics to insure that the signals from a light barcode
`element are Within a preselected range as the barcode is
`scanned. After the gain is adjusted to achieve a desired
`exposure range, the softWare can compensate for lighter or
`darker regions of the image arising from variations in the
`brightness or re?ectance of a display.
`In the case of a re?ective display, the re?ectance and/or
`contrast can also vary across the screen, Which can be
`compensated With softWare, as above for emissive displays,
`and/or the brightness of the illumination lamp can be
`adjusted. Thus, the re?ected light Will produce an output in
`a desired range. The illumination can also be adjusted
`according to the detected contrast betWeen light and dark
`code elements, With or Without associated gain control.
`Preferably, the gain of the CCD and the light level should
`be adjusted so that the brightest portion of the image does
`not saturate the detector. For example, if the detector has 8
`bits of resolution, the detected brightness should be in the
`0—255 range or a slightly smaller range. The image should
`not be too dim (eg all data in the 0—8 range) or loW contrast
`(eg all data in the 5—16 range) or too bright (e.g. Where
`many pixels Would read at 255).
`There are quite a feW methods for adjusting the
`brightness, including adjusting the gain of the detector, the
`exposure time, the brightness of the illumination source
`(When scanning re?ective displays) and the camera lens
`aperture, if such control is available. Contrast adjustment
`and compensation for brightness variations across an image
`can be adjusted using computer softWare methods. For
`example, each pixel could be compared to the average of all
`pixels out to some ?xed distance from the pixel to determine
`if it is brighter or darker than the local average to determine
`if it is part of a dark or light barcode element. Barcodes are
`typically designed so that there are never large contiguous
`regions of all light or all dark pixels. Without threshold
`brightness adjustment across the scanned image, a barcode
`rendered on a non-uniform screen might have one end
`portion that is all light or all dark.
`Another feature of the scanning system is that it can adapt
`to non-square display pixels. The scanning system includes
`softWare that adjusts the resolution and scale of the scanned
`data to accommodate rectangular pixels. For example,
`although various tWo-dimensional barcode standards require
`square elements (Within a speci?ed tolerance), the reader
`softWare of the scanning system can accommodate non
`square elements. The lack of squareness might arise from
`
`10
`
`15
`
`25
`
`35
`
`45
`
`55
`
`65
`
`8
`non-square pixels on the display, and/or image distortion,
`that might arise from a slightly curved screen or the scanner
`being held off-angle (not perpendicular) to the barcode
`image. Initially, the softWare steps Would recogniZe and
`locate a barcode image. Then, the softWare Would locate
`portions of the image that might represent barcode elements,
`and then measure the siZe of those portions. For example,
`DATAMATRIXTM barcodes have a particular border pattern.
`These measurements are then used to identify the individual
`element locations of the barcode Within the image, correct
`ing for any distortions. It is possible that the maximum
`alloWable distortion tolerance speci?cation may be broad
`ened compared to corresponding printed (paper) barcodes.
`5. Scanning of Color Pixels
`In a further embodiment, the scanning system further
`corrects for differences betWeen black and White displays
`and color displays. Referring to FIG. 2A, in black and White
`displays a pixel 80 typically consists of a square active
`region 82 surrounded by a border 84. For typical LCDs and
`CRTs the active region exceeds 50% of the pixel area, and
`can be as high as 90% in some displays. HoWever, in color
`LCD screens pixels are divided vertically into three colored
`subpixels that combined make up about 30—60% of the pixel
`area. The subpixels emit or re?ect red, green, and blue light,
`Which are mixed to display other colors.
`In CRTs the red, green, and blue regions may be small
`dots or stripes of colored phosphor. There may be more than
`one dot of each color in each complete subpixel. When the
`colored elements of the subpixel are imaged With the mono
`chrome CCD camera in the scanner, the colors Will appear
`as different shades of gray. Black and White (monochrome)
`cameras are often designed to mimic the color response of
`the human eye. These cameras are called photopic cameras.
`Imaged With a photopic camera, the green components of a
`White color subpixel Will appear brighter than the red and
`blue components. Some cameras are more sensitive to
`near-infrared light and Will image the red portion as bright
`est.
`FIG. 2B illustrates the internal structure of a White
`colored pixel 86 on a backlit color screen. The pixel includes
`a red stripe 88, a green stripe 90, and a blue stripe 92. FIG.
`2C illustrates an image 94 of the White-colored pixel shoWn
`in FIG. 2B as imaged With a photopic black and White
`camera. All subpixel images are shades of gray, the imaged
`green subpixel 90‘ being the brightest, and the imaged blue
`subpixel 92‘ being the darkest. The imaged color pixel
`generally has reduced brightness, thus reducing contrast
`betWeen the light and dark code elements. In a preferred
`embodiment, the CCD camera is not a photopic camera, bu