I lllll llllllll II llllll lllll lllll lllll lllll lllll lllll lllll 111111111111111111111111111111111
`US 20020075389Al
`
`(19) United States
`(12) Patent Application Publication
`Seeger et al.
`
`(10) Pub. No.: US 2002/0075389 Al
`Jun. 20, 2002
`(43) Pub. Date:
`
`(54) APPARATUS AND METHOD FOR
`CAPTURING A DIGITAL IMAGE
`
`(75)
`
`Inventors: Mauritius Seeger, Royston (GB);
`Stuart A. Tuylor, Cambridge (GB)
`
`Correspondence Address:
`John E. Beck
`Xerox Corporation
`Xerox Square-20A
`Rochester, NY 14644 (US)
`
`(73) Assignee: Xerox Corporation
`
`(21) Appl. No.:
`
`09/737,965
`
`(22) Filed:
`
`Dec. 18, 2000
`
`Publication Classification
`
`Int. Cl.7
`(51)
`......... ........................ ......... ...... ..... H04N 5/228
`(52) U.S. Cl. . ........................................................ , .... 348/222
`
`(57)
`
`ABSTRACT
`
`A camera apparatus obtains a sharp, high-resolution image
`of an object which is difficult to focus in a single image at
`a fixed focus, such as the imaging of a document using a
`document camera held at an oblique angle to the document.
`The apparatus composites an image of an object from plural
`image segments of the object acquired at different focusing
`distances. Higb quality regions (i.e. sharply focused and/or
`high resolution) are extracted from each image to form the
`segments for compositing.
`
`3g
`,------------------------------------ ---·----------- --------------------
`I 4Lt.
`
`I
`I
`I
`I
`
`l'vfEMORY
`
`CCD
`
`ZOOM
`
`'
`I
`I
`I
`I
`I
`I
`I
`I
`I
`
`I 42
`
`I/ p
`
`DEWARPING
`
`INTER-
`FACE
`
`:
`' :
`
`I
`I
`I
`
`ANALYSIS
`COMPOSITING
`-------------------------------------------------------------------------'
`
`I
`I
`
`0001
`
`exocad GmbH, et al.
`Exhibit 1011
`
`

`

`Patent Application Publication
`
`.Tun. 20, 2002 Sheet 1 of 4
`
`US 2002/0075389 Al
`
`12
`
`14
`
`10
`
`FlG-.1
`
`'Fl(r. 2
`
`18
`
`Fl&.~
`
`. '·
`
`0002
`
`

`

`Patent Application Publication
`
`Jun. 20, 2002 Sheet 2 of 4
`
`US 2002/0075389 Al
`
`t.ffti r.,... - L ~
`I
`I
`
`FlG.4
`
`r---------------------------------- ------------------- --~
`44
`: I
`' I
`
`- - - - - - I
`
`:MEMORY
`
`CCD
`
`FOCUS
`
`I /P
`
`INTER·
`FACE
`
`~40
`1 - - - - - - - - - -<
`so
`DEWARPING
`52
`ANALYSIS
`51t
`CO:MPOSITING
`'- ----- ---------------------- - ------------------~----------
`FlC:r.5
`
`0003
`
`

`

`Patent Application Publication
`
`.Jun. 20, 2002 Sheet 3 of 4
`
`US 2002/0075389 Al
`
`Go .
`
`&2
`
`Acquire images ll!ling sweeping focus
`
`Find geometric transform for each image
`
`Determine registration of images
`
`Find regions of low blur
`
`(o ~
`
`Composite image using regions of best focus
`
`Camera Orientation:
`Focus
`Depth from defocus
`Accelc:rometc:r
`Ranging
`Motion detection
`
`FIG-. t0
`
`,------- --- ---- ----- ------------ -- -- ~ --------------- -----·-··-·-----·--i
`' i Lia
`
`. . . - - . . . . / - - - - .
`
`I
`I
`I
`I
`
`ZOOM
`
`MEMORY
`
`CCD
`
`4-2
`
`I/ p
`
`DEWARPING
`
`INTER(cid:173)
`FACE
`
`I
`
`ANALYSIS
`COMPOSITING
`----·-·--------------------------------------------------------------- ---·
`
`FIG-.7-
`
`I
`I
`I
`
`I ' • I
`
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`
`Determine Zoom Range
`
`Acquire images usin& sweeping focus & zoom
`
`Find geometric transfoml for each image
`
`Determine registration of images
`
`Find regions of!ow blur&. high resolution
`
`Composite image using regions of best focus &.
`resolution
`
`Camera Orientation:
`Focus
`14--- - - l Depth from defocus
`Accelerometer
`Ranging
`Motion detection
`
`Fre:r. g
`
`0004
`
`

`

`Patent Application Publication
`
`.Jun. 20, 2002 Sheet 4 of 4
`
`US 2002/0075389 Al
`
`\
`
`\
`
`I
`
`. 8
`
`'I
`
`0005
`
`

`

`US 2002/0075389 Al
`
`Jun.20,2002
`
`1
`
`APPARATUS AND METHOD FOR CAPTURING A
`DIGITAL IMAGE
`
`BACKGROUND OF THE INVENTION
`
`[0001] 1. Field of the Invention
`
`[0002] This invention relates to capturing a digital image .
`In a preferred form, the invention relates to a digital camera
`and to a method of operation.
`
`[0003] The invention is especially suitable for use with, or
`for inclusion in, so-called document cameras for capturing
`digital images of documents, for example, for storage, or for
`processing by optical character recognition (OCR). How(cid:173)
`ever, the invention is not limited only to such a field, and
`may find application for use with, or for inclusion in, general
`digital-photography cameras.
`
`[0004] The invention is also especially suitable for use
`with, or for inclusion in, handheld cameras, but it is not
`limited exclusively to such cameras.
`
`[0005] 2. Description of Related Art
`
`[0006] Many designs of camera for capturing a digital
`image of a document are known, including hand-held cam(cid:173)
`eras.
`
`[0007] However, when using a hand-held document cam(cid:173)
`era, the camera will often be held at an oblique angle relative
`to the document (in other words, it is often impractical to
`hold the document camera in a plane parallel to the plane of
`the document). In such a case, the captured document image
`can suffer from distortion including perspective distortion,
`and from out of focus blur. Although perspective distortion
`may be corrected by dewarping techniques, this can lead to
`low-resolution and poor image quality. In addition, out-of(cid:173)
`focus blur may be present in parts of the image due to the
`oblique angle.
`
`[0008] An example of such problems in a captured image
`is illustrated in FIGS. 1, 2 and 3 . FIG. 1 depicts a camera
`operator 10 holding a camera 12 at an oblique angle to
`capture an image of a document 14.
`[0009] FIG. 2 shows a typical captured document image
`16 in such a case. It is immediately evident that the captured
`image suffers from perspective distortion. The edges of the
`text columns are not "vertical" (i.e. perpendicular to the text
`lines); the text appears compressed in a horizontal direction
`and the text varies in size from top to bottom of the image;
`and individual letters incline towards the edges of the image .
`
`[0010] Dewarping techniques arc known for geometrically
`transforming an image to correct the perspective distortion.
`For example, the image may be dewarped by expanding the
`horizontal image width progressively from bottom to top,
`and also by expanding the image vertically to correct the
`horizontal compression.
`[0011] FIG. 3 shows the result of such a dewarping
`technique applied to the image of FIG. 2. Although the
`perspective is restored, the image contains poor quality
`regions 18 and 20 which suffer from out-of-focus blur. The
`upper region 18 of the document is too distant from the
`camera to be focused correctly, and the lower region 20 of
`the document is too close to the camera to be focused
`correctly. Only the central region 22 of the image is of clear
`quality. Additionally, the resolution of the upper (distant)
`
`portion 18 is very low as a result of the perspective distortion
`(which causes distant portions to appear smaller, and hence
`have a reduced resolution).
`
`[0012] The above problem is not limited to document
`cameras. There are many situations in which it is impossible
`to capture an image in which an object is in focus throughout
`the image. For example, the object may be too large to be
`focused correctly. Additionally, it often impossible to cap(cid:173)
`ture both a foreground and background together in focus.
`
`[0013] Although not relevant to the present field, reference
`may be made to the bar-code readers described in U.S. Pat.
`Nos. 5,798,516 and 5,386,107. These documents describe
`arrangements for reading barcodes at unknown distance
`ranges. However, these documents do not address the prob(cid:173)
`lem of achieving a completely blur free image of an object
`at an oblique angle which may never be in perfect focus.
`
`SUMMARY OF Tl-IE INVENTION
`
`It would be advantageous to overcome or reduce
`[0014]
`the above problems.
`
`[0015] A first aspect of the invention addresses the prob(cid:173)
`lem of out-of-focus blur in images. Broadly speaking, in
`contrast to the prior art technique of capturing an image at
`a fixed focus, one aspect of the present invention is to
`composite an image of an object from plural image segments
`of the object acquired at different focusing distances.
`
`[0016] Such a technique can avoid the problems associ(cid:173)
`ated with out-of-focus blur occurring in images which are
`difficult to capture at a fixed focus.
`
`In one form, the invention provides a technique in
`[0017]
`which plural images of an object are acquired at different
`focus distances, and the composited image is composited
`from plural segments derived from the plural captured
`images.
`
`[0018] By acquiring plural images at different focus dis(cid:173)
`tances, there is a much higher probability that a region of one
`image which suffers from out-of-focus blur will be sharply
`focused in another captured image. Also, by compositing the
`optimum quality segments from the different captured
`images, a final image can be produced which would be
`impossible to capture in a single image with a fixed focus.
`
`[0019] Preferably, the apparatus includes a processor for
`determining a geometric transform to apply to a captured
`image (or to a region thereof) to correct for image distortion
`(e.g., perspective distortion). Such a correction transform is
`also referred to herein as dewarping. The composited image
`is thus composited from perspective corrected segments, to
`produce a perspective corrected image.
`
`[0020] Preferably, the apparatus comprises an image ana(cid:173)
`lyzer for analyzing the captured images for selection of a
`segment therefrom to use in the composited image. Prefer(cid:173)
`ably, the image analyzer analyses the quality of one or more
`regions of each captured image; indicative of the quality
`may be the sharpness of the image region.
`
`[0021] Preferably, the apparatus comprises a variable
`focus mechanism which is controlled to vary the focus
`distance as the plural images are acquired.
`
`In a particularly preferred form, the invention also
`[0022]
`addresses the problem of reduced resolution resulting from
`
`0006
`
`

`

`US 2002/0075389 Al
`
`Jun.20,2002
`
`2
`
`perspective distortion of relatively distant portions of an
`object. To address this, the apparatus preferably comprises a
`zoom mechanism for varying the magnification at which the
`image is captured, and means for controlling the zoom
`mechanism.
`[0023]
`In one form, the zoom mechanism may be con(cid:173)
`trolled in accordance with the focusing distance.
`[0024] This can enable more distantly focused portions of
`an object to be acquired al a magnified resolution, to
`compensate at least to some degree for loss of resolution
`caused by perspective distortion of the distant portion.
`[0025] A highly preferred feature of the invention, in
`whichever form it is used, is that the apparatus comprises a
`device for determining the registration of one captured
`image (or image segment) with another. In other words, the
`device identifies one or more points of registration between
`the images, so that the relative alignment and positions of
`the captured images is known. This is advantageous to
`enable the quality of image regions lo be compared accu(cid:173)
`rately in the different captured images, and to enable image
`segments to be selected and composited together to form a
`seamless composited image.
`[0026]
`In a preferred form, the invention is implemented
`in a digital camera . However, in an alternative form, at least
`a portion of the image processing (e.g. dewarping, registra(cid:173)
`tion, quality analysis and compositing) may be performed
`using a separate image processor external to the camera.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`[0027] Two non-limiting embodiments of the invention
`are now described, by way of example only, with reference
`to the accompanying drawings, in which:
`[0028] FIG. 1 is a schematic view of a user capturing an
`image of a document;
`[0029] FIG. 2 is a schematic image of the document
`captured in FIG. 1;
`[0030] FIG. 3 shows the effect of dewarping the image of
`FIG.2;
`[0031] FIG. 4 is a schematic perspective view of a docu(cid:173)
`ment camera;
`[0032] FIG. 5 is a block diagram showing some of the
`functional elements of the camera of FIG. 4 including a
`variable focus mechanism;
`[0033] FIG. 6 is a flow diagram showing the principle of
`operation of the camera of FIG. 4;
`[0034] FIG. 7 is a block diagram showing some of the
`functional elements of a second embodiment of the camera;
`[0035] FIGS. 8(a)-(d) are schematic diagrams of images
`captured by a camera; and
`[0036] FIG. 9 is schematic representation of the final
`image composited from the images of FIG. 8.
`
`DETAILED DESCRIPTION
`
`[0037] Referring to FIGS. 4 and 5, a first embodiment of
`a document camera 30 comprises a case 32 carrying an
`objective lens 34, and housing a photoelectric detector 36
`(typically a charge coupled device (CCD)), a focus mecha-
`
`nism 38 for controllably varying the focusing distance of the
`Jens 34, a control and proces.<;ing circuit 40, one or more user
`inputs 42 including a "capture" button, and a storage device
`44 for storing captured images. The storage device may
`consist of any suitable storage medium, for example, a
`semiconductor memory, or an optical medium, or a magnetic
`medium .
`[0038] The camera additionally comprises an interface 46
`(e.g . a connection port or a wireless interface) for uploading
`images from the camera and/or for downloading infonnation
`or images into the camera. Additionally, the camera 30 may
`comprise a display 48 for displaying images.
`[0039] One of the operating principles of this embodiment
`(described in more detail below) is to capture plural images
`of an object taken at different focus settings. Each image can
`be processed to determine a geometrical transformation to
`correct for image distortion. The images are analyzed to
`identify or quantify the quality of one or more regions of the
`image. Dewarped segments from the plural images are then
`composited, according to the quality of the segments, to
`form a final composited image.
`[0040] By capturing plural images at different focus set(cid:173)
`tings, many of the prior art problems of a single image at a
`single focus setting can be avoided. The final image is
`generated by compositing together optimum quality seg(cid:173)
`ments from the different captured images.
`[0041]
`In this embodiment, the control and processing
`circuit 40 comprises a dewarping processor 50 for deter(cid:173)
`mining a geometric transform for correcting or dewarping an
`image, an image analyzer 52 for performing image regis(cid:173)
`tration and quality analysis, and an image compositor 54 for
`compositing the final image. Although Lhe elemenL<; of Lhe
`circuit 40 are shown as separate functional parts, it will he
`appreciated that the control circuit may comprise a processor
`and executable code for perfonning one or more of these
`functions.
`[0042] The above image capture/dewarping/analysis/com(cid:173)
`position process is described in more detail with reference to
`FIG. 6. The process starts at step 60 when the camera
`operator presses the "capture" or "shu\ler release" bullon of
`the camera. Al step 60, the control circuit 40 controls the
`camera to capture plural images of the object taken al
`different focus settings. In this preferred embodiment, the
`focus settings are swept over the focusing range of the
`camera. Typically, the number of images captured would be
`about 3, 4 or 5. However, this range is merely an example;
`the number may be smaller or greater, and may depend for
`example, on the range of possible focus settings, or on a user
`settable parameter, or on the quality results of previous
`images.
`
`[0043] The plural images are preferably acquired suffi(cid:173)
`ciently quickly to avoid large motions of the hand-held
`camera. However, some camera motion may still occur.
`
`[0044] At step 62, each captured image is processed by the
`dewarping processor 50, to determine a geometric transform
`to correct the image for one or more of perspective distor(cid:173)
`tion, scaling, rotation, barrel distortion, and page warp. The
`dewarping transform may be derived only on the basis of the
`image itself (e.g. based on identifying straight columns and
`lines of text, or based on the size ratio of letters). Alterna(cid:173)
`tively, it may be faster and more reliable to use additional
`
`0007
`
`

`

`US 2002/0075389 Al
`
`Jun.20,2002
`
`3
`
`information regarding the object and the relative position
`and/or orientation of the camera. Although not shown
`explicitly in FIG. 5, one or more of the normal camera
`sensors may be used, for example, accelerometers, range
`sensors, focus, motion detection, etc. For example, the
`amount of perspective could be inferred from the detected
`orientation of the camera, it being assumed that the docu(cid:173)
`ment is lying horizontal. Alternatively, the camera could be
`orientated initially to be "parallel" to the plane of the
`document, and then moved to the desired more comfortable
`orientation of use at which the images are to be captured. By
`detecting the initial orientation and the orientation ofuse, the
`degree of perspective can be inferred.
`
`[0045] Allemalive !echniques are also known based on the
`projection of a known image shape on to the document, from
`a position offset from the optical axis. The parallax between
`the projection position and the optical axis of the camera
`causes the image to have a different shape when viewed
`along the optical axis of the camera. The difference between
`the viewed shape and the known projected shape provides a
`direct indication of the perspective distortion, and also other
`distortions such as page curvature. Generally such a tech(cid:173)
`nique is performed by capturing one or more images prior to
`the main image captures, and the projected image is turned
`off during the main image captures so as not to interfere with
`the object. More information about this type of technique
`can be found in U.S. Pat. No. 5,835,241, and also in
`DoncescuA. ct al, "Former Books Digital Processing: Image
`Warping", Proc. Workshop on Document Image Analysis,
`San Juan, Puerto Rico, Jun. 20, 1997, Eds. L. Vincent & G.
`E. Kopec. The teachings of these documents are incorpo(cid:173)
`rated herein by reference.
`
`[0046] Many allemalive algorithms for dewarping images
`lo correct geometrically for perspective distortion, scaling,
`rotation, barrel distortion and page warp, are well known to
`one skilled in the art, and need not be described in detail
`herein.
`
`[0047] The output from the dcwarping processor 50 may
`either be a geometric transform (to be applied later), or it
`may be in the form of a dewarpcd image to which the
`transform has already been applied.
`
`[0048] At step 63, the images (whether or not dewarped)
`are processed by the image analyzer to identify the regis(cid:173)
`tration or correspondence of one image with respect to
`another. In the present embodiment, the camera's field of
`view does not change between image captures, and so any
`difference between image correspondence from one image
`to the next is a direct result of camera movement (normally
`accidental, but not necessarily).
`
`[0049] A suitable registration algorithm is described, for
`example, in "An interactive image registration technique
`with an application to stereo vision" by B. D. Lucas and T.
`Kanade, Proc. DARPA Image Understanding Workshop
`1981, pages 121-130. Other suitable registration algorithms
`are known to one skilled in the art, and so need not be
`described here in detail.
`
`[0050] Depending on the embodiment, the registration
`may be carried out either on the images without dewarping,
`or on dewarped images. If the perspective distortion is the
`same (or is assumed to be the same) in each captured image,
`then the registration can be carried out on the original
`
`images without dewarping. However, if all situations are to
`be fully accommodated, then the registration can be carried
`out on the dewarped images.
`
`IL will be appreciated lhat, if desired, movement of
`[0051]
`the camera may be detected, for example, by one or more
`camera accelerometers, and such movement could be pro(cid:173)
`vided as an input to aid registration.
`
`[0052] At step 64, the images (whether or not dewarped)
`are processed by the image analyzer 52 to identify the
`quality of image regions for selection lo be included in the
`final image. For example, image blur can be identified using
`a maximum variance test or an analysis of the frequency
`components in the image. Correctly focused areas have high
`frequency components and high variance.
`
`[0053] The analysis step may, for example, grade a region
`of an image according to its quality, or it may simply identify
`one or more regions which are suitable for the final image.
`The analyzer can also determine the relative qualities of an
`image region in the different captured images, to determine
`which captured image will provide lhe highest quality
`segment for composition.
`
`[0054] Al step 66, the image compositor 54 composites
`segments selected from the captured images to form a final
`image. If the geometric transforms have not yet been
`applied, then these are applied to each segment during
`composition of the final image. The image compositor
`preferably selects regions of highest quality to form the
`segments of the image, to provide the best possible com(cid:173)
`posited image from the available captured images.
`
`[0055] A second embodiment of the invention is illus(cid:173)
`lraled in FIG. 7. Ibis is similar lo lhe first embodiment
`described above, and like reference numerals are used where
`appropriate.
`
`[0056] The second embodiment further improves on the
`first embodiment, by enabling the resolution of distaDI
`portions of an image to be increased. In the first embodi(cid:173)
`ment, the resolution of an image is constant. This means that,
`as a result of perspective distortion at an oblique angle,
`distant portions of an object will appear smaller than, and
`hence will have a reduced resolution relative to, close
`portions. Even when a distant portion is correctly focused in
`the first embodiment, the resolution might in certain circum(cid:173)
`stances (particularly when the camera angle is very oblique)
`be insufficiently high to obtain a sharp image once the image
`is dewarped.
`
`[0057] Referring to FIG. 7, in the second embodiment, the
`camera further comprises a zoom mechanism 68, for adjust(cid:173)
`ing lhe focal length (and hence the magnification) of lhe lens
`assembly, under the control of the control circuit 40. By
`providing a zoom facility, it is possible to capture distant
`portions of an object at a higher resolution, in order to
`compensate for loss of resolution caused by perspective
`distortion.
`
`[0058] Referring to FIG. 8, the image capture process is
`very similar to that of FIG. 6 except that, at step 60a, the
`control circuit 40 controls the zoom mechanism such that
`plural images are acquired at different zoom settings.
`Although it is possible to vary the focus and zoom settings
`independently, in this preferred embodiment the zoom level
`is controlled lo increase as the focusing distance increases
`
`0008
`
`

`

`US 2002/0075389 Al
`
`Jun.20,2002
`
`4
`
`(or to decrease as the focusing distance decreases). Distant
`focused regions of the object are thus automatically captured
`al a higher resolution.
`
`It will be appreciated that the amount of zoom
`[0059]
`required to compensate for the loss of resolution caused by
`perspective distortion, will depend on the amount of per(cid:173)
`spective distortion itself. In other words, the appropriate
`level of zoom will depend on the angle of the camera relative
`to the document. For example, camera is held at a very
`oblique angle relative to the document, the perspective
`distortion is severe; however, at a less oblique angle, the
`amount of perspective distortion (and the amount of zoom
`required to compensate for resolution) is reduced.
`
`In the present embodiment, the amount of zoom is
`[0060]
`inferred at step 60a from camera sensors (e.g. accelerom(cid:173)
`eters) which indicate the angle at which the camera is held
`(it being assumed that the document lies horizontally).
`Alternatively, an additional step 69 may be included prior to
`step 60a. Al step 69, a low-resolution image of the object is
`acquired, and is processed to obtain optical information
`indicative of the amount of perspective distortion in the
`image. (For example, referring to FIG. 2, such optical
`information may be the inclination of edges of Lexi or other
`objecl identifiable in the low-resolution image). However, it
`will be appreciated that such information could also be
`inputted to the camera manually by the camera user.
`
`[0061] At step 62a, the zoom level is included as part of
`the determination of the geometric transform, in order for
`the correction to match the zoomed image.
`
`[0062] At step 64a, the analysis also takes into account the
`resolution of image regions. The purpose of the analysis is
`to determine in-focus regions which also have high resolu(cid:173)
`tion, such that the image will still be sharp after dewarping.
`
`[0063] An example of an image capture and processing
`using the second embodiment is shown in FIGS. 9 and 10.
`FIGS. 9a-9d depict a sequence of four images captured as
`the focus is varied from a distant setting (FIG. 9a) through
`progressively closer distances (FIGS. 9b and 9c) to a near
`setting (FIG. 9d). The zoom level is also controlled from a
`large zoom setting (FIG. 9a) at the largest focusing distance,
`through progressively less magnified settings (FIGS. 9b and
`9c) to a least magnified setting (FIG. 9d) at the nearest
`focusing distance.
`
`[0064] FIG. 10 shows the final image 70 produced by
`compositing together dewarped segments of the plural
`images of FlGS. 9a-9d. The image 70 is made up of a first
`segment 72 taken from the first image (FlG. 9a), a second
`segment 74 taken from the second image (FIG. 9b), a third
`segment 76 taken from the third image (FIG. 9c), and a
`fourth segment 78 taken from the fourth image (FIG. 9d). As
`can be seen in FIG. 10, there is some degree of possible
`overlap between the image segments which have acceptable
`quality (sharp focus and high resolution). This indicates that
`the number of plural images is adequate to provide a high
`quality image of the entire object, and that there are no
`quality "gaps" in the composited image.
`
`[0065] The invention, particularly as described in the
`preferred embodiments, can enable sharp, high quality
`images to be acquired from documents even when the
`camera is held at an oblique angle. The invention can be
`used to correct for book curvature and the resulting out-of-
`
`focus and low-resolution areas resulting from a single image
`at a fixed focus. The invention can also be used to scan
`books from an oblique angle when the opening of the book
`is restricted, for example, for valuable or old books.
`
`[0066] Although the invention is especially suitable for
`document capture, the invention may be used for any digital
`camera for imaging three-dimensional objects of all kinds
`which might be difficult to bring in to focus in a single
`image.
`
`In the above embodiments, the camera comprises a
`[0067]
`variable focus mechanism 38 for varying the camera focus(cid:173)
`ing distance under the control of the control circuit 40. In an
`alternative embodiment, the focus may be swept manually
`by physical camera movement.
`
`[0068] The invention may also be used in combination
`with image mosaicing, so that larger areas may be imaged.
`In this way, images suffering from extreme distortion can be
`recovered with a sulliciently high resolution (without mosa(cid:173)
`icing, the maximum resolution is limited by the size of the
`document).
`
`[0069] Such a combination would include motion of the
`camera as well as focus (and focal length) sweeping. A
`suitable mosaicing technique is described in U.S. patent
`application Ser. No. 09/408,873 entitled "Mosaicing images
`with an offset lens", the contents of which is incorporated
`herein by reference.
`
`[0070] The focus and focal length sweeping may also be
`used in conjunction with a moving linear sensor (instead of
`a traditional area sensor). This allows an image to be
`acquired with variable resolution in one direction. The basis
`for such a technique is also described in the above-incor(cid:173)
`porated U.S. patent application Ser. No . 09/408,873.
`
`[0071] The method of the present invention may be com(cid:173)
`bined with a shift lens or any other image-shifting device so
`that the position of the image can be adjusted when the focal
`length is increased. Such a system is especially useful if
`unwanted motion is present, as in a portable camera, for
`example.
`
`In the preferred embodiments, the invention is
`[0072]
`implemented within a camera unit. This can provide an
`extremely powerful camera technique. However, in olher
`embodiments for less complicated or less expensive cam(cid:173)
`eras, it is possible to perform at least some of the image
`processing (for example, dewarping, registration, quality
`analysis, and composition) using an external image proces(cid:173)
`sor (for example, oftl.ine processing). In such alternative
`embodiments, it is preferred that data representing physical
`characteristics of the camera (such as orientation, focus
`setting, zoom setting, etc.) be recorded with each image to
`assist in later processing of the images.
`
`It will be appreciated that the foregoing description
`[0073]
`is merely illustrative of preferred embodiments of the inven(cid:173)
`tion, and that many modifications and equivalents will occur
`to one skilled in the art within the spirit and scope of the
`present invention.
`
`1. An apparatus for capturing an image comprising
`
`a) an input device for providing at least plural image
`segments of an object scene captured at different focus
`distances;
`
`0009
`
`

`

`US 2002/0075389 Al
`
`Jun.20,2002
`
`5
`
`b) a perspective correction device for determining at least
`one geometric transform to correct said image seg(cid:173)
`ments for perspective distortion; and
`
`c) an image compositor for compositing a perspective
`corrected image of said object from the plural image
`segments to which said at least one geometric trans(cid:173)
`form bas been applied.
`2. An apparatus according to claim 1, wherein the input
`device comprises an image capture device for capturing
`plural images of the object at said different focus distances,
`and wherein the image compositor composites the image
`from plural segments derived from the plural captured
`images.
`3. An apparatus according to claim 2, further comprising
`an image analyzer for analyzing the captured images for
`selection of a segment of the captured image for the com(cid:173)
`positor.
`4. An apparatus according to claim 4, wherein the image
`analyzer comprises a registration detector for identifying the
`registration of one image with respect lo another.
`5. An apparatus according to claim 3, wherein the image
`analyzer is operative to analyze the quality of at least a
`region of a captured image, for selection of a segment
`therefrom for the compositor according to the image quality
`of the segment.
`6. An apparatus according to claim 5, wherein the quality
`is determined by the sharpness of the image.
`7. An apparatus according to claim 5 wherein the quality
`is determined by the resolution of the image.
`8. An apparatus according to claim 2, further comprising
`a variable focus mechanism for varying the focus distance,
`and a controller for controlling the image capture device and
`the variable focus mechanism to capture said at least plural
`image segments of an object at different focus distances.
`9. An apparatus according to claim 2, further comprising
`a variable zoom mechanism for varying the zoom setting for
`the image capture device, and a controller for controlling the
`image capture device to capture said at least plural image
`segments of an object at different zoom settings.
`10. An apparatus according to claim 2, further comprising
`a variable focus mechanism for varying the focus distance,
`a variable zoom mechanism for varying the zoom selling for
`the image caplure device, and a controller operative lo
`conlrol lhe variable focus mechanism and !he variable zoom
`mechanism to vary the focus setting and tbe zoom setting in
`combination.
`11. An apparatus according to claim 2, wherein the
`apparatus is implemented in a camera.
`12. An apparatus comprising:
`a) an input device for providing at least plural image
`segments of an objecl scene captured at di!Ierenl focus
`distances;
`
`b) an image anal