1111111111111111 IIIIII IIIII 11111 1111111111 lllll lllll lllll lllll lllll 111111111111111 11111111
`US 20060146144Al
`
`c19) United States
`c12) Patent Application Publication
`Salmelin et al.
`
`c10) Pub. No.: US 2006/0146144 Al
`Jul. 6, 2006
`(43) Pub. Date:
`
`(54) DIGITAL IMAGING WITH AUTOFOCUS
`
`Publication Classification
`
`(75)
`
`Inventors: Eero Salmelin, Tampere (FI); Janne
`Haavisto, Kangasala (FI); Ossi Kalevo,
`Toijala (FI)
`
`(51)
`
`Int. Cl.
`H04N 51228
`(2006.01)
`(52) U.S. Cl. .......................................................... 348/222.1
`
`Correspondence Address:
`WARE FRESSOLA VAN DER SLUYS &
`ADOLPHSON,LLP
`BRADFORD GREEN, BUILDING 5
`755 MAIN STREET, P O BOX 224
`MONROE, CT 06468 (US)
`
`(73) Assignee: Nokia Corporation
`
`(21) Appl. No.:
`
`11/030,724
`
`(22) Filed:
`
`Jan. 5, 2005
`
`100~
`
`(57)
`
`ABSTRACT
`
`In digital imaging, in a case where the full resolution digital
`image needs to be scaled down before presenting it on a
`display, before the image is scaled down, statistical data is
`gathered from said the full resolution image data, whereby
`the gathered statistical data is attached to the image for
`further processing. The invention relates to a method, to a
`device, to an imaging module and to a computer program
`product.
`
`132
`
`DISPLAY
`
`T
`I
`
`DSP CTR I 130
`
`11
`
`31
`
`r CAM
`120 PROCESSOR
`
`1121 AF
`I
`;
`t
`t
`!
`IMG ~ I STATI
`SENSOR 116
`119
`LENSES ~ IL-SENSI
`
`MEMORY
`
`133
`
`MAIN UNIT
`
`118
`
`110
`CAM MODULE
`
`111
`
`115
`
`1
`
`Exhibit 1050
`Apple v. Qualcomm
`IPR2018-01277
`
`

`

`Patent Application Publication
`
`Jul. 6, 2006 Sheet 1 of 5
`
`US 2006/0146144 Al
`
`0
`M
`T"'""
`
`O::'.
`I-u
`
`0..
`Cf)
`0
`
`T"'""
`M
`T"'""
`
`>-<!
`
`__J
`0..
`Cf)
`....
`0 ~
`
`N
`M
`T"'""
`
`•I'
`
`.~
`
`,I,
`
`4-
`
`++
`
`-~
`
`LL..
`<!
`
`T"'""
`N
`T"'""
`
`O::'.
`0
`Cf)
`Cf)
`
`w u ~o
`<! O::'.
`u 0..
`\_~
`
`T"'""
`
`0
`0
`T"'""
`
`>-0::'.
`0
`~ w
`~
`
`t
`
`•
`C)
`■-LL
`
`:=)
`
`I-
`z
`z
`<!
`~
`
`Cf) z
`w
`Cf) co
`T"'""
`I
`__J T"'""
`
`1~E1
`'[:;]~
`
`c.o
`
`T"'""
`T"'""
`
`Cf) r-
`
`T"'""
`T"'""
`
`Cf)
`
`O::'.
`0
`Cf)
`l9 z
`~ w
`-
`w
`__J
`:=)
`0
`Cf)
`w
`0
`~ Cf)
`~ z
`w
`0
`'r'""" <!
`'r'""" u
`
`__J
`
`I.(')
`T"'""
`T"'""
`
`T"'""
`T"'""
`T"'""
`
`2
`
`

`

`Patent Application Publication
`
`Jul. 6, 2006 Sheet 2 of 5
`
`US 2006/0146144 Al
`
`Cf) -~
`
`O:'.'.
`0
`5
`I-w
`z
`
`0
`M
`'I""""
`
`,;;:t"
`M
`N
`
`I
`
`0:::
`
`I-u - a..
`
`CJ)
`0
`
`'I""""
`M
`'I'"""
`
`•
`
`'
`
`LL
`<(
`
`'I'"""
`N
`'I'"""
`
`'
`O:'.'.
`0
`Cf)
`Cf)
`
`w u ~o
`<( 0:::
`u a..
`'--2 'I""""
`
`>-<(
`_J a..
`CJ)
`0
`
`N
`M
`'I""""
`
`It-
`
`0
`0
`N
`
`■
`
`N
`C)
`
`■ -LL
`
`>-
`0:::
`0
`~
`w
`~
`
`M
`M
`'I'"""
`
`t
`
`0::: Cf)
`<( I-
`~ _J z
`::::) w
`_J ~
`_J w w .....J
`u w
`
`l+t 0
`-
`0
`::::)
`<(
`
`I-
`z
`:::::,
`z
`<(
`~
`
`LO
`M
`N
`
`<.O
`M
`N
`
`I+-
`
`+i
`
`Cf) z
`w
`Cf) CX)
`__'._J :=
`
`1~~1
`- c~~ <.O
`
`CJ) r-
`
`'I'"""
`'I'"""_
`
`'I""""
`'I""""
`
`Cf)
`
`O:'.'.
`0
`Cf)
`(.9 z
`~ w
`-
`w
`_J
`::::)
`0
`(/)
`0
`w
`~ Cf)
`~ z
`w
`0
`'I"""" <(
`.....J
`'I"""" u
`
`LO
`'I'"""
`'I'"""
`
`'I'"""
`'I'"""
`'I'"""
`
`3
`
`

`

`> ....
`....
`Cl's --- 0 ....
`
`.i;...
`.i;...
`Cl's
`.i;...
`
`0
`0
`N
`rJJ
`c
`
`Ul
`
`~
`
`0 ....
`.....
`rJJ =(cid:173)
`
`('D
`('D
`
`Cl's
`0
`0
`N
`
`2' -~Cl's
`.... 0 =
`""O = O" -....
`.... 0 =
`t "e -....
`
`~ .....
`
`(')
`
`('D = .....
`~ .....
`""O
`
`~ .....
`
`(')
`
`Fig. 3
`
`130
`
`BASEBAND
`
`338
`STEREO OUT
`STEREO IN
`
`120
`
`322
`
`PROCESSOR
`I MULTIMEDIA I
`CAM PROCESSOR
`
`SENSOR
`
`110
`
`MODULE
`CAM
`
`s
`
`~
`
`300 ~
`
`4
`
`

`

`Patent Application Publication
`
`Jul. 6, 2006 Sheet 4 of 5
`
`US 2006/0146144 Al
`
`• en
`■-
`LL
`
`~'
`
`z z
`<( 0 -
`I- I-
`z -(/)
`0 <( 0
`(/) I- ~ 0..
`z Uz 0::: (/)
`0 I- 0 ozw
`0 ( / ) - WO
`V) w _J <(
`w -
`I- -
`(/) <( u >w~
`<( I- w 0
`I
`(f)
`co (/) 0 ~ I- -
`
`0
`,--
`V
`
`0
`N .---
`
`clOSS3:)0cld Vcl 3 l/\J'v'J
`
`3 :)'v' .:lcl3l.NI Vcl 3 l/\JV:)
`
`0
`c.o
`v
`
`z
`<(
`z
`0
`0
`-
`I-
`u I-
`0
`-
`0
`<(
`0
`<(
`-
`(./)
`0
`I- _J
`w w
`w
`0:::
`(/) :::>
`w
`I- u
`(.9 _J
`(.9
`<( I~ >
`<( _J f------i <( <( :--t
`~ u
`z ~ I- <(
`~
`-
`(/) u
`-
`0
`(./)
`s
`u
`0
`N <(
`0
`V
`
`z
`s
`
`w
`0
`(.9
`w <(
`0::: ~
`w -
`I- LL
`zo
`w
`<( 0
`I- z
`<( w
`0 w
`I- I
`<(
`<( I- I-
`I.{) I- I- <(
`0
`(/) <( 0
`V
`
`,
`
`<(
`0:::
`
`.---
`0
`V
`
`L.O
`
`.---.---
`
`M
`0
`V
`
`V
`0
`V
`
`clOSN3S 38Vl/\JI
`
`5
`
`

`

`Patent Application Publication
`
`Jul. 6, 2006 Sheet 5 of 5
`
`US 2006/0146144 Al
`
`It)
`
`■
`
`C)
`■-LL
`
`0
`N
`LO
`
`6
`
`

`

`US 2006/0146144 Al
`
`Jul. 6, 2006
`
`1
`
`DIGITAL IMAGING WITH AUTOFOCUS
`
`FIELD OF THE INVENTION
`
`[0001] This invention relates generally to digital imaging,
`and particularly to autofocusing in digital imaging. The
`invention relates especially to those situations, wherein the
`digital image needs to be scaled down, for example, for
`preview on a viewfinder display during autofocusing.
`
`BACKGROUND OF THE INVENTION
`
`[0002]
`In the past years, digital imaging devices, such as
`digital cameras have taken remarkable role in imaging
`technology. Traditional cameras have got a successor that,
`however, is very different from them. Traditional cameras
`rely entirely on chemical and mechanical processes, and
`there is necessarily no need for electricity to operate with
`one. However, digital cameras have one or more built-in
`processors and these cameras record images in an entirely
`digital form. Because of their electronic nature, digital
`cameras (or digital camera modules) can be readily inte(cid:173)
`grated to other electronic devices, of which mobile telecom(cid:173)
`munication devices (mobile terminals) are nowadays the
`most common examples. Depending on the master device
`(i.e. the device the camera module is integrated with) the
`camera module can communicate with several other com(cid:173)
`ponents and systems of said device. E.g. in a camera phone,
`the camera module is typically operatively communicating
`with one or more processors, and in the case of a digital
`camera, the device can comprise some other type of dedi(cid:173)
`cated signal processing components.
`
`[0003] A digital camera has a series of lenses, a lens
`system that focuses light to create an image of a scene. But
`instead of focusing this light onto a piece of film, it focuses
`it onto a semiconductor device that records light electroni(cid:173)
`cally. This semiconductor device can be e.g. a CMOS
`(Complementary Metal Oxide Semiconductor) or CCD
`(Charge-Coupled Device) sensor. The sensor is mainly com(cid:173)
`posed of a collection of light-sensitive pixels, which convert
`light into electrical charge and this electronic charge is
`further converted into digital image data. Typically, digital
`image data is mainly processed outside the sensor compo(cid:173)
`nent itself, but nowadays it is also possible to integrate logic
`and memory into the CMOS sensors. U.S. Pat. No. 6,570,
`617 B2 discloses a single chip camera, where certain signal
`and control electronics have been integrated on the same
`substrate with the photosensitive element.
`
`[0004] Deviating significantly from the conventional film
`type cameras, the modern digital cameras usually have
`integrated color displays that provide a preview of the frame
`that the user is capturing. This display can be used as a
`digital viewfinder instead of a traditional optical viewfinder.
`The image that is seen on the display is typically taken
`directly from the image sensor and after scaling down from
`its original resolution displayed on the viewfinder display.
`This procedure ensures rapidly updating live display during
`such preview mode. The preview image is also used for
`image based autofocusing purposes in the manner described
`below.
`
`[0005] Typically, in the digital camera the image sensor
`outputs data in lower resolution QVGA (Quarter Video
`Graphic Array) for the display because of the aforemen(cid:173)
`tioned requirement to have rapidly updating preview display
`
`and because of the limited pixel resolution of the display.
`During preview mode image based autofocusing can per(cid:173)
`formed by analysing the low-resolution image and by adjust(cid:173)
`ing the lens system accordingly to improve the focusing. The
`user can check the image displayed on the viewfinder
`display to see if the autofocusing has been successfully
`applied. Also non-image based focusing can be applied here
`by using, for example, various distance measuring methods
`known in the art. This lowered image resolution intended for
`viewfinder preview is, however, not high enough to perform
`accurate image based focusing in the modern multimega(cid:173)
`pixel, e.g. 3 megapixel cameras. In other words, the scaled
`down image resolution is not high enough to provide basis
`for accurate image analysis to fine tune autofocusing There(cid:173)
`fore, before the image is captured, in addition to the afore(cid:173)
`mentioned prefocusing measures the camera needs to typi(cid:173)
`cally perform some final focusing steps based on higher
`resolution image data.
`
`[0006]
`In most digital camera devices, when the user
`pushes capture button half way down, the autofocusing is
`performed based on the scaled down viewfinder image data
`as described above. When the user decides to take the picture
`and pushes the capture button all the way down, the camera
`device performs the final focusing steps. During the final
`focusing the image sensor starts to output high-resolution
`image data in order to provide the best possible basis for the
`focusing algorithms. From these high-resolution images
`focus measure values are gathered/calculated according to
`selected image blocks (e.g. from the center).
`
`[0007] Depending on how much the results of the prefo(cid:173)
`cusing based on the scaled down images, and the results of
`the final focusing based on the high resolution images
`deviate from each other, the final focusing steps create a
`certain delay before the image can actually be recorded. In
`addition, during the final focusing low resolution images for
`the viewfinder may not be produced or updated properly. In
`some digital cameras, even when the capture button is
`pressed half way down and the autofocusing (prefocusing)
`function is enabled, regular updating of the viewfinder
`image may be neglected.
`
`[0008] The operation of the autofocusing and previewing
`functions of the prior art digital cameras may deviate in their
`details somewhat from the description given above, but it
`can be realized that the prior art systems have certain severe
`limitations especially in respect to providing exact focus
`data from megapixel images continuously and without
`unnecessary delays. It can be realized that in prior art
`systems exact focus data is not available all the time,
`because at certain times only scaled down low resolution
`image intended for the viewfinder might be made available.
`Also, in some cases when the high-resolution images are
`captured for final focusing, the viewfinder images might not
`be updated properly.
`
`[0009] User experiences in digital imaging are most
`affected by the delay that occurs within the capturing opera(cid:173)
`tion. In other words, the delay between pressing the capture
`button and the actual image capture moment should be
`minimized. In current digital cameras the focusing is initi(cid:173)
`ated by the user pushing the capturing button half way down.
`When the push button is pushed halfway, the digital camera
`usually tries to find the best focus and locks there. If the
`focus is not good enough, the button can be released and
`
`7
`
`

`

`US 2006/0146144 Al
`
`Jul. 6, 2006
`
`2
`
`pushed again to find a better focus. When the button is then
`pushed all the way down, the image is captured immediately
`if the focusing was already finished properly at the time
`when the button was pushed half way. If the focusing was
`not ready and the button is pushed all the way down, the
`camera needs to finalize the focusing at the latest at this
`point, and this consumes undesirably more time. This
`means, that after the user has pushed the button, he/she
`needs to wait a while for the camera to complete the
`focusing. If the user, during this short moment of time,
`moves the camera, the image will naturally become unfo(cid:173)
`cused. This delay affects also the general use of the camera.
`With a digital camera a series of images can quite easily be
`captured, which means pushing the capturing button con(cid:173)
`stantly, e.g., during a target operation, for capturing every
`phase of the operation. However with said moment, even
`though the button is pushed repeatedly, the "waiting time"
`typical for prior art devices prevents "every phase of the
`operation" to be captured.
`
`[0010]
`It can be seen that even though digital cameras
`provide new and interesting ways for imaging, there still is
`a need for improved focusing methods, which are more
`accurate and faster than the ones in current technology, and
`that can cope with the ever increasing sensor resolutions. In
`addition, such methods should also work in situations where
`the image needs to be scaled down for generating a higher
`picture frequency, for example, for preview purposes. The
`following description discloses a solution that addresses
`these needs.
`
`SUMMARY OF THE INVENTION
`
`[0011] The solution includes a method, a device, an imag(cid:173)
`ing module and computer program product according to
`which image focusing can be improved.
`
`[0012] The method according to one embodiment com(cid:173)
`prises steps for providing a full resolution image, providing
`a scaled down partial resolution image based on said full
`resolution image, calculating statistical data based on the
`full resolution image data and providing said statistical data
`together with the scaled down image data for further pro(cid:173)
`cessing steps.
`
`[0013] The device for digital imaging comprises an image
`sensor for providing a full resolution image, a processor for
`providing a scaled down partial resolution image based on
`said full resolution image, and calculating means for deter(cid:173)
`mining statistical data based on the full resolution image
`data and providing said statistical data together with the
`scaled down image data for further processing.
`
`[0014] The imaging module for digital imaging comprises
`an image sensor for providing a full resolution image, a
`processor for providing a scaled down partial resolution
`image based on said full resolution image, whereby said
`imaging module further comprises calculating means for
`determining statistical data based on the full resolution
`image data and providing said statistical data together with
`the scaled down image data for further processing.
`
`[0015] The computer program product in digital imaging
`comprises computer readable instructions being configured
`to provide a full resolution image, to provide a scaled down
`partial resolution image based on said full resolution image,
`wherein by means of said computer readable instructions
`
`statistical data is calculated based on the full resolution
`image data and said statistical data is provided together with
`the scaled down image data for further processing.
`
`[0016] This invention has considerable advantages when it
`comes to digital imaging and especially to the focusing
`operation. Because of the invention, exact autofocus is
`available all the time at the target frame rate. In addition the
`image can always be kept in focus. At the time the user
`pushes the capture button the focused image can be captured
`instantly as there is no need to delay. Not only is this method
`applicable with still images, but also in video imaging the
`autofocus is more accurate.
`
`DESCRIPTION OF THE DRAWINGS
`
`[0017] A better understanding of the invention may be
`obtained from the following examples taken in conjunction
`with the accompanying drawings.
`
`[0018] FIG.1 illustrates an example of the imaging device
`structure,
`
`[0019] FIG. 2 illustrates another example of the imaging
`device structure,
`
`[0020] FIG. 3 illustrates yet another example of the imag(cid:173)
`ing device structure,
`
`[0021] FIG. 4 illustrates an example of method steps and
`the division of them between two camera components, and
`
`[0022] FIG. 5 illustrates an example of an image with
`blocks for statistical data calculation.
`
`DETAILED DESCRIPTION OF THE
`INVENTION
`
`[0023] Although specific terms are used in the following
`description for the sake of clarity, these terms are intended
`to refer only to the particular structure of the invention
`selected for illustration in the drawings and they are not
`intended to induly define or limit the scope of the invention.
`In the description a general term "image" is used for defining
`such forms of an image data that are visual. Examples of
`images are still image, video image and a preview image that
`is used during capturing process. "Raw image" in this
`description relates to such an image data that is not pro(cid:173)
`cessed in any manner. Raw image is data that is captured by
`the image sensor but not interpreted. "Processed image" is
`a result from processed raw image data. The processed
`image can be interpreted visually, but not necessarily. An
`"imaging device" can be any device comprising means for
`digital imaging. Therefore, the imaging device can be a
`digital camera, imaging phone or some other device, that can
`either include imaging means or be connected to one ( e.g. a
`web camera).
`
`[0024] An imaging device that implements the method
`disclosed by this application is now described by referring to
`FIG. 1, where an imaging device structure is presented in
`very simplified manner. The drawing of the imaging device
`presents such components of a digital camera, to which this
`method significantly contributes.
`
`[0025] The imaging device 100 comprises a camera mod(cid:173)
`ule 110, a camera processor 120 that can be an audiovisual
`engine, and a main unit 130. The purpose of the main unit
`130 is to be in main charge of the various applications of the
`
`8
`
`

`

`US 2006/0146144 Al
`
`Jul. 6, 2006
`
`3
`
`device 100. The main unit 130 comprises, for example, a
`display controller 131 for controlling the display 132 of the
`imaging device 100. The main unit 130 is also arranged to
`store images and other information to a memory card 133 or
`other memory means. In addition, the main unit 130 can
`have communication means for forming a data transfer
`connection to external memory means such as a personal
`computer. These communication means can be a data cable
`or a wireless system. The main unit 130 may further have
`other capabilities or connections to other systems, compo(cid:173)
`nents or databases, which improve efficiency and function(cid:173)
`ality of the imaging device.
`
`In this example the camera processor 120 com(cid:173)
`[0026]
`prises at least means for determining autofocus 121. The
`camera processor 120 may ( depending on the situation) also
`be capable of processing the images according to known
`image processing methods. In this example, the camera
`processor 120 is illustrated as an independent component,
`but it should be appreciated that the camera processing
`functions can be carried out by a processor of the main unit
`130. Other configurations are possible as well. For example,
`statistical data needed for a statistical calculation may be
`gathered by the camera module 110, the statistical calcula(cid:173)
`tion done by the camera processor 120 and the autofocus
`control done in the main unit 130. By understanding these
`variations of the configuration, the other possible configu(cid:173)
`rations are understood as well.
`
`[0027] The camera module 110 comprises lenses 111 via
`which the light is produced to an image sensor 115. In this
`example, the image sensor 115 comprises an analog-to(cid:173)
`digital converter 116 and means to introduce a gain. Some
`image sensors can comprise only a gain, and some image
`sensors may not include either. In this example the image
`sensor comprises also a video image scaler 117 and a light
`sensitive area 118. The image sensor 115 can also be
`introduced with statistical means 119 for providing statisti(cid:173)
`cal data relating to said image. The statistical means 119 may
`be capable of both gathering and calculating autofocus
`statistics, but the statistical means 119 can also be arranged
`only for gathering statistical data, whereby the statistical
`calculation is carried out by other calculation means.
`
`[0028]
`In relation to some other image sensors, such as
`CCD-sensors, the sensor may comprise only the light(cid:173)
`sensitive area and charge transmitters. In addition to these,
`the sensor needs an interface to the electronic components
`described earlier. The interface is arranged to transmit
`analog charge values from the sensor and timing needed by
`the charge transmitters to the sensor.
`
`[0029] Another example of the imaging device is illus(cid:173)
`trated in FIG. 2, wherein an imaging phone 200 is presented.
`imaging phone 200 comprises
`the components
`The
`described with reference to FIG. 1 but also such components
`that are needed with wireless communication functions. For
`example, in this FIG. 2 the device 200 comprises audio
`means 236, various networking means 234 and basic cellular
`components 235 (e.g. keyboard, power management). One
`skilled in the art will appreciate that the device 200 can
`incorporate other functions and components with the ones
`mentioned here.
`
`[0030] Yet another example of the imaging device is
`presented in FIG. 3. The imaging phone 300 illustrated here
`is a bit more detailed drawing of the imaging phone 200
`
`(FIG. 2). The means for determining the autofocus 121 are
`in this figure arranged into multimedia processor 322. In this
`FIG. 3, the imaging phone comprises also a WLAN con(cid:173)
`nection 234, but it will be evident that also this additional
`short range network capability can be some other short range
`network, e.g. Bluetooth or Infrared. Also into this FIG. 3 a
`receiver-transmitter 337 is illustrated. As a main memory,
`the imaging phone 300 of this example comprises SDRAM
`memory 339. As a main unit 130 in this figure operates a
`phone in baseband. Further the imaging phone 300 is
`capable of stereo input and stereo output 338. It will be
`appreciated by the man skilled in the art that, instead of the
`main unit, also the camera processor may be in connection
`with components, such as a display, an audio, a memory card
`or an external memory.
`
`[0031] One example of the imaging procedure is illus(cid:173)
`trated in FIG. 4. The procedure starts by the user viewing an
`imaging target by the imaging device. Raw image data 401
`is acquired through the lenses to the image sensor 115. The
`image sensor 115 is running at high-resolution mode. In
`traditional cameras, when the viewfinding is done the image
`is usually a low resolution image, and for capturing still
`images and for focusing purposes the image sensor outputs
`high-resolution image. However, in this description the
`focusing can be done by using any selected area in the
`image, as will be evident from the following description.
`
`[0032] The invention aims to remove the prior art limita(cid:173)
`tions and to present a method, where scaled down image
`data is available for preview, but at the same time high
`quality statistical data from a high-resolution image is also
`available for image-based autofocusing. When such high
`quality statistical data is substantially continuously available
`for focusing, this speeds up the autofocusing process and
`minimizes the delays during the final image capture.
`
`[0033] The procedure before image-based autofocusing
`can be implemented according to the invention in various
`ways. The basic idea is that after full resolution image is
`captured 401 in the image sensor and analog-to-digital
`conversion 402 is done, a statistical data 403 concerning the
`image is gathered. The statistical data is calculated in the
`image sensor from several areas of the image, for example
`see FIG. 5 where, in the image 510, the statistical data is
`calculated from blocks 520. The number of blocks is not
`relevant, but it is more advantageous to have more than one
`block for the calculation. Usually the statistical gathering is
`carried out in a center part of the image, in several areas and
`in several different frequency bands. When there are more
`blocks to be used in the statistical calculation, it is possible
`e.g. to select a certain group, which represents the best
`focusing.
`
`[0034] When the statistical data is gathered/calculated
`403, the image is scaled down 404 to a desired partial
`resolution, which is normally used in preview, and which
`can be for instance QVGA-resolution. In this example the
`statistical data is attached 405 at the end of the image data.
`However in some situations it may be more advantageous to
`transfer the statistical data at another time, e.g. right after it
`has been calculated. It should also be noticed that not all the
`image data is needed at the same time, but the gathering and
`scaling can proceed accordingly as the image data is getting
`ready. Therefore, the statistical gathering may be done one
`line at the time by repeating steps 401-404 and once in a
`
`9
`
`

`

`US 2006/0146144 Al
`
`Jul. 6, 2006
`
`4
`
`while sending data further. The sending can also in this case
`be operated at the end of the image 405. It will be appre(cid:173)
`ciated by anyone skilled in the art, that the procedure may
`incorporate other image processing functions between steps
`401-410, which are not disclosed here. As an example,
`processes such as clamping, noise reduction are mentioned.
`
`[0035]
`It should be appreciated, that according to the
`current invention statistical data for auto focusing purposes is
`determined from the high-resolution image, and only after
`that is the image scaled down for other purposes, such as for
`a live viewfinder display. Therefore, statistical data to facili(cid:173)
`tate high accuracy autofocusing is available continuously as
`well as a scaled down image for preview. In prior art
`solutions this has not been possible, because the system
`needs to switch from preview mode to a specific autofocus(cid:173)
`ing mode. During the first mentioned mode a scaled down
`image for preview is available, but there is no high quality
`statistical data calculated from the high-resolution original
`image. In the later mode high-resolution autofocusing data
`is available, but the preview display cannot be updated
`regularly, because scaled down image data is missing.
`
`[0036] The step of statistical gathering, i.e. the actions
`required to analyze an image to produce statistical data
`describing the state of focus of said image, can comprise any
`steps or methods known in the art or any other steps suitable
`for such purpose. The invention should not be limited by the
`nature of those steps, but it is only important when and
`where those steps are performed.
`
`[0037] The image data with the statistical data is further
`delivered to camera processor 120 via camera interface 460.
`The camera processor 120 makes a decision for autofocus
`410, i.e. for moving the lenses or for maintaining the current
`position of the lenses. The camera processor 120 can decide,
`based on the statistical data gathered already at the sensor,
`which way the lenses should be moved without going
`through the entire image data. The camera processor 120
`may decide which block in the image is used for focusing.
`A decision for lens movement based only on analysis of the
`downscaled image would be inaccurate, and therefore the
`method according to the invention is preferable over the
`prior art methods.
`
`[0038] The focusing decision is based on statistical data.
`Depending on the characteristics of the image sensor and a
`last focused value, at least one target value can be defined to
`be used for decisions. The statistical data is compared to the
`target value, and lenses are moved according to how much
`the current statistical data deviated from said target value.
`
`[0039] When a statistical datum is less (worse) than the
`target value, the lenses are moved into that direction which
`is supposed to be the better direction. When the moving is
`done, the statistical datum is checked and if the result is good
`enough, the lenses are kept at that place. If the result is
`better, but not good enough, the moving is continued into
`that direction. If the result is worse the lenses are moved into
`opposite direction, where the statistical datum is checked
`and a decision of the next step is made. If the result
`improved sufficiently, the lenses are kept at that place or, if
`needed, the next place can be checked as well. If the result
`is worse also here, the lenses are brought back to the original
`place. The target value is redefined. In every step the target
`value is redefined when the lenses are moved to a new place
`or back to a previous place. If the target value becomes too
`
`bad, it may indicate that the focus is lost, whereby the
`scanning can be redone in the whole focus area. It should be
`also noticed that whether there are statistical data from many
`different frequencies, it is possible to examine from their
`changes when the focus is improved and into which direc(cid:173)
`tion the lenses should be moved. It enables also observations
`for when the focusing is needed. For example, the lower the
`frequency band best target value ends up, the more probably
`the lenses should be moved into some direction.
`
`[0040] The above describes a situation, where the whole
`focus area has been scanned through and the lenses are in
`good focus. The first scanning can be carried out by starting
`the scanning from a certain point and by scanning the whole
`image until the best new focusing is found. The best focus
`can be found by going through all the lens positions and
`determining at which position the focus value reaches maxi(cid:173)
`mum value or by quick scanning and then by focusing.
`
`[0041]
`In the previous examples the image sensor is
`capable of gathering statistical data, scaling the image and
`entering the statistical data to image. The camera processor
`is hence responsible for processing, autofocusing and func(cid:173)
`tions relating to it. However, it is possible to divide the tasks
`differently between the camera module and camera proces(cid:173)
`sor. However it should be kept in mind, that even if the tasks
`are shared differently, the basic idea is not dismissed, which
`is to gather the statistical data before scaling down the
`image. Therefore, when the basic idea is remembered, it will
`be appreciated that either the camera module or the camera
`processor or main unit may implement the processing. It is
`possible to deliver pre-gathered statistical data or smaller
`high-resolution blocks from the camera module, from which
`the statistical data can be calculated afterwards. It is also
`possible to deliver the gathered or calculated statistical data
`with the raw image data or the processed image data (and
`possibly even scaled image data) and it is possible as well to
`gather the statistical data from the raw image data or the
`processed image data. However, in the later situation the
`processing and scaling should be carried out by the camera
`module. The autofocus is decided by the camera processor
`or the main unit.
`
`[0042]
`In these examples the image sensor is a CMOS
`sensor, where logic and memory can be integrated on the
`same substrate as the photosensitive elements. However, the
`invention is not limited to the use of CMOS sensors. CCD
`sensors and also other image sensors may be used as well
`when they are arranged to incorporate suitable circuitry and
`logic for determining statistical data for autofocusing pur(cid:173)
`poses.
`
`[0043] Anyone skilled in the art will appreciate that the
`system may incorporate any number of capabilities and
`functionalities, which are suitable to enhance the efficiency
`of the focusing functions. Additionally the system may
`provide other manual or automatic services for maintaining
`data.
`
`[0044] Therefore, it will become evident that variations
`and modifications of the examples of embodiment described
`are possible without departing from the scope of protection
`of the invention as set forth in the claims.
`
`10
`
`

`

`US 2006/0146144 Al
`
`Jul. 6, 2006
`
`5
`
`We claim:
`1. A method in digital imaging, comprising at least the
`steps of
`
`providing a full resolution image,
`
`providing a scaled down partial resolution image based on
`said full resolution image, and
`
`calculating statistical data based on data of the full
`resolution image and providing said statistical data
`together with data of the scaled down image for further
`processing steps.
`2. A method according to claim 1, w