`
`
`
`
`
`
`
`US 8,896,697 B2
`(10) Patent No.:
`(12) United States Patent
`
`
`
`
`
`
`
`
`Nov. 25, 2014
`(45) Date of Patent:
`Golan et al.
`
`
`
`
`USOO8896697B2
`
`
`
`
`
`
`
`(54) VIDEO MOTION COMPENSATION AND
`
`
`STABILIZATION GIMBALED IMAGING
`
`SYSTEM
`
`
`
`
`
`
`Inventors: Chen Golan, Ein Vered (IL); Boris
`.
`.
`.
`
`
`
`Klpnls, Tel-Av1v (IL)
`
`
`
`
`
`
`
`Subject to any disclaimer, the term of this
`
`
`
`
`patent is extended or adjusted under 35
`
`
`
`
`U.S.C. 154(b) by 586 days.
`
`13/259,250
`
`
`APE 6, 2010
`PCT/IL2010/000280
`
`
`
`
`
`
`
`
`(76)
`
`
`
`
`( * ) Notice:
`
`
`
`
`
`
`(21) Appl. No.:
`
`
`
`(22) PCT Filed:
`
`
`
`(86) PCT No.:
`
`
`§ 371 (00):
`
`
`
`(2), (4) Date:
`
`
`
`Sep. 23, 2011
`
`
`
`
`
`
`
`(200601)
`(200601)
`(200601)
`
`
`
`
`
`
`
`
`
`(87) PCT Pub. No.: W02010/116366
`
`
`
`
`
`
`PCT P1111 Date: OCt- 14: 2010
`
`
`
`Prior Publication Data
`
`
`
`
`
`US 2012/0019660 A1
`Jan. 26, 2012
`
`
`
`
`Related US. Application Data
`
`
`
`
`
`
`(60) Provisional application No. 61/167 226 filed on Apr.
`
`
`7 2009
`’
`’
`.
`’
`
`
`Int. Cl-
`
`
`H04N 7/18
`
`
`H04N 5032
`
`
`H04N 50‘”
`
`
`
`(52) US Cl-
`
`
`
`
`
`
`
`CPC ~~~~~~~ H04N 5/23258 (201301); H04N 5/23248
`
`
`
`
`
`(2013.01); H04N 5/232 (2013.01); H04N
`
`
`
`
`
`5/23264 (2013.01); H04N5/2628 (2013.01)
`
`
`
`USPC .......................................................... 348/144
`
`(65)
`
`
`
`
`
`(51)
`
`
`
`
`
`(58) Field of Classification Search
`
`
`
`
`
`
`
`CPC .......... H04N 5/23248; H04N 5/23258; H04N
`
`
`
`
`
`
`5/232; H04N 5/23264; H04N 5/2628; G02B
`
`
`
`27/646; GOIC 11/025
`
`
`
`
`USPC ............................................... 348/144, 208.4
`.
`.
`.
`
`
`
`
`
`
`
`See application file for complete search history.
`
`(56)
`
`
`
`
`References Cited
`
`
`
`
`
`US. PATENT DOCUMENTS
`
`
`
`
`
`
`
`2008/0151064 A1 *
`6/2008 Saito et al.
`................. 348/208.4
`
`
`
`* cited by examiner
`
`
`Primary Examiner 7 Allen Wong
`
`
`
`
`
`
`
`
`
`ABSTRACT
`(57)
`.
`.
`.
`.
`
`
`
`
`
`
`
`
`A system and method for compensating for image d1stortlons
`
`
`
`
`
`
`
`formed by the motion of a computerized camera system
`
`
`
`
`
`
`
`mounted on a moving platform. The camera system includes
`
`
`
`
`
`
`
`
`a camera, wherein the camera acquires a plurality of image
`
`
`
`
`
`
`
`frames including images of the environment viewed from
`within the field of view of the camera. The distortion is
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`formed in the acquired image frame, during and in between
`
`
`
`
`
`
`
`
`image acquisitions. During the image acquisition the camera
`
`
`
`
`
`
`
`may be maneuvered in space, typically, in the pan and tilt axis.
`
`
`
`
`
`
`
`
`The method includes the steps of providing camera maneu-
`
`
`
`
`
`
`
`
`vering signals, providing sensors for detecting other motions
`
`
`
`
`
`
`
`
`of the camera, computing the pre acquisition aggregated
`
`
`
`
`
`
`
`
`motion vector of the camera, thereby determining the pre
`
`
`
`
`
`
`
`
`acquisition image distortion, and compensating for the deter-
`
`
`
`
`
`
`g
`(1
`P
`y
`(1
`mined re ac uisitionima e distortionb ane uivalent vec-
`
`
`
`
`
`
`
`tor, in a direction opposite to the direction of the pre acquisi-
`
`
`
`
`tion aggregated motion vector.
`
`
`
`
`46 Claims, 5 Drawing Sheets
`
`
`
`100
`
`160 v// 190
`
`
`
`
`
`Post-Capture
`(Optional)
`Gym-stabilized
`
`
`
`
`Digital Zoom &
`Digital Image
`Captured Image
`
`
`
`
`
`
`stabilization
`
`
`Cropping
`
`
`
`
`
`
`Continuous, real-time
`
`
`pan/tilt pusitiun correction
`
`
`
`
`
`Sensnr Pan/Tilt
`Znnm Cunlml
`Sensor control
`
`
`
`
`
`IRIG IN Ji—
`
`
`
`
`pan/tiltlwmilnw size
`
`
`
`
`
`:nnlininus
`
`
`rEal-time pan/till
`/,,,,,,,,,,,,,,,,,,,
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`150
`
`
`
`
`
`
`
`Pan/Till Cnmrnl
`Coordinate offset
`
`
`
`
`calculation unit
`
`
`
`
`
`
`
`
`Displacement
`sensor
`
`
`
`
`1
`ggggg
`1 ______________________________________i
`
`APPL—1022/ Page 1 of 13
`Apple v. Corephotonics
`
`APPL-1022 / Page 1 of 13
`Apple v. Corephotonics
`
`

`

`
`U.S. Patent
`
`
`
`2w.
`
`41
`
`teeh
`
`5
`
`
`
`69
`
`
`2B79
`
`
`
`i,mE_.._mm._“3352.5
`
`
`
`m:EEEEUSEES.=E=ua
`
`Nco=fi=E£mmEnaEu
`
`
`
`=532>”mac:_E_m_n_=533>wEoow_E_m_n_mane..5538
`
`
`
`.$2250.32.5%..th2:35-33
`
`09
`
`oar09
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`SQ|_._IEOE...an?gnuEgbztdna
`Ema—N09soon:_Hm=_._..___._an:_um_._um0.—C00LOW—LO_n:..—=OU
`
`
`
`
`
`
`
`6,BE
`
`
`
`
`
`8:......................................_39.38m||||IIHmm,s:1was:anr\..Ee§§§U25__INN.mofm_I...m
`
`_n:c::o=m_:o_mo
`
`
`
`
`
`12.25.53
`
`
`
`cm\\zEznamE=._nEONr
`
`
`
`
`
`_ummto825.500.225=E=£N:\n
`
`
`
`om
`
`
`3]
`
`APPL-1022/ Page 2 of 13
`
`APPL-1022 / Page 2 of 13
`
`
`

`

`
`US. Patent
`
`
`
`
`
`Nov. 25, 2014
`
`
`
`
`Sheet 2 of5
`
`
`
`US 8,896,697 B2
`
`
`
`
`
`Fig2
`
`APPL-1022/ Page 3 of 13
`
`APPL-1022 / Page 3 of 13
`
`

`

`
`U.S. Patent
`
`
`
`
`
`NOV. 25, 2014
`
`
`
`
`Sheet 3 of 5
`
`
`
`US 8,896,697 B2
`
`02‘
`
`EzcuuBEE
`
`mat
`
`use0.5%}“awn—*0umEOON>
`
`
`
`
`
`EaEmE33>
`
`
`
` MW54%
`
`we
`
`warmm?
`
`o:
`
`
`
`
`
`
`
`APPL-1022/ Page 4 of 13
`
`APPL-1022 / Page 4 of 13
`
`
`
`
`

`

`
`US. Patent
`
`
`
`
`
`Nov. 25, 2014
`
`
`
`
`Sheet 4 of5
`
`
`
`US 8,896,697 B2
`
`
`
`400
` "Jideunut
`Cnntiniuus
`
`
`
` ZuumContrul
`
`
` EinfSkip
`
`Fig5
`
`APPL-1022/ Page 5 of 13
`
`APPL-1022 / Page 5 of 13
`
`

`

`
`U.S. Patent
`
`
`
`
`
`NOV. 25, 2014
`
`
`
`
`Sheet 5 of 5
`
`
`
`US 8,896,697 B2
`
`CNN
`
`
`
`VJ
`
`
`
`
`
`
`
`
`
`awash1%mewEEpofiom
`
`moumwnfifim
`
`
`
`
`
`
`
`
`
`
`93:825M:38wsufisofimo7
`
`0FN
`
`omr
`
`
`
`
`
`motion33328wfihfiom
`
`53%09238HOE33we
`
`
`
`>
`
`
`
`83%owmz:mEE>uo<
`
`
`
`CNN
`
`omN
`
`9mm
`
`
`
`wEEOONEREwEEHotom“
`
`:5:2twcuosbmsooom
`
`.
`
`
`
`APPL-1022/ Page 6 of 13
`
`28$owwzzBo:mchsfiuo<
`
`
`
`>0mm
`
`
`
`808mmowns:mumbflmmfih
`
`
`
`ommfl:wscsawoémom
`
`
`
`in:wfimmoogm
`
`0mm
`
`
`
`
`
`
`
`
`
`
`APPL-1022 / Page 6 of 13
`
`
`
`
`

`

`
`
`US 8,896,697 B2
`
`
`1
`VIDEO MOTION COMPENSATION AND
`
`
`
`STABILIZATION GIMBALED IMAGING
`
`
`SYSTEM
`
`
`
`
`
`RELATED APPLICATION
`
`
`
`
`
`
`
`
`
`
`
`The present application claims the benefit of US. provi-
`
`
`
`
`
`
`
`sional application 61/167,226 filed on Apr. 7, 2009, the dis-
`
`
`
`
`
`closure of which is incorporated herein by reference.
`FIELD OF THE INVENTION
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`The present invention relates to imaging systems, and more
`
`
`
`
`
`
`
`particularly, the present invention relates to an imaging sys-
`
`
`
`
`
`
`
`tem, operatively mounted on an air-born vehicle, that can
`
`
`
`
`
`
`
`transmit high resolution images of a selected region of inter-
`
`
`
`
`
`
`
`
`est, whereas the images are continuously compensated for
`vehicle motion.
`
`
`
`BACKGROUND OF THE INVENTION AND
`
`
`
`PRIOR ART
`
`
`
`
`
`
`
`
`
`
`
`
`
`An image sensor is generally subject to motion and vibra-
`
`
`
`
`
`
`
`
`tions which might distort a detected image of a scene. The
`
`
`
`
`
`
`
`
`motion can be linear, where the image sensor undergoes a
`
`
`
`
`
`
`
`linear displacement or scaling, and the motion can be angular,
`
`
`
`
`
`
`
`
`
`where the image sensor rotates about one or more axes. In
`
`
`
`
`
`
`
`case of an image sensor mounted on a marine vessel, the
`
`
`
`
`
`
`
`image can be distorted as a result of ocean waves. Likewise,
`
`
`
`
`
`
`
`image distortion can occur in images detected by an image
`
`
`
`
`
`
`sensor mounted to a ground vehicle, an airborne platform,
`
`
`
`
`such as an aircraft, a helicopter or a satellite.
`
`
`
`
`
`
`
`Methods for compensating for the vibrations and noise in
`
`
`
`
`
`
`
`
`
`order to obtain a stabilized image are known in the art. For
`
`
`
`
`
`
`
`example, a gyroscope connected to the image sensor detects
`
`
`
`
`
`
`
`
`the inertial rotations of the image sensor, and a servo system
`
`
`
`
`
`
`
`
`(including a servo motor and a controller) rotates the gimbals
`
`
`
`
`
`
`
`
`
`on which the image sensor is mounted, in the opposite direc-
`
`
`
`
`
`
`
`
`tion and by the same amount, according to the output of the
`
`
`
`
`
`
`
`gyroscope. The image can be further refined by employing
`
`
`
`
`
`
`
`additional gyroscopes and by providing each gyroscope addi-
`
`
`
`tional degrees of freedom.
`
`
`
`
`
`
`
`
`Prior art imaging systems are typically large in size and
`
`
`
`
`
`
`thereby in relative weight. Furthermore, prior art imaging
`
`
`
`
`
`
`systems require extensive image processing on the whole
`
`
`
`
`
`
`
`image frame acquired, particularly for high resolution imag-
`
`
`ing systems.
`
`
`
`
`
`
`
`There is a need for and it would be advantageous to have
`
`
`
`
`
`
`image sensors, mounted on an airborne vehicle, such as
`
`
`
`
`
`
`
`
`unmanned aerial vehicle (UAV), having high resolution and
`
`
`
`
`
`
`
`capability to select in real the region-of—interest (ROI), low
`
`
`
`
`
`
`
`cost, low weight and low power consumption.
`SUMMARY OF THE INVENTION
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`The present invention describes a motion-compensation
`
`
`
`
`
`
`
`and stabilization gimbaled camera system for performing
`
`
`
`
`
`
`
`
`image acquisition and image transmission. The present inven-
`tion is often described herein in terms of an air-bom camera
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`system, but the present invention is not limited to an air-born
`
`
`
`
`
`
`
`motion compensation and stabilization gimbaled camera sys-
`
`
`
`
`
`
`
`
`
`tem, and the system can be used in any video acquisition
`
`
`
`
`
`
`system, such as on hand held cameras, land-vehicle mounted,
`
`
`
`
`
`
`traffic control systems, waterways-vehicle mounted, etc.
`
`
`
`
`
`
`According to teachings of the present invention, there is
`
`
`
`
`
`
`
`provided a camera system including camera motion compen-
`
`
`
`
`
`
`
`
`sation and stabilization units, using a high resolution image
`
`10
`
`
`
`15
`
`
`
`20
`
`25
`
`
`
`30
`
`
`
`35
`
`
`
`40
`
`
`
`45
`
`
`
`50
`
`
`
`55
`
`
`
`60
`
`
`
`65
`
`
`
`
`
`
`
`
`
`
`sensor, such as a multi-megapixel CMOS (“CMOS image
`
`
`
`
`
`
`
`
`
`sensor”) or a camera module with digital pan, tilt and option-
`
`
`
`
`
`
`
`ally zoom capability, mounted on a moving platform and
`
`
`
`
`
`
`
`having a simple mechanical gimbals support. The camera
`
`
`
`
`
`
`
`system facilitates, each time before an image is captured, to
`
`
`
`
`
`
`
`compensate for unwanted image motion or jitter caused by
`
`
`
`
`
`
`
`the camera platform motion, by pointing to relevant image
`
`
`
`
`
`
`array region corresponding to a selected geographical region
`
`
`
`
`
`
`
`
`
`
`of interest, just before capturing the image. The correct win-
`
`
`
`
`
`
`
`
`dow offset is calculated using platform angular motion sen-
`
`
`
`
`sors, such as gyro or rate-gyro.
`
`
`
`
`
`
`According to further teachings of the present invention,
`
`
`
`
`
`
`
`
`there is provided a method for compensating for image dis-
`
`
`
`
`
`
`
`tortions formed by the motion of a computerized camera
`
`
`
`
`
`
`
`system mounted on a moving platform. The camera system
`
`
`
`
`
`
`
`
`includes a camera having one or more image sensor arrays,
`
`
`
`
`
`
`
`wherein the camera acquires consecutively, in real time, a
`
`
`
`
`
`
`
`
`plurality of image frames including images of the environ-
`ment viewed from within the field of view of the camera. The
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`distortion is formed in the acquired image frame, during and
`
`
`
`
`
`
`
`
`in between image acquisitions. During the image acquisition
`
`
`
`
`
`
`
`
`the camera may be maneuvered in space, typically, in the pan
`
`
`
`
`
`
`
`
`and tilt axis. The platform can be an air born vehicle, a land
`
`
`
`
`
`
`
`vehicle, a waterway vehicle, a living body, carried by hand or
`
`
`
`
`
`
`any other moving and/or vibrating platform.
`
`
`
`
`
`
`
`The method for compensating for image distortions in the
`
`
`
`
`
`
`
`
`acquired image frames includes the steps ofproviding camera
`
`
`
`
`
`
`
`maneuvering signals, providing one or more sensors for
`
`
`
`
`
`
`
`
`detecting the motion of the camera, computing the pre acqui-
`
`
`
`
`
`
`
`sition aggregated motion vector of the camera, thereby deter-
`
`
`
`
`
`
`
`
`mining the pre acquisition image distortion caused by the pre
`
`
`
`
`
`
`acquisition aggregated motion vector ofthe camera, compen-
`
`
`
`
`
`
`
`
`sating for the determined pre acquisition image distortion by
`
`
`
`
`
`
`
`
`a magnitude equal to the magnitude of the pre acquisition
`
`
`
`
`
`
`
`aggregated motion vector, in a direction opposite to the direc-
`
`
`
`
`
`
`
`
`
`tion of the pre acquisition aggregated motion vector, and
`
`
`
`acquiring an image frame.
`
`
`
`
`
`
`The camera maneuvering signals are maneuvering com-
`
`
`
`
`
`
`mands as provided by the steering control of the camera.
`
`
`
`
`
`
`
`The one or more sensors are typically displacement sensors
`
`
`
`
`
`
`
`for sensing changes in spatial position such as angular rate
`
`
`
`
`
`
`sensors, gyroscope sensors, rate gyroscope sensors or smart
`
`
`
`
`inertial navigation system units.
`
`
`
`
`
`
`
`
`Preferably, the method further includes the steps ofprovid-
`
`
`
`
`
`
`
`ing an environmental region of interest within the environ-
`
`
`
`
`
`
`
`
`
`ment viewed from within the field of view of the camera, and
`
`
`
`
`
`
`
`determining the array of pixels being a portion of the one or
`
`
`
`
`
`
`
`
`more image sensor arrays acquiring the image ofthe environ-
`
`
`
`
`
`
`
`
`
`mental region of interest and thereby obtaining an image
`
`
`
`
`
`
`
`
`region of interest. The compensation for the determined pre
`
`
`
`
`
`
`acquisition image distortion is performed on the image region
`of interest.
`
`
`
`
`
`
`
`
`
`Optionally, the method for compensating for image distor-
`
`
`
`
`
`
`
`
`
`tions in the acquired image frames includes steps for further
`
`
`
`
`
`
`
`compensating for distortions not attended by the pre acquisi-
`
`
`
`
`
`
`
`
`tion compensation steps. The method further includes post
`
`
`
`
`
`
`
`acquisition compensation steps of determining the post
`
`
`
`
`
`
`acquisition image distortion caused by the detected motion of
`
`
`
`
`
`
`
`
`the camera from the instant of issuing of a command for
`
`
`
`
`
`
`
`
`acquiring an image frame until the actual acquisition of the
`
`
`
`
`
`
`
`
`image frame, and compensating for the determined post
`
`
`
`
`
`
`
`acquisition image distortion, wherein the compensation for
`
`
`
`
`
`
`
`the determined post acquisition image distortion is applied to
`
`
`
`
`
`
`
`
`the image region of interest, whereby creating a final image
`
`
`
`
`
`
`
`
`region of interest. The compensation for the determined post
`
`
`
`
`
`acquisition image distortion is performed by a magnitude
`
`
`
`
`
`
`
`equal to the magnitude of the post acquisition aggregated
`
`2
`
`
`
`
`
`
`
`APPL-1022/ Page 7 of 13
`
`APPL-1022 / Page 7 of 13
`
`

`

`
`3
`
`
`
`
`
`
`
`motion vector and in a direction opposite to the direction of
`
`
`
`
`
`
`the post acquisition aggregated motion vector.
`
`
`
`
`
`
`
`
`Preferably, the method further includes the steps of pad-
`
`
`
`
`
`
`
`
`ding the image region of interest with a predefined margin,
`
`
`
`
`
`
`
`
`before determining the post acquisition image distortion, and
`
`
`
`
`
`
`
`
`cropping the image region of interest to remove the margin,
`
`
`
`
`
`
`
`
`after compensating for the determined post acquisition image
`
`
`
`
`
`
`
`
`
`distortion, and before forming the final image region of inter-
`est.
`
`
`
`
`
`
`
`
`Optionally, when using a camera having a rolling shutter,
`
`
`
`
`
`
`
`
`
`the method further includes the steps of determining the roll-
`
`
`
`
`
`
`
`ing shutter image distortion, typically a wobble distortion,
`
`
`
`
`
`
`
`
`and compensating for the determined rolling shutter image
`
`
`
`
`
`
`
`
`
`
`distortion in an opposite direction to the direction of the
`
`
`
`
`
`
`
`
`
`rolling shutter image distortion for each line or pixel in the
`
`
`
`
`
`
`
`image region of interest. It should be noted that determining
`
`
`
`
`
`
`
`
`the rolling shutter image distortion and the compensation for
`
`
`
`
`
`
`
`
`the determined rolling shutter image distortion are performed
`
`
`
`
`
`
`in either line, pixel or sub-pixel resolution.
`
`
`
`
`
`
`In embodiments ofthe present invention, the compensation
`
`
`
`
`
`
`
`
`
`for the determined rolling shutter image distortion are per-
`
`
`
`
`
`
`
`
`formed in the X-axis by line shifts to the opposite direction of
`
`
`
`
`
`
`
`
`
`the rolling shutter motion during the image acquisition scan.
`
`
`
`
`
`
`In embodiments ofthe present invention, the compensation
`
`
`
`
`
`
`
`
`for the determined rolling shutter image distortion is per-
`
`
`
`
`
`
`
`
`formed in the Y-axis by calculating and changing the line to
`line distances.
`
`
`
`
`
`
`
`
`
`
`Optionally, the method further includes the steps of pro-
`
`
`
`
`
`
`viding a zooming mechanism, providing a zoom request
`
`
`
`
`
`
`
`
`including zoom parameters, and computing the final image
`
`
`
`
`
`
`
`
`region with the provided parameters ofthe zoom request. The
`
`
`
`
`
`
`zooming mechanism can be an optical zoom, an electronic
`
`
`
`
`
`
`
`zoom or a combination of optical zoom and electronic zoom.
`
`
`
`
`
`
`
`
`It should be noted that the resolution ofthe acquired image
`
`
`
`
`
`
`
`
`frame may be larger than the resolution ofthe image region of
`
`
`
`
`
`
`
`
`interest and the final image region of interest. It should be
`
`
`
`
`
`
`
`
`
`noted that the original resolution ofthe acquired image frame
`
`
`
`
`
`
`may be extended using digital zooming methods.
`
`
`
`
`
`
`
`Optionally, the method for compensating for image distor-
`
`
`
`
`
`
`
`
`
`tions in the acquired image frames includes steps for further
`
`
`
`
`
`
`
`
`compensating for distortions not attended by the pre acquisi-
`
`
`
`
`
`
`
`
`tion compensation steps and the post acquisition compensa-
`
`
`
`
`
`
`
`
`
`tion steps. The method further includes the steps ofproviding
`
`
`
`
`
`
`
`
`a digital image stabilization unit, determining residual image
`
`
`
`
`
`
`
`
`distortions, and compensating for the residual image distor-
`
`
`
`
`
`
`
`tions. The step of determining of residual image distortions
`
`
`
`
`
`
`includes computing the correlation between a previously
`
`
`
`
`
`
`
`
`
`computed final image region of interest and the currently
`
`
`
`
`
`computed final image region of interest.
`50
`
`
`
`
`
`
`Preferably, after completion of the post acquisition com-
`
`
`
`
`
`
`
`
`pensation steps the final image region of interest is transmit-
`
`
`
`
`
`
`
`ted to a predetermined video receiving unit,
`typically a
`
`
`
`
`remote video receiving unit.
`
`
`
`
`
`
`
`The camera system my further include a motorized
`
`
`
`
`
`
`
`
`mechanical gimbal that extends the camera dynamic range
`
`
`
`
`
`
`
`
`with an additional degree of freedom. The motorized
`
`
`
`
`
`
`mechanical gimbal can be operated by a variety of motors,
`
`
`
`
`
`
`
`including a step motor, a DC motor, a brushless motor, etc.,
`
`
`
`
`
`
`
`and is preferable operated by a DC motor with pulse width
`
`
`
`
`
`
`modulation, to control motor force and speed.
`
`
`
`
`
`
`In variations of the present invention, in a computerized
`
`
`
`
`
`
`
`
`
`gimbaled camera system, the method further includes the step
`
`
`
`
`
`
`
`of activating the motorized mechanical gimbal to maintain
`
`
`
`
`
`
`
`
`the central pixel of the image region of interest, representing
`
`
`
`
`
`
`the center of the environmental region of interest, within a
`
`
`
`
`
`
`
`
`distance less than a predefined threshold value from the center
`
`
`
`
`of the image sensor array.
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`55
`
`60
`
`65
`
`
`
`US 8,896,697 B2
`
`
`4
`
`
`
`
`
`
`In variations of the present invention, in a computerized
`
`
`
`
`
`
`
`
`gimbaled camera system, the method further includes the
`
`
`
`
`
`
`
`
`steps of computing the distance of each edge of the image
`
`
`
`
`
`
`
`
`region ofinterest from the respective edge ofthe image sensor
`
`
`
`
`
`
`
`array, and activating the motorized mechanical gimbal to
`
`
`
`
`
`
`
`maintain each of the edges of the image region of interest at a
`
`
`
`
`
`
`
`
`distance less than a predefined threshold value from the
`
`
`
`
`
`
`
`
`respective edge of the image sensor array. Optionally, the
`
`
`
`
`
`computation of the distance of each of the edges of the image
`
`
`
`
`
`
`
`
`
`region of interest, from the respective edge of the image
`
`
`
`
`
`
`
`
`sensor array, uses a hysteresis function. The hysteresis values
`
`
`
`
`
`
`of the hysteresis function may be calculated as a function of
`
`
`
`
`
`zoom and motion changes prediction.
`
`
`
`
`
`
`An aspect of the present invention is to provide a comput-
`
`
`
`
`
`
`
`erized camera system mounted on a moving platform, option-
`
`
`
`
`
`
`
`
`ally having a steering control, for compensating for image
`
`
`
`
`
`
`
`
`distortions in the acquired image frames, wherein the distor-
`
`
`
`
`
`
`
`tions are caused by movements and/or vibrations of the cam-
`era.
`
`
`
`
`
`
`
`
`The computerized camera system includes a camera hav-
`
`
`
`
`
`
`
`
`
`ing one or more image sensor arrays, wherein the camera
`
`
`
`
`
`
`
`
`acquires consecutively,
`in real time, a plurality of image
`
`
`
`
`
`
`
`frames including images of the environment viewed from
`
`
`
`
`
`
`
`
`
`
`within a field of view of the camera, the camera system
`
`
`
`
`
`
`
`including a coordinate offset calculation unit, a camera steer-
`
`
`
`
`
`
`
`ing control, a displacement sensor, an image sensor configu-
`
`
`
`
`
`
`
`ration control unit, and a video timing unit.
`
`
`
`
`
`
`
`
`
`The video timing unit determines the frame acquisition rate
`
`
`
`
`
`
`
`
`
`
`of the camera and wherein the video timing unit begins a
`
`
`
`
`
`
`
`frame acquisition cycle having a pre acquisition portion and a
`
`
`
`
`
`
`
`
`post acquisition portion. The camera steering control pro-
`
`
`
`
`
`
`
`
`
`vides tilt and/or pan motional data of the camera. The dis-
`
`
`
`
`
`
`
`
`placement sensor senses the camera motion in space. The
`
`
`
`
`
`
`
`coordinate offset calculation unit continuously aggregates the
`
`
`
`
`
`
`
`sensed motions of the camera and thereby determining a pre
`
`
`
`
`
`
`
`
`acquisition aggregated motion vector. The image sensor con-
`
`
`
`
`
`
`
`
`figuration control unit determines the pre acquisition image
`
`
`
`
`
`
`
`distortion caused by the pre acquisition aggregated motion
`
`
`
`
`
`
`
`
`vector. The image sensor configuration control unit compen-
`
`
`
`
`
`
`
`sates for the determined pre acquisition image distortion by a
`
`
`
`
`
`
`
`magnitude equal to the magnitude of the pre acquisition
`
`
`
`
`
`
`
`aggregated motion vector, in a direction opposite to the direc-
`
`
`
`
`
`
`tion of the pre acquisition aggregated motion vector.
`
`
`
`
`
`
`
`In preferred embodiments of the present invention, the
`
`
`
`
`
`
`
`camera system further includes a computation unit and a
`
`
`
`
`
`
`
`post-capturing image processing unit. The coordinate offset
`
`
`
`
`
`
`
`
`calculation unit and the image sensor configuration control
`
`
`
`
`
`
`
`
`
`unit provide the computation unit with timing on motion data.
`
`
`
`
`
`
`
`The computation unit continuously aggregates the sensed
`
`
`
`
`
`
`
`motions of the camera from the instant of issuing of a com-
`
`
`
`
`
`
`
`
`mand for acquiring an image frame until the actual acquisi-
`
`
`
`
`
`
`
`
`
`tion ofthe image frame and thereby determining a post acqui-
`
`
`
`
`
`
`
`sition aggregated motion vector. The post-capturing image
`
`
`
`
`
`
`
`
`processing unit determines the post acquisition image distor-
`
`
`
`
`
`
`
`tion caused by the post acquisition aggregated motion vector.
`
`
`
`
`
`
`
`The post-capturing image processing unit compensates for
`
`
`
`
`
`
`
`the determined post acquisition image distortion by a magni-
`
`
`
`
`
`
`
`
`tude equal to the magnitude ofthe post acquisition aggregated
`
`
`
`
`
`
`motion vector, in a direction opposite to the direction of the
`
`
`
`
`
`post acquisition aggregated motion vector.
`
`
`
`
`
`
`
`In variations of the present invention, the camera systems
`
`
`
`
`
`
`
`further includes a mechanism for adjusting the zoom of the
`camera,
`
`
`
`
`
`
`
`In variations of the present invention, the camera systems
`
`
`
`
`
`
`further includes a motorized gimbaled device, wherein the
`
`
`
`
`
`
`motorized gimbaled device extends the camera dynamic
`
`
`
`
`
`range by providing an additional degree of freedom; and
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`APPL-1022/ Page 8 of 13
`
`APPL-1022 / Page 8 of 13
`
`

`

`
`5
`
`
`
`
`
`
`
`wherein the motorized gimbaled device facilitates maintain-
`
`
`
`
`
`
`
`
`
`ing an environmental region of interest within the field of
`view of the camera.
`
`
`
`
`
`
`
`
`
`In variations of the present invention, the camera systems
`
`
`
`
`
`
`
`
`further includes a digital image stabilization unit, wherein the
`
`
`
`
`
`
`
`
`digital image stabilization unit performs final digital image
`
`
`
`
`
`stabilization and small jitter correction.
`
`
`
`
`
`
`Preferably, the camera system further includes a transmit-
`
`
`
`
`
`
`
`
`
`ter for transmitting the final region of interest to a video
`
`
`
`
`
`
`
`receiving unit, typically a remote video receiving unit.
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`The present invention will become fully understood from
`
`
`
`
`
`
`
`
`
`the detailed description given herein below and the accom-
`
`
`
`
`
`
`
`panying drawings, which are given by way of illustration and
`
`
`
`
`
`
`
`
`
`example only and thus not limitative ofthe present invention,
`and wherein:
`
`
`
`
`
`
`
`
`FIG. 1 is a block diagram illustration of an air-born camera
`
`
`
`
`
`
`
`
`system for performing image acquisition and image transmis-
`
`
`
`
`
`
`sion, according to the preferred embodiments of the present
`
`invention;
`
`
`
`
`
`
`FIG. 2 is a schematic illustration of an example spatial
`
`
`
`
`
`
`
`environment, in which the air-bom camera system shown in
`
`
`FIG. 1 operates.
`
`
`
`
`
`FIG. 3 is a block diagram illustration of the post-capture
`
`
`
`
`
`
`
`
`digital zoom and cropping unit ofthe air-born camera system,
`
`
`shown in FIG. 1;
`
`
`
`
`
`
`FIG. 4 is a block diagram illustration of the motorized
`
`
`
`
`compensation unit, shown in FIG. 1;
`
`
`
`
`
`FIG. 5 is a block diagram illustration of a zoom control
`
`
`
`
`
`
`sub-system for an air-bom camera system, according to varia-
`
`
`
`
`tions of the present invention; and
`
`
`
`
`
`
`
`FIG. 6 is a data flow diagram illustration one cycle of an
`
`
`
`
`
`
`
`image acquisition process, according to variations of the
`
`
`present invention.
`DESCRIPTION OF THE PREFERRED
`
`
`EMBODIMENTS
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`Before explaining embodiments of the invention in detail,
`it is to be understood that the invention is not limited in its
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`application to the details of construction and the arrangement
`
`
`
`
`
`
`
`
`of the components set forth in the host description or illus-
`
`
`
`trated in the drawings.
`
`
`
`
`
`
`
`Unless otherwise defined, all technical and scientific terms
`
`
`
`
`
`
`
`
`used herein have the same meaning as commonly understood
`
`
`
`
`
`
`
`
`
`by one ofordinary skill in the art ofthe inventionbelongs. The
`
`
`
`
`
`
`
`
`methods and examples provided herein are illustrative only
`
`
`
`
`and not intended to be limiting.
`
`
`
`
`
`
`
`Reference is now made to the drawings. FIG. 1 is a block
`
`
`
`
`
`
`
`
`diagram illustration of an air-bom camera system 100 for
`
`
`
`
`
`
`performing image acquisition and image transmission,
`
`
`
`
`
`
`
`according to the preferred embodiments ofthe present inven-
`
`
`
`
`
`
`
`
`tion. Air-born camera system 100 includes a high resolution
`
`
`
`
`
`
`
`
`
`
`digital image sensor (typically, in current state of the art,
`
`
`
`
`
`
`
`higher than 1 mega pixels) 110, a coordinate offset calcula-
`
`
`
`
`
`
`
`
`tion unit 120, a displacement sensor 122, a video timing
`
`
`
`
`
`
`
`
`(clock) unit 130, an image sensor configuration control unit
`
`
`
`
`
`
`
`140, a computation unit 150, a post-capturing image process-
`
`
`
`
`
`
`
`
`
`ing unit 160, an X/Y motorized compensation unit 170, pref-
`
`
`
`
`
`
`
`
`
`erably a gimbaled device 180 (on which image sensor 110 is
`
`
`
`
`
`
`
`mounted) and optionally, a digital image stabilization unit
`190.
`
`
`
`
`
`
`
`
`
`It should be noted that although the present invention is
`
`
`
`
`
`
`described in terms ofa computerized camera system mounted
`
`
`
`
`
`
`
`
`on an air born vehicle, the computerized camera system ofthe
`
`
`
`US 8,896,697 B2
`
`
`6
`
`
`
`
`
`
`present invention is not limited to be mounted only on air
`
`
`
`
`
`
`
`vehicles. Similar computerized camera systems can be
`
`
`
`
`
`
`mounted on/in land vehicles, waterway vehicles, carried by a
`
`
`
`
`
`
`
`
`living body, for example by hand, or mounted on any other
`
`
`
`
`
`
`
`
`moving and/or vibrating platform. Similar motion and vibra-
`
`
`
`
`
`
`
`
`tion problems exist in land vehicles, waterway vehicles and
`
`
`
`
`
`
`
`
`other platforms. It should be further noted that typically, the
`
`
`
`
`
`
`
`
`
`smaller the vehicle is the less stable the vehicle is, whereas an
`
`
`
`
`
`
`
`
`air vehicles for carrying camera are typically small.
`
`
`
`
`
`
`Typically, camera system 100 is operatively mounted on an
`
`
`
`
`
`
`
`air-bom vehicle. When in operation, the air-born vehicle
`
`
`
`
`
`
`
`maneuvers to stay in a desired path using a manual or remote
`
`
`
`
`
`
`
`
`steering control. Digital image sensor 110 of camera system
`
`
`
`
`
`
`
`
`100 may encounter two types of motions which need to be
`
`
`
`
`
`
`
`compensated for in order to stabilize the acquired image
`
`
`
`
`
`
`
`
`
`
`streams. Digital image sensor 110 has Pan and Tilt degrees of
`
`
`
`
`
`
`freedom. Regardless of the platform maneuvering, the Pan
`
`
`
`
`
`
`
`
`and Tilt motion of digital image sensor 110 is controlled, on
`
`
`
`
`
`
`
`board or remotely, by a camera steering controller 50. The
`
`
`
`
`
`
`
`camera steering signals sent by camera steering controller 50
`
`
`
`
`
`
`
`
`
`
`carry the data regarding the Pan and Tilt motion of digital
`
`
`
`
`
`
`
`
`image sensor 110. Furthermore, the vehicle typically encoun-
`
`
`
`
`
`
`
`
`
`ters unstable conditions such as air pockets, and incurs vari-
`
`
`
`
`
`
`
`
`ous motions, vibrations and trembling caused by units such as
`
`
`
`engine, motors etc.
`
`
`
`
`
`
`Reference is also made to FIG. 2, which is a schematic
`
`
`
`
`
`
`illustration of an example spatial environment, in which cam-
`
`
`
`
`
`
`
`
`
`era system 100 operates. In the example shown in FIG. 2, the
`
`
`
`
`
`
`
`
`array image sensor 110 images a geographical zone 40, being
`
`
`
`
`
`
`
`
`part of a larger geographical region 30. The operator selects a
`
`
`
`
`
`
`geographical region of interest (GROI) 22 which is imaged
`
`
`
`
`
`
`
`
`onto image region of interest (IROI) 112, being a virtual
`
`
`
`
`
`
`
`
`portion of image sensor 110. Configuration control unit 140
`
`
`
`
`
`
`
`
`continuously tracks the position of IROI 112 and maintains
`
`
`
`
`
`
`
`
`
`IROI 112 within the boundaries of active array of image
`
`
`
`
`
`
`
`
`sensor 110. It should be noted that GROI 22 may also be
`
`
`
`
`
`
`
`selected automatically, for example by tracking a static or
`
`
`
`
`
`
`
`
`moving object. It should be further noted that in some appli-
`
`
`
`
`
`
`
`
`cations, the region of interest is selected from the environ-
`
`
`
`
`
`
`
`
`ment as viewed by the camera. Therefore, the terms “geo-
`
`
`
`
`
`
`graphical region of interest” and “environmental region of
`
`
`
`
`
`interest” are used herein interchangeably.
`
`
`
`
`
`
`
`
`
`To facilitate a high image frame transfer rate, while main-
`
`
`
`
`
`
`
`taining high resolution image sampling by a high resolution
`
`
`
`
`
`
`
`
`image sensor, only the portion of the image frame acquired
`
`
`
`
`
`
`from IROI 112 is preferably transferred to post-capturing
`
`
`
`
`
`
`
`
`image processing unit 160, for further processing. But, to
`
`
`
`
`
`
`
`enable post-capturing image processing unit 160 to perform
`
`
`
`
`
`
`
`
`
`the post processing task more accurately, a raw IROI 114 is
`
`
`
`
`
`
`
`transferred to post-capturing image processing unit 160,
`
`
`
`
`
`
`
`
`
`wherein raw IROI 114 is larger than IROI 112 by a predefined
`
`
`margin of pixels.
`
`
`
`
`
`
`The camera is controlled by a camera steering controller
`
`
`
`
`
`
`
`50, manned or remotely, which typically, enables maneuver-
`
`
`
`
`
`
`
`
`
`
`
`ing the camera in the Pan a