US008908041B2
`
`(12) United States Patent
`Stein et al.
`
`(10) Patent No.:
`(45) Date of Patent:
`
`US 8,908,041 B2
`Dec. 9, 2014
`
`(*) Notice:
`
`(54) STEREO ASSIST WITH ROLLING SHUTTERS
`(71) Applicants: Gideon Stein, Jerusalem (IL); Efim
`Belman, Jerusalem (IL)
`(72) Inventors: Gideon Stein, Jerusalem (IL); Efim
`Belman, Jerusalem (IL)
`(73) Assignee: Mobileye Vision Technologies Ltd.,
`Jerusalem (IL)
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`U.S.C. 154(b) by Oda
`M
`YW-
`(b) by
`yS.
`(21) Appl. No.: 14/156,096
`(22) Filed:
`Jan. 15, 2014
`(65)
`Prior Publication Data
`US 2014/O1981.84 A1
`Jul. 17, 2014
`O
`O
`Related U.S. Application Data
`(60) Provisional application No. 61/752,515, filed on Jan.
`15, 2013, provisional application No. 61/761.724,
`filed on Feb. 7, 2013.
`(51) Int. Cl.
`(2006.01)
`H04N 7/8
`(2006.01)
`H04N 5/232
`(2006.01)
`H04N I3/02
`(2006.01)
`G6K 9/00
`o
`(52) U.S. Cl
`AV e. we
`CPC ........... Hity 13/0282 (2013.01); Hity 5/232
`(2013.01); H04N 13/0296 (2013.01); G06K
`9/00791 (2013.01)
`USPC ........................................... 348/148; 348/153
`(58) Field of Classification Search
`USPC .......................... 348/111-118, 142-160, 837
`See application file for complete search history.
`
`(56)
`
`References Cited
`U.S. PATENT DOCUMENTS
`
`7,112,774 B2
`7,486,803 B2
`
`9, 2006 Baer
`2/2009 Camus
`(Continued)
`
`
`
`FOREIGN PATENT DOCUMENTS
`WO WO 2010, 136344
`12/2010
`WO WO 2012/076274
`6, 2012
`(Continued)
`OTHER PUBLICATIONS
`Zilly et al. "Depth Based Content Creation Targeting Stereoscopic
`and Auto-Stereoscopic Displays.” Fraunhofer Heinrich Hertz Insti
`tute, Germany, 2012 (8 pages).
`Winkler et al. “Stereo/Multiview Picture Quality. Overview and
`Recent Advances.” University of Illinois at Urbana-Champaign,
`Elsevier, 2013 (16 pages).
`Llorca et al., “Stereo Regions-of-Interest Selection for Pedestrian
`Protection: A Survey.” University of Alcata, Madrid, Spain, Elsevier,
`2012 (12 pages).
`
`(Continued)
`Primary Examiner — Behrooz Senfi
`(74) Attorney, Agent, or Firm — Finnegan, Henderson,
`Farabow, Garrett & Dunner, LLP
`(57)
`ABSTRACT
`An imaging system for a vehicle may include a first image
`capture device having a first field of view and configured to
`acquire a first image relative to a scene associated with the
`vehicle, the first image being acquired as a first series of
`image Scanlines captured using a rolling shutter. The imaging
`system may also include a second image capture device hav
`ing a second field of view different from the first field of view
`and that at least partially overlaps the first field of view, the
`second image capture device being configured to acquire a
`second image relative to the scene associated with the vehicle,
`the second image being acquired as a second series of image
`scan lines captured using a rolling shutter. As a result of
`overlap between the first field of view and the second field of
`view, a first overlap portion of the first image corresponds
`with a second overlap portion of the second image. The first
`image capture device has a first scan rate associated with
`acquisition of the first series of image scan lines that is dif
`ferent from a second scan rate associated with acquisition of
`the second series of image scan lines, such that the first image
`capture device acquires the first overlap portion of the first
`image over a period of time during which the second overlap
`portion of the second image is acquired.
`23 Claims, 18 Drawing Sheets
`
`APPL-1023 / Page 1 of 35
`APPLE INC. v. COREPHOTONICS LTD.
`
`

`

`US 8,908,041 B2
`Page 2
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`7,817,187
`7.961,906
`8,330,796
`8,358,359
`8,378,851
`8,417,058
`8,456,517
`8.497,932
`2005/OO77450
`2005/OO781.85
`2009,0201361
`2010, O253784
`2011/0222757
`2011/0242342
`2011/0285982
`2013, OO 10084
`2013/0101176
`2013/O141579
`2013/0147921
`2013,025 0046
`
`10, 2010
`6, 2011
`12, 2012
`1, 2013
`2, 2013
`4, 2013
`6, 2013
`T/2013
`4, 2005
`4, 2005
`8, 2009
`10, 2010
`9, 2011
`10, 2011
`11, 2011
`1, 2013
`4, 2013
`6, 2013
`6, 2013
`9, 2013
`
`Silsby et al.
`Ruedin
`Schmidt et al.
`Baker et al.
`Stein et al.
`Tardif
`Spektor et al.
`Morimoto
`Baer
`Suzuki et al. ................. 348,148
`Lyon et al.
`Oleg ............................. 348,187
`Yeatman et al.
`Goma et al.
`Breed
`Hatano
`Park et al.
`Schofield et al. ............. 348,148
`Mor et al.
`Schofield et al. ............... 348/36
`
`FOREIGN PATENT DOCUMENTS
`
`WO WO 2013/O12335
`WO WO 2013,029608
`
`1, 2013
`3, 2013
`
`OTHER PUBLICATIONS
`Henzle et al., "Computational Stereo Camera System with Program
`mable Control Loop.” Disney Research, Zurich, 2011 (10 pages).
`Gu et al., “Coded Rolling Shutter Photography; FlexibleSpace-Time
`Sampling.” Columbia University, 2010 (8 pages).
`Darms et al., “Data Fusion Strategies in Advanced Driver Assistance
`Systems.” SAE International, oct. 19, 2010 (8 pages).
`Bradley et al., “Synchronization and Rolling Shutter Compensation
`for Consumer video Camera Arrays.” University of British Colum
`bia, 2009 (8 pages).
`Tiemann et al., “Eln Beitrag zur Situationsanalyse im
`vorausschauenden FuBgangershutz. Universitat Dulsburg-Essen,
`Germany, 2012 (173 pages).
`Communication from the International Searching Authority entitled
`“Invitation to Pay Additional Fees and, Where Applicable, Protest
`Fee” and "Annex to Form PCT/ISA206, Communication Relating to
`the Results of the Partial International Search.” dated Aug. 1 2014 (8
`pages).
`Bennett Wilburn et al., “High Performance Imaging Using Large
`Camera Arrays.” Association for Computing Machinery, Inc., vol. 24.
`No. 3 (2006) (12 pages).
`Andrew Adams et al., “The Frankencamera: An Experimental Plat
`form for Computational Photography.” ACM Transactions on Graph
`ics, vol. 29, No. 4. Article 29 (Jul. 2010) (12 pages).
`* cited by examiner
`
`APPL-1023 / Page 2 of 35
`APPLE INC. v. COREPHOTONICS LTD.
`
`

`

`U.S. Patent
`
`Dec. 9, 2014
`
`Sheet 1 of 18
`
`US 8,908,041 B2
`
`
`
`APPL-1023 / Page 3 of 35
`APPLE INC. v. COREPHOTONICS LTD.
`
`

`

`U.S. Patent
`
`Dec. 9, 2014
`
`Sheet 2 of 18
`
`US 8,908,041 B2
`
`
`
`APPL-1023 / Page 4 of 35
`APPLE INC. v. COREPHOTONICS LTD.
`
`

`

`U.S. Patent
`
`Dec. 9, 2014
`
`Sheet 3 of 18
`
`US 8,908,041 B2
`
`
`
`s
`
`APPL-1023 / Page 5 of 35
`APPLE INC. v. COREPHOTONICS LTD.
`
`

`

`U.S. Patent
`
`Dec. 9, 2014
`
`Sheet 4 of 18
`
`US 8,908,041 B2
`
`
`
`FIG. 2a
`
`APPL-1023 / Page 6 of 35
`APPLE INC. v. COREPHOTONICS LTD.
`
`

`

`APPL-1023 / Page 7 of 35
`APPLE INC. v. COREPHOTONICS LTD.
`
`

`

`U.S. Patent
`
`Dec. 9, 2014
`
`Sheet 6 of 18
`
`US 8,908,041 B2
`
`310
`
`320
`
`ON
`
`PERFORMRECTIFICATION
`STEP
`
`332
`
`CROP THE WIDE FOW
`CAMERAMAGE
`
`SMOOTH AND SUBSAMPLE THE
`NARROW FOW IMAGE
`
`
`
`
`
`330
`
`MATCHING
`PAIR
`OBTAINED?
`
`
`
`
`
`
`
`COMPLETE STEREO PROCESSING
`
`FIG. 3
`
`APPL-1023 / Page 8 of 35
`APPLE INC. v. COREPHOTONICS LTD.
`
`

`

`U.S. Patent
`
`Dec. 9, 2014
`
`Sheet 7 of 18
`
`US 8,908,041 B2
`
`- - - - - - - - -
`- - - - - - -
`
`
`
`y (pixels)
`FIG. 4
`
`- - - - - - - - - - - -
`
`r
`
`i
`---------
`s:----------
`
`A.
`
`-
`
`--------
`f
`:
`
`-------------
`
`---------
`
`WF-100
`O
`
`rOW-60
`row-40
`OW
`OW
`-
`- - - row=20
`OWE40
`
`Ow=80
`
`---------
`as me us
`
`-
`scSL
`s
`sq m up ako as as a
`to -m or mollus so man no um man
`F:
`a vil
`
`------------------------------------------4-----
`
`i----------------------------
`- - - - - - - - - - - - - - - - - - w -- - - - - - - - - - - - - - - - - - - - -
`
`Lateral position (m)
`FIG. 5
`
`APPL-1023 / Page 9 of 35
`APPLE INC. v. COREPHOTONICS LTD.
`
`

`

`U.S. Patent
`
`Dec. 9, 2014
`
`Sheet 8 of 18
`
`US 8,908,041 B2
`
`
`
`- V - - - - -
`
`-------
`
`--------------
`
`or - - - - -
`
`Trrier rr way v rersy wres
`wa
`
`- a a 1-1 was dates'
`
`tet-at-a le.
`
`ypixels
`
`FIG. 6
`
`APPL-1023 / Page 10 of 35
`APPLE INC. v. COREPHOTONICS LTD.
`
`

`

`U.S. Patent
`
`Dec. 9, 2014
`
`Sheet 9 of 18
`
`US 8,908,041 B2
`
`CHOOSE OPTIMALLINE TOSYNCHRONIZE
`
`
`
`
`
`PREPROCESSNARROW FOW IMAGE BY
`A HOMOGRAPHY OFATARGET PLANE
`
`710
`
`720
`
`PERFORM DENSE DEPTH MAP
`COMPUTATION INA HORIZONTAL STRIP
`
`
`
`730
`
`740
`
`MATCHING
`LARGE
`DISPARITIES
`
`YES
`
`SEARCHADJACENT LINESTO
`DETERMINEMATCHES
`
`742
`
`741
`
`ESTIMATE FOE
`
`750
`
`SOLVE FOR DISTANCEESTIMATES
`
`FIG. 7
`
`
`
`
`
`
`
`
`
`NO
`
`
`
`751
`
`APPL-1023 / Page 11 of 35
`APPLE INC. v. COREPHOTONICS LTD.
`
`

`

`U.S. Patent
`
`Dec. 9, 2014
`
`Sheet 10 of 18
`
`US 8,908,041 B2
`
`-------------------
`
`• • • • • • 4. - - - - - - -
`
`
`
`0.5
`O
`-0.5
`Lateral position (m)
`FIG. 8a
`
`15
`
`- - - - - - - - - - -
`
`--wra aasses - - -
`
`an as wre---
`
`-----
`
`---a st-was-- wer
`---
`
`- - - - - - - - - -
`
`- - - - - - - - - - - - - - - - - -
`
`Lateral position (m)
`FIG. 8b
`
`APPL-1023 / Page 12 of 35
`APPLE INC. v. COREPHOTONICS LTD.
`
`

`

`U.S. Patent
`
`Dec. 9, 2014
`
`Sheet 11 of 18
`
`US 8,908,041 B2
`
`
`
`Disparity for points on road (H
`
`30m/s)
`1.25m, V
`
`O
`
`0
`
`original
`with time delay
`no time delay
`
`wn
`
`or a on and on m r -
`
`u
`
`- - - - - - - - - - - - - - - - - - - - - - - - - - - - -
`
`to go 0
`
`o o o o
`
`4.
`
`to so to e to on to on
`
`400
`
`do
`0 0
`
`Khe
`
`O
`0e
`
`0.
`
`-o-o-o-o-o-
`
`MM polo Mae MAOM
`
`-o-o-o-o-o-o-o-o-o-4-00-00-00-4-4s-4------
`popo popo dope M topo Ape dopo popo
`
`Mean MAn Albao MoMAs
`
`MMAM AM As eas Mao Mao Mad M
`- - - - - - - - - - -- - - - - - - - - - -- - - - - - - - -
`- - - - - - - - - - -
`-----------
`Popedopop
`Poo poppuppopo Pope
`
`torprorooper pre-ore
`
`boopee de Popaed
`
`opodoped
`
`popee PPPopodott
`
`rrrrrrrrrrt
`
`--ee-to------
`
`100
`
`200
`
`300
`
`400
`
`O
`pixels
`
`FIG. 9
`
`APPL-1023 / Page 13 of 35
`APPLE INC. v. COREPHOTONICS LTD.
`
`

`

`U.S. Patent
`
`Dec. 9, 2014
`
`Sheet 12 of 18
`
`US 8,908,041 B2
`
`
`
`- w - - - - - - - - - - - - - - - - - - - - - - - - - - - m r -
`
`- - - - - - - - - -
`
`-
`
`- - - - -
`
`- - a rare rr.
`
`or
`
`- - - was as - - - - - - - - m r - - - - - - - - - - - - -
`
`- - - r or - - - - - - - ---n - - - - - - - - - -A- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
`- - - - - - - - - - - - - - - - -
`
`-------------------------------
`- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -N-- - - - - - - - - - - - -
`
`FIG. 10
`
`APPL-1023 / Page 14 of 35
`APPLE INC. v. COREPHOTONICS LTD.
`
`

`

`U.S. Patent
`
`Dec. 9, 2014
`
`Sheet 13 of 18
`
`US 8,908,041 B2
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`PREWARPNARROW FOW IMAGE
`ACCORDING TO SPEED DEPENDENT
`FUNCTION
`
`1110
`
`
`
`COMPUTE HOMOGRAPHY MATRIXDUETO
`MOTION OF THE VEHICLE OVER THE
`ROAD PLANE PERLINE TIMING
`
`1120
`
`COMPUTE HOMOGRAPHY FOR
`EACH LINE'S SKEW
`
`
`
`
`
`1140
`
`LINE'S
`HOMOGRAPHY
`KNOWN
`
`
`
`
`
`WARPLINEACCORDING
`TO THE LINE'S HOMOGRAPHY
`
`
`
`
`
`
`
`
`
`
`
`
`
`1130
`
`NO
`
`COMPUTE A HOMOGRAPHY
`MATRIX FOR THE TIME
`DIFFERENCE BETWEEN ONE
`FRAME AND THE NEXT
`
`1142
`
`COMPUTE THE IMAGE
`MOTION DUE TO THE
`HOMOGRAPHY
`
`1150
`
`WARP EACH LINE WITH A
`FRACTION OF THE MOTION
`DEPENDING ON THE TIME
`SKEW
`
`FIG. 11
`
`APPL-1023 / Page 15 of 35
`APPLE INC. v. COREPHOTONICS LTD.
`
`

`

`U.S. Patent
`
`Dec. 9, 2014
`
`Sheet 14 of 18
`
`US 8,908,041 B2
`
`6----------
`---------
`
`---------------------
`
`4----------
`
`- - - - - - - - - - - - - - - - - - - - -
`- - - - - - - - - - m - - - - - - - - - - - - - - - - - - -
`
`1 or w we -
`
`1293
`1292
`1291
`1290
`128O
`1270
`1260
`1250
`1240
`
`1210
`
`
`
`FIG. 12a
`
`------------------.
`
`t
`
`FIG.12b
`
`APPL-1023 / Page 16 of 35
`APPLE INC. v. COREPHOTONICS LTD.
`
`

`

`U.S. Patent
`
`Dec. 9, 2014
`
`Sheet 15 of 18
`
`US 8,908,041 B2
`
`
`
`- - - - -A- - - - - - - - -
`
`--------------
`ls man - ww- - - - - -an n w w a
`
`- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
`
`1320
`
`- - - - -a-
`-- - - - - - - - - - - w - - - -
`
`- - - - - - - - - -
`
`- - - - - - - - - - - - - - - - -
`
`- w w w -
`
`us
`
`as
`
`N
`- - - - - - - - - - - - - - - - - - m as - - - - - - - - - - -
`... - - - near - an e-m- - - - - - - - - - - - - -n - - - - - - - - - -
`- - - - - - - - - - - - - - -
`
`- - - - - - - ess - - - - - - - - - - - - - - - - - -F
`
`-200
`
`-100
`-150
`row y (pixels)
`
`FIG. 13
`
`APPL-1023 / Page 17 of 35
`APPLE INC. v. COREPHOTONICS LTD.
`
`

`

`U.S. Patent
`
`Dec. 9, 2014
`
`Sheet 16 of 18
`
`US 8,908,041 B2
`
`
`
`1410
`
`- - - - - - - - - - - - - - - - - - - - - - -
`
`as - - - - - - - - - - - - - - N - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
`
`- - - - - - - - - - - - - - - - - - - - - - - - - - - -a- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
`
`----------------------
`-N-------
`
`- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -ae - - - - - - - - - - - - - - - - -
`
`- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - ww-r- rs -- - - - - - - - - - - - - - as as as as a rew - - - - - - - - - - - N - - - - - -
`
`image row (pixels)
`
`FIG. 14a
`
`APPL-1023 / Page 18 of 35
`APPLE INC. v. COREPHOTONICS LTD.
`
`

`

`U.S. Patent
`
`Dec. 9, 2014
`
`Sheet 17 of 18
`
`US 8,908,041 B2
`
`
`
`1420
`
`- m - - - - - - raz - - - - - - www-rro - w - - - - -
`
`up w w
`
`- - - - - - - --- was .
`
`.
`
`.
`
`.
`
`... - - - - - - - - - - - - - - -
`
`-----------
`------N-----
`------
`
`-----------------
`-------
`
`- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
`
`- - - - - - - - -
`- - - - - - - - - - - - - - - - - - - -
`H - - - - - - w
`
`ur r - r - N-H - - - - - a m + are m r - us ur ur or - - - - -
`
`- - - - - - - - - - - - - - - - - - - - - - - -
`
`-- - - - - - -
`
`image row (pixels)
`
`FIG. 14b
`
`APPL-1023 / Page 19 of 35
`APPLE INC. v. COREPHOTONICS LTD.
`
`

`

`U.S. Patent
`
`Dec. 9, 2014
`
`Sheet 18 of 18
`
`US 8,908,041 B2
`
`SPLT LATERAL DISPARTY INTOMOTION
`DISPARITY ANDTRUE DISPARITY
`
`
`
`APPROXIMATE MOTION DISPARTY BASED
`ON THE TIME SKEW BETWEEN THE ROWS
`INTHE NARROW AND WIDE FOY CAMERAS
`
`DETERMINE LATERAL DISPARTY AT TWO
`DIFFERENT KNOWN TIMES
`
`SOLVE FOR TRUE DISPARITY
`
`1510
`
`1520
`
`1530
`
`1540
`
`FIG. 15
`
`APPL-1023 / Page 20 of 35
`APPLE INC. v. COREPHOTONICS LTD.
`
`

`

`US 8,908,041 B2
`
`1.
`STEREO ASSIST WITH ROLLING SHUTTERS
`
`This application claims priority under 35 U.S.C. S 119 to
`U.S. Provisional Application No. 61/752,515, filed Jan. 15,
`2013, and U.S. Provisional Application No. 61/761,724, filed
`Feb. 7, 2013. Each aforementioned application is incorpo
`rated herein by reference in its entirety.
`
`5
`
`BACKGROUND
`
`2
`applications that use cameras having different fields of view.
`For instance, if both a wide FOV camera and a narrow FOV
`camera are aimed at a common scene, then the narrow FOV
`camera may overlap with only a portion of the FOV of the
`wide FOV camera. If both cameras acquire images as a simi
`lar number of image scan lines acquired at a similar line Scan
`rate, then the acquired image scan lines in the area of the
`overlap in the fields of view of the two cameras will lack
`synchronization. Such a lack of synchronization may intro
`duce difficulties in determining a correspondence of image
`points in a first image from the wide FOV camera with image
`points in a second image from the narrow FOV camera, which
`can lead to significant inaccuracies object distance measure
`mentS.
`
`SUMMARY
`
`Consistent with disclosed embodiments, an imaging sys
`tem for a vehicle is provided, the system comprising a first
`image capture device having a first field of view and config
`ured to acquire a first image relative to a scene associated with
`the vehicle, the first image being acquired as a first series of
`image Scanlines captured using a rolling shutter. The imaging
`system may also include a second image capture device hav
`ing a second field of view different from the first field of view
`and that at least partially overlaps the first field of view, the
`second image capture device being configured to acquire a
`second image relative to the scene associated with the vehicle,
`the second image being acquired as a second series of image
`scan lines captured using a rolling shutter. As a result of
`overlap between the first field of view and the second field of
`view, a first overlap portion of the first image may correspond
`with a second overlap portion of the second image. The first
`image capture device may have a first scan rate associated
`with acquisition of the first series of image Scanlines that may
`be different from a second scan rate associated with acquisi
`tion of the second series of image Scanlines, such that the first
`image capture device acquires the first overlap portion of the
`first image over a period of time during which the second
`overlap portion of the second image is acquired.
`Consistent with disclosed embodiments, a vehicle is dis
`closed, the vehicle including a body and an imaging system
`for a vehicle, the system comprising a first image capture
`device having a first field of view and configured to acquire a
`first image relative to a scene associated with the vehicle, the
`first image being acquired as a first series of image scan lines
`captured using a rolling shutter. The imaging system may also
`include a second image capture device having a second field
`of view different from the first field of view and that at least
`partially overlaps the first field of view, the second image
`capture device being configured to acquire a second image
`relative to the scene associated with the vehicle, the second
`image being acquired as a second series of image scan lines
`captured using a rolling shutter. As a result of overlap between
`the first field of view and the second field of view, a first
`overlap portion of the first image may correspond with a
`second overlap portion of the second image. The first image
`capture device may have a first scan rate associated with
`acquisition of the first series of image scan lines that may be
`different from a second scan rate associated with acquisition
`of the second series of image scan lines, such that the first
`image capture device acquires the first overlap portion of the
`first image over a period of time during which the second
`overlap portion of the second image is acquired, wherein the
`period of time is associated with a ratio between the first scan
`rate and the second scan rate.
`
`I. Technical Field
`The present disclosure relates generally to camera imaging
`systems and more specifically to devices and techniques for
`capturing images in a rolling shutter, Stereo image acquisition
`system that may be included on a vehicle.
`II. Background Information
`Camera based driver assistance systems for use in vehicles
`may include monocular object detection systems that rely
`primarily on a single camera to collect images. Because the
`images in these types of systems are captured from a single
`point of view, direct determination of distance to a target
`object can be challenging. Therefore, monocular object
`detection systems may rely upon estimation techniques to
`indirectly determine distances to a target object based on
`information about the object class and/or contextual informa
`tion relative to the context of the object (e.g., aspects of a road
`plane on which the target object resides). In some cases,
`monocular systems may use pattern recognition to detect a
`specific object class prior to monocular range estimation.
`Camera based driver assistance systems for use in vehicles
`may also include stereo systems that employ two cameras. In
`some systems, these cameras may be mounted side-by-side
`where epipolar lines are aligned with the horizontal image
`scan lines. Such a system may use a dense disparity map to
`create a 3D map of the environment. The system may then use
`this 3D representation for foreground and/or background seg
`mentation, for instance, to find candidate regions for further
`processing. The system may also use the 3D representation to
`locate interesting objects or to estimate range and/or range
`rate to detected objects. Such stereo systems may work well
`with close and medium range targets and in good weather, and
`may give depth maps on general targets. However, Such stereo
`systems may experience difficulties during adverse weather
`conditions or where cluttered scenes exist. Additionally, these
`systems may have difficulty imaging objects at longer dis
`tances from the vehicle.
`Some imaging systems, such as systems described in U.S.
`Pat. No. 7,786.898, may fuse information from both a
`monocular system and a stereo system. This type of system
`may include a primary camera responsible for target detec
`tion/selection and range estimation. A secondary camera may
`provide stereo-range on selected targets for purposes of target
`verification.
`Some stereo systems may include an asymmetric configu
`ration that may combine Stereo-depth and monocular depth
`together. For instance, two asymmetric cameras (e.g., with
`different fields of view (FOV) and focal lengths) may be
`employed for independent applications. Additionally, image
`information from these cameras may be combined to provide
`Stereo depth. For cameras with global shutters, such stereo
`processing may involve, among other things, cropping the
`wider FOV camera, Smoothing and Subsampling of images,
`and/orrectification in order to provide a matching image pair.
`Recent generations of image sensors, including those that
`may be used in automotive sensors, may include a rolling
`shutter. Such a rolling shutter may introduce complications in
`Stereo image processing, especially in asymmetric stereo
`
`10
`
`15
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`APPL-1023 / Page 21 of 35
`APPLE INC. v. COREPHOTONICS LTD.
`
`

`

`US 8,908,041 B2
`
`3
`Consistent with disclosed embodiments, an imaging sys
`tem for a vehicle is provided, the system comprising a first
`image capture device having a first field of view and config
`ured to acquire a first image relative to a scene associated with
`the vehicle, the first image being acquired as a first series of
`image scan lines captured using a rolling shutter. The system
`may also include a second image capture device having a
`second field of view different from the first field of view and
`that at least partially overlaps the first field of view, the second
`image capture device being configured to acquire a second
`image relative to the scene associated with the vehicle, the
`second image being acquired as a second series of image scan
`lines captured using a rolling shutter. As a result of overlap
`between the first field of view and the second field of view, a
`first overlap portion of the first image may correspond to a
`second overlap portion of the second image. They system
`may include at least one processing device configured to:
`receive the first image from the first image capture device;
`receive the second image from the second image capture
`device; and correlate at least a first area of the first overlap
`portion of the first image with a corresponding second area of
`the second overlap portion of the second image.
`It is to be understood that both the foregoing general
`description and the following detailed description are exem
`25
`plary and explanatory only and are not restrictive of the
`claims.
`
`5
`
`10
`
`15
`
`4
`FIG. 10 is an example of the shift in they direction as a
`function of the image row:
`FIG. 11 shows exemplary process for use in systems to
`estimate distances to road features;
`FIG.12a is an example of the disparity error for different
`rows in the image introduced by the vehicle forward motion
`and the time delay;
`FIG.12b is an example of the disparity error as a fraction of
`the disparity on that row:
`FIG. 13 is an example of the expected y shift due to time
`delay and forward motion if a given row is synchronized;
`FIG. 14a is an example of the ratio of disparity error due to
`motion over true disparity due to baseline as a function of row
`(y);
`FIG. 14b is an example of the ratio of disparity error due to
`motion over true disparity due to baseline for the case where
`the object is moving laterally;
`FIG. 15 shows an exemplary process for use in systems
`where there is motion disparity.
`
`DETAILED DESCRIPTION
`
`The following detailed description refers to the accompa
`nying drawings. Wherever possible, the same reference num
`bers are used in the drawings and the following description to
`refer to the same or similar parts. While several illustrative
`embodiments are described herein, modifications, adapta
`tions and other implementations are possible. For example,
`Substitutions, additions or modifications may be made to the
`components illustrated in the drawings, and the illustrative
`methods described herein may be modified by substituting,
`reordering, removing, or adding steps to the disclosed meth
`ods. Accordingly, the following detailed description is not
`limiting of the disclosed embodiments. Instead, the proper
`Scope is defined by the appended claims.
`Referring to the accompanying drawings, FIG. 1a is a
`diagrammatic, side view representation of an exemplary
`vehicle imaging system consistent with the presently dis
`closed. FIG. 1b is diagrammatic, top view illustration of the
`embodiment shown in FIG. 1a. As illustrated in FIG. 1a, a
`disclosed embodiment of the present invention may include a
`vehicle 150 having a system 100 with a first image capture
`device 110 and a second image capture device 120 and a
`processor 130. While two image capture devices 110 and 120
`are shown, it should be understood that other embodiments
`may include more than two image capture devices.
`It is to be understood that the disclosed embodiments are
`not limited to vehicles and could be applied in other contexts.
`It is also to be understood that disclosed embodiments are not
`limited to a particular type of vehicle 150 and may be appli
`cable to all types of vehicles including automobiles, truck
`trailers and other types of vehicles.
`The processor 130 may comprise various types of devices.
`For example, processor 130 may include a controller unit, an
`image preprocessor, a central processing unit (CPU), Support
`circuits, digital signal processors, integrated circuits,
`memory, or any other types of devices for image processing
`and analysis. The image preprocessor may include a video
`processor for capturing, digitizing and processing the imag
`ery from the image sensors. The CPU may comprise any
`number of microcontrollers or microprocessors. The Support
`circuits may be any number of circuits generally well known
`in the art, including cache, power Supply, clock and input
`output circuits. The memory may store software that when
`executed by the processor, controls the operation of the sys
`tem. The memory may include databases and image process
`ing software. The memory may comprise any number of
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`30
`
`35
`
`40
`
`The accompanying drawings, which are incorporated in
`and constitute a part of this disclosure, illustrate various dis
`closed embodiments. In the drawings:
`FIG. 1a is a diagrammatic, side view representation of an
`exemplary vehicle imaging system consistent with the pres
`ently disclosed embodiments:
`FIG. 1b is a diagrammatic, top view illustration of the
`embodiment shown in FIG. 1a,
`FIG. 1c is a diagrammatic, top view illustration of an
`exemplary vehicle imaging system with another camera con
`figuration consistent with the presently disclosed embodi
`ments;
`FIG.2a represents overlapping fields of view of two cam
`eras having different fields of view, according to an exem
`plary disclosed embodiment;
`45
`FIG.2b represents overlapping fields of view of two cam
`eras having different fields of view, according to another
`exemplary disclosed embodiment;
`FIG.3 represents an exemplary process for use in systems
`employing cameras with global shutters;
`FIG. 4 is an example of displacement of points in the image
`from the synchronized line for targets at various distances
`from the vehicle;
`FIG. 5 is an example of depth estimates for a target at a
`given distance from a vehicle:
`FIG. 6 shows displacement iny for various distance values
`after a fix involving pre-processing an image with the homog
`raphy of a target plane;
`FIG. 7 shows exemplary process for use in systems to
`perform depth measurements for upright objects;
`FIG. 8a shows depth estimates for a target at a given dis
`tance from the vehicle for a given vehicle forward motion and
`lateral velocity;
`FIG. 8b shows corrected depth estimates for forward
`vehicle motion and no lateral velocity:
`FIG. 9 is an example of the disparity expected on the road
`Surface due to the camera baseline;
`
`50
`
`55
`
`60
`
`65
`
`APPL-1023 / Page 22 of 35
`APPLE INC. v. COREPHOTONICS LTD.
`
`

`

`US 8,908,041 B2
`
`15
`
`25
`
`30
`
`40
`
`5
`random access memory, read only memory, flash memory,
`disk drives, optical storage, tape storage, removable storage
`and other types of storage. In one instance, the memory may
`be separate from the processor 130. In another instance, the
`memory may be integrated into the processor 130.
`The first image capture device 110 may include any suit
`able type of image capture device. Image capture device 110
`may include an optical axis 116. In one instance, the image
`capture device 110 may include an Aptina M9V024 WVGA
`sensor with a global shutter. In other embodiments, image
`10
`capture device 110 may include a rolling shutter. Image cap
`ture device 110 may include various optical elements. In
`Some embodiments one or more lenses may be included, for
`example, to provide a desired focal length and field of view
`for the image capture device. In some embodiments, image
`capture device 110 may be associated with a 6 mm lens or a 12
`mm lens. In some embodiments, image capture device may be
`configured to capture images having a desired FOV, includ
`ing, for example, a wide FOV, such as a 46 degree FOV, 50
`degree FOV, 52 degree FOV, or greater. In some embodi
`ments, image capture device 110 may include a wide angle
`bumper camera or one with up to a 180 degree FOV.
`The first image capture device 110 may acquire a plurality
`of first images relative to a scene associated with the vehicle
`150. Each of the plurality of first images may be acquired as
`a series of image scan lines, which may be captured using a
`rolling shutter. Each scan line may include a plurality of
`pixels.
`The first image capture device 110 may have a scan rate
`associated with acquisition of each of the first series of image
`Scanlines. The scan rate may refer to a rate at which an image
`sensor can acquire image data associated with each pixel
`included in a particular scan line.
`The first image capture device 110 may contain any Suit
`able type of image sensor, including CCD sensors or CMOS
`35
`sensors, for example. In one embodiment, a CMOS image
`sensor may be employed along with a rolling shutter, Such
`that each pixel in a row is read one at a time, and Scanning of
`the rows proceeds on a row-by-row basis until an entire image
`frame has been captured. In some embodiments, the rows
`may be captured sequentially from top to bottom relative to
`the frame.
`The use of a rolling shutter may result in pixels in different
`rows being exposed and captured at different times, which
`may cause skew and other image artifacts in the captured
`image frame. On the other hand, when the image capture
`device 110 is configured to operate with a global or synchro
`nous shutter, all of the pixels may be exposed for the same
`amount of time and during a common exposure period. As a
`result, the image data in a frame collected from a system
`employing a global shutter represents a Snapshot of the entire
`FOV at a particular time. In contrast, in a rolling shutter
`application, each row in a frame is exposed and data is capture
`at different times. Thus, moving objects may appear distorted
`in an image capture device having a rolling shutter. This
`phenomenon will be described in greater detail below.
`The second image capture device 120 may be any type of
`image capture device. Like the first image capture device 110.
`image capture device 120 may include an optical axis 126. In
`one embodiment, image capture device 120 may include an
`Aptina M9V024 WVGA sensor with a global shutter. Alter
`natively, image capture device 120 may include a rolling
`shutter. Like image capture device 110, image capture device
`120 may be configured to include various lenses and optical
`elements. In some embodiments, lenses associated with
`image capture device 120 may provide a FOV that is the same
`as or narrower than a FOV associated with image capture
`
`45
`
`50
`
`55
`
`60
`
`65
`
`6
`device 110. For example, image capture de