as) United States
`a2) Patent Application Publication co) Pub. No.: US 2013/0101176 Al
`
`(43) Pub. Date: Apr. 25, 2013
`PARK et al.
`
`US 20130101176A1
`
`(54)
`
`(75)
`
`3D IMAGE ACQUISITION APPARATUS AND
`METHOD OF CALCULATING DEPTH
`INFORMATIONIN THE 3D IMAGE
`ACQUISITION APPARATUS
`
`Inventors: Yong-hwa PARK,Yongin-si (KR);
`Jang-woo YOU,Yongin-si (KR);
`Hee-sun YOON,Seoul (KR)
`
`Publication Classification
`
`(51)
`
`Int. Cl.
`GO06K 9/00
`HOAN 13/02
`(52) U.S.CL
`USPC.
`eesssssssseeseeeee 382/106; 348/49; 348/E13.074
`
`(2006.01)
`(2006.01)
`
`ABSTRACT
`(57)
`A 3-dimensional (3D) image acquisition apparatus and a
`method of calculating depth information in the 3D image
`acquisition apparatus, the 3D image acquisition apparatus
`including: an optical modulator for modulating light reflected
`(21) Appl. No.: 13/594,094
`from a subject by sequentially projected N (N is 3 oralarger
`natural number) light beams; an image sensorfor generating
`(22)
`Filed:
`Aug. 24, 2012
`N sub-imagesby capturing the light modulated bythe optical
`modulator; and a signal processorfor calculating depth infor-
`mation regarding a distance to the subject by using the N
`sub-images.
`
`(73) Assignee: SAMSUNG ELECTRONIC CO.,.LTD.,
`Suwon-si (KR)
`
`(30)
`
`Foreign Application Priority Data
`
`Oct. 25,2011
`
`(KR) wee 10-2011-0109431
`
`INTENSIVELY PROJECT N DIFFERENT
`
`PROJECTION LIGHT BEAMS TO SUBJECT
`
`MODULATE N REFLECTION LIGHT
`BEAMS REFLECTED FROM SUBJECT
`
`GENERATE N SUB-IMAGES BY CAPTURING N
`MODULATED REFLECTION LIGHT BEAMS
`
`PHASES OF PROJECTION LIGHT BEAMS FROM MEMORY
`
`READ WEIGHTING FACTORS CORRESPONDING
`TO NUMBER OF USED PROJECTION LIGHT BEAMS,
`INTENSITIES OF PROJECTION LIGHT BEAMS, AND
`
`
`CALCULATE FIRST AVERAGE IMAGE V BY MULTIPLYING N
`SUB-IMAGES BY FIRST WEIGHTING FACTORS
`
`CALCULATE SECOND AVERAGE IMAGE U BY MULTIPLYING
`N SUB-IMAGES BY SECOND WEIGHTING FACTORS
`
`CALCULATE DEPTH INFORMATION FROM FIRST
`AVERAGE IMAGE V AND SECOND AVERAGE IMAGE U
`
`St
`
`$2
`
`34
`
`qn 6
`
`S7
`
`Align Ex. 1016
`U.S. Patent No. 9,962,244
`
`0001
`
`Align Ex. 1016
`U.S. Patent No. 9,962,244
`
`0001
`
`

`

`Patent Application Publication
`
`US 2013/0101176 Al
`
`T“Old
`
`
`
`s“iydagat_|NSISTO3cuideq——L-fTWIELHONGNOIZSéd2||7WaHOMNOWLOSTA3YNOILOFTISY
`
`
`HOM~|—NOILOTT234eda|TWNOIS|icNOLLYINGON5||USTIOULNOOWold|«|YadLHOIM
`1SHI4
`
`
`
`NOILOSTI34=cujdeq-——y—|AffttegWYSELHOMTNOLLOFTISYWN2OSSS00ud
`
`NOLLOSPOudTIN|
`\ujdeq-=--t--f}--f-4--(LdWol4)HOMldSdagLEY.
`
`
`
`.|(SdWold)
`
`
`901GOLOLLOLSOL601fo:—~{--4-=e—VSSLHDITCNOOSS|arW36NOILOS143
`
`
`||||WW39LHOMNOLLOIPOHdNODS
`
`baaWv3aLHOMNOLLOAPOHdLSE
`
`
`LHONTNOLLOSTISYSHI
`
`N38LHOMNOLOTISIYYIN
`
`NOILOAROLd
`
`LHOM002
`
`0002
`
`0002
`
`
`
`

`

`Patent Application Publication
`
`Apr. 25, 2013 Sheet 2 of 10
`
`US 2013/0101176 Al
`
`CqWol4
`
`tdWOl{
`
`
`
`WdLHO!1NOILOS1454YIN
`
`(9)
`
`EdWol4
`
`édWJ
`
`LdWwo4
`
`ahrdgbdgledg]4g
`
`OKrepbeplepl4g
`
`
`SdOy.rdapltdefedgdgCOL
`
`
`
`N96ew)ans
`
`TOlLdO
`
`NOILVINGOW
`
`TYNOIS
`
`Zs62W]ng
`
`TWNOIS
`
`NOLVINGOA
`
`
`“TTFT4
`
`TWolldO|AysvaquyipeyeinpoyTWNOIS
`
`oSWolldO
`
`NOILVINGOW
`
`|edegns
`
`0003
`
`GdWol-
`
`GdWOL4
`
`bdW014Pdwol4
`
`OdWOlYEdWOL4
`
`ZqWoZdWO4|woLdWol4
`
`eowld
`
`(9)(Vi
`
`
`
`WV¥49LHOMNO|LOd444GNOOASWY49LHDITNO|LOd1gSYLSdlz
`
`
`
`
`
`
`
`0003
`
`
`
`
`

`

`Patent Application Publication
`
`Apr. 25, 2013 Sheet 3 of 10
`
`US 2013/0101176 Al
`
`Ssta‘
`4fsberu|ipere|npoy
`»—-WO|LdOYIN
`
`WOILdO
`
`NOILVINGOW
`
`TYNDIS
`
`No6ewang
`
`(0)
`
`(9)&Ole
`
`(¥)
`
`
`
`W¥dLHOMNOILOI1454YIN
`
`AV4dLHOMNOWO1334GNOQSS
`
`
`
`
`W349LHOMNOILOS143¢LSul4
`
`Gdwol4
`
`tdWO
`
`EqWO
`
`cdWOl4
`
`LdWo
`
`GdWO
`
`vdWo
`
`edWo
`
`édWO
`
`LdWoJ4
`
`pdWOl4
`
`SdWOolJ
`
`GqWO|
`
`Zdwo
`
`LdWol-
`
`
`
`oYragedgledg)eg
`
`opyraghagleag|'4q
`
`
`GdLDRTTINTOADTORTILSdldQ.
`
`
`_regaloIdcol
`
`TWOLdO
`
`0004
`
`
`
`zo6e|qng
`NOILYINGOW
`
`WolldO
`
`TWNDIS
`
`L
`
`
`
`apewlans
`
`
`
`
`
`
`
`QNOOAS!fstulpareynpoyTYNOIS|NOILVINGON
`
`0004
`
`
`
`
`
`
`
`

`

`Patent Application Publication
`
`Apr. 25, 2013 Sheet 4 of 10
`
`US 2013/0101176 Al
`
`FIG. 4A
`
`OPTICAL POWER
`
` AoJ
`Aai
`-_
`
`ee Wele | 8BBe po a epee
`L EXPOSURETIME OF IMAGE PICKUP DEVICE .
`
`
`
`
`DUTY RATE 100%
`
`GAIN
`
`E OF OPTICAL MODULATOR
`
`0005
`
`

`

`Patent Application Publication
`
`Apr. 25, 2013 Sheet 5 of 10
`
`US 2013/0101176 Al
`
`FIG. 4B
`
`OPTICAL POWER
`
`
`
`
`
`EXPOSURE TIME OF INAGE PICKUP DEVICE
`
`
`DUTY RATE 20%
`es
`
`0006
`
`0006
`
`

`

`Patent Application Publication
`
`Apr. 25, 2013 Sheet 6 of 10
`
`US 2013/0101176 Al
`
`ONINGOI-FOVAIFI 440
`
`
`NOILA1dWOO
`NOILAIdWOO
`NOILAIdWOO
`NOILAF1dWOO
`
`ONINGOS-JOVIT€1
`
`ONINYOS-JOVMIcl
`ONIMYOS-JOVWILI
`
`G‘Old
`
`<O8lL
`
`oO
`
`NOILO414344
`
`LHOM
`
`WOlLdO
`
`qOLVINGOW
`
`a‘4SNIT8€21
`::
`<—_
`
`Inopesy
`Sul,
`eAnsodxy
`
`aul
`
`™
`
`ann
`
`JOVI
`
`dNwWOld
`
`SOIAAG
`
`0007
`
`0007
`
`

`

`Patent Application Publication
`
`Apr. 25, 2013 Sheet 7 of 10
`
`US 2013/0101176 Al
`
`ONINHOS-JOVNIPl=ONINGOS-JOVNIE!=ONINWOS-JOVNIcl=ONINHOS-JOVTLI AWIL
`TWOlLdO LHONM
`NOILAIdNOONOILAIdWOONOILATdNOONOILAIdNOOD
`
`
`
`
`aa.aINI1619dnd
` 440YOLVINGOW
`
`!a|JOINIC
`
`inoavay
`
`
`
`ANI]8EC1
`
`
`
`NOILOAIssa
`
`:AASANSOdXS
`
`4Ban
`
`MiJOVI
`
`0008
`
`0008
`
`

`

`Patent Application Publication
`
`Apr. 25, 2013 Sheet 8 of 10
`
`US 2013/0101176 Al
`
`
`
`
`
`f)ebew)pabeloayA96eul|pabeleay
`
`AZ“Ole
`
`andetnt
`
`
`
`1=¥Npl<p=1nl'V<>=A
`al=7Nso
`
`0009
`
`
`
`
`
`abe]uIdeq
`
`LD
`
`wh
`
`(DARI?
`AOLv
`=yjdap
`
`0009
`
`
`
`

`

`Patent Application Publication
`
`Apr. 25, 2013 Sheet 9 of 10
`
`US 2013/0101176 Al
`
` o1ao0,|6ao0gco0£099
`
`9099gq90ya90€q0020091099siojoe4‘sea||||||||||YB93$0ON
`
`
`
`ee
`
`
`
`
`
`
`
`
`
`
`
`(BOE‘oLGZ‘0907‘obGL‘oLOL‘ob‘o0)ZEN‘(cO0E‘oOPZ‘o08L‘oOZL“009‘o0)
`
`
`
`
`
`
`
`
`
`
`(0882‘o9LZ‘oPPL‘o2Z‘o0)S=N(0022‘o08L‘006‘00)PEN‘(0077‘o0ZI‘o0)€=N
`2099'06ree0Shee'0-2068'0-S}S0'}-2068'0-ByeE'0-
`
`
`
`
`
`0819°00819°00819°0-
`
`
`
`
`
`
`
`
`
`
` (cO0ZE‘008%‘oOPZ‘000%‘0091‘oOZL‘008‘OF‘o0)
`
`
`
`
`
`
`
`
`
`
`
`
`
`(oGL€‘oOLZ‘oG7Z‘oO8L‘oGEL‘o06‘oGP‘o0)S=
`
`8Old
`
`
`
`yluseseud
`
`
` CIO74ptg<<dD74andVKN
`
`1=¥
`
`
`
`0819°0-
`
`2098'0
`
`
`SISO’g
`0y
`=FJie
`(
`_dOue]=
`5OL=N
`(DAF)
`
`gy
`
`
`
`0010
`
`0010
`
`
`

`

`Patent Application Publication
`
`Apr. 25, 2013 Sheet 10 of 10
`
`US 2013/0101176 Al
`
`S|
`
`se
`
`54
`
`FIG. 9
`
`INTENSIVELY PROJECT N DIFFERENT
`PROJECTION LIGHT BEAMS TO SUBJECT
`
`MODULATE N REFLECTION LIGHT
`BEAMS REFLECTED FROM SUBJECT
`
`GENERATE N SUB-IMAGES BY CAPTURING N
`MODULATED REFLECTION LIGHT BEAMS
`
`PHASES OF PROJECTION LIGHT BEAMS FROM MEMORY
`
`READ WEIGHTING FACTORS CORRESPONDING
`TO NUMBER OF USED PROJECTION LIGHT BEAMS,
`INTENSITIES OF PROJECTION LIGHT BEAMS, AND
`
`CALCULATE FIRST AVERAGE IMAGE V BY MULTIPLYING N
`SUB-IMAGES BY FIRST WEIGHTING FACTORS
`
`CALCULATE SECOND AVERAGE IMAGE U BY MULTIPLYING
`N SUB-IMAGES BY SECOND WEIGHTING FACTORS
`
`CALCULATE DEPTH INFORMATION FROM FIRST
`AVERAGE IMAGE V AND SECOND AVERAGE IMAGE U
`
`0011
`
`0011
`
`

`

`US 2013/0101176 Al
`
`Apr. 25, 2013
`
`3D IMAGE ACQUISITION APPARATUS AND
`METHOD OF CALCULATING DEPTH
`INFORMATIONIN THE 3D IMAGE
`ACQUISITION APPARATUS
`
`CROSS-REFERENCE TO RELATED
`APPLICATIONS
`
`BACKGROUND
`
`1. Field
`[0002]
`[0003] The present disclosure relates to 3-dimensional
`(3D) image acquisition apparatuses and methodsofcalculat-
`ing depth information in the 3D image acquisition appara-
`tuses.
`
`[0011] Additional aspects will be set forth in part in the
`description which followsand, in part, will be apparent from
`the description, or may be learned by practice of the exem-
`plary embodiments.
`[0012] According to an aspect of an exemplary embodi-
`ment, a 3-dimensional (D) image acquisition apparatus
`includes: an optical modulator for modulating light reflected
`from a subject by sequentially projected N (N is 3 oralarger
`[0001] This application claimspriority from Korean Patent
`natural number) light beams; an image sensorfor generating
`Application No. 10-2011-0109431, filed on Oct. 25, 2011, in
`N sub-imagesby capturing the light modulated bythe optical
`the Korean Intellectual Property Office, the disclosures of
`modulator; and a signal processorfor calculating depth infor-
`which are incorporated herein in their entirety by reference.
`mation regarding a distance to the subject by using the N
`sub-images.
`[0013] The N light beams may be discontinuously pro-
`jected.
`[0014] The N projected light beams maybe different from
`each other and be emitted by one or morelight sources.
`[0015] The one or more light sources may sequentially
`project the N light beams with a predeterminedtimeinterval.
`[0016] An operating time of the optical modulator may be
`synchronized with a projecting time of each of the N light
`beams.
`
`2. Description of the Related Art
`[0004]
`[0005] Recently, the importance of 3-dimensional GD)
`contentis increasing with the developmentandthe increase in
`demandof 3D display devices for displaying images having
`[0017] The operating time of the optical modulator may be
`depth perception. Accordingly, there is research being con-
`shorter than the projecting time.
`ducted into 3D image acquisition apparatuses, such as a 3D
`[0018] An exposure time of the image sensor may be syn-
`camera by which a user personally creates 3D content. Such
`chronized with the operating time of the optical modulator.
`a 3D camera acquires depth information in addition to exist-
`[0019] The image sensor may be exposed during the light-
`ing 2D color image information in one capture.
`projecting time to capture the modulated light and may form
`[0006] Depth information regarding distances between sur-
`the N sub-imagesduringat least a portion ofa remaining time
`faces of a subject and a 3D camera may be acquired using a
`of the light-projecting time.
`stereo vision method using two camerasor a triangulation
`[0020] All pixels ofthe image sensor maybe exposedto the
`method using structured light and a camera. However, since
`modulated light during the light-projecting time.
`the accuracy of depth information in these methods rapidly
`[0021] TheNlight beams maybe periodic waves having the
`decreases as a distance to a subject increases and these meth-
`sameperiod andat least one selected from the group consist-
`ods depend on a surface state of the subject, it is difficult to
`ing of a different intensity and a different phase.
`acquire accurate depth information.
`[0022] The optical modulator may modulate the reflected
`[0007]
`To improve this problem, a Time-of-Flight (TOF)
`method has been introduced. The TOF methodis a method of
`light with the same modulation signal.
`[0023] TheNlight beams maybethe sameperiodic waves.
`measuring a light beam’s flight time until the light reflected
`from a subject is received by a light-receiving unit after an
`[0024] The optical modulator may modulate the reflected
`illumination light is projected to the subject. Accordingto the
`light with different modulation signals.
`TOF method,light of a predetermined wavelength (e.g., Near
`[0025] A phase difference between any two light beams
`Infrared (NIR) light of 850 nm)is irradiated to a subject by
`projected at adjacent times from amongthe N light beams
`using an illumination optical system including a Light-Emit-
`may be a value obtained by equally dividing 360° by N.
`ting Diode (LED) or a Laser Diode (LD). A light having the
`[0026] The reflected light may includeNreflection light
`same wavelengthis reflected from the subject and is received
`beams obtained by reflecting the N light beams from the
`by a light-receiving unit. Thereafter, a series of processing
`subject.
`processes for calculating depth information are performed.
`[0027] The N sub-images generated by the image sensor
`Various TOF technologies are introduced according to the
`may sequentially one-to-one match the N reflection light
`beams.
`series of processing processes.
`[0008]
`Inthe TOF methoddescribed above, depth informa-
`tion is calculated by assuming an ideal environment without
`noise. However, when a 3D camera is used, ambient light,
`such as illumination in an indoor environmentand sunlight in
`an outdoor environment, always exists in the surroundings.
`The ambientlight is incident to the 3D camera and becomes
`noise in a process of calculating depth information.
`[0009] Accordingly, it is necessary to reduce ambientlight
`causing noise in the process of calculating depth information.
`
`Ifthe N sub-imagesdo not one-to-one match the N
`[0028]
`reflection light beams, the signal processor may convert the N
`sub-images ona line by line basis and sequentially one-to-one
`match the N line-based sub-imageswith the N reflectionlight
`beams.
`
`[0029] The signal processor may generate a first average
`image by averaging the N sub-images multiplied by first
`weighting factors, generate a second average image by aver-
`aging the N sub-images multiplied by second weighting fac-
`tors, and calculate the depth information from the first aver-
`age image and the second average image.
`[0030] The depth information may be calculated from an
`arctangent value ofa ratio of the first average image to the
`second average image.
`
`SUMMARY
`
`Provided are a methodof calculating depth infor-
`[0010]
`mation by reducing captured ambient light and a 3D image
`acquisition apparatus therefor.
`
`0012
`
`0012
`
`

`

`US 2013/0101176 Al
`
`Apr. 25, 2013
`
`FIGS. 3A to 3C illustrate a process of generating N
`[0049]
`different sub-images with one projection light beam and N
`different optical modulation signals, according to an exem-
`plary embodiment;
`[0050]
`FIGS. 4A and 4Bare time graphs when a 3D image
`is captured whena duty rate of projection light is 100% and a
`case where a duty rate ofprojection light is 20%,respectively,
`according to an exemplary embodiment;
`[0051]
`FIG. 5is a time graph of when an imageis captured
`by synchronizing a light source, an optical modulator, and an
`image pickup device with each other, according to an exem-
`plary embodiment;
`[0052]
`FIG. 6 is a time graph when an imageis captured
`when notall pixels of an image pickup device are exposed
`during a single operating time of an optical modulator;
`[0053]
`FIG. 71is a schematic diagram for describing a pro-
`cess of calculating depth information from N different
`images, according to an exemplary embodiment;
`[0054]
`FIG. 8isa table illustrating weighting factors A, and
`B,, according to an exemplary embodiment; and
`[0055]
`FIG. 9 is a flowchart illustrating a methodofcalcu-
`lating depth information, according to an exemplary embodi-
`ment.
`
`[0031] According to another aspect of an exemplary
`embodiment, a method of calculating depth information
`includes: modulating light reflected from a subject by sequen-
`tially projecting N (N is 3 or a larger natural number) light
`beams; generating N sub-images by capturing the light modu-
`lated by the optical modulator; and calculating depth infor-
`mation regarding a distance to the subject by using the N
`sub-images.
`[0032] The N light beams may be discontinuously pro-
`jected.
`[0033] The N projected light beams maybe different from
`each other and be emitted by one or morelight sources.
`[0034] The N light beams may be sequentially projected
`with a predetermined time interval.
`[0035] An operating time of an optical modulator for
`modulating the light may be synchronized with a projecting
`time of each of the N light beams.
`[0036] The operating time of the optical modulator may be
`shorter than the projecting time.
`[0037] An exposure time of an image sensorfor capturing
`the light may be synchronized with the operating time of the
`optical modulator.
`[0038] All pixels ofthe image sensor maybe exposedto the
`modulated light during the light-projecting time.
`DETAILED DESCRIPTION
`[0039] TheNlight beams maybe periodic waves having the
`sameperiod andat least one selected from the group consist-
`[0056] Reference will now be madein detail to exemplary
`ing of a different intensity and a different phase, and the
`embodiments, examples of which are illustrated in the
`reflected light may be modulated with the same modulation
`accompanying drawings. In the drawings, the widths and
`signal.
`thicknesses of layers and regions are exaggerated for the
`[0040] The N light beams maybe the sameperiodic waves,
`clarity of the specification. In the description, like reference
`and thereflected light may be modulated with different modu-
`numerals refer to like elements throughout. Expressions such
`lation signals.
`as “at least one of,’ when precedinga list of elements, modify
`the entire list of elements and do not modify the individual
`[0041] A phase difference between any two light beams
`elements ofthe list.
`projected at adjacent times from among the N light beams
`FIG. 1 is a schematic diagram of a 3-dimensional
`[0057]
`maybe a value obtained by equally dividing 360° by N.
`(3D) image acquisition apparatus 100 according to an exem-
`[0042] The generated N sub-images may sequentially one-
`plary embodiment. Referring to FIG. 1, the 3D image acqui-
`to-one match the N reflection light beams.
`sition apparatus 100 may include a light source 101 for gen-
`[0043] The method mayfurther include, if the N sub-im-
`erating light having a predetermined wavelength, an optical
`ages do not one-to-one match the N reflection light beams,
`modulator 103 for modulating light reflected from a subject
`converting the N sub-images on a line by line basis and
`200, an image pickup device 105 (e.g., an image sensor) for
`sequentially one-to-one matching the N line-based sub-im-
`generating a sub-image from the modulated light, a signal
`ages with the N reflection light beams.
`processor 106 for calculating depth information based on a
`[0044]Afirst average image may be generated by averag-
`sub-image formed by the image pickup device 105 and gen-
`ing the N sub-images multiplied by first weighting factors, a
`erating an image including the depth information, and a con-
`second average image may be generated by averaging the N
`troller 107 for controlling operationsof the light source 101,
`sub-images multiplied by second weighting factors, and the
`the optical modulator 103, the image pickup device 105, and
`depth information may be calculated from thefirst average
`the signal processor 106.
`image and the second average image.
`[0058]
`In addition, the 3D image acquisition apparatus 100
`[0045] The depth information maybe calculated from an
`mayfurther include, in front of a light-incident face of the
`arctangent value of a ratio of the first average image to the
`optical modulator 103, a filter 108 for transmitting only light
`second average image.
`having a predetermined wavelength from among the light
`BRIEF DESCRIPTION OF THE DRAWINGS
`reflected from the subject 200 andafirst lens 109 for concen-
`trating the reflected light within an area of the optical modu-
`lator 103, and a second lens 110 for concentrating the modu-
`lated light within an area of the image pickup device 105
`between the optical modulator 103 and the image pickup
`device 105.
`
`[0046] These and/or other aspects will become apparent
`and more readily appreciated from the following description
`of the embodiments, taken in conjunction with the accompa-
`nying drawings in which:
`[0047]
`FIG. 1 is a schematic diagram of a 3-dimensional
`(3D) image acquisition apparatus according to an exemplary
`embodiment;
`[0048]
`FIGS. 2A to 2C illustrate a process of generating N
`different sub-images by modulating N different reflection
`light beams, according to an exemplary embodiment;
`
`[0059] The light source 101 may be for example a Light-
`Emitting Diode (LED) or a Laser Diode (LD) capable of
`emitting light having a Near Infrared (NIR) wavelength of
`about 850 nm that is invisible to human eyes for safety.
`However, the light source 101 is not limited to a wavelength
`bandor type.
`
`0013
`
`0013
`
`

`

`US 2013/0101176 Al
`
`Apr. 25, 2013
`
`period and waveform to the subject 200 under control of the
`controller 107, wherein N maybe3 ora larger natural num-
`ber. The light source 101 may sequentially project the N
`different light beams continuously or within a predetermined
`time interval.
`
`[0060] Light projected from the light source 101 to the
`subject 200 may have a form of a periodic continuous func-
`tion having a predetermined period. For example, the pro-
`jected light may have a specifically defined waveform such as
`a sine wave, a ramp wave, or a square wave, or an undefined
`general waveform. In addition, the light source 101 may
`For example, when 4 different projection light
`[0068]
`intensively project light to the subject 200 for only a prede-
`beamsare used, the light source 101 may generate and project
`termined time in a periodic manner under control of the
`a first projection light beam to the subject 200 for a time T1,
`controller 107. A timethat light is projected to the subject 200
`a second projection light beam to the subject 200 for a time
`is called a light-projecting time.
`T2, a third projection light beam to the subject 200 for a time
`[0061] The optical modulator 103 modulateslightreflected
`T3, andafourth projection light beam to the subject 200 for a
`from the subject 200 under control of the controller 107. For
`time T4. Thesefirst to fourth projection light beams sequen-
`example, the optical modulator 103 may modulate the inten-
`tially projected to the subject 200 may have a form of a
`sity of the reflected light by changing a gain in response to an
`continuousfunction having a predeterminedperiod, such as a
`optical modulation signal having a predetermined wave-
`sine wave. For example, the first to fourth projection light
`length. To do this, the optical modulator 103 may have a
`beams maybe periodic waves having the same period and
`variable gain.
`waveform and different intensities or phases.
`[0062] The optical modulator 103 may operate at a high
`[0069] When the N different light beams are projected, a
`modulation frequency of tens to hundreds of MHzto identify
`phase difference between any twoof the light beams pro-
`a phase differenceor a traveling time of light according to a
`jected at the same time may be 360°/N,andthe period of each
`distance. The optical modulator 103 satisfying this condition
`projected light beam maybeshorter than the operating time of
`maybe at least one of a sub-image intensifier including a
`the light source 101. All ofthe N different light beams may be
`Multi-Channel Plate (MCP), a solid optical modulatorof the
`sequentially projected to the subject 200 within the operating
`GaAsseries, or a thin-type optical modulator using an elec-
`time ofthe light source 101.
`tro-optic material. Although the optical modulator 103 is a
`[0070] A light beam projectedto the subject 200 is reflected
`transmission-type optical modulator in FIG.1, a reflection-
`on the surface of the subject 200 and incidentto thefirst lens
`type optical modulator mayalso be used.
`109. In general, the subject 200 has a plurality of surfaces
`[0063] Like the light source 101, the optical modulator 103
`having different distances, i.e., depths, from the 3D image
`may also operate for a predetermined time to modulate the
`acquisition apparatus 100. FIG.1 illustrates the subject 200
`light reflected from the subject 200. A time that the optical
`having 5 surfaces P1 to P5 having different depths for sim-
`modulator 103 operates to modulate light is called an operat-
`plification of description. When the projected light beam is
`ing time of the optical modulator 103. The light-projecting
`reflected from the 5 surfaces P1 to P5 having different depths,
`time of the light source 101 may be synchronized with the
`5 differently time-delayed (i.e., different phases) reflection
`operating time of the optical modulator 103. Thus, the oper-
`light beamsare generated.
`ating timeofthe optical modulator 103 may be the same as or
`[0071]
`For example, 5 first reflection light beams having
`shorter than the light-projecting time of the light source 101.
`different phases are generated whena first projection light
`[0064] The image pickup device 105 generates a sub-image
`beam is reflected from the 5 surfaces P1 to P5 of the subject
`by detecting the reflected light modulated by the optical
`200, and 5 second reflection light beams having different
`modulator 103 under control of the controller 107. If only a
`phases are generated when a secondprojection light beam is
`distance to any one point on the subject 200 is to be measured,
`reflected from the 5 surfaces P1 to P5 of the subject 200.
`the image pickup device 105 may use a single optical sensor
`Likewise, 5xN reflection light beams having different phases
`such as, for example, a photodiodeor an integrator. However,
`are generated when an Nth projection light beam is reflected
`if distances to a plurality ofpoints on the subject 200 are to be
`from the 5 surfaces P1 to P5 of the subject 200. A reflection
`measured, the image pickup device 105 may havea plurality
`light beam reflected from the surface P1 that is farthest from
`of photodiodes or a 2D or 1D array of other optical detectors.
`the 3D image acquisition apparatus 100 mayarrive atthe first
`For example, the image pickup device 105 may include a
`lens 109 witha phase delay of®,,, anda reflection light beam
`reflected from the surface P5 that is nearest from the 3D
`Charge-Coupled Device (CCD) image sensor or a Compli-
`mentary Metal-Oxide Semiconductor (CMOS) imagesensor.
`image acquisition apparatus 100 mayarrive at the first lens
`The image pickup device 105 may generate a single sub-
`109 with a phase delay of ®,, that is less than ®,,.
`imageperreflected light beam.
`[0072]
`Thefirst lens 109 focusesthe reflection light within
`[0065] The signal processor 106 calculates depth informa-
`an area of the optical modulator 103. The filter 108 for trans-
`tion based on a sub-image formedby the image pickup device
`mitting only light having a predetermined wavelength may be
`105 and generates a 3D image including the depth informa-
`disposed betweenthefirst lens 109 and the optical modulator
`tion. The signal processor 106 may be implemented by, for
`103 to remove ambientlight, such as backgroundlight, except
`example, an exclusive Integrated Circuit (IC) or software
`for the predetermined wavelength. For example, when the
`installed in the 3D image acquisition apparatus 100. When the
`light source 101 emits light having an NIR wavelength of
`signal processor 106 is implemented by software, the signal
`about 850 nm,thefilter 108 may be an NIR bandpassfilter for
`processor 106 may be stored in a separate portable storage
`transmitting an NIR wavelength bandof about 850 nm. Thus,
`medium.
`although light incident to the optical modulator 103 may be
`mostly light emitted from the light source 101 andreflected
`from the subject 200, ambient light is also included therein.
`Although FIG.1 showsthatthe filter 108 is disposed between
`the first lens 109 and the optical modulator 103, positions of
`the first lens 109 and the filter 108 may be exchanged. For
`
`[0066] Hereinafter, an operation of the 3D image acquisi-
`tion apparatus 100 having the above-described structure is
`described.
`
`[0067] The light source 101 sequentially and intensively
`projects N different light beams having a predetermined
`
`0014
`
`0014
`
`

`

`US 2013/0101176 Al
`
`Apr. 25, 2013
`
`example, NIR lightfirst passing through thefilter 108 may be
`focused on the optical modulator 103 bythefirst lens 109.
`[0073] The optical modulator 103 modulates the reflection
`light into an optical modulation signal having a predeter-
`mined wavelength. For convenience of description,
`it
`is
`assumed that the 5 surfaces P1 to P5 of the subject 200
`correspond to pixels divided in 5 areas of the image pickup
`device 105. A period of a gain wavelength of the optical
`modulator 103 may be the sameas a period of a projection
`light wavelength. In FIG. 1, the optical modulator 103 may
`modulate the 5 first reflection light beamsreflected from the
`5 surfaces P1 to P5 of the subject 200 and provide the modu-
`lated light beams to the image pickup device 105 and, in
`succession, may sequentially modulate the 5 secondreflec-
`tion light beams into the 5xN reflection light beams and
`provide the modulated light beamsto the image pickup device
`105. The intensity ofthe reflection light may be modulated by
`an amount obtained by multiplying it by an optical modula-
`tion signal when thereflection light passes throughthe optical
`modulator 103. A period ofthe optical modulation signal may
`be the sameasthat of the projectionlight.
`[0074] The intensity-modulated light output from the opti-
`cal modulator 103 is multiplication-adjusted and refocused
`by the second lens 110 andarrives at the image pickup device
`105. Thus, the modulated light is concentrated within the area
`of the image pickup device 105 by the second lens 110. The
`image pickup device 105 may generate sub-images by receiv-
`ing the modulated light for a predetermined time through
`synchronization with the light source 101 and the optical
`modulator 103. A time that the image pickup device 105 is
`exposedto receive the modulatedlight is an exposure time of
`the image pickup device 105.
`[0075] A method of generating N sub-images from N
`reflection light beams will now bedescribed.
`[0076]
`FIGS. 2A to 2D illustrate a process of generating N
`different sub-images by modulating N different reflection
`light beams, according to an exemplary embodiment.
`[0077] As shown in FIG. 2A, the image pickup device 105
`generates a first sub-image by receiving, for a predetermined
`exposuretime,5 first reflection light beams modulated after
`being reflected from the 5 surfaces P1 to P5 ofthe subject 200.
`Thereafter, as shown in FIG. 2B, the image pickup device 105
`generates a second sub-imagebyreceiving, for the predeter-
`mined exposure time, 5 secondreflection light beams modu-
`lated after being reflected from the 5 surfaces P1 to P5 of the
`subject 200. After repeating these procedures, as shown in
`FIG. 2C, the image pickup device 105 finally generates an
`Nth sub-imageby receiving, for the predetermined exposure
`time, 5xN reflection light beams modulated after being
`reflected from the 5 surfaces P1 to P5 of the subject 200. In
`this manner, the N different sub-images may be sequentially
`obtained as shown in FIG. 2D.
`
`to Nth sub-images may be sub-frame
`[0078] The first
`images for generating a single frame of an image. For
`example, assuming that a period of a single frame is Td, an
`exposure timeof the image pickup device 105 to obtain each
`of the first to Nth sub-images may be about Td/N.
`[0079]
`In FIGS. 2A to 2D, a case of generating N different
`sub-imagesby using N different projection light beams and N
`different reflection light beams has been described. However,
`it is also possible that the samereflection light beam is used
`for all sub-images and the optical modulator 103 modulates a
`reflection light beam for each of the sub-images with a dif-
`ferent gain waveform.
`
`FIGS. 3A to 3D illustrate a process of generating N
`[0080]
`different sub-images with one sameprojection light beam and
`N different optical modulation signals, according to an exem-
`plary embodiment. Referring to FIG.3, reflection light beams
`generated byreflecting the projection light beam from the
`subject 200 have the same waveform and phase forall sub-
`images. As described above, reflection light beams for each
`sub-image havedifferent phase delays ®,, to ®,, according
`to the surfaces P1 to P5 ofthe subject 200. As shown in FIGS.
`3A to 3C,the optical modulator 103 modulates 5 first reflec-
`tion light beams by using a first optical modulation signal,
`modulates 5 secondreflection light beams by using a second
`optical modulation signal different from the first optical
`modulation signal, and modulates 5xNreflection light beams
`by using an Nth optical modulation signal different from any
`other optical modulation signal. Here, the first to Nth optical
`modulation signals may have waveforms totally different
`from each otheror have the sameperiod and waveform except
`for their phases. Accordingly, as shown in FIG. 3D,the image
`pickup device 105 may obtain N first to Nth sub-images that
`are different from each other.
`
`[0081] Hereinafter, a method of generating sub-images by
`using signal waveformsis described.
`[0082]
`For convenience of description, an embodimentin
`which the light source 101 projects N different projection
`light beamsto the subject 200 and the optical modulator 103
`uses a single same optical modulation signal is described as
`an example. However, the theoretical description below may
`be equally applied to a case where one same projection light
`beam and N different optical modulation signals are used. In
`addition, since a methodof calculating depth information is
`equally applied to each pixel even for a case where a sub-
`image formed by the image pickup device 105 is a 2D array
`sub-image, only a method applied to a single pixel
`is
`described. However, when depth information is calculated
`from a plurality of pixels in a 2D array sub-imageat the same
`time, a computation amount may be reduced by omitting a
`portion to be repetitively processed by efficiently processing
`data management and memory allocation.
`[0083]
`First, a waveform P, of general projection light hav-
`ing a period T, may be expressed by Equations 1-1 and 1-2.
`
`P9)(7) = asin(wr — O)) + Pave
`
`mm
`—
`PIG) = » {alsin(keot) + BMcos(ket)} + Pave
`k=l
`
`(1-1)
`
`1-2
`(1-2)
`
`[0084] Here, s denotes an identifier for identifyingfirst to
`Nth projection light beamsthatare different from each other.
`For example, when N projection light beamsare used, s=1, 2,
`...