ONBEING`UNDIGITAL'WITHDIGITALCAMERAS:
`EXTENDINGDYNAMICRANGEBYCOMBINING
`DIFFERENTLYEXPOSEDPICTURES
`S.MannandR.W.Picard
`MassachusettsInstituteofTechnology,E -  ,AmesStreet,Cambridge,MA
`CorrespondingauthorcurrentlywithUniversityofToronto
`Contactinfo:Tel=( ) - ;Fax=( )  - 
`steve@media.mit.edu
`http://n nlf- .eecg.toronto.edu
`http://eecg.toronto.edu/~mann
`http://www.wearcam.org
` .INTRODUCTION
`BibTeXinfo(originalappearances):
` . .Advantageofbeing`undigital'
`@techreport{mann b,
`Digitalphotographyallowsustodomanythingswecannotdo
`author="S.MannandR.W.Picard",
`withtraditionalanalogphotography.However,beingdigital
`title="Being`undigital'withdigitalcameras:
`isnotdesirableinandofitself{itisdesirableforwhatit
`ExtendingDynamicRangebyCombining
`facilitates(instantfeedback,abilitytorapidlytransmithigh
`DifferentlyExposedPictures",
`qualityanywhereintheworld,easeofmanipulation,etc).
`number="  ",
`Digitalimagingimposescertainlimitationsontheways
`institution="M.I.T.MediaLabPerceptual
`wethinkaboutimages.
`Ideally,whatwewantisnotbits,
`ComputingSection",
`but,rather,amathematicalorparametricrepresentationof
`address="Boston,Massachusetts",
`thecontinuousunderlyingintensityvariationsprojectedonto
`year=" ",
`animageplane,representedinaformthatallowsforeasy
`note="Alsoappears,{IS\&T}'sthannualconference,
`transmission,storage,andanalysis.
`Cambridge,Massachusetts,May ",
`Asthespatialresolutionofdigitalimageshasimproved
`pages="-",
`overtheyears,weareapproachingalevelwheretheimage
`}
`mayberegardedasessentiallycontinuous{itisessentially
`freeofpixels.Thushighresolutiondigital
`imagesgiveus
`ABSTRACT
`thespatialcontinuityofanalogphotography,togetherwith
`theabilitytoviewpicturesrightaway,transmitthemover
`Mosteverydaysceneshaveafargreaterdynamicrangethan
`wirelesslinks,analyzethemcomputationally,etc.
`canberecordedonaphotographic(cid:12)lmorelectronicimaging
`However,whiletheremaybesomanypixelsthatwecan,
`apparatus(whetheritbeadigitalstillcamera,video,etc.).
`forallpracticalpurposes,assumetheimageisafunctionof
`However,asetofpictures,thatareidenticalexceptfortheir
`tworealcoordinates,eachofthesepixelsarestillrepresented
`exposure,collectivelyshowusmuchmoredynamicrangethan
`asanarrayofintegersthatcanassumeonlydi(cid:11)erent
`anysinglepicture.Thedarkpicturesshowushighlightdetails
`values,foreachcolorchannel.So-calledbitcolor,also
`ofthescenethatwouldbewashedoutina\properlyexposed"
`knownasfullcolor,truecolordirectvisual,etc.,isnotas
`picture,whilethelightpicturesshowussomeshadowdetail
`\full"or\true"asthesenamesimply.
`Inparticular,these
`thatwouldalsonotappearina\properlyexposed"picture.
`imagesarealsotypicallymanipulatedusing-bitprecision
`Weproposeameansofcombiningdi(cid:11)erentlyexposedpic-
`arithmetic.Anysimplemanipulationsinanimageediting
`turestoobtainasinglepictureofextendeddynamicrange,
`program,suchasPhotoshop,quicklydegradethequalityof
`andimprovedcolor(cid:12)delity.Givenasetofdigitalpictures,
`theimages,introducinggapsinthehistogramsthatgrowwith
`wemayproduceasinglepicturewhichis,forallpractical
`eachsuccessivecomputation.
`purposes,`undigital',inthesensethatitisa(cid:13)oatingpoint
`Thepurposeofthispaperistoexaminetherecoveryof
`image,withthekindofdynamicrangeweareaccustomedto
`the`trueimage',areal-valuedquantityoflightprojectedonto
`seeingintypical(cid:13)oatingpointrepresentations,asopposedto
`a(cid:13)atsurface.Weregardthe`trueimage'asacollectionof
`theintegerimagesfromwhichitwasgenerated.
`analogphotometricquantitiesthatmighthavebeenmea-
`Themethodiscompletelyautomatic;itrequiresnohu-
`suredwithanarrayoflinearizedlightmetershaving(cid:13)oating-
`manintervention,anditrequiresnoknowledgeofthere-
`pointprecision,andthus,beingessentially,forallpractical
`sponsefunctionoftheimagingdevice.Itworksreliablywith
`purposes,`undigital'.
`imagesfromadigitalcameraofunknownresponse,orfrom
`Ofcourse,allimagesthatarestoredonacomputerare
`ascannerwithunknownresponse,scanninganunknown(cid:12)lm
`digital.A(cid:13)oatingpointnumberisdigital.Butadouble-
`type.
`
`

`

`precision(bit)(cid:13)oatingpointnumberisclosetoanalogin
`spiritandintent.
`Withthegrowingwordsizeofdesktopcomputational
`hardware,(cid:13)oatingpointarithmeticisbecomingmoreprac-
`ticalforlargeimages.ThenewDEC (Alpha)computer
`hasawordsizeofbits,andcaneasilyhandleimagesas
`doubleprecisionarrays.Doubleprecisionisnothingnew.
`Foryears,languageslikeFORTRANhavesupported(cid:13)oating
`pointarithmetic,usedwidelybythescienti(cid:12)ccommunity,but
`(cid:13)oatingpointcalculationsarenotsupportedinanyofthe
`popularimagemanipulationsoftwaresuchasPhotoshopor
`Livepicture.Capturingimagesthatareessentiallyunlimited
`indynamicrange,and,whiledigitallyrepresented,behaveas
`analogimages,allowsustocaptureandsurpassthebene(cid:12)ts
`traditionallyo(cid:11)eredbytrulyanalogimageformatslike(cid:12)lm.
`.WHATISACAMERA
`Weregardanimageasacollectionofphotometricmeasure-
`ments,andacameraasanarrayoflightmeters.However,
`intraditionalimaging,eachofthesemeasurements(pixels)
`aremadewithalightmeter(sensorelement)thathassome
`unknownnonlinearityfollowedbyaquantizationtoamea-
`surementhaving-bitprecision.
`. .Dynamicrangeandamplituderesolution
`Manyeverydayscenescontainatremendousdynamicrange.
`Forexample,thescenemightbeadimlylitroom,witha
`windowinthebackground;throughthewindowwemight
`observeabeautifulbluesummerskywithpu(cid:11)ywhiteclouds.
`Yetapicturethatisexposedfortheindoorscenewillrender
`thewindowasawhiteblob,bloomingoutintotheroom,
`wherewecanscarcelydiscerntheshapeofthewindow,let
`alone,seebeyondit.Ofcourse,ifweexposedforthesky
`outside,theinteriorwouldappearcompletelyblack.
`Cameras(whetheranalogordigital)tendtohaveavery
`limiteddynamicrange.Itispossibletoextendthedynamic
`rangebyvariousmeans.Forexample,inthecaseofphoto-
`graphicemulsion,the(cid:12)lmcanbemadethicker,butthereare
`tradeo(cid:11)s(e.g.thickeremulsionresultsinincreasedscatter-
`ing,whichresultsindecreasedspatialresolution).Nyquist
`showedhowasignalcanbereconstructedfromasamplingof
`(cid:12)niteresolutioninthedomain(e.g.spaceortime),butas-
`sumedin(cid:12)nitedynamicrange.Ontheotherhand,ifwehave
`in(cid:12)nitespatialresolution,butlimiteddynamicrange(evenif
`wehaveonly bitofimagedepth),CurtisandOppenheim[ ]
`showedthatwecanalsoobtainperfectreconstruction.This
`tradeo(cid:11)betweenimageresolution,andimagedepthisalsoat
`workinaslightlydi(cid:11)erentwayinimagehalftoning.
`Beforethedaysofdigitalimageprocessing,CharlesWyck-
`o(cid:11)formulatedamultiplelayerphotographicemulsion[][ ].
`TheWycko(cid:11)(cid:12)lmhadthreelayersthatwereidenticalintheir
`spectralsensitivities(eachwasroughlyequallysensitivetoall
`wavelengthsoflight),anddi(cid:11)eredonlyintheiroverallsensi-
`tivitiestolight(e.g.thebottomlayerwasveryslow,withan
`ISOratingof,whilethetoplayerwasveryfastwithanISO
`ratingof).
`ApicturetakenonWycko(cid:11)(cid:12)lmcanbothrecordady-
`namicrangeofonetoahundredmillionandcaptureverysub-
`tledi(cid:11)erencesinexposure.Furthermore,theWycko(cid:11)picture
`hasverygoodspatialresolution,andthusappearstoover-
`cometheresolution/depthtradeo(cid:11),byusingdi(cid:11)erentcolor
`
`dyesineachlayer,whichhaveaspeculardensityasopposed
`thedi(cid:11)usedensityofsilver.Wycko(cid:11)printedhisgreyscale
`picturesoncolorpaper,sothefast(yellow)layerwouldprint
`blue,themedium(magenta)layerwouldprintgreen,andthe
`slow(cyan)layerwouldprintred.Hisresultwasapseudo-
`colorimagesimilartothoseusednowindatavisualization
`systemstodisplay(cid:13)oatingpointarraysonacomputerscreen
`oflimiteddynamicrange.
`Wycko(cid:11)'smostwell-knownpicturesareperhapshismo-
`tionpicturesofnuclearexplosions{onecouldclearlyseethe
`faintglowofabombjustbeforeitexploded(whichwould
`appearasblue,sinceitonlyexposedthefasttoplayer),as
`wellasthedetailsinthehighlightsoftheexplosion(which
`appearedwhitesincetheyexposedall layers{thedetails
`discernableprimarilyonaccountoftheslowbottomlayer).
`..Combiningmultiplepicturesofthesamescene
`Theideaofcomputationallycombiningdi(cid:11)erentlyexposed
`picturesofthesamescenetoobtainextendeddynamicrange
`hasbeenrecentlyproposed[],wheretheimageswereas-
`sumedtohavebeentakenfromroughlythesameposition
`inspace,withpossiblydi(cid:11)erentcameraorientations(pan,
`tilt,rotationaboutopticalaxis),anddi(cid:11)erentzoomsettings.
`Inthispaperwedescribe,infurtherdetail,thecomputa-
`tionalmeansofcombiningdi(cid:11)erentlyexposedpicturesintoa
`(cid:13)oating-pointimagearray,andassumeasimplercase,namely
`thatallpicturesaretakenfromacameraata(cid:12)xedlocation
`inspaceanda(cid:12)xedorientation,witha(cid:12)xedfocallengthlens.
`Thissimplercasecorrespondstopicturesthatdi(cid:11)eronlyin
`exposure.Werefertoacollectionofpicturesthatdi(cid:11)eronlyinexpo-
`sureasaWycko(cid:11)set,inhonorofCharlesWycko(cid:11),whowas
`the(cid:12)rsttoexploitsuchasetofpicturescollectively.Pho-
`tographers,throughaprocedurecalledexposurebracketing
`(tryingavarietyofexposuresettingsandlaterselectingthe
`oneexposurethattheymostprefer)alsoproduceWycko(cid:11)
`setsbutgenerallywiththeintentoflatermerelyselectingthe
`bestimagefromtheset,withoutexploitingthefullpotential
`valueofusingtheimagescollectively.
` .EXPOSUREBRACKETINGOFDIGITAL
`IMAGES
`Wheneverthedynamicrangeofthesceneexceedstherangeof
`therecordingmedium(whichisalmostalways)photographers
`tendtoexposeforareasofinterestinthescene.Forexample,
`ascenecontainingpeopleisusuallyexposedtoshowthemost
`detail
`inthem(Fig. )attheexpenseofdetailselsewhere
`inthescene.Additionally,inourcase,apicturewastaken
`immediatelyafterward(Fig.),withfourtimestheexposure
`time,sothatthesurroundingcontextualdetailsofthescene
`wouldshowupnicely.
`Ideally,onlyonepicturewouldbeneededtocapturethe
`entiredynamicrangeofthescene,andwewouldn'teven
`needtoworryaboutwhetherthepicturewasoverexposed
`orunderexposedbecausewecouldlightenordarkenitlater
`on,bysimplyusingtheappropriate`lookupoperator'.By
``lookupoperator',wemeananyspatiallyinvariantnonlinear-
`ity:g(x;y)=g(f(x;y)).A`lookupoperator'isthecontinu-
`ousanalogofalookuptable.Gammacorrectionisanexample
`ofa`lookupoperator'.
`However,duetovariousnoisesources,suchasquantiza-
`tionnoise,a`lookupoperator'willonlybeabletocompensate
`
`

`

`Figure :TheMannfamilystandingoutsideanoldbuilding
`withthecamerainside.Heretheexposurewasselectedso
`thatthepeoplewouldshowupnicely.
`foraverylimitedamountofoverexposureorunderexposure.
`Forexample,wewillneverrecoverthedetailinthefacesof
`thepeoplefromFig..Theincreasedexposurehascaused
`thisinformationtobelostbythecombinede(cid:11)ectofsatu-
`rationandnoise.Similarly,nothingcanbedonetorecover
`theshadowdetailsinthedarkerportionsofFig. ,because
`theseareashavepixelvaluesthatareuniformlyzero.Even
`inslightlybrighterareas,wherethereisvariationinthepix-
`els,thisvariationissubjecttoextremequantizationnoise.
`Forexample,indarkareaswherethepixelvalues(cid:13)uctuate
`betweenzeroandone,thereisonlyonebitofprecision.A
`camerawithasmallnumberofbitsofdepth(suchasaone-
`bitcamera),butwhichhasveryhighspatialresolution,may
`beusedtocaptureacontinuoustoneimage[ ].Indeed,astat
`camera,usedinaphotomechanicaltransfer(PMT)machine,
`isabletocaptureimagesthatappeartobecontinuous-tone
`(duetothehalftoningscreen),eventhoughthe(cid:12)lmcanonly
`recordtwodistinctlevels.Thisispossiblebecausethe(cid:12)lm
`hasessentiallyunlimitedspatialresolution,andisrecording
`throughascreenofmuchlower(e.g.dpi)spatialresolution.
`However,inmostdigitalphotographyandvideoapplica-
`tions,spatialresolutionismuchlowerthanwewouldlike.We
`donothavetheluxuryofessentiallyin(cid:12)nitespatialresolution
`thatPMTsystemshave,andsowearenotatlibertytotrade
`spatialresolutionforimproveddynamicrange.
`Therefore,weproposetheuseofexposurebracketingas
`analternative,wherebywemakethetradeo(cid:11)alongthetime
`axis,exchangingreducedframe-rateforimproveddynamic
`range,ratherthanreducedspatialresolutionforimproved
`dynamicrange.Inparticular,oftenastillimageisallthatis
`desiredfromavideocamera,andinmanyotherdigitalvideo
`applications,allthatisneededisafewframespersecond,
`fromacameracapableofproducing framespersecondor
`more.
`.SELF-CALIBRATINGCAMERA
`Thenumericalquantityappearingatapixelintheimageis
`seldomlinearlyrelated tothequantityoflightfallingonthe
`correspondingsensorelement.Inthecaseofanimagescanned
`from(cid:12)lm,thedensityofthe(cid:12)lmvariesnonlinearlywiththe
`quantityoflighttowhichitisexposed.Furthermore,the
` Infact,quiteoften,photographersdesireanonlinearrelation-
`ship:thenonlinearitiestendtomaketheimagelookbetterwhen
`printedonmediathathavelimiteddynamicrange.
`
`Figure:Theexposurewasincreasedbyafactorofk=,
`comparedtoFig. ;asaresult,theinteriorofthebuildingis
`nicelyvisible.
`scannerwillmostlikelyintroduceafurtherunknownnonlin-
`earity.Weproposeasimplealgorithmfor(cid:12)ndingthepointwise
`nonlinearityoftheentireprocess,f,thatmapsthelightq
`projectedonapointintheimageplanetothepointwisevalue
`inthepicture,f(q),uptoaconstantscalefactor.Weignore,
`untilSection. ,thefactthateachpixelcanonlyassumea
`(cid:12)nitenumberofvalues,thefactthattherearea(cid:12)nitenumber
`ofpixelsintheimage,andthee(cid:11)ectsofimagenoise:
` .Selectarelativelydarkpixelfromimagea,andobserveboth
`itslocation,(x;y),anditsnumericalvalue,f.Wedonot
`knowtheactualquantityoflightthatgaverisetof,butwe
`willcallthisunknownquantityq.Sincefistheresultof
`someunknownmapping,f,appliedtotheunknownquantity
`oflight,q,wedenotea(x;y)byf(q).
`.Locatethecorrespondingpixelinimageb,namelyb(x;y).
`Weknowthatktimesasmuchlightgaverisetob(x;y)as
`toa(x;y).Thereforeb(x;y)=f(kq).Forconvenience,
`wedenoteb(x;y)byf(q ),sothatf(q )=f(kq).Now
`searcharoundinimageaforapixelthathasthenumeri-
`calvaluef(q ),andmakeanoteofthecoordinatesofthe
`foundpixel.Callthesecoordinates(x ;y ),sothatwehave
`a(x ;y )=f(q ).
` .Lookatthesamecoordinatesinimagebandobservethe
`numericalquantityb(x ;y ).Weknowthatktimesas
`muchlightfellonb(x ;y )asdidona(x ;y ).Therefore
`b(x ;y )=f(kq ).Forconvenience,wedenoteb(x ;y )by
`f(q).Sofarwehavethatf(q)=f(kq )=f(kq).Now
`searcharoundinimageaforapixelthathasthenumerical
`valuef(q)andnotethesecoordinates(x;y).
`.Continuinginthisfashion,weobtainthenonlinearityof
`theimagesensoratthepointsf(q),f(kq),f(kq),...,
`f(knq).
`Nowwecanconstructpointsonaplotoff(q)asafunc-
`tionofq,whereqisthequantityoflightmeasuredinarbitrary
`(reference)units.Weillustratethisprocessdiagrammatically
`(Fig (a)),wherewehaveintroducedaplotofthenumeri-
`calvaluesinthe(cid:12)rstimage,a=f(q)againstthenumerical
`valuesinthesecondimage,b=f(kq),whichwecallthe
``range-range'plot,astheaxesareboththerangeoff,with
`aconstantdomainratio,k.Oncethecameraiscalibrated,
`wemayusethecalibrationcurvetocombinesetsofpictures
`liketheonesinFig. and.Thepicturesthatareused
`tocalibratethecameraneednotbethesameonesusedto
`makethecomposite.
`Infact,hadweusedasmallervalue
`forktocalibratethecamera(e.g. .orinsteadof),
`
`

`

`f(q)
`
`f(q )
`0
`
`f(q )
`1
`
`f(q )
`2
`
`1
`
`k
`
`q/q0
`
`k2
`
`(RANGE-RANGE PLOT)
`
`(RESPONSE CURVE)
`
`f(kq)
`
`f(q)
`
`f(kq )
`1
`
`f(kq )
`0
`
`Figure :Procedurefor(cid:12)ndingthepointwisenonlinearityof
`animagesensorfromtwopicturesdi(cid:11)eringonlyintheirexpo-
`sures.(RANGE-RANGEPLOT)Plotofpixelvaluesinone
`imageagainstcorrespondingpixelvaluesintheother,which
`wecallthe`range-range'plot.(RESPONSECURVE)Points
`ontheresponsecurve,foundfromonlythetwopictures,with-
`outanyknowledgeaboutthecharacteristicsoftheimagesen-
`sor.Ifweusealogarithmicexposurescale(asmostphotog-
`raphersdo)thenthesamplesfalluniformlyonthelog(q=q)
`axis.wewouldhaveobtainedmoresamplepointsontheresponse
`curve(Fig (b)).
`Ingeneral,estimatingafunction,f(q),fromagraphof
`f(q)versusf(kq),isadi(cid:14)cultproblem.However,wecan
`placecertainrestrictionsonf.Forexample,wesupposethat
`fissemi-monotonic(increasesorremainsconstantwithin-
`creasingq).Sincetheresponsecurveissemi-monotonic,so
`istheplotdepictedinFig (a).Wecanalsoimposethat
`f()=bytakingapicturewiththelenscapon,andsub-
`tractingtheresultingpixelvaluefromeachofthetwo(or
`more)images.ThisstepwillinsurethattheplotofFig (a)
`passesthroughtheorigin.
`Wemaybewillingtoplaceevenstrongerrestrictionson
`theresponsecurve.Forexample,acommonlyusedempirical
`lawfor(cid:12)lmisf(q)=(cid:11)+(cid:12)q(cid:13).Thisgivesrisetothecanonical
`DlogE(densityversuslogexposure)curvemuchofwhichis
`linear.TheDmin(minimumdensity),(cid:11),wouldbesubtracted
`o(cid:11)assuggested,usingapicturewiththelenscapon,andthe
`(a;b)plotwouldtaketheformb=k(cid:13)a,fromwhichwecould
`(cid:12)ndthe(cid:12)lm'scontrastparameter(cid:13)byapplyingregressionto
`thepointsknownontherange-rangeplot.
`. .Quantizationandothernoise
`Inpractice,thepixelvaluesarequantized,sothattherange-
`rangeplotisreallyastaircasefunction.
`Itisstillsemi-
`monotonic,sinceitisaquantizedversionofacontinuous
`semi-monotonicfunction.
`Inadditiontoquantizatione(cid:11)ects,wealsohavenoise,
`whichmaybeduetoavarietyofcauses,suchasthermalnoise
`intheimagesensor,graininthe(cid:12)lm,slightmisregistration
`oftheimages,orslightchangesincameraposition,scene
`content,andlighting.Weconsidera`jointhistogram'ofthe
`twoimages(Fig.(a)),whichisthediscreteequivalentofthe
`Theonlypracticalsituationthatwouldlikelyviolatethisas-
`sumption,iswhereanegative(cid:12)lmisbeingused,thesunisinthe
`picture,andthesun'sraysareconcentratedonthe(cid:12)lmforasu(cid:14)-
`cientlylongtimetoburnaholethroughanegative(cid:12)lm.Theresult
`isaprintwherethebrightestobjectinthescene(thesun)appears
`black.
`
`Density
`
`Log E
`
`Density
`
`(b)
`(a)
`Figure:CrosshistogramoftheimagesinFigs. and.The
`cross-histogramoftwoimagesisitselfanimage.Sincethetwo
`imageshaveadepthofbits,thecrosshistogramisa(cid:2)
`imageregardlessofthesizesofthetwoimagesfromwhich
`itisobtained.Thebincountattheorigin(lowerleftcorner)
`indicateshowmanypixelswereblack(hadavalueofzero)
`atthesamelocationinbothimages.
`(a)Crosshistogram
`displayedasanimage.Darkerareascorrespondtogreaterbin
`counts.(b)Allnon-emptybinsareshownasblack.Ideally,
`thereshouldonlybeaslender\staircased"curveofnonempty
`bins,butduetonoiseintheimages,thecurvefattens.
`Figure:ResponsecurvesoftheWycko(cid:11)set(notethelog
`scaleasopposedtothescaleofFig (RESPONSECURVE)
`whichwaslinear).(a)Responsecurvescorrespondingtotwo
`di(cid:11)erentexposures,depictedasthoughtheyweretakenon
`twodi(cid:11)erent(cid:12)lms.Thedashedlinemaybethoughtofas
`eitheralongerexposureorthefasterlayerofa-layerWyck-
`o(cid:11)(cid:12)lm,whilethedottedlinemayberegardedasashorter
`exposureortheslowlayerofa-layerWycko(cid:11)(cid:12)lm.
`(b)
`\Certaintyfunctions"calculatedbydi(cid:11)erentiatingthetwo
`responsecurves.(c)Hypotheticalcurvesre-alignedasthey
`willbewhentheimagesarelatercombined.Theresponseof
`theidealcompositeisindicatedbythethinsolidline.Using
`moreexposurebracketing(ormorelayersonaWycko(cid:11)(cid:12)lm),
`wecanextendthisresponseinde(cid:12)nitely.
``range-range'plotofFig .Itisabyarraysinceeach
`pixelofthetwoimagescanassumedistinctvalues.Due
`tonoise,weseeafatridge,ratherthanaslender\staircase".
`Ideallythereshouldbenopointso(cid:11)ofthestaircase,de(cid:12)ned
`byquantizingtherange-rangeplot,butinpracticewe(cid:12)nda
`considerablenumberofsuchnon-emptybins(Fig.(b)).
`.COMBININGIMAGESOFDIFFERENT
`EXPOSURE
`Atthispointwehavefoundtheresponsecurve(by(cid:12)ttingto
`thedataintherange-rangeplot,asinFig),andcanshift
`theresponsecurvetotheleftorrighttogetthecurvesofthe
`twoormoreexposures(Fig.(a)).Intheshadowareas(areas
`oflowexposure,E)thesamequantityoflightinthescene
`hashadamorepronouncede(cid:11)ectonthedashed-exposure,so
`thattheshadowdetailinthescenewillstillbeonaportionof
`thedashedlinethatisrelativelysteep.Thehighlightdetail
`willsaturatethisexposure,butnotthedotted-exposure.
`Ingeneral,forpartsofthe(cid:12)lmthatareexposedinthe
`
`(a)
`Certainty
`
`Log E
`
`Log E
`
`(b)
`
`(c)
`
`

`

`Figure:`Crossoverimage'correspondingtothetwopictures
`inFigs. and.BlackdenotespixellocationswhereFig. is
`themore\certain"ofthetwoimages,andthuswhereFig.
`shouldcontributetothecomposite.Whitedenotespixello-
`cationswhereFig.isthemore\certain"ofthetwoimages,
`andthuswhereitshouldcontributetothecomposite.
`In
`practice,wetakeaweightedsumoftheimagesratherthan
`theabruptswitchoverdepictedinthis(cid:12)gure.
`extremes(greatlyoverexposedorgreatlyunderexposed),de-
`tailislost{wecannolongerdistinguishsmallchangesin
`lightlevelsincetheresultingchangesin(cid:12)lmdensityareso
`smallthattheyfallbelowthenoise(cid:13)oor(e.g.weareoper-
`atingonthe(cid:13)atpartsofFig).Ontheotherhand,steep
`portionsoftheresponsecurvescorrespondtodetailthatcan
`bemoreaccuratelyrecovered,andarethusdesirableoperat-
`ingpoints.Intheseregions,smallchangesinlightwillcause
`largechangesinthemeasuredvalueoftheresponsefunction,
`andevenifthemeasurementsarehighlyquantized(e.g.only
`madewithbitprecision),smalldi(cid:11)erencesinthemeasured
`quantitieswillremaindiscernable.
`Thuswearetemptedtoplotthederivativesofthesehy-
`potheticalresponsecurves(Fig.(c)),whichwecallthecer-
`taintyfunctions.
`At(cid:12)rstglance,onemightbetemptedtomakeacompos-
`itefromtwoormoredi(cid:11)erentlyexposedpicturesbymanually
`combiningthelightregionsfromthedarkerpicturesandthe
`darkregionsfromthelighterpictures(e.g.manuallyselecting
`themiddleofFig. andpastingontopofFig.).However,we
`wishtohavethealgorithmautomaticallycombinetheimages.
`Furthermore,theboundary(Fig.)betweenlightregionsand
`darkregionsis,ingeneral,notasmoothshape,andwouldbe
`di(cid:14)culttotraceoutbyhand.Pastingthisirregularregion
`ofFig. intoFig.,amountstochoosing,ateachpointof
`thecomposite,thesourceimagethathasthehighercertainty
`ofthetwo.However,abruptchangesresultingfromsuddenly
`switchingfromoneimagetoanotherocasionallyintroduce
`unpleasantartifacts,soinstead,wecomputeaweightedaver-
`age.Everypixelofthecomposite,whethershadoworhigh-
`light,orinthetransitionregion,isdrawnfromalloftheinput
`images,byweightingbasedonthecertaintyfunctions.This
`providesagradualtransitionbetweentheimages,wherethe
`shadowdetailcomesprimarilyfromthelighterimage,and
`thehighlightdetailcomesprimarilyfromthedarkerimage.
`Theextended-responseimagearrayfromthetwopictures
`ofFigs. andisa(cid:13)oatingpointarraywhichhasmorethan
`distinctvalues,andthereforecannotbedisplayedona
`conventional-bitdisplaydevice.
`
`Figure:Wycko(cid:11)composite,derivedfromFig. andFig.,
`reducedincontrastandthenquantizedtobitimagedepth.
`.DYNAMICRANGE;`DYNAMICDOMAIN'
`Tekalp,Ozkan,andSezan[],IraniandPeleg[],andMann
`andPicard[]haveproposedmethodsofcombiningmultiple
`picturesthatareidenticalinexposure,butdi(cid:11)erincamera
`position.Theresultisincreasedspatialresolution.When
`oneoftheseimagesistoobigto(cid:12)tonthescreen,welook
`atitthroughasmallmovableviewport,scrollingaroundand
`exploringonepartofthe`imagedomain'atatime.
`Inthispaper,thecompositeimageisa(cid:13)oatingpointarray,
`andisthereforetoodeepforconventionalscreendepthsof
`bits(bitsforeachcolorchannel),soweconstructedaslider
`controltoallowtheusertointeractivelylookatonlypartof
`the`imagerange'atatime.Theuserslidesthecontrolback
`andforthdependingontheareaofinterestinthecomposite
`image.Thiscontrolistoscreenrangeasthescrollingwindow
`istoscreendomain{showingthevasttonalrangeonepiece
`atatime.Ofcoursewewereabletoobtaintheunderexposed
`viewmuchlikeFig ,byslidingthecontrol
`left,andthe
`overexposedviewmuchlikeFigbyslidingthecontrolright.
`Whenanimageistoobigto(cid:12)tonthescreen,onecan
`alsosubsampleitsdomaintomakeit(cid:12)tonthescreen.Anal-
`ogously,weappliedtheappropriaterange-subsampling(quan-
`tizationtobits)toour(cid:13)oating-pointcompositeimagefor
`screendisplay,orprint(Fig).Beforequantization,weap-
`pliedanonlinearitywhichrestoredtheappearanceoftheim-
`agetothefamiliartonalscaletowhichphotographersare
`accustomed,andweaddedtheappropriateamountofnoise
`(dither) .Itisworthmentioningthatthe(cid:12)nalnonlinearity
`beforequantizationselectsthetonalrangeofinterest.We
`canregarditsderivative(the`certaintyfunction')asdepict-
`ingthe`Wycko(cid:11)spectrum'(whichregionsofgreyvalueare
`emphasizedandbyhowmuch)analogoustoaconventional
`bandpass(cid:12)lterwhichselectsthefrequenciesofinterest.The
`elementsofaWycko(cid:11)set,havingequallyspacedcertainty
`functionsofidenticalshape,areanalogoustoabankofcon-
`stantQ(cid:12)lters.
`Ifallthatisdesiredisasingleprint,whynotjusttryto
`formulateasuper-low-contrast(cid:12)lmorimagesensor?Thesu-
`periorityoftheWycko(cid:11)compositeliesintheabilitytocontrol
`theprocessofgoingtothelowcontrastmedium.Forexam-
`ple,wemightapplyahomomorphic[](cid:12)lteringoperationto
` Theditherdidnothaveaperceivablee(cid:11)ectonanbitimage,
`butwhenreducingaWycko(cid:11)compositetobitsorless,thedither
`madeanoticableimprovement.
`
`

`

`.Linearizetheimages(undothenonlinearresponseofeach),
`ifdesired,ormaptheresponsecurvesontoonedesired(cid:12)nal
`responsecurve.
` .Computethecertaintyfunctionbydi(cid:11)erentiatingthere-
`sponsefunction.Thecertaintyfunctionofeachimageis
`foundbyappropriatelyshiftingthisonecertaintyfunction
`alongtheexposureaxis.
`.Computetheweightedsumoftheseimages,weightingby
`thecertaintyfunctions.
`Thecompositemaybeexploredinteractivelyorcontrast-
`reducedandquantized,foraconventionaldisplaydevice.Fur-
`thermore,wecanregardtheWycko(cid:11)(cid:12)lm(orexposurebrack-
`eting)asperformingananalysisbydecomposingthelight
`fallingonthesensorintoits`Wycko(cid:11)layers'.Theproposed
`algorithmprovidesthesynthesistoreconstructa(cid:13)oatingpoint
`imagearraywiththedynamicrangeoftheoriginal
`light
`fallingontheimageplane.Thissuggeststhepossibilityof
`a`Wycko(cid:11)(cid:12)lter'thatcould,forexample,blurthehighlights
`ofanimagewhilesharpeningthemidtonesandshadows.
`Wycko(cid:11)(cid:12)ltersworkinthe`amplitudedomain',incontrastto
`Fourier(cid:12)lterswhichworkinthefrequencydomain,orspatio-
`temporal(cid:12)lterswhichworkinthespaceandtimedomains.
` .REFERENCES
`[ ]S.R.CurtisandA.V.Oppenheim,\Signalreconstructionfrom
`Fouriertransformsigninformation,"TechnicalReportNo.,
`MITResearchLaboratoryofElectronics,May .
`[]C.W.Wycko(cid:11),\Anexperimentalextendedresponse(cid:12)lm,"Tech.
`Rep.NO.B-  ,Edgerton,Germeshausen&Grier,Inc.,Boston,
`Massachusetts,MARCH  .
`[ ]C.W.Wycko(cid:11),\Anexperimentalextendedresponse(cid:12)lm,"
`S.P.I.E.NEWSLETTER,JUNE-JULY .
`[]S.Mann,\Compositingmultiplepicturesofthesamescene,"in
`ProceedingsofthethAnnualIS&TConference,(Cambridge,
`Massachusetts),TheSocietyofImagingScienceandTechnology,
`May -  .
`[]A.Tekalp,M.Ozkan,andM.Sezan,\High-resolutionimagerecon-
`structionfromlower-resolutionimagesequencesandspace-varying
`imagerestoration,"inProc.oftheInt.Conf.onAcoust.,Speech
`andSig.Proc.,(SanFrancisco,CA),pp.III{  ,IEEE,Mar. -
`, .
`[]M.IraniandS.Peleg,\ImprovingResolutionbyImageRegistra-
`tion,"CVGIP,vol. ,pp. { ,May .
`[]S.MannandR.W.Picard,\Virtualbellows:constructinghigh-
`qualityimagesfromvideo,"inProceedingsoftheIEEE(cid:12)rst
`internationalconferenceonimageprocessing,(Austin,Texas),
`Nov. -  .
`[]T.G.Stockham,Jr.,\Imageprocessinginthecontextofavisual
`model,"Proc.IEEE,vol.,pp.{,July .
`[ ]C.Ryals,\Lightspace:Anewlanguageofimaging,"PHOTO
`Electronic
`Imaging,
`vol.
` ,
`no.,
`pp.
` { ,
` .
`http://www.novalink.com/pei/mann.html.
`[ ]S.Mann,\Lightspace."Unpublishedreport(Paperavailablefrom
`author).SubmittedtoSIGGRAPH .Alsoseeexampleimages
`inhttp://wearcam.org/lightspace,July .
`
`the(cid:12)nalcomposite,whichwouldbringoutimproveddetails
`athighspatialfrequencies,whilereducingtheunimportant
`overallchangesindensityatlowspatialfrequencies.
`.WYCKOFFANALYSISANDSYNTHESIS
`FILTERBANKS
`WecanregardtheWycko(cid:11)(cid:12)lm(orexposurebracketing)as
`performingananalysisbydecomposingthelightfallingon
`thesensorintoits`Wycko(cid:11)layers'.Theproposedalgorithm
`providesthesynthesistoreconstructa(cid:13)oatingpointimage
`arraywiththedynamicrangeoftheoriginallightfallingon
`theimageplane.Thisanalysis-synthesisconceptisillustrated
`inFig.Theanalysis-synthesisconceptsuggeststhepossibilityof
`usingtheWycko(cid:11)layerdecompositionasa\Wycko(cid:11)(cid:12)lter"
`thatcould,treattheshadows,midtones,andhighlightsof
`animagedi(cid:11)erently.Forexample,wemightwishtosharpen
`thehighlightsofanimagewithouta(cid:11)ectingthemidtonesand
`shadows.TheWycko(cid:11)(cid:12)lterprovidesanewkindof(cid:12)ltering{`am-
`plitudedomain'(cid:12)ltering{asopposedtotheclassicFourier
`domain,spatialdomain,temporaldomain,andspatiotempo-
`ral(cid:12)lters.WeenvisionageneralizedNyquist-liketheoryfor
`reconstructionfrom`amplitudesamples',toaugmentclassic
`samplingtheory.
`.
``LIGHTSPACE'
`Theconceptpresentedinthispaperispartofalargerframe-
`workcalled`lightspace'[ ],whichisadescriptionoftheway
`ascenerespondstolight.
``Lightspace'isthespaceofall
`possiblephotometricmeasurementstakenforeachpossible
`photometricexcitation.
`RegardinganimageofsizeM(cid:2)Npixelsasapointorvec-
`torinIRMN,allowsustoconsidereachofasetofdi(cid:11)erently
`exposedimages,priortononlinearitiesandquantization,as
`colinearvectorsinIRMN.
`Furthermore,ifweobtainmultiplepicturesofthesame
`scenedi(cid:11)eringonlyinlighting,theyspanasubspaceofIRMN,
`whichwecallthe`lightvectorsubspace'.Fromanysetof
``lightvectors'(picturesofascenetakenwithparticularlight-
`ing)thatspanaparticular`lightvectorsubspace'wecansyn-
`thesizepicturestakenwithanycombinationofthelightsources.
`Totheextentthatamultichannelimage(suchascolor,
`havingthreechannels:R,G,B),havingLchannelsisacollec-
`tionofLvectors,thenforeachofasetofmultiplechannel
`picturesdi(cid:11)eringonlyinlighting,wecanassociateLvectors.
`WecallthesetofLvectorsa`lightmodule'.
`Ithasbeenshown[ ]thatasetof`lightmodules'(which
`wecalla`lightmodulesubspace')alsospansausefulspace.
`Forexample,asetofcolorpicturesofascenedi(cid:11)eringonly
`inlighting,takenwithwhitelightsatvariousplacesinthe
`scene,wasusedtosynthesizetheresultofhavingtakena
`picturewithcoloredlightsatthesesamelocations.
` .SUMMARY
`Wehavepresentedameansofcombiningmultipledigital
`imagesthatdi(cid:11)eronlyintheirexposure,toarriveatan
`extended-response(cid:13)oatingpointimagearray.Themethod
`proceedsasfollows:
` .Fromthesetofpictures(orfromanothersetofpictures
`takenwiththesamecamera)determinethecamera'spoint-
`wiseresponsefunctionusingthe\self-calibration"method
`ofSection.
`
`

`

`Light falling on image sensor
`
`underexposure
`
`‘‘proper’’ exposure
`
`overexposure
`
`Wyckoff filters
`(analysis)
`
`Wyckoff set
`
`underexposed
`
`‘‘properly’’
`exposed
`
`overexposed
`
`Figure:ThelayersofaWycko(cid:11)(cid:12)lmdecomposethelightfallingonthe(cid:12)lmintodi(cid:11)erentlyexposedimages.Eachofthese
`imagesmayberegardedasa(cid:12)lteredversionofthelightfallingontheimagesensor.These`Wycko(cid:11)(cid:12)lters'actasa(cid:12)lterbank
`tocaptureoverlappingportionsoftheexposure\spectrum",andperformananalysisofthelightfallingontheimagesensor.
`Thesetofpicturescanthenbeusedtoobtainperfectreconstructionoftheoriginallightintensityfallingontheimagesensor.
`
`Combiner (synthesis)
`
`Reconstruction of light
`falling on image sensor
`
`