APPL-1028 / Page 1 of 9
`APPLE INC. v. COREPHOTONICS LTD.
`
`

`

`Bae and Durand / Defocus Magnification
`
`background, called bokeh, has a real cult following among
`some photographers.
`
`(a) large sensor (22.2 x 14.8 mm)
`
`(b) small sensor (7.18 x 5.32 mm)
`
`Figure 2: Given the same field of view and the same f-
`number (f/2.8), a large sensor (a) yields more defocus than
`a small sensor (b) does.
`
`Our technique takes a single input image where the depth
`of field is too large and increases the amount of defocus
`present in out-of-focus regions. That is, our goal is oppo-
`site to that of work that seeks to create images that are sharp
`everywhere.
`
`Our approach first estimates the spatially-varying amount
`of blur over the image, and then uses an off-the-shelf image-
`based technique to increase defocus. We first estimate the
`size of the blur kernel at edges, building on the method by
`Elder and Zucker [EZ98], and then propagate this defocus
`measure over the image with a non-homogeneous optimiza-
`tion. Using our defocus map, we can magnify the existing
`blurriness, which means that we further blur blurry regions
`and keep sharp regions sharp.
`
`Note that in contrast to more difficult problems such as
`depth from defocus, we do not require precise depth estima-
`tion and do not need to accurately disambiguate smooth re-
`gions of the image, since such regions are not much affected
`by extra blur due to defocus. The fundamental ambiguity be-
`tween out-of-focus edges and originally smooth edges is out
`of the scope of our work. We also do not need to disam-
`biguate between objects in front and behind the plane of fo-
`cus. We simply compute the amount of blur and increase it.
`While our method does not produce outputs that perfectly
`matches images captured with a larger-aperture lens, it qual-
`itatively reproduces the amount of defocus. We refer inter-
`ested readers to Appendix A where we review thin-lens op-
`tics and defocus.
`
`1.1. Related work
`
`Defocus effects have been an interest of the Computer Vi-
`sion community in the context of recovering 3D from 2D.
`Camera focus and defocus have been used to reconstruct
`depth or 3D scenes from multiple images: depth from fo-
`cus [Hor68, DW88, EL93, NN94, HK06] and depth of defo-
`cus [Pen87,EL93,WN98,FS02,JF02,FS05]. These methods
`use multiple images with different focus settings and esti-
`mate the corresponding depth for each pixel. They have to
`
`know the focus distance and focal length to computer the
`depth map. In contrast, we do not estimate the depth but the
`blur kernel. We want to treat this problem without the help
`of any special camera settings, but only with image post-
`processing techniques.
`
`Image processing methods have been introduced to mod-
`ify defocus effects without reconstructing depth. Eltoukhy
`and Kavusi [EK03] use multiple photos with different focus
`settings and fuse them to produce an image with extended
`depth of field. Özkan et al. [OTS94] and Trussell and Fo-
`gel [TF92] have developed a system to restore space-varying
`blurred images and Reeves and Mersereau [RM92] find a
`blur model to restore blurred images. This is the opposite of
`what we want to do. They want to restore blurred images,
`while we want to increase existing blurriness.
`
`Kubota and Aizawa [KA05] use linear filters to recon-
`struct arbitrarily focused images from two differently fo-
`cused images. On the contrary, we want to modify defocus
`effects only with a single image. Lai et al. [LFC92] use a sin-
`gle image to estimate the defocus kernel and corresponding
`depth. But their method only works on an image composed
`of straight lines at a spatially fixed depth.
`
`Given an image with a corresponding depth map, depth
`of field can be approximated using a spatially-varying blur,
`e.g. [PC81, BHK∗03], but note that special attention must be
`paid to occlusion boundaries [BTCH05]. Similar techniques
`are now available in commercial software such as Adobe R°
`Photoshop R° (lens blur) and Depth of Field Generator Pro
`(dofpro.com). In our work we simply use these features and
`instead of providing a depth map, we provide a blurriness
`map estimated from the photo. While the amount of blurri-
`ness is only related to depth and is not strictly the same as
`depth, we have found that the results qualitatively achieve
`the desired effect and correctly increase defocus where ap-
`propriate. Note that a simple remapping of blurriness would
`yield a map that resembles more closely a depth map.
`
`2. Overview of Our Approach
`
`For each pixel, we estimate the spatially-varying amount of
`blur. We call our blur estimation the defocus map. We es-
`timate the defocus map in two steps. First, we estimate the
`amount of blur at edges. Then, we propagate this blur mea-
`sure to the rest of the image.
`
`We model an edge as a step function and the blur of this
`edge as a Gaussian blurring kernel. We adapt the method
`by Elder and Zucker [EZ98], which uses multiscale filter re-
`sponses to determine the size of this kernel. We add a cross-
`bilateral filtering step [ED04, PAH∗04] to remove outlier es-
`timates.
`
`We propagate the blur measure using non-homogeneous
`optimization [LLW04]. Our assumption is that blurriness
`varies smoothly over the image except where the color is dis-
`
`c° The Eurographics Association and Blackwell Publishing 2007
`
`APPL-1015 / Page 2 of 9
`
`APPL-1028 / Page 2 of 9
`APPLE INC. v. COREPHOTONICS LTD.
`
`

`

`APPL-1028 / Page 3 of 9
`APPLE INC. v. COREPHOTONICS LTD.
`
`

`

`Bae and Durand / Defocus Magnification
`
`(a) input
`
`(b) actual blur sigma
`
`(c) the zero-crossing of
`the third derivative
`
`(d) blur measure
`using our approach
`
`Figure 4: The zero-crossing of the third derivative (c) is greatly affected by neighboring edges and cannot localize the second
`derivative extrema. In contrast, our approach (d) can estimate the blur sigma that is close to the actual blur sigma (b). The
`input (a) is generated using the blur sigma (b).
`
`intensities and colors. We minimize the difference between
`the blurriness B(p) and a weighted average of blurriness of
`neighboring pixels:
`
`wpqB(q))2
`
`with the distance in space and with the range difference of a
`reference image.
`
`In addition to the original cross bilateral filtering weights,
`we use a sharpness bias, b(BM) = exp(−BM/2). The sharp-
`ness bias corrects blur measures in soft shadows and glossy
`highlights that are higher than they are supposed to be.
`With gσ(x) = exp(−x2/2σ2), a Gaussian function, we de-
`fine the biased cross bilateral filtering of a sparse set of blur
`measures, BM at an edge pixel p as the following:
`
`wpq b(BMq) BMq
`
`(3a)
`
`gσs (||p− q||) gσr (|Ci(p)− Ci(q)|)
`(3b)
`
`1 k
`
`bCBF(BM)p =
`
`∑
`q∈BM
`with: wpq ∝ ∑
`i∈{R,G,B}
`
`(4a)
`
`(4b)
`
`)
`
`(4c)
`
`E(B) = ∑ (B(p)− ∑
`q∈N(p)
`+ ∑ αp (B(p)− BM(p))2
`exp(−(Ci(p)− Ci(q))2
`with: wpq ∝ ∑
`2σ2
`i∈{R,G,B}
`ip
`where σp is the standard deviation of the intensities and col-
`ors of neiboring pixels in a window around p. The window
`size used is 7× 7. We have experimented both with setting
`the second term as hard constraints vs. as a quadratic data
`term, and have found that the latter is more robust to poten-
`tial remaining errors in the blur measure.
`
`and
`
`k = ∑
`q∈BM
`
`wpq b(BMq)
`
`(3c)
`
`where σs controls the spatial neighborhood, and σr the
`influence of the intensity difference, and k normalizes the
`weights. We use the RGB color channels of the original input
`image as the reference and set σr = 10% of the image range
`and σs = 10% of the image size. This refinement process
`does not generate much change but refines a few outliers as
`shown in Figure 5. The cross bilateral filtering refines out-
`liers such as yellow and green measures (b) in the focused
`regions to be blue (c).
`
`4. Blur Propagation
`
`Our blur estimation provides blur kernels only at edges
`and we need to propagate this blur measure. We use non-
`homogeneous optimization [LLW04] and assume that the
`amount of defocus is smooth when intensity and color are
`smooth.
`
`4.1. Propagate using optimization
`
`Our propagation is inspired by the colorization paper by
`Levin et al. [LLW04]. We impose the constraint that neigh-
`boring pixels p, q have similar blurriness if they have similar
`
`We solve this optimization problem by solving the cor-
`responding sparse linear system. Figure 6 shows the defocus
`map for various values of α. We use α = 0.5 for edge pixels.
`
`5. Results
`
`We have implemented our blur estimation using Matlab. Our
`defocus map enables defocus magnification. We rely on Pho-
`toshop’s lens blur to compute the defocused output. We crop
`the upper and lower 5% of the defocus map and clamp its
`minimum value to 0. In addition, we apply Gaussian blur to
`the defocus map to use it as a depth map. The Gaussian blur
`radius is set to 0.5% of the image size.
`
`Using our defocus map, we can simulate the effect of dou-
`bling the aperture size. Figure 7 compares two input defocus
`maps of two images with the f-number 8 (a) and 4 (b). As
`we double the defocus map (c) of the f/8 image, we obtain a
`result similar to the defocus map (d) of the f/4 image. While
`the simulated defocused map (e) is not exactly the same as
`the real map (d), the output image with magnified defocus
`(f) is visually close to the f/4 photograph (b).
`
`In Figure 11, we show the results of using our defocus
`map to magnify the existing defocus effects in the original
`images. The results preserve the sharpness of the focused re-
`gions but increase the blurriness of the out-of-focus regions.
`
`c° The Eurographics Association and Blackwell Publishing 2007
`
`APPL-1015 / Page 4 of 9
`
`APPL-1028 / Page 4 of 9
`APPLE INC. v. COREPHOTONICS LTD.
`
`

`

`APPL-1028 / Page 5 of 9
`APPLE INC. v. COREPHOTONICS LTD.
`
`

`

`APPL-1028 / Page 6 of 9
`APPLE INC. v. COREPHOTONICS LTD.
`
`

`

`APPL-1028 / Page 7 of 9
`APPLE INC. v. COREPHOTONICS LTD.
`
`

`

`APPL-1028 / Page 8 of 9
`APPLE INC. v. COREPHOTONICS LTD.
`
`

`

`Bae and Durand / Defocus Magnification
`
`(a)
`
`(b)
`
`(c)
`
`(d)
`
`(e)
`
`(f)
`
`Input
`
`Our defocus map
`
`Our result with magnified defocus
`
`Figure 11: Results. The original images, their defocus maps, and results blurred using our approach. The inputs were taken
`by (a) a Nikon D50 with a sensor size of 23.7× 15.6 mm and a 180.0 mm lens at f/4.8, (b) a Canon 1D Mark II with a sensor
`size of 28.7× 19.1 mm and a Canon EF 85mm f/1.2L lens, and (c, d) a Canon PowerShot A80, a point-and-shoot camera with
`a sensor size of 7.18× 5.32 mm, and a 7.8 mm lens at f/2.8. The two at the bottom are from bigfoto.com.
`
`c° The Eurographics Association and Blackwell Publishing 2007
`
`APPL-1015 / Page 9 of 9
`
`APPL-1028 / Page 9 of 9
`APPLE INC. v. COREPHOTONICS LTD.
`
`