UNITED STATES PATENT AND TRADEMARK OFFICE
`_______________
`
`BEFORE THE PATENT TRIAL AND APPEAL BOARD
`_____________
`
`APPLE INC.,
`Petitioner
`v.
`COREPHOTONICS LTD.
`Patent Owner
`
`_______________
`IPR2020-00905 & IPR2020-00906
`U.S. Patent 10,255,479
`_______________
`DECLARATION OF FRÉDO DURAND, PH.D.
`UNDER 37 C.F.R. § 1.68 IN SUPPORT OF PETITIONER’S REPLIES
`
`Apple v. Corephotonics
`
`- 1 -
`
`APPL-1038
`
`

`

`
`
`
`
` Declaration of Frédo Durand, Ph.D. in support of
`Petitioner’s Replies in IPR2020-00905; -00906
`
`
`TABLE OF CONTENTS
`
`I.
`II.
`
`INTRODUCTION .......................................................................................... 4
`CLAIM CONSTRUCTION ........................................................................... 5
`A.
`“fused image with a point of view (POV) of the Wide camera” (claims
`1 and 23) ............................................................................................... 5
`“to find translations between matching points in the images to
`calculate depth information and to create a fused image suited for
`portrait photos” (claim 19) ................................................................. 10
`III. OBVIOUSNESS .......................................................................................... 12
`A.
`Claims 1, 10-14, 16, 18, 23, 32-36, 38, and 40 are obvious over the
`combination of Parulski and Konno. .................................................. 12
`1.
`A POSITA would have implemented Parulski’s Fig. 14 method
`of outputting a combined image with a broadened depth of field
`by using Parulski’s range mapping method in Fig. 11 to identify
`and extract objects. ................................................................... 13
`Parulski teaches outputting a “fused image with a point of view
`(POV) of the Wide camera” when this term is properly
`construed to include Wide position POV. ................................ 20
`Claims 2-4 and 24-26 are obvious over the combination of Parulski,
`Konno, and Szeliski. .......................................................................... 23
`Claims 5-9 and 27-31 are obvious over the combination of Parulski,
`Konno, Szeliski, and Segall ............................................................... 24
`Claims 15 and 37 are obvious over the combination of Parulski,
`Konno, and Stein. ............................................................................... 25
`Claims 19 and 20 are obvious over the combination of Parulski,
`Ogata, Kawamura, and Soga because Patent Owner does not dispute
`that “a camera controller configured to … to process the Wide and Tele
`image to find translations between matching point in the images to
`calculate depth information and to create a fused image suited for
`portrait photos” is satisfied under Petitioner’s construction. ............. 27
`Claims 21 and 22 are obvious over the combination of Parulski,
`Ogata, Kawamura, Soga, and Morgan-Mar. ...................................... 27
`
`B.
`
`B.
`
`C.
`
`2.
`
`D.
`
`E.
`
`F.
`
`Apple v. Corephotonics
`
`
`
`- 2 -
`
`APPL-1038
`
`

`

`
`
`
`
` Declaration of Frédo Durand, Ph.D. in support of
`Petitioner’s Replies in IPR2020-00905; -00906
`
`IV. SECONDARY CONSIDERATIONS .......................................................... 28
`A.
`Patent Owner’s secondary considerations of non-obviousness are not
`relevant because there is no nexus. .................................................... 28
`The industry praise/licensing has no nexus and much of the alleged is
`from biased sources. ........................................................................... 29
`Patent Owner did not show commercial success. .............................. 44
`C.
`There was no failure of others, and Petitioner did not copy. ............. 45
`D.
`V. DECLARATION .......................................................................................... 49
`
`
`B.
`
`
`
`
`
`
`
`
`
`Apple v. Corephotonics
`
`
`
`- 3 -
`
`APPL-1038
`
`

`

`
`
`INTRODUCTION
`
`
`
`I.
`
` Declaration of Frédo Durand, Ph.D. in support of
`Petitioner’s Replies in IPR2020-00905; -00906
`
`
`1.
`
`I, Frédo Durand, who previously submitted a declaration as APPL-
`
`1003 in both IPR2020-00905 (905 IPR) and IPR2020-00906 (906 IPR). The terms
`
`of my engagement, my background, qualifications and prior testimony, and the
`
`legal standards and claim constructions I am applying are set forth in my previous
`
`CV and declaration. See APPL-1003; APPL-1004. I offer this declaration in reply
`
`to the Response the Patent Owner filed in these proceeding. In forming my
`
`opinion, I have considered the materials noted in my previous declaration, as well
`
`as the following additional materials:
`
`(1) Dr. Hart’s Declaration, Ex. 2001;
`
`(2) Declaration of Eran Kali, Ex. 2013;
`
`(3)
`
`(4)
`
`Patent Owner’s Response in IPR2020-00905, Paper No. 16;
`
`Patent Owner’s Response in IPR2020-00906, Paper No. 16;
`
`(5) Ralph E. Jacobson et al., The Manual of Photography: photographic
`
`and digital imaging, 9th Edition, 2000 (“Jacobson”), APPL-1017;
`
`(6) U.S. Patent App. Pub. No. 2010/0321511 to Koskinen et al.
`
`(“Koskinen”), APPL-1016;
`
`(7) Dr. Hart’s deposition transcript, April 29, 2021 (“Hart Deposition”),
`
`APPL-1037;
`
`Apple v. Corephotonics
`
`
`
`- 4 -
`
`APPL-1038
`
`

`

`
`
`
`
` Declaration of Frédo Durand, Ph.D. in support of
`Petitioner’s Replies in IPR2020-00905; -00906
`
`
`(8) Any additional documents discussed below.
`
`II. CLAIM CONSTRUCTION
`
`A.
`
`2.
`
`“fused image with a point of view (POV) of the Wide camera”
`(claims 1 and 23)
`
`Dr. Hart incorrectly proposes to construe “fused image with a point of
`
`view (POV) of the Wide camera” as “fused image in which the positions and
`
`shapes of objects reflect the POV of the Wide camera.” Ex. 2001, ¶46. As an initial
`
`matter, Dr. Hart’s construction redundantly replaces “with a point of view (POV)”
`
`with “in which the positions and shapes of objects reflect the POV.” This proposal
`
`unhelpfully fails to provide any meaning to the construed term “point of view
`
`(POV),” as the construction repeats the term within the construction. Dr. Hart
`
`asserts that “POV” itself refers to a particular position of objects in the image and
`
`perspective/shape of objects in an image. Ex. 2001, ¶43. Thus, a more appropriate
`
`construction under Patent Owner’s theory would be “fused image in which the
`
`positions and shapes of objects reflect those of the Wide camera.”
`
`3.
`
`But Dr. Hart’s construction, even when clarified to remove the
`
`redundancy, is still incorrect because it uses the word “and” instead of the word
`
`“or.” Specifically, Dr. Hart’s erroneously requires a fused image in which both
`
`object positions and shapes reflect those of the Wide camera, which is not required
`
`by the claim language and conflicts with the specification. The ’479 patent refers
`
`Apple v. Corephotonics
`
`
`
`- 5 -
`
`APPL-1038
`
`

`

`
`
`
`
` Declaration of Frédo Durand, Ph.D. in support of
`Petitioner’s Replies in IPR2020-00905; -00906
`
`
`to two different types of Wide POV: “Wide position POV” describes objects
`
`maintaining the same position as in the wide image, and “Wide perspective POV”
`
`describes objects maintaining the same shape. APPL-1005, 5:16-19 (“If the output
`
`image retains the Wide image shape then it has the Wide perspective POV. If it
`
`retains the Wide camera position then it has the Wide position POV.”). Critically,
`
`maintaining either one of those types of Wide POV can satisfy the claim language,
`
`which requires only “a” point of view (POV) of the Wide camera. Dr. Hart admits
`
`that the ‘479 patent contemplates embodiments in which the output image reflects
`
`only the “Wide position POV” or “Wide perspective POV”. Ex. 2001, ¶38.
`
`4.
`
`Dr. Hart incorrectly argues, however, that when the ’479 patent
`
`“refers to ‘Wide POV,’ without qualification, it is referring to the complete Wide
`
`POV, both perspective and position.” Id., ¶45 (emphasis added). Dr. Hart is wrong.
`
`The ‘479 Patent discloses two alternative embodiments of fusion maintaining Wide
`
`POV: one that maintains both Wide perspective POV and Wide position POV, and
`
`one that maintains only the Wide position POV. Dr. Hart erroneously exclude the
`
`latter embodiment. Specifically, the ’479 Patent discusses “Wide POV” in the
`
`context of fusion as the fused image obtained by registering Tele image pixels to a
`
`matching pixel set with in the Wide image pixels, maintaining only the Wide
`
`position POV: “it is possible to register Telephoto image pixels to a matching pixel
`
`Apple v. Corephotonics
`
`
`
`- 6 -
`
`APPL-1038
`
`

`

`
`
`
`
` Declaration of Frédo Durand, Ph.D. in support of
`Petitioner’s Replies in IPR2020-00905; -00906
`
`
`set within the Wide image pixels, in which case the output image will retain the
`
`Wide POV (‘Wide fusion’).” APPL-1005, 5:23-26. While this sentence refers to
`
`“Wide POV,” the specification continues on to confirm that this embodiment
`
`maintains only Wide position POV. Only when an additional function is performed
`
`(the Telephoto image is “shifted” prior to registration mapping) is the Wide
`
`perspective POV also retained: “It is also possible to perform the registration after
`
`either sub-camera image is shifted, in which case the output image will retain the
`
`respective Wide or Tele perspective POV.” Id., 5:30-33 (emphasis added), 5:14-16
`
`(“[t]he system output image can have the shape and position of either sub-camera
`
`image or the shape or position of a combination thereof.”). In other words, an
`
`additional “possible” step is required in order to retain the perspective POV (i.e.,
`
`retain the shapes), and the embodiment that does not perform that optional step
`
`maintains only the position POV, not the perspective POV.
`
`5.
`
`Consequently, Patent Owner’s construction, that requires both the
`
`position (i.e., Wide position POV) and shape (i.e., Wide perspective POV) of
`
`objects be maintained from the wide image, improperly excludes the embodiment
`
`that maintains only the Wide position POV. Patent Owner’s construction
`
`erroneously imports an unclaimed limitation, ignoring the specification’s teaching
`
`that Wide POV can be only the Wide position POV.
`
`Apple v. Corephotonics
`
`
`
`- 7 -
`
`APPL-1038
`
`

`

`
`
`
`
` Declaration of Frédo Durand, Ph.D. in support of
`Petitioner’s Replies in IPR2020-00905; -00906
`
`Third, Dr. Hart raises a red herring by contrasting the use of “FOV” in
`
`6.
`
`Petitioner’s construction with “POV”. Ex. 2001, ¶¶ 37-39. This is an immaterial
`
`distinction for present purposes. My previous analysis proposed the construction:
`
`“fused image that maintains [1] the Wide camera’s field of view or [2] the Wide
`
`camera’s position.” APPL-1003, ¶¶ 32-33. I phrased it this way to reflect that the
`
`Wide camera POV is based on either [1] the scene seen through the wide camera’s
`
`field of view (i.e., the position of objects seen in the camera); or [2] how the wide
`
`camera was positioned in 3D space when the image was captured (i.e., determining
`
`the perspective or shape of the objects). Ex. 2036, Durand Depo. at 21:3–7. Thus,
`
`maintaining the Wide camera FOV in my construction would ensure that the
`
`position of objects in the image is that of the Wide camera, as required by Dr.
`
`Hart’s construction. Similarly, maintaining the Wide camera 3D position in my
`
`construction would ensure that the perspective/shape of objects in the image is that
`
`of the Wide camera, as required by Dr. Hart’s construction.
`
`7.
`
`Dr. Hart asserts that FOV can be described using a single number, and
`
`thus cannot be relevant to the POV. Ex. 2001, ¶40. But, as Dr. Hart discusses,
`
`object positions in images depend on multiple factors like the 3D location of the
`
`camera, the orientation of the camera, and its field of view. My original
`
`construction referred to the “fused image” maintaining the Wide camera FOV to
`
`Apple v. Corephotonics
`
`
`
`- 8 -
`
`APPL-1038
`
`

`

`
`
`
`
` Declaration of Frédo Durand, Ph.D. in support of
`Petitioner’s Replies in IPR2020-00905; -00906
`
`
`indicate the camera’s viewing direction at the captured scene. While the Wide
`
`camera FOV is a single number, by saying the fused image maintains that FOV,
`
`my construction indicates that the image shows the same objects/scene as seen
`
`from the Wide camera FOV. While FOV can be described as a single number, Dr.
`
`Hart gives no explanation as to why the image maintaining the FOV must
`
`somehow be described by a single number.
`
`8.
`
`As previously explained, the parties’ constructions reflect the same
`
`concepts in capturing a scene but merely use different terminology. The only
`
`significant difference between the parties’ constructions is whether the claim
`
`language encompasses maintaining Wide position POV or Wide perspective POV
`
`(as Petitioner contends) or requires maintaining both Wide position POV and Wide
`
`perspective POV (as Patent Owner incorrectly contends). To narrow the disputes
`
`for resolution by the Board, Petitioner’s construction can be rephrased using Patent
`
`Owner’s terminology, consistent with the specification, as “fused image in which
`
`the positions or shapes of objects reflect those of the Wide camera.” As shown in
`
`my previous analysis and explained further below, Parulski satisfies the meaning
`
`required by the claim language, including under this alternative construction.
`
`Apple v. Corephotonics
`
`
`
`- 9 -
`
`APPL-1038
`
`

`

`
`
`
`
` Declaration of Frédo Durand, Ph.D. in support of
`Petitioner’s Replies in IPR2020-00905; -00906
`
`
`“to find translations between matching points in the images to
`calculate depth information and to create a fused image suited for
`portrait photos” (claim 19)
`
`B.
`
`9.
`
`Dr. Hart’s construction improperly reads the entire limitation as one
`
`process where “translations between matching points in the images to calculate
`
`depth information” is used to “create a fused image suited for portrait photos.” See
`
`Ex. 2001, ¶47.
`
`10. Dr. Hart first argues this construction using the plain and ordinary
`
`meaning based on English grammar rules. Ex. 2001, ¶48. This argument fails
`
`because the limitation requires a camera controller “configured” to perform three
`
`functions, numbered below:
`
`a camera controller … configured [1] to control the AF
`mechanisms, [2] to process the Wide and Tele images to
`find translations between matching points in the images to
`calculate depth information and [3] to create a fused
`image suited for portrait photos, the fused image having a
`DOF shallower
`than DOFT and having a blurred
`background.
`
`APPL-1001, claim 19 (annotated). The omission of the Oxford comma before the
`
`“and” in the three-item list above is consistent with other claims in the ’479 patent.
`
`E.g., id., claim 20 (omitting Oxford comma before the “fifth lens element— “a first
`
`lens element …, a second lens element …, a fourth lens element with negative
`
`Apple v. Corephotonics
`
`
`
`- 10 -
`
`APPL-1038
`
`

`

`
`
`
`
` Declaration of Frédo Durand, Ph.D. in support of
`Petitioner’s Replies in IPR2020-00905; -00906
`
`
`power and a fifth lens element”). If the “and” connects the “to calculate” and the
`
`“to create”, as Dr. Hart argues (Ex. 2001, ¶48), then the list of “configured”
`
`functions performed by the camera controller has no conjunction, and forms an
`
`incomplete list. Thus, my construction properly interprets the plain meaning of the
`
`claim as a list of three functions that the camera controller is “configured” to
`
`perform.
`
`11. Second, Dr. Hart simply asserts that the process for “‘creat[ing] a
`
`fused image’ is part of the broader ‘to process the Wide Tele’” and should
`
`therefore include all of the steps in between these two parts. Ex. 2001, ¶49. As
`
`shown above, this is grammatically incorrect.
`
`12. Moreover, neither of these arguments rebut my previous analysis—
`
`that the specification does not support a construction where “find[ing] translations
`
`between matching points to calculate depth information” is used to “create a fused
`
`image.” See APPL-1003, ¶¶ 31-35. Rather, the ’479 specification only describes
`
`the process of “find[ing] translation between matching points … to calculate depth
`
`information” for use in faster focusing:
`
`The transformation coefficient includes the translation
`between matching points in the two images. … Different
`translations will result in a different number of pixel
`movements between matching points in the images. This
`
`Apple v. Corephotonics
`
`
`
`- 11 -
`
`APPL-1038
`
`

`

`
`
`
`
` Declaration of Frédo Durand, Ph.D. in support of
`Petitioner’s Replies in IPR2020-00905; -00906
`
`
`movement can be translated into depth and the depth can
`be translated into an AF position. This enables to set the
`AF position by only analyzing two images (Wide & Tele).
`The result is fast focusing.
`
`APPL-1001, 12:11-15; see APPL-1003, ¶33. Adopting Patent Owner’s construction
`
`read out “fast focusing,” the only embodiment in the specification for calculating and
`
`using depth information.
`
`13. Dr. Hart does not dispute that creating a fused image using calculated
`
`depth information is not described anywhere in the specification. See Ex. 2001, ¶49;
`
`APPL-1001, 7:44-59, 9:39-60. For example, Fig. 5 describes a processing algorithm
`
`ending with fusion step 512 where “the decision output, re-sampled Tele image and
`
`the Wide image are fused into a single zoom image.” Id., 9:58-60. This fusion step
`
`notably excludes calculating depth information or using calculated depth information.
`
`Consequently, Petitioner’s construction is the only one that encompasses the depth
`
`calculation and use described in the specification, finds written description support in
`
`the specification, and properly divides this limitation into the two distinct functions
`
`described in the specification.
`
`III. OBVIOUSNESS
`
`A. Claims 1, 10-14, 16, 18, 23, 32-36, 38, and 40 are obvious over the
`combination of Parulski and Konno.
`
`14. Dr. Hart’s believes that Parulski does not teach a “fused image with a
`
`- 12 -
`
`APPL-1038
`
`Apple v. Corephotonics
`
`
`

`

`
`
`
`
` Declaration of Frédo Durand, Ph.D. in support of
`Petitioner’s Replies in IPR2020-00905; -00906
`
`
`point of view (POV) of the Wide camera” since “nothing in the discussion of
`
`Figure 11 describes using a range map as part of a system that outputs a ‘fused
`
`image’” and “Figure 14 … neither mentions using a range map” nor “provide[s]
`
`any detail on how the two images are fused, if at all.” Ex. 2001, ¶¶ 73, 77. Both of
`
`these arguments fail.
`
`1.
`
`A POSITA would have implemented Parulski’s Fig. 14
`method of outputting a combined image with a broadened
`depth of field by using Parulski’s range mapping method in
`Fig. 11 to identify and extract objects.
`
`15. Dr. Hart does not dispute that Parulski’s Fig. 11 teaches a method for
`
`creating a range map that “map[s] tele image pixels to matching pixels within the
`
`wide image.” See id., ¶¶ 73-76. But Dr. Hart does not believe that a POSITA would
`
`have combined the range mapping method in Fig. 11 with the image combination
`
`method in Fig. 14 because “[n]owhere … does Parulski describe using the Figure
`
`11 method together with image fusion.” Id., ¶74.
`
`16. Dr. Hart impermissibly views the POSITA with no creativity,
`
`knowledge, or skill, incapable of using two of Parulski’s compatible teachings—
`
`range mapping to identify and extract objects and combining objects from one
`
`image with corresponding objects in another image—because Parulski explicitly
`
`say to do so. Id., ¶¶ 74-76; APPL-1037, 61:22-10, 62:14-63:17, 100:25-101:13,
`
`111:17-112:5, 114:3-13.
`
`Apple v. Corephotonics
`
`
`
`- 13 -
`
`APPL-1038
`
`

`

`
`
`
`
` Declaration of Frédo Durand, Ph.D. in support of
`Petitioner’s Replies in IPR2020-00905; -00906
`
`
`17. Dr. Hart agrees that “[i]dentifying objects within an image,” among
`
`others are “operations that could utilize a range map ….” Ex. 2001, ¶73. One
`
`operation that Dr. Hart fails to mention, though, is to “enable object extraction
`
`from an image by identifying the continuous boundaries of the object so it can be
`
`segmented within the image.” APPL-1005, 20:56-59. While Fig. 14 may not
`
`explicitly state that it uses a range map, a POSITA, not being an automaton, would
`
`have recognized the applicability of using object identification and extraction to
`
`the last step in Fig. 14 (block 514) which teaches that the second image (e.g., tele
`
`image) “is used to enhance the depth of field of the primary image” (e.g., wide
`
`image), “for instance, where the secondary still image is used to … sharpen
`
`portions of the primary still image that are positioned near the secondary focus
`
`distance.” APPL-1005, 22:37-42. A range map is the only method taught by
`
`Parulski that a POSITA could have used for identifying and extracting the portions
`
`of the tele image “positioned near the secondary focus distance” to “sharpen
`
`portions” of the wide image, thereby broadening the depth of field of the wide
`
`image.
`
`18. Rather than use what Parulski already teaches to implement Fig. 14
`
`(i.e., identifying and extracting portions of the tele image to combine with the
`
`wide), Dr. Hart believes that a POSITA would have looked to another method
`
`Apple v. Corephotonics
`
`
`
`- 14 -
`
`APPL-1038
`
`

`

`
`
`
`
` Declaration of Frédo Durand, Ph.D. in support of
`Petitioner’s Replies in IPR2020-00905; -00906
`
`
`outside of Parulski that does not “incorporat[e] image data from the tele image”
`
`like “sharpening” by “enhance[ing] edges based solely on the data from a single
`
`image.” Ex. 2001, ¶75. If Parulski was only concerned with sharpening edges of an
`
`object, it would have explained its enhancement process as sharpening edges to
`
`enhance the depth of field, not “sharpening portions” by combining images. But
`
`instead, step 514 uses portions of the tele image “captured near” the tele camera’s
`
`“focus distance” (i.e., tele camera’s depth of field) to “sharpen portions” of the
`
`wide image, or in other words, combining two images “into a modified image with
`
`a broadened depth of field.” APPL-1005, 22:37-43, 28:48-53. This is evident from
`
`Parulski’s example with the mountains, the flowers, and the dog. See APPL-1005,
`
`21:7-44.
`
`19. Dr. Hart argues that Parulski’s example does not provide motivation
`
`for using a range map to combine image data from the tele image with the wide
`
`image to broaden the depth of field. Ex. 2001, ¶¶ 77-78. But Parulski’s example is
`
`included to instruct a POSITA about the “use of a range map for purposes such as
`
`noted above,” which include object identification and extraction. See APPL-1005,
`
`20:50-21:8. Parulski then uses the example to explain that “the depth of field can
`
`be adjusted so that, e.g., the dog is in focus, the mountains are in focus and so are
`
`those great flowers.” APPL-1005, 21:25-27. This as well as other depth of field
`
`Apple v. Corephotonics
`
`
`
`- 15 -
`
`APPL-1038
`
`

`

`
`
`
`
` Declaration of Frédo Durand, Ph.D. in support of
`Petitioner’s Replies in IPR2020-00905; -00906
`
`enhancements is achieved in part by sharpening the dog using the in-focus portions
`
`of the tele image, thereby broadening the depth of field to include objects closer to
`
`the wide camera that may be outside of the wide camera’s depth of field. See
`
`APPL-1005, 21:27-31.
`
`20. Dr. Hart believes that a POSITA would not have read Parulski’s
`
`example this way because Parulski’s states, at the bottom of column 21 in
`
`reference to Figs. 12 and 13, that an ultra-wide lens can capture an image focused
`
`at 8 feet “so that objects from 4 feet to infinity are in focus.” See Ex. 2001, ¶76;
`
`APPL-1005, 21:57-65. This would presumably include the dog in the example at 5
`
`feet, which Patent Owner interprets as evidence that a POSITA would not use the
`
`focused portions of the dog from the tele image to broaden the depth of field of the
`
`wide image. See Ex. 2001, ¶76; APPL-1005, 21:12-13.
`
`21. This argument fails, though, because it does not consider the
`
`combination with Konno’s wide lens (which Patent Owner does not dispute)
`
`which, when similarly focused at 8 feet, has a smaller depth of field of about 6 feet
`
`to infinity.
`
`22. Depth of field (“DOF”) of a combination lens and image sensor can
`
`be determined using the following equations provided in Jacobson (APPL-1016):
`
`Considering the general photographic case, for distant
`subjects where m is less than unity, the object plane is near
`
`- 16 -
`
`Apple v. Corephotonics
`
`
`APPL-1038
`
`

`

`
`
`
`
` Declaration of Frédo Durand, Ph.D. in support of
`Petitioner’s Replies in IPR2020-00905; -00906
`
`
`infinity, so we may put L' approximately equal to f
`Likewise, N', the effective aperture, may be replaced by N,
`the relative aperture. For simple and symmetrical lenses,
`the pupils are located in the principal planes, so we may
`take L = u. Hence
`
` 𝑆= !"!
`"!#$%!
` 𝑅= !"!
`"!&$%!
`
`(15)
`
`(16)
`
`The values of S and R define the far and near limits
`respectively of the depth of field, when the lens is focused
`on distance u. The values may be tabulated in various
`ways, displayed in graphical form or provided as depth of
`field scales on the focusing mounts of lenses (Figure 4.20),
`
`If depth of field (T) is defined as T = S – R then
`
`𝑇= ’"!!!$%
`""#$!%!!!
`
`(17)
`
`APPL-1016, p.54. The variables in the above equations are:
`
`f is the focal length of the lens,
`u is the distance from the focus to the lens,
`N is the F number of the lens,
`C is the circle of confusion of the image sensor,
`
`See id., pp.53-55.
`
`23. According to Jacobson, the circle of confusion (C) “is taken as a
`
`constant for a given format, for the whole range of lenses.” APPL-1016, p.53. One
`
`Apple v. Corephotonics
`
`
`
`- 17 -
`
`APPL-1038
`
`

`

`
`
`
`
` Declaration of Frédo Durand, Ph.D. in support of
`Petitioner’s Replies in IPR2020-00905; -00906
`
`
`way that C can be determined, according to Koskinen (APPL-1017), is by using
`
`the Zeiss formula (C=d/1730) where d is the diagonal of the image sensor. APPL-
`
`1017, ¶29 (“There are different ways to find the CoC diameter limit, for example
`
`the Zeiss formula d/1730.”). Using a sensor such as the ICX652CQZ (see APPL-
`
`1032), circle of confusion (C) using the Zeiss formula would approximately be
`
`0.005 (i.e., 9.3 mm/1730). The equation from Jacobson with a circle of confusion
`
`C=0.005, yields the following approximate DOFW and DOFT for Konno’s Example
`
`2 dual lens module. Because DOF is relative to a specific distance of focus from
`
`the lens, the below equation uses 8 ft., as in Dr. Hart’s example, for the focus
`
`distance (u) of the wide camera which converts to 2438.4 mm and 5 ft. for the
`
`focus distance of the telephoto camera which converts to 1524 mm.
`
`24. Konno’s wide lens has a focal length of 3.70 mm and an f-number of
`
`3.0. APPL-1015, p.21. Using these values in the DOF calculations for the wide
`
`lens yields the following:
`
`𝑫𝑶𝑭𝑾= (2)(3.7)’(2438.4)’(3)(0.005)
`(3.7))−((3)’(0.005)’(2438.4)’)=−𝟐𝟏𝟐𝟑 𝒎𝒎
`(2438.4)(3.7)’
`𝑭𝑨𝑹𝑾=
`(3.7)’−((3.0)(0.005)(2438.4))=−𝟏𝟒𝟓𝟗 𝒎𝒎
`
`The far and near DOFW limits are:
`
`Apple v. Corephotonics
`
`
`
`- 18 -
`
`APPL-1038
`
`

`

`
`
`
`
`𝑵𝒆𝒂𝒓𝑾=
`
` Declaration of Frédo Durand, Ph.D. in support of
`Petitioner’s Replies in IPR2020-00905; -00906
`
`
`(2438.4)(3.7)’
`(3.7)’+((3.0)(0.005)(2438.4))=𝟔𝟔𝟒 𝒎𝒎
`
`25. The negative result in the above equations indicates infinity meaning
`
`that the DOFW when focused at 8 ft. has a depth of field that extends to infinity.
`
`This is also indicated by the far limit FARW of the depth of field also being
`
`negative. The near limit NEARW of the depth of field is 664 mm which converts to
`
`about 2.2 ft. Applied to the focus distance of 8 ft. indicates that the depth of field
`
`of Konno’s wide lens would be from 2.2 ft. in front of the focus distance (i.e.,
`
`about 6 ft.) to infinity.
`
`26.
`
`In this case, the dog being at 5 feet would be slightly out of focus in
`
`the wide image which, based on Parulski’s example, could be sharpened by using a
`
`range map to identify the in-focus portions of the dog from the tele image,
`
`extracting these in-focus portions, and outputting an image that combines these
`
`portions with the wide image to sharpen the slightly-out-of-focus dog.
`
`27. Thus, Dr. Hart fails to show that a POSITA would not have used
`
`Parulski’s range mapping methods for object identification and extraction to
`
`broaden the depth of field of the wide image via combination with corresponding
`
`portions of the tele image captured at the tele camera’s focus distance.
`
`Apple v. Corephotonics
`
`
`
`- 19 -
`
`APPL-1038
`
`

`

`
`
`
`
`2.
`
` Declaration of Frédo Durand, Ph.D. in support of
`Petitioner’s Replies in IPR2020-00905; -00906
`
`
`Parulski teaches outputting a “fused image with a point of
`view (POV) of the Wide camera” when this term is properly
`construed to include Wide position POV.
`
`28. Dr. Hart states that “[n]othing in Parulski suggests that whatever
`
`image data from the tele image that might be ‘fused’ into the output would be
`
`modified to have the shapes and positions from the wide image POV,” or in other
`
`words, that the analysis in my declaration does not shown how the combination
`
`would satisfy the “fused image with a point of view (POV) of the Wide camera”
`
`under his construction. Ex. 2001, ¶79. As discussed above, though, Dr. Hart’s
`
`construction improperly imports a limitation requiring that Wide perspective POV
`
`must be maintained, when in fact the claim also encompasses maintaining only
`
`Wide position POV. See Ex. 2001, ¶79. Dr. Hart does not directly dispute that
`
`Parulski teaches maintaining the Wide position POV when combining portions of
`
`the tele image with the wide image. See Ex. 2001, ¶¶ 79-80.
`
`29. As discussed previously, the ‘479 specification describes that Wide
`
`position POV is maintained by “register[ing] Tele image pixels to a matching pixel
`
`set within the Wide image pixels, in which case the output image will retain the
`
`Wide POV (‘Wide fusion’).” APPL-1001, 5:23-25. The Wide position POV is
`
`therefore maintained when the FOV of the Wide camera is maintained.
`
`30. Only when the pixels from the Tele image are “shifted” is the Wide
`
`Apple v. Corephotonics
`
`
`
`- 20 -
`
`APPL-1038
`
`

`

`
`
`
`
` Declaration of Frédo Durand, Ph.D. in support of
`Petitioner’s Replies in IPR2020-00905; -00906
`
`
`perspective POV for the objects also maintained: “It is also possible to perform the
`
`registration after either sub-camera image is shifted, in which case the output
`
`image will retain the respective Wide or Tele perspective POV.” APPL-1001,
`
`5:30-33. Claims 1 and 23 only requires that the POV of the wide camera be
`
`maintained “by mapping Tele image pixels to matching pixels within the Wide
`
`image” (e.g., using a registration map as in Parulski’s Fig. 11). APPL-1001, 13:48-
`
`50. Nothing in the claims also requires shifting the pixels in the Tele image to also
`
`maintain the Wide perspective POV. While the Wide perspective POV could also
`
`be maintained by also performing shifting, the claims do not require it.
`
`31. Parulski teaches in Fig. 14 that when the first capture stage is the wide
`
`camera and the second capture state is the tele camera, that portions of the tele
`
`image are combined with the wide image, thereby producing an output image that
`
`maintains the Wide position POV or the field of view of the Wide camera when the
`
`image was captured:
`
`Apple v. Corephotonics
`
`
`
`- 21 -
`
`APPL-1038
`
`

`

`
`
`
`
` Declaration of Frédo Durand, Ph.D. in support of
`Petitioner’s Replies in IPR2020-00905; -00906
`
`
`
`
`APPL-1003, p.48; APPL-1005, 28:48-53 (“an image is captured from the primary
`
`capture unit at one focus position and another image is captured from the scene
`
`analysis capture unit (the secondary image capture unit) at another focus position.
`
`Then, the two images are combined into a modified image with a broadened depth
`
`of field.”).
`
`32. My previous analysis explains the same thing—that Parulski
`
`maintains the Wide position POV in terms of the field of view of the wide camera
`
`when the image was captured by outputting an image that maps Tele image pixels
`
`- 22 -
`
`APPL-1038
`
`Apple v. Corephotonics
`
`
`

`

`
`
`
`
` Declaration of Frédo Durand, Ph.D. in support of
`Petitioner’s Replies in IPR2020-00905; -00906
`
`
`to matching pixels within the Wide image. Petition at 29-30; APPL-1003, p.51-52.
`
`B. Claims 2-4 and 24-26 are obvious over the combination of
`Parulski, Konno, and Szeliski.
`
`33. Dr. Hart admits that “image rectification was already well known to a
`
`POSITA” at the time Paru