`
`BEFORE THE PATENT TRIAL AND APPEAL BOARD
`
`ALIGN TECHNOLOGY, INC.
`Petitioner
`
`v.
`
`3SHAPE A/S
`Patent Owner
`
`Case No. IPR2019-00118
`Patent No. 9,962,244
`
`DECLARATION OF DR. CHANDRAJIT L. BAJAJ, PH.D.
`IN SUPPORT OF INTER PARTES REVIEW OF U.S. PATENT NO.
`9,962,244
`
`Mail Stop “PATENT BOARD”
`Patent Trial and Appeal Board
`U.S. Patent and Trademark Office
`P.O. Box 1450
`Alexandria, VA 22313-1450
`
`Align Ex. 1003
`U.S. Patent No. 9,962,244
`
`
`
`
`I.
`II.
`III.
`
`IV.
`
`V.
`VI.
`VII.
`
`
`
`
`
`
`TABLE OF CONTENTS
`
`Petition for IPR2019-00118
` U.S. Patent No. 9,962,244
`
`Introduction ................................................................................................. 22
`Qualifications and Expertise ....................................................................... 26
`Legal Understanding ................................................................................... 31
`A. My Understanding of Claim Construction .......................................... 31
`B. A Person of Ordinary Skill in the Art ................................................. 32
`C. My Understanding of Obviousness ..................................................... 33
`D. My Understanding of Written Description ......................................... 35
`Background of the Technologies Disclosed in the ’244 Patent .................. 36
`A. Technical Overview of Intraoral Scanners .......................................... 36
`1.
`Early Medical Imaging ............................................................. 36
`2.
`Image Stitching and Blending ................................................... 38
`3.
`Image Processing ...................................................................... 39
`4.
`3D Modeling ............................................................................. 40
`5.
`Color 3D Modeling using Intraoral Scanners ........................... 42
`B. Overview of the ’244 Patent ................................................................ 48
`[Intentionally Omitted – Included in PGR2018- 00103 and -00104] ......... 51
`Claim Construction ..................................................................................... 52
`The combinations of (a) Fisker and Szeliski and (b) Fisker and
`Matsumoto render claims 1-5, 7-10, 15, 16, 18, 21, 22, 24, 26, and
`28 obvious. .................................................................................................. 52
`A. Overview of Fisker .............................................................................. 52
`B. Overview of Szeliski ........................................................................... 54
`C. Overview of Matsumoto ...................................................................... 57
`D. Claim 1 ................................................................................................ 58
`1.
`[1.P]: “A focus scanner for recording surface geometry
`and surface color of an object” ................................................. 58
`[1.1]: “a multichromatic light source configured for
`providing a multichromatic probe light for illumination
`of the object.” ............................................................................ 58
`
`2.
`
`- 2 -
`
`
`
`3.
`
`4.
`
`5.
`
`6.
`
`7.
`
`8.
`
`9.
`
`10.
`
`11.
`
`
`
`Petition for IPR2019-00118
`
` U.S. Patent No. 9,962,244
`
`
`[1.2]: “a color image sensor comprising an array of
`image sensor pixels for capturing one or more 2D images
`of light received from said object” ........................................... 59
`[1.3.a]: “wherein the focus scanner is configured to
`operate by translating a focus plane along an optical axis
`of the focus scanner” ................................................................. 60
`[1.3.b]: “wherein the focus scanner is configured to
`operate by…capturing a series of the 2D images, each 2D
`image of the series is at a different focus plane position
`such that the series of captured 2D images forms a stack
`of 2D images” ........................................................................... 61
`[1.4.a]: “a data processing system configured to derive
`surface geometry information for a block of said image
`sensor pixels from the 2D images in the stack of 2D
`images captured by said color image sensor” ........................... 62
`[1.4.b]: “the data processing system also configured to
`derive surface color information for the block of said
`image sensor pixels from at least one of the 2D images
`used to derive the surface geometry information” .................... 65
`[1.5.a]: “wherein the data processing system further is
`configured to combining [sic] a number of sub-scans to
`generate a digital 3D representation of the object” .................. 67
`[1.5.b]: “determining [sic] object color of a least one
`point of the generated digital 3D representation of the
`object from sub-scan color of the sub-scans combined to
`generate the digital 3D representation” .................................... 69
`[1.5.c]: “such that the digital 3D representation expresses
`both geometry and color profile of the object” ......................... 72
`[1.6]: “wherein determining the object color comprises
`computing a weighted average of sub-scan color values
`derived for corresponding points in overlapping sub-
`scans at that point of the object surface.” ................................. 73
`a)
`Fisker .............................................................................. 73
`b)
`Szeliski ............................................................................ 75
`c) Matsumoto ...................................................................... 77
`
`- 3 -
`
`
`
`
`
`Petition for IPR2019-00118
`
` U.S. Patent No. 9,962,244
`
`
`d) Motivation to Combine ................................................... 81
`E. Claim 2: “The focus scanner according to claim 1, wherein the
`data processing system is configured for generating a sub-scan
`of a part of the object surface based on surface geometry
`information and surface color information derived from a
`plurality of blocks of image sensor pixels.” ........................................ 89
`F. Claim 3: “The focus scanner according to claim 1, where the
`scanner system comprises a pattern generating element
`configured for incorporating a spatial pattern in said probe
`light.” ................................................................................................... 90
`G. Claim 4: “The focus scanner according to claim 1, where
`deriving the surface geometry information and surface color
`information comprises calculating for several 2D images a
`correlation measure between the portion of the 2D image
`captured by said block of image sensor pixels and a weight
`function, where the weight function is determined based on
`information of the configuration of the spatial pattern.” .................... 91
`H. Claim 5: “The focus scanner according to claim 4, wherein
`deriving the surface geometry information and the surface color
`information for a block of image sensor pixels comprises
`identifying the position along the optical axis at which the
`corresponding correlation measure has a maximum value.” .............. 94
`I. Claim 7: “The focus scanner according to claim 6, where the
`maximum correlation measure value is the highest calculated
`correlation measure value for the block of image sensor pixels
`and/or the highest maximum value of the correlation measure
`function for the block of image sensor pixels.” .................................. 95
`J. Claim 8: “The focus scanner according to claim 5, wherein the
`data processing system is configured for determining a sub-scan
`color for a point on a generated sub-scan based on the surface
`color information of the 2D image in the series in which the
`correlation measure has its maximum value for the
`corresponding block of image sensor pixels.” .................................... 96
`K. Claim 9: “The focus scanner according to claim 8, wherein the
`data processing system is configured for deriving the sub-scan
`color for a point on a generated sub-scan based on the surface
`color information of the 2D images in the series in which the
`
`- 4 -
`
`
`
`Petition for IPR2019-00118
`
` U.S. Patent No. 9,962,244
`
`
`correlation measure has its maximum value for the
`corresponding block of image sensor pixels and on at least one
`additional 2D image.” ......................................................................... 99
`L. Claim 10: “The focus scanner according to claim 9, where the
`data processing system is configured for interpolating surface
`color information of at least two 2D images in a series when
`determining the sub-scan color.” .......................................................101
`M. Claim 15: “The focus scanner according to claim 1, where the
`color image sensor comprises a color filter array comprising at
`least three types of colors filters, each allowing light in a known
`wavelength range, W1, W2, and W3 respectively, to propagate
`through the color filter.” ....................................................................102
`N. Claim 16: “The focus scanner according to claim 15, where the
`surface geometry information is derived from light in a selected
`wavelength range of the spectrum provided by the
`multichromatic light source.” ............................................................103
`O. Claim 18: “The focus scanner according to claim 16, wherein
`the selected wavelength range matches the W2 wavelength
`range.” ...............................................................................................104
`P. Claim 21: “The focus scanner according to claim 3, where the
`pattern generating element is configured to provide that the
`spatial pattern comprises alternating dark and bright regions
`arranged in a checkerboard pattern.”.................................................104
`Q. Claim 22 ............................................................................................105
`1.
`[22.P]: “A method of recording surface geometry and
`surface color of an object” ......................................................106
`[22.1]: “obtaining a focus scanner according to claim 1.” .....106
`[22.2]: “illuminating the surface of said object with
`multichromatic probe light from said multichromatic
`light source” ............................................................................107
`[22.3]: “capturing a series of 2D images of said object
`using said color image sensor.” ..............................................108
`[22.4]: “deriving both surface geometry information and
`surface color information for a block of image sensor
`pixels at least partly from one captured 2D image.” ..............109
`
`2.
`3.
`
`4.
`
`5.
`
`
`
`- 5 -
`
`
`
`
`
`2.
`
`Petition for IPR2019-00118
`
` U.S. Patent No. 9,962,244
`
`
`R. Claim 24: “The focus scanner according to claim 1, wherein the
`multichromatic light source, the color image sensor, and at least
`a portion of the data processing system are included in a hand
`held unit.” ..........................................................................................112
`S. Claim 26: “The focus scanner according to claim 9, wherein
`said at least one additional 2D image comprises a neighboring
`2D image from the series of captured 2D images.” ..........................114
`T. Claim 28: “The focus scanner according to claim 10, where the
`interpolation is of surface color information of neighboring 2D
`images in a series.” ............................................................................115
`VIII. The combinations of (a) Fisker and Yamada and (b) Fisker and
`Suzuki render claim 29 obvious. ..............................................................116
`A. Overview of Yamada.........................................................................116
`B. Overview of Suzuki ...........................................................................116
`C. Claim 29 ............................................................................................117
`1.
`[29.P]: “A focus scanner for recording surface geometry
`and surface color of an object” ...............................................117
`[29.1]: “a multichromatic light source configured for
`providing a multichromatic probe light for illumination
`of the object” ...........................................................................118
`[29.2]: “a color image sensor comprising an array of
`image sensor pixels for capturing one or more 2D images
`of light received from said object” .........................................119
`[29.3.a-b]: “wherein the focus scanner is configured to
`operate by translating a focus plane along an optical axis
`of the focus scanner and capturing a series of the 2D
`images, each 2D image of the series is at a different focus
`plane position such that the series of captured 2D images
`forms a stack of 2D images” ...................................................120
`[29.4.a-b]: “a data processing system configured to
`derive surface geometry information for a block of said
`image sensor pixels from the 2D images in the stack of
`2D images captured by said color image sensor, the data
`processing system also configured to derive surface color
`information for the block of said image sensor pixels
`
`5.
`
`3.
`
`4.
`
`- 6 -
`
`
`
`
`
`6.
`
`Petition for IPR2019-00118
`
` U.S. Patent No. 9,962,244
`
`
`from at least one of the 2D images used to derive the
`surface geometry information” ...............................................122
`[29.5]: “where the data processing system further is
`configured to detecting saturated pixels in the captured
`2D images and for mitigating or removing the error in the
`derived surface color information or the sub-scan color
`caused by the pixel saturation.” ..............................................125
`a)
`Fisker ............................................................................125
`b)
`Yamada .........................................................................126
`c)
`Suzuki ...........................................................................127
`d) Motivation to Combine .................................................128
`The combinations of (a) Fisker, Szeliski, and Yamada, (b) Fisker,
`Szelski, and Suzuki, (c) Fisker, Matsumoto, and Yamada, and (d)
`Fisker, Matsumoto, and Suzuki render claim 12 obvious. .......................132
`A. Claim 12: “The focus scanner according to claim 1, wherein the
`data processing system is configured for detecting saturated
`pixels in the captured 2D images and for mitigating or
`removing the error in the derived surface color information or
`the sub-scan color caused by the pixel saturation.” ..........................132
`a)
`Fisker ............................................................................133
`b)
`Yamada .........................................................................133
`c)
`Suzuki ...........................................................................134
`d) Motivation to Combine .................................................135
`The combination of Fisker and Tanaka renders claims 31 and 32
`obvious. .....................................................................................................136
`A. Overview of Tanaka ..........................................................................136
`B. Claim 31 ............................................................................................137
`1.
`[31.P]: “A focus scanner for recording surface geometry
`and surface color of an object” ...............................................137
`[31.1]: “a multichromatic light source configured for
`providing a multichromatic probe light for illumination
`of the object” ...........................................................................138
`
`2.
`
`- 7 -
`
`IX.
`
`X.
`
`
`
`
`
`3.
`
`4.
`
`Petition for IPR2019-00118
`
` U.S. Patent No. 9,962,244
`
`
`[31.2.a]: “a color image sensor comprising an array of
`image sensor pixels for capturing one or more 2D images
`of light received from said object” .........................................139
`[31.2.b]: “where the color image sensor comprises a
`color filter array comprising at least three types of colors
`filters, each allowing light in a known wavelength range,
`W1, W2, and W3 respectively, to propagate through the
`color filter” ..............................................................................140
`[31.3.a-b] “wherein the focus scanner is configured to
`operate by translating a focus plane along an optical axis
`of the focus scanner and capturing a series of the 2D
`images, each 2D image of the series is at a different focus
`plane position such that the series of captured 2D images
`forms a stack of 2D images” ...................................................141
`[31.4.a-b]: “a data processing system configured to
`derive surface geometry information for a block of said
`image sensor pixels from the 2D images in the stack of
`2D images captured by said color image sensor, the data
`processing system also configured to derive surface color
`information for the block of said image sensor pixels
`from at least one of the 2D images used to derive the
`surface geometry information” ...............................................142
`[31.5.a]: “where the data processing system further is
`configured to derive the surface geometry information is
`derived from light in a selected wavelength range of the
`spectrum provided by the multichromatic light source” .........147
`[31.5.b]: “where the color filter array is such that its
`proportion of pixels with color filters that match the
`selected wavelength range of the spectrum is larger than
`50%.” .......................................................................................147
`a)
`Fisker ............................................................................148
`b)
`Tanaka ...........................................................................148
`c) Motivation to Combine .................................................149
`C. Claim 32 ............................................................................................152
`1.
`[32.P]: “A focus scanner for recording surface geometry
`and surface color of an object” ...............................................152
`
`5.
`
`6.
`
`7.
`
`8.
`
`- 8 -
`
`
`
`4.
`
`5.
`
`6.
`
`7.
`
`3.
`
`Petition for IPR2019-00118
`
` U.S. Patent No. 9,962,244
`
`
`[32.1]: “a multichromatic light source configured for
`providing a multichromatic probe light for illumination
`of the object” ...........................................................................153
`[32.2]: “a color image sensor comprising an array of
`image sensor pixels for capturing one or more 2D images
`of light received from said object” .........................................154
`[32.3.a-b]: “wherein the focus scanner is configured to
`operate by translating a focus plane along an optical axis
`of the focus scanner and capturing a series of the 2D
`images, each 2D image of the series is at a different focus
`plane position such that the series of captured 2D images
`forms a stack of 2D images” ...................................................155
`[32.4.a-b]: “a data processing system configured to
`derive surface geometry information for a block of said
`image sensor pixels from the 2D images in the stack of
`2D images captured by said color image sensor, the data
`processing system also configured to derive surface color
`information for the block of said image sensor pixels
`from at least one of the 2D images used to derive the
`surface geometry information” ...............................................157
`[32.5.a]: “where the color image sensor comprises a color
`filter array comprising at least three types of colors
`filters, each allowing light in a known wavelength range,
`W1, W2, and W3 respectively, to propagate through the
`color filter” ..............................................................................161
`[32.5.b]: “the filters are arranged in a plurality of cells of
`6×6 color filters, where the color filters in positions (2,2)
`and (5,5) of each cell are of the W1 type, the color filters
`in positions (2,5) and (5,2) are of the W3 type.” ....................162
`a)
`Fisker ............................................................................162
`b)
`Tanaka ...........................................................................162
`c) Motivation to Combine .................................................165
`The combinations of (a) Fisker, Szeliski, and Tanaka and (b) Fisker,
`Matsumoto, and Tanaka render claims 17 and 19 obvious. .....................168
`A. Claim 17: “The focus scanner according to claim 16, where the
`color filter array is such that the proportion of the image sensor
`
`
`
`2.
`
`XI.
`
`- 9 -
`
`
`
`
`
`XII.
`
`2.
`
`Petition for IPR2019-00118
`
` U.S. Patent No. 9,962,244
`
`
`pixels of the color image sensor with color filters that match the
`selected wavelength range of the spectrum is larger than 50%.” .....168
`B. Claim 19: “The focus scanner according to claim 15, wherein
`the color filter array comprises a plurality of cells of 6×6 color
`filters, where the color filters in positions (2,2) and (5,5) of
`each cell are of the W1 type, the color filters in positions (2,5)
`and (5,2) are of the W3 type.” ...........................................................168
`The combinations of (a) Fisker and Suzuki and (b) Fisker and Cai
`render claim 34 obvious. ..........................................................................169
`A. Overview of Cai ................................................................................169
`B. Claim 34 ............................................................................................169
`1.
`[34.P]: “A focus scanner for recording surface geometry
`and surface color of an object” ...............................................170
`[34.1]: “a multichromatic light source configured for
`providing a multichromatic probe light for illumination
`of the object” ...........................................................................170
`[34.2]: “a color image sensor comprising an array of
`image sensor pixels for capturing one or more 2D images
`of light received from said object” .........................................171
`[34.3.a-b]: “wherein the focus scanner is configured to
`operate by translating a focus plane along an optical axis
`of the focus scanner and capturing a series of the 2D
`images, each 2D image of the series is at a different focus
`plane position such that the series of captured 2D images
`forms a stack of 2D images” ...................................................172
`[34.4.a-b]: “a data processing system configured to
`derive surface geometry information for a block of said
`image sensor pixels from the 2D images in the stack of
`2D images captured by said color image sensor, the data
`processing system also configured to derive surface color
`information for the block of said image sensor pixels
`from at least one of the 2D images used to derive the
`surface geometry information” ...............................................174
`[34.4.c]: “where deriving the surface geometry
`information and surface color information comprises
`calculating for several 2D images a correlation measure
`
`5.
`
`3.
`
`4.
`
`6.
`
`- 10 -
`
`
`
`8.
`
`9.
`
`7.
`
`Petition for IPR2019-00118
`
` U.S. Patent No. 9,962,244
`
`
`between the portion of the 2D image captured by said
`block of image sensor pixels and a weight function,
`where the weight function is determined based on
`information of the configuration of the spatial pattern” .........178
`[34.4.d]: “identifying the position along the optical axis
`at which the corresponding correlation measure has a
`maximum value” .....................................................................179
`[34.4.e]: “where the data processing system further is
`configured for determining a sub-scan color for a point
`on a generated sub-scan based on the surface color
`information of the 2D image in the series in which the
`correlation measure has its maximum value for the
`corresponding block of image sensor pixels.” ........................180
`[34.4.f]: “where the data processing system further is
`configured for…computing an averaged sub-scan color
`for a number of points of the sub-scan, where the
`computing comprises an averaging of sub-scan colors of
`surrounding points on the sub-scan.” ......................................183
`a)
`Fisker ............................................................................183
`b)
`Suzuki ...........................................................................185
`c)
`Cai .................................................................................186
`d) Motivation to Combine .................................................187
`XIII. The combinations of (a) Fisker, Szeliski, Suzuki, and Cai and (b)
`Fisker, Matsumoto, Suzuki, and Cai render claim 11 obvious ................191
`A. Claim 11: “The focus scanner according to claim 9, wherein the
`data processing system is configured for computing an
`averaged sub-scan color for a number of points of the sub-scan,
`where the computing comprises an averaging of sub-scan colors
`of different points.” ...........................................................................191
`XIV. The combinations of (a) Thiel425, Thiel576, and Szeliski and (b)
`Thiel425, Thiel576, and Matsumoto render claims 1, 22, and 24
`obvious. .....................................................................................................192
`A. Overview of Thiel425 .......................................................................192
`B. Overview of Thiel576 .......................................................................193
`C. Claim 1 ..............................................................................................194
`
`
`
`- 11 -
`
`
`
`1.
`
`2.
`
`3.
`
`4.
`
`5.
`
`6.
`
`7.
`
`8.
`
`9.
`
`
`
`Petition for IPR2019-00118
`
` U.S. Patent No. 9,962,244
`
`
`[1.P]: “A focus scanner for recording surface geometry
`and surface color of an object” ...............................................194
`a)
`Thiel425 ........................................................................194
`b)
`Thiel576 ........................................................................195
`c) Motivation to Combine .................................................196
`[1.1]: “a multichromatic light source configured for
`providing a multichromatic probe light for illumination
`of the object” ...........................................................................199
`[1.2]: “a color image sensor comprising an array of
`image sensor pixels for capturing one or more 2D images
`of light received from said object” .........................................200
`[1.3.a]: “wherein the focus scanner is configured to
`operate by translating a focus plane along an optical axis
`of the focus scanner” ...............................................................202
`[1.3.b]: “wherein the focus scanner is configured to
`operate by…capturing a series of the 2D images, each 2D
`image of the series is at a different focus plane position
`such that the series of captured 2D images forms a stack
`of 2D images” .........................................................................202
`[1.4.a]: “a data processing system configured to derive
`surface geometry information for a block of said image
`sensor pixels from the 2D images in the stack of 2D
`images captured by said color image sensor” .........................203
`[1.4.b]: “the data processing system also configured to
`derive surface color information for the block of said
`image sensor pixels from at least one of the 2D images
`used to derive the surface geometry information” ..................204
`a)
`Thiel425 ........................................................................204
`b)
`Thiel576 ........................................................................205
`c) Motivation to Combine .................................................206
`[1.5.a]: “wherein the data processing system further is
`configured to combining [sic] a number of sub-scans to
`generate a digital 3D representation of the object” ................206
`[1.5.b]: “wherein the data processing system further is
`configured to … determining [sic] object color of a least
`
`- 12 -
`
`
`
`
`
`10.
`
`Petition for IPR2019-00118
`
` U.S. Patent No. 9,962,244
`
`
`one point of the generated digital 3D representation of
`the object from sub-scan color of the sub-scans combined
`to generate the digital 3D representation, such that the
`digital 3D representation expresses both geometry and
`color profile of the object” ......................................................208
`[1.6]: “wherein determining the object color comprises
`computing a weighted average of sub-scan color values
`derived for corresponding points in overlapping sub-
`scans at that point of the object surface.” ...............................209
`a)
`Thiel425 and Thiel576 ..................................................209
`b)
`Szeliski ..........................................................................209
`c) Matsumoto ....................................................................211
`d) Motivation to Combine .................................................214
`D. Claim 22 ............................................................................................218
`1.
`[22.P]: “A method of recording surface geometry and
`surface color of an object” ......................................................218
`a)
`Thiel425 ........................................................................218
`b)
`Thiel576 ........................................................................219
`c) Motivation to Combine .................................................220
`[22.1]: “obtaining a focus scanner according to claim 1.” .....220
`[22.2]: “illuminating the surface of said object with
`multichromatic probe light from said multichromatic
`light source” ............................................................................221
`[22.3]: “capturing a series of 2D images of said object
`using said color image sensor” ...............................................222
`[22.4]: “deriving both surface geometry information and
`surface color information for a block of image sensor
`pixels at least partly from one captured 2D image.” ..............224
`a)
`Thiel425 ........................................................................224
`b)
`Thiel576 ........................................................................225
`c) Motivation to Combine .................................................226
`
`2.
`3.
`
`4.
`
`5.
`
`- 13 -
`
`
`
`
`
`XV.
`
`Petition for IPR2019-00118
`
` U.S. Patent No. 9,962,244
`
`
`E. Claim 24: “wherein the multichromatic light source, the color
`image sensor, and at least a portion of the data processing
`system are included in a hand held unit.” .........................................226
`The combinations of (a) Thiel425, Thiel576, and Yamada and (b)
`Thiel425, Thiel576, and Suzuki render claim 29 obvious. ......................227
`1.
`[29.P]: “A focus scanner for recording surface geometry
`and surface color of an object” ...............................................227
`a)
`Thiel425 ........................................................................227
`b)
`Thiel576 ........................................................................228
`c) Motivation to Combine .................................................229
`[29.1]: “a multichromatic light source configured for
`providing a multichromatic probe light for illumination
`of the object” ...........................................................................229
`[29.2]: “a color image sensor comprising an array of
`image sensor pixels for capturing one or more 2D images
`of light received from said object” .........................................230
`[29.3.a-b]: “wherein the focus scanner is configured to
`operate by translating a focus plane along an optical axis
`of the focus scanner and capturing a series of the 2D
`images, each 2D image of the series is at a different focus
`plane position such that the series of captured 2D images
`forms a stack of 2D images” ...................................................231
`[29.4.a-b]: “a data processing system configured to
`derive surface geometry information for a block of said
`image sensor pixels from the 2D images in the