`
`
`
`
`
`
`
`
`UNITED STATES PATENT AND TRADEMARK OFFICE
`
`
`
`
`
`
`BEFORE THE PATENT TRIAL AND APPEAL BOARD
`
`
`
`
`
`
`
`ALIGN TECHNOLOGY, INC.
`Petitioner
`
`v.
`
`3SHAPE A/S
`Patent Owner
`
`
`
`Case Nos. PGR2018-00103
`Patent No. 9,962,244
`
`
`
`
`CORRECTED DECLARATION OF DR. CHANDRAJIT L. BAJAJ, PH.D.
`IN SUPPORT OF POST-GRANT REVIEW OF U.S. PATENT NO. 9,962,244
`
`
`
`
`
`
`
`
`
`
`
`
`
`Mail Stop “PATENT BOARD”
`Patent Trial and Appeal Board
`U.S. Patent and Trademark Office
`P.O. Box 1450
`Alexandria, VA 22313-1450
`
`
`
`
`
`
`Align Ex. 1003
`U.S. Patent No. 9,962,244
`
`

`

`
`
`
`I.
`II.
`III.
`
`IV.
`
`V.
`
`VI.
`VII.
`
`
`
`TABLE OF CONTENTS
`
`Introduction ................................................................................................... 1
`Qualifications and Expertise ......................................................................... 5
`Legal Understanding ...................................................................................10
`A. My Understanding of Claim Construction ..........................................10
`B. A Person of Ordinary Skill in the Art .................................................11
`C. My Understanding of Obviousness .....................................................12
`D. My Understanding of Written Description .........................................14
`Background of the Technologies Disclosed in the ’244 Patent ..................15
`A. Technical Overview of Intraoral Scanners ..........................................15
`1.
`Early Medical Imaging .............................................................15
`2.
`Image Stitching and Blending ...................................................17
`3.
`Image Processing ......................................................................18
`4.
`3D Modeling .............................................................................19
`5.
`Color 3D Modeling using Intraoral Scanners ...........................21
`B. Overview of the ’244 Patent ................................................................27
`Claims 19, 25, and 32 lack support in the Provisional Application
`requiring PGR eligibility for the ’244 Patent. ............................................30
`A. Claims 19 and 32 .................................................................................30
`B. Claim 25 ..............................................................................................32
`Claim Construction .....................................................................................34
`The combinations of (a) Fisker and Szeliski and (b) Fisker and
`Matsumoto render claims 1-5, 7-10, 15, 16, 18, 21, 22, 24, 26, and
`28 obvious. ..................................................................................................34
`A. Overview of Fisker ..............................................................................34
`B. Overview of Szeliski ...........................................................................36
`C. Overview of Matsumoto ......................................................................38
`D. Claim 1 ................................................................................................39
`1.
`[1.P]: “A focus scanner for recording surface geometry
`and surface color of an object” .................................................39
`
`
`
`- i -
`
`

`

`
`
`
`
`
`
`2.
`
`3.
`
`4.
`
`5.
`
`6.
`
`7.
`
`8.
`
`9.
`
`10.
`
`11.
`
`[1.1]: “a multichromatic light source configured for
`providing a multichromatic probe light for illumination
`of the object.” ............................................................................40
`[1.2]: “a color image sensor comprising an array of
`image sensor pixels for capturing one or more 2D images
`of light received from said object” ...........................................40
`[1.3.a]: “wherein the focus scanner is configured to
`operate by translating a focus plane along an optical axis
`of the focus scanner” .................................................................41
`[1.3.b]: “wherein the focus scanner is configured to
`operate by…capturing a series of the 2D images, each 2D
`image of the series is at a different focus plane position
`such that the series of captured 2D images forms a stack
`of 2D images” ...........................................................................42
`[1.4.a]: “a data processing system configured to derive
`surface geometry information for a block of said image
`sensor pixels from the 2D images in the stack of 2D
`images captured by said color image sensor” ...........................43
`[1.4.b]: “the data processing system also configured to
`derive surface color information for the block of said
`image sensor pixels from at least one of the 2D images
`used to derive the surface geometry information” ....................46
`[1.5.a]: “wherein the data processing system further is
`configured to combining [sic] a number of sub-scans to
`generate a digital 3D representation of the object” ..................48
`[1.5.b]: “determining [sic] object color of a least one
`point of the generated digital 3D representation of the
`object from sub-scan color of the sub-scans combined to
`generate the digital 3D representation” ....................................51
`[1.5.c]: “such that the digital 3D representation expresses
`both geometry and color profile of the object” .........................53
`[1.6]: “wherein determining the object color comprises
`computing a weighted average of sub-scan color values
`derived for corresponding points in overlapping sub-
`scans at that point of the object surface.” .................................54
`a)
`Fisker ..............................................................................54
`
`- ii -
`
`

`

`
`
`
`
`
`
`b)
`Szeliski ............................................................................57
`c) Matsumoto ......................................................................59
`d) Motivation to Combine ...................................................62
`E. Claim 2: “The focus scanner according to claim 1, wherein the
`data processing system is configured for generating a sub-scan
`of a part of the object surface based on surface geometry
`information and surface color information derived from a
`plurality of blocks of image sensor pixels.” ........................................70
`F. Claim 3: “The focus scanner according to claim 1, where the
`scanner system comprises a pattern generating element
`configured for incorporating a spatial pattern in said probe
`light.” ...................................................................................................71
`G. Claim 4: “The focus scanner according to claim 1, where
`deriving the surface geometry information and surface color
`information comprises calculating for several 2D images a
`correlation measure between the portion of the 2D image
`captured by said block of image sensor pixels and a weight
`function, where the weight function is determined based on
`information of the configuration of the spatial pattern.” ....................72
`H. Claim 5: “The focus scanner according to claim 4, wherein
`deriving the surface geometry information and the surface color
`information for a block of image sensor pixels comprises
`identifying the position along the optical axis at which the
`corresponding correlation measure has a maximum value.” ..............75
`I. Claim 7: “The focus scanner according to claim 6, where the
`maximum correlation measure value is the highest calculated
`correlation measure value for the block of image sensor pixels
`and/or the highest maximum value of the correlation measure
`function for the block of image sensor pixels.” ..................................76
`J. Claim 8: “The focus scanner according to claim 5, wherein the
`data processing system is configured for determining a sub-scan
`color for a point on a generated sub-scan based on the surface
`color information of the 2D image in the series in which the
`correlation measure has its maximum value for the
`corresponding block of image sensor pixels.” ....................................77
`
`- iii -
`
`

`

`
`
`
`
`
`
`K. Claim 9: “The focus scanner according to claim 8, wherein the
`data processing system is configured for deriving the sub-scan
`color for a point on a generated sub-scan based on the surface
`color information of the 2D images in the series in which the
`correlation measure has its maximum value for the
`corresponding block of image sensor pixels and on at least one
`additional 2D image.” .........................................................................80
`L. Claim 10: “The focus scanner according to claim 9, where the
`data processing system is configured for interpolating surface
`color information of at least two 2D images in a series when
`determining the sub-scan color.” .........................................................82
`M. Claim 15: “The focus scanner according to claim 1, where the
`color image sensor comprises a color filter array comprising at
`least three types of colors filters, each allowing light in a known
`wavelength range, W1, W2, and W3 respectively, to propagate
`through the color filter.” ......................................................................83
`N. Claim 16: “The focus scanner according to claim 15, where the
`surface geometry information is derived from light in a selected
`wavelength range of the spectrum provided by the
`multichromatic light source.” ..............................................................84
`O. Claim 18: “The focus scanner according to claim 16, wherein
`the selected wavelength range matches the W2 wavelength
`range.” .................................................................................................85
`P. Claim 21: “The focus scanner according to claim 3, where the
`information of the saturated pixel in the computing of the
`pattern generating element is configured to provide that the
`spatial pattern comprises alternating dark and bright regions
`arranged in a checkerboard pattern.” ..................................................85
`Q. Claim 22 ..............................................................................................86
`1.
`[22.P]: “A method of recording surface geometry and
`surface color of an object” ........................................................87
`[22.1]: “obtaining a focus scanner according to claim 1.” .......87
`[22.2]: “illuminating the surface of said object with
`multichromatic probe light from said multichromatic
`light source” ..............................................................................88
`
`2.
`3.
`
`- iv -
`
`

`

`
`
`
`
`4.
`
`5.
`
`2.
`
`[22.3]: “capturing a series of 2D images of said object
`using said color image sensor.” ................................................89
`[22.4]: “deriving both surface geometry information and
`surface color information for a block of image sensor
`pixels at least partly from one captured 2D image.” ................90
`R. Claim 24: “The focus scanner according to claim 1, wherein the
`multichromatic light source, the color image sensor, and at least
`a portion of the data processing system are included in a hand
`held unit.” ............................................................................................93
`S. Claim 26: “The focus scanner according to claim 9, wherein
`said at least one additional 2D image comprises a neighboring
`2D image from the series of captured 2D images.” ............................95
`T. Claim 28: “The focus scanner according to claim 10, where the
`interpolation is of surface color information of neighboring 2D
`images in a series.” ..............................................................................96
`VIII. The combinations of (a) Fisker and Yamada and (b) Fisker and
`Suzuki render claim 29 obvious. ................................................................97
`A. Overview of Yamada...........................................................................97
`B. Overview of Suzuki .............................................................................97
`C. Claim 29 ..............................................................................................98
`1.
`[29.P]: “A focus scanner for recording surface geometry
`and surface color of an object” .................................................98
`[29.1]: “a multichromatic light source configured for
`providing a multichromatic probe light for illumination
`of the object” .............................................................................99
`[29.2]: “a color image sensor comprising an array of
`image sensor pixels for capturing one or more 2D images
`of light received from said object” .........................................100
`[29.3.a-b]: “wherein the focus scanner is configured to
`operate by translating a focus plane along an optical axis
`of the focus scanner and capturing a series of the 2D
`images, each 2D image of the series is at a different focus
`plane position such that the series of captured 2D images
`forms a stack of 2D images” ...................................................101
`
`3.
`
`4.
`
`
`
`- v -
`
`

`

`
`
`
`
`6.
`
`5.
`
`[29.4.a-b]: “a data processing system configured to
`derive surface geometry information for a block of said
`image sensor pixels from the 2D images in the stack of
`2D images captured by said color image sensor, the data
`processing system also configured to derive surface color
`information for the block of said image sensor pixels
`from at least one of the 2D images used to derive the
`surface geometry information” ...............................................103
`[29.5]: “where the data processing system further is
`configured to detecting saturated pixels in the captured
`2D images and for mitigating or removing the error in the
`derived surface color information or the sub-scan color
`caused by the pixel saturation.” ..............................................106
`a)
`Fisker ............................................................................106
`b)
`Yamada .........................................................................107
`c)
`Suzuki ...........................................................................108
`d) Motivation to Combine .................................................109
`The combinations of (a) Fisker, Szeliski, and Yamada, (b) Fisker,
`Szelski, and Suzuki, (c) Fisker, Matsumoto, and Yamada, and (d)
`Fisker, Matsumoto, and Yamada render claim 12 obvious. .....................113
`A. Claim 12: “The focus scanner according to claim 1, wherein the
`data processing system is configured for detecting saturated
`pixels in the captured 2D images and for mitigating or
`removing the error in the derived surface color information or
`the sub-scan color caused by the pixel saturation.” ..........................113
`a)
`Fisker ............................................................................114
`b)
`Yamada .........................................................................114
`c)
`Suzuki ...........................................................................115
`d) Motivation to Combine .................................................116
`The combination of Fisker and Tanaka renders claims 31 and 32
`obvious. .....................................................................................................117
`A. Overview of Tanaka ..........................................................................117
`B. Claim 31 ............................................................................................118
`
`- vi -
`
`IX.
`
`X.
`
`
`
`

`

`
`
`
`
`1.
`
`2.
`
`3.
`
`4.
`
`5.
`
`6.
`
`7.
`
`8.
`
`
`
`[31.P]: “A focus scanner for recording surface geometry
`and surface color of an object” ...............................................118
`[31.1]: “a multichromatic light source configured for
`providing a multichromatic probe light for illumination
`of the object” ...........................................................................119
`[31.2.a]: “a color image sensor comprising an array of
`image sensor pixels for capturing one or more 2D images
`of light received from said object” .........................................120
`[31.2.b]: “where the color image sensor comprises a
`color filter array comprising at least three types of colors
`filters, each allowing light in a known wavelength range,
`W1, W2, and W3 respectively, to propagate through the
`color filter” ..............................................................................121
`[31.3.a-b] “wherein the focus scanner is configured to
`operate by translating a focus plane along an optical axis
`of the focus scanner and capturing a series of the 2D
`images, each 2D image of the series is at a different focus
`plane position such that the series of captured 2D images
`forms a stack of 2D images” ...................................................122
`[31.4.a-b]: “a data processing system configured to
`derive surface geometry information for a block of said
`image sensor pixels from the 2D images in the stack of
`2D images captured by said color image sensor, the data
`processing system also configured to derive surface color
`information for the block of said image sensor pixels
`from at least one of the 2D images used to derive the
`surface geometry information” ...............................................123
`[31.5.a]: “where the data processing system further is
`configured to derive the surface geometry information is
`derived from light in a selected wavelength range of the
`spectrum provided by the multichromatic light source”.........128
`[31.5.b]: “where the color filter array is such that its
`proportion of pixels with color filters that match the
`selected wavelength range of the spectrum is larger than
`50%.” .......................................................................................129
`a)
`Fisker ............................................................................129
`
`- vii -
`
`

`

`
`
`
`
`b)
`Tanaka ...........................................................................129
`c) Motivation to Combine .................................................130
`C. Claim 32 ............................................................................................133
`1.
`[32.P]: “A focus scanner for recording surface geometry
`and surface color of an object” ...............................................133
`[32.1]: “a multichromatic light source configured for
`providing a multichromatic probe light for illumination
`of the object.” ..........................................................................134
`[32.2]: “a color image sensor comprising an array of
`image sensor pixels for capturing one or more 2D images
`of light received from said object” .........................................135
`[32.3.a-b]: “wherein the focus scanner is configured to
`operate by translating a focus plane along an optical axis
`of the focus scanner and capturing a series of the 2D
`images, each 2D image of the series is at a different focus
`plane position such that the series of captured 2D images
`forms a stack of 2D images” ...................................................136
`[32.4.a-b]: “a data processing system configured to
`derive surface geometry information for a block of said
`image sensor pixels from the 2D images in the stack of
`2D images captured by said color image sensor, the data
`processing system also configured to derive surface color
`information for the block of said image sensor pixels
`from at least one of the 2D images used to derive the
`surface geometry information” ...............................................138
`[32.5.a]: “where the color image sensor comprises a color
`filter array comprising at least three types of colors
`filters, each allowing light in a known wavelength range,
`W1, W2, and W3 respectively, to propagate through the
`color filter” ..............................................................................142
`[32.5.b]: “the filters are arranged in a plurality of cells of
`6×6 color filters, where the color filters in positions (2,2)
`and (5,5) of each cell are of the W1 type, the color filters
`in positions (2,5) and (5,2) are of the W3 type.” ....................143
`a)
`Fisker ............................................................................143
`b)
`Tanaka ...........................................................................143
`
`5.
`
`6.
`
`7.
`
`2.
`
`3.
`
`4.
`
`
`
`- viii -
`
`

`

`
`
`
`
`XI.
`
`XII.
`
`2.
`
`3.
`
`c) Motivation to Combine .................................................146
`The combinations of (a) Fisker, Szeliski, and Tanaka and (b) Fisker,
`Matsumoto, and Tanaka render claims 17 and 19 obvious. .....................149
`A. Claim 17: “The focus scanner according to claim 16, where the
`color filter array is such that the proportion of the image sensor
`pixels of the color image sensor with color filters that match the
`selected wavelength range of the spectrum is larger than 50%.” .....149
`B. Claim 19: “The focus scanner according to claim 15, wherein
`the color filter array comprises a plurality of cells of 6×6 color
`filters, where the color filters in positions (2,2) and (5,5) of
`each cell are of the W1 type, the color filters in positions (2,5)
`and (5,2) are of the W3 type.” ...........................................................149
`The combinations of (a) Fisker and Suzuki and (b) Fisker and Cai
`render claim 34 obvious. ..........................................................................150
`A. Overview of Cai ................................................................................150
`B. Claim 34 ............................................................................................150
`1.
`[34.P]: “A focus scanner for recording surface geometry
`and surface color of an object” ...............................................150
`[34.1]: “a multichromatic light source configured for
`providing a multichromatic probe light for illumination
`of the object” ...........................................................................151
`[34.2]: “a color image sensor comprising an array of
`image sensor pixels for capturing one or more 2D images
`of light received from said object” .........................................152
`[34.3.a-b]: “wherein the focus scanner is configured to
`operate by translating a focus plane along an optical axis
`of the focus scanner and capturing a series of the 2D
`images, each 2D image of the series is at a different focus
`plane position such that the series of captured 2D images
`forms a stack of 2D images” ...................................................153
`[34.4.a-b]: “a data processing system configured to
`derive surface geometry information for a block of said
`image sensor pixels from the 2D images in the stack of
`2D images captured by said color image sensor, the data
`processing system also configured to derive surface color
`information for the block of said image sensor pixels
`
`4.
`
`5.
`
`
`
`- ix -
`
`

`

`7.
`
`8.
`
`9.
`
`
`
`
`
`6.
`
`from at least one of the 2D images used to derive the
`surface geometry information” ...............................................155
`[34.4.c]: “where deriving the surface geometry
`information and surface color information comprises
`calculating for several 2D images a correlation measure
`between the portion of the 2D image captured by said
`block of image sensor pixels and a weight function,
`where the weight function is determined based on
`information of the configuration of the spatial pattern” .........159
`[34.4.d]: “identifying the position along the optical axis
`at which the corresponding correlation measure has a
`maximum value” .....................................................................160
`[34.4.e]: “where the data processing system further is
`configured for determining a sub-scan color for a point
`on a generated sub-scan based on the surface color
`information of the 2D image in the series in which the
`correlation measure has its maximum value for the
`corresponding block of image sensor pixels.” ........................161
`[34.4.f]: “where the data processing system further is
`configured for…computing an averaged sub-scan color
`for a number of points of the sub-scan, where the
`computing comprises an averaging of sub-scan colors of
`surrounding points on the sub-scan.” ......................................164
`a)
`Fisker ............................................................................164
`b)
`Suzuki ...........................................................................166
`c)
`Cai .................................................................................167
`d) Motivation to Combine .................................................168
`XIII. The combinations of (a) Fisker, Szeliski, Suzuki, and Cai and (b)
`Fisker, Matsumoto, Suzuki, and Cai render claim 11 obvious ................172
`A. Claim 11: “The focus scanner according to claim 9, wherein the
`data processing system is configured for computing an
`averaged sub-scan color for a number of points of the sub-scan,
`where the computing comprises an averaging of sub-scan colors
`of different points.” ...........................................................................172
`
`
`
`- x -
`
`

`

`
`
`
`
`XIV. The combinations of (a) Thiel425, Thiel576, and Szeliski and (b)
`Thiel425, Thiel576, and Matsumoto render claims 1, 22, and 24
`obvious. .....................................................................................................173
`A. Overview of Thiel425 .......................................................................173
`B. Overview of Thiel576 .......................................................................174
`C. Claim 1 ..............................................................................................175
`1.
`[1.P]: “A focus scanner for recording surface geometry
`and surface color of an object” ...............................................175
`a)
`Thiel425 ........................................................................175
`b)
`Thiel576 ........................................................................176
`c) Motivation to Combine .................................................177
`[1.1]: “a multichromatic light source configured for
`providing a multichromatic probe light for illumination
`of the object” ...........................................................................180
`[1.2]: “a color image sensor comprising an array of
`image sensor pixels for capturing one or more 2D images
`of light received from said object” .........................................181
`[1.3.a]: “wherein the focus scanner is configured to
`operate by translating a focus plane along an optical axis
`of the focus scanner” ...............................................................183
`[1.3.b]: “wherein the focus scanner is configured to
`operate by…capturing a series of the 2D images, each 2D
`image of the series is at a different focus plane position
`such that the series of captured 2D images forms a stack
`of 2D images” .........................................................................184
`[1.4.a]: “a data processing system configured to derive
`surface geometry information for a block of said image
`sensor pixels from the 2D images in the stack of 2D
`images captured by said color image sensor” .........................184
`[1.4.b]: “the data processing system also configured to
`derive surface color information for the block of said
`image sensor pixels from at least one of the 2D images
`used to derive the surface geometry information” ..................186
`a)
`Thiel425 ........................................................................186
`
`2.
`
`3.
`
`4.
`
`5.
`
`6.
`
`7.
`
`
`
`- xi -
`
`

`

`
`
`
`
`
`
`8.
`
`9.
`
`b)
`Thiel576 ........................................................................186
`c) Motivation to Combine .................................................187
`[1.5.a]: “wherein the data processing system further is
`configured to combining [sic] a number of sub-scans to
`generate a digital 3D representation of the object” ................187
`[1.5.b]: “wherein the data processing system further is
`configured to … determining [sic] object color of a least
`one point of the generated digital 3D representation of
`the object from sub-scan color of the sub-scans combined
`to generate the digital 3D representation, such that the
`digital 3D representation expresses both geometry and
`color profile of the object” ......................................................188
`[1.6]: “wherein determining the object color comprises
`computing a weighted average of sub-scan color values
`derived for corresponding points in overlapping sub-
`scans at that point of the object surface.” ...............................190
`a)
`Thiel425 and Thiel576 ..................................................190
`b)
`Szeliski ..........................................................................190
`c) Matsumoto ....................................................................192
`d) Motivation to Combine .................................................195
`D. Claim 22 ............................................................................................199
`1.
`[22.P]: “A method of recording surface geometry and
`surface color of an object” ......................................................199
`a)
`Thiel425 ........................................................................199
`b)
`Thiel576 ........................................................................200
`c) Motivation to Combine .................................................201
`[22.1]: “obtaining a focus scanner according to claim 1.” .....201
`[22.2]: “illuminating the surface of said object with
`multichromatic probe light from said multichromatic
`light source” ............................................................................202
`[22.3]: “capturing a series of 2D images of said object
`using said color image sensor” ...............................................203
`
`10.
`
`2.
`3.
`
`4.
`
`- xii -
`
`

`

`
`
`
`
`5.
`
`2.
`
`XV.
`
`[22.4]: “deriving both surface geometry information and
`surface color information for a block of image sensor
`pixels at least partly from one captured 2D image.” ..............205
`a)
`Thiel425 ........................................................................205
`b)
`Thiel576 ........................................................................206
`c) Motivation to Combine .................................................207
`E. Claim 24: “wherein the multichromatic light source, the color
`image sensor, and at least a portion of the data processing
`system are included in a hand held unit.” .........................................207
`The combinations of (a) Thiel425, Thiel576, and Yamada and (b)
`Thiel425, Thiel576, and Suzuki render claim 29 obvious. ......................208
`1.
`[29.P]: “A focus scanner for recording surface geometry
`and surface color of an object” ...............................................208
`a)
`Thiel425 ........................................................................208
`b)
`Thiel576 ........................................................................209
`c) Motivation to Combine .................................................210
`[29.1]: “a multichromatic light source configured for
`providing a multichromatic probe light for illumination
`of the object” ...........................................................................210
`[29.2]: “a color image sensor comprising an array of
`image sensor pixels for capturing one or more 2D images
`of light received from said object” .........................................211
`[29.3.a-b]: “wherein the focus scanner is configured to
`operate by translating a focus plane along an optical axis
`of the focus scanner and capturing a series of the 2D
`images, each 2D image of the series is at a different focus
`plane position such that the series of captured 2D images
`forms a stack of 2D images” ...................................................212
`[29.4.a-b]: “a data processing system configured to
`derive surface geometry information for a block of said
`image sensor pixels from the 2D images in the stack of
`2D images captured by said color image sensor, the data
`processing system also configured to derive surface color
`information for the block of said image sensor pixels
`
`3.
`
`4.
`
`5.
`
`
`
`- xiii -
`
`

`

`
`
`
`
`6.
`
`2.
`
`from at least one of the 2D images used to derive the
`surface geometry information” ..........................