UNITED STATES PATENT AND TRADEMARK OFFICE
`____________________
`
`BEFORE THE PATENT TRIAL AND APPEAL BOARD
`
`____________________
`
`SAMSUNG ELECTRONICS CO., LTD. AND
`SAMSUNG ELECTRONICS AMERICA, INC.
`Petitioner
`
`v.
`
`IMAGE PROCESSING TECHNOLOGIES LLC,
`Patent Owner
`____________________
`
`CASE IPR2017-01190
`Patent No. 6,717,518
`____________________
`
`IMAGE PROCESSING TECHNOLOGIES LLC’S
`PATENT OWNER RESPONSE PURSUANT TO 37 C.F.R. § 42.120
`
`
`
`
`
`
`
`

`

`IPR2017-01190 (’518 Patent) Patent Owner Response
`
`TABLE OF CONTENTS
`
`I.
`
`II.
`
`Introduction ...................................................................................................... 1
`
`Summary .......................................................................................................... 3
`
`III. Constitutionality of Inter Partes Review ........................................................ 5
`
`IV. Technical Background ..................................................................................... 5
`
`A. Anatomy of the Eye ............................................................................... 5
`
`B.
`
`Facial Detection ..................................................................................... 7
`
`V. Overview of the ’518 Patent ............................................................................ 8
`
`A. Histogram Calculation Based on Image Data ..................................... 10
`
`B.
`
`Detection of Eye Area—Two Preferred Embodiments ...................... 16
`
`1. Detection of Eye Area Based on Head Frame .............................. 17
`
`2. Detection of Eye Area Based on Location of a Facial
`Characteristic ................................................................................ 21
`
`C.
`
`D.
`
`Claim 39 .............................................................................................. 27
`
`Claim Construction ............................................................................. 28
`
`1. “histogram” ................................................................................... 28
`
`2. “anthropomorphic model” ............................................................ 31
`
`3. “characteristic of the face” / “facial characteristic” ...................... 32
`
`4. “Selecting pixels of the portion of the image having
`characteristics corresponding to the feature to be detected;
`forming at least one histogram of the selected pixels” ................. 34
`
`VI. Legal Standards ............................................................................................. 41
`
`VII. Claim 39 Is Not Obvious Over the Asserted Prior Art Combinations .......... 44
`
`A.
`
`Eriksson (Ex. 1005) in View of Stringa (Ex. 1006) Does Not
`Teach or Suggest All Elements of Claim 39. ...................................... 45
`
`i
`
`

`

`IPR2017-01190 (’518 Patent) Patent Owner Response
`
`B.
`
`C.
`
`1. Neither Reference Teaches or Suggests a “Histogram” ............... 45
`
`2. Eriksson in Combination with Stringa Does Not Teach or
`Suggest a Histogram “of the Selected Pixels”. ............................. 48
`
`A POSA Would Not Have Selected and Combined Eriksson and
`Stringa in the Manner Proposed by Petitioner to Achieve the
`Claimed Invention. .............................................................................. 52
`
`Ando (Exhibit 1009) in View of Suenaga (Exhibit 1007) Does Not
`Teach or Suggest All Elements of Claim 39. ...................................... 57
`
`1. Neither References Teaches or Suggests Forming a Histogram
`“of the Selected Pixels”. ............................................................... 57
`
`2. Suenaga Does Not Teach or Suggest Identifying a Facial
`Characteristic Other than the Feature to Be Detected. ................. 66
`
`D. A POSA Would Not Have Selected and Combined Ando and
`Suenaga in the Manner Proposed by Petitioner to Achieve the
`Claimed Invention. .............................................................................. 66
`
`VIII. Conclusion ..................................................................................................... 74
`
`
`
`
`
`
`
`ii
`
`

`

`IPR2017-01190 (’518 Patent) Patent Owner Response
`
`TABLE OF AUTHORITIES
`
`Cases
`
`Apple Inc. v. Contentguard Holdings, Inc.,
` IPR2015-00442, Paper 9 (P.T.A.B. July 13, 2015) ............................... 41, 43, 44
`
`Google, Inc. v. Everymd.com LLC,
`IPR2014-00347, Paper 9 (P.T.A.B. May 22, 2014) ............................................. 42
`
`Grain Processing v. American-Maize Prods,
` 840 F.2d 902 (Fed. Cir. 1988) ............................................................................. 45
`
`In re Magnum Oil Tools Int’l.,
` 829 F.3d 1364 (Fed. Cir. 2016) .................................................................... 41, 43
`
`In re NTP, Inc.,
` 654 F.3d 1279 (Fed. Cir. 2011) ........................................................................... 45
`
`In re Omeprazole Patent Litigation,
` 536 F.3d 1361 (Fed. Cir. 2008) ........................................................................... 45
`
`InTouch Tech., Inc. v. VGo Communs., Inc.,
` 751 F.3d 1327 (Fed. Cir. 2014) ........................................................................... 45
`
`Kinetic Concepts, Inc. v. Smith & Nephew, Inc.,
` 688 F.3d 1342 (Fed. Cir. 2012) ........................................................................... 42
`
`KSR Int’l Co. v. Teleflex Inc.,
`550 U.S. 398 (2007) ................................................................................ 44, 45, 46
`
`Liberty Mut. Ins. Co. v. Progressive Cas. Ins. Co.,
` CBM-2012-00003, Paper 7 (P.T.A.B. Nov. 26, 2012) ................................ 42, 43
`
`Ortho-McNeil Pharm. v. Mylan Labs,
` 520 F.3d 1358 (Fed. Cir. 2008) ........................................................................... 45
`
`Proctor & Gamble Co. v. Teva Pharm. USA, Inc.,
` 566 F.3d 989 (Fed. Cir. 2009) ...................................................................... 43, 44
`
`Rolls–Royce PLC v. United Techs. Corp.,
`603 F.3d 1325, 1339 (Fed. Cir. 2010) .................................................................. 54
`
`iii
`
`

`

`IPR2017-01190 (’518 Patent) Patent Owner Response
`
`Unigene Labs., Inc. v. Apotex, Inc.,
` 655 F.3d 1352 (Fed. Cir. 2011) .................................................................... 44, 45
`
`Whole Space Indus Ltd. v. Zipshade Indus.,
` IPR2015-00488, Paper 14 (P.T.A.B. July 24, 2015) .......................................... 42
`
`Statutes
`
`35 U.S.C. § 103 ........................................................................................................ 41
`
`35 U.S.C. § 314 ........................................................................................................ 41
`
`
`
`
`
`
`
`iv
`
`

`

`IPR2017-01190 (’518 Patent) Patent Owner Response
`
`TABLE OF EXHIBITS
`
`Exhibit
`
`Ex. 2001. ............
`
`Description
`Claim construction opinion in the
`Image Processing Technologies,
`LLC v. Samsung Elecs. Co., et al.,
`No. 16-cv-00505-JRG (E.D. Tex.),
`dated June 21, 2017.
`Ex. 2002. ............ The American Heritage College
`Dictionary (1997)
`Ex. 2003. ............ Dec. 22 2017 Deposition Transcript,
`Dr. John Hart
`Declaration of Michael N. Zachary
`In Support of Patent Owner’s
`Motion for Pro Hac Vice
`Ex. 2005. ............ (Not used)
`Ex. 2006. ............ (Not used)
`Ex. 2007. ............ (Not used)
`Ex. 2008. ............ (Not used)
`Ex. 2009. ............ (Not used)
`Ex. 2010. ............ (Not used)
`Ex. 2011. ............ (Not used)
`Website printout, Parallel
`Computing Institute, Dr. John C.
`Hart
`Ex. 2013. ............ Website printout, Computer
`Graphics Illinois, Dr. John C. Hart
`Ex. 2014. ............ Hand-drawn sketch
`Ex. 2015. ............ Diagram of Eye
`
`Ex. 2004. ............
`
`Ex. 2012. ............
`
`
`
`
`
`v
`
`

`

`IPR2017-01190 (’518 Patent) Patent Owner Response
`
`Patent Owner Image Processing Technologies LLC (“Patent Owner”) hereby
`
`submits this Patent Owner Response to the Petition filed by Samsung Electronics
`
`Co. Ltd. and Samsung Electronics America, Inc. (collectively, “Petitioner”). On
`
`October 3, 2017, the Board instituted this inter partes review no. IPR2017-01190
`
`for claim 39 on only two grounds: whether claim 39 of U.S. Patent No. 6,717,518
`
`(the “’518 patent”) is unpatentable under 35 U.S. C. § 103 as obvious over:
`
`•
`
`•
`
`Ground 1: Eriksson (Ex. 1005) in light of Stringa (Ex. 1006);
`
`Ground 2: Ando (Ex. 1009) in light of Suenaga (Ex. 1007).
`
`No other grounds were authorized for this inter partes review. Paper 11
`
`at 26.
`
`I.
`
`INTRODUCTION
`
`The construction of the following language of claim 39 is, if interpreted as
`
`suggested by Patent Owner, dispositive of this inter partes review:
`
`[d] selecting pixels of the portion of the image having
`characteristics corresponding
`to
`the feature
`to be
`detected;
`[e] forming at least one histogram of the selected pixels;
`
`At issue is whether the claim requires that a histogram be formed that includes
`
`only values for pixels that have characteristics of the feature to be detected, as
`
`advocated by Patent Owner, or whether a histogram of pixels in an area of an
`
`image (such as a rectangular area) also satisfy the claim.
`
`1
`
`

`

`IPR2017-01190 (’518 Patent) Patent Owner Response
`
`Although the Board determined for purposes of institution that the claim
`
`does not require a histogram formed of selected pixels “having characteristics
`
`corresponding to the feature” to be limited to only those pixels having
`
`characteristics of the feature, Paper 11 at 16, Patent Owner respectfully submits
`
`that the language of the claim in light of the specification requires this limitation,
`
`see Section V.D.4, and, importantly, that the portions of the specification that the
`
`Board cited in reaching its preliminary conclusion support Patent Owner’s
`
`position. The cited portions teach the formation of a histogram of only the pixels
`
`that have a characteristic (DP=1, indicating a changing pixel value over time) of a
`
`feature (an eyelid). Ex. 1001, 27:58–59 (cited on page 16 of Institution Decision).
`
`The ’518 patent figure cited by the Board also shows histograms of only “pixels
`
`with selected criteria corresponding to the driver’s eyelids,” the selected criteria
`
`being a combination of parameter values of DP=1 and downward movement
`
`direction (DI value).1 Ex. 1001, 28:15–18.
`
`Patent Owner also respectfully submits that construing the term “histogram”
`
`will resolve Ground 1 because, under any reasonable interpretation of the term,
`
`
`1 The parameter DP indicates whether the intensity value for with the pixel is
`significantly different from its intensity value for the last frame (DP=1) or not
`(DP=0). Ex. 1001, 7:59–8:21. The parameter DI represents direction of
`movement, which is represented in Freemen code (numbers 0 through 7). Id.,
`11:49–63, 12:1–5, 13:7–19, 14:50–54. Both DP and DI are determined by
`analyzing a matrix of pixels surrounding a particular pixel. Id., 11:55–13:41. See
`id., Figs. 4, 6–7 and 9.
`
`2
`
`

`

`IPR2017-01190 (’518 Patent) Patent Owner Response
`
`Eriksson and Stringa do not each or suggest a histogram. (See Section V.D.1).
`
`Patent Owner further submits that construing the “facial characteristic” term will
`
`assist in resolving Samsung’s challenge because Samsung’s application of the term
`
`to Eriksson and Suenaga is overly broad. (See Section V.D.3).
`
`Patent Owner respectfully requests that the Board construe this claim
`
`language, because doing so would resolve Petitioner’s challenge to claim 39.2
`
`II.
`
`SUMMARY
`
`Based on the language of claim 39, interpreted in light of the specification,
`
`claim 39 is not obvious over either asserted prior art combination.
`
`As to ground 1, Eriksson and Stringa do not teach or suggest elements [d],
`
`[e], [f], and [g]. First, the references do not disclosure “histograms” as required by
`
`element [e], but instead merely disclose the use of intensity values for certain
`
`horizontal lines within the image.3 Second, even if the disclosures are interpreted
`
`as histograms, the histograms are formed of all pixels in a particular line, and not
`
`
`2 For clarity of these proceedings, Patent Owner also submits a definition of
`“anthropomorphic model” that was previously agreed to in the Litigation. (Section
`V.D.2).
`3 Patent Owner did not to raise all of the arguments asserted here in its Preliminary
`Response. Patent Owner therefore did not contest the “histogram” element in that
`paper. Patent Owner’s use of the term “histogram” in the Preliminary Response
`was not an admission or waiver as to this claim element. Patent Owner properly
`raises this argument in its Patent Owner response. See D.I. 12 at 3 (“any arguments
`for patentability not raised in the response will be deemed waived.”).
`
`3
`
`

`

`IPR2017-01190 (’518 Patent) Patent Owner Response
`
`of “selected pixels” that have “characteristics corresponding to the feature to be
`
`detected” as required by the claim (elements [d], [e], and [g]).
`
`A POSA would not have combined the references to achieve claim 39.
`
`Eriksson does not identify a first facial feature as required by the claim and thus
`
`fails to disclose element [b]. If a POSA were to combine the two references, the
`
`POSA would have used the simpler method of finding a facial center that fails to
`
`disclose this element [b], and thus the POSA would not have arrived at the
`
`invention of claim 39.
`
`As to Ground 2, Ando and Suenaga do not teach or suggest forming a
`
`histogram of “selected pixels” that have “characteristics corresponding to the
`
`feature to be detected” as required by the claim as required by elements [d], [e], [f],
`
`and [g]. Ando discloses an intensity histogram of all pixels in a rectangular area.
`
`And Suenaga discloses x and y histograms of an entire eye area to analyze shape of
`
`the full eye area, not, as the claim would require, an iris, pupil, or cornea. Ex.
`
`2003, 174:23–175:7.
`
`A POSA would not have combined the references to achieve claim 39. A
`
`POSA would not look to Suenaga to provide increased accuracy for Ando as
`
`suggested by Petitioner because Ando already discloses an asserted improved
`
`method that is asserted to be accurate.
`
`4
`
`

`

`IPR2017-01190 (’518 Patent) Patent Owner Response
`
`Therefore, Claim 39 is not obvious over either of the asserted grounds and
`
`the Board should hold that the claims are patentable.
`
`III. CONSTITUTIONALITY OF INTER PARTES REVIEW
`Patent Owner challenges the constitutionality of this proceeding for the
`
`reasons asserted in Oil States Energy Svcs. v. Greene’s Energy Group, No. 16-712
`
`(cert. granted June 12, 2017). Inter partes review proceedings violate the
`
`Constitution by extinguishing private property rights through a non-Article III
`
`forum without a jury.
`
`IV. TECHNICAL BACKGROUND
`A. Anatomy of the Eye
`The ’518 patent distinguishes between the pupil, iris, or cornea and the
`
`overall shape of the eye. Ex. 1001, 30:51–59. Claim 39 recites the selection of
`
`pixels having characteristics of the feature to be detected, such feature being the
`
`iris, pupil, or cornea.
`
`The pupil is black, round, and at the center of the eye. The pupil is
`
`surrounded by the iris, which may be blue, brown, or other colors. The cornea is
`
`the clear covering over the iris-pupil area. The below diagram taken from the
`
`website of the American Academy of Ophthalmology shows these parts of the
`
`5
`
`

`

`IPR2017-01190 (’518 Patent) Patent Owner Response
`
`eye.4 Dr. Hart did not dispute the accuracy of the below diagrams. Ex. 2003,
`
`132:4-133:19; Ex. 2015 (eye diagrams)
`
`The cornea is distinct from the sclera, as shown below:
`
`
`
`
`
`
`4 Downloaded from the web address https://www.aao.org/eye-
`health/anatomy/parts-of-eye on December 7, 2017.
`
`6
`
`

`

`IPR2017-01190 (’518 Patent) Patent Owner Response
`
`In the above graphic,5 the cornea is light blue, while the sclera is purple. As
`
`shown in the graphic, much of the visible eye surface is the sclera, or the white of
`
`the eye:
`
`
`
`Facial Detection
`
`B.
`At the time of the invention, there were many possible algorithms for
`
`detection of faces and eyes, and it was “rather difficult to assess the state of the
`
`art.” Ex. 1006, 4. As of 1993, numerous applications had been developed based
`
`on “various techniques,” including “template matching,” “isodensity maps,” and
`
`“feature extraction by neural and Hopfield-type networks.” Ex. 1006, 4. A 1991
`
`evaluation of the state of the art reported that “the best results were obtained with a
`
`template matching approach,” (id.), which is not the technique of claim 39 of the
`
`’518 patent. Eriksson also explained that a “popular” technique was based on
`
`template-matching using deformable templates. Ex. 1005, 5. Eriksson noted that
`
`as of 1997, “[m]any methods have been proposed for localizing facial features in
`
`images.” Ex. 1005, 5 (right column at bottom). It was also understood that
`
`5 Downloaded from the web address https://en.wikipedia.org/wiki/Sclera
`(Wikipedia entry for Sclera) on December 7, 2017.
`
`7
`
`

`

`IPR2017-01190 (’518 Patent) Patent Owner Response
`
`“[v]arious approaches can be used for the purpose of locating the position of the
`
`eye.” Ex. 1006, 8. One example is eye template matching. Id.
`
`V. OVERVIEW OF THE ’518 PATENT
`The ’518 patent is directed to the novel use of histograms to detect and
`
`analyze faces and eyes. As stated in the ’518 patent Abstract:
`
`In a process of detecting a person falling asleep, an image
`of the face of the person is acquired. Pixels of the image
`having characteristics corresponding to an eye of the
`person are selected and a histogram is formed of the
`selected pixels. The histogram is analyzed over time to
`identify each opening and closing of the eye, and
`characteristics indicative of the person falling asleep are
`determined. A sub-area of the image including the eye
`may be determined by identifying the head or a facial
`characteristic of the person, and then identifying the sub-
`area using an anthropomorphic model. To determine
`openings and closings of the eyes, histograms of
`shadowed pixels of the eye are analyzed to determine the
`width and height of the shadowing, or histograms of
`movement corresponding to blinking are analyzed.
`
`Ex. 1001, 1 (Abstract).
`
`In order to form a histogram of pixels having characteristics corresponding
`
`to characteristics of at least one eye, the ’518 patent teaches two preferred methods
`
`of efficiently finding the eye area—one method where the entire head is located,
`
`8
`
`

`

`IPR2017-01190 (’518 Patent) Patent Owner Response
`
`and a second method where a specific facial feature is located. The two methods
`
`operate differently. The head-location method identifies a rectangle for the head
`
`area based on a general characteristic of the head, such as skin color or the fact that
`
`the head will likely be moving. The head-location method applies an
`
`anthropomorphic ratio to the head rectangle to find an eye area.
`
`The facial-feature method works based on a particular property of a specific
`
`facial feature. For example, nostrils are dark. The method may identify dark
`
`pixels and form a histogram of those dark pixels, and then check for spacing and
`
`dimensions of nostrils by analyzing pixels to distinguish dark nostril pixels from,
`
`say, black hair or pupil pixels, as shown in Fig. 32. Ex. 1001, 30:1–2.
`
`Ex. 1001, Figure 32
`
`
`
`An anthropomorphic model is then used to locate the eye area based on the
`
`expected location of eyes and noses on a face.
`
`Once the eye area is identified, the eye itself is analyzed. As confirmation of
`
`the detection of nostrils or other facial feature being detected, a search box
`
`9
`
`

`

`IPR2017-01190 (’518 Patent) Patent Owner Response
`
`established around the eye area using an anthropomorphic model is analyzed for
`
`characteristics of an eye present. Id., 30:51–56. “These characteristics may
`
`include, for example, a moving eyelid, a pupil, iris or cornea, a shape
`
`corresponding to an eye, a shadow corresponding to an eye, or any other indicia
`
`indicative of an eye.” Id., 30:56–58. For example, a histogram of low-luminance,
`
`high-gloss pixels for the pupil, as shown in Figure 36. Id., 30:61–64.
`
`Ex. 1001, Figure 36
`
`
`
`The histogram can then be analyzed over time to identify each opening and
`
`closing of the eye. Ex. 1001, 2:37–40.
`
`A. Histogram Calculation Based on Image Data
`In order to accomplish the efficient, real-time identification and localization
`
`of facial characteristics by processing image data using histograms, e.g., Ex. 1001,
`
`2:1–3:13, the patent teaches the use of histogram units coupled with validation
`
`units that operate on image data comprised of frames of pixel data.
`
`The operation of the histogram units and validation units on pixel data is
`
`described in this section. Figure 1 of the ’518 patent teaches the relationship of the
`
`10
`
`

`

`IPR2017-01190 (’518 Patent) Patent Owner Response
`
`exemplary image processing system to a digital camera and the information
`
`captured by the camera.
`
`Ex. 1001, Figure 1 (see Ex. 1001, 5:8–9, 6:39–45).
`
`
`
`Signal S(PI) is a digital signal output of a succession of frames by a camera
`
`(13). See Ex. 1001, 6:44–7:13. Each frame comprises pixels associated with a
`
`particular x, y (or i, j) horizontal and vertical location in the frame. See Ex. 1001,
`
`Figure 1 at 6:58–65. The digital signal S(PI) includes data reflecting properties
`
`associated with each pixel, for example luminance value for a black-and-white
`
`image, or, for example HSL (hue, saturation, luminance) values associated with the
`
`11
`
`

`

`IPR2017-01190 (’518 Patent) Patent Owner Response
`
`pixel for a color image. See Ex. 1001, 14:28–42, 25:66–26:12, 29:57–60, 32:14–
`
`20.
`
`Figure 5 of the ’518 patent illustrates a series of frames, and a pixel location
`
`i, j in each frame. See Ex. 1001, 9:51–54. Note the arrows indicating x and y
`
`direction. Ex. 1001, 5:16–17, 9:51–57.
`
`Ex. 1001, Figure 5 (page 4).
`
`
`
`The signal S(PI) is processed to calculate properties associated with pixels.
`
`Domains of data associated with pixels include luminance, hue, saturation, speed,
`
`oriented direction, and x-axis and y-axis position. Id., 2:21–25, 14:24–42.
`
`In Figure 11 of the ’518 patent, spatial and temporal processing unit (11) is
`
`shown in connection with a histogram formation unit 22a. Spatial and temporal
`
`processing unit (11) receives digital video signal S(PI). Ex. 1001, 6:39–45. S(PI)
`
`represents the pixel values PI of video signal S, in a succession of frames, each
`
`representing an instant in time. Ex. 1001, 6:58–65, 7:16–24; 9:6–10. Spatial and
`
`temporal processing unit (11) outputs signals SR (delayed video signal) and also
`
`12
`
`

`

`IPR2017-01190 (’518 Patent) Patent Owner Response
`
`calculated values such as speed (V) and oriented direction of displacement (DI) for
`
`pixels in the image. Ex. 1001, 7:15–7:30. Speed and oriented direction are
`
`calculated by image processing system (11), based on a matrix of pixels centered
`
`on a particular pixel. Ex. 1001, 7:13–20, 13:7–14:5. A bus Z–Z1 (the dotted line
`
`which appears in Figures 11 and also Figure 12 which is discussed below) transfers
`
`output signals of the image processing system (11) to histogram formation unit
`
`(22a). Ex. 1001, 14:15–23.
`
`Ex. 1001, Figure 11 (page 6).
`
`
`
`Figure 12 of the ’518 patent shows an example of a histogram processor 22a
`
`with multiple histogram formation blocks 24–29. Ex. 1001, 14:24–32. For
`
`example, Block 24 enables a histogram to be formed in the luminance domain
`
`(ranging from 0–255). Id.,14:34–42. The histogram formation blocks and other
`
`components are interconnected by a bus 23. Id., 14:24–28.
`
`13
`
`

`

`IPR2017-01190 (’518 Patent) Patent Owner Response
`
`Ex. 1001, Figure 12 (page 7).
`
`
`
`A validation unit accompanies each histogram formation block of Figure 12.
`
`Ex. 1001, 16:22–27. Figure 14, referring to histogram formation block 25 and
`
`validation unit 31 of Figure 12, shows a histogram formation block with a
`
`classifier 25b. The classifier has registers that permit classification criteria to be
`
`individually selected: Ex. 1001, 15:62–67.
`
`14
`
`

`

`IPR2017-01190 (’518 Patent) Patent Owner Response
`
`Ex. 1001, Figure 14 (page 9).
`
`
`
`Histogram memory 100 stores the histogram for the histogram formation
`
`block. Id., 17:42–58. When the pixel has the correct values for the categories
`
`being considered, e.g., direction, speed, or luminance, the memory 100 is
`
`incremented at the address corresponding to the pixel value for the domain being
`
`considered. For example, if the pixel has a speed value of 6, the content in
`
`histogram memory at location 6 will be incremented. Id., 17:2–11. If the pixel
`
`does not have the correct values for all categories being considered, the pixel is not
`
`used in forming a histogram. Id., 17:12–18.
`
`15
`
`

`

`IPR2017-01190 (’518 Patent) Patent Owner Response
`
`The ’518 patent teaches the use of the histogram formation blocks described
`
`above in forming histograms to detect features of an eye.
`
`B. Detection of Eye Area—Two Preferred Embodiments
`In one embodiment, the ’518 patent teaches two preferred embodiments of
`
`determining “a sub-area of the image comprising the eye” prior to “the step of
`
`selecting pixels having characteristics corresponding to characteristics of an eye”:
`
`(See Ex. 1001, 2:41–44.)
`
`In this embodiment, the step of selecting pixels of the
`image having characteristics of an eye involves selecting
`pixels within the sub-area of the image. The step of
`identifying a sub-area of the image preferably involves
`identifying [i.] the head of the driver, or [ii.] a facial
`characteristic of the driver, such as the driver's nostrils,
`and then identifying the sub-area of the image using an
`anthropomorphic model.
`
`Ex. 1001, 2:44–50 (numerals “i.” and “ii.” added in brackets). These methods are
`
`alternatives to one another. Compare Ex. 1001, Figure 30, 24:59–29:10 (“head”
`
`method), and Figures 31–36, 29:11–37 (“FIGS. 31–36 show an alternative
`
`method”).
`
`These two methods are also claimed using distinct claim language. For
`
`example, the head-identification method is reflected in the language of claim 2:
`
`“selecting pixels of the image having characteristics corresponding to edges of the
`
`16
`
`

`

`IPR2017-01190 (’518 Patent) Patent Owner Response
`
`head of the person; forming histograms of the selected pixels . . . analyzing the
`
`histograms of the selected pixels to identify the edges of the head….” Ex. 1001,
`
`32:48–52. The facial-feature method is reflected in, for example, claim 4:
`
`“selecting pixels of the image having characteristics corresponding to the facial
`
`characteristic; forming histograms of the selected pixels.” Id., 33:10–13. Claim 5
`
`which depends on claim 4 provides an example: a “facial characteristic” is nostrils,
`
`and the selected pixels for nostrils are low-luminance pixels. Id., 33:18–21.
`
`Each preferred embodiment is taught in the specification as explained below.
`
`1.
`Detection of Eye Area Based on Head Frame
`A first embodiment teaches detecting the head of a person. Id., 2:51–53.
`
`The head is located by searching the image for the head, identifying a rectangular
`
`frame for the head, and using an anthropomorphic ratio to locate an eye area based
`
`on the rectangular head area. See, e.g., Id., 2:41–3:13. Histograms are then used to
`
`analyze the eye area. The embodiment in general is illustrated by Figure 30:
`
`17
`
`

`

`IPR2017-01190 (’518 Patent) Patent Owner Response
`
`Ex. 1001, Figure 30 (page 17, annotations in red added).
`
`
`
`The head may be identified by selecting pixels that have characteristics
`
`corresponding to the edges of the head. Id., 2:51–53. The characteristics of edges
`
`18
`
`

`

`IPR2017-01190 (’518 Patent) Patent Owner Response
`
`of the head are generally applicable to the entire facial region (i.e., the border
`
`characteristics are not specific features of the face), and include “hue
`
`corresponding to human skin,” slow movement, and significant change in the pixel
`
`value from past values for that pixel location (corresponding to a pixel DP value of
`
`1, i.e. “DP = 1”).6 Id., 26:16–45. Figure 24 illustrates detection of the edges of the
`
`head:
`
`
`Ex. 1001, Fig. 24 (Excerpt) (see 5:64–65).
`
`Once the edges of the head are detected, the area surrounding the face is
`
`masked such that only the face area is considered further. Id., 26:66–27:1. Figure
`
`25 illustrates this process:
`
`
`6 Ex. 1001, 8:3–21.
`
`19
`
`

`

`IPR2017-01190 (’518 Patent) Patent Owner Response
`
`
`Ex. 1001, Figure 25 (page 15); see also Ex. 1001, 27:1–17.
`
`An anthropomorphic ratio is applied based on the rectangle identified as the
`
`face area (the face “frame”). Id., 27:28–51. Figure 26 illustrates the application of
`
`a ratio “between the zone of the eyes and the entire face for a human being,
`
`especially in the vertical direction”. Id., 27:34–35. Only the area within Z’ is
`
`considered further:
`
`
`Ex. 1001, Figure 26 (page 15); see also Ex. 1001, 27:28–51.
`
`20
`
`

`

`IPR2017-01190 (’518 Patent) Patent Owner Response
`
`2.
`
`Detection of Eye Area Based on Location of a Facial
`Characteristic
`
`In the second embodiment discussed here, the process of detecting an eye
`
`area first comprises identifying specific facial features instead of using the general
`
`characteristics like those used for edge-detection of the head. Id., 29:12–14. The
`
`controller may execute a search mode to scan the image for facial characteristics,
`
`for example the nostrils, nose, ears, eyebrows, mouth, etc. Id., 29:12–17, 52–57.
`
`In one embodiment, for example, the patent teaches searching for facial features in
`
`a series of sub-images, which are portions of the initial image. Id., 29:12–51. The
`
`process of this “facial feature” embodiment is illustrated in the flowchart of
`
`Figure 35:
`
`21
`
`

`

`IPR2017-01190 (’518 Patent) Patent Owner Response
`
`
`Ex. 1001, Figure 35 (page 20 excerpt, annotated in red); see also 29:12–18).
`
`The controller, if searching for nostrils, for example, may search for nostrils
`
`in a first sub-image A by forming x-direction and y-direction histograms of pixels
`
`having low luminance to identify characteristics indicative of nostrils. Id., 29:24–
`
`35. If nostrils are not identified in a sub-image, the process is repeated for next
`
`sub-image and subsequent sub-images. Id., 29:35–40. Figure 31 illustrates sub-
`
`22
`
`

`

`IPR2017-01190 (’518 Patent) Patent Owner Response
`
`images A through F of the main image, which are searched in sequence to identify
`
`a facial characteristic:
`
`Ex. 1001, Figure 31 (page 18).
`
`
`
`Figure 32 illustrates an exemplary analysis of histograms to identify characteristics
`
`of nostrils.
`
`
`Ex. 1001, Figure 32 (page 18); see also Ex. 1001, 30:1–2.
`
`23
`
`

`

`IPR2017-01190 (’518 Patent) Patent Owner Response
`
`Nostrils are characterized, for example, by a peak in the y-direction histogram and
`
`two peaks in the x-direction histogram. Id., 30:2–4. Histograms of the low-
`
`luminance pixels may also be analyzed to ensure a sufficient number of pixels with
`
`low luminance have been identified and to ensure thresholds are exceeded for the
`
`maximum (peak) of the histograms, among other things. Id., 30:4–12.
`
`Once the nostrils are identified, an anthropomorphic model based on the
`
`spatial relationship between the eyes and nose of humans can be used to establish a
`
`search box around the eye. Id., 30:40–45.
`
`The search box may then be analyzed for characteristics of an eye present in
`
`the search box. Id., 30:51–56. The characteristics may include, for example, a
`
`moving eyelid, a pupil, iris, or cornea, a shape corresponding to an eye, or a
`
`shadow corresponding to an eye. Id., 30:56–59.
`
`Figure 36, for example, illustrates a histogram calculation targeted to
`
`detecting the pupil, in which the histogram units are set to detect pixels with low
`
`luminance levels and high gloss levels, characteristic of a pupil (item 432). Id.,
`
`30:61–64. Only pixels with both low luminance values and high gloss values are
`
`used to form the histogram. Id.
`
`24
`
`

`

`IPR2017-01190 (’518 Patent) Patent Owner Response
`
`Ex. 1001, Figure 36 (page 21)
`
`
`
`The pupil may