`
`
`
`UNITED STATES PATENT AND TRADEMARK OFFICE
`
`____________________
`
`BEFORE THE PATENT TRIAL AND APPEAL BOARD
`
`____________________
`
`SAMSUNG ELECTRONICS CO., LTD.; AND
`SAMSUNG ELECTRONICS AMERICA, INC.
`Petitioner
`
`v.
`
`IMAGE PROCESSING TECHNOLOGIES, LLC
`Patent Owner
`
`____________________
`
`Patent No. 7,650,015
`____________________
`
`DECLARATION OF DR. JOHN C. HART
`IN SUPPORT OF PETITION FOR INTER PARTES REVIEW
`OF U.S. PATENT NO. 7,650,015
`
`
`
`Page 1 of 278
`
`SAMSUNG EXHIBIT 1002
`Samsung v. Image Processing Techs.
`
`

`

`Declaration of Dr. John C. Hart
`Inter Partes Review of Patent No. 7,650,015
`
`
`
`I.
`
`II.
`
`TABLE OF CONTENTS
`
`Contents
`INTRODUCTION ........................................................................................... 1
`
`BACKGROUND AND EXPERIENCE .......................................................... 1
`
`A. Qualifications ........................................................................................ 1
`
`B.
`
`Previous Testimony ............................................................................... 4
`
`III. TECHNOLOGICAL BACKGROUND .......................................................... 5
`
`IV. THE ’015 PATENT ....................................................................................... 12
`
`V.
`
`SUMMARY OF OPINIONS ......................................................................... 19
`
`VI. LEVEL OF ORDINARY SKILL IN THE ART ........................................... 20
`
`VII. CLAIM CONSTRUCTION .......................................................................... 21
`
`VIII. THE PRIOR ART TEACHES OR SUGGESTS EVERY STEP AND
`FEATURE OF THE CHALLENGED CLAIMS OF THE ’015
`PATENT ........................................................................................................ 21
`
`A. Overview Of The Prior Art References .............................................. 21
`
`1.
`
`2.
`
`3.
`
`U.S. Patent No. 5,481,622 to Gerhardt (Ex. 1013) ................... 21
`
`U.S. Patent No. 6,044,166 to Bassman (Ex. 1014) ................... 28
`
`Alton L. Gilbert et al., A Real-Time Video Tracking
`System, PAMI-2 No. 1 IEEE Transactions on Pattern
`Analysis and Machine Intelligence 47 (Jan. 1980)
`(“Gilbert”) (Ex. 1005) ............................................................... 31
`
`4.
`
`U.S. Patent No. 5,521,843 (“Hashima”) (Ex. 1006) ................. 43
`
`5. W. B. Schaming, Adaptive Gate Multifeature Bayesian
`Statistical Tracker, 359 Applications of Digital Image
`Processing IV 68 (1982) (“Schaming”) (Ex. 1008) .................. 50
`
`i
`
`SAMSUNG EXHIBIT 1002
`Page 2 of 278
`
`

`

`Declaration of Dr. John C. Hart
`Inter Partes Review of U.S. Patent No. 7,650,015
`Gerhardt In View Of Bassman Renders Obvious Claims 1-2
`and 4-5 ................................................................................................. 55
`
`B.
`
`1.
`
`2.
`
`3.
`
`4.
`
`5.
`
`6.
`
`7.
`
`Reasons To Combine Gerhardt and Bassman .......................... 55
`
`Claim 1 ...................................................................................... 57
`
`Claim 2: “The process according to claim 1, comprising
`centering the tracking box relative to an optical axis of
`the frame” .................................................................................. 66
`
`Claim 4: “The process according to claim 1, wherein said
`image processing system comprises at least one
`component selected from a memory, a temporal
`processing unit, and a spatial processing unit.” ........................ 68
`
`Claim 5: “The process according to claim 1, wherein said
`image processing system comprises at least two
`components selected from a memory, a temporal
`processing unit, and a spatial processing unit” ......................... 72
`
`Gerhardt and Bassman Are Not Cumulative ............................ 72
`
`Detailed Application Of Gerhardt And Bassman To
`Claims 1, 2, 4, And 5 ................................................................ 73
`
`C.
`
`Gerhardt In View Of Bassman And Further In View Of
`Hashima Renders Obvious Claims 3 and 7 .......................................116
`
`1.
`
`2.
`
`Reasons To Combine Gerhardt and Bassman with
`Hashima ..................................................................................116
`
`Claim 3: “The process according to claim 1, comprising
`calculating a histogram according to a projection axis in a
`region delimited by an associated classifier, between two
`points on the projection axis, creating a histogram of the
`same points with orientation and intensity of motion as
`input parameters and modifying the values corresponding
`to said two points of the classifier and calculating an
`anticipated next frame” ...........................................................117
`
`3.
`
`Claim 7 ....................................................................................120
`
`
`
`
`
`-ii-
`
`
`
`SAMSUNG EXHIBIT 1002
`Page 3 of 278
`
`

`

`Declaration of Dr. John C. Hart
`Inter Partes Review of U.S. Patent No. 7,650,015
`Gerhardt, Bassman, and Hashima Are Not Cumulative .........127
`
`Detailed Application Of Gerhardt, Bassman And
`Hashima To The Challenged Claims ......................................128
`
`4.
`
`5.
`
`D. Gilbert In View Of Gerhardt And Further In View Of Schaming
`Renders Obvious Claims 1-5 and 7 ...................................................155
`
`1.
`
`2.
`
`3.
`
`4.
`
`5.
`
`6.
`
`7.
`
`8.
`
`9.
`
`Reasons To Combine Gilbert, Gerhardt and Schaming ..........155
`
`Claim 1 ....................................................................................158
`
`Claim 2: “The process according to claim 1, comprising
`centering the tracking box relative to an optical axis of
`the frame” ................................................................................170
`
`Claim 3: “The process according to claim 1, comprising
`calculating a histogram according to a projection axis in a
`region delimited by an associated classifier, between two
`points on the projection axis, creating a histogram of the
`same points with orientation and intensity of motion as
`input parameters and modifying the values corresponding
`to said two points of the classifier and calculating an
`anticipated next frame” ...........................................................172
`
`Claim 4: “The process according to claim 1, wherein said
`image processing system comprises at least one
`component selected from a memory, a temporal
`processing unit, and a spatial processing unit.” ......................174
`
`Claim 5: “The process according to claim 1, wherein said
`image processing system comprises at least two
`components selected from a memory, a temporal
`processing unit, and a spatial processing unit” .......................176
`
`Claim 7 ....................................................................................177
`
`Gilbert, Gerhardt, and Schaming Are Not Cumulative ..........179
`
`Detailed Application Of Gilbert, Gerhardt, And
`Schaming To The Challenged Claims ....................................181
`
`
`
`
`
`-iii-
`
`
`
`SAMSUNG EXHIBIT 1002
`Page 4 of 278
`
`

`

`Declaration of Dr. John C. Hart
`Inter Partes Review of U.S. Patent No. 7,650,015
`IX. CONCLUSION ............................................................................................273
`
`
`
`
`
`
`
`-iv-
`
`
`
`SAMSUNG EXHIBIT 1002
`Page 5 of 278
`
`

`

`Declaration of Dr. John C. Hart
`Inter Partes Review of U.S. Patent No. 7,650,015
`I , John C. Hart, declare as follows:
`
`1.
`
`I.
`
`INTRODUCTION
`2.
`
`I have been retained by Samsung Electronics Co., Ltd. and Samsung
`
`Electronics America, Inc. (collectively, “Petitioner”) as an independent expert
`
`consultant in this proceeding before the United States Patent and Trademark Office
`
`(“PTO”).
`
`3.
`
`I have been asked to consider whether certain references teach or
`
`suggest the features recited in Claims 1-5 and 7 (the “Challenged Claims”) of U.S.
`
`Patent No. 7,650,015 (“the ’015 Patent”) (Ex. 1001), which I understand is
`
`allegedly owned by Image Processing Technologies, LLC (“Patent Owner”). My
`
`opinions and the bases for my opinions are set forth below.
`
`4.
`
`I am being compensated at my ordinary and customary consulting rate
`
`for my work.
`
`5. My compensation is in no way contingent on the nature of my
`
`findings, the presentation of my findings in testimony, or the outcome of this or
`
`any other proceeding. I have no other interest in this proceeding.
`
`II. BACKGROUND AND EXPERIENCE
`A. Qualifications
`6.
`I have more than 25 years of experience in computer graphics and
`
`image processing technologies. In particular, I have devoted much of my career to
`
`1
`
`SAMSUNG EXHIBIT 1002
`Page 6 of 278
`
`

`

`Declaration of Dr. John C. Hart
`Inter Partes Review of U.S. Patent No. 7,650,015
`researching and designing graphics hardware and systems for a wide range of
`
`applications.
`
`7. My research has resulted in the publication of more than 80 peer-
`
`reviewed scientific articles, and more than 50 invited papers, and talks in the area
`
`of computer graphics and image processing.
`
`8.
`
`I have authored or co-authored several publications that are directly
`
`related to target identification and tracking in image processing systems. Some
`
`recent publications include:
`
`• P.R. Khorrami, V.V. Le, J.C. Hart, T.S. Huang. A System for
`
`Monitoring the Engagement of Remote Online Students using Eye
`
`Gaze Estimation. Proc. IEEE ICME Workshop on Emerging
`
`Multimedia Systems and Applications, July 2014.
`
`• V. Lu, I. Endres, M. Stroila and J.C. Hart. Accelerating Arrays of
`
`Linear Classifiers Using Approximate Range Queries. Proc. Winter
`
`Conference on Applications of Computer Vision, Mar. 2014.
`
`• M. Kamali, E. Ofek, F. Iandola, I. Omer, J.C. Hart Linear Clutter
`
`Removal from Urban Panoramas. Proc. International Symposium on
`
`Visual Computing. Sep. 2011.
`
`9.
`
`From 2008-2012, as a Co-PI of the $18M Intel/Microsoft Universal
`
`Parallelism Computing Research Center at the University of Illinois, I led the
`
`2
`
`SAMSUNG EXHIBIT 1002
`Page 7 of 278
`
`

`

`Declaration of Dr. John C. Hart
`Inter Partes Review of U.S. Patent No. 7,650,015
`AvaScholar project for visual processing of images that included face
`
`identification, tracking and image histograms.
`
`10.
`
`11.
`
`I am a co-inventor of U.S. Patent No. 7,365,744.
`
`I have served as the Director for Graduate Studies for the Department
`
`of Computer Science, an Associate Dean for the Graduate College, and I am
`
`currently serving as the Executive Associate Dean of the Graduate College at the
`
`University of Illinois. I am also a professor in the Department of Computer
`
`Science at the University of Illinois, where I have served on the faculty since
`
`August 2000. As a professor I have taught classes on image processing and
`
`graphics technology and have conducted research into specific applications of
`
`these technologies.
`
`12. From 1992 to 2000, I worked first as an Assistant Professor and then
`
`as an Associate Professor in the School of Electrical Engineering and Computer
`
`Science at Washington State University.
`
`13. From 1991-1992, I was a Postdoctoral Research Associate at the
`
`Electronic Visualization Laboratory at the University of Illinois at Chicago, and at
`
`the National Center for Supercomputing Applications at the University of Illinois
`
`at Urbana-Champaign.
`
`14.
`
`I earned a Doctor of Philosophy in Electrical Engineering and
`
`Computer Science from the University of Illinois at Chicago in 1991.
`
`3
`
`SAMSUNG EXHIBIT 1002
`Page 8 of 278
`
`

`

`Declaration of Dr. John C. Hart
`Inter Partes Review of U.S. Patent No. 7,650,015
`I earned a Master’s Degree in Electrical Engineering and Computer
`
`15.
`
`Science from the University of Illinois at Chicago in 1989.
`
`16.
`
`I earned a Bachelor of Science in Computer Science from Aurora
`
`University in 1987.
`
`17.
`
`I have been an expert in the field of graphics and image processing
`
`since prior to 1996. I am qualified to provide an opinion as to what a person of
`
`ordinary skill in the art (“POSA”) would have understood, known, or concluded as
`
`of 1996.
`
`18. Additional qualifications are detailed in my curriculum vitae, which I
`
`understand has been submitted as Exhibit 1003 in this proceeding.
`
`B.
`19.
`
`Previous Testimony
`
`In the previous five years, I have testified as an expert at trial or by
`
`deposition or have submitted declarations in the following cases:
`
`20. Certain Computing or Graphics Systems, Components Thereof, and
`
`Vehicles Containing Same, Inv. No. 337-TA-984.
`
`21. ZiiLabs Inc., Ltd v. Samsung Electronics Co. Ltd. et al., No. 2:14-cv-
`
`00203 (E.D. Tex. Feb. 4, 2016).
`
`22. Certain Consumer Electronics with Display and Processing
`
`Capabilities, Inv. No. 337-TA-884.
`
`4
`
`SAMSUNG EXHIBIT 1002
`Page 9 of 278
`
`

`

`Declaration of Dr. John C. Hart
`Inter Partes Review of U.S. Patent No. 7,650,015
`I have also submitted declarations in support of the following
`
`23.
`
`Petitions for Inter Partes Review in Samsung v. Image Processing Technologies,
`
`LLC:
`
`• IPR2017-00355 against the ’015 Patent, filed 11/30/2016.
`
`• IPR2017-00347 against U.S. Patent No. 8,805,001, filed 11/29/2016.
`
`• IPR2017-00336 against U.S. Patent No. 6,959,293, filed 11/29/2016.
`
`• IPR2017-00357 against U.S. Patent No. 8,989,445, filed 11/30/2016.
`
`• IPR2017-00353 against U.S. Patent No. 8,983,134, filed 11/30/2016.
`
`• IPR2017-01190 against U.S. Patent No. 6,717,518, filed 3/29/2017.
`
`• IPR2017-01212 against U.S. Patent No. 8,989,445, filed 3/30/2017.
`
`• IPR2017-01189 against U.S. Patent No. 6,959,293, filed 3/30/2017.
`
`III. TECHNOLOGICAL BACKGROUND
`24.
`Image processing systems have long used histograms as a
`
`mathematical tool to identify and track image features and to adjust image
`
`properties. The use of histograms to identify and track image features dates back
`
`to well before 1997. D. Trier, A. K. Jain and T. Taxt, “Feature Extraction Methods
`
`for Character Recognition-A Survey”, Pattern Recognition, vol. 29, no. 4, 1996,
`
`pp. 641–662 (Ex. 1009) (citing M. H. Glauberman, “Character recognition for
`
`business machines,” Electronics, vol. 29, pp. 132(136), Feb. 1956(Ex. 1010)).
`
`5
`
`SAMSUNG EXHIBIT 1002
`Page 10 of 278
`
`

`

`Declaration of Dr. John C. Hart
`Inter Partes Review of U.S. Patent No. 7,650,015
`25. A digital image is represented by a number of picture elements, or
`
`pixels, where each pixel has certain properties, such as brightness, color, position,
`
`velocity, etc., which may be referred to as domains. For each pixel property or
`
`domain, a histogram may be formed. A histogram is a type of statistical tool. In
`
`image processing, histograms are often used to count the number of pixels in an
`
`image in a certain domain of the pixel. Histograms have multiple bins, where each
`
`bin in the histogram counts the pixels that fall within a range for that domain. For
`
`example, for the continuous variable of luminance (also called brightness), the
`
`luminance value for each pixel can be sampled by a camera and then digitized and
`
`represented by an 8-bit value. Then, those luminance values could be loaded into a
`
`luminance histogram. The histogram would have one bin for each range of
`
`luminance values, and each bin would count the number of pixels in the image that
`
`fall within that luminance value range. As shown below, a luminance histogram
`
`may reveal certain properties of an image, such as whether it is properly exposed,
`
`based on whether an excessive number of pixels fall on the dark end or light end of
`
`the luminance range.
`
`6
`
`SAMSUNG EXHIBIT 1002
`Page 11 of 278
`
`

`

`Declaration of Dr. John C. Hart
`Inter Partes Review of U.S. Patent No. 7,650,015
`
`
`
`26. Histograms of other pixel properties can also be formed. For
`
`example, the figure below illustrates two histograms formed by counting the
`
`number of black pixels having each X-coordinate value (i.e., the X-coordinate
`
`domain) and the number having each Y-coordinate value (i.e., the Y-coordinate
`
`domain).
`
`
`
`7
`
`SAMSUNG EXHIBIT 1002
`Page 12 of 278
`
`

`

`Declaration of Dr. John C. Hart
`Inter Partes Review of U.S. Patent No. 7,650,015
`27. Such histograms are sometimes called “projection histograms”
`
`because they represent the image projected onto each axis. In the example above,
`
`the image was pure black and white, but projection histograms of a greyscale
`
`image can also be formed in a similar manner by defining a luminance threshold
`
`and projecting, for example, only those pixels that have a luminance value lower
`
`than 100.
`
`28. A more complex greyscale image is shown below, along with its
`
`luminance histogram (black = 0; white = 255):
`
`
`
`8
`
`SAMSUNG EXHIBIT 1002
`Page 13 of 278
`
`

`

`Declaration of Dr. John C. Hart
`Inter Partes Review of U.S. Patent No. 7,650,015
`
`
`
`29. Here, the peak in the dark luminance region (luminance = 0-50)
`
`corresponds to the dark suit and tie and relatively dark background. The peak in
`
`the light luminance region (luminance > 230) corresponds to the white shirt, while
`
`the central peak (between luminance 130 and 170) corresponds largely to the
`
`medium brightness of the face. If one were to select only the subset of pixels with
`
`brightness between 130 and 170 and plot them according to their x and y position,
`
`one would get the following image:
`
`9
`
`SAMSUNG EXHIBIT 1002
`Page 14 of 278
`
`

`

`Declaration of Dr. John C. Hart
`Inter Partes Review of U.S. Patent No. 7,650,015
`
`
`
`30. Taking projection histograms of this subset of pixels with luminance
`
`between 130 and 170, then, provides an indication of location of the face in the
`
`image. On the left, below, is a projection of this subset of pixels onto the x axis,
`
`and on the right is a similar projection onto the y axis.
`
`10
`
`
`
`SAMSUNG EXHIBIT 1002
`Page 15 of 278
`
`

`

`Declaration of Dr. John C. Hart
`Inter Partes Review of U.S. Patent No. 7,650,015
`31. Histograms may also be formed of pixel color properties in much the
`
`same way. Color is typically represented by three values: hue, saturation and
`
`luminance. Hue (aka “tone”) is an angle ranging from 0° to 360° around a color
`
`wheel that indicates which “color” is bring represented, e.g. 0° = red, 60° = yellow,
`
`120° = green, 180° = cyan, 240° = blue, and 300° = magenta. Saturation, which
`
`may also range from 0 to 255, represents how “brilliant” the color is. For example,
`
`if a color with a saturation of 255 represents red, then a saturation of 128 would
`
`represent pink and a saturation of 0 would represent gray. Luminance ranges from
`
`0 to 255 and represents the “brightness” of the color. If luminance = 0, then the
`
`color is black, regardless of the other values. Given a color image, the luminance
`
`values of the pixels would yield the “black-and-white” or grayscale version of the
`
`image.
`
`
`
`11
`
`SAMSUNG EXHIBIT 1002
`Page 16 of 278
`
`

`

`Declaration of Dr. John C. Hart
`Inter Partes Review of U.S. Patent No. 7,650,015
`
`IV. THE ’015 PATENT
`32. The ’015 Patent, entitled “Image Processing Method,” was filed on
`
`February 20, 2007, and issued on January 19, 2010. The ’015 Patent names
`
`Patrick Pirim as the sole inventor. I understand that the ’015 Patent claims a
`
`priority date of July 26, 1996.
`
`33. The ’015 Patent is generally directed to tracking a target using an
`
`image processing system. For example, in the Abstract, the ’015 Patent notes that
`
`the invention relates to “[a] method and apparatus for localizing an area in relative
`
`movement and for determining the speed and direction thereof.” Ex. 1001, ’015
`
`Patent at Abstract.
`
`34. The ’015 Patent uses the pixels in a frame of an image of a video
`
`signal to form one or more histograms in order to identify and track a target. See
`
`e.g., Ex. 1001, ’015 Patent at Claims 1-5, 7. The input signal employed in the ’015
`
`Patent is comprised of a “succession of frames, each frame having a succession of
`
`pixels.” Ex. 1001, ’015 Patent at 3:13-23. Although the disclosed embodiments
`
`relate primarily to a video signal, the ’015 Patent also teaches that the input signal
`
`could correspond to other types of signals, for example “ultrasound, IR, Radar,
`
`tactile array, etc.” Ex. 1001, ’015 Patent at 9:6-16.
`
`35. The ’015 Patent teaches constructing a histogram showing the
`
`frequency of the pixels meeting a certain characteristic. In the ’015 Patent, these
`
`12
`
`SAMSUNG EXHIBIT 1002
`Page 17 of 278
`
`

`

`Declaration of Dr. John C. Hart
`Inter Partes Review of U.S. Patent No. 7,650,015
`characteristics—such as luminance or speed—are referred to as “domains.” Ex.
`
`1001, 3:46-58. The ’015 Patent teaches that “the domains are preferably selected
`
`from the group consisting of i) luminance, ii) speed (V), iii) oriented direction
`
`(DI), iv) time constant (CO), v) hue, vi) saturation, and vii) first axis (x(m)), and
`
`viii) second axis (y(m)).” Ex. 1001, ’015 Patent at 3:54-58.
`
`36. A domain can be further subdivided into classes, each class consisting
`
`of the subset of pixels with similar domain values. Figure 14a (and the
`
`accompanying text) illustrates an example of “classes” within a domain:
`
`
`
`FIG. 14 a shows an example of the successive classes C1
`
`C2 . . . Cn−1 Cn, each representing a particular velocity, for
`
`a hypothetical velocity histogram, with their being
`
`categorization for up to 16 velocities (15 are shown) in
`
`this example. Also shown is envelope 38, which is a
`
`smoothed representation of the histogram.
`
`Ex. 1001, ’015 Patent at 20:47-52.
`
`13
`
`SAMSUNG EXHIBIT 1002
`Page 18 of 278
`
`

`

`Declaration of Dr. John C. Hart
`Inter Partes Review of U.S. Patent No. 7,650,015
`37. The hypothetical histogram in Figure 14a would be constructed using
`
`histogram formation block 25 in Figure 11. In this figure, various histogram
`
`processors (numbered 24-29) are shown that allow creation of histograms in
`
`various domains. Block 25 is disclosed as creating velocity histograms. Id. at
`
`17:3-9.
`
`
`A detailed depiction of histogram block 25 is shown in Figure 13:
`
`14
`
`SAMSUNG EXHIBIT 1002
`Page 19 of 278
`
`

`

`Declaration of Dr. John C. Hart
`Inter Partes Review of U.S. Patent No. 7,650,015
`
`
`38. The ’015 Patent then uses the histograms to identify a target in the
`
`input signal. For example, one embodiment of the ’015 Patent performs
`
`“automatic framing of a person… during a video conference.” Ex. 1001, 22:4-17;
`
`see also id., Figure 15:
`
`
`
`15
`
`SAMSUNG EXHIBIT 1002
`Page 20 of 278
`
`

`

`Declaration of Dr. John C. Hart
`Inter Partes Review of U.S. Patent No. 7,650,015
`39. The system constructs histograms in the X and Y domains counting
`
`the number of pixels that have a difference in luminance between successive
`
`frames above certain threshold values:
`
`The pixels with greatest movement within the image will
`
`normally occur at the peripheral edges of the head of the
`
`subject, where even due to slight movements, the pixels
`
`will vary between the luminance of the head of the
`
`subject and the luminance of the background. Thus, if
`
`the system of the invention is set to identify only pixels
`
`with DP=1, and to form a histogram of these pixels, the
`
`histogram will detect movement peaks along the edges of
`
`the face where variations in brightness, and therefore in
`
`pixel value, are the greatest, both in the horizontal
`
`projection along Ox and in the vertical projection along
`
`Oy.
`
`Ex. 1001, 22:44-54 and 10:29-44 (explaining that DP is set to “1” when the pixel
`
`value of the pixel under consideration has “undergone significant variation as
`
`compared to…the same pixel in the prior frame.
`
`40. Figures 16 and 17 show camera setup and the histogram constructed
`
`using this method:
`
`16
`
`SAMSUNG EXHIBIT 1002
`Page 21 of 278
`
`

`

`Declaration of Dr. John C. Hart
`Inter Partes Review of U.S. Patent No. 7,650,015
`
`Ex. 1001, Fig. 16
`
`
`
`Ex. 1001, Fig. 17
`
`41.
`
`In addition, the system may also be used to automatically track a
`
`
`
`target by “a spotlight or a camera. Using a spotlight the invention might be used
`
`on a helicopter to track a moving target on the ground, or to track a performer on a
`
`stage during an exhibition. The invention would similarly be applicable to
`
`weapons targeting systems.” Ex. 1001, 23:35-40. In such applications, the system
`
`uses X and Y minima and maxima of the histograms in X and Y domains to
`
`determine the center of the target. Ex. 1001, 24:46-51. The patent defines “the
`
`positions of the minima” of a projection histogram to be the smallest X (and Y)
`
`coordinate of any pixel in the image region whose validation signal is “1.”
`
`Similarly the maximum is the largest X (and Y) coordinate of any pixel in the
`
`image region whose validation signal is “1.” Id.
`
`42. Once the center of the target is determined, the center is used to adjust
`
`the camera or spotlight to be directed to the moving target:
`
`17
`
`SAMSUNG EXHIBIT 1002
`Page 22 of 278
`
`

`

`Declaration of Dr. John C. Hart
`Inter Partes Review of U.S. Patent No. 7,650,015
`Having acquired the target, controller 206 controls
`
`servomotors 208 to maintain the center of the target in
`
`the center of the image….
`
`It will be appreciated that as the target moves, the
`
`targeting box will move with the target, constantly
`
`adjusting the center of the targeting box based upon the
`
`movement of the target, and enlarging and reducing the
`
`size of the targeting box. The targeting box may be
`
`displayed on monitor 212, or on another monitor as
`
`desired to visually track the target.
`
`Ex. 1001, 25:8-21. Figure 23 shows an example of the targeting box in a frame:
`
`Ex. 1001 at Fig. 23
`
`
`
`18
`
`SAMSUNG EXHIBIT 1002
`Page 23 of 278
`
`

`

`Declaration of Dr. John C. Hart
`Inter Partes Review of U.S. Patent No. 7,650,015
`SUMMARY OF OPINIONS
`43.
`
`In preparing this declaration, I have reviewed at least the documents
`
`V.
`
`labeled Exhibits 1001-1014 and other materials referred to herein in connection
`
`with providing this declaration. In addition to these materials, I have relied on my
`
`education, experience, and my knowledge of practices and principles in the
`
`relevant field, e.g., image processing. My opinions have also been guided by my
`
`appreciation of how one of ordinary skill in the art would have understood the
`
`claims and specification of the ’015 Patent around the time of the alleged
`
`invention, which I have been asked to assume is the earliest claimed priority date
`
`of July 26, 1996.
`
`44. Based on my experience and expertise, it is my opinion that certain
`
`references teach or suggest all the features recited in Claims 1-5 and 7 of the ’015
`
`Patent, as explained in detail below. Specifically, it is my opinion that Claims 1-2,
`
`4-5 are disclosed by U.S. Patent No. 5,481,622 (“Gerhardt”) in combination with
`
`U.S. Patent No. 6,044,166 (“Bassman”). It is also my opinion that Claims 3 and 7
`
`are also disclosed by Gerhardt in combination with Bassman and U.S. Patent No.
`
`5,521,843 (“Hashima”). It is also my opinion that Claims 1-5 and 7 are also
`
`disclosed by Alton L. Gilbert et. al., A Real-Time Video Tracking System
`
`(“Gilbert”) in combination with Gerhardt, and further in combination with . B.
`
`Schaming, Adaptive Gate Multifeature Bayesian Statistical Tracker (“Schaming”).
`
`19
`
`SAMSUNG EXHIBIT 1002
`Page 24 of 278
`
`

`

`Declaration of Dr. John C. Hart
`Inter Partes Review of U.S. Patent No. 7,650,015
`VI. LEVEL OF ORDINARY SKILL IN THE ART
`45. Based on my review of the ’015 Patent specification, claims, file
`
`history, and prior art, I believe one of ordinary skill in the art around the time of
`
`the alleged invention of the ’015 Patent would have had either (1) a Master’s
`
`Degree in Electrical Engineering or Computer Science or the equivalent plus at
`
`least a year of experience in the field of image processing, image recognition,
`
`machine vision, or a related field or (2) a Bachelor’s Degree in Electrical
`
`Engineering or Computer Science or the equivalent plus at least three years of
`
`experience in the field of image processing, image recognition, machine vision, or
`
`a related field. Additional education could substitute for work experience and vice
`
`versa.
`
`46.
`
`In determining the level of ordinary skill in the art, I was asked to
`
`consider, for example, the type of problems encountered in the art, prior art
`
`solutions to those problems, the rapidity with which innovations are made, the
`
`sophistication of the technology, and the educational level of active workers in the
`
`field.
`
`47. My opinions concerning the ’015 Patent claims are from the
`
`perspective of a person of ordinary skill in the art (“POSA”), as set forth above.
`
`20
`
`SAMSUNG EXHIBIT 1002
`Page 25 of 278
`
`

`

`Declaration of Dr. John C. Hart
`Inter Partes Review of U.S. Patent No. 7,650,015
`VII. CLAIM CONSTRUCTION
`48. For my analysis, I have interpreted all claim terms according to their
`
`plain and ordinary meaning.
`
`VIII. THE PRIOR ART TEACHES OR SUGGESTS EVERY STEP AND
`FEATURE OF THE CHALLENGED CLAIMS OF THE ’015 PATENT
`A. Overview Of The Prior Art References
`1.
`U.S. Patent No. 5,481,622 to Gerhardt (Ex. 1013)
`49. Lester A. Gerhardt and Ross M. Sabolcik, researchers at Rensselaer
`
`Polytechnic Institute, disclosed using a histogram of pixel characteristics to
`
`identify and track a target in digitized visual input in U.S. Patent No. 5,481,622
`
`(“Gerhardt”). Gerhardt issued on January 2, 1996.
`
`50. Gerhardt’s system tracks the position of a user’s pupil to generate
`
`input to the computer. This allows one to interact with a computer without using
`
`one’s hands. In one example, Gerhardt’s system uses a video camera mounted on
`
`a helmet, as shown in Figures 1 and 2.
`
`
`
`21
`
`SAMSUNG EXHIBIT 1002
`Page 26 of 278
`
`

`

`Declaration of Dr. John C. Hart
`Inter Partes Review of U.S. Patent No. 7,650,015
`51. Gerhardt’s system receives an input signal from a “camera means for
`
`acquiring a video image” and a “frame grabber means [that is] coupled to the
`
`camera means.” Ex. 1013, Gerhardt at 2:25-44. The “frame grabber” converts
`
`video data (which inherently contains a plurality of frames) to digital pixel data
`
`(plurality of pixels). For each frame input, Gerhardt generates a histogram based
`
`on the pixels’ intensity values to identify and track the user’s pupil. Ex. 1013,
`
`Gerhardt at 9:39-61. Gerhardt forms a histogram of the eye image with bins along
`
`the horizontal axis, where the “vertical axis indicates the pixel count of each bin,
`
`and the horizontal axis indicates the magnitude of the pixel intensity of each bin.”
`
`Ex. 1013, Gerhardt at 9:39-61. In one embodiment, Gerhardt teaches classification
`
`according to the continuous variable of intensity and that intensity may be
`
`“represented by a 7-bit greyscale, or in other words, divided up into 128 bins.” Id.
`
`Figure 5 shows an example histogram formed based on the eye image:
`
`22
`
`SAMSUNG EXHIBIT 1002
`Page 27 of 278
`
`

`

`Declaration of Dr. John C. Hart
`Inter Partes Review of U.S. Patent No. 7,650,015
`
`
`
`52. Gerhardt next identifies the pupil (i.e., the target) from the intensity
`
`histogram. Gerhardt divides the pixels into two sets based on an intensity
`
`threshold level—a darker set (pixels with intensity below the threshold) that has
`
`total pixel area substantially equal to the expected size of the use’s pupil in the eye
`
`image, and a lighter set (the remaining pixels). In the example shown in Figure 5,
`
`the threshold intensity (about 61) is chosen such that the pixels below the threshold
`
`(shown in black in Figure 5 above) take up about 5% of the image area.
`
`53. After finding the intensity threshold corresponding to the pupil (i.e.,
`
`the target), Gerhardt creates a binary image that shows only the pixels belonging to
`
`the pupil. Ex. 1013, Gerhardt at 10:6-34. A binary image created from the eye
`
`image is shown in Figure 6.
`
`23
`
`SAMSUNG EXHIBIT 1002
`Page 28 of 278
`
`

`

`Declaration of Dr. John C. Hart
`Inter Partes Review of U.S. Patent No. 7,650,015
`
`
`
`54. Once pixels belonging to the target (pupil) are identified in the
`
`histogram, Gerhardt “locat[es] the pupil, map[s] the pupil coordinates to display
`
`screen coordinate, and inform[s] peripheral devices of the pupil location.” Ex.
`
`1013, Gerhardt at 8:34-37. This is done by first identifying the “blobs” or “set[s]
`
`of contiguous pixels” in the image using a region-growing method. Ex. 1013,
`
`Gerhardt at 12:32-61. The system then “selects one of these blobs as
`
`corresponding to the user’s pupil” based on the blob’s properties (such as its size,
`
`centroid, X- and Y-minima and maxima of the pixels in the blob, the length-to-
`
`width ratio of the blob’s bounding rectangle, the perimeter of…the blo