UNITED STATES PATENT AND TRADEMARK OFFICE
`
`_________________________
`
`BEFORE THE PATENT TRIAL AND APPEAL BOARD
`
`_________________________
`
`Robert Bosch LLC,
`Petitioner
`
`v.
`
`Monument Peak Ventures, LLC,
`Patent Owner
`_________________________
`
`Case IPR2019-01473
`U.S. Patent No. 6,654,507
`_________________________
`
`DECLARATION OF DR. JOHN R. GRINDON, D.SC.
`
`BOSCH EXHIBIT 1004
`
`Page 1 of 43
`
`

`

`
`
`
`
`I.
`
`
`
`
`IPR2019-01473
`U.S. Patent No. 6,654,507
`
`TABLE OF CONTENTS
`Introduction ...................................................................................................... 1
`
`II.
`
`Qualifications ................................................................................................... 1
`
`III. Legal Standards ............................................................................................... 3
`
`IV. Level of Ordinary Skill in the Art and Materials Considered ......................... 7
`
`V.
`
`The ’507 Patent ................................................................................................ 8
`
`A. Disclosure and Claims ......................................................................... 10
`
`B.
`
`Prosecution History ............................................................................. 14
`
`VI. Claim Construction ........................................................................................ 15
`
`A.
`
`Belief Map ........................................................................................... 15
`
`B. Main Subject ........................................................................................ 17
`
`VII. Claims 1, 8, and 14 are unpatentable over the prior art ................................. 18
`
`A. Ground 1: Toyama, Itti, and Neubauer render claims 1, 8, and 14
`obvious. ............................................................................................... 18
`
`1.
`
`Prior Art Summary .................................................................... 18
`
`Toyama ........................................................................... 18
`a.
`Itti .................................................................................... 20
`b.
`Neubauer ......................................................................... 23
`c.
`Claim 1 ...................................................................................... 25
`
`e.
`
`d.
`
`Claim 1 [preamble]: “A method of producing an image of
`at least a portion of a digital image, comprising:” ......... 25
`Claim 1[a]: “a) providing a digital image having pixels;”
` ........................................................................................ 26
`Claim 8 ...................................................................................... 38
`
`2.
`
`3.
`
`f.
`
`“The method of claim 1 wherein the crop window is
`completely within the digital image.” ............................ 38
`
`
`
`i
`
`Page 2 of 43
`
`

`

`
`
`
`
`
`
`
`IPR2019-01473
`U.S. Patent No. 6,654,507
`
`4.
`
`Claim 14 .................................................................................... 38
`
`g.
`
`“A computer storage product having at least one
`computer storage medium having instructions stored
`therein causing one or more computers to perform the
`method of claim 1.” ........................................................ 38
`VIII. Conclusion ..................................................................................................... 39
`
`
`
`
`
`
`
`ii
`
`Page 3 of 43
`
`

`

`
`
`
`I.
`
`Introduction
`
`
`
`
`IPR2019-01473
`U.S. Patent No. 6,654,507
`
`I, Dr. John R. Grindon, declare as follows:
`
`1.
`
`I have been asked to submit this declaration on behalf of Robert Bosch
`
`LLC (“Petitioner”) in connection with a petition for inter partes review of U.S. Pa-
`
`tent No. 6,654,507 (“the ’507 patent,” Ex. 1001). Specifically, I have been retained
`
`as an independent expert consultant by Petitioner to provide my opinions on the
`
`technology claimed in, and the patentability or unpatentability of certain claims of
`
`the ’507 patent (“the challenged claims”). Although I am being compensated at my
`
`usual rate of $500 per hour for the time I spend on this matter, no part of my com-
`
`pensation depends on the outcome of this proceeding, I have no financial interest in
`
`any of the parties, and I have no other interest in this proceeding.
`
`II. Qualifications
`
`2. My qualifications for forming the opinions set forth in this report are
`
`summarized below and described in more detail in my curriculum vitae (attached as
`
`Ex. 1003). I am currently an independent consultant and provide services related to
`
`digital image processing and software algorithm development. For more than forty
`
`years, I have studied, researched, and worked in the analysis, design, and develop-
`
`ment of electronic systems, devices, and software related to acquiring, processing
`
`and analyzing signals and images. I hold various US and foreign patents in these
`
`areas.
`
`
`
`1
`
`Page 4 of 43
`
`

`

`
`
`
`
`
`
`
`IPR2019-01473
`U.S. Patent No. 6,654,507
`
`3.
`
`I received a Bachelor of Science in Electrical Engineering from the Uni-
`
`versity of Missouri at Rolla (now Missouri University of Science and Technology),
`
`in Rolla, Missouri in 1961. I then received a Master of Science in Electrical Engi-
`
`neering from MIT in Cambridge, Massachusetts in 1962. In 1960, I designed RF
`
`electronics for military radar systems for Westinghouse Electric Corporation, and in
`
`1961, I designed microwave electronics for Hughes Aircraft Corporation. I began
`
`working at McDonnell Aircraft Corp., which became McDonnell Douglas Corpora-
`
`tion and now Boeing, in 1962, and I received my Doctor of Science in Electrical
`
`Engineering from Washington University in St. Louis in 1970 while employed at
`
`McDonnell Douglas.
`
`4.
`
`I held various positions during my 25-year tenure at McDonnell Doug-
`
`las, including Branch Chief of Electronics. I led various research and development
`
`teams, including projects related to (1) tracking and guidance for the Tomahawk
`
`Cruise Missile; (2) developing scene analysis algorithms for 3D-imagery for auton-
`
`omous Cruise Missile guidance; and (3) developing image-based automatic target
`
`classification and recognition algorithms for anti-ship missiles.
`
`5.
`
`In 1987, I became Executive Vice President and Director of Research
`
`for Cencit, Inc., a start-up company engaged in the research and development of 3D
`
`electronic imaging systems based on digital video image processing electronics and
`
`software algorithms. While at Cencit, I developed the concept and led the analysis,
`
`
`
`2
`
`Page 5 of 43
`
`

`

`
`
`
`design, and implementation of a 3D computer vision system for non-contact shape
`
`IPR2019-01473
`U.S. Patent No. 6,654,507
`
`
`
`
`digitization and a computer-controlled four-axis DNC milling machine to replicate
`
`physical models of scanned objects. In 1990, I became an independent consultant
`
`working on various digital image processing algorithms.
`
`III. Legal Standards
`
`6.
`
`I have been informed by Petitioner’s counsel that the following legal
`
`principles apply to analysis of patentability based on 35 U.S.C. § 103 for obvious-
`
`ness. I have also been informed that, in an inter partes review proceeding, a patent
`
`claim is unpatentable if it is shown by a preponderance of the evidence that the claim
`
`would have been anticipated by a prior art patent or publication, or obvious by one
`
`or more properly-combined prior art patents or publications.
`
`7.
`
`I understand that under 35 U.S.C. § 103, “[a] patent for a claimed in-
`
`vention may not be obtained, notwithstanding that the claimed invention is not iden-
`
`tically disclosed as set forth in Section 102, if the differences between the claimed
`
`invention and the prior art are such that the claimed invention as a whole would have
`
`been obvious before the effective filing date of the claimed invention to a person
`
`having ordinary skill in the art to which the claimed invention pertains.”
`
`8. When considering the issues of obviousness, I have been instructed by
`
`counsel for Petitioner to do the following:
`
`(a) determine the scope and content of the prior art;
`
`
`
`3
`
`Page 6 of 43
`
`

`

`
`
`
`
`
`
`
`IPR2019-01473
`U.S. Patent No. 6,654,507
`
`(b) ascertain the differences between the prior art and the claims at is-
`
`sue;
`
`(c) resolve the level of ordinary skill in the pertinent art; and
`
`(d) consider evidence of secondary indicia of non-obviousness (if avail-
`
`able).
`
`9.
`
`I understand that the relevant time for considering whether a claim
`
`would have been obvious to a person having ordinary skill in the art (“POSA”) is
`
`the time of alleged invention. I have been asked to assume a priority date for the
`
`challenged claims of December 14, 2000.
`
`10.
`
`I have been told that a reference may be modified or combined with
`
`other references or with the POSA’s own knowledge if the person would have found
`
`the modification or combination obvious. A POSA is presumed to know all the rel-
`
`evant prior art, and the obviousness analysis may consider the inferences and crea-
`
`tive steps that a POSA would employ.
`
`11.
`
`In determining whether a prior-art reference could have been combined
`
`with another prior-art reference or other information known to a person having or-
`
`dinary skill in the art, I have been told that the following principles may be consid-
`
`ered:
`
`
`
`(a) a combination of familiar elements according to known methods is likely
`
`to be obvious if it yields predictable results;
`
`4
`
`Page 7 of 43
`
`

`

`
`
`
`
`
`
`
`IPR2019-01473
`U.S. Patent No. 6,654,507
`
`(b) the substitution of one known element for another is likely to be obvious
`
`if it yields predictable results;
`
`(c) the use of a known technique to improve similar items or methods in the
`
`same way is likely to be obvious if it yields predictable results;
`
`(d) the application of a known technique to a prior art reference that is ready
`
`for improvement is likely obvious if it yields predictable results;
`
`(e) any need or problem known in the field and addressed by the reference can
`
`provide a reason for combining the elements in the manner claimed;
`
`(f) a person of ordinary skill often will be able to fit the teachings of multiple
`
`references together like a puzzle; and
`
`(g) the proper analysis of obviousness requires a determination of whether a
`
`person of ordinary skill in the art would have a “reasonable expectation of
`
`success”—not “absolute predictability” of success—in achieving the claimed
`
`invention by combining prior art references.
`
`12.
`
`I have been told that whether a prior art reference renders a patent claim
`
`unpatentable as obvious is determined from the perspective of a POSA. Further, I
`
`understand that while there is no requirement that the prior art contain an express
`
`suggestion to combine known elements to achieve the claimed invention, a sugges-
`
`tion to combine known elements to achieve the claimed invention may come from
`
`the prior art as a whole or individually, as filtered through the knowledge of one
`
`
`
`5
`
`Page 8 of 43
`
`

`

`
`
`
`skilled in the art. I have also been told that the inferences and creative steps a person
`
`IPR2019-01473
`U.S. Patent No. 6,654,507
`
`
`
`
`of ordinary skill in the art would employ are also relevant to the determination of
`
`obviousness.
`
`13.
`
`I have been told that when a work is available in one field, design alter-
`
`natives and other market forces can prompt variations of it, either in the same field
`
`or in another. If a POSA can implement a predictable variation and would see the
`
`benefit of doing so, that variation is likely to be obvious. In many fields, there may
`
`be little discussion of obvious combinations, and in these fields market demand—
`
`not scientific literature—may drive design trends. When there is a design need or
`
`market pressure and there are a finite number of predictable solutions, a POSA has
`
`good reason to pursue those known options.
`
`14.
`
`I have been told that there is no rigid rule that a reference or combina-
`
`tion of references must contain a “teaching, suggestion, or motivation” to combine
`
`references. But I also have been told that the “teaching, suggestion, or motivation”
`
`test can be a useful guide in establishing a rationale for combining elements of the
`
`prior art. This test poses the question whether there is an express or implied teaching,
`
`suggestion, or motivation to combine prior art elements in a way that yields the
`
`claimed invention and avoids impermissible hindsight analysis.
`
`
`
`6
`
`Page 9 of 43
`
`

`

`
`
`
`
`
`
`
`IPR2019-01473
`U.S. Patent No. 6,654,507
`
`15.
`
`I have been informed that, in this proceeding, claim terms in the ’507
`
`patent are given their ordinary and customary meaning to a person of ordinary skill
`
`in the art at the time of the invention.
`
`16.
`
`I have been asked to assume that December 14, 2000, is the date of the
`
`invention, as that is the earliest date to which the ’507 patent claims priority. I have
`
`not conducted a separate analysis to determine whether the challenged claims are
`
`entitled to this priority date, as I understand that the prior art references discussed
`
`below qualify as prior art even if all claims are deemed to be entitled to the December
`
`14, 2000, priority date.
`
`IV. Level of Ordinary Skill in the Art and Materials Considered
`
`17. The ’507 patent is directed to cropping a digital image by using a belief
`
`map of the image to identify main subjects. In my opinion, a person of ordinary skill
`
`in the art (“POSA”) of the ’507 patent would have a bachelor’s degree in electrical
`
`engineering, computer science, physics, or a related field, and two to four years of
`
`experience (or the academic equivalent) in the field of computer/machine vision or
`
`image processing. Less work experience may be compensated by a higher level of
`
`education, and vice versa.
`
`18.
`
`In forming my opinions expressed in this declaration, I have consid-
`
`ered, among other things, the following documents. I understand the documents have
`
`been given the following exhibit numbers in this proceeding:
`
`
`
`7
`
`Page 10 of 43
`
`

`

`
`
`
`
`Exhibit
`Ex. 1001
`Ex. 1002
`Ex. 1005
`Ex. 1006
`
`Ex. 1007
`
`
`
`
`IPR2019-01473
`U.S. Patent No. 6,654,507
`
`Description
`U.S. Patent No. 6,654,507 to Jiebo Luo et al. (the ’507 patent”).
`Patent Prosecution History of U.S. Patent No. 6,654,507.
`U.S. Patent No. 6,792,135 to Toyama. (“Toyama”).
`Itti, Laurent et al., “A Model of Saliency-Based Visual Attention for
`Rapid Scene Analysis,” IEEE Transactions on Pattern Analysis and
`Machine Intelligence, Vol. 20, No. 11, November 1998 (“Itti”).
`U.S. Patent No. 6,553,131 to Neubauer et al. (“Neubauer”).
`
`
`In forming my opinions, I have also relied on my education and experience.
`
`V. The ’507 Patent
`
`19. The challenged claims are directed to a system and method for auto-
`
`matically creating cropped and zoomed versions of photographic images. ’507 pa-
`
`tent, Abstract. As the ’507 patent admits, conventional systems already cropped im-
`
`ages based on, for example, “relatively homogenous margins around the borders of
`
`an image” or “different intensity levels within the image.” Id., 2:22-23, 32-33. The
`
`’507 patent claims that the major drawback of these techniques is that “both tech-
`
`niques cannot deal with images with nonuniform backgrounds.” Id., 2:50-51. The
`
`patent asserts that this resulted in manual cropping still being preferred over those
`
`automated methods.
`
`20. The ’507 patent asserts that the advantage of the disclosed invention is
`
`the ability to automatically crop and zoom images based on a so-named “belief”
`
`map, which helps identify areas of interest in an image based upon their saliency in
`
`the map. ’507, 3:53-60. But belief, or saliency, maps were not novel as of the filing
`
`
`
`8
`
`Page 11 of 43
`
`

`

`
`
`
`date of the ’507 patent. In fact, such maps (and algorithms for developing such maps)
`
`IPR2019-01473
`U.S. Patent No. 6,654,507
`
`
`
`
`were already known and taught by others, including algorithms based on research
`
`attempting to model the way humans (and other primates) detect objects of interest
`
`in an image. Itti, p. 1254. Simply automating human behavior by applying a known
`
`technique for modeling such behavior is not inventive. See Princeton Biochemicals,
`
`Inc. v. Beckman Coulter, Inc., No. CIV.A. 96-5541 (MLC), 2004 WL 1398227, at
`
`*18 (D.N.J. June 17, 2004), aff'd, 411 F.3d 1332 (Fed. Cir. 2005) (“It has been well-
`
`established that the obvious automation of a known manual process is not patenta-
`
`ble.”) (citing In re Vener & Bowser, 262 F.2d 91(C.C.P.A. 1958), and In re Rundell,
`
`48 F.2d 958 (C.C.P.A. 1931)). And the technique claimed by the ’507 patent of lo-
`
`cating an object of interest in an image and cropping around it was similarly con-
`
`ventional. See Itti, p. 1256 (discussing the focus of attention as “a simple disk”).
`
`Indeed, the challenged claims recite nothing more than automating manual processes
`
`by combining well-known machine vision principles to determine main subjects in
`
`a photograph (that model the first step of the manual process) and well-known tech-
`
`niques to crop and zoom images (that model the second step of the manual process).
`
`Such automation of a manual process is not patentable. See Princeton Biochemical ,
`
`2004 WL 1398227, at *18. Therefore, with further reference to the following infor-
`
`mation discussed below, the Board should institute inter partes review of the ’507
`
`patent and cancel claims 1, 8, and 14 based on the grounds presented.
`
`
`
`9
`
`Page 12 of 43
`
`

`

`
`
`
`
`
`
`
`IPR2019-01473
`U.S. Patent No. 6,654,507
`
`21. The ’507 patent provides “[a] method of producing an image of at least
`
`a portion of a digital image.” ’507 patent, Abstract. Before the ’507 patent, there
`
`were two main types of prior art: one examines images in a line-by-line fashion and
`
`the other, examines images block-by-block to identify uniform areas. The first type
`
`automatically cropped images by (1) examining image borders; (2) determining, by
`
`using grayscale images, whether a row of pixels is to be cropped based on pixel
`
`variation; (3) determining if a row of pixels passes the criterion to be included in the
`
`final cropped image; (4) examining the next row inward until all rows are complete;
`
`and (5) determining the final cropped image. Id., 2:8-21. In other words, these prior
`
`art programs removed homogenous margins around the image border and did not
`
`examine image contents. Id., 2:22-24. The second type cropped images based on
`
`different intensity levels within an image. Id., 2:30-40.
`
`22. The ’507 patent purports to improve prior art systems by using a belief
`
`map of the image to identify main subjects. Id., 3:28-32. Belief mapping has been
`
`used in various computer vision applications, and using it to crop an image would
`
`have been obvious to one of ordinary skill.
`
`A. Disclosure and Claims
`
`23. The ’507 patent claims methods of and systems for cropping a digital
`
`image comprising three main steps: (1) developing a belief map indicative of main
`
`subjects in the image; (2) determining a crop window; and (3) cropping the image to
`
`
`
`10
`
`Page 13 of 43
`
`

`

`
`
`
`include a portion of the image with high subject content. See generally ’507 Patent,
`
`IPR2019-01473
`U.S. Patent No. 6,654,507
`
`
`
`
`Claims 1-28. Claim 1 is illustrative of the challenged claims:
`
`1. A method of producing an image of at least a portion of a digital im-
`age, comprising:
`
`a) providing a digital image having pixels;
`
`b) computing a belief map of the digital image by using the pixels
`of the digital image to determine a series of features and using such fea-
`tures to assign a probability of a location of a main subject of the digital
`image in the belief map;
`
`c) determining a crop window having a shape factor and a zoom
`factor, the shape and the zoom factors determining a size of the crop
`window; and
`
`d) cropping the digital image to include a portion of the image of
`high Subject content in response to the belief map and the crop window.
`
`Id., 11:57-12:3.
`
`24. The specification of the ’507 patent describes in significantly more de-
`
`tail a process for automatically cropping an image. For example, in explaining how
`
`a belief map is created, the ’507 patent describes a main subject detection (“MSD”)
`
`system that provides a “discriminative” “measure of saliency or relative importance
`
`for different regions” in an image for a number of applications, “including automatic
`
`crop and zoom.” Id., 4:48-51, 4:53. The MSD system performs various sub-tasks,
`
`including: “region segmentation, perceptual grouping, feature extraction, and prob-
`
`
`
`11
`
`Page 14 of 43
`
`

`

`
`
`
`abilistic and semantic reasoning.” Id., 4:65-66. Features are extracted for each seg-
`
`IPR2019-01473
`U.S. Patent No. 6,654,507
`
`
`
`
`mented image region to represent a variety of visual saliency properties which are
`
`input into a probability network to create a non-binary belief map. Id., 4:67-5:4.
`
`25. Through MSD, regions belonging to a main subject are differentiated
`
`from the image background. Id., 5:5-7. The MSD system outputs a “list of segmented
`
`regions ranked in descending order of their likelihood (or belief) as potential main
`
`subjects for a generic or specific application” and can also be converted into a belief
`
`map, which can be represented for visual purposes as an image wherein the region’s
`
`brightness is proportional to the belief or probability that the region is a main subject.
`
`Id., 5:17-20, 5:25-27 (“The associated likelihood is also attached to each region so
`
`that regions with large values correspond to regions with high confidence or be-
`
`lief…”). These “brighter” regions, i.e. those having larger belief values, represent
`
`regions with high confidence of being part of the main subject. Id., 5:26-28.
`
`26. FIG. 3 of the ’507 patent further explains how it uses the belief map
`
`results for additional image processing . In step 200, the image is input, and a belief
`
`map is created using MSD. Id., 7:1. A zoom factor, based on either an operator’s
`
`selection or determined automatically on a main subject size estimate, and a cropped
`
`window are selected in step 201. Id., 7:2-7.
`
`
`
`12
`
`Page 15 of 43
`
`

`

`
`
`
`
`
`
`
`IPR2019-01473
`U.S. Patent No. 6,654,507
`
`
`
`27.
`
`In step 201, belief map regions are clustered using a k-means clustering
`
`process and the lowest belief cluster, or the background, is zeroed using a predefined
`
`threshold. Id., 7:16-18. Next, the centroid of non-zero beliefs are computed in step
`
`202, and a crop window is centered at the centroid in step 204. Id., 7:23-25, 8:5-6.
`
`The centroid is calculated by summing the values of the beliefs at each pixel loca-
`
`tion. Id., 7:29-38.
`
`28.
`
`In step 205, the crop window is moved so that the entire crop window
`
`is within the original image. Id., 8:7-8. The cropped window is then moved again in
`
`
`
`13
`
`Page 16 of 43
`
`

`

`
`
`
`item 206 so that all regions of the highest belief subject are included in the cropped
`
`IPR2019-01473
`U.S. Patent No. 6,654,507
`
`
`
`
`window, thereby capturing the entire subject of interest. Id., 8:9-12. Step 207 deter-
`
`mines whether an “acceptable solution” has been found, meaning whether it is pos-
`
`sible to include “at least the regions of the highest belief values” in the cropping
`
`window. Id., 8:16-19. If an acceptable solution exists, the window is moved again in
`
`item 208 to maximize the belief sums within the window. Id., 8:20-22. If an accepta-
`
`ble solution is not produced, the final position of the crop window is restored to that
`
`of step 205. Id., 9:1-3.
`
`B.
`
`Prosecution History
`
`29. During prosecution of the application that issued as the ’507 patent, the
`
`Examiner rejected challenged claims 1, 8, and 14 under 35 U.S.C. § 102(e) as antic-
`
`ipated by Jia (U.S. Patent No. 6,430,320). Ex. 1002, p. 77-80. The examiner also
`
`rejected claims 1, 8, and 14 for obviousness-type double patenting. Id., p. 81
`
`30. The applicant also argued that Jia did not disclose “belief maps” and
`
`agreed to file a terminal disclaimer upon an indication of allowability. Id., 275-79.
`
`31. The Examiner allowed claims 1-28 because “none of the prior art, teach
`
`or fairly suggest computing a belief map of the digital image, by using the pixels of
`
`the digital image to determine a series of features, and using such features to assign
`
`a probability of a location of a main subject of the digital image in the belief map.”
`
`Id., 291. The examiner noted that Jia (US Patent 6,430,320) discloses “a method of
`
`
`
`14
`
`Page 17 of 43
`
`

`

`
`
`
`automatically cropping an image according to a belief of where in the image a main
`
`IPR2019-01473
`U.S. Patent No. 6,654,507
`
`
`
`
`subject is located …. [but] does not each or fairly suggest the use of a belief map in
`
`which a probability of the location of the main subject is assign[ed].” Id., 291-92.
`
`32. The Examiner, however, never considered the references applied in the
`
`petition— Toyama, Itti or Neubauer—nor were these references cited by the appli-
`
`cant. See Ex. 1002; ’507 patent, cover page. The proposed ground teaches a belief
`
`map with corresponding belief values that the Examiner found lacking in the prior
`
`art considered and which led to the allowance of the claims.
`
`VI. Claim Construction
`
`33. A claim in a patent subject to inter partes review is subject to the claim
`
`construction standard of Phillips v. AWH Corp., 415 F.3d 1303 (Fed. Cir. 2005) (en
`
`banc). See 83 Fed. Reg. 51,340 (October 11, 2018). Under the Phillips standard,
`
`claims are given their ordinary and customary meaning to a person of ordinary skill
`
`in the art at the time of the invention. Phillips, 415 F.3d at 1313. The following terms
`
`requires construction for this proceeding. Claim terms not addressed below should
`
`be given their plain and ordinary meaning.
`
`A. Belief Map
`
`34.
`
`In the ’507 patent, a “belief map” as used in claim 1 means a probabil-
`
`ity-related map indicating importance of a region by providing a measure of relative
`
`
`
`15
`
`Page 18 of 43
`
`

`

`
`
`
`importance for different regions in an image. Synonyms for “belief” maps are “like-
`
`IPR2019-01473
`U.S. Patent No. 6,654,507
`
`
`
`
`lihood,” “probability,” or “saliency” maps.
`
`35. The ’507 patent states that the “‘belief’ map is more than a binary map
`
`that only indicates location of the determined main subject;” instead, the “associated
`
`likelihood is also attached to each region so that regions with large values correspond
`
`to regions with high confidence or belief of being part of the main subject.” ’507
`
`patent, 5:23-28 (emphasis added); see also id., 5:17-22 (explaining that the main
`
`subject detection (MSD) output used “is a list of segmented regions ranked in de-
`
`scending order of their likelihood (or belief) as potential main subjects for a generic
`
`or specific application. This list can be readily converted into a map in which the
`
`brightness of a region is proportional to the main subject belief of the region.”).
`
`36. Each pixel in the belief map corresponds to a pixel in the original image
`
`and is assigned a likelihood value of the region where it is in the original image. See
`
`id., 7:36-38. The region(s) with the highest likelihood value are designated as the
`
`main subject. Thresholds may be applied to select other belief values—those that are
`
`lower than the highest but higher than the lowest belief values—and designate them
`
`as secondary main subjects. Id., 5:51-54 (explaining that these secondary main sub-
`
`jects can be “included according to a descending order of belief values once the main
`
`subject of highest belief values are included.”); see also id., 8:43-46, 9:59-67. Ac-
`
`cordingly, Petitioner requests that “belief map” be construed as “a map indicating
`
`
`
`16
`
`Page 19 of 43
`
`

`

`
`
`
`importance of a region by providing a measure of relative importance for different
`
`IPR2019-01473
`U.S. Patent No. 6,654,507
`
`
`
`
`regions in an image.”
`
`B. Main Subject
`
`37. A “main subject,” as used in claim 1 is a photographic subject having a
`
`highest belief value in a belief map. See ’507 patent, 4:38-40 (“Main subject detec-
`
`tion provides a measure of saliency or relative importance for different regions that
`
`are associated with different subjects in an image.”) The ’507 patent states that the
`
`“output of [main subject detection] … is a list of segmented regions ranked in de-
`
`scending order of their likelihood (or belief) as potential main subjects.” Id., 5:17-
`
`19. The ’507 patent explains that the belief map “reflects the inherent uncertainty”
`
`of determining a main subject because “different observers may disagree on certain
`
`subject matter while agreeing on other subject matter in terms of main subjects.” Id.,
`
`5:29-33.
`
`38. This list of regions can be converted to a map where the region bright-
`
`ness is proportional to the main subject belief of the region. Id., 5:20-22; see also
`
`id., 5:25-28 (explaining that the belief map attaches a likelihood “to each region so
`
`that regions with large values correspond to regions with high confidence or belief
`
`of being part of the main subject”) (emphases added). However, a “binary decision”
`
`can also be obtained by “using an appropriate threshold on the belief map.” Id., 5:34-
`
`36.
`
`
`
`17
`
`Page 20 of 43
`
`

`

`
`
`
`VII. Claims 1, 8, and 14 are unpatentable over the prior art
`
`
`
`
`IPR2019-01473
`U.S. Patent No. 6,654,507
`
`A. Ground 1: Toyama, Itti, and Neubauer render claims 1, 8, and 14
`obvious.
`
`39. Toyama discloses a system and method for producing at least a portion
`
`of an image by determining a crop window and cropping the image to a portion of
`
`the image containing a main subject as claimed. Toyama suggests computing a belief
`
`map to determine the location of the main subject, and Itti and Neubauer resolve any
`
`dispute that computing a belief map and assigning a probability of a main subject’s
`
`location was known. Itti discloses a belief map through its saliency map that repre-
`
`sents “the conspicuity—or ‘saliency’—at every location” in the image created from
`
`normalized conspicuity maps representing various features. Itti, p. 1255. Neubauer
`
`similarly discloses a probability-like saliency map where the “highest peak values
`
`in the intensity regions have the highest probability of being the license plate and
`
`are selected for further processing.” Neubauer., 4:29-31, 5:4-7.
`
`1.
`
`Prior Art Summary
`
`a.
`
`Toyama
`
`40. As shown below in FIGS. 3 and 4, Toyama discloses a face detection
`
`system 210 and method for detecting a face within an image. Toyama, 2:18-23. The
`
`system includes: (1) a hypothesis module that defines a sub-region in which to search
`
`for a face; (2) a feature extraction module for extracting image feature values based
`
`on a non-intensity image property; and (3) a relational template module that uses a
`
`
`
`18
`
`Page 21 of 43
`
`

`

`
`
`
`relational template and facial regions to determine whether a face has been detected.
`
`IPR2019-01473
`U.S. Patent No. 6,654,507
`
`
`
`
`Id., 2:30-42.
`
`
`
`41. First, the face detection system receives the raw image 200 and sends
`
`it to a hypothesis module 300. Id., 6:32-36. The hypothesis module generates a hy-
`
`pothesis and defines the dimensions of a sub-region in the raw image 200 (or cropped
`
`image) where a face may be found. Id. The cropped image is preprocessed by pre-
`
`processing module 320, id., 6:36-38, before being sent to a feature extraction mod-
`
`ule, id., 6:38-40.
`
`
`
`19
`
`Page 22 of 43
`
`

`

`
`
`
`
`
`
`
`IPR2019-01473
`U.S. Patent No. 6,654,507
`
`
`
`42. Based on a relational analysis of various features extracted from the
`
`cropped image, the system 210 determines whether a face has been detected in a
`
`cropped image (box 360). Id., 6:53-55. If a face is not found in the sub-region, a
`
`different sub-region is produced (box 370) by returning to the hypothesis module
`
`300 and another hypothesis is generated about where a face may be located within
`
`the image 200. Id., 6:55-60. If a face is detected in the cropped image, the face in-
`
`formation, including the location and dimensions of the cropped image, is sent as an
`
`output (380). Id., 6:62-67.
`
`b.
`
`Itti
`
`
`
`20
`
`Page 23 of 43
`
`

`

`
`
`
`
`
`
`
`IPR2019-01473
`U.S. Patent No. 6,654,507
`
`43.
`
`“A Model of Saliency-Based Visual Attention for Rapid Scene Analy-
`
`sis” (“Itti”) is an article published by the IEEE in November 1998. Ex. 1006. Ac-
`
`cording to the IEEE Xplore database, the article appears in volume 20, issue 11 of
`
`the “IEEE Transactions on Pattern Analysis and Machine Intelligence magazine,