Do Applicant Patent Citations Matter?1
`Christopher A. Cotropia2
`
`Mark Lemley3
`
`Bhaven Sampat4
`
`
`
`
`
`1 © 2012 Christopher A. Cotropia, Mark A. Lemley, & Bhaven Sampat. We thank Tejas Narenchania for
`excellent research assistance and Ashish Arora, Ernie Beffel, Dawn-Marie Bey, Dennis Crouch, Andrew
`Daughety, Rose Hagan, Arti Rai, David Schwartz, Robert Schwartzman, Ted Sichelman, Polk Wagner,
`two anonymous reviewers, and participants in workshops at the Stanford Law School, the Berkeley Law
`School, Northwestern University Law School, and the Cardozo Law School for comments on a previous
`draft.
`2 Professor of Law, University of Richmond School of Law.
`3 William H. Neukom Professor, Stanford Law School; partner, Durie Tangri LLP.
`4 Assistant Professor, Mailman School of Public Health, Columbia University.
`
`PMC Exhibit 2023
`Apple v. PMC
`IPR2016-00754
`Page 1
`
`

`
`Do Applicant Patent Citations Matter?
`
`Abstract
`
`Patent law both imposes a duty on patent applicants to submit relevant prior art to the
`
`PTO and assumes that examiners use this information to determine an application's
`
`patentability. In this paper, we examine the validity of these assumptions by studying the use
`
`made of applicant-submitted prior art by delving into the actual prosecution process in over a
`
`thousand different cases. We find that patent examiners rarely use applicant-submitted art in
`
`their rejections to narrow patents, relying almost exclusively on prior art they find themselves.
`
`Our findings have implications for a number of important legal and policy disputes, including
`
`initiatives to improve patent quality and the strong presumption of validity the law grants issued
`
`patents—a presumption that makes patents more difficult to challenge in court.
`
`1.
`
`INTRODUCTION
`
`
`
`Patent law imposes a duty of candor on patent applicants. They must disclose any
`
`material prior inventions, uses, and publications (“prior art”) of which they are aware to the
`
`Patent and Trademark Office (PTO); failure to do so can render the resulting patent
`
`unenforceable. The idea is that applicants should help patent examiners decide whether an
`
`invention is patentable by submitting what is likely to be the most relevant information. And we
`
`trust that examiners will do so; when the patent issues we imbue it with a strong presumption of
`
`validity.
`
`
`
`In this paper, we study the use made of those submitted prior art references by delving
`
`into the actual prosecution process in over a thousand different issued patents. We find, to our
`
`surprise, that patent examiners did not use applicant-submitted art in the rejections that narrowed
`
`PMC Exhibit 2023
`Apple v. PMC
`IPR2016-00754
`Page 2
`
`

`
`claims before these patents issued, relying almost exclusively on prior art they find themselves.
`
`This is not simply because the applicant has “drafted around” the art they submitted. Even late-
`
`submitted art is not commonly used by examiners in their rejections. Nor does the explanation
`
`appear to be that applicant art is uniformly weak. We also provide evidence suggesting
`
`examiners are less likely to use prior art discovered by foreign search authorities for the same
`
`invention, art that is presumably of better than average quality and relevance. Taken together, the
`
`evidence points towards myopia as a plausible explanation: examiners tend to focus on
`
`references that they themselves identify.
`
`
`
`Subject to the caveats discussed below, our findings have implications for policy
`
`initiatives that aim to improve patent quality through bringing more “prior art” before examiners,
`
`under the theory that with better access to prior art they would be less likely to issue patents of
`
`questionable validity. These include proposals that encourage outsourcing of search to
`
`applicants, third party searchers, or worksharing with foreign patent offices. However, if
`
`examiners pay attention mainly to art they find for themselves, these proposals might generate
`
`prior art that will fall on deaf ears and go unused. Our results also have implications for patent
`
`law: it is far from clear that the law should presume a patent valid over applicant cited art if the
`
`examiner has not given much consideration to these references. The presumption of validity,
`
`which makes a patent harder to challenge in a patent infringement litigation, is based on the
`
`assumption that patent office thoroughly tested the patent’s claims for validity. Our findings
`
`bring this assumption into question. They therefore have implications for current policy debates
`
`that have occupied both the courts and Congress. Our findings also have implications for the
`
`legal doctrine of inequitable conduct, the willful failure to submit prior art to the PTO. Such
`
`conduct may be less of a problem than previously thought – not because applicants don’t try to
`
`PMC Exhibit 2023
`Apple v. PMC
`IPR2016-00754
`Page 3
`
`

`
`deceive the PTO, but because any effort to do so may be wasted.5 Finally, our findings appear to
`
`challenge some of the assumptions underlying the use of citation data as indicators by scholars of
`
`research policy, joining a growing literature analyzing the economic and social meaning of
`
`citation-based indicators (e.g. Alcacer et al. 2009; Harhoff et al . 2003; Meyer 2000).
`
`
`
`In Part 2 we provide background on the collection of information in the patenting
`
`process, and the presumption of validity that results. We present our data in Part 3. In Part 4 we
`
`discuss implications. In Part 5 we summarize and conclude.
`
`2.
`
`PATENT EXAMINATION, PRIOR ART, AND THE PRESUMPTION OF VALIDITY
`
`
`
`The Patent and Trademark Office (PTO) examines patent applications to decide whether
`
`the government should issue a patent. An applicant submits a description of the invention, along
`
`with what they propose to claim as their own. Applicants must also submit any “prior art”
`
`(relevant prior publications and inventions) of which they are aware, but have no obligation to
`
`search for prior art. (Cotropia 2009).
`
`
`
`Applications are assigned to examiners skilled in the general field of the invention.
`
`Those examiners are tasked with reading the application, conducting their own prior art search,
`
`reading and evaluating that art, and evaluating the application in a back-and-forth written (and
`
`sometimes oral) colloquy with the applicant. Examiners have a significant caseload, and can
`
`devote on average only about 18 hours over three to five years to searching for, evaluating, and
`
`5 The doctrine of inequitable conduct may still be important when applied to non-prior-art information
`like false assertions of unexpected results to overcome obviousness or representations regarding whether
`a prior use by the applicant was experimental or not. Thus, our findings do not argue for elimination of
`the inequitable conduct doctrine entirely, but they do call into question the most common use of that
`doctrine.
`
`
`
`PMC Exhibit 2023
`Apple v. PMC
`IPR2016-00754
`Page 4
`
`

`
`applying the prior art to the application. (Lemley 2001). Applicants dissatisfied with the
`
`examiner’s decision can (and often do) refile one of several forms of continuation application to
`
`try again. At the end of the day, applicants obtain patents in about 75% of all cases. (Lemley
`
`and Sampat 2009).
`
`
`
`Issued patents enjoy a strong presumption of validity that can be overcome only by clear
`
`and convincing evidence. The theory underlying that presumption is that the PTO has vetted the
`
`patent and their expert opinion is entitled to deference. If a scientific expert has considered
`
`whether the patent should be granted, a court is reluctant to second-guess that judgment lightly.
`
`
`
`In recent years the strong presumption of validity afforded patents has come in for
`
`substantial criticism. (Lemley, Lichtman and Sampat 2005; Lichtman and Lemley 2007). Critics
`
`point out that the PTO operates under substantial resource constraints (Lemley 2001), with
`
`skewed incentives (Lemley and Moore 2004), and without the benefit of third-party participation
`
`(Thomas 2001). Further, the PTO ultimately issues a patent to a large majority of the applicants
`
`who seek one. (Lemley and Sampat 2008). And nearly half of the patents that do issue and are
`
`later litigated turn out to be invalid. (Allison and Lemley 1998). The PTO seems positioned to
`
`narrow claims in patent applications, but generally not to reject applications (Lemley and
`
`Sampat 2012).
`
`
`
`Some of those bad patents have costs. (Farrell and Merges 2004). At a minimum they
`
`impose substantial attorneys’ fees on defendants, a median of $5 million per case. (AIPLA
`
`Survey 2009). They may also lead small companies to drop products rather than defend their
`
`legality (Chien 2009; Graham and Sichelman 2008) and cause others to pay too much money to
`
`license the patent rather than face the risk of an injunction. (Lemley and Shapiro 2007). And in
`
`PMC Exhibit 2023
`Apple v. PMC
`IPR2016-00754
`Page 5
`
`

`
`the biomedical industries, the mere issuance of a patent invokes a set of regulatory mechanisms
`
`that blocks market entry by competitors for a substantial period of time. (Hemphill and Sampat
`
`2012).
`
`
`
`Despite these problems, it seems unlikely that the courts or Congress will soon eliminate
`
`or weaken the presumption of validity. It is more plausible that they will change the structure of
`
`the presumption. Before 1982, the presumption was largely limited to prior art actually
`
`considered by the patent examiner6 (Lemley and Lichtman 2007). The Federal Circuit expanded
`
`that presumption to apply to all invalidity arguments, whether or not they had been considered by
`
`the examiner (Ultra-Tex Surfaces, Inc. v. Hill Bros. Chem. Co. [Fed. Cir. 2000]). The
`
`application of the presumption to prior art not before the PTO was called into question by the
`
`Supreme Court in KSR Int'l Co. v. Teleflex Inc. which said "the rationale underlying the
`
`presumption—that the PTO, in its expertise, has approved the claim—seems much diminished"
`
`with regard to art not before the PTO. But the Supreme Court declined to change that
`
`presumption in 2011 in Microsoft Corp. v. i4i LP. It did, however, hold that courts should
`
`instruct jurors that it was easier to prove a patent invalid using evidence not considered by the
`
`patent examiner.
`
`
`
`As a practical matter, whether prior art was before the examiner turns out to matter
`
`whether or not the law says it does. Allison and Lemley find that factfinders are far more likely
`
`to invalidate a patent on the basis of prior art not before the PTO. (Allison and Lemley 1998).
`
`The reason is intuitive: judges and juries not technically trained are unlikely to second-guess the
`
`decision of the expert examiner to allow the patent, but they may be much more receptive to
`
`6 Before 2001, it was impossible to distinguish applicant-provided from examiner-provided art, so the
`rule was applied to art cited on the face of the patent.
`
`PMC Exhibit 2023
`Apple v. PMC
`IPR2016-00754
`Page 6
`
`

`
`arguments that the examiner never got to hear about a particular piece of prior art and would
`
`have rejected the application if she had.
`
`
`
`That fact in turn gives applicants who hope to enforce their patents an incentive to
`
`disclose, and perhaps even to search for, prior art (Sampat 2010). Disclosure of prior art to the
`
`PTO can help “bulletproof” a patent in later litigation. There are other incentives for disclosure
`
`as well. The PTO imposes a duty of candor on applicants, requiring them to disclose what they
`
`know. A deliberate failure to disclose material prior art is inequitable conduct that renders the
`
`entire patent unenforceable. (Cotropia 2009). Applicants may weigh these incentives to disclose
`
`against the perceived risk of disclosure: that the PTO will refuse to grant a patent at all, or will
`
`narrow it in ways that render it less useful. Prior work suggests that that balance differs by
`
`industry; at a minimum, applicants are much more likely to search in some industries than in
`
`others. (Lemley and Sampat 2012).
`
`
`
`It is reasonable ex ante to assume that applicant-submitted prior art will be the most
`
`important to the evaluation of the application. After all, the applicant is closest to the invention,
`
`and ought to be best positioned to know about the most relevant prior art. (Cotropia 2009).
`
`Further, important categories of prior art such as prior sales and public uses do not show up in
`
`publications, and are accordingly unlikely to be found by patent examiners. That information is
`
`more likely to be in the possession of applicants. (Cotropia 2009).
`
`
`
`Thus, it matters to the law whether an examiner actually has prior art in front of them
`
`when conducting the examination. It is quite plausible that it will matter more in the near future.
`
`And there are ex ante reasons to assume that the most significant prior art will be that submitted
`
`to the examiner by the applicant (Cotropia 2009).
`
`PMC Exhibit 2023
`Apple v. PMC
`IPR2016-00754
`Page 7
`
`

`
`3.
`
`HOW EXAMINERS EVALUATE PRIOR ART
`
`
`
`
`
`3.1.
`
`Sources of Prior Art
`
`Given the importance of cited prior art in later litigation, we set out to examine how
`
`examiners actually consider prior art, recognizing the individual variation in examiner behavior.
`
`(Cockburn et al., 2003; Tu, 2012; Lemley and Sampat 2012). In particular, we explore in this
`
`paper the roles different types of prior art from different sources play in the prosecution process
`
`for issued patents. Do examiners—themselves constrained in their searching ability—actually
`
`pay attention to applicant disclosed art, relying on it in rejecting applications during
`
`prosecution?7
`
`
`
`Answering this question requires case-by-case evaluation of prosecution histories. We
`
`started with a 1% random sample of all utility patents issued in 2007, or 1564 patents. For each
`
`of these patents, we collected all applicant and examiner references, by type.
`
`
`
`Since 2001, front pages indicate whether references come from examiner searches or
`
`applicant Information Disclosure Statements (IDS) (Sampat 2010; Alcacer, Gittleman and
`
`Sampat 2009; Thompson 2006). We present the summary results in Table 1.
`
`
`
`
`
`7 We do not look at use of applicant information outside of references, like prior sales, public uses, etc.,
`that could render a patent claim invalid.
`
`PMC Exhibit 2023
`Apple v. PMC
`IPR2016-00754
`Page 8
`
`

`
`Table 1: Prior art references in a 1% sample of patents issued in 2007, by type and source
`
`
`Type of Prior Art
`
`Total Number of Citations
`
`Share From Applicants
`
`Foreign
`
`Non-Patent Literature
`
`Patent
`
`Total
`
`
`
`6021
`
`9444
`
`26,932
`
`42,397
`
`94%
`
`94%
`
`66%
`
`76%
`
`Over three-quarters of the submitted art against which patentability is evaluated (32,208 /
`
`42,397 references) comes from applicants. Overall, most of the art (64%) is previous U.S.
`
`patents or patent applications. Notably, examiners account for a much larger share of citations to
`
`U.S. patents than of other types of art. Examiners account for 34% of citations to U.S. patents,
`
`versus 6 percent for non-patent art and for foreign patents.8 This is consistent with prior
`
`suggestions that patent examiners primarily search prior U.S. patents (Thomas 2001). They have
`
`less ability to search foreign patents and unpublished sources of non-patent art, so the
`
`overwhelming majority of other references are those provided by applicants. 9
`
`
`
`3.2.
`
`Examiner Rejections
`
`8 Using data on citing patents issued between 2001 and 2003, Alcacer et al. (2009) find that 59 percent of
`citations to U.S. patents are from applicants (page 420; Table 1, Column 3). The difference between this
`figure and ours, 66 percent, reflects a fairly smooth increase over time and is an interesting issue to
`explore, though beyond the scope of our paper. See http://dvn.iq.harvard.edu/dvn/dv/boffindata for the
`raw data.
`
` For example, Thomas (2001) argues that“[i]n comparison to much of the secondary literature [non-
`patent art], patents are readily accessible conveniently classified, and printed in a common format.
`Identification of a promising secondary reference, and full comprehension of its contents, often prove to
`be more difficult tasks” (Thomas 2001, 318).
`
` 9
`
`PMC Exhibit 2023
`Apple v. PMC
`IPR2016-00754
`Page 9
`
`

`
`
`
`We are interested in the extent to which these different types of art are used in examiner
`
`rejections. To assess this, we collected image file wrappers for each of the citing patents from
`
`PTO’s the Patent Application Information Retrieval (PAIR) database. The PAIR file wrappers
`
`are generally (though not always) in image PDF form, making them difficult to use for large
`
`sample analysis. Accordingly, we transcribed all examiner office actions—final and non-final
`
`rejections--and converted them to ASCII format. We similarly coded information from
`
`Information Disclosure Statements—the forms on which applicants typically submit prior art,
`
`also kept as PDF files in the PAIR database--since in some of the analyses below it is important
`
`to know not only whether the prior art reference comes from an examiner, but also the timing of
`
`when it was submitted. We also collected the transaction history from PAIR. That transaction
`
`history shows every filing and procedural step in the patent prosecution process (Lemley and
`
`Sampat 2012). These data are used in analyses analyzing the timing of prior art submission.
`
`
`
`Under the US Patent code, claims can be rejected for several reasons. We categorized
`
`examiner rejections into several basic categories:
`
`• Section 101: Subject Matter Eligibility or Utility: rejecting the claimed invention because
`
`it either is directed to ineligible subject matter, such as a law of nature, physical
`
`phenomena, or abstract idea, or is not useful.
`
`• Section 102: Novelty: rejecting the claimed subject matter because it is not novel at the
`
`time of invention and is described in a printed publication or publicly used or sold in the
`
`United States more than one prior to the filing of the patent application.
`
`• Section 103: Non-Obviousness: rejecting the claimed invention because it is an obvious
`
`advance over what was known at the time of invention.
`
`PMC Exhibit 2023
`Apple v. PMC
`IPR2016-00754
`Page 10
`
`

`
`• Section 112: Disclosure: rejecting the claimed invention because the patent fails to
`
`adequately describe and enable others to practice the invention or fails to clearly define
`
`what is claimed.
`
`• Section 121: Restriction requirements: restricting the patent application to a single
`
`invention because the application includes two or more independent and distinct
`
`inventions.
`
`The office actions cite the specific reasons for rejection, generally in standardized form.10 This
`
`allows us to parse the file for the type of rejection, and, when rejections are based on prior art,
`
`what art is used in the rejections. Consistent with Lemley and Sampat (2009), 17% of the patents
`
`in our sample were issued with no rejection, leaving 1316 of the patents with at least one
`
`rejection.11 We parsed the office actions to assess types of rejections employed in either initial or
`
`“final” office actions, by broad art unit. We present the results in Table 2.
`
`
`
`
`
`10 For example, a non-final rejection of claims in patent 7235478 reads “Claims 1-5, 7-13, 16, 18, 20 are
`rejected under 35 U.S.C. 102(b) as being unpatentable by Hsu (US 6,562,696).” This standard language
`is dictated by the Manual of Patent Examining Procedure ("MPEP").
`
`11 This information comes from the office actions. It is consistent with that from transaction history.
`
`PMC Exhibit 2023
`Apple v. PMC
`IPR2016-00754
`Page 11
`
`

`
`Table 2: Share of Patents With Different Types of Rejections, By Art Unit
`
`Art Unit
`
`N
`
`101
`
`102
`
`103
`
`112
`
`121
`
`Any Rejections
`
`1600 Biotechnology
`
`1700 Chemicals
`
`2100 Computers
`
`2600
`Communication
`
`2800
`Semiconductors
`
`143
`
`177
`
`153
`
`236
`
`24%
`
`66%
`
`50%
`
`72%
`
`21%
`
`8%
`
`84%
`
`73%
`
`56%
`
`25%
`
`29%
`
`85%
`
`67%
`
`52%
`
`9%
`
`16%
`
`78%
`
`62%
`
`35%
`
`11%
`
`457
`
`9%
`
`72%
`
`58%
`
`24%
`
`11%
`
`3600 Transportation
`
`3700 Mechanical
`
`183
`
`215
`
`4%
`
`7%
`
`72%
`
`50%
`
`46%
`
`79%
`
`62%
`
`35%
`
`9%
`
`9%
`
`Total
`
`
`1564
`
`12%
`
`76%
`
`60%
`
`40%
`
`13%
`
`82%
`
`91%
`
`93%
`
`82%
`
`79%
`
`78%
`
`84%
`
`83%
`
`We note that “restriction requirements” under section 121 are much more common in
`
`chemicals and biotechnology, as some anecdotal evidence has previously suggested. But
`
`because they are not rejections on the merits, we don’t consider them further in this paper.
`
`Section 112 rejections (primarily written description and enablement are also much more
`
`common in biotechnology than in other fields: 72% of all patents in this art unit had a written
`
`description or enablement rejection. Section 101 rejections are much more common in computers
`
`and biotechnology than in other technologies. These are fields where issues about patent-eligible
`
`subject matter have been most pronounced. Because these are not prior-art based rejections, we
`
`don’t consider them further here.
`
`3.3.
`
`Types of Prior Art Used in Novelty (102) and Non-Obviousness (103) Rejections
`
`PMC Exhibit 2023
`Apple v. PMC
`IPR2016-00754
`Page 12
`
`

`
`
`
`The bulk of all patents with rejections (1192 of 1316) have at least one 102 or 103
`
`rejection, i.e. one rejection relying on prior art. To determine who supplied the reference actually
`
`used by examiners in the 1192 patents with novelty (section 102) and non-obviousness (section
`
`103) rejections, we coded all of the art cited in the 1192 patents that had at least one claim
`
`rejected under sections 102 or 103, and used text-matching algorithms to map each reference to
`
`an observation in the dataset of all applicant- and examiner-submitted references. For a small
`
`number of patents (25) chosen at random, we also determined whether each of the cited
`
`references were used by examiners through reading the full text of the office actions; the
`
`algorithm and hand-coded results were in agreement for 97 percent of references.
`
`
`
`As in the overall sample, most of the art cited in the patents where there were prior art-
`
`based rejections comes from applicants, not examiners. Applicants submitted 23,664 of the
`
`32,181 prior art references cited in patents with at least one prior art-based rejection, or 73.5%.
`
`Of the 32,181 total references, 3,358, or about 11%, are used in examiners’ rejections. That fact
`
`itself is worthy of note; the overwhelming majority of art that appears on the face of the patent is
`
`not in fact discussed in the course of patent prosecution or used as the basis for a prior art
`
`rejection.
`
`
`
`Table 3 also provides evidence that examiners tend overwhelmingly to rely on examiner-
`
`supplied references: only about 2% of applicant references are cited in rejections, while over a
`
`third of examiner references are. Viewed another way, of the references examiners use to reject
`
`claims, only 12.7% come from the applicants, while 87.2% come from examiners. And that
`
`applicant share represents an upper bound. Art is listed as applicant-submitted if applicants
`
`submit it, even if it is also found independently by an examiner during a search (Manual of
`
`PMC Exhibit 2023
`Apple v. PMC
`IPR2016-00754
`Page 13
`
`

`
`Patent Examination Procedure, Section 1302.12). We have no way to tell what fraction of the
`
`12.7% was in fact also found independently by examiners during a search.
`
`Table 3: Applicant and Examiner References, and whether they are used in 102 or 103
`rejections (Based on Citations in Patents with at least one rejection)
`
`
`Source of Reference
`
`Not used in a rejection
`
`Used in at least one rejection
`
`Total
`
`Applicant
`
`Examiner
`
`Total
`
`
`
`23,209
`
`5,414
`
`28,623
`
`455
`
`3,103
`
`3,558
`
`23,664
`
`8,517
`
`32,181
`
`Clearly, then, examiners focus almost exclusively on art they find themselves in
`
`considering whether a patent application is new and nonobvious. The question is why.
`
`
`
`One possible explanation relates to differences in the types of prior art cited by applicants
`
`and examiners. There surely are such differences; as we noted above, applicants are more likely
`
`to submit foreign patents and non-patent prior art, both because examiners tend to confine their
`
`searches to prior U.S. patents and because many types of non-patent prior art are not easily
`
`searchable.12 So if it happens to be that prior United States patents are the most important source
`
`of prior art, the fact that examiners disproportionately search those might explain our results.
`
`For a first cut assessment of this, we recalculated these numbers based on types of prior art cited.
`
`Table 4 shows the results:
`
`
`
`
`
`12 Our definition of prior art includes 102(b) art created by the applicant as well as third-party prior art.
`
`
`PMC Exhibit 2023
`Apple v. PMC
`IPR2016-00754
`Page 14
`
`

`
`Table 4: Whether a reference is used in a rejection, by type of prior art and source
`
`Type of Prior Art
`
`Share of All Applicant
`References Used in Making a
`Rejection
`
`Share of All Examiner
`References Used in Making a
`Rejection
`
`Foreign Patent
`(n=4,779)
`
`Non-Patent Literature
`(n=6,834)
`
`U.S. Patents
`(n=20,568)
`
`Total
`(n=32,181)
`
`2.9%
`
`1.1%
`
`1.9%
`
`2%
`
`49.2%
`
`29.5%
`
`36.4%
`
`39.5%
`
`
`
`Of the applicant citations to U.S. patents, over 98% are not used by examiners in
`
`rejections.13 Over 90% of the U.S. patents used in rejections emanate from examiner searches,
`
`rather than from applicants. The percentage of applicant-submitted foreign and non-patent art
`
`used in rejections is also low: examiners used only 2.9% of the applicant-submitted foreign
`
`patents and 1.1% of the applicant-submitted non-patent art. However, applicant-submitted
`
`foreign patents and non-patent art account for a higher share of all rejections made by examiners
`
`using foreign and non-patent art (46% and 32%, based on unreported calculations). This is
`
`because examiners themselves are less likely to have access to, and therefore less likely to cite,
`
`these other types of prior art than applicants.
`
`
`
`The data above don’t account for potential industry effects. Perhaps the differences can
`
`be explained by the behavior of examiners or applicants in different industries. To assess this we
`
`13 Note that we are conditioning here on 102 or 103 rejections. The actual share of applicant references
`employed in rejections across all patents would be even lower.
`
`PMC Exhibit 2023
`Apple v. PMC
`IPR2016-00754
`Page 15
`
`

`
`also control for 3-digit patent classes, and estimated linear probability models relating whether a
`
`citation was used to the source (examiner or applicant) and the type of citation (U.S. patent,
`
`foreign patent, non-patent literature).14 The unit of analysis is a cited reference. We also
`
`controlled for several patent characteristics. These include whether the citing patent was a
`
`continuation or divisional of a previous application (rather than an original application), as well
`
`as the patent application year. Table 5 shows the baseline results.15
`
`
`
`
`
`14 We estimated logit analogues of each of the ordinary least squares regressions in this paper. The
`estimated marginal effects are similar. These results are available on request.
`15 We report robust standard errors, clustered on citing patents.
`
`PMC Exhibit 2023
`Apple v. PMC
`IPR2016-00754
`Page 16
`
`

`
`Table 5: Linear Probability Models with Class and Year Effects [Dependent Variable: Was
`Citation Used in a Rejection?]
`
`
`
`
`Non-Patent Citation
`
`
`
`Foreign Patent Citation
`
`
`
`Citing Patent a Continuation
`
`
`
`Citing Patent a Divisional
`
`
`
`Applicant Citation
`
`
`
`Applicant Citation * Non-Patent Citation
`
`
`
`Applicant Citation * Foreign Patent Citation
`
`
`
`Applicant Citation * U.S. Patent Citation
`
`
`
`Applicant Citation * Total Citations in Bottom Quartile
`
`
`
`Applicant Citation * Total Citations in Second Quartile
`
`
`
`Applicant Citation * Total Citations in Third Quartile
`
`
`
`Applicant Citation * Total Citations in Top Quartile
`
`
`
`Constant
`
`
`
`
`
`(1)
`
`
`
`(2)
`
`
`-0.0979***
`(0.00840)
`
`
`-0.0125*
`(0.00511)
`
`
`-0.104***
`(0.00816)
`
`
`0.00452
`(0.00548)
`
`
`-0.0644***
`(0.00928)
`
`
`-0.00845
`(0.00711)
`
`
`-0.0457***
`(0.0126)
`
`
`-0.00359
`(0.00782)
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`0.250***
`(0.0519)
`
`
`-0.351***
`(0.00994)
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`0.419***
`(0.0418)
`
`PMC Exhibit 2023
`Apple v. PMC
`IPR2016-00754
`Page 17
`
`

`
`
`Observations
`
`
`
`
`
`32181
`32181
`
`
`Model 1 includes patent characteristics and dummy variables indicating whether a
`
`citation is a non-patent reference or a foreign patent citation. The left-out category is U.S. patent
`
`citations. The results show that after controlling for patent class and application characteristics,
`
`non-patent prior art and foreign prior art respectively are 9.8 and 10.4 percentage points less
`
`likely to be used in a rejection than U.S. patent citations.
`
`
`
`Model 2 includes a dummy variable indicating whether a citation is an applicant citation.
`
`All else equal, applicant citations are 35 percentage points less likely to be used in a rejection
`
`than examiner citations. Moreover, on including the applicant citation indicator, the magnitude
`
`of the differences (across types of prior art) in the chance that a citation is used in a rejection
`
`diminish sharply; that is because that most citations to non-patent literature and foreign patents
`
`come from applicants.
`
`
`
`Model 3 examines this in more detail, including interaction terms between types and
`
`sources of prior art. The results show that the probability a given reference will be used in a
`
`rejection is significantly higher for examiner citations than applicant citations for all categories
`
`of prior art, though the difference varies from 28 percentage points less use of applicant-provided
`
`non-patent art to as much as 48 percentage points less use of applicant-provided foreign patents.
`
`Interestingly, on inclusion of these interaction terms, the coefficient on the foreign prior art
`
`dummy is positive and significant, indicating that examiner-located foreign patents are more
`
`likely to be used in a rejection than examiner-identified U.S. patents (the left-out category).
`
`PMC Exhibit 2023
`Apple v. PMC
`IPR2016-00754
`Page 18
`
`

`
`
`
`Each of these models demonstrates that the examiner preference for examiner-found
`
`rather than applicant-submitted art cannot be explained as a function of the type of art being
`
`submitted or the field of technology.16
`
`
`
`What, then, is going on? Possible explanations are that the applicant-cited art may just
`
`not be very relevant, or it may be submitted in such a way that examiners can’t evaluate it
`
`effectively. A particular version of this concern is “flooding the patent office”: that patent
`
`applicants provide so much prior art as to be useless. For example, Popp et al. (2004), who
`
`interviewed numerous examiners for their study of patent office practice, note “All of the
`
`examiners we interviewed complained about applications which arrive with box loads of
`
`supporting references. Examiners often suggested that an application with no references at all
`
`would be preferable, since the examiner could research it himself instead of being saddled with
`
`the existing reference list to edit” (13). A recent blog entry by an examiner also observes: “The
`
`trend today seems to be towards more and more extensive IDSes, usually with less and less
`
`relevant art cited.”
`
`(http://just-n-examiner.livejournal.com/33286.html). Citing
`
`too many
`
`references may be beneficial to the patentee if it causes an examiner to fail to process those truly
`
`material to patentability: if granted, these patents would earn a presumption of validity as against
`
`the cited art. (Sampat 2010; Cotropia 2009).
`
`
`
`We can’t identify relevance of individual prior art references given the large number of
`
`patents and citations in our sample. Instead, to test for the flooding effect, in Model 4 we re-ran
`
`the specification in Model 2, but this time interacted the indicator for whether a reference was an
`
`applicant citation with an indicator for whether the total number of applicant references in the
`
`16 We also estimated models with citing patent fixed effects, controlling for the quality of the application
`as a whole. Results are similar, and available on request.