I.
`
`“The experimental record showed that the
`use of indirect relationships generally
`resulted in worse search results”
`
`Jacobs Dec. at ¶¶ 217-254; 287-296. ‘352 POR at 25-26.
`
`EXHIBIT 2119
`Facebook, Inc. et al. v. Software Rights Archive, LLC
`CASES IPR2013-00478
`IPR2013-00479
`IPR2013-00480
`IPR2013-00481
`
`1
`
`
`
`“The experimental record showed that the
`use of indirect relationships generally
`resulted in worse search results”
`
`Jacobs Dec. at ¶¶ 217-254; 287-296. ‘352 POR at 25-26.
`
`EXHIBIT 2119
`Facebook, Inc. et al. v. Software Rights Archive, LLC
`CASES IPR2013-00478
`IPR2013-00479
`IPR2013-00480
`IPR2013-00481
`
`1
`
`
`
`Jacobs’s Opinions:
`I. “The experimental record showed that the use of
`indirect relationships generally resulted in worse search
`results”. Jacobs Dec. at ¶¶ 217-254; 287-296. ‘352 POR
`at 25-26.
`
`II. “The experimental record showed that the use of indirect
`relationships … was not sufficiently reliable to be used in an
`automated retrieval system.” Jacobs Dec. at ¶¶ 205-216.
`‘352 POR at 25-26.
`
`2
`
`
`
`Claims Require Indirect Relationships for Search:
`‘352 Claim 26 requires, inter alia:
`“A non-semantical method … for computerized searching … comprising:
`…
`analyzing the first numerical representation for indirect relationships existing
`between or among objects in the database;
`generating a second numerical representation of each object based on the analysis of
`the first numerical representation;
`storing the second numerical representation for use in computerized searching; and
`searching the objects in the database using a computer and the stored second
`numerical representations, wherein the search identifies one or more of the objects
`in the database.” ‘352 patent at 35:28-53; see also IPR2013-00478 POR at 1-4, 50-
`60.
`
`3
`
`
`
`‘494 Claim 1 requires , inter alia:
`“A method of analyzing a database with indirect relationships, using links and
`nodes, comprising … generating candidate cluster links for the selected node,
`wherein the step of generating comprises an analysis of one or more indirect
`relationships in the database … [and] identifying one or more nodes for display;
`and displaying the identity of one or more nodes using the actual cluster links.”
`‘494 patent at 51:38-50; see also IPR2013-00478 POR at 1-4, 50-60.
`
`4
`
`
`
`‘494 Claim 18 requires, inter alia:
`“A method of analyzing a database having objects and a first
`numerical representation of direct relationships in the database,
`comprising … generating a second numerical representation using
`the first numerical representation, wherein the second numerical
`representation accounts for indirect relationships in the
`database ... [and] identifying at least one object in the database,
`wherein the stored numerical representation is used to identify
`objects” ‘494 patent at 53:43-45; IPR2013-00478 POR at 1-4, 50-60.
`
`5
`
`
`
`‘571 Claim 12 requires, inter alia:
`“A method for visually displaying data related to a web having
`identifiable web pages and Universal Resource Locators with
`pointers, comprising … analyzing Universal Resource Locators,
`wherein Universal Resource Locators which have an indirect
`relationship to the chosen web pages are located … wherein the
`located Universal Resource Locators are used to identify web
`pages” ‘571 Patent at 52:37-56; IPR2013-00478 POR at 1-4, 50-60.
`
`6
`
`
`
`Petition: Studies directed to “bc” and “cc” bibliographic
`relationships contained in experimental paper collections of
`ISI and CACM demonstrate it would be obvious to use
`indirect relationships for search.
`‘352 Pet. at 10-11.
`PO Response:
`“The experimental record showed that the use of indirect
`relationships generally resulted in worse search results and
`was not sufficiently reliable to be used in an automated
`retrieval system.”
`‘352 PO Res. at 25-26.
`
`7
`
`
`
`Jacobs Testifies:
`“Until the filing of the ‘352, ‘494, and ‘571 patents, the little work
`that had been done pertaining to the possible use of indirect, non-
`semantic relationships in information retrieval was entirely
`experimental.
`This work was conducted in isolated research experiments, using
`small-scale, highly specialized and prepared data (such as the CACM
`collection).”
`‘352 Jacobs Dec., IPR2013-00478 Exhibit 2113 at ¶ 218; IPR2013-00478 POR at 50-60
`
`8
`
`
`
`1963 – Gerard Salton
`“Gerard Salton was the most prominent figure in the field of information
`retrieval between 1970-1994 … and is largely credited for creating the
`vector space model.”
`‘352 Jacobs Dec., IPR2013-00478 Exhibit 2113 at ¶ 220.
`Q: Who was Gerard Salton?
`A: He’s often considered to be the father of the field of
`information retrieval. Very prolific author. Editor of many
`different publications of various kinds, editor or author of a
`number of well-regarded books and many, many works. Very
`excellent scientist.
`Fox Depo. Tr. Pt 1 at 24-25; see also IPR2013-00478 POR at 50-55; ‘352 Jacobs Dec., IPR2013-00478
`Exhibit 2113 at ¶ 217-254.
`.
`
`9
`
`
`
`1963 – Gerard Salton
`“An attempt is therefore made in the next few sections to evaluate the
`utility of bibliographic citations as an aid to automatic content analysis.
`When this problem, is first considered, the initial reaction must clearly
`be one of skepticism. … Because of these and other variations, citation
`and reference lists have not generally been used as an indication of
`document content.”
`
`Salton 1963, Ex. 2009, at 445-446; ‘352 Jacobs Dec., IPR2013-00478 Exhibit 2113 at ¶ 220; see also ‘352
`Jacobs Dec., IPR2013-00478 Exhibit 2113 at ¶ 217-254; IPR2013-00478 POR at 50-55.
`
`10
`
`
`
`1963 – Gerard Salton
`“A cursory comparison of actual similarity coefficients obtained in
`the present experiment with the theoretical values shown in Figure
`8(b) reveals that very few (less than five percent) of the values are
`significant at the ninety-eight percent level.”
`Salton 1963 at 455; ‘352 Jacobs Dec., IPR2013-00478 Exhibit 2113 at ¶ 220.
`Salton concludes from his results:
`“Clearly no proof has been presented in this study that citations
`do in fact play a significant role in automatic document
`retrieval... ”
`Salton 1963, Ex. 2009, at 455; ‘352 Jacobs Dec., IPR2013-00478 Exhibit 2113 at ¶ 221; see also ‘352
`Jacobs Dec., IPR2013-00478 Exhibit 2113 at ¶ 217-254; IPR2013-00478 POR at 50-55.
`
`11
`
`
`
`1976-1982 – Tapper
`“Table 2 shows the upper part of the correlation table for the cases in
`the British database. The first column gives the numbers allocated to
`the cases. The second gives the correlation value indicated by the
`conventional VEXT technique without weighting, and the third the
`rank order according to those values. The fourth and fifth correspond
`to the second and third except that the values there are derived from
`the Oxford technique with Oxford weighting. Great care must be
`taken in interpreting this table. It is necessary to bear in mind that the
`absolute values are very low, and that only the first two or three
`pairs have a value of any real significance at all.”
`Tapper 1982, Ex. 1005, at 148 (emphasis added); ‘494 Jacobs Decl., IPR2013-00479 at ¶ 350; see also ‘352
`Jacobs Dec., IPR2013-00478 Exhibit 2113 at ¶ 217-254; IPR2013-00478 POR at 50-55.
`
`12
`
`
`
`1976-1982 – Jacobs Testifies Regarding Tapper:
`“Indeed, Tapper 1982, a reference relied upon by, Petitioners from this time
`period, stated about Tapper’s experiments:”
`‘352 Jacobs Dec., IPR2013-00478 Exhibit 2113 at ¶ 222.
`“What [the citation experiment’s results] do not, and cannot, demonstrate is
`whether or not the technique can be improved to yield still better results, and
`whether or not it can be developed into a component part of an operational
`commercial system. It is sadly little appreciated just how wide a gap separates a
`system which operates perfectly satisfactorily in an experimental or limited
`environment, and one which can take the strains imposed by the massive
`databases, constant use, and variety of misuse, much of it quite possibly
`unimaginable, to which a commercial system is inevitably exposed.”
`Tapper 1982, IPR2013-00478 Exhibit 1006, at 157; ‘352 Jacobs Dec., IPR2013-00478 Exhibit 2113 at ¶ 222;
`see also ‘352 Jacobs Dec., IPR2013-00478 Exhibit 2113 at ¶ 217-254; IPR2013-00478 POR at 50-55.
`
`13
`
`
`
`1982 – Tapper
`
`“The former question can be answered by the conduct of a series of
`careful and systematic experiments. … It is now necessary to
`consider the question of testing the system. Some of the further
`research described above could and should be tested by the same
`methods as those employed hitherto. Some testing should however
`be organized on a more scientific and practical basis.”
`Tapper 1982, Ex. 1006, at 157; see also ‘352 Jacobs Dec., IPR2013-00478 Exhibit 2113 at ¶ 217-254;
`IPR2013-00478 POR at 50-55.
`
`14
`
`
`
`1983 – Fox Thesis precision tests
`Jacobs testifies: “Dr. Fox concluded that the use of indirect
`relationships was likely to degrade retrieval accuracy”
`
`Fox Thesis, IPR2013-00478 Exhibit 1009 at 241-256; ‘352 Jacobs Dec., IPR2013-00478 Exhibit 2113 at ¶ 222; see also ‘352 Jacobs Dec.,
`IPR2013-00478 Exhibit 2113 at ¶ 217-254, 287; IPR2013-00478 POR at 50-55.
`
`15
`
`
`
`Jacobs Testifies: “co-citation and bibliographic coupling
`(corresponding to indirect relationships) would hurt the results”
`Experiment
`Result
`Experiment
`Result
`-37% (vs terms)
`-6.3% (vs terms)
`8.2-ISI (cc)
`8.11-CACM (bc)
`-54% (vs terms)
`-1.4% (vs terms)
`8.2-ISI (cd)
`8.11-CACM (cc)
`-6.2% (vs terms)
`-2.9% (vs terms)
`8.3-ISI (cc+terms)
`8.11-CACM (bc+cc)
`-7.8% (vs terms)
`-7.4% (vs ln)
`8.3-ISI (cc+other
`8.11-CACM (bc)
`subvectors)
`8.3-ISI (cc+terms
`emphasized)
`8.7-ISI
`(cc+terms regression)
`-26.5% (vs terms)
`-62% (vs terms)
`8.12-CACM (bc)
`8.8-CACM (bc)
`-29% (vs terms)
`-51% (vs terms)
`8.8-CACM (cc)
`8.12-CACM (cc)
`‘352 Jacobs Dec., IPR2013-00478 Exhibit 2113 at ¶ 287, 225; see also IPR2013-00478 POR at 25-28, 50-55.
`
`+4.5 to 6.1% (vs
`terms)
`+5 to 6% (vs terms)
`
`8.11-CACM (cc)
`
`-6% (vs ln)
`
`8.11-CACM (bc+cc)
`
`-4.1% (vs ln)
`
`16
`
`
`
`Jacobs Testifies: “co-citation and bibliographic coupling
`(corresponding to indirect relationships) would hurt the results”
`
`Experiment
`
`Result
`
`8.13-CACM (bc)
`8.13-CACM (cc)
`8.13-CACM (bc+cc)
`8.13-CACM (all
`subvectors regression)
`
`-11.1% (vs ln)
`-3.7% (vs terms)
`-11.5% (vs ln)
`undetermined but less than +2.1%
`divided among au, cr, bc, and cc
`(no positive contribution from indirect
`relationships because bc and cc weighed
`at less than 1% and 0% respectively)
`
`‘352 Jacobs Dec., IPR2013-00478 Exhibit 2113 at ¶ 287, 225; see also IPR2013-00478 POR at 25-28, 50-55.
`
`17
`
`
`
`Jacobs Testifies: “Dr. Fox … identifies the experimental
`results of Table 8.13 as indicating his strongest case for
`using indirect relationships.”
`“Even more improvement was found with a more advanced
`feedback technique. See Section 8.3.3 and Table 8.13 at 257.”
`Fox Dec., IPR2013-00478, Ex. 1016 at ¶ 65.
`
`Q: So you would say that the experiment reported in table 8.13 is
`the most complete experiment of all the experiments in Chapter 8?
`A: It’s the one with the best results.
`Fox Dep. Tr. Pt. 1 at 129:4-9; ‘352 Jacobs Dec., IPR2013-00478 Ex. 2113, at ¶ 292; see also
`IPR2013-00478 POR at 25-28, 50-55.
`
`18
`
`
`
`Jacobs Testifies:
`
`“of the 29.7%
`improvement of the
`combined vector,
`approximately 27% can be
`attributed to terms and
`direct links alone.”
`“The remaining
`approximate “2.7%”
`improvement cannot be
`attributed to any
`particular subvector or
`subvector combination
`among the following
`remaining subvectors: au,
`cr, bc, and cc.”
`
`Fox Thesis, IPR2013-00478 Exhibit 1009 at 241-257;; ‘352 Jacobs Dec., IPR2013-00478 Exhibit 2113 at ¶ 292; see also ‘352 Jacobs Dec., IPR2013-
`00478 Exhibit 2113 at ¶ 217-254, 287; IPR2013-00478 POR at 25-28, 50-55.
`
`19
`
`
`
`“An average effectiveness improvement above
`10 percent is normally considered important
`enough to warrant serious attention…”
`Salton 1986, Ex. 2009, at 9; ‘352 Jacobs Dec., IPR2013-00478 Exhibit 2113 at ¶ 294; see also
`‘352 Jacobs Dec., IPR2013-00478 Exhibit 2113 at ¶ 217-254; IPR2013-00478 POR at 50-55.
`
`20
`
`
`
`Fox Thesis, IPR2013-
`00478 Exhibit 1009 at
`241-256; see also
`‘352 Jacobs Dec.,
`IPR2013-00478
`Exhibit 2113 at ¶ 217-
`254; IPR2013-00478
`POR at 25-28, 50-55.
`
`21
`
`
`
`Jacobs testifies: “There is no case in
`Fox 1983 where bc and cc improve
`results when direct links are present.”
`
`‘352 Jacobs Decl., IPR2013-00478, Exhibit 2113 at ¶ 227, 287; see also IPR2013-00478 POR at 25-28, 50-55.
`
`Claim 26: “creating a first numerical representation for each identified
`object in the database based upon the object’s direct relationship with
`other objects in the database; storing the first numerical representations
`for use in computerized searching”
`
`‘352 pat. at claim 26.
`
`22
`
`
`
`1983 – Fox Papers
`
`Fox states about Table 8.13:
`“[I]t can be inferred, however, that with the
`other subvectors present cc is not really
`needed and bc is probably not either.”
`
`Fox Thesis at 258 (emphasis added); see also ‘352 Jacobs Decl., IPR2013-00478, Exhibit 2113 at ¶ 227,
`287; IPR2013-00478 POR at 52.
`
`23
`
`
`
`1983 – Fox Papers
`“The recipe proposed is to at least employ terms (tm), some manually
`assigned categorization scheme (cr), and direct links between
`documents (ln). When bibliographic information is only available
`among articles in a collection the simplest form of that information,
`references (ln) [i.e., the direct links vector], seems to be the most reliable
`and most useful of all the types considered (bc, ln, cc) [(bc) and (cc) are
`the indirect bibliographic vectors]. The ln subvectors are typically
`longer than the other two and are easier to obtain so use of them is
`encouraged by practicality considerations as well as effectiveness tests.”
`Fox Thesis at 253; see also ‘352 Jacobs Dec., IPR2013-00478 Exhibit 2113 at ¶ 217-254; IPR2013-00478
`POR at 50-55.
`
`24
`
`
`
`1986 – Salton Precision Tests (Fox’s Thesis Advisor )
`Salton’s 1986 paper states:
`“Since no obvious way exists for distinguishing the positive from the
`negative effects, the citation methodology cannot be recommended for
`inclusion in practical retrieval environments. It appears that any term
`association process ... must include strict syntactical and/or semantic
`controls if the generation of inappropriate related term groups is to be
`prevented.”
`Salton 1986, IPR2013-00478 Exhibit 2006, at at 9-11.
`
`“Overall, the procedure is not sufficiently reliable to warrant
`incorporation into operational automatic retrieval systems.”
`
`Salton 1986 at 9; see also ‘352 Jacobs Dec., IPR2013-00478 Exhibit 2113 at ¶ 217-254; IPR2013-00478 POR at 50-55.
`
`.
`
`25
`
`
`
`1986 – Nunn Precision Tests
`Jacobs Testifies:
`
`approximately -13% degradation
`Nunn, Exhibit 2027, at 37; see also Jacobs Depo. Tr. Pt. 2 at 397-402; ‘352 Jacobs Decl., IPR 2013-00478 at ¶ 213;
`IPR2013-00478 POR at 50-55.
`.
`
`26
`
`
`
`1991 – Thompson
`“Roger Thompson, who testified for the
`Petitioners, replicated Fox’ experiments on the
`CACM collection. He specifically chose not
`use the bc and cc data.”
`
`‘352 Jacobs Dec., IPR2013-00478 Exhibit 2113 at ¶ 234 (citing Thompson Part 3, IPR2013-00478 Exhibit
`2012, at 96); see also ‘352 Jacobs Dec., IPR2013-00478 Exhibit 2113 at ¶ 217-254; IPR2013-00478 POR at
`50-55.
`
`27
`
`
`
`1992 - Frei & Steiger
`“When users browse, they tend to follow only few of the existing links. Conversely, a retrieval
`algorithm may follow too many links with the consequence of both high retrieval cost and
`doubtful results. Retrieval experiments in a collection of bibliographic references showed that
`following citations–a kind of referential links–may produce ambiguous results. This result is
`not very surprising. In connection with the Cranfield Project, Cleverdon et al, (1966) asked
`authors to grade the relevance of their own citations in their papers. Most considered the
`background papers marginally relevant. Some stated that many of their citations were not
`really relevant and that some citations just amplified a detail in the text.
`…
`The hope is that our semantic links contain the information necessary to decide whether a
`further nodes should be visited by the retrieval algorithm or not.”
`‘352 Jacobs Dec., IPR2013-00478 Exhibit 2113 at ¶ 237; Frei & Stieger, IPR2013-00478 Exhibit 2030, at 104
`(emphasis added); see also ‘352 Jacobs Dec., IPR2013-00478 Exhibit 2113 at ¶ 217-254; IPR2013-00478 POR at
`50-55.
`
`28
`
`
`
`1992 – Ledwith
`“However, despite the significant efforts to explore and
`develop these models, there remain concerns about the
`models’ utility for the searching of large scientific
`databases. Using the p-norm retrieval experiment described
`in Fox (1983) as an example, I will present my three major
`concerns…the reliability of extrapolating the
`performance of research systems that use the collection
`to a system to search a file over 750 times larger than
`the collection is highly questionable…”
`
`‘352 Jacobs Dec., IPR2013-00478 Exhibit 2113 at ¶ 238; Ledwith 1992, IPR2013-00478 Exhibit 2028, at 451-452; see also ‘352
`Jacobs Dec., IPR2013-00478 Exhibit 2113 at ¶ 217-254; IPR2013-00478 POR at 50-55.
`
`29
`
`
`
`1994 – Salton
`“It has been suggested that links be placed between related
`pieces of text, connecting, for example, particular text
`paragraphs to other paragraphs covering related subject
`matter. Such a linked text structure, often called hypertext,
`makes it possible for the reader to start with particular text
`passages and use the linked structure to find related text
`elements (1). Unfortunately, until now, viable methods for
`automatically building large hypertext structures and for
`using such structures in a sophisticated way have not been
`available.”
`Salton 1994, IPR2013-00478 Exhibit 2031, at 1421; see also ‘352 Jacobs Dec., IPR2013-00478 Exhibit 2113 at ¶ 217-254;
`IPR2013-00478 POR at 50-55.
`
`30
`
`
`
`1998 – Langville Testifies:
`Pre-1998, “no one at the web search companies mentioned
`using links. The links were the reason that a research project
`running on a computer in a Stanford dorm room had become the
`top performer.”
`Michael Specter, “Search and Deploy,” The New Yorker, May 29, 2000 (IPR2013-00479 Exhibit 2041) (emphasis added).
`“Before [1998], people were only looking at the content. They
`were completely ignoring the fact that people were going to the
`effort of putting a link from one page to another and that there
`must be a meaning to that.”
`
`Professor Rajeev Motwani, IPR2013-00479 Exhibit 2041 (emphasis added); see also Langville Decl., Ex.
`2114 at ¶ 1-28; IPR2013-00478 POR at 50-60.
`
`31
`
`
`
`1998 – Langville Testifies
`“We originally developed PageRank kind of playing around with
`all the links on the web and that too was a pretty revolutionary
`idea, though it seems very simple, the fact you can even just
`collect [the links of the web] and do anything meaningful with
`them …
`I really credit Larry for pursuing that idea that it is even worth
`collecting the graph and then that you could run any kind of
`processing on it.”
`
`Brin transcript, IPR2013-00478 Ex. 2032, at 6:15-6:20, 15:17 – 15:33 (emphasis added); see also Langville
`Decl., Ex. 2114 at ¶ 1-28; IPR2013-00478 POR at 50-60.
`
`32
`
`
`
`1998 – Langville Testifies:
`SERGEY BRIN: But what we found was we-- kind of by
`accident almost-- we found that this processing of the link
`structure of the web, we could create a search that was better in
`important ways. In ways that these search engines had
`ignored. … And we sort of stumbled upon a way to do that
`by studying links. . .We originally developed PageRank kind
`of playing around with all the links.”
`Brin transcript, IPR2013-00478 Ex. 2032, at 6:15-6:20, 15:17 – 15:33 (emphasis added); see also Langville
`Decl., Ex. 2114 at ¶ 1-28; IPR2013-00478 POR at 50-60.
`
`33
`
`
`
`“The heart of our software is PageRank™, a system for ranking web pages
`developed by our founders Larry Page and Sergey Brin”
`
`“PageRank was absolutely crucial for Google’s Success”
`
`www.google.com/technology/
`
`Turkei, A Survey to Google’s PageRank
`
`“PageRank was the backbone of the Google success”
`
`Fleischner, 5 Myths about Google
`
`“Google surpassed its competition by serving superior and relevant results using
`a formula [i.e. PageRank] that was difficult to manipulate”
`
`Ormsby, PageRank Explained
`Langville Decl., IPR2013-00478 Exhibit 2114 at 1-28; see also IPR2013-00478 POR at 50-60.
`
`34
Accessing this document will incur an additional charge of $.
After purchase, you can access this document again without charge.
Accept $ Charge
Still Working On It
This document is taking longer than usual to download. This can happen if we need to contact the court directly to obtain the document and their servers are running slowly.
Give it another minute or two to complete, and then try the refresh button.
A few More Minutes ... Still Working
It can take up to 5 minutes for us to download a document if the court servers are running slowly.
Thank you for your continued patience.
This document could not be displayed.
We could not find this document within its docket. Please go back to the docket page and check the link. If that does not work, go back to the docket and refresh it to pull the newest information.
Your account does not support viewing this document.
You need a Paid Account to view this document. Click here to change your account type.
Your account does not support viewing this document.
Set your membership
status to view this document.
With a Docket Alarm membership, you'll
get a whole lot more, including:
- Up-to-date information for this case.
- Email alerts whenever there is an update.
- Full text search for other cases.
- Get email alerts whenever a new case matches your search.
One Moment Please
The filing “” is large (MB) and is being downloaded.
Please refresh this page in a few minutes to see if the filing has been downloaded. The filing will also be emailed to you when the download completes.
Your document is on its way!
If you do not receive the document in five minutes, contact support at support@docketalarm.com.
Sealed Document
We are unable to display this document, it may be under a court ordered seal.
If you have proper credentials to access the file, you may proceed directly to the court's system using your government issued username and password.
Access Government Site
