II. “The experimental record showed that the
`use of indirect relationships … was not
`sufficiently reliable to be used in an
`automated retrieval system.”
`
`Jacobs Dec. at ¶¶ 205-216. ‘352 POR at 25-26.
`
`EXHIBIT 2120
`Facebook, Inc. et al. v. Software Rights Archive, LLC
`CASES IPR2013-00478
`IPR2013-00479
`IPR2013-00480
`IPR2013-00481
`
`1
`
`
`use of indirect relationships … was not
`sufficiently reliable to be used in an
`automated retrieval system.”
`
`Jacobs Dec. at ¶¶ 205-216. ‘352 POR at 25-26.
`
`EXHIBIT 2120
`Facebook, Inc. et al. v. Software Rights Archive, LLC
`CASES IPR2013-00478
`IPR2013-00479
`IPR2013-00480
`IPR2013-00481
`
`1
`
`
`
`Jacobs’s Opinions:
`I. “The experimental record showed that the use of indirect
`relationships generally resulted in worse search results”.
`Jacobs Dec. at ¶¶ 217-254; 287-296. ‘352 POR at 25-26.
`
`II. “The experimental record showed that the use of
`indirect relationships … was not sufficiently reliable to
`be used in an automated retrieval system.” Jacobs Dec.
`at ¶¶ 205-216. ‘352 POR at 25-26.
`
`2
`
`
`
`Jacobs Testifies: “Dr. Fox Relies on Regression Results,
`Which Is a Method of Data Analysis, Not a Generalizable
`Search Method”
`“All of the results relied upon by Dr. Fox are based on regression
`weights, without which Fox Thesis teaches that bc and cc would
`detract from terms and from ln even more.
`A skilled artisan however would understand that regression weights
`are a means of trying to explain data, not a general method of
`computerized searching, and certainly not applicable to the claimed
`method.”
`
`IPR2013-00478 POR at 28; ‘352 Jacobs Dec., IPR2013-00478 Exhibit 2113 at ¶ 205.
`
`3
`
`
`
`Jacobs Testifies: “Dr. Fox Relies on Regression Results,
`Which Is a Method of Data Analysis, Not a Generalizable
`Search Method”
`“Regression is a method of statistical analysis that attempts to explain
`the fit between variables.
`Since one has to know the queries and answers in advance, this
`process is not usable for search in [actual] retrieval environments.
`If one has both the queries and responses required to create the
`regression weights, one does not need Dr. Fox’s alleged method to
`search for responses.”
`
`‘352 Jacobs Dec., IPR2013-00478 Exhibit 2113 at ¶ 205-206; IPR2013-00478 POR at 28.
`
`4
`
`
`
`Dr. Salton States:
`“Various attempts have been made over the years to construct
`enhanced document representations by using thesauruses of related
`terms, term association maps, or knowledge frameworks that can be
`used to extract appropriate terms and concepts. None of the
`proposed methods for the improvement of document
`representation has proved to be generally useful when applied to a
`variety of different retrieval environments.”
`
`Salton 86, Ex. 2006, at Abstract; see also ‘352 Jacobs Decl., IPR2013-00478 Ex. 2113 at ¶ 211; IPR2013-
`00478 POR at 2.
`
`5
`
`
`
`Fox Admits:
`“Now that the various measures have been defined and
`illustrated, it remains to be seen how they can be utilized to
`enhance document representations by incorporation into
`additional concept types.”
`Fox Thesis at 169.
`Q: Is the term “knowledge framework for extracting terms and
`concepts,” could that be used to describe aspects of your work
`that describe the Fox ‘83 papers?
`A: I guess so.
`Fox Depo. Tr. Pt. 1 at 16:17-24; see also ‘352 Jacobs Decl., IPR2013-00478 Ex. 2113 at ¶ 211; IPR2013-
`00478 POR at 2.
`
`6
`
`
`
`Dr. Salton Observes:
`“The CACM results of Table 3 show clearly that the addition to the document
`terms of title words from bibliographically related documents is beneficial,
`since the retrieval effectiveness improves by about 30 percent on average
`for the citing + cited method using tf weights, and by 15 percent for the
`citing + cited method using the more powerful tf x idf weights…
`Unfortunately, this optimistic conclusion is not maintainable when the
`CISI results of Table 4 are considered. In that case, none of the
`bibliographic expansion method proved beneficial for the more effective tf x
`idF weighting system, the deterioration in effectiveness ranging from
`about 1 percent to as much as 7 percent for the citing + cited method.”
`
`Salton 1986, Ex. 2006, at 9 (emphasis added); see also ‘352 Jacobs Decl., IPR2013-00478 Ex. 2113 at ¶ 211;
`IPR2013-00478 POR at 50-60.
`
`7
`
`
`
`“In other words, the same methods that appeared to be effective
`on the CACM collection actually degraded results when applied
`to the ISI collection.”
`
`Salton concludes:
`“Since no obvious way exists for distinguishing the positive from
`the negative effects, the citation methodology cannot be
`recommended for inclusion in practical retrieval environments.
`
`Salton 1986, Ex. 2006, at 11, abstract (emphasis added); see also ‘352 Jacobs Decl., IPR2013-00478 Ex.
`2113 at ¶ 211; IPR2013-00478 POR at 50-60.
`
`8
`
`
`
`“[S]ince no obvious way exists for
`distinguishing the positive from the negative
`effects, the citation methodology cannot be
`recommended for inclusion in practical
`retrieval environments.”
`Salton 1986, Ex. 2006, at 11, abstract (emphasis added).
`“Overall, the procedure is not sufficiently
`reliable to warrant incorporation into
`operational automatic retrieval systems.”
`Salton 1986, Ex. 2006, at 11, abstract (emphasis added); see also ‘352 Jacobs Decl.,
`IPR2013-00478 Ex. 2113 at ¶ 211; IPR2013-00478 POR at 50-60.
`
`9
`
`
`
`Q: Who was Gerard Salton?
`A: He’s often considered to be the father of the field of
`information retrieval. Very prolific author. Editor of many
`different publications of various kinds, editor or author of a
`number of well-regarded books and many, many works. Very
`excellent scientist.
`
`Fox Depo. Tr. Pt 1 at 24-25; see also IPR2013-00478 POR at 50-55; ‘352 Jacobs Dec., IPR2013-00478
`Exhibit 2113 at ¶ 220.
`
`10 A
`
`
`
`Jacobs Testifies:
`“Dr. Fox’s graduate student Gary Nunn in his 1987 Master’s
`Thesis, “Regression Analysis of Extended Vectors to Obtain
`Coefficients for use in Probabilistic Information Retrieval,”
`describes experimentation using the bc and cc subvectors on
`the CACM and ISI collection as a follow-up to the regression
`analysis discussed in Fox’s 1983 thesis.”
`
`‘352 Jacobs Dec., IPR2013-00478 Exhibit 2113 at ¶ 213; Jacobs Depo. Tr. Pt. 2 at 397-402; see also
`IPR2013-00478 POR at 50-60.
`
`11
`
`
`
`Jacobs Testifies:
`“Nunn’s experiment took the original ISI and CACM
`collections and divided them in half. From one half of each
`collection, coefficients were generated using the subvectors
`and procedures described in Fox. 1983. The same coefficients
`and subvector selections were then applied to the other half
`of each respective collection to determine average precision.”
`‘352 Jacobs Dec., IPR2013-00478 Exhibit 2113 at ¶ 213-214; Jacobs Depo. Tr. Pt. 2 at 397-402; see also
`IPR2013-00478 POR at 50-60.
`
`12
`
`
`
`Jacobs Testifies:
`“The results demonstrated that the coefficient and feedback queries did not
`generalize even when tested against a different part of the same collection:”
`“The base runs in CACM2 and ISI2 of each collection are not as high as those of
`CACM1 and ISI1…all of the runs which used coefficients had lower precision
`values than the corresponding base run…Though disappointing, these results are
`not surprising – term relevance weights and coefficients for concept types were derived
`through a feedback sampling process on a different half of the collection.”
`Nunn, Ex. 2027, at 32 (emphasis added).
`
`“[Nunn’s results confirm that,] considering the teachings of the Fox Papers, one of
`ordinary skill would not have expected any predictable results using the Fox methods on
`electronic databases having direct and indirect relationships as claimed.”
`‘352 Jacobs Dec., IPR2013-00478 Exhibit 2113 at ¶ 213-214; Jacobs Depo. Tr. Pt. 2 at 397-402; see also
`IPR2013-00478 POR at 50-60.
`
`13
`
`
`
`“However, despite the significant efforts to explore and develop these
`models, there remain concerns about the models’ utility for the
`searching of large scientific databases. Using the p-norm retrieval
`experiment described in Fox (1983) as an example, I will present my
`three major concerns… the reliability of extrapolating the
`performance of research systems that use the collection to a
`system to search a file over 750 times larger than the collection is
`highly questionable…”
`
`Ledwith 1992, IPR2013-00478 Exhibit 2028, at 451-452; see also ‘352 Jacobs Dec., IPR2013-00478 Exhibit
`2113 at ¶ 214; IPR2013-00478 POR at 50-60.
`
`14
`
`
`
`Dr. Fox Recognizes:
`“You have to understand the overall experimental
`paradigm here. We had a particular set of queries, we had
`a particular set of documents. We had a particular
`technique that we were using. And it’s hard to
`extrapolate that to other kinds of settings.”
`
`Fox Deposition Transcript Pt. I at 76:8-14 (emphasis added); see also ‘352 Jacobs Dec., IPR2013-00478
`Exhibit 2113 at ¶ 215; IPR2013-00478 POR at 54-55.
`
`15
`
`
`
`Dr. Fox Recognizes:
`“Q: Do you have any reason to dismiss the results of [the experiment
`of Fig. 8.11] you conducted?
`A: When you run an experiment, you get results. The results as far as I
`know were accurate in that particular situation. Whether those
`results are useful in any other situation is unclear. “
`
`Fox Deposition Transcript Pt. I at 76:8-14 (emphasis added); ‘352 Jacobs Dec., IPR2013-00478 Exhibit 2113
`at ¶ 215; see also IPR2013-00478 POR at 54-55.
`
`16
`
`
`
`Jacobs Testifies:
`“If it is “unclear” even to Dr. Fox whether these
`teachings could be extended beyond the specific
`conditions of a test on a limited experimental paper
`collection, it certainly would not have been obvious to
`one of ordinary skill to extend them to a computer
`database having direct and indirect relationships as
`claimed.”
`‘352 Jacobs Dec., IPR2013-00478 Exhibit 2113 at ¶ 215; ‘352 Patent Owner Response, IPR2013-
`00478 Paper 38 at 54-55.
`
`17
`
`
`
`•
`
`•
`
`“Dr. Fox cannot identify a single non-research search system that used the relied-
`upon bc and cc subvectors.” ‘352 Jacobs Dec., IPR2013-00478 Ex. 2113 at ¶ 188
`(citing Fox Transcript Pt. II. at 289-90. POR at 50.
`
`“[N]o commercial web search engine used indirect relationships until nearly 15
`years later after Dr. Fox’s experiments in 1998 when Google deployed the PageRank
`algorithm.” ‘352 Jacobs Dec., IPR2013-00478 Ex. 2113 at ¶ 188. POR at 50.
`
`• During this period, [Turtle and Thompson] were among those who used the CACM
`collection for their experiments. [They] did not use the bc and cc …” ‘352 Jacobs
`Dec., IPR2013-00478 Ex. 2113 at ¶ 188-189. POR at 50.
`
`•
`
`is commonly accepted that PageRank revolutionized the search industry
`“It
`generally.” Langville Dec., IPR2013-00478 Ex. 2114 at ¶ 73. POR at 59.
`
`18 B,C
`
`
`
`Dr. Langville testified:
`Dr. Fox’s focus on co-citation and bibliographic coupling would not
`have led to the significant breakthrough of the inventions claimed in
`the challenged claims of the SRA patents and embodied in PageRank
`because Dr. Fox’s research makes use of only the two most
`straightforward indirect relationships, indirect relationships of length
`involving two links). There are many other indirect
`two (i.e.,
`relationships representing higher order relationships of increasing
`length and complexity of links. Compared to Fox’s indirect relationships
`of co-citation and bibliographic coupling, PageRank, HITS, and the
`cluster link generator claimed in the challenged claims of the ’494 patent
`examine far more types of indirect relationships, thereby providing
`much more complete and less manipulable information.
`Langville Declaration, Exhibit 2114, at ¶ 50; see also IPR2013-00478 POR at 50-60.
`
`19
`
`
`
`Jacobs’s Opinions:
`I. “The experimental record showed that the use of indirect
`relationships generally resulted in worse search results”.
`Jacobs Dec. at ¶¶ 217-254; 287-296. ‘352 POR at 25-26.
`
`II. “The experimental record showed that the use of
`indirect relationships … was not sufficiently reliable to
`be used in an automated retrieval system.” Jacobs Dec.
`at ¶¶ 205-216. ‘352 POR at 25-26.
`
`20
Accessing this document will incur an additional charge of $.
After purchase, you can access this document again without charge.
Accept $ Charge
Still Working On It
This document is taking longer than usual to download. This can happen if we need to contact the court directly to obtain the document and their servers are running slowly.
Give it another minute or two to complete, and then try the refresh button.
A few More Minutes ... Still Working
It can take up to 5 minutes for us to download a document if the court servers are running slowly.
Thank you for your continued patience.
This document could not be displayed.
We could not find this document within its docket. Please go back to the docket page and check the link. If that does not work, go back to the docket and refresh it to pull the newest information.
Your account does not support viewing this document.
You need a Paid Account to view this document. Click here to change your account type.
Your account does not support viewing this document.
Set your membership
status to view this document.
With a Docket Alarm membership, you'll
get a whole lot more, including:
- Up-to-date information for this case.
- Email alerts whenever there is an update.
- Full text search for other cases.
- Get email alerts whenever a new case matches your search.
One Moment Please
The filing “” is large (MB) and is being downloaded.
Please refresh this page in a few minutes to see if the filing has been downloaded. The filing will also be emailed to you when the download completes.
Your document is on its way!
If you do not receive the document in five minutes, contact support at support@docketalarm.com.
Sealed Document
We are unable to display this document, it may be under a court ordered seal.
If you have proper credentials to access the file, you may proceed directly to the court's system using your government issued username and password.
Access Government Site
