Case 4:19-cv-03074-YGR Document 183 Filed 12/10/20 Page 1 of 46
`
` Pages 1 - 46
`
`UNITED STATES DISTRICT COURT
`
`NORTHERN DISTRICT OF CALIFORNIA
`
`BEFORE THE HONORABLE THOMAS S. HIXSON, MAGISTRATE JUDGE
`
`
`IN RE APPLE IPHONE ANTITRUST )
`LITIGATION. ) No. 11-cv-06714-YGR (TSH)
` )
`DONALD R. CAMERON, et al., )
` )
` Plaintiffs, )
` )
` VS. ) No. 19-cv-03074-YGR (TSH)
` )
`APPLE INC.,
`)
` )
` Defendant.
`)
` )
`EPIC GAMES, INC.,
`)
` )
` Plaintiff/ )
` Counter-defendant, )
` )
` VS. ) No. 20-cv-05640-YGR (TSH)
` )
`APPLE INC.,
`)
` )
` Defendant/ )
` Counterclaimant. )
` )
` San Francisco, California
` Wednesday, December 9, 2020
`
`
`
`TRANSCRIPT OF PROCEEDINGS VIA ZOOM WEBINAR
`
`
`
`(Appearances on next page)
`
`
`
`Reported by: Katherine Powell Sullivan, CSR #5812, CRR, RMR
` Official Reporter - U.S. District Court
`
`
`
` Pages 1 - 46
`
`UNITED STATES DISTRICT COURT
`
`NORTHERN DISTRICT OF CALIFORNIA
`
`BEFORE THE HONORABLE THOMAS S. HIXSON, MAGISTRATE JUDGE
`
`
`IN RE APPLE IPHONE ANTITRUST )
`LITIGATION. ) No. 11-cv-06714-YGR (TSH)
` )
`DONALD R. CAMERON, et al., )
` )
` Plaintiffs, )
` )
` VS. ) No. 19-cv-03074-YGR (TSH)
` )
`APPLE INC.,
`)
` )
` Defendant.
`)
` )
`EPIC GAMES, INC.,
`)
` )
` Plaintiff/ )
` Counter-defendant, )
` )
` VS. ) No. 20-cv-05640-YGR (TSH)
` )
`APPLE INC.,
`)
` )
` Defendant/ )
` Counterclaimant. )
` )
` San Francisco, California
` Wednesday, December 9, 2020
`
`
`
`TRANSCRIPT OF PROCEEDINGS VIA ZOOM WEBINAR
`
`
`
`(Appearances on next page)
`
`
`
`Reported by: Katherine Powell Sullivan, CSR #5812, CRR, RMR
` Official Reporter - U.S. District Court
`
`
`
`Case 4:19-cv-03074-YGR Document 183 Filed 12/10/20 Page 2 of 46
`
` 2
`
`APPEARANCES: (via Zoom Webinar)
`
`For Plaintiffs in In re Apple iPhone Antitrust Litigation,
`11-cv-06714-YGR:
` WOLF HALDENSTEIN ADLER FREEMAN & HERZ LLP
` 750 B Street, Suite 1820
` San Diego, California 92101
` BY: RACHELE R. BYRD, ESQ.
`
`For Plaintiffs in Cameron, et. al v. Apple Inc.,
`19-cv-03074-YGR:
` HAGENS BERMAN SOBOL SHAPIRO LLP
` 1301 Second Avenue, Suite 2000
` Seattle, Washington 98101
` BY: ROBERT F. LOPEZ, ESQ.
`
`For Plaintiff Epic Games, Inc.:
` CRAVATH, SWAINE & MOORE LLP
` 825 Eighth Avenue
` New York, New York 10019
` BY: LAUREN A. MOSKOWITZ, ESQ.
`
`For Defendant Apple Inc.:
` GIBSON, DUNN & CRUTCHER LLP
` 555 Mission Street
` San Francisco, California 94105-0921
` BY: ETHAN D. DETTMER, ESQ.
`
` 1
`
` 2
`
` 3
`
` 4
`
` 5
`
` 6
`
` 7
`
` 8
`
` 9
`
`10
`
`11
`
`12
`
`13
`
`14
`
`15
`
`16
`
`17
`
`18
`
`19
`
`20
`
`21
`
`22
`
`23
`
`24
`
`25
`
`
`
`Case 4:19-cv-03074-YGR Document 183 Filed 12/10/20 Page 3 of 46
`
` 3
`
` 1
`
` 2
`
` 3
`
` 4
`
` 5
`
` 6
`
` 7
`
` 8
`
` 9
`
`10
`
`11
`
`12
`
`13
`
`14
`
`15
`
`16
`
`17
`
`18
`
`19
`
`20
`
`21
`
`22
`
`23
`
`24
`
`25
`
`Wednesday - December 9, 2020
`
` 1:01 p.m.
`
`P R O C E E D I N G S
`
`---000---
`
`THE CLERK: All right, everyone. Good afternoon.
`
`Thank you for all joining us on the Zoom call.
`
`We're here in Civil Action 11-6714, In re Apple iPhone
`
`Antitrust Litigation, and in Case Number 19-3074, Cameron, et
`
`al. versus Apple Inc. And the last case is 20-5640, Epic
`
`Games, Inc. versus Apple Inc.
`
`Counsel, please state your appearances for the record.
`
`The Honorable Thomas S. Hixson, presiding.
`
`Let's start with the Apple Antitrust Litigation first.
`
`MS. BYRD: Good afternoon, Your Honor. This is
`
`Rachele Byrd, with Wolf Haldenstein Adler Freeman & Herz, on
`
`behalf of the plaintiffs.
`
`THE COURT: Good afternoon.
`
`THE CLERK: Thank you.
`
`And the Cameron v. Apple.
`
`MR. LOPEZ: Good afternoon, Your Honor. This is Rob
`
`Lopez, of Hagens Berman, for the developer plaintiffs in the
`
`Cameron matter.
`
`THE COURT: Good afternoon.
`
`THE CLERK: And Epic Games, Inc., versus Apple Inc.
`
`MS. MOSKOWITZ: Good afternoon, Your Honor. Lauren
`
`Moskowitz, from Cravath, Swaine & Moore, on behalf of Epic
`
`
`
`Case 4:19-cv-03074-YGR Document 183 Filed 12/10/20 Page 4 of 46
`
` 4
`
` 1
`
` 2
`
` 3
`
` 4
`
` 5
`
` 6
`
` 7
`
` 8
`
` 9
`
`10
`
`11
`
`12
`
`13
`
`14
`
`15
`
`16
`
`17
`
`18
`
`19
`
`20
`
`21
`
`22
`
`23
`
`24
`
`25
`
`Games.
`
`THE COURT: Good afternoon.
`
`MR. DETTMER: And, Your Honor, Ethan Dettmer, from
`
`Gibson Dunn & Crutcher, on behalf of Apple in all three
`
`matters.
`
`THE COURT: Good afternoon.
`
`MR. DETTMER: Good afternoon, Your Honor.
`
`THE COURT: So let me tell you what my agenda is for
`
`the hearing. I have some questions that I want to ask Epic
`
`Games and Apple, and then I have some thoughts that I want to
`
`discuss with those two parties.
`
`And then what I'm likely to do is tell you to go meet and
`
`confer a little bit more in light of my -- the ideas I discuss
`
`with the two parties.
`
`I know that you've met and conferred for several weeks
`
`already in a good-faith attempt to get to an answer, and I'm
`
`hoping that with some feedback from me we can get you over the
`
`finish line.
`
`So, anyway, there's not going to be an order coming out of
`
`this hearing. Or the order will be that you talk with each a
`
`little bit more.
`
`First, I want to ask Epic, Inc., just to make sure that I
`
`understand how you searched for documents, I understand that
`
`you had a set of search terms and that was shared with Apple.
`
`Is that correct?
`
`
`
`Case 4:19-cv-03074-YGR Document 183 Filed 12/10/20 Page 5 of 46
`
` 5
`
` 1
`
` 2
`
` 3
`
` 4
`
` 5
`
` 6
`
` 7
`
` 8
`
` 9
`
`10
`
`11
`
`12
`
`13
`
`14
`
`15
`
`16
`
`17
`
`18
`
`19
`
`20
`
`21
`
`22
`
`23
`
`24
`
`25
`
`MS. MOSKOWITZ: Yes, Your Honor. We had initially
`
`proposed a set of terms, and Apple responded with a lengthy set
`
`of proposed conditions that we met and conferred and reached
`
`agreement on.
`
`THE COURT: Okay. So there was -- at the start of the
`
`process, there was an agreement on a group of search terms so
`
`Apple would know if a document didn't have one of their search
`
`terms it wasn't going to come up. And if it did, then the
`
`search term would pull it out. Is that right?
`
`MS. MOSKOWITZ: Correct. Correct.
`
`THE COURT: Okay. Then after you ran the search
`
`terms, did you have a team of document reviewers do manual
`
`review to see which documents were, in fact, responsive?
`
`MS. MOSKOWITZ: That's correct, Your Honor. We are
`
`performing a linear review of all of the search term hit
`
`results, which is in the order of over 3.5 million documents.
`
`THE COURT: Wow. That sounds like a big task.
`
`MS. MOSKOWITZ: It is.
`
`THE COURT: Was there ever a time when you took a
`
`sample of the documents that hit on the search terms and then
`
`disclosed to Apple, here's how we're making responsiveness
`
`calls with respect to this set, and let us know what you --
`
`what you think?
`
`In other words, I knew that they had input on your search
`
`terms. What I'm wondering is whether they also had any input
`
`
`
`Case 4:19-cv-03074-YGR Document 183 Filed 12/10/20 Page 6 of 46
`
` 6
`
` 1
`
` 2
`
` 3
`
` 4
`
` 5
`
` 6
`
` 7
`
` 8
`
` 9
`
`10
`
`11
`
`12
`
`13
`
`14
`
`15
`
`16
`
`17
`
`18
`
`19
`
`20
`
`21
`
`22
`
`23
`
`24
`
`25
`
`into your responsiveness calls or if -- and I know this a term
`
`that no one -- it sounds pejorative, but whether what the
`
`manual reviewers were doing is not just a black box to Apple?
`
`MS. MOSKOWITZ: Your Honor, we have not disclosed a
`
`set of specific documents with the up or down determinations
`
`that we are making. We haven't discussed that with Apple.
`
`We are -- we can describe that we are taking quite a broad
`
`responsiveness on view, and we have disclosed that concept to
`
`Apple, that we are not making very fine responsiveness calls,
`
`but rather we are erring on the side of responsiveness.
`
`THE COURT: I see. So, presumably, there's some
`
`attorney-client privilege review memo, and your team are
`
`working off of that. Is that right?
`
`MS. MOSKOWITZ: A responsiveness protocol that
`
`includes both privilege calls and the types of documents that
`
`really shouldn't be marked "responsive," along the lines of
`
`technical documents, coding documents, that really are not
`
`relevant to the issues in the case. But separate from that,
`
`the instructions have been to mark "responsive."
`
`THE COURT: Okay. And then in your validation
`
`protocol, I think you had two aspects to it. One is you would
`
`validate that the search terms did, in fact, pull up a certain
`
`recall level. Is that the idea?
`
`MS. MOSKOWITZ: For step one, Your Honor, that's
`
`correct. What we had proposed to do upfront, so that we could
`
`
`
`Case 4:19-cv-03074-YGR Document 183 Filed 12/10/20 Page 7 of 46
`
` 7
`
` 1
`
` 2
`
` 3
`
` 4
`
` 5
`
` 6
`
` 7
`
` 8
`
` 9
`
`10
`
`11
`
`12
`
`13
`
`14
`
`15
`
`16
`
`17
`
`18
`
`19
`
`20
`
`21
`
`22
`
`23
`
`24
`
`25
`
`know with certainty the universe that we were going to be
`
`performing the linear review on, would be to take a sample of
`
`the overall population.
`
`And that's, I think, on the order of 6-and-a-half million
`
`documents after de-duplication, and to take a sample of 1500
`
`documents from that full universe, some of which would be from
`
`the hit results and some from the non-hit results, review those
`
`by an expert for responsiveness, and calculate a recall
`
`percentage based on the responsive hit in that sample
`
`population.
`
`And we had proposed to Apple that we would reach an
`
`80 percent recall on that such that 80 percent of the
`
`responsive documents were being returned by the search terms
`
`and that we would be prepared to move forward on that basis.
`
`THE COURT: Okay. So that's the first aspect.
`
`Then there was another aspect, and that was the -- to take
`
`a random sample of the ones coded "not responsive" to determine
`
`which ones should have been responsive, and you would calculate
`
`the elusion rate.
`
`And for the benefit of the court reporter, I'm saying
`
`e-l-u-s-i-o-n. I'm not saying i-l-l-u-s-i-o-n.
`
`So is that the other aspect of it?
`
`MS. MOSKOWITZ: That's correct, Your Honor. That is
`
`what our proposal was, as a compromise, given that we -- we
`
`personally have never seen anyone go that far on a search terms
`
`
`
`Case 4:19-cv-03074-YGR Document 183 Filed 12/10/20 Page 8 of 46
`
` 8
`
` 1
`
` 2
`
` 3
`
` 4
`
` 5
`
` 6
`
` 7
`
` 8
`
` 9
`
`10
`
`11
`
`12
`
`13
`
`14
`
`15
`
`16
`
`17
`
`18
`
`19
`
`20
`
`21
`
`22
`
`23
`
`24
`
`25
`
`protocol.
`
`I'm happy to give more background on how we ended up
`
`there, but that is the end result of what we were proposing to
`
`do.
`
`THE COURT: I see.
`
`So I had two concerns that maybe you can speak to. The
`
`one is that the recall rate that you're estimating is not a
`
`true recall rate because your doc review process had two steps.
`
`First you had search terms, and then we had the manual
`
`review. And so this 80 percent, it doesn't capture the manual
`
`review part, and that's kind of what I'm struggling with.
`
`I understand that your elusion rate is designed to deal
`
`with the manual review problem, but the concern there -- I
`
`think Apple's right, that if the search terms got in a lot of
`
`nonresponsive documents, then you could pass the elusion rate
`
`test even if actual recall wasn't that great.
`
`So I'm wondering if you could speak to those concerns.
`
`MS. MOSKOWITZ: Yes, Your Honor. I think, just from
`
`a -- backing up to first principles on what you're trying to
`
`achieve in discovery, I think that we have a disagreement on
`
`what that is.
`
`And we don't view the exercise of discovery, especially in
`
`cases with just millions and millions and millions and millions
`
`of documents, that you are trying to find every responsive
`
`document.
`
`
`
`Case 4:19-cv-03074-YGR Document 183 Filed 12/10/20 Page 9 of 46
`
` 9
`
` 1
`
` 2
`
` 3
`
` 4
`
` 5
`
` 6
`
` 7
`
` 8
`
` 9
`
`10
`
`11
`
`12
`
`13
`
`14
`
`15
`
`16
`
`17
`
`18
`
`19
`
`20
`
`21
`
`22
`
`23
`
`24
`
`25
`
`We've all served bilateral requests that, if read
`
`literally, call for every document in our files. And the real
`
`idea is to get not just responsive documents but what's the
`
`meat of the documents in this case.
`
`An exhibit list in this trial is going to be a fraction of
`
`a fraction of the documents produced. And so search terms are
`
`not just targeted at getting responsive documents but getting
`
`what the critical documents are.
`
`Apple got to say, we think that the documents that you
`
`have, that we really care about, are going to say the following
`
`types of words, and go look for those documents, and then
`
`review those for responsiveness and produce them.
`
`That is the whole idea behind search terms. It's not a
`
`blind responsiveness review but, rather, a targeted meet and
`
`confer on targeted terms that Apple specifically can request us
`
`to look at to get the documents that they think they need in
`
`the case.
`
`And so the -- the 80 percent offer that we did was
`
`really -- it does look at responsiveness, but it's really just
`
`a check on is there a huge number of responsive documents that
`
`aren't hitting on the terms?
`
`And that gives you the comfort that that's not happening,
`
`that only 20 percent of technically responsive documents are
`
`being left behind.
`
`But Apple has already told us the specific documents they
`
`
`
`Case 4:19-cv-03074-YGR Document 183 Filed 12/10/20 Page 10 of 46
`
` 10
`
` 1
`
` 2
`
` 3
`
` 4
`
` 5
`
` 6
`
` 7
`
` 8
`
` 9
`
`10
`
`11
`
`12
`
`13
`
`14
`
`15
`
`16
`
`17
`
`18
`
`19
`
`20
`
`21
`
`22
`
`23
`
`24
`
`25
`
`want by virtue of the search terms. So it's really the search
`
`terms themselves are the validation of our protocol. And the
`
`80 percent is just a double check or a triple check because
`
`they really do have so much involvement and had so much
`
`involvement in search terms themselves.
`
`So that's why we proposed that on the search terms piece.
`
`And I can certainly speak to the second aspect of it, but I may
`
`have lost your question if I haven't yet answered it.
`
`THE COURT: I think you have. I think the concept
`
`here, that you're getting at, is sometimes referred to as
`
`precision. The only way you can get every relevant document is
`
`you produce all the documents. But no one wants all the
`
`documents, they want the good stuff. And so okay.
`
`All right. Let me -- let me ask Apple some questions.
`
`If I understand the way you did TAR, at the beginning you
`
`would have to -- I mean, the machine learns. So you start by
`
`making some responsiveness calls so the machine can learn from
`
`that and it can find documents that are similar to the ones
`
`that are responsive and not so much of the nonresponsive ones.
`
`Let me ask you a version of the same question I asked Epic
`
`Games, which is: Was there a time early on when you had a
`
`sample set of documents, you made responsiveness calls, and you
`
`shared those with Epic Games to get their input?
`
`MR. DETTMER: No, Your Honor.
`
`First of all, we obviously started this review long before
`
`
`
`Case 4:19-cv-03074-YGR Document 183 Filed 12/10/20 Page 11 of 46
`
` 11
`
` 1
`
` 2
`
` 3
`
` 4
`
` 5
`
` 6
`
` 7
`
` 8
`
` 9
`
`10
`
`11
`
`12
`
`13
`
`14
`
`15
`
`16
`
`17
`
`18
`
`19
`
`20
`
`21
`
`22
`
`23
`
`24
`
`25
`
`Epic filed its lawsuit. You know, so there's two reviews,
`
`right? There's the App Store antitrust class action reviews
`
`that have been going on, you know, for better part of a year
`
`now, and then the new Epic ones, which are separate but
`
`related.
`
`But in direct answer to your question, the way this
`
`particular TAR system works is it does not operate, you know,
`
`with any kind of seed set or anything like that. It's a
`
`continuous active learning-type system where the review starts
`
`and each and every call that a reviewer makes then looks at
`
`each document, looks at the -- you know, the characteristics of
`
`that document, and looks for other documents in the whole
`
`collection that are either like or unlike it, and then re-sorts
`
`the stack of documents to put the more likely to be a
`
`responsive documents toward the top.
`
`But that system keeps going throughout the whole review,
`
`and you keep doing it until you get to a point where, you know,
`
`the richness of the documents you're reviewing gets to a very
`
`low point. And then, you know, you do some validation and you
`
`stop because there aren't enough documents left.
`
`So this system is great in that it makes the review more
`
`efficient by pushing the more likely to be responsive documents
`
`toward the top, but also has the advantage of allowing you to
`
`review, you know, each and every document as you go along until
`
`you get to a point where, you know, there just aren't many
`
`
`
`Case 4:19-cv-03074-YGR Document 183 Filed 12/10/20 Page 12 of 46
`
` 12
`
` 1
`
` 2
`
` 3
`
` 4
`
` 5
`
` 6
`
` 7
`
` 8
`
` 9
`
`10
`
`11
`
`12
`
`13
`
`14
`
`15
`
`16
`
`17
`
`18
`
`19
`
`20
`
`21
`
`22
`
`23
`
`24
`
`25
`
`responsive documents left in what you're reviewing.
`
`Now, I will just -- and sorry to drag this out a little
`
`bit. I'll give you a caveat in that, as Your Honor probably
`
`knows, we have a much more aggressive timeline in place right
`
`now than we had before the Epic case started.
`
`So we have made a slight change to that now where we're
`
`actually going to use kind of a hybrid system of what I just
`
`described and a more traditional TAR 1 system where, you know,
`
`the seed set, so to speak, is what we've done already, which is
`
`obviously millions of documents reviewed, and hundreds of
`
`thousands in the Epic case, which then has taught the system.
`
`So it's complicated, but that's the -- that's the high
`
`level. And I will say we have described this in detail to the
`
`plaintiffs, including, you know, allowing them to talk with our
`
`consultant about it.
`
`THE COURT: Okay. Thank you. That's helpful.
`
`And then, as I understand your validation proposal,
`
`there's a symmetry with Epic Games in the sense that what you
`
`want to validate is 75 percent recall of what you think is
`
`responsive, but they don't really know what that is.
`
`Like, just as the -- they haven't -- you don't have any
`
`input into what their manual reviewers are saying, responsive
`
`or not responsive. The calls that Apple has been making, you
`
`know, you've been making, but they don't get to see that and
`
`they wouldn't as part of the validation. Is that right?
`
`
`
`Case 4:19-cv-03074-YGR Document 183 Filed 12/10/20 Page 13 of 46
`
` 13
`
` 1
`
` 2
`
` 3
`
` 4
`
` 5
`
` 6
`
` 7
`
` 8
`
` 9
`
`10
`
`11
`
`12
`
`13
`
`14
`
`15
`
`16
`
`17
`
`18
`
`19
`
`20
`
`21
`
`22
`
`23
`
`24
`
`25
`
`MR. DETTMER: Well, yes and no, Your Honor. I mean,
`
`we did produce to Epic, you know, as soon as -- well, shortly
`
`after they started the case, we produced to them the
`
`approximately 3.7 million documents that we produced in the
`
`class actions to date.
`
`So they have that production and, obviously, know that
`
`those documents were coded as responsive. So to that extent,
`
`they do know what we are coding is responsive. And as Your
`
`Honor knows, they take issue with at least part of that. And,
`
`obviously, I'm happy to talk about that further.
`
`So I suppose, in a sense, they do know at least --
`
`THE COURT: Well, I mean, you both know that, what the
`
`other side has produced. I understand that. But how the
`
`sausage got made, you don't know. You know the output of it
`
`and they know the output of it.
`
`There are some times with TAR where people do exchange
`
`seed sets, and they do, as part of validation, exchange all
`
`that. I took from the letter brief that that didn't happen
`
`here. I'm just asking the question to confirm that that
`
`understanding is correct.
`
`MR. DETTMER: That's correct, Your Honor. And, as I
`
`described, we don't really have a seed set in the first case.
`
`THE COURT: You started it before there was an Epic,
`
`Inc. lawsuit, so I guess they -- they really couldn't have had
`
`much input.
`
`
`
`Case 4:19-cv-03074-YGR Document 183 Filed 12/10/20 Page 14 of 46
`
` 14
`
` 1
`
` 2
`
` 3
`
` 4
`
` 5
`
` 6
`
` 7
`
` 8
`
` 9
`
`10
`
`11
`
`12
`
`13
`
`14
`
`15
`
`16
`
`17
`
`18
`
`19
`
`20
`
`21
`
`22
`
`23
`
`24
`
`25
`
`MR. DETTMER: Right.
`
`THE COURT: Okay. So here's a thought that I have.
`
`At a high level, I -- I don't think search terms are any better
`
`than TAR. They both have pluses and minuses. They have, you
`
`know, pros and cons depending on the kind of case, but there's
`
`no reason why one of them is inherently better than the other.
`
`So I want to set a recall burden that's roughly similar for the
`
`two sides because I do think that's appropriate.
`
`Apple's proposal of -- I think it was 75 percent? Is that
`
`what you were looking at?
`
`MR. DETTMER: Yes, Your Honor.
`
`THE COURT: That seems right to me.
`
`There is this tension between recall and precision that if
`
`you go up too high then you start producing a lot of junk,
`
`which, you know, no one wants.
`
`But here's a thought that I have, is that maybe we need to
`
`measure recall differently for the two sides. And let me ask
`
`Apple first.
`
`So with the search terms it seems a little unfair to
`
`measure recall against the population of documents that were
`
`collected, because you know and Epic Games knows that
`
`everything that didn't have a search term is just never going
`
`to get reviewed.
`
`And so what I'm thinking is that maybe the real issue is
`
`making sure that their manual review was consistent and found
`
`
`
`Case 4:19-cv-03074-YGR Document 183 Filed 12/10/20 Page 15 of 46
`
` 15
`
` 1
`
` 2
`
` 3
`
` 4
`
` 5
`
` 6
`
` 7
`
` 8
`
` 9
`
`10
`
`11
`
`12
`
`13
`
`14
`
`15
`
`16
`
`17
`
`18
`
`19
`
`20
`
`21
`
`22
`
`23
`
`24
`
`25
`
`the stuff that it's supposed to find.
`
`And so I'm thinking that maybe we -- for Epic Games, we
`
`measure recall by the manual review as compared to the
`
`documents that hit on the search terms, and we want 75 percent
`
`of recall in that universe.
`
`Another way of doing it is to compare what the manual
`
`review says is responsive against not just what the search
`
`terms found but what they were run against and didn't pull up.
`
`But I'm wondering if that might be too big a burden on Epic
`
`Games.
`
`What are your thoughts?
`
`MS. MOSKOWITZ: Your Honor, I do --
`
`THE COURT: First I was asking Apple --
`
`MS. MOSKOWITZ: Oh. Pardon me.
`
`THE COURT: -- because I'm proposing something more
`
`favorable to you, and I want to hear their strenuous objections
`
`to it before --
`
`MR. DETTMER: Well, I guess -- I guess a concern, Your
`
`Honor, is twofold. One, you know, the -- the idea of search
`
`terms, as Your Honor pointed out, there are pluses and minuses.
`
`One of the minuses, particularly in a case like this one,
`
`where, you know, we obviously did have some input into what the
`
`search terms were going to be, you know, at the outset we
`
`obviously didn't have any idea what their documents were at the
`
`time.
`
`
`
`Case 4:19-cv-03074-YGR Document 183 Filed 12/10/20 Page 16 of 46
`
` 16
`
` 1
`
` 2
`
` 3
`
` 4
`
` 5
`
` 6
`
` 7
`
` 8
`
` 9
`
`10
`
`11
`
`12
`
`13
`
`14
`
`15
`
`16
`
`17
`
`18
`
`19
`
`20
`
`21
`
`22
`
`23
`
`24
`
`25
`
`So, you know, while we can make a, you know, not
`
`particularly well-educated guess about what the good search
`
`terms might be, there's a lot of blind spots that we had in
`
`giving our input into those search terms.
`
`So I think, you know, I would have a lot of concern with
`
`that, that there could be a lot of responsive documents that
`
`were left out of the search term call. I mean, in, you know, a
`
`lot of these cases, as I'm sure Your Honor knows, there's an
`
`iterative search process where you have search terms, you get
`
`documents back, you look at the documents, you do another round
`
`of that where you have sort of more knowledge about what the
`
`documents are so that you can then, you know, say something
`
`more intelligent about -- about the search terms.
`
`And, frankly, we just don't have time for that here given
`
`the really aggressive schedule in the case.
`
`THE COURT: Let me sort of short-circuit this a little
`
`bit by saying, I understand and agree with all the things
`
`you're saying. It's always frustrating when one side says,
`
`"What search terms do you want us to run?" You know, I don't
`
`know, I have your documents. That's a logical thing to say.
`
`I guess I'm more asking you about here we are getting
`
`toward the end, and you knew that they were using search terms.
`
`You had some limited ability to thoughtfully comment on them,
`
`and you did what you could. But the known risk is that if a
`
`document didn't hit on a search term, they weren't going to
`
`
`
`Case 4:19-cv-03074-YGR Document 183 Filed 12/10/20 Page 17 of 46
`
` 17
`
` 1
`
` 2
`
` 3
`
` 4
`
` 5
`
` 6
`
` 7
`
` 8
`
` 9
`
`10
`
`11
`
`12
`
`13
`
`14
`
`15
`
`16
`
`17
`
`18
`
`19
`
`20
`
`21
`
`22
`
`23
`
`24
`
`25
`
`find it. So that was -- you knew that at the beginning. And
`
`so I'm wondering if it's fair to hold you to that.
`
`The thing you had no ability to evaluate was their manual
`
`review. That really was a black box. And I'm wondering if --
`
`should we at least hold Epic Games to the burden to prove that
`
`they found the things they were trying to find?
`
`That's really all your validation proposal is showing, as
`
`well, that you found the things that you think were responsive.
`
`And so I'm wondering if just because you walked into this
`
`known risk -- like, at the end of the document production we
`
`can't have Epic Games redo all their search terms. If they
`
`weren't any good, that problem existed from the beginning. But
`
`we just don't have time to do that now.
`
`And so that's why I'm wondering if recall should be
`
`measured against what hit on the search terms. Can you respond
`
`more to that, rather than addressing why search terms aren't
`
`great, which I know already.
`
`MR. DETTMER: Well, Your Honor, I didn't mean to
`
`sidetrack. I guess the reason I raise that is because that
`
`creates, at the end of the day, a validation protocol that
`
`really treats the two parties very differently.
`
`And I understand we're using different methodologies, but
`
`at the end of the day our belief -- Apple's belief is that,
`
`regardless of whether we're using search terms or TAR, each
`
`party should -- and, frankly, the lawyers should be held to the
`
`
`
`Case 4:19-cv-03074-YGR Document 183 Filed 12/10/20 Page 18 of 46
`
` 18
`
` 1
`
` 2
`
` 3
`
` 4
`
` 5
`
` 6
`
` 7
`
` 8
`
` 9
`
`10
`
`11
`
`12
`
`13
`
`14
`
`15
`
`16
`
`17
`
`18
`
`19
`
`20
`
`21
`
`22
`
`23
`
`24
`
`25
`
`same ultimate recall standard.
`
`And I think if Apple is being held to a recall standard
`
`against all the documents that are sort of in the search
`
`universe and Epic is being held to one that's, you know, a much
`
`narrower universe of documents that have already been searched
`
`and winnowed, that's really not a -- you know, a real
`
`comparison. And Epic is being held to a much lower standard
`
`under that different protocol than we are.
`
`THE COURT: Well, that's true in one sense. They're
`
`certainly being held to a different standard. But I guess at
`
`the beginning Apple said, we're running TAR against all of the
`
`documents that we collect.
`
`And Epic Games has said, we're not going to do a manual
`
`review of all the documents, we're only going to do it with the
`
`ones that come up with search terms.
`
`So the way that these are symmetrical is that we're trying
`
`to decide how good were you at finding the things you wanted to
`
`find in the places you were looking? And Apple and Epic were
`
`looking at different places.
`
`But, anyway, let me hear Epic Games' view on that.
`
`MS. MOSKOWITZ: Thank you, Your Honor.
`
`We do think that what Your Honor is proposing is better
`
`than what Apple is proposing, certainly. Though the way the
`
`math would work out, if we were including both the search terms
`
`recall, meaning hits and non-hits, and the recall on the
`
`
`
`Case 4:19-cv-03074-YGR Document 183 Filed 12/10/20 Page 19 of 46
`
` 19
`
` 1
`
` 2
`
` 3
`
` 4
`
` 5
`
` 6
`
` 7
`
` 8
`
` 9
`
`10
`
`11
`
`12
`
`13
`
`14
`
`15
`
`16
`
`17
`
`18
`
`19
`
`20
`
`21
`
`22
`
`23
`
`24
`
`25
`
`review, we would have to hit -- almost 97 out of every hundred
`
`needs to be in agreement. And I couldn't even agree with
`
`myself 97 out of a hundred times.
`
`And so we do -- we do think search terms are different, as
`
`Your Honor pointed out. We do think that the very nature of
`
`them allowed Apple to have input on what was going to be
`
`reviewed. And so we should move past that and then do the
`
`recall on the review.
`
`We had proposed elusion because, frankly, whether or not
`
`they got extra documents as responsive that maybe they would
`
`have thought were nonresponsive, so what.
`
`And, really, what we should be looking at is, were there
`
`really responsive documents in what we were not producing and
`
`marking "nonresponsive." That's what we thought, really, we
`
`should be trying to get at, because is anyone really going to
`
`complain from Epic producing a few documents that maybe could
`
`have been marked "nonresponsive" when people were being really
`
`careful and circumscribed in what we were calling responsive,
`
`which we're not doing.
`
`So that is why we proposed it. We thought that would give
`
`Apple the comfort that there weren't a whole batch of documents
`
`sitting as a nonresponsive code, that they wouldn't get. That
`
`is the thought behind it.
`
`But if Your Honor was focused on doing an overall recall
`
`number, we would be prepared to meet and confer with Apple and
`
`
`
`Case 4:19-cv-03074-YGR Document 183 Filed 12/10/20 Page 20 of 46
`
` 20
`
` 1
`
` 2
`
` 3
`
` 4
`
` 5
`
` 6
`
` 7
`
` 8
`
` 9
`
`10
`
`11
`
`12
`
`13
`
`14
`
`15
`
`16
`
`17
`
`18
`
`19
`
`20
`
`21
`
`22
`
`23
`
`24
`
`25
`
`to do that on the percentage of documents that hit on search
`
`terms and, thus, were actually linearly reviewed.
`
`THE COURT: All right. Thank you. That is what I'm
`
`thinking.
`
`Let me turn to the -- let me just look at my notes to make
`
`sure that I -- so now I have a question for Apple.
`
`Epic Games says that of the 3.8 million documents you
`
`produced, 2.2 were these bulk emails. If I were to say you
`
`have to exclude those before you calculate the recall rate, how
`
`would that affect -- how would that affect you?
`
`MR. DETTMER: It would be a dramatic problem in the
`
`whole review, Your Honor, for a host of reasons.
`
`One is, as Your Honor probably knows, Judge Gonzalez
`
`Rogers talked about how we should not be relitigating the
`
`earlier production and review, and this would be exactly that.
`
`You know, Epic said twice in their letter to you that
`
`these are responsive documents. They're certainly responsive
`
`documents. It was admitted twice in the letter. They claim
`
`that they're not relevant. That is just not true.
`
`I mean, these documents are part of what Apple does to
`
`review apps and app providers, developers in the App Store
`
`ecosystem. It does things -- these emails show things like
`
`approval of selling apps in different areas. It shows, you
`
`know, changes to apps that may be needed to be followed up for
`
`security or quality reasons. It shows, you know, when there
`
`
`
`Case 4:19-cv-03074-YGR Document 183 Filed 12/10/20 Page 21 of 46
`
` 21
`
` 1
`
` 2
`
` 3
`
` 4
`
` 5
`
` 6
`
` 7
`
` 8
`
` 9
`
`10
`
`11
`
`12
`
`13
`
`14
`
`15
`
`16
`
`17
`
`18
`
`19
`
`20
`
`21
`
`22
`
`23
`
`24
`
`25
`
`have been changes to how the apps are actually operating that
`
`Apple needs to follow up on.
`
` So these are all part of showing how Apple takes these
`
`dramatic steps and then puts all this investment into making
`
`the App Store ecosystem safer, better for developers and
`
`consumers. So these are, you know, very relevant to the claims
`
`and defenses in this case. And the fact that Epic doesn't
`
`think they are is just, A, not right. They are part of our
`
`defenses in the case and certainly relevant to those.
`
`So, you know, in calculating recall, which, as Your Honor
`
`knows, is just the percentage of documents that are responsive,
`
`they're responsive. I mean, it's admitted. It's just true.
`
`And if we had to go back and redo our production now
`
`because of sort of a second-guessing of, you know, whether
`
`these documents were relevant enough, which I don't even know
`
`how you impose that kind of standard, would --
`
`THE COURT: You're getting a little bit away from my
`
`question. I'm certainly not proposing that you redo your
`
`document production. And I understand the philosophy behind
`
`including those documents.
`
`The concern I had is that there -- it looked like most of
`
`your document production are these highly repetitive --
`
`relevant, yes, but also low-value documents. And I'm worried
`
`that -- that the inclusion of those could mean that if your
`
`recall rate was terrible on everything else, you would,
`
`
`
`Case 4:19-cv-03074-YGR Document 183 Filed 12/10/20 Page 22 of 46
`
` 22
`
` 1
`
` 2
`
` 3
`
` 4
`
` 5
`
` 6
`
` 7
`
` 8
`
` 9
`
`10
`
`11
`
`12
`
`13
`
`14
`
`15
`
`16
`
`17
`
`18
`
`19
`
`20
`
`21
`
`22
`
`23
`
`24
`
`25
`
`nonetheless, pass the 75 percent threshold.
`
`And so I just mean operationally, from a TAR validation
`
`standpoint, what if I said, look, knock out the 2.2 million.
`
`Can you describe -- would this have any -- I don't want to do
`
`something that would make it too difficult for Apple to satisfy
`
`the validation. That's the point I'm getting at. I don't want
`
`to tell them to produce something that's impossible or very
`
`difficult.
`
`And so I'm asking, as an operational method, if you had to
`
`block out the 2.2 million, those documents, what effect, if
`
`any, would that have on your ability to get a recall rate of
`
`75 percent?
`
`MR. DETTMER: It would -- I don't -- we haven't done
`
`it at this point, but it would certainly affect, probably very
`
`negatively, our ability to do that in a way that I think is
`
`both, you know, unfair -- I mean, look, Your Honor, we reviewed
`
`all these documents. I mean, these weren't bulk-coded
`
`documents. They weren't, you know, machine coded.
`
`People went
Accessing this document will incur an additional charge of $.
After purchase, you can access this document again without charge.
Accept $ Charge
Still Working On It
This document is taking longer than usual to download. This can happen if we need to contact the court directly to obtain the document and their servers are running slowly.
Give it another minute or two to complete, and then try the refresh button.
A few More Minutes ... Still Working
It can take up to 5 minutes for us to download a document if the court servers are running slowly.
Thank you for your continued patience.
This document could not be displayed.
We could not find this document within its docket. Please go back to the docket page and check the link. If that does not work, go back to the docket and refresh it to pull the newest information.
Your account does not support viewing this document.
You need a Paid Account to view this document. Click here to change your account type.
Your account does not support viewing this document.
Set your membership
status to view this document.
With a Docket Alarm membership, you'll
get a whole lot more, including:
- Up-to-date information for this case.
- Email alerts whenever there is an update.
- Full text search for other cases.
- Get email alerts whenever a new case matches your search.
One Moment Please
The filing “” is large (MB) and is being downloaded.
Please refresh this page in a few minutes to see if the filing has been downloaded. The filing will also be emailed to you when the download completes.
Your document is on its way!
If you do not receive the document in five minutes, contact support at support@docketalarm.com.
Sealed Document
We are unable to display this document, it may be under a court ordered seal.
If you have proper credentials to access the file, you may proceed directly to the court's system using your government issued username and password.
Access Government Site
