`· · ·BEFORE THE PATENT TRIAL AND APPEAL BOARD
`·2· ·-----------------------------------------X
`· · ·TOYOTA MOTOR CORPORATION,
`·3
`
`·4· · · · · · · · · · ·Petitioner,
`
`·5· · · · · · · · · · · · · · ·IPR2013-00412
`· · ·VS.· · · · · · · · · · · ·IPR2013-00413
`·6· · · · · · · · · · · · · · ·IPR2013-00416
`
`·7
`· · ·AMERICAN VEHICULAR SCIENCES, LLC,
`·8
`· · · · · · · · · · · ·Patent Owner.
`·9· ·-----------------------------------------X
`
`10
`
`11
`
`12· · · · · · · · · DEPOSITION OF
`
`13· · · · ·NIKOLAOS PAPANIKOLOPOULOS, PH.D.
`
`14· · · · · · ·Monday, February 24, 2014
`
`15· · · · · · · · New York, New York
`
`16
`
`17
`
`18
`
`19
`
`20
`
`21
`
`22
`
`23· ·Reported By:
`
`24· ·LINDA J. GREENSTEIN
`
`25· ·JOB NO. 89371
`
`AVS EXHIBIT 2003
`Toyota Inc. v. American Vehicular Sciences LLC
`IPR2013-00424
`
`
`
`·1· · · · Nikolaos Papanikolopoulos, Ph.D.
`
`·2· ·exploited -- we had a camera mounted inside
`
`·3· ·the car, and we use a location of the
`
`·4· ·license plate with respect to the rest of
`
`·5· ·the car, and we tried to compute also other
`
`·6· ·characteristics; say, characteristics of
`
`·7· ·the license plate.
`
`·8· · · · · · · And then based on the size, we
`
`·9· ·try to keep the size constant, it allows us
`
`10· ·to keep constant distance as part of a
`
`11· ·vehicle following application.
`
`12· · · · Q.· · Did that system use a pattern
`
`13· ·recognition algorithm?
`
`14· · · · · · · MR. BERKOWITZ:· Object to form.
`
`15· · · · A.· · Again, let me clarify since we
`
`16· ·have these terms in my -- the claim
`
`17· ·construction I was given.· Do you want me
`
`18· ·to answer --
`
`19· · · · Q.· · As you normally use that term,
`
`20· ·did that system, the one that you refer to
`
`21· ·in paragraph 4 of your 419 application, use
`
`22· ·a pattern recognition algorithm?
`
`23· · · · · · · MR. BERKOWITZ:· Object to form.
`
`24· · · · · · · Lack of foundation.
`
`25· · · · A.· · Normally use -- again, under the
`
`
`
`·1· · · · Nikolaos Papanikolopoulos, Ph.D.
`
`·2· ·assumption of the claim construction that I
`
`·3· ·was given, the answer is yes.
`
`·4· · · · Q.· · What about under your ordinary
`
`·5· ·usage of that term?
`
`·6· · · · · · · MR. BERKOWITZ:· Object to form
`
`·7· ·and lack of foundation.
`
`·8· · · · A.· · By using the claim construction
`
`·9· ·of pattern recognition, yes.
`
`10· · · · Q.· · I wasn't asking about the claim
`
`11· ·construction in this case.
`
`12· · · · · · · I was just asking about as you
`
`13· ·would normally use that term in your
`
`14· ·research, did the system of paragraph 4 use
`
`15· ·a pattern recognition algorithm?
`
`16· · · · · · · MR. BERKOWITZ:· Object to form.
`
`17· · · · · · · Lack of foundation.
`
`18· · · · A.· · Given the precision that I need
`
`19· ·to provide with respect -- since we have
`
`20· ·many terms which are used, I think I need
`
`21· ·to stick to use of the term in my
`
`22· ·declaration.· So in this case, yes.
`
`23· · · · Q.· · So you've used the term "pattern
`
`24· ·recognition" in some of your papers,
`
`25· ·haven't you?
`
`
`
`·1· · · · Nikolaos Papanikolopoulos, Ph.D.
`
`·2· ·part of the claim construction, so I'm
`
`·3· ·going to ask you the same clarification.
`
`·4· · · · Q.· · First, as you've construed the
`
`·5· ·claims in this case, did your license plate
`
`·6· ·system involve trained pattern recognition?
`
`·7· · · · · · · MR. BERKOWITZ:· Object to form
`
`·8· ·and foundation.
`
`·9· · · · A.· · So given the claim construction
`
`10· ·I have here, we didn't use trained pattern
`
`11· ·recognition means.
`
`12· · · · Q.· · And then as you used that term
`
`13· ·before these IPR matters, would you have
`
`14· ·said that that used trained pattern
`
`15· ·recognition?
`
`16· · · · · · · MR. BERKOWITZ:· Object to form
`
`17· ·and lack of foundation.
`
`18· · · · A.· · The term "trained pattern
`
`19· ·recognition" is not really common.· I mean,
`
`20· ·we use other terms, so I would say no.
`
`21· · · · Q.· · Okay.
`
`22· · · · · · · How did the system detect a
`
`23· ·license plate?
`
`24· · · · A.· · So I can give you my
`
`25· ·recollection.
`
`
`
`·1· · · · Nikolaos Papanikolopoulos, Ph.D.
`
`·2· ·perimeter of the license plate, so it was a
`
`·3· ·scale or a number, to I don't want to be --
`
`·4· ·but mainly images.
`
`·5· · · · Q.· · Okay.
`
`·6· · · · · · · And did you have on the system
`
`·7· ·any data or images relating to license
`
`·8· ·plates?
`
`·9· · · · A.· · I don't recall this, to be
`
`10· ·precise.
`
`11· · · · · · · One thing I recall is actually
`
`12· ·we used to have rectangles, or models of
`
`13· ·rectangles.
`
`14· · · · · · · I don't recall exactly if we had
`
`15· ·the license plates or -- and also, I'm not
`
`16· ·sure in Yue Du's thesis -- I asked him do
`
`17· ·this, but I'm not sure if this was ever
`
`18· ·implemented.
`
`19· · · · Q.· · So it's possible in a system of
`
`20· ·this type to have the system look for
`
`21· ·rectangles rather than specifically compare
`
`22· ·captured images to stored images of license
`
`23· ·plates?
`
`24· · · · · · · MR. BERKOWITZ:· Object to form.
`
`25· · · · · · · Foundation.
`
`
`
`NIKOLAOS PAPANIKOLOPOULOS, PH.D.
`TOYOEAMOTORvaAMS
`
`_
`
`,
`
`February 24, 2014
`48
`
`Nikolaos Papanikolopoulos, Ph.D.
`
`In a system like that, could you
`
`have used simulated data rather than
`
`images?
`
`MR. BERKOWITZ: Object to form.
`
`A.
`
`Under the assumption that
`
`different situations, different
`
`applications might require different ~—
`
`‘ it's a possibility.
`
`But for us, it was a
`
`requirement, a contractual requirement,
`
`to
`
`demonstrate something like this.
`
`o.
`
`Okay.
`
`MR. BERKOWITZ: Whenever it's a
`
`convenient time, we've been going around an
`
`hour and ten.
`
`BY MR. SCHARFF:.
`
`Q.
`
`Earlier I asked you about the
`
`claim constructions that you applied in
`
`your declaration, and you said that those
`
`were proyided to you.
`
`Were those provided to you by
`
`the attorneys for Toyota?
`
`A.
`
`Yes.
`
`From Kenyon & Kenyon,
`
`just
`
`
`
`1.
`
`2.
`
`3
`
`4
`
`5
`
`6
`
`7
`
`8
`
`9
`
`10
`
`11
`
`12
`
`13
`
`14
`
`15
`
`16
`
`17
`
`18
`
`19
`
`20
`
`21
`
`22
`
`23
`
`24
`
`25
`
`
`
`
`
`1
`
`to be ~~
`5i: ESQUIRE
`
`s o .. u 1
`
`. o r. s
`
`~ 800.211.0EPO (3376)
`
`EsquireSo/utions.oom
`
`
`
`NlKOLAOS PAPANIKOLOPOULOS, PHD.
`TOYOTAMOTORvsANS
`
`February 24, 2014
`89
`
`Nikolaos Papanikolopoulos, Ph.D.
`
`Q.
`
`So the claim doesn't just say "a
`
`trained pattern recognition algorithm
`generated from examples," does it?
`
`A.
`
`Q.
`
`It doesn't say.
`
`You understand then that Claim 1
`
`requires that the trained pattern
`
`recognition algorithm be generated from’
`
`data of possible exterior objects and
`
`patterns of received waves from the
`
`possible exterior objeCts; correct?
`
`A.
`
`Q.
`
`Yes.
`
`Not every trained pattern
`
`recognition algorithm would be generated
`
`using those specific requirements, would
`
`they?
`'
`
`I
`
`MR. BERKOWITZ: Object to form.
`
`Foundation.
`
`A.
`
`Can you clarify, because this is
`
`a little bit broad?
`broad.
`
`I would say it's too
`
`Q.
`
`What do you regard as the
`
`difference between just a trained pattern
`
`recognition algorithm in general, versus a
`
`trained pattern recognition algorithm
`
`\qumU'II-PUJNH
`
`H O
`
`11
`
`12
`
`13
`
`14
`
`15
`
`16
`
`17
`
`18
`
`19
`
`20
`
`21
`
`22
`
`23
`
`24
`
`25
`
`
`
`
`
`ESQUIRE
`
`5 o . u T .
`
`u N s
`
`800.211.DEPO (3376)
`
`EsquireSo/utions.com
`
`
`
`NIKOLAOS PAPANIKOLOPOULOS, PHD.
`TOYOTA MOTOR VS. AVS
`
`
`February 24, 2014
`90
`
`koooqmmithH
`
`l“o
`
`11
`
`12
`
`13
`
`14
`
`15
`
`16
`
`1'7
`
`18
`
`19
`
`20
`
`21
`
`22
`
`23
`
`24;
`
`25
`
`
`
`
`
`Nikolaos PapanikolopoulOs, Ph.D.
`
`generated from data of possible exterior
`
`objects and patterns of received waves from
`
`the possible exterior objects?
`
`MR. BERKOWITZ: Object to form.
`
`A.
`You see, again,
`I have to stick
`to the construction that was given, so I
`
`checked the trained pattern recognition
`
`algorithm, as you stated.
`
`Q.
`
`I'm sorry,
`
`that wasn't my
`
`question.
`
`Can you think of examples of a
`
`trained pattern recognition algorithm that
`
`are not generated from data of possible
`
`exterior objects and patterns of received
`
`waves from the possible exterior objects.
`
`MR. BERKOWITZ: Object to form.
`
`MR. SCHARFF:
`
`I'll restate.
`
`BY MR.. SCHARFF:
`
`Q.
`
`So can you think of an example
`
`of a trained pattern recognition algorithm
`
`that is not generated from data of possible
`
`exterior objects and patterns of received
`
`waves from the possible exterior objects?
`
`A.
`
`With respect to the specific
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`r” ESQUIRE
`
`s o . u T . a n s
`
`800.21 1.05130 (3376)
`EsquireSo/utions.com
`
`
`
`NMOLAOSPAPANMOLOPOULOSJWiD.
`TOYOTA MOTOR vs. AVS
`
`February 24., 2014
`91
`
`Nikolaos Papanikolopoulos, Ph.D.
`
`claim construction that I had to examine,
`
`no.
`
`Q.
`
`I'm sorry, it was an open—ended
`
`question.
`
`So you can't think of any
`
`examples of a trained pattern recognition
`
`algorithm that are not generated from data
`
`of possible exterior objects and patterns
`of received waves from the possible
`
`exterior objects?
`
`A.
`
`Again, my answer is with respect
`
`to this particular claim construction that
`
`I had to examine,
`
`I had to look at the
`
`whole claim.
`
`The claim construction I was
`
`given,
`
`I had to offer an opinion.
`
`Q.
`
`You have a claim construction
`
`for trained pattern recognition algorithm,
`
`but I'm asking you about the next phrase.
`
`A.
`
`Q.
`
`And this is the one -*
`
`Did you ignore, for purposes of
`
`your opinions,
`
`the phrase "generated from
`
`data of possible exterior objects and
`
`patterns of received waves from the
`
`possible exterior objects"?
`
`10
`
`11
`
`12
`
`13
`
`14
`
`15
`
`16
`
`17
`
`18
`
`19
`
`20
`
`21
`
`22
`
`23
`
`24
`
`25
`
`
`
`
`
`
`
`5%”? ‘ ESQUIRE
`
`
`s o L u T
`
`. a r. s
`
`800.211.DEPO (3376)
`
`EsquireSo/utions.com
`
`
`
`LOUJQG‘XU'II‘DLMNH
`HHHHHHmswmHo
`
`16
`
`17
`
`18
`
`19
`
`20
`
`21
`
`22
`
`23
`
`24
`
`25
`
`IWKOLAOSPAPANMOLOPOULOS,HiD.
`TOYOTAMOTORvaANS
`
`February 24, 2014
`92
`
`Nikolaos Papanikolopoulos, Ph.D.
`
`A.
`
`No.
`
`It is included in my
`
`declaration. That is a statement from
`
`Lemelson which addresses this.
`
`So if you look at my declaration
`
`~~ this is the '000 —— you want me to do
`
`the '000?
`
`Q.
`
`The '057. We're on the '057
`
`patent right now.
`
`The IPR.
`
`A.
`
`Q.
`
`Which one do you want?
`
`For the '057 patent.
`
`In your opinion, Lemelson
`
`
`
`discloses:
`
`"A trained pattern recognition
`
`algorithm generated from data of possible
`
`exterior objects and patterns of received
`
`waves from the possible exterior objects."
`
`Is that right?
`
`A.
`
`What I am saying is actually
`
`Lemelson explicitly teaches ~~ explicitly
`
`discloses the teaching based on data of
`
`possible exterior objects, and I have a
`
`statement to that effect.
`
`So I have to look at the claim
`
`construction that was given and the
`
`particular claims in order to ~—
`
` fig ESQUIRE
`“’
`s r.
`,. u T
`. o N s
`
`800.211.DEPO (3376)
`Esqu1reSo/ut10ns.oom
`
`
`
`NIKOLAOS PAPANIKOLOPOULOS, PHD.
`TOYOIAMOTORvaANS
`
`February 24, 2014
`93
`
`
`
`Nikolaos Papanikolopoulos, Ph.D.
`
`Q.
`
`So I'm asking you, what type of
`
`trained pattern recognition algorithm is
`
`not generated from data of possible
`
`exterior objects and patterns of received
`
`waves from the possible exterior objects?
`
`MR. BERKQWITZ: Objection.
`
`Foundation.
`
`
`
`A.
`
`My reply to this is that I was
`
`not asked this.
`
`I was asked to examine *—
`
`Q.
`
`I understand.
`
`I think that's the difficultly
`
`with you responding to my questions.
`
`I'm allowed to ask you these
`
`questions, and I understand you weren't
`
`previously asked this specific type of
`
`question, but I'm asking this now, and I'm
`
`asking what your answer is.
`
`And you can't just say:
`
`I don't
`
`have an answer because I wasn't asked this
`
`previously or because it wasn't this
`
`specific question that you addressed with
`
`respect to Lemelson.
`
`
`
`
`ommqmmswml—I
`
`23
`
`24
`
`25
`
`slaw
`
`My question is:
`
`
`ESQUIRE
`800.21 1.05130 (3376)
`
`s o L u T . o u s
`
`EsquireSo/utions.com
`
`
`
`NIKOLAOS PAPANIKOLOPOULOS, PHD.
`TOYOTA MOTOR VS. AVS
`
`February 24, 2014
`
` Nikolaos Papanikolopoulos, Ph.D.'
`
`Are there trained pattern
`
`recognition algorithms that are not
`
`generated from data of possible exterior
`
`objects and patterns of received waves from
`the possible exterior objects?
`
`MR. BERKOWITZ: Object to form;
`
`Foundation.
`
`A.
`
`You see, it would be highly
`
`speculative on my part to enumerate cases
`
`like this when I have —— my declaration is
`
`very specific.
`
`Q.
`
`I'm sorry, but you do have to
`
`answer the question.
`
`MR. BERKOWITZ: Object as
`
`argumentative.
`
`I think he is answering the
`
`question.
`
`MR. SCHARFF: No, he isn't.
`
`He's saying he's refusing to
`
`
`
`answer it because it wasn't part of the
`
`specific analysis he did from Lemelson.
`MR. BERKOWITZ:
`I disagree with
`
`all that.
`
`MR. SCHARFF: And, by the way,
`
`that isn't an allowed objection under the
`
`
`
`omm4mmswwH<
`
`15
`
`l6
`
`l7
`
`18
`
`19
`
`20
`
`21
`
`22
`
`23
`
`24:
`
`25
`
`
`, ESQUIRE
`
`s o .. u T . a u s
`
`'
`
`800.211.DEPO (3376)
`
`EsquireSo/utions.com
`
`
`
`NIKOLAOS PAPANIKOLOPOULOS, PHD.
`TOYOTAMOTORvsAVS
`
`February 24,- 2014
`95
`
`Nikolaos Papanikolopoulos, Ph.D.
`
`rules.
`
`MR. BERKOWITZ:
`
`If you start to
`
`argue with my witness,
`
`I can make that
`
`objection.
`
`MR. SCHARFF:
`
`It‘s not in the ~~
`
`MR. BERKOWITZ:
`
`In the Appendix?
`
`I disagree with that.
`
`Can you read the question back.
`
`(Requested portion of record
`
`read.)
`
`BY MR. SCHARFF:
`
`Q.
`
`That should be a simple
`
`yes—or—no answer.
`
`Are there or aren't there?
`
`MR. BERKOWITZ: Object to form.
`
`A.
`
`So let me state, page ~~
`
`paragraph 55 of my '057,
`
`this is from
`
`column 8,
`
`row I through 8.
`
`"Lemelson explains that neural
`
`networks used in the vehicle warning system
`
`are trained to recognize roadway hazards
`
`which the vehicle is approaching,
`
`including
`
`automobiles,
`
`trucks and pedestrians.
`
`"Training involves providing
`
`DUN
`
`\OCOQOWUTFP
`
`10
`
`ll
`
`12
`
`13
`
`14
`
`15
`
`16
`
`17
`
`18
`
`:9
`
`20
`
`21
`
`22
`
`23
`
`24
`
`25
`
`
`
`
`9‘37 ESQUIRE
`
`
`so.u1.u"s
`
`‘
`
`800.211.DEPO (3376)
`
`EmeeSoMMwmtom
`
`
`
`BHKOLAOSFVUWVMKOLOPOULOS,PHll
`February 24, 2014
`96
`TOYOTA MOTOR VS. AVS
`
`j
`
`
`
`Nikolaos Papanikolopdulos, Ph.D.
`
`known inputs to the network resulting in
`
`desired output responses.
`
`"The weights are automatically
`
`adjusted based on error signal measurements
`
`until the desired outputs are generated.
`
`"Various learning algorithms may
`
`be applied."
`
`And if you look at figure 5 of
`
`Lemelson, basically makes very clear that
`
`the training is done ~- explicitly
`
`disclosed training based on basically data.
`
`Q.
`
`That was wasn't my question.
`
`I
`
`wasn‘t asking you what Lemelson discloses.
`
`I wasn't asking you what does satisfy
`
`generating a trained pattern recognition.
`
`I was asking you specifically:
`
`Are there ways to train a
`
`pattern recognition algorithm that don't
`
`involve generating from data of possible
`
`exterior objects and patterns of received
`
`waves from the possible exterior objects?
`
`Just based on your knowledge,
`
`your experience, your research.
`
`MR. BERKOWITZ: Object to form.
`
`15
`
`16
`
`17
`
`18
`
`19
`
`20
`
`21
`
`22
`
`23
`
`24
`
`25
`
`
`
`
`
` ESQUIRE
`
`s o .. u T
`
`. o r. s
`
`800.21 1.DEPO (3376)
`EsquireSo/utions.com
`
`
`
`NHfilAOSPAPAMKOLOPOULOSJWiD.
`TOYOTAMOTORVSAVS
`
`February 24, 2014
`97
`
`
`
`Nikolaos Papanikolopoulos, Ph.D.
`
`
`
`WM
`
`koooqmmhb
`
`10
`
`11
`
`12
`
`13
`
`14
`
`15
`
`16
`
`17
`
`18
`
`19
`
`20
`
`21
`
`22
`
`23'
`
`24
`
`25
`
`A.
`
`Again, it will be highly
`
`speculative because it depends on the
`
`application.
`
`And I will mention, if you go to
`
`my bio, you are going to see I have a paper
`
`about using trainable ~— trained pattern
`
`recognition algorithms to recognize vehicle
`
`occupants.
`
`I think it's journal paper 50
`
`or 51.
`
`The training there involve
`
`images and patterns, so given the analysis,
`
`I cannot think in the context of other
`
`examples.
`
`Q.
`
`Okay.
`
`So you're unable to answer the
`
`question? You don't know whether or not
`
`there are ways that you can train a pattern
`
`recognition algorithm that don't involve
`
`generating it from data of possible
`
`exterior objects and patterns of received
`
`waves from possible exterior objects?
`
`MR. BERKOWITZ: Object to form.
`
`A.
`
`' You see, given the assumption
`
`that we're dealing with a particular
`
`
` 5” J ESQUIRE
`
`'
`
`s
`
`r.-
`
`.v u T . o N s
`
`800.211.DEPO (3376)
`
`EsquireSo/utions.com
`
`
`
`MKOLAOSPAPAMKOLOPOULOSJWiD
`TOYOEAMOTORVSAVS
`
`February 24, 2014
`102
`
`Nikolaos Papanikolopoulos, Ph.D.
`
`But if you have access to real
`
`data, you‘re going to have useless
`
`information in—useless information out.
`
`Q.
`
`Is it technologically possible
`
`to use simulated data for a pattern
`
`recognition system for training that system
`
`for detecting automobiles?
`
`MR. BERKOWITZ: Object to form.
`
`A.
`
`If you look at my declaration,
`
`I
`
`mention several systems in this domain.
`
`In this particular domain, you
`
`go to simulated data, or if you don't have
`
`access to real data,
`
`to real images.
`
`So the fact that someone might
`
`go that way will be highly speculative
`
`because ——
`
`Q.
`
`I understand it may not make
`
`sense to you, but is it technologically
`
`possible to use simulated data ——
`
`
`
`A.
`
`Q.
`
`Now? Then?
`
`Now.
`
`Is it possible today to use
`
`simulated data to train a pattern
`
`recognition algorithm to detect an
`
`ES QUIRE
`
`s o L u T
`
`. o u s
`
`800.21 1.DEPO (3376)
`
`EsquireSo/utions.com
`
`DJN
`
`qmmpb
`
`10
`
`11‘
`
`12
`
`13
`
`14
`
`15
`
`16
`
`17
`
`18
`
`19
`
`20
`
`21
`
`22
`
`23
`
`24
`
`25
`
`
`
`NIKOLAOS PAPANIKOLOPOULOS, PHD.
`TOYOTA MOTOR VS. AVS
`
`February 24, 2014
`103
`
`Nikolaos Papanikolopoulos, Ph.D.
`
`automobile?
`
`A.
`
`It's one of the questions that
`
`is not a yes or no, but sometimes you can
`
`use very advanced simulated data that are
`
`really as close. But also you have very
`
`inexpensive means to access the real data,
`
`the real images, so I don't see any —~
`
`Q.
`
`I'm just asking if it's
`
`technologically possible. Not whether or
`
`not that would be the way that you would do
`
`it.
`
`A.
`
`I suppose.
`
`But you don't want useless stuff
`
`in~useless stuff out.
`
`Q.
`
`Could training a pattern
`
`recognition algorithm with simulated data
`
`involve less data that needs to be stored
`
`on the system?
`
`MR. BERKOWITZ: Object to form.
`
`Q.
`
`As compared to training with
`
`images, for example, real life images.
`
`MR. BERKOWITZ:
`
`Same objection.
`
`
`
`LoooqmmiwaI—l
`mHHHHHHHHHHOKDODQG‘AU'ipD-UJNHO
`
`21
`
`22
`
`23
`
`24:
`
`25
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`A.
`‘You use different —— with the
`
` fig; ESQUIRE
`800.211.DEPO (3376)
`
`assumption that you can use different
`
`s o . u T
`
`. a r. s
`
`EsquireSo/uz‘ions.com
`
`
`
`·1· · · · Nikolaos Papanikolopoulos, Ph.D.
`
`·2· ·obstructions of images -- in other words,
`
`·3· ·you can use edge images, boundary images,
`
`·4· ·the answer is probably yes.
`
`·5· · · · · · · But, again, it goes back to my
`
`·6· ·earlier comment.· You don't want to have
`
`·7· ·useless stuff in or noise in, because it
`
`·8· ·will be noise out.
`
`·9· · · · Q.· · Back in 1995, would it have been
`
`10· ·technologically possible to train a pattern
`
`11· ·recognition algorithm for detecting
`
`12· ·vehicles using simulated data?
`
`13· · · · A.· · Simulated data at that time,
`
`14· ·technologically possible is a -- let me
`
`15· ·explain to you.
`
`16· · · · · · · If you use a silicon graphics
`
`17· ·machine processor, the data that could be
`
`18· ·produced has nothing to do with what the
`
`19· ·real image would have been.
`
`20· · · · · · · So I can speculate that anyone
`
`21· ·can produce collections of 0s and 1s, but
`
`22· ·if they had any relevance with reality and
`
`23· ·the task is a different story.
`
`24· · · · Q.· · For example, back in 1995, was
`
`25· ·it technologically possible to just train a
`
`
`
`·1· · · · Nikolaos Papanikolopoulos, Ph.D.
`
`·2· · · · Q.· · Okay.
`
`·3· · · · · · · Would it have been possible in
`
`·4· ·1995 to use a trained pattern recognition
`
`·5· ·system to detect something other than a
`
`·6· ·vehicle and assume that a vehicle was
`
`·7· ·detected; for example, taillights?
`
`·8· · · · · · · MR. BERKOWITZ:· Object to form.
`
`·9· · · · A.· · Simulated data or real data?
`
`10· · · · Q.· · Either simulated or real.
`
`11· · · · A.· · We have some of the references
`
`12· ·here discussing this, and I think the --
`
`13· ·it's an easier problem than -- the overall
`
`14· ·problem that --
`
`15· · · · Q.· · So in 1995, it would have been
`
`16· ·possible to train a pattern recognition
`
`17· ·algorithm to detect taillights, and from
`
`18· ·that, assume that a vehicle was present?
`
`19· · · · A.· · Can you repeat the question?
`
`20· · · · Q.· · Sure.· In 1995, it was possible
`
`21· ·to train a pattern recognition algorithm to
`
`22· ·detect taillights; and then, from that,
`
`23· ·assume a vehicle was present?
`
`24· · · · A.· · You're asking by saying -- it's
`
`25· ·very easy to confuse -- in cases like this,
`
`
`
`NIKOLAOS PAPANIKOLOPOULOS, PHD.
`TOYOTA MOTOR VS. AVS
`
`February 24, 2014
`123
`
`
`
`KDOOQOWU‘IHBUJNH
`HHHHHHHHqmmamNHo
`
`18
`
`19
`
`20
`
`21
`
`22
`
`23
`
`24
`
`25
`
`Nikolaos Papanikolopoulos, Ph.D.
`
`less storage space requirements than using
`
`digital image data for vehicle detection?
`
`A.
`
`To me, it makes no sense to use
`
`simulated data in a scenario like this.
`
`Images —~ and the assumption is
`
`
`
`actually,
`
`I'm talking about
`
`'95 —~
`
`the
`
`simulation couldn't have reached the level
`
`of complexity that a picture could provide.
`And if you are a little bit more,
`
`careful with the management of the data and
`
`the processing and the storage of the data,
`
`this was, as it has been demonstrated by
`
`some of the papers that I'm including in my
`
`writeup, you can do it, so I don't see the
`
`reason why you would go to simulated data.
`
`Now, if you take a raw image and
`
`you just save it...
`
`Q.
`
`Could in 1995 simulated data
`
`involve smaller file sizes than digital
`
`image data?
`
`A.
`
`Potentially, yes.
`But, again,
`I don't see -— I
`
`don't see the usefulness.
`
`Again, why do we need to have
`
`
`
`
`ESQUIRE
`
`s o L u T
`
`. a u s
`
`800.211.DEPO (3376)
`
`EsquireSo/utions.com
`
`
`
`·1· · · · Nikolaos Papanikolopoulos, Ph.D.
`
`·2· · · · A.· · So I'm on page 45, above
`
`·3· ·paragraph 110, the Figure 1, that
`
`·4· ·basically, the camera 12 -- to me, this is
`
`·5· ·a disclosure.
`
`·6· · · · Q.· · Again, I wasn't asking about
`
`·7· ·Borcherts.
`
`·8· · · · · · · In your opinion, in a
`
`·9· ·hypothetical prior art reference that
`
`10· ·discloses a receiver that's arranged near
`
`11· ·the rearview mirror, but is definitely not
`
`12· ·on the rearview mirror, is that good
`
`13· ·enough, in your opinion, to satisfy the
`
`14· ·claim requirement in Claim 30 of "at least
`
`15· ·one receiver arranged on a rearview mirror
`
`16· ·of the vehicle"?
`
`17· · · · A.· · Even the claim construction --
`
`18· ·yes.
`
`19· · · · Q.· · So in your opinion, near the
`
`20· ·rearview mirror, that's good enough to
`
`21· ·count as on a rearview mirror?
`
`22· · · · · · · MR. BERKOWITZ:· Object to form.
`
`23· · · · A.· · For someone with ordinary skill
`
`24· ·in the art, I think this will be
`
`25· ·sufficient.
`
`
`
`NIKOLAOS PAPANIKOLOPOULOS, PHD.
`TOYOTA MOTOR VS. AVS
`
`February 24, 2014
`162
`
`
`
`KOCOQGNU‘II‘P-UJNI-J
`mNNHHHHHHHHHHmHommqmmswMHo
`
`Nikolaos Papanikolopoulos, Ph.D.
`
`Q.
`
`A.
`
`Q.
`
`Yes,
`
`'057.
`
`Yes.
`
`So,
`
`in particular, you state
`
`that:
`
`"Lemelson discloses a processor that
`
`processes the signals derived from the
`
`camera image(s) and provides a
`
`classification,
`
`identification or location
`
`of the exterior object.
`"In particular, Lemelson teaches
`
`that this can be accomplished using a
`
`neural network. Specifically,
`
`the signal
`
`output from the camera(s)
`
`is digitized and
`
`passed to an image analyzing computer
`
`(which, as set forth in greater detail
`
`below, meets the 'trained pattern
`
`recognition means'
`
`limitation of Claims 1
`
`and 56)."
`
`So,
`
`in addition, as we talked
`
`about earlier, Claim 1 of the '057 patent,
`
`for example, requires, among other things,
`
`a trained patent recognition algorithm;
`
`23~
`
`right?
`
`24
`
`25
`
`A.
`
`Q.
`
`Yes.
`
`So let's go to page 55 of your
`
`
`
`'J‘V‘
`
` ESQUIRE
`
`s a .. u 7
`
`, o r. s
`
`V
`
`800.211.DEPO (3376)
`
`EsquireSo/utions.com
`
`
`
`NIKOLAOS PAPANIKOLOPOULOS, PHD.
`TOYOTA MOTOR VS. AVS
`
`February 24, 2014
`163
`
`Nikolaos Papanikolopoulos, Ph.D.
`
`declaration.
`
`A.
`
`Q.
`
`Okay.
`
`There you state that:
`
`
`
`"Lemelson explains that neural
`
`networks used in the vehicle warning system
`
`are trained to recognize roadway hazards
`
`which the vehicle is approaching,
`
`including
`
`automobiles,
`
`trucks and pedestrians.
`
`"Training involves providing
`
`known inputs to the network resulting in
`
`desired output responses."
`
`Now,
`
`in your declaration —4 so
`
`then,
`
`in your opinion —~ where Lemelson
`
`describes "providing known inputs," that's
`
`the disclosure that you point to in
`
`Lemelson regarding training with the neural
`
`network;
`
`is that right?
`
`MR. BERKOWITZ: Object to form.
`
`It's one of the places.
`
`Q.
`
`Where else in Lemelson in your
`
`opinion does it discuss training of the
`
`patent recognition algorithm?
`
`Let me give you Lemelson.
`
`I
`
`don't believe it's already in evidence.
`
`g?”4», M
` ESQUIRE
`
`s o L u 1
`
`.
`
`.3 u s
`
`800.211.0590 (3376)
`
`EsquireSo/utions.com
`
`
`
`N IKOLAOS PAPANIKOLOPOULOS, PHD.
`February 24, 2014
`164
`TOYOTAMOTORvaAVS
`
`
`
`Nikolaos Papanikolopoulos, Ph.D.
`
`(Toyota Exhibit 1002 for
`
`identification in IPR 419, Lemelson
`
`document.)
`
`BY MR. SCHARFF:
`
`Q.
`
`-
`
`I show you what Toyota has
`
`marked as 1002 in the 419 IPR.
`
`Do you recognize this document?
`
`.Yes.
`
`It is the Lemelson.
`
`Other than the sentence that you
`
`A.
`
`Q.
`
`quote in your declaration that states
`
`“training involves providing known inputs
`
`to the network resulting in desired output
`
`responses," what else in Lemelson in your
`
`opinion relates to training of the pattern
`
`recognition algorithm?
`
`A.
`
`I think, first of all, we have
`
`to mention Figure 1.
`
`I think I copied my declaration,
`
`because it shows a structure and how —— for
`
`example,
`
`the processor means in 19. And
`
`then I want to mention Figure 2.
`
`Q.
`
`Sure.
`
`Before we move on from Figure 1,
`
`there's nothing in Figure 1 that
`
`
`
`LIJN
`
`4mms
`
`(I)
`
`10
`
`ll
`
`12
`
`13
`
`14
`
`15
`
`16
`
`17
`
`18
`
`19
`
`20
`
`21
`
`22
`
`23
`
`24
`
`25
`
`#7“
`
`
`i‘x ESQUIRE
`
`5 o . u T
`
`. o r. s
`
`
`
`800.211.DEPO (3376)
`
`EsquireSo/utions.com
`
`
`
`NIKOLAOS PAPANIKOLOPOULOS, PHD.
`TOYOTA MOTOR VS. AVS
`
`February 24, 2014
`165
`
`ODQONU'IeP-UJNH
`
`k0
`
`10
`
`11
`
`12
`
`13
`
`14
`
`15
`
`16
`
`17
`
`18
`
`19
`
`20
`
`21
`
`22
`
`23
`
`24
`
`25
`
`
`
`Nikolaos Papanikolopoulos, Ph.D.
`
`specifically refers to training,
`
`is there?
`
`A.
`
`If you look at image analyzing
`
`computer, and the way that is discussed in
`
`the —— let me find the exact paragraph
`
`somewhere.
`
`If you look at the description
`
`of the drawing, Figure 1 says it's a
`
`"diagram of the overall model, basically
`
`warning and control system,
`
`illustrating
`
`system sensors, computer displays,
`
`inputeoutput device and other key
`
`elements."
`
`So we go to ~-
`
`Q.
`
`Before we move on,
`
`that sentence
`
`that you just read to me ~—
`
`MR. BERKOWITZ: He‘s got to be
`
`able to finish here.
`
`MR. SCHARFF: Well, if he's
`
`going to read ten different sentences, and
`
`then I have to go back to them individually
`
`and ask him questions about that —~
`
`MR. BERKOWITZ: You could do
`
`that at the end.
`
`And referring to the Appendix
`
`I am required to
`that you mentioned before,
`
`
`
`
`
`* ESQUIRE I
`
`s a L u T
`
`. a N s
`
`'
`
`800.211.DEPO (3376)
`
`EsquireSo/utions.com
`
`
`
`February 24, 2014
`NIKOLAOS PAPANIKOLOPOULOS, PHD.
`166
`TOYOTA MOTOR vs. AVS
`
`
`1
`
`2
`
`3
`
`4
`
`5
`
`6
`
`7
`
`8
`
`9
`
`10
`
`11‘
`
`12
`
`13
`
`14
`
`15
`
`16
`
`17
`
`18
`
`19
`
`20
`
`21
`
`22
`
`23
`
`24
`
`
`
`Nikolaos Papanikolopoulos, Ph.D.
`
`object
`
`to opposing parties' conduct, so I
`
`do object to the fact that you're cutting
`
`him off from his answers.
`
`MR. SCHARFF:
`
`I let him complete
`
`his sentence. Before he moves on to
`
`another sentence,
`
`I can ask if we can ask
`
`questions about that sentence.
`
`Again,
`
`this isn't an allowable
`
`objection. You're coaching the witness.
`
`MR. BERKOWITZ:
`
`I am actually
`
`required to make that objection.
`
`So let the record reflect that
`
`counsel is cutting the witness off.
`
`MR. SCHARFF: And, again,
`
`let
`
`the record reflect I didn't cut him off in
`
`the middle of a sentence.
`
`MR. BERKOWITZ:
`
`In the middle of
`
`an answer.
`
`BY MR. SCHARFF:
`
`Q.
`
`Let‘s take your support for the
`
`pattern recognition training one item at a
`
`time.
`
`We talked previously about the
`
`. 25
`
`training with known inputs sentence in
`
`
`
`ESQUIRE
`
`s a . u T
`
`.
`
`o N s
`
`800.211.DEPO (3376)
`
`EsquireSo/utions.com
`
`
`
`mmqmmewmr—I
`
`1..) O
`
`ll
`
`12
`
`13
`
`14
`
`15
`
`16
`
`17
`
`18
`
`19
`
`20
`
`21
`
`22
`
`23
`
`24
`
`25
`
`NIKOLAOS PAPANIKOLOPOULOS, PHD.
`
`TOYOTA MOTOR VS. AVS
`
`February 24, 2014
`167
`
`
`
`Nikolaos Papanikolopoulos, Ph.D.
`
`Lemelson that you point to in your
`
`declaration.
`
`What's the next instance?
`
`A.
`
`With all due respect, may I
`
`explain a little bit the thought process
`
`here?
`
`Q.
`A.
`
`Okay.
`There are issues of text and
`
`figures that are interconnected, so if you
`
`let me mention the figures and the texts,
`
`then I can answer the questions here and
`
`the logic that I followed here.
`
`Q.
`
`That's fine.
`
`If that's what you
`
`need to do, go ahead.
`
`I was just trying to
`
`make it a little more concise.
`
`A.
`
`So the ones I consider is
`
`actually Figure l and its expected
`
`description; Figure 2 and Figure 5.
`
`And also, as part of this,
`
`the
`
`Figure 3 and 4.
`
`I go to the ~- so column 4,
`
`where it has descriptions of the different
`
`figures. And then I'm using 515 down to
`
`55. And then I'm using 81, and I think I
`
`
`5” ESQUIRE
`
`s o . u T . u r: s
`
`800.211.DEPO (3376)
`EsquireSo/utions.com
`
`
`
`NIKOLAOS PAPANIKOLOPOULOS, PHD.
`TOYOTA MOTOR vs. AVS
`
`..
`
`February 24, 2014
`168
`
`1
`
`2
`
`3
`
`4
`
`5
`
`6
`
`7
`
`8
`
`9
`
`10
`
`11
`
`12
`
`13
`
`14
`15
`
`16
`
`17
`
`18
`
`19
`
`20
`
`21
`
`22
`
`23
`
`24
`
`25
`
`
`
`Nikolaos Papanikolopoulos, Ph.D.
`
`stopped at row 8.
`
`The idea here is actually trying
`
`to understand how the components are put
`
`together, and also how the different claim
`
`construction in terms of what was given to
`
`me map this combination of text and
`
`figures.
`
`So if you see the column 8 text,
`
`it says some other interesting things that
`
`teach the training based on data of
`
`possible exterior objects.
`
`It says, for example:
`
`"The neural network of the image
`,analysis computer 19" ~— so that's why 1
`
`want to use the Figure 1
`
`~— "provides a
`
`highly parallel image processing structure
`
`with rapid, real~time image recognition
`
`necessary for the Motor Vehicle Warning and
`
`Control System."
`
`"Very Large Scale Integrated
`
`(VLSI) Circuit implementation of the neural
`
`processing elements permits low—cost,
`
`low—weight limitation."
`
`And I want to emphasize this:
`
`.7; ESQUIRE
` w:
`
`s
`
`r:
`
`L u w ‘ a r: s
`
`800.211.0sPo (3376)
`
`EsquireSo/utions.com
`
`
`
`NIKOLAOS PAPANIKOLOPOULOS, PHD.
`Febnmwy24,2014
`TOYOTA MOTOR VS. AVS
`169
`
`
`
`
`[\3
`
`DJ
`
`\1CH.U'IEPA
`
`(I)
`
`10
`
`11
`
`12
`
`13'
`
`14
`
`15
`
`16
`
`17
`
`18
`
`19
`
`20
`
`21
`
`22
`
`23
`
`24
`
`25
`
`Nikolaos Papanikolopoulos, Ph.D.
`
`"Also, a neural network has
`certain reliability advantages important in
`
`a safety warning system.
`
`Loss of one
`
`processing element does not necessarily
`
`result in a processing system failure."
`
`In other words, if one of the
`
`elements goes off for whatever reason,
`
`the
`
`system is able to learn online and adopt,
`
`so all these are elements that I took into
`
`consideration.
`
`Now, one important question that
`
`I had to face as part of this is actually
`
`the data and patterns are actually used.
`
`This is for the training. This
`
`is the Figure 5, because it shows an image
`
`processer that receives image data, and an
`
`image data bus that connects the virtual
`
`processing elements,
`
`so all of these were a
`
`part of this.
`
`So I don‘t know if this has been
`
`helpful.
`
`Q.
`
`A.
`
`Are you done with your answer?
`
`One last thing.
`
`Another thing that I had to
`
`
`fig: ESQUIRE
`
`
`s o . u T . o u s
`
`800.211.DEPO (3376)
`
`EsquireSo/utions.com
`
`
`
`NIKOLAOS PAPANIKOLOPOULOS, PH.D.
`TOYOTA MOTOR vs. AVS
`
`February 24, 2014
`170
`
`Nikolaos Papanikolopoulos, Ph.D.
`consider as part of the claim was actually
`
`the idea of patterns. And for me, patterns
`
`are actually relationships of pixels.
`
`Q.
`
`Okay.
`
`So let's go to your
`
`declaration.
`
`In your 419 declaration,
`
`paragraphs 47 to 64 relates to anticipation
`
`by Lemelson;
`
`is that right?
`
`A.
`
`Q.
`
`Yes.
`
`And out of those paragraphs in
`
`your declaration,
`
`the only spot where you
`
`refer to "training" is in paragraph 55,
`
`when you quote to where Lemelson says:
`
`”Training involves providing
`
`known inputs to the network resulting in
`
`desired output responses."
`
`MR. BERKOWITZ: Object to form.
`
`Foundation.
`
`A.
`
`There is also various learning
`
`elements may be applied and also the
`
`weights are automatically adjusted.
`
`Q.
`
`I'm talking specifically where
`
`you use the word "training."
`
`That only appears in paragraph
`
`UJN
`
`4mma
`
`10
`
`11
`
`12
`
`13
`
`14c
`
`15
`
`16
`
`17
`
`18
`
`19
`
`20
`
`21
`
`22
`
`23
`
`24:
`
`25
`
`
`
`
`
`5W ESQUIRE
`
`
`s c:
`
`L u T
`
`. o n s
`
`800.211.0EPO (3376)
`
`EsquireSo/utions.com
`
`
`
`NlKOLAOS PAPANIKOLOPOULOS, PHD.
`TOYOTA MOTOR VS. AVS
`
`February 24, 2014
`17.1
`
`Nikolaos Papanikolopoulos, Ph.D.
`
`55 in the sentence:
`
`"Training involves
`
`providing known inputs to the network
`
`resulting in desired output responses."
`
`Is that correct?
`
`A.
`
`To me, we need to look at
`
`training as part of the learning in the
`
`adjustments of weights, given the structure
`
`that I have here.
`
`Q.
`
`So, again,
`
`the only place
`
`between paragraphs 47 and 64 where you use
`
`the word "training" is in paragraph 55;
`
`is
`
`that correct?
`
`MR. BERKOWITZ: Object to form
`
`and foundation.
`
`
`
`A.
`
`These are components of the
`
`training.
`
`I cannot
`
`ignore this.
`
`I gave
`
`you the line of thought I followed in order
`
`to form an opinion.
`
`Q.
`
`Sure.
`
`I get that. But my
`
`question is:
`
`Is the only instance in which
`
`you use the word "training" in paragraphs
`
`47 to 64 in parag