`v.
`American Vehicular Sciences (“AVS”), Patent Owner
`
`Thomas J. Wimbiscus, Lead Counsel
`Scott P. McBride, Backup Counsel
`Christopher M. Scharff, Backup Counsel
`
`1
`
`
`
`Before Jameson Lee, Trevor M. Jefferson, and Barbara A.
`Parvis, Administrative Patent Judges
`
`No. IPR2013-00424
`Patent No. 5,845,000
`
`CORRECTED AVS EXHIBIT 2006
`Toyota Inc. v. American Vehicular Sciences LLC
`IPR2013-00424
`
`
`
`Grounds for Institution
`
`Trial Instituted on Claims:
` 10, 11, 16, 17, 19, 20, 23
`
`• Claims 10, 11, 16, 17, 19, 20, 23 for anticipation by Lemelson
`
`• Claims 10, 11, 19, 23 for obviousness by Lemelson and Asayama
`
`• Claims 16, 17, 20 for obviousness by Yanagawa and Lemelson
`
`2
`
`CORRECTED AVS EXHIBIT 2006
`Toyota Inc. v. American Vehicular Sciences LLC
`IPR2013-00424
`
`
`
`Grounds for Institution by Type
`
`
`• All reviewed claims for anticipation by Lemelson
`
`o All reviewed claims require “generating a pattern recognition algorithm” or a
`“pattern recognition algorithm generated” “from data of possible exterior
`objects and patterns of received electromagnetic illumination from the
`possible exterior objects”
`
`
`
`• Claims 10, 11, 19, 23 for obviousness by Lemelson and Asayama
`
`o Claims require “transmitting electromagnetic waves”
`
`
`
`• Claims 16, 17, 20 for obviousness by Lemelson and Asayama
`
`o Claims require “output means . . . for dimming the headlights in said vehicle”
`3
`
`
`
`CORRECTED AVS EXHIBIT 2006
`Toyota Inc. v. American Vehicular Sciences LLC
`IPR2013-00424
`
`
`
`Independent Claim 10
`
`10. In a motor vehicle having an interior and an exterior, a monitoring system for monitoring at
`least one object exterior to said vehicle comprising: a) transmitter means for transmitting
`electromagnetic waves to illuminate the at least one exterior object;
`
`b) reception means for receiving reflected electromagnetic illumination from the at least
`one exterior object;
`
`c) processor means coupled to said reception means for processing said received
`illumination and creating an electronic signal characteristic of said exterior object based
`thereon;
`
`d) categorization means coupled to said processor means for categorizing said electronic
`signal to identify said exterior object, said categorization means comprising trained
`pattern recognition means for processing said electronic signal based on said received
`illumination from said exterior object to provide an identification of said exterior object
`based thereon, said pattern recognition means being structured and arranged to apply a
`pattern recognition algorithm generated from data of possible exterior objects and
`patterns of received electromagnetic illumination from the possible exterior objects; and
`
`e) output means coupled to said categorization means for affecting another system in the
`vehicle in response to the identification of said exterior object.
`
`4
`
`CORRECTED AVS EXHIBIT 2006
`Toyota Inc. v. American Vehicular Sciences LLC
`IPR2013-00424
`
`
`
`Independent Claim 16
`
`16. In a motor vehicle having an interior and an exterior, an automatic headlight dimming system
`comprising:
`
`a) reception means for receiving electromagnetic radiation from the exterior of the vehicle;
`
`b) processor means coupled to said reception means for processing the received radiation
`and creating an electronic signal characteristic of the received radiation;
`
`c) categorization means coupled to said processor means for categorizing said electronic
`signal to identify a source of the radiation, said categorization means comprising trained
`pattern recognition means for processing said electronic signal based on said received
`radiation to provide an identification of the source of the radiation based thereon, said
`pattern recognition means being structured and arranged to apply a pattern recognition
`algorithm generated from data of possible sources of radiation including lights of vehicles
`and patterns of received radiation from the possible sources; and
`
`d) output means coupled to said categorization means for dimming the headlights in said
`vehicle in response to the identification of the source of the radiation.
`
`5
`
`CORRECTED AVS EXHIBIT 2006
`Toyota Inc. v. American Vehicular Sciences LLC
`IPR2013-00424
`
`
`
`Independent Claim 23
`
`23. A method for affecting a system in a vehicle based on an object exterior of the
`vehicle, comprising the steps of: a) transmitting electromagnetic waves to illuminate the
`exterior object;
`
`b) receiving reflected electromagnetic illumination from the object on an array;
`
`c) processing the received illumination and creating an electronic signal
`characteristic of the exterior object based thereon;
`
`d) processing the electronic signal based on the received illumination from the
`exterior object to identify the exterior object, said processing step comprising the
`steps of generating a pattern recognition algorithm from data of possible exterior
`objects and patterns of received electromagnetic illumination from the possible
`exterior objects, storing the algorithm within a pattern recognition system and
`applying the pattern recognition algorithm using the electronic signal as input to
`obtain the identification of the exterior object; and
`
`e) affecting the system in the vehicle in response to the identification of the
`exterior object.
`
`
`
`6
`
`CORRECTED AVS EXHIBIT 2006
`Toyota Inc. v. American Vehicular Sciences LLC
`IPR2013-00424
`
`
`
`Qualifications of Prof. Koutsougeras
`(AVS expert)
`
`4. I received my B.S. in 1983 from the National Technical University of
`Athens, my M.S. degree in 1984 from the University of Cincinnati, and my
`PhD in 1988 from Case Western Reserve University. My Ph.D. research
`and dissertation was on the topic of neural networks and more specifically
`on algorithms for training feed-forward types of neural networks.
`
`5. I also have experience in automotive technology involving external
`object detection and collision warning systems, and the use of pattern
`recognition technology in such systems, as I have participated in the
`DARPA 2005 Grand Challenge competition with a team that built an
`autonomous vehicle designed to drive completely unassisted in unknown
`and unrehearsed cross-country environments. The vehicle was a regular
`production SUV which was modified to be controlled by computers aided
`by sensors including Ladars and GPS.
`
`
`Ex. 2002, Koutsougeras Declaration at ¶¶ 4-5.
`
`7
`
`CORRECTED AVS EXHIBIT 2006
`Toyota Inc. v. American Vehicular Sciences LLC
`IPR2013-00424
`
`
`
`Qualifications of Prof. Koutsougeras
`(AVS expert)
`
`Q. I’ll start by asking whether you can summarize for me your experience with
`neural networks?
`
`A. I first got familiar with the concept of neural networks while I was doing my
`Ph.D. dissertation. And I did my dissertation on neural networks, particularly
`methods for training neural networks. Subsequently I – as a faculty, I directed
`thesis of students and I taught classes that were either specifically on neural
`networks and substitute neural networks or for which neural networks was a
`substantial component. And then I can also say that I did various applications
`of neural networks, works with applications of neural networks, and that there
`were a substantial number of different domains in which I tried to apply them.
`
`Ex. 1019, Koutsougeras Dep. Tr. at 19:19-20:12
`
`8
`
`CORRECTED AVS EXHIBIT 2006
`Toyota Inc. v. American Vehicular Sciences LLC
`IPR2013-00424
`
`
`
`Qualifications of Prof. Koutsougeras
`(AVS expert)
`
`Q. So, prior to and including 1995, did you yourself have any experience working
`with vehicle exterior monitoring systems?
`
`A. I –yes, I was aware of some systems, and I had used them in the lab.
`
`Q. What systems had you used in the lab prior to and including 1995?
`
`A. Proximity sensors, ultrasound-based.
`
`
`
`*
`*
`*
`Q. Have you ever had any experience working with pattern recognition algorithms
`in vehicle exterior monitoring systems?
`
`A. Yes.
`
`Ex. 1019, Koutsougeras Dep. Tr. at 23:25-24:8
`
`9
`
`CORRECTED AVS EXHIBIT 2006
`Toyota Inc. v. American Vehicular Sciences LLC
`IPR2013-00424
`
`
`
`Claim Construction
`Prof. Koutsougeras (AVS expert)
`
`The ‘000 patent discloses and claims a specific method for generating the “training set.”
`The ‘000 patent discloses training the algorithm with (1) data of possible exterior objects or
`data of possible radiation sources and (2) patterns of received waves (e.g., patterns of
`received electromagnetic illumination from the possible exterior objects or patterns of
`received radiation from the possible sources.) For example, if the vehicle uses a radar
`receiver, a neural network could be trained with examples of received radar waves from
`possible objects such as cars, plus labels to indicate the identification and possibly other
`information relating to the object. This “other information” might include, for example,
`location information. The statements “received waves from the possible exterior objects”
`and “received electromagnetic illumination from the possible exterior objects” are
`understood to describe actual readings of real waves (e.g., real electromagnetic
`illumination) from actual possible exterior objects. The statement “received radiation from
`the possible sources” is understood to describe actual readings of real waves (e.g., real
`radiation) from actual possible radiation sources.
`
`Ex. 2002, Koutsougeras Declaration at ¶ 19.
`
`10
`
`CORRECTED AVS EXHIBIT 2006
`Toyota Inc. v. American Vehicular Sciences LLC
`IPR2013-00424
`
`
`
`Claim Construction
`Prof. Koutsougeras (AVS expert)
`
`In the case of a radar system, the examples of received radar waves from possible
`objects used to train the system are real radar waves, so that the system knows how to
`recognize radar waves received from that same object or a similar one when the
`vehicle is later driving in actual conditions. This can be done, for example, by putting
`actual examples of a possible object in front of a vehicle radar system, subjecting the
`object with radar waves that are received back by the system, and then labeling the
`object for the system. This is different from other possible ways to train a pattern
`recognition system, such as through completely simulated data, which I discuss below.
`
`
`Ex. 2002, Koutsougeras Declaration at ¶ 20.
`
`11
`
`CORRECTED AVS EXHIBIT 2006
`Toyota Inc. v. American Vehicular Sciences LLC
`IPR2013-00424
`
`
`
`Lemelson
`
`“Neural networks used in the vehicle warning system are trained
`to recognize roadway hazards which the vehicle is approaching
`including automobiles, trucks, and pedestrians. Training involves
`providing known inputs to the network resulting in desired output
`responses. The weights are automatically adjusted based on
`error signal measurements until the desired outputs are
`generated. Various learning algorithms may be applied.”
`
`Lemelson at 8:1-8
`
`12
`
`CORRECTED AVS EXHIBIT 2006
`Toyota Inc. v. American Vehicular Sciences LLC
`IPR2013-00424
`
`
`
`13
`
`Prof. Koutsougeras (AVS’s Expert)
`
`46. Lemelson discloses a system for identifying objects exterior to a vehicle, and it does
`disclose using a type of pattern recognition algorithm, a neural network. (See Lemelson at
`Fig 3, 7:47, 8:1) The only pertinent discussion in Lemelson, however, relating to generating
`the neural network, is found at column 8, line 4, which states “[t]raining involves providing
`known inputs to the network resulting in desired output responses.” (Lemelson at 8:4-6.)
`
`47. I note that this is also the only reference to training in Lemelson found in the declaration
`of Dr. Papanikolopoulos. (See Exhibit 1013, Papanikolopoulos Decl. at ¶¶ 47-65.)
`Specifically, Dr. Papanikolopoulos only quotes the sentence from Lemelson that states
`“[t]raining involves providing known inputs to the network resulting in desired output
`responses.” (Id. at ¶59.) Nowhere else in his declaration does he use the words “training”,
`“trained,” or “generate” when discussing the Lemelson reference.
`
`
`
`Ex. 2002, Koutsougeras Declaration at ¶ 46-47.
`
`CORRECTED AVS EXHIBIT 2006
`Toyota Inc. v. American Vehicular Sciences LLC
`IPR2013-00424
`
`
`
`Prof. Koutsougeras (AVS expert)
`
`57. First, in my opinion, the system in Lemelson could have been trained using completely
`simulated data, rather than data from possible exterior objects and patterns of received
`electromagnetic illumination from the possible exterior objects, or rather than data of possible
`sources of radiation and patterns of received radiation from the possible sources.
`
`56. Simulated data is data that is not obtained from or based on actual real-time readings of
`real experiments or actual rehearsals in the domain of interest. Instead, it is generated by
`programs that simulate the target domain and produce estimates of what the sensors would
`be reading in a real instance. As such, it is essentially created by a programmer, who
`specifies parameters expected from an object (e.g., it would be expected to have a certain
`range of shapes and sizes, have varying positions and orientations, etc.). For example, data
`that represents scan readings of a road ahead of a vehicle can be obtained by actually
`driving the vehicle and recording the scans, or it can be estimated by computational
`geometric means on a computer-simulated road. Simulated data is therefore not data from
`objects or patterns of waves from objects—it is completely made-up data. . . .
`
`
`Ex. 2002, Koutsougeras Declaration at ¶ 55-56.
`
`14
`
`CORRECTED AVS EXHIBIT 2006
`Toyota Inc. v. American Vehicular Sciences LLC
`IPR2013-00424
`
`
`
`Prof. Koutsougeras (AVS expert)
`
`57. As an analogy, simulated data is similar to a cartoon movie compared to a
`movie made with actors. The cartoon would provide a rough approximation for
`what a person is expected to look like, but would not be nearly as accurate as
`a video with a real actor. Or as another analogy, training with simulated data
`is analogous to training with only drawings of cars or other objects. One would
`not say that the cartoon or drawing is generated from “patterns of waves” from
`objects. They are made up.
`
`
`Ex. 2002, Koutsougeras Declaration at ¶ 59.
`
`15
`
`CORRECTED AVS EXHIBIT 2006
`Toyota Inc. v. American Vehicular Sciences LLC
`IPR2013-00424
`
`
`
`Simulated Data
`
`claim 4 “wherein said neural network training is
`accomplished by the use of simulated data”
`
`claim 6 “wherein said neural network training is
`accomplished by applying actual data”
`
`U.S. Pat. No. 5,537,327
`
`16
`
`CORRECTED AVS EXHIBIT 2006
`Toyota Inc. v. American Vehicular Sciences LLC
`IPR2013-00424
`
`
`
`Prof. Koutsougeras, Cont.
`
`Simulated data can be more desirable in some aspects compared to real-world
`data. Using simulated data has certain advantages in being able to generate a
`large training set easily and ensure balance. “Balance” means that the training
`set is not disproportionately weighted. For example, if I am training a system to
`detect roads, the system would not be balanced if it had mostly right-hand
`turns, and very few left-hand turns.
`
`Ex. 2002, Koutsougeras Decl. at ¶ 62
`
`17
`
`CORRECTED AVS EXHIBIT 2006
`Toyota Inc. v. American Vehicular Sciences LLC
`IPR2013-00424
`
`
`
`Prof. Papanikolopolous (Toyota’s Expert)
`
`A.
`
`Q. Could training a pattern recognition algorithm with simulated data involve
`less data that needs to be stored on the system. … As compared to
`training with images, for example, real life images?
`
`. . . [T]he answer is probably yes.
`
`Q. Could in 1995 simulated data involve smaller file sizes than digital image
`data?
`
`A. Potentially, yes.
`
`Exhibit 2003, Papanikolopoulos Dep. Tr. at 103:16-104:4, 123:19-22.
`
`18
`
`CORRECTED AVS EXHIBIT 2006
`Toyota Inc. v. American Vehicular Sciences LLC
`IPR2013-00424
`
`
`
`Lemelson at col. 8, lines 8-10
`
`“Adaptive operation is also possible with on-line adjustment of
`neural networks to meet imaging requirements.”
`
`Lemelson at 8:8-10
`
`“pattern recognition algorithm generated from data of possible
`exterior objects and patterns of received electromagnetic
`illumination from the possible exterior objects”
`
`‘000 patent at claims 10
`
`19
`
`CORRECTED AVS EXHIBIT 2006
`Toyota Inc. v. American Vehicular Sciences LLC
`IPR2013-00424
`
`
`
`Prof. Koutsougeras, Cont.
`
`“The fact that a computer was used to analyze images obtained
`when the system is in use (after it has been trained) tells
`nothing about the nature and extent of the training phase. An
`image analysis computer could be used to analyze received
`camera images when the vehicle is driving along the road, but
`the system could have been trained with something else
`entirely (such as simulated inputs as I discuss below).”
`
`Ex. 2002, Koutsougeras Decl. at ¶ 48
`
`20
`
`CORRECTED AVS EXHIBIT 2006
`Toyota Inc. v. American Vehicular Sciences LLC
`IPR2013-00424
`
`
`
`Prof. Koutsougeras, Cont.
`
`Accordingly, in my opinion . . . Lemelson’s neural network was not
`“necessarily” generated using data and patterns of received waves from
`possible objects
`
`Koutsougeras Decl. at ¶ 64
`
`21
`
`CORRECTED AVS EXHIBIT 2006
`Toyota Inc. v. American Vehicular Sciences LLC
`IPR2013-00424
`
`
`
`Papanikolopoulos Deposition Testimony
`
`Q. Is it technologically possible to use simulated data for a pattern
`recognition system for training that system for detecting
`automobiles?
`
`A. If you look at my declaration, I mention several systems in this
`domain. In this particular domain, you go to simulated data, or if
`you don’t have access to real data, to real images.
`
`Exhibit 2003, Papanikolopoulos Dep. Tr. at 102:5-14.
`
`22
`
`CORRECTED AVS EXHIBIT 2006
`Toyota Inc. v. American Vehicular Sciences LLC
`IPR2013-00424
`
`
`
`Papanikolopoulos 2nd Declaration
`
`In my opinion, one of ordinary skill in the art would have understood the phrase “known
`inputs” in Lemelson to refer to “real data” because Lemelson’s neural network was trained
`to identify exterior objects, and one of ordinary skill in 1995 would have known that training
`with “real data” would have yielded the best results for this purpose. One of ordinary skill in
`the art would not have understood that “known inputs” referred to simulated data or partial
`data in the context of Lemelson’s disclosure, since one of ordinary skill would not have had
`any reason to believe that those categories of data would have been effective for the
`purpose of identifying exterior objects or sources of radiation.
`
`Exhibit 1020, Papanikolopoulos Reply Decl. at 9.
`
`23
`
`CORRECTED AVS EXHIBIT 2006
`Toyota Inc. v. American Vehicular Sciences LLC
`IPR2013-00424
`
`
`
`Terms Construed in Toyota’s Petition
`
`24
`
`“pattern recognition algorithm”
`
`“trained pattern recognition means” and “trained pattern recognition
`algorithm”
`
`“identify”/”identification”
`
`“measurement means for measuring a distance between the exterior
`object and the vehicle”
`
`“rear view mirror”
`
`
`
`Toyota’s Petition at pp. 6-8.
`
`CORRECTED AVS EXHIBIT 2006
`Toyota Inc. v. American Vehicular Sciences LLC
`IPR2013-00424
`
`
`
`25
`
`Term Construed in Toyota’s Reply
`
`“The ‘generated from’ language is not a limitation for purposes of the
`patentability analysis.”
`
`
`
`Toyota’s Reply at pp. -.
`
`CORRECTED AVS EXHIBIT 2006
`Toyota Inc. v. American Vehicular Sciences LLC
`IPR2013-00424
`
`
`
`Patent Trial Practice Guide
`
`“While replies can help crystalize issues for decision, a reply that
`raises a new issue or belatedly presents evidence will not be
`considered and may be returned.”
`
`Patent Trial Practice Guide, Fed. Reg. Vol. 77, No. 157 at p. 48767.
`
`26
`
`CORRECTED AVS EXHIBIT 2006
`Toyota Inc. v. American Vehicular Sciences LLC
`IPR2013-00424
`
`
`
`27
`
`Prof. Papanikolopoulos’s Deposition
`
`Q. So can you think of an example of a trained pattern recognition
`algorithm that is not generated from data of possible exterior objects
`and patterns of received waves from the possible exterior objects?
`
`A. With respect to the specific claim construction that I had to
`examine, no.
`
`
`
` Papanikolopoulos Dep. Tr. at 90:20-91:3.
`
`CORRECTED AVS EXHIBIT 2006
`Toyota Inc. v. American Vehicular Sciences LLC
`IPR2013-00424
`
`
`
`Claim 16
`
`16. In a motor vehicle having an interior and an exterior, an automatic headlight dimming
`system comprising:
`
`a) reception means for receiving electromagnetic radiation from the exterior of the vehicle;
`
`b) processor means coupled to said reception means for processing the received radiation
`and creating an electronic signal characteristic of the received radiation;
`
`c) categorization means coupled to said processor means for categorizing said electronic
`signal to identify a source of the radiation, said categorization means comprising trained
`pattern recognition means for processing said electronic signal based on said received
`radiation to provide an identification of the source of the radiation based thereon, said
`pattern recognition means being structured and arranged to apply a pattern recognition
`algorithm generated from data of possible sources of radiation including lights of vehicles
`and patterns of received radiation from the possible sources; and
`
`d) output means coupled to said categorization means for dimming the headlights in said
`vehicle in response to the identification of the source of the radiation.
`
`28
`
`CORRECTED AVS EXHIBIT 2006
`Toyota Inc. v. American Vehicular Sciences LLC
`IPR2013-00424
`
`
`
`Yanagawa
`
`Yanagawa at p. 3.
`
`29
`
`CORRECTED AVS EXHIBIT 2006
`Toyota Inc. v. American Vehicular Sciences LLC
`IPR2013-00424
`
`
`
`Prof. Koutsougeras (“AVS expert”)
`
`In my opinion, combining the neural network of Lemelson with the
`headlight and taillight recognition device of Yanagawa does not make
`sense. Yanagawa describes a way to detect taillights and headlights of
`other vehicles based on very specific and relatively simple calculations
`and geometric considerations (see, e.g., Exhibit 1009, Yanagawa at 3,
`equations (3)-(5)). The onboard device receives an image from ahead
`of the vehicle (see, e.g., id. at 2; and Figures 1 and 2), and performs a
`simple filtering of the pixels (see, e.g., id., equations (1) and (2)) to
`determine whether taillights or headlights are present.
`
`Ex. 2002, Koutsougeras Decl. at ¶ 80.
`
`30
`
`CORRECTED AVS EXHIBIT 2006
`Toyota Inc. v. American Vehicular Sciences LLC
`IPR2013-00424
`
`
`
`Prof. Koutsougeras (“AVS expert”)
`
`It would not be reasonable to try to replace the simple calculations of
`Yanagawa with a neural network as in Lemelson. A neural network might
`assist with problems for which the functional relation between inputs and
`outputs is not known or is not expressible in finite terms. But if the function
`is explicitly known such as, for example, in the form of Yanagawa’s explicit
`equations or is expressible in finite terms such as, for example, a set of
`rules, then there are no good reasons to even try to use a neural network
`to perform that explicitly known and well specified function through
`training. It would be simpler and more reliable to code the equation or the
`set of rules into a computer program.
`
`Ex. 2002, Koutsougeras Decl. at ¶ 81.
`
`31
`
`CORRECTED AVS EXHIBIT 2006
`Toyota Inc. v. American Vehicular Sciences LLC
`IPR2013-00424
`
`
`
`Prof. Koutsougeras (“AVS expert”)
`
`This is why it is hard to understand why one of ordinary skill in the art with
`knowledge of Yanagawa’s equations and Yanagawa’s device would even
`contemplate trying to use a neural network with its uncertainties instead of
`the certainty of its equations and calculations that could be encoded as
`software into Yanagawa’s device. Using Yanagawa’s equations and
`device, a programmer would know exactly what routine the computer is
`performing because the programmer has instructed step-by-step the
`computer in Yanagawa, for example, to use the given equations and/or to
`follow a specific set of rules. When such knowledge and certainty is
`available in Yanagawa, it makes no sense to inject or to have to deal with
`the uncertainties of a neural network as in Lemelson.
`
`Ex. 2002, Koutsougeras Decl. at ¶ 82.
`
`32
`
`CORRECTED AVS EXHIBIT 2006
`Toyota Inc. v. American Vehicular Sciences LLC
`IPR2013-00424
`
`
`
`Prof. Koutsougeras (“AVS expert”)
`
`The fact that an unacceptable degradation or a failure may result negates
`Dr. Papanikolopoulos’s conclusion that the Lemelson neural network
`possesses a reliability advantage.
`
`Ex. 2002, Koutsougeras Decl. at ¶ 83.
`
`33
`
`CORRECTED AVS EXHIBIT 2006
`Toyota Inc. v. American Vehicular Sciences LLC
`IPR2013-00424