UNITED STATES PATENT AND TRADEMARK OFFICE
`__________________
`
`BEFORE THE PATENT TRIAL AND APPEAL BOARD
`__________________________________________________________________
`
`TOYOTA MOTOR CORPORATION
`
`Petitioner
`
`v.
`
`AMERICAN VEHICULAR SCIENCES,
`
`Patent Owner
`
`Patent No. 6,772,057
`Issue Date: Aug. 3, 2004
`Title: VEHICULAR MONITORING SYSTEMS
`__________________________________________________________________
`
`REPLY DECLARATION OF NIKOLAOS PAPANIKOLOPOULOS, PH.D.
`
`
`Case No. IPR2013-00419
`__________________________________________________________________
`
`IPR2013-00419 - Ex. 1023
`Toyota Motor Corp., Petitioner
`1
`
`
`__________________
`
`BEFORE THE PATENT TRIAL AND APPEAL BOARD
`__________________________________________________________________
`
`TOYOTA MOTOR CORPORATION
`
`Petitioner
`
`v.
`
`AMERICAN VEHICULAR SCIENCES,
`
`Patent Owner
`
`Patent No. 6,772,057
`Issue Date: Aug. 3, 2004
`Title: VEHICULAR MONITORING SYSTEMS
`__________________________________________________________________
`
`REPLY DECLARATION OF NIKOLAOS PAPANIKOLOPOULOS, PH.D.
`
`
`Case No. IPR2013-00419
`__________________________________________________________________
`
`IPR2013-00419 - Ex. 1023
`Toyota Motor Corp., Petitioner
`1
`
`
`
`I, Nikolaos Papanikolopoulos, Ph.D., hereby further declare and state as follows:
`
`I.
`BACKGROUND
`1. My employment and compensation information have not changed since I
`
`submitted my original declaration in support of Toyota’s Petition for Inter Partes
`
`Review of U.S. Patent No. 6,772,057 (“the ’057 patent”).
`
`2.
`
`A copy of my updated curriculum vitae is included herewith.
`
`II. ASSIGNMENT AND COMPENSATION
`3.
`I submit this declaration in support of Toyota’s Reply to Patent Owner’s
`
`Response (Paper 33, hereinafter “Response”) and in response to the Declaration
`
`(Exhibit 2001) and Deposition Testimony (Exhibit 1022) of Cris Koutsougeras.
`
`4.
`
`Specifically, I have been asked to respond to Dr. Koutsougeras’s opinions
`
`regarding the disclosure in U.S. Patent No. 6,553,130 (“Lemelson”) relating to neural
`
`network training.
`
`5.
`
`The opinions expressed in this declaration are not exhaustive of my opinions
`
`on the patentability of any of the claims in the ’057 patent. Therefore, the fact that I
`
`do not address a particular point should not be understood to indicate any agreement
`
`on my part that any claim otherwise complies with the patentability requirements.
`
`In forming my opinion I have reviewed the following additional sources:
`
`
`•
`
`Declaration of Chris Koutsougeras, PhD in Support of AVS’s Response
`
`Under 37 CFR §42.120 (Ex. 2001).
`
`•
`
`Decision on Institution of Inter Partes Review for U.S. Patent No.
`
`
`
`1
`
`
`
` 2
`
`
`
`•
`
`•
`
`•
`
`•
`
`6,772,057 (Paper 19).
`
`Patent Owner’s Response (Paper 33).
`
`U.S. Patent No. 5,537,327 (Exhibit 2003).
`
`Pomerleau, “Neural Network Perception for Mobile Robot Guidance”
`
`(Exhibit 2004).1
`
`The transcript from the deposition of Dr. Cris Koutsougeras in
`
`connection with this case (Exhibit 1022).
`
`6.
`
`The opinions expressed in this declaration are my personal opinions and do not
`
`reflect the views of University of Minnesota.
`
`III. ANALYSIS
`A.
`Preliminary Understanding of Dr. Koutsougeras’ Positions
`As a preliminary matter, I understand from Dr. Koutsougeras’s declaration and
`
`7.
`
`deposition that he divided the data that could have been used for pattern recognition
`
`algorithm training (in 1995) into three areas: training with data and waves from actual
`
`objects (“real data”), training with simulated data and waves (“simulated data”), and
`
`training with “data and waves not representing exterior objects to be detected”
`
`(“partial data”). Ex. 1022 at 86:25-87:14, 132:24-138:5, 163:18-164:7. As I understand
`
`
`1
`I was previously familiar with this paper in the context of my work on
`
`IPR2013-00424 and from my interaction with Dean Pomerleau and the NavLab
`
`vehicle.
`
`
`
`2
`
`
`
` 3
`
`
`
`it, Dr. Koutsougeras opined that only training with real data would meet the claim
`
`limitation “pattern recognition algorithm generated from data of possible exterior
`
`objects and patterns of received waves.”
`
`8.
`
`As I understand it, Dr. Koutsougeras further opined that Lemelson’s disclosure
`
`of training with “known inputs” does not necessarily mean training with “real data”
`
`because it could have been referring instead to “simulated data” or “partial data.”
`
`9.
`
`For the reasons I discuss below, I disagree with Dr. Koutsougeras’s
`
`interpretation of Lemelson. In my opinion, one of ordinary skill in the art would have
`
`understood the phrase “known inputs” in Lemelson to refer to “real data” because
`
`Lemelson’s neural network was trained to identify exterior objects, and one of
`
`ordinary skill in 1995 would have known that training with “real data” would have
`
`yielded the best results for this purpose. One of ordinary skill in the art would not
`
`have understood that “known inputs” referred to simulated data or partial data in the
`
`context of Lemelson’s disclosure, since one of ordinary skill would not have had any
`
`reason to believe that those categories of data would have been effective for the
`
`purpose of object identification (e.g., for the purpose of identification of a pedestrian).
`
`B. One of Ordinary Skill Would Have Understood that Training of
`the Lemelson Neural Network Would Have Used Real Data
`10. Lemelson discloses a collision avoidance system, wherein a neural network is
`
`used to identify many different types of objects that could present themselves as
`
`hazards on a roadway, including, for example, road barriers, trucks, automobiles,
`
`
`
`3
`
`
`
` 4
`
`
`
`pedestrians, signs and symbols, etc. Ex. 1002 at 5:41-43; 8:1-6. Lemelson explains:
`
`Neural networks used in the vehicle . . . warning system are trained to
`
`recognize roadway hazards which the vehicle is approaching including
`
`automobiles, trucks, and pedestrians. Training involves providing
`
`known inputs to the network resulting in desired output responses. The
`
`weights are automatically adjusted based on error signal measurements
`
`until the desired outputs are generated. Various learning algorithms may
`
`be applied. Adaptive operation is also possible with on-line adjustment
`
`of network weights to meet imaging requirements.
`
`Ex. 1002 at 8:1-6.
`
`11. One of ordinary skill in the art would have understood the phrase “known
`
`inputs,” and would have understood that it referred to the use of real sensor data in
`
`the context of Lemelson. For example, one of ordinary skill would have understood
`
`that training a neural network could involve putting actual examples of real-world
`
`objects in front of a camera, imaging them, and providing feedback to the neural
`
`network as to the desired output responses corresponding to those images.
`
`12. As set forth below, it is my opinion that one of ordinary skill would not have
`
`understood the phrase “known inputs” in the context of Lemelson to refer to “partial
`
`data” or “simulated data” because one of ordinary skill would have recognized that
`
`neither of these categories would have been effective for the intended purpose of
`
`training a neural network to identify various types of exterior objects. I often refer to
`4
`
`
`
`
`
` 5
`
`
`
`these ineffective training routines to my students as “garbage in–garbage out.”
`
`C. Dr. Koutsougeras’ Description of Partial Data is Inaccurate
`13. One of ordinary skill in the art would not have understood the phrase “known
`
`inputs” to refer to partial data, such as from license plates, tail lights or rear windows
`
`because they would have recognized that training a neural network with partial data
`
`would not have been successful for the purpose of identifying different exterior
`
`objects.
`
`14.
`
`For example, one of ordinary skill in the art would not have been able to use
`
`partial data, such as rear windows, license plates, or taillights to identify, classify, or
`
`locate pedestrians and to distinguish them from vehicles. While partial data may be
`
`useful in certain isolated situations, such as when there is only a single object of
`
`interest, partial data is not useful when there are many possible objects that need to be
`
`identified, such as is the case in Lemelson.
`
`15. Also, the presence of occlusions and/or shadows in the environment exterior
`
`to a vehicle complicates training since these occlusions and shadows may completely
`
`mask partial data.
`
`16. One of ordinary skill in the art would not have understood that a neural
`
`network used for exterior object identification would be trained with partial data.
`
`Rather, one of ordinary skill would have understood that a neural network would be
`
`trained with all available sensor information to associate particular sensor information
`
`with desired output responses. The purpose of training a neural network is to identify
`
`
`
`5
`
`
`
` 6
`
`
`
`the particular features in a scene that are important and that are indicative of the
`
`exterior object of interest. On the other hand, to detect objects using partial data, one
`
`of ordinary skill would already expect to know the particular features that are
`
`important and indicative of the exterior object of interest (e.g., the outline and corner
`
`points of a license plate). None of the examples that Dr. Koutsougeras points to in
`
`his declaration involved training a neural network with partial data prior to 1995. It is
`
`also worth noting that Dr. Koutsougeras points to a pattern recognition system I
`
`worked on to detect license plates. Contrary to Dr. Koutsougeras’s assumption, this
`
`system did not involve trained pattern recognition.
`
`17. Accordingly, one of ordinary skill would not have expected that training the
`
`neural network of Lemelson with partial data would have resulted in a system capable
`
`of identifying the exterior objects required by Lemelson, such as “pedestrians, barriers
`
`and dividers, turns in the road, signs and symbols.” Ex. 1002 at 5:42-43.
`
`D. Dr. Koutsougeras’ Description of Simulated Data and His
`Characterizations of Pomerleau are Inaccurate
`18. One of ordinary skill in the art would not have understood the phrase “known
`
`inputs” to refer to simulated data because they would have recognized that training a
`
`neural network with simulated data would not have been successful for the purpose of
`
`identifying different exterior objects. One of ordinary skill in the art in 1995 would
`
`have known that the generation of simulated data was not sophisticated enough to
`
`allow for training the type of neural network described by Lemelson. Thus, even if
`
`
`
`6
`
`
`
` 7
`
`
`
`simulated data were used, the result would have likely been “garbage in–garbage out.2”
`
`19. One of ordinary skill in the art would have known that simulated data suffered
`
`from several problems in 1995.
`
`20.
`
`First, generation of simulated data would have required a lot of computer
`
`power and special equipment, neither of which were disclosed by Lemelson. See, e.g.,
`
`the Warp supercomputer used by Pomerleau, Ex. 2004 at 40. Lemelson does not
`
`disclose any computer hardware or methods for generating simulated data. See, e.g.,
`
`1002 at Fig. 1.
`
`21.
`
`Second, the Lemelson neural network was trained to identify “other vehicles,
`
`pedestrians, barriers and dividers, turns in the road, signs and symbols.” As of 1995,
`
`
`2. I note that AVS has taken two quotes from my deposition regarding simulated data
`
`out of context. First, the system I was talking about on page 48 of the transcript was
`
`my own system for identifying only license plates. See Ex. 1025 at 41:4-49:21. This
`
`system was not a collision avoidance system where different types of exterior objects
`
`needed to be identified. I was only locating license plates. Second, the question on
`
`page 102 referred to a system from the present day and not from 1995. See Ex. 1025
`
`at 102:18-22. Both of my statements at my deposition are fully consistent with my
`
`opinion here that one of ordinary skill in the art at the time of the publication of
`
`Lemelson would only have expected to use real data as a known input to the neural
`
`network.
`
`
`
`7
`
`
`
` 8
`
`
`
`one of ordinary skill in the art would not have expected that a simulated data set could
`
`be readily generated that could accurately represent all exterior objects described by
`
`Lemelson as perceived by sensors on a vehicle. This type of simulation would have
`
`required modeling of both a moving camera and moving objects in a scene, such as
`
`pedestrians, which would have resulted in a very complex data set. Furthermore, one
`
`of ordinary skill would have recognized that all of these complexities would have been
`
`obviated by simply training the system with real data in a variety of situations.
`
`22. Dr. Koutsougeras cites to a document that I discussed at length in a declaration
`
`I submitted in connection with IPR2013-00424: Dean Pomerleau’s 1992 Thesis. Ex.
`
`2004. Dr. Koutsougeras relies on the disclosure of Pomerleau’s Thesis as proof that
`
`one of ordinary skill would have used simulated data in the Lemelson system.
`
`However, Dr. Koutsougeras fails to read and comprehend the entirety of Pomerleau’s
`
`disclosures in several ways.
`
`23.
`
`First, the Board determined that a similar Pomerleau reference (Ex. 1008) only
`
`attempts to detect the road surface and does not identify “exterior objects,” as that
`
`term was construed. Paper 19, Decision on Institution at 12-14, 35-36. Pomerleau
`
`does not stand for the proposition that one of ordinary skill in the art would have
`
`thought one could use simulated data to train the system of Lemelson to identify
`
`exterior objects.
`
`24.
`
`Second, the Pomerleau thesis concluded that training with simulated data “has
`
`serious drawbacks.” Ex. 2004 at p. 40. Ultimately, Pomerleau concluded that
`8
`
`
`
`
`
` 9
`
`
`
`simulated data should not be used to train a system. Id. at pp. 40, 56. He reached this
`
`conclusion despite the fact that Pomerleau’s computational needs were much less
`
`demanding than required by Lemelson. Pomerleau explained that “differences
`
`between the synthetic road images on which the network was trained and the real
`
`situations on which the network was tested often resulted in poor performance in real
`
`driving situations.” Ex. 2004 at p. 40. Pomerleau further stated: “[W]hile relatively
`
`effective at training the network to drive under the limited conditions of a single-lane
`
`road, it quickly became apparent that extending the synthetic training paradigm to deal
`
`with more complex situations such as multi-lane and off-road driving would require
`
`prohibitively complex training data generators.” Id. Because of these drawbacks,
`
`Pomerleau concluded that, “[g]enerating realistic artificial training data proved
`
`impractical for all but the simplest driving situations.” Ex. 2004 at p. 56.
`
`25. One of ordinary skill in the art would not have understood that the phrase
`
`“known inputs” in Lemelson referred to “simulated data.”
`
`26. Dr. Koutsougeras also cites to U.S. Patent No. 5,537,327, in his discussion of
`
`simulated training data. However, his citation to the ’327 patent is misplaced. The
`
`disclosure of the ’327 patent relates to a different subject matter from that disclosed
`
`by the ’057 patent or the Lemelson reference. It relates to the use of neural networks
`
`to identify fault impedances in electrical power systems. The ’327 patent does not
`
`involve classification, identification, or location of objects exterior to a vehicle, or for
`
`that matter, any exterior monitoring from a vehicle at all. Furthermore, the ’327
`9
`
`
`
`
`
`10
`
`
`
`patent accounts for none of the complications that would have arisen when
`
`identifying possible exterior objects that could collide with a vehicle, as in Lemelson.
`
`Accordingly, the ’327 patent would not have indicated to one of ordinary skill in the
`
`art that “simulated data” could have been used as a known input to the Lemelson
`
`system.
`
`
`
`Date: May 26, 2014
`
`
`
`______________________________
`Nikolaos Papanikolopoulos, Ph.D.
`
`
`
`10
`
`
`
`11
`
`
Accessing this document will incur an additional charge of $.
After purchase, you can access this document again without charge.
Accept $ Charge
Still Working On It
This document is taking longer than usual to download. This can happen if we need to contact the court directly to obtain the document and their servers are running slowly.
Give it another minute or two to complete, and then try the refresh button.
A few More Minutes ... Still Working
It can take up to 5 minutes for us to download a document if the court servers are running slowly.
Thank you for your continued patience.
This document could not be displayed.
We could not find this document within its docket. Please go back to the docket page and check the link. If that does not work, go back to the docket and refresh it to pull the newest information.
Your account does not support viewing this document.
You need a Paid Account to view this document. Click here to change your account type.
Your account does not support viewing this document.
Set your membership
status to view this document.
With a Docket Alarm membership, you'll
get a whole lot more, including:
- Up-to-date information for this case.
- Email alerts whenever there is an update.
- Full text search for other cases.
- Get email alerts whenever a new case matches your search.
One Moment Please
The filing “” is large (MB) and is being downloaded.
Please refresh this page in a few minutes to see if the filing has been downloaded. The filing will also be emailed to you when the download completes.
Your document is on its way!
If you do not receive the document in five minutes, contact support at support@docketalarm.com.
Sealed Document
We are unable to display this document, it may be under a court ordered seal.
If you have proper credentials to access the file, you may proceed directly to the court's system using your government issued username and password.
Access Government Site
