`
`
`
`Filed on behalf of ABB, Inc.
`
`By: Richard D. Mc Leod (Reg. No. 46,921)
`Rick.mcleod@klarquist.com
`Klarquist Sparkman LLP
`One World Trade Center, Suite 1600
`121 S.W. Salmon Street
`Portland, Oregon 97204
`Telephone: (503) 595-5300
`Facsimile: (503) 595-5301
`
`UNITED STATES PATENT AND TRADEMARK OFFICE
`
`____________
`
`BEFORE THE PATENT TRIAL AND APPEAL BOARD
`
`____________
`
`ABB, INC.
`Petitioner
`
`v.
`
`ROY-G-BIV CORPORATION
`Patent Owner
`
`____________
`
`Trial No. IPR2013-00062
`Patent 6,516,236 B1
`____________
`
`DECLARATION OF NIKOLAOS PAPANIKOLOPOULOS, PH.D.
`
`
`
`
`
`
`
`
`
`Page 1 of 69
`
`
`
`
`
`I, Nikolaos Papanikolopoulos, Ph.D., hereby declare and state as follows:
`
`I.
`1.
`
`BACKGROUND
`
`I am currently employed by the University of Minnesota as a Distinguished
`
`McKnight University Professor of Computer Science and Engineering. I have been a
`
`professor at the University of Minnesota (originally as an assistant professor, and then
`
`as an associate professor) since the Fall of 1992. Between Fall 2001 and Spring 2004,
`
`and between Fall 2010 and Spring 2013, I was the Director of Undergraduate Studies
`
`of the College of Science and Engineering.
`
`2.
`
`In 1992, I received my Ph.D. in Electrical and Computer Engineering from
`
`Carnegie Mellon University. My thesis, supervised by Prof. Pradeep Khosla, was
`
`entitled “Controlled Active Vision” and focused on using computer vision in a
`
`controlled fashion to monitor and manipulate objects in the environment. The
`
`computer vision was implemented on a camera and a computer system that processed
`
`the images and issued motion control commands to a robot manipulator. My group
`
`was the Advanced Manipulators Laboratory (AML). In 1988, I also received my M.S.
`
`in Electrical and Computer Engineering from Carnegie Mellon University. My B.S. in
`
`Electrical Engineering was received in 1987 from the National Technical University in
`
`Athens, Greece.
`
`3.
`
`Over the last twenty three years, my research and teaching work has focused on
`
`robotics, computer vision, real-time systems, and intelligent transportation systems.
`
`This research has included control and sensing software for robots like manipulators
`
`
`
`
`
`Page 2 of 69
`
`
`
`
`
`and mobile robots.
`
`4.
`
`I have founded a company named ReconRobotics Inc. which is one of the
`
`largest manufacturers in the world of miniature robots like the UMN Scout. More
`
`than 4,000 Scout robots have been deployed around the globe.
`
`5. My research in the early 1990’s focused on solving sensor-based control
`
`problems for robots including using sensory systems and algorithms to move a
`
`manipulator to detect an object, track it, estimate its position and orientation, and
`
`finally grasp it. Some of our efforts were described under the term “visual servoing.”
`
`Our robots where the algorithms were implemented ranged from a Direct Drive Arm
`
`(DDArmII) to a Puma 560 robot and the Chimera II system was used to coordinate
`
`the various software tasks. A screenshot of the pertinent system is shown in Figure 1.
`
`
`
`Figure 1: Experimental setup for the visual servoing task that used Chimera II.
`6. When I moved to the University of Minnesota, I continued the work and in
`
`fact I received funding from the Sandia Labs (the initial source was the Department of
`
`Energy). The same unit partially funded the Chimera and Onika development efforts.
`
`
`
`
`
`Page 3 of 69
`
`
`
`
`
`In particular, we developed a system (using a CCD camera) that could detect the
`
`depth of an object from a camera by moving the robot in a controlled way. We also
`
`created a robot-based system that could track non-rigid objects by using active
`
`deformable models.
`
`7.
`
`I currently teach three courses relating to intelligent systems: (i) CSci 5551
`
`Introduction to Intelligent Robotic Systems, (ii) CSci 5511 Artificial Intelligence, and
`
`(iii) CSci 5561Computer Vision.
`
`8. My research has produced more than 320 journal and conference publications.
`
`More than 70 publications are in refereed journals. Many of my publications relate to
`
`control software for motion control devices (including robots). Some examples
`
`include:
`
`1) Janssen, M., and Papanikolopoulos, N.P., “Utilizing Queued Actions to
`Increase Interaction Efficiency in Robot Control Interfaces”, Proceedings of
`the 21st Mediterranean Conference on Control and Automation (MED’ 2013),
`Platanias, Greece, 2013, pp 34-39.
`
`
`2) McMillen, C., Stubbs, K., Rybski, P., Stoeter, S., Gini, M., and
`Papanikolopoulos, N.P., “Resource Scheduling and Load Balancing in
`Distributed Robotic Control Systems”, Robotics and Autonomous Systems,
`Volume 44, No. 3-4, September 2003, pp 251-259.
`
`
`3) Rybski, P., Stoeter, S., Gini, M., Hougen, D., and Papanikolopoulos, N.P.,
`"Performance of a Distributed Robotic System Using Shared
`Communications Channels: A Framework for the Operation and
`Coordination of Multiple Miniature Robots", IEEE Trans. on Robotics and
`Automation, Volume 18, No. 5, October 2002, pp 713-727.
`
`
`4) Papanikolopoulos, N.P., and Khosla, P.K., “Adaptive Robotic Visual
`Tracking: Theory and Experiments”, IEEE Trans. on Automatic Control,
`Volume 38, No. 3, March 1993, pp 429 - 445.
`
`
`
`
`Page 4 of 69
`
`
`
`
`
`9.
`
`
`5) Papanikolopoulos, N.P., Khosla, P.K., and Kanade, T., “Visual Tracking
`of a Moving Target by a Camera Mounted on a Robot: A Combination of
`Control and Vision”, IEEE Trans. on Robotics and Automation, Volume 9,
`No. 1, February 1993, pp 14 - 35.
`
`
`As a result of my work and research, I am familiar with the design, control,
`
`operation and functionality of software for motion control devices including the ones
`
`used in robots (manipulators and mobile ones).
`
`10. A copy of my curriculum vitae is attached as included herewith.
`
`
`II. ASSIGNMENT AND COMPENSATION
`11.
`I submit this declaration to oppose the Patent Owner’s Responses filed by
`
`RGB in the Inter Partes Review of U.S. Patent No. 5,516,236 (“the ’236 patent”), which
`
`includes Trial Nos. IPR2013-0062 and IPR2013-00282.
`
`12.
`
`13.
`
`I am not an employee of Petitioner ABB or any affiliate or subsidiary thereof.
`
`I am being compensated for my time at my usual rate of $700 per hour. My
`
`compensation is in no way dependent upon the substance of the opinions I offer
`
`below, or upon the outcome of this inter partes review.
`
`14.
`
`I have been asked to provide certain opinions relating to the ’236 Patent and
`
`the prior art that has been cited against the patent. Specifically, I have been asked to
`
`provide my opinion regarding (i) the level of ordinary skill in the art to which the
`
`patent pertains, and (ii) the patentability of claims of the patent.
`
`15. The opinions expressed in this declaration are not exhaustive of my opinions
`
`
`
`
`
`Page 5 of 69
`
`
`
`
`
`on the patentability of any of the claims in the patent. Therefore, the fact that I do
`
`not address a particular point should not be understood to indicate any agreement on
`
`my part that any claim otherwise complies with the patentability requirements.
`
`16. The opinions expressed in this declaration are my personal opinions and do not
`
`reflect the views of University of Minnesota.
`
`III. LEGAL STANDARDS
`17.
`I have been informed and I understand that a patentability analysis is
`
`performed from the viewpoint of a hypothetical person of ordinary skill in the art. I
`
`understand that “the person of ordinary skill” is a hypothetical person who is
`
`presumed to be aware of the universe of available prior art as of the time of the
`
`invention at issue.
`
`18.
`
`I understand that a patent claim is invalid as anticipated when a single piece of
`
`prior art describes every element of the claimed invention, either expressly or
`
`inherently, and arranged in the same way as in the claim. For inherent anticipation to
`
`be found, it is required that the missing descriptive material is necessarily present in
`
`the prior art. I understand that, for the purpose of an inter partes review, prior art that
`
`anticipates a claim can include both patents and printed publications from anywhere
`
`in the world.
`
`19.
`
`I understand that some claims are written in dependent form, in which case
`
`they incorporate all of the limitations of the claim(s) on which they depend. I
`
`understand that there are different claim construction standards use by the Patent
`
`
`
`
`
`Page 6 of 69
`
`
`
`
`
`Office and the district courts in evaluating the scope of a patent claim. It is my
`
`understanding that the Patent Office uses the “broadest reasonable interpretation in
`
`light of the specification” in reviewing claims. It is also my understanding that the
`
`Patent Trial and Appeals Board has adopted specification constructions for certain
`
`claim terms. I am also aware that the district court has adopted certain constructions
`
`that are broader than those assigned by the Board.
`
`20.
`
`I understand that a patent claim is invalid as obvious if the subject matter of
`
`the claim as a whole would have been obvious to a person of ordinary skill in the art
`
`as of the time of the invention at issue. I understand that the following factors must
`
`be evaluated to determine whether the claimed subject matter is obvious: (1) the
`
`scope and content of the prior art; (2) the difference or differences, if any, between
`
`the scope of the claim of the patent under consideration and the scope of the prior
`
`art; and (3) the level of ordinary skill in the art at the time the patent was filed. Unlike
`
`anticipation, which allows consideration of only one item of prior art, I understand
`
`that obviousness may be shown by considering more than one item of prior art. I
`
`have also been informed and I understand that so-called objective indicia of non-
`
`obviousness, also known as “secondary considerations,” like the following are also to
`
`be considered when assessing obviousness: (1) commercial success; (2) long-felt but
`
`unresolved needs; (3) copying of the invention by others in the field; (4) initial
`
`expressions of disbelief by experts in the field; (5) failure of others to solve the
`
`problem that the inventor solved; and (6) unexpected results. I also understand that
`
`
`
`
`Page 7 of 69
`
`
`
`
`
`evidence of objective indicia of non-obviousness must be commensurate in scope
`
`with the claimed subject matter. It is my understanding that RGB has not presented
`
`evidence of “secondary considerations” in this matter.
`
`21. As an initial matter, I have been informed that claim terms may be written in
`
`means-plus-function format. In this situation, the means-plus-function claim terms
`
`cover the corresponding structure identified in the specification for performing the
`
`claimed function, and equivalents thereof.
`
`22.
`
`I have applied these principles with respect to my analysis set forth below.
`
`IV. BACKGROUND OF THE PATENT
`23. The ’236 patent generally describes a software framework for motion control
`
`devices/systems. The patent alleges that the framework could potentially enable users
`
`to write software for motion control devices without having to write low-level code.
`
`The framework includes an Application Programming Interface (API) and a Service
`
`Provider Interface (SPI). The API includes component functions while the SPI
`
`includes driver functions. The driver functions are classified as core and extended
`
`ones. (According to certain RGB documents, the architecture was derived from
`
`Microsoft’s Graphic Display Interface / Windows Open Services Architecture)
`
`(Ex. 1025; also IPR2013-00282 Ex. 1006).
`
`24. Core driver functions are associated with primitive operations while extended
`
`ones are associated with non-primitive operations.
`
`
`
`
`
`
`
`Page 8 of 69
`
`
`
`
`
`V.
`25.
`
`CLAIM CONSTRUCTION
`
`I have not performed my own independent claim construction analysis.
`
`Rather, I have been asked to apply the claim constructions that have been adopted by
`
`the Board in analyzing the patentability of the identified claims.
`
`26. As noted above, I have been informed that claim terms may be written in
`
`means-plus-function format. In this situation, the means-plus-function claim terms
`
`cover the corresponding structure identified in the specification for performing the
`
`claimed function and equivalents thereof.
`
`27.
`
`I have reviewed and evaluated the “meta” interpretations that David Stewart
`
`has presented in his declaration in this matter, particularly where he appears to have
`
`adopted claim interpretations that would be narrower than the “broadest reasonable
`
`interpretation” or otherwise in conflict with an interpretation adopted by the Board.
`
`VI. BACKGROUND ON THE STATE OF THE ART
`28. The following is a brief exemplary discussion of the state of the art prior to
`
`May 1995.
`
`29. During the last forty years, there has been a growing interest in programming
`
`environments for motion control devices. The variety of hardware involved, the
`
`complexity of sensors and actuators, the rare reuse of existing software, and the
`
`existence of legacy proprietary systems forced the researchers to develop new levels of
`
`abstraction that separate the user from the low-level details and hardware. The real-
`
`time requirements pushed these abstractions to levels that involve agnostic use of the
`
`
`
`
`
`Page 9 of 69
`
`
`
`
`
`low-level device drivers.
`
`30. Robotics is a prime candidate for testing the efficacy of some of the motion
`
`control systems abstractions. Robotics involves actuators of various types and the
`
`controllers are often ones that require calls to hardware components for which the
`
`user knows or understands very little.
`
`31. Researchers have tried to develop programming languages for robotics since
`
`the early days of the field. For example, VAL was first written in 1975 and run on a
`
`PDP 11/45 with a very limited set of capabilities. For example, motions from point to
`
`point were at the center of the effort. VAL was reworked in 1976 to include structures
`
`like complex trajectory generation. Other languages like TEACH and AML also
`
`surfaced. TEACH included ideas like concurrency and device independence. Many of
`
`these efforts met with little success since they were too much focused on the robot
`
`instead of focusing on the variety of tasks that a robot might have encountered.
`
`32. Task planning was introduced to address some of these issues. Robotics
`
`experts would try to encode with fine granularity all the possible scenarios that a robot
`
`might encounter. Task planning was effective up to a point. For example, rapidly
`
`evolving environments like the ones found outdoors would render some these plans
`
`unrealistic. Attempts to re-plan were computationally challenging since they required
`
`calculation of hundreds of parameters.
`
`33. Later efforts tried to introduce uncertainty in the underlying models. Although
`
`these efforts were somewhat successful, they required robust estimates for the
`
`
`
`
`Page 10 of 69
`
`
`
`
`
`uncertainty. Limitations in hardware created impediments in the way that these
`
`systems were implemented and operated.
`
`34. Additionally, there was extensive research that was performed with respect to
`
`the application of layered software development approaches that separated device
`
`drivers from high-level programming for motion control systems (robots being one
`
`large category capitalizing on advances in motion control). This was research
`
`published in a number of different articles. The list of publications that I used in this
`
`write-up is:
`
`1)
`
`Ingimarson, D.; Stewart, D.; and Khosla, P., Chimera 3.2: The Real-
`time Operating System for Reconfigurable Sensor-Based Control
`Systems, Program Documentation, Carnegie Mellon University,
`February 1995 (“ChimeraManual”). (Ex. 1133).
`2) Stewart, D.; and Khosla, P., The Chimera Methodology: Designing
`Dynamically Reconfigurable Real-time Software Using Port-Based
`Objects, Proceedings of the First Workshop on Object-Oriented
`Real-time Dependable Systems (WORDS 1994), October 1994,
`Dana Point, CA (“StewartWorkshop”). (Ex. 1134).
`3) Morrow, J.D., Sensorimotor Primitives for Programming Robotic
`Assembly Skills, Ph.D. dissertation, Carnegie Mellon University,
`May 1997 (“MorrowThesis”). (Ex. 1131).
`4) Morrison, J.P., Data Responsive Modular, Interleaved Task
`Programming System, IBM Technical Disclosure Bulletin, Vol. 13,
`No. 8, pp. 2425-2426, January 1971 (“Morrison”). (Ex. 1135).
`5) Ramamritham, K.; Stemple, D.; Briggs, D.; and Vinter, S., Privilege
`Transfer and Revocation in a Port-Based System, IEEE
`Transactions on Software Engineering, Vol. 12, No. 5, pp. 635-648,
`1986 (“PortBasedObject”). (Ex. 1136).
`
`
`
`
`
`
`
`Page 11 of 69
`
`
`
`
`
`
`
`VII. ANALYSIS
`A.
`35.
`
`Level of Ordinary Skill in the Art
`
`I have been asked to provide my opinion regarding the level of ordinary skill in
`
`the art in May 1995, which is the date that RGB alleges to be the effective filing date
`
`of the claims.
`
`36.
`
`It is my opinion that, in 1995, a person of ordinary skill in the art would have
`
`had one of the following: (i) a bachelor’s degree in computer science, or electrical, or
`
`computer engineering (or a closely related field) with at least four years of experience
`
`working with software for motion control systems, (ii) a master’s degree in computer
`
`science, or electrical, or computer engineering (or a closely related field) with at least
`
`two years of experience working with software for motion control systems, or (iii) a
`
`PhD in computer science, or electrical, or computer engineering (or a closely related
`
`field).1
`
`37.
`
`In my opinion, the level of ordinary skill in the art would have been the same in
`
`in May 1995 (and at any time between July 1994 and May 1995).
`
`38.
`
`In opining on the level of ordinary skill in the art, I have considered the
`
`following factors: (i) the education level of the inventor; (ii) the type of problems
`
`encountered in the art; (iii) prior art solutions to those problems; (iv) the rapidity with
`
`
`1 Although I have applied this level of ordinary skill in analyzing the obviousness
`issues, it is my opinion that claims 1-10 are, for the reasons set forth below, so clearly
`obvious that even a person of lesser skill would have found them obvious.
`
`
`
`
`Page 12 of 69
`
`
`
`
`
`which innovations are made; (v) the sophistication of the technology; and (vi) the
`
`education level of active workers in the field.
`
`39. Based on my experience and education, I consider myself to be an expert in the
`
`field of technology implicated by the patent.
`
`B.
`Scope and Content of the Prior Art
`40. The scope and content of the prior art as of July 1994 would have broadly
`
`included patents and publications regarding motion control systems as well as
`
`software engineering and control (regardless of whether specifically applied in robots
`
`or otherwise).
`
`41.
`
`In my opinion, the references disclosed below would all have been considered
`
`to be within the same technical field as the subject matter of the ’236 patent.
`
`Furthermore, all of these references would be considered highly relevant prior art to
`
`claims 1-10 of the ’236 patent.
`
`42. My opinion is the same with respect to the scope and content of the prior art as
`
`of July 1994 and any time between July 1994 and May 1995.
`
`C.
`43.
`
`List of Prior Art References Discussed in Analysis
`
`In my analysis, I discuss the following references, which I introduce here to
`
`provide abbreviations. I understand that all of these references qualify as prior art to
`
`the identified claims of the ’236 patent.
`
`1) Gertz, M.W., A Visual Programming Environment for Real-Time Control
`
`Systems. Ph.D. dissertation, Carnegie Mellon University, Nov. 22, 1994 (“Gertz”).
`
`
`
`
`
`Page 13 of 69
`
`
`
`
`
`2) Stewart, D.B., Real-Time Software Design and Analysis of Reconfigurable
`
`Multi-Sensor Based Systems. Ph.D. dissertation, Carnegie Mellon University, April 1,
`
`1994 (“Stewart”).
`
`3) Morrow, J. Dan; Nelson, Bradley J.; and Khosla, Pradeep, Vision and Force
`
`Driven Sensorimotor Primitives for Robotic Assembly Skills, Institute for Software
`
`Research, paper 574, January 1, 1995 (“Morrow”).
`
`4) Microsoft Press, MS Windows 3.1 Device Driver Adaptation Guide, ©
`
`1991, Chs. 1-2, 4, 10-12 (“DDAG”).
`
`5) Hewlett Packard, Interface and Programming Manual, HP 7550 Graphics
`
`Plotter, 3rd ed., 1986 (“HP86”).
`
`6) Brockschmidt, K., Inside OLE 2: The Fast Track to Building Powerful
`
`Object-Oriented Applications, Microsoft Press Programming Series, 1994
`
`(“Brockschmidt”).
`
` Overview of Stewart
`
`D.
`Stewart describes a software system (Chimera) that allowed the development of
`
`44.
`
`reconfigurable software for robotics. The approach had a wider applicability to other
`
`automation systems. Furthermore, the system enabled the creation of reusable
`
`software as well. The Advanced Manipulators Laboratory (AML) at Carnegie Mellon
`
`University experimented with many robotic platforms (e.g., DDArm II, Puma 560,
`
`
`
`
`
`Page 14 of 69
`
`
`
`
`
`etc.). These robotic platforms incorporated many motion control devices which
`
`move them.
`
`45. One initial challenge for researchers in the AML (e.g. Richard Volpe, Nikos
`
`Papanikolopoulos, Bradley Nelson, Richard Voyles, and Dan Morrow) was the lack of
`
`a software framework that would enable the writing of code at a higher level without
`
`worrying about the lower level drivers for the actuators, sensors, and other hardware
`
`components of the system. The sensors among others included force sensors, cameras
`
`that could be placed in an eye-in-hand configuration and/or on different locations
`
`around the manipulator. Stewart’s work enabled the members of the AML group to
`
`focus on writing high level application programs like visual servoing ones (Stewart
`
`Figures 3.15 and 3.16 and a reference [46] to my servoing work). It should be noted
`
`that Figure 3.15 illustrates inverse kinematics, not inverse dynamics as the caption
`
`suggests.
`
`
`
`
`
`
`
`Page 15 of 69
`
`
`
`
`
`
`
`
`46. The aforementioned diagrams show the challenge that a lot of researchers
`
`faced at that time. They did not want to have to modify or rewrite the device drivers
`
`for the robot controllers or the vision subsystem.
`
`47.
`
`Stewart discloses a “port-based object” at the core of his approach. Port-based
`
`objects have been studied extensively in software engineering. (PortBasedObject,
`
`&1). A port-based object enabled the connection of different “control modules” in
`
`order to accomplish a particular goal. In fact, the ports enabled the input and output
`
`variables to enforce the right connectivity between the different port-based objects.
`
`48.
`
`Stewart discloses: “An application can be decomposed into multiple
`
`subsystems. Within a subsystem we defined a new software model called the port-
`
`based object, which combines the use of objects with the port-automaton theory. Our
`
`structure of the port-based object was presented in detail. The port-based object
`
`forms the heart of a reconfigurable task set, and is integrated through the use of a
`
`
`
`
`
`Page 16 of 69
`
`
`
`
`
`global state variable table mechanism. This mechanism allows for the multi-threaded
`
`real-time communication between multiple tasks on multiple processors.” (Stewart,
`
`&68). It is clearly a description of a decomposition that takes place to facilitate the
`
`reuse of software by a user who may not be aware of the intricate details of the low-
`
`level drivers.
`
`49. Like classical objected-oriented architectures, Stewart’s port-based objects
`
`encapsulate both the data that the object represents and the functions that manipulate
`
`the data. Stewart discloses: “The code for a control module is decomposed into
`
`several components, which are implemented as methods of the control object, or as
`
`subroutines if the control object is defined as an abstract data type. The components
`
`are init, on, cycle, off, kill, error, and clear.” (Stewart, &47). “Methods” and
`
`“subroutines” are examples of types of functions. Most of these functions derive
`
`from the object architecture. For example, Stewart discloses: “When using an object-
`
`oriented programming language such as C++ for implementation, the init component
`
`is the constructor method of the object.” (Stewart, &48). Stewart teaches that the
`
`“cycle” function performs the “work” defined by a specific object: “For as long as the
`
`task is in the ON state, the cycle component is executed every time the task receives a
`
`wakeup signal.” (Stewart, &50). It is of no consequence that the function is triggered
`
`by means of a signaling protocol. For example, in event driven programming (as
`
`found in Apple and Windows programs), active processes have an “event loop” that
`
`waits for an input signal of some type. Mouse clicks and keystrokes are examples of
`
`
`
`
`Page 17 of 69
`
`
`
`
`
`signals that the main event loop of a program will process typically by calling a
`
`function to perform the requested action.
`
`50. A person having ordinary skill would recognize that Stewart teaches that each
`
`port-based object has this cycle function, that the function has code that performs a
`
`specific task (such as performing calculations on input variables), and that this
`
`function is activated/called by another process.
`
`51. This is easily confirmed by reference to another publication of Dr. Stewart: “A
`
`port-based object has all the properties associated with standard objects, including
`
`internal state, code and data encapsulation, and characterization by its methods.”
`
`(StewartWorkshop, &2). Again, “method” is formal terminology for functions in
`
`object-oriented systems.
`
`52.
`
`Stewart discloses the operation “move to point x.” (Stewart, &39). The ’236
`
`patent alleges that the “MOVE RELATIVE” is a primitive operation. In addition,
`
`Stewart discloses operations like “Read” and “Write” (Stewart, &159). One with
`
`ordinary skill in the art might also consider these “Read” and “Write” operations as
`
`primitive ones.
`
`
`
`53.
`
`
`E. Overview of Gertz
`I have read the Declaration of David Stewart, Ph.D. and his cross-examination.
`
`Dr. Stewart’s summary of Gertz ignored several features that are relevant to the
`
`
`
`
`
`Page 18 of 69
`
`
`
`
`
`patentability of the claims. I provide a more detailed analysis below to address those
`
`issues.
`
`54. Gertz’s research in his thesis provided a visual tool that enabled engineers and
`
`users of robot control systems to develop complex systems without resorting to the
`
`tedious tasks of writing low-level code. His system was called “Onika.”
`
`55. Dr. Stewart inaccurately characterized Onika as “only” a visual programming
`
`environment based on simple configuration files, ignoring the many portions of the
`
`Gertz thesis that describe not only these capabilities of the program, but also actual
`
`demonstrations performed at Sandia National Laboratory and CMU along with user
`
`testing.
`
`56. Onika is an example of “flow based programming,” a technique developed
`
`more than forty years ago. (Morrison, &2,4). It is also a simulation and execution
`
`environment. (Gertz, &28-29, 29-30, 34, 54-56). That is, Onika can execute the
`
`programs that the user creates to control a device such as a PUMA arm.
`
`57. Matt Gertz utilized the Chimera environment as the backbone of his
`
`development work.
`
`58. Gertz discloses: “Any routine can be defined by (or decomposed into) a
`
`collection of common lower level routines: The software at any level of the system is
`
`reusable, modular, and generic, and can be combined logically with other such
`
`software to create higher level routines.” (Gertz, &120). The distinction between
`
`“lower level” and ‘higher level” routines for one with ordinary skill in the art leads to
`
`
`
`
`Page 19 of 69
`
`
`
`
`
`the concept of primitive and non-primitive operations of RGB.
`
`59. Gertz discloses the software architecture shown in Figure 1.
`
`
`
`60.
`
`Figure 1 in Gertz discloses three device drivers, one associated with a unique
`
`motion control device (an actuator). (Gertz, &42).
`
`61. Gertz, as shown in Figure 1, discloses “tasks,” which are at the lowest level of
`
`granularity. One with ordinary skill in the art would have understood that tasks
`
`mapped to core driver functions associated with primitive operations. (Gertz, &48).
`
`Gertz also discloses “configurations” which map to extended driver functions which
`
`are associated with non-primitive operations.
`
`62. Gertz also states: “A job is a high-level port-based object, which refers to a
`
`specific configuration of control tasks. Whereas lower-level objects have definite input
`
`
`
`
`
`Page 20 of 69
`
`
`
`
`
`and output ports based on state variables, these high-level port-based objects merely
`
`have ports to receive user-specified input. For example, a job which performs motion
`
`in joint space requires data specifying the endpoint of the trajectory.” (Gertz, &45).
`
`Of course, persons of ordinary skill in the art would understand that the “lower-level”
`
`objects are Stewart’s port-based objects.
`
`63. Gertz points to operations such as: “(Cartesian move to x, joint move to q,
`
`open gripper, and close gripper) were already available in the laboratory's user
`
`libraries, and had originally been assembled from reconfigurable software tasks within
`
`minutes using Onika's engineering-level interface.” (Gertz, &126). One with ordinary
`
`skill in the art would also consider an operation like “joint move to q” a primitive one
`
`similar to the “MOVE RELATIVE” in the ’236 patent.
`
`64. Dr. Stewart alleged that Onika’s ability to communicate with Chimera was
`
`limited, such that it could only create a “configuration file” that “needed” to be
`
`downloaded to Chimera. His opinion is incorrect for at least two reasons.
`
`65.
`
`First, Gertz discloses a mechanism for communication with Chimera for
`
`controlling the execution flow of the tasks. Gertz discloses read and write operations:
`
`“The functions enetSend(SFD,type,size,buffer) and
`
`enetReceive(SFD,&type,&size,buffer) are similar to (and in fact use) the standard
`
`Unix write() and read() commands.” (Gertz, &110).
`
`66.
`
`Second, Onika can use the same file system as Chimera or a remote one (Gertz,
`
`&70). Gertz states: “In the latter case, Onika runs faster, but must upload any
`
`
`
`
`Page 21 of 69
`
`
`
`
`
`executables it compiles to the Chimera file system.” (Gertz, &70). Naturally, when
`
`the two have local access to the same file system, it is not necessary to use a file
`
`transfer protocol as Gertz teaches.
`
`67. Dr. Stewart suggested that Onika and Chimera could not have been run on the
`
`same computer system. However, Chimera was typically run on a multiprocessor
`
`system that included a “host workstation” connected to a VME bus.
`
`(ChimeraManual, &27). (At his deposition, Dr. Stewart confirmed that he was an
`
`author of the Chimera Manual.) Examples of the host workstation include Sun 4 or
`
`Sun Sparcstation computers. (ChimeraManual, &27). Sun also provided X Windows
`
`software for these systems. For example, the Chimera package included an X
`
`Windows based application called Ugraph. (ChimeraManual, &50).
`
`68. The Onika GUI is implemented in X-Windows, and thus it could have been
`
`executed on the Chimera host workstation. Alternatively, one with ordinary skill in
`
`the art could have used an X Windows emulator to run Onika on the same computer
`
`with Chimera assuming that the necessary computational resources were available.
`
`These configurations are consistent with Onika and Chimera running on the same file
`
`system, contrary to Dr. Stewart’s testimony. (Gertz, &70).
`
`69. Gertz specifically explained how Onika and Chimera could be run on different
`
`computers at remote locations. (Gertz, &70-71).
`
`70.
`
`In fact, Matt Gertz demonstrated a working software product in various demos
`
`(e.g., NIST, Sandia Lab). Gertz discloses: “Onika has been demonstrated at the
`
`
`
`
`Page 22 of 69
`
`
`
`
`
`NASA Langley Research Center and in several joint Sandia National Laboratory and
`
`Carnegie Mellon University distributed laboratory demonstrations, and is in regular
`
`use at several outside laboratory sites, including Wright Patterson Air Force Base and
`
`NIST.” (Gertz, &22). This is in sharp contrast to the RGB efforts at the time.
`
`71. Gertz describes an extensive user testing in Chapter 6 of his thesis. (Gertz,
`
`&123-139). He states: “In this chapter, we present the results from our user tests of
`
`Onika. These tests were performed in order to determine the efficiency of Onika as a
`
`programming environment. In particul
Accessing this document will incur an additional charge of $.
After purchase, you can access this document again without charge.
Accept $ Charge
Still Working On It
This document is taking longer than usual to download. This can happen if we need to contact the court directly to obtain the document and their servers are running slowly.
Give it another minute or two to complete, and then try the refresh button.
A few More Minutes ... Still Working
It can take up to 5 minutes for us to download a document if the court servers are running slowly.
Thank you for your continued patience.
This document could not be displayed.
We could not find this document within its docket. Please go back to the docket page and check the link. If that does not work, go back to the docket and refresh it to pull the newest information.
Your account does not support viewing this document.
You need a Paid Account to view this document. Click here to change your account type.
Your account does not support viewing this document.
Set your membership
status to view this document.
With a Docket Alarm membership, you'll
get a whole lot more, including:
- Up-to-date information for this case.
- Email alerts whenever there is an update.
- Full text search for other cases.
- Get email alerts whenever a new case matches your search.
One Moment Please
The filing “” is large (MB) and is being downloaded.
Please refresh this page in a few minutes to see if the filing has been downloaded. The filing will also be emailed to you when the download completes.
Your document is on its way!
If you do not receive the document in five minutes, contact support at support@docketalarm.com.
Sealed Document
We are unable to display this document, it may be under a court ordered seal.
If you have proper credentials to access the file, you may proceed directly to the court's system using your government issued username and password.
Access Government Site
