`
`
`
`Filed on behalf of ABB, Inc.
`
`By: Richard D. Mc Leod (Reg. No. 46,921)
`Rick.mcleod@klarquist.com
`Klarquist Sparkman LLP
`One World Trade Center, Suite 1600
`121 S.W. Salmon Street
`Portland, Oregon 97204
`Telephone: (503) 595-5300
`Facsimile: (503) 595-5301
`
`UNITED STATES PATENT AND TRADEMARK OFFICE
`
`____________
`
`BEFORE THE PATENT TRIAL AND APPEAL BOARD
`
`____________
`
`ABB, INC.
`Petitioner
`
`v.
`
`ROY-G-BIV CORPORATION
`Patent Owner
`
`____________
`
`Trial No. IPR2013-00062
`Patent 6,516,236 B1
`____________
`
`DECLARATION OF NIKOLAOS PAPANIKOLOPOULOS, PH.D.
`
`
`
`
`
`
`
`
`
`Page 1 of 69
`
`
`
`
`
`I, Nikolaos Papanikolopoulos, Ph.D., hereby declare and state as follows:
`
`I.
`1.
`
`BACKGROUND
`
`I am currently employed by the University of Minnesota as a Distinguished
`
`McKnight University Professor of Computer Science and Engineering. I have been a
`
`professor at the University of Minnesota (originally as an assistant professor, and then
`
`as an associate professor) since the Fall of 1992. Between Fall 2001 and Spring 2004,
`
`and between Fall 2010 and Spring 2013, I was the Director of Undergraduate Studies
`
`of the College of Science and Engineering.
`
`2.
`
`In 1992, I received my Ph.D. in Electrical and Computer Engineering from
`
`Carnegie Mellon University. My thesis, supervised by Prof. Pradeep Khosla, was
`
`entitled “Controlled Active Vision” and focused on using computer vision in a
`
`controlled fashion to monitor and manipulate objects in the environment. The
`
`computer vision was implemented on a camera and a computer system that processed
`
`the images and issued motion control commands to a robot manipulator. My group
`
`was the Advanced Manipulators Laboratory (AML). In 1988, I also received my M.S.
`
`in Electrical and Computer Engineering from Carnegie Mellon University. My B.S. in
`
`Electrical Engineering was received in 1987 from the National Technical University in
`
`Athens, Greece.
`
`3.
`
`Over the last twenty three years, my research and teaching work has focused on
`
`robotics, computer vision, real-time systems, and intelligent transportation systems.
`
`This research has included control and sensing software for robots like manipulators
`
`
`
`
`
`Page 2 of 69
`
`
`
`
`
`and mobile robots.
`
`4.
`
`I have founded a company named ReconRobotics Inc. which is one of the
`
`largest manufacturers in the world of miniature robots like the UMN Scout. More
`
`than 4,000 Scout robots have been deployed around the globe.
`
`5. My research in the early 1990’s focused on solving sensor-based control
`
`problems for robots including using sensory systems and algorithms to move a
`
`manipulator to detect an object, track it, estimate its position and orientation, and
`
`finally grasp it. Some of our efforts were described under the term “visual servoing.”
`
`Our robots where the algorithms were implemented ranged from a Direct Drive Arm
`
`(DDArmII) to a Puma 560 robot and the Chimera II system was used to coordinate
`
`the various software tasks. A screenshot of the pertinent system is shown in Figure 1.
`
`
`
`Figure 1: Experimental setup for the visual servoing task that used Chimera II.
`6. When I moved to the University of Minnesota, I continued the work and in
`
`fact I received funding from the Sandia Labs (the initial source was the Department of
`
`Energy). The same unit partially funded the Chimera and Onika development efforts.
`
`
`
`
`
`Page 3 of 69
`
`
`
`
`
`In particular, we developed a system (using a CCD camera) that could detect the
`
`depth of an object from a camera by moving the robot in a controlled way. We also
`
`created a robot-based system that could track non-rigid objects by using active
`
`deformable models.
`
`7.
`
`I currently teach three courses relating to intelligent systems: (i) CSci 5551
`
`Introduction to Intelligent Robotic Systems, (ii) CSci 5511 Artificial Intelligence, and
`
`(iii) CSci 5561Computer Vision.
`
`8. My research has produced more than 320 journal and conference publications.
`
`More than 70 publications are in refereed journals. Many of my publications relate to
`
`control software for motion control devices (including robots). Some examples
`
`include:
`
`1) Janssen, M., and Papanikolopoulos, N.P., “Utilizing Queued Actions to
`Increase Interaction Efficiency in Robot Control Interfaces”, Proceedings of
`the 21st Mediterranean Conference on Control and Automation (MED’ 2013),
`Platanias, Greece, 2013, pp 34-39.
`
`
`2) McMillen, C., Stubbs, K., Rybski, P., Stoeter, S., Gini, M., and
`Papanikolopoulos, N.P., “Resource Scheduling and Load Balancing in
`Distributed Robotic Control Systems”, Robotics and Autonomous Systems,
`Volume 44, No. 3-4, September 2003, pp 251-259.
`
`
`3) Rybski, P., Stoeter, S., Gini, M., Hougen, D., and Papanikolopoulos, N.P.,
`"Performance of a Distributed Robotic System Using Shared
`Communications Channels: A Framework for the Operation and
`Coordination of Multiple Miniature Robots", IEEE Trans. on Robotics and
`Automation, Volume 18, No. 5, October 2002, pp 713-727.
`
`
`4) Papanikolopoulos, N.P., and Khosla, P.K., “Adaptive Robotic Visual
`Tracking: Theory and Experiments”, IEEE Trans. on Automatic Control,
`Volume 38, No. 3, March 1993, pp 429 - 445.
`
`
`
`
`Page 4 of 69
`
`
`
`
`
`9.
`
`
`5) Papanikolopoulos, N.P., Khosla, P.K., and Kanade, T., “Visual Tracking
`of a Moving Target by a Camera Mounted on a Robot: A Combination of
`Control and Vision”, IEEE Trans. on Robotics and Automation, Volume 9,
`No. 1, February 1993, pp 14 - 35.
`
`
`As a result of my work and research, I am familiar with the design, control,
`
`operation and functionality of software for motion control devices including the ones
`
`used in robots (manipulators and mobile ones).
`
`10. A copy of my curriculum vitae is attached as included herewith.
`
`
`II. ASSIGNMENT AND COMPENSATION
`11.
`I submit this declaration to oppose the Patent Owner’s Responses filed by
`
`RGB in the Inter Partes Review of U.S. Patent No. 5,516,236 (“the ’236 patent”), which
`
`includes Trial Nos. IPR2013-0062 and IPR2013-00282.
`
`12.
`
`13.
`
`I am not an employee of Petitioner ABB or any affiliate or subsidiary thereof.
`
`I am being compensated for my time at my usual rate of $700 per hour. My
`
`compensation is in no way dependent upon the substance of the opinions I offer
`
`below, or upon the outcome of this inter partes review.
`
`14.
`
`I have been asked to provide certain opinions relating to the ’236 Patent and
`
`the prior art that has been cited against the patent. Specifically, I have been asked to
`
`provide my opinion regarding (i) the level of ordinary skill in the art to which the
`
`patent pertains, and (ii) the patentability of claims of the patent.
`
`15. The opinions expressed in this declaration are not exhaustive of my opinions
`
`
`
`
`
`Page 5 of 69
`
`
`
`
`
`on the patentability of any of the claims in the patent. Therefore, the fact that I do
`
`not address a particular point should not be understood to indicate any agreement on
`
`my part that any claim otherwise complies with the patentability requirements.
`
`16. The opinions expressed in this declaration are my personal opinions and do not
`
`reflect the views of University of Minnesota.
`
`III. LEGAL STANDARDS
`17.
`I have been informed and I understand that a patentability analysis is
`
`performed from the viewpoint of a hypothetical person of ordinary skill in the art. I
`
`understand that “the person of ordinary skill” is a hypothetical person who is
`
`presumed to be aware of the universe of available prior art as of the time of the
`
`invention at issue.
`
`18.
`
`I understand that a patent claim is invalid as anticipated when a single piece of
`
`prior art describes every element of the claimed invention, either expressly or
`
`inherently, and arranged in the same way as in the claim. For inherent anticipation to
`
`be found, it is required that the missing descriptive material is necessarily present in
`
`the prior art. I understand that, for the purpose of an inter partes review, prior art that
`
`anticipates a claim can include both patents and printed publications from anywhere
`
`in the world.
`
`19.
`
`I understand that some claims are written in dependent form, in which case
`
`they incorporate all of the limitations of the claim(s) on which they depend. I
`
`understand that there are different claim construction standards use by the Patent
`
`
`
`
`
`Page 6 of 69
`
`
`
`
`
`Office and the district courts in evaluating the scope of a patent claim. It is my
`
`understanding that the Patent Office uses the “broadest reasonable interpretation in
`
`light of the specification” in reviewing claims. It is also my understanding that the
`
`Patent Trial and Appeals Board has adopted specification constructions for certain
`
`claim terms. I am also aware that the district court has adopted certain constructions
`
`that are broader than those assigned by the Board.
`
`20.
`
`I understand that a patent claim is invalid as obvious if the subject matter of
`
`the claim as a whole would have been obvious to a person of ordinary skill in the art
`
`as of the time of the invention at issue. I understand that the following factors must
`
`be evaluated to determine whether the claimed subject matter is obvious: (1) the
`
`scope and content of the prior art; (2) the difference or differences, if any, between
`
`the scope of the claim of the patent under consideration and the scope of the prior
`
`art; and (3) the level of ordinary skill in the art at the time the patent was filed. Unlike
`
`anticipation, which allows consideration of only one item of prior art, I understand
`
`that obviousness may be shown by considering more than one item of prior art. I
`
`have also been informed and I understand that so-called objective indicia of non-
`
`obviousness, also known as “secondary considerations,” like the following are also to
`
`be considered when assessing obviousness: (1) commercial success; (2) long-felt but
`
`unresolved needs; (3) copying of the invention by others in the field; (4) initial
`
`expressions of disbelief by experts in the field; (5) failure of others to solve the
`
`problem that the inventor solved; and (6) unexpected results. I also understand that
`
`
`
`
`Page 7 of 69
`
`
`
`
`
`evidence of objective indicia of non-obviousness must be commensurate in scope
`
`with the claimed subject matter. It is my understanding that RGB has not presented
`
`evidence of “secondary considerations” in this matter.
`
`21. As an initial matter, I have been informed that claim terms may be written in
`
`means-plus-function format. In this situation, the means-plus-function claim terms
`
`cover the corresponding structure identified in the specification for performing the
`
`claimed function, and equivalents thereof.
`
`22.
`
`I have applied these principles with respect to my analysis set forth below.
`
`IV. BACKGROUND OF THE PATENT
`23. The ’236 patent generally describes a software framework for motion control
`
`devices/systems. The patent alleges that the framework could potentially enable users
`
`to write software for motion control devices without having to write low-level code.
`
`The framework includes an Application Programming Interface (API) and a Service
`
`Provider Interface (SPI). The API includes component functions while the SPI
`
`includes driver functions. The driver functions are classified as core and extended
`
`ones. (According to certain RGB documents, the architecture was derived from
`
`Microsoft’s Graphic Display Interface / Windows Open Services Architecture)
`
`(Ex. 1025; also IPR2013-00282 Ex. 1006).
`
`24. Core driver functions are associated with primitive operations while extended
`
`ones are associated with non-primitive operations.
`
`
`
`
`
`
`
`Page 8 of 69
`
`
`
`
`
`V.
`25.
`
`CLAIM CONSTRUCTION
`
`I have not performed my own independent claim construction analysis.
`
`Rather, I have been asked to apply the claim constructions that have been adopted by
`
`the Board in analyzing the patentability of the identified claims.
`
`26. As noted above, I have been informed that claim terms may be written in
`
`means-plus-function format. In this situation, the means-plus-function claim terms
`
`cover the corresponding structure identified in the specification for performing the
`
`claimed function and equivalents thereof.
`
`27.
`
`I have reviewed and evaluated the “meta” interpretations that David Stewart
`
`has presented in his declaration in this matter, particularly where he appears to have
`
`adopted claim interpretations that would be narrower than the “broadest reasonable
`
`interpretation” or otherwise in conflict with an interpretation adopted by the Board.
`
`VI. BACKGROUND ON THE STATE OF THE ART
`28. The following is a brief exemplary discussion of the state of the art prior to
`
`May 1995.
`
`29. During the last forty years, there has been a growing interest in programming
`
`environments for motion control devices. The variety of hardware involved, the
`
`complexity of sensors and actuators, the rare reuse of existing software, and the
`
`existence of legacy proprietary systems forced the researchers to develop new levels of
`
`abstraction that separate the user from the low-level details and hardware. The real-
`
`time requirements pushed these abstractions to levels that involve agnostic use of the
`
`
`
`
`
`Page 9 of 69
`
`
`
`
`
`low-level device drivers.
`
`30. Robotics is a prime candidate for testing the efficacy of some of the motion
`
`control systems abstractions. Robotics involves actuators of various types and the
`
`controllers are often ones that require calls to hardware components for which the
`
`user knows or understands very little.
`
`31. Researchers have tried to develop programming languages for robotics since
`
`the early days of the field. For example, VAL was first written in 1975 and run on a
`
`PDP 11/45 with a very limited set of capabilities. For example, motions from point to
`
`point were at the center of the effort. VAL was reworked in 1976 to include structures
`
`like complex trajectory generation. Other languages like TEACH and AML also
`
`surfaced. TEACH included ideas like concurrency and device independence. Many of
`
`these efforts met with little success since they were too much focused on the robot
`
`instead of focusing on the variety of tasks that a robot might have encountered.
`
`32. Task planning was introduced to address some of these issues. Robotics
`
`experts would try to encode with fine granularity all the possible scenarios that a robot
`
`might encounter. Task planning was effective up to a point. For example, rapidly
`
`evolving environments like the ones found outdoors would render some these plans
`
`unrealistic. Attempts to re-plan were computationally challenging since they required
`
`calculation of hundreds of parameters.
`
`33. Later efforts tried to introduce uncertainty in the underlying models. Although
`
`these efforts were somewhat successful, they required robust estimates for the
`
`
`
`
`Page 10 of 69
`
`
`
`
`
`uncertainty. Limitations in hardware created impediments in the way that these
`
`systems were implemented and operated.
`
`34. Additionally, there was extensive research that was performed with respect to
`
`the application of layered software development approaches that separated device
`
`drivers from high-level programming for motion control systems (robots being one
`
`large category capitalizing on advances in motion control). This was research
`
`published in a number of different articles. The list of publications that I used in this
`
`write-up is:
`
`1)
`
`Ingimarson, D.; Stewart, D.; and Khosla, P., Chimera 3.2: The Real-
`time Operating System for Reconfigurable Sensor-Based Control
`Systems, Program Documentation, Carnegie Mellon University,
`February 1995 (“ChimeraManual”). (Ex. 1133).
`2) Stewart, D.; and Khosla, P., The Chimera Methodology: Designing
`Dynamically Reconfigurable Real-time Software Using Port-Based
`Objects, Proceedings of the First Workshop on Object-Oriented
`Real-time Dependable Systems (WORDS 1994), October 1994,
`Dana Point, CA (“StewartWorkshop”). (Ex. 1134).
`3) Morrow, J.D., Sensorimotor Primitives for Programming Robotic
`Assembly Skills, Ph.D. dissertation, Carnegie Mellon University,
`May 1997 (“MorrowThesis”). (Ex. 1131).
`4) Morrison, J.P., Data Responsive Modular, Interleaved Task
`Programming System, IBM Technical Disclosure Bulletin, Vol. 13,
`No. 8, pp. 2425-2426, January 1971 (“Morrison”). (Ex. 1135).
`5) Ramamritham, K.; Stemple, D.; Briggs, D.; and Vinter, S., Privilege
`Transfer and Revocation in a Port-Based System, IEEE
`Transactions on Software Engineering, Vol. 12, No. 5, pp. 635-648,
`1986 (“PortBasedObject”). (Ex. 1136).
`
`
`
`
`
`
`
`Page 11 of 69
`
`
`
`
`
`
`
`VII. ANALYSIS
`A.
`35.
`
`Level of Ordinary Skill in the Art
`
`I have been asked to provide my opinion regarding the level of ordinary skill in
`
`the art in May 1995, which is the date that RGB alleges to be the effective filing date
`
`of the claims.
`
`36.
`
`It is my opinion that, in 1995, a person of ordinary skill in the art would have
`
`had one of the following: (i) a bachelor’s degree in computer science, or electrical, or
`
`computer engineering (or a closely related field) with at least four years of experience
`
`working with software for motion control systems, (ii) a master’s degree in computer
`
`science, or electrical, or computer engineering (or a closely related field) with at least
`
`two years of experience working with software for motion control systems, or (iii) a
`
`PhD in computer science, or electrical, or computer engineering (or a closely related
`
`field).1
`
`37.
`
`In my opinion, the level of ordinary skill in the art would have been the same in
`
`in May 1995 (and at any time between July 1994 and May 1995).
`
`38.
`
`In opining on the level of ordinary skill in the art, I have considered the
`
`following factors: (i) the education level of the inventor; (ii) the type of problems
`
`encountered in the art; (iii) prior art solutions to those problems; (iv) the rapidity with
`
`
`1 Although I have applied this level of ordinary skill in analyzing the obviousness
`issues, it is my opinion that claims 1-10 are, for the reasons set forth below, so clearly
`obvious that even a person of lesser skill would have found them obvious.
`
`
`
`
`Page 12 of 69
`
`
`
`
`
`which innovations are made; (v) the sophistication of the technology; and (vi) the
`
`education level of active workers in the field.
`
`39. Based on my experience and education, I consider myself to be an expert in the
`
`field of technology implicated by the patent.
`
`B.
`Scope and Content of the Prior Art
`40. The scope and content of the prior art as of July 1994 would have broadly
`
`included patents and publications regarding motion control systems as well as
`
`software engineering and control (regardless of whether specifically applied in robots
`
`or otherwise).
`
`41.
`
`In my opinion, the references disclosed below would all have been considered
`
`to be within the same technical field as the subject matter of the ’236 patent.
`
`Furthermore, all of these references would be considered highly relevant prior art to
`
`claims 1-10 of the ’236 patent.
`
`42. My opinion is the same with respect to the scope and content of the prior art as
`
`of July 1994 and any time between July 1994 and May 1995.
`
`C.
`43.
`
`List of Prior Art References Discussed in Analysis
`
`In my analysis, I discuss the following references, which I introduce here to
`
`provide abbreviations. I understand that all of these references qualify as prior art to
`
`the identified claims of the ’236 patent.
`
`1) Gertz, M.W., A Visual Programming Environment for Real-Time Control
`
`Systems. Ph.D. dissertation, Carnegie Mellon University, Nov. 22, 1994 (“Gertz”).
`
`
`
`
`
`Page 13 of 69
`
`
`
`
`
`2) Stewart, D.B., Real-Time Software Design and Analysis of Reconfigurable
`
`Multi-Sensor Based Systems. Ph.D. dissertation, Carnegie Mellon University, April 1,
`
`1994 (“Stewart”).
`
`3) Morrow, J. Dan; Nelson, Bradley J.; and Khosla, Pradeep, Vision and Force
`
`Driven Sensorimotor Primitives for Robotic Assembly Skills, Institute for Software
`
`Research, paper 574, January 1, 1995 (“Morrow”).
`
`4) Microsoft Press, MS Windows 3.1 Device Driver Adaptation Guide, ©
`
`1991, Chs. 1-2, 4, 10-12 (“DDAG”).
`
`5) Hewlett Packard, Interface and Programming Manual, HP 7550 Graphics
`
`Plotter, 3rd ed., 1986 (“HP86”).
`
`6) Brockschmidt, K., Inside OLE 2: The Fast Track to Building Powerful
`
`Object-Oriented Applications, Microsoft Press Programming Series, 1994
`
`(“Brockschmidt”).
`
` Overview of Stewart
`
`D.
`Stewart describes a software system (Chimera) that allowed the development of
`
`44.
`
`reconfigurable software for robotics. The approach had a wider applicability to other
`
`automation systems. Furthermore, the system enabled the creation of reusable
`
`software as well. The Advanced Manipulators Laboratory (AML) at Carnegie Mellon
`
`University experimented with many robotic platforms (e.g., DDArm II, Puma 560,
`
`
`
`
`
`Page 14 of 69
`
`
`
`
`
`etc.). These robotic platforms incorporated many motion control devices which
`
`move them.
`
`45. One initial challenge for researchers in the AML (e.g. Richard Volpe, Nikos
`
`Papanikolopoulos, Bradley Nelson, Richard Voyles, and Dan Morrow) was the lack of
`
`a software framework that would enable the writing of code at a higher level without
`
`worrying about the lower level drivers for the actuators, sensors, and other hardware
`
`components of the system. The sensors among others included force sensors, cameras
`
`that could be placed in an eye-in-hand configuration and/or on different locations
`
`around the manipulator. Stewart’s work enabled the members of the AML group to
`
`focus on writing high level application programs like visual servoing ones (Stewart
`
`Figures 3.15 and 3.16 and a reference [46] to my servoing work). It should be noted
`
`that Figure 3.15 illustrates inverse kinematics, not inverse dynamics as the caption
`
`suggests.
`
`
`
`
`
`
`
`Page 15 of 69
`
`
`
`
`
`
`
`
`46. The aforementioned diagrams show the challenge that a lot of researchers
`
`faced at that time. They did not want to have to modify or rewrite the device drivers
`
`for the robot controllers or the vision subsystem.
`
`47.
`
`Stewart discloses a “port-based object” at the core of his approach. Port-based
`
`objects have been studied extensively in software engineering. (PortBasedObject,
`
`&1). A port-based object enabled the connection of different “control modules” in
`
`order to accomplish a particular goal. In fact, the ports enabled the input and output
`
`variables to enforce the right connectivity between the different port-based objects.
`
`48.
`
`Stewart discloses: “An application can be decomposed into multiple
`
`subsystems. Within a subsystem we defined a new software model called the port-
`
`based object, which combines the use of objects with the port-automaton theory. Our
`
`structure of the port-based object was presented in detail. The port-based object
`
`forms the heart of a reconfigurable task set, and is integrated through the use of a
`
`
`
`
`
`Page 16 of 69
`
`
`
`
`
`global state variable table mechanism. This mechanism allows for the multi-threaded
`
`real-time communication between multiple tasks on multiple processors.” (Stewart,
`
`&68). It is clearly a description of a decomposition that takes place to facilitate the
`
`reuse of software by a user who may not be aware of the intricate details of the low-
`
`level drivers.
`
`49. Like classical objected-oriented architectures, Stewart’s port-based objects
`
`encapsulate both the data that the object represents and the functions that manipulate
`
`the data. Stewart discloses: “The code for a control module is decomposed into
`
`several components, which are implemented as methods of the control object, or as
`
`subroutines if the control object is defined as an abstract data type. The components
`
`are init, on, cycle, off, kill, error, and clear.” (Stewart, &47). “Methods” and
`
`“subroutines” are examples of types of functions. Most of these functions derive
`
`from the object architecture. For example, Stewart discloses: “When using an object-
`
`oriented programming language such as C++ for implementation, the init component
`
`is the constructor method of the object.” (Stewart, &48). Stewart teaches that the
`
`“cycle” function performs the “work” defined by a specific object: “For as long as the
`
`task is in the ON state, the cycle component is executed every time the task receives a
`
`wakeup signal.” (Stewart, &50). It is of no consequence that the function is triggered
`
`by means of a signaling protocol. For example, in event driven programming (as
`
`found in Apple and Windows programs), active processes have an “event loop” that
`
`waits for an input signal of some type. Mouse clicks and keystrokes are examples of
`
`
`
`
`Page 17 of 69
`
`
`
`
`
`signals that the main event loop of a program will process typically by calling a
`
`function to perform the requested action.
`
`50. A person having ordinary skill would recognize that Stewart teaches that each
`
`port-based object has this cycle function, that the function has code that performs a
`
`specific task (such as performing calculations on input variables), and that this
`
`function is activated/called by another process.
`
`51. This is easily confirmed by reference to another publication of Dr. Stewart: “A
`
`port-based object has all the properties associated with standard objects, including
`
`internal state, code and data encapsulation, and characterization by its methods.”
`
`(StewartWorkshop, &2). Again, “method” is formal terminology for functions in
`
`object-oriented systems.
`
`52.
`
`Stewart discloses the operation “move to point x.” (Stewart, &39). The ’236
`
`patent alleges that the “MOVE RELATIVE” is a primitive operation. In addition,
`
`Stewart discloses operations like “Read” and “Write” (Stewart, &159). One with
`
`ordinary skill in the art might also consider these “Read” and “Write” operations as
`
`primitive ones.
`
`
`
`53.
`
`
`E. Overview of Gertz
`I have read the Declaration of David Stewart, Ph.D. and his cross-examination.
`
`Dr. Stewart’s summary of Gertz ignored several features that are relevant to the
`
`
`
`
`
`Page 18 of 69
`
`
`
`
`
`patentability of the claims. I provide a more detailed analysis below to address those
`
`issues.
`
`54. Gertz’s research in his thesis provided a visual tool that enabled engineers and
`
`users of robot control systems to develop complex systems without resorting to the
`
`tedious tasks of writing low-level code. His system was called “Onika.”
`
`55. Dr. Stewart inaccurately characterized Onika as “only” a visual programming
`
`environment based on simple configuration files, ignoring the many portions of the
`
`Gertz thesis that describe not only these capabilities of the program, but also actual
`
`demonstrations performed at Sandia National Laboratory and CMU along with user
`
`testing.
`
`56. Onika is an example of “flow based programming,” a technique developed
`
`more than forty years ago. (Morrison, &2,4). It is also a simulation and execution
`
`environment. (Gertz, &28-29, 29-30, 34, 54-56). That is, Onika can execute the
`
`programs that the user creates to control a device such as a PUMA arm.
`
`57. Matt Gertz utilized the Chimera environment as the backbone of his
`
`development work.
`
`58. Gertz discloses: “Any routine can be defined by (or decomposed into) a
`
`collection of common lower level routines: The software at any level of the system is
`
`reusable, modular, and generic, and can be combined logically with other such
`
`software to create higher level routines.” (Gertz, &120). The distinction between
`
`“lower level” and ‘higher level” routines for one with ordinary skill in the art leads to
`
`
`
`
`Page 19 of 69
`
`
`
`
`
`the concept of primitive and non-primitive operations of RGB.
`
`59. Gertz discloses the software architecture shown in Figure 1.
`
`
`
`60.
`
`Figure 1 in Gertz discloses three device drivers, one associated with a unique
`
`motion control device (an actuator). (Gertz, &42).
`
`61. Gertz, as shown in Figure 1, discloses “tasks,” which are at the lowest level of
`
`granularity. One with ordinary skill in the art would have understood that tasks
`
`mapped to core driver functions associated with primitive operations. (Gertz, &48).
`
`Gertz also discloses “configurations” which map to extended driver functions which
`
`are associated with non-primitive operations.
`
`62. Gertz also states: “A job is a high-level port-based object, which refers to a
`
`specific configuration of control tasks. Whereas lower-level objects have definite input
`
`
`
`
`
`Page 20 of 69
`
`
`
`
`
`and output ports based on state variables, these high-level port-based objects merely
`
`have ports to receive user-specified input. For example, a job which performs motion
`
`in joint space requires data specifying the endpoint of the trajectory.” (Gertz, &45).
`
`Of course, persons of ordinary skill in the art would understand that the “lower-level”
`
`objects are Stewart’s port-based objects.
`
`63. Gertz points to operations such as: “(Cartesian move to x, joint move to q,
`
`open gripper, and close gripper) were already available in the laboratory's user
`
`libraries, and had originally been assembled from reconfigurable software tasks within
`
`minutes using Onika's engineering-level interface.” (Gertz, &126). One with ordinary
`
`skill in the art would also consider an operation like “joint move to q” a primitive one
`
`similar to the “MOVE RELATIVE” in the ’236 patent.
`
`64. Dr. Stewart alleged that Onika’s ability to communicate with Chimera was
`
`limited, such that it could only create a “configuration file” that “needed” to be
`
`downloaded to Chimera. His opinion is incorrect for at least two reasons.
`
`65.
`
`First, Gertz discloses a mechanism for communication with Chimera for
`
`controlling the execution flow of the tasks. Gertz discloses read and write operations:
`
`“The functions enetSend(SFD,type,size,buffer) and
`
`enetReceive(SFD,&type,&size,buffer) are similar to (and in fact use) the standard
`
`Unix write() and read() commands.” (Gertz, &110).
`
`66.
`
`Second, Onika can use the same file system as Chimera or a remote one (Gertz,
`
`&70). Gertz states: “In the latter case, Onika runs faster, but must upload any
`
`
`
`
`Page 21 of 69
`
`
`
`
`
`executables it compiles to the Chimera file system.” (Gertz, &70). Naturally, when
`
`the two have local access to the same file system, it is not necessary to use a file
`
`transfer protocol as Gertz teaches.
`
`67. Dr. Stewart suggested that Onika and Chimera could not have been run on the
`
`same computer system. However, Chimera was typically run on a multiprocessor
`
`system that included a “host workstation” connected to a VME bus.
`
`(ChimeraManual, &27). (At his deposition, Dr. Stewart confirmed that he was an
`
`author of the Chimera Manual.) Examples of the host workstation include Sun 4 or
`
`Sun Sparcstation computers. (ChimeraManual, &27). Sun also provided X Windows
`
`software for these systems. For example, the Chimera package included an X
`
`Windows based application called Ugraph. (ChimeraManual, &50).
`
`68. The Onika GUI is implemented in X-Windows, and thus it could have been
`
`executed on the Chimera host workstation. Alternatively, one with ordinary skill in
`
`the art could have used an X Windows emulator to run Onika on the same computer
`
`with Chimera assuming that the necessary computational resources were available.
`
`These configurations are consistent with Onika and Chimera running on the same file
`
`system, contrary to Dr. Stewart’s testimony. (Gertz, &70).
`
`69. Gertz specifically explained how Onika and Chimera could be run on different
`
`computers at remote locations. (Gertz, &70-71).
`
`70.
`
`In fact, Matt Gertz demonstrated a working software product in various demos
`
`(e.g., NIST, Sandia Lab). Gertz discloses: “Onika has been demonstrated at the
`
`
`
`
`Page 22 of 69
`
`
`
`
`
`NASA Langley Research Center and in several joint Sandia National Laboratory and
`
`Carnegie Mellon University distributed laboratory demonstrations, and is in regular
`
`use at several outside laboratory sites, including Wright Patterson Air Force Base and
`
`NIST.” (Gertz, &22). This is in sharp contrast to the RGB efforts at the time.
`
`71. Gertz describes an extensive user testing in Chapter 6 of his thesis. (Gertz,
`
`&123-139). He states: “In this chapter, we present the results from our user tests of
`
`Onika. These tests were performed in order to determine the efficiency of Onika as a
`
`programming environment. In particul