Declaration of Richard Voyles Ph. D.
`
`
`
`Filed on behalf of ABB, Inc.
`
`By: Richard D. Mc Leod (Reg. No. 46,921)
`rick.mcleod@klarquist.com
`Klarquist Sparkman LLP
`One World Trade Center, Suite 1600
`121 S.W. Salmon Street
`Portland, Oregon 97204
`Telephone: (503) 595-5300
`Facsimile: (503) 595-5301
`
`
`
`
`
`
`UNITED STATES PATENT AND TRADEMARK OFFICE
`
`____________
`
`BEFORE THE PATENT TRIAL AND APPEAL BOARD
`
`____________
`
`ABB, INC.
`Petitioner
`
`v.
`
`ROY-G-BIV CORPORATION
`Patent Owner
`
`____________
`
`Trial No. IPR2013-00062 (joined with IPR2013-00282)
`Patent 6,516,236 B1
`
`____________
`
`DECLARATION OF RICHARD VOYLES, PH.D.
`PURSANT TO 37 CFR § 42.53
`
`
`
`1
`
`Page 1 of 30
`
`

`

`Declaration of Richard Voyles Ph. D.
`
`
`
`I, Richard Voyles, Ph. D., declare as follows:
`
`
`1. I am currently employed by the College of Technology at Purdue
`
`University as Associate Dean for Research. I have been a professor at
`
`Purdue University since July of 2013 and was previously a tenured
`
`associate professor of robotics and mechatronics at the University of
`
`Denver (2006 – 2013) and a tenured associate professor of computer
`
`science at the University of Minnesota (1997 – 2007). Over the past
`
`three years I have been concurrently serving as a “rotator” at the
`
`National Science Foundation in the capacity of lead Program Director
`
`for the National Robotics Initiative (NRI) in the Computer and
`
`Information Science and Engineering directorate. The NRI is a multi-
`
`agency initiative of the federal government and I have been leading it
`
`since it was announced by President Obama in June of 2011. This fall,
`
`I plan to transition to the Office of Science and Technology Policy to
`
`assume the position of Assistant Director for Robotics and Cyber-
`
`Physical Systems (concurrent with my role at Purdue). A copy of my
`
`c.v. is attached at the end of this declaration.
`
`2. In 1997, I received my Ph.D. in Robotics from Carnegie Mellon
`
`University. My thesis, supervised by Prof. Pradeep Khosla, was
`
`entitled “Toward Gesture-Based Programming: Agent-Based Haptic
`
`2
`
`Page 2 of 30
`
`

`

`Declaration of Richard Voyles Ph. D.
`
`
`
`Skill Acquisition and Interpretation” and focused on a method for
`
`programming robots by human demonstration, rather than explicit
`
`programming of software code. Instead, a computer system would
`
`extract the intention of the human user by observing a physical
`
`demonstration of the desired task through a variety of sensors and
`
`build software from existing primitives (by the Morrow definition) to
`
`autonomously execute the learned program. My work focused on
`
`tactile sensors, force sensors, and various sensors of motion control,
`
`rather than computer vision. (Sing Bing Kang and Brad Nelson were
`
`working on vision problems in the lab.)
`
`3. I made extensive use of Chimera and Onika during my time in the lab
`
`and the demonstration of my thesis work used the tools provided by
`
`David Stewart and Dan Morrow, as well as my own software code.
`
`4. In 1989, I received my M.S. in Manufacturing Systems Engineering
`
`from the Mechanical Engineering Department of Stanford University,
`
`during which I worked in the robotics lab of Oussama Khatib. My
`
`B.S. in Electrical Engineering was received in 1983 from Purdue
`
`University in West Lafayette, IN where I worked in the robotics lab of
`
`Richard Paul.
`
`3
`
`Page 3 of 30
`
`

`

`Declaration of Richard Voyles Ph. D.
`
`
`
`5. Over the last 30 years, my research and teaching work has focused on
`
`robotics, real-time systems, mechatronics, computer engineering,
`
`sensors, artificial intelligence and cyber-physical systems. As part of
`
`my research work since graduating from Carnegie Mellon, I have
`
`developed a software package called “PBO/RT” (Port-Based
`
`Objects/Real-Time) which is a direct descendant of Chimera 3.
`
`6. Interspersed with my research and teaching experience over the past
`
`30 years, I have also gained significant experience in nearly all
`
`aspects of the commercial motion control sector as employee and
`
`entrepreneur . I have worked on the design and manufacture of low-
`
`level motor controllers, the development of multi-axis motion
`
`controllers, the development and application of hardware/software
`
`systems for the rapid prototyping of real-time motion control
`
`solutions, and the integration of robotic manipulators into assembly
`
`lines.
`
`Assignment and Compensation
`
`
`7. I submit this declaration to oppose the Patent Owner’s Responses filed
`
`by RGB in the Inter Partes Review of U.S. Patent No. 5,516,236 (“the
`
`’236 patent”), which includes Trial Nos. IPR2013-0062 and IPR2013-
`
`00282.
`
`4
`
`Page 4 of 30
`
`

`

`8. I am not an employee of Petitioner ABB or any affiliate or subsidiary
`
`Declaration of Richard Voyles Ph. D.
`
`
`
`thereof.
`
`9. I am being compensated for my time at my usual consulting rate of
`
`$500 per hour. My compensation is not dependent upon the substance
`
`of my testimony, or upon the outcome of this proceeding.
`
`10. I have been retained by ABB, Inc. to review and discuss certain
`
`factual events relating to research that was conducted at Carnegie
`
`Mellon University’s Advanced Manipulators Lab. I have also been
`
`asked to provide opinions regarding the testimony of David Stewart,
`
`Ph.D. that has been given in this proceeding.
`
`Materials Reviewed
`
`
`11. In preparing this declaration, I have reviewed ABB’s Petition for
`
`Inter Partes Review, ABB’s 2nd Petition for Inter Partes Review,
`
`PTAB Decisions Instituting Trial on claim 1-10, the Patent Owner’s
`
`Responses, Declaration of David Stewart, Ph. D. (and his
`
`Supplemental Declaration), Declaration of David Brown, Deposition
`
`Transcript of David Stewart, and the prior art that has been cited
`
`against the claims.
`
`12. I have further relied on a number of contemporaneous documents that
`
`reflect the state of the art prior to July 10, 1994, the research being
`
`5
`
`Page 5 of 30
`
`

`

`Declaration of Richard Voyles Ph. D.
`
`
`
`conducted at CMU’s Advanced Manipulator Lab and other facilities,
`
`and my experience in the field.
`
`13. Prior to 1995, Dr. Pradeep Khosla was the director of the AML. He
`
`supervised the research conducted there. I joined CMU in 1990. The
`
`students in the lab included Matt Gertz, David Stewart, Brad Nelson,
`
`Dan Morrow, Nikos Papanikolopoulos, and others. Many of these
`
`people are named in the acknowledgments in the Gertz and Stewart
`
`theses.
`
`14. I had numerous conversations with the above persons prior to and
`
`during 1994 and 1995. Not all of the students in the AML were
`
`conducting research on problems related to kinematic theory or
`
`applications.
`
`15. For example, David Stewart was insulated from the primary research
`
`into advanced manipulators that was the focus of the bulk of the work
`
`in the lab. By this, I mean he did not have a background in
`
`kinematics, robotic sensors, or visual servoing nor was he considered
`
`a user of robotic manipulators. He was the primary architect of the
`
`real-time software environment that provided the infrastructural
`
`support to most of the rest of the researchers in the lab.
`
`6
`
`Page 6 of 30
`
`

`

`Declaration of Richard Voyles Ph. D.
`
`
`
`16. Likewise, Matt Gertz was a software developer working on non-real-
`
`time programming environments and did not have a background in
`
`robotic manipulators.
`
`17. The AML functioned symbiotically in that the robotics researchers
`
`made use of the programming environments developed by Gertz and
`
`Stewart and they incorporated the feedback of the robotics experts and
`
`Dr. Khosla, himself, into their work. Most of the researchers in the lab
`
`used Chimera, while few researchers used Onika. I was the most
`
`experienced user of Onika in the lab.
`
`
`
`18. While Dr. Stewart stated that he could see no motivation to combine
`
`the teaching of Gertz and Morrow, I personally suggested this to Dan
`
`Morrow while I was studying at CMU as detailed below.
`
`19. Using a real-time operating system was not a novel concept in 1992,
`
`nor was the concept of visual programming. Prior to joining CMU, I
`
`had worked at Integrated Systems, Inc., where we used our own
`
`RTOS, tightly coupled to a multiprocessor computer system, and
`
`visual programming environment for robotic applications.
`
`20. Integrated Systems had commercial products which included
`
`MatrixX, a competitor to Matlab, SystemBuild, a graphical
`
`7
`
`Page 7 of 30
`
`

`

`Declaration of Richard Voyles Ph. D.
`
`
`
`programming environment for complex real-time systems, and
`
`AutoCode, a code generator for real-time applications that executed
`
`on the custom multiprocessor computer system.
`
`21. AutoCode was a software product that automatically compiled real-
`
`time executable code for the custom multiprocessor system (similar to
`
`the Chimera real-time execution environment) from collections of
`
`configuration files that consisted of input/output specifications
`
`(similar to Onika) and software templates for primitive functions
`
`(similar to Chimera port-based object modules), plus associated
`
`libraries and object files. Taken together, these configuration files
`
`comprised the “software code” of the real-time executable. Similarly,
`
`the host workstation provided the non-real-time control of software
`
`execution.
`
`22. Although the system from Integrated Systems compiled its software
`
`codes into executables and Onika’s software codes were interpreted at
`
`run-time, this distinction does not disqualify a particular file from
`
`containing “software code.” Many popular languages can be
`
`interpreted at run-time, as opposed to being compiled. “Software
`
`code” instantiates a program based on a formal language. The Onika
`
`visual programming environment employs a formal language and,
`
`8
`
`Page 8 of 30
`
`

`

`Declaration of Richard Voyles Ph. D.
`
`
`
`therefore, to a computer scientist of ordinary skill, produces software
`
`code.
`
`23. I brought a prototype of the Integrated Systems real-time
`
`multiprocessor computer system with me to CMU around 1991.
`
`Integrated Systems, Inc. was subsequently purchased by WindRiver
`
`Systems. WindRiver was a commercial competitor that licensed its
`
`own RTOS, VxWorks, for robotic and other applications.
`
`24. Rather than purchase VxWorks licenses, Dr. Khosla directed the
`
`development of an alternative RTOS, which led to the creation of
`
`Chimera II and Chimera 3 systems. Low-level access to the internals
`
`of the OS was necessary for the work of David Stewart.
`
`25. In addition, by 1991, Dr. Khosla had already invested considerable
`
`effort into the development of the Chimera hardware/software system
`
`and was not interested in purchasing the commercially-available
`
`hardware/software system from Integrated Systems due to its
`
`prohibitively high cost. This system, apparently still available today
`
`from National Instruments, incorporated a similar notion of
`
`“primitive” motion control functions and composable “non-
`
`primitives” made up of collections of the basic primitives that were
`
`automatically assembled into software code and, through selectable
`
`9
`
`Page 9 of 30
`
`

`

`Declaration of Richard Voyles Ph. D.
`
`
`
`device-specific software drivers, executed on a real-time
`
`multiprocessor to control a selectable variety of mechanical systems
`
`through a sequence of motion commands.
`
`26. Although Integrated Systems did not use the “primitive” and “non-
`
`primitive” terminology, they implemented the complete range of non-
`
`decomposable mathematical functions mirroring the functionality of
`
`the popular Matlab software product, which constitute the primitives.
`
`From these basic primitive functions, they also provided the ability to
`
`assemble, via “Super Blocks,” any conceivable non-primitive function
`
`for motion control or otherwise.
`
`27. The primary architect of the system from Integrated Systems was an
`
`engineer of ordinary skill by the name of Dipak Patel who neither held
`
`a PhD nor was a PhD candidate during the development of this
`
`hardware/software system. (Though the system implementation was a
`
`group effort with many significant contributors.)
`
`28. The above description highlights an important problem with the
`
`definition of a “primitive” employed by the 236 Patent and Dr.
`
`Stewart’s subsequent overly narrow interpretation of a primitive in his
`
`declaration (exhibit 2011). The Patent asserts that, for example, a
`
`MOVE_RELATIVE command is a primitive and is “necessary for
`
`10
`
`Page 10 of 30
`
`

`

`Declaration of Richard Voyles Ph. D.
`
`
`
`motion control.” To a person of ordinary skill in the field, this
`
`statement is untrue as a general proposition. Further, if it assumed to
`
`be true, this would be seen as an arbitrary definition with limited and
`
`subjective applicability.
`
`29. For example, MOVE_RELATIVE can be emulated if the commands
`
`GET_POSITION and MOVE_ABSOLUTE are available. Likewise,
`
`MOVE_ABSOLUTE can be emulated with the commands
`
`GET_POSITION and MOVE_RELATIVE. If both are available,
`
`neither can be primitive, assuming the “non-emulation” part of the
`
`definition of primitive operation.
`
`30. Furthermore, GET_POSITION is a low-level command that reads a
`
`sensor. It is natural (and common, but not necessary) that the
`
`complementary command SET_TORQUE would similarly be
`
`available. If GET_POSITION and SET_TORQUE are available, both
`
`MOVE_RELATIVE and MOVE_ABSOLUTE can be emulated
`
`regardless of the availability of the other.
`
`31. In my opinion, the 236 Patent provides no meaningful basis for what
`
`it might mean for a “primitive” to be “necessary for motion control”
`
`other than to provide examples that are self-contradictory, as noted
`
`above.
`
`11
`
`Page 11 of 30
`
`

`

`Declaration of Richard Voyles Ph. D.
`
`
`
`32. Nevertheless, I have considered the cited art under the constructions
`
`that have been adopted for “primitive operation” by the Board and the
`
`district court.
`
`33. Even given the arbitrary and narrow definition of “primitive” of the
`
`236 Patent, I disagree with Stewart’s conclusion that, “The operation
`
`performed by a control task in the Gertz Reference is not a “primitive
`
`operation”…” (#35). In the broader sense, it appears that “primitives”
`
`are simply an arbitrary subset of the commands the controller
`
`understands. Under this interpretation, the Chimera system would be
`
`the analog to the “controller” mentioned in the 236 Patent and the
`
`“control tasks” running on the Chimera system would represent the
`
`types of commands the “controller” understands.
`
`34. If “GET_POSITION” is assumed to be primitive (as asserted by
`
`RGB and the Patent), then the “Sensor Modules” disclosed in
`
`Stewart’s thesis, Fig. 3.14 are examples of a “GET POSITION”
`
`operation, while the “torque mode robot interface” would correspond
`
`to a SET_TORQUE operation (sending a “raw torque command”) to a
`
`robot. Stewart states: “Onika,… uses the software framework
`
`described in this dissertation…and uses hypermedia techniques for
`
`searching and selecting modules from them” (Stewart at 64). In
`
`12
`
`Page 12 of 30
`
`

`

`Declaration of Richard Voyles Ph. D.
`
`
`
`reference to an actual test of Onika, Gertz stated: “The only module
`
`which needed to be created for the mobile manipulator was the one
`
`that actually communicated with the robot’s hardware; other modules,
`
`such as trajectory, kinematics, and visual servoing modules, were
`
`already available.” (Gertz at 72). As Gertz provide a wrapper for any
`
`module and given the examples that are stated in the Patent, Gertz
`
`teaches at least one “primitive operation.”
`
`35. While Chimera simplified the development of real-time software code
`
`and facilitated the sharing of reusable code modules for control
`
`engineers, Onika simplified the programming of applications for
`
`lesser skilled users. Onika was an attempt to address what many of the
`
`robotics experts in the lab began to refer to as the “fourth level” (and
`
`higher) programming and execution environment. It was referred to as
`
`such because Chimera 3 provided effective tools for three levels: I/O
`
`drivers, sensor/actuator interface, and port-based objects, all in the
`
`real-time domain. The “fourth level,” as envisioned by the robotics
`
`researchers, represented a transition from the time-deterministic, hard-
`
`real-time, primarily periodic components of the module level to the
`
`soft-real-time, primarily state-based applications of the real world.
`
`13
`
`Page 13 of 30
`
`

`

`Declaration of Richard Voyles Ph. D.
`
`
`
`36. Dan Morrow investigated sensorimotor primitives for robotic tasks.
`
`His work was aimed at robotics engineers trying to develop useful
`
`tasks and applications and attempted to address the “fourth level”
`
`head-on.
`
`37. Initially, Onika and Chimera 3 were inadequate for fully addressing
`
`the “fourth level” as Chimera was focused on lower levels and Onika
`
`was focused on higher levels. I urged Matt Gertz to include more
`
`state-based control constructs into Onika. He incorporated if-then and
`
`loop constructs, which are Turing equivalents to what we needed. (By
`
`Turing equivalent, I mean that Onika had sufficient functionality to
`
`perform generic computation and could emulate state-based control,
`
`though in a somewhat cumbersome manner.) As stated in section
`
`4.4.7.2 IF/THEN/ELSE of the Gertz thesis, “Onika fully supports this;
`
`in fact, in Onika conditionals are implemented as “case” statements,
`
`making them more general. When a conditional application icon is
`
`encountered within a higher-level application, the return value from
`
`the previous action is used to choose which “case” is followed at run-
`
`time.” (This is in direct contradiction to Dr. Stewart’s declaration, ¶
`
`21).
`
`14
`
`Page 14 of 30
`
`

`

`Declaration of Richard Voyles Ph. D.
`
`
`
`38. The use of “case statements” to implement Onika’s conditionals was
`
`explicitly employed to allow for future expansion of state-based, run-
`
`time control, as noted by Gertz: “Future work on Onika will enhance
`
`the implementation and presentation of conditionals.” He intended
`
`them to be more general and, in fact, were implemented in a more
`
`general way, as noted above.
`
`39. Onika’s state-based control structures (Gertz, § 5.9 at 115-118) permit
`
`a type of programming called “flow based programming.” As such,
`
`Onika provides both a programming and execution environment
`
`(Gertz at 29-30), and the “visual programs” created by Onika would
`
`be considered examples of both “computer code” and “software code”
`
`as those terms are broadly understood in computer science, contrary to
`
`Dr. Stewart’s declaration.
`
`40. I specifically encouraged Dan Morrow to work with Matt Gertz.
`
`Additionally, I urged Dan Morrow to use Onika in his research.
`
`41. Ultimately, Dr. Morrow wrote his own “fourth level” environment,
`
`the “Agent Level” which is discussed in his own Ph.D. thesis. See Ex.
`
`1031, § 9.1, Fig. 1, and comparing it to Onika, § 9.1.6.
`
`42. Within the context of Prof. Khosla’s lab, my work, Morrow’s work,
`
`Gertz’ work and Stewart’s work were all various forms of
`
`15
`
`Page 15 of 30
`
`

`

`Declaration of Richard Voyles Ph. D.
`
`
`programming systems. I think they all would be recognized as such by
`
`the majority of computer scientists because they implement formal
`
`languages and, therefore, all represent systems that produce “software
`
`code”.
`
`43. In fact, most computer scientists recognize the Jacquard Loom as an
`
`implementation of one of the earliest programming languages and in
`
`1801, it long pre-dated computers as Stewart refers to them. However,
`
`in 1994, CNC machines used computers to interpret “paper tape” that
`
`are, in concept, nearly identical to the programs of the Jacquard Loom
`
`and this form of programming is certainly considered “software code”
`
`in my opinion. It is a form of programming language for motion
`
`control that is even more restrictive than the “software code” that
`
`Onika implements.
`
`44. Therefore, I feel it is far too restrictive to a person of ordinary skill to
`
`say there is a distinction between “software code” and “downloaded
`
`configuration files” in reference to that which Onika produces.
`
`45. Furthermore, Onika had several functionalities which included a run-
`
`time control facility that implemented optional conditional and
`
`looping programming constructs, as mentioned above. These elements
`
`of the realized “application flows” were executed from the Onika
`
`16
`
`Page 16 of 30
`
`

`

`Declaration of Richard Voyles Ph. D.
`
`
`
`environment on a workstation remote from, but networked to, the
`
`Chimera run-time environment.
`
`46. Like the Jacquard Loom or CNC machine, Onika implements an
`
`optional real-time interface that can start and stop programs that have
`
`been “downloaded” to their respective real-time execution
`
`environments. (A Chimera system represents the real-time execution
`
`environment in the case of Onika.) However, Onika could do much
`
`more than a CNC or Loom. The Onika run-time environment could
`
`actually monitor and alter the program flow during execution.
`
`47. These are the state-based control structures referenced above and
`
`represent the primary commonality between the Gertz and Morrow
`
`work.
`
`
`
`
`17
`
`Page 17 of 30
`
`

`

`Declaration of Richard Voyles Ph. D.
`
`I declare that all statements made herein of my own knowledge are true; that
`all statements made on information and belief are believed to be true; and
`further that these statements were made with the knowledge that willful false
`statements and the like are punishable by fine or imprisonment, or both,
`.
`.C. § 1001 and that such willful false statements may
`
`
`:2 flay/4W 10/2518
`
`char Voyles, Ph. D.
`
`Date
`
`18
`
`Page 18 of 30
`
`Page 18 of 30
`
`

`

`Richard M. Voyles, Ph.D.
`Purdue University:
`401 N. Grant Street
`W. Lafayette, IN 47907-2021
`cell: (651) 285-1079
`rvoyles@purdue.edu
`
`National Science Foundation
`4201 Wilson Blvd
`Arlington, VA 22230
`(703) 292-4541
`rvoyles@nsf.gov
`
`EDUCATION:
`
`Ph.D.
`Carnegie
`Mellon
`University
`
`MSE
`Stanford
`BSEE
`Purdue
`
`Robotics Ph.D. Program (School of Computer Science), 9/90 - 8/97. My thesis focused
`on Gesture-Based Programming for robotic contact tasks, a paradigm for non-textual
`programming by human demonstration. This required the design of novel capacitive
`tactile sensors and actuators and a powerful, autonomous calibration technique I call
`Shape from Motion Primordial Learning. My advisor was Pradeep Khosla. GPA = 3.7/4.0
`Manufacturing Systems Engineering (division of Mechanical Engineering), 9/87 - 6/89.
`Emphasis on robotics and controls as well as manufacturing. GPA = 3.8/4.0
`Electrical Engineering, 8/80 - 5/84. Completed course work for the BSEE in three years
`with GPA = 5.6/6.0. Left MSEE program when my advisor, Richard Paul, went to Penn.
`
`TEACHING INTERESTS:
`Robotics, Mechatronic Systems, Real-Time Systems, Embedded Systems, Intelligent Systems, Computer
`Architecture, Computer Engineering, MEMS, Manufacturing Methods, Entrepreneurship
`
`ACADEMIC POSITIONS:
`July 13 - present
`Purdue University, West Lafayette, IN
`Associate Dean for Research, College of Technology, and Professor of Electrical and Computer
`Engineering Technology.
`Dec 13 - Nov 14
`Office of Science and Technology Policy, White House
`Pending security clearance, I will serve as Assistant Director of Robotics and Cyber-Physical Sys-
`tems under the Technoogy and Innovation Division.
`Sept 11 - present
`National Science Foundation, Arlington, VA
`Program Director, Division of Information and Intelligent Systems, CISE. As the sole lead Program
`Director launching the new National Robotics Initiative, a $50M per year civilian robotics program
`focused on co-robots, it was my responsibility to coordinate the four federal agencies involved,
`develop detailed procedures for reviewing over 700 individual proposals, build a network of NSF
`PDs to recruit reviewers, and coordinate the decision and award process. This program alone
`accounted for almost 25% of the entire proposal volume of the IIS division and the total requested
`funding topped $1B. This was achieved without missing a single deadline and while I was carrying
`all the normal duties of a Robust Intelligence cluster PD, including CAREER proposals and the RI
`core program for robotics. In addition, I was nominated as one of three founding PDs charged with
`creating the Innovation-Corps program. Both the I-Corps and NRI were anounced by President
`Obama in 2011. I received three Director’s Awards for my efforts in 2012. I am a "rotator" at NSF,
`on leave from the University of Denver with 8 students under my advisorship.
`Sept. 10 - Sept 11
`National Science Foundation, Arlington, VA
`Program Director, Division of Computer and Network Systems, CISE. My responsibilities as Pro-
`gram Director included the Cyber Physical Systems program, Major Research Instrumentation, and
`Industry/University Cooperative Research Centers. I was a "rotator" on leave from the University
`of Denver.
`Sept. 06 - July 13
`University of Denver, Denver, CO
`Associate professor, with tenure, department of electrical and computer engineering with joint
`appointment in mechanical engineering. DU Site Director of the NSF Safety, Security, and Rescue
`Research Center through 2010. This center, which I founded at UMN, has expanded to include the
`University of Pennsylvania. Teaching Mechatronic Systems, Embedded Systems, Real-Time Sys-
`tems, Robotics, and Computer Organization
`Aug. 04 - Dec. 07
`University of Minnesota, Minneapolis, MN
`Associate professor, with tenure, computer science and engineering and Site Director of the NSF
`Safety, Security, and Rescue Research Center (on leave from 9/06 - 12/07). The NSF SSR-RC is an
`Industry/University Cooperative Research Center founded by myself and Dr. Robin Murphy that
`
`Page 19 of 30
`
`

`

`focuses on issues of homeland security and robotic search and rescue in collaboration with the Uni-
`versity of South Florida. I have created classes in Real-Time and Embedded Systems and Microas-
`sembly and Microfabrication.
`Sept. 97 - July 04
`University of Minnesota, Minneapolis, MN
`Assistant professor of computer science and engineering teaching courses on data structures, C++,
`robotics, real-time systems, pattern recognition, machine organization, computer architecture. The
`Real-Time Systems course was a new course created to fill industry and student demand. I have
`also adapted it for an executive Master’s program in software engineering. Class size has ranged
`from 6 to 195 students.
`May 86 - Dec. 86
`Broome Community College, Binghamton, NY
`Adjunct physics instructor teaching one lab and two lecture sections per semester to associate’s-
`and bachelor’s-degree candidates while full-time at IBM. I was fully responsible for developing
`syllabi, setting class policy, delivering lectures, and grading assignments.
`
`RESEARCH PROJECTS
`Heterogeneous Wireless Control Networks
`Wireless Sensor Networks have received much attention recently for their scalability and rapid
`deployability in applications requiring distributed, synchronized sensing such as environmental
`monitoring and surveillance. Based on homogeneous nodes of very low power, these systems are
`ideal for low bandwidth, long duration studies. However, Wireless Sensor Networks are limited by
`ad hoc programming methodologies, one-size-fits-all computing platforms and the lack of deter-
`minism across the network. We aim to build heterogeneous networks with nodes of vastly different
`capabilities programmed with an Embedded Virtual Machine paradigm and strict determinism
`maintained, in hardware, across the network.
`Structured Computational Polymers - Smart Materials
`Smart Meta-Materials promise a wave of new capabilities for the design of complex, cyber-physcal
`systems. We have begun prototyping 1-D and 2-D smart polymers incorporating organic sensors,
`organic actuators, and organic, printable electronics. An example application of such Scturctured
`Computational Polymers (SCP) is in active tethers for small, search and rescue robots. We have
`built an active tether, driven by the water hammer effect, out of a 1-D type of SCP. This novel actu-
`ation means adds motive force to a small robot and the embedded computation, in the form of a
`Synthetic Neural Network, predicts the direction of the impulse. Applications also exist for novel
`colonoscopy devices and self-locomoting fire hoses.
`TerminatorBot - Search and Rescue Robots
`The TerminatorBot (also known as CRAWLER) is a miniature crawling robot for search-and-res-
`cue and planetary exploration involving rough terrain. It is an outgrowth of the DARPA-sponsored
`Scout project, providing novel, mission-specific capabilities for heterogeneous robot teams. The
`CRAWLER (Cylindrical Robot for Autonomous Walking and Lifting during Emergency Response)
`is a unique design from the ground up that relies on a pair of arms for both locomotion and manip-
`ulation. Themes of research include mechanism design for operational constraints, locomotion gait
`development, gait self-adaptation, visual and force servoing, and terrain identification. The
`research has been partially funded through DARPA ($5M Distributed Robotics and $135K Self-
`Adaptive Software), a Dosdall Fellowship, and internal sources.
`Surface Mount Magnetics with 3-D Silicon Interconnect
`Surface Mount Optics is an approach to photonics manufacturing that achieves high-speed, sub-
`micron tolerance assemblies using existing, low-tolerance (10 microns) pick-and-place machines.
`The novel manufacturing approach is based on pre-alignment of optical parts with respect to high-
`accuracy, silicon-micromachined features embedded in standardized packaging. We analyzed the
`GigaHertz-level electrical performance of the resulting structures. The research has been partially
`funded by a leave of absence funded by Avanti Optics Corp., an Army phase I STTR ($30K) sub-
`contracted to Avanti Optics Corp., and generous donations of equipment from CyberOptics Corp.
`as well as internal sources. With the collapse of the large-scale photonics industry, we are now
`developing the technology as Surface Mount Magnetics for the disk drive industry. Heat assisted
`magentic recording is the future of the hard disk industry, which requires high-precision assembly
`of dissimilar materials.
`Smart Tupperware
`The intelligent home of the future will include ubiquitous computing devices that are virtually
`invisible to its human occupants, but are constantly improving their quality of life. I envision multi-
`purpose, over-sized, TerminatorBot-like devices that connect to video games for realistic flight
`simulators, can be ridden by children like intelligent bicycles, and haul materials like intelligent
`wheelbarrows. But this pervasive network of devices and capabilities holds many challenges in
`
`Page 20 of 30
`
`

`

`wireless networking, security, artificial intelligence, and user interfaces. A likely focal point of
`early adoption will be the kitchen. Through undergraduate projects, we have begun preliminary
`investigation of intelligent food containers for automating the maintenance of a grocery list. We
`have re-designed the TerminatorBot’s microcontroller, added short-range wireless capability, and
`embedded it in kitchen containers.
`Gesture-Based Programming
`Gesture-Based Programming is a form of programming by human demonstration that focuses on
`task experts rather than programming experts. The idea is to make programming accessible to lay
`people that understand how to achieve the desired task, rather than training an expert programmer
`how to do the desired task so that she can program it. My interest is specifically in the area of
`robotic contact-based tasks, so the forces of touch -- for both human and robot -- are important.
`This study has necessitated tactile sensor and actuator design, as well as novel learning paradigms.
`The system learns basic sensorimotor primitives from human demonstration, then learns to map the
`human demonstration of a task to the primitives the robot knows how to execute, thereby learning
`the task. Funding for this research was provided, in part, by a $17K Grant-in-Aid, a $135K contract
`from DARPA on Self-Adaptive Software, a $30K seed grant from DTC, and internal sources.
`
`DU RESEARCH FUNDING (PI ON OVER $3M IN FUNDING, CO-PI ON $767K)
`While an NSF PD, I am required to name an alternate PI to manage my grants, hence "PI-on-leave".
`
`Smart Tupperware: Low Power Conformable Displays for Kitchen Containers Sept 11 - Aug 12
`PI-on-leave of $28K grant to develop low power displays for TupperwareEarth -- an intelligent
`web of automous Smart Tupperware containers (SSR-RC).
`S