Ph.D. Dissertation
`
`A Visual Programming Environment for
`Real-Time Control Systems
`
`Matthew Wayne Gertz
`
`Submitted in partial fulfillment
`of the requirements for the degree of
`Doctor of Philosophy
`in Electrical and Computer Engineering
`
`Department of Electrical and Computer Engineering
`Carnegie Mellon University
`Pittsburgh, Pennsylvania 15213-3890
`
`November 22, 1994
`
`
`Copyright ©1994 Carnegie Mellon University
`
`ABB Inc.
`
`EXHIBIT 1002
`
`Page 1 of 198
`
`

`

`Page 2 of 198
`Page 2 of 198
`
`

`

`dedicated to the memory of my grandfather
`
`HARVEY H. GERTZ
`1916-1985
`
`who taught me that no matter how bad the weather is outside, the cows still need to be milked
`
`Page 3 of 198
`
`

`

`Page 4 of 198
`Page 4 of 198
`
`

`

`Table of Contents
`
`List of Figures............................................................................................. vii
`List of Code...................................................................................................xi
`List of Tables ............................................................................................. xiii
`List of Abbreviations...................................................................................xv
`Abstract..................................................................................................... xvii
`Acknowledgments ......................................................................................xix
`1. Introduction...............................................................................................1
`1.1 Overview.........................................................................................................1
`1.2 Motivation.......................................................................................................1
`1.3 Goals and Contributions .................................................................................3
`1.4 Organization of Thesis....................................................................................5
`2. Related Work.............................................................................................7
`2.1 Introduction.....................................................................................................7
`2.2 Textual Languages..........................................................................................7
`2.3 Visual Programming and Program Visualization ...........................................8
`2.4 Presentation of Graphics.................................................................................9
`2.5 Modeling Visual Programming.....................................................................11
`2.6 Syntax and Grammar ....................................................................................11
`2.7 Usability........................................................................................................13
`2.8 Modularity and Hypermedia.........................................................................13
`2.9 Environmental Interfaces..............................................................................14
`2.10 Summary.......................................................................................................14
`
`Page 5 of 198
`i
`
`

`

`3. Terminology.............................................................................................15
`3.1 Introduction...................................................................................................15
`3.2 Framework Terminology ..............................................................................15
`3.2.1 Control Modules ..................................................................................... 15
`3.2.2 Control Tasks.......................................................................................... 17
`3.2.3
`Task Libraries ......................................................................................... 18
`3.2.4 Configurations......................................................................................... 19
`3.2.5
`Jobs ......................................................................................................... 19
`3.2.6
`Targets..................................................................................................... 20
`3.2.7
`Job Dictionaries ...................................................................................... 20
`3.2.8 Actions .................................................................................................... 21
`3.2.9 Control Subsystems ................................................................................ 21
`3.2.10 Applications ............................................................................................ 22
`3.3 Summary.......................................................................................................22
`4. Overview of Onika ..................................................................................25
`4.1 Introduction...................................................................................................25
`4.2 Lower Level Interface...................................................................................26
`4.2.1 Combining task routines ......................................................................... 27
`4.2.2
`Task combination rules........................................................................... 31
`4.2.3 Creation of higher level routines............................................................. 33
`4.3 Upper Level Interface ...................................................................................33
`4.3.1 Combining job routines .......................................................................... 33
`4.3.2
`Icon combination rules............................................................................ 35
`4.4 Completeness of the Upper Level Language................................................38
`4.4.1 Block statements ..................................................................................... 39
`4.4.1.1 BEGIN-END........................................................................................... 39
`4.4.1.2 COBEGIN-COEND ............................................................................... 39
`Joint control statements........................................................................... 40
`4.4.2.1 DRIVE JOINT <i> of ARM <a> TO <theta> WITH DURATION=<d>..
`40
`4.4.2.2 DRIVE JOINT <i> of ARM <a> USING <p> TO <theta> WITH DURA-
`TION=<d>................................................................................................... 40
`End effector control (Cartesian control) ................................................. 40
`4.4.3.1 MOVE ARM <a> TO <x> WITH DURATION=<d> ........................... 40
`4.4.3.2 MOVE ARM <a> TO <x> WITH YVECTOR=<y>, DURATION=<d>..
`40
`4.4.3.3 MOVE ARM <a> TO <x> USING SENSOR AT <loc> WITH VELOCI-
`TY=<v>, DURATION=<d> ....................................................................... 41
`4.4.3.4 MOVE ARM <a> USING <xp> WITH DURATION=<d> .................. 41
`4.4.4 Base control ............................................................................................ 41
`4.4.5 Hand control............................................................................................ 41
`4.4.5.1 CLOSE HAND <a>................................................................................ 41
`4.4.5.2 OPEN HAND <a>.................................................................................. 41
`Pause control........................................................................................... 42
`4.4.6.1 PAUSE ARM <a> WITH DURATION=<d> ........................................ 42
`Flow Control Constructs......................................................................... 42
`Page 6 of 198
`
`4.4.2
`
`4.4.3
`
`4.4.6
`
`4.4.7
`
`ii
`
`

`

`4.4.7.1 GOTO <label>........................................................................................ 42
`4.4.7.2 IF/THEN/ELSE ...................................................................................... 42
`4.4.7.3 WHILE-DO ............................................................................................ 42
`4.5 Distributed Laboratories and Technology Transfer......................................43
`4.5.1 Overview................................................................................................. 43
`4.5.2
`Shared Hardware Resources ................................................................... 44
`4.5.3
`Shared Software Resources..................................................................... 46
`4.6 Summary.......................................................................................................47
`5. Implementation of Onika .......................................................................49
`5.1 Introduction...................................................................................................49
`5.2 Task Representation......................................................................................49
`5.2.1
`Loading tasks .......................................................................................... 50
`5.2.2 Manipulating tasks.................................................................................. 53
`5.2.3 Aliasing to local conventions.................................................................. 53
`5.3 Configurations...............................................................................................55
`5.3.1 Configuration formats............................................................................. 55
`5.3.2
`The starting configuration....................................................................... 55
`5.3.3
`Laws governing the placement of tasks.................................................. 57
`5.3.4 Reconfiguration....................................................................................... 59
`5.3.5 Rendering of tasks and connections........................................................ 61
`5.3.6
`Task manipulation and control................................................................ 65
`5.4 Job Dictionaries ............................................................................................66
`5.4.1
`Loading jobs and targets ......................................................................... 66
`5.4.2 Manipulating icons.................................................................................. 69
`5.5 Applications ..................................................................................................70
`5.5.1 Application formats ................................................................................ 70
`5.5.2 Application traversal............................................................................... 71
`5.5.3
`Laws governing the placement of icons.................................................. 72
`5.5.4
`Icon manipulation within an application................................................. 74
`5.5.5 Rendering of icons in applications.......................................................... 76
`5.6 Creating and Representing Icons ..................................................................76
`5.6.1
`Icon creation and modification ............................................................... 76
`5.6.2
`Job representation ................................................................................... 77
`5.6.3
`Target representation .............................................................................. 78
`5.6.4 Application representation...................................................................... 79
`5.7 User Preferences ...........................................................................................79
`5.7.1
`Syntax preferences.................................................................................. 80
`5.7.2 Basic preferences .................................................................................... 81
`5.7.3 Additional preferences............................................................................ 82
`5.8 Communication.............................................................................................83
`5.8.1
`The ENET socket package...................................................................... 83
`5.8.2 Onika-Chimera RTOS communication................................................... 84
`5.8.3
`Task-to-subsystem communication ........................................................ 86
`5.8.4 Hypermedia communication................................................................... 88
`Page 7 of 198
`iii
`
`

`

`6.2.3
`
`5.9 Execution of Applications.............................................................................89
`5.9.1
`Simulation and redirection of applications ............................................. 89
`5.9.2
`The execution loop.................................................................................. 90
`5.10 Error Handling ..............................................................................................93
`5.10.1 File errors................................................................................................ 93
`5.10.2 Network errors ........................................................................................ 93
`5.11 Portability of the Onika Algorithms to Other Systems.................................94
`5.12 Summary.......................................................................................................94
`6. User Testing .............................................................................................97
`6.1 Introduction...................................................................................................97
`6.2 Method ..........................................................................................................98
`6.2.1
`Subject demographics ............................................................................. 98
`6.2.2 Apparatus ................................................................................................ 99
`6.2.2.1 Onika....................................................................................................... 99
`6.2.2.2 Structural hardware................................................................................. 99
`6.2.2.3 Icons...................................................................................................... 100
`6.2.2.4 Analysis Software and Criteria............................................................. 100
`Procedure .............................................................................................. 102
`6.2.3.1 Preparation............................................................................................ 102
`6.2.3.2 Administration of tutorial ..................................................................... 103
`6.2.3.3 Warm-up............................................................................................... 103
`6.2.3.4 Assembly of structure........................................................................... 104
`6.2.4 Results................................................................................................... 104
`6.2.5
`Significance of the statistical data......................................................... 107
`6.3 Discussion...................................................................................................108
`6.3.1 Usability of Onika by non-programmers .............................................. 108
`6.3.2 Reduced set-up time.............................................................................. 109
`6.3.3
`Effects of cognitive style upon performance ........................................ 109
`6.3.4
`Performance motivation........................................................................ 111
`6.4 Summary.....................................................................................................112
`7. Summary, Contributions, and Future Work......................................115
`Appendix A: File Formats........................................................................119
`A.1 Task Parameter (.rmod) file ........................................................................119
`A.2 Task Configuration (.conf) file ...................................................................121
`A.3 Upper-level Icon (.onk) file.........................................................................122
`A.4 Application (.appl) file................................................................................123
`A.5 Preference file ($HOME/.onika).................................................................125
`A.6 Onika Default Functions File ($HOME/.onika.buttons).............................127
`A.7 The Syntax Definition File (.OnikaUpperSetUp.dta).................................127
`A.8 Start and Stop Icon Files.............................................................................129
`
`iv
`
`Page 8 of 198
`
`

`

`Appendix B: Mouse Clicks .......................................................................131
`B.1 The Task Library.........................................................................................131
`B.2 The Configuration Canvas ..........................................................................132
`B.3 The Job Dictionary......................................................................................133
`B.4 The Application Workspace .......................................................................134
`Appendix C: How to Create Tasks..........................................................137
`C.1 Trigger Modules..........................................................................................137
`C.2 SBS_OFF vs. errInvoke()............................................................................139
`C.3 Signal Response..........................................................................................140
`C.4 User Input (the UI packet) ..........................................................................141
`Appendix D: Future Hypermedia Engine Code.....................................145
`Appendix E: User Testing ........................................................................151
`E.1 Checklist .....................................................................................................151
`E.2 Raw Data.....................................................................................................161
`E.3 Processed Data............................................................................................163
`E.4 Timelined Data............................................................................................163
`Bibliography ..............................................................................................167
`
`Page 9 of 198
`v
`
`

`

`vi
`vi
`
`Page 10 of 198
`Page 10 of 198
`
`

`

`List of Figures
`
`Figure 1: The Reconfigurable Software Framework developed at Carnegie Mellon Univer-
`sity. ............................................................................................................16
`Figure 2: The relationship between tasks and modules. Note that the task can “alias’ the
`names of the internal variables of the modules to more preferred names. 18
`Figure 3: Clockwise from the top left: The Onika configuration canvas, the Onika task li-
`brary, and the Onika control panel. Tasks are presented to the user in the li-
`brary; a pop-up menu helps the user locate tasks in libraries containing
`many tasks. There is no limit on the number of open libraries within Oni-
`ka. ..............................................................................................................28
`Figure 4: The user has placed a PUMA simulator task onto the canvas. Tasks which would
`conflict with its operations are “greyed-out” in the library, to prevent the
`user from placing them. ............................................................................28
`Figure 5: Tasks are automatically connected together as they are placed into the configura-
`tion. ...........................................................................................................29
`Figure 6: The user has “cycled” most of the tasks. The final task is a trajectory task, and
`requires an endpoint and a duration to begin execution. ..........................30
`Figure 8: The Onika state variable display window. ........................................................30
`Figure 7: The Onika status window. .................................................................................31
`Figure 9: Tasks parameters can be changed easily within Onika. The names in the I/O ports
`can be aliased without needing to recompile code. ...................................32
`Figure 10: The user is saving the configuration of Figure 5 as a job, creating a picture for
`it. Onika has determined that this will be a “joint space job.” ..................34
`Figure 11: Entering the information for a joint target. ......................................................35
`Figure 12: The Onika job dictionary (inset: the Onika control panel). Job and target icons
`are sorted according to type, and presented to the user with a description of
`their function. ............................................................................................36
`
`List of Figures
`
`Page 11 of 198
`vii
`
`

`

`Figure 13: An application in the process of being built. The “go light” and “stop light”
`icons indicate the beginning and ending of the application, respectively. The
`job created in Figure 10 has been inserted into the application; since its right
`edge doesn’t match the left edge of the icon which follows it, a hole was in-
`serted. The hole must be filled by a joint target icon which starts out yellow
`and ends up red. ........................................................................................37
`Figure 14: A complete parallel application. The two jobs marked “b” are synchronized so
`that they will begin execution at the same time, as are those marked “a.” 38
`Figure 15: This application has a bottom-test loop tagged “assemble.” It also has three
`breakpoints within the loop so that the user can step through the jobs includ-
`ed in the loop. ............................................................................................39
`Figure 16: The distributed laboratory setup demonstrating control across a network. .....45
`Figure 17: An example of subset dynamic reconfiguration from a joint position control
`configuration to a Cartesian control configuration. The shaded tasks are
`common to both configurations, and thus were not removed during recon-
`figuration. ..................................................................................................60
`Figure 18: The ONIKON nodes for an application are linked in the order in which they were
`placed into the application, regardless of their location in the application.
`The ONIFLOW nodes are smaller and kept in the order in which they will
`be executed (i.e. the order that the user sees pictured in the application), and
`include blank nodes (such as the shaded one in this figure) to indicate places
`where “holes” occur. .................................................................................72
`Figure 19: Two applications have been included in a “super” application. One is incom-
`plete, and requires that the “super” application complete it with an object
`when it is used. The other application is complete, and requires no object
`when used. .................................................................................................73
`Figure 20: The use of ONIFLOW nodes anticipates the future development of graphically
`representing conditionals as a graph, wherein fast, concise routines will be
`needed to traverse and manipulate icons within a flow. ...........................73
`Figure 21: The user can define frequently-used system calls as buttons to be used from
`within Onika. .............................................................................................82
`Figure 22: Connecting to the real-time operating system. ................................................85
`Figure 23: In (a), the retrieval and sending of I/O to and from the path planner is completely
`controlled by the visual programming interface. If the path planner I/O pro-
`tocol is changed, then the visual programming interface itself must be
`changed. In (b), the path planner I/O is handled by the trajectory generator
`(a port-based object). If the path planner I/O is changed, then only the tra-
`jectory generator must be changed. ..........................................................87
`Figure 24: A depiction of the four boxes which the user programmed the robot to assemble
`in the user tests. Note that each box is labelled with a different symbol for
`Page 12 of 198
`List of Figures
`
`viii
`
`

`

`the purposes of identification, and that the top box has a light in its center
`which illuminates when the structure is assembled correctly. In this figure,
`the boxes are shown stacked in the correct order. ....................................99
`Figure 25: A graphical representation of a section of timeline data taken from a subject
`test. ..........................................................................................................102
`Figure 26: A comparison example of creating an Onika real-time pick-and-place applica-
`tion versus creating a similar application using traditional methods. No sim-
`ulation port is required for Onika, since physical code can be simulated
`without changes. Furthermore, unlike the textual case, the Onika code can,
`after completion, also be executed on another manipulator without changes.
`As noted previously, training times are omitted from this figure; this would
`be negligible in Onika compared to the days of training required for textual
`systems. ...................................................................................................110
`
`List of Figures
`
`Page 13 of 198
`ix
`
`

`

`x
`
`Page 14 of 198
`Page 14 of 198
`List of Figures
`List ofFigures
`
`

`

`List of Code
`
`Code 1: An example parameter file for a task. Onika uses these files to create icons “on the
`fly” for each task. ......................................................................................27
`Code 2: The structure of an Onika TASK node. The field show is only used when the task
`is in a library; the fields inleft, spawned, stask, and rtpu are used only when
`the task is in a configuration. ....................................................................52
`Code 3: The structure of an Onika CONFIG node. When used as a library node, only the
`fields prev, next, name, dir, task_header, task_tailer, selected, and button
`are used; the rest are useful only for configurations. ................................56
`Code 4: The structure of a LOCUS node, used for automatic connections between tasks. ..
`62
`Code 5: The algorithms (in pseudocode) used for autoconnecting a pin on a task with the
`appropriate connection locus. ...................................................................64
`Code 7: The structure of an Onika ONIKON node. The fields after and including line are
`used only when the icon is in an application. ...........................................67
`Code 6: An example target icon’s .onk file. ......................................................................68
`Code 8: The structure of application nodes. When used as a dictionary node, the fields
`flow_header, flow_tailer, parallel, dirty, alias, aliasnum, and lines are un-
`used, and numflow is always 1. .................................................................70
`Code 9: The structure of an ONIFLOW node. ..................................................................71
`Code 10: The ONEX node, used for holding the current status of an executing application
`(and subapplications, via a linked list) within a flow. ..............................91
`
`List of Code
`
`Page 15 of 198
`xi
`
`

`

`xii
`xii
`
`Page 16 of 198
`Page 16 of 198
`List of Code
`List of Code
`
`

`

`List of Tables
`
`Table 1: Performances when no descriptions were given............................................... 105
`Table 2: Performance when descriptions were given ..................................................... 105
`Table 3: Performance when no graphics were given...................................................... 106
`Table 4: Subject group comparisons (HSD test)............................................................. 108
`Table 5: The Task Library .............................................................................................. 131
`Table 6: The Configuration Canvas................................................................................ 132
`Table 7: The Job Dictionary ........................................................................................... 133
`Table 8: The Application Workspace ............................................................................. 134
`Table 9: Signal Response in Onika................................................................................. 140
`
`List of Tables
`
`Page 17 of 198
`xiii
`
`

`

`xiv
`xiv
`
`Page 18 of 198
`Page 18 of 198
`List of Tables
`List of Tables
`
`

`

`List of Abbreviations
`
`CFIG: Chimera Configuration File Reading Utility
`
`CMU: Carnegie Mellon University
`
`DH: Denavit-Hartenberg Robot Parameters
`
`DOF: Degrees-of-Freedom (generally, the number of joints a robot has)
`
`ENET: Network interface designed for Onika and Chimera
`
`GUI: Graphical User Interface
`
`HMI: Human-Machine Interface
`
`IPL: Iconic Programming Language
`
`INBOTH: Input port declared as both INCONST and INVAR
`
`INCONST: Input Port Constant
`
`INVAR: Input Port Variable
`
`I/O: Input/Output
`
`NDOF: Number of Degrees of Freedom
`
`ONIFLOW: Onika Upper-Level Flow Element
`
`ONIKON: Onika Upper-Level Icon
`
`OUTBOTH: Output port declared as both OUTCONST and OUTVAR
`
`OUTCONST: Output Port Constant
`
`OUTVAR: Output Port Variable
`
`List of Abbreviations
`
`Page 19 of 198
`xv
`
`

`

`PV: Program Visualization
`
`RTOS: Real-Time Operating System
`
`RTPU: Real-Time Processing Unit
`
`SIG: Signal
`
`SVAR: State Variable
`
`VL: Visual Language
`
`VP: Visual Programming
`
`VPE: Visual Programming Environmant
`
`VPL: Visual Programming Language
`
`xvi
`
`Page 20 of 198
`List of Abbreviations
`
`

`

`Abstract
`
`The development of real-time software for control systems is an expensive process,
`accounting for a significant portion of total application costs. This expense can be reduced
`by automating the software development procedure; for instance, by providing a software
`framework in which coded routines are small, portable, and reusable. However, to make
`the integration and control of these routines accessible to programmers of various exper-
`tise, and thus further reduce the amount of required resources, a user-friendly high-level
`programming environment designed for the creation of reusable real-time software is
`required. A programming interface of this type would not only allow for the rapid devel-
`opment of software, but would also considerably ease the process of debugging real-time
`code.
`
`Much of the expense and tedium of software development is caused by the limitations of
`textual code. To use a textual language properly, the programmer must undergo expensive
`training. The deciphering, debugging, and use of real-time textual code is particularly
`time-consuming, especially when the code is cryptic, non-portable, and uncommented.
`The sequential nature of text can cause confusion when tracing a program flow through
`subroutines, recursion, and processes spawned in parallel. These “coding complexities”
`inevitably consume many resources, adding to the mounting costs associated with the sys-
`tem. This limits the amount of applications which may be developed for the system and, in
`turn, limits the system’s usefulness.
`
`In the past, researchers have created visual programming languages to address the prob-
`lems of textual coding. However, these interfaces have been, in general, either very high-
`level and narrow in scope, or low-level and cryptic. Furthermore, these interfaces have not
`been designed with the specific requirements of real-time programming in mind. These
`requirements include the need to switch from one job to the next with minimal time loss,
`the need to modify the code of a job while it is executing, and the need to coordinate many
`Page 21 of 198
`xvii
`
`Abstract
`
`

`

`jobs running in parallel.
`
`In this dissertation, we present a multilevel/iconically-programmed visual programming
`environment called Onika. Its multiple interfaces directly connect with the underlying
`real-time operating system to coordinate the activities of several routines running in paral-
`lel. Each task on the real-time operating system is represented by an icon, which is manip-
`ulated by a mouse. These tasks are co