Direct manipulation systems offer the satisfying experience
`of operating on visible objects. The computer becomes transparent,
`and users can concentrate on their tasks.
`
`Direct Manipulation:
`A Step Beyond Programming
`Languages
`
`Ben Shneiderman, University of Maryland
`
`Leibniz sought to make the form of a symbol reflect its
`content. "In signs," he wrote, "one sees an adsantage for
`discovery that is greatest wshen they express the exact
`nature of a thinlg briefly and, as it were, picture it; then, in-
`deed, the labor of thought is sonderfully diminished."
`Frederick Kreiling, "Leibniz,"
`Scientific A merican, M ay 1 968
`
`Certain interactive systems generate glowing en-
`thusiasm among users-in marked contrast with the
`more common reaction of grudging acceptance or out-
`right hostility. The enthusiastic users' reports are filled
`with positive feelings regarding
`
`* mastery of the system,
`* competence in the performance of their task,
`* ease in learning the system originally and in assimi-
`lating advanced features,
`* confidence in their capacity to retain mastery over
`time,
`* enjoyment in using the system,
`* eagerness to show it off to novices, and
`* desire to explore more powerful aspects of the
`system.
`
`These feelings are not, of course, universal, but the
`amalgam does convey an image of the truly pleased user.
`As I talked with these enthusiasts and examined the sys-
`tems they used, I began to develop a model of the fea-
`tures that produced such delight. The central ideas
`seemed to be visibility of the object of interest; rapid,
`reversible, incremental actions; and replacement of com-
`plex command language syntax by direct manipulation
`of the object of interest-hence the term "direct manip-
`ulation. "
`
`Examples of direct manipulation systems
`
`No single system has all the attributes or design fea-
`tures that I admire-that may be impossible-but those
`described below have enough to win the enthusiastic sup-
`port of many users.
`
`Display editors. "Once you've used a display editor,
`you'll never want to go back to a line editor. You'll be
`spoiled." This reaction is typical of those who use full-
`page display editors, who are great advocates of their
`systems over line-oriented text editors. I heard similar
`comments from users of stand-alone word processors
`such as the Wang system and from users of display
`editors such as EMACS on the MIT/Honeywell Multics
`system or "vi" (for visual editor) on the Unix system. A
`beaming advocate called EMACS "the one true editor."
`Robertsl found that the overall performance time of
`display editors is only half that of line-oriented editors,
`and since display editors also reduce training time, the
`evidence supports the enthusiasm of display editor devo-
`tees. Furthermore, office automation evaluations consis-
`tently favor full-page display editors for secretarial and
`executive use.
`The advantages of display editors include
`Display of afull 24 to 66 lines of text. This full display
`enables viewing each sentence in context and simplifies
`reading and scanning the document. By contrast, the
`
`A portion of this article was derived from the author's keynote address at
`the NYU Symposium on User Interfaces, "The Future of Interactive
`Systems and the Emergence of Direct Manipulation," published in
`Human Factors in Interactiue Computer Systems, Y. Vassiliou, ed.,
`Ablex Publishing Co., Norwood, N.J., 1983.
`
`August 1983
`
`s 9162 83 08(
`
`)-00'-15(I -00
`
`1i983
`
`LEEF
`
`57
`
`GOOGLE EX. 1029
`Google v. Philips
`
`

`
`one-line-at-a-time view offered by line editors is like see-
`ing the world through a narrow cardboard tube.
`Display of the document in its final form. Eliminat-
`ing the clutter of formatting commands also simplifies
`reading and scanning the document. Tables, lists, page
`breaks, skipped lines, section headings, centered text,
`and figures can be viewed in the form that will be printed.
`The annoyance and delay of debugging the format com-
`mands is eliminated because the errors are immediately
`apparent.
`Cursor action that is visible to the user. Seeing an ar-
`row, underscore, or blinking box on the screen gives the
`operator a clear sense of where to focus attention and ap-
`ply action.
`
`Cursor motion through physically obvious and intui-
`tively natural means. Arrow keys or devices such as a
`mouse, joystick, or graphics tablet provide natural
`physical mechanisms for moving the cursor. This is in
`marked contrast with commands such as UP 6, which re-
`quire an operator to convert the physical action into cor-
`rect syntactic form and which may be difficult to learn,
`hard to recall, and a source of frustrating errors.
`Labeled buttons for action. Many display editors
`have buttons etched with commands such as INSERT,
`DELETE, CENTER, UNDERLINE, SUPERSCRIPT,
`BOLD, or LOCATE. They act as a permanent mehu se-
`lection display, reminding the operator of the features
`and obviating memorization of a complex command-lan-
`
`EDIT --- SPFDEMO.MYLIB.PLI(COINS) - 01.04 ------------------- COLUMNS 001 072
`SCROLL ===> HALF
`COMMAND INPUT =>
`*********************~****** TOP OF DATA *
`*X****
`000100
`COINS:
`PROCEDURE OPTIONS (MAIN);
`000Z00
`DECLARE
`000*300
`CCUNT
`FIXED BINARY (31) AUTOMATIC INIT (1),
`000400
`HALVES
`000500
`FIXED BINARY (31),
`QUARTERS FIXED BINARY (31),
`000600
`000700
`FIXED BINARY (31),
`DIMES
`I3
`NICKELS
`FIXED BINARY (31),
`000900
`SYSPRINT FILE STREAM OUTPUT PRINT;
`001000
`DO HALVES = 100 TO 0 BY -50;
`001100
`DO QUARTERS = (100 - HALVES) TO 0 BY -25;
`DO DIMES = ((100 - HALVES - QUARTERS)/10)*10 TO 0 BY -10;
`001200
`NICKELS = 100 - HALVES - QUARTERS - DIMES;
`001300
`PUT FILE(SYSPRINT) DATA(COUNT,HALVES,QUARTERS,DIMES,NICKELS);
`D
`COUNT = COUNT + 1;
`001500
`001600
`END;
`END;
`001700
`END;
`001800
`001900
`END COINS;
`******
`
`BOTTOM OF DATA ******
`
`******** *
`
`EDIT --- SPFDEMO.MYLIB.PLI(COINS) - 01.04 ------------------- COLUMNS 001 072
`COMMAND INPUT =
`SCROLL =
`HALF
`***** ***************** ********** TOP OF DATA *****
`**************
`COItS:
`000100
`PROCEDURE OPTIONS (MAIN);
`000200
`DECLARE
`000300
`FIXED BINARY (31) AUTOMATIC INIT (1),
`000400
`COUNT
`000500
`FIXED BINARY (31),
`HALVES
`000600
`QUARTERS FIXED BINARY (31),
`FIXED BINARY (31),
`000700
`DIMES
`FIXED BINARY (31),
`000800
`NICKELS
`
`000900
`SYSPRINT FILE STREAM OUTPUT PRINT;
`DO HALVES
`001000
`100 TO 0 BY -50;
`DO QUARTERS = (100 - HALVES) TO 0 BY -25;
`001100
`001^00
`((100 - HALVES - QUARTERS)/10)*10 TO 0 BY -10;
`DO DIMES
`NICKELS = 100 - HALVES - QUARTERS - DIMES;
`001300
`COUNT = COUNT 4
`1;
`001500
`END;
`001600
`001700
`END;
`END;
`001800
`001C00
`END COINS;
`****** *************************** BOTTOM OF DATA ***
`
`*******
`
`*****
`
`58
`
`COMPUTER
`
`GOOGLE EX. 1029
`Google v. Philips
`
`

`
`guage syntax. Some editors provide basic functionality
`with only 10 or 15 labeled buttons, and a specially
`marked button may be the gateway to advanced or infre-
`quently used features offered on the screen in menu
`form.
`Immediate display of the results of an action. When a
`button is pressed to move the cursor or center the text,
`the results appear on the screen immediately. Deletions
`are apparent at once, since the character, word, or line is
`erased and the remaining text rearranged. Similarly, in-
`sertions or text movements are shown after each key-
`stroke or function button press. Line editors, on the
`other hand, require a print or display command before
`the results of a change can be seen.
`Rapid action and display. Most display editors are
`designed to operate at high speeds: 120 characters per
`second (1200 baud), a full page in a second (9600 baud),
`or even faster. This high display rate coupled with short
`response time produces a thrilling sense of power and
`speed. Cursors can be moved quickly, large amounts of
`text can be scanned rapidly, and the results of commands
`can be shown almost instantaneously. Rapid action also
`reduces the need for additional commands, thereby sim-
`plifying product design and decreasing learning time.
`Line editors operating at 30 characters per second with
`three- to eight-second response times seem sluggish in
`comparison. Speeding up line editors adds to their attrac-
`tiveness, but they still lack features such as direct over-
`typing, deletion, and insertion.
`Easily reversible commands. Mistakes in entering text
`can be easily corrected by backspacing and overstriking.
`Simple changes can be made by moving the cursor to the
`problem area and overstriking, inserting, or deleting
`characters, words, or lines. A useful design strategy is to
`include natural inverse operations for each operation.
`Carroll2 has shown that congruent pairs of operations
`are easy to learn. As an alternative, many display editors
`offer a simple UNDO command that cancels the previous
`command or command sequence and returns the text to
`its previous state. This easy reversibility reduces user anx-
`iety about making mistakes or destroying a file.
`
`The large market for display editors generates active
`competition, which accelerates evolutionary design re-
`finements. Figure I illustrates the current capabilities of
`an IBM display editor.
`
`Visicaic. Visicorp's innovative financial forecasting
`program, called Visicalc, was the product of a Harvard
`MBA student, who was frustrated by the time needed to
`carry out multiple calculations in a graduate business
`course. Described as an "instantly calculating electronic
`worksheet" in the user's manual, it permits computation
`and display of results across 254 rows and 63 columns
`and is programmed without a traditional procedural con-
`trol structure. For example, positional declarations can
`prescribe that column 4 displays the sum of columns I
`through 3; then every time a value in the first three col-
`umns changes, the fourth column changes as well. Com-
`plex dependencies among manufacturing costs, distribu-
`tion costs, sales revenue, commissions, and profits can
`
`August 1983
`
`be stored for several sales districts and months so that the
`impact of changes on profits is immediately apparent.
`Since Visicalc simulates an accountant's worksheet, it
`is easy for novices to comprehend. The display of 20 rows
`and up to nine columns, with the provision for multiple
`windows, gives the user sufficient visibility to easily scan
`information and explore relationships among entries (see
`Figure 2). The command language for setting up the
`worksheet can be tricky for novices to learn and for infre-
`quent users to remember, but most users need learn only
`the basic commands. According to Visicalc's distributor,
`"It jumps," and the user's delight in watching this prop-
`agation of changes cross the screen helps explain its
`appeal.
`
`Figure 2. This simple Visicalc program display (top) shows four col-
`umns and 20 rows of home budget information. The cursor, an inverse
`video light bar controlled by key presses, is in position C2. The top
`command line shows that C2 is a value (as opposed to a text string)
`that has been set up to have the same value as position B2.
`The second display (above) shows two windows over the home budget
`data with row sums to the right. The last row shows leisure dollar
`amounts, which are established by the top command line formula as
`the income minus the sum of expenses. A change to the income or ex-
`pense values would immediately propagate to all affected values.
`(Displays reproduced by permission of Visicorp.)
`
`59
`
`GOOGLE EX. 1029
`Google v. Philips
`
`

`
`Spatial data management. The developers of the pro-
`totype spatial data management system3 attribute the
`basic idea to Nicholas Negroponte of MIT.
`In one scenario, a user seated before a color graphics
`display of the world zooms in on the Pacific to see
`markers for military ship convoys. Moving a joystick fills
`the screen with silhouettes of individual ships, which can
`be zoomed in on to display structural details or, ultimate-
`ly, a full-color picture of the captain. (See Figure 3.)
`In another scenario, icons representing different
`aspects of a corporation, such as personnel, organiza-
`tion, travel, production, or schedules, are shown on a
`screen. Moving the joystick and zooming in on objects
`takes users through complex "information spaces" or
`"I-spaces" to locate the item of interest. For example,
`when they select a department from a building floor
`
`Figure 3. A spatial data management system has been in-
`stalled on the aircraft carrier USS Carl Vinson. In the
`photo at top left, the operator has a world map on the left
`screen and a videodisc map of selected areas on the
`center screen. After some command selections with the
`data tablet and puck, the operator can zoom in on specif-
`ic data such as the set of ships shown in the second
`photo. With further selections the operator can get de-
`tailed information about each ship, such as the length,
`speed, and fuel. (Photos courtesy of Computer Corporation of
`America.)
`
`In 1971, about the only people playing video games were students in computer science laboratories. By 1973, however,
`millions of people were familiar with at least one video game-Pong (above left). A few years later came Breakout (above
`right), which, according to many designers was the first true video game and the best one ever invented. Pong and other
`early games imitated real life, but Breakout could not have existed in any medium other than video. In the game, a single
`paddle directed a ball toward a wall of color bricks; contact made a brick vanish and changed the ball's speed.
`
`When the first arcade video game, Computer Space, went on location in a Sears store, its joystick was torn off before
`the end of the first day. As a result, game designers have sought controls that were both easy to use and hard to
`destroy. Centipede (above left) uses simple controls-a trackball and one button. On the other hand, Defender (above
`right) has fIve buttons and a joystick; novice players are confused by these relatively complex controls and usually
`give up after a few seconds.
`
`60
`
`COMPUTER
`
`GOOGLE EX. 1029
`Google v. Philips
`
`

`
`plan, individual offices become visible. Moving the cur-
`sor into a room brings the room's details onto the screen.
`If they choose the wrong room, they merely back out and
`try another. The lost effort is minimal, and no stigma is
`attached to the error.
`The success of a spatial data management system de-
`pends on the designer's skill in choosing icons, graphical
`representations, and data layouts that are natural and
`easily understood. Even anxious users enjoy zooming in
`and out or gliding over data with a joystick, and they
`quickly demand additional power and data.
`
`Video games. Perhaps the most exciting, well-engi-
`neered-certainly, the most successful-application of
`direct manipulation is in the world of video games. An
`early, but simple and popular, game called Pong re-
`quired the user to rotate a knob, which moved a white
`rectangle on the screen. A white spot acted as a Ping-
`Pong ball, which ricocheted off the wall and had to be hit
`back by the movable white rectangle. The user developed
`skill involving speed and accuracy in placement of the
`"paddle" to keep the increasingly speedy ball from get-
`ting by, while the speaker emitted a ponging sound when
`the ball bounced. Watching someone else play for 30
`seconds was all the training needed to become a compe-
`tent novice, but many hours of practice were required to
`become a skilled expert.
`
`Contemporary games such as Missile Command, Don-
`key Kong, Pac Man, Tempest, Tron, Centipede, or
`Space Invaders are far more sophisticated in their rules,
`color graphics, and sound effects (see sidebar below and
`on facing page). The designers of these games have pro-
`vided stimulating entertainment, a challenge for novices
`and experts, and many intriguing lessons in the human
`factors of interface design-somehow they have found a
`way to get people to put coins into the sides of com-
`puters. The strong attraction of these games contrasts
`markedly with the anxiety and resistance many users ex-
`perience toward office automation equipment.
`Because their fields of action are abstractions of reali-
`ty, these games are easily understood-learning is by
`analogy. A general idea of the game can be gained by
`watching the on-line automatic demonstration that runs
`continuously on the screen, and the basic principles can
`be learned in a few minutes by watching a knowledgeable
`player. But there are ample complexities to entice many
`hours and quarters from experts. The range of skill ac-
`commodated is admirable.
`The commands are physical actions, such as button
`presses, joystick motions, or knob rotations, whose
`results appear immediately on the screen. Since there is
`no syntax, there are no syntax error messages. If users
`move their spaceships too far left, then they merely use
`the natural inverse operation of moving back to the right.
`Error messages are unnecessary because the results of ac-
`
`GOOGLE EX. 1029
`Google v. Philips
`
`

`
`tions are so obvious and easily reversed. These principles
`can be applied to office automation, personal comput-
`ing, and other interactive environments.
`Every game that I have seen keeps a continuous score
`so that users can measure their progress and compete
`with their previous performance, with friends, or with
`the highest scorers. Typicallv, the 10 highest scorers get
`to storc their initials in the game for regular display, a
`form of positive reinforcement that encourages mastery.
`Malone's4 and our own studies with elementary school
`children have shown that continuous display of scores is
`extremely valuable. Machine-generated value judgments
`"Very good" or "You're doing great!"
`are not as
`effective, since the same score means different things to
`different people. Users prefer to make their own subjec-
`tive judgments and may perceive machine-generated
`messages as an annoyance and a deception.
`Carroll and Thomas' draw productive analogies be-
`tween game-playing environments and application sys-
`tems. However, game players seek entertainment and the
`challenge of mastery, while application-system users
`focus on the task and may resent forced learning of
`system constraints. The random events that occur in
`most games are meant to challenge the user, but predict-
`able system behavior is preferable in nongame designs.
`Game players compete with the system, but application-
`system users apparently prefer a strong internal locus of
`control, which gives them the sense of being in charge.
`
`The pleasure in using these systems stems
`from the capacity to manipulate the object
`of interest directly and to generate multiple
`alternatives rapidly.
`
`Computer-aided design/manufacturing. Many com-
`puter-aided design systems for automobiles, electronic
`circuitry, architecture, aircraft, or newspaper layout use
`direct manipulation principles. The operator may see a
`schematic on the screen and with the touch of a lightpen
`can move resistors or capacitors into or out of the pro-
`posed circuit. When the design is complete, the computer
`can provide information about current, voltage drops,
`fabrication costs, and warnings about inconsistencies or
`manufacturing problems. Similarly, newspaper layout
`artists or automobile body designers can try multiple
`designs in minutes and record promising approaches
`until a better one is found.
`The pleasure in using these systems stems from the
`capacity to manipulate the object of interest directly and
`to generate multiple alternatives rapidly. Some systems
`have complex command languages, but others have
`moved to cursor action and graphics-oriented commands.
`Another, related application is
`in computer-aided
`manufacturing and process control. Honeywell's process
`control system provides an oil refinery, paper mill, or
`power utility plant manager with a colored schematic
`view of the plant. The schematic may be on eight
`displays, with red lines indicatine a sensor value that is
`
`out of normal range. By pressing a single numbered but-
`ton (there are no commands to learn or remember), the
`operator can get a more detailed view of the troublesome
`component and, with a second press, move the tree struc-
`ture down to examine individual sensors or to reset valves
`and circuits.
`The design's basic strategy precludes the necessity of
`recalling complex commands in once-a-year emergency
`conditions. The plant schematic facilitates problem solv-
`ing by analogy, since the link between real-world high
`temperatures or low pressures and screen representations
`is so close.
`
`Further examples. Driving an automobile is my
`favorite example of direct manipulation. The scene is
`directly visible through the windshield, and actions such
`as braking or steering have become common skills in our
`culture. To turn to the left, simply rotate the steering
`wheel to the left. The response is immediate, and the
`changing scene provides feedback to refine the turn. Im-
`agine trying to turn by issuing a LEFT 30 DEGREES
`command and then issuing another command to check
`your position, but this is the operational level of many
`office automation tools today.
`The term direct manipulation accurately describes the
`programming of some industrial robots. Here, the opera-
`tor holds the robot's "hand" and guides it through a
`spray painting or welding task while the controlling com-
`puter records every action. The control computer then
`repeats the action to operate the robot automatically.
`A large part of the success and appeal of the Query-
`by-Example6 approach to data manipulation is due to its
`direct representation of relations on the screen. The user
`moves a cursor through the columns of the relational
`table and enters examples of what the result should look
`like. Just a few single-letter keywords supplement this
`direct manipulation style. Of course, complex Booleans
`or mathematical operations require knowledge of syntac-
`tic forms. Still, the basic ideas and language facilities can
`be learned within a half hour by many nonprogrammers.
`Query-by-Example succeeds because novices can begin
`work with just a little training, yet there is ample power
`for the expert. Directly manipulating the cursor across
`the relation skeleton is a simple task, and how to provide
`an example that shows the linking variable is intuitively
`clear to someone who understands tabular data. Zloof7
`recently expanded his ideas into Office-by-Example,
`which elegantly integrates database search with word
`processing, electronic mail, business graphics, and menu
`creation.
`Designers of advanced office automation systems have
`used direct manipulation principles. The Xerox Star8 of-
`fers sophisticated text formatting options, graphics,
`multiple fonts, and a rapid, high-resolution, cursor-
`based user interface. Users can drag a document icon and
`drop it into a printer icon to generate a hardcopy print-
`out. Apple's recently announced Lisa system elegantly
`applies many of the principles of direct manipulation.
`Researchers at IBM's Yorktown Heights facility have
`proposed a future office system, called Pictureworld, in
`which graphic icons represent file cabinets, mailboxes,
`notebooks, phone messages, etc. The user could com-
`
`62
`
`COMPUTER
`
`GOOGLE EX. 1029
`Google v. Philips
`
`

`
`pose a memo on a display editor and then indicate distri-
`bution and filing operations by selecting from the menu
`of icons. In another project, Yedwab et al.9 have de-
`scribed a generalized office system, which they call the
`"automated desk."
`Direct manipulation can be applied to replace tradi-
`tional question-and-answer computer-assisted instruc-
`tion with more attractive alternatives. Several CDC Plato
`lessons employ direct manipulation concepts, enabling
`students to trace inherited characteristics by breeding
`drosophilla, perform medical procedures to save an
`emergency room patient, draw and move shapes by
`finger touches, do chemistry lab projects (see Figure 4),
`or play games.
`
`"virtuality" -a representation of reality that can be ma-
`nipulated. RutkowskitI conveys a similar concept in his
`principle of transparency: "The user is able to apply in-
`tellect directly to the task; the tool itself seems to disap-
`pear." MacDonald12 proposes "visual programming" as
`a solution to the shortage of application progammers.
`He feels that visual programming speeds system con-
`struction and allows end users to generate or modify
`applications systems to suit their needs.
`Each of these writers has helped increase awareness
`of the new form that is emerging for interactive sys-
`tems. Much credit also goes to individual designers who
`have created systems exemplifying aspects of direct
`manipulation.
`
`Explanations of direct manipulation
`
`Several people have attempted to describe the com-
`ponent principles of direct manipulation. "What you see
`is what you get," is a phrase used by Don Hatfield of
`IBM and others to describe the general approach. Hat-
`field is applying many direct manipulation principles in
`his work on an advanced office automation system. Ex-
`panding Hatfield's premise, Harold Thimbleby of the
`University of York, England, suggests, "What you see is
`what you have got." The display should indicate a com-
`plete image of what the current status is, what errors have
`occurred, and what actions are appropriate, according to
`Thimbleby.
`Another imaginative observer of interactive system
`designs, Ted Nelson,10 has noticed user excitement over
`interfaces constructed by what he calls the principle of
`
`Problem-solving and learning research. Another
`perspective on direct manipulation comes from psychol-
`ogy literature on problem solving. It shows that suitable
`representations of problems are crucial to solution find-
`ing and to learning.
`PolyaI3 suggests drawing a picture to represent math-
`ematical problems. This approach is in harmony with
`Maria Montessori's teaching methods for children. 14 She
`proposed use of physical objects such as beads or wood-
`en sticks to convey mathematical principles such as addi-
`tion, multiplication, or size comparison. BrunerI5 ex-
`tends the physical representation idea to cover polynom-
`ial factoring and other mathematical principles. In a re-
`cent experiment, Carroll, Thomas, and MalhotraI6
`found that subjects given a spatial representation solved
`problems more rapidly and successfully than subjects
`given an isomorphic problem with temporal representa-
`
`A
`hA
`
`hA
`hA
`
`-
`T
`
`- C )r_DL -r J)9Fr
`+ -
`1
`- e fre-t -r!
`
`-
`
`-
`
`."IN
`
`t:-
`
`'
`
`A
`
`.
`
`iAkA
`hkA
`
`hA
`hA
`
`Is
`
`-. -I
`
`-: .l
`
`Figure 4. Computer-assisted instruction can become more appealing with direct manipulation, rather than simple question and
`answer scenarios. This CDC Plato lesson written by Stanley Smith of the Department of Chemistry at the University of Illinois
`allows students to construct a distillation apparatus by proper finger actions on a touch-sensitive screen (figure at left). Once the
`student has assembled the apparatus and begun the experiment, the real-time display gives a realistic view of the process with the
`graph of distillation temperature vs. volume. The student controls the experiment by touching light buttons. The figure at right
`shows that the student experimenter has gotten into trouble.
`
`August 1983
`
`63
`
`GOOGLE EX. 1029
`Google v. Philips
`
`

`
`tion. (Deeper understanding of visual perception can be
`obtained from Arnheimt7 and McKim 18)
`Physical, spatial, or visual representations are also
`easier to retain and manipulate. WertheimerI9 found that
`subjects who memorized the formula for the area of a
`parallelogram, A = h x b, mastered such calculations
`rapidly. On the other hand, subjects who were given a
`structural explanation (cut a triangle from one end and
`place it on the other) retained the knowledge and applied
`it in similar circumstances more effectively. In plane
`geometry theorem proving, a spatial representation facil-
`itates discovery of proof procedures more than an ax-
`iomatic representation. The diagram provides heuristics
`that are difficult to extract from the axioms. Similarly,
`studenits of algebra are often encouraged to draw a pic-
`ture to represent a word problem.
`Papert's Logo language20 creates a mathematical
`microworld in which the principles of geometry are visi-
`ble. Influenced by the Swiss psychologist Jean Piaget's
`theory of child development, Logo offers students the
`opportunitv to create line drawings with an electronic
`turtle displayed on a screen. In this environment, users
`can receive rapid feedback about their programs, can
`easily determine wFhat has happened, can quickly spot and
`repair errors, and can experience creative satisfaction.
`
`Problems with direct manipulation. Some profession-
`al programming tasks can be aided by the u.se of graphic
`representations such as high-level flowcharts, record
`structures, or database schema diagrams, but additionai
`effort may be required to absorb the rules of the repre-
`sentationi. Graphic representations can be especially
`helpful when there are multiple relationships among ob-
`jects andi when the representation is more compact than
`the detailed object. In these cases, selectively screening
`out detail and piesenrting a suitable abstraction can
`facilitate performaince.
`However, using spatial or graphic representations of
`the problem does not necessarily improve performance.
`In a series of studies, subjects given a detailed flowchart
`did no better in conmprehension, debugging, or modifica-
`tion thani those given the code only.2t In a program com-
`prehensioni task, subjects given a graphic representation
`of control flow or data structure did no better than those
`giv%en a textual description.-2 On the other hand, subjects
`given the data structure documenitationi consistently did
`better than subjects given the control flow documenta-
`tioIl. This study sllgeests that the content of graphic
`represenitations is a critical determinant of their utility.
`The wrong information, or a cluttered presentation, can
`lead to greater confusioni.
`A second problem is that users must learn the meaning
`of the components of the graphic representation. A
`graphic iconi, although meaningful to the designer, may
`require as much
`learning time as a word.
`or more
`Some airports serving multilingual communities use
`graphic icons extensively, but their meaning may not be
`obvious. Similarly, some computer terminals designed
`for international use have icons in place of names, but
`the meaning is not always clear.
`A third problem is that the graphic represenltation may
`be misleadine. The Lisci mav rapidly grasp the analogical
`
`representation, but then make incorrect conclusions
`about permissible operations. Designers must be cau-
`tious in selecting the displayed representation and the op-
`erations. Ample testing must be carried out to refine the
`representation and minimize negative side effects.
`A fourth problem is that graphic representations may
`take excessive screen display space. For experienced
`users, a tabular textual display of 50 document names is
`far more appealing than only 10 document graphic icons
`with the names abbreviated to fit the icon size. Icons
`
`Choosing the right representations and
`operations is not easy. Simple metaphors,
`analogies, or models with a minimal set of
`concepts seem most appropriate.
`
`should be evaluated first for their power in displaying
`static information about objects and their relationship,
`and second for their utility in the dynamic processes of
`selection, movement, and deletion.
`Choosing the right representations and operations is
`not easy. Simple metaphors, analogies, or models with a
`minimal set of concepts seem most appropriate. Mixing
`metaphors from two sources adds complexity, which
`contributes to confusion. The emotional tone of the
`metaphor should be inviting rather than distasteful or in-
`appropriate16
`sewage disposal systems are an inap-
`propriate metaphor for electronic message systems.
`Since users may not share the designer's metaphor, anal-
`ogy, or conceptual model, ample testing is required.
`
`The syntactic/semantic model. The attraction of sys-
`tems that use principles of direct manipulation is con-
`firmed by the enthusiasm of their users. The designers of
`the examples given had an innovative inspiration and an
`intuitive grasp of what users wanted. Each example has
`features that could be criticized, but it seems more pro-
`ductive to construct an integrated portrait of direct
`manipulation:
`
`* Continuous representation of the object of interest.
`* Physical actions (movement and selection by mouse,
`etc.) or labeled button
`joystick, touch screen,
`presses instead of complex syntax.
`* Rapid, incremental, reversible operations whose im-
`pact on the object of interest is immediately visible.
`* Layered or spiral approach to learning that permits
`usage with minimal knowledge. Novices can learn a
`modest and useful set of commands, which they can
`exercise till they become an "expert" at level I of the
`system. After obtaining reinforcing feedback from
`successful operation, users can gracefully expand
`their knowledge of features and gain fluency.23
`
`By using thes