Direct manipulation systems offer the satisfying experience
`of operating on visible objects. The computer becomes transparent,
`and users can concentrate on their tasks.
`
`Direct Manipulation:
`A Step Beyond Programming
`Languages
`
`Ben Shneiderman, University of Maryland
`
`Leibniz sought to make the form of a symbol reflect its
`content. "In signs," he wrote, "one sees an advantage for
`discovery that is greatest when they express the exact
`nature of a thing brietlv and, as it were, picture it; then, in(cid:173)
`deed, the labor of thought is wonderfully diminished."
`
`Frederick Kreiling, "Leibniz."
`Scienu/ic American, \1ay 1968
`
`Certain interactive systems generate glowing en(cid:173)
`thusiasm among users-in marked contrast with the
`more common reaction of grudging acceptance or out(cid:173)
`right hostility. The enthusiastic users' reports are filled
`with positive feelings regarding
`
`• mastery of the system,
`• competence in the performance of their task,
`• ease in learning the system originally and in assimi(cid:173)
`lating advanced features,
`• confidence in their capacity to retain mastery over
`time,
`• enjoyment in using the system,
`• eagerness to show it off to novices, and
`• desire to explore more powerful aspects of the
`system.
`
`These feelings are not, of course, universal, but the
`amalgam does convey an image of the truly pleased user.
`As I talked with these enthusiasts and examined the sys(cid:173)
`tems they used, I began to develop a model of the fea(cid:173)
`tures that produced such delight. The central ideas
`seemed to be visibility of the object of interest; rapid,
`reversible, incremental actions; and replacement of com(cid:173)
`plex command language syntax by direct manipulation
`of the object of interest-hence the term "direct manip(cid:173)
`ulation."
`
`Examples of direct manipulation systems
`
`No single system has all the attributes or design fea(cid:173)
`tures that I admire-that may be impossible-but those
`described below have enough to win the enthusiastic sup(cid:173)
`port of many users.
`
`Display editors. ''Once you've used a display editor,
`you 'II never want to go back to a line editor. You 'II be
`spoiled." This reaction is typical of those who use full(cid:173)
`page display editors, who are great advocates of their
`systems over line-oriented text editors. I heard similar
`comments from users of stand-alone word processors
`such as the Wang system and from users of display
`editors such as EMACS on the MIT /Honeywell Multics
`system or "vi" (for visual editor) on the Unix system. A
`beaming advocate called EMACS ''the one true editor.''
`Roberts 1 found that the overall performance time of
`display editors is only half that of line-oriented editors,
`and since display editors also reduce training time, the
`evidence supports the enthusiasm of display editor devo(cid:173)
`tees. Furthermore, office automation evaluations consis(cid:173)
`tently favor full-page display editors for secretarial and
`executive use.
`The advantages of display editors include
`Display of a full 24 to 66 lines of text. This full display
`enables viewing each sentence in context and simplifies
`reading and scanning the document. By contrast, the
`
`A portion of this article was derived from the author's keynote address at
`the NYU Symposium on Cser Interfaces, "The Future of Interactive
`Systems and the Emergence of Direct Manipulation," published in
`Human Factors in lnteracril:e Computer S_vstems, Y. Vassiliou, ed.,
`Ablex Publishing Co., Nornood. N.J .. 1983.
`
`August 1983
`
`tK118-9162 8) 081Kl-IXlS-sn1.oo
`
`198) llXF
`
`57
`
`MSFT EX. 1011
`Page 1 of 13
`
`

`

`one-line-at-a-time view offered by line editors is like see(cid:173)
`ing the world through a narrow cardboard tube.
`Display of the document in its final form. Eliminat(cid:173)
`ing the clutter of formatting commands also simplifies
`reading and scanning the document. Tables, lists, page
`breaks, skipped lines, section headings, centered text,
`and figures can be viewed in the form that will be printed.
`The annoyance and delay of debugging the format com(cid:173)
`mands is eliminated because the errors are immediately
`apparent.
`Cursor action that is visible to the user. Seeing an ar(cid:173)
`row, underscore, or blinking box on the screen gives the
`operator a clear sense of where to focus attention and ap(cid:173)
`ply action.
`
`Cursor motion through physically obvious and intui(cid:173)
`tively natural means. Arrow keys or devices such as a
`mouse, joystick, or graphics tablet provide natural
`physical mechanisms for moving the cursor. This is in
`marked contrast with commands such as UP 6, which re(cid:173)
`quire an operator to convert the physical action into cor(cid:173)
`rect syntactic form and which may be difficult to learn,
`hard to recall, and a source of frustrating errors.
`Labeled buttons for action. Many display editors
`have buttons etched with commands such as INSERT,
`DELETE, CENTER, UNDERLINE, SUPERSCRIPT,
`BOLD, or LOCATE. They act as a permanent mehu se(cid:173)
`lection display, reminding the operator of the features
`and obviating memorization of a complex command-Ian-
`
`EDIT --- SPFOEMO. MYLIB . PL! I COINS l - 01. 04 ------------------- COLUNNS 001 072
`cot:t1ANO INPUT === >
`SCROLL ===> HALF
`*~**** *************************** TOP OF DATA ********************************
`000100 con;s :
`000200
`PROCEDURE OPTIONS IMAINJ;
`DECLARE
`00~!00
`FIXED BINARY 1311 AUTOMATIC !NIT (J),
`000400
`CCUNT
`000500
`FI XED BINARY (3JJ,
`HALVES
`000600
`QUARTERS FI XED BWARY ( 3J l,
`000700
`DIMES
`FIXED BINARY 13JJ,
`I3
`NICKE LS FIXED BHl~RY ( 31 l,
`000900
`SYSPRINT FILE STREAM OUTPUT PRINT;
`001000
`DO HALVES = 100 TO 0 BY -50;
`00 QUARTERS= (JOO - HALVES! TO 0 BY -25;
`001100
`001200
`00 Dltt!OS = I I JOO - HALVES - QUARTERSl/JOl*JO TO 0 BY -JO;
`NICf(ELS = JOO - HALVES - QUARTERS - DIMES;
`001300
`0
`PUT FILE! SYS PR INTI DATA! COUNT ,HALVES,QUARTERS,OIMES,NICKELS l;
`COUNT = COUNT + J ;
`001500
`001600
`ENO;
`Et:O;
`001700
`001800
`ENO;
`001900
`END COINS;
`****** *************************** BOTTOM OF DATA *****************************
`
`C>
`
`C>
`
`EDIT --- SPFDEMO . MYLIB. PLll COINS l - 01. 04 ------------------- COLUMNS 001 072
`cm::1ANO ItlPUT ===>
`SCROLL ===> HALF
`****** *************************** TOP OF DATA ********************************
`000100 corns:
`000200
`PROCEDURE OPTIONS IMAINJ;
`000300
`DECLARE
`000400
`FIXED BINARY ( 31 l AUTOMATIC !NIT ( J l,
`COUNT
`000500
`FIXED BINARY 13JJ,
`HALVES
`000600
`QUARTERS FIXED BINARY ( 31 l,
`000700
`DIMES
`FI XED BINARY ( 31 l,
`000800
`NICKELS FIXED BINARY I 3J l,
`
`C>
`
`000900
`SYSPRINT FILE STREAM OUTPUT PRINT;
`001000
`00 HALVES = 100 TO 0 BY -50;
`001100
`00 COUARTERS = (JOO - HALVES I TO 0 BY -25;
`DO DIMES= ((JOO - HALVES - QUARTERSJ/101*10 TO 0 BY -10;
`001~00
`001300
`NICKELS = J 00 - HALVES - QUARTERS - DIMES;
`COUNT = COIJNT + 1;
`001500
`ENO;
`OOJ600
`OOJ 700
`ENO;
`ENO;
`OOJSOO
`mo corns;
`001900
`*~**** ***********~*************** BOTTOM OF DATA *****************************
`
`Figure 1. This example from the IBM SPF display editor shows 19 lines of a PL/I program. The commands to in·
`sert three lines (13) and to delete one line (Dor 01) are typed on the appropriate lines in the first screen display.
`Pressing ENTER causes commands to be executed and the cursor to be placed at the beginning of the in·
`serted line. New program statements can be typed directly in their required positions. Control keys move the
`cursor around the text to positions where changes are made by overstriking. A delete key causes the character
`under the cur~or to be deleted and the text to the left to be shifted over. After pressing an insert key, the user
`can type text in place. Programmed function keys allow movement of the window forwards backwards left
`and right over the text. (Examples courtesy of I BM.)
`'
`'
`'
`
`58
`
`COMPUTER
`
`MSFT EX. 1011
`Page 2 of 13
`
`

`

`guage syntax. Some editors provide basic functionality
`with only 10 or 15 labeled buttons, and a specially
`marked button may be the gateway to advanced or infre(cid:173)
`quently used features offered on the screen in menu
`form.
`Immediate display of the results of an action. When a
`button is pressed to move the cursor or center the text,
`the results appear on the screen immediately. Deletions
`are apparent at once, since the character, word, or line is
`erased and the remaining text rearranged . Similarly, in(cid:173)
`sertions or text movements are shown after each key(cid:173)
`stroke or function button press. Line editors, on the
`other hand, require a print or display command before
`the r'esults of a change can be seen.
`Rapid action and display. Most display editors are
`designed to operate at high speeds: 120 characters per
`second (1200 baud), a full page in a second (9600 baud),
`or even faster. This high display rate coupled with short
`response time produces a thrilling sense of power and
`speed. Cursors can be moved quickly, large amounts of
`text can be scanned rapidly, and the results of commands
`can be shown almost instantaneously. Rapid action also
`reduces the need for additional commands, thereby sim(cid:173)
`plifying product design and decreasing learning time.
`Line editors operating at 30 characters per second with
`three- to eight-second response times seem sluggish in
`comparison. Speeding up line editors adds to their attrac(cid:173)
`tiveness , but they still lack features such as direct over(cid:173)
`typing, deletion, and insertion.
`
`Easily reversible commands. Mistakes in entering text
`can be easily corrected by backspacing and overstriking.
`Simple changes can be made by moving the cursor to the
`problem area and overstriking, inserting, or deleting
`characters, words, or lines. A useful design strategy is to
`include natural inverse operations for each operation.
`Carroll2 has shown that congruent pairs of operations
`are easy to learn. As an alternative, many display editors
`offer a simple UNDO command that cancels the previous
`command or command sequence and returns the text to
`its previous state. This easy reversibility reduces user anx(cid:173)
`iety about making mistakes or destroying a file.
`
`The large market for display editors generates active
`competition, which accelerates evolutionary design re(cid:173)
`finements. Figure 1 illustrates the current capabilities of
`an IBM display editor.
`
`Visicalc. Visicorp's innovative financial forecasting
`program, called Visicalc, was the product of a Harvard
`MBA student, who was frustrated by the time needed to
`carry out multiple calculations in a graduate business
`course. Described as an "instantly calculating electronic
`worksheet" in the user's manual, it permits computation
`and display of results across 254 rows and 63 columns
`and is programmed without a traditional procedural con(cid:173)
`trol structure. For example, positional declarations can
`prescribe that column 4 displays the sum of columns 1
`through 3; then every time a value in the first three col(cid:173)
`umns changes, the fourth column changes as well. Com(cid:173)
`plex dependencies among manufacturing costs, distribu(cid:173)
`tion costs, sales revenue, commissions, and profits can
`
`August 1983
`
`be stored for several sales districts and months so that the
`impact of changes on profits is immediately apparent.
`Since Visicalc simulates an accountant's worksheet, it
`is easy for novices to comprehend. The display of 20 rows
`and up to nine columns, with the provision for multiple
`windows, gives the user sufficient visibility to easily scan
`information and explore relationships among entries (see
`Figure 2). The command language for setting up the
`worksheet can be tricky for novices to learn and for infre(cid:173)
`quent users to remember, but most users need learn only
`the basic commands. According to Visicalc's distributor,
`"It jumps," and the user's delight in watching this prop(cid:173)
`agation of changes cross the screen helps explain its
`appeal.
`
`Figure 2. This simple Visicalc program display (top) shows four col·
`umns and 20 rows of home budget information. The cursor, an inverse
`video light bar controlled by key presses, is in position C2. The top
`command line shows that C2 is a value (as opposed to a text string)
`that has been set up to have the same value as position B2.
`The second display (above) shows two windows over the home budget
`data with row sums to the right. The last row shows leisure dollar
`amounts, which are established by the top command line formula as
`the income minus the sum of expenses. A change to the income or ex·
`pense values would immediately propagate to all affected values.
`(Displays reproduced by permission of Visicorp. )
`
`59
`
`MSFT EX. 1011
`Page 3 of 13
`
`

`

`Spatial data management. The developers of the pro(cid:173)
`totype spatial data management system3 attribute the
`basic idea to Nicholas Negroponte of MIT.
`In one scenario, a user seated before a color graphics
`display of the world zooms in on the Pacific to see
`markers for military ship convoys. Moving a joystick fills
`the screen with silhouettes of individual ships, which can
`be zoomed in on to display structural details or, ultimate(cid:173)
`ly, a full-color picture of the captain. (See Figure 3.)
`In another scenario, icons representing different
`aspects of a corporation, such as personnel, organiza(cid:173)
`tion, travel, production, or schedules, are shown on a
`screen. Moving the joystick and zooming in on objects
`takes users through complex "information spaces" or
`"I-spaces" to locate the item of interest. For example,
`when they select a department from a building floor
`
`Figure 3. A spatial data management system has been in·
`stalled on the aircraft carrier USS Carl Vinson. In the
`photo at top left, the operator has a world map on the left
`screen and a videodisc map of selected areas on the
`center screen. After some command selections with the
`data tablet and puck, the operator can zoom in on specif·
`ic data such as the set of ships shown in the second
`photo. With further selections the operator can get de·
`tailed information about each ship, such as the length,
`speed, and fuel. (Photos courtesy of Computer Corporation of
`America.)
`
`In 1971, about the only people playing video games were students in computer science laboratories. By 1973, however,
`millions of people were familiar with at least one video game-Pong (above left). A few years later came Breakout (above
`right), which, according to many designers, was the first true video game and the best one ever invented. Pong and other
`early games imitated real life, but Breakout could not have existed in any medium other than video. In the game, a single
`paddle directed a ball toward a wall of color bricks; contact made a brick vanish and changed the ball's speed.
`
`When the first arcade video game, Computer Space, went on location in a Sears store, its joystick was torn off before
`the end of the first day. As a result, game designers have sought controls that were both easy to use and hard to
`destroy. Centipede (above left) uses simple controls-a trackball and one button. On the other hand, Defender (above
`right) has five buttons and a joystick; novice players are confused by these relatively complex controls and usually
`give up after a few seconds.
`
`60
`
`COMPUTER
`
`MSFT EX. 1011
`Page 4 of 13
`
`

`

`plan, individual offices become visible. Moving the cur(cid:173)
`sor into a room brings the room's details onto the screen.
`If they choose the wrong room, they merely back out and
`try another. The lost effort is minimal, and no stigma is
`attached to the error.
`The success of a spatial data management system de(cid:173)
`pends on the designer's skill in choosing icons, graphical
`representations, and data layouts that are natural and
`easily understood. Even anxious users enjoy zooming in
`and out or gliding over data with a joystick, and they
`quickly demand additional power and data.
`
`Video games. Perhaps the most exciting, well-engi(cid:173)
`neered-certainly, the most successful-application of
`direct manipulation is in the world of video games. An
`early, but simple and popular, game called Pong re(cid:173)
`quired the user to rotate a knob, which moved a white
`rectangle on the screen. A white spot acted as a Ping(cid:173)
`Pong ball, which ricocheted off the wall and had to be hit
`back by the movable white rectangle. The user developed
`skill involving speed and accuracy in placement of the
`"paddle" to keep the increasingly speedy ball from get(cid:173)
`ting by, while the speaker emitted a ponging sound when
`the ball bounced. Watching someone else play for 30
`seconds was all the training needed to become a compe(cid:173)
`tent novice, but many hours of practice were required to
`become a skilled expert.
`
`Contemporary games such as Missile Command, Don(cid:173)
`key Kong, Pac Man, Tempest, Tron, Centipede, or
`. Space Invaders are far more sophisticated in their rules,
`color graphics, and sound effects (see sidebar below and
`on facing page). The designers of these games have pro(cid:173)
`vided stimulating entertainment, a challenge for novices
`and experts, and many intriguing lessons in the human
`factors of interface design-somehow they have found a
`way to get people to put coins into the sides of com(cid:173)
`puters. The strong attraction of these games contrasts
`markedly with the anxiety and resistance many users ex(cid:173)
`perience toward office automation equipment.
`Because their fields of action are abstractions of reali(cid:173)
`ty, these games are easily understood-learning is by
`analogy. A general idea of the game can be gained by
`watching the on-line automatic demonstration that runs
`continuously on the screen, and the basic principles can
`be learned in a few minutes by watching a knowledgeable
`player. But there are ample complexities to entice many
`hours and quarters from experts. The range of skill ac(cid:173)
`commodated is admirable.
`The commands are physical actions, such as button
`presses, joystick motions, or knob rotations, whose
`results appear immediately on the screen. Since there is
`no syntax, there are no syntax error messages. If users
`move their spaceships too far left, then they merely use
`the natural inverse operation of moving back to the right.
`Error messages are unnecessary because the results of ac-
`
`nkey Kong, Space Invaders, and Tron (clockwise from above)
`mplify the lively variety of video games now Inviting the
`r's loose change. As of mid· 1981, according to Steve Bloom,
`hor of Video lnradetS, more than four billion quarters had
`n dropped Into Space Invaders games around the
`rid-that's roughly "one game per earthling."
`
`eo game photos reprinted courtesy of IEEE Spectrum. For a more complete
`ort on the topic. see '"Video Games: The Electronic Big Bang'" by Tekla
`ry. Carol Truxal. and Paul Wallich in IEEE Spectrum, Vol. 19, No. 12. Dec.
`82, pp. 20-33.
`
`August1983
`
`MSFT EX. 1011
`Page 5 of 13
`
`

`

`tions are so obvious and easily reversed. These principles
`can be applied to office automation, personal comput(cid:173)
`ing, and other interactive environments.
`Every game that I have seen keeps a continuous score
`so that users can measure their progress and compete
`with their previous performance, with friends, or with
`the highest scorers. Typically, the IO highest scorers get
`to store their initials in the game for regular display, a
`form of positive reinforcement that encourages mastery.
`Malone's4 and our own studies with elementary school
`children have shown that continuous display of scores is
`extremely valuable. Machine-generated value judgments
`-"Very good" or "You're doing great!"-are not as
`effective, since the same score means different things to
`different people. Users prefer to make their own subjec(cid:173)
`tive judgments and may perceive machine-generated
`messages as an annoyance and a deception.
`Carroll and Thomas 5 draw productive analogies be(cid:173)
`tween game-playing environments and application sys(cid:173)
`tems. However, game players seek entertainment and the
`challenge of mastery, while application-system users
`focus on the task and may resent forced learning of
`system constraints. The random event' that occur in
`most games are meant to challenge the user, but predict(cid:173)
`able system behavior is preferable in nongame designs.
`Game players compete with the system, but application(cid:173)
`system users apparently prefer a strong internal locus of
`control, which gives them the sense of being in charge.
`
`The pleasure in using these systems stems
`from the capacity to manipulate the object
`of interest directly and to generate multiple
`alternatives rapidly.
`
`Computer-aided design/manufacturing. Many com(cid:173)
`puter-aided design systems for automobiles, electronic
`circuitry, architecture, aircraft, or newspaper layout use
`direct manipulation principles. The operator may see a
`schematic on the screen and with the touch of a lightpen
`can move resistors or capacitors into or out of the pro(cid:173)
`posed circuit. When the design is complete, the computer
`can provide information about current, voltage drops,
`fabrication costs, and warnings about inconsistencies or
`manufacturing problems. Similarly, newspaper layout
`artists or automobile body designers can try multiple
`designs in minutes and record promising approaches
`until a better one is found.
`The pleasure in using these systems stems from the
`capacity to manipulate the object of interest directly and
`to generate multiple alternatives rapidly. Some systems
`have complex command languages, but others have
`moved to cursor action and graphics-oriented commands.
`Another, related application is in computer-aided
`manufacturing and process control. Honeywell's process
`control system provides an oil refinery, paper mill, or
`power utility plant manager with a colored schematic
`view of the plant. The schematic may be on eight
`displays, with red lines indicating a sensor value that is
`
`out of normal range. By pressing a single numbered but(cid:173)
`ton (there are no commands to learn or remember), the
`operator can get a more detailed view of the troublesome
`component and, with a second press, move the tree struc(cid:173)
`ture down to examine individual sensors or to reset valves
`and circuits.
`The design's basic strategy precludes the necessity of
`recalling complex commands in once-a-year emergency
`conditions. The plant schematic facilitates problem solv(cid:173)
`ing by analogy, since the link between real-world high
`temperatures or low pressures and screen representations
`is so close.
`
`Further examples. Driving an automobile is my
`favorite example of direct manipulation. The scene is
`directly visible through the windshield, and actions such
`as braking or steering have become common skills in our
`culture. To turn to the left, simply rotate the steering
`wheel to the left. The response is immediate, and the
`changing scene provides feedback to refine the turn. Im(cid:173)
`agine trying to turn by issuing a LEFT 30 DEGREES
`command and then issuing another command to check
`your position, but this is the operational level of many
`office automation tools today.
`The term direct manipulation accurately describes the
`programming of some industrial robots. Here, the opera(cid:173)
`tor holds the robot's "hand" and guides it through a
`spray painting or welding task while the controlling com(cid:173)
`puter records every action. The control computer then
`repeats the action to operate the robot automatically.
`A large part of the success and appeal of the Query(cid:173)
`by-Example6 approach to data manipulation is due to its
`direct representation of relations on the screen. The user
`moves a cursor through the columns of the relational
`table and enters examples of what the result should look
`like. Just a few single-letter keywords supplement this
`direct manipulation style. Of course, complex Booleans
`or mathematical operations require knowledge of syntac(cid:173)
`tic forms. Still, the basic ideas and language facilities can
`be learned within a half hour by many non programmers.
`Query-by-Example succeeds because novices can begin
`work with just a little training, yet there is ample power
`for the expert. Directly manipulating the cursor across
`the relation skeleton is a simple task, and how to provide
`an example that shows the linking variable is intuitively
`clear to someone who understands tabular data. Zloof7
`recently expanded his ideas into Office-by-Example,
`which elegantly integrates database search with word
`processing, electronic mail, business graphics, and menu
`creation.
`Designers of advanced office automation systems have
`used direct manipulation principles. The Xerox Star8 of(cid:173)
`fers sophisticated text formatting options, graphics,
`multiple fonts, and a rapid, high-resolution, cursor(cid:173)
`based user interface. Users can drag a document icon and
`drop it into a printer icon to generate a hardcopy print(cid:173)
`out. Apple's recently announced Lisa system elegantly
`applies many of the principles of direct manipulation.
`Researchers at IBM's Yorktown Heights facility have
`proposed a future office system, called Pictureworld, in
`which graphic icons represent file cabinets, mailboxes,
`notebooks, phone messages, etc. The user could com-
`
`62
`
`COMPUTER
`
`MSFT EX. 1011
`Page 6 of 13
`
`

`

`pose a memo on a display editor and then indicate distri(cid:173)
`bution and filing operations by selecting from the menu
`of icons. In another project, Yedwab et al. 9 have de(cid:173)
`scribed a generalized office system, which they call the
`"automated desk."
`Direct manipulation can be applied to replace tradi(cid:173)
`tional question-and-answer computer-assisted instruc(cid:173)
`tion with more attractive alternatives. Several CDC Plato
`lessons employ direct manipulation concepts, enabling
`students to trace inherited characteristics by breeding
`drosophilla, perform medical procedures to save an
`emergency room patient, draw and move shapes by
`finger touches, do chemistry lab projects (see Figure 4),
`or play games.
`
`''virtuality' '-a representation of reality that can be ma(cid:173)
`nipulated. Rutkowski 11 conveys a similar concept in his
`principle of transparency: "The user is able to apply in(cid:173)
`tellect directly to the task; the tool itself seems to disap(cid:173)
`pear." MacDonald 12 proposes "visual programming" as
`a solution to the shortage of application progammers.
`He feels that visual programming speeds system con(cid:173)
`struction and allows end users to generate or modify
`applications systems to suit their needs.
`Each of these writers has helped increase awareness
`of the new form that is emerging for interactive sys(cid:173)
`tems. Much credit also goes to individual designers who
`have created systems exemplifying aspects of direct
`manipulation.
`
`Explanations of direct manipulation
`
`Several people have attempted to describe the com(cid:173)
`ponent principles of direct manipulation. "What you see
`is what you get," is a phrase used by Don Hatfield of
`IBM and others to describe the general approach. Hat(cid:173)
`field is applying many direct manipulation principles in
`his work on an advanced office automation system. Ex(cid:173)
`panding Hatfield's premise, Harold Thimbleby of the
`University of York, England, suggests, "What you see is
`what you have got." The display should indicate a com(cid:173)
`plete image of what the current status is, what errors have
`occurred, and what actions are appropriate, according to
`Thimbleby.
`Another imaginative observer of interactive system
`designs, Ted Nelson, IO has noticed user excitement over
`interfaces constructed by what he calls the principle of
`
`Problem-solving and learning research. Another
`perspective on direct manipulation comes from psychol(cid:173)
`ogy literature on problem solving. It shows that suitable
`representations of problems are crucial to solution find(cid:173)
`ing and to learning.
`Polya 13 suggests drawing a picture to represent math(cid:173)
`ematical problems. This approach is in harmony with
`Maria Montessori's teaching methods for children. 14 She
`proposed use of physical objects such as beads or wood(cid:173)
`en sticks to convey mathematical principles such as addi(cid:173)
`tion, multiplication, or size comparison. Bruner 15 ex(cid:173)
`tends the physical representation idea to cover polynom(cid:173)
`ial factoring and other mathematical principles. In a re(cid:173)
`cent experiment, Carroll, Thomas, and Malhotra 16
`found that subjects given a spatial representation solved
`problems more rapidly and successfully than subjects
`given an isomorphic problem with temporal representa-
`
`-,.,-
`
`..:.:·
`
`--.-----
`
`__ ,/ _______ -:.__
`
`I
`
`I
`
`- - - - - - · - - - -
`
`Figure 4. Computer·assisted instruction can become more appealing with direct manipulation, rather than simple question and
`answer scenarios. This CDC Plato lesson written by Stanley Smith of the Department of Chemistry at the University of Illinois
`allows students to construct a distillation apparatus by proper linger actions on a touch-sensitive screen (figure at left). Once the
`student has assembled the apparatus and begun the experiment, the real-time display gives a realistic view of the process with the
`graph of distillation temperature vs. volume. The student controls the experiment by touching light buttons. The figure at right
`shows that the student experimenter has gotten into trouble.
`
`August 1983
`
`63
`
`MSFT EX. 1011
`Page 7 of 13
`
`

`

`tion. (Deeper understanding of visual perception can be
`obtained from Arnheim 17 and Mc Kim. 18)
`Physical, spatial, or visual representations are also
`easier to retain and manipulate. Wertheimer 19 found that
`subjects who memorized the formula for the area of a
`parallelogram, A = h x b, mastered such calculations
`rapidly. On the other hand, subjects who were given a
`structural explanation (cut a triangle from one end and
`place it on the other) retained the knowledge and applied
`it in similar circumstances more effectively. In plane
`geometry theorem proving, a spatial representation facil(cid:173)
`itates discovery of proof procedures more than an ax(cid:173)
`iomatic representation. The diagram provides heuristics
`that arc difficult to extract from the axioms. Similarly,
`students of algebra are often encouraged to draw a pic(cid:173)
`ture to represent a word problem.
`Pa pert' s Logo language20 creates a mathematical
`microworld in which the principles of geometry are visi(cid:173)
`ble. Influenced by the Swiss psychologist Jean Piaget's
`theory of child development, Logo offers students the
`opportunity to create line drawings with an electronic
`turtle displayed on a screen. In this environment, users
`can receive rapid feedback about their programs, can
`easily determine what has happened, can quickly spot and
`repair errors, and can experience creative satisfaction.
`
`Problems with direct manipulation. Some profession(cid:173)
`al programming tasks can be aided by the use of graphic
`representations such as high-level flowcharts, record
`structures, or database schema diagrams, but additionai
`effort may be required to absorb the rules of the repre(cid:173)
`sentation. Graphic representations can be especially
`helpful when there are multiple relationships among ob(cid:173)
`jects and when the representation is more compact than
`the detailed object. In these cases, selectively screening
`out detail and presenting a suitable abstraction can
`facilitate performance.
`However, using spatial or graphic representations of
`the problem does not necessarily improve performance.
`In a series of studies, subjects given a detailed flowchart
`did no better in comprehension, debugging, or modifica(cid:173)
`tion than those given the code only. 21 In a program com(cid:173)
`prehension task, subjects given a graphic representation
`of control flow or data structure did no better than those
`given a textual description. 22 On the other hand, subjects
`given the data structure documentation consistently did
`better than subjects given the control flow documenta(cid:173)
`tion. This study suggests that the content of graphic
`representations is a critical determinant of their utility.
`The wrong information, or a cluttered presentation, can
`lead to greater confusion.
`A second problem is that users must learn the meaning
`of the components of the graphic representation. A
`graphic icon, alt hough meaningful to the designer, may
`require as much-or more-learning time as a word.
`Some airports serving multilingual com