`Human-Computer
`Interaction Teclinology
`
`Brad A. Myers
`
`-
`
`i n tc r a cti o n s ... ma r ch +ap r il
`
`1 998
`
`Page 1 ofll
`
`TRADING TECH EXHIBIT 2003
`IBG ET AL. v. TRADING TECH
`CBM2015-00182
`
`
`
`article
`
`Brad A. Myers
`Human-Computer
`Interaction Institute
`School of Computer
`Science
`Carnegie Mellon
`University
`Pittsburgh, PA 15213-
`3891
`
`bam@cs.cmu.edu
`
`Research in human-computer interac-
`
`tion (HCI) has been spectacularly sue-
`
`cessful and has fundamentally changed
`
`computing. One example is the ubiqui-
`
`tous graphical
`
`interface used by
`
`Microsoft Windows 95, which is based
`
`on the Macintosh, which is based on
`
`work at Xerox PARC, which in turn is
`
`based on early research at the Stanford
`
`Research Laboratory (now SRI) and at
`
`Massachusetts Institute of Technology.
`
`Another example is that virtually all soft(cid:173)
`ware written today employs user interface
`toolkits and interface builder~oncepts that
`were developed first at universities. Even the
`remarkable growth of the World Wide Web is
`a direct result of HCI research: applying
`hypertext technology to browsers allows one
`to traverse a link across the world with a click
`of the mouse. More than anything else,
`improvements to interfaces have triggered this
`explosive growth. Furthermore, the research
`that will lead to the user interfaces for the
`computers of tomorrow is being carried out
`today at universities and a few corporate
`research labs.
`T his paper attempts to briefly summarize
`many of the important research developments
`in HCI technology, emphasizing the role of
`university research, which may not be widely
`recognized. By "research," I mean exploratory
`work at universities and government and cor(cid:173)
`porate research labs (such as Xerox PARC)
`that is not directly related to products. By
`"HCI technology," I am referring to the com(cid:173)
`puter side ofHCI. A companion article on the
`history of the "human side," discussing the
`contributions
`from psychology, design,
`human factors, and ergonomics would also be
`appropriate.
`
`Figure 1 shows the time span for some of
`the technologies discussed in this article.
`including when they were introduced. Of
`course, a deeper analysis would reveal signifi(cid:173)
`cant interaction among the university, corpo(cid:173)
`rate research, and commercial activity lines. It
`is important to appreciate that years of
`research are involved in creating and making
`these technologies ready for widespread use.
`The same will be true for the HCI technolo(cid:173)
`gies currently being developed that will pro(cid:173)
`vide the interfaces of tomorrow.
`Clearly it is impossible to list every system
`and source in a paper of this scope, but I have
`tried to represent the earliest and most influ(cid:173)
`ential systems. Further information can be
`found in other surveys of H CI topics (see, for
`example, [1, 11, 36, 41]). Another useful
`resource is the video All The Widgets, which
`shows the historical progression of a number
`of user interface ideas [27].
`The technologies discussed in this paper
`include fu ndamental interaction styles such as
`direct manipulation, the mouse pointing
`device, and windows; several important kinds
`of application areas, such as drawing, text
`editing, and spreadsheets; the technologies
`that will likely have the biggest impact on
`interfaces of the future, such as gesture recog(cid:173)
`nition, multimedia, and three-dimensionality;
`and the technologies used to create interfaces
`using the other technologies, such as user
`interface management systems, toolkits, and
`interface builders.
`
`Basic Interactions
`DIRECT MANIPULATION OF
`
`GRAPHICAL OBJECTS
`The now ubiquitous direct manipulation inter(cid:173)
`face, where visible objects on the screen are
`directly manipulated with a pointing device,
`was first demonstrated by Ivan Sutherland in
`Sketchpad [47], the thesis of his doctoral di~
`serration in 1963. Sketchpad supported the
`manipulation of objects using a light pen,
`including grabbing objects, moving them,
`changing size, and using constraints. It con(cid:173)
`tained the seeds of myriad important interface
`ideas. The system was built at Lincoln Labs
`with support from the U.S. Air Force and the
`National Science Foundation (NSF).
`
`i nte r a ct ion s .. . ma rc h
`
`• apr il
`
`1 998
`
`Page 2 ofll
`
`
`
`KEY:
`• • • University Research
`
`• • • Corporate Research
`
`• • • Commercial Products
`
`Direct Manipulation of Graphical Objects
`
`Figure 1. Approximate time lines showing where
`and when work was performed on some major
`technologies discussed in this article.
`
`I
`
`1945
`
`I
`
`I
`
`1955
`
`I
`
`I
`1965
`
`I
`
`I
`
`1975
`
`I
`
`I
`
`1985
`
`I
`
`The Mouse
`
`..
`I
`
`1995
`
`I
`
`1945
`
`I
`
`I
`
`1955
`
`I
`
`I
`1965
`
`I
`
`I
`
`1975
`
`I
`
`I
`
`1985
`
`I
`
`I
`
`1995
`
`Windows
`
`..
`
`I
`
`I
`
`I
`
`I
`
`I
`
`I
`
`I
`
`I
`
`I
`
`1985
`
`I
`
`I
`1995
`
`1945
`
`1955
`
`1965
`
`1975
`
`Text Editing
`
`I
`
`I
`
`1955
`
`I
`
`I
`
`1965
`
`I
`
`I
`
`1975
`
`I
`
`I
`
`1985
`
`I
`
`I
`
`1955
`
`I
`
`I
`
`1965
`
`I
`
`I
`
`1975
`
`I
`
`I
`
`1985
`
`..
`I
`
`1995
`
`..
`I
`
`1995
`
`I
`
`I
`
`I
`
`1945
`
`HyperText -
`
`I
`
`1945
`
`Gesture Recognition
`
`1945
`
`1955
`
`1965
`
`1975
`
`1985
`
`1995
`
`i nt er a ct ion s .. . mar c h
`
`..,. apr il
`
`1 998
`
`Page 3 ofll
`
`
`
`article
`
`-
`
`NLS project (funding from ARPA, NASA,
`and Rome ADC) [10]. It was intended to be a
`cheap replacement for light pens, which had
`been used at least since 1954 [11, p. 68].
`Many of the current uses of the mouse were
`demonstrated by Doug Engelhart as parr of
`NLS in a movie created in 1968 [9]. The
`mouse was then made famous as a
`practical input device by Xerox
`in the 1970s. Ir first
`PARC
`appeared commercially as parr
`of the Xerox Star (1981), the
`Three Rivers Computer
`Company's PERQ (1981)
`[25],
`the Apple Lisa
`(1982), and the Apple
`Macintosh (1984).
`
`"THE SKETCHPAD
`
`SYSTEM MAKES IT
`
`POSSIBLE FOR A
`
`MAN AND A COM-
`
`PUTER TO CON-
`
`William Newman's Reaction Handler [33],
`created at Imperial College, London during
`1966 and 1967, provided direct manipulation
`of graphics and introduced Light H andles
`[32], a form of graphical potentiometer that
`was probably the fust "widget." Another early
`system was AMBIT/G (implemented at the
`Massachusetts Institute of Technology's
`(MIT) Lincoln Labs in 1968 and
`funded by the Advanced Research
`Projects Agency (ARPA). Ir
`employed, among other inter(cid:173)
`face techniques, iconic repre(cid:173)
`sentations, gesture
`recog(cid:173)
`nition, dynamic menus with
`items selected using a point-
`ing device, selection of icons
`by pointing, and moded
`and mode-free styles of
`interaction.
`David Canfield Smith
`coined the term "icons" in his
`1975 doctoral thesis on Pyg(cid:173)
`malion [44] (funded by ARPA
`and National Institute of Mental
`Health - NIMH). Smith later popu-
`larized icons as one of the chief designers
`of the Xerox Star [45]. Many of the interac(cid:173)
`tion techniques popular in direct manipula(cid:173)
`tion interfaces, such as how objects and rexr
`are selected, opened, and manipulated, were
`researched at Xerox PARC in the 1970s. In
`particular, the idea of "WYSIWYG" (what
`you see is what you get) originated there with
`systems such as the Bravo text editor and the
`Draw drawing program [11]
`The concept of direct manipulation inter(cid:173)
`faces for everyone was envisioned by Alan Kay
`of Xerox PARC in a 1977 article about the
`Dynabook [18]. The first commercial systems
`to use direct manipulation extensively were the
`Xerox Star (1981) [45], the Apple Lisa (1982)
`[54], and the Macintosh (1984) [55]. Ben
`Shneiderman at the University of Maryland
`coined the term "direct manipulation" in 1982,
`identified the components, and gave psycholog(cid:173)
`ical motivations for direct manipulation [43].
`
`VERSE RAPIDLY
`
`THROUGH THE
`
`MEDIUM OF LINE
`
`DRAWINGS."
`
`-
`
`I VAN SUTHERLAND
`[47, P. 329]
`
`WrNoows
`Multiple riled windows
`were demonstrated
`10
`Engelhart's NLS in 1968
`[9]. Early research at Stan(cid:173)
`ford on systems like C OPI(cid:173)
`LOT (1974) [49] and at MIT
`with the EMACS rext editor (1974)
`[46] also demonstrated tiled windows.
`Alan Kay proposed the idea of overlapping
`windows in his 1969 doctoral thesis [17], and
`overlapping windows first appeared in 1974 in
`his Smalltalk system [12] at Xerox PARC, and
`soon afterward in the InterLisp system [50].
`Some of rhe first commercial uses of win(cid:173)
`dows were on Lisp Machines Inc. (LMI) and
`Symbolics Lisp Machines (1979), which grew
`out of MIT Artificial Intelligence (AI) Lab pro(cid:173)
`jects. The Cedar Window Manager from Xerox
`PARC was the first major riled window manag(cid:173)
`er (1981) [48], followed soon by the Andrew
`window manager [35] by Carnegie Mellon
`University's (CMU) Information
`Technology Center (1983, funded
`by IBM). The main commercial
`systems popularizing windows
`were the Xerox Star (1981), the
`Apple Lisa (1982), and most
`important, the Apple Macin-
`tosh (1984). The early versions of
`the Star and Microsofr Windows
`were tiled, but eventually they sup(cid:173)
`ported overlapping windows like the
`
`MousE
`The mouse wa.s developed at Stanford
`Research Laboratory in 1965 as part of the
`
`interact ion s .. . march +april 1998
`
`Page 4 of ll
`
`
`
`Lisa and Macintosh. The X Window System, a
`current international standard, was developed
`at MIT in 1984 [42] . For a survey of window
`managers, see [26].
`
`"THE DIFFICULTY
`
`OF MANKIND'S
`
`PROBLEMS WAS
`
`INCREASING AT A
`
`based display editors that was widely used
`[51]. The Hypertext Editing System [53, p.
`1 08] from Brown University had screen edit(cid:173)
`ing and formatting of arbitrary-sized strings
`with a light pen in 1967 (funding from
`IBM). NLS demonstrated mouse(cid:173)
`based editing in 1968 (Figure 4).
`TECO from MIT was an early
`screen editor (1967), and
`EMACS [46] was developed
`from it in 1974. Xerox
`PARC's Bravo [11, p. 284]
`was the first WYSIWYG
`editor-formatter (1974) .
`It was designed by Butler
`Lampson and Charles
`Simonyi, who had started
`working on these con(cid:173)
`cepts about 1970 while at
`Berkeley. The first com(cid:173)
`mercial WYSIWYG edi(cid:173)
`tors were
`the Star,
`LisaWrite,
`and,
`later,
`MacWrite. For surveys of
`text editors, see [24] and [53].
`
`Applications
`DRAWING PROGRAMS
`Much of the current technology
`for drawing programs was
`demonstrated in Sutherland's
`1963 Sketchpad system
`[47] . The use of a mouse to
`manipulate graphics was
`demonstrated
`in NLS
`(1965). In 1968 Ken
`Pulfer
`and
`Grant
`Bechthold at the Nation-
`al Research Council of
`Canada built a mouse out
`of wood patterned after
`Engelhart's mouse and
`used it with a key-frame
`animation system to draw
`all the frames of a movie. A
`subsequent movie in 1971,
`Hunger, won a number of
`awards and was drawn using a
`tablet instead of the mouse (fund-
`ing by the National Film Board of
`Canada) [3]. William Newman's Markup
`(1975) was the first drawing program for
`Xerox PARC's Alto, followed shortly by
`Patrick Baudelaire's Draw, which added han(cid:173)
`dling of lines and curves [11, p. 326]. The
`first computer painting program was probably
`Dick Shoup's
`Superpaint
`at PARC
`(1974-1975). The first widely used drawing
`programs were MacPaint and MacDraw on
`the Macintosh (1984).
`
`GREATER RATE THAN
`
`OUR ABILITY TO
`
`COPE. (WE ARE IN
`
`TROUBLE.) [SO] I ...
`
`COMMITTED MY
`
`CAREER TO
`
`'AUGMENTING THE
`
`HUMAN INTELLECT."'
`
`- DOUG ENGELBART, ON THE
`NLS SYSTEM [8, P. 189]
`
`SPREADSHEETS
`The initial spreadsheet was Visi(cid:173)
`Calc which was developed by
`Frankston and Bricklin (1977 - 1978) for
`the Apple II while they were students at MIT
`and the Harvard Business School, respectively.
`The solver was based on a dependency-direct(cid:173)
`ed backtracking algorithm by Sussman and
`Stallman at the MIT AI Lab.
`
`HYPERTEXT
`The idea for hypertext (by which documents are
`linked to related documents) is credited to Van(cid:173)
`nevar Bush's famous MEMEX idea from 1945
`[4]. Ted Nelson coined the term "hypertext" in
`1965 [31]. Engelhart's NLS system [9] at the
`Stanford Research Laboratories in 1965 made
`extensive use of linking (funding from ARPA,
`NASA, and Rome ADC). The "NLS Journal"
`[11, p. 212] was one of the first on-line journals
`and included full linking of articles (1970). The
`Hypertext Editing System, jointly designed by
`Andy van Dam, Ted Nelson, and two students
`at Brown University (funding from IBM) was
`distributed extensively [52].
`
`TEXT EDITING
`In 1962, at the Stanford Research
`Laboratory, Engelhart proposed,
`and later implemented, a word
`processor with automatic word
`, wrap, search and replace, user(cid:173)
`/ definable macros, scrolling text,
`and commands to move, copy,
`and delete characters, words, or
`'blocks of text. Stanford's TVEdit
`(1965) was one of the first CRT-
`
`i nteractions .. . mar c h
`
`..,. apri l
`
`1 998
`
`Page 5 ofll
`
`
`
`article
`
`manufacturing
`aided
`(CAM) was probably
`General Motor's DAC-
`1 (about 1963).
`
`VIDEO GAMES
`The first graphi(cid:173)
`cal video game
`was probably
`Space War by Slug
`Russel of MIT in 1962 for
`the PDP-1 [21, p. 49], including the
`first computer joysticks. The early computer
`Adventure game was created by Will
`Crowther at Bolt, Beranek & Newman
`(BBN), and Don Woods developed it into a
`more sophisticated Adventure game at Stan-
`ford in 1966 [21, p. 132]. Conway's
`game of LIFE was developed on
`computers at MIT and Stan(cid:173)
`ford in 1970. The first popu(cid:173)
`lar commercial game was
`Pong (about 1976).
`
`"MULTIPLE
`
`WINDOWS ALLOW A
`
`DOCUMENT
`
`The University of Vermont's
`PROMIS (1976) was the first
`hypertext system released to
`the user community. It
`was used to link patient
`and patient care infor(cid:173)
`mation at the Univer(cid:173)
`sity of Vermont's
`medical center. T he
`ZOG project (1977) from
`CMU was another early hypertext
`system and was funded by Office of
`Naval Research (ONR) and ARPA [39]. Ben
`Shneiderman's Hyperties was the first system
`in which highlighted items in the text could
`be clicked on to go to other pages (1983, Uni(cid:173)
`versity of Maryland) [19]. HyperCard from
`Apple (1988) significantly helped to
`bring the idea to a wide audience.
`Tim Berners-Lee used the
`hypertext idea to create the
`World Wide Web in 1990 at
`the
`government-funded
`European Particle Physics
`Laboratory
`(CERN).
`Mosaic, the first popular
`hypertext browser for
`the World-Wide Web,
`was developed at the
`University of Illinois'
`National Center
`for
`Supercomputer Appli(cid:173)
`cations (NCSA). For a
`more complete history of
`HyperText, see [34].
`
`(COMPOSED OF TEXT,
`
`PICTURES, MUSICAL
`
`NOTATION) TO BE
`
`CREATED AND
`
`VIEWED
`
`SIMULTANEOUSLY AT
`
`SEVERAL LEVELS OF
`
`REFINEMENT. II
`
`Up-and-Coming Areas
`GESTURE REcoGNITION
`The
`first pen-based
`input device, the Rand
`tablet, was funded by
`ARPA. Sketchpad used
`light-pen
`gestures
`(1963). Teitelman in
`1964 developed the first
`trainable gesture recog(cid:173)
`nizer. A very early demon-
`stration
`of
`gesture
`recognition was Tom Ellis's
`GRAIL system on the Rand
`tablet (1964, funded by ARPA).
`It was quite common in light-
`pen- based systems to include some
`gesture recognition, for example in the
`AMBIT/G system (1968, funded by ARPA). A
`gesture-based text editor using proofreading
`symbols was developed at CMU by Michael
`Coleman in 1969. Bill Buxton at the Universi(cid:173)
`ty of Toronto has been studying gesture-based
`interactions since 1980. Gesture recognition
`has been used in commercial CAD systems
`since the 1970s and came to universal notice
`with the Apple Newton in 1992.
`
`- ALAN KAY.
`ON SMALLTALK-72
`[ 18, P. 34].
`
`COMPUTER-AIDED DESIGN
`The same 1963 International
`Federation of Information Pro-
`cessing Societies (IFIPS) confer-
`ence at which Sketchpad was
`presented also contained a number of com(cid:173)
`puter-aided design (CAD) systems, includ(cid:173)
`ing Doug Ross's Computer-Aided Design
`Project at MIT in the Electronic Systems Lab
`[40] and Coons's work at MIT with Sketch(cid:173)
`Pad [7]. Timothy Johnson's pioneering work
`on the interactive 3D CAD system Sketch(cid:173)
`pad 3 [15] was his doctoral thesis in 1963
`(funded by the U.S. Air Force) . The first sys(cid:173)
`tem in industry using CAD and computer-
`
`interactions . . . march
`
`• apr i l
`
`1998
`
`Page 6 ofll
`
`
`
`by NASA, U.S. Navy, and others). Another
`important center of current research on 3-D
`interfaces is Fred Brooks's lab at University of
`North Carolina at Chapel H ill (e.g., [2]).
`
`MULTIMEDIA
`The FRESS project at Brown University used
`multiple windows and integrated text and
`graphics (1968, funding from industry). The
`Interactive Graphical Documents project at
`Brown was the first hypermedia (as opposed
`to hypertext) system and used raster graphics
`and text but not video (1979- 1983, funded
`by ONR and NSF). The Diamond
`project at BBN (starting in 1982,
`funded by ARPA) explored
`combining multimedia infor-
`mation (text, spreadsheets,
`graphics, speech). The
`Movie Manual at the
`Architecture Machine
`Group (Min was one
`of the first to demon(cid:173)
`strate mixed video
`and computer graph-
`ics in 1983 (funded
`by ARPA).
`
`THE 'MOUSE' WAS
`
`DEVELOPED BY SRI. IT
`
`IS CONSTRUCTED FROM
`
`TWO POTENTIOMETERS,
`
`MOUNTED ORTHOGONALLY,
`
`EACH OF WHICH HAS A
`
`WHEEL ATTACHED TO ITS
`
`V IRTUAL REALITY AND "AUGMENTED REALITY"
`The original work on virtual reality (VR) was
`performed by Ivan Sutherland when he was at
`Harvard (1965- 1968, funding by Air
`Force, Central Intelligence Agency,
`and Bell Laboratories). Very
`important early work was
`done by Tom Furness when
`he was at Wright-Patter-
`son AFB. Myron
`Krueger's early work at
`the University of Con(cid:173)
`necticut on interfaces
`that incorporate and
`interpret a video of
`the user was influen(cid:173)
`tial. Fred Brooks's and
`Henry Fuch's groups
`at UNC did a lot of
`early research, includ(cid:173)
`ing the study of force
`feedback (1971, fund(cid:173)
`ing from U.S. Atomic
`Energy Commission and
`NSF). Much of the early
`research on head-mounted
`displays and on the DataGlove
`was supported by NASA.
`
`DIMENSIONALITY
`The first 3D system
`was probably Timothy
`Johnson's 3D CAD sys(cid:173)
`tem mentioned earlier
`(1963, funded by the U.S.
`Air Force) [15]. The Lin-
`coln Wand by Larry Roberts
`was an ultrasonic 3D location
`sensing system developed at Lincoln
`Labs (1966, funded by ARPA). That system
`also had the first interactive 3D hidden line
`elimination. An early use was for molecular
`modeling [20]. The late 1960s and early
`1970s saw the flowering of 3D raster graph-
`ics research at the University of Utah with
`Dave Evans, Ivan Sutherland, G.W.
`Romney, Henri Gouraud, Bui-Tuong
`Phong, and G.S. Watkins, much
`of it government funded. Also,
`the military-industrial flight
`simulation work of the 1960s
`, and 1970s led the way to mak-
`1ing 3D interfaces operate in real
`time on commercial systems
`from General Electric, Evans &
`utherland, Singer/Link (funded
`
`SHAFT. THE MOUNTING
`
`FRAME ... IS ENCLOSED
`
`IN A 2 INCH X 2 INCH X 4
`
`INCH WOODEN CASE."
`
`- W.K. ENGLISH, D.C. ENGELBART, AND
`M.L. BERMAN, ON THE NLS SYSTEM
`[10, P. 7]
`
`COMPUTER-SUPPORTED
`
`CooPERATIVE WoRK
`Doug Engelhart's 1968 demonstration of
`NLS [9] included the remote participation of
`multiple people at various sites (funding from
`ARPA, NASA, and Rome ADC). Licklider
`and Taylor predicted online interactive com(cid:173)
`munities in a 1968 article [22] and speculat(cid:173)
`ed about the problem of access being limited
`to the privileged. Electronic mail, still the
`most widespread multiuser software, was
`enabled by the ARPAnet, which became
`operational in 1969, and by the Ethernet
`from Xerox PARC in 1973. An early comput(cid:173)
`er conferencing system was Turoff's EIES sys(cid:173)
`tem at the New Jersey Institute ofTechnology
`(1975).
`
`THREE(cid:173)
`
`1S
`
`i ntera c tion s . . . mar c h
`
`..,. apr il
`
`1 998
`
`Page 7 ofll
`
`
`
`art icl e
`
`-
`
`NATURAL LANGUAGE AND SPEECH
`The fundamental research for speech and nat(cid:173)
`ural language understanding and generation
`has been performed at CMU, MIT, SRI,
`BBN, IBM, AT&T Bell Labs and BellCore,
`much of it government funded. See, for exam(cid:173)
`ple, [37] for a survey of the early work.
`
`Software Tools and Architectures
`The area of user interface (UI) sofrware tools
`is quite active now, and many companies are
`selling tools. Most of today's applications are
`implemented using various forms of sofrware
`tools. For a more complete survey and discus(cid:173)
`sion of UI tools, see [28].
`
`USER INTERFACE MANAGEMENT SYSTEMS
`AND Toouars
`User interface management systems (UIMS)
`and toolkits are sofrware libraries and tools
`that help to create user interfaces. The fust
`UIMS was William Newman's Reaction Han(cid:173)
`dler [33] created at Imperial College, London
`(1966-1967 with SRC funding). Most of the
`early work was done at universities (University
`of Toronto with Canadian government fund(cid:173)
`ing; George Washington University with
`NASA, NSF, Department of Energy, and
`National Bureau of Standards (NBS) funding;
`Brigham Young University with industrial
`funding; and so on). The term "UIMS" was
`coined by David Kasik at Boeing (1982) [16] .
`Early window managers such as Smalltalk
`(1974) and ImerLisp, both from Xerox PARC,
`came with a few widgets, such as popup menus
`and scrollbars. The Xerox Star (1981) was the
`first commercial system to have a large collec(cid:173)
`tion of widgets. The Apple Macintosh (1984)
`was the first to actively promote its toolkit for
`use by other developers to enforce a consistent
`interface. An early C++ toolkit was InterViews
`[23], developed at Stanford (1988, industrial
`funding). Much of the modern research is
`being performed at universities, for example,
`the Garnet (1988) [29] and Amulet (1994)
`[30] projects at CMU (funded by DARPA)
`and SubArctic at Georgia Tech (1 996, funded
`by Intel and NSF) [14].
`
`INTERFACE BUILDERS
`Interface builders are interactive tools that
`
`allow interfaces composed of widgets such as
`buttons, menus, and scrollbars to be placed
`using a mouse. The Steamer project at BBN
`(1979-1985; funded by ONR) demonstrated
`many of the ideas later incorporated into
`interface builders and was probably the first
`object-oriented graphics system. Trillium [13]
`was developed at Xerox PARC in 1981.
`Another early interface builder was the
`MenuLay system [5] developed by Bill Buxton
`at the University ofToronto (1983, funded by
`the Canadian Government). The Macintosh
`(1984) included a Resource Editor that
`allowed widgets to be placed and edited. Jean(cid:173)
`Marie Hullot created SOS Interface in Lisp
`for the Macintosh while working at The
`French National Institute for Research in
`Computer Science and Control (INRIA)
`(1984, funded by the French government)
`which was the first modern interface builder.
`Hullot built this into a commercial product in
`1986 and then went to work for NeXT and
`created the NeXT Interface Builder (1988),
`which popularized this type of tooL Now
`there are literally hundreds of commercial
`interface builders.
`
`COMPONENT ARCHITECTURES
`The idea of creating interfaces by connecting
`separately written components was first
`demonstrated in the Andrew project [35] by
`Carnegie Mellon University's Information
`Technology Center (1983, funded by IBM).It
`is now being widely popularized by Microsoft's
`object linking and embedding (OLE), Apple
`and IBM's OpenDoc, and the Java Beans
`architectures.
`
`Discussion
`It is dear that all of the most impor(cid:173)
`tant innovations in HCI have
`benefited from research at
`both corporate research labs
`and universities. One moti(cid:173)
`vation for this article is to
`overcome the impressions
`some people may have that
`much of the important work
`in human-computer interacr
`tion occurred in industry and
`that if university research in
`
`interact ion s .. . march +a p ril
`
`1 998
`
`Page 8 of ll
`
`
`
`PERMISSION T O MAKE DIGITAL! HARD
`
`COPY OF PART OR All OF T HIS WORK
`
`FOR PERSONAL OR CLASSROOM USE 15
`
`GRANTED WITHOUT FEE PROVIDED
`
`THAT COPIES ARE NOT MADE OR
`
`DISTit.IBUTED FOR PROFIT OR. 00.\t
`
`Mf.R.CALADVANTACE. ntE OOPYRJGHT
`
`NOTICE. T HE rrnE Of ntE PUBUO.
`
`T ION AND m DATE AI"PEAR. AND
`
`NOTICE IS GIVEN THAT COPYING IS BY
`
`PER.MlSSION OF ACM, INC. To COPY
`
`OTHERWISE., TO REPUBUSH, TO POST
`
`ON SERVERS, OR TO RWISTRJBtlfE TO
`
`LISTS REQUfRES PRIOR SPECJFJC PERMIS
`
`SION AND/ OR A FEE.
`
`C ACM 1072 5220/ 98/0300$3.50
`
`"IN EMBEDDED
`MENUS, HIGHLIGHTED
`
`OR UNDERLINED
`
`WORDS OR PHASES
`
`WITHIN THE TEXT
`
`BECOME THE MENU
`ITEMS, AND ARE
`
`industry needs. It seems that only through
`computer science does HCI research dissemi(cid:173)
`nate to products. Furthermore, without
`appropriate levels of funding of academic
`HCI research, there will be fewer doc(cid:173)
`toral graduates in HCI to perform
`research in corporate labs, and
`fewer top-notch graduates in
`this area will be interested
`in being professors, so
`the needed user inter(cid:173)
`face courses will not be
`offered.
`The conventional
`style of graphical
`user interfaces that
`use windows, icons,
`menus, and a mouse
`are in a phase of
`standardization,
`where almost every(cid:173)
`one
`is using
`the
`same, standard tech(cid:173)
`nology and making
`small,
`incremental
`changes. As computers
`perform faster, more of
`the processing power is
`being devoted to the user
`interface. The interfaces of the
`future will use gesture recognition,
`speech recognition, "intelligent agents,"
`adaptive interfaces, video, and many other
`technologies now being investigated by
`research groups at universities and corporate
`labs [38]. Therefore, it is imperative that uni(cid:173)
`versity, corporate, and government-supported
`research continue and be well-supported, so
`that we can develop the science and technolo(cid:173)
`gy needed for the user interfaces of the future.
`
`SELECTABLE USING THE
`COMMONLY USED
`
`TOUCH SCREEN,
`
`CURSOR AND MOUSE
`
`METHODS."
`
`L. KOVED AND
`-
`B. SHNEIDERMAN ON THE
`HYPERTIES SYSTEM [19, P. 312].
`
`HCI is not supported, then industry will just
`carry on anyway. This is simply not true. I
`hope that this paper has shown that many of
`the most famous HCI successes developed by
`companies are deeply rooted in univer-
`sity research. In fact, virtually all of
`today's major interface styles
`and applications have been
`significantly influenced by
`research at universities
`and labs, often with gov(cid:173)
`ernment funding. To
`illustrate this point,
`this article has listed
`the funding sources
`of some of the major
`advances. Without
`this research, many
`of the advances in
`the field of HCI
`would probably not
`have taken place, and
`as a consequence, the
`user interfaces of com(cid:173)
`mercial products would
`be far more difficult to
`use and learn than they are
`today. As described by Stu
`Card [6, p. 162]:
`of
`funding
`Government
`advanced human-computer interac-
`tion technologies built the intellectual capi-
`tal and trained the research teams for pioneer
`systems that, over a period of 25 years, revolu(cid:173)
`tionized how people interact with computers.
`Industrial research laboratories at the corporate
`level in Xerox, IBM, AT&T, and others played a
`strong role in developing this technology and
`bringing it into a form suitable for the commer(cid:173)
`cial arena.
`Another important argument in favor
`of HCI research at universities is that
`, computer science students need to
`know about user interface issues.
`User interfaces are likely to be
`one of the main value-added
`:competitive advantages of the
`! future as both hardware and basic
`software become commodities. If
`students do not know about user
`1interfaces, they will not serve
`
`Acknowledgments
`I must thank a large number of people who
`responded to posts of earlier versions of this
`article on the announcements.chi mailing list
`for their very generous help, and to Jim Hollan
`who helped write the short excerpt of this arti(cid:173)
`cle. Some of the information in this article was
`supplied by (in alphabetical order): Stacey Ash(cid:173)
`lund, Meera M. Blattner, Keith Butler, Stuart
`K Card, Bill Curtis, David E. Damouth, Dan
`
`i nteractions .. . m ar ch
`
`..,. apri l
`
`1 998
`
`Page 9 ofll
`
`
`
`article
`
`Diaper, Dick Duda, Tim T.K
`Dudley, Steven Feiner, Harry
`Forsdick, Bjorn Freeman(cid:173)
`Benson, John Gould,
`Wayne Gray, Mark
`Green, Fred Hansen,
`Bill Hefley, D.
`Austin Henderson,
`Jim Hollan, Jean(cid:173)
`Marie Hullot, Rob Jacob,
`Bonnie
`John,
`Sandy
`Kobayashi, T.K Landauer, John
`Leggett, Roger Lighty, Marilyn Mantei, Jim
`Miller, William Newman, Jakob Nielsen, Don
`Norman, Dan Olsen, Rarnesh Patil, Gary Perl(cid:173)
`man, Dick Pew, Ken Pier, Jim Rhyne, Ben
`Shneiderman, John Sibert, David C. Smith,
`Elliot Soloway, Richard Stallman, Ivan Suther(cid:173)
`land, Dan Swinehart, John Thomas, Alex
`Waibel, Marceli Wein, Mark Weiser, Alan
`Wexelblat, and Terry Winograd. Editorial
`comments were also provided by the forego(cid:173)
`ing as well as by Jim Hollan, Ellen Borison,
`
`Rich McDaniel, Rob
`Miller, Bernita Myers,
`Yoshihiro Tsujino, and
`the reviewers.
`A
`short excerpt
`this
`article
`from
`appeared as part of
`"Strategic Direc(cid:173)
`tions in Human
`Computer Inter(cid:173)
`action," B. Myers, J. Hollan,
`I. Cruz (eds.). ACM Computing Sur(cid:173)
`veys 28, 4 (Dec. 1996), pp. 794-809.
`This research was sponsored partly by
`NCCOSC under Contract No. N66001-94-
`C-6037, ARPA Order No. B326, and partly
`by the National Science Foundation under
`Grant No. IRI-9319969. The views and con(cid:173)
`clusions contained in this document are those
`of the author and should not be interpreted as
`representing the official policies, either
`expressed or implied, of NCCOSC or the
`U.S. Government.
`
`References
`1. Baecker, R., et al. ''A Historical and Intel(cid:173)
`lectual Perspective." In Readings in
`Human- Computer Interaction: Toward the Year
`2000. (R. Baecker eta!., eds.). 2nd ed. Mor(cid:173)
`gan Kaufmann Publishers, Inc., San Francisco,
`
`1995, pp. 35-47.
`2. Brooks, F. "The computer 'scientist' as tool(cid:173)
`smith-Studies in interactive computer graph(cid:173)
`ics." In Proceedings of IFIP Conference, 1977,
`pp. 625-634.
`3. Burtnyk, N. and Wein, M . "Computer
`generated key frame animation." journal of the
`Society of Motion Picture and Television Engi(cid:173)
`neers 8, 3 (1971 ), pp. 149- 153.
`4. Bush, V. "As we may think." The Atlantic
`Monthly 1760u1y 1945), pp. 101- 108.
`Reprinted and discussed in interactions, 3, 2
`(Mar. 1996), pp. 35-67.
`5. Buxton, W. et a!. "Towards a comprehen(cid:173)
`sive user interface management system." In
`Computer Graphics 17 (1983), pp. 35-42. Pro(cid:173)
`ceedings ofSIGGRAPH'83, Detroit, MI.
`6. Card, S.K " Pioneers and Settlers: Methods
`Used in Successful User Interface Design." In
`Human-Computer Interface Design: Success Sto(cid:173)
`ries, Emerging Methoth, and Real-World Con-
`
`text (M. Rudisill et al., eds.). Morgan Kauf(cid:173)
`mann Publishers, San Francisco, 1996, pp.
`122- 169.
`7. Coons, S. "An Outline of the Requirements
`for a Computer-Aided Design System." In
`Proceedings of AF/PS Spring joint Computer
`Conference 23 (1963), pp. 299- 304.
`8. Engelhart, D . "The Augmented Knowledge
`Workshop." In A History of Personal Worksta(cid:173)
`tions (A Goldberg, ed.). Addison-Wesley Pub(cid:173)
`lishing Company, New York, 1988, pp.
`187-232.
`9. Engelhart, D. and English, W. "A Research
`Center for Augmenting Human InteUect."
`Reprinted in ACM S/GGRAPH Video Review
`106(1994), video made in 1968
`10. English, W.K, Engelhart, D .C., and
`Berman, M.L. "Display selection techniques
`for text manipulation." IEEE Transactions on
`Human Factors in Electronics HFE-8, 1 (1967),
`pp . 5-15.
`11. Goldberg, A., ed. A History of Personal
`Workstations. Addison-Wesley Publishing
`Company, New York, 1988.
`12. Goldberg, A. and Robson, D. ''A
`Metaphor for User Interface Design." In Pro(cid:173)
`ceedings of the 12th Hawaii International Con-
`
`f~nce on System Sciences 1 (1979), pp.
`148-157.
`13. Henderson J r., D.A "The Trillium user
`interface design environment." In Proceedings
`ofSIGCHI '86, Human Factors in Computing
`Systems, Boston, MA, 1986, pp. 221- 227.
`14. H udson, S.E. and Smith, I. "Ultra(cid:173)
`Lightweight Constraints." In Proceedings of
`ACM S/GGRAPH Symposium on User Inter(cid:173)
`face Software and Technology (UIST '96), Seat(cid:173)
`de, WA, 1996, pp. 147- 155. System available
`from
`http://www.cc.gatech.edu/gvului/sub_atctic/
`15. Johnson, T. "Sketchpad III: Three
`Dimensional Graphical Communication with
`a Digital Computer." In Proceedings of AF/PS
`Spring joi