TRADING EXHIBIT 2007
`TRADESTIATION ET AL. v. TRADING TECH
`CBM2016-00051
`
`

`
`artlcle —
`
`
`
`Brad A. Myers
`Human-Computer
`Interaction Institute
`
`School of Computer
`Science
`
`Carnegie Mellon
`
`University
`Pittsburgh, PA 15213-
`3891
`
`bam0cs.cmu.edu
`
`
`
`Research in human—computer
`
`interac-
`
`tion (HCI) has been spectacularly suc-
`
`cessful and has fundamentally changed
`
`computing. One example is the ubiqui-
`
`tous
`
`graphical
`
`interface
`
`used
`
`by
`
`Microsoft Windows 95, which is based
`
`on the Macintosh, which is based on
`
`work at Xerox PARC, which in turn is
`
`based on early research at the Stanford
`
`Research Laboratory (now SRI) and at
`
`Massachusetts Institute of Technology.
`
`Another example is that virtually all soft-
`ware written today employs user
`interface
`toolkits and interface builders—concepts that
`were developed first at universities. Even the
`remarkable growth of the World Wide Web is
`a direct
`result of HCI research: applying
`hypertext technology to browsers allows one
`to traverse a link across the world with a click
`
`of the mouse. More than anything else,
`improvements to interfaces have triggered this
`explosive growth. Furthermore, the research
`that will lead to the user interfaces for the
`
`computers of tomorrow is being carried out
`today at universities and a few corporate
`research labs.
`
`This paper attempts to briefly summarize
`many of the important research developments
`in HCI teclmology, emphasizing the role of
`university research, which may not be widely
`recognized. By “research,” I mean exploratory
`work at universities and government and cor-
`porate research labs (such as Xerox PARC)
`that is not directly related to products. By
`“HCI technology,” I am referring to the com-
`puter side of HCI. A companion article on the
`history of the “human side,” discussing the
`contributions
`from psychology,
`design,
`human factors, and ergonomics would also be
`appropriate.
`
`Figure 1 shows the time span for some of
`the technologies discussed in this article.
`including when they were introduced. Of
`course, a deeper analysis would reveal signifi-
`cant interaction among the university, corpo-
`rate research, and commercial activity lines. It
`is
`important
`to appreciate that years of
`research are involved in creating and making
`these technologies ready for widespread use.
`The same will be true for the HCI technolo-
`
`gies currently being developed that will pro-
`vide the interfaces of tomorrow.
`
`Clearly it is impossible to list every system
`and source in a paper of this scope, but I have
`tried to represent the earliest and most influ-
`ential systems. Further information can be
`
`found in other surveys of HCI topics (see, for
`example,
`[1, 11, 36, 41]). Another useful
`resource is the video All The Widgets, which
`shows the historical progression of a number
`of user interface ideas [27].
`
`The technologies discussed in this paper
`include fundamental interaction styles such as
`
`the mouse pointing
`direct manipulation,
`device, and windows; several important kinds
`of application areas, such as drawing,
`text
`editing, and spreadsheets;
`the technologies
`that will likely have the biggest impact on
`interfaces of the future, such as gesture recog-
`nition, multimedia, and three-dimensionality;
`and the technologies used to create interfaces
`using the other technologies, such as user
`interface management systems, toolkits, and
`interface builders.
`
`Basic Interactions
`Dmecr Mmmuumou or
`
`GRAPHICAL OBJECTS
`
`The now ubiquitous direct manipulation inter-
`face, where visible objects on the screen are
`directly manipulated with a pointing device,
`was first demonstrated by Ivan Sutherland in
`Sketchpad [47], the thesis of his doctoral dis-
`sertation in 1963. Sketchpad supported the
`manipulation of objects using a light pen,
`including grabbing objects, moving them,
`changing size, and using constraints. It con-
`tained the seeds of myriad important interface
`ideas. The system was built at Lincoln Labs
`with support from the U.S. Air Force and the
`National Science Foundation (NSF).
`
`interactions..
`
`.march -1. april
`
`I998
`
`Page 2 of 11
`
`

`
`KEY:
`
`_ University Research
`
`technologies discussed in this article.
`
`2 C°'P°'at° Re‘°a"h
`: Commercial Products
`
`Figure 1. Approximate time lines showing where
`and when work was performed on some major
`
`Direct Manipulation of Graphical Objects
`
`The Mouse
`
`interactions...march + april
`
`I998
`
`Page 3 of 11
`
`

`
`William Newmans Rmction Handler [33],
`
`created at Imperial College, London during
`1966 and 1967, provided direct manipulation
`of graphics and introduced Light Handles
`[32], a fonn of graphical potentiometer that
`was probably the first “widget.” Another early
`system was AMBIT/G (implemented at the
`Massachusetts Institute of Technology’s
`(MIT) Lincoln labs in 1968 and
`
`article —
`
`NLS project (funding from ARPA, NASA.
`and Rome ADC) [10].1t was intended to be a
`
`cheap replacement for light pens, which had
`been used at least since 1954 [11, p. 68].
`Many of the current uses of the mouse were
`demonstrated by Doug Engelbart as part of
`NLS in a movie created in 1968 [9]. The
`mouse was then made famous as a
`
`input device by Xerox
`practical
`PARC in the 19705.
`It
`first
`
`appeared commercially as part
`of the Xerox Star (1981), the
`
`Three Rivers Computer
`Company's PERQ (1981)
`[25].
`the Apple Lisa
`(1982), and the Apple
`Macintosh (1984).
`
`WINDOWS
`Multiple tiled windows
`were
`demonstrated
`in
`
`funded by the Advanced Research
`Projects Agency (ARPA).
`It
`employed, among other inter-
`face techniques, iconic repre-
`sentations, gesture
`recog-
`nition, dynamic menus with
`items selected usingapoint-
`ing device, selection of icons
`by pointing, and moded
`and mode-free styles of
`interaction.
`David Canfield Smith
`
`coined the term “icons” in his
`1975 doctoral
`thesis on Pyg-
`malion [441 (ftmded by ARPA
`and National Institute of Mental
`
`"‘-THE's;l(E,T'C’HPAb
`
`svsrejr/4-V M_Ag(res 7 IT
`.|?9SS,J,Bl£E. ran
`MANAND A;coM_
`(
`___
`Pu7TE_R5lr?ic°N'
`'VER§E‘RAPlDL;_Y-
`THROUGH-THE
`'
`'
`a’
`MEDlUM-DFKLWE
`
`nnfiwjuagg
`,
`—"V“‘.' "’v"-“"‘,"‘."."
`(‘,7-I’-I-.72-9)
`
`Engelbart's NLS in 1968
`[9]. Early research at Stan-
`ford on systems like com-
`LOT (1974)
`[49] and at MIT
`with the EMACS text editor (1974)
`[46] also demonstrated tiled windows.
`
`Alan Kay proposed the idea of overlapping
`windows in his 1969 doctoral thesis [17], and
`
`overlapping windows first appeared in 1974 in
`his Smalltalk system [12] at Xerox PARC, and
`soon afterward in the InterLisp system [50].
`Some of the first commercial uses of win-
`
`dows were on Lisp Machines lnc. (LMI) and
`Symbolics Lisp Machines (1979). which grew
`out of MIT Artificial Intelligence (Al) Lab pro-
`jects. The Cedar VVindow Manager from Xerox
`PARC was the first major tiled window manag-
`er (I981) [48], followed soon by the Andrew
`window manager [35] by Carnegie Mellon
`University's
`(CMU)
`lnfonnation
`Technology Center (1983. funded
`by IBM). The main commercial
`systems popularizing windows
`were the Xerox Star (1981), the
`
`Apple Lisa (1982), and most
`important,
`the Apple Macin-
`tosh (1984). The early versions of
`the Star and Microsoft \X/indows
`
`were tiled, but eventually they sup-
`ported overlapping windows like the
`
`
`
`Health — NIMH). Smith later popu-
`larized icons as one of the chief designers
`of the Xerox Star [45]. Many of the interac-
`tion techniques popular in direct manipula-
`tion interfaces, sudt as how objects and text
`are selected, opened, and manipulated, were
`researched at Xerox PARC in the 1970s. In
`
`the idea of “WYSIWYG” (what
`particular.
`you see is what you get) originated there with
`systems such as the Bravo text editor and the
`
`Draw drawing program [11]
`The concept of direct manipulation inter-
`faces for everyone was envisioned by Alan Kay
`of Xerox PARC in a 1977 article about the
`
`Dynabook [18]. The first commercial systems
`to use direct manipulation extensively were the
`Xerox Star (1981) [45], the Apple Lisa (I982)
`[54], and the Macintosh (1984)
`[55]. Ben
`
`the University of Maryland
`Shneiderrnan at
`coined the term “direct manipulation” in 1982,
`identified the components, and gave psycholog-
`ical motivations for direct manipulation [43].
`
`Mouse
`
`The mouse was developed at Stanford
`Research Laboratory in 1965 as part of the
`
`interactions..
`
`.marcb + april
`
`1998
`
`Page 4 of 11
`
`

`
`
`
`
`
`
`
`
`Lisa and Macintosh. The X Window System, a
`
`
`
`
`
`current international standard, was developed
`
`
`
`
`
`
`
`
`
`at MIT in 1984 [42]. For a survey of window
`
`
`
`managers, see [26].
`
`
`
`
`
`
`
`
`based display editors that was widely used
`
`
`
`
`
`
`
`[51]. The Hypertext Editing System [53, p.
`
`
`
`
`
`
`
`108] from Brown University had screen edit-
`
`
`
`
`
`
`ing and formatting of arbitrary-sized strings
`
`
`
`
`
`
`
`with a light pen in 1967 (fiinding from
`
`
`
`IBM). NLS demonstrated mouse-
`
`
`
`
`
`
`based editing in 1968 (Figure 4).
`
`
`
`
`
`
`TECO from MIT was an early
`
`
`
`
`screen editor
`(1967), and
`
`
`
`
`EMACS [46] was developed
`from it
`in 1974. Xerox
`
`
`
`
`
`
`
`
`
`
`PARC's Bravo [11, p. 284]
`was the first WYSIWYG
`
`
`
`
`
`
`(1974).
`editor-formatter
`
`
`
`
`
`It was designed by Butler
`
`
`
`Lampson and Charles
`
`
`
`
`Simonyi, who had started
`
`
`
`working on these con-
`
`
`
`
`cepts about 1970 while at
`
`
`
`Berkeley. The first com-
`mercial WYSIWYG edi-
`
`
`Star,
`tors
`the
`were
`
`
`later,
`and,
`LisaWrite,
`
`
`
`
`
`
`
`Macwrite. For surveys of
`
`
`
`
`
`
`text editors, see [24] and [53].
`
`"THE o'u=|'=i'cuI.tv
`
`
`
`
`oi? MANI(lN_D"S
`
`
`
`
`
`PROBLEMS WAS
`
`
`
`
`
`
`'|nc_n£As|_I_\|G; AT; A
`GREATER-'RiATE THAN"
`
`
`
`
`
`
`
`oun J‘-i\'_Bl.L-ITY to
`cope. ARE.IN
`
`
`
`
`
`IRQUBLE.) [so] I
`
`
`
`
`
`c'oMMI1'rEb MY.
`
`
`
`
`C1-'_.lREE'R to
`
`
`
`
`
`‘AUGMENTING THE-
`
`
`HUMAN I._NTEL'_LE.CT."'
`
`
`
`
`
`
`
`
`
`rH.i.=
`— nous. Euastmin-1;
`
`
`
`
`NL5 SYSTEM ' [8, P. 189]
`
`
`
`
`
`
`
`Applications
`DRAWING PROGRAMS
`
`
`
`
`
`
`Much of the current technology
`
`
`
`
`for drawing programs was
`demonstrated in Sutherland's
`
`
`
`
`
`1963 Sketchpad system
`
`
`
`
`
`
`[47]. The use ofa mouse to
`
`
`
`manipulate graphics was
`demonstrated in NLS
`
`
`
`In 1968 Ken
`(1965).
`
`
`
`and
`Grant
`Pulfer
`
`
`
`Bechthold at the Nation-
`
`
`
`:11 Research Council of
`
`
`
`
`Canada built a mouse out
`
`
`
`
`
`
`
`
`of wood patterned after
`
`
`
`Engelbart’s mouse and
`
`
`
`
`
`used it with a key-frame
`
`
`
`
`animation system to draw
`all the frames of a movie. A
`
`
`
`
`
`
`
`
`
`
`subsequent movie in 1971,
`
`
`
`
`
`Hunger; won a number of
`
`
`
`
`
`
`awards and was drawn using a
`
`
`
`
`
`tablet instead of the mouse (fund-
`
`
`
`
`
`
`
`ing by the National Film Board of
`
`
`
`
`
`Canada)
`‘William Newman’s Markup
`
`
`
`
`
`
`
`(1975) was the first drawing program for
`
`
`
`
`
`
`Xerox PARC's Alto,
`followed shortly by
`Patrick Baudelaire’s Draw, which added han-
`
`
`
`
`
`
`
`
`
`
`
`
`
`dling of lines and curves [1], p. 326]. The
`
`
`
`
`
`first computer painting program was probably
`
`
`
`
`Dick
`Shoup’s
`Superpaint
`at
`PARC
`
`
`
`
`
`(197<i—1975). The first widely used drawing
`
`
`
`
`
`programs were MacPaint and MacDraw on
`
`
`
`the Macintosh (1984).
`
`
`
`
`
`
`
`Sr-nnaosussrs
`
`
`
`
`
`
`The initial spreadsheet was Visi-
`
`
`
`
`
`Calc which was developed by
`
`
`
`
`
`
`Franlcston and Bricldin (1977-1978) for
`
`
`
`
`
`
`
`
`
`the Apple II while they were students at MIT
`
`
`
`
`
`
`and the Harvard Business School, respectively.
`
`
`
`
`
`The solver was based on a dependency-direcu
`
`
`
`
`
`ed backtracking algoridlm by Sussman and
`Stallman at the MIT AI Lab.
`
`
`
`
`
`
`
`
`
`HYI’ERTEXT
`
`
`
`
`
`
`
`
`
`
`
`The idea for hypertext (by which documents are
`
`
`
`
`
`
`
`linked to related documents) is credited to Van-
`nevar Bush's famous MEMEX idea from 1945
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`[4}. Ted Nelson coined the term “hypertext” in
`
`
`
`
`
`
`
`
`1965 [31]. Engelbart’s NLS system [9] at the
`Stanford Research Laboratories in 1965 made
`
`
`
`
`
`
`
`
`
`
`
`
`
`extensive use of linking (funding from ARPA,
`
`
`
`
`
`
`
`NASA, and Rome ADC). The “N15 journal”
`
`
`
`
`
`
`
`
`
`
`[11, p. 212] was one of the first on-line journals
`
`
`
`
`
`
`
`
`and included full linking of articles (1970). The
`
`
`
`
`
`
`Hypertext
`System, jointly designed by
`
`
`
`
`
`
`
`
`Andy van Dam, Ted Nelson, and two students
`
`
`
`
`
`
`
`at Brown University (funding fi'om IBM} was
`
`
`
`distributed extensively [5 2].
`
`
`
`TEXT EDITING
`
`
`In 1962, at the Stanford Research
`
`
`
`
`
`
`
`
`
`Laboratory, Engelbart proposed,
`
`
`
`
`
`and later implemented, a word
`
`
`
`
`processor with automatic word
`
`
`
`
`
`wrap, search and replace, user-
`
`
`
`
`
`definable macros, scrolling text,
`
`
`
`
`
`and commands to move, copy,
`and delete characters, words, or
`
`
`
`
`
`blocks of text. Stanford’s TX/Edit
`
`
`
`
`
`(1965) was one of the first CRT-
`
`
`
`
`
`
`
`interactions... march + april
`
`
`
`
`
`
`
`1998
`
`
`
`
`Page 5 of 11
`
`
`
`
`
`
`
`
`
`

`
`artlcle —
`
`manufacturing
`aided
`(CAM) was probably
`General Motors DAC-
`
`1 (about 1963).
`
`VIDEO GAMES
`
`The first graphi-
`cal video game
`was
`probably
`SpaceWar by Slug
`Russel of MIT in 1962 for
`
`
`
`The University of Vermont’s
`PROMIS (1976) was the first
`
`hypertext system released to
`the user community.
`It
`was used to link patient
`and patient care infor-
`mation at the Univer-
`
`sity of Vermont’s
`medical
`cemer. The
`
`from
`(1977)
`ZOG project
`CMU was another early hypertext
`system and was funded by Office of
`Naval Research (ONR) and ARPA [39]. Ben
`
`Shneiderman’s Hyperties was the first system
`in which highlighted items in the text could
`be clicked on to go to other pages (1983, Uni-
`versity of Maryland) [19]. HyperCard from
`Apple (1988) significantly helped to
`bring the idea to a wide audience.
`Tim Berners-Lee used the
`
`the PDP-1 [21, p. 49], including the
`first computer joysticks. The early computer
`Adventure game was
`created by Will
`Crowther at Bolt, Beranek 86 Newman
`
`(BBN), and Don Woods developed it into a
`more sophisticated Adventure game at Stan-
`ford in 1966 [21, p. 132]. Conway’s
`game of LIFE was developed on
`computers at MIT and Stan-
`ford in 1970. The first popu-
`lat commercial game was
`Pong (about 1976).
`
`If/[v|uL1|pLE
`..
`_
`w|>ND°ws ALLOW A-
`DOCUMENT.
`
`idea to create the
`hypertext
`World Wide Web in 1990 at
`the
`government-funded
`European Particle Physics
`laboratory
`(CERN).
`Mosaic, the first popular
`hypertext browser for
`the World-Wide Web,
`
`was developed at
`the
`University of Illinois’
`National Center
`for
`Supercomputer Appli-
`cations (NCSA). For a
`more complete history of
`HyperText, see [34].
`
`COMPUTER-AIDED DESIGN
`The same 1963 International
`Federation of Information Pro-
`
`‘(COMPOSED o‘I'='rExT.‘
`
`Up-and-Coming Areas
`GESTURE RECOGNITION
`
`»p|ct.untE's} .Mus'|<:AL
`
`NOTATION) TO. BE]
`
`CREATED AND
`VlEWE‘D
`5'MuL-|-IANEOUSLY AT,
`I
`'
`i
`\
`SEVERAL -1~'E~vE-Ls of
`REFINEMENT."
`_ AMNKAY
`9)., 5M,qu_1_-,g_K.'72
`‘"33’-i‘341-I
`
`pen-based
`first
`The
`input device, the Rand
`tablet, was funded by
`ARPA. Sketchpad used
`light-pen
`gestures
`(1963). Teitelman in
`1964 developed the first
`trainable gesture recog-
`nizer. Avery early demon-
`stration
`of
`gesture
`recognition was Tom Ellis’s
`GRAIL system on the Rand
`tablet (1964, funded by ARPA).
`It was quite common in light-
`pen—based systems to include some
`gesture recognition,
`for example in the
`AMBIT/G system (1968, funded by ARPA). A
`gesture-based text editor using proofreading
`symbols was developed at CMU by Michael
`Coleman in 1969. Bill Buxton at the Universi-
`
`ty of Toronto has been studying gesture-based
`interactions since 1980. Gesture recognition
`has been used in commercial CAD systems
`since the 1970s and came to universal notice
`
`with the Apple Newton in 1992.
`
`cessing Societies (IFIPS) confer-
`ence
`at which
`Sketchpad was
`presented also contained a number of com-
`puter-aided design (CAD) systems, includ-
`ing Doug Ross’s Computer-Aided Design
`Project at MIT in the Electronic Systems Lab
`[40] and Coons’s work at MIT with Sketch-
`
`Pad [7]. Timothy ]ohnson’s pioneering work
`on the interactive 3D CAD system Sketch-
`pad 3 [15] was his doctoral thesis in 1963
`(funded by the U.S. Air Force). The first sys-
`tem in industry using CAD and computer-
`
`interactions..
`
`.march + april
`
`I998
`
`Page 6 of 11
`
`

`
`MULTIMEDLA
`
`
`
`
`
`
`
`
`by NASA, U.S. Navy, and others). Another
`
`
`
`
`
`
`
`important center of current research on 3-D
`
`
`
`
`
`
`
`
`interfaces is Fred Brool-rs’s lab at University of
`
`
`
`
`
`
`
`Nort-h Carolina at Chapel Hill (e.g., [2]).
`
`
`
`
`
`
`
`
`
`The PRESS project at Brown University used
`
`
`
`
`
`
`multiple windows and integrated text and
`
`
`
`
`
`
`graphics (1968, funding from industry). The
`
`
`
`
`
`Interactive Graphical Documents project at
`
`
`
`
`
`
`
`Brown was the first hypermedia (as opposed
`
`
`
`
`
`
`
`to hypertext) system and used raster graphics
`
`
`
`
`
`
`
`and text but not video (1979-1983, funded
`
`
`
`
`
`
`by ONR and NSF). The Diamond
`
`
`
`
`
`
`project at BBN (starting in 1982,
`
`
`
`
`funded by ARPA) explored
`
`
`combining multimedia infor-
`
`
`
`mation (text, spreadsheets,
`
`
`
`graphics,
`speech). The
`Movie Manual at
`the
`
`
`
`
`Architecture Machine
`
`
`
`
`
`
`Group (MIT) was one
`of the first to demon-
`
`
`
`
`strate mixed video
`
`
`
`
`
`and oomputer graph-
`
`
`
`ics in 1983 (funded
`
`
`by ARPA).
`
`
`
`
`T|_-IE "Mo_u_‘sE-* was‘
`
`
`
`
`
`DEVELOPEIQ av sm.
`‘Is cons-mucrrn FROM
`
`
`
`-I-'I'
`
`
`
`
`
`- T-W0 Po1En"Ir1'o'MErErt3s,
`M_6___Ul\_lTED _IORTl-il_i0GONA_L_LY.
`
`
`
`
`
`
`' EACH o_I=_ wHI_ci-I I-IA-S--A
`WHEEL ATTACHED TO ITS
`
`
`
`
`
`
`
`
`
`
`VIRTUAL REALITY AND “AUGMENIED REALITY”
`
`
`
`
`
`
`
`
`
`
`
`
`
`The original work on virtual reality (VR) was
`
`
`
`
`
`
`
`
`performed by Ivan Sutherland when he was at
`
`
`
`
`
`Harvard (1965-1968, fimding by Air
`
`
`
`
`Force, Central Intelligence Agency,
`
`
`
`
`and Bell Laboratories). Very
`
`
`
`
`important early work was
`
`
`
`
`
`done by Tom Furness when
`
`
`
`he was at \Wri.ght-Patter-
`
`
`
`son
`AFB.
`Myron
`
`
`
`
`Krueger’s early work at
`
`
`
`the University of Con-
`necticut on interfaces
`
`
`
`
`
`
`that
`incorporate and
`
`
`
`
`interpret a video of
`the user was influen-
`
`
`
`Fred Brool-cs's and
`
`
`
`
`
`
`
`Henry Fuclfs groups
`at UNC did a lot of
`
`
`
`
`
`
`
`
`early research, includ-
`
`
`
`
`
`ing the study of force
`
`
`feedback (1971,
`fund-
`
`
`
`
`ing fi'om U.S. Atomic
`
`
`
`Energy Commission and
`
`
`
`
`
`NSF). Much of the early
`research on head-mounted
`
`
`
`
`
`
`
`
`displays and on the DataGlove
`
`
`
`
`was supported by NASA.
`
`
`
`
`
`
`
`
`
`COMPUTER-SUPPORTED
`COOPERATIVE WORK
`
`
`
`
`
`
`
`
`
`
`
`Doug Engelbart's 1968 demonstration of
`
`
`
`
`
`
`
`NLS [9] included the remote participation of
`
`
`
`
`
`
`
`multiple people at various sites (funding from
`
`
`
`
`
`
`ARPA, NASA, and Rome ADC). Licldider
`
`
`
`
`
`and Taylor predicted online interactive com-
`
`
`
`
`
`
`munities in a 1968 article [22] and speculat-
`
`
`
`
`
`
`
`
`ed about the problem of access being limited
`
`
`
`
`
`
`
`to the privileged. Electronic mail, still the
`
`
`
`
`
`most widespread multiuser software, was
`
`
`
`
`
`
`enabled by the ARPAnet, which became
`
`
`
`
`
`
`
`operational
`-in 1969, and by the Ethernet
`
`
`
`
`
`
`
`from Xerox PARC in 1973. An early comput-
`
`
`
`
`
`
`er conferencing system was Turoff’s EIES sys-
`
`
`
`
`
`
`tem at the New Jersey Institute of Technology
`
`{I975}.
`
`
`
`-sI'-un-*r.. 1-HE.-Mo.uN'1ii1__ma
`
`
`I=_'s‘AME-
`_I_s _sN'cLos-an
`
`
`
`
`
`
`
`
`
`
`IN A.-2 _IN.CH :x- 2_ INCH x -4
`INCH WOODEN EASE."
`
`
`
`— FIEK. ENGLISH,-D;C. ENGELBARI; AND-
`
`
`
`
`M.:.. s.-.=':=rn_»IAN,'_ oiv THE. ms svsrsm
`
`
`
`
`
`
`
`
`.1’'9 3 7?
`
`THREE-
`DIMENSIONALITY
`
`
`
`
`
`
`
`
`
`
`
`
`The first 3D system
`
`
`
`was probably Timothy
`
`
`
`Johnson's 3D CAD sys-
`tem mentioned earlier
`
`
`
`
`
`
`
`
`{1963, funded by the U.S.
`
`
`
`
`Air Force)
`[15]. The Lin-
`
`
`
`
`
`coln Wand by Larry Roberts
`was an ultrasonic 3D location
`
`
`
`
`
`
`
`
`
`
`sensing system developed at Lincoln
`
`
`
`
`
`
`
`I_.abs (1966, funded by ARPA). That system
`also -had the first interactive 3D hidden line
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`elimination. An early use was for molecular
`
`
`
`
`
`
`
`modeling [20]. The late 1960s and early
`
`
`
`
`
`
`
`1970s saw the flowering of 3D raster graph-
`
`
`
`
`
`
`
`
`ics research at the University of Utah with
`
`
`
`
`
`Dave Evans,
`Ivan Sutherland, G.W.
`
`
`
`
`Romney, Henri Gouraud, Bui-Tuong
`
`
`
`
`
`Phong, and G.S. Watkins, much
`
`
`
`
`
`of it government filnded. Also,
`
`
`
`the military-industrial
`flight
`simulation work of the 19605
`
`
`
`
`
`
`
`
`
`
`
`
`and 19705 led the way to malt-
`
`
`
`
`
`
`"ing 3D interfaces operate in real
`
`
`
`
`time on commercial
`systems
`
`
`
`
`from General Electric, Evans 86
`
`
`
`
`Sutherland, Singer.~‘Lini( (funded
`
`interacrions..
`
`
`.rnarcl1 + april
`
`
`
`
`
`
`I993
`
`
`
`
`Page 7 of 11
`
`
`
`

`
`article —
`
`NATURAL LANGUAGE AND SPEECH
`
`The fundamental research for speech and nat-
`ural language understanding and generation
`has been performed at CMU, MIT, SRI,
`BBN, IBM, AT8cT Bell Labs and BellCo1e,
`
`much of it government funded. See, for exam-
`ple, [37] for a survey of the early work.
`
`Software Tools and Architectures
`
`The area of user interface (UI) software tools
`
`is quite active now. and many companies are
`selling tools. Most of today’s applications are
`implemented using various forms of software
`tools. For a more complete survey and discus-
`sion of UI tools, see [28].
`
`User: INTERFACE MANAGEMENT Svsrms
`AND Toouurs
`
`User interface management systems (UIMS)
`and toolkits are software libraries and tools
`
`allow interfaces composed of widgets such as
`buttons, menus. and scrollbars to be placed
`using a mouse. The Steamer project at BBN
`(1979-1985: funded by ONR) demonstrated
`many of the ideas later incorporated into
`interface builders and was probably the first
`object-oriented graphics system. Trillium [13]
`was developed at Xerox PARC in 1981.
`Another early interface builder was the
`MenuLay system [5] developed by Bill Buxton
`at the University of Toronto (I983, funded by
`the Canadian Government). The Macintosh
`(1984)
`included a Resource Editor
`that
`
`allowed widgets to be placed and edited. Jean-
`Marie Hullot created SOS Interface in Lisp
`for
`the Macintosh while working at The
`French National
`Institute for Research in
`
`(INRIA)
`Computer Science and Control
`(1984, funded by the French government)
`which was the first modern interface builder.
`
`that help to create user interfaces. The first
`UIMS was William Newman’s Raction Han-
`
`Hullot built this into a commercial product in
`1986 and then went to work for NeXT and
`
`created the NeXT Interface Builder (1988),
`
`which popularized this type of tool. Now
`there are literally hundreds of commercial
`interface builders.
`
`Coupomanr Ancurrecrunns
`
`The idea of creating interfaces by connecting
`separately written components was
`first
`demonstrated in the Andrew project [35] by
`Carnegie Mellon University’s
`Information
`Technology Center (1983, funded by IBM). It
`is now being widely popularized by Microsoft’s
`object linking and embedding (OLE), Apple
`and IBM’s OpenDoc, and the java Beans
`architectures.
`
`dler [33] created at Imperial College, London
`(1966-1967 widi SRC funding). Most of the
`early work was done at universities (University
`of Toronto with Canadian government fiind-
`ing; George Washington University with
`NASA, NSF, Department of Energy, and
`National Bureau of Standards (NBS) funding;
`Brigham Young University with industrial
`funding; and so on). The term “UIMS” was
`coined by David Kasik at Boeing (1982) [16].
`Early window managers such as Smalltalk
`(1974) and InterLisp, both from Xerox PARC,
`came with a few widgets, such as popup menus
`and scrollbars. The Xerox Star (1981) was the
`
`first commercial system to have a large collec-
`tion of widgets. The Apple Macintosh (1984)
`was the first to actively promote its toolkit for
`use by other developers to enforce a consistent
`interface. An early C++ toolkit was InterViews
`[23], developed at Stanford (1988, industrial
`funding). Much of the modern research is
`being performed at universities, for example,
`the Garnet (I988) [29] and Amulet (1994)
`
`[30] projects at CMU (funded by DARPA)
`and SubArctic at Georgia Tech (1996, funded
`by Intel and NSF) [I4].
`
`INTERFACE BUILDERS
`
`Interface builders are interactive tools that
`
`interactions..
`
`.marcl1 + april
`
`I998
`
`Page 8 of 11
`
`Discussion
`
`It is clear that all of the most impor-
`tant
`innovations in HCI have
`benefited from research at
`
`both corporate research labs
`and universities. One moti-
`vation for this article is to
`
`overcome the impressions
`some people may have that
`much of the important work
`in human—computer interac-
`tion occurred in industry and
`that
`if university research in
`
`
`
`

`
`
`
`
`
`
`
`industry needs. It seems that only through
`
`
`
`
`
`computer science does HCI research dissemi-
`
`
`
`
`nate to products. Furthermore, without
`
`
`
`
`
`appropriate levels of funding of academic
`HCI research, there will be fewer doc-
`
`
`
`
`
`
`
`
`
`
`
`
`toral graduates in HCI to perform
`
`
`
`
`
`research in corporate labs, and
`
`
`
`
`fewer top-notch graduates in
`this area will be interested
`
`
`
`
`
`
`
`
`
`in being professors, so
`the needed user inter-
`
`
`
`face courses will not be
`
`
`
`
`
`offered.
`
`The conventional
`
`
`
`
`
`style of graphical
`user interfaces that
`
`
`
`use windows, icons,
`
`
`
`
`
`
`menus, and a mouse
`
`
`
`
`are in a phase of
`standardization,
`
`
`where almost every-
`
`
`
`is using the
`one
`same, standard tech-
`
`
`
`
`
`nology and making
`
`
`incremental
`small,
`
`
`
`changes. As computers
`
`
`
`
`perform fiister, more of
`
`
`
`
`the processing power
`is
`
`
`
`
`
`being devoted to the user
`interface. The interfaces of the
`
`
`
`
`
`
`
`
`
`
`future will use gesture recognition,
`
`
`
`
`speech recognition, “intelligent agents,”
`
`
`
`
`
`
`adaptive interfaces, video, and many other
`
`
`
`
`
`technologies now being investigated by
`
`
`
`
`
`
`research groups at universities and corporate
`
`
`
`
`
`
`
`labs B8}. Therefore, it is imperative that uni-
`
`
`
`versity, corporate, and government-supported
`
`
`
`
`
`research continue and be well-supported, so
`
`
`
`
`
`
`
`that we can develop the science and technolo-
`
`
`
`
`
`
`
`
`
`gy needed for the user interfaces of tl1e future.
`
`
`
`
`
`
`
`
`
`HCl is not supported, then industry will just
`
`
`
`
`
`
`
`
`
`carry on anyway. This is simply not true.
`I
`
`
`
`
`
`
`
`
`
`hope that this paper has shown that many of
`
`
`
`
`
`
`
`die most famous HCI successes developed by
`
`
`
`
`
`companies are deeply rooted in univer-
`
`
`
`
`
`
`
`sity research. In fact, virtually all of
`
`
`
`
`today’s major interface styles
`
`
`
`
`and applications have been
`
`
`
`significantly influenced by
`research at universities
`
`
`
`
`
`
`
`and labs, often with gov-
`
`
`
`ernment
`funding. To
`
`
`
`illustrate this point,
`this article has listed
`
`
`
`
`
`
`
`the funding sources
`
`
`
`
`
`of some of the major
`advances.
`Without
`
`
`
`
`
`this research, many
`of the advances in
`
`
`
`
`the
`field of HG]
`
`
`
`
`
`
`
`would probably not
`
`
`
`
`have taken place, and
`
`
`
`
`35 a consequence,
`[ll-E
`user interfaces of com-
`
`
`
`
`
`
`mercial products would
`be far more difficult
`to
`
`
`
`
`
`
`
`
`
`
`
`use and learn than they are
`
`
`
`
`today. As described by Stu
`
`
`
`
`Card [6, p. 162]:
`
`
`
`of
`Gorremmem funding
`
`
`imtemev
`advanced fmman—mmpm‘er
`
`
`
`
`
`riot: technologies (mil: dye fnrefieaual capi-
`
`
`
`
`
`
`
`
`mi’ and trained the research teams fin’ pioneer
`
`
`
`
`
`
`
`
`syrrerm rfmr, ever a period r;f25 years, revolu-
`
`
`
`
`
`
`tionized bow pe.~:_Jp/e’
`interact with computers.
`
`
`
`
`
`
`Indummf research laboratories at the rerporme
`
`
`
`
`
`
`
`
`level in Xerox, IBM ATC§"iC and ether: played it
`
`
`
`
`
`
`
`m'ang mil:
`in devefoping dais tedmofogy and
`
`
`
`
`
`
`
`
`bringing it info :1 fiirm ruimblefiar the commer-
`cial’ creme.
`
`
`
`
`
`
`
`Another important argument in favor
`of HCI research at universities is that
`
`
`
`
`
`
`
`
`
`
`
`
`
`- computer science students need to
`know about user interface issues.
`
`
`
`
`
`
`
`
`
`
`
`User interfaces are likely to be
`one of the main value-added
`
`
`
`
`
`
`
`
`
`competitive advantages of the
`future as both hardware and basic
`
`
`
`
`
`
`
`software become commodities. If
`
`
`
`
`students do not know about user
`
`
`
`
`
`
`
`
`
`interfaces,
`they will not
`serve
`
`
`Acknowledgments
`
`
`
`
`
`
`
`
`
`I must thank a large number of people who
`
`
`
`
`
`
`
`
`responded to posts of earlier versions of this
`
`
`
`
`
`
`article on the announcements.chi mailing list
`
`
`
`
`
`
`
`
`
`for their very generous help, and to Jim Hollan
`
`
`
`
`
`
`
`
`who helped write the short excerpt of this arti-
`cle. Some of dle information in this article was
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`supplied by (in alphabetical order): Stacey Ash-
`
`
`
`
`
`
`
`lund, Meera M. Blattner, Keith Butler, Stuart
`
`
`
`
`
`
`
`
`K. Card, Bill Curtis, David E. Damouth, Dan
`
`
`
`
`
`
`
`ir1teracrions.. .rnarch + april
`
`
`
`
`
`
`
`1993
`
`
`
`
`Page 9 of 11
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`PERMISSION TO MAKE D.lCFI'ALIr HARD
`
`
`
`
`COPY OF PART OR A].L BE THE WORK
`
`
`
`
`
`
`
`
`FOR PERSONAL on ct.-\ss.nooM us: rs
`
`
`
`
`
`
`cmurrto w1THo1.rr FEE mom-1or_o
`
`
`
`‘mar cones AR: No? mm on
`
`
`
`
`
`
`D|S]'|'Ll'BUTED FOR PROET DR CXJM
`
`
`
`
`
`MEIICIJLL ADVANTAGE, THE CXZPPYRIGHT
`
`
`
`NOTICE. THE ITFLE OF THE Pl."-EEJCA
`
`
`
`
`
`nor.’ .-um rrs oars )\.P?EAJl. nun
`
`
`
`
`
`
`NOTICE Is cnnrn ‘n-urr comm: rs 3':
`
`
`
`
`
`
`
`‘PERMISSION or ACM, INC. To cow
`
`
`
`
`
`DTHERWWSE. TD |1EI"U'BLI5l-(. TO 'PCnT
`
`
`
`
`
`ON SERVERS, DR TD RED']5I'R!BUTE TD
`
`
`
`
`
`
`LISTS ‘REQUIRE PRIOR SPE('_'l.FJC PERMIS
`
`
`
`
`
`SEDN motor .1 FE.
`
`
`
`
`Q ACM I072 522039330300 $3.50
`
`
`
`
`
`
`
`
`
`
`
`
`"IN EMBEDDED
`
`
`
`
`
`MENUS. HI.G"HLlG"H1'"ED
`
`
`
`on--uN'oE'fiI‘.INE'D
`wonins--.on -PI-fA5_ES
`
`
`
`
`
`
`WITHIN THE TEXT
`BE'C'iOlUI'E' “i"H'E' M'E'N'.U
`
`
`
`
`
`ITE}Iv'I'§,. AND ARE-
`5EiL'EcT_ABI.=e-us'|_'N‘c -‘rat
`
`
`
`
`
`'C.'0M'MONl-Y -USE-.9
`
`
`
`
`
`‘TOUCH SCREEN."
`
`
`
`cunstin AND1 Mouse
`
`
`
`METI.-_l.OD:S-._"
`
`
`
`-1;-, K_0,v_Efi-Aflfi
`
`
`
`
`
`
`
`.
`.
`.'.IiI':.. §H'NEtti£Ei§t?rN on: rat
`HYPERIIES15 Y5 TEN! [19, P. 312}-.
`
`
`
`
`
`
`
`

`
`article —
`
`Rob
`
`Rich McDaniel,
`
`Miller, Bernita Myers,
`Yoshihiro Tsujino, and
`the reviewers.
`A short
`
`
`
`Diaper, Dick Duda, Tim T.K.
`Dudley, Steven Feiner, Harry
`Forsdick, Bjorn Freeman-
`Benson, John Gould,
`Wayne Gray, Mark
`Green, Fred Hansen,
`
`D.
`Hefley,
`Bill
`Austin Henderson,
`
`Jean-
`Jim Hollan,
`Marie Hullot, Rob Jacob,
`
`Sandy
`John,
`Bonnie
`Kobayashi, T.K. Iandauet, John
`Ieggett, Roger Lighty, Marilyn Mantei, Jim
`Miller, \Villiam Newman, Jakob Nielsen, Don
`
`Norman, Dan Olsen, Ramesh Patil, Gary Perl-
`man, Dick Pew, Ken Pier, Jim Rhyne, Ben
`Shneiderman, John Sibert, David C. Smith,
`Elliot Soloway, Richard Stallman, Ivan Suther-
`land, Dan Swinehart, John Thomas, Alex
`Waibd, Marceli Wein, Mark Weiser, Alan
`
`Wexelblat, and Terry Winogmd. Editorial
`comments were also provided by the forego-
`ing as well as by Jim Hollan, Ellen Borison,
`
`from this
`
`excerpt
`article
`
`appeared as part of
`“Strategic Direc-
`tions in Human
`
`Inter-
`Computer
`action,” B. Myers, J. Hollan,
`I. Cruz (eds.). ACM Computing Sur-
`veys 28, 4 (Dec. 1996). pp. 794-809.
`This research was sponsored partly by
`NCCOSC under Contract No. N6600l—94-
`
`C-6037, ARPA Order No. B326, and partly
`by the National Science Foundation under
`Grant No. IRI-9319969. The views and con-
`clusions contained in this document are those
`
`of the author and should not be interpreted as
`representing the official policies, either
`expressed or implied, of NCCOSC or the
`U.S. Government.
`
`References
`1. Baecker, R., et al. “A Historical and Intel-
`
`lectual Perspective.” In Readings in
`Human—Computer Intemetion: Toward the Year
`2000. (R. Baeclter et al.,
`2nd ed. Mor-
`
`gan Kaufinann Publishers, Inc., San Francisco,
`1995, pp. 35-47.
`2. Brooks, F. “The computer ‘scientist’ as tool-
`smith—Studies in interactive computer graph-
`
`ics." In Proceeding: ofIFIP Confirmte, 1977.
`pp. 625-634.
`3. Burtnyk, N. and Wein, M. “Computer
`
`Morgan Kauf-
`tart (M. Rudisill et al.,
`mann Publishers, San Francisco, 1996, pp.
`I22-I69.
`
`7. Coons, S. “An Outline of the Requirements
`for a Computer-Aided Design System.” In
`
`Proceedings of/IFIPS Springfaint Computer
`Confirenee 23 (1963), pp. 299-304.
`8. Engdbart, D. “The Augmented Knowledge
`
`Workshop.” In A History qfPer:onaI W1:rl'.rta-
`tions (A. Goldberg, ed.). Addison-\WesIey Pub-
`lishing Company, New York, 1988, pp.
`187-232.
`
`generated key frame animation.” fournal ofthe
`Society ofMotion Picture and Yelemirion Engi-
`neers 3. 3 (1971). pp. 149-153.
`4. Bush, V. “As we may think.” The Arlantir
`
`9. Engelbart, D. and English, W. “A Research
`Center for Augmenting Human Intellect.”
`Reprinted in ACM SIGGRAPH Video Review
`1060994), video made in 1968
`
`Mandi!) 176 (July 1945), pp. 101-108.
`Reprinte