THE DESIGN, SIMULATION, AND EVALUATION OF A MENU DRIVEN USER INTERFACE
`
`Ricky E. Savage, James K. Habinek, and Thomas W. Barnhart
`
`IBM System Products Division
`Rochester, Minnesota 5590]
`
`INTRODUCTION
`
`The design and evaluation of a menu-driven
`interface
`for
`a general purpose
`system
`user
`described.
`Analysis
`of
`errors made
`by
`are
`participants
`in
`simulation.
`studies
`of
`the
`interface led to the development of hypotheses
`concerning user choice behavior.
`For example,
`novice
`users
`had difficulty selecting menu
`options based on job titles rather than tasks and
`functions. Redesign of the interface to reflect
`these
`hypotheses
`resulted
`in
`significantly
`improved performance.
`The current version of the
`interface appears to accommodate both the novice
`user,
`through an extensive hierarchy 13f menus,
`and the experienced user,
`through a variety of
`shortcuts to system functions.
`
`Many practical alternatives for user-computer
`communication exist
`(Martin [5] Ramsey & Atwood
`[7], Shneiderman [8]). All have advantages and
`drawbacks,
`usually
`as
`a
`function
`of
`the
`characteristics of the typical operator. At one
`extreme,
`programming
`languages with
`precise
`syntax and vocabularies may be used for dialog
`between person and
`computer.
`These
`languages
`have the advantage that complex concepts can be
`communicated unambiguously by the operator.
`The
`associated drawback,
`is
`that
`the use of
`the
`language requires extensive training and strict
`adherence to rules.
`
`At the other extreme, a hierarchy of detailed
`menus
`may
`be
`used
`for
`person-computer
`interaction.
`Since this technique relies heavily
`on
`the user's
`recognition memory
`and passive
`response
`to
`computer
`prompts,
`little formal
`training is
`required.
`As might be
`expected,
`menus have
`their drawbacks,
`too.
`For example,
`the creator of the menu must have a perception of
`all the possible or desirable options to include.
`Furthermore, use of the menus may become tedious
`if the choices are too finely detailed or if the
`user has extensive experience.
`
`the
`the number of system users increases,
`As
`degree of
`formal
`training of
`the typical user
`@1981 ASSOCIATION FOR COMPUTING MACHINERY
`Permission to copy without fee all or part of this material is granted provided that
`the copies are not made or distributed for direct commerical advantage. the ACM
`copyright notice and the title of the publication and its date appear. and notice is
`given that copying is by permission of the Association for Computing Machinery.
`To copy otherwise. or to republish. requires a fee andlor specific permission.
`
`selection,
`such as menu
`Techniques
`declines.
`which
`can best
`accommodate
`the novice user,
`almost necessarily must be included in a strategy
`for person-computer communication. Yet care must
`be
`taken that
`the experienced or sophisticated
`user
`is not encumbered with an interface that
`involves frustratingly slow entry of commands or
`procedures.
`This paper details the process and
`techniques
`required to
`develop
`and
`test
`an
`interface that would satisfy the needs of a broad
`spectrum of users.
`Two design and evaluation
`iterations are described.
`
`DESIGN OF THE INTERFACE (PHASE I)
`
`Initial Human Factors Considerations
`
`The primary consideration for designing this
`interface was
`to provide an easy access to
`user
`the entire system without constant reference to
`manuals.
`The design was
`targeted to assist the
`novice user, but at the same time not to penalize
`the
`experienced user.
`This
`second point
`is
`important,
`and to accomplish this goal, systems
`may have to be designed with multiple levels of
`user interface. General purpose systems normally
`have all levels of users. For experienced users,
`there would be
`a highly abbreviated and quick
`access to system functions.
`The novice user, on
`the other hand, would
`need
`a very specific
`step-by-step
`interface
`to
`lead
`him to
`the
`required system function.
`Both Shneiderman [8]
`and Martin [5]
`have
`stated that novice users
`normally require a menu driven interface.
`
`A consideration in designing the interface
`was the structure of the menus.
`The hierarchical
`or
`tree structure was used (see Shneiderman [8]
`Chapter 7
`for
`a
`brief
`discussion
`on
`data
`structure modes)
`because of
`the
`experimental
`evidence favoring such a structure (Brosey and
`Shneiderman
`[1]),
`and because of
`the natural
`hierarchical
`structure
`of
`this
`interface
`(Durding, Becker, and Gould [3]).
`
`initial consideration was consistent
`Another
`screen design.
`Based on the work of Engel and
`Granda
`[4]
`and Peterson [6],
`screen standards
`were developed to ensure consistency.
`It was
`also determined that menus should have no more
`than nine options, and that the paths or levels
`of menus
`to a
`function should be
`relatively
`short, generally three or four levels.
`
`36
`
`1
`1
`
`ELASTIC - EXHIBIT 1013
`
`ELASTIC - EXHIBIT 1013
`
`

`

`the less novice user, several shortcuts
`For
`to commands and procedures exist.
`For example,
`any
`given menu may
`be
`displayed simply by
`entering its name.
`In addition,
`the user may
`directly enter a known command or procedure name
`to obtain a prompt
`screen for that
`command or
`procedure.
`Finally,
`the command or procedure
`name
`may
`be
`entered with
`its
`appropriate
`positional parameters to most directly accomplish
`a desired function.
`
`Implementation of the Simulation
`
`Very little supporting software was required
`since most of
`the support was already available
`on the IBM System/34.
`The Source Entry Utility
`(SEU)
`and Screen Design Aid
`(SDA) were used
`extensively to create the menus
`and prompts.
`System/34 had some restrictions which prohibited
`running
`some
`commands
`from a menu.
`These
`restrictions were removed and code was added to
`process the command keys and chaining of menus.
`These operating system changes were primarily
`modifications
`to existing System/34 Assembler
`routines.
`
`EVALUATION OF THE USER INTERFACE (PHASE I)
`Tasks
`
`field
`from individuals with
`help
`With
`experience,
`tasks were developed for programmers,
`system operators,
`and work station operators.
`Careful consideration was given to importance,
`frequency of use,
`and difficulty of
`the many
`possible functions before they were selected for
`use.
`Some
`examples
`of
`tasks were building
`libraries, copying files from diskette, changing
`printer IDs, printing information, and modifying
`source.
`Each participant had at least ten tasks
`to perform.
`
`Participants
`
`Twelve participants performed the programmer
`tasks.
`Nine participants performed the system
`operator tasks and ten participants performed the
`work station operator tasks. These participants'
`experience ranged from total novice to operators
`with
`less
`than
`six months
`experience.
`All
`participants were
`employees of
`IBM Rochester.
`
`Equipment
`
`interface was
`the user
`simulation of
`The
`developed to run on the System/34 which,
`in turn,
`became
`the vehicle for running the study.
`The
`system captured response and time on each menu
`for the protocol analysis. Video recorders and
`cameras were used as
`a backup and to record
`comments
`by
`the participants.
`Finally,
`an
`attitude questionnaire was administered.
`
`Procedure
`
`system
`the
`operated
`participant
`Each
`session.
`a
`two-hour
`to
`individually for
`up
`and
`the
`were
`given
`General
`instructions
`participants were allowed to ask questions before
`
`assistance.
`experimenter
`continuing without
`Participants read the task instructions and used
`the menus
`to accomplish their
`tasks. Manuals
`were
`available
`through
`the
`session.
`The
`participants chose options from a series of menus
`to lead them to a command, procedure, or utility.
`Parameters were then entered on a prompt screen
`from which
`the
`function would
`execute
`and
`accomplish the task.
`If participants could not
`accomplish a particular task,
`they were allowed
`to continue with the next task.
`
`Data Analysis
`
`interface posed
`the user
`analysis of
`The
`special problems because the evaluation was not
`an experimental comparison of alternatives.
`The
`analysis
`procedures
`had
`to
`be
`sensitive
`in
`finding problems,
`such as
`'the wording of
`the
`menus,
`the hierarchical structure of the menus,
`and organization of
`a specific menu path or a
`specific menu, and to be sensitive in describing
`user behavior, particularly user errors.
`Time was recorded for each menu and each task.
`The primary benefit of measuring time was
`to
`serve as
`a baseline for future evaluation (as a
`result of modifications to the interface) and to
`determine if an unusually large amount of
`time
`was spent on a particular menu or task.
`The protocol analysis consisted of mapping
`subject's
`responses
`on
`paper
`for
`a
`each
`comparison with the optimal path, which was
`defined as
`the shortest
`route to the desired
`function.
`The
`error
`analysis
`consisted
`of
`categorizing user errors (defined as an incorrect
`menu option chosen or
`an incorrect parameter
`specified)
`into types of errors.
`A probability
`analysis was performed by regarding each menu as
`a decision point, and determining the probability
`of a correct decision.
`The probability analysis
`pointed
`to
`specific menus which were
`not
`communicating adequately.
`
`Results and Discussion
`
`general
`four
`found
`analysis
`error
`The
`The first category was
`categories of errors.
`called an "inconvenience error" and resulted from
`three different actions:
`1)
`the user taking the
`wrong path but ending up at the correct function;
`2)
`the user searching or exploring various menu
`options
`and paths
`and
`eventually taking the
`correct path to the correction function;
`and
`3)
`the user searching or exploring and eventually
`taking the wrong path but
`ending up at
`the
`correct
`function.
`These
`inconvenience
`errors
`were not serious errors, but the user took a less
`than optimal path to the correct function.
`
`"path
`called a
`category was
`second
`The
`error." Three different actions could cause this
`error:
`1)
`the user taking the wrong path to the
`wrong
`function;
`2) the
`user
`searching
`or
`exploring and taking the wrong path to the wrong
`function; and 3) the user taking the correct path
`but
`on
`the last menu selecting the incorrect
`option which led to the wrong function.
`The path
`
`37
`
`2
`
`

`

`errors were serious because the user ended up
`using the incorrect command or procedure.
`
`A "function error" was the third category and
`resulted when
`the user
`filled in the wrong
`parameters on the prompt
`screen when.
`they" had
`successfully reached the correct function. This
`error did not
`indicate problems with the menus,
`but did point out problems the user had with the
`prompts.
`
`the percentage of errors for
`shows
`Table 1
`each category and for each type of
`task.
`The
`inconvenience errors were relatively consistent
`for all three types of tasks and they seemed to
`be due to unclear wording of the menu options and
`some misinterpretation of how the task should be
`achomplished.
`The path errors were rather high
`for system and work station operator tasks.
`Some
`of
`these
`errors will
`be
`explained
`in the
`discussion of
`the probability analysis.
`The
`function errors
`for
`system and work
`station
`operator
`tasks
`seemed to be due
`to £1
`lack of
`experience with the command and procedure prompts
`and
`a
`lack
`of
`help
`text
`explaining
`the
`parameters.
`Finally,
`searching and
`exploring
`various menu options by the users accounted for
`most of
`the inconvenience and path errors.
`The
`absence
`of detailed help-text
`for
`each menu
`option
`in
`the
`simulation
`may
`have
`been
`responsible
`for much of
`the user
`confusion.
`
`Table 1--The Percentage of Errors for Each
`Category
`
`Inconv
`errors
`
`Path
`errors
`
`Function
`errors
`
`The results of the questionnaire administered
`to the participants showed that
`the wording of
`the menus was
`the primary problem. Generally,
`the participants felt that the interface was easy
`to use, helped them to learn about
`the system,
`and would aid in their productivity.
`
`from this
`results
`primary
`the
`of
`One
`programmer
`evaluation was the high success rate:
`tasks, 92%; system operator tasks, 79%; and work
`station operator
`tasks,
`81%.
`Another major
`result was
`that the participants did not use the
`manuals.
`These
`results suggested that
`a menu
`driven user
`interface offers a viable method of
`assisting users
`in their work.
`On
`the other
`hand,
`the fact that failures did occur, and that
`frequently many
`false starts and backtracking
`were
`required before acceptable solutions were
`found,
`suggested that
`the interface needed some
`redesign.
`
`REDESIGN AND REEVALUATION (PHASE II)
`
`Many of the changes to the interface involved
`breaking up and rewording complex and cluttered
`menus
`to make them simpler in appearance and to
`reduce the information. overload.
`As discussed
`earlier,
`two menu paths
`had to be
`redesigned
`which affected some of the structure of the menu
`hierarchy.
`These paths required users to choose
`options which corresponded to their job:
`"work
`station tasky' or
`"system console tasks."
`The
`results from Phase I clearly showed that users
`were not
`inclined to use
`the menus
`in this
`fashion.
`They looked for task oriented options
`rather
`than job classification options.
`The
`redesign eliminated the two job classification
`options and developed four task oriented options.
`The purpose of
`the Phase II evaluation was
`to
`test the success of these changes.
`
`Programming
`tasks
`
`61.1
`
`System operator
`tasks
`
`33.1
`
`Work station
`operator tasks 25.8
`
`29.6
`
`9.3
`
`Method
`
`36.5
`
`30.3
`
`61.6
`
`12.6
`
`second
`Twenty people participated in the
`phase of
`testing.
`Six performed the programmer
`tasks,
`and seven performed the system console
`operator
`and
`the work station operator
`tasks.
`The
`tasks,
`procedures,
`and
`equipment were
`identical to Phase I.
`'
`
`in
`successful
`The probability analysis was
`pointing to menus with a
`low probability of
`correct option selection.
`In general,
`these
`problem menus were of
`two types.
`In the first
`type, one or possibly two incorrect alternatives
`had a high probability of selection relative to
`the correct alternative. These errors appeared to
`result
`from a discrimination problem.
`In the
`other type, participants used a shotgun approach:
`a
`large variety of
`incorrect alternatives was
`selected by
`the participants
`in lieu of
`the
`correct option.
`In this case,
`the users seemed
`to have little notion as to which alternative was
`correct.
`The path errors for the system and work
`station operator
`tasks
`illustrate this.
`These
`two
`groups
`of participants were
`constantly
`confusing each other's options
`from the first
`menu. At
`the next
`level of menus,
`the shotgun
`approach resulted.
`
`Results and Discussions
`
`The second phase of testing the interface was
`completed with significant
`improvements
`in user
`performance.
`Table 2 shows the time and success
`rate
`for
`the
`three
`groups
`of participants,
`comparing the result of
`the first
`and second
`phases of
`testing.
`As can be seen, significant
`improvements were obtained for both types of
`operators
`in terms
`of
`the
`average
`time
`to
`a
`complete
`task and
`the
`rate of
`successful
`completion of the tasks.
`
`are
`analysis
`error
`the
`of
`results
`The
`presented in Table 3. Clearly the path errors
`are the most serious, and they were significantly
`reduced during the second phase of
`testing for
`both
`types
`of operators.
`These
`two
`tables
`demonstrate the success of the changes that were
`
`38
`
`3
`
`

`

`the
`Finally,
`operators.
`the
`help
`to
`made
`questionnaire
`produced
`similar
`attitudinal
`results to Phase I, with the wording of the menus
`again being the major problem.
`
`Table 2--Average time to complete each task and
`the success rate of both phases of testing
`
`Task Type
`
`Test Phase
`
`Time (Mins) Success
`
`Programmer
`
`System
`console
`
`Work
`station
`
`1
`
`2
`
`1
`
`2
`
`1
`
`2
`
`11.53
`
`12.85
`
`5.41
`
`1.69
`
`6.30
`
`2.92
`
`92%
`
`100%
`
`79%
`
`95%
`
`81%
`
`95%
`
`Table 3--Average errors per participant
`by error category for both phases
`of testing
`
`Test
`Task Type Phase
`
`Error Category
`Inconvenience Path Function
`
`Programmer
`
`System
`console
`
`Work
`station
`
`1
`
`2
`
`I
`
`2
`
`1
`
`2
`
`6.00
`
`7.30
`
`4.80
`
`3.71
`
`4.33
`
`5.71
`
`1.60
`
`2.20
`
`2.00
`
`.83
`
`5.30
`
`4.40
`
`.71
`
`.29
`
`10.33
`
`2.11
`
`.43
`
`1.14
`
`several
`found
`analysis
`probability
`The
`problems, but
`they were minor compared to those
`in
`the
`first
`test
`phase.
`These
`problems
`consisted of some menu options being misleading,
`missing
`functions,
`wording
`problems,
`and
`discrimination problems. Generally, all of these
`problems had obvious
`solutions and were easily
`correctable.
`
`CONCLUSION
`
`these
`in
`found
`problems
`the
`of
`Many
`terminology.
`to ambiguous
`evaluations were due
`Participants did not know the difference between
`work station, display station,
`and device, nor
`
`did they understand the difference between saving
`and copying a file, or
`removing and deleting a
`file.
`This problem needs to be solved not only
`with simpler terminology, but also with help text
`for each menu to explain in more detail what each
`particular menu option means.
`
`evaluations
`these
`from
`results
`Two
`a
`better
`to
`significantly
`contributed
`of
`user
`behavior with menus.
`understanding
`First,
`it was evident
`from these studies that a
`user's job title was not important, whereas tasks
`and
`functions were
`important when developing
`menus.
`This was demonstrated by the success of
`the
`changes made
`from Phase I.
`Second,
`it
`appears
`that users
`in these studies preferred
`shorter menus’with more levels than the opposite
`case. Many of the changes from Phase I consisted
`of breaking up a
`complex menu into two menus.
`The
`second study showed
`the success of
`these
`changes.
`This
`result
`is in apparent conflict
`with a recent study by Dray, Ogden, and Vestewig
`[2].
`These differences may be attributed to the
`realism of this interface (in that it dealt with
`an actual
`system)
`and to the fact
`that menu
`options were word phrases as opposed to one or
`two words in the Dray, et a1 [2] study.
`
`this menu driven user
`The basic concept of
`interface is clearly sound.
`Comments
`received
`from participants were generally favorable and
`performance
`reasonably successful.
`Any
`future
`changes made
`to the
`interface will
`be more
`"fine-tuning" than "major overhaul."
`
`ACKNOWLEDGEMENTS
`
`Many people were involved in the design and
`evaluation of
`this interface and in reviewing
`this paper.
`Those who contributed significantly
`were:
`Nancy Blackstad,
`Steve Dahl, Karen
`Eikenhorst,
`Dave
`Peterson,
`and Mike Temple.
`
`REFERENCES
`
`Two
`1 Brosey, M. K. and Shneiderman, B.
`relational
`experimental
`comparisons
`of
`and hierarchical data base models.
`International Journal of Man-Machine
`Studies, 1978, 10, 625-637.
`
`2 Dray, S. M., Ogden, W. G., and
`Vestewig, R. E. Measuring performance
`with a menu-selection human-computer
`interface. Proceedings of the 25th
`Annual Meeting of the Human Factors
`Society, 1981.
`
`3 Durding, B. M., Becker, C. A., and
`Gould, J.D. Data organization.
`Human Factors, 1977, 19, 1, 1-14.
`
`4 Engel, S. E. and Granda, R. E. Guidelines
`for man/display interfaces.
`IBM
`Poughkeepsie Laboratory Technidal
`Report TR00.2720, December 19, 1975.
`
`39
`
`4
`
`

`

`5 Martin, J. Design of Man-Computer Dialogues.
`Englewood Cliffs, NJ: Prentice Hall, 1973.
`
`6
`
`Screen design guidelines.
`Peterson, D. E.
`
`Small Systems World, February 1979, 19-21,
`34-37.
`
`7 Ramsey, H. R. and Atwood, M. E. Human factors
`in computer
`systems:
`a
`review of
`the
`literature. Englewood, CO: Science
`Application,
`Inc., Technical Report SAI-
`79-lll-DEN, September, 1979.
`
`Software Psychologx.
`8 Shneiderman, B.
`Cambridge, MA: Winthrop Publishers, Inc.,
`1980.
`
`40
`
`5
`
`