THE DESIGN AND EVALUATION OF
`MARKING MENUS
`
`
`
`
`
`
`
`
`
`
`
`
`
`by
`
`Gordon Paul Kurtenbach
`
`
`
`
`
`
`
`A thesis submitted in conformity with the requirements
`of the Degree of Doctor of Philosophy
`Graduate Department of Computer Science
`University of Toronto
`
`
`
`
`
`Valve Exhibit 1070
`Valve v. Immersion
`
`

`

`Copyright © 1993 Gordon Paul Kurtenbach
`
`

`

`

`

`

`

`Abstract
`
`This research focuses on the use of hand drawn marks as a human-computer input
`technique. Drawing a mark is an efficient command input technique in many
`situations. However, marks are not intrinsically self-explanatory as are other
`interactive techniques such as buttons and menus. This research develops and
`evaluates an interaction technique called marking menus which integrates menus
`and marks such that both self-explanation and efficient interaction can be provided.
`
`A marking menu allows a user to perform a menu selection by either popping up a
`radial menu and then selecting an item, or by drawing a straight mark in the
`direction of the desired menu item. Drawing a mark avoids popping up the menu.
`Marking menus can also be hierarchic. In this case, hierarchic radial menus and
`“zig-zag” marks are used. Marking menus are based on three design principles:
`self-revelation, guidance and rehearsal. Self-revelation means a marking menu
`reveals to a user what functions or items are available. Guidance means a marking
`menu guides a user in selecting an item. Rehearsal means that the guidance
`provided by the marking menu is a rehearsal of making the mark needed to select
`an item. Self-revelation helps a novice determine what functions are available, while
`guidance and rehearsal train a novice to use the marks like an expert. The intention
`is to allow a user to make a smooth and efficient transition from novice to expert
`behavior.
`
`This research evaluates marking menus through empirical experiments, a case
`study, and a design study. Results shows that (1) 4, 8 and 12 item menus are
`advantageous when selecting using marks, (2) marks can be used to reliably select
`from four-item menus that are up to four levels deep or from eight-item menus that
`are up to two levels deep, (3) marks can be performed more accurately with a pen
`than a mouse, but the difference is not large, (4) in a practical application, users
`tended towards using the marks 100% of the time, (5) using a mark, in this
`application, was 3.5 times faster than selection using the menu, (6) the design
`principles of marking menus can be generalized to other types of marks.
`
`

`

`

`

`
`
`THE DESIGN AND EVALUATION OF MARKING MENUS
`Gordon Paul Kurtenbach
`Degree of Doctor of Philosophy
`Graduate Department of Computer Science, University of Toronto, 1993
`
`abstract
`This research focuses on the use of hand drawn marks as a human-computer input
`technique. Drawing a mark is an efficient command input technique in many
`situations. However, marks are not intrinsically self-explanatory as are other
`interactive techniques such as buttons and menus. This research develops and
`evaluates an interaction technique called marking menus which integrates menus
`and marks such that both self-explanation and efficient interaction can be provided.
`
`A marking menu allows a user to perform a menu selection by either popping up a
`radial menu and then selecting an item, or by drawing a straight mark in the
`direction of the desired menu item. Drawing a mark avoids popping up the menu.
`Marking menus can also be hierarchic. In this case, hierarchic radial menus and
`“zig-zag” marks are used. Marking menus are based on three design principles:
`self-revelation, guidance and rehearsal. Self-revelation means a marking menu
`reveals to a user what functions or items are available. Guidance means a marking
`menu guides a user in selecting an item. Rehearsal means that the guidance
`provided by the marking menu is a rehearsal of making the mark needed to select
`an item. Self-revelation helps a novice determine what functions are available, while
`guidance and rehearsal train a novice to use the marks like an expert. The intention
`is to allow a user to make a smooth and efficient transition from novice to expert
`behavior.
`
`This research evaluates marking menus through empirical experiments, a case
`study, and a design study. Results shows that (1) 4, 8 and 12 item menus are
`advantageous when selecting using marks, (2) marks can be used to reliably select
`from four-item menus that are up to four levels deep or from eight-item menus that
`are up to two levels deep, (3) marks can be performed more accurately with a pen
`than a mouse, but the difference is not large, (4) in a practical application, users
`tended towards using the marks 100% of the time, (5) using a mark, in this
`application, was 3.5 times faster than selection using the menu, (6) the design
`principles of marking menus can be generalized to other types of marks.
`
`

`

`
`
`

`

`
`
`Acknowledgments
`
`Many years ago when I was in high school my classmates and I spent three days
`writing occupation aptitude tests. Months later the computer graded tests were
`returned to us. I remember my friends’ and my own excitement as we ripped open
`the envelopes to see what the computer had recommended. My friends cheered as
`they read out their long list of possibilities: doctor! lawyer! pilot! writer! scientist!
`With great anticipation I opened my computer recommendation. There, before my
`eyes, was one lonely recommendation: pre-cast concrete worker.
`
`Although I have failed to fulfill my destiny as pre-cast concrete worker, I have
`created this thesis with the support of many people. In particular, I would like to
`thank:
`
`• My supervisor and friend, Bill Buxton. Bill’s creativity, intellect, and humor
`inspired me to pursue research and make bad jokes.
`
`• The members of my committee: Ron Baecker, Mark Chignell, Marilyn Mantei,
`Ken Sevcik, and Cathy Wolf. Each contributed in helping me polish my research
`into a doctoral thesis.
`
`• Great researchers and friends. Abigail Sellen greatly helped by designing
`experiments, writing, and putting on excellent parties; Tom Moran, Stuart Card,
`and Ken Pier provided creative insights and guidance; George Fitzmaurice and
`Beverly Harrison waded through treacherous drafts of my thesis, helped me make it
`a better document, and listened to my concerns over many a cappuccino; Gary
`Hardock utilized my work in his research and put up with my kidding; George
`Drettakis and Dimitri Nastos kept the lab systems running, humored me, and
`organized the most delicious Greek barbecues; Tim Brecht advised me, made me
`laugh way too loud and long, yet still managed to keep me sane.
`
`

`

` I
`
` don’t think I’ll thank the computer that graded the aptitude tests...
`
`

`

`
`
`
`
`

`

`

`

`
`
`
`
`
`
`
`
`
`To my parents, Helen and Leo,
`
`and my brother and sisters,
`
`Robert, Beverly, Donna, Carole, and Tammy:
`
`“My thesis is done, you can probably reach me at home now”
`
`

`

`
`
`

`

`
`
`Table of Contents
`
`Chapter 1: Introduction....................................................................................................1
`1.1. General area and definitions ......................................................................3
`1.2. Why use marks? ...........................................................................................4
`1.2.1. Symbolic nature ...........................................................................5
`1.2.2.
`Intrinsic advantages ....................................................................7
`1.3. Self-revelation, guidance and rehearsal....................................................7
`1.3.1. The problem: learning and using marks ..................................8
`Self-revelation.......................................................................10
`Guidance ...............................................................................12
`Rehearsal ...............................................................................12
`1.3.2. Unfolding interfaces....................................................................13
`1.3.3. Solution: ways of learning and using marks ...........................14
`Off-line documentation.......................................................14
`On-line documentation .......................................................15
`On-line interactive methods ...............................................16
`On-line interactive rehearsal methods..............................18
`1.4. Thesis statement...........................................................................................20
`1.5 Summary .......................................................................................................21
`
`Chapter 2: Marking menus..............................................................................................23
`2.1. Definition.......................................................................................................23
`2.2. Motivation for study....................................................................................26
`2.2.1. Advantages over traditional menus .........................................26
`Keyboardless acceleration ..................................................26
`Acceleration on all items.....................................................27
`Menu selection mimics acceleration..................................27
`Combining pointing and selecting ....................................27
`Spatial mnemonics...............................................................28
`2.2.2. Ease of drawing and recognition...............................................28
`2.2.3. Marks when no obvious marks exists ......................................29
`2.2.4. Compatibility with unfolding interfaces..................................29
`2.2.5. Compatibility with existing interfaces......................................29
`2.2.6. Novices, experts, and rehearsal.................................................30
`i
`
`
`
`

`

`
`
`2.2.7. Utilizing motor skills...................................................................31
`2.2.8. “Eyes-free” selection ...................................................................31
`2.3. Related work and open problems .............................................................31
`2.3.1. Pie menus......................................................................................32
`2.3.2. Command compass.....................................................................34
`2.4. Research Issues.............................................................................................35
`2.4.1. Articulation...................................................................................35
`2.4.2. Memory.........................................................................................36
`2.4.3. Hierarchic structuring.................................................................38
`2.4.4. Command parameters and design rationale ...........................41
`2.4.5. Generalizing self-revelation, guidance and rehearsal............42
`2.5. Design rationale ...........................................................................................42
`2.5.1. Fundamental design goals .........................................................42
`2.5.2. The design space..........................................................................43
`2.5.3. Discrimination method...............................................................44
`2.5.4. Control methods ..........................................................................46
`2.5.5. Selection events: preview, confirm and terminate..................47
`2.5.6. Mark ambiguities.........................................................................50
`2.5.7. Display methods ..........................................................................54
`2.5.8. Backing-up the hierarchy ...........................................................54
`2.5.9. Aborting selection........................................................................56
`2.5.10. Graphic designs and layout .......................................................57
`2.5.11. Summary of design......................................................................58
`2.6. Summary .......................................................................................................59
`
`Chapter 3: An empirical evaluation of non-hierarchic marking menus ...................61
`3.1. The experiment.............................................................................................62
`3.1.1. Design............................................................................................62
`3.1.2. Hypotheses ...................................................................................63
`3.1.3. Method ..........................................................................................64
`3.2. Results and discussion ................................................................................68
`3.2.1. Effects due to number of items per menu................................68
`3.2.2. Device effects................................................................................70
`3.2.3. Mark analysis ...............................................................................72
`3.2.4. Learning effects............................................................................74
`3.3. Conclusions...................................................................................................75
`3.4. Summary .......................................................................................................79
`
`Chapter 4: A case study of marking menus ..................................................................81
`4.1. Description of the test application.............................................................81
`4.2. How marking menus were used................................................................83
`4.2.1. The design.....................................................................................83
`4.2.2. Discussion of design....................................................................86
`Menu item choice.................................................................86
`ii
`
`
`
`

`

`
`
`Spatial aspects.......................................................................86
`Temporal aspects .................................................................87
`Inverting semantics of menu items ...................................88
`The role of command feedback..........................................88
`4.3. Analysis of use..............................................................................................89
`4.3.1.
`Issues of use and hypotheses .....................................................90
`4.3.2. Results ...........................................................................................91
`Menu versus mark usage....................................................91
`Mark confirmation and reselection ...................................94
`Reselection ............................................................................96
`Selection time and length of mark.....................................96
`Users' perceptions ................................................................98
`Marking menus versus linear menus................................99
`4.4. Summary .......................................................................................................101
`
`Chapter 5: An empirical evaluation of hierarchic marking menus ...........................103
`5.1. The experiment.............................................................................................105
`5.1.1. Design............................................................................................105
`5.1.2. Hypotheses ...................................................................................107
`5.1.3. Method ..........................................................................................109
`5.2. Results and discussion ................................................................................112
`5.3. Conclusions...................................................................................................119
`5.4. Summary .......................................................................................................121
`
`Chapter 6: Generalizing the concepts of marking menus...........................................123
`6.1. Introduction ..................................................................................................123
`6.2. Integrating marking menus into a pen-based interface .........................126
`6.2.1. Adapting to drawing and editing modes.................................127
`6.2.2. Avoiding ambiguity ....................................................................128
`6.2.3. Dealing with screen limits..........................................................134
`6.3. Applying the principles to iconic markings.............................................137
`6.3.1. Problems with the marking menu approach...........................139
`Overlap ..................................................................................139
`Not enough information .....................................................139
`6.3.2. Solutions........................................................................................140
`Crib-sheets.............................................................................140
`Animated, annotated demonstrations ..............................142
`6.4. Usage experiences........................................................................................150
`6.5. Summary .......................................................................................................151
`
`Chapter 7: Conclusions ....................................................................................................155
`7.1. Summary .......................................................................................................155
`7.2. Contributions................................................................................................157
`7.2.1. Marking menus............................................................................157
`
`
`
`iii
`
`

`

`
`
`Issues of human computer interaction. ....................................158
`7.2.2.
`7.3. Future Research............................................................................................160
`7.4. Final Remarks...............................................................................................161
`
`References ..........................................................................................................................163
`
`Appendix A: Statistical Methods....................................................................................171
`
`
`
`iv
`
`

`

`
`
`
`
`
`
`v
`
`

`

`

`

`
`
`Chapter 1: Introduction
`
`Research in the last forty years has brought great improvements in the quality of
`human-computer
`interactions.
` In the past, human-computer dialogs were
`optimized for the computer; humans communicated with computers using protocols
`that were easy for the computer to understand but were hard for a human to
`understand and use, for example, machine languages. Advances in human-
`computer interaction have changed this situation. Controlling a computer no longer
`requires memorizing obtuse, cryptic codes or an intimate understanding of the
`internal workings of the computer. In well-designed systems, human-computer
`interactions are optimized for the human. Interfaces now make use of sophisticated
`graphics, sound, and pointing devices to make the human's job easier.
`
`The major advances in human-computer interaction have been in making computers
`easier to use. Specifically, research on methods to reduce the amount of training a
`person needs before being able to operate a computer has come a long way. For
`example, the Apple Macintosh has set standards for the minimal amount of
`instruction that a person needs before operating a computer. Because of these
`advances, the world of computers opened up for people who otherwise would not
`have invested the time in training to operate a computer system.
`
`Given these advances in human-computer interaction, we can think of the interface
`as currently being optimized for the human, specifically, the novice computer user.
`Clearly, this is of great value, but we can consider another important class of user—
`the expert. Human capacity for the development of skills is great. Virtuoso pianists
`are proof of this. Virtuosos invest a great deal of time in practicing their skills—
`eight hours of practice a day is not uncommon. Now consider expert computer
`users. It is not uncommon for an expert computer user to spend eight hours a day
`working on the computer. Therefore, there is untapped potential for human skill
`1
`
`
`
`

`

`
`
`development in human-computer interactions. A good interface should take
`advantage of this potential and not limit the efficiency of a skilled user.
`
`In order for this skill potential to be tapped, an interface must have certain
`properties. First, the interface must provide interaction methods that are suitable for
`an expert. Experts require efficient interactions. As a result, interactions may be
`terse and unprompted. Second, and most critically, the interface must also provide
`support for a novice to become expert. We look at the interface design not so much
`as making the interface easier to use but rather as accelerating the rate at which novices
`begin to perform like experts. This goal demands three components: support for the
`novice, support for the expert, and an efficient mechanism to support the transition
`from novice to expert (see Figure 1.1).
`
`Novice component
`(recognition)
`• exploration
`
`• explanation
`
`• learning
`
`Skill development
`
`Transition component
`(recognition and recall)
`• practice expert
`behavior
`• return to novice
`behavior when needed
`• seamlessly switch
`between the two
`
`Expert component
`(recall)
`• terse, unprompted
`and efficient actions
`
`
`
`Figure 1.1: The components required to accelerate the rate at which users begin to
`perform like experts. The novice component allows a user to issue commands by
`searching for them and recognizing them. The expert component allows a user to
`efficiently issue commands by recalling the action associated with the command.
`The transition component allows a user to efficiently switch between these two
`methods to learn and practice command action associations.
`
`In this dissertation, we focus on an interaction technique that is intended to take
`advantage of this skill potential and support the development of skill. We propose
`an interaction technique which has a two modes. In the first mode, the style of
`interaction is intended to facilitate novice use. In the second mode, the style of
`interaction is intended for skilled expert behavior. The first mode is also designed to
`allow a novice to practice the skills required in the second mode. A user can switch
`to the second mode by operating the technique quickly. One can think of this in
`metaphorical terms. When you are learning to drive a car, its suitable to have a car
`2
`
`
`
`

`

`
`
`that is designed for a student driver. However, as your driving skills improve, the
`car incrementally transforms into a Ferrari.
`
`1.1. GENERAL AREA AND DEFINITIONS
`
`To support the expert component described in the previous section, we focus on a
`style of human computer interaction in which a user “writes” on the display surface.
`This style of interaction is similar to writing or drawing with a pen on ordinary
`paper. Writing on a display, however, is accomplished with a special pen and the
`computer simulates the appearance of ink.1
`
`We define a mark as the series of pixels that are changed to a special “ink” color
`when the pen is pressed and then moved across the display. The pixels that are
`changed to an ink color are those which lay directly under the tip of the pen as it is
`moved across the display. Free hand drawings, ranging from meaningless scribbles
`to meaningful line drawings and symbols, including handwriting, are examples of
`marks. The act of drawing a mark is referred to as marking.
`
`Marks can be created not only with a pen but also with other types of input devices.
`For example, a mouse can leave a trail of ink (commonly referred to as an ink-trail)
`behind the tracking symbol when the mouse button is pressed and the mouse is
`dragged. Some systems use a pen and tablet. In this case, marks are made on the
`display by writing on the tablet instead of the display.
`
`From a user’s point of view, these interfaces allow one to make marks and then have
`the system interpret those marks. There are, however, systems in which marks can
`be made but not recognized by the system. They are interpreted strictly as
`annotations, for example, Freestyle (Perkins, Blatt, Workman, & Ehrlich, 1989). The
`focus of this dissertation, however, is on systems in which marks are interpreted as
`commands and parameters.
`
`Much of the literature refers to marks as gestures. However, the term gesture is
`inappropriate in this context. Indeed creating a mark does involve a physical
`
`
`
`1 The pen, in these types of systems, is sometimes referred to as a stylus.
`
`
`
`3
`
`

`

`
`
`gesture but the real object of interpretation is the mark itself.2 For example, the “X”
`mark requires a completely different physical gesture if performed with a pen
`instead of a mouse. Gesture is an important aspect of mark because some marks
`may require awkward physical gestures with the input device. However, the two
`terms should be distinguished. The term gesture is more appropriate for systems in
`which the gestures leave no marks, for example, VideoPlace (Krueger, Giofriddo &
`Hinrichsen, 1985). The term mark is more appropriate for pen-based computer
`systems or applications that emulate paper and pen.
`
`1.2. WHY USE MARKS?
`
`Current human-computer interfaces are asymmetric in terms of input and output
`capabilities. There a number of computer output modes: visual, audio and tactile.
`Most computers extensively utilize the visual mode; high resolution images which
`use thousands of colors of can be displayed quickly and in meaningful ways to a
`user. In contrast, a computer's ability to sense user input is limited. Humans have a
`wide range of communication skills such as speech and touch, but most computers
`sense only a small subset of these. For example, keyboards only sense finger presses
`(but not pressure) and mice only sense very simple arm or wrist movements.
`Therefore, we believe the advent of the pen as a computer input device provides the
`opportunity to increase input bandwidth through the use of marks.3
`
`There are two major motivations for using marks. The first addresses the problem
`of efficiently accessing the increasing number of functions in applications. The
`second motivation is that there are some intrinsic qualities that marks have which
`can provide a more “natural” way to articulate otherwise difficult or awkward
`concepts (such as spatial or temporal information). Both of these motivations will
`now be examined in more detail.
`
`
`
`2 There are systems where interpretation depends not only on what is drawn but also how it is drawn. For
`example, an "X" drawn quickly may have a different interpretation from a "X" that is drawn slowly. By this
`dissertation's terminology, these systems would contain a combination of marking and gesture recognition.
`
`3 It is ironic that one of the first input devices for graphics was a light pen which wrote directly on the display
`surface (Sutherland, 1963).
`
`
`
`4
`
`

`

`
`
`1.2.1.
`
`Symbolic nature
`
`The inadequacy of mouse and keyboard interfaces is exemplified by applications
`that are controlled through button presses and position information.4 Buttons must
`be accessible and thus require physical space. Problems occur when an application
`has more functions than can be mapped to buttons or reasonably managed on the
`display. Other problems also exist: arbitrary mappings between functions and
`buttons can be confusing, and user management of the display and removal of
`graphical buttons can be tedious.
`
`Expert users of these types of systems find the interface inadequate because button
`interfaces are inefficient. The existence of interaction techniques that override
`buttons for the sake of efficiency is evidence of this. Experts, having great
`familiarity with the interface, are aware of the set of available commands. Menus
`are no longer needed to remind them of available commands and invoking
`commands through menu display becomes very tedious.
`
`Designers have addressed this problem in several ways. One solution is accelerators
`keys which allow experts direct access to commands. An accelerator key is a key on
`the keyboard which, when pressed, immediately executes a function associated with
`a menu item or button. The intention is that using an accelerator key saves the user
`the time required to display and select a menu item or button. Many systems
`display the names of accelerator keys next to menu items or buttons to help users
`learn and recall the associations between accelerator keys and functions.
`
`Another way of supporting an expert is by supplying a command line interface in
`addition to a direct manipulation interface. Commodore’s command line interface,
`CLI, and graphical user interface, Intuition, are an example of this approach.
`
`Both these approaches have their problems. In the case of accelerator keys, arbitrary
`mappings between functions and keys can be hard to learn and remember.
`Sometimes mnemonics can be established between accelerator key and function
`(e.g., control-o for “open”), but mnemonics quickly run out as the number of
`accelerator keys increases. Further confusion can be caused by different applications
`
`
`
`4 The term buttons is used as a generic way of describing menus items, dialog box items, icons, keys on a
`keyboard, etc., which are typical of direct manipulation interfaces.
`
`
`
`5
`
`

`

`
`
`using a common key for different functions or by different applications using
`different keys for a common function. Experts must then remember arbitrary or
`complex mappings between keys and functions depending on application.
`Command line interfaces are problematic because they are radically different from
`direct manipulation interfaces. To become an expert, a novice must learn another
`entirely different interface.
`
`Marks, because of their symbolic nature, can make functions more immediately
`accessible. Rather than triggering a function by a button press, a mark can signal a
`command. For example, a symbolic mark can be associated with a function and a
`user can invoke the function by drawing the symbol. In theory, because marks can
`be used to draw any symbol or series of symbols, marks can provide a quicker
`method of choosing a command than searching for a physical or graphical button
`and pressing it. In practice, the number of marks is limited by the system's ability to
`recognize symbols and a human's ability to remember the set of symbols.
`Nevertheless, even if only a small set of marks are used, a user can invoke the
`associated functions immediately.
`
`Marks can also be used to hide functions because they are user generated symbols.
`For example, researchers at Xerox PARC made use of this property when faced with
`a dilemma during the design of a pen-based application. This application runs on a
`wall sized display where a user can write on the display using an electric pen (Elrod
`et. al., 1992). There were two major design requirements. First, the designers
`wanted the application to look and operate like a whiteboard and maximize the size
`of the area where drawing could take place. Second, they wanted to provide
`additional functions commonly found in computer drawing programs. This second
`requirement meant that many graphical buttons would need to appear on the
`screen. This, however, violated the first design requirement because the numerous
`graphical buttons would consume too much of the drawing area and make the
`interface look complicated.
`
`The design solution was to assign many of the drawing functions to marks. M