United States Patent 19
`Kurtenbach
`
`5,689,667
`[11] Patent Number:
`Nov. 18, 1997
`145] Date of Patent:
`
`
`US005689667A
`
`[54] METHODS AND SYSTEM OF
`CONTROLLING MENUS WITH RADIAL
`AND LINEAR PORTIONS
`
`[75]
`
`Inventor: Gordon. P. Kurtenbach, Toronto,
`Canada
`sre
`i
`.
`oo
`[73] Assignee: Silicon Graphics, Inc., Mountain View,
`Calif.
`
`[21} Appl. No.: 469,913
`[22] Filed:
`Jun. 6, 1995
`[51]
`Tint, Co ccccsccssessssesescnsscssscssscenscenscnnscere GO6F 3/00
`[SZ] WS. CU. cacccsssssscsssssensee 395/352; 395/353; 395/358;
`345/146
`[58] Field of Searcy .ssososctssscueseusnsen 395/155-161,
`.
`1
`395/326-358; 345/117-120, 146, 902
`References Cited
`
`[56]
`
`Implementation, Evaluation, and
`Hopkins et al, “Pies:
`Application of Circular Menus”, UM CS Dept. Tech. Report,
`1988, pp. 1-25.
`Gordon P. Kurtenbach, “An Empirical Evaluation of Some
`Articulatory and Cognitive Aspects of Marking Menus”,
`Human-Computer Interaction, 1993, vol. 8, pp. 1-23.
`Gordon Kurtenbach and William Buxton, “The Limits Of
`Expert Performance Using Hierarchic Marking Menus”,
`Interchi °93, Apr. 24-29, 1993, pp. 482-487.
`G. Kurtenbach, TP. Moran and W. Buxton, “Contextual
`Animation of Gestural Commands”, Computer Graphics
`forum, vol. 13, (1994) no. 5, pp. 305-314.
`Don Hopkins, “The Design and Implementation of Pie
`Menus”, Dr. Dobb’s Journal, Dec. 1991.
`Gordon Paul Kurtenbach, “The Design and Evaluation of
`Marking Menus”, 1993, pp. 1-173.
`Gordon Kurtenbach and William Buxton, “User Learning
`and Performance with Marking Menus”, Human Factors in
`Computing Systems CHI 94, Apr. 24-28, 1994, pp.
`258-264.
`
`U.S. PATENT DOCUMENTS
`
`Primary Examiner—John E. Breene
`
`4,496,699
`5,303,388
`5,500,935.
`5,524,196
`5,559,944
`
`1/1985 Driskell .........cccsssssssssessesssensece 395/352
`
`4/1994 Kreitman etal.
`- 395/159
`3/1996 Moran etal. .....
`. 395/156
`
`~ 395/155
`6/1996 Blades......
`
`
`9/1996 OM0 w.e.rescsssosscasearee
`.- 395/156
`OTHER PUBLICATIONS
`LaLondeetal, “Pie Menus”, Joop,v. 2, n. 1, May/Jun. 1989,
`pp. 47-54.
`Callahanet al, “An Empirical Comparison of Pie vs. Linear
`Menus”, ACM, 1988, pp. 95-100, (with illustrations).
`
`[57]
`
`ABSTRACT
`«
`:
`:
`:
` Asystem that combines a radial marking menu portion with
`a linear menu portion in a single menu display. Item selec-
`tion in the linear portion is performed by location selection
`using a pointing device. Item selection in the marker portion
`is determined by thepattern of a stroke made by thepointing
`device with the system ignoring linear menu items across
`which the stroke completely passes.
`
`4 Claims, 9 Drawing Sheets
`
`MENU ITEM 1
`
`MENU ITEM 8
`
`MENU ITEM 2
`
`MENU ITEM 7
`MENU ITEM 6
`
`MENU ITEM 3
`MENU ITEM 4
`
`62 MARKING
`
`MENU ITEMS
`
`MENU ITEM 5
`
`
`
`MENU ITEM 9
`
`MENU ITEM 10
`
`
`64 LINEAR
`
`MENU ITEM 12|MENU ITEMS
`
` MENU ITEM 14
`
`MENU ITEM 11
`
`MENUITEM13 |
`
`eo
`
`HP Inc. - Exhibit 1015 - Page 1
`
`HP Inc. - Exhibit 1015 - Page 1
`
`

`

`5,689,667
`
`Nov. 18, 1997
`
`Sheet 1 of 9
`
`U.S. Patent
`
`FIG. 1
`
`MENU ITEM 3
`
`so”
`
`HP Inc. - Exhibit 1015 - Page 2
`
`HP Inc. - Exhibit 1015 - Page 2
`
`

`

`U.S. Patent
`
`Nov. 18, 1997
`
`Sheet 2 of 9
`
`5,689,667
`
`FIG. 2
`
`MENU ITEM 14
`
`MENU ITEM 1
`
`MENU ITEM 2
`
`MENU ITEM 3
`
`MENU ITEM 4
`
`MENU ITEM 5
`
`MENU ITEM 6
`
`MENU ITEM 7
`
`MENU ITEM 8
`
`MENU ITEM 9
`
`MENU ITEM 10
`
`MENU ITEM 11
`
`MENU ITEM 12
`
`MENU ITEM 13
`
`LINEAR MENU
`
`HP Inc. - Exhibit 1015 - Page 3
`
`HP Inc. - Exhibit 1015 - Page 3
`
`

`

`U.S. Patent
`
`Nov. 18, 1997
`
`Sheet 3 of 9
`
`5,689,667
`
`FIG. 4
`
`56 —~
`
`
`
`[STAPLES] *
`
`
`[MISC]_ 52
`
`54—~,
`
`
`
`HP Inc. - Exhibit 1015 - Page 4
`
`HP Inc. - Exhibit 1015 - Page 4
`
`

`

`U.S. Patent
`
`Nov. 18, 1997
`
`Sheet 4 of 9
`
`5,689,667
`
`FIG. 6
`
`MENU ITEM 1
`
`MENUITEM 8
`
`MENU ITEM 2
`
`MENUITEM 7
`MENU ITEM 6
`
`MENUITEM 3
`MENU ITEM 4
`
`62 MARKING
`MENU ITEMS
`
`MENU ITEM 9
`
`MENU ITEM 5
`
`
`
`
`64 LINEAR
`MENU ITEM 12|MENU ITEMS
`
`
`
`
`
`
`
`607
`
`
`
`HP Inc. - Exhibit 1015 - Page 5
`
`HP Inc. - Exhibit 1015 - Page 5
`
`

`

`U.S. Patent
`
`Nov. 18, 1997
`
`Sheet 5 of 9
`
`5,689,667
`
`
`
`74
`
`HP Inc. - Exhibit 1015 - Page 6
`
`HP Inc. - Exhibit 1015 - Page 6
`
`

`

`U.S. Patent
`
`Nov. 18, 1997
`
`Sheet 6 of 9
`
`5,689,667
`
`FIG. 9
`
`120
`
`140
` MOUSE
`MOVED MOUSE
`
`
`DOWN EVENT
`
`
`
`YES
`
`NO
`
` RECORD X.Y
`
`SET CURRENT
`MODE TO
`MARKING
`
`SETCURRENT
`ENUT
`STARTING ME
`“_
`
`SET TIMER
`INTERRUPT
`FOR 1/3 SEC.
`
`RECORD X,Y
`
`Suh
`A =“
`
`
`
`
`42
`MOUSE
`MOVED MORE THAN
`5 PIXELS
`9
`
`
`
`144
`|
`veg {RESET TIMER
`INTERRUPT
`FOR 1/3 SEC.
`
`
`
`446
`
`NO
`
`
`
`YES
`
`152
`
`OF CURSOR
`AND THE TIME
`‘a
`DRAW LINE FROM
`PREVIOUSX,Y
`TO CURRENTX,Y
`
`143
`HILITE MENU
`ITEM ASSOCIATED
`WITH X,Y LOCATION
`
`NO}
`
`150
`
`RECORDTHIS
`
`ITEM AS CHOICE
`FOR THE CURRENT
`feCU
`
`
`
`3
`
`WAIT
`
`120
`
`WAIT
`
`HP Inc. - Exhibit 1015 - Page 7
`
`HP Inc. - Exhibit 1015 - Page 7
`
`

`

`U.S. Patent
`
`Nov. 18, 1997
`
`Sheet 7 of 9
`
`5,689,667
`
`120
`
`WAIT
`
`_ FIG. 10
`
`
`FOR 1/3 SEC.
`
`YES
`
`170
`
`RESET TIMER
`
`SELECTED
`
`
`
`YES
`
`182
`
`DISPLAY SUBMENU
`
`184
`
`MAKE SUBMENU THE
`CURRENT MENU
`
`120
`
`HP Inc. - Exhibit 1015 - Page 8
`
`FOR 1/3 SEC.
`
`YES
`
`SET MODE
`TO
`MENUING
`
`1
`
`64
`
`1
`
`66
`
`REPLACE THE
`MARK WITH
`MENUS
`
`168
`
`RESET TIMER
`
`400
`
`WAIT
`
`
`ITEM HAS SUB-
`
`
`MENU ?
`
`HP Inc. - Exhibit 1015 - Page 8
`
`

`

`U.S. Patent
`
`Nov. 18, 1997
`
`Sheet 8 of 9
`
`420
`
`FIG. 117
`
`
`
`MENUS FROM
`SCREEN
`
` EXECUTE THE
`
` No.|REMOVE ALL
`COMMAND
`ASSOCIATED
`WITH THE SERIES
`OF MENU CHOICES
`
`
`
`
`
`RECORD X,Y
`POSITION
`
`AND TIME
`
` DRAW LINE FROM
`
`PREVIOUS X,Y
`TO CURRENTX,Y
`
`
`
`
`204
`
`ERASE MARK
`FROM SCREEN
`
`206
`
`
`
`USE SAVED X,Y POSITIONS
`
`AND TIMES TO DETERMINE
`MENU CHOICES FROM
`THE MARK
`
`
`HP Inc. - Exhibit 1015 - Page 9
`
`HP Inc. - Exhibit 1015 - Page 9
`
`

`

`Nov. 18, 1997
`
`Sheet 9 of 9
`
`5,689,667
`
`U.S. Patent
`
`21210)
`
`ELIOReLioeN92]|Nelo]|Nhefot|NI(>lolo)|[plolo}|[p[ajo)|[>]
`
`pe1BUS
`
`HP Inc. - Exhibit 1015 - Page 10
`
`HP Inc. - Exhibit 1015 - Page 10
`
`

`

`5,689,667
`
`1
`METHODS AND SYSTEM OF
`CONTROLLING MENUS WITH RADIAL
`AND LINEAR PORTIONS
`
`BACKGROUND OF THE INVENTION
`
`1. Field of the Invention
`
`2
`It is also an object of the present invention to provide a
`processthat distinguishes between linear (or location) based
`menu selection and pattern-based menu selection.
`The above objects can be attained by a system that
`combines a radial marking menu, in which item selection is
`performed by marking a menuitem with a cursor mark, with
`a linear menu, in which item selection is performed by
`selection at a particular location. The system distinguishes
`between parts of the display in which the linear menu is
`located and parts of the display in which the radial marking
`menu is located.
`
`These together with other objects and advantages which
`will be subsequently apparent, reside in the details of
`construction and operation as more fully hereinafter
`described and claimed, reference being had to the accom-
`panying drawings forming a part hereof, wherein like
`numerals refer to like parts throughout.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`10
`
`15
`
`20
`
`25
`
`The present invention is directed to a method and appa-
`ratus for producing, controlling and displaying menus that
`combine radial marking menus and linear menus and, more
`particularly, to a system that combinesa pattern selectable
`menu with a location selectable menuinto a single display.
`2. Description of the Related Art
`Menusselection in modern user interfaces is a fundamen-
`tal and frequently used operation. In general, the length of
`time it takes to select from a menu is small (seconds or
`fractions of second). However, because menuselection is a
`high frequency operation in some types of applications,
`reducing the time involved in selecting from a menu can
`result in significant time savings in the overall time it takes
`FIG.1illustrates typical hardware of the present inven-
`to accomplish a larger task. It is for this very reason that
`tion;
`menuselection speed up (“accelerator”) techniques like “hot
`keys” and marking menusare popular.
`FIG.2 illustrates a linear or location selection type menu;
`Each accelerator technique has its own advantages and
`FIG.3 illustrates a radial marking type menu;
`disadvantages. For example, hot keys can select from a large
`FIG. 4 illustrates submenu selection using a marking
`set of menu items, for example, every key on the keyboard
`menu;
`can be usedas hot key. However, this is at the cost of the user
`remembering the associations between menu items and
`keys. Hot keys are also noteffective if a user is working on
`a system that does not include a keyboard or the user must
`move between the keyboard and another device while oper-
`ating the interface. For example,the user may have to move
`the same hand from a mouse,to the keyboard, and then back
`to the mouse.
`
`30
`
`35
`
`FIG. 5 depicts selection using a marking pattern without
`producing a display;
`FIG. 6 shows a combined radial marker and linear menu
`according to the present invention;
`FIG.7 illustrates selection operations of the invention:
`FIGS. 8-11 comprise flowcharts of the steps of the present
`invention; and
`FIGS. 12 and 13 depict an icon menus and expansion
`thereof from a tool pallet.
`
`Marking menusis a pop up menu technology that displays
`menu items in a circle, around the cursor whereastraditional
`menus(linear menus) display menu items in a linear fashion,
`for example, from top to bottom. Because ofthis difference,
`item selection can be performed more rapidly with marking
`menus than with linear menus. For example, a user can
`select from marking menusby “flicking” or making a stroke
`of the cursorin the direction of a desired menu item. Thus,
`the user does not haveto wait for the menu to bedisplayed
`to select from it. However, as the number of items in a
`marking menuincreases, rapid selection without popping up
`the menu becomesdifficult because the angular difference
`between menu items becomes small anddifficult for the user
`to articulate. To combatthis problem, the number of items
`in a marking menusis generally limited to eight orless.
`Whatis needed is a combinationofa radial marking menu
`and a linear menu in the same display such that the number
`of items in the menu can be increased beyond eight items
`while still permitting rapid selection for the items of the
`marking menu using a marking or stroke pattern and selec-
`tion of the items of the linear menu using a locational
`method.
`
`45
`
`50
`
`SUMMARY OF THE INVENTION
`It is an object of the present invention to combine radial
`marking menus with linear menus in the samedisplay.
`It is another object of the present invention to combine
`marking menu selection techniques with location selection
`techniques.
`It is an additional object of the present invention to
`provide a menu display in which selection of some items are
`accelerated.
`
`65
`
`DESCRIPTION OF THE PREFERRED
`EMBODIMENTS
`
`invention typically has a hardware
`The present
`configuration, as illustrated in FIG. 1, that includes a com-
`puter 20 which generates and displays menus on a display
`22. A.userinteracts with the display 22 through the computer
`system 20 using an interface device, such as a keyboard 24,
`and a pointing device 26, such as a mouse. The computer 20
`includes the conventional hardware necessary to interact
`with a pointing device 26, such as motion detection circuits
`and button depression detection circuits, and the appropriate
`storage media, such as a hard disk or a floppy disk on which
`the process discussed herein is stored. The computer 20 can
`be any type of computer from a personal/portable computer
`to a main frame computer with the typical computer being
`a workstation type computer that is used for high volume
`production operations in which menuselection is a signifi-
`cant portion of the operations performed by a user, such as
`in CAD/CAM and animation operations. The display 22 can
`also be any type of display that is capable of displaying a
`menu, such as a CRT display or a liquid crystal display. The
`pointing device 26 can be any type of pointing device
`including a pen, a mouse,a track ball, a track point or a joy
`stick, either separate from or part of the keyboard 24. The
`display 22 and pointing device 26 can also be combined into
`a single device, such as touch sensitive screen.
`The present invention combines a radial marking menu
`with a linear menu into a single display where the radial and
`linear portions are associated with each other and displayed
`
`HP Inc. - Exhibit 1015 - Page 11
`
`HP Inc. - Exhibit 1015 - Page 11
`
`

`

`5,689,667
`
`3
`simultaneously. The present invention allows the menusto
`be selected using the pattern and location techniquesof these
`two types of menus. The selection and display arrangement
`of linear and radial menus will be discussed before discuss-
`ing their combination.
`A linear menu 30,as illustrated in FIG. 2, is typically a
`linear array of menu items where the number shownin FIG.
`2 is fourteen items (1~14) but where the array shownis a
`single dimensional array. The number of items can, of
`course, be less than or greater than 14 and the items can be
`arranged in two or more dimensions or even in an arbitrary
`arrangement. The main feature of a linear menu is the way
`in which it is used to select a desired item. Typically, a
`pointer (or cursor) 32, such as an arrow as shown, is moved
`to the location or area defined by the menu item desired to
`be selected by manipulating the pointing device 26 and the
`selection is made by depressing (or releasing) a button that
`is typically associated with the pointing device 26 but which
`could be in a separate interface device, such as the keyboard
`24. The computer 10 determines the location of the pointer
`32 at the time the button is depressed (or released) and if the
`pointer is within the area defined by a menu item, that item
`is selected. If the pointer is not within any areas defined by
`menu items nothing is selected. That is, linear menususe the
`location of the pointer 32 to determine the item selected. If
`the user depresses(or releases) the selection button while the
`pointer 32 is positioned over Menu Item 1, as shownin FIG.
`2, Menu item 1 would be selected. Using a linear menu
`requires that the user position the pointer 32 in the area of
`the display defined by the item to be selected. This can take
`a considerable amount of time for the user and accurate
`positioning is required. The accurate positioning is even
`more critical when the item is an icon that can be rather
`small.
`
`Radial menus include two types: pie menus and marking
`menus. Pie menusare typically used in item selection using
`the location principles of linear menus as discussed above.
`Marking menus operate on the principle of the direction of
`cursor or pointer motion as being the basis for item selec-
`tion. Marking is a menu selection operation which works as
`follows. A user activates the selection indicator of the
`pointing device 26, such as by pressing down on the screen
`with a pen or holding downthe button on a mouse and waits
`for a short interval of time (approximately 4 second). A
`radial menu 40, such as illustrated in FIG. 3, then appears
`(“pops-up”) directly underthetip of the pen 52. A user then
`highlights an item by keeping the pen pressed and making a
`stroke 42 towardsthe desired item. The stroke 42 is created
`by the computer constantly redrawing a straight
`line
`between the center 44 of the menu 40 and the current
`location of the pointer (that is, the stroke 42 behaveslike a
`“subber band line” between the center of the menu and the
`pointer). The computer 20 highlights a particular item when
`the pointer movesinto the angular range associated with that
`menu item (for example, angle range 46, associated with
`MenuItem 1 in FIG. 3). If the item has no submenu, the
`highlighted item can be selected by lifting the pen. If the
`item does have a sub-menu, and the user stops moving the
`pointer, the submenu is displayed with the center 48 of the
`new menu under the pen 52 (see FIG. 4). The user then
`continues, from the new center 48, selecting the newly
`displayed sub-menu by providing another stroke 50 with the
`pen 52. Lifting the pen 52 will cause the current series of
`highlighted items to be selected. In the example of FIG.4 the
`items selected are “Groceries” in the main menu 54 and
`“Fruit & Veg” in the submenu 56. The menus are then
`removed from the screen. At any time a user can indicate “no
`
`4
`selection”of the submenu 56 by moving the pen backto the
`center (48) of the submenu before lifting, or change the
`selection by movingthe pen to highlight another item before
`lifting. Finally, a user can “back-up” to a previous menu by
`pointing to its center 44.
`Another, faster, way to make a selection without popping
`up the menu is by drawing a mark 58 as illustrated in FIG.
`5. A mark can be drawn by pressing the pen down and
`immediately moving. The direction of the mark 58 dictates
`the particular item selected from the menu. In the present
`invention if only a mark is made, as will be discussed in
`moredetail later, and the menuis not allowed to pop-up, the
`system interprets the mark as if the linear portion of the
`present invention doesnotexist. This is called the “marking”
`mode as opposed to the “menuing” mode when the com-
`bined marking and linear menuis displayed which will be
`discussed in more detail later.
`
`Thus, marking menus allow items to be selected in two
`different ways. Using the method of FIG. 4 radial menus can
`be sequentially displayed and selections made. The method
`of FIG. 5 uses marking without menu display to make the
`sameselection. The first method is good when the user is
`unfamiliar with the menu. The second method is good when
`the user is familiar with the menu and wants to avoid waiting
`for the display of the menu.
`Marking menus avoid someofthe problems of hot keys.
`A.user does not have to remember the association between
`keys and menu items. The user only needs to rememberthe
`spatial layout of menu items. Typically users very quickly
`learn the spatial location of menu items, especially for
`frequently used commands. Marking menus also do not
`require a keyboard for operation. However, because human
`capability is limited in the accuracy in which angles can
`drawn, the number of menu items in marking menus is
`generally limited to eight items.
`In manysituations modern user interfaces use menus with
`more than eight items. For example, the number of items in
`pop-up menusin the ALIAS V6 package ranges from 8 to 15
`items, with an average of 12.75 items per pop-menu. Higher
`number of items can be used in marking menus by making
`the menus hierarchic. In this case rapid menu selection
`requires a zig-zag type line with pauses rather than just a
`simple stroke. For example, a 64 item menu can be made up
`of an eight item menu where each item leads to another eight
`item submenu. Thus, if a user interface designer wishes to
`convert an application which uses linear menus into one
`which uses marking menus, the designer must convert any
`menu that takes more than eight items into an hierarchic
`menu. This conversion process requires extra work and
`linear menu items may not decompose naturally into hier-
`archic categories.
`The present
`invention combines marking menus and
`linear menusinto a single menu 60,as illustrated in FIG. 6,
`such that converting linear menus into a marking menu does
`not require the decomposition of menuitems into categories.
`In the invention upto the first eight items of the traditional
`linear menu that are the most highly selected are mapped
`directly to a marking menu portion 62. The additional
`(“overflow”) items are mappedto a linear menu portion 64,
`which is displayed at the same time the marking menu
`portion 62 is displayed. The linear portion 64 is shown
`below the marker portion 62, however, the linear portion
`position can be varied as desired as long as it appears
`simultaneously on the same display.
`The present invention, as depicted in FIG. 6, has the
`following advantages.First, as stated before, conversion of
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`65
`
`HP Inc. - Exhibit 1015 - Page 12
`
`HP Inc. - Exhibit 1015 - Page 12
`
`

`

`5,689,667
`
`5
`linear menusto this format is simple. No decomposition into
`hierarchic categories is needed and morethan eight item per
`menu can be used. Second, accelerated selection can be
`performedonthefirst eight items, such that time savings can
`be realized over traditional linear menus. Third, because
`someof the items in the menu 60 are laid outside by side
`as opposed to top to bottom, the overall height of menu 60
`is reduced. Finally, this type of menu 60 takes up approxi-
`mately the same screen spaceasa tradition linear menu and
`therefore this new menu 60 can be displayed at the same
`The present invention is directed to the combination of
`location as tradition linear menus (for example, the menu
`radial menus and linear menus in the same display. If the
`can be pop-up or pop-down,etc.).
`invention were to follow the typical rules for menu selection
`The invention has some characteristics different from
`in the prior art this would mean that all menuselections
`traditional linear menus.First, the user must pause to display
`(either by displaying a menu or by drawing a mark) would
`the menubefore selecting an item from the linear portion 64
`be based onthe location ofthe cursorat the end ofthe stroke.
`of the menu. ‘This is not a great disadvantage since these
`This in turn would result in selection errors whenauser tries
`items are by definition supposed to be infrequently used
`to select from the menu of the present invention quickly.
`commands. If the position where the menu pops-up is very
`Thus, the advantage anddistinction ofthe present invention
`close the edge of screen there may not be enough room to
`is apparent.
`select some of the items from the radial portion 62 of menu
`The operation of the process of determining the selection
`using a flick. For example, suppose the menu 60 pops up
`is illustrated in the flowcharts of FIGS. 8-11. Typically such
`whena user presses the mouse button downright in the top
`a process is interrupt driven with interrupts occurring based
`left corner of the screen. At this point there is no room to
`on an interrupt
`timer or based on events, such as the
`“flick” the cursorup or to the left to select some of the menu
`depression/release of a mouse button or the detection of
`items. In practice, this is not a serious problem since there
`mouse movement. As can be seen from a visual review of
`are very few situations where a menu is popped up right
`FIGS. 8-11 the operations performed begin with an exit
`along the edge for the screen. Also, the amount of room
`from a wait state 120 and return to this state 120. The exit
`needed for a flick can be very small. Finally, the radial
`from the wait state is caused by an event, which eventis
`portion 62 of menu 6¢@ represents a grouping of menu items
`checked to determinethe type of event as discussed in more
`that is based on frequency of selection. In linear menus,
`detail below. The operations can generally be broken into
`items are generally grouped together based on function.
`pen-down—FIG. 8, pen-drag—FIG. 9, pen-still—FIG. 11
`Therefore, the conversion to this new menu system might
`and pen-up—FIG.11.
`require rearrangementof items in a menu. Fortunately, the
`The pen-down operation of FIG. 8 first checks 122 to
`particular grouping of menu items haslittle value once a user
`determine whether the mouse button down event has
`becomes familiar with a menu layout and therefore this is
`occurred. If not, the system returns to the wait state 120.
`not a serious problem in the long run.
`Otherwise, the mode is set 124 to the marking menu mode
`The creation of such a combined menu requires that a
`and the menu selection is set 126to thestarting or highest
`displayed menu 79,as illustrated in FIG. 7, be divided into
`menuin a hierarchy. A system timer is then set 128 to ensure
`different regions. In the linear region which is defined by
`that the system detects that the pen/mouse is no longer
`each of the areas of the individual menu items, “display
`movingor is still. The current position of the cursor and the
`buttons” or selection regions 72 are defined. In the marking
`time are then recorded 130.
`region, whichis all of the display outside the display buttons
`72 and 73, the selection regions are defined by angular
`ranges 74. During operation if a stroke or mark, such as
`stroke 76, has an endpoint 78 that is within a linear selection
`region 72 or within a label 73 (and the pen is lifted at that
`point), the item of the end point 78 is selected which in this
`example is Item 5.If a stroke 80 is in a region of the marking
`menu portion, the item selected is the item ofthe coinciding
`range or wedge which in this particular example is Item 4.
`If a stroke 82 ends in a marker region 74 after crossing one
`or morelinear regions 72, the item selected is the item of the
`coinciding wedge or marker region whichin this particular
`example is item 3. That is, whenever the end point of the
`stroke does notlie within a linear region 72 (or label 73), the
`coinciding or corresponding marker regionitem is selected.
`A radial menu label can fall outside an angular range
`associated with that label, however, pointing to a menu item
`overrides the angular ranges. For example, a portion of
`MenuItem 2 (the area of the label for Item 2) in FIG. 6 can
`lie, depending on how the angular ranges of each item are
`defined, in the angular range for Menu Item 3. However,
`pointing directly to the label of Menu item 2 will highlight
`it evenif the pointer is in the angular range of Menu Item 3.
`Linear menu items, such as 72 in FIG. 7, have no “angular
`tange” features and the user must position the pointer
`directly over the label for the item to be selected.
`
`In the pen-drag operation the first check performed, as
`depicted in FIG. 9, is to determine 140 whether the event of
`the mouse moving while the button is down has occurred. If
`not, again the system returns to the wait state 120. If so, a
`check 142 is madeof the distance of movementofthe cursor
`or pointer against a movementthreshold. If the threshold has
`been exceeded the timer is reset 144. In both cases the next
`step is a determination 146 concerning the mode. If the
`mode is not the marking mode, the item of the particular
`location of the cursor or pointer is highlighted and the item
`is recorded 150 as the current choice. If the mode is the
`marker mode, the current cursor location and the time is
`recorded.
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`45
`
`50
`
`55
`
`65
`
`6
`The invention behaves differently from the description
`above whenthe user does not display the menus but draws
`a mark. In this case, the mark is interpreted as if the linear
`menuitems do not exist. For example, if a mark like 76 were
`drawn this would not result in the selection of linear menu
`item 5, but in radial item menu 3. This has the advantagethat
`a user can quickly select radial menu items with a quick
`mark (a “flick”) without accidently selecting linear menu
`items.
`
`In the pen-still operation (FIG. 10), the first check 160 is
`to determine whetherthe timer event has occurred, if not, the
`wait state 120 is entered. In this operation the system also
`checks 162 on the mode. In the marker mode the modeisset
`164 to the menu mode and the mark is replaced 166 by the
`appropriate menus as previously discussed. The timeis then
`set 168 and the wait state is entered. When not in the marker
`mode, the system sets 170 the timer and determines 172
`whether the cursor is in the center of the parent menu.If so,
`the child menuis removed 174 from thedisplay.If not in the
`center of the parent menu, a determination 176 is made as to
`whether the position is over a label menu item. If not, a
`
`HP Inc. - Exhibit 1015 - Page 13
`
`HP Inc. - Exhibit 1015 - Page 13
`
`

`

`5,689,667
`
`7
`determination 178 is made as to whether the cursor is
`beyondthe edgeof the radial menu.If it is beyond the edge,
`a determination 180 is then made as to whether the item has
`a submenu.If a submenuexists for the item, the submenu is
`displayed 182 and the submenu is made 184 the current
`menu.
`
`The pen-up operation as depicted in FIG. 11 performs the
`item selection determination. The system determines 192
`whether the mouse button up event has occurred and if it is
`up, stops 194 the timer. If the current mode is determined
`194 not to be the marking mode, the menusare all erased
`196 and the selection is made 198 based on the recorded
`menu choices. If the mode is the marker mode, the current
`cursor position and time is recorded 200. A line is then
`drawn 202 from the between the previous and current cursor
`positions. The mark is then erased 204, the saved positions
`and times are used to determine 206 the menu choices from
`the mark and the selection is executed 198.
`
`It is important that an implementation take certain char-
`acteristics of typical window operating systems, with which
`the invention is typically used,
`into account as well as
`accounting for how such operating systems interface with
`applications. One problem is that many operating systems
`do not issue an event when a cursor or pointer moved by a
`pointing device stops moving andthe velocity drops to zero.
`Another problem is that some systems will not relinquish
`control unless an interrupt is generated. As a result, for such
`systems the implementation must include a provision for
`periodically sampling pointing device (cursor) position,
`such as by including timer interrupts, so that the various
`steps discussed above will work properly. Some operating
`systems do not include provisions for saving a portion of a
`screen into which a menu such as that of the present
`invention is “popped.” In such systems the implementation
`needs to provide for saving and restoring the portion of the
`display over written by the pop-up menu when or after the
`particular selection by the user is made. Because of such
`problems it is preferred that the invention be implemented in
`an XWINDOWSsystem, such as available from Silicon
`Graphics, Inc.
`Additional details concerning the preferred implementa-
`tion of the process depicted by FIGS. 8 and 11 are set forth
`in pseudocode included herein as an Appendix. It is pre-
`ferred that the process as described herein be implemented
`in a language such as C. However, any language suitable for
`user interface display generation and processing with a
`pointing device can be used.
`to
`The invention has been described with respect
`examples that depict menus with text labels. However, in
`many cases the preferred modeof display is to use icons. An
`example of an icon menu that includes radial as well as
`linear menu items that are arranged in a three-column array
`is illustrated in FIG. 12. In this example eight icons 210-224
`are shown in the marking menu portion 62 arranged around.
`a center 226 at which a pointer is located. The linear or
`location selection portion 64 includes six icons 228-238.
`This menuis selected in the same wayas the text label menu
`previously described with the user being required to under-
`stand the meaning of the icons. The linear menus can be
`arranged in a two dimensional arrangementas shownin FIG.
`12 and the linear menu can bearbitrarily located at any
`location around the radial menuthat is desired depending on
`wherethe radial menu actually pops up, something generally
`decided in the original design of the menu.
`FIG.13illustrates icon sets 250-264 of a tool box 266 and
`the expansion of the tool pallet 268 of a selected one of the
`
`8
`tool sets 252 with the pointer being positioned at the center
`of the pallet when expanded allowing continued marking or
`location dependentselections from the center of the pallet.
`Thisillustrates that the pallet can be expanded from a small
`and generally hard to decipher icon display representation.
`In addition, the marking selection process can be used to
`select icons from the small representations of the icons on
`the left in this figure even when the icons have not been
`popped-up or expandedto full size, when they might not be
`“readable” and when the user would have to remember
`which icon represented which selection.
`The menu items shown and discussed herein have
`included text labels and icons, however, other types of menu
`items such as moving pictures, symbols, sound, etc. can be
`substituted. The invention can also be used with any type of
`menu in a variety of contexts, such as a tool pallet, pulldown
`menu and object hot spots. The stroke or “ink trail” or
`“rubber band line” also need not be displayed. The number
`of items in each portion of the menu can vary and the items
`can be in any desired arrangement.
`The many features and advantages of the invention are
`apparent from the detailed specification and,
`thus,
`it is
`intended by the appended claims to cover all such features
`and advantages of the invention which fall within the true
`spirit and scope of the invention. Further, since numerous
`modifications and changes will readily occur to thoseskilled
`in the art, it is not desired to limit the invention to the exact
`construction and operation illustrated and described, and
`accordingly all suitable modifications and equival