p15
`United States Patent
`6,094,197
`Patent Number:
`[11]
`[45] Date of Patent:
`Jul. 25, 2000
`Buxton etal.
`
`US006094197A
`
`[54] GRAPHICAL KEYBOARD
`
`[75]
`
`Inventors: William A. S. Buxton, Toronto,
`Canada; Gordon P. Kurtenbach,
`Mountain View, Calif.
`
`[73] Assignee: Xerox Corporation, Stamford, Conn.
`
`[21] Appl. No.: 08/443,259
`
`Kurtenbachetal. “Issues in Combining Marking and Direct
`Manipulation Techniques,” User
`Interface Software and
`Technology (UIST), ITilton IIead, SC (Nov. 11-13, 1991),
`137-144.
`
`
`
`Kurtenbach, Gordon P. “The Design and Evaluation of
`Marking Menus,” Doctrate Thesis, Graduate Department of
`Computer Science, University of ‘loronto, 1993.
`Kurtenbach, Gordon “Making Marks Self—Revealing,” SIG-
`CHI Bulletin, vol. 23(4):60-61 (Abstract) (Oct. 1991).
`[22] Filed:|May 17, 1995
`Kurtenbachet al. “An Empirical Evaulation of Some Articu-
`latory and Cognitive Aspects of Marking Menus,” Human—
`Computer Intcraction 8:1-23 (1993).
`Robertson et al., “Buttons as First Class Objects on an X
`Desktop,” user Interface Software and Technology (UIST),
`Hilton Head, SC (Nov. 11-13, 1991), 35-44.
`
`
`
`Related U.S. Application Data
`
`[63] Continuation of application No. 08/171,288, Dec. 21, 1993,
`abandoned.
`
`Tint, C07 ec cccssscssssssscessesssessesssesessesnse G09G 5/00
`PSD]
`[52] U.S. Chee 345/358; 345/168; 345/179;
`341/22
`[58] Field of Search oe 345/156, 157,
`345/168, 169, 170, 172, 173, 174, 175,
`176, 179, 180, 181, 182, 183; 341/22, 23;
`170/18, 19; 395/326, 335, 347, 350, 351,
`352; 382/188, 189
`
`[56]
`
`References Cited
`U.S. PATENT DOCUMENTS
`
`Primary Examiner—Steven Saras
`Attorney, Agent, or Firm—Townsend & Townsend Crew
`LLP
`
`[57]
`
`ABSTRACT
`
`Asystem and method for a graphical keyboard that benefits
`from the expressive power andintuitive ease of use associ-
`ated with pen strokes and gestures, yet does not require
`complex character-recognition software. The graphical key-
`board responds differently to different kinds of pen strokes.
`For example, lowercase “a” is entered by tapping a stylus on
`the “a” key of the graphical keyboard. Uppercase “A” is
`entered by an upward stroke initiated over the “a” key.
`Likewise, by stroking in other dircctions, a user can express
`other modifiers to the basic character, such as “control’,
`“alt”, “command,” etc. Also, strokes in distinct directions
`can be used to express space, backspace, delete, and return
`European Pat. Off.
`12/1993
`0 603 669A1
`characters, for example. Multiple strokes can be applied in
`Japan .
`7/1985
`0144817
`sequence to a single key to express multiple modifiers.
`Japan .
`7/1985
`60-144817
`Visual feedback of pen strokes can be provided in the form
`0011144=11/1989) WIPO woes cece cneees 178/18
`of marks made by the pen. Pop-up menus can be used to
`89/11144
`11/1989 WIPO .
`
`
`92/09944=6/1992 WIPO wo..eeeeecetteeeeeee GO6F 3/033 provide help with and to facilitate memorization of com-
`mandstrokes and stroke sequences. The graphical keyboard
`OTHER PUBLICATIONS
`thus lets the expert user proceed by “feel,” and prompts the
`novice user when and as necessary.
`
`4,725,694
`5,050,105
`
`.. 345/157 5,148,155
`
`.ucecceccccccee cess 345/173
`2/1988 Aueret al.
`9/1991 Peters ..........
`9/1992 Martin etal.
`11/1993 Jones........
`5,258,748
`345/173
`1/1994 Lamb..........
`5,276,794
`. 345/156
`9/1994 Agulnick etal.
`5,347,295
`2/1997 Goughet al. oie eee 345/179
`5,603,053
`FOREIGN PATENT DOCUMENTS
`
`........ GO6F 3/033
`
`Knodt, Ruediger W. “Smart Virtual Keyboard System Suit-
`able for Mouse or Stylus Entry,” Xerox Disclosure Journal,
`vol. 18(3):245—245, May/Jun. 1993.
`
`18 Claims, 20 Drawing Sheets
`
`
`
`MSFT EX. 1006
`Page 1 of 35
`
`MSFT EX. 1006
`Page 1 of 35
`
`

`

`Jul. 25, 2000
`
`Sheet 1 of 20
`
`U.S. Patent
`
`6,094,197
`
`STROKE.a => "A"
`
`
`
`FIG. 3.
`
`MSFT EX. 1006
`Page 2 of 35
`
`MSFT EX. 1006
`Page 2 of 35
`
`

`

`U.S. Patent
`
`Jul. 25, 2000
`
`Sheet 2 of 20
`
`6,094,197
`
`UPPER CASE
`A
`
`OPTION ==--
`
`| ~~" CONTROL
`
`
`
`t
`COMMAND
`
`FIG.
`
`4.
`
`™ =>"CONTROL c”
`
`-= =>"CONTROL Cc"
`
`i
`
` BACKSPACE
`RETURN
`XN
`4
`
`
`
`MSFT EX. 1006
`Page 3 of 35
`
`MSFT EX. 1006
`Page 3 of 35
`
`

`

`U.S. Patent
`
`Jul. 25, 2000
`
`Sheet 3 of 20
`
`6,094,197
`
`TSE|CUSsaisk YE
`frablq[wleIrit|y[uj|ffablatwletrIth \j
`aislalflalh 4
`FAFACATALIIN
`
`a ‘eizlHA
`
`[|RPSI4)5|6bzie
`fabawnfist. SSS
`| (Pit,
`
`
`
`MSFT EX. 1006
`Page 4 of 35
`
`MSFT EX. 1006
`Page 4 of 35
`
`

`

`U.S. Patent
`
`Jul. 25, 2000
`
`Sheet 4 of 20
`
`6,094,197
`
`|“tt(2{3]4[5]61716 ||12[3[4i5[6[71¢
`
`
`Fabla{wlelritiyiul|fabla] fl yTu
`wileir]t|
`atedlFIOTRTA
`leper9TAT|
`Zix}cAy
`
`oN
`
`
`
`FIG 3.
`
`
`
`FIC 10,
`
`MSFT EX. 1006
`Page 5 of 35
`
`MSFT EX. 1006
`Page 5 of 35
`
`

`

`U.S. Patent
`
`Jul. 25, 2000
`
`Sheet 5 of 20
`
`6,094,197
`
`
` Gas
`
`
`CLIPBOARD} SELECTION USING
`
`
`
`SELECTION USING
`MENU MODE
`MARK MODE
`
`
`FIG.
`
`VL.
`
`/
`|
`
`2s
`
`173
`
`\
`4
`
`Q @Q@
`%ZING
`SEA
`3X0)
`<a?
`OS)
`ox
`
`3a
`
`3b
`
`££ Kaz
`2:0 a ol
`xe
`Y Z 6 ~
`\
`\
`NOO™
`
`3:c
`
`3:d
`
`4:4
`
`4:5
`4c
`MENU HIERARCHY
`MARK SET
`
`FIG.
`&2.
`
`4:d
`
`MSFT EX. 1006
`Page6 of 35
`
`MSFT EX. 1006
`Page 6 of 35
`
`

`

`U.S. Patent
`
`Jul. 25, 2000
`
`Sheet 6 of 20
`
`6,094,197
`
`(1)
`
`(2)
`
`FIG.
`
`13.
`
`UPPER CASE
`
`. !
`
`COMMAND
`
`SPACE” '
`
`MSFT EX. 1006
`Page7 of 35
`
`MSFT EX. 1006
`Page 7 of 35
`
`

`

`U.S. Patent
`
`Jul. 25, 2000
`
`Sheet 7 of 20
`
`6,094,197
`
`15.
`
`FIG.
`
`ea
`
`
`MSFT EX. 1006
`Page 8 of 35
`
`MSFT EX. 1006
`Page 8 of 35
`
`

`

`U.S. Patent
`
`Jul. 25, 2000
`
`Sheet 8 of 20
`
`6,094,197
`
`negere5161714SeTreyie)_|WPPERISER oF
`LL!12(ameeinen7|é
`abtgTHEHRCAS4
`OPMOMTGTALTRein
`
`feereeSFLLatCOMMAND vi
`
`
`
`FIFRA
`Eola.
`Chawiny24
`
`Ldlelol7
`
`MSFT EX. 1006
`Page 9 of 35
`
`MSFT EX. 1006
`Page 9 of 35
`
`

`

`U.S. Patent
`
`Jul. 25, 2000
`
`Sheet 9 of 20
`
`6,094,197
`
`NOILWONdd¥
`
`WVH90Ud
`
`AYOWIN
`
`
`
`BlOld
`
`Yasn
`
`JOVIUIINI
`
`MSFTEX. 1006
`Page 10 of 35
`
`MSFT EX. 1006
`Page 10 of 35
`
`
`

`

`U.S. Patent
`
`Jul. 25, 2000
`
`Sheet 10 of 20
`
`Yas
`
`NOWLVONdd¥
`
`WY¥U90Ud
`
`6,094,197
`
`
`
`J0VIU3INI
`
`MSFTEX. 1006
`Page 11 of 35
`
`MSFT EX. 1006
`Page 11 of 35
`
`

`

`Jul. 25, 2000
`
`Sheet 11 of 20
`
`U.S. Patent
`
`6,094,197
`
`FIG.
`
`IQA.
`
`MSFT EX. 1006
`Page 12 of 35
`
`MSFT EX. 1006
`Page 12 of 35
`
`

`

`U.S. Patent
`
`Jul. 25, 2000
`
`Sheet 12 of 20
`
`6,094,197
`
`OgOA
`
`4asn
`
`
`
`JOVINIINI
`
`Sb
`
`Nd)
`
`p—+KOHN
`
`LT
`
`MSFTEX. 1006
`Page 13 of 35
`
`NOLVONdd¥
`
`WVU90Ud
`
`9
`
`MSFT EX. 1006
`Page 13 of 35
`
`
`

`

`U.S. Patent
`
`Jul. 25, 2000
`
`Sheet 13 of 20
`
`6,094,197
`
`BEGIN
`
`
`
`
`
`DETERMINE
`EVENT TYPE
`
`
`
`D
`
`
`
`FIG. 2t
`
`MSFT EX. 1006
`Page 14 of 35
`
`MSFT EX. 1006
`Page 14 of 35
`
`

`

`U.S. Patent
`
`Jul. 25, 2000
`
`Sheet 14 of 20
`
`6,094,197
`
`DETERMINE
`KEY
`PRESSED
`
`HIGHLIGHT
`THAT KEY
`
` SET MODE
`
`TO MARK
`MODE
`
`SAVE PEN
`LOCATION AND
`TIME IN
`POINT LIST
`
`BEGIN
`
`
`
`
`
`
`
`
`RETURN
`
`FIG.
`
`22.
`
`MSFT EX. 1006
`Page 15 of 35
`
`MSFT EX. 1006
`Page 15 of 35
`
`

`

`U.S. Patent
`
`Jul. 25, 2000
`
`Sheet 15 of 20
`
`6,094,197
`
`BEGIN
`MARK MODE
`
`MENU MODE
`
`
`
`DETERMINE MENU
`SAVE
`
`ITEM BEING
`PEN LOCATION
`
`
`SELECTED BASED
`AND TIME
`
`
`
`ON PEN LOCATION
`
`
`IN POINTLIST
`
`
`
`
`
`ORAW INK TRAIL
`
`STORE IN
`FROM PREVIOUS
`
`PEN LOCATION
`“SELECTED ITEM"
`
`TO CURRENT
`OF CURRENT MENU
`
`PEN LOCATION
`
`
`
`
`
`
`
`RETURN
`
`FIG. 23.
`
`MSFT EX. 1006
`Page 16 of 35
`
`MSFT EX. 1006
`Page 16 of 35
`
`

`

`U.S. Patent
`
`Jul. 25, 2000
`
`Sheet 16 of 20
`
`6,094,197
`
`MARK MODE
`
`
`
`
`
` SET MODE
`TO MENU
`MODE DOES
`
`
`ITEM HAVE A
`SUBMENU?
`
`
`
`REPLACE THE
`MARK WITH MENUS BY
`
`
`DETERMINING ITEMS
`UNDISPLAY THE
`
`SELECTED BY MARK
`CURRENT MENU
`
`
`
`
`DISPLAY SUBMENU
`AND MAKE IT THE
`CURRENT MENU
`
`
`
` RETURN
`
`FIG. 24
`
`MSFT EX. 1006
`Page 17 of 35
`
`MSFT EX. 1006
`Page 17 of 35
`
`

`

`U.S. Patent
`
`Jul. 25, 2000
`
`Sheet 17 of 20
`
`6,094,197
`
`MENU MODE
`
` ERASE
`ALL MENUS
`
` DETERMINE MENU
`
`
`ITEMS SELECTED
`BY MARK
`
`BEGIN
`MARK MODE
`
`
`
`
`TRAVERSE THE MM
`
`HIERARCHY TO DETERMINE
`WHICH ITEMS/ MODIFIERS
`WERE CHOSEN
`
`
`
`
`
`
`
`
`
`
`
`GENERATE CHARACTER
`AND PLACE INTO BUFFER
`
`UNHIGHLIGHT THE
`PRESSED KEY
`
`RETURN
`
`FIG, 25.
`
`MSFT EX. 1006
`Page 18 of 35
`
`MSFT EX. 1006
`Page 18 of 35
`
`

`

`U.S. Patent
`
`Jul. 25, 2000
`
`Sheet 18 of 20
`
`6,094,197
`
`IS MARK LESS THAN
`5 MM LONG?
`
`NO MODIFIER
`WAS SELECTED
`
`RETURN
`
`GET ARTICULATIONPOINTS
`
`ARTICULATION POINTS TO MENU ITEMS
`
`GENERATE MODIFIERS ACCORDING
`TO MENU SELECTIONS
`
`FIG. 26.
`
`MSFT EX. 1006
`Page 19 of 35
`
`MSFT EX. 1006
`Page 19 of 35
`
`

`

`U.S. Patent
`
`Jul. 25, 2000
`
`Sheet 19 of 20
`
`6,094,197
`
`TIME BETWEEN POINTS IS GREATER THAN .5 SECONDS)
`
`GET THE POINTS IN THE INPUT POINTLIST WHERE THE USER
`PAUSED WHILE DRAWING (POINTS WHERE THE ELAPSED
`
`FA
`
`ARE
`
`THERE AT LEAST
`2 PAUSE POINTS
`
`REMOVE INACCURACIES IN THE POINTLIST DUE
`TO THE PENS FIRST TOUCHING THE DISPLAY
`
`TAKE PAUSE POINTS TO
`BE ARTICULATION POINTS.
`ADD TO OUTPUT POINTLIST
`
`
`
`
`
`
`
`
`
`REMOVE SMALL ANGLE CHANGES IN THE
`POINTLIST DUE TO PEN JITTER
`
`
`THE LOCATION OF THE
`FIRST AND LAST
`ANY
`
` POINT & LAST POINT OF
`
`POINTS OF THE MARK
`SIGNIFICANT
`
`
`THE INPUT POINTLIST
`AND THE LARGEST ANGLE
`ANGLE CHANGES
`
`
`DEFINE A SINGLE MENU
`CHANGE(S) DEFINE THE
`FOUND?
`
`
`
`SELECTION. ADD THEM
`
`ARTICULATION POINTS.
`TO THE eT POINT-
`ADD THEM TO THE OUT-
`
`PUT POINTLIST.
`
`
`
`REMOVE INACCURACIES IN THE POINTLIST DUE
`TO THE PEN'S LIFTING FROM THE DISPLAY
`
`REMOVE SNALL ANGLE CHANGES IN THE
`POINTLIST DUE TO PEN JITTER
`
`FE
`
`ADD FINISHING POINT TO
`OUTPUT POINTLIST
`
`FM
`
`
`
`
`RETURN
`
`FIG. 27.
`
`MSFT EX. 1006
`Page 20 of 35
`
`MSFT EX. 1006
`Page 20 of 35
`
`

`

`U.S. Patent
`
`Jul. 25, 2000
`
`Sheet 20 of 20
`
`6,094,197
`
`MAKE THE TOP MM IN
`THE HIERARCHY THE
`CURRENT MENU
`
`
`
`
`
`
`
`
`FOR EACH
`
`
`ARTICULATION
`
`POINT EXCEPT
`THE LAST ONE:
`
`
`
`
`
`
`
`
`DETERMINE THE ITEM THAT WOULD BE
`IF THE CURRENT MENU WERE CENTERED
`ON THE ARTICULATION POINT AND THE
`PEN TIP WERE THE LOCATION OF THE
`
`NEXT ARTICULATION POINT.
`
`
`
`STORE THIS AS THE SELECTED
`ITEM FOR THE CURRENT MENU
`
`
`CONTINUE LOOPING- GO
`PROCESS NEXT
`
`
`ARTICULATION POINT
`
`
`DONE --LOOP
`TERMINATED
`
`IMMEDIATELY
`
`
`YES
`GG
`
`
`
`
`MAKE THE SELECTED ITEM
`SUBMENU THE CURRENT MENU
`
`GH
`
`DROP OUT OF LOOP
`
`RETURN
`
`FIG. 28.
`
`MSFT EX. 1006
`Page 21 of 35
`
`MSFT EX. 1006
`Page 21 of 35
`
`

`

`6,094,197
`
`1
`GRAPHICAL KEYBOARD
`
`This is a continuation Ser. No. 08/171,288,filed Dec. 21,
`1993, now abandoned.
`A portion of the disclosure of this patent document
`contains material which is subject to copyright protection.
`The copyright owners have no objection to the facsimile
`reproduction, by anyone, of the patent document or the
`patent disclosure, as it appears in the Patent and Trademark
`Office patent
`file or records, but otherwise reserve all
`copyright rights whatsoever.
`SOFTWARE
`
`10
`
`The application recites provides a source code software
`program for implementation of an embodiment of the
`method of the invention on a digital computer. The source
`code includesa single file written in the HyperCard scripting
`language. The code mins under the HyperCard program
`(version 2.0 or higher) for Apple Macintosh computers
`(Macintosh II or better hardware running System 7.0 or
`higher operating software), well knownto those of skill in
`the art.
`
`is
`The source code represents unpublished work that
`Copyright ©1993 Xerox Corporation. All rights reserved.
`Copyright protection claimed includesall forms and matters
`of copyrightable material and information now allowed by
`statutory or judicial
`law or hereafter granted,
`including
`without limitation, material generated from the software
`programs which are displayed on the screen such as icons,
`serecn display looks, ctc.
`Although a particular software implementation is
`provided, it will be apparent to those of skill in the art that
`a wide variety of programming languages and hardware
`configurations can readily be used based onthis disclosure
`without departing from the scope of the invention.
`BACKGROUND OF THE INVENTION
`
`The present invention relates to graphical user interfaces
`and more particularly to graphical user interfaces for pen-
`based computing.
`Pen-based computing is becoming increasingly popular.
`The user of a pen-based computer inputs commands and
`data into the computer using a pen or stylus, typically by
`printing or writing characters with the pen or stylus on the
`computer’s display screen in a manneranalogousto printing
`or writing with a pen on ordinary paper. Certain commands
`can be entered as pen strokes (gestures); for example, a word
`may be deleted by crossing it out, or a file may be opened
`by writing an “o” on its representative icon. Examples of
`pen-based computing are found in systems ranging from
`palm-sized personal digital assistants,
`through slate-sized
`machines such as notepad computers, and up to wall-sized
`interactive displays. Also, pen-based interfaces can some-
`times be added or retrofitted to computer systems not
`originally designed to be pen-based, such as certain personal
`computer systems.
`The usefulness of pen-based computingis limited because
`of the limited capabilities of character recognition software.
`In many pen-bascd systems, the user cannot write in cursive
`handwriting, but must print. This slows down input.
`Furthermore, the system hasdifficulty in recognizing printed
`characters. Consequently, the user must print carefully (viz.,
`slowly) and often must print in special windows and within
`designated boxes, and the system can require special train-
`ing for each individual user. Even so, character recognition
`performance can berelatively poor. In addition, pen-based
`
`30
`
`40
`
`50
`
`60
`
`2
`systems that use character recognition do not adapt well to
`run on conventional computer workstations equipped with a
`mouse, for example.
`Because of the limitations of the character-recognition
`approach to pen-based input, an alternative approach is
`sometimes used. In this approach, a graphical keyboard,
`visually analogous to the conventional mechanical
`(typewriter) keyboard commonly used for input in non-pen-
`based computer systems, appears on the computer display.
`The user enters text by using the stylus to tap on desired keys
`of the graphical keyboard.
`The graphical-keyboard approach offers certain advan-
`tages over the character-recognition approach. Entering data
`through a graphical keyboard can be faster than printing
`(especially with numeric keypads), can be less susceptible to
`errors when properly implemented(i.e., when the graphical
`keys are not too small), and does not require the system to
`be specially trained for each individual user. Moreover, the
`approach is simple to implementand integrate into existing
`software applications.
`Unfortunately, known graphical keyboards suffer from
`numerous drawbacks. Although the graphical keyboard is
`visually analogous to a mechanical keyboard,it is function-
`ally far more limited. With a graphical keyboard, a user
`cannot “touch type” using all fingers of both hands, but
`instead must “hunt and peck” with the pen or stylus, as if
`typing with one finger on a mechanical keyboard. Typing
`speed on the graphical keyboard is constrained by the time
`required for the user to move the finger/stylus from key to
`key on the keyboard. Also, the user cannot tap multiple keys
`simultaneously. In particular, the user cannot tap a modifier
`key, such as the “shift,” “ctrl,” “option,” or “alt” key, at the
`same timeas an ordinary key, such as a letter or number key,
`but instead mustfirst tap the modifier key and then tap the
`letter or numberkey. In some systems the user must tap the
`modifier key once to set
`the modifier,
`tap the letter or
`number key, and then tap the modifier key again to release
`the modifier (for example, tap “shift,” “a,” “shift” to input an
`uppercase “A”. This can be especially awkward when mul-
`tiple modifiers must be composed to form a single character
`(for example, a doubly modified character such as ctrl-alt-x
`or a triply modified character such as ctrl-alt-X).
`Another problem with the graphical keyboard approach is
`that it places a heavy visual load on the user. Whereas with
`a mechanical keyboard, the user can memorize commonly
`used sequences of keystrokes and perform them “by feel”
`without looking at the keyboard, with the graphical key-
`board the user typically must look at the display before each
`tap of the stylus to ensure that the stylus is correctly placed.
`Furthermore, the graphical keyboard tendsto take up a large
`portion of the screen display, in part because the keyboard
`contains manyorall of the modificr and function keys found
`in its mechanical counterpart.
`the graphical-keyboard
`Because of its drawbacks,
`approach to pen-based input
`is often considered
`cumbersome, and is used in pen-bascd computing systems
`only as a fallback when character recognition fails or is
`insufficiently reliable. The character-recognition and
`graphical-keyboard approaches to pen-based computing are
`typically seen as mutually exclusive alternatives, not as
`complements.
`In particular,
`it has not been possible to
`combine the intuitive ease of use promised (if not always
`afforded) by the character-recognition approach with the
`speed, robustness and ease of implementation offered by the
`graphical keyboard approach.
`SUMMARYOF THE INVENTION
`
`The present invention is a graphical keyboard that permits
`the user to do more than simple key-tapping. It lets the
`
`MSFT EX. 1006
`Page 22 of 35
`
`MSFT EX. 1006
`Page 22 of 35
`
`

`

`6,094,197
`
`10
`
`4
`3
`FIG. 17 is a series of views showing how an eight-way
`expert user proceed by “feel,” and prompts the novice user
`marking menu can be used with the graphical keyboard of
`when and as necessary. The graphical keyboard benefits
`the invention;
`from the expressive powerand intuitive ease of use associ-
`ated with pen strokes and gestures, yet does not require
`FIG. 18 illustrates a pen-based system suitable for imple-
`complex character-recognition software.
`menting the invention;
`The graphical keyboard of the invention responds differ-
`FIG. 19 illustrates an alternative pen-based system in
`ently to different kinds of pen strokes. For example, lower-
`which the display screen and input sensor are decoupled;
`case “a” is entered by tapping a stylus on the “a” key of the
`FIG. 19A illustrates an input sensor with an affixed
`graphical keyboard. Uppercase “A”is entered by an upward
`keyboard image;
`stroke initiated over the “a” key. Likewise, by stroking in
`FIG. 20 illustrates a mouse-based system suitable for
`other directions, a user can express other modifiers to the
`implementing the invention;
`basic character, such as “control”, “alt”, “command,” etc.
`FIG. 21 is a flowchart of steps for responding to an event
`Also, strokes in distinct directions can be used to express
`according to an embodiment of the invention that incorpo-
`space, backspace, delete, and return characters, for example.
`rates marking menus;
`Multiple strokes can be applied in sequence to a single key
`to express multiple modifiers. Visual feedback of pen strokes
`FIG. 22 is a flowchart of steps for processing a PenDown
`event;
`can be provided in the form of marks made by the pen.
`Pop-up menus can be used to provide help with and to
`FIG. 23 is a flowchart of steps for processing a PenDrag
`facilitate memorization of command strokes and_stroke
`event;
`sequences.
`TIG. 24 is a flowchart of steps for processing a PenStill
`invention provides an
`the present
`In another aspect,
`event;
`improved method for entering characters into a pen-based
`FIG. 25 is a flowchart of steps for processing a PenUp
`computer using a graphical keyboard.
`event;
`A further understanding of the nature and advantages of
`FIG. 26 is a flowchart of steps for determining the
`the invention will become apparent by reference to the
`modifiers corresponding to a mark;
`remaining portions of the specification and drawings.
`FIG. 27 is a flowchart of steps for determining points of
`BRIE DESCRIPTION OF THE DRAWINGS
`articulation along a mark; and
`FIG. 28 is a flowchart of steps for matching points of
`articulation along a mark with centers of corresponding
`menus.
`
`30
`
`FIG. 1 illustrates a sample graphical keyboard;
`FIG. 2 schematically depicts how a lowercase character
`can be entered by tapping a key of the graphical keyboard of
`the invention;
`FIG. 3 schematically depicts how a uppercase character
`can be entered by an upwardstroke on a keyof the graphical
`keyboard of the invention;
`The present invention provides a system and method for
`FIG. 4 schematically depicts how each modifier can be
`an improvedgraphical keyboard that respondsdifferently to
`distinguished by the direction of its particular stroke;
`different kinds of pen strokes. By applying appropriate
`FIG. 5 schematically depicts an example of how modifiers
`strokes, the user can easily type characters that on mechani-
`are combined;
`cal keyboards are typed bypressing multiple keys simulta-
`FIG. 6 schematically depicts an example of how certain
`neously. Also, the user can type certain frequently used keys,
`frequently used keys can be represented as diagonalstrokes;
`such as space, backspace, delete, and return, from anywhere
`FIG. 7 is a series of views in which a user enters a
`on the keyboard. Using pen strokes makes typing easier and
`lowercase character by tapping a key of the graphical
`faster, reduces the visual load on the user, and makesit
`keyboard;
`possible for the keyboard to take up less space in the screen
`FIG. 8 is a series of views in which a user enters an
`display.
`uppercase character by making an upward stroke onakey of
`the graphical keyboard;
`2. Examples
`FIG. 9 is a series of views in which a user enters a control
`character by making a rightward stroke on a key of the
`graphical keyboard;
`FIG. 10 is a series of views in which a user enters a control
`uppercase character by making a compound rightward-
`upward stroke on a key of the graphical keyboard;
`TIG. 11 depicts the two modes of selecting from a
`marking menu;
`FIG. 12 is example of a radial menu hierarchy and the
`marks that select from it;
`FIG. 13 is an example of mark confirmation;
`FIG. 14 schematically depicts a pop-up menurevealing a
`stroke vocabulary of eight stroke directions;
`TIG. 15 is a series of views in which a user is presented
`with and selects from a marking menu;
`FIG. 16 is a series of views showing an example of
`marking menus used to combine modifiers in the context of
`the present invention;
`
`To understand the invention, it is helpful to begin with
`some illustrative examples of its use, as depicted in FIGS.
`1-10. A displayed image of a sample graphical keyboard
`according to the invention is shown in FIG. 1. Tapping on
`any of the keys with the pen or stylus has the sameeffect as
`pushing the analogous key on a mechanical keyboard. Thus
`to enter a lowercase character, a digit, or another unmodified
`character such as a period or comma,the uscr simplytaps the
`desired key.
`To enter an uppercase character, rather than tapping, the
`user makes an upwardstroke, initiated on the desired key.
`lor example, as illustrated in FIG. 2, tapping on the “a” key
`results in “a.” An upward stroke on “a”results in “A” (FIG.
`3).
`
`DESCRIPTION OF SPECIFIC EMBODIMENTS
`
`1. Introduction
`
`40
`
`50
`
`60
`
`‘The user accesses other modifier keys, such as control,
`alternate, option, or command, in a similar manner. Each
`modifier is distinguished by the direction of its particular
`stroke. An example mapping is shown in I'IG. 4. According
`
`MSFT EX. 1006
`Page 23 of 35
`
`MSFT EX. 1006
`Page 23 of 35
`
`

`

`6,094,197
`
`5
`to the mapping, of FIG. 4, a stroke to the right indicates the
`control (“ctrl”) modifier. Thus a rightward pen stroke initi-
`ated over the character “c” causes the graphical keyboard to
`output the character “ctrl-c.” A stroke to the left indicates the
`option modifier, and a downward stroke indicates the com-
`mand modifier.
`
`Modifiers can easily be combined. For example, once
`again applying the mapping of FIG. 4,a stroke to the right,
`starting on the character “c,” followed by an upward stroke
`expresses “ctrl-C,” as distinct from “ctrl-c.” This is illus-
`trated in FIG. 5.
`Additional stroke directions can be used to substitute for
`certain frequently used keys. For example, pen strokes in the
`diagonal directions can be used to express space, backspace,
`delete, and return. A possible layout of these commands is
`shown in FIG. 6. This class of strokes is distinct from the
`modificr strokes discussed above in connection with FIGS.
`3, 4, and 5,
`in that the meaning of these strokes is not
`affected by where the stroke is made. Thus a diagonal stroke
`up and to the left is interpreted in the mapping of FIG. 6 as
`a backspace regardless of whetherit is made onthe “aa” key,
`the “c” key, or any other key of the graphical keyboard.
`Visual fecdback canassist the uscr in applying pen strokes
`to the graphical keyboard. As the user moves the pen across
`the screen, a line appears on the screen to show where the
`pen has been, as though the pen contained ink that lefi a
`mark on the screen. Once the pen stroke is recognized, the
`character is entered and the “ink trail” disappears. This is
`illustrated, for example, in FIG. 8.
`Each of FIGS. 7-10 is a series of views showing how a
`user enters input into the graphical keyboard of the inven-
`tion. The series of two views of FIG. 7 showsa user entering
`a lowercase character using the graphical keyboard of the
`invention. The user does this by tapping the desired key of
`the graphical keyboard with the pen or stylus. More
`preciscly, the uscr taps the pen against the surface of the
`screen on which the keyboard image of the graphical key-
`board is displayed, with the point of the pen striking the
`screen surface in proximity to the location of the desired key
`within the keyboard image.
`the user places the pen
`In the first view of FIG. 7,
`momentarily on the key correspondingto the letter or other
`character lo be entered (an “a” in this example). The key is
`highlighted in the display. ‘The character butter, which is
`represented by the small rectangle next to the view of the
`keyboard, pen, and user, is empty at this stage. In the second
`view, the user removes the pen from the graphical keyboard,
`thereby completing the tapping motion. The graphical key-
`board producesthe character “a” and sendsit to the character
`buffer. The highlighting disappears from the display.
`The series of three views of FIG. 8 shows a user entering
`an uppercase character using the graphical keyboard of the
`invention. The user does this by making an upwardstroke or
`mark on the desired key of the graphical keyboard.
`In thefirst vicw of FIG. 8, the user places the pen or stylus
`on the key corresponding to the character to be entered (an
`“A” in this example). The key is highlighted in the display.
`The character buffer is empty.
`In the sccond vicw of FIG. 8, the uscr moves the pen
`“upward’—that
`is,
`toward the topmost
`row of the
`keyboard—while continuing to hold the pen against
`the
`keyboard. The user maintains contact between the pen point
`and the surtace of the screen on which the keyboard image
`is displayed substantially throughout the upward motion.
`This is called dragging the pen. The display showsa line or
`mark (an ink trail) where the user has dragged the pen. The
`
`10
`
`30
`
`40
`
`50
`
`60
`
`6
`character buffer remains empty at this stage, because the
`character to be entered has not yet been recognized. The key
`remains highlighted in the display.
`In the third view of FIG. 8, the uscr removesthe pen from
`the keyboard surface, completing the upward stroke. The
`line or mark disappears, as does the highlighting, and the
`character produced by the graphical keyboard in response to
`the upward stroke appears in the character buffer.
`The series of three views of FIG. 9 showsa user entering
`a control character using the graphical keyboard of the
`invention. A “control character” is produced on a
`conventional, mechanical keyboard by holding down the
`modifier key marked “control” or “ctrl” while pressing a
`character key. Pressing the “ctrl” key while striking the key
`for the letter “a,” for example, produces the character called
`“control-a,” which can also be written as “ctrl-a” or “*a.” On
`the graphical kcyboard of the invention, the uscr can enter
`a control character by making a rightward stroke or mark on
`the desired key of the graphical keyboard.
`In the first view of FIG. 9, the user places the pen or stylus
`on the key corresponding to the character to be entered (the
`key “a” correspondingto the character “‘a” in this example).
`The key is highlighted in display. The character buffer is
`empty.
`In the second view of FIG. 9, the user drags the pento the
`right: The user moves the pen to the right while continuing
`to hold the pen against the keyboard, and maintains contact
`between the pen point and the surface of the screen on which
`the keyboard imageis displayed substantially throughout the
`rightward motion. The display shows a line or mark where
`the user has dragged the pen. The key remains highlighted.
`The character buffer remains empty atthis stage, because the
`character to be entered has not yet been recognized.
`In the third view of FIG. 9, the user removes the pen from
`the keyboard surface, completing the rightward stroke. The
`line or mark disappears, as does the highlighting of the key,
`and the character produced by the graphical keyboard in
`response to the rightward stroke appears in the character
`buffer.
`
`FIG. 10 showsthat modifiers can be combined by making
`compoundstrokes that combine strokes of different direc-
`tions. The series of three views of FIG. 10 shows a user
`entering a control uppercase character using the graphical
`keyboard of the invention. A “control uppercase character”
`is produced on a conventional, mechanical keyboard by
`holding down the “shift” and “ctrl” modifier keys simulta-
`neously while pressing a character key. Pressing the “shift”
`and “ctrl” keys while striking the key for the letter “a,” for
`example, produces the character called “control-shift-a,”
`which can also be written as “ctrl-A” or ““A.” On the
`graphical keyboard of the invention, the user can enter a
`control uppercase character by making compound
`rightward-upward stroke or mark on the desired key of the
`graphical keyboard.
`In the first view of FIG. 10, the uscr places the pen or
`stylus on the key corresponding to the character to be
`entered (the key “a” corresponding to the character “*A” in
`this example). The key is highlighted in the display. The
`character butter is empty.
`In the second view of FIG. 10, the user drags the pen a
`distance to the right and then a distance “upward”—thatis,
`toward the topmostrowof the keyboard. The user maintains
`contact between the pen point and the surface of the screen
`on which the keyboard image is displayed substantially
`throughout the rightward-and-then-upward motion. The dis-
`play shows a line or mark where the user has dragged the
`
`MSFT EX. 1006
`Page 24 of 35
`
`MSFT EX. 1006
`Page 24 of 35
`
`

`

`6,094,197
`
`7
`pen. The key remains highlighted in the display. The char-
`acter buffer remains empty at this stage, because the char-
`acter to be entered has not yet been recognized.
`In the third vicw of FIG. 10, the uscr removes the pen
`from the keyboard surface, completing the compound
`stroke. The line or mark disappears, as does the key
`highlighting, and the character produced by the graphical
`keyboard in response to the stroke appears in the character
`buffer.
`
`10
`
`30
`
`40
`
`Although the example of FIG. 10 shows the rightward
`portion of the compound stroke preceding the upward
`portion, the same “*A” character is produced if the upward
`portion of the stroke is performed before the rightward
`portion. Other kinds of modifiers can be combined by
`making compound marksin other dircctions. For example,
`if a leftward stroke indicates the “alt” modifier,
`then a
`leftward-upwardor upward-leftward compound stroke com-
`bines the “alt” and “shift” (uppercase) modifiers, and a
`leftward-rightward or rightward-lettward compoundstroke
`combines the “ctrl” and “alt” modifiers.
`
`8
`is true that creating a mark does involve a
`Although it
`physical gesture, the thing that is interpreted by the system
`is the mark itself. For example, the “X” mark requires a
`completely different physical gesture if performed with a
`pen instead of a mouse. Gesture is an important aspect of
`marking because some marks may require awkward physical
`gestures with the input device. However, the two terms can
`be distinguished. The term mark is more appropriate for
`pen-based computer systems or applications that emulate
`paper and pen, and will be used hereinbelow.
`3.2 Why Use Marks?
`Conventional human-computer interfaces are asymmetric
`in terms of input and output capabilities. There are number
`of computer output modes: visual, audio and tactile. Most
`computers extensively utilize the visual mode; high-
`resolution images that use thousands of colors of can be
`displayed quickly and in meaningful ways to a user. In
`contrast, a computer’s ability to sense user input is limited.
`Humanshave a wide range of communication skills such as
`speech and touch, but most computers sense only a small
`subsct of these. For example, keyboards only scnsc finger
`3. Marking Menus
`presses (but not pressure) and mice only sense very simple
`arm or wrist movement