[19]
`United States Patent
`6,094,197
`Patent Number:
`[11
`
`Buxton et al.
`[45 Date of Patent:
`Jul. 25, 2000
`
`USOO6094197A
`
`
`
`[54] GRAPHICAL KEYBOARD
`
`[75]
`
`Inventors: William A. S. Buxton, Toronto,
`Canada; Gordon P. Kurtenbach,
`Mountain View, Calif.
`
`[73] Assignee: Xerox Corporation, Stamford, Conn.
`
`[21] Appl. No.2 08/443,259
`
`[22]
`
`Filed:
`
`May 17,1995
`
`Related US. Application Data
`
`[63] Continuation of application No. 08/171,288, Dec. 21, 1993,
`abandoned.
`
`Int. Cl.7 ..................................................... .. G09G 5/00
`[51]
`[52] US. Cl.
`........................ .. 345/358; 345/168; 345/179;
`341/22
`[58] Field of Search ................................... .. 345/156, 157,
`345/168, 169, 170, 172, 173, 174, 175,
`176, 179, 180, 181, 182, 183; 341/22, 23,
`170/18, 19; 395/326, 335, 347, 350, 351,
`352; 382/188, 189
`
`[56]
`
`References Cited
`US. PATENT DOCUMENTS
`
`............................ .. 345/173
`2/1988 Auer et a1.
`4,725,694
`9/1991 Peters ........ ..
`.. 345/157
`5,050,105
`178/18
`9/1992 Martin et al.
`5,148,155
`
`.. 345/172
`11/1993 Jones ...... ..
`5,258,748
`
`1/1994 Lamb ........ ..
`.. 345/173
`5,276,794
`
`
`9/1994 Agulnick et a .
`. 345/156
`5,347,295
`...................... .. 345/179
`2/1997 Gough et al.
`5,603,053
`FOREIGN PATENT DOCUMENTS
`
`0 603 669A1
`0144817
`607144817
`0011144
`89/11144
`92/09944
`
`G06F 3/033
`
`European Pat. Off.
`12/1993
`Japan .
`7/1985
`Japan .
`7/1985
`11/1989 WIPO .................................... .. 178/18
`11/1989 WIPO .
`..... .. 178/18
`
`6/1992 WIPO ........................... .. G06F 3/033
`OTHER PUBLICATIONS
`
`Knodt, Ruediger W. “Smart Virtual Keyboard System Suit-
`able for Mouse or Stylus Entry,” Xerox Disclosure Journal,
`vol. 18(3):245—245, May/Jun. 1993.
`
`Kurtenaach et a1. “Issues in Combining Marking and Direct
`Manipulation Techniques,” User
`Interface Software and
`Technology (UIST), Hilton Head, SC (Nov. 11—13, 1991),
`137—144.
`
`Kurtenaach, Gordon P. “The Design and Evaluation of
`Marking Menus,” Doctrate Thesis, Graduate Department of
`Computer Science, University of Toronto, 1993.
`Kurtenaach, Gordon “Making Marks Self—Revealing,” SIG-
`CHI Bulletin, vol. 23(4):60—61 (Abstract) (Oct. 1991).
`Kurtenjach et al. “An Empirical Evaulation of Some Articu-
`Iatory and Cognitive Aspects of Marking Menus,” Human—
`Computcr Intcraction 821—23 (1993).
`Robertson et al., “Buttons as First Class Objects on an X
`Desktop,” user Interface Software and Technology (UIST),
`Hilton Head, SC (Nov. 11—13, 1991), 35—44.
`
`Primary Examiner—Steven Saras
`Attorney, Agent, or Firm—Townsend & Townsend Crew
`LIP
`
`[57]
`
`ABSTRACT
`
`A system and method for a graphical keyboard that benefits
`from the expressive power and intuitive ease of use associ-
`ated with pen strokes and gestures, yet does not require
`complex character—recognition software. The graphical key—
`board responds differently to different kinds of pen strokes.
`For example, lowercase “a” is entered by tapping a stylus on
`the “a” key of the graphical keyboard. Uppercase “A” is
`entered by an upward stroke initiated over the “a” key.
`Likcwisc, by stroking in other directions, a user can express
`other modifiers to the basic character, such as “control”,
`“alt”, “command,” etc. Also, strokes in distinct directions
`can be used to express space, backspace, delete, and return
`characters, for example. Multiple strokes can be applied in
`sequence to a single key to express multiple modifiers.
`Visual feedback of pen strokes can be provided in the form
`of marks made by the pen. Pop-up mcnus can be used to
`provide help with and to facilitate memorization of com-
`mand strokes and stroke sequences. The graphical keyboard
`thus lets the expert user proceed by “feel,” and prompts the
`novice user when and as necessary.
`
`18 Claims, 20 Drawing Sheets
`
` K
`Bflflfllfllflfi
`
`[mm-m.
`mummy.
`EEEEIEJIHII
`
`
`
`MSFT EX. 1006
`
`Page 1 of 35
`
`MSFT EX. 1006
`Page 1 of 35
`
`

`

`US. Patent
`
`Jul. 25, 2000
`
`Sheet 1 0f 20
`
`6,094,197
`
`
`
`STROKE\@ => "All
`
`
`
`FIG: 3.
`
`MSFT EX. 1006
`
`Page 2 of 35
`
`MSFT EX. 1006
`Page 2 of 35
`
`

`

`US. Patent
`
`Jul. 25, 2000
`
`Sheet 2 0f 20
`
`6,094,197
`
`UPPER CASE
`
`OPTION “"‘
`
`“""CONTROL
`
`l
`
`t
`
`COMMAND
`
`
`
`- =>"CONTROL c"
`
`I
`---I => "CONTROL c“
`
`
`
`
`
`MSFT EX. 1006
`
`Page 3 of 35
`
`MSFT EX. 1006
`Page 3 of 35
`
`

`

`US. Patent
`
`Jul. 25, 2000
`
`Sheet 3 0f 20
`
`6,094,197
`
`-‘
`
`IBZEEEEWE
`awnmanm V
`flflflflflh'
`amnmug
`
`
`
`IEEEHEEZE
`
`Hflflmflflflflm
`
`.
`
`“
`
`E??- \
`
`_
`
`'
`
`lflzalqfiflfi
`
`Hflflmflkfifii
`
`MSFT EX. 1006
`
`Page 4 of 35
`
`MSFT EX. 1006
`Page 4 of 35
`
`

`

`US. Patent
`
`Jul. 25, 2000
`
`Sheet 4 0f 20
`
`6,094,197
`
`\ —
`
`B
`lflfl Elli
`IIIEEJEIEEE
`mmanm mammal: ll.
`ragaqnmmn
`mgr-Juana
`
`E "r‘ Blatant-mu
`
`F/GI
`
`IO.
`
`MSFT EX. 1006
`
`Page 5 of 35
`
`MSFT EX. 1006
`Page 5 of 35
`
`

`

`US. Patent
`
`Jul. 25, 2000
`
`Sheet 5 0f 20
`
`6,094,197
`
`LIPBOARD
`
`
` SELECTION USING
`
`SELECTION USING
`MENU MODE
`MARK MODE
`
`
`
`
`FIG:
`
`/I.
`
`/
`I
`
`2\
`
`/3
`
`\
`4
`
`egg
`egg
`\an
`/\
`Q3
`0
`9&3 Q;
`
`a
`'5 f .4
`2%
`2:0 2} 2D
`I/ 4 K _/
`
`3d
`
`3:0
`
`3b
`
`3c
`
`K
`
`Y
`
`I\ ‘\
`
`4:0
`
`4:b
`
`4c
`
`4d
`
`MENU HIERARCHY
`
`MARK SET
`
`
`
`FIG?
`
`l2.
`
`MSFT EX. 1006
`
`Page 6 of 35
`
`MSFT EX. 1006
`Page 6 of 35
`
`

`

`US. Patent
`
`Jul. 25, 2000
`
`Sheet 6 0f 20
`
`6,094,197
`
`(I)
`
`
`
`FIG /3.
`
`UPPER CASE
`
`BACKSPACE\
`
`0
`
`MSFT EX. 1006
`
`Page 7 of 35
`
`MSFT EX. 1006
`Page 7 of 35
`
`

`

`US. Patent
`
`Jul. 25, 2000
`
`Sheet 7 0f 20
`
`6,094,197
`
`I' ..
`.
`‘ 5
`7 '
`,
`I
`:
`nsysg-m'fig' mm 33‘5"":
`- - I - a
`‘
`-
`y
`Egmmmmlgn 012mm lei-1mg
`«121100
`101ml:
`-
`
`(010W:
`
`FIG /5.
`
`llltu'ul
`
`l m t. n g
`
`Hw-9‘‘qgumm
`
`
`
`MSFT EX. 1006
`
`Page 8 of 35
`
`MSFT EX. 1006
`Page 8 of 35
`
`

`

`US. Patent
`
`Jul. 25, 2000
`
`Sheet 8 0f 20
`
`6,094,197
`
`‘55)“; -
`(021010 EWL‘HK‘V'IIHII
`”‘‘ "' "—
`
`Ema- ‘-
`
`WWW
`
`IfiPnnfi
`mum“ “WWII
`W'
`
`MSFT EX. 1006
`
`Page 9 of 35
`
`MSFT EX. 1006
`Page 9 of 35
`
`

`

`US. Patent
`
`Jul. 25, 2000
`
`Sheet 9 0f 20
`
`6,094,197
`
`zoz<u3m¢<
`
`z<zocmm
`
`.3“9k
`
`
`
`m
`
`n
`
`mum:
`
`uuqmzmbz_
`
`all22::
`
`MSFT EX. 1006
`
`Page100f35
`
`MSFT EX. 1006
`Page 10 of 35
`
`
`

`

`US. Patent
`
`Jul. 25, 2000
`
`Sheet 10 0f 20
`
`6,094,197
`
`552.5%
`
`EESE
`
`SE52:Em:
`
`.96?.
`
`MSFT EX. 1006
`
`Page 11 of 35
`
`MSFT EX. 1006
`Page 11 of 35
`
`

`

`US. Patent
`
`Jul. 25, 2000
`
`Sheet 11 0f 20
`
`6,094,197
`
`@QGBQQMBOOOO
`
`FIG I94
`
`MSFT EX. 1006
`
`Page 12 of 35
`
`MSFT EX. 1006
`Page 12 of 35
`
`

`

`US. Patent
`
`Jul. 25, 2000
`
`Sheet 12 0f 20
`
`6,094,197
`
`zoz<u3mm<
`
`=<=womm
`
`mum:
`
`6Nnot
`
`wc<m=w~=_
`
`MSFT EX. 1006
`
`Page13of35
`
`MSFT EX. 1006
`Page 13 of 35
`
`

`

`US. Patent
`
`Jul. 25, 2000
`
`Sheet 13 0f 20
`
`6,094,197
`
`
`
` DETERMINE
`EVENT TYPE
`
`0
`
`PEHDRAG
`
`PENSTILL
`
`
`
`FIG 2!.
`
`MSFT EX. 1006
`
`Page 14 of 35
`
`MSFT EX. 1006
`Page 14 of 35
`
`

`

`
`
`
`
`
`
`
`
`
`
`
`
`
`SAVE PEN
`LOCATION AND
`TIME IN
`POINTLIST
`
`US. Patent
`
`Jul. 25, 2000
`
`Sheet 14 0f 20
`
`6,094,197
`
`BEGIN
`
`AA
`
`DETERMINE
`KEY
`PRESSED
`
`AB
`
`AC
`
`HIGHLIGHT
`THAT KEY
`
` SET MODE
`TO MARK
`MODE
`
`AD
`
`
`
`FIG.
`
`22.
`
`MSFT EX. 1006
`
`Page 15 of 35
`
`MSFT EX. 1006
`Page 15 of 35
`
`

`

`US. Patent
`
`Jul. 25, 2000
`
`Sheet 15 0f 20
`
`6,094,197
`
` MARK MODE
`
`MENU MODE
`
`BD
`
`
`
`DETERMINE MENU
`SAVE
`ITEM BEING
`PEN LOCATION
`
`
`SELECTED BASED
`AND TIME
`
`
`ON PEN LOCATION
`IN POINTLIST
`
`
`
`
`
`
`
`DRAW INK TRAIL
`FROM PREVIOUS
`PEN LOCATION
`T0 CURRENT
`PEN LOCATION
`
`
`
`
`
`STORE IN
`
`
`
`
`BE
`
`
`
`
`OF CURRENT MENU
`
`
`"SELECTED ITEM"
`
`
`
`RETURN
`
`FIG. 23.
`
`MSFT EX. 1006
`
`Page 16 of 35
`
`MSFT EX. 1006
`Page 16 of 35
`
`

`

`US. Patent
`
`Jul. 25,2000
`
`Sheet 16 0f 20
`
`6,094,197
`
` MARK MODE
`
`MENU MODE
`
`
`
`IS PEN
`
`LOCATED IN A MENU
`ITEM?
`
`
`
`
`
`YES
` SET MODE
`
`CE
`T0 MENU
`
`MODE
` DOES
`
`
`ITEM HAVE A
`SUBMENU?
`
`
`
`REPLACE THE
`MARK WITH MENUS BY
`
`DETERMINING ITEMS
`
`
`SELECTED BY MARK
`
`
`
`
` RETURN
`
`UNDISPLAY THE
`CURRENT MENU
`
`DISPLAY SUBMENU
`AND MAKE IT THE
`CURRENT MENU
`
`FIG: 24.
`
`MSFT EX. 1006
`
`Page 17 of 35
`
`MSFT EX. 1006
`Page 17 of 35
`
`

`

`US. Patent
`
`Jul. 25, 2000
`
`Sheet 17 0f 20
`
`6,094,197
`
` MARK MODE
`
`MENU MODE
`
`
`
`DD
`
`
`
` ERASE
`ALL MENUS
`
` DETERMINE MENU
`ITEMS SELECTED
`BY MARK
`
`
`
`
`TRAVERSE THE MN
`
`HIERARCHY TO DETERMINE
`WHICH ITEMS/MODIFIERS
`WERE CHOSEN
`
`
` DC
`
`DF
`
`GENERATE CHARACTER
`AND PLACE INTO BUFFER
`
`UNHIGHLIGHT THE
`
`PRESSED KEY
`
`
`
`
`
`FIG 25.
`
`MSFT EX. 1006
`
`Page 18 of 35
`
`MSFT EX. 1006
`Page 18 of 35
`
`

`

`US. Patent
`
`Jul. 25,2000
`
`Sheet 18 0f 20
`
`6,094,197
`
` IS MARK LESS THAN
`
`5 MM LONG?
`
`N0 NODlFlER
`WAS SELECTED
`
`
`
`
`
`
`
`GET ARTICULATIONPOINTS
`
`ARTICULATION POINTS T0 MENU ITEMS
`
`GENERATE MODIFIERS ACCORDING
`TO MENU SELECTIONS
`
`RETURN
`
`FIG. 26.
`
`MSFT EX. 1006
`
`Page 19 of 35
`
`MSFT EX. 1006
`Page 19 of 35
`
`

`

`US. Patent
`
`Jul. 25, 2000
`
`Sheet 19 0f 20
`
`6,094,197
`
`TIME BETWEEN POINTS IS GREATER THAN .5 SECONDS)
`
`GET THE POINTS IN THE INPUT POINTLIST WHERE THE USER
`PAUSED WHILE DRAWING (POINTS WHERE THE ELAPSED
`
`F8
`FC
`
`
`ADD STARTING POINT T0
`OUTPUT POINTLIST IF IT
`
`
`IS NOT AMONG THE
` 2 PAUSE POINTS
`
`
`PAUSE POINTS
`
`
`TAKE PAUSE POINTS TO
`BE ARTICULATION POINTS.
`ADD TO OUTPUT POINTLIST
`
`
`
`
`REMOVE INACCURACIES IN THE POINTLIST DUE
`TO THE PEN'S LIFTING FROM THE DISPLAY
`
`
`
`REMOVE SMALL ANGLE CHANGES IN THE
`POINTLIST DUE TO PEN JITTER
`
`REMOVE INACCURACIES IN THE POINTUST DUE
`TO THE PEN'S FIRST TOUCHING THE DISPLAY
`
`FD
`
`FE
`
`ADD FINISHING POINT T0
`OUTPUT POINTLIST
`
`F"
`
`
`REMOVE SMALL ANGLE CHANGES IN THE
`POINTLIST DUE TO PEN JITTER
`
`
`
`THE LOCATION OF THE
`MARK IS A STRAIGHT
`
`
`LINE, SO THE FIRST
`FIRST AND LAST
` ANY
`
`
`
`POINT & LAST POINT OF
`SIGNIFICANT
`
`POINTS OF THE MARK
`
`
`THE INPUT POINTLIST
`ANGLE CHANGES
`AND THE LARGESTANGLE
`
`
`DEFINE A SINGLE MENU
`FOUND?
`CHANGEISI DEFINE THE
`
`
`
`SELECTION. ADD THEM
`ARTICULATION POINTS.
`
`
`TO THE DUEL” POINT-
`ADD THEM TO THE OUT-
`
`
`PUT POINTUST.
`
`
`
`
`
`
`
`
`
`
`RETURN
`
`FIG? 27.
`
`MSFT EX. 1006
`
`Page 20 of 35
`
`MSFT EX. 1006
`Page 20 of 35
`
`

`

`US. Patent
`
`Jul. 25, 2000
`
`Sheet 20 0f 20
`
`6,094,197
`
`GA
`
`
`
`MAKE THE TOP MM IN
`THE HIERARCHY THE
`CURRENT MENU
`
`
`
`FOR EACH
`
`ARTICULATION
`
`
`POINT EXCEPT
`THE LAST ONE:
`
`CB
`
`
`
`
`
`CC
`
`
`DETERMINE THE ITEM THAT WOULD BE
`IF THE CURRENT MENU WERE CENTERED
`ON THE ARTICULATION POINT AND THE
`PEN TIP WERE THE LOCATION OF THE
`NEXT ARTICULATION POINT.
`
`
`GE
`
`
`
`
`
`
`
`STORE THIS AS THE SELECTED
`ITEM FOR THE CURRENT MENU
`
`CONTINUE LOOPINC- GO
`PROCESS NEXT
`ARTICULATION POINT
`
`
`
`
`
` DOES
`
`SELECTED ITEM
`
`HAVE A
`
`UBMENU?
`
`NO
`
`
`
`
`
`DONE -- LOOP
`TERM INATED
`
`IMMEDIATELY
`
`
`YES
`
`CG
`
`MAKE THE SELECTED ITEM
`SUDMENU THE CURRENT MENU
`
`CH
`
`DROP OUT OF LOOP
`
`RETURN
`
`FIG. 28.
`
`MSFT EX. 1006
`
`Page 21 of 35
`
`MSFT EX. 1006
`Page 21 of 35
`
`

`

`6,094,197
`
`1
`GRAPHICAL KEYBOARD
`
`This is a continuation Ser. No. 08/171,288, filed Dec. 21,
`1993, now abandoned.
`A portion of the disclosure of this patent document
`contains material which is subject to copyright protection.
`The copyright owners have no objection to the facsimile
`reproduction, by anyone, of the patent document or the
`patent disclosure, as it appears in the Patent and Trademark
`Office patent
`file or records, but otherwise reserve all
`copyright rights whatsoever.
`SOFTWARE
`
`10
`
`The application recites provides a source code software
`program for implementation of an embodiment of the
`method of the invention on a digital computer. The source
`code includes a single file written in the HyperCard scripting
`language. The code runs under the HyperCard program
`(version 2.0 or higher) for Apple Macintosh computers
`(Macintosh II or better hardware running System 7.0 or
`higher operating software), well known to those of skill in
`the art.
`
`is
`The source code represents unpublished work that
`Copyright ©1993 Xerox Corporation. All rights reserved.
`Copyright protection claimed includes all forms and matters
`of copyrightable material and information now allowed by
`statutory or judicial
`law or hereafter granted,
`including
`without limitation, material generated from the software
`programs which are displayed on the screen such as icons,
`screen display looks, etc.
`Although a particular software implementation is
`provided, it will be apparent to those of skill in the art that
`a wide variety of programming languages and hardware
`configurations can readily be used based on this disclosure
`without departing from the scope of the invention.
`BACKGROUND OF THE INVENTION
`
`The present invention relates to graphical user interfaces
`and more particularly to graphical user interfaces for pen-
`based computing.
`Pen—based computing is becoming increasingly popular.
`The user of a pen-based computer inputs commands and
`data into the computer using a pen or stylus, typically by
`printing or writing characters with the pen or stylus on the
`computer’s display screen in a manner analogous to printing
`or writing with a pen on ordinary paper. Certain commands
`can be entered as pen strokes (gestures); for example, a word
`may be deleted by crossing it out, or a file may be opened
`by writing an “0” on its representative icon. Examples of
`pen—based computing are found in systems ranging from
`palm-sized personal digital assistants,
`through slate-sized
`machines such as notepad computers, and up to wall—sized
`interactive displays. Also, pen-based interfaces can some-
`times be added or retrofitted to computer systems not
`originally designed to be pen-based, such as certain personal
`computer systems.
`The usefulness of pen-based computing is limited because
`of the limited capabilities of character recognition software.
`In many pen-based systems, the user cannot write in cursive
`handwriting, but must print. This slows down input.
`Furthermore, the system has difficulty in recognizing printed
`characters. Consequently, the user must print carefully (viz.,
`slowly) and often must print in special windows and within
`designated boxes, and the system can require special train-
`ing for each individual user. Even so, character recognition
`performance can be relatively poor. In addition, pen-based
`
`30
`
`40
`
`50
`
`60
`
`2
`systems that use character recognition do not adapt well to
`run on conventional computer workstations equipped with a
`mouse, for example.
`Because of the limitations of the character-recognition
`approach to pen—based input, an alternative approach is
`sometimes used. In this approach, a graphical keyboard,
`Visually analogous to the conventional mechanical
`(typewriter) keyboard commonly used for input in non-pen-
`based computer systems, appears on the computer display.
`The user enters text by using the stylus to tap on desired keys
`of the graphical keyboard.
`The graphical-keyboard approach offers certain advan-
`tages over the character-recognition approach. Entering data
`through a graphical keyboard can be faster than printing
`(especially with numeric keypads), can be less susceptible to
`errors when properly implemented (i.e., when the graphical
`keys are not too small), and does not require the system to
`be specially trained for each individual user. Moreover, the
`approach is simple to implement and integrate into existing
`software applications.
`Unfortunately, known graphical keyboards sulIer from
`numerous drawbacks. Although the graphical keyboard is
`Visually analogous to a mechanical keyboard, it is function-
`ally far more limited. With a graphical keyboard, a user
`cannot “touch type” using all fingers of both hands, but
`instead must “hunt and peck” with the pen or stylus, as if
`typing with one finger on a mechanical keyboard. Typing
`speed on the graphical keyboard is constrained by the time
`required for the user to move the finger/stylus from key to
`key on the keyboard. Also, the user cannot tap multiple keys
`simultaneously. In particular, the riser cannot tap a modifier
`key, such as the “shift,” “ctrl,” “option,” or “alt” key, at the
`same time as an ordinary key, such as a letter or number key,
`but instead must first tap the modifier key and then tap the
`letter or number key. In some systems the user must tap the
`modifier key once to set
`the modifier,
`tap the letter or
`number key, and then tap the modifier key again to release
`the modifier (for example, tap “shift,” “a,” “shift” to input an
`uppercase “A”. This can be especially awkward when mul-
`tiple modifiers must be composed to form a single character
`(for example, a doubly modified character such as ctrl-alt-x
`or a triply modified character such as ctrl-alt-X).
`Another problem with the graphical keyboard approach is
`that it places a heavy Visual load on the user. Whereas with
`a mechanical keyboard, the user can memorize commonly
`used sequences of keystrokes and perform them “by feel”
`without looking at the keyboard, with the graphical key—
`board the user typically must look at the display before each
`tap of the stylus to ensure that the stylus is correctly placed.
`Furthermore, the graphical keyboard tends to take up a large
`portion of the screen display, in part because the keyboard
`contains many or all of the modifier and function keys found
`in its mechanical counterpart.
`the graphical-keyboard
`Because of its drawbacks,
`approach to pen-based input
`is often considered
`cumbersome, and is used in pen-based computing systems
`only as a fallback when character recognition fails or is
`insufficiently reliable. The character—recognition and
`graphical-keyboard approaches to pen-based computing are
`typically seen as mutually exclusive alternatives, not as
`complements.
`In particular,
`it has not been possible to
`combine the intuitive ease of use promised (if not always
`afforded) by the character-recognition approach with the
`speed, robustness and ease of implementation offered by the
`graphical keyboard approach.
`SUMMARY OF THE INVENTION
`
`The present invention is a graphical keyboard that permits
`the user to do more than simple key-tapping. It lets the
`
`MSFT EX. 1006
`
`Page 22 of 35
`
`MSFT EX. 1006
`Page 22 of 35
`
`

`

`6,094,197
`
`3
`expert user proceed by “feel,” and prompts the novice user
`when and as necessary. The graphical keyboard benefits
`from the expressive power and intuitive ease of use associ-
`ated with pen strokes and gestures, yet does not require
`complex character-recognition software.
`The graphical keyboard of the invention responds differ—
`ently to different kinds of pen strokes. For example, lower-
`case “a” is entered by tapping a stylus on the “a” key of the
`graphical keyboard. Uppercase “A” is entered by an upward
`stroke initiated over the “a” key. Likewise, by stroking in
`other directions, a user can express other modifiers to the
`basic character, such as “control”, “alt”, “command,” etc.
`Also, strokes in distinct directions can be used to express
`space, backspace, delete, and return characters, for example.
`Multiple strokes can be applied in sequence to a single key
`to express multiple modifiers. Visual feedback of pen strokes
`can be provided in the form of marks made by the pen.
`Pop-up menus can be used to provide help with and to
`facilitate memorization of command strokes and stroke
`sequences.
`invention provides an
`the present
`In another aspect,
`improved method for entering characters into a pen-based
`computer using a graphical keyboard.
`A filrther understanding of the nature and advantages of
`the invention will become apparent by reference to the
`remaining portions of the specification and drawings.
`BRIEF DESCRIPTION OF TIIE DRAWINGS
`
`FIG. 1 illustrates a sample graphical keyboard;
`FIG. 2 schematically depicts how a lowercase character
`can be entered by tapping a key of the graphical keyboard of
`thc invention;
`FIG. 3 schematically depicts how a uppercase character
`can be entered by an upward stroke on a key of the graphical
`keyboard of the invention;
`FIG. 4 schematically depicts how each modifier can be
`distinguished by the direction of its particular stroke;
`FIG. 5 schematically depicts an example of how modifiers
`are combined;
`FIG. 6 schematically depicts an example of how certain
`frequently used keys can be represented as diagonal strokes;
`FIG. 7 is a series of Views in which a user enters a
`lowercase character by tapping a key of the graphical
`keyboard;
`FIG. 8 is a series of views in which a user enters an
`uppercase character by making an upward stroke on a key of
`the graphical keyboard;
`FIG. 9 is a series of views in which a user enters a control
`character by making a rightward stroke on a key of the
`graphical keyboard;
`FIG. 10 is a series of Views in which a user enters a control
`uppercase character by making a compound rightward-
`upward stroke on a key of the graphical keyboard;
`FIG. 11 depicts the two modes of selecting from a
`marking menu;
`FIG. 12 is example of a radial menu hierarchy and the
`marks that select from it;
`FIG. 13 is an example of mark confirmation;
`FIG. 14 schematically depicts a pop-up menu revealing a
`stroke vocabulary of eight stroke directions;
`FIG. 15 is a series of Views in which a user is presented
`with and selects from a marking menu;
`FIG. 16 is a series of Views showing an example of
`marking menus used to combine modifiers in the context of
`the present invention;
`
`10
`
`30
`
`40
`
`50
`
`60
`
`4
`FIG. 17 is a series of views showing how an eight-way
`marking menu can be used with the graphical keyboard of
`the invention;
`FIG. 18 illustrates a pen-based system suitable for imple-
`menting the invention;
`FIG. 19 illustrates an alternative pen-based system in
`which the display screen and input sensor are decoupled;
`FIG. 19A illustrates an input sensor with an aflixed
`keyboard image;
`FIG. 20 illustrates a mouse-based system suitable for
`implementing the invention;
`FIG. 21 is a flowchart of steps for responding to an event
`according to an embodiment of the invention that incorpo-
`rates marking menus;
`FIG. 22 is a flowchart of steps for processing a PenDown
`event;
`FIG. 23 is a flowchart of steps for processing a PenDrag
`event;
`FIG. 24 is a flowchart of steps for processing a PenStill
`event;
`FIG. 25 is a flowchart of steps for processing a PenUp
`event;
`FIG. 26 is a flowchart of steps for determining the
`modifiers corresponding to a mark;
`FIG. 27 is a flowchart of steps for determining points of
`articulation along a mark; and
`FIG. 28 is a flowchart of steps for matching points of
`articulation along a mark with centers of corresponding
`menus.
`
`DESCRIPTION OF SPECIFIC EMBODIMENTS
`
`1. Introduction
`
`The present invention provides a system and method for
`an improved graphical keyboard that responds differently to
`different kinds of pen strokes. By applying appropriate
`strokes, the user can easily type characters that on mechani-
`cal keyboards are typed by pressing multiple keys simulta-
`neously. Also, the user can type certain frequently used keys,
`such as space, backspace, delete, and return, from anywhere
`on the keyboard. Using pen strokes makes typing easier and
`faster, reduces the Visual load on the user, and makes it
`possible for the keyboard to take up less space in the screen
`display.
`
`2. Examples
`
`To understand the invention, it is helpful to begin with
`some illustrative examples of its use, as depicted in FIGS.
`1410. A displayed image of a sample graphical keyboard
`according to the invention is shown in FIG. 1. Tapping on
`any of the keys with the pen or stylus has the same effect as
`pushing the analogous key on a mechanical keyboard. Thus
`to enter a lowercase character, a digit, or another unmodified
`charactcr such as a period or comma, thc uscr simply taps thc
`desired key.
`To enter an uppercase character, rather than tapping, the
`user makes an upward stroke, initiated on the desired key.
`For example, as illustrated in FIG. 2, tapping on the “a” key
`results in “a.” An upward stroke on “a” results in “A” (FIG.
`3).
`The user accesses other modifier keys, such as control,
`alternate, option, or command, in a similar manner. Each
`modifier is distinguished by the direction of its particular
`stroke. An example mapping is shown in FIG. 4. According
`
`MSFT EX. 1006
`
`Page 23 of 35
`
`MSFT EX. 1006
`Page 23 of 35
`
`

`

`6,094,197
`
`5
`to the mapping of FIG. 4, a stroke to the right indicates the
`control (“ctrl”) modifier. Thus a rightward pen stroke initi-
`ated over the character “c” causes the graphical keyboard to
`output the character “ctrl-c.” Astroke to the left indicates the
`option modifier, and a downward stroke indicates the com-
`mand modifier.
`
`Modifiers can easily be combined. For example, once
`again applying the mapping of FIG. 4, a stroke to the right,
`starting on the character “c,” followed by an upward stroke
`expresses “ctrl-C,” as distinct from “ctrl-c.” This is illus-
`trated in FIG. 5.
`Additional stroke directions can be used to substitute for
`certain frequently used keys. For example, pen strokes in the
`diagonal directions can be used to express space, backspace,
`delete, and return. Apossible layout of these commands is
`shown in FIG. 6. This class of strokes is distinct from the
`modifier strokes discussed above in connection with FIGS.
`3, 4, and 5,
`in that the meaning of these strokes is not
`affected by where the stroke is made. Thus a diagonal stroke
`up and to the left is interpreted in the mapping of FIG. 6 as
`a backspace regardless of whether it is made on the “aa” key,
`the “c” key, or any other key of the graphical keyboard.
`Visual feedback can assist the user in applying pen strokes
`to the graphical keyboard. As the user moves the pen across
`the screen, a line appears on the screen to show where the
`pen has been, as though the pen contained ink that left a
`mark on the screen. Once the pen stroke is recognized, the
`character is entered and the “ink trail” disappears. This is
`illustrated, for example, in FIG. 8.
`Each of FIGS. 7—10 is a series of views showing how a
`user enters input into the graphical keyboard of the inven—
`tion. The series of two views of FIG. 7 shows a user entering
`a lowercase character using the graphical keyboard of the
`invention. The user does this by tapping the desired key of
`the graphical keyboard with the pen or stylus. More
`precisely, the user taps the pen against the surface of the
`screen on which the keyboard image of the graphical key-
`board is displayed, with the point of the pen striking the
`screen surface in proximity to the location of the desired key
`within the keyboard image.
`the user places the pen
`In the first view of FIG. 7,
`momentarily on the key corresponding to the letter or other
`character to be entered (an “a” in this example). The key is
`highlighted in the display.
`'lhe character buffer, which is
`represented by the small rectangle next to the view of the
`keyboard, pen, and user, is empty at this stage. In the second
`view, the user removes the pen from the graphical keyboard,
`thereby completing the tapping motion. The graphical key-
`board produces the character “a” and sends it to the character
`buffer. The highlighting disappears from the display.
`The series of three views of FIG. 8 shows a user entering
`an uppercase character using the graphical keyboard of the
`invention. The user does this by making an upward stroke or
`mark on the desired key of the graphical keyboard.
`In the first view of FIG. 8, the user places the pen or stylus
`on the key corresponding to the character to be entered (an
`“A” in this example). The key is highlighted in the display.
`The character buffer is empty.
`In the second view of FIG. 8, the user moves the pen
`“upward”—that
`is,
`toward the topmost
`row of the
`keyboardiwhile continuing to hold the pen against
`the
`keyboard. The user maintains contact between the pen point
`and the surface of the screen on which the keyboard image
`is displayed substantially throughout the upward motion.
`This is called dragging the pen. The display shows a line or
`mark (an ink trail) where the user has dragged the pen. The
`
`10
`
`30
`
`40
`
`50
`
`60
`
`6
`character buffer remains empty at this stage, because the
`character to be entered has not yet been recognized. The key
`remains highlighted in the display.
`In the third view of FIG. 8, the user removes the pen from
`the keyboard surface, completing the upward stroke. The
`line or mark disappears, as does the highlighting, and the
`character produced by the graphical keyboard in response to
`the upward stroke appears in the character buffer.
`The series of three views of FIG. 9 shows a user entering
`a control character using the graphical keyboard of the
`invention. A “control character” is produced on a
`conventional, mechanical keyboard by holding down the
`modifier key marked “control” or “ctrl” while pressing a
`character key. Pressing the “ctrl” key while striking the key
`for the letter “a,” for example, produces the character called
`“control-a,” which can also be written as “ctrl-a” or “a.” On
`the graphical keyboard of the invention, the user can enter
`a control character by making a rightward stroke or mark on
`the desired key of the graphical keyboard.
`In the first view of FIG. 9, the user places the pen or stylus
`on the key corresponding to the character to be entered (the
`key “a” corresponding to the character “‘a” in this example).
`The key is highlighted in display. The character buffer is
`empty.
`In the second view of FIG. 9, the user drags the pen to the
`right: The user moves the pen to the right while continuing
`to hold the pen against the keyboard, and maintains contact
`between the pen point and the surface of the screen on which
`the keyboard image is displayed substantially throughout the
`rightward motion. The display shows a line or mark where
`the user has dragged the pen. The key remains highlighted.
`The character buffer remains empty at this stage, because the
`character to be entered has not yet been recognized.
`In the third view of FIG. 9, the user removes the pen from
`the keyboard surface, completing the rightward stroke. The
`line or mark disappears, as does the highlightng of the key,
`and the character produced by the graphical keyboard in
`response to the rightward stroke appears in the character
`buffer.
`
`FIG. 10 shows that modifiers can be combined by making
`compound strokes that combine strokes of different direc-
`tions. The series of three views of FIG. 10 shows a user
`entering a control uppercase character using the graphical
`keyboard of the invention. A “control uppercase character”
`is produced on a conventional, mechanical keyboard by
`holding down the “shift” and “ctrl” modifier keys simulta—
`neously while pressing a character key. Pressing the “shift”
`and “ctrl” keys while striking the key for the letter “a,” for
`example, produces the character called “control-shift-a,”
`which can also be written as “ctrl-A” or “A.” On the
`graphical keyboard of the invention, the user can enter a
`control uppercase character by making compound
`rightward-upward stroke or mark on the desired key of the
`graphical keyboard.
`In the first view of FIG. 10, the user places the pen or
`stylus on the key corresponding to the character to be
`entered (the key “a” corresponding to the character “A” in
`this example). The key is highlighted in the display. The
`character buffer is empty.
`In the second view of FIG. 10, the user drags the pen a
`distance to the right and then a distance “upward”ithat is,
`toward the topmost row of the keyboard. The user maintains
`contact between the pen point and the surface of the screen
`on which the keyboard image is displayed substantially
`throughout the rightward-and-then-upward motion. The dis-
`play shows a line or mark where the user has dragged the
`
`MSFT EX. 1006
`
`Page 24 of 35
`
`MSFT EX. 1006
`Page 24 of 35
`
`

`

`6,094,197
`
`7
`pen. The key remains highlighted in the display. The char-
`acter buffer remains empty at this stage, because the char-
`acter to be entered has not yet been recognized.
`In the third view of FIG. 10, the user removes the pen
`from the keyboard surface, completing the compound
`stroke. The line or mark disappears, as does the key
`highlighting, and the character produced by the graphical
`keyboard in response to the stroke appears in the character
`buffer.
`
`Although the example of FIG. 10 shows the rightward
`portion of the compound stroke preceding the upward
`portion, the same “A” character is produced if the upward
`portion of the stroke is performed before the rightward
`portion. Other kinds of modifiers can be combined by
`making compound marks in other directions. For example,
`if a leftward stroke indicates the “alt” modifier,
`then a
`leftward—upward or upward—leftward compound stroke com—
`bines the “alt” and “shift” (uppercase) modifiers, and a
`leftward—rightward or rightward—leftward compound stroke
`combines the “ctrl” and “alt” modifiers.
`
`3. Marking Menus
`An interaction technique known as marking menus is used
`to implement the presentation of and response to pen strokes
`in the graphical keyboard of the present invention in the
`specific embodiment to be described. This section presents
`a general description of the technique; section 4 describes
`the technique more specifically as it applies to the present
`invention. A complete description of the marking menus
`technique can be found in Gordon P. Kurtenbach, “The
`Design and Evaluation of Marking Menus,” Ph. D.
`dissertation, Dept. of Computer Science, Univ. of Toronto,
`1993, which is incorporated herein by reference as though
`set out in full.
`3.1 Mark and Related Conceits
`The marking menus technique is intended