United States Patent
`
`[19]
`
`[11] Patent Number:
`
`6,094,197
`
`Buxton et al.
`
`[45] Date of Patent:
`
`Jul. 25, 2000
`
`US006094197A
`
`[54] GRAPHICAL KEYBOARD
`
`[75]
`
`Inventors: William A- S- Buxton, Toronto,
`Canada; Gordon P. Kurtenbach,
`Mountain View’ Calif’
`[73] Assignee: Xerox Corporation, Stamford, Conn.
`
`I21] APPL N05 08/443359
`[22]
`Filed,
`May 17, 1995
`
`Related U.S. Application Data
`
`,
`,
`,
`_
`[63] Continuation of application No. 08/171,288, Dec. 21, 1993,
`abandoned.
`Int. Cl.7 ....................................................... G09G 5/00
`[51]
`[52] U.S. Cl.
`........................ .. 345/358; 345/168, 345/179,
`341/22
`
`[58] Field of Search ................................... .. 345/156, 157,
`345/168, 169, 170, 172, 173, 174, 175,
`176, 179, 180, 181, 182, 183; 341/22, 23;
`170/18, 19; 395/326, 335, 347, 350, 351,
`352, 382/188, 189
`
`.
`References Cited
`
`U.S. PATENT DOCUMENTS
`2/1988 Auer et al.
`.............................. 345/173
`4,725,694
`9/1991
`345/157
`5,050,105
`9/1992
`5,148,155
`178/18
`5,258,748 11/1993
`345/172
`5,276,794
`1/1994 Lamb . . . . . . . . . . . . . . .
`. . . .. 345/173
`5,347,295
`9/1994 Agulnick et al.
`345/156
`5,603,053
`2/1997 Gough et al.
`......................... .. 345/179
`FOREIGN PA1-ENT DOCUMENTS
`
`European Pat 00 ~~~~~~ -- G061: 3/033
`12/1993
`0 603 669/34
`60012123 3/ gapan ~
`‘
`/
`apan ‘
`0011144
`11/1989 WIPO .................................... .. 178/18
`89/1114,‘
`11/1989
`92/09944
`6/1992 WIPO ........................... .. G06F 3/033
`
`OTHER PUBLTCATTONS
`Knodt, Ruediger W. “Smart Virtual Keyboard System Suit-
`able for Mouse or Stylus Entry,” Xerox Disclosure Journal,
`vol. 18(3):245—245, May/Jun. 1993.
`
`Kurtenbach et al. “Issues in Combining Marking and Direct
`Manipulation Techniques,” User Interface Software and
`Technology (UIST), Hilton Head, sc (Nov. 11-13, 1991),
`137-144.
`Kurtenbach, Gordon P. “The Design and Evaluation of
`Marking Menus,” Doctrate Thesis, Graduate Department of
`Computer Science, University of Toronto, 1993.
`
`Kurtenbach, Gordon “Making Marks Self—Revealing,” SIG-
`CHI Bulletin, vol. 23(4):60—61 (Abstract) (Oct. 1991).
`Kurtenbach et al. “An Empirical Evaulation of Some Articu-
`latory and Cognitive Aspects of Marking Menus,” Human-
`Computer Interaction 8:1—23 (1993).
`
`Robertson et al., “Buttons as First Class Objects on an X
`D k
`”
`1
`1
`s 1
`or h
`1
`UIS
`HiiogogieatlsegCnggoicel1:’1‘;Va1r;9a1I; 3;:£° °gY(
`T)»
`>
`'
`>
`>
`'
`
`Primary Examiner—SteVen Saras
`Attorney, Agent, or Firm—Townsend & Townsend Crew
`LLP
`
`ABSTRACT
`[57]
`A system and method for a graphical keyboard that benefits
`.
`.
`.
`.
`.
`from the expressive power and intuitive ease of use associ-
`ated with pen strokes and gestures, yet does not require
`Complex CharaCtef'reC°g“1t1°“ .S°fiWare‘.The graph1°a1key'
`board responds differently to different kinds of pen strokes.
`1
`1
`“
`db
`.
`1
`For examp e, owercase a is entere
`y tapping a sty/‘ usjon
`a key of the graphical keyboard. Uppercase A is
`the
`entered by an upward stroke rnrtrated over the “a” key.
`Likewise, by stroking in other directions, a user can express
`other modifiers to the basic character, such as “control”,
`“alt”, “command,” etc. Also, strokes in distinct directions
`can be used to express space, backspace, delete, and return
`characters, for example. Multiple strokes can be applied in
`sequence to a single key to express multiple modifiers.
`Visual feedback of pen strokes can be provided in the form
`_
`of milrkshriliade thg penf. P.c1)p ltlp menuscatn be lised to
`PTO“ ‘’
`6 P W1
`a“
`0 acl 1 a C memorlza 10“ 0 C°m'
`mand strokes and stroke sequences. The graphical keyboard
`thus lets the expert user proceed by “feel,” and prompts the
`TTOVTCC user when and 35 0eCe55arY~
`
`18 Claims, 20 Drawing Sheets
`
`E
`III Eta.
`EEEEEIEEII-.
`E!E!':£!£lL-'llIlIl
`BEBE!-Ilflli
`—
`
`GOOGLE EX. 1006 pg. 1
`Google v. Philips
`
`GOOGLE EX. 1006 pg. 1
`Google v. Philips
`
`

`
`Jul. 25, 2000
`
`Sheet 1 of 20
`
`6,094,197
`
`STROKE\ =_-> HAM
`
`GOOGLE EX. 10
`
`. 2
`
`Google v.
`
`'
`
`'ps
`
`GOOGLE EX. 1006 pg. 2
`Google v. Philips
`
`

`
`Jul. 25, 2000
`
`Sheet 2 of 20
`
`6,094,197
`
`UPPER CASE
`
`A
`
`OPTION ““‘
`
`“""CONTROL
`
`V
`COMMAND
`
`[::}-*- =>"CONTROL d‘
`
`I
`
`@--I => "CONTROL C“
`
`GOOGLE EX. 1006 pg. 3
`Google v. Philips
`
`GOOGLE EX. 1006 pg. 3
`Google v. Philips
`
`

`
`Jul. 25, 2000
`
`Sheet 3 of 20
`
`6,094,197
`
`[:E:l
`
`IBEEEEEWE
`IIEEEEEEE
`I-IIIIHII EIIIIEL ‘I
`E'{"'Elflll
`EIEEII-‘ll.
`‘
`Bu.
`EB!-II.
`-6
`
`HEEE
`
`EH-IEIEIIJIZEI
`
`.-
`
`' *
`
`‘
`
`‘ III. "-‘BE
`
`EEEEIIHBKEEE \
`
`GOOGLE EX. 1006 pg. 4
`Google v. Philips
`
`GOOGLE EX. 1006 pg. 4
`Google v. Philips
`
`

`
`Jul. 25, 2000
`
`Sheet 4 of 20
`
`6,094,197
`
`E
`IIIEBEIEEE HIIEI I-VIE.
`fllflfllllfl HUGE I
`I-:'.:'ElliE]lfll‘I
`EEE.£'JIfllflll
`9-. """‘
`Bfllilll.-1l'flli
`
`|.\< I.
`
`'94‘.
`
`GOOGLE EX. 1006 pg. 5
`Google v. Philips
`
`GOOGLE EX. 1006 pg. 5
`Google v. Philips
`
`

`
`Jul. 25, 2000
`
`Sheet 5 of 20
`
`6,094,197
`
`LIPBOARD
`
`_.
`
`3"“-'
`
`'
`
`\
`3\\ ‘'
`\\\\\
`
`V\
`
`“aw
`
`SELECTION usws
`MENU MODE
`
`SELECTION USING
`MARK MODE
`
`MENU HIERARCHY
`
`GOOGLE EX. 1006 pg. 6
`Google v. Philips
`
`GOOGLE EX. 1006 pg. 6
`Google v. Philips
`
`

`
`U.S. Patent
`
`Jul. 25, 2000
`
`Sheet 6 of 20
`
`6,094,197
`
`UPPER CASE
`
`BACKSPACE
`
`RETURN
`
`\ ‘ I
`OPWON - - 3,. :- —— CONTROL
`I :
`SPACE
`1 \ DELETE
`
`\
`
`\
`
`COMMAND
`
`GOOGLE EX. 1006 pg. 7
`Google v. Philips
`
`GOOGLE EX. 1006 pg. 7
`Google v. Philips
`
`

`
`U.S. Patent
`
`Jul. 25, 2000
`
`Sheet 7 of 20
`
`6,094,197
`
`I".'?".""'
`=
`n:.'.'.Js:-mi"
`<o1auo10ré:-.57‘ an II
`01o1:'Iz'I.-'10»
`
`..
`.
`«
`:
`'«'na~r.:=r.-02.:
`0210.; nowvelun‘I
`‘-1
`cr '
`0m:Iz'r'sIT?n'»
`
`»-
`
`_'
`
`;
`
`LIl[(lI'I'I:l';Il-Fiifllg
`=~«W»'ra|_z¢:1i_IIm
`.~ ~ \¢
`
`GOOGLE EX. 1006 pg. 8
`Google v. Philips
`
`GOOGLE EX. 1006 pg. 8
`Google v. Philips
`
`

`
`U.S. Patent
`
`Jul. 25, 2000
`
`Sheet 8 of 20
`
`6,094,197
`
`0- -
`
`-\
`
`''012:'i::
`
`(flIuI}{:.'(:§(%1i‘lal1L:I._l§ :L 01'IIIIH!II»
`
`$93.?‘
`
`”’*?o1nm:m‘;'"‘
`
`GOOGLE EX. 1006 pg. 9
`Google v. Philips
`
`GOOGLE EX. 1006 pg. 9
`Google v. Philips
`
`

`
`
`
`111313cI'S'fl
`
`000z‘sz‘I111’
`
`m'
`
`:a‘en
`enFF
`\D
`c:
`H:
`
`N©
`
`L6I‘v60‘9
`
`e9/"'55,: ’
`137-744’I 7/3 ’ 71747117179)/Ill!)
`4rziévz2£%cwyz3zvzz4r“_
`
`6
`TI
`
`I MEMORY
`
`USER
`INTERFACE
`
`APPUCAHON
`PROGRAM
`
`GOOGLE EX. 1006 pg. 10
`Google v. Philips
`
`

`
`
`
`111313cI'S'fl
`
`000z‘sz‘Inf
`
`@I
`
`n‘(‘D
`(‘DFF
`>—|
`
`G OH
`
`:
`[0
`G
`
`I?!V/1}! 7/3/91117/77lI{'J/J/J//
`arzazkusuézagusvzmrnr
`
`APPLICATION
`PROGRAM
`
`USER
`INT ERFAOE
`
`L6I‘v60‘9
`
`GOOGLE EX. 1006 pg. 11
`Google v. Philips
`
`

`
`U.S. Patent
`
`Jul. 25,2000
`
`Sheet 11 of 20
`
`6,094,197
`
`GOOGLE EX. 1006 pg. 12
`Google v. Philips
`
`GOOGLE EX. 1006 pg. 12
`Google v. Philips
`
`

`
`
`
`
`
`00oz‘sz°1nr1ua112(['3'[1
`
`m'
`
`.=a‘en
`enFF
`>—t
`
`N cH
`
`: NG
`
`L6I‘v60‘9
`
`6
`— 5
`
`MEMORY -
`
`45
`
`40
`
`ussn
`INTERFACE
`
`APPLICATION
`PROGRAM
`
`GOOGLE EX. 1006 pg. 13
`Google v. Philips
`
`

`
`U.S. Patent
`
`Jul. 25,2000
`
`Sheet 13 of 20
`
`6,094,197
`
`DETERMINE
`
`EVENT TYPE
`
`PENDRAG
`
`PENSTILL
`
`0
`
`GOOGLE EX. 1006 pg. 14
`Google v. Philips
`
`GOOGLE EX. 1006 pg. 14
`Google v. Philips
`
`

`
`U.S. Patent
`
`Jul. 25,2000
`
`Sheet 14 of 20
`
`6,094,197
`
`BEGIN
`
`DETERMINE
`KEY
`PRESSED
`
`HIGHLIGHT
`THAT KEY
`
`SET MODE
`T0 MARK
`MODE
`
`SAVE PEN
`LOCATION AND
`TIME IN
`PDINTLIST
`
`GOOGLE EX. 1006 pg. 15
`Google v. Philips
`
`GOOGLE EX. 1006 pg. 15
`Google v. Philips
`
`

`
`U.S. Patent
`
`Jul. 25,2000
`
`Sheet 15 of 20
`
`6,094,197
`
`SAVE
`PEN LOCATION
`AND TIME
`IN POINTLIST
`
`DRAW INK TRAIL
`FROM PREVIOUS
`PEN LOCATION
`P3}
`
`DETERMINE MENU
`ITEM BEING
`SELECTED BASED
`ON PEN LOCATION
`
`STORE IN
`‘SELECTED new
`or cuanenr mu
`
`RETURN
`
`GOOGLE EX. 1006 pg. 16
`Google v. Philips
`
`GOOGLE EX. 1006 pg. 16
`Google v. Philips
`
`

`
`U.S. Patent
`
`Jul. 25,2000
`
`Sheet 16 of 20
`
`6,094,197
`
`REPLACE THE
`
`’6'E'}"E;¥"»'N' "c”:'%'§nBs'
`SELECTED av MARK
`
`I m
`
`RETURN
`
`IS PEN
`LOCATEDiN A MENU
`ITEM?
`
`DOES
`ITEM HAVE A
`SUBMENU?
`
`c‘l’,',',§’,'E~°'NP,LA,‘fE,I,',‘E
`
`NSPLAY SUBMENU
`AND MAKE IT THE
`CURRENT MENU
`
`GOOGLE EX. 1006 pg. 17
`Google v. Philips
`
`GOOGLE EX. 1006 pg. 17
`Google v. Philips
`
`

`
`U.S. Patent
`
`Jul. 25,2000
`
`Sheet 17 of 20
`
`6,094,197
`
`BEGIN
`
`DA
`
`DETERMINE MENU
`ITEMS SELECTED
`BY MARK
`
`TRAVERSE THE MM
`HIERARCHY TO DETERMINE
`WHICH ITEMS/MDDIFIERS
`WERE CHOSEN
`
`GENERATE CHARACTER
`
`AND PLACE INTO BUFFER
`
`DF
`
`DC
`
`UNHIGHLIGHT THE
`
`PRESSED KEY
`
`RETURN
`
`
`
`FIG! 25.
`
`GOOGLE EX. 1006 pg. 18
`Google v. Philips
`
`GOOGLE EX. 1006 pg. 18
`Google v. Philips
`
`

`
`U.S. Patent
`
`Jul. 25,2000
`
`Sheet 18 of 20
`
`6,094,197
`
`IS MARK LESS THAN
`5 MM LONG?
`
`_
`
`NO MODIFIER
`WAS SELECTED
`
`GET ARTICULATION POINTS
`
`ARTICULATION POINTS TO MENU ITEMS
`
`GENERATE I-IODIFIERS ACCORDING
`TO MENU SELECTIONS
`
`RETURN
`
`GOOGLE EX. 1006 pg. 19
`Google v. Philips
`
`GOOGLE EX. 1006 pg. 19
`Google v. Philips
`
`

`
`U.S. Patent
`
`Jul. 25,2000
`
`Sheet 19 of 20
`
`6,094,197
`
`GET THE POINTS IN THE INPUT POINTLIST WHERE THE USER
`PAUSED WHILE DRAWING (POINTS WHERE THE ELAPSED
`TIME BETWEEN POINTS IS GREATER THAN .5 SECONDS)
`
`2 PAUSE POINTS
`
`REMOVE INACCURACIES IN THE POINTUST DUE
`TO THE PENS FIRST TOUCHING THE DISPLAY
`
`REMOVE INACCURACIES IN THE POINTLIST DUE
`TO THE PEN'S LIFTING FROM THE DISPLAY
`
`REMOVE SMALL ANGLE CHANGES IN THE
`POINTLIST DUE TO PEN JITTER
`
`ADD STARTING POINT T0
`OUTPUT POINTLIST IF IT
`IS NOT AMONG THE
`PAUSE POINTS
`
`TAIIE PAUSE POINTS TO
`BE ARTIGULATION POINTS.
`ADD TO OUTPUT POINTLIST
`
`FE
`
`ADD FINISHING POINT TO
`OUTPUT POINTLIST
`
`REMOVE SHALL ANGLE CHANGES IN THE
`POINTLIST DUE TO PEN JITTER
`
`F"
`
`THE LOCATION OF THE
`FIRST AND LAST
`POINTS OF THE MARK
`AND THE LARGESTANGLE
`CHANGEISI DEFINE THE
`ARTICULATION POINTS.
`ADO THEM TO THE OUT-
`PUT POINTUST.
`
`YES
`
`ANY
`SIGNIFICANT
`ANGLE CHANGES
`FOUND?
`
`MARK IS A STRAIGHT
`LINE, SO THE FIRST
`POINT In LAST POINT OF
`THE INPUT POINTLIST
`DEFINE A SINGLE MENU
`SELECTION. ADD THEM
`TO THE OUTPUT POINT-
`LIST
`
`RETURN
`
`FIG 27.
`
`GOOGLE EX. 1006 pg. 20
`Google v. Philips
`
`GOOGLE EX. 1006 pg. 20
`Google v. Philips
`
`

`
`U.S. Patent
`
`Jul. 25,2000
`
`Sheet 20 of 20
`
`6,094,197
`
`MAKE THE TOP MM IN
`THE HIERARCHY THE
`CURRENT MENU
`
`FOR EACH
`ARTICULATION
`POINT EXCEPT
`THE LAST ONE3
`
`DETERMINE THE ITEM THAT WOULD BE
`IF THE CURRENT MENU WERE CENTERED
`ON THE ARTICULATION POINT AND THE
`PEN TIP WERE THE LOCATION OF THE
`NEXT ARTICULATION POINT.
`
`STORE THIS AS THE SELECTED
`ITEM FOR THE CURRENT MENU
`
`CONTINUE LOOPINC- GO
`PROCESS NEXT
`ARTICIILATION POINT
`
`GE
`
`YES
`
`GO
`
`MAKE THE SELECTED ITEM
`SUBMENU THE CURRENT MENU
`
`GH
`
`DROP OUT OF LOOP
`
`RETURN
`
`DONE -- LOOP
`TERMINATE D
`IMMEDIATELY
`
`GOOGLE EX. 1006 pg. 21
`Google v. Philips
`
`GOOGLE EX. 1006 pg. 21
`Google v. Philips
`
`

`
`6,094,197
`
`1
`GRAPHICAL KEYBOARD
`
`This is a continuation Ser. No. 08/171,288, filed Dec. 21,
`1993, now abandoned.
`A portion of the disclosure of this patent document
`contains material which is subject to copyright protection.
`The copyright owners have no objection to the facsimile
`reproduction, by anyone, of the patent document or the
`patent disclosure, as it appears in the Patent and Trademark
`Office patent
`file or records, but otherwise reserve all
`copyright rights whatsoever.
`SOFTWARE
`
`The application recites provides a source code software
`program for implementation of an embodiment of the
`method of the invention on a digital computer. The source
`code includes a single file written in the HyperCard scripting
`language. The code runs under the HyperCard program
`(version 2.0 or higher) for Apple Macintosh computers
`(Macintosh II or better hardware running System 7.0 or
`higher operating software), well known to those of skill in
`the art.
`
`is
`The source code represents unpublished work that
`Copyright ©1993 Xerox Corporation. All rights reserved.
`Copyright protection claimed includes all forms and matters
`of copyrightable material and information now allowed by
`statutory or judicial
`law or hereafter granted,
`including
`without limitation, material generated from the software
`programs which are displayed on the screen such as icons,
`screen display looks, etc.
`Although a particular software implementation is
`provided, it will be apparent to those of skill in the art that
`a wide variety of programming languages and hardware
`configurations can readily be used based on this disclosure
`without departing from the scope of the invention.
`BACKGROUND OF THE INVENTION
`
`The present invention relates to graphical user interfaces
`and more particularly to graphical user interfaces for pen-
`based computing.
`Pen-based computing is becoming increasingly popular.
`The user of a pen-based computer inputs commands and
`data into the computer using a pen or stylus, typically by
`printing or writing characters with the pen or stylus on the
`computer’s display screen in a manner analogous to printing
`or writing with a pen on ordinary paper. Certain commands
`can be entered as pen strokes (gestures); for example, a word
`may be deleted by crossing it out, or a file may be opened
`by writing an “o” on its representative icon. Examples of
`pen-based computing are found in systems ranging from
`palm-sized personal digital assistants, through slate-sized
`machines such as notepad computers, and up to wall-sized
`interactive displays. Also, pen-based interfaces can some-
`times be added or retrofitted to computer systems not
`originally designed to be pen-b ased, such as certain personal
`computer systems.
`The usefulness of pen-based computing is limited because
`of the limited capabilities of character recognition software.
`In many pen-based systems, the user cannot write in cursive
`handwriting, but must print. This slows down input.
`Furthermore, the system has difficulty in recognizing printed
`characters. Consequently, the user must print carefully (viz.,
`slowly) and often must print in special windows and within
`designated boxes, and the system can require special train-
`ing for each individual user. Even so, character recognition
`performance can be relatively poor. In addition, pen-based
`
`2
`systems that use character recognition do not adapt well to
`run on conventional computer workstations equipped with a
`mouse, for example.
`Because of the limitations of the character-recognition
`approach to pen-based input, an alternative approach is
`sometimes used. In this approach, a graphical keyboard,
`visually analogous to the conventional mechanical
`(typewriter) keyboard commonly used for input in non-pen-
`based computer systems, appears on the computer display.
`The user enters text by using the stylus to tap on desired keys
`of the graphical keyboard.
`The graphical-keyboard approach offers certain advan-
`tages over the character-recognition approach. Entering data
`through a graphical keyboard can be faster than printing
`(especially with numeric keyp ads), can be less susceptible to
`errors when properly implemented (i.e., when the graphical
`keys are not too small), and does not require the system to
`be specially trained for each individual user. Moreover, the
`approach is simple to implement and integrate into existing
`software applications.
`Unfortunately, known graphical keyboards suffer from
`numerous drawbacks. Although the graphical keyboard is
`visually analogous to a mechanical keyboard, it is function-
`ally far more limited. With a graphical keyboard, a user
`cannot “touch type” using all fingers of both hands, but
`instead must “hunt and peck” with the pen or stylus, as if
`typing with one finger on a mechanical keyboard. Typing
`speed on the graphical keyboard is constrained by the time
`required for the user to move the finger/stylus from key to
`key on the keyboard. Also, the user cannot tap multiple keys
`simultaneously. In particular, the user cannot tap a modifier
`key, such as the “shift,” “ctrl,” “option,” or “alt” key, at the
`same time as an ordinary key, such as a letter or number key,
`but instead must first tap the modifier key and then tap the
`letter or number key. In some systems the user must tap the
`modifier key once to set the modifier,
`tap the letter or
`number key, and then tap the modifier key again to release
`the modifier (for example, tap “shif ,” “a,” “shif ” to input an
`uppercase “A”. This can be especially awkward when mul-
`tiple modifiers must be composed to form a single character
`(for example, a doubly modified character such as ctrl-alt-x
`or a triply modified character such as ctrl-alt-X).
`Another problem with the graphical keyboard approach is
`that it places a heavy visual load on the user. Whereas with
`a mechanical keyboard, the user can memorize commonly
`used sequences of keystrokes and perform them “by feel”
`without looking at the keyboard, with the graphical key-
`board the user typically must look at the display before each
`tap of the stylus to ensure that the stylus is correctly placed.
`Furthermore, the graphical keyboard tends to take up a large
`portion of the screen display, in part because the keyboard
`contains many or all of the modifier and function keys found
`in its mechanical counterpart.
`the graphical-keyboard
`Because of its drawbacks,
`approach to pen-based input
`is often considered
`cumbersome, and is used in pen-based computing systems
`only as a fallback when character recognition fails or is
`insufliciently reliable. The character-recognition and
`graphical-keyboard approaches to pen-based computing are
`typically seen as mutually exclusive alternatives, not as
`complements.
`In particular,
`it has not been possible to
`combine the intuitive ease of use promised (if not always
`afforded) by the character-recognition approach with the
`speed, robustness and ease of implementation offered by the
`graphical keyboard approach.
`SUMMARY OF THE INVENTION
`
`The present invention is a graphical keyboard that permits
`the user to do more than simple key-tapping. It lets the
`
`GOOGLE EX. 1006 pg. 22
`Google v. Philips
`
`GOOGLE EX. 1006 pg. 22
`Google v. Philips
`
`

`
`6,094,197
`
`3
`expert user proceed by “feel,” and prompts the novice user
`when and as necessary. The graphical keyboard benefits
`from the expressive power and intuitive ease of use associ-
`ated with pen strokes and gestures, yet does not require
`complex character-recognition software.
`The graphical keyboard of the invention responds differ-
`ently to different kinds of pen strokes. For example, lower-
`case “a” is entered by tapping a stylus on the “a” key of the
`graphical keyboard. Uppercase “A” is entered by an upward
`stroke initiated over the “a” key. Likewise, by stroking in
`other directions, a user can express other modifiers to the
`basic character, such as “control”, “alt”, “command,” etc.
`Also, strokes in distinct directions can be used to express
`space, backspace, delete, and return characters, for example.
`Multiple strokes can be applied in sequence to a single key
`to express multiple modifiers. Visual feedback of pen strokes
`can be provided in the form of marks made by the pen.
`Pop-up menus can be used to provide help with and to
`facilitate memorization of command strokes and stroke
`sequences.
`
`invention provides an
`the present
`In another aspect,
`improved method for entering characters into a pen-based
`computer using a graphical keyboard.
`A further understanding of the nature and advantages of
`the invention will become apparent by reference to the
`remaining portions of the specification and drawings.
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`FIG. 1 illustrates a sample graphical keyboard;
`FIG. 2 schematically depicts how a lowercase character
`can be entered by tapping a key of the graphical keyboard of
`the invention;
`FIG. 3 schematically depicts how a uppercase character
`can be entered by an upward stroke on a key of the graphical
`keyboard of the invention;
`FIG. 4 schematically depicts how each modifier can be
`distinguished by the direction of its particular stroke;
`FIG. 5 schematically depicts an example of how modifiers
`are combined;
`FIG. 6 schematically depicts an example of how certain
`frequently used keys can be represented as diagonal strokes;
`FIG. 7 is a series of views in which a user enters a
`
`lowercase character by tapping a key of the graphical
`keyboard;
`FIG. 8 is a series of views in which a user enters an
`
`uppercase character by making an upward stroke on a key of
`the graphical keyboard;
`FIG. 9 is a series of views in which a user enters a control
`character by making a rightward stroke on a key of the
`graphical keyboard;
`FIG. 10 is a series of views in which a user enters a control
`
`uppercase character by making a compound rightward-
`upward stroke on a key of the graphical keyboard;
`FIG. 11 depicts the two modes of selecting from a
`marking menu;
`FIG. 12 is example of a radial menu hierarchy and the
`marks that select from it;
`FIG. 13 is an example of mark confirmation;
`FIG. 14 schematically depicts a pop-up menu revealing a
`stroke vocabulary of eight stroke directions;
`FIG. 15 is a series of views in which a user is presented
`with and selects from a marking menu;
`FIG. 16 is a series of views showing an example of
`marking menus used to combine modifiers in the context of
`the present invention;
`
`4
`FIG. 17 is a series of views showing how an eight-way
`marking menu can be used with the graphical keyboard of
`the invention;
`FIG. 18 illustrates a pen-based system suitable for imple-
`menting the invention;
`FIG. 19 illustrates an alternative pen-based system in
`which the display screen and input sensor are decoupled;
`FIG. 19A illustrates an input sensor with an affixed
`keyboard image;
`FIG. 20 illustrates a mouse-based system suitable for
`implementing the invention;
`FIG. 21 is a flowchart of steps for responding to an event
`according to an embodiment of the invention that incorpo-
`rates marking menus;
`FIG. 22 is a flowchart of steps for processing a PenDown
`event;
`
`FIG. 23 is a flowchart of steps for processing a PenDrag
`event;
`
`FIG. 24 is a flowchart of steps for processing a PenStill
`event;
`
`FIG. 25 is a flowchart of steps for processing a PenUp
`event;
`
`FIG. 26 is a flowchart of steps for determining the
`modifiers corresponding to a mark;
`FIG. 27 is a flowchart of steps for determining points of
`articulation along a mark; and
`FIG. 28 is a flowchart of steps for matching points of
`articulation along a mark with centers of corresponding
`menus.
`
`DESCRIPTION OF SPECIFIC EMBODIMENTS
`
`1. Introduction
`
`The present invention provides a system and method for
`an improved graphical keyboard that responds differently to
`different kinds of pen strokes. By applying appropriate
`strokes, the user can easily type characters that on mechani-
`cal keyboards are typed by pressing multiple keys simulta-
`neously. Also, the user can type certain frequently used keys,
`such as space, backspace, delete, and return, from anywhere
`on the keyboard. Using pen strokes makes typing easier and
`faster, reduces the visual load on the user, and makes it
`possible for the keyboard to take up less space in the screen
`display.
`
`2. Examples
`
`To understand the invention, it is helpful to begin with
`some illustrative examples of its use, as depicted in FIGS.
`1-10. A displayed image of a sample graphical keyboard
`according to the invention is shown in FIG. 1. Tapping on
`any of the keys with the pen or stylus has the same effect as
`pushing the analogous key on a mechanical keyboard. Thus
`to enter a lowercase character, a digit, or another unmodified
`character such as a period or comma, the user simply taps the
`desired key.
`To enter an uppercase character, rather than tapping, the
`user makes an upward stroke, initiated on the desired key.
`For example, as illustrated in FIG. 2, tapping on the “a” key
`results in “a.” An upward stroke on “a” results in “A” (FIG.
`3).
`
`The user accesses other modifier keys, such as control,
`alternate, option, or command, in a similar manner. Each
`modifier is distinguished by the direction of its particular
`stroke. An example mapping is shown in FIG. 4. According
`
`GOOGLE EX. 1006 pg. 23
`Google v. Philips
`
`GOOGLE EX. 1006 pg. 23
`Google v. Philips
`
`

`
`6,094,197
`
`5
`to the mapping of FIG. 4, a stroke to the right indicates the
`control (“ctrl”) modifier. Thus a rightward pen stroke initi-
`ated over the character “c” causes the graphical keyboard to
`output the character “ctrl-c.” A stroke to the left indicates the
`option modifier, and a downward stroke indicates the com-
`mand modifier.
`
`Modifiers can easily be combined. For example, once
`again applying the mapping of FIG. 4, a stroke to the right,
`starting on the character “c,” followed by an upward stroke
`expresses “ctrl-C,” as distinct from “ctrl-c.” This is illus-
`trated in FIG. 5.
`Additional stroke directions can be used to substitute for
`certain frequently used keys. For example, pen strokes in the
`diagonal directions can be used to express space, backspace,
`delete, and return. A possible layout of these commands is
`shown in FIG. 6. This class of strokes is distinct from the
`modifier strokes discussed above in connection with FIGS.
`
`in that the meaning of these strokes is not
`3, 4, and 5,
`affected by where the stroke is made. Thus a diagonal stroke
`up and to the left is interpreted in the mapping of FIG. 6 as
`a backspace regardless of whether it is made on the “aa” key,
`the “c” key, or any other key of the graphical keyboard.
`Visual feedback can assist the user in applying pen strokes
`to the graphical keyboard. As the user moves the pen across
`the screen, a line appears on the screen to show where the
`pen has been, as though the pen contained ink that left a
`mark on the screen. Once the pen stroke is recognized, the
`character is entered and the “ink trail” disappears. This is
`illustrated, for example, in FIG. 8.
`Each of FIGS. 7-10 is a series of views showing how a
`user enters input into the graphical keyboard of the inven-
`tion. The series of two views of FIG. 7 shows a user entering
`a lowercase character using the graphical keyboard of the
`invention. The user does this by tapping the desired key of
`the graphical keyboard with the pen or stylus. More
`precisely, the user taps the pen against the surface of the
`screen on which the keyboard image of the graphical key-
`board is displayed, with the point of the pen striking the
`screen surface in proximity to the location of the desired key
`within the keyboard image.
`the user places the pen
`In the first view of FIG. 7,
`momentarily on the key corresponding to the letter or other
`character to be entered (an “a” in this example). The key is
`highlighted in the display. The character buffer, which is
`represented by the small rectangle next to the view of the
`keyboard, pen, and user, is empty at this stage. In the second
`view, the user removes the pen from the graphical keyboard,
`thereby completing the tapping motion. The graphical key-
`board produces the character “a” and sends it to the character
`buffer. The highlighting disappears from the display.
`The series of three views of FIG. 8 shows a user entering
`an uppercase character using the graphical keyboard of the
`invention. The user does this by making an upward stroke or
`mark on the desired key of the graphical keyboard.
`In the first View of FIG. 8, the user places the pen or stylus
`on the key corresponding to the character to be entered (an
`“A” in this example). The key is highlighted in the display.
`The character buffer is empty.
`In the second view of FIG. 8, the user moves the pen
`“upward”—that
`is,
`toward the topmost
`row of the
`keyboard—while continuing to hold the pen against
`the
`keyboard. The user maintains contact between the pen point
`and the surface of the screen on which the keyboard image
`is displayed substantially throughout the upward motion.
`This is called dragging the pen. The display shows a line or
`mark (an ink trail) where the user has dragged the pen. The
`
`6
`character buffer remains empty at this stage, because the
`character to be entered has not yet been recognized. The key
`remains highlighted in the display.
`In the third view of FIG. 8, the user removes the pen from
`the keyboard surface, completing the upward stroke. The
`line or mark disappears, as does the highlighting, and the
`character produced by the graphical keyboard in response to
`the upward stroke appears in the character buffer.
`The series of three views of FIG. 9 shows a user entering
`a control character using the graphical keyboard of the
`invention. A “control character” is produced on a
`conventional, mechanical keyboard by holding down the
`modifier key marked “control” or “ctrl” while pressing a
`character key. Pressing the “ctrl” key while striking the key
`for the letter “a,” for example, produces the character called
`“control-a,” which can also be written as “ctrl-a” or ‘“a.” On
`the graphical keyboard of the invention, the user can enter
`a control character by making a rightward stroke or mark on
`the desired key of the graphical keyboard.
`In the first View of FIG. 9, the user places the pen or stylus
`on the key corresponding to the character to be entered (the
`key “a” corresponding to the character ‘“a” in this example).
`The key is highlighted in display. The character buffer is
`empty.
`In the second View of FIG. 9, the user drags the pen to the
`right: The user moves the pen to the right while continuing
`to hold the pen against the keyboard, and maintains contact
`between the pen point and the surface of the screen on which
`the keyboard image is displayed substantially throughout the
`rightward motion. The display shows a line or mark where
`the user has dragged the pen. The key remains highlighted.
`The character buffer remains empty at this stage, because the
`character to be entered has not yet been recognized.
`In the third view of FIG. 9, the user removes the pen from
`the keyboard surface, completing the rightward stroke. The
`line or mark disappears, as does the highlighting of the key,
`and the character produced by the graphical keyboard in
`response to the rightward stroke appears in the character
`buflbr
`
`FIG. 10 shows that modifiers can be combined by making
`compound strokes that combine strokes of different direc-
`tions. The series of three views of FIG. 10 shows a user
`entering a control uppercase character using the graphical
`keyboard of the invention. A “control uppercase character”
`is produced on a conventional, mechanical keyboard by
`holding down the “shift” and “ctrl” modifier keys simulta-
`neously while pressing a character key. Pressing the “shift”
`and “ctrl” keys while striking the key for the letter “a,” for
`example, produces the character called “control-shift-a,”
`which can also be written as “ctrl-A” or “”A.” On the
`
`graphical keyboard of the invention, the user can enter a
`control uppercase character by making compound
`rightward-upward stroke or mark on the desired key of the
`graphical keyboard.
`In the first View of FIG. 10, the user places the pen or
`stylus on the key corresponding to the character to be
`entered (the key “a” corresponding to the character “”A” in
`this example). The key is highlighted in the display. The
`character buffer is empty.
`In the second view of FIG. 10, the user drags the pen a
`distance to the right and then a distance “upward”—that is,
`toward the topmost row of the keyboard. The user maintains
`contact between the pen point and the surface of the screen
`on which the keyboard image is displayed substantially
`throughout the rightward-and-then-upward motion. The dis-
`play shows a line or mark where the user has dragged the
`
`GOOGLE EX. 1006 pg. 24
`Google v. Philips
`
`GOOGLE EX. 1006 pg. 24
`Google v. Philips
`
`

`
`6,094,197
`
`7
`pen. The key remains highlighted in the display. The char-
`acter buffer remains empty at this stage, because the char-
`acter to be entered has not yet been recognized.
`In the third View of FIG. 10, the user removes the pen
`from the keyboard surface, completing the compound
`stroke. The line or mark disappears, as does the key
`highlighting, and the character produced by the graphical
`keyboard in response to the stroke appears in the character
`buffer.
`
`Although the example of FIG. 10 shows the rightward
`portion of the compound stroke preceding the upward
`portion, the same ““A” character is produced if the upward
`portion of the stroke is performed before the rightward
`portion. Other kinds of modifiers can be combined by
`making compound marks in other directions. For example,
`if a leftward stroke indicates the “alt” modifier,
`then a
`leftward-upward or upward-leftward compound stroke com-
`bines the “alt” and “shift” (uppercase) modifiers, and a
`leftward-rightward or rightward-leftward compound stroke
`combines the “ctrl” and “alt” modifiers.
`
`3. Marking Menus
`An interaction technique known as marking menus is used
`to implement the presentation of and response to pen strokes
`in the graphical keyboard of the present invention in the
`specific embodiment to be described. This section presents
`a general description of the technique; section 4 describes
`the technique more specifically as it applies to the present
`invention. A complete description of the marking menus
`technique can be found in Gordon P. Kurtenbach, “The
`Design and Evaluation of Marking Menus,” Ph. D.
`dissertation, Dept. of Computer Science, Univ. of Toronto,
`1993, which is incorporated herein by reference as though
`set out in full.
`3.1 Mark and Related Conceits
`
`The marking menus technique is intended to support the
`novice user and expert user alike, and to provide an efficient
`mechanism to support the us