`
`Ex. GOOG 1108
`
`EX. GOOG 1108
`
`
`
`
`
`

`

`United States Patent [19]
`Berman et al.
`
`111111
`
`lllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll
`5,760,773
`Jun.2, 1998
`
`US005760773A
`[ll] Patent Number:
`[451 Date of Patent:
`
`[54] METHODS AND APPARATUS FOR
`INTERACTING WITH DATA OBJECTS
`USING ACTION HANDLES
`
`[75]
`
`Inventors: Eric Robert Berman. Redmond;
`Edward Low Mills. Bellevue; Michael
`Hinkley Van KJeeck. Duvall; Vmayak
`A. Bhalerao. Redmond. all of Wash.
`
`[73) Assignee: Microsoft Corporation, Redmond.
`Wash.
`
`[21) Appl. No.: 369,688
`
`Jan. 6, 1995
`
`[22] Filed:
`Int. CL6
`•••••••••••••••••••••••••••••••••••••••••••••••••••••••• G06F 3100
`[51]
`[52] U.S. Cl •............................................. 3451347; 345/121
`[58] Field of Search ..................................... 395/155. 159.
`395/156. 161. 146, 133. 347. 339. 348,
`354, 352. 334, 335; 345/145. 146. 121.
`347,339.348.354,352.334.335.433
`
`[56)
`
`References Cited
`
`U.S. PATENf DOCUMENTS
`
`5,196,838
`5,345,543
`5,347,295
`5,367,453
`5,390,281
`
`3/1993 Meier et al ............................. 345/118
`9/1994 Capps et al ............................. 345/437
`9/1994 Agulnick et al ........................ 345/156
`1111994 Capps et al ............................. 707/531
`211995 Luciw et al ............................... 395/12
`
`5,396,590
`5,450,539
`5,513,309
`5,515,496
`5.570,281
`
`3/1995 Kreegar ................................... 345/347
`9/1995 Ruben ..................................... 345/354
`4/1996 Meier et al ............................. 345/339
`5/1996 Kaehler et al .......................... 345/334
`10/1996 Berry ...................................... 3641146
`
`Primary Emminer-Matthew M. Kim
`Assistant EXaminer--Crescelle N. dela Torre
`Attome~ Agent, or Finn-Jones & Askew, lLP
`[57]
`ABSTRACT
`
`A central processing unit (CPU) is coupled to a computer
`display for displaying graphic and other information. The
`CPU is further coupled to a pointer control device such as a
`pen. stylus or mouse that permits the user to selectively
`position a pointer and activate an action handle on the
`display associated with a data object. Activation of the
`action handle signals the CPU of selections associated with
`the data object. Tapping or clicking on the action handle
`causes display of a context menu containing at least one
`command that may be invoked with respect to the data
`object. Dragging the action handle indicates movement or
`dragging of the action handle and the corresponding data
`object for an operation associated with dragging such as
`drag-and-drop. The use of action handles in association with
`data objects eliminates ambiguity between actions intended
`to create digital ink with respect to a pen-based computer
`system and actions intended to invoke a command or other
`interaction with the computer system.
`
`30 Claims, 12 Drawing Sheets
`
`Ex. GOOG 1108
`
`

`

`U.S. Patent
`
`Jun. 2, 1998
`
`Sheet 1 of 12
`
`5,760,773
`
`10~
`
`x=
`
`0
`
`, I . 25
`\ \ i 1
`I
`
`\
`
`\
`
`I
`
`I
`
`/11'
`IJ.
`j;/.
`x = HS2-+-+...,.._- /. ~~ 13
`x = HS1-~--Yif-.
`
`.. / /
`
`//i/
`/,/ /
`
`42
`
`'
`
`-f--+-1~::1{V (DATA~ OBJ~f1
`r ;; \~---------J
`I ·
`TO DISPLAY
`\ JtO~
`\
`SCREEN
`. >--------~-_j
`22
`~
`L ___ , ___ _J--,
`-~ 110
`t---
`I
`20~ __ _j ___ ,
`I
`;;7L ___ r __ .J
`CPU
`/I
`I
`/ r----L--1
`I MEMORY
`I
`L ______ J
`
`19
`
`18
`
`27
`
`31
`
`~+WI~ PCMCIA
`CARD
`
`29
`
`J
`
`,
`
`x=O
`
`x=W
`
`FIC.l
`
`Ex. GOOG 1108
`
`

`

`U.S. Patent
`
`Jun. 2, 1998
`
`Sheet 2 of 12
`
`5,760,773
`
`10
`\.,..
`
`Undo
`Undo
`
`40d 42d
`
`Addresses
`LJ Adrian Wyard
`LJ Anthony Discolo
`L::J Barbara Ellsworth
`[Bi~taintoo ... · • · · ·· · · · ·
`L::J Bjorn Hovstadius
`LJ Brett Marl
`
`Show I by First Name I vI.___~
`206-456-3332
`206-456-3332
`206-456-3332
`.·. /::;.~~2]
`206-456-3332
`
`428
`\
`
`4
`
`18
`
`L:J Bowie, David
`0 Chad Schwitters
`D Cynthia Anderson
`
`206-456-3332
`206-456-3332
`206-456-3332
`
`Description:
`
`0
`
`News
`
`Calendar
`
`To Do
`
`Mail
`
`Notes
`
`FIG.2
`
`Ex. GOOG 1108
`
`

`

`U.S. Patent
`
`Jun. 2, 1998
`
`Sheet 3 of 12
`
`5,760,773
`
`40a~ lal§l
`40b~ 1...:.1
`40c~ l-=·1
`40d~u
`
`TEXT
`
`TO DO ITEM
`
`MESSAGE
`
`ADDRESS
`
`40e~
`
`CALENDAR ITEM
`
`40f~
`
`NOTE
`
`40g~
`
`MULTIPLE OBJECTS
`
`40h~1 ... .. ,
`
`INSERTION POINT I CARET
`
`EXEMPLARY STATIC ACTION HANDLES
`
`FIG.J
`
`~
`
`40a
`
`40c~ {S)
`
`40g~~
`FIG.4
`
`EXEMPLARY DRAG ICONS
`(DYNAMIC OR STATUS INDICATING ACTION HANDLES
`
`Ex. GOOG 1108
`
`

`

`42
`
`I
`
`I
`
`I
`
`I
`
`I
`
`I
`
`...
`
`I
`
`I
`
`I
`
`I
`
`I
`
`...
`
`I
`
`I
`
`I
`
`I
`
`I
`
`I
`
`I
`
`I
`
`I
`
`I
`
`I
`
`I
`
`I
`
`"
`'
`I
`I
`
`I
`
`I
`
`I
`
`I
`
`I
`
`I
`
`I
`
`I
`
`I
`
`I
`
`I
`
`I
`
`I
`
`I
`
`I
`
`I
`
`I
`
`I
`
`I
`
`I
`
`I
`
`I
`
`I
`
`I
`
`c en •
`~ ~
`g
`
`~
`
`~ c::
`?
`,.
`N
`"""' ~
`
`00
`
`("tt
`
`CfJ =-("tt
`.....
`.a;:..
`~
`"""' N
`
`
`
`13319.21'S'fl
`
`8661‘Z'unf
`
`21J017199*18
`
`€LL‘09L‘S
`
`"••••••••••~~••••••••i'•••••••: I
`•tt••••••••••
`
`1
`
`J!
`
`•
`•
`
`FIG.~
`
`Ex. GOOG 1108
`
`

`

`U.S. Patent
`
`Jun. 2, 1998
`
`Sheet 5 of 12
`
`5,760,773
`
`Address
`
`To Do
`
`Notes
`
`Mail
`
`To
`Ce
`Bee
`
`Priority:
`
`18
`\,.,.
`
`Subject
`
`Proposal
`
`Encl: I El ~~~6fera1
`
`Notes about the proposal:
`
`1. Under 10 pages in length.
`2. Include specs for both product lines.
`3. Analyze competitive information
`FIC.6
`
`Ex. GOOG 1108
`
`

`

`U.S. Patent
`
`Jun.2, 1998
`
`Sheet 6 of 12
`
`5,760,773
`
`40 )
`j Notes
`
`Properties ...
`Delete
`Clear Ink
`Send ...
`Print..
`
`lOa
`
`I 1-
`
`.
`Apphcat1on action handle
`
`FIC.7A
`
`40
`
`62329
`67385
`64453
`64861
`
`lOb
`
`FIG.7B
`
`Ex. GOOG 1108
`
`

`

`U.S. Patent
`
`Jun. 2, 1998
`
`Sheet 7 of 12
`
`5,760,773
`
`Description:
`
`18_/
`
`82
`FIG.8A
`
`FIG.8B
`
`FIG.8C
`
`z
`
`84
`
`84
`
`Ex. GOOG 1108
`
`

`

`U.S. Patent
`
`Jun.2, 1998
`
`Sheet 8 of 12
`
`5,760,773
`
`Description:
`
`40a
`
`ich is mine, is mine.
`that I have; that is to al!
`This is a test of thP. foxy hue
`
`88
`
`89
`
`z
`
`FIG.9
`
`84
`
`:. dog
`that is lazy. This theory
`This is a test of
`that I have, that is
`which is mine, is mine.
`·~· I
`This is a test of t .1e
`hue.
`
`, I
`
`FIC.l 0
`
`Ex. GOOG 1108
`
`

`

`U.S. Patent
`
`Jun.2, 1998
`
`Sheet 9 of 12
`
`5,760,773
`
`90
`
`40a ~ 92
`
`40a-
`
`FIG.ll A
`
`FIG.ll B
`
`FIG.llC
`
`FIG.ll D
`
`FIG.ll E
`
`FIG.ll F
`
`FIG.llC
`
`Ex. GOOG 1108
`
`

`

`U.S. Patent
`
`Jun.2, 1998
`
`Sheet 10 of 12
`
`5,760,773
`
`100\
`
`CLASS: RECTANGLE
`
`INSTANCE VARIABLES:
`
`UPPER LEFT
`UPPER RIGHT
`LOWER LEFT
`LOWER RIGHT
`
`CORNER
`CORNER
`CORNER
`CORNER
`
`MESSAGES: METHODS:
`
`LINEWIDTH
`FILLPATTERN
`
`DRAW
`ERASE
`MOVE
`
`DRAW LINE FROM POINT TO POINT
`ERASE LINES
`TRANSlATE OR OFFSET CORNER POINTS
`
`FIG.12
`
`Ex. GOOG 1108
`
`

`

`U.S. Patent
`
`Jun.2, 1998
`
`Sheet 11 of 12
`
`5,760,773
`
`105'\
`
`CLASS: ICON
`
`INSTANCE VARIABLES:
`BITMAP
`LOCATION
`UNDERLYING OBJECT
`
`MESSAGES: METHODS:
`
`SELECT
`ACTIVATE
`
`• DISPLAY BITMAP IN REVERSE VIDEO
`• SEND ACTIVATE MESSAGE TO
`UNDERLYING OBJECT:
`DISPLAY REVERSE VIDEO ALTERNATELY
`SHOW MENU • DISPLAY MENU
`DRAG
`• TRANSLATE OR OFFSET LOCATION
`DROP
`• REDRAW AT NEW LOCATION; EXECUTE FUNCTION
`• DELETE BITMAP AND UNDERLYING OBJECT
`DELETE
`DOUBLETAP ·EXECUTE 1ST COMMAND ON COMMAND LIST
`
`108\. 1
`
`110
`~
`SUBCLASS: PROGRAM ICON SUBCLASS: ACTION HANDLE ICON
`
`\
`
`INSTANCE VARIABLES:
`•
`•
`•
`MEMORY ALLOCATION
`
`INSTANCE VARIABLES:
`BITMAP 1
`BITMAP2
`BITMAP3
`
`MESSAGES: METHODS:
`
`MESSAGES: METHODS:
`
`ACTIVATE: LOADPROGRAM ACTIVATE: POPUPCONTEXTMENU
`DRAG
`DRAG
`
`FIC.l "J
`
`Ex. GOOG 1108
`
`

`

`U.S. Patent
`
`Jun.2, 1998
`
`Sheet 12 of 12
`
`5,760,773
`
`OPERATING
`SYSTEM
`SOFTWARE
`
`\.120
`
`I
`EVENT
`HANDLER
`
`\.125
`
`HHWJNDOW
`
`\.. 135
`
`HHAPPLICATION
`
`\.130
`
`HHPANE
`
`\.138
`
`HHBUITON
`
`HHEDIT
`
`HHINK
`
`\.
`140
`(DESCRIBED)
`
`HHLISTBOX
`'= 150
`(DESCRIBED)
`
`FIG.14
`
`Ex. GOOG 1108
`
`

`

`5.760.773
`
`1
`METHODS AND APPARATUS FOR
`INTERACTING WITH DATA OBJECTS
`USING ACTION HANDLES
`
`TECHNICAL FIELD
`
`The present invention relates generally to apparatus and
`methods for displaying and manipulating information on a
`computer display screen. and more particularly relates to a
`computer controlled display system for displaying action
`handles or icons associated with varying types of data
`objects on a display screen. and for allowing user interaction
`with selected data objects by way of the action handles or
`icons.
`
`20
`
`25
`
`BACKGROUND OF THE INVENTION
`Present day computer systems often employ "object(cid:173)
`oriented" displays with icons, pictures. text. pop-up or
`drop-down menus. dialog boxes. and other graphical items
`or objects in what is known as a "Qraphical Jlser
`Interface" or "GUI". In such systems. various combinations
`of graphical items or objects are utilized to convey infor(cid:173)
`mation to users via a cathode ray tube (CRf) monitor or
`display. Users generally interact with the computer system
`by activating graphical items or objects on the display. The
`graphical items or objects are often activated by pointing at
`them with a mouse. pen. or other pointing device. and
`clicking or tapping a button.
`Object oriented display systems often utilize small graphi-
`cal items or objects called "icons". which symbolically 30
`indicate a type of operation the computer system will
`execute if the icon is selected by a user interface device such
`as a mouse. pen. or keyboard. These icons are frequently
`disposed within a window on a display. Icons are often used
`to delete or copy information. move information from one 35
`window to another. cause operations to the data in the
`window. and the like. An example of a computer graphical
`user interface that uses icons is the Microsoft WINDOWSTM
`operating system manufactured by the assignee of the
`present invention. Microsoft COipOration.
`New generations of computer systems that use a pen or
`stylus as a user input device are being introduced. Some of
`the new pen-based computers are small handheld devices
`with a display screen but no keyboard. Such devices are
`sometimes called "Personal Digital Assistants". "PDAs". or 45
`"portable pen pads". With such devices. a user interacts with
`a display screen like a sheet of paper. using an ink-less pen
`or stylus to draw pictures. to enter text via handwriting or
`printing. and to interact with the computer system. The
`computer interprets the motion of the pen or stylus relative 50
`to the display screen. and often displays lines or markings on
`the screen that track the movement of the pen. Markings on
`the screen displayed as a result of movement of the stylus are
`often called "digital ink".
`The pen or stylus in a pen-based computing system may 55
`be considered a "direct pointing device", that is. a pointing
`device consisting of a display and an indicator such as a
`finger, stylus. or other object. The pointing-device actions of
`"pointing" and ''tapping" -which are analogous to "click(cid:173)
`ing" with a mouse-are achieved by moving the indicator on 60
`or near the display screen and touching the display screen,
`respectively. With a mouse or trackball type pointing device.
`there is an indirection. The user moves some physical device
`(positioned somewhere away from the screen) in order to
`move an artificial pointer (i.e. a cursor) on the display 65
`screen. Without the cursor, there is no way for the user to
`"see where you are" on the screen. With a direct pointing
`
`2
`device. the ''where you are" is represented by the indicator
`itself. and. optionally. by a cursor.
`Because there is no keyboard or mouse. pen-based com(cid:173)
`puters have tighter restrictions on what a user can and cannot
`5 do. Common user interfaces such as those is found in the
`Microsoft WINDOWS™ operating system and other Gill(cid:173)
`based computer systems typically rely on a keyboard and a
`pointing device with buttons such as a mouse or trackball for
`user interaction. The user points to a location on the screen
`IO where an action is desired. and invokes a command by
`pressing a button on the pointing device or entering a
`keyboard command. For example. to insert a space between
`words it is generally assumed the user can simply select a
`location for insertion of the space by using the mouse to
`15 position a cursor at the point of insertion and hit the space
`bar. Consequently. many present-day application software
`programs do not offer an "insert space" command for text
`editing.
`Some known computer application programs add com-
`mands to existing drop-down menu bars. However. this
`method is not suitable for handheld pen-based applications
`since most menu bars are impracticably large on devices
`with a small screen.
`Therefore. systems that are required to work with a pen or
`stylus must find alternative methods to accomplish such
`actions.
`In pen-based computers. it is difficult for the computer to
`distinguish actions of a user intended as digital ink from
`actions intended to invoke operations of or provide com(cid:173)
`mands to the computer system. It is desirable that a user
`interface for a pen-based computer system be able to inter(cid:173)
`pret movement of the stylus in a consistent. unambiguous
`fashion. so that users can enter data or invoke commands
`quickly and without frustration. It is also important that the
`user interface be intuitive and easy to remember. so that the
`user can readily determine how to tell the computer that he
`or she intends to invoke a command. or make digital ink (e.g.
`enter text via handwriting)
`One example of a known pen-based computer system is
`the Newton™ MessagePadTM manufactured by Apple
`Computer. Inc .. Cupertino. Calif. This device is a small.
`hand-held device with a liqnid crystal display (LCD) screen.
`no keyboards or buttons (other than an on-off switch). and
`a stylus that is used to interact with the system. The user
`manipulates the stylus to enter handwriting. tap on regions
`of the screen. or draw pictures. The Apple Newton attempts
`to interpret patterns of markings by the stylus on the screen
`or "gestures" as commands. A gesture is a symbol or pattern
`of markings that the user can draw with a stylus (or finger)
`that is recognized by the system as a command. rather than
`as text or a picture.
`For example, in the Apple Newton a squiggly gesture
`made on top of a word like this: ~ means "delete this"
`and (if interpreted properly) causes the deletion of the object
`or text upon which the squiggly mark was drawn. However.
`the computer might interpret the marking as the character
`"W". and the result would be a misspelled word ''wordW".
`
`40
`
`toqeose means to
`likewise, a caret gesture like this:
`"insert space". Moreover. the user must remember that a
`squiggly gesture means "delete". and that a caret gesture
`means "insert". since there is no readily apparent means for
`telling the user what the gestures are if forgotten.
`Stated in other words, gestures are "hidden" from the user
`and can be ambiguous. That is. there is no way for a user to
`determine what particular gesture to draw when a particular
`
`Ex. GOOG 1108
`
`

`

`5.760.773
`
`5
`
`3
`command is desired. except to be told or remembered Once
`learned. however. gestures can be easily forgotten. if not
`repeatedly used. Another disadvantage is that of recognition.
`Gestures can be and frequently are misrecognized. The
`results of a misrecognized gesture range from simple annoy-
`ance to unintended loss of data.
`Even with traditional keyboard and/or mouse-type graphi(cid:173)
`cal user interfaces. there is sometimes ambiguity as to the
`function of an icon or the manner for invoking a command
`for a data object displayed on screen. For example. in a 10
`mouse-driven GUI. if users wish to delete a word from
`within a sentence. they must remember that they must (1)
`first select the word to be deleted by pointing and dragging
`the cursor (thereby forming a "selected data object") and (2)
`then invoke the appropriate "delete" command by either (a) 15
`remembering the keyboard equivalent command. (b) finding
`the command from a drop-down or pop-up menu which may
`be remotely located on the screen relative to the selected
`data object. or (c) activate a disassociated "delete" icon
`somewhere on the screen. assuming they can find it.
`While all of such methods (keyboard command. menu
`command. icon command) are presently employed with
`success in modern application computer programs. it would
`advance the art of GUI design if tl!e object itself upon which
`an action is desired could provide the user with the options 25
`to invoke actions in a concise. consistent and uniform
`methodology. Such an advance would not force users to
`remember keyboard commands. locate commands in menu
`located away from tl!e object. or locate appropriate action
`icons located remotely from the object. In yet other words. 30
`it would be desirable that selected data objects themselves
`provide a context for prompting and receiving user interac(cid:173)
`tion.
`In order to reduce ambiguity to some degree. some 35
`application programs for the Apple Newton Message Pad
`system employ fixed-position icons that display pop-up
`menus of selectable items when tapped. (A "tap" is a form
`of gesture wherein the user lightly strikes a spot on the
`display screen with the stylus.) These fixed-position icons 40
`are typically a small diamond shaped region (e.g. +)whose
`very shape signifies that it is a "hot spot". It has proven easy
`for users to remember that tl!e diamond icon means "tap here
`for display of selectable options."
`However. such pop-up menus are associated with a series
`of selectable items. and are not context-sensitive. There is no
`predictable one-to-one connection between an object and an
`icon. In other words. selected data objects themselves do not
`provide a context for prompting and receiving user interac(cid:173)
`tion. Moreover. such pop-up menu icons do not completely
`duplicate gesture functionality, and cannot be moved or
`repositioned.
`It is expected that the use of pen-based computing sys(cid:173)
`tems will increase in the foreseeable future because of the
`flexibility afforded by such systems to people who either 55
`cannot or do not wish to type using a keyboard Moreover.
`pen-based computers offer the advantage of greater port(cid:173)
`ability since no keyboards or pointing devices are required
`other than a simple stylus. It is foreseen that even future
`desktop computer systems may utilize flat panel display
`screens that serve both as an upright display when a user
`wishes to use a keyboard and/or mouse for interaction. but
`may also be removed from a stand, placed in a lap or on a
`desktop. and employed like "digital paper" with a stylus, as
`the user enters "digital ink''. Therefore, there is a need for
`computer operating systems that offer increased command
`accessibility and reduced ambiguity, especially for pen-
`
`4
`based computer systems, but for that matter with any types
`of computer systems that rely upon undiscoverable or easily
`forgettable actions that are associated with particular icons.
`
`SUMMARY OF THE INVENTION
`
`The present invention overcomes the foregoing described
`difficulties with present computer user interface design by
`providing action handles or icons in association with data
`objects. typically displayed in close proximity to a related
`data object displayed on the screen. The action handles
`provide a method for performing actions on text and other
`data objects with a pen or other pointing device. and also a
`method for displaying a context menu for the text or other
`objects. A context menu, for purposes of the present
`application, is a menu of commands that apply to only a
`particular object within a particular context. and that pops up
`on screen over. or in proximity to the particular object.
`Briefly described. the present invention comprises com(cid:173)
`puter systems and methods that comprise displaying a small
`20 icon or "action handle" next to or in close proximity to a data
`object. such as a selected text item. or a calendar item. or a
`list of other data items. or at an insertion point within a range
`of text. The action handles receive event messages indicative
`of a tapping or dragging action with a stylus or other
`pointing device such as a mouse. In response to a tapping
`action. preferably a context menu displaying commands
`available for the object or text is displayed In further
`response to selection of a command in the context menu. the
`system carries out the appropriate selected command. In
`response to a dragging action. preferably the selected object
`is moved to a new location.
`For example. an action handle next to an address in an
`address book can be dragged to copy or move the address to
`another application. for example. to an electronic mail
`("e-mail") application to send mail to a person. Or. the action
`handle can be tapped to cause display of the context menu
`that offers commands that are appropriate in a given context.
`such as "delete". "print". or "edit". that act on the address.
`As another example. the action handle associated with a
`selected text data object might be used to move the text
`around by dragging. and might contain commands in its
`context menu such as "delete". "make upper case". "make
`bold face". Similarly. an action handle associated with an
`45 insertion point (which may include a draggable cursor) can
`be dragged to select text. which then yields an action handle
`associated with the selected text. The selected text action
`handle itself. like the draggable cursor action handle, can be
`tapped to reveal commands such as "insert space" or "start
`50 new line".
`Preferably. the action handle will contain graphical infor(cid:173)
`mation to assist in identifying the object in question. For
`example. the action handle for an address item in an address
`book might look like a business card
`More particularly described. the present invention relates
`to methods and systems for displaying and controlling
`actions relating to a data object displayed on a display in a
`computer controlled display system having a display
`coupled to a central processing unit (CPU). The methods and
`60 systems include the steps of generating a graphic image
`associated with the data object on the display, and generating
`a graphic image of an action handle proximate to the data
`object. An activatable region on the display associated with
`the action handle graphic image is provided. A user positions
`65 a pointer on the display using a pointing control device
`operatively associated with the CPU, which thereby pro(cid:173)
`vides a signal to the CPU upon correspondence of the
`
`Ex. GOOG 1108
`
`

`

`5,760.773
`
`25
`
`30
`
`5
`pointer with the activatable region. In response to a signal
`indicative of a first type of interaction between the pointer
`and the activatable region, the method involves displaying at
`least one selectable command relating to the data object. In
`response to signal indicative of a second type of interaction
`between the pointer and the activatable region. the method
`involves moving the graphic image of the data object on the
`display.
`In the preferred embodiment. the first type of interaction
`is a tap. while the second type of interaction is a drag. These
`types of interactions are particularly suitable for pen-based
`computer systems.
`The preferred step of displaying at least one selectable
`command comprises displaying a context menu on the
`display including a plurality of selectable commands. In the
`context menu. there is provided an activatable second region
`on the display associated with the selectable command. In
`response to interaction between the second region and the
`pointer, a computer command corresponding to the select(cid:173)
`able command is executed. Preferably. again. the interaction 20
`between the pointer and the second region comprises a tap.
`Still more particularly described. the step of moving the
`graphic image of the data object on the display comprises
`moving the graphic image of the action handle on the display
`during a dragging operation from a first position to a second
`position on the display. In response to a release command
`provided at the second position on the display (typically by
`withdrawing the pen or stylus from the display screen). a
`predetermined operation is carried out. e.g. the graphic
`image of the data object is moved to a new position on the
`display associated with the second position. or if the object
`is "dropped" onto a clip box region, the data object is copied.
`In the invention, the data object may be a data item
`associated with a predetermined application program. For
`example. the data item could be a selected string of text
`associated with a text processing application program. or a
`calendar entry associated with a calendar application pro(cid:173)
`gram.
`It will be appreciated that the invention has utility for both .w
`pen-based and mouse-based (e.g. desktop) computer sys(cid:173)
`tems. Advantageously. action handles are very discoverable
`and cannot be forgotten since they are always displayed in
`association with the data object. To find out what command
`are available in the context of a particular object, the user 45
`can simply tap on the action handle to display the context
`menu. and then selects an appropriate command from the list
`of choices in the context menu. Execution of a command in
`the context menu operates like other menus.
`Action handles also have no ambiguity about them. The so
`actions available at any given time are always displayed by
`tapping the action handle to reveal the context menu. and if
`the item is movable. the action handle icon. as it is dragged
`with the stylus or pointing device on the screen, can change
`its graphical properties to indicate that movement is permit- 55
`ted. If movement is not permitted, the action handle simply
`never moves or pops back to the original location. indicating
`non-movability to a particular location. This would be handy
`in applications wherein a particular data item cannot be
`discarded. as when a user attempts to drag the non- 60
`discardable data object to the trash can or a delete icon. and
`the icon pops back to its location original.
`Action handles therefore eliminate the need for gestures.
`Action handles always provide a visible place where the
`stylus always acts as a pointing device rather than as a pen 65
`(that is. no digital ink is created when touching an action
`handle). In text entry areas. mouse operations such as
`
`6
`drag-and-drop and selection are very difficult for a computer
`to distinguish from inking and writing operations. Action
`handles solve this problem.
`It will thus be appreciated that action handles work with
`5 any type of pointing device. and provide a mechanism
`whereby a user can readily determine that an action is
`possible. This solves the problem of creating context menus
`in computer application programs that rely upon a "right
`mouse" button (for which fingers and pens have no analog)
`10 for activation of a context menu. With an action handle.
`there is always a visible indication of what a user can click
`or tap on to get a context menu.
`The action handle upon its creation can indicate a visible
`encapsulation of an object being manipulated. In other
`words. the icon can represent the object for actions to be
`15 taken with the object Action handles further permit a
`single-tap or click method to cause display of the context
`menu. This contrasts with certain double-click or other
`multiple tap or click methodologies which are difficult for
`users to remember.
`The action handle provides a single focus of the user's
`attention for all actions on an object, both dragging and
`command execution. Again, this helps eliminate ambiguity
`and promotes consistency in the user interface across dif(cid:173)
`ferent applications and through the operating system.
`Action handles, as contemplated by the present invention,
`are automatically generated and exposed in the appropriate
`context. If any particular action is available for an particular
`object. the action handle is displayed. Thus. the user need
`not remember any gestures. and can always find out what
`action to take by either dragging the action handle or tapping
`the action handle. Even common insertion-point icons or
`cursors can automatically appear in edit controls.
`Accordingly. it is an object of the present invention to
`35 provide an improved graphical user interface for computer
`systems.
`It is another object of the present invention to provide an
`improved graphical user interface particularly suitable for
`pen-based computer systems.
`It is another object of the present invention to provide an
`improved graphic user interface for pen-based computer
`systems that reduces ambiguity in the possible actions that
`can be taken with respect to an object displayed on a display
`screen.
`It is another object of the present invention to provide an
`unambiguous method for selecting text and other objects in
`a pen-based computer system graphic user interface.
`It is another object of the present invention to provide for
`an improved user interface for pen-based computers that
`allows a computer system to distinguish actions of a user
`intended as digital ink from actions intended to invoke
`operations of or provide commands to the computer system.
`It is another object of the present invention to provide a
`pen-based user interface for a computer system that is able
`to interpret movement of a pen or stylus in a consistent.
`unambiguous fashion. so that users can enter data or invoke
`commands quickly and without frustration.
`These and other objects. features. and advantages of the
`present invention may be more clearly understood and
`appreciated from a review of the following detailed descrip(cid:173)
`tion of the disclosed embodiments and by reference to the
`appended drawings and claims.
`BRIEF DESCRIPTION OF THE DRAWING
`fiGURES
`FIG. 1 illustrates the general architecture of a handheld
`pen-based computer that is constructed in accordance with
`
`Ex. GOOG 1108
`
`

`

`5.760.773
`
`7
`and utilizes the teachings of the present invention to provide
`action handles in association with data objects.
`FIG. 2 illustrates an exemplary display screen for the
`computer of FIG. 1. that illustrates various relationships of
`action handles and associated data objects.
`FIG. 3 illustrates exemplary static action handles utilized
`in the preferred embodiment of the invention to indicate
`different types of data objects or items.
`FIG. 4 illustrates exemplary dynamic (in dragging
`motion) and status indicating action handles or "drag icons"
`utilized in the preferred embodiment of the invention to
`indicate status of a drag or a drag-and-drop operation.
`FIG. 5 illustrates a general drag-and-drop operation uti(cid:173)
`lizing an action handle.
`FIG. 6 illustrates a drag-and-drop operation to a clip box,
`for copying a data object associated with an action handle.
`FIGS. 7Aand 7B illustrates pop-up or context menus that
`are generated in the preferred embodiment of the present
`invention when a user taps on an action handle associated 20
`with a predetermined application and a predetermined data
`item. respectively.
`FIGS. 8A--SC illustrates a drag-and-drop operation
`applied to an action handle associated with a range of
`selected text in an exemplary text-based application 25
`program. in accordance with the invention.
`FIG. 9 illustrates an action handle that is "pinned" to a
`border of a window in accordance with the invention.
`indicating that the associated data object has been scrolled
`off-screen.
`FIG. 10 illustrates an action handle associated with a
`small range of selected text of a size approximating that of
`the action handle itself. displayed beneath the selected text.
`in accordance with the invention.
`FIGS. llA-llG illustrates an action handle in association 35
`with a flashing cursor data object, for purposes of text
`selection and manipulation. in accordance with the inven(cid:173)
`tion.
`FIG. 12 is a table illustrating a general class of objects (a 40
`rectangle) with associated instance variables. messages. and
`methods. in connection with object-oriented programming.
`FIG. 13 is a table illustrating an exemplary Icon class of
`objects with associated instance variables. messages, and
`methods. with corresponding subclasses of Program Icon 4s
`and Action Handle Icon. in accordance with the invention.
`FIG. 14 is a table illustrating a hierarchy of object classes
`utilized in constructing a computer program that implements
`the preferred embodiment of the present invention.
`
`DETAILED DESCRIPfiON OF THE
`PREFERRED EMBODIMENTS
`Referring now to the drawings. in which lilre numerals
`illustrate like elements throughout the several views, FIG. 1
`illustrates a pen-based computer system 10 for gen