a2) United States Patent
`US 6,791,536 B2
`(10) Patent No.:
`Sep. 14, 2004
`(45) Date of Patent:
`Keely etal.
`
`US006791536B2
`
`(54)
`
`SIMULATING GESTURESOF A POINTING
`DEVICE USING A STYLUS AND PROVIDING
`FEEDBACK THERETO
`
`(56)
`
`References Cited
`U.S. PATENT DOCUMENTS
`
`(75)
`
`Inventors:
`
`Leroy B. Keely, Portola Valley, CA
`(US); David F. Jones, Redmond, WA
`(US); David Switzer, Redmond, WA
`(US); Michael Hin-cheung Tsang,
`Bellevue, WA (US); William Hong
`Vong, Seattle, WA (US)
`
`(73)
`
`Assignee:
`
`Microsoft Corporation, Redmond, WA
`(US)
`
`Notice:
`
`Subject to any disclaimer, the term ofthis
`patent is extended or adjusted under 35
`US.C. 154(b) by 69 days.
`
`Appl. No.:
`Filed:
`
`09/815,272
`
`Mar.23, 2001
`Prior Publication Data
`
`US 2002/0057263 Al May 16, 2002
`
`(60)
`
`61)
`(52)
`
`(58)
`
`Related U.S. Application Data
`Provisional
`application No. 60/247,844, filed on Nov. 10,
`2000.
`
`Int. Cl.” ..
`US. Cl.
`..
`
`secessessssuusesssssasesssssuussseesiusessses G09G. 5/00
`secetusssssteeesseen 345/173; 345/161; 345/163;
`345/168; 345/173; 345/179; 345/863
`Field of Search ..............0ccceecceeeee 345/173, 156,
`345/161, 163, 167, 168, 179, 863; 178/19.01,
`19.04, 20.01; 703/17, 19, 21-24
`
`4,899,138 A *
`5,404,458 A *
`5,592,566 A *
`5,602,570 A *
`5,666,113 A *
`5,864,635 A *
`5,880,411 A *
`6,049,329 A *
`6,262,719 B1 *
`6,266,050 Bl *
`
`2/1990 Araki etal. 0... 345/175
`4/1995 Zetts voececccsscesseseeseeeeee 710/73
`
`1/1997 Pagallo et al. ........... 382/187
`
`ee 345/173
`.........
`2/1997 Capps et al.
`
`9/1997 Logan... eee 341/34
`
`1/1999 Zetts etal. oo 382/187
`......... 178/18.01
`3/1999 Gillespie et al.
`4/2000 Zetts et al. owe 345/179
`7/2001 Biet al. we 345/179
`
`
`7/2001 Ohetal. we 345/173
`
`* cited by examiner
`
`Primary Examiner—Henry N. Tran
`(74) Attorney, Agent, or Firm—Banner & Witcoff, Ltd.
`
`(57)
`
`ABSTRACT
`
`A method and apparatus for simulating at least one gesture
`of a pointing device such as a mouse. A left click, right click,
`left drag, right drag, and/or mouse movement may be
`simulated using a stylus in conjunction with a touch-
`sensitive display surface. For example, a computer having
`the display surface may detect whether a stylus is being held
`down on a touch-sensitive display surface for at least a
`threshold amount of time. The computer may further detect
`whetherthe stylus is then removed from the touch-sensitive
`display surface after at least the threshold amountof time.
`Responsive to the stylus being removed, the computer may
`generate at least one event representing a right mouse button
`being pressed.
`
`47 Claims, 4 Drawing Sheets
`
`314
`
`rm.
`LEFTDRAG
`STYLUS MOVES
`LEFT_MOUSE_BUTTON_DOWN
`BEFORETIMEO!
`EVENT
`
`
`
`
`
`SUBSEQUENT
`STYLUS MOVEMENT
`
`
`LEFT_MOUSE_BUTTON_DOWN
`EVENT
`
`
`
`
`
`EVENT
`
`VENT ¥
`
`320
`wo
`
`37
`
`
`
`MOUSE_MOVE
`EVENT
`
`
`STYLUS UP
`
`
`
`3
`18
`LEFT_MOUSE_BUTTON_UP
`
`
`
`
`
`HOLD
`
`FEEDBACK
`
`
`
`305
`
`306
`
`RIGHT_MOUSE_BUTTON_DOWN
`
`
`
`SUBSEQUENT
`STYLUS MOVEMENT
`
`310
`
`
`
`
`EVENT
`
`it
`
`307
`MOUSE_MOVE
`BIGHTCLICK)
`rm
`EVENT
`FIGHT_MOUSE_BUTTON_DOWN
`EVENT
`
`
`
`|
`
`DELAY
`
`RIGHT_MOUSE_BUTTON_UP-
`
`
`313.
`
`Valve Exhibit 1011
`Valve Exhibit 1011
`Valve v. Immersion
`Valve v. Immersion
`
`

`

`U.S. Patent
`
`Sep. 14, 2004
`
`Sheet 1 of 4
`
`US 6,791,536 B2
`
`SOL
`
`Ol
`
`Z0L
`
`ELL
`
`~~
`
`NN
`
`~
`
`NN
`
`~
`
`~
`
`“N
`
`OL
`
`
`
`
`
`
`[Avonan|604
`YALNdWOS
`SLOW3Y
`HaZILISIAi
`YOLINOW
`MYOMLAN
`
`[|wagon>"
`901O€L
`YYGC)O61¥$L60g
`sndWALSAS
`XX\
`dAldvav
`
`
`
`
`WMOMIANYauW507qiuasqWOlLdOOLLENOWNauvH
`LLLNOLLVONddvAsnow86hsoug|48!NSHIO96)Nolvonady2°!onvelsdo
`
`
`
`
`
`140dAAGJAMaSiaysiaZLZ\aovsuaini||sovsusini||aovsuant3OV4UAINIOMLaN.
`
`
`
`SWVHDOudWALSASSWWuo0udvivaSON
`
`
`
`
`bblc6L
`
`
`
`
`
`|
`
`
`
`801
`
`OAQIA
`
`OLL
`
`LINN
`
`Z6L
`
`YaHLO
`
`W3LSAS
`
`ONISSADONd
`
`
`|ost(Wve)|
`
`
`
`
`
`‘091sold
`56)ontyuado
`
`961NOILVWOMddV¥
`
`SNVeoOdd
`Wvdo0dd
`SAINGOW
`
`
`
`
`
`AYOWAWWaALSAS
`
`
`
`Ovl(WOH)
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`

`

`U.S. Patent
`
`US 6,791,536 B2
`
`Sep. 14, 2004
`
`Sheet 2 of 4
`
`
`
`
`
`

`

`U.S. Patent
`
`Sep. 14, 2004
`
`Sheet 3 of 4
`
`US 6,791,536 B2
`
`
`STYLUS MOVES
`EFORE TIMEOUT?
`
`
`
`LEFT DRAG
`
`YES
`
`
`
`
`
`304
`
`314
`
`LEFT_MOUSE_BUTTON_DOWN
`EVENT
`
`
`
`
`
`LEFT_MOUSE_BUTTON_DOWN
`EVENT
`
`
`
`STYLUS UP
`BEFORE
`TIMEOUT?
`
`LEFT CLICK
`
`319
`
`NO
`
`HOLD
`FEEDBACK
`
`309
`
`RIGHT_MOUSE_BUTTON_DOWN
`EVENT
`
`
`SUBSEQUENT
`
`
`STYLUS MOVEMENT
`
`
`
`
`
`
`MOUSE_MOVE
`EVENT
`
`
`
`
`
`
`305
`
`
`
`
`
`RIGHT DRAG
`
`310
`
`SUBSEQUENT
`STYLUS MOVEMENT
`
`LEFT_MOUSE_BUTTON_UP
`EVENT
`
`
`
` 313
`
`EVENT
`
`
`
`
`311
`
`RIGHT_MOUSE_BUTTON_UP
`
`FIG. 3
`
`

`

`U.S. Patent
`
`Sep. 14, 2004
`
`Sheet 4 of 4
`
`US 6,791,536 B2
`
`400
`
`401
`
`402
`
`FIG. 4
`
`

`

`US 6,791,536 B2
`
`1
`SIMULATING GESTURES OF A POINTING
`DEVICE USING A STYLUS AND PROVIDING
`FEEDBACK THERETO
`
`CROSS-REFERENCE TO RELATED
`APPLICATIONS
`
`2
`operability as well. The user may generate other gestures
`using the mouse such as by clicking the left or right button
`of the mouse and moving the mouse while the button is
`depressed (a gesture commonly referred to as either a “left
`drag” or a “right drag,” respectively).
`While such mouse gestures are of course easily done with
`an actual mouse, it is not always convenientfora user to use
`The present application claimspriority to U.S. provisional
`a mouse with a computer even though the computer may
`patent application Serial No. 60/247,844, entitled Simulat-
`have a GUI system. For instance, when using a portable
`ing Gestures of a Mouse Using a Stylus and Providing
`system there may not always be a surface available upon
`Feedback Thereto, filed Nov. 10, 2000, which is hereby
`which to place a mouse. Some systems utilize a touch-
`incorporated by reference as to its entirety. The present
`sensitive display screen with a stylus as an input device. The
`application is also related to application Ser. No. 09/760,
`user places the stylus tip upon the touch-sensitive display
`631, entitled Method and Apparatus For Improving the
`screen at various locations to control objects displayed on
`Appearance of Digitally Represented Handwriting, filed
`the display screen. A problem with this type of system is that
`Nov. 10, 2000; to application Ser. No. 60/247,841, entitled
`the stylusis limited in the types of actions that may beeasily
`Highlevel Active Pen Matrix, filed Nov. 10, 2000; to appli-
`performed as compared with a mouse. For instance,
`the
`cation Ser. No. 60/247/973, entitled Selection Handles in
`typical stylus can perform only three types of movements:
`Editing Electronic Documents,filed Nov. 10, 200; to appli-
`placing the stylus tip onto the screen, moving the stylus tip
`cation Ser. No. 60,247,842, entitled Insertion Point Bungee
`across the screen, and removing the stylus tip from the
`Space Tool, filed Nov. 10, 2000; to application Ser. No.
`screen. While a user may operate a remote toolbarto select
`60/247,400, entitled System and Method for Accepting
`a different tool, the excessive movement renders the control
`Disparate Types of User Input, filed Nov. 10, 2000;
`to
`of the system laborious, inconvenient, and slow. Using the
`application Ser. No. 60/247,972, entitled In Air Gestures,
`tip of the stylus-type device, there is presently no convenient
`filed Nov. 10, 2000;
`to application Ser. No. 60/247,831,
`wayto simulate or distinguish between, for example, a right
`entitled Mouse Input Panel WindowsClass List, filed Nov.
`click as opposedtoaleft click as interpreted by the operating
`10, 2000; to application Ser. No. 60/247,843, entitled Mouse
`system.
`Input Panel and User Interface, filed Nov. 10, 2000;
`to
`In order to make the full range of interface features
`application Ser. No. 60/247,479, entitled System and
`accessible to users of such a stylus-based computer,there is
`Method for Inserting Implicit Page Breaks, filed Nov. 10,
`a need for an intuitive way of simulating mouse gestures
`2000;
`to application Ser. No. 09/750,288, entitled
`with the stylus. Although not necessary, it is preferable and
`Classifying, Anchoring, and Transforming Ink, filed Dec.
`more useful that any new way of simulating mouse gestures
`29, 2000;
`to application Ser. No. 09/736,170, entitled
`be compatible with existing software applications that con-
`Highlevel Active Pen Matrix, filed Dec. 15, 2000; and to
`ventionally are used with a mouse.
`application Ser. No. 09/741,107, entitled Mode Hinting/
`Switching, filed Dec. 21, 2000; each of which is incorpo-
`There is also a need for providing helpful feedback to the
`rated by reference herein as to their entireties.
`user of the stylus to indicate whether gestures made with the
`stylus are those that are intended by the user. This is useful
`with a stylus-type input device since there is conventionally
`little to no natural tactile or auditory feedback when using a
`stylus than there is when using a mouse(e.g., the buttons of
`a mouse often have a definitive mechanical “click” feel and
`sound whenpressed). As will be seen, such feedback is even
`more useful when provided in conjunction with various
`gesture simulation techniques of the present invention.
`
`10
`
`15
`
`20
`
`25
`
`30
`
`40
`
`45
`
`SUMMARYOF THE INVENTION
`
`50
`
`55
`
`60
`
`65
`
`As discussed in the various copending patent applications
`incorporated herein by reference, aspects of the present
`invention are directed to a tablet-like computer that may be
`used for directly writing on a touch-sensitive display surface
`using a stylus. The computer may allow the user to write and
`to edit, manipulate, and create objects throughthe use of the
`stylus. Many of the features discussed in these copending
`applications are even more useful
`in combination with
`various aspects of the present invention discussed herein.
`An aspect of the present invention is directed to methods
`and apparatus for simulating gestures of a pointing device,
`such as a mouse, trackball, or joystick, by use of a stylus on
`a touch-sensitive display surface. For example, a left click of
`a mouse may be simulated, e.g., where the user holds the
`stylus down on the touch-sensitive display surface without
`substantial movement and then removesthe stylus from the
`display surface before the expiration of a threshold amount
`of time. A right click of a mouse may be simulated, e.g.,
`where the user waits until at least the threshold amountof
`
`time before removing the stylus. Where the stylus is moved
`
`FIELD OF THE INVENTION
`
`The present invention is directed generally to apparatus
`and methods for simulating various gestures of a pointing
`device with multiple switches, such as a two button mouse
`in a computer system, and for providing feedback thereto,
`and morespecifically to simulating gestures such asthe left
`click of a mouse, the right click of a mouse, and mouse
`dragging by manipulation of a stylus in conjunction with a
`touch-sensitive computer display, as well as generating
`appropriate visual or other feedback in response to certain
`gestures.
`
`BACKGROUND OF THE INVENTION
`
`Typical computer systems, especially computer systems
`using graphical user interface (GUI) systems such as
`Microsoft WINDOWS, are optimized for accepting user
`input from two discrete input devices: a keyboard for
`entering text, and a pointing device such as a mouse with
`two or more buttons or other selectors for driving the user
`interface. Virtually all software applications designed to run
`on Microsoft WINDOWSare optimized to accept user input
`in the same manner. For instance, many applications make
`extensive use of a primary switch of the pointing device such
`as the left button of a mouse (a gesture commonly referred
`to as a “left click” of a mouse) to select an object, and a
`secondary switch such as the right button of the mouse (a
`gesture commonlyreferred to as a “right click” of a mouse)
`to display context-sensitive command menus. It is noted that
`other operating systems incorporate left and right click
`
`

`

`US 6,791,536 B2
`
`3
`along the display surface during certain stylus gestures, left
`or right dragging may thereby be simulated.
`Another aspect of the present invention is directed to
`methods and apparatus for providing feedback to a user in a
`stylus-based touch-sensitive display surface computer sys-
`tem. The user may accordingly be given someindication that
`the user has performed a particular stylus gesture, e.g., by
`displaying a visual object on the display surface to indicate
`a particular status. Such feedback may be responsive to
`certain stylus gestures such as pressing and holding the
`stylus against
`the display surface for at
`least a certain
`minimum amountof time. More generally, the feedback may
`be in the form of a “state change indicator,” which may be
`visual on the display surface such as an icon, bitmap, or
`other visual object or indicator, and/or auditory such as a
`sound emitted from the computer’s speakers. The state
`change indicator may be animated and or otherwise vary
`with stylus hold time. Significant advantages of this aspect
`of the invention are realized in conjunction with the various
`features discussed in the copending patent applications
`incorporated herein by reference.
`These and other features of the invention will be apparent
`upon consideration of the following detailed description of
`preferred embodiments. Although the invention has been
`defined using the appended claims, these claims are exem-
`plary in that the invention is intendedto include the elements
`and steps described herein in any combination or subcom-
`bination. Accordingly, there are any number ofalternative
`combinations for defining the invention, which incorporate
`one or more elements from the specification, including the
`description, claims, and drawings, in various combinations
`or subcombinations.It will be apparentto those skilled in the
`relevant technology, in light of the present specification, that
`alternate combinations of aspects of the invention, either
`alone or in combination with one or more elements or steps
`defined herein, may be utilized as modifications or alter-
`ations of the invention or as part of the invention. It is
`intended that the written description of the invention con-
`tained herein covers all such modifications and alterations.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`The foregoing summary of the invention, as well as the
`following detailed description of preferred embodiments,is
`better understood when read in conjunction with the accom-
`panying drawings, which are included by way of example,
`and not by way of limitation with regard to the claimed
`invention.
`In the accompanying drawings, elements are
`labeled with three-digit reference numbers, whereinthe first
`digit of a reference numberindicates the drawing numberin
`which the element is first illustrated. The same reference
`numberin different drawings refers to the same element.
`FIG. 1 is a schematic diagram of a general-purpose digital
`computing environment
`that can be used to implement
`various aspects of the invention.
`FIG. 2 is a plan view of a tablet computer and stylus that
`can be used in accordance with various aspects of the present
`invention.
`
`FIG. 3 is a flow chart showing an exemplary set of steps
`that may be performed in order to simulate a right click of
`a mouse according to aspects of the present invention.
`FIG. 4 is a plan view ofanillustrative embodiment of a
`trackball having left and right buttons.
`DETAILED DESCRIPTION OF PREFERRED
`EMBODIMENTS
`
`10
`
`15
`
`20
`
`25
`
`30
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`4
`In this
`implement various aspects of the invention.
`embodiment, computer 100 includes a processing unit 110,
`a system memory 120, and a system bus 130 that couples
`various system components including the system memoryto
`the processing unit 110. The system bus 130 may be any of
`several types of bus structures including a memory bus or
`memory controller, a peripheral bus, and a local bus using
`any of a variety of bus architectures. The system memory
`includes read only memory (ROM) 140 and random access
`memory (RAM) 150.
`A basic input/output system 160 (BIOS), containing the
`basic routines that help to transfer information between
`elements within the computer 100, such as during start-up,
`is stored in ROM 140. Computer 100 also includes a hard
`disk drive 170 for reading from and writing to a hard disk
`(not shown), a magnetic disk drive 180 for reading from or
`writing to a removable magnetic disk 190, and an optical
`disk drive 191 for reading from or writing to a removable
`optical disk 192 such as a CD ROMorother optical media.
`The hard disk drive 170, magnetic disk drive 180, and
`optical disk drive 191 are connected to the system bus 130
`by a hard disk drive interface 192, a magnetic disk drive
`interface 193, and an optical disk drive interface 194,
`respectively. The drives and their associated computer-
`readable media provide nonvolatile storage of computer
`readable instructions, data structures, program modules and
`other data for the personal computer 100. It will be appre-
`ciated by those skilled in theart that other types of computer
`readable media which can store data that is accessible by a
`computer, such as magnetic cassettes, flash memory cards,
`digital video disks, Bernoulli cartridges, random access
`memories (RAMs), read only memories (ROMs), and the
`like, may also be used in the example operating environ-
`ment.
`
`A numberof program modules can be stored on the hard
`disk, magnetic disk 190, optical disk 192, ROM 140 or
`RAM 150, including an operating system 195, one or more
`application programs 196, other program modules 197, and
`program data 198. A user can enter commandsand infor-
`mation into the computer 100 through input devices such as
`a keyboard 101 and pointing device 102. Other input devices
`(not shown) may include a microphone, joystick, game pad,
`satellite dish, scanner, or the like. These and other input
`devices are often connected to the processing unit 110
`through a serial port interface 106 that is coupled to the
`system bus, but may be connected by other interfaces, such
`as a parallel port, game port or a universal serial bus (USB).
`A monitor 107 or other type of display device is also
`connected to the system bus 130 via an interface, such as a
`video adapter 108. In addition to the monitor, personal
`computers typically include other peripheral output devices
`(not shown), such as speakers and printers.
`The computer 100 can operate in a networked environ-
`ment using logical connections to one or more remote
`computers, such as a remote computer 109. Remote com-
`puter 109 can be a server, a router, a network PC, a peer
`device or other common network node, and typically
`includes many or all of the elements described above
`relative to computer 100, although only a memorystorage
`device 111 has been illustrated in FIG. 1.
`‘The logical
`connections depicted in FIG. 1 include a local area network
`(LAN) 112 and a wide area network (WAN) 113. Such
`networking environments are commonplace in offices,
`enterprise-wide computer networks, intranets and the Inter-
`net.
`
`FIG. 1 shows an exemplary embodiment of a general-
`purpose digital computing environment that can be used to
`
`When used in a LAN networking environment, the com-
`puter 100 is connected to the local network 112 through a
`
`

`

`US 6,791,536 B2
`
`10
`
`15
`
`20
`
`30
`
`5
`network interface or adapter 114. When used in a WAN
`networking environment, the personal computer 100 typi-
`cally includes a modem 115 or other means for establishing
`a communications over the wide area network 113, such as
`the Internet. The modem 115, which may be internal or
`external, is connected to the system bus 130 via the serial
`port interface 106. In a networked environment, program
`modules depicted relative to the personal computer 100, or
`portions thereof, may be stored in the remote memory
`storage device.
`It will be appreciated that the network connections shown
`are example and other means of establishing a communica-
`tions link between the computers can be used. The existence
`of any of various well-known protocols such as TCP/IP,
`Ethernet, FTP, HTTP and the like is presumed, and the
`system can be operated in a client-server configuration to
`permit a user to retrieve web pages from a web-based server.
`Anyof various conventional web browsers can be used to
`display and manipulate data on web pages.
`FIG. 2 illustrates a tablet PC 201 that can be used in
`
`6
`milliseconds. However, the threshold amountof time may be
`any amount of time, such as between 300 and 600
`milliseconds, or between 600 milliseconds and 1 second, or
`greater than 1 second. If the computer 201 detects that the
`stylus movesprior to the timeout condition occurring (step
`302), then in response the computer 201 may generate a
`standard Microsoft WINDOWS LeftMouseButtonDown
`event (or other event that represents the primary switch of
`the pointing device being activated) in the original position
`where the stylus 204 was first brought down (step 314),
`which may be sent to the software application presently in
`use. At this point, any subsequent movementof the stylus
`204 across the display surface 202 may be detected by the
`computer 201 (step 315), and in response the computer 201
`may generate one or more Microsoft WINDOWS Mouse-
`Moveevents (step 316) (or other events representing move-
`ment of the pointing device). When the stylus 204 is
`eventually removed from the display surface 202 (hereafter
`referred to as “bringing the stylus up”) (step 317),
`the
`computer 201 may detect this and in response may generate
`a Microsoft WINDOWSLeftMouseButtonUp event (step
`accordance with various aspects of the present invention.
`318) (or other event that represents the primary switch being
`Anyorall of the features, subsystems, and functions in the
`deactivated). Thus, via steps 301, 302, and 314-318, a left
`system of FIG. 1 can be included in the computer of FIG. 2.
`drag of a mouseis simulated.
`Tablet PC 201 includes a large touch-sensitive display
`25
`To simulate a left click of a mouse without dragging,if the
`surface 202, e.g., a digitizing flat panel display, for example,
`computer 201 detects that the stylus 204 is not moved prior
`a liquid crystal display (LCD) screen, on whichaplurality
`to the timeout condition (step 302), and if instead the stylus
`of windows203 is displayed. Using a stylus 204, a user can
`204 is brought up prior to the timeout condition (step 303),
`select, highlight, and write on the digitizing display area.
`then in response the computer 201 may generate first a
`Examples of suitable digitizing display pancls include clec-
`LeftMouseButtonDownevent(or other event that represents
`tromagnetic pen digitizers, such as the Mutoh or Wacom pen
`the primary switch being activated), then optionally delay
`digitizers. Other
`types of pen digitizers, e.g., optical
`for a period oftime (step 320) such as about 20 milliseconds
`digitizers, may also be used. Tablet PC 201 interprets marks
`or some other amount of time, and then generate a Left-
`made using stylus 204 in order to manipulate data, entertext,
`MouseButtonUp event (steps 304 and 318) (or other event
`and execute conventional computer application tasks such as
`that represents the primary switch being deactivated). Thus,
`spreadsheets, word processing programs, and the like. The
`via steps 301-304, 318, and 320,a slightly delayedleft click
`touch-sensitive display surface 202 may also be proximity-
`of a mouseis simulated.
`sensitive such thatthe tip of the stylus 204 maybe sensed by
`To simulate a right click of a mouse without dragging,if
`the display surface 202 when proximate thereto without
`the computer 201 detects that the stylus 204 is not moved
`actually contacting the display surface 202.
`either prior to the timeout condition or after the timeout
`The stylus 204 may be equipped with or without buttons
`condition (steps 302 and 305), and if the computer 201
`or other features to augment its selection capabilities. In one
`further detects that the stylus 204 is not brought up until after
`embodiment,
`the stylus 204 may be implemented as a
`the timeout condition (steps 303 and 306), and thus thatthe
`“pencil” or “pen”, in which one end constitutes a writing, or
`stylus 204 has been held down for at least the threshold
`tip, portion and the other end constitutes an “eraser” end
`amount of time, then in response the computer 201 may
`that, when movedacrossthe display indicates portionsof the
`generate first a Microsoft WINDOWSRightMouseButton-
`display that are to be erased. The term “stylus” as used
`herein is intended to have a broad definition that includes,
`Down event (step 307) (or other event that represents the
`secondary switch of the pointing device being activated).
`€.g.,
`a user’s ownfinger, a pen, a pencil, a stick, a rod, and/or
`The computer 201 may,further in response, optionally delay
`any other similar device suitable for pointing at a particular
`for a period oftime (step 308), in one embodiment, about 20
`location on the display surface 202. Other types of pointing
`milliseconds, and then generate a Microsoft WINDOWS
`devices, such as a mouse,trackball, or the like may be used
`RightMouseButtonUp event (step 313) (or other event that
`with the tablet PC 201 in addition to or conjunction with the
`represents the secondary switch being deactivated). The
`stylus 204.
`computer 201 may also generate and display a state change
`Referring now to FIG. 3, an example of howaleft drag
`55
`indicator (step 319) that indicates to the user that the stylus
`of a mouseis simulated is now discussed. Anyorall of steps
`204 has been placed downfor a certain minimum amount of
`set forth herein and in FIG. 3 may be performed by special
`time, as will be discussed below in more detail.
`software, hardware, and/or firmwarethat runs in conjunction
`It is noted that steps 319 may be in a variety of places
`with, or is part of, the operating system of the computer 201.
`within the flowchart of FIG. 3 other than the one shown,
`To simulate a left drag, the user may touch the stylus 204 or
`such as between steps 305 and 306. Steps 301-303,
`other similar user pointing device to the display surface 202,
`305-308, and 313, thus simulate a delayed right click of a
`an action referred to hereafter as “placing the stylus down”
`mouse.
`(step 301). Responsive to the computer 201 detecting the
`stylus being placed down,
`the computer 201 may begin
`counting time, e.g., by using a timer, up to at
`least a
`threshold amount of time. A timeout condition is defined
`responsive to the threshold amountof time having passed. In
`one embodiment, the threshold amountof time is about 600
`
`40
`
`45
`
`50
`
`60
`
`65
`
`To simulate a nght drag of a mouse, if the computer 201
`detects that the stylus 204 is not moved until after the
`timeout condition (step 305), then in response the computer
`201 may generate a RightMouseButtonDown event (step
`309) (or other event that represents the secondary switch
`
`

`

`US 6,791,536 B2
`
`7
`being activated). Upon the computer 201 detecting any
`subsequent movement of the stylus 204 along the display
`surface 202 (step 310), the computer 201 may in response
`generate one or more MouseMove events (step 311) (or
`other events representing movementof the pointing device).
`When the computer detects that the stylus 204 is brought up
`(312), in response the computer 201 may generate a Right-
`MouseButtonUp event (step 313) (or other event that rep-
`resents the secondary switch being deactivated). This
`simulates, via steps 301-303, 305, and 309-313, a right drag
`of a mouse.
`
`As a further feature of the right click simulation, a special
`hold event may be defined wherein the stylus 204 is held
`against the display screen 202 without substantial movement
`for at least another threshold amountof time, say about 700
`milliseconds, or between 0.5 and 1 second,or between about
`1 and 3 seconds or more, the computer 201 may detect such
`a hold event and in response act by immediately interpreting
`any subsequent holding of the stylus 204 as holding the left
`mouse button down (or, more generally, as activating the
`primary switch of the pointing device). Thus, the computer
`201 may, after 600 milliseconds of stylus holding, go on to
`steps 305 and 319, and wherethe stylus 204 is released prior
`to, say, 5 seconds or some other time limit,
`then steps
`306-308 and 313 may be implemented. However,if instead
`the stylus 204 is further held at least 5 seconds, then the
`special subroutine may, in response, immediately generate a
`LeftMouseButtonDownevent(or other event that represents
`the primary switch of the pointing device being activated).
`Then, when the stylus is eventually released, a LeftMouse-
`ButtonUp event (or other event that represents the primary
`switch being deactivated) would be generated by the special
`subroutine. If the stylus 204 is moved along the display
`surface 202, then the computer 201 may immediately in
`response generate one or more MouseMoveevents(or other
`events representing movement of the pointing device).
`Whenthe stylus 204 is eventually removed from the display
`surface 202, all pending RightMouseevents (or other events
`that represent action by the secondary switch of the pointing
`device) that would normally have been generated in accor-
`dance with the method shown in FIG. 3 would be canceled.
`
`10
`
`15
`
`20
`
`25
`
`30
`
`40
`
`8
`feel the click of a mouse button to ensure that a click has
`been performed, the user of the stylus 204 may not receive
`clear enough feedback that the intended gesture has been
`madeproperly. Also, it is desirable to indicate to the user that
`a particular requested mode has really been switched to,
`such as selection mode. Further, where the user must hold
`down the stylus 204 for at least a certain threshold amount
`of time to simulate, for example, a right click, feedback to
`the user indicating that the threshold time has passed would
`be useful.
`
`Such feedback may be in the form of a state change
`indicator, which may be visual (e.g., a graphical icon, a
`symbol, a bitmapped or other image, etc., displayed on the
`display surface 202 such as state change indicator 205 in
`FIG. 2), auditory (e.g., a click, a beep, tone, etc. emanating
`from the computer 201 and/or stylus 204), and/ortactile
`(e.g., vibration of the computer 201 and/or stylus 204). For
`example, where the user selects the pen icon in the appli-
`cation toolbar (as describedin at least one of the co-pending
`patent applications incorporated herein by reference), the
`user may drag the stylus 204 over a documentdisplayed on
`the display surface 202 such that digital ink is generated on
`the display surface 202. At this point, the user may switch to
`selection mode by pressing and holdingthe stylus 204 for at
`least a threshold amountof time that may be the sameas the
`threshold amount of time or that may be different. For
`instance, the threshold minimum amount oftime that the
`user must hold the stylus 204 against the display surface 202
`may be, in one embodiment, about 400 milliseconds, or
`between about 400 milliseconds and about 1 second, or more
`than about 1 second. Responsiveto the user holding without
`substantial movement the stylus 204 against
`the display
`surface 202 for the minimum threshold amountof time, the
`computer 201 may generate a state change indicator on the
`display surface 202 which may be located at a location
`depending uponthe location of the tip of the stylus 204. For
`example, the state change indicator may be located ator near
`the location of the tip of the stylus 204.
`In some embodiments, the state change indicator may be
`animated and/or may otherwise change over time. For
`instance,
`the state change indicator 205 is shown as a
`partially completed clock-like loop that may become more
`complete over time in, e.g., a clockwise direction. In such
`embodiments with animation,the state change indicator may
`actually begin prior to the threshold amount of time passing,
`as an indication to the user that the threshold amountof time
`
`In the above and remaining discussion, it is assumed that
`a Microsoft WINDOWSoperating system is being used with
`the computer 201. However, any operating system, espe-
`cially a GUI operating system, may be used in accordance
`with the present
`invention. Further, although the above
`will soon pass. For instance, where a first threshold amount
`embodiments are sample methods for simulating mouse
`of time is 1 second, holding the stylus 204 against the
`gestures using a stylus, there are many variations that are
`display surface 202 for 500 milliseconds may cause an icon
`encompassed by the present invention. For instance,
`the
`50
`(a state change indicator) in the shape of a wandtip to appear
`stylus gestures required for simulationaleft click and a right
`on the display surface 202 at or nearthe location ofthe stylus
`click might be reversed, or a user may not need to always
`204. As the stylus 204 is continued to be held, the icon may
`keep the stylus 204 down onthe display surface 202 during
`animate and begin to glow to indicate that a gesture is
`the entire time that a drag is executed, such thatafirst
`beginning to form. At the first threshold amountof time, the
`touch/touch-and-hold of the stylus 204at a first location on
`animation may have rea