United States Patent 15
`5,491,495
`[11] Patent Number:
`
`[45] Date of Patent: Feb. 13, 1996
`Ward et al.
`
`VANEAA
`
`[54] USER INTERFACE HAVING SIMULATED
`DEVICES
`
`[75]
`
`Inventors: Jean R. Ward, Arlington; David M.
`Barrett, Tyngsboro; Patricia A.
`Martin, Groton; Christopher D.
`Mokoski, Auburn, all of Mass.
`
`[73]
`
`Assignee: Wang Laboratories, Inc., Lowell,
`Mass.
`
`OTHER PUBLICATIONS
`
`Brad A. Myers; “Creating Interaction Techniques by Dem-
`onstration”; IEE CG&A; Sep. 1987; pp. 51-60.
`Randall B. Smith; “Experiences with the Alternate Reality
`Kit:An example of the tension between literalism and
`magic”; IEEE CG&A; Sep. 1989; pp. 42-50.
`Apple Computer, Macintosh Plus Owner’s Guide, 1987, pp.
`72-73, 148-149.
`Baran, “Agilis Hand-Held Workstations: Computing Power
`in the Field”, Byte, Aug. 1989, pp. 91-94.
`Fisher, “New Computer Reads Handwriting”, New York
`Times, Sep. 28, 1989.
`
`(List continued on next page.)
`
`Primary Examiner—Ulysses Weldon
`Assistant Examiner—M.Fatahiyar
`Attorney, Agent, or Firm—Kenneth L. Milik
`
`[57]
`
`ABSTRACT
`
`[21]
`
`[22]
`
`[51]
`[52]
`[58]
`
`[56]
`
`Appl. No.: 613,416
`
`Filed:
`
`Noy. 13, 1990
`
`Tint. C18 oecccsccssssessssssssesesssecssssssessessseessese G09G 3/02
`
`TLS. Ch. oc
`ccccecesseeesseene 345/173; 345/168; 345/112
`Field of Search ou...ccceceseeseessessees 340/706, 707,
`340/710, 711, 712; 178/18; 382/3, 13; 345/156,
`157, 161-168, 173-175, 112, 179, 127,
`145; 395/159
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`FOREIGN PATENT DOCUMENTS
`
`1/1988 European Pat. Off..
`0254561A2
`6/1988 European Pat. Off........... GO6F 3/03
`0271280
`0395469 10/1990 European Pat. Off.
`.
`2127720
`7/1990
`Japan .
`WO08911695
`11/1989 WIPO.
`
`A computer system hating a digitizing tablet overlaying the
`display screen. The tablet serves as a user’s primary input
`device. Various features of the system makeit possible for
`the user to run and interact with standard programs designed
`for keystroke and mouse input and not designed for use with
`a tablet. In addition to the main processor, on which the
`user’s programsare executed, there is an interface processor.
`In addition to a standard display buffer, there is an ink plane
`6/1977 Saich ....ececcccscessescseecsersseeenes 340/711
`4,028,695
`buffer for interface display data that is combined with the
`5/1989 Kaplow et al. oe eeee 340/712
`4,202,041
`data from the standard display buffer on a pixel-by-pixel
`4,562,304 12/1985 Ward et al.
`.
`basis according to data from a mask plane buffer. The
`4,587,633
`5/1986 Wangetal. .
`interface processor manages input from the tablet, presents
`4,602,286
`7/1986 Kellar etal. .
`feedback to the user by meansof the ink and mask planes,
`4,641,354
`2/1987 Fukunagaetal. .
`and provides keystroke and mouse data to the main proces-
`4,803,463=2/1989 Sado oueeeeceessecseeccecesececenseneecees 340/712
`4,839,634
`6/1989 More etal.
`...
`»- 340/707
`sor as if from a standard keyboard controller. The interface
`
`4,859,995
`8/1989 Hamsen et dl. oe. eescssesseeceeee 340/710
`processor presents the user with a collection of simulated
`4,899,136
`2/1990 Beard et al.
`.
`devices, including standard devices such as a keyboard and
`4,901,221
`2/1990 Kodoskyetal. .
`a mouse. A nonstandard simulated device performs character
`4,972,496 11/1990 Sklarew .
`recognition, permitting handwritten characters to be used for
`program input. During interaction with one of the user’s
`programs, the user can activate and deactivate simulated
`devices (by removing them from and returning them to a
`device tray) and can make adjustmentsin their operation and
`location on the screen.
`
`18 Claims, 12 Drawing Sheets
`
`38
`
`a
`MAIN PROCESSOR
`
`
`VGA CONTROLLER AND MEMORY
`
`43
`
`DISPLAY
`
`46
`
`54
`
`KEYBOARD
`INTERFACE
`52
`
`OPTIONAL KEYBCARD
`
`TABLET/STYLUS,
`
`
`
`Valve Exhibit 1057
`Valve Exhibit 1057
`Valve v. Immersion
`Valve v. Immersion
`
`
`meoS
`==
`
`INTERPROCESSOR
`
`MASK PLANE
`
`
`INTERFACE INTERFACE PROCESSOR
`
`
`36
`
`
`TABLET INTERFACE
`
`
`
`
`

`

`5,491,495
`Page 2
`
`
`
`OTHER PUBLICATIONS
`
`Microsoft Corp., “Microsoft Windows Desktop Applications
`User’s Guide”, pp. 63-67, in Microsoft Windows User’s
`Guide, Version 2.0, 1987.
`Schmeupe,“A Pair of Digitizing Tablets”, Macworld, Mar.
`1987, pp. 144-145.
`Ward & Philips, “Digitizer Technology: Performance Char-
`
`acteristics and the Effects on the User Interface”, TEEE
`Computer Graphics and Applications, Apr. 1987, pp. 31-44.
`Berlis & Borden, “Building a Better Mouse”, Macuser, Oct.
`1989, pp. 124-139.
`IBM Technical Disclosure Bulletin, vol. 32, No. 10B, Mar.
`1990, (Armonk, US) “Briefcase icon for take-home and
`disconnected user support”pp. 88-89.
`
`

`

`U.S. Patent
`
`Feb. 13, 1996
`
`Sheet 1 of 12
`
`5,491,495
`
`
`
`

`

`U.S. Patent
`
`Feb. 13, 1996
`
`Sheet 2 of 12
`
`5,491,495
`
`é
`
`ve
`
`*C)|4SNTALS/LIT8VL
`
`QUVOSGASyN 9¢bS
`SOVSYSLN!LIIGVL
`
`
`
`
`GHVOGA3SyINOlLdO
`
`SOVSYSLNI
`
`AV1dS10
`
`ee
`CsSNV1dNI
`
`8v
`
`oP
`
`
`
`YOSS300ddJOVSYHSLNI
`
`
`
`YOSSADONdYSLNI
`
`SOVsYSLNI
`
`BeO¢
`
`AYOWSWGNVYSTIONLNOOVOA
`YOSS390UdNIV
`
`
`
`
`
`

`

`U.S. Patent
`
`Feb. 13, 1996
`
`Sheet 3 of 12
`
`5,491,495
`
`ose
`
`OL
`
`
`
`AYOWSAWV9OA
`
`DOV
`
`ASVW/HNI
`
`AYOWSW
`
`JOVISYSLNI
`
`SOTVNY
`
`VLIVO/TELS
`
`vg
`
`TLS
`
`1W3071dSd
`
`AYOWSN
`
`ydaav‘UU
`
`09
`
`bOE
`
`ip
`
`pL
`
`nNS
`
`LASdIHD
`
`qx'98¢JYLXS
`
`CO-EDVOW
`
`HL
`
`Oe
`
`viva
`
`TLOWot}eee
`YySSINLLdOa
`teosvaeN
`ao0937
`AYOWSWawl—udevsugiine
`WLISIdt9wsen|xso2t08
`JYOD2/Sdmi
`
`YSTIOULNOD
`
`SAIN
`
`VOA
`
`
`
`4S10GUVH
`
`YATIONLNOD
`
`ISS
`
`
`
` >oOQoawngugziaWOudsola
`
`a=3344
`
`go¢e
`
`e838
`
`L3TEVL
`
`S0VAY3SLNI
`
`YATIOWLNOS©
`
`TaTVivuvd
`
`WIS3AS
`
`ed
`
`WVYSOWD
`
`gy
`
`vs
`
`
`
`
`
`
`
`
`
`

`

`U.S. Patent
`
`Feb. 13, 1996
`
`Sheet 4 of 12
`
`5,491,495
`
`L___?
`[>
`_— STATUS
`
`mS GENERATION
`Pe
`r
`
`TT sevaoane
`
`Uy
`
`CONTROLLER
`DATA
`REGISTER
`CPU.ADDR [CPU SIDE DECODER
`415.0) = Pr WR_CNTL A. DATAX oT
`F—WR_CNTILR_STATX=<[TT] SE 120
`
`[—>G0-SETUPX -—J|_ DAT >; na {>
`
`
`eT.OD
`—_
`{>
`in 122
`
`=PoWROLINK.STATX==[TT <<} lee
`_?
`=.
`M/IOX
`LC
`—
`RD_DATAX
`||
`SS oo
`ADL¥
`{>
`Ccmox
`RD.CNTLR_DATAX
`L_ RD CNTLRUSTATX i
`L>
`CHRST
`Dp
`LY»
`=
`=
`= P= RD_LINK DATA sy
`
`B
`KEYBOARD
`CONTROLLER
`STATUS REG
`:
`yi 124
`>.
`|_<—} 126
`
`DATALINK
`DATA
`REGISTER
`c>
`PT 9
`2 12
`
`|)
`KI ei
`
`>
`b—
`
`DATALINK
`STATUS
`REGISTER
`Lp
`C114
`
`cj] 6
`
`<7...
`_
`CPU_DATAS7.
`
`.0)
`
`>
`KEYBOARD_CLOCK
`
`KEYBOARD.DATA
`
`RECEIVER
`
`KEYBOARD
`DATA
`REGISTER
`
`
`
`
`
`KEYBOARD
`STATUS
`REGISTER
`
`
`
`FIG. 4a
`
`KEYBOARD STATUS
`FLAG GENERATION
`
`

`

`U.S. Patent
`
`Feb. 13, 1996
`
`Sheet 5 of 12
`
`5,491,495
`
`STATUS
`FLAG
`
`P
`
`GENERATION
`
` V25 SIDE DECODER
`
`
`
`SEN
`CONT
`CTV25.WRCNTLR_DATAN --——] — 2PICS
`REGISTER
`[5V25-WR-CNTLRALINKX
`<
`..
`120
`C= VZS-WRELINK-DATAX _ V25R/WX —
`ic’ Fe
`V25-WR_KYBD_DATA¥
`Kd
`<_]
`7
`el
`10STBX
`lzee
`an
`J
`\22
`V25_RD.DATAX
`a Leso.omcamSf
`— Y25.F0CTL.DATAX
`RD. LINK_DATAX
`=, V25_RD.KYBD.DATAX
`
`KEYBOARD
`CONTROLLER
`STATUS REG.
`
`rteeens
`BSOo faWm
`
`a
`
`DATALINK
`DATA
`REGISTER
`
`NO nut
`
`12
`
`DATALINK
`STATUS
`REGISTER
`
`4 E>
`
`16
`
`KEYBOARD
`DATA
`REGISTER
`
`130 E>
`rrr
`
`KEYBOARD
`STATUS
`REGISTER
`132
`7s Lo
`
`|_|
`i—d
`/—<
`__
`
`c>
`
`V25_DATAC7..0)
`
`FIG. 4b
`
`

`

`TOMLNOD097
`
`8
`
`¢
`
`GSls
`
`U.S. Patent
`
`Feb. 13, 1996
`
`Sheet 6 of 12
`
`5,491,495
`
`7OYLNOD¥/G!uaisioay|"ONLNOD||GCA|SIg3aoo30||SNSSSAYGIVWIVA!hsnaoz
`
`3a00gd! |VivaIviINasWvEA|o|NAINIOD|YSLNNOD
`
`
`SZAAOMLNODWVYA|454x“JONLNOD|9LuOdSZA|‘WId3siISN@TONLNOD
`
`wivado7|;|jTOMLNOO/waWON9
`szZA||8|VIVOWWHAfe| ss3yaay{*|vsza||SnaSSaxdav
`z,|1uOd:Twiggs|SngVLVASZA
`
`
`lg NaLSIOSY||muHALNNOD|YaLNNOD5|434XNVOS
`
`|INSWS0V1d3yLuOd
`O3CIAVOA
`vivapviva
`
`
`
`
`
`

`

`U.S. Patent
`
`Feb. 13, 1996
`
`Sheet 7 of 12
`
`5,491,495
`
`go71OL
`
`9Sls
`
`VOAWOYS
`
`x N
`
`I
`
`OSI
`
`
`

`

`an
`918SiGvidcidcid118OIG688E4898GSvafhcE
`SILSILvilcllellIllOllL61S8121OLGiplEL@l
`
`JIVHdOLAVIdSId
`
`SIVHWOLLOEAV1IdSId
`
`19
`
`5,491,495
`
`Z 9
`
`Is
`
`U.S. Patent
`
`Feb. 13, 1996
`
`Sheet 8 of 12
`
`

`

`U.S. Patent
`
`Feb. 13, 1996
`
`Sheet 9 of 12
`
`5,491,495
`
`BITMAP
`
`SbdO_
`
`SDI
`
`$b2
`
`SD3
`
`VRAM2
`
`PROGRAM/DATA
`PROGRAM/DATA
`
`BITMAP CONTINUED
`
`SD8
`
`SbDgS
`
`SDIO SDI
`
`--3FFFFH->
`
`<--395FFH->
`
`<--30003H->
`
`<-- 3000IH-->
`
`<--30002H->
`<-30000H->
`
`<~QO000H->
`
`<-3FFFFH—>
`
`<-- 32BFFH-->
`
`<~30000H-->
`
`<--QOO0OH-->
`
`FIG. 8
`
`MBI4|MBI5
`<--39603H->
`<~ 39602H->
`<--3960IH- >
`<- ~39600H->
`
`VRAMI
`
`ITI6
`
`mri2
`Mra
`
`1T5
`
`MTI
`
`PROGRAM/DATA
`
`SD4
`
`SD5
`
`$D6
`
`SD7
`
`VRAM3
`
`
`
`
`
`
`
`
`
`BITMAP CONTINUED
`
`PROGRAM /DATA
`
`SDi2
`
`SDIZ.
`
`SDI4— SspI5
`
`

`

`U.S. Patent
`
`Feb. 13, 1996
`
`Sheet 10 of 12
`
`5,491,495
`
`7FFFFH
`
`—_>
`
`TOSFFH
`
`->
`
`74BO0OH
`
`_—>
`
`7OQO00H
`
`—>
`
`395FFH
`
`—>
`
`34BOOH
`
`30000H
`
`~>
`
`OOOOOH
`
`—_>
`
`PROGRAM/DATA
`
`MASK BOTTOM
`
`MBS MBIO MBI| MBIi2 MBI3 MBI4 MBI5 MBI6
`MBI MB2 MB3 MB4 MBS MB6 MB? MB8
`
`MASK TOP
`
`MTIO MTIl MTl2 MTI3 MTI4 MTIS MTI6
`MT2 MT3 MT4 Mi5 MT6 MT? MTB
`
`PROGRAM/DATA
`
`PROGRAM/DATA IT?
`
`INK BOTTOM
`
`IB9
`IB|
`
`IBIO IBil
`I1B2
`[B35
`
`IBl2 1613
`184
`185
`
`IBl4 IBIS IBI6
`IB6
`IB7
`I[B8
`
`INK TOP
`
`ITiO ITH ITi2 (Tis
`IT3
`174
`ITS
`iT2
`
`ITM ITI5 [Tl6
`IT6
`IT8
`
`FIG. 9
`
`

`

`Feb. 13, 1996
`
`Sheet 11 of 12
`
`5,491,495
`
`
`
`N e
`
`aH
`
`NIN
`SSS
`or
`awe lu
`
`
`
`7@4D4S4OdiusyojuoF!/DSjuo{IDS
`
`
`
`
`
`(diaH)aanPLESLSS
`
`U.S. Patent amewaTeepireOre.rrbrrareberrrryar2}=LELILLY
`{54,097°"tL]=-O68L9SPE"21ZAXMANLSYOdONATI?IH94ARGORV SeeeeeeeeeeeeG4DBI9PuDBuosss
`
`
`rrr?ELIAaTILIALIALIEIRLALLATESaeraeLePowe.ZHVWwdaucisasYA7VETala2SELalwlcfulofafolslof
`
`
`
`
`
`
`
`
`09S14,UodnYs4s0yJYUBNOSGS4ayjD}-4N0}INO‘obosipakuaasspun..-..---asOdSIMOY
`
`
`
`
`
`pajpesaaapuaw|jD4Du;Uo!yIsodoidautOfPAaLDDIPSppud‘Ayaaqi|UlPsAtBsdoUuOD‘UO;yDUMauD4jUusUl4uU
`zy.yTECPPRRRereiSorFel--PPatof?IlslFalol!TYiedsepoqn
`
`
`Sene-i*C]=-O6829StE?71ZAXMANLSYODONWIWT[HOAS0908V
`
`SeeeLeeweEIET()RBVLGHOIzAxmanisibdouw|yf1ybsapoqo
`
`‘uorunyo‘uo|fIDS
`
`
`
`
`
`Leeeeeeeee‘apoosayyOf}padsadyoyayosowjoa@yyubnoupau4
`IYIAIPIIITLALILATITY?STTELELILATATRAMALILAAVEDEPILLLLPIRELTLIVTTILILILTITTREELEDaPIMoe
`
`
`
`beeeneeeeeecesseeeesse+ppnba
`
`
`
`
`
`G@00SSasIMOUSJOYYIMSapylsdywouduoY
`
`

`

`U.S. Patent
`
`Feb. 13, 1996
`
`Sheet 12 of 12
`
`5,491,495
`
`ll‘Sls
`
`2 L
`
`378VL
`
`822
`
`AAveeeee
`
`OINI
`
`
`
`ONILLIGMQNVHASNOW|QUVOEAAm
`
`
`
`
`
`WVY9ONdSSOIAAGG3ALVINWIS
`
`
`
`YOSS300NdJOVAYSLNI
`
`SJYVMWdIS
`
`Ode
`
`Sola
`
`YOSS300Ud
`
`NIVW WVY9Osd
`F
`
`vg!
`
`AV1dSI0
`
`002
`
`NOILVOINIdd¥
`
`
`
`
`

`

`5,491,495
`
`1
`USER INTERFACE HAVING SIMULATED
`DEVICES
`
`CROSS-REFERENCES TO RELATED
`APPLICATIONS
`
`This application is related to the following applications:
`COMPUTER WITH TABLET INPUT TO STANDARD
`PROGRAMS,Martin,et al., (Ser. No. 07/613,324) filed the
`same day as the present application;
`COMPUTER WITH SEPARATE DISPLAY PLANE
`AND USER INTERFACE PROCESSOR,Barrett, et al.,
`(Ser. No. 07/613,325) filed the same day as the present
`application; and
`SYSTEM CLOCK SPEED CONTROLLER,David Bar-
`rett, et al., (Ser. No. 07/611,990)filed Nov. 9, 1990.
`The first
`two of these cases are based on the same
`disclosure as in the presentcase.
`
`FIELD OF THE INVENTION
`
`The present invention relates to the operation of and
`supporting facilities for user interfaces for computer sys-
`tems. The invention is particularly useful in systems where
`a stylus is the user’s primary input device.
`
`BACKGROUND
`
`Driven both by the needs of users and the capabilities of
`various implementation technologies, computers and the
`waysthat people interact with them are redesigned again and
`again.
`In doingso,it is desirable to optimize the design of each
`system according to the current technology and current
`understanding of human factors. However, another, often
`opposing, force shaping the design of computer systemsis
`the need for new systems to be capable of utilizing pre-
`existing components, both hardware and software. This
`force flows from various sources, including the desire to
`continue to exploit large investments in pre-existing designs,
`and the desire to use new systems in environments that
`continue to be populated with pre-existing systems.
`It is particularly advantageous for a new system to be able
`to utilize pre-existing software components, both application
`programs (programs that adapt a system to address real
`world tasks—often, referred to simply as “applications’)
`and system programs(providing system management func-
`tions and services used by application programs). Because
`some pre-existing applications have become de facto stan-
`dards within certain user communities, it is of particular
`importance that a new system be able to run these standard
`applications.
`is
`Most software that provides for user interaction,
`designed for primary user input in the form of keystroke
`data. A growing body of software, especially that designed
`for desktop systems, accepts input from a mouse,in addition
`to keystroke data; generally,
`these systems still require
`keystroke data for certain functions.
`To make computer systems easier for people to use,
`various systems have included stylus-type input devices
`(typically used with a digitizing tablet). Because of their
`similarity to conventional writing implements, stylus-type
`devices are, among pointing devices, particularly well suited
`to be used for character input, in addition to functioning as
`a pointing device. When used, stylus-type input devices
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`55
`
`60
`
`65
`
`2
`have generally been provided in addition to a keyboard. In
`some cases, systems have been designed with a stylus-type
`device as the primary user input device.
`A keyboard has characteristics that differ radically from
`those of a stylus and digitizing tablet. With a keyboard,
`precise character identification is straightforward: a user
`generates data representing alphabetic characters simply by
`pressing corresponding keys. Because a keyboard can be
`used to create a character with a single event (i.c., press of
`a binary switch), all the feedback to the user that is needed
`is display of the character and perhaps accompanied by an
`audible click. In contrast, while a stylus can be used for
`character input, very different feedback to the user
`is
`required. Handwritten characters are generated by a process
`that (although familiar to a user) is much more complex,
`both for the user and for the computer system.
`For making function selections with a keyboard, a wide
`range of alternative approaches are used; ofthese, pointing
`to a visible representation of a function or object to be
`manipulated is awkward, because a keyboard is a very poor
`pointing device.
`A mouseis-often used in systems that permit a user to use
`the mouse to drag objects on the screen. This is accom-
`plished by providing display of a particular type of mouse
`cursor or pointer: this cursor has an absolute location known
`to the system, and the system adjusts the cursor’s location
`according to relative movements of the mouse; this cursor
`functions as an absolute intermediary between the relative
`movements of a mouse and the object drag operations.
`If programs designed to use a mouse cursor are used with
`a stylus that operates directly on the display screen, the user
`might expect to be able to directly manipulate the cursor
`with thestylus, i.e., by placing the stylus over the cursor and
`dragging it. Typically, this works in a way that users are
`likely to find frustrating: the relationship between an amount
`of mouse movement and the distance moved by the cursor
`is not fixed; this may vary from application to application or
`even vary within a single application (e.g.,
`it may be a
`function of velocity); as a result, although the stylus may
`start on the object, the object will not track the position of
`the stylus.
`In a system in which a stylus is used, it is particularly
`advantageousforthe userto be able to usethestylus directly
`with the display. For example, it should be possible to write
`directly on the display screen, and, when appropriate,a trail
`of “ink” should appear on the display in the path taken by the
`stylus, thus appearing to a user muchlike writing on paper.
`A pointing-type input device to be used directly on a
`screen should provide its absolute position (so that the ink
`appears on the screen at the location of the stylus). This
`differs from the input provided by a mouse, which provides
`relative position: with a mouse, the user can generate mouse
`data indicating nothing but a series of rightward movements,
`and still have the mouse end atits starting location (e.g.,
`move a mouseto the fight, pick up the mouse,place it to the
`left of its original position, and move the mouseto the right
`to return to the original locate on). Thus, although both a
`mouse and a stylus are “pointer-type” devices (locators),
`their characteristics differ.
`
`The input of character codes presents another type of
`impediment to running existing (non-stylus) programs with
`a stylus. Input of handwritten characters is a sufficiently
`complex as to require feedbackto the user. Yet a program not
`contemplating handwritten input will have defined a user
`interface with no provision for such feedback.
`These differences in input device characteristics are major
`impediments to effective utilization of much existing soft-
`
`

`

`5,491,495
`
`3
`ware with a stylus-based computer system. As result, the
`practice has been that most of the uses of stylus-based input
`have relied upon application programs written specifically
`for a system that includes a stylus.
`A system built according to the present invention can run
`pre-existing software and permits a user of the system to use
`a stylus to interact with such software.
`
`SUMMARYOF THE INVENTION
`
`A system according to the present invention provides for
`user operation of pre-existing programs with a pointer-lype
`device (in particular, a stylus/tablet) that the pre-existing
`programs were not designed to accommodate. This is
`accomplished by providing a user interface that logically
`overlays, but does not interfere with, the operation of the
`pre-existing program. This user interface provides the user
`with feedback concerning operation of the pointing-device.
`This separate user interface presents to a user a set of
`simulated devices. Each of these is operated by the user as
`a separate input device, similar to how the user might treat
`the various input devices with which the user may already be
`familiar, although these devices have no physical existence
`independent of the computer system on which their mani-
`festations are displayed. For example, a simulated mouse
`device is available to a user; the user observes mouse pad
`and button areas on the display and operates the simulated
`mouse device by meansof the system’s pointer-type device
`(e.g., stylus).
`
`BRIEF DESCRIPTION OF THE DRAWING
`
`The invention is pointed out with particularity in the
`appended claims. The above and other advantages of the
`invention may be better understood by referring to the
`following detailed description in conjunction with the draw-
`ing, in which:
`FIG. 1 is a perspective drawing of a computer system in
`which the present invention is advantageously employed.
`FIG.2 is a block diagram, emphasizing logical organiza-
`tion, of circuitry of a computer system embodying the
`present invention.
`FIG. 3 is a block diagram, emphasizing physical imple-
`mentation, of circuitry of a computer system embodying the
`present invention.
`FIG.4 is a block diagram ofcircuitry used in communi-
`cations between the main processor and the interface pro-
`CESSOT.
`
`FIG. 5 is a block diagram of circuitry in the display
`subsystem.
`FIG.6 illustrates the display data multiplexing circuitry of
`the display subsystem.
`FIGS. 7, 8, and 9 illustrate the correspondence between
`pixel locations on the display screen and the storage of ink
`and mask data. FIG. 7 shows someofthe pixel locations on
`a display screen. FIG. 8 has four memory maps showing
`how the ink and mask data is stored in four VRAM chips.
`FIG. 9 shows how the ink and mask data appears in the
`address space of the interface processor.
`FIG. 10 is a screen display showing three simulated
`device icons: keyboard, handwriting, and mouse.
`FIG. 11 illustrates relationships among the tablet, the
`display, and certain software components.
`
`10
`
`20
`
`_
`
`25
`
`30
`
`35
`
`40
`
`45
`
`55
`
`60
`
`65
`
`4
`DETAILED DESCRIPTION OF AN ILLUSTRATIVE
`EMBODIMENT
`
`1 Platform
`
`1.1 The Computing Environment
`The invention is advantageously used in a computer
`system with which the user interacts by use of a writing
`stylus or other pointing device. When the user can write on
`the display, the keyboard is eliminated (or at least made
`optional), and the computer system is made compact, the
`resulting system is sometimes referred to as a “notebook”
`computer. The present invention will be described in the
`context of such a notebook computer. Based on this descrip-
`tion, other embodiments of the invention will be apparent.
`FIG. 1 is a perspective drawing of a notebook computer
`10. It includes a combination digitizing tablet and display
`12. A stylus 14 operates in conjunction with the digitizing
`tablet to provide X-Y coordinate input to the system. The
`display is implemented using liquid crystal technology; a
`backlight is provided. Along one edge of the tablet area are
`a series of labeled areas 16a, 16b, 16c, and 16d; the system
`is responsive to stylus touches in these areas to initiate
`certain system functions. In certain modes of operation the
`system operates so as to provide electronic ink, in which
`case movement of the stylus on the tablet leaves a trail
`visible on the display, as if the user was writing on a piece
`of paper.
`Other features of this illustrative notebook computer
`include: voice digitizing and playback (a headset/micro-
`phone can be plugged in), data communications (including
`FAX), internal hard disk drive, standard serial and parallel
`ports, a SCSI port connection, among other things, to an
`external floppy disk drive). Instead of the SCSI port, a
`connector can be provided to connect to a bus compatible
`with the micro channel of many of IBM’s PS/2 computers.
`The main processoris an Inte] 80386SX.The core of the
`system is generally compatible with IBM’s PS/2 Model 80,
`Type 1.
`Thoseaspects of the circuitry that are most closely related
`to the present invention are described below, including the
`interface processor, the ink/mask plane portion of the dis-
`play control, and an ASIC (referred to below as the opti-
`mizer) that provides various support logic.
`1.2 Major System Components
`The major hardware components of the computer system
`are illustrated in FIG. 2:
`main processor 30,
`interface processor 32,
`pointing device (tablet/stylus 34 and interface 36),
`main display buffer and controller 38,
`display control circuitry 40, including ink plane buffer 42,
`mask plane buffer 44, and video multiplexer 46,
`display screen 48,
`interprocessor interface 50, and
`optional keyboard 52, connected to the interface proces-
`sor 32 by interface circuitry 54.
`FIG. 3 is a block diagram of the computer system that
`emphasizes the physical implementation of the circuitry,
`much of which is implemented in specialized large scale
`integrated circuit chips.
`1.2.1 Main Processor
`The main processor 30 is implemented using an 80386SX
`microprocessor,
`from Intel Corporation. This processor
`executes the operating system and application programs.
`There is 1 or 4 Mbyte of base main memory 30a for this
`
`

`

`5,491,495
`
`5
`processor, with the option of adding two 1 Mbyte or 4 Mbyte
`modules (thus, providing up to 12 Mbytes). In keeping with
`the portable nature of the notebook computer, the main
`memory uses pseudo-static memory chips controlled in a
`way similar to that described in U.S. Pat. No. 4,710,903.
`There is 128 Kbytes of memory 30b (ROM)for certain
`system service software known as the basic input/output
`system (BIOS).
`1.2.2 Display Screen
`Visual output is provided by a display screen 48. In the
`preferred embodiment, this is a fiat panel display that is
`physically integrated with a digitizing tablet 34 (described
`below). In particular, it is a liquid crystal display with 640
`by 480 pixels and having a switching time.of 250 millisec-
`onds or better.
`1.2.3 Main Display Buffer and Controller
`The main display buffer and display controller 38 (Chips
`& Technologies 82C455 VGAcontroller) are controlled by
`the main processor 30. This appears to the applications
`tumning on the main processor as a conventional video
`display, such as is commonly known as VGA.This provides
`for control of 640 by 480 pixels, which are stored in four
`128K by 8 bit pseudo-static memory devices. This memory,
`is large enough to work with applications written for 256
`colors. Although the pixels of the display screen are binary,
`the VGA controller time multiplexes these to simulate 16
`levels of gray.
`1.2.4 Interface Processor
`The interface processor 32 is a V25 microprocessor,
`available from NEC Electronics Inc. The V25 is similar to
`Intel’s 80188. This processor performs the following func-
`tions:
`connects to main processor and simulates (down to the
`hardware level) a keyboard controller of the type in a
`PS/2-type computer. This protocol provides for data from an
`auxiliary device,
`typically a mouse.
`In addition to the
`keyboard port, another communication channel to the main
`processor is used for other communications.
`connects to the optional physical keyboard and provides
`this keyboard data to the main processor (as would an 8042
`controller in a PS/2-type computer).
`connects to and provides data to the ink and mask planes.
`runs a simulated devices user interface that uses the tablet
`input, simulated keyboard port output, and writes to the ink
`and mask planes.
`runs a handwriting recognition algorithm (a part of the
`simulated devices interface).
`1.2.5 Interface Processor Firmware
`There is 16 Kbytes of ROM in the V25. This ROM
`includes program codeinitialization, diagnostics, support of
`data transfer from the main processor to the interface pro-
`cessor’s main memory (which includes the buffers for the
`ink and mask planes), and keyboard controller emulation
`(permitting use of the physical keyboard prior to download
`of any software into the interface processor’s main
`memory).
`The diagnosticstest the interface processor’s memory, the
`data bus connecting the interface processor to the optimizer,
`and the serial port (including obtaining the status of the
`tablet that is connected to the serial port).
`1.2.5.1 Keyboard Controller Emulation
`Thefirmware provides the functions typically provided by
`an 8042 keyboard controller. In addition to passing key-
`stroke and mousedata to the main processor, these functions
`includeinstalling and testing passwords, reading and writing
`“8042RAM”, and enabling, disabling and testing the physi-
`cal keyboard interface.
`
`6
`The firmware services that send data to the main proces-
`sor across the keyboard port can deal with data being sent in
`three ways: translated data, untranslated data, auxiliary data.
`Translated data is handled as coming from a physical
`keyboard. This data is monitored for changes that affect the
`shift state of the system as perceived by the interface
`processor. This data may be translated from one scan code
`to another, depending uponthestatus of the PC compatibil-
`ity bit of the 8042 command byte.
`Untranslated data is sent to the system without change.
`Untranslated data monitored for changes in shift state; the
`interface processor assumes that any changes in the shift
`state due to the untranslated data will be undone by the
`stream of untranslated data.
`Auxiliary data is sent to the main processor without
`translation. In sending auxiliary data, a bit of the keyboard
`status registeris set, indicating to the main processorthat the
`data is auxiliary data, rather than keyboard data. Auxiliary
`data is typically mouse data.
`1.2.5.2 Services for Simulated Devices
`The firmware provides a variety of services to support the
`use of the keyboard port by simulated devices.
`There are services to disable and enable the flow of data
`from a physical keyboard. When a disable is requested, the
`. service waits until the stream may be interrupted without
`splitting any multibyte keycodes, and then returns with an
`indication of the current shift state.
`The submit buffer service provides for the submission of
`a block of “keyboard” data to be sent. The data is identified
`as translated, untranslated, or auxiliary, and is treated
`accordingly. A pointer to a “when done” routine is provided
`along with the block of data; the service executesthis routine
`when the data in the buffer has been sent. An error is
`returned if anther buffer is currently being sent. If the
`physical keyboard is enabled, the submit buffer service will
`effect a disable physical keyboard before buffer transmission
`and an enable physical keyboard after data transmission.
`An abort buffer service is provided to permit a buffer
`transmission in progress to be canceled.
`A set buffer transmit rate service is provided to control the
`rate at which buffers of keystroke data are sent to the main
`processor. This is provided to control against overflow of the
`keyboard buffer in the main processor. This delay does not
`apply to buffers of 8 bytes orless; this permits the byte string
`for any single keystroke to be sent without delay.
`A service is provided to contro] the clock speed of the
`interface processor. This clock can beset to 2, 4, or 8 MHz.
`The hardware supporting this speed controlis integral to the
`V25 chip.
`A service is provided that will wait for an event from the
`interface processor serial port (connected to the tablet), an
`event from the datalink, or a period of time (specified by the
`caller in milliseconds). The service can wait for any com-
`bination of these three events. While waiting, the service
`will set the speed of the interface processor to its slowest
`speed. When the event occurs,
`the service restores the
`interface processor's speed to its original value and the
`service returns to the caller.
`
`A service is also provided so thatit is possible to chain
`interrupt vectors to learn about communications from the
`system to the keyboard or auxiliary device.
`The firmware also provides a free-running 32-bit counter.
`This counter is incremented once each millisecond. It is also
`possible to chain to these one millisecond interrupts.
`1.2.5.3 Datalink Control
`The firmware also provides some of the support for
`communication over the datalink. This communication is
`described below.
`
`20
`
`25
`
`30
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`

`

`7
`
`8
`
`5,491,495
`
`7
`
`5
`
`4
`
`3
`
`2
`
`1
`
`0
`
`1.2.6 Pointing Device
`Preferably, the pointing device is of a type that provides
`absolute location (e.g., digitizing tablet, light pen). A mouse
`PROX
`0
`0
`0
`SIDE
`TIP
`1
`0
`1
`typically provides relative location (i.e., movement, not
`
`
`
`
`
`
`
`2 X2=OXI0 0 XS x4 X30 xO
`position); a mouse could be used, but with less advantage.
`
`
`
`
`
`
`3 X9=X80 0 xii X10 xX? X6
`
`The pointing device of the preferred embodiment is a
`
`
`
`
`
`
`
`4 Y2=Yi0 0 Y5 Y¥4 Y3 YO
`transparent digitizing tablet and its associated stylus (tablet
`5
`0
`0
`YU
`Y10
`Y9
`Y8
`Y7
`Y¥6
`and stylus together are illustrated in FIG. 2 as 34), such as
`available from Scriptel Corporation.
`An appropriate tablet should have characteristics such as
`the following:
`Absolute accuracy should permit pointing to a single pixel
`in the underlying display; for an underlying display with 100
`pixels/inch, this meansthattotal errors from all sources must
`be less than + 0.005 inch.
`Becausethe tablet overlays the display, it must be trans-
`parent, light diffusion should be minimal, and height of the
`writing surface over the display should be minimized to
`reduce visual parallax.
`Thetablet should be abrasion resistant, so as to withstand
`use of the stylus as well as impact and abrasion from other
`objects.
`The tablet must have sufficient noise immunity to operate
`accurately in close proximity to the display and to other RFI
`sources within the computer.
`The small size of a notebook-type computer makesit
`particularly useful as a portable device; thus, low poweris
`also a useful characteristic.
`Thetablet should beableto digitize at least 120 points per
`second.
`There should be provision for sensing proximity of the
`stylus to the tablet so that the system can display a tracking
`cursor when the user is not writing or dragging, but the
`stylus is near or on the surface; “touching”the tablet should
`be sensed, and this can be implemented bya tip switch. The
`switch force should be such that a user can comfortably
`write with minimal skipping.
`An article titled “Digitizer Technology: Performance
`Characteristics and the Effects on the User Interface” EEE
`Computer Graphics and Applications, April 1987, pp
`31-44) describes digitizer technology.
`A tablet interface 36 transfers X-Y coordinate data from
`the tablet to the interface processor. This interface includes
`an analog front end, a D/A converter, a 68HC11 microcon-
`troller (from Motorola Corporation) and associated logic,
`and a PROM.The PROM contains information that is used
`by the microcontroller to correct the raw tablet data. The
`microcontroller has a serial link to the interface processor
`32.
`The connection between the tablet interface 36 and the
`interface processor 32 is a bidirectional, asynchronousserial
`link operating at 19.2 Kbits/second. The following are the
`most important of the commandsthatthe interface processor
`can send to the tablet controller:
`Reset
`The tablet is reinitialized and tested; success or failure
`code