United States Patent [191
`Greanias et a1.
`
`[54] ADVANCED USER INTERFACE
`
`[75]
`
`[21]
`
`I21]
`
`[51]
`[52]
`[53]
`
`[56]
`
`Inventors: Evon C. Greanias, Chevy Chase,
`Md.; Guy F, Verrier, Reston, V3,;
`Gordon W. Arbeitman, Gaithersburg;
`Alan R. Tannenbaum, Washington
`Grove, both of Md.; Yu L. An,
`vlennai Va-
`_
`.
`_
`_
`Asslgneei lntel'natlfmal 3115111855 Machines
`Corporation, Armonk, N-Y-
`
`Appl No‘: 344,379
`
`Filed:
`
`Apr. 28, 1989
`
`Int. Cl.5 ............................................. .. G09G 5/00
`U.S. CI. ......................... ._ 340/706; 340/712
`Field of Search .............. ._ 340/706, 712; 364/188,
`364/189, 190; 178/18
`
`References Cited
`v
`U'S' PATENT DOCUMENTS
`4.451.895 5/1984 Sliwkowski ...... ............... _. 364/521
`4,587.630 5/1986 Straton et a1. .
`.... .. 340/706
`4‘689761 3/1937 Yurchenco
`~ 364/708
`2,2522;
`lsrrgessl ................................ .. 381/41
`‘
`J’
`a 0.‘
`4,899.138 2/1990 Aralu et a]. ....................... ,. 340/712
`4 903 012 M1990 Ohuchi _________ n
`“ 340/712
`4,908,612 3/1990 Bromley et al.
`.0 340/706
`4931.950 6/1990 Isle e131. ........................... .. 364/188
`4.972.496 ll/l990 Sklarew .
`
`IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII
`USOO5157384A
`Patent Number:
`5,157,384
`Oct. 20, 1992
`Date of Patent:
`
`[11]
`[45]
`
`OTHER PUBLICATIONS
`“Multimedia User Interface for Office Workstation” By
`Ishii, et a1. Euromicro, pp. 433-440, 1985.
`P,_-
`E -
`__J 5- r A Brier
`"nary xammer e e y '
`[57]
`ABSTRACT
`An advanced user interface for use with a computer
`system operating on an integrated operating environ
`ment. The integrated operating environment allows a
`plurality of application programs to be running simulta~
`neously, one of which is designated the active applica
`tion program to which all input data is directed. The
`advanced user interface allows a user to select among
`user-friendly input devices to operate any application
`program according to his individual preferences with
`out change to the application program code- The ad
`vanced user interface includes alternate input modules
`which translate the input signals transmitted from the
`various input devices into input messages useable by the
`rest of the interface. The advanced user interface also
`includes interface pro?les which contain mappings of
`the input messages against corresponding commands
`useable by the application programs, the integrated
`operating environment or other modules of the ad
`vanced user interface itself. An environment link mod
`ule refers to the interface profiles and matches the input
`message against the corresponding command for the
`application program active at the time the input signal
`.
`.
`was transmitted and send the corresponding command
`‘0 that aPPllcanon Program
`
`47 Claims, 7 Drawing Sheets
`
`I
`
`I
`
`111
`
`j
`
`+113
`
`1,115
`
`117
`_’
`
`,AIQ
`
`SEND ELECTRICAL SIGNALS FROM INPUT DEVICE
`TO RECOGNITION UNIT
`I
`SEND INPUT SIGNAL TO A.1.s.
`I
`SEND INPUT MESSAGE TO ENVIRONMENT LINK
`I
`QUERY OPERATING ENVIRONMENT FOR
`ACTIVE APPLICATION
`I
`REFER TO APPLICATION PROFILE FOR
`CORRESPONDING COMMAND
`I
`REFER TO USER PROFILE FOR CORRESPONDING COMMAND /'121
`I
`DETERMINE WHICH COMMAND HAS HIGHER PRIORITY
`I
`SEND HIGHER PRIORITY COMMAND TO ACTIVE APPLICATION ,v125
`I
`ACTIVE APPLICATION PROGRAM TAKES ACTION
`
`/_123
`
`__,i..127
`
`Petitioners Ex. 1003 Page 1
`
`

`

`US. Patent
`
`Oct. 20, 1992
`
`Sheet 1 0f 7
`
`5,157,384
`
`FIG. 1PRIOR ART
`
`APPLICATION
`A
`50
`
`APPLICATION
`B
`51
`
`APPLICATION
`C
`52
`
`APPLICATION
`D
`53
`
`INTEGRATED OPERATING ENVIRONMENT
`56
`
`DISPLAY
`28
`
`OPERATING
`SYSTEM
`54
`
`BIOS
`70
`
`DRIVER
`A
`60
`
`B
`62
`TOUCH
`DRIVER
`s3
`
`DEVICE A
`32
`
`DEVICE B 34
`
`TOUCH SENSOR
`as
`
`Petitioners Ex. 1003 Page 2
`
`

`

`2
`
`9
`
`483,7.b
`
`amu
`
`0,2520
`
`.m88mm58mm
`s:88ozEEEo
`i,awwmmm48?BEEP;
`mm5§<zo#555m8mmo
`2mm8_vm8<
`
`m<EEO555@2530zoEoEnz
`m0_O>mmD7zoEBEnz
`
`.m8EZEEmzzomsfi285.4%:
`
`mmmZmozo:.<o_._mn_<
`
`3
`
`88mm
`
`
`
`oEEmeEz5m
`
`
`
`Em:omoz<>a<
`
`Petitioners Ex. 1003 Page 3
`
`0
`
`U.S. Patent
`
`N.0.“—
`
`v553mg;85>:88mosma858359%:8Eu
`gNV838<0mm8a8
`momzmwmowzmmmowzmmm<woa
`
`
`
`
`Petitioners Ex. 1003 Page 3
`
`
`

`

`US. Patent
`
`Oct. 20, 1992
`
`Sheet 3 of 7
`
`5,157,384
`
`.1
`
`8mmmeg.
`
`8:2:
`
`3%$9988
`
`we
`
`
`
`
`
`mofimmhz.5%.$22meA?375mm
`
`mop
`
`mww:
`
`mjiomm
`
`n9
`
`kszzomSzm
`
`2:v2:
`
`mjlomm.
`
`mm:23
`
`commwo_o>mo:2:00mm
`
`mmko<m<Io
`
`Commmmahmmw
`
`«o.22:
`
`zo¢<oan<
`
`._.:.<
`
`
`
`mo.>.:.=._.D
`
`mmmowzmw
`
`1029.X”mmoSwo
`
`I<moSmo
`
`no
`
`
`
`memoIQDOH
`
`2mos
`
`
`
`
`
`HszzomSzmwzF/Ewmoomzmomfiz
`
`mmoz..Em<
`
`mmo2.4%?
`
`
`
`Emam<z..En_<z..Em<
`
`ow<mw>Eo
`
`mm
`
`wzfiémmo
`
`‘I2mm>waw
`
`><.Em_o
`
`mowzmmwo<§
`
`ow
`
`wo_o>
`
`anmowzww
`
`Petitioners Ex. 1003 Page 4
`
`Petitioners Ex. 1003 Page 4
`
`
`
`
`
`
`
`
`
`
`
`

`

`US. Patent
`
`Oct. 20, 1992
`
`Sheet 4 of 7
`
`5,157,384
`
`FIG. 4
`
`SEND ELECTRICAL SIGNALS FROM INPUT DEVICE
`TO RECOGNITION UNIT
`I
`SEND INPUT SIGNAL TO A.I.S.
`W
`SEND INPUT MESSAGE TO ENVIRONMENT LINK
`I
`QUERY OPERATING ENVIRONMENT FOR
`ACTIVE APPLICATION
`
`\~-
`
`---113
`
`/,115
`
`‘
`
`I
`REFER TO APPLICATION PROFILE FOR
`CORRESPONDING COMMAND
`I
`REFER TO USER PROFILE FOR CORRESPONDING COMMAND /'121
`
`/,I‘I9
`
`I
`DETERMINE WHICH COMMAND HAS HIGHER PRIORITY
`I
`SEND HIGHER PRIORITY COMMAND TO ACTIVE APPLICATION /v125
`
`,423
`
`I
`ACTIVE APPLICATION PROGRAM TAKES ACTION
`
`__....-I27
`
`FIG. 5
`
`INPUT
`MESSAGE
`
`GS ,Cmc
`
`COMMENT ON
`USER ACTION
`
`COMMAND
`
`COMMENT ON
`COMMAND
`
`CIRCLE GESTURE
`ON TOUCH SENSOR
`
`KEYBD-
`UTI-
`
`INVOKE THE POP-UP
`KEYBOARD
`
`VS/CUT ‘CUT’ COMMAND TO
`VOICE SENSOR
`
`__ CUT”
`
`GS/LA
`
`LEFT ARROW
`GESTURE 0N
`TOUCH SENSOR
`
`SEND A MENU SELECT
`MESSAGE TO SELECT
`'CUT‘ OPTION
`EXECUTE. ‘FORM FILLER‘
`MACRO WHICH CONTAINS
`ASEQUENCE 0F COMMANDS
`AND UTILITIES
`
`Petitioners Ex. 1003 Page 5
`
`

`

`US. Patent
`
`0a. 20, 1992
`
`Sheet 5 of 7
`
`5,157,384
`
`| _ ” _ _ _ - _ _ '1
`
`APP E
`150
`
`APP F
`151
`
`APP I-I
`153
`
`APP G
`152
`r
`
`PM- LINK 201
`
`l
`
`I
`
`_ _. _ _ _ _ _ _ _ _ _ __ ..._ _ _ _- -- - >- — -— ‘ _ _ _ _ -- - ‘
`
`A.u.I.
`I
`II" UTILITIES 209
`:
`APPLICATION :
`PROFILE 205
`
`I
`
`I
`
`I
`I
`I
`
`'
`
`I
`'
`I
`I
`
`I
`I
`I
`
`I
`
`I
`
`I
`I
`'
`l
`
`L _ _ _ _ _ _ _ _ _ _ _ _ __ __ W L l
`I
`I
`USER
`I
`I
`I
`PROFILE 207
`I
`I
`204 j " —
`
`PRESENTATION MANAGER 15s
`l
`
`_ _ __ _l
`
`08/2 154
`II
`
`II
`
`'
`I
`|
`
`_ _ _ _ _ _ _ _ _ _ _ _ _.
`
`l
`I
`
`I
`I
`I
`I
`
`ALTERNATIVE INPIIT
`SUBSYSTEM
`203
`II
`
`II
`
`II
`
`I
`
`CHAR REG GEsTuRE
`211
`REG 213
`
`VOICE REC
`
`215
`
`IMAGE REC
`217
`
`I______.______ _ _ _ _ _ _ _ _ _ _ _ _ __ __J
`
`1200
`
`DRIVER
`140
`
`II
`
`D80
`141
`
`II
`
`D80
`142
`
`II
`
`D80
`143
`
`II
`
`D80
`144
`
`II
`
`KEYBOARD
`130
`
`MOUSE
`132
`
`TOUCH
`SENSOR 134
`
`VOICE
`SENSOR 136
`
`IMAGE
`SENSOR 138
`
`Petitioners Ex. 1003 Page 6
`
`

`

`US. Patent
`
`Oct. 20, 1992
`
`Sheet 60f 7
`
`5,157,384
`
`SEND ELECTRICAL SIGNALS FROM INPUT DEvICE
`TO DEVICE SPECIFIC CODE
`
`221
`. ?/
`
`II
`SEND INPUT SIGNAL TO A.I.S.
`
`I
`SEND “MOUSE” INPUT MESSAGE TO OS/2 AND
`PRESENTATION MANAGER
`
`"
`INTERCEPT MESSAGE TO APPLICATION
`
`,_ 223
`
`225
`,
`
`227
`v
`
`OUERY A.I.S. TO DETERMINE TRUE INPUT MESSAGE
`
`__,229
`
`SEND TRUE INPUT MESSAGE TO APPLICATION
`
`,/231
`
`I
`RECEIvE MESSAGE FROM APPLICATION
`
`—--233
`
`235
`
`I _
`
`DID APPLICATION UNDERSTAND
`TRUE INPUT MESSAGE?
`
`END
`
`237
`
`REFER TO RECOGNITION UNIT?
`23S
`SEND TOKEN TO PM-LINK /
`I
`REFER TO APPLICATION PROFILE FOR CORRESPONDING COMMAND -—241
`
`II
`REFER TO USER PROFILE FOR CORRESPONDING COMMAND \L243
`
`II
`DETERMINE WHICH COMMAND HAS HIGHER PRIORITY
`
`---245
`
`I
`SEND HIGHER PRIORITY COMMAND TO ACTIVE APPLICATION
`
`——247
`
`II
`ACTIVE APPLICATION TAKES ACTION
`
`249
`~—-/
`
`Petitioners Ex. 1003 Page 7
`
`

`

`US. Patent
`
`0a. 20, 1992
`
`Sheet 7 of 7
`
`5,157,384
`
`SEND ELECTRICAL SIGNALS FROM INPUT DEvICE TO DEvICE SPECIFIC CODE -—-—261
`I
`SEND INPUT SIGNAL TO A.I.S.
`Y
`SEND 'MOUSE“ INPUT MESSAGE TO OS/2 AND PRESENTATION MANAGER ,/265
`1
`INTERCEPT MESSAGE TO APPLICATION
`6
`OUERY A.I.S. TO DETERMINE TRUE INPUT MESSAGE
`i
`REFER TO USER PROFILE FOR TIME DELAY PERIOD
`+
`COLLECT TOUCH INPUT POINTS IN STROKE UNTIL STYLUS LIFTOFF ,-_ 273
`
`,/263
`
`‘~26?
`
`__I- 269
`
`,__ 271
`
`“PERIOD OF
`STABILITY"?
`YES
`
`275
`
`EXCEEDS “ NO
`TIME DELAY?
`
`YES
`
`277
`
`"
`
`279
`RECOGNIZE TOUCH INPUT STROKE AS GESTURE OR CHARACTER _/
`
`II
`
`GENERATE MOUSE COMMAND
`
`<———
`
`+
`SEND TRUE INPUT MESSAGE TO APPLICATION
`T
`RECEIvE MESSAGE FROM APPLICATION
`
`283
`,/
`
`~42“
`
`287
`
`DID APPLICATION UNDERSTAND TRUE
`
`REFER TO APPLICATION PROFILE FOR CORRESPONDING COMMAND ,/2s9
`i
`REFER TO USER PROFILE FOR CORRESPONDING COMMAND //291
`I
`DETERMINE WHICH COMMAND HAS HIGHER PRIORITY
`
`../293
`
`*
`SEND COMMAND TO A.U.I. UTILITIES
`
`.
`—" 295
`
`Petitioners Ex. 1003 Page 8
`
`

`

`1
`
`ADVANCED USER INTERFACE
`
`5
`
`15
`
`25
`
`BACKGROUND OF THE INVENTION
`This invention relates to improvements in entering
`input data into digital computers. More particularly, it
`relates to an advanced userinterface which allows a
`user to select one or more input devices to input data
`into a computer running a program originally written
`for a different input device in a multiapplication envi
`ronment.
`In the past, computers were used only by scientists,
`mathematicians, and other high-level, sophisticated
`computer users. As computer technology progressed,
`and particularly with the advent of the personal com
`puter, data processing has reached every level of soci
`ety, and every level of user. The trend is for fewer
`computer users to be computer professionals or sophis
`ticated in data processing techniques. Access to com
`puters will increase even further in the future as com
`puter hardware and software increase in power and
`ef?ciency.
`However, despite the prevalence of computers in our
`society, many are apprehensive about using them. The
`traditional text display and alphanumeric keyboard
`input device are not readily learned, requiring the mem
`orization of the correct format of dozens, if not hun
`dreds, of commands. Unless the user is a reasonably
`good typist, data entry can be inconvenient and time
`consuming. The hours of experimentation which may
`be necessary to become pro?cient with a single piece of
`software becomes extremely frustrating for the unso
`phisticated user who is likely to avoid the effort alto
`gether.
`It has therefore become necessary to design what
`have become known in the art as “user friendly” input
`devices and computer programs. Such “user friendly”
`devices and software are designed to allow an unsophis- ‘
`ticated user to perform desired tasks without extensive
`training. One of the more popular input devices is the
`mouse pointing device. The mouse generates signals
`corresponding to the direction and distance it is moved
`across a flat surface by the user. The computer responds
`to this information by changing the position of the cur
`sor on the computer display to allow the user to point to
`any displayed object. Once the cursor is correctly posi
`tioned, the user can perform a number of functions by
`depressing one or more buttons on top of the mouse.
`Human factor studies have shown that a device
`which allows the user to input data directly on the
`visual display screen of a computer, generally known in
`the art as a touch input device, achieves greatest imme'
`diacy and accuracy between man and machine. One of
`the first input devices for use at the display surface was
`the light pen. The light pen is an optical detector in a
`hand held stylus, which is placed against the face of a
`cathode ray tube. The location of the light pen is deter
`mined by detecting the coordinates of the dot of light
`which is the scanning raster of the display. A second
`type of touch input device is a mechanical deformation
`60
`membrane which is placed over the display screen. The
`membrane is a transparent overlay which consists of
`two transparent conductor planes disposed on a ?exible
`surface. When a selection is made, the user mechani
`cally displaces one of the conductor planes to touch the
`other by a ?nger or stylus touch, thereby bringing the
`conductors into electrical contact with each other. Ap
`propriate electronics and software translate the electri
`
`5,157,384
`2
`cal signals generated by the ?nger or stylus touch to the
`position on the display surface. Another touch input
`device is a capacitive transparent overlay placed over
`the display screen, which includes transparent conduc‘
`tors driven by an electromagnetic signal. The input
`device can detect the location of a ?nger touch by the
`change in capacitance of the overlay or, alternately, a
`stylus is used to return the electromagnetic signals from
`the overlay back to the computer to determine the sty
`lus position. Yet another touch input device uses a
`frame which ?ts around the display screen having a
`number of infrared or visible light transmitters and
`receptors arranged in parallel horizontal and vertical
`directions. When the user’s ?nger blocks the light
`beams, the horizontal and vertical receptors note the
`absence of the signals, thereby locating the position of
`the action desired by the user.
`Many other user friendly input devices which re’
`spond to actions of the user such as voice recognition
`units or digital tablets, have been developed.
`In addition to user friendly input devices, designers
`have made efforts to develop more user friendly soft
`ware. One technique which has been employed is to
`provide the user with a menu of choices of the particu
`lar tasks or functions which can be performed. In this
`way, the user is not required to commit long lists of
`commands to ‘memory. The menu can be a full or partial
`screen display with spaces adjacent to the menu entries
`to which the cursor is moved by keyboard or by other
`cursor moving device to select a particular action. A1
`tematively, the user can select an action by entering an
`alphanumeric character associated with the menu selec
`tion on a command line.
`Another recent trend is to provide some sort of inte
`gration of computer program applications. Without
`integration, the user must employ separate application
`programs for word processing, database manipulation,
`graphics and electronic mail functions, and so forth. It is
`often quite difficult to integrate the outputs of the differ
`ent programs into a single desired output. One solution
`has been to write a single integrated piece of software
`which incorporates a variety of applications which is
`called a multiple-function program. Typically, these
`multiple-function programs include text, spreadsheet,
`and business graphing applications.
`Another approach is to provide an integrated operat
`ing environment as implemented by Microsoft WIN
`DOWS TM or IBM Presentation Manager TM. In this
`approach, individual application programs share infor
`mation and often appear on the display screen at the
`same time, each in its own window. By selecting the
`strongest individual application programs, a much more
`powerful environment can be tailored for the user’s
`particular needs, in contrast to multi‘function programs
`where the user is limited to whatever applications have
`been programmed into the package.
`Unfortunately, while many user friendly input de
`vices and many user friendly computer programs have
`been developed, these efforts have not been well inte~
`grated. For example, there are a large number of pro
`grams which have been written to accept keyboard or
`mouse input which do not recognize information from a
`touch input device or voice recognition unit. This situa
`tion presents a severe obstacle to the introduction and
`widespread use of new user friendly input devices. Ei
`ther a programmer must make extensive revisions to a
`large body of existing software, or must write a new
`
`35
`
`45
`
`65
`
`Petitioners Ex. 1003 Page 9
`
`

`

`5,157,384
`3
`multiple function program, or set of application pro
`grams each time a new input device is developed. Ei
`ther alternative adds signi?cantly to the costs and time
`of a development effort and may be beyond the capabil
`ities of the organization developing the input device or
`render the effort too expensive for the expected bene
`?ts. In addition, copyrights or other proprietary rights
`may prevent the input device developer from adapting
`existing software. Further, newly developed software
`may not be as ef?cient or commercially acceptable to
`users as the established application programs already
`written for a particular purpose.
`A ?nal consideration is that the individual user is
`probably best suited to assessing his own needs, yet is
`least likely to be able or interested in undertaking exten
`sive revisions in application programming. It would be
`advantageous to allow a user to select the particular
`input devices and application programs which would
`best serve his purposes and yet with only a relatively
`low level of expertise, allow him to assemble the com
`ponents in a user friendly interface between himself and
`the computer. The general solution developed by the
`programmer may not be optimum for a particular user’s
`needs. It would also be advantageous to assure a user
`that future advances in input devices and application
`programs can be accommodated by the interface with a
`minimum of revision.
`
`4
`ent manner from the way in which the application pro
`gram was originally written. An entirely new input
`device may be used with the application program, or
`optionally, the user may change the input signals of the
`existing input device from those to which the applica
`tion program originally responded. Each application
`pro?le contains a list of input messages mapped to cor
`responding commands useable by the application pro
`gram. The interface pro?le module may also comprise a
`set of user pro?les for each user who utilizes the com
`puter system. The user pro?le contains a list of input
`messages and their corresponding commands which can
`be common to several application programs.
`The environment link module communicates with
`integrated operating environment and contains algo
`rithms for the execution of the advanced user interface
`procedures. For example, it takes the input messages
`generated by the alternate input subsystem, queries the
`integrated operating environment as to which applica
`tion program is active, matches the input messages to
`the corresponding commands in the appropriate appli
`cation pro?le within the interface pro?le module, and
`initiates an action based on the set of instructions such as
`sending messages to the active application program via
`the operating environment. The advanced user inter
`face may also include a set of utility programs such as a
`pop-up keyboard or an image magni?er utility.
`
`25
`
`30
`
`35
`
`SUMMARY OF THE INVENTION
`It is therefore an object of the invention to allow a
`user to select between a plurality of input devices to
`input data into a computer system.
`It is another object of the invention to input informa
`tion to a computer system via a different input device
`than that for which the active application program was
`originally written without revising the application pro
`gram code.
`It is still another object of the invention to allow a
`relatively unsophisticated user to select among avail
`able application programs and input devices and con
`struct a user interface designed particularly to his needs.
`It is yet another object of the invention to accommo
`date future advances in user friendly input devices and
`application programs in a user interface.
`It is still yet another object of the invention to allow
`different users who operate a single computer to con
`struct different user interfaces to accommodate individ
`ual preferences in inputting data.
`These objects and others are accomplished by an
`advanced user interface which operates with an inte
`grated operating environment capable of running a
`plurality of application programs and which utilizes the
`message and focusing functions provided by the envi
`ronment. The advanceduser interface is comprised of
`three major modules, an alternative input subsystem
`55
`module, an interface pro?le module, and an environ
`ment link module. The alternative input subsystem
`module provides communication between the attached
`user friendly input devices and the remainder of the
`advanced user interface as well as application programs
`through the integrated operating environment. The
`alternative input subsystem module receives the input
`signals generated by the input devices and translates
`them to input messages useable in the advanced user
`interface.
`The interface pro?le module comprises a set of appli
`cation pro?les, one for each application program for
`which the user wishes to use an input device in a differ
`
`45
`
`65
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`FIG. 1 is a block diagram showing the interaction of
`computer system software and hardware in response to
`actions of user using an integrated operating environ
`ment such as Presentation Manager TM.
`FIG. 2 is an architectural block diagram of a com
`puter system in accordance with the present invention,
`depicting the memory having read therein a series of
`application and operating system programs.
`FIG. 3 is a block diagram showing the interaction of
`computer system software and hardware in response to
`actions of a user designed in accordance with a ?rst
`embodiment of the present invention.
`FIG. 4 is a flow diagram of the operation of the ?rst
`embodiment of the present invention in response to a
`voice input signal.
`FIG. 5 is a diagram of an application pro?le.
`FIG. 6 is a block diagram showing the interaction of
`computer system software and hardware in response to
`actions of a user designed in accordance with a second
`embodiment of the present invention.
`FIG. 7 is a flow diagram of the operation of the sec
`ond embodiment of the present invention in response to
`a voice input signal.
`FIG. 8 is a ?ow diagram of the operation of a com
`puter module which distinguishes between touch input
`meant to emulate a mouse pointing device and touch
`input meant to be a gesture or character.
`
`DETAILED DESCRIPTION OF THE
`PREFERRED EMBODIMENT
`FIG. 1 shows the interaction of the user and the
`computer system utilizing application programs A, B, C
`and D 50-53 in a typical integrated operating environ
`ment such as Presentation Manager TM 56.
`The user inputs data into the computer system via
`device A (keyboard) 32, device B (mouse) 34 or touch
`sensor 36. As the user inputs data, the selected device
`generates an interrupt to activate the appropriate
`driver. In the case of the keyboard 32, driver A 60 is
`
`Petitioners Ex. 1003 Page 10
`
`

`

`5,157,384
`
`15
`
`25
`
`35
`
`20
`
`5
`activated; in the case of the mouse 34. driver B 62 is
`energized; and input to the touch sensor 36 activates the
`touch driver 63. The appropriate driver signals the
`integrated operating environment 56. The integrated
`operating environment 56 keeps track of which applica
`tion program is active and sends all input data to that
`application program. Alternatively, the user can change
`application programs, and the integrated operating en
`vironment 56 will route the message to the new active
`application program. In one preferred embodiment,
`application program A 50 is a spreadsheet program,
`application program B 51 is a word processor, applica
`tion program C 52 is a drawing program, and applica
`tion program D 53 is an electronic mail facility. Keep~
`ing track of the active program is a “focusing function",
`as the integrated operating environment 56 determines
`the “focus” of the user at a given time. For purposes of
`illustration, assume the user is inputting data via the
`keyboard 32 and the spreadsheet 50 is the active appli
`cation program. The spreadsheet 50 then decides what
`action should be performed. Generally, this action will
`fall into one of three categories: the spreadsheet 50 will
`change the display 28, the spreadsheet 50 will communi
`cate with other I/O devices such as a printer, or the
`spreadsheet 50 will communicate with other applica
`tions. All of the messages are handled by the integrated
`operating environment 56.
`While the prior art integrated operating environ
`ments allow easier access to multiple application pro
`grams, and therefore, the user can select the particular
`application programs which best suit his needs, the user
`can only communicate with the application program
`with the input devices for which it has been written.
`For example, spreadsheet 50 is a program which has
`been written for a combination of keyboard 32 and
`mouse 34. Although a touch driver 63 can be developed
`to communicate coherently with the integrated operat
`ing environment 56, by converting input signals gener-_
`ated by the touch sensor 63 into mouse movements and
`mouse clicks, the capabilities of the touch input device
`36 are limited by the repertoire of the mouse device.
`New categories of touch commands such as handwrit
`ing recognition or gestures cannot be easily accommo
`dated by this method, especially by an application pro
`gram such as spreadsheet 50 which has no understand
`ing of what the types of touch commands mean. Other
`new input devices are more drastically limited. Thus,
`although the system designer or user would like to
`include new user-friendly input devices into an individ
`ual workstation, he may find it impossible or severely
`limiting to use existing software.
`The preferred embodiment of the invention com
`prises a set of computer programs for controlling the
`interaction between a user and a computer system as
`shown in FIG. 2. The invention is primarily envisioned
`55
`for use with a personal computer such as the IBM
`PS/2 TM ; however, the principles of this invention can
`be extended to other individual workstations or to much
`larger data processing systems. The architectural block
`diagram of FIG. 2 includes a central processing unit
`(CPU) 20 connected by means of a system bus 22 to a
`read-only memory (ROM) 24 and a random access
`memory (RAM) 26. Also included in the computer
`system in FIG. 2 are a display 28 by which the com
`puter presents information to the user, and plurality of
`65
`input devices including a keyboard 30, and devices A 32
`and B 34, which for the purposes of illustration in the
`following description are a second alphanumeric key
`
`6
`board 32 and a mouse 34, but which could be a special
`ized keyboard such as a numeric keyboard, and a re
`mote pointing device or a variety of other input devices.
`The computer system further comprises a touch sensor
`36 or touch input device for use at the surface of the
`display 28, a voice sensor 38 and an image sensor 40.
`The I/O 42 can be connected to communication lines or
`other I/O equipment.
`The random access memory- (RAM) 26 includes ap
`plication program A 50, application program B 51,
`application program C 42, and application program D
`53. Examples of typical application programs would
`include word processors, spreadsheets, graphic pro
`grams, electronic mail, and data base programs. The
`RAM 26 also stores an operating system 54, such as
`DOS TM or OS/2 TM , and an integrated operating
`environment program 56, such as WINDOWS TM or
`Presentation Manager TM , which allow several applica
`tion programs to be running concurrently. Other soft
`ware includes driver A 60 and driver B 62, which inter
`pret the electrical signals generated by the second key
`board 32 and the mouse 34 respectively for the operat
`ing environment 56. The touch driver 63, the gesture
`recognition unit 64 and character recognition unit 65
`handle input signals from the touch sensor 36. Input
`signals from the voice sensor 38 and the image sensor 40
`are processed by the voice recognition unit 66 and
`image recognition unit 68 respectively. Although de
`picted in FIG. 2 as entirely software, driver A 60, driver
`B 62, touch driver 63, gesture recognition unit 64, voice
`recognition unit 66 and image recognition unit 68 can
`also be a combination of hardware and software. Fi
`nally, also resident in RAM 26 are the set of software
`modules which make up the advanced user interface
`100.
`
`40
`
`45
`
`First Embodiment
`FIG. 3 shows a computer system designed in accor
`dance with the present invention. Rather than being
`connected to the integrated operating environment 56
`via a standard device driver at least one of the plurality
`input devices is connected to the advanced user inter
`face 100, via a recognition unit. In FIG. 3, the voice
`sensor 38 and image sensor 40 are connected to the
`advanced user interface (AUI) 100 via their respective
`recognition units, voice recognition unit 65 and image
`recognition unit 68. A keyboard 32, and a mouse 34,
`remain connected to integrated operating environment
`56 via their respective drivers driver A 60 and driver B
`62, since many application programs recognize key
`board and mouse input. Finally, in FIG. 3, the touch
`sensor 36 is connected to both the integrated operating
`environment 56 via the standard touch driver 63 and to
`the advanced user interface 100 via the gesture recogni
`tion unit 64 and the character recognition unit 65. The
`touch driver 63, gesture recognition unit 64 and charac
`ter recognition unit 65 translate signals from the touch
`sensor 36 to produce gesture or character “tokens”
`respectively. The series of positions of a ?nger or stylus
`on the surface of the touch sensor 36 can be recognized
`as handwritten alphanumeric characters by the charac
`ter recognition unit 65, or as a gesture such as a circle or
`right hand arrow by the gesture recognition unit 64.
`The voice recognition unit 66 digitizes the sound re
`ceived by the voice sensor 38 and performs pattern
`matching against previously de?ned sound patterns to
`produce voice “tokens”. Image recognition unit 68
`processes information from the image sensor 40 and
`
`Petitioners Ex. 1003 Page 11
`
`

`

`5
`
`25
`
`5,157,384
`7
`8
`the AIS 103 from the input signals received by input
`outputs image "tokens". A "token” is an input message
`which represents a series of electrical signals collected
`devices 36, 38, 40, mapped to keyboard, mouse or other
`from an input device. For example, the series of points
`commands which are usable by existing application
`programs, e.g., mouse clicks, keystroke messages,
`collected when a user draws a circle on the touch sensor
`can be de?ned as the “circle token". Coupling the touch
`MACROs, utility programs, etc. The Advanced User
`sensor 36 to integrated operating environment 56 allows
`Interface utility (AUI utility) 109 is a set of utility pro
`the user to take advantage of application programs
`grams such as a pop-up keyboard or an image magni?er
`which are speci?cally written for the touch sensor 36 or
`utility which can be used with AUI 100. As shown, the
`other existing application programs for which only
`AUI utility 109 can be written as a separate application
`primitive mouse input signals are needed. Yet as the
`program, or set of application programs, so that it can
`use the message passing capabilities of the integrated
`touch sensor 36 is also connected to AUI 100 through
`the gesture recognition unit 64 and the character recog
`operating environment 56 to communicate with other
`application programs.
`nition unit 65, application programs which do not rec
`ognize touch input can still accept gesture or character
`A few examples of utilities which might be found in
`“tokens” due to the translation capabilities of the ad
`the advanced user interface utilities 109 include a “pop
`up” keyboard, a “pop-up” numeric keypad, an image
`vanced user interface 100.
`magnifying utility and an electronic mail routing slip.
`In other embodiments, the keyboard 32 and mouse 34
`The “pop-up” keyboard and “pop-up” keypad are func
`are connected to both the AU] 100 and integrated oper
`ating environment 56 as is the touch sensor 36. In still
`tions which are preferably designed to operate with a
`touch input device at the display surface. By issuing the
`other embodiments, all input devices are attached di
`rectly to the AUI 100 alone. By using a keyboard AUI
`proper command, e.g., a circle gesture or touching a
`keyboard icon, the keyboard or keypad will “pop-up”
`link, a user could change the normal typewritten com
`mands for an application program to those which he
`on the screen. A user can quickly enter columns of
`alphanumeric information directly on the screen by
`?nds more convenient to input or memorize. As the
`touching the keys displayed on the screen, thus elimi
`AUI 100 is written as an application program, it is able
`to use the communication and focusing capabilities of
`nating the need to divert their attention