United States Patent
`Luciw et al.
`
`[III Patent Number:
`1451 Date of Patent:
`
`5,644,735
`Jul. 1, 1997
`
`[54] METHOD AND APPARATUS FOR
`PROVIDING IMPLICIT COMPUTER-
`IMPLEMENTED ASSISTANCE
`
`r/5] Inventors: William W. Luciw, Morgan Hill;
`Stephen P. Capps, San Carlos;
`Lawrence G. Tesler, Portola Valley, all
`of Calif.
`
`[73] Assignee: Apple Computer, Inc., Cupertino,
`Calif.
`
`[21] Appl. No.: 42Q959
`Apr. 19, 1995
`
`[22] Filed:
`
`Related U.S. Application Data
`
`[60] Division of Ser. No. 99,861, Jd. 30, 1993, Pat. No. 5,477,
`447, which is a continuation-in-part of Ser. No. 889,225,
`May 27, 1992, Pat. No. 5,390,281.
`[5 11 ht. C1.6 ...................................................... G06F 17/30
`[52] U.S. C1. ........................................ 395/338; 3951336
`[58] Field of Search ..................................... 3951155, 156,
`3951157, 159, 161, 149,336,337, 338,
`968, 759, 12, 51, 62; 364/419.08,419.19,
`419.1,419.14, 419.15, 419.13
`
`[5 61
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`................................
`Re. 34,476
`1211993 Norwood
`382f186
`611987 Schramm et al. ........................ 395160
`4,670,848
`................................
`1211987 Scott et al.
`4,713,775
`395150
`......................
`411988 Katayama et al.
`4,736,2%
`3951759
`................................
`3951336 X
`4,862,390
`811989 Weiner
`......................................
`1011989 Smith
`4,875,187
`3951141
`411990 Iggulden et al. ........................ 3791100
`4,918,723
`711990 Nakama et al. ................... 3641709.11
`4,945,504
`811990 Gansner et al. ........................ 3951140
`4,953,106
`1111990 Amirghdi et al. ................... 399759
`4,974,191
`...............................
`211992 Silverberg
`5,091,790
`3581434
`.......................................
`411992 Lanier
`5,103,498
`395168
`411992 Katayama et al. ...................... 395I759
`5,109,509
`811993 Gardner et al. ........................... 395112
`5,239,617
`(List continued on next page.)
`
`FOREIGN PMENT DOCUMENTS
`0441089A2 811991 European Pat. off. .
`1-130291 511989 Japan.
`
`OTHER PUBLICATIONS
`
`Wilensky, Robert; Arens, Yigal, and Chin, David, 'Talking
`to UNM in English: An Overview of UC," Communications
`of the ACM, Jun. 1984, vol. 27, No. 6, pp. 574 to 593.
`Tello, Ernest R., "Natural Language Systems," Savvy PC,
`Clout 2, Q&A, Lotus HAL, Mastering A1 Tools and Tech-
`niques, Chapter 2, pp. 25 to 64.
`
`(List continued on next page.)
`
`Primary Emminer-Kee M. Tung
`Assistant Examiner--Crescelle N. dela Torre
`Attorney, Agent, or Fim+Hickman Beyer & Weaver
`
`ABSTRACT
`[571
`A method and apparatus for providing computer-assisted
`implicit and explicit assistance. If no implicit assist actions
`are desired or indicated, a logical process is initiated to
`determine whether explicit assistance should be undertaken.
`If implicit assistance is indicated, a list of action alternatives
`is displayed for the user. If explicit assistance is desired by
`the user, particular object(s) from which the assistance may
`be inferred are entered into an assistance operation. An
`attempt is made to recognize possible intents expressed by
`the objects entered into the assistance process. If no user
`intent is, in fact, recognized, the assist operation is usually
`terminated. If a possible intent is recognized, the actual
`intent is hypothesized. A check is further undertaken, to
`determine whether a hypothesis is in fact available. If no
`hypothesis is found, the process permits the user to supply
`a proposed action. If no hypothesis is found and no user
`action is proposed, assistance efforts terminate. However, if
`a hypothesis is available, preparations for execution are
`undertaken. A final inquiry is made as to whether to under-
`take the hypothesized assist. If the response to an inquiry
`whether to assist as hypothesized is aftinnative, execution of
`the hypothesized action is undertaken. A pen-based com-
`puter preferably implements the indicated functions.
`
`20 Claims, 18 Drawing Sheets
`
`GOOGLE INC. 1010
`
`1
`
`

`
`5,644,735
`Page 2
`
`U.S. PAENT DOCUMENTS
`.....................................
`5,255,386 1011993 Prager
`3951605
`..............................
`5,390,281 211995 Luciw et al.
`395112
`5,432,W 711995 Matsumoto ............................. 3951338
`5,477,447 1211995 Luciw et al. ............................ 3951759
`5335,323 711996 Miller et al. ............................ 3951338
`OTHER PuBLIcKrIONS
`O'Connor, Rory J., "Apple Banking on Newton's Brain",
`Sun Jose Mercury News, Wednesday, Apr. 22, 1992.
`Hendrix. Gary G. and Walter, Brett A., 'The Intelligent
`Assistant: Technical Considerations Involved in Designing
`Q&A's Natural-language Interface", Byte Magazine, Dec.
`1987, vol. 12, Issue 14, p. 251.
`Edwards, John R., "Q&A: Integrated Software with Macros
`and an Intelligent Assistant", Byte Magazine, Jan. 1986, vol.
`11, Issue 1, pp. 120-122.
`Goldberg, Cheryl, "IBM Drawing Assistant: Graphics for
`the EGA", PC Magazine, Dec. 24, 1985, vol. 4, Issue 26, p.
`255.
`Garretson, R., "TBM Adds 'Drawing Assistant' Design Tool
`to Graphics Series", PC Week, Aug. 13, 1985, vol. 2, Issue
`32, p. 8.
`GlinertStevens, Susan, "Microsoft Publisher: Desktop
`Wizardry", PC Sources, Feb., 1992, vol. 3, Issue 2, p. 357.
`Nilsson, B.A., "Microsoft Publisher is an Honorable Start
`for DTP Beginners", Computer Shoppel; Feb. 1992, vol. 12,
`Issue 2, p. 426.
`Poor, Alfred, "Microsoft Publisher", PC Magazine, Nov. 26,
`1991, vol. 10, Issue 20, p. 40, evaluates Microsoft Publisher.
`
`Rampe, Dan, et al. In a Jan. 9, 1989 news release, Claris
`Corporation announced two products, SmartForm Designer
`and SmartForm Assistant, which provide "Intelligent Assis-
`tance", such as custom help messages, choice lists, and
`data-entq validation and formatting.
`Berry, Deanne, et al. In an Apr. 10, 1990 news release,
`Symantec announced a new version of MORE (I'M).
`Elofson, G. and Konsynski, B., "Delegation Technologies:
`Environmental Scanning with Intelligent Agents", Joumal
`of Management Information Systems, Summer 1991, vol. 8,
`Issue 1, pp. 37-62.
`Nadoli, Gajanana and Biegel, John, "Intelligent Agents in
`the Simulation of Manufacturing Systems", Proceedings of
`the SCS Multiconference on AI and Simulation, 1989.
`Sharif Heger, A. and Koen, B. V., "KNOWBOT: an Adaptive
`Data Base Interface", Nuclear Science and Engineering,
`Feb. 1991, vol. 107, No. 2, pp. 142-157, describes an
`adaptive interface KNOWBOT.
`Ohsawa, I. and Yonezawa, A., "A Computational Model of
`an Intelligent Agent Who Talks with a Person", Research
`Reports on Information Sciences, Series C, Apr. 1989, No.
`92, pp. 1-18.
`RatcliBe, Mitch and Gore, Andrew, "Intelligent Agents take
`U.S. Bows.", MacWeek, Mar. 2, 1992, vol. 6, No. 9, p. 1.
`Boy, Guy A., Intelligent Assistant Systems, Harcolllt Brace
`Jovanovich, 1991, uses the term "Intelligent Assistant Sys-
`tems".
`Microsofr Windows User's Guide for the Windows Graphi-
`cal Environment; Version 3.0; Microsoft Press copyright
`1990-1995.
`
`2
`
`

`
`U.S. Patent
`
`Jul. 1, 1997
`
`Sheet 1 of 18
`
`3
`
`

`
`U.S. Patent
`
`Jul. 1, 1997
`
`Sheet 2 of 18
`
`4
`
`

`
`U.S. Patent
`
`Jul. 1, 1997
`
`Sheet 3 of 18
`
`102
`
`Provide List Of
`Alternatives
`
`Selection
`
`Update Information
`& Linked
`Information
`
`Figure 3
`
`4
`
`Prepare TO
`Execute
`
`-- 143
`
`5
`
`

`
`U.S. Patent
`
`J U ~ . I, 1997
`
`Sheet 4 of 1s
`
`154
`
`I Yes
`
`Recognize
`User Input
`
`4
`
`Figure
`
`6
`
`

`
`U.S. Patent
`
`Jul. 1, 1997
`
`Sheet 5 of 18
`
`Name
`
`Phone
`
`Speaker
`
`d ~ o d e m
`
`179
`
`185
`
`189
`
`Name
`
`Isaac
`
`Phone
`
`/ 175
`
`11 177
`
`7
`
`

`
`U.S. Patent
`
`Jul. 1, 1997
`
`Sheet 6 of 18
`
`<PERSON>
`NAME:
`BIRTHDAY:
`TELEPHONE:
`FAX:
`ADDRESS:
`HEIGHT:
`WEIGHT:
`
`INSTANCES: <PERSON- I> <PERSON-2> <PERSON-3>
`
`182c
`
`182a
`
`<PERSON-3> <ISAAC-3>
`182b
`NAME: ISAAC JONES
`?
`<PERSON-2> <ISAAC-2>
`NAME: ISAAC ASIMOV
`<PERSON- 1 > <ISAAC- 1 >
`NAME: ISAAC NEWTON
`BIRTHDAY: NULL
`ZLEPHONE: 408-555-1212
`FAX: NULL
`ADDRESS: 123 MAIN STREET,
`
`HEIGHT: NULL
`WEIGHT: NULL
`
`8
`
`

`
`U.S. Patent
`
`Jul. 1, 1997
`
`Sheet 7 of 18
`
`Phone
`
`179
`
`Speaker
`
`181
`d ~ o d e m
`
`185
`
`189
`
`ISAAC NEWTON
`
`I ISAAC ASIMOV I m
`
`ISAAC JONES
`
`I
`
`,
`+ N a m e
`
`Phone
`
`179
`
`F Speaker d ~ o d e m
`
`181
`
`183
`2
`
`/ 177
`
`185
`
`189
`J
`
`175' c,,,
`
`Phone
`
`179
`
`Isaac ~ s i m o v
`
`181
`Speaker d ~ o d e m
`J 183
`
`585
`
`589
`
`9
`
`

`
`U.S. Patent
`
`Jul. 1, 1997
`
`Sheet 8 of 18
`
`Based Upon A
`Selected Heuristic
`
`Figure 7
`
`Smart Fields?
`I Yes
`
`Corresponding
`Field Types
`
`Data Into
`Smart Fields
`
`I
`
`Figure 8a
`
`10
`
`

`
`U.S. Patent
`
`Jul. 1, 1997
`
`Sheet 9 of 18
`
`175' 6 N E ~ W Isaac Asimov
`
`Phone 1-202-555-1234
`181
`r) Speaker d M o d e m
`179
`
`Options
`
`k'
`
`177
`
`11
`
`

`
`US. Patent
`
`JUI. I, 1997
`
`Sheet 10 of 1s
`
`yi a
`
`135 in
`Fig.3
`
`O:e>
`Explicitly
`Selected?
`
`Figure 9a
`
`12
`
`

`
`I LASTUSED
`
`INSTANCE
`ISAAC- 1
`ISAAC-2
`ISAAC-3
`
`INSTANCE
`ISAAC- 1
`ISAAC-2
`ISAAC-3
`
`ORDER
`1
`2
`3
`
`INSTANCE
`ISAAC- I
`ISAAC-2
`ISAAC3
`
`TIMES USED
`2
`12
`1 0 0
`
`CALL ISAAC
`BBure 96
`
`13
`
`

`
`US. Patent
`
`Jul. 1, 1997
`
`Sheet 12 of 18
`
`&Please CALL lSAAC
`
`175'
`6 ~ 1 e a s e CALL ISAAC
`
`14
`
`

`
`U.S. Patent
`
`J U ~ . 1, 1997
`
`Sheet 13 of 18
`
`Figure I l a
`
`15
`
`

`
`U.S. Patent
`
`Jul. 1, 1997
`
`Sheet 14 of 18
`
`Scheduling
`
`Finding
`
`-Person(
`
`1x1
`1
`
`
`-
`
`-1
`
`~armatting'
`
`Format
`
`Mailing
`
`Faxing
`
`Print
`
`Calling
`
`16
`
`

`
`U.S. Patent
`
`Jul. 1, 1997
`
`Sheet 15 of 18
`
`Intents
`
`Choose Best
`
`Figure 12a
`
`276 0
`
`User Selection?
`
`F
`
`7
`2
`r
`Enter Action AS
`Another Object
`
`8
`
`
`
`gure 12b
`Fi
`
`17
`
`

`
`U.S. Patent
`
`Jul. 1, 1997
`
`Sheet 16 of 18
`
`6 Please
`
`I SCHEDULE I
`
`)75'
`please CALL ISAAC
`
`How Do I?
`
`18
`
`

`
`U.S. Patent
`
`Jul. 1, 1997
`
`Sheet 17 of 18
`
`& Identify Missing
`Preconditions
`
`Preconditions To
`Extent Possible
`
`h&
`
`Figure 13
`
`Do Button
`Pressed?
`
`Yes
`
`Display
`Confirmation
`Slip, If Any
`1
`
`,304
`
`Figure 14a
`
`19
`
`

`
`U.S.
`
`Jul. 1, 1997
`
`Sheet 18 of 18
`
`Name
`
`Fax #
`
`1 -1
`
`/ 1 1 1
`
`) Format Plain
`d Cover Page
`(Preview) (Notes) (TzGE-)(Fax)E
`
`Manual Dialing
`
`20
`
`

`
`5,644,735
`
`1
`METHOD AND APPARATUS FOR
`PROVIDING IMPLICIT COMPUTER-
`IMPLEMENTED ASSISTANCE
`
`2
`assistance could be provided is recognized. However, such
`an approach may provide unsatisfactory results when the
`user is provided with assistance that is unwanted or dispro-
`portionate.
`Cross-Reference to a Related Application
`Simply stated, concerns have arisen about assist functions
`This application is divisional of U.S. patent application
`being undertaken by the computer without adequate user
`Ser. No. 081099,861, filed on Jul. 30, 1993, now U.S. Pat.
`control and interaction. When the assist function has been
`No. 5,4775147, under the title '
`' M ~ O D AND APPARA-
`undertaken without adequate user control, assistance would
`TUS FOR PROVIDING COMPUTER-IMPLEhENIED
`be provided awkwardly and at times when the assistance
`ASSISTANCE" on behalf of Luciw et. al. and assigned to 10 was not necessarily desired.
`the same assignee as herein; which application is incorpo-
`It is essential that the control, the timing, and the appli-
`rated herein by reference in its entirety. Application Ser. NO.
`cation of the assistance is considered appropriate and well-
`081099,861 is a continuation-in-pat of S a NO. 889,225,
`tuned. The confidence of the user is undermined when the
`issued u.S. Pat. NO. 5990281 filed May 27, 1992 and
`assistance provided only obliquely addresses particularized
`issued Feb. 14, 1995 and which is incorporated herein by 15 us, needs without precisely providing the paaic*
`results
`reference in its entirety. Priority rights based upon this
`objectively and subjectively required.
`earlier filed United States Patent are claimed.
`
`5
`
`30
`
`35
`
`SUMMARY OF THE INVENTION
`According to the invention. a method and aoDaratus has
`20 been developed for providing computer-assisted implicit
`and explicit assistance for a variety of user-supportive
`information functions. If no implicit assist actions are
`desired or indicated, then a logical process is initiated to
`determine whether explicit assistance should be undertaken.
`25 If implicit assistance is indicated, a list of action alternatives
`is displayed for the user. Alternatively, a process can be
`undertaken to automatically select a best action alternative
`of several identified alternatives.
`If explicit assistance is desired by the user, particular
`object(s) from which the assistance may be inferred are
`entered into an assistance operation. An attempt is then made
`to recognize possible intents expressed by the objects
`entered into the assistance process. If no user intent is, in
`fact, recognized, the assist operation is usually terminated If
`a possible intent is recognized, the actual intent is hypoth-
`esized. A check is further undertaken, to determine whether
`a hypothesis is in fact available. If no hypothesis is found,
`the process permits the user to supply a proposed action. If
`no hypothesis is found and no user action is proposed,
`assistance efforts terminate. However, if a hypothesis is
`available, preparations for execution are undertaken. A final
`inquiry is made as to whether to undertake the hypothesized
`assist. If the response to an inquiry whether to assist as
`hypothesized is aikmtive, execution of the hypothesized
`action is undertaken.
`Accordingly, it is an intent of the invention to provide user
`explicit assist functions which are provided under oversight
`and with the interaction of the user, and implicit assist
`functions in certain structured instances. The involvement of
`the user in the implementation of implicit assist operations
`is a key aspect of the invention. User intent is not merely
`deduced, but specifc user queries are made to ensure
`controlled application of assist operations.
`These and other advantages of the present invention will
`become apparent upon reading the following detailed
`descriptions and studying the various figures of the draw-
`ings. -
`
`BACKGROUND OF THE INVENTION
`The present invention relates generally to computer
`systems, and more particularly to computer-implemented
`assistance methods and apparatus.
`Computerized personal organizers are becoming increas-
`ingly popular with a large segment of the population.
`Computerized personal organizers tend to be small,
`lightweight, and relatively inexpensive, and can perform
`such functions as keeping a calendar, an address book, a
`to-do list, etc. While many of these functions can also be
`provided in conventional computer systems, personal orga-
`nizers are very well suited to the personal organization task
`due to their small size and portability. Personal organizers
`are available from such companies as Sharp and Casio of
`Japan.
`A relatively new form of computer, the pen-based com-
`puter system, holds forth the promise of a marriage of the
`power of a general purpose computer with the functionality
`and small size of a personal organizer. A pen-based com-
`puter system is typically a small, hand-held computer where
`the primary method for inputting data includes a "pen" or
`stylus. Apen-based computer sy~temis commonly housed in
`a generally rectangular enclosure, and has a dual-function
`display assembly providing a viewing screen along one of
`the planar sides of the enclosure. The dual-function display
`assembly serves as both an input device and an output
`device. When operating as an input device, the display
`assembly senses the position of the tip of a stylus on the
`viewing screen and provides this positional information to
`the computer's central processing unit (CPU). Some display
`assemblies can also sense the pressure of the stylus on the
`screen to provide further information to the CPU. When
`operating as an output device, the display assembly presents
`computer-generated images on the screen.
`The dual-function display assemblies of pen-based com-
`puter systems permit users to operate the computer as a
`computerized notepad. For example, graphical images can
`be input into the pen-based computer by merely moving the
`stylus on the surface of the screen. As the CPU senses the
`position and movement of the stylus, it generates a corre-
`sponding image on the screen to create the illusion that the 60
`stylus is drawing the image directly upon the screen, i.e. that
`the stylus is "inking" an image on the screen. With suitable
`recognition software, text and numeric information can also
`be entered into the pen-based computer system in a similar
`fashion.
`One approach to computerized assist operations is to
`provide assistance automatically when a situation in which
`
`45
`
`55
`
`BRIEF DESCXETION OF THE DRAWINGS
`FIG. 1 is a block diagram of a computer system in
`accordance with the present invention.
`FIG. 2 is a top plan view of the screen, case, and keypad
`of the computer system of FIG. 1.
`FIG. 3 is a flow diagram of a process according to the
`invention for providing controlled computer-assisted user
`assistance.
`
`65
`
`21
`
`

`
`5,644,735
`
`10 --
`
`faxing
`
`3
`4
`FIG. 13 illustrates the process for preparing for execution
`FIG. 4a shows a process for determining whether or not
`implicit assistance is desired by the user, in connection with
`of a particular activity.
`providing computer assisted support to the user.
`FIG. 14a is a short flow diagram of a process for deter-
`FIG. 4b shows an example of an @licit assist operation mining whether to proceed with execution of a particular
`with a phone slip window having a smart name field evoked, 5 selected assist activity.
`for example, by either highlighting the verb ''call" or by
`FIG. 14b illustrates an example of a c o m t i o n of
`writing it on the note field before evoking window.
`action slip that could be produced upon completion of a
`FIG. 4' shows the phone "P window of
`4bw"h the
`of the process of
`activity, in this case
`name formal font form. ISAAC having been recognized and
`to another party.
`established in the name field in
`FIG. 5 shows an example of a generic <PERSON> type
`DEXAZLED DESCFSFlION OF THE
`frame along with a particular set of specific frames of the
`PREFERRED EMBODIMENTS
`<PERSON> type.
`The present invention is well suited for pointer based
`FIGS. 6a-6c show respective assist windows in succes-
`systems such as the
`pen-awae and
`sive stages of an assist process, including first a window
`mouse controlled systems that are currently popular. For the
`containing a first informational level directed at the name
`purposes of illustration, the invention will be described in
`ISAAC alone, a second window with a popup menu offer-
`connection with a ~en-based system.
`ing a user choice among several known ISAACs, and a tbird
`window showing the selection of a particular ISAAC, that is
`As shown in FIG. 1, a pen-based computer system 10 in
`ISAAC ASIMOV, having been accomplished.
`accordance with the present invention includes a central
`FIG. 7 illustrates a brief flow diagram illustrating a 20 processing unit (CPU) 12, read only memory (ROM) 14,
`heuristic process for the selection of a particular choice
`random access memory (RAM) 16, inputloutput (T/O) cir-
`among alternatives when the user is not queried for selection
`cuitry 18, and a display assembly 20. The pen-based com-
`of alternatives.
`puter system 10 may also optionally include a mass storage
`unit 22, a keypad (or keyboard) 24, a serial port 26, an
`FIGS. 7a-7c show selected examples of heuristic rules of
`thumb which are effective in making automatic choices 25 infrared (m) Port 28, and a clock 30.
`between alternative ISAACs, respectively directed toward
`The CPU l2 is preferably a commercially available,
`selection schemes such as selecting the last used ISAAC,
`single chip microprocessor. While CPU 12 can be a complex
`selecting the last in order of ISAACs according to particular
`instruction set computer (CISC) chip, it is preferable that
`position within a selected table, and finally selecting a 30 CPU 12 be one of the commercially available, reduced
`instruction set computer WSC) chips which are known to
`particular ISAAC based upon prior frequency of choice of
`that particular ISAAC.
`be of generally higher performance than CISC chips. CPU
`FIG. &I is a flow diagram illustrating the updating process
`12 is coupled to ROM 14 by a unidirectional data bus 32.
`for data base information in linked smart fields.
`ROM 14 contains the basic operating system for the pen-
`FIG. 8b is a call slip illustration of an updated smart field 35 based computer system 10. CPU
`is connected to RAM 16
`by a hi-directional data bus 34 to pennit the use of RAM 16
`window in which the phone number field information has
`as scratch pad memory. ROM 14 and RAM 16 are also
`been updated.
`coupled to CPU 12 by appropriate control and address
`FIG. 9a is a flow diagram of the process
`to the
`busses, as is well h 0 w n to those skilled in the art. CPU 12
`invention in which a query is made as to whether a specific
`40 is also coupled to the 110 circuitry 18 by bi-directional data
`selection has been made as to a particular object.
`bus 36 to pennit data transfers with peripheral devices.
`indicate graphically the perfonnance of the
`9
`~
`9
`~
`
`UO circuitry 18 typically includes a number of latches,
`selection query operation as expressed in FIG. 9A.
`FIG. 9d illustrates the transferal of the higwghted objects
`registers and direct memory access (DMA) controllers. he
`purpose of UO circuitry 18 is to provide an interface between
`of FIG. 9c transferred to a selected window.
`l2 and such peripheral devices as
`FIG. loa flustrates the input of a handwritten object into 45
`20,
`mass storage 22, keypad 24, serial port 26, and I/R port 28.
`a smart field in a window.
`20 of
`computer system lo is
`FIG. lob illustrates the recognition of the handwritten
`and an Output device.
`an
`object of FIG. loa arid its conversion into f&
`it is
`font form.
`coupled to UO from a variety of vendors. The input device
`FTG. l l a is a flow diagram illustrating the recognition of
`so of display assembly 20 is preferably a thin, clear membrane
`objects process.
`which covers the LCD display and which is sensitive to the
`FIG. lib illustrates an object combination under Opera-
`position of a stylus 38 on its surface. With such a structure,
`tion.
`the display assembly 20 can serve as an input "tablet." These
`FIG. l l c shows a template for o%aniAng in Preset form
`position sensitive membranes are also readily available on
`a variety of object combinations.
`5s the commercial market. Alternatively, other types of tablets
`FIG.
`flustrates a Process for hypothesizing user
`can be used, such as inductively coupled tablets. Combina-
`intent as to particular activities.
`tion display assemblies such as display assembly 20 which
`FIG. l2b is a flow diagram setting forth a Process for
`include both the LCD and the i n ~ u t membrane are c o r n . -
`determining whether the user wishes to provide or supply a
`cially available from such vendois as Scriptel Corporation of
`60 Columbus, Ohio.
`particular action.
`FIG. l2c shows a window with a menu partially overlap
`Some type of mass storage 22 is generally considered
`ping its topside in order to provide the user with an array of
`desirable. Mass storage 22 can be coupled to UO circuitry 18
`activity choices.
`by a bi-directional data bus 40. However, the mass storage
`FIG. 12d shows the window of FTG. 12c with the activity
`22 can be eliminated by providing a sufficient amount of
`of calling having been selected, establishing the combined 65 RAM 16 to store user application programs and data. In that
`objects C A U and ISAAC in the activity field of the
`case, the RAM 16 can be provided with a backup battery to
`window.
`prevent the loss of data even when the pen-based computer
`
`22
`
`

`
`-
`
`5,644,735
`
`6
`5
`system 10 is turned off. However, it is generally desirable to
`In FTG. 2, the pen-based computer system 10 of FIG. 1 is
`have some type of long term mass storage 22 such as a
`shown housed within a generally rectangular enclosure 50.
`commercially available miniature hard disk drive, nonvola-
`12, ROM 14, RAM 16,110 circuitry 18, and clock
`The
`backed RAM7 a
`tile memory such as flash memory, baa-
`26 are preferably fully enclosed within the enclosure 50. The
`Personal Computers Memory Card International Association 5 display assembly 20 (FIG. 1) is mostly enclosed within the
`(,PCMCIA) card, or the like.
`enclosure 50, but a viewing screen 52 of the display assem-
`The keypad 24 can comprise an array of rnd-~anical bly is exposed to the user. As used herein, the term "screen"
`buttons or switches coupled to y o circuitry 18 by a data bus
`will refer to the portion of the display assembly 20 which
`42.
`an entire, stan-
`keypad 24 can
`can display an image that can be viewed by a user. Also
`dard Qm
`keyboard In the present
`a 10 accessible to the user is the pseudo keypad 24' that was
`separate keypad 24 is not used in favor of a "pseudo" keypad
`described with reference to FTG.
`24'. This "pseudo" keypad 24' comprises "button" areas
`power-up, pen based computer system 10 displays
`which are associated with a bottom edge of the tablet
`on screen 52 an initial "note" area 54u including a header bar
`membrane that extends beyond the lower edge of the LCD
`and a number Of guidelines 58. The header bar 56u
`display. These button areas are defined by a printed or
`silk-saeened icons which can be seen through the transpar- 15 preferably includes the date of creation of the note area 54a
`ent membrane of the input tablet. when the 'buttons* are
`and a number of icons and "soft" buttons, not particularly
`the stylus 38 with the membrane over
`germane to the discussion of the present invention. For this
`selected by
`the header bar 56a will not be discussed in detail
`these printed icons, the membrane senses the pressure and
`communicates that fact to the CPU 12 via data bus 38 and
`herein. The optional guidelines 58 aid a user in entering text,
`I/O 18. An example of pseudo keypad 24 is shown in FIG. 20 g r ~ h i c s , and data into the pen-based computer system 10.
`A graphic object Gin the form of a triangle is shown entered
`2.
`within note area
`Other types of pointing devices can also be used in
`Additional note areas, such as a note area 5 4 , can be
`conjunction with the present invention, for example, an
`interrupt port of the CPU 12 which can count the clock 25 formedb~ theuser by drawing a substantially horizontalline
`acrOSS the S aW l l 52 with the stylus 38. The ~~b~talltially
`pulses to provide the time function. However, this alterna-
`tive clock embodiment tends to be wasteful of CPU pro-
`horizontal line is recognized by the system 10 and is
`cessing power. Clock 30 is coupled to CPU 12 by a data bus
`converted into a second header bar 56b. Additional text,
`graphical, and other data can then be entered into this second
`48.
`is input into the pen-based
`~n operation, idonnation
`note area 54b. For example, the text object T comprising
`"ISAAC" has been entered into second note area 54.
`system 10 by c%ting>~ on the
`of display
`In this preferred embodiment, the keypad 24', as
`assembly 20 with the stylus 38. Information concerning the
`explained previously, comprises a printed or silk-screened
`location of the stylus 38 on the screen of the display
`assembly 20 is input into the CPU 12 via data bus 38 and I/O member 60 provided beneath a lower edge of a thin, clear,
`circuitry 18. Typically, this information comprises the Car- 35 stylus-sensitive membrane 62 of the input "tablet."
`tesian (i.e.x& y) coordinates of a pixel of the screen of
`Alternatively, a keypad could comprise a mechanical keypad
`(or keyboard) 24, or a keypad could comprise "soft buttons"
`display assembly 20 over which the tip of the stylus is
`i.e. images generated at convenient locations on the screen
`positioned. Commercially available combination display
`assemblies such as the aforementioned assemblies available
`52, in which case a "button" would be activated by touching
`from Scriptel Corporation include appropriate circuitry to
`the stylus to the screen over the image of the button. The
`keypad 24' preferably includes a number of dedicated func-
`provide the stylus location information as digitally encoded
`tion buttons 64 and a pair of scroll buttons 66a and 66b. The
`data to the UO circuitry of the present invention. The CPU
`12 then processes the data under control of an operating
`operation of the note areas 54a, 5 4 , etc., scroll buttons 66a
`system. While the method of the present invention is
`and 66b, and other aspects of computer system 10 are
`described in the context of a pen-based system, other point- 45 discussed in greater detail in co-pending U.S. patent appli-
`ing devices such as a computer mouse, a track ball, or a
`cation 07/868,013, filed Apr. 13,1992 on behalf of Tchao et
`al., assigned to the assignee of the present invention and
`tablet can be used to manipulate a pointer on a screen of a
`general purpose computer. Therefore, as used herein, the
`incorporated herein by reference in its entirety.
`The screen illustrated in HG. 2 is referred to as the
`terms "pointer", "pointing device", "pointing means", and
`the like will refer to any mechanism or device for pointing 50 "notepad", and is preferably an application program running
`to a particular location on a screen of a CO~PUter display.
`under the operating system of the pen based computer
`Serial port 26 is coupled to UO circuitry by a
`system 10. In this preferred embodiment, the notepad is a
`bi-directional bus 44. The serial port 26 can be used to
`special or "base" application which is always available
`couple the CPU to external devices and networks.
`beneath higher level applications. The notepad application,
`Infrared (I/R) port 28 is coupled to 110 circuitry by a 55 like other applications, NnS within a window, which in this
`bi-directional bus 46. The I/R port can be used for outgoing
`instance comprises the entire viewing screen 52. Therefore,
`as used herein, a "window" is the entire screen or any
`information (e.g. to control a printer or some other external
`device, or to communicate with other computer systems) or
`portion of an entire screen which is dedicated to a particular
`for incoming information from other computers or devices.
`application program.
`Clock 30 preferably comprises a real-time clock to pro- 60 A status bar 68 is provided at the bottom of the notepad
`vide real-time information to the system 10. Alternatively,
`application. The status bar 68 is provided with a number of
`clock 30 can simply provide regular clock pulses td possibly
`active and display areas, which again are not particularly
`an application program stored in ROM 14, RAM 16, or mass
`germane to the present invention and will therefore not be
`storage 22. The CPU 12 next produces data which is
`discussed in detail herein. U.S. patent application Ser. No.
`transferred to the display assembly 20 via YO circuitry 18 65 071976,970 filed Nov. 16, 1992 on behalf of Foster et. al,
`and data bus 38 to produce appropriate images on the screen
`entitled "Status Bar for Application Windows" and assigned
`portion of the display assembly 20.
`to the assignee of the present invention desdbes how to
`
`23
`
`

`
`5,644,735
`
`= the present invention, objects are preferably @Ie-
`
`8
`7
`Amethod or process 100 for providing implicit or explicit
`make and use the status bar, and is incorporated herein by
`reference in its entirety.
`assistance in the provision of computer implemented ser-
`vices in accordance with the present invention is shown in
`The enclosure 50 is preferably provided with apertures 70
`FIG. 3. The Process begins at step 102 on power-up of the
`which pennit the free transmission of sound from a sp&
`(not shown) which is housed within enclosure 50. The 5 computer system 10 and runs concurrently with other system
`functions.
`speaker can be driven by the CPU 12, by UO circuitry 18, or
`by specialized sound chips, as is well known to those skilled
`At step 104, the process recognizes whether or not an
`in the art. The speaker can be used to provide user feedback,
`implicit assistance function is to be provided by computer
`or to transmit audible information to a user.
`system 10. As will be seen, implicit assistance may, for
`will be used extensively in the follow- 10 example, arise from an entry into a smart field by a users. If
`The term
`a user does enter inf~rmation into a ''smart field? the
`ing discussions. As is well known to software developers, an
`computer database will be queried at step 106 to determin