US005644735A
`
`
`[19]
`5,644,735
`[11] Patent Number:
`Umted States Patent
`
`
`
`
`
`
`
`
`
`
`
`Luciw et al.
`[45] Date of Patent:
`Jul. 1, 1997
`
`
`
`
`
`
`
`
`
`
`
`
`[54] METHOD AND APPARATUS FOR
`
`PROVIDING INIPLICIT CONIPUTER-
`
`
`
`
`IMPLEMENTED ASSISTANCE
`
`
`
`
`
`Inventors: William W Luciw, Morgan Hill;
`
`
`
`Stephen P. Capps, San Carlos;
`
`
`
`
`
`
`
`Lawrence G. Tesler, 1301101a Valley, all
`
`
`0f Calif.
`
`
`
`
`
`[75]
`
`
`
`
`
`
`
`
`[73] As51gnee: éfie Computer, Inc., Cupertino,
`
`
`
`
`[21] Appl No . 424,959
`
`
`
`
`.
`.
`,
`[22] Filed'
`Apr 19 1995
`
`
`
`
`
`
`
`
`FOREIGN PATENT DOCUNIENTS
`
`
`
`
`
`
`$133; $3;me Pa" 05' '
`
`
`
`
`
`
`0411133233?
`
`
`
`
`
`
`
`OTHER PUBLICATIONS
`
`.
`
`
`
`
`
`
`
`Wilensky, Robert; Arens, Ylgal; and Chin, David, ‘Talking
`
`
`
`
`
`
`
`
`to UNIX in English: An Overview of UC,” Communications
`
`
`
`
`
`
`
`
`
`
`
`
`
`of the ACM, Jun. 1984, V01. 27, N0. 6, pp. 574 to 593.
`Tello, Ernest R., “Natural Language Systems,” Savvy PC,
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`Clout 2, Q&A, Lotus HAL, Mastering A1 Tools and Tech-
`
`
`
`
`
`
`
`niques, Chapter 2, pp. 25 to 64.
`
`
`
`
`(List continued on next page.)
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`-
`.
`
`
`
`Related U'S' Appheation Data
`[60] Division of Ser. No. 99,861, Jul. 30, 1993, Pat. No. 5,477,
`
`
`
`
`
`
`
`
`
`447, which is a continuation-impart of Ser. No. 889,225,
`
`
`
`
`
`
`
`
`May 27, 1992, Pat No. 5,390,281.
`
`
`
`
`
`
`
`
`Int. Cl.‘5
`[51]
`.............................................. G06F 17/30
`
`
`
`
`
`
`[52] us. Cl. .................... 395/338; 395/336
`
`
`[58] Field of Search ............................... 395/155 156
`
`
`
`
`
`395/157 159, 161 149 336, 337 338
`
`
`
`
`968 759 i2 51 62" 364/419.08 4i9. 19’
`
`
`
`
`
`
`4191’ 41’9- 14 41915: 419 13:
`’
`’
`
`
`
`’
`’
`’
`'
`Referencos Cited
`
`
`US. PATENT DOCUMENTS
`
`
`
`[56]
`
`
`
`
`
`
`
`
`
`
`Re. 34,476 12/1993 Norwood ................................ 382/186
`
`4,670,848
`6/1987 Schramm et a1.
`..
`
`
`
`
`
`
`" 332/20
`
`
`4,713,775 12/1987 Scott et a1.
`..........
`
`
`
`
`
`I 0
`"
`395/759
`4,736,296
`4/1988 Katayama et a].
`..
`
`
`
`
`
`
`
`
`4,862,390
`8/1989 Weiner ...............
`395I336 X
`
`
`
`
`
`4,875,187 10/1989 Smith .................
`_____ 395,141
`
`
`
`
`
`
`379/100
`4/1990 Iggulden et 31.
`4,918,723
`
`
`
`
`
`
`
`7/1990 Nakama et a1.
`4,945,504
`364/709,11
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`395/140
`8/1990 Gansner et al.
`4,953,106
`
`
`
`
`
`
`11/199O Amirghodsi Ct 31-
`395F759
`--
`4i974,191
`
`
`
`
`
`5,0913% 2/1992 Silverberg ................... 358/434
`
`
`
`
`
`
`4/1992 Lanier ....................... 395/68
`5,103,493
`
`395/759
`4/1992 Katayama et a1.
`..
`5,109,509
`
`
`
`
`
`
`
`
`395,12
`8/1993 G
`1 er et a1
`5239 617
`
`
`
`
`
`
`
`
`
`
`
`
`
`Primary Examiner—Kee M. Tung
`Assistant Examiner—Crescelle N. dela Torre
`
`
`
`
`
`
`
`
`
`
`
`Anome)’: Agent, 0’ Fi'm—Ificmn Beyer & Weaver
`
`
`
`
`ABSTRACT
`[57]
`
`
`
`
`
`
`
`A method and apparatus for providing computer-assisted
`
`
`
`
`
`
`
`
`implicit and explicit assistance If no implicit assist actions
`
`
`
`
`
`
`
`
`
`
`are “Shed 0‘ indicated, 3108“ Process is initiated to
`
`
`
`
`
`
`
`determine whether explicit assistance should be undertaken.
`
`
`
`
`
`
`
`
`If implicit assistance is indicated, a list of action alternatives
`
`
`
`
`
`
`
`
`
`
`
`is displayed for the user. If explicit assistance is desired by
`
`
`
`
`
`
`
`
`
`the user, particular object(s) from which the assistance may
`
`
`
`
`
`
`
`
`
`be inferred are entered into an assistance operation. An
`
`
`
`
`
`
`
`
`
`attempt is made to recognize possible intents expressed by
`
`
`
`
`
`
`
`
`
`the objects entered into the assistance process. If no user
`
`
`
`
`
`
`
`
`
`
`intent is, in fact, recognized, the assist operation is usually
`
`
`
`
`
`
`
`
`
`terminated. If a possible intent is recognized, the actual
`
`
`
`
`
`
`
`
`
`intent is hypothesized. A check is further undertaken, to
`determine whether a hypothesis is in fact available. If no
`
`
`
`
`
`
`
`
`
`
`.
`.
`.
`
`
`
`
`
`
`
`
`
`
`hypothes1s 1s found, the process permits the user to supply
`
`
`
`
`
`
`
`
`
`
`a proposed action. If no hypothesis is found and no user
`
`
`
`
`
`
`
`action is proposed, assistance eiforts terminate. However, if
`
`
`
`
`
`
`
`
`a hypothesis is available, preparations for execution are
`
`
`
`
`
`
`
`
`
`
`
`undertaken. A final inquiry is made as to whether to under-
`
`
`
`
`
`
`
`
`
`
`take the hypothesized assist. If the response to an inquiry
`
`
`
`
`
`
`
`
`whether to assist as hypothesized is afinnafive, execution of
`the, d
`- undemk _ A
`4,, d
`th h
`d
`_
`
`
`
`
`
`
`
`
`6 ypo m ac on ls
`com
`Pen
`en
`86
`
`
`
`
`
`
`puter preferably implements the indicated functions.
`
`
`
`
`
`(List continued on next page.)
`
`
`
`
`
`
`
`20 Claims, 18 Drawing Sheets
`
`
`
`
`
`
`
`0001
`
`Google Inc. 1006
`Google Inc. 1006
`
`0001
`
`

`

`
`5,644,735
`Page 2
`
`
`. U.S. PATENT DOCUMENTS
`
`
`
`
`5,255,386 10/1993 Prager ..................................... 395/605
`
`
`
`
`2/1995 Luciw et al. .......... 395/12
`5,390,281
`
`
`
`
`
`
`395/338
`7/1995 Matsumoto
`5,432,902
`
`
`
`
`395f759
`5,477,447 12/1995 Luciw et a1.
`
`
`
`
`
`
`............................ ' 395/338
`7/1996 Miller et a1.
`5,535,323
`
`
`
`
`
`OTHER PUBLICATIONS
`
`
`
`
`
`
`
`
`
`
`
`
`
`O’Connor, Rory 1., “Apple Banking on Newton’s Brain”,
`
`
`
`
`
`
`
`
`San Jose Mercury News, Wednesday, Apr. 22, 1992.
`
`
`
`
`
`
`
`
`Hendrix, Gary G. and Walter, Brett A., ‘The Intelligent
`
`
`
`
`
`Assistant: Technical Considerations Involved in Designing
`
`
`
`
`
`Q&A’s Natural—language Interface”, Byte Magazine, Dec.
`1987, vol. 12, Issue 14, p. 251.
`
`
`
`
`
`
`Edwards, John R., “Q&A: Integrated Software with Macros
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`and an Intelligent Assistant”, Byte Magazine, Jan. 1986, vol.
`11, Issue 1, pp. 120—122.
`
`
`
`
`
`
`
`
`
`
`
`Goldberg, Cheryl, “IBM Drawing Assistant: Graphics for
`
`
`
`
`
`
`
`
`
`
`
`the EGA”, PC Magazine, Dec. 24, 1985, vol. 4, Issue 26, p.
`255.
`
`
`
`
`
`
`
`
`Garretson, R., “IBM Adds ‘Drawing Assistant’ Design Tool
`
`
`
`
`
`
`
`
`
`
`to Graphics Series”, PC Week, Aug. 13, 1985, vol. 2, Issue
`32, p. 8.
`
`
`
`
`
`
`
`Glinert—Stevens, Susan, “Microsoft Publisher: Deshop
`
`
`
`
`
`
`
`
`
`
`“Wizardry”, PC Sources, Feb., 1992, vol. 3, Issue 2, p. 357.
`Nilsson, B.A., “Microsoft Publisher is an Honorable Start
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`for DTP Beginners”, Computer Shopper; Feb. 1992, vol. 12,
`Issue 2, p. 426.
`
`
`
`
`
`
`
`
`
`
`
`Poor, Alfred, “Microsoft Publisher”, PC Magazine, Nov. 26,
`
`
`
`
`
`
`
`
`
`1991, vol. 10, Issue 20, p. 40, evaluates Microsoft Publisher.
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`Rampe, Dan, et al. In a Jan. 9, 1989 news release, Clan's
`
`
`
`
`
`Corporation announced two products, SmartForm Designer
`
`
`
`
`
`
`and SmartForm Assistant, which provide “Intelligent Assis-
`
`
`
`
`
`
`
`
`tance”, such as custom help messages, choice lists, and
`
`
`
`
`data—entry validation and formatting.
`
`
`
`
`
`
`
`
`
`
`Berry, Deanne, et al. In an Apr. 10, 1990 news release,
`
`
`
`
`
`
`
`Symantec announced a new version of MORE (TM).
`
`
`
`
`
`
`Elofson, G. and Konsynski, B., “Delegation Technologies:
`
`
`
`
`
`Environmental Scanning with Intelligent Agents”, Journal
`
`
`
`
`
`
`
`of Management Information Systems, Summer 1991, vol. 8,
`
`
`
`
`Issue 1, pp. 37—62.
`
`
`
`
`
`
`
`Nadoli, Gajanana and Biegel, John, “Intelligent Agents in
`
`
`
`
`
`
`the Simulation of Manufacturing Systems”, Proceedings of
`
`
`
`
`
`
`
`
`the SCS Multiconference an AI and Simulation, 1989.
`
`
`
`
`
`
`
`
`
`Sharif Heger, A. and Koen, B. V., “KNOWBOT: an Adaptive
`
`
`
`
`
`
`Data Base Interface”, Nuclear Science and Engineering,
`
`
`
`
`
`
`
`
`
`Feb. 1991, vol. 107, No. 2, pp. 142—157, describes an
`
`
`
`adaptive interface KNOWBOT.
`
`
`
`
`
`
`
`Ohsawa, I. and Yonezawa, A., “A Computational Model of
`
`
`
`
`
`
`
`
`an Intelligent Agent Who Talks with a Person”, Research
`
`
`
`
`
`
`
`
`Reports on Infomzation Sciences, Series C, Apr. 1989, No.
`
`
`
`92, pp. 1—18.
`
`
`
`
`
`
`
`Ratcliffe, Mitch and Gore, Andrew, “Intelligent Agents take
`
`
`
`
`
`
`
`
`
`
`
`
`U.S. Bows.”, MacWeek, Mar. 2, 1992, vol. 6, No. 9, p. 1.
`
`
`
`
`
`
`
`Boy, Guy A., Intelligent Assistant Systems, Harcourt Brace
`
`
`
`
`
`
`
`Jovanovich, 1991, uses the term “Intelligent Assistant Sys—
`tems”.
`
`
`
`
`
`
`
`
`Microsoft Windows User’s Guide for the Windows Graphi-
`
`
`
`
`
`
`cal Environment; Version 3.0; Microsoft Press copyright
`1990—1995.
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`0002
`
`0002
`
`

`

`
`US. Patent
`
`
`
`
`
`Jul. 1, 1997
`
`
`
`
`
`
`Sheetlvof18
`
`
`
`5,644,735
`
`NEsmg
`
`NW
`
`m0<m0hm
`
`Q“3
`
`.4
`
`\9.ON
`
`.m><._n_m_o
`
`Fan—Z—
`
`>4m2mmw<
`
`mm,
`
`N“
`
`WW
`
`D<m>m¥
`
`mm
`
`
`
`EEO...—Aim—mm
`
`mmhmOmm:
`
`
`
`NV
`
`V?
`
`m».
`
`0:
`
`0v
`
`mm.
`
`3.
`
`Fm.Wm.
`
`3....0
`
`Q»
`
`x0040
`
`0m.
`
`0003
`
`0003
`
`
`

`

`
`US. Patent
`
`
`
`Jul. 1, 1997
`
`
`
`
`
`Sheet 2 of 18
`
`
`
`
`
`
`5,644,735
`
`ASSIS‘
`
`Names Dates
`
`E tras
`
`Undo
`
`Find
`
`fgure 2
`
`0004
`
`0004
`
`

`

`
`US. Patent
`
`
`
`Jul. 1, 1997
`
`
`
`
`
`Sheet 3 of 18
`
`
`
`
`
`
`5,644,735
`
`100
`
`\
`
`.
`
`102
`
`
`
`Q)
`
`112
`
`Q
`uery
`
`106
`
`‘
`
`Yes
`
`No
`
`121
`
`No
`
`Select Best
`Alternative
`
`Yes
`
`No
`
`Yes
`
`130
`
`Selection
`
`132
`
`
`Yes Enter Selected
`Object(s) Into
`Made?
`
`Assistant
`
`Enter Object(s)
`Into Assistant
`
`
`
`
`
`Recognize
`@ Possible Intent?
`
`
`Y
`65
`
`
`135
`
`137
`
`Yes
`
`HypotheSLZe Intent
`
`ser Supplies
`Action?
`
` 139
`
`Hypothesis?
`
`
`Is There a
`
`Yes
`
`
`
`
`
`Prepare To
`
`Execute
`
`
`
`143
`
`
`
`
`
`
`Yes
`
`147
`
`EXCCUIB
`
`0005
`
`
`
`
`
`Provide List Of
`
`
`
`Alternatives
`
`
`User
`
`Selection
`
`
`
`117
`
`l 1
`
`9
`
`
`
`
`
`123
`
`Update Information
`
`
`& Linked
`
`Information
`
`
`
`
`Figure 3
`
`
`
`0005
`
`

`

`
`US. Patent
`
`
`
`
`Jul. 1, 1997
`
`
`
`
`
`Sheet 4 of 18
`
`
`
`5,644,735
`
`
`
`154
`
`
`
`
`
`
`
`
`
`Recognize
`
`
`User Input
`
`
`
`
`
`106 in Fig.3
`
`0006
`
`0006
`
`

`

`
`US. Patent
`
`
`
`
`
`Jul. 1, 1997
`
`
`
`
`
`Sheet 5 of 18
`
`
`
`
`5,644,735
`
`170
`
`Lama/c
`
`
`Name ______________________
`
`‘/
`
`
`
`
`
`r. Speaker 6Modem
`
`
`179
`183
`
`
`Phone
`
`
`
`181
`
`1/
`
`
`
`185
`
`
`
`189
`
`
`
`figure 415
`
`170
`
`rQarneg
`
`
`
`:[ESC]C](:
`
`
`
`Phone
`
`
`
`181
`
`At’//’
`
`
`
`“{,/”
`
`
`
`
`
`179
`
`183
`
`f. Speaker Q”Modem
`“62>
`
`185
`
`189
`
`figure 4c
`
`0007
`
`0007
`
`

`

`
`US. Patent
`
`
`
`Jul. 1, 1997
`
`
`
`
`
`Sheet 6 of 18
`
`
`
`
`
`
`5,644,735
`
`
`
`
`BIRTHDAY:
`
`TELEPHONE:
`FAX:
`
`ADDRESS:
`HEIGHT:
`
`
`
`
`
`
`
`
`
`180
`
`
`
`
`
`
`
`
`
`NAME:
`
`WEIGHT:
`
`INSTANCES: <PERSON-1> <PERSON-2> <PERSON—3>
`
`184
`
`1820
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`<PERSON-3> <ISAAC—3>
`182b
`NAME: ISAAC JONES
`
`
`
`
`<PERSON—2> <ISAAC—2>
`
`
`
`
`NAME: ISAAC ASIMOV
`
`
`
`<PERSON-1> <ISAAC-1>
`
`
`
`NAL'IE: ISAAC NEWTON
`
`
`
`BIRTHDAY: NULL
`
`
`TELEPHONE: 408-555-1212
`
`
`FAX: NULL
`
`
`
`
`
`
`ADDRESS: 123 MAIN STREET,
` HEIGHT: NULL
`
`
`
`WEIGHT: NULL
`
`
`
`
`
`[Figure 5
`
`0008
`
`0008
`
`

`

`
`US. Patent
`
`
`
`Jul. 1, 1997
`
`
`
`
`
`Sheet 7 of 18
`
`
`
`
`
`
`5,644,735
`
`170
`
`Isaac
`
`175
`
`/
`
`175'
`H
`.Name
`
`Phone
`
`
`
`181
`
`r. Speaker dModem
`
`
`179
`
`-183 185189
`
`/177 figure 6a
`
`
`
`
`
`
`
`170'
`
`\/ ISAAC NEWTON
`
`
`
`
`I SAAC AS IMOV
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`I SAAC JONES
`r1179
`
`
`
`
`
`. Name
`
`
`Phone
`
`
`
`
`
`
`
`
`170
`
`
`
`/ /
`
`
`177
`
`
`
`
`
`figure 65
`
`181
`
`
`r. Speaker OJModem
`
`
`
`179
`183
`185
`189
`
`
`
`‘-®
`
`
`
`
`
`I saac As imov
`
`
`
`
`
`181
`
`7’. Speaker 6Modem
`
`183
`179
`
`
`
`ii)
`
`170
`
`175
`
`177
`
`A(//
`
`A//’
`
`
`
`185189
`
`
`
`
`0009
`
`0009
`
`

`

`
`US. Patent
`
`
`
`Jul. 1, 1997
`
`
`
`
`
`Sheet 8 of 18
`
`
`
`
`
`5,644,735
`
`121
`
`l 80
`
`Make Best Guess
`
`
`
`Based Upon A
`
`Selected Heuristic
`
`
`
`
`
`
`
`123
`
`Obtain Data From
`
`
`Corresponding
`
`
`Field Types
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`Enter Obtained
`
`
`Data Into
`
`Smart Fields
`
`
`
`206
`
`
`
`0010
`
`
`
`
`Figure 8a
`
`0010
`
`

`

`
`US. Patent
`
`
`
`
`
`Jul. 1, 1997
`
`
`
`
`
`
`Sheet 9 of 18
`
`
`
`5,644,735
`
`170
`
`
`
`Isaac Asimov
`
`
`
`
`phone 1—202—555-1234
`
`
`
`181
`C:jjflodem
`
`183
`
`NS
`
`/
`
`177
`/ ‘
`
`
`figure 85
`
`189
`
`dfié)
`
`
`
`
`
`
`
`0011
`
`0011
`
`

`

`
`US. Patent
`
`
`
`
`Jul. 1, 1997
`
`
`
`
`
`
`
`
`Sheet 10 of 18
`
`5,644,735
`
`130
`
`220
`
`222
`
`
`
`Any Objects
`
`
`Explicitly
`
`Selected?
`
`
`
`224
`
`
`
`
`
`226
`
`Any Objects
`Since Last
`Timeout?
`
`Yes
`
`
`Figure 9a
`
`
`
`0012
`
`0012
`
`

`

`
`
`mama'S'fl
`
`L661‘ITHE
`
`SIJ0[I”WIS
`
`SSL‘W9‘S
`
`INSTANCE
`
`
`
`
`
`
`2 1
`
`
`2
`
`
`100
`
`
`IS AAC- 1
`
`ISAAC-2
`
`
`ISAAC—3
`
`
`
`
`
`TIMES USED
`
`
`
`figure 76
`
`figure 7c
`
`
`CALL ISAAC
`
`
`
`
`
`figure 96
`
` CALL ISAAC
`
`figure 9c
`
`0013
`
`INSTANCE
`
`
`
`
`
`ORDER ISAAC- 1
`
`
`ISAAC-2
`
`
`
`ISAAC—3
`
`1
`
`2
`
`3
`
`
`
`
`
`INS TANCE
`
`
`
`LAST USED
`
`
`
`
`
`
`T-l
`
`ISAAC-1
`
`ISAAC-2
`T-2
`
`
`ISAAC—3
`T-3
`
`
`. figure 7a
`
`0013
`
`

`

`
`US. Patent
`
`
`
`Jul. 1, 1997
`
`
`
`
`
`Sheet 12 of 18
`
`
`
`
`
`
`5,644,735
`
`170
`
`
`
`175'
`
`(Haas, CALL ISAAC
`
`
`figure 9d
`
`' 170
`
`figure 10a
`
`170
`
`175'
`
`
`(Haas, CALL ISAAC
`
`
`figure 106
`
`
`
`0014
`
`0014
`
`

`

`
`US. Patent
`
`
`
`
`Jul. 1, 1997
`
`
`
`'
`
`
`
`
`Sheet 13 of 18
`
`
`
`5,644,735
`
`135
`
`240
`
`242
`
`244
`
`
`I o The Object(s
`
`
`Match At Least The
`
`Template?
`
`
`
`
`
`Figure 11a
`
`
`
`0015
`
`0015
`
`

`

`
`US. Patent
`
`
`
`Jul. 1, 1997
`
`
`
`
`
`Sheet 14 of 18
`
`
`
`
`
`
`5,644,735
`
`<PERSON—2>
`
`<ACTION— 10>
`
`
`
`
`
`
`
`
`"ISAAC"
`
`figure 115
`
`H
`
`
`
`
`
`Find
`
`
`
`
`
`
`
`
`-_..-
`
`
`
`
`
`
`
`
`
`
`
`
`
`{Figure 11c
`
`0016
`
`0016
`
`

`

`
`US. Patent
`
`
`
`Jul. 1, 1997
`
`
`
`
`
`Sheet 15 of 18
`
`
`
`
`
`
`5,644,735
`
`137
`\
`
`141
`
`260
`
`Enumerate Possible
`Intents
`
`
`262
`
`Prion'tize And
`
`
`Choose Best
`Guess
`
`264
`
`
`
`
`
`Figure 12a
`
`
`
`
`
`
`Yes
`
`Present List Of
`
`274
`
`276
`
`
`
`
`
`
`
`Applicable Actions
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`278
`
`
`
`
`
`Figure 12b
`
`
`
`0017
`
`0017
`
`

`

`
`US. Patent
`
`
`
`Jul. 1, 1997
`
`
`
`
`
`Sheet 16 of 18
`
`
`
`
`
`
`5,644,735
`
`170"
`
`
`CALL
`
`FAX
`
`
`F IND
`
`170
`
`
`
`
`
`
`MAI L
`
`
`SCHEDULE
`
`
`
`
`
`figure 125
`
`figure 121
`
`
`r373
`
`.Please CALL ISAAC
`
`
`
`
`
`
`189
`
`170
`
`185
`
`0018
`
`0018
`
`

`

`
`US. Patent
`
`
`
`Jul. 1, 1997
`
`
`
`
`
`Sheet 17 of 18
`
`
`
`
`
`
`5,644,735
`
`290
`
`143
`
`
`
`
`Fill In Plan Template
`
`
`
`& Identify Missing
`Preconditions
`
`
`292
`
`293
`
`Resolve Missing
`Preconditions To
`
`
`Extent Possible
`
`
`
`
`
`294
`
`ait For Additional
`
`Precondinons?
`
`,
`
`
`
`No
`
`
`
`
`
`
`
`
`
`
`Continuation
`
`
`
`Slip, If Any
`
`
`
`
`
`
`
`
`
`
`
`
`
`Figure 14a
`
`
`
`0019
`
`0019
`
`

`

`US. Patent
`
`Jul. 1, 1997
`
`Sheet 13 of 18
`
`5,644,735
`
`270
`
`Name
`
`Fax #
`
`175
`
`177
`
`/
`
`/
`
`189
`
`Format Plain
`Ii] Cover Page
`
`[I Manual Dialing
`
`figure 146
`
`0020
`
`0020
`
`

`

`
`5fi4¢735
`
`
`
`
`
`
`
`
`2
`
`
`
`
`
`
`
`assistance could be provided is recognized. However, such
`
`
`
`
`
`
`
`an approach may provide unsatisfactory results when the
`
`
`
`
`
`
`
`
`
`user is provided with assistance that is unwanted or dispro-
`portionate.
`
`
`
`
`
`
`
`
`
`Simply stated, concerns have arisen about assist functions
`
`
`
`
`
`
`
`
`being undertaken by the computer without adequate user
`control and interaction. When the assist function has been
`
`
`
`
`
`
`
`
`
`undertaken without adequate user control, assistance would
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`be provided awkwardly and at times when the assistance
`
`
`
`
`was not necessarily desired.
`
`
`
`
`
`
`
`
`
`
`It is essential that the control, the timing, and the appli-
`
`
`
`
`
`
`
`
`
`cation of the assistance is considered appropriate and well—
`tuned. The confidence of the user is undermined when the
`
`
`
`
`
`
`
`
`
`
`assistance provided only obliquely addresses particularized
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`user needs without precisely providing the particular results
`objectively and subjectively required.
`
`
`
`
`SUMMARY OF THE INVENTION
`
`
`
`
`According to the invention, a method and apparatus has
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`been developed for providing computer—assisted implicit
`
`
`
`
`
`
`
`
`and explicit assistance for a variety of user~supportive
`
`
`
`
`
`
`
`
`information functions. If no implicit assist actions are
`
`
`
`
`
`
`
`
`
`
`desired or indicated, then a logical process is initiated to
`
`
`
`
`
`
`
`determine whether explicit assistance should be undertaken.
`
`
`
`
`
`
`
`
`If implicit assistance is indicated, a list of action alternatives
`
`
`
`
`
`
`
`
`
`
`is displayed for the user. Alternatively, a process can be
`undertaken to automatically select a best action alternative
`
`
`
`
`
`
`
`of several identified alternatives.
`
`
`
`
`
`
`
`
`
`
`
`
`
`If explicit assistance is desired by the user, particular
`
`
`
`
`
`
`
`
`
`object(s) from which the assistance may be inferred are
`
`entered into an assistance operation. An attempt is then made
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`to recognize possible intents expressed by the objects
`entered into the assistance process. If no user intent is, in
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`fact, recognized, the assist operation is usually terminated. If
`
`
`
`
`
`
`
`
`
`
`a possible intent is recognized, the actual intent is hypoth—
`esized. A check is further undertaken, to determine whether
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`a hypothesis is in fact available. If no hypothesis is found,
`
`
`
`
`
`
`
`
`
`the process permits the user to supply a proposed action. If
`
`
`
`
`
`
`
`
`
`
`no hypothesis is found and no user action is proposed,
`
`
`
`
`
`
`
`
`assistance efforts terminate. However, if a hypothesis is
`available, preparations for execution are undertaken. A final
`
`
`
`
`
`
`
`
`inquiry is made as to whether to undertake the hypothesized
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`assist. If the response to an inquiry whether to assist as
`
`
`
`
`
`
`
`hypothesized is affirmative, execution of the hypothesized
`action is undertaken.
`
`
`
`
`
`
`
`
`
`
`
`
`
`Accordingly, it is an intent of the invention to provide user
`
`
`
`
`
`
`
`
`explicit assist functions which are provided under oversight
`
`
`
`
`
`
`
`
`
`
`and with the interaction of the user, and implicit assist
`functions in certain structured instances. The involvement of
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`the user in the implementation of implicit assist operations
`is a key aspect of the invention. User intent is not merely
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`deduced, but specific user queries are made to ensure
`
`
`
`
`
`controlled application of assist operations.
`
`
`
`
`
`
`
`
`These and other advantages of the present invention will
`
`
`
`
`
`
`
`become apparent upon reading the following detailed
`
`
`
`
`
`
`
`
`
`descriptions and studying the various figures of the draw—
`ings.
`
`
`
`
`10
`
`
`15
`
`
`
`20
`
`
`25
`
`
`
`30
`
`35
`
`
`
`
`
`45
`
`
`
`50
`
`55
`
`
`
`
`1
`METHOD AND APPARATUS FOR
`
`
`
`
`PROVIDING IMPLICIT COMPUTER-
`
`
`IMPLEMENTED ASSISTANCE
`
`
`
`
`
`
`
`
`
`
`
`
`
`Cross-Reference to a Related Application
`
`
`
`
`
`
`
`This application is divisional of US. patent application
`Ser. No. 08/099,861, filed on Jul. 30, 1993, now US. Pat.
`
`
`
`
`
`
`
`
`
`
`
`No. 5,477,447, under the title “METHOD AND APPARA-
`
`
`
`
`
`
`
`
`TUS FOR PROVIDING COMPUTER-IMPLEMENTED
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`ASSISTANCE” on behalf of Luciw et. a1. and assigned to
`the same assignee as herein; which application is incorpo—
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`rated herein by reference in its entirety. Application Ser. No.
`08/099,861 is a continuation—in—part of Ser. No. 889,225,
`
`
`
`
`
`
`
`
`issued US. Pat. No. 5,390,281 filed May 27, 1992 and
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`issued Feb. 14, 1995 and which is incorporated herein by
`reference in its entirety. Priority rights based upon this
`
`
`
`
`
`
`
`
`
`earlier filed United States Patent are claimed.
`
`
`
`
`
`
`
`BACKGROUND OF THE INVENTION
`
`
`
`
`
`
`
`
`
`
`
`invention relates generally to computer
`The present
`
`
`
`
`
`systems, and more particularly to computer-implemented
`assistance methods and apparatus.
`
`
`
`
`
`
`
`
`
`
`Computerized personal organizers are becoming increas-
`
`
`
`
`
`
`
`
`
`ingly popular with a large segment of the population.
`Computerized personal organizers tend to be small,
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`lightweight, and relatively inexpensive, and can perform
`such functions as keeping a calendar, an address book, a
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`to-do list, etc. While many of these functions can also be
`
`
`
`
`
`
`
`provided in conventional computer systems, personal orga-
`
`nizers are very well suited to the personal organization task
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`due to their small size and portability. Personal organizers
`
`
`
`
`
`
`
`
`
`
`are available from such companies as Sharp and Casio of
`Japan.
`
`
`
`
`
`
`
`
`
`
`A relatively new form of computer, the pen-based com-
`
`
`
`
`
`
`
`
`
`
`
`puter system, holds forth the promise of a marriage of the
`power of a general purpose computer with the functionality
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`and small size of a personal organizer. A pen-based com-
`
`
`
`
`
`
`
`
`puter system is typically a small, hand-held computer where
`
`
`
`
`
`
`
`
`
`the primary method for inputting data includes a “pen” or
`
`
`
`
`
`
`
`stylus. A pen—based computer system is commonly housed in
`a generally rectangular enclosure, and has a dual-function
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`display assembly providing a viewing screen along one of
`
`
`
`
`
`
`
`
`
`the planar sides of the enclosure. The dual-function display
`
`
`
`
`
`
`
`
`
`
`assembly serves as both an input device and an output
`
`
`
`
`
`
`
`
`
`device. When operating as an input device, the display
`
`
`
`
`
`
`
`
`
`
`
`
`assembly senses the position of the tip of a stylus on the
`
`
`
`
`
`
`
`
`viewing screen and provides this positional information to
`
`the computer’s central processing unit (CPU). Some display
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`assemblies can also sense the pressure of the stylus on the
`
`
`
`
`
`
`
`
`
`screen to provide further information to the CPU. When
`
`
`
`
`
`
`
`
`
`operating as an output device, the display assembly presents
`
`
`
`
`
`computer-generated images on the screen.
`
`
`
`
`
`
`
`The dual-function display assemblies of pen-based com-
`
`
`
`
`
`
`
`
`
`
`puter systems permit users to operate the computer as a
`
`
`
`
`
`
`
`computerized notepad. For example, graphical images can
`be input into the pen-based computer by merely moving the
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`stylus on the surface of the screen. As the CPU senses the
`
`
`
`
`
`
`
`
`
`position and movement of the stylus, it generates a corre-
`
`
`
`
`
`
`
`
`
`
`
`sponding image on the screen to create the illusion that the
`
`
`
`
`
`
`
`
`
`
`
`stylus is drawing the image directly upon the screen, i.e. that
`
`
`
`
`
`
`
`
`
`
`
`the stylus is “inking” an image on the screen. With suitable
`recognition software, text and numeric information can also
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`be entered into the pen—based computer system in a similar
`fashion.
`
`
`
`
`
`
`
`
`
`One approach to computerized assist operations is to
`
`
`
`
`
`
`
`
`provide assistance automatically when a situation in which
`
`
`
`65
`
`
`
`BRJEF DESCRIPTION OF THE DRAWINGS
`
`
`
`
`
`
`FIG. 1 is a block diagram of a computer system in
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`accordance with the present invention.
`
`
`
`
`
`
`
`
`
`
`
`FIG. 2 is a top plan view of the screen, case, and keypad
`
`
`
`
`
`
`
`of the computer system of FIG. 1.
`
`FIG. 3 is a flow diagram of a process according to the
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`invention for providing controlled computer—assisted user
`assistance.
`
`
`
`
`
`
`0021
`
`0021
`
`

`

`i64£735
`
`
`
`
`
`
`
`
`
`
`3
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`FIG. 4a shows a process for determining Whether or not
`
`
`
`
`
`
`
`
`implicit assistance is desired by the user, in connection with
`providing computer assisted support to the user.
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`FIG. 4b shows an example of an implicit assist operation
`with a phone slip window having a smart name field evoked,
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`for example, by either highlighting the verb “call” or by
`
`
`
`
`
`
`
`
`
`writing it on the note field before evoking window.
`
`
`FIG. 4c shows the phone slip window of FIG. 4b with the
`
`
`
`
`
`
`
`
`
`
`
`
`
`name formal font form. ISAAC having been recognized and
`established in the name field in
`
`
`
`
`
`
`FIG. 5 shows an example of a generic <PERSON> type
`
`
`
`
`
`
`
`
`
`frame along with a particular set of specific frames of the
`
`
`
`
`
`
`
`
`
`
`<PERSON> type.
`
`
`
`
`
`
`
`
`
`FIGS. 6a—60 show respective assist windows in succes-
`
`
`
`
`
`
`
`
`
`sive stages of an assist process, including first a window
`containing a first informational level directed at the name
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`ISAAC alone, a second window with a pop-up menu offer-
`
`
`
`
`
`
`
`
`ing a user choice among several known ISAACs, and a third
`
`
`
`
`
`
`
`
`
`window showing the selection of a particular ISAAC, that is
`
`
`
`
`
`ISAAC ASIMOV, having been accomplished.
`
`
`
`
`
`
`
`
`FIG. 7 illustrates a brief flow diagram illustrating a
`
`
`
`
`
`
`
`
`heuristic process for the selection of a particular choice
`
`
`
`
`
`
`
`
`
`among alternatives when the user is not queried for selection
`of alternatives.
`
`
`
`
`
`
`
`
`
`
`
`FIGS. 7a—7c show selected examples of heuristic rules of
`
`
`
`
`
`
`
`
`thumb which are effective in making automatic choices
`between alternative ISAACs, respectively directed toward
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`selection schemes such as selecting the last used ISAAC,
`
`
`
`
`
`
`
`
`
`
`selecting the last in order of ISAACs according to particular
`
`
`
`
`
`
`
`
`
`position within a selected table, and finally selecting a
`particular ISAAC based upon prior frequency of choice of
`
`
`
`
`
`
`
`
`
`that particular ISAAC.
`
`
`
`
`
`
`
`
`
`
`FIG. 8a is a flow diagram illustrating the updating process
`for data base information in linked smart fields.
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`FIG. 8b is a call slip illustration of an updated smart field
`
`
`
`
`
`
`
`
`Window in which the phone number field information has
`been updated.
`
`
`
`
`
`
`
`
`
`
`FIG. 9a is a flow diagram of the process according to the
`
`
`
`
`
`
`
`
`
`invention in which a query is made as to whether a specific
`
`
`
`
`
`
`
`
`selection has been made as to a particular object.
`
`
`
`
`
`
`FIG. 9b-9c indicate graphically the performance of the
`selection query operation as expressed in FIG. 9A.
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`FIG. 9d illustrates the transferal of the highlighted objects
`of FIG. 9c transferred to a selected window.
`'
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`FIG. 10a illustrates the input of a handwritten object into
`a smart field in a window.
`
`
`
`
`
`
`
`
`
`
`
`
`
`FIG. 10b illustrates the recognition of the handwritten
`
`object of FIG. 10a arid its conversion into formal font form.
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`FIG. 11a is a flow diagram illustrating the recognition of
`objects process.
`
`
`
`
`
`
`
`
`
`FIG. 11b illustrates an object combination under opera-
`tion.
`
`FIG. 110 shows a template for organizing in preset form
`
`
`
`
`
`
`
`
`a variety of object combinations.
`
`
`
`
`
`
`
`
`
`
`
`FIG. 12a illustrates a process for hypothesizing user
`intent as to particular activities.
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`FIG. 12b is a flow diagram setting forth a process for
`
`
`
`
`
`
`
`
`
`determining whether the user wishes to provide or supply a
`particular action.
`
`
`
`
`
`
`
`
`
`FIG. 120 shows a window with a menu partially overlap-
`
`
`
`
`
`
`
`
`
`
`
`
`ping its topside in order to provide the user with an array of
`activity choices.
`
`
`
`
`
`
`
`
`
`
`
`
`FIG. 12d shows the window of FIG. 12c with the activity
`
`
`
`
`
`
`
`of calling having been selected, establishing the combined
`
`
`
`
`
`
`
`
`
`objects CALL and ISAAC in the activity field of the
`window.
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`4
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`FIG. 13 illustrates the process for preparing for execution
`of a particular activity.
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`FIG. 14a is a short flow diagram of a process for deter-
`
`
`
`
`
`
`
`
`mining whether to proceed with execution of a particular
`
`
`
`selected assist activity.
`
`
`
`
`
`
`
`
`FIG. 14b illustrates an example of a confirmation of
`
`
`
`
`
`
`
`
`
`
`action slip that could be produced upon completion of a
`
`
`
`
`
`
`
`
`particular activity, in this case completion of the process of
`
`
`
`
`faxing information to another party.
`DETAILED DESCRIPTION OF THE
`
`
`
`PREFERRED EMBODIMENTS
`
`
`
`
`
`
`
`
`
`
`The present invention is well suited for pointer based
`
`
`
`
`
`computer systems such as the pen-based, pen-aware and
`
`
`
`
`
`
`
`
`mouse controlled systems that are currently popular. For the
`purposes of illustration, the invention will be described in
`
`
`
`
`
`
`
`
`
`
`
`
`
`connection with a pen—based system.
`
`
`
`
`
`
`
`
`
`As shown in FIG. 1, a pen-based computer system 10 in
`accordance with the present invention includes a central
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`processing unit (CPU) 12, read only memory (ROM) 14,
`
`
`
`
`
`
`
`random access memory (RAM) 16, input/output (I/O) cir-
`
`
`
`
`
`
`
`
`cuitry 18, and a display assembly 20. The pen-based com-
`
`
`
`
`
`
`
`
`puter system 10 may also optionally include a mass storage
`
`
`
`
`
`
`
`
`
`
`
`unit 22, a keypad (or keyboard) 24, a serial port 26, an
`
`infrared (I/R) port 28, and a clock 30.
`
`
`
`
`
`
`
`
`
`
`
`The CPU 12 is preferably a commercially available,
`
`
`
`
`
`
`
`
`single chip microprocessor. While CPU 12 can be a complex
`
`
`
`
`
`
`
`
`instruction set computer (CISC) chip, it is preferable that
`
`
`
`
`
`
`
`
`CPU 12 be one of the commercially available, reduced
`
`
`
`
`
`
`
`
`instruction set computer (RISC) chips which are known to
`be of generally higher performance than CISC chips. CPU
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`12 is coupled to ROM 14 by a unidirectional data bus 32.
`
`
`
`
`
`
`
`
`
`ROM 14 contains the basic operating system for the pen-
`based computer system 10. CPU 12 is connected to RAM 16
`
`
`
`
`
`
`
`
`
`
`by a hi-directional data bus 34 to permit the use of RAM 16
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`as scratch pad memory. ROM 14 and RAM 16 are also
`
`
`
`
`
`
`
`
`coupled to CPU 12 by appropriate control and address
`busses, as is well known to those skilled in the art. CPU 12
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`is also coupled to the I/O circuitry 18 by bi-directional data
`
`
`
`
`
`
`
`
`
`bus 36 to permit data transfers with peripheral devices.
`I/O circuitry 18 typically includes a number of latches,
`
`
`
`
`
`
`
`
`registers and direct memory access (DMA) controllers. The
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`purpose of I/O circuitry 18 is to provide an interface between
`
`
`
`
`
`CPU 12 and such peripheral devices as display assembly 20,
`
`
`
`
`
`
`
`
`
`
`
`
`
`mass storage 22, keypad 24, serial port 26, and HR port 28.
`
`
`
`
`
`Display assembly 20 of pen-based computer system 10 is
`
`
`
`
`
`
`
`both an input and an output device. Accordingly,
`it is
`
`
`
`
`
`
`
`
`
`
`coupled to I/O fi‘om a variety of vendors. The input device
`
`
`
`
`
`
`
`
`
`of display assembly 20 is preferably a thin, clear membrane
`
`
`
`
`
`
`
`
`
`
`
`which covers the LCD display and which is sensitive to the
`
`position of a stylus 38 on its surface. With such a structure,
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`the display assembly 20 can serve as an input “tablet.” These
`
`
`
`
`
`
`
`
`position sensitive membranes are also readily available on
`
`
`
`
`
`
`
`
`the commercial market Alternatively, other types of tablets
`
`
`
`
`
`
`
`
`
`can be used, such as inductively coupled tablets. Combina-
`
`
`
`
`
`
`
`
`
`tion display assemblies such as display assembly 20 which
`
`
`
`
`
`
`
`
`
`include both the LCD and the input membrane are cormner-
`
`
`
`
`
`
`
`
`cially available from such vendors as Scriptel Corporation of
`
`
`Columbus, Ohio.
`
`
`
`
`
`
`
`
`
`Some type of mass storage 22 is generally considered
`
`
`
`
`
`
`
`
`
`
`
`desirable. Mass storage 22 can be coupled to I/O circuitry 18
`
`
`
`
`
`
`
`
`
`
`by a bi-directional data bus 40. However, the mass storage
`
`
`
`
`
`
`
`
`
`
`22 can be eliminated by providing a sufficient amount of
`
`
`
`
`
`
`
`
`
`
`
`RAM 16 to store user application programs and data. In that
`
`
`
`
`
`
`
`
`
`
`
`case, the RAM 16 can be provided with a backup battery to
`
`
`
`
`
`
`
`
`
`
`prevent the loss of data even when the pen-based computer
`
`
`
`10
`
`
`15
`
`20
`
`
`
`
`
`25
`
`30
`
`
`
`35
`
`
`
`
`
`45
`
`
`
`50
`
`
`
`55
`
`
`
`
`
`65
`
`
`
`0022
`
`0022
`
`

`

`5
`
`
`system 10 is turned off. However, it is generally desirable to
`
`
`
`
`
`
`
`
`
`
`have some type of long term mass storage 22 such as a
`
`
`
`
`
`
`
`
`
`
`commercially available miniature hard disk drive, nonvola-
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`tile memory such as flash memory, battery backed RAM, a
`Personal Computers Memory Card International Association
`
`
`
`
`
`
`(PCMCIA) card, or the like.
`
`
`
`
`
`The keypad 24 can comprise an array of mechanical
`
`
`
`
`
`
`buttons or switches coupled to I/O circuitry 18 by a d