United States Patent
`
`[191
`
`[11] Patent Number:
`
`4,972,496
`
`Sklarew
`
`[54]
`
`[75]
`
`[73]
`
`[21]
`
`[22]
`
`[63]
`
`[51]
`[52]
`
`[53]
`
`[56]
`
`HANDWRI’ITEN KEYBOARDLESS ENTRY
`COMPUTER SYSTEM
`
`Inventor: Ralph Sklarew, Reston, Va.
`
`Assignee: Grid Systems Corporation, Fremont,
`Calif.
`
`App]. No.: 29,772
`
`Filed:
`
`Mar. 24, 1987
`
`Related US. Application Data
`' Continuation-impart of Ser. No. 889,513, Jul. 25, 1986,
`abandoned.
`
`Int. Cl.5 ....................................... G06K 9/22
`US. Cl. ........................................ 382/13; 382/57;
`382/59; 178/18
`Field of Search ....................... 382/13, 24, 57, 14;
`340/706, 707, 709; 178/18, 19, 20
`References Cited
`U.S. PATENT DOCUMENTS
`
`3,699,439 10/1972 Turner .
`3,832,693 8/ 1974 Ishizaki et al. ...................... 340/707
`3,903,517 9/1975 Hafner
`382/57
`
`4,112,415 9/1978 Hilbrink
`382/13
`4,177,354 12/1979 Mathews
`..... 178/ 18
`4,177,448 12/1979 Brayton
`..... 382/14
`4,184,147
`1/1980 Seelbach ..........
`382/13
`
`..... 382/13
`4,365,235 12/1982 Greanias et a1.
`6/1984 Schlosser et a1. ........... 178/ 19
`4,456,787
`
`4,475,239 10/1984 van Raamsdonk
`..... 382/57
`
`3/1986 Inagaki ........................ 382/13
`4,578,811
`.................... 382/13
`4,641,354
`2/1987 Fukunaga et a1.
`
`[45] Date of Patent:
`
`Nov. 20, 1990
`
`6/1987 Yamakawa ............................ 382/ 13
`4,672,677
`7/1987 Lukis ...................
`178/19
`4,679,241
`4,680,430 7/1987 Yoshikawa et a1.
`178/19
`
`4,680,804 7/ 1987 Kuzunuki et a1.
`.................... 382/ 13
`
`FOREIGN PATENT DOCUMENTS
`
`55-143681 10/1980 Japan ..................................... 382/13
`2117154 10/1983 United Kingdom .................. 382/13
`
`Primary Examiner—Leo H. Boudreau
`Attorney, Agent, or Firm—Townsend & Townsend
`
`[5 7]
`
`ABSTRACT
`
`A keyboardless entry computer system includes a trans-
`parent input screen that generates positional informa-
`tion when contacted by a stylus, and a display screen
`mounted physically below the input screen such that a
`character that is displayed can be seen below the input
`screen. The system includes a computer that has been
`programmed to compile the positional information into
`Strokes, to calculate Stroke characteristics, and then
`compare the Stroke characteristics with those stored in
`a database in order to recognize the symbol drawn by
`the stylus. Key features of the system are: (1) transpar-
`ent position sensing subsystem; (2) underlying display
`on which to mimic drawing of sensed positions and to
`show characters or symbols;
`(3) means to convert
`sensed positions first into plotted Points and then into
`recognized characters or symbols; and (4) means to
`“learn” to associate sensed input positions with a char-
`acter or symbol.
`
`21 Claims, 14 Drawing Sheets
`
`
`
`FANDANGO EXHIBIT 1071
`
`FANDANGO EXHIBIT 1071
`
`

`

`US. Patent
`
`Nov. 20, 1990
`
`Sheet 1 of 15
`
`4,972,496
`
`
`
`FIG.
`
`2
`
`21G. 2A
`
`

`

`US. Patent
`
`Nov. 20, 1990
`
`Sheet 2 of 15
`
`4,972,496
`
`3?
`
`=
`
`33
`
`=
`
`37
`
`3e \
`
`~ (Xhfl)
`
`37
`
`=
`
`38
`
`~
`
`=
`
`37
`
`INTERFACE
`AND
`MULTIPLEXER
`
`42
`‘
`
`'
`
`43
`
`46
`
`ANALOG T0
`
`DIGITAL CONVERTER
`
`44
`
`PORT
`
`MICROCONTROLLER
`
`
`
`62
`
`58
`
`|
`l
`:
`I
`
`RS 232
`
`INTERFACE
`
`
`
`‘ BATTEE/E,
`
`LITHIUM
`
`_
`BATTERY
`PACK
`
`L'
`
`60
`
`_J
`
`INPUT SCREEN
`
`64
`
`66
`
`7
`DISPLAY
`iNTERFACE
`I(fi|%%fi)A§0EN%%ElEgI§¥
`1 0
`T
`t
`INPUT SCREEN
`.
`DISPLAY
`‘6
`I»
`STYLUS
`.
`___2‘- 79.4
`
`SCREEN
`
`

`

`US. Patent
`
`Nov. 20, 1990
`
`Sheet 3 of 15
`
`4,972,496
`
`75
`
` CONVERT x, Y VOLTAGES
`
`73
`
`80
`
`
`T0 DIGITAL X,Y
`
`~LOCATING SIGNALS
`
` CONVERT X,Y LOCATING
`
`
`SIGNALS T0 STROKE
`CHARACTERISTICS
`
`
` COMPARE STROKE
`
`
`CHARACTERISTICS
`TO DATA BASE
`
`82
`
`
`
`
`
`SYMBOL
`
`
`RECOGNIZED
`
`
`REQUEST MORE
`INFORMATION
`
`DISPLAY
`
` 86
`SYMBOL
`
`

`

`US. Patent
`
`Nov. 20, 1990
`
`Sheet 4 of 15
`
`4,972,496
`
`COMPUTER
`OPERATING
`SYSTEM
`
`ATAAATAN
`(8.9. EDITOR)
`
`90
`
`92
`
`HANDWRiTlNG
`RECOGNITION
`PROGRAM
`
`95
`
`94
`
`FUNCTION
`
`x,y POINT ENCODING
`INTO STROKES
`SUBROUTINE
`
`STROKE
`CHARACTERIZATION
`SUBROUTINE
`
`STROKE
`COMPARISON
`SUBROUTINE
`
`UPDATE STROKE
`COMPARISON
`SUBROUTINE
`
`NO
`
`NZ
`
`
`”4
`
`X,Y POINTS
`
`
`
`INCORPORATE
`-
`INTO
`STROKES
`
`CHARACTERIZE STROKES
`(LENGTH, SLOPE,
`CURVATURE, ETC.)
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`||8
`
`II6
`
`RECOGNIZE
`CHARACTERS
`
`FOUND
`
`COMPARE STROKE AND
`SEQUENCE OF STROKES
`WITH DATA BASE
`
`NOT FOUND
`
`SHUFFLE DATA BASE
`T0 PRIORITIZE
`SYMBOL USAGE
`
`
`QUERY USER TO
`
`ADD TO DATA BASE
`
`Fig. 7
`
`

`

`US. Patent
`
`Nov. 20, 1990 \
`
`Sheet 5 of 15
`
`4,972,496
`
`-1"79- 84
`
`START
`
`'50
`
`
`
`READ X,Y
`
`VOLTAGES
`
`
`YES
`
`(STROKE
`I56
`DONE)
`
`
`‘<3
`
`
`POINTS IN
`STROKE
`
`
`STROKE
`
`CHARACTERIZATION
`ROUTINE
`
`
`no
`
`'
`
`‘
`
`SCALE
`VOLTACES
`
`
`
`
`
`
`
`
`FILTER
`
`ERRONEOUS
`
`
`POINTS
`
`
`IDENTIFY STROKE,
`AS DOT
`
`
`
`CALCULATE
`AVERAGE HHCHT
`
`
`
`IYI
`
`
`
`INCREMENT
`POINT
`COUNTER
`
`
`
`mmC:
`
`COMPARE STROKEISI
`WITH DATA BASE ENTRY
`
`YES
`
`DIFFERENT
`4* STROKES
`
`IT4
`
`NO
`
`I?
`
`6
`
`(GET NEXT POINT)
`
`SELECT NEW
`DATA BASE
`ENTRY
`
`r
`
`EVG-HABL
`>éLINE
`
`NO
`
`#*PTS.
`DIFFER>ID
`
`NO
`
`I78
`
`

`

`US. Patent
`
`Nov. 20, 1990 .
`
`Sheet 6 of 15
`
`4,972,496
`
`-
`o fl - 85'
`
`CALCULATE
`GAUGE
`
`SAVE MINIMUM
`COMPARISON
`
`N0
`
`LAST
`DATA BASE
`ENTRY
`
`I82
`
`I83
`
`I84
`
`YES
`
`'
`
`ADD
`“0
`TO DATA
`
`BASE
` ADD TO BOTTOM
`
`'ACT ON
`COMMANDS
`
`OF DATA BASE
`
`ASK USER IF STROKE
`TO BE ADDED TO
`DATA BASE
`
`I9 5
`
`SHUTTLE
`DATA BASE
`
`ZERO POINTS AND
`INCREMENT COUNTER
`
`mSPLAY CHARACTER
`AT STROKE POSITION
`
`I96
`
`I98
`
`200
`
`

`

`US. Patent
`
`Nov. 20, 1990 |
`
`Sheet 7 of 15
`
`4,972,496
`
`STROKE
`CHARACTERIZATION
`SUBROUTI NE
`
`
`
`
`ITO
`
`CALCULATE
`POINT T0 POINT SLOPE
`
`
`
`
`
`
`
`
`
`
`
`
`CALCULATE POINT
`ABOVE BASELINEI
`
`
`
`HE
`HA
`
`IGHT
`BL)
`
`220
`
`222
`
`224
`
`226
`
`228
`
`Fig. 9
`
`
`
`

`

`US. Patent
`
`Nov. 20, 1990 .
`
`-
`
`Sheet 8 of 15
`
`4,972,496
`
`I - Printed Circuit hoard
`_
`, Thehoard’ssstze, cost, power consuuption, and
`reliability are critical to tie sucess of our product.
`
`The hoard Must ilt in our package, which llwlts lts
`.
`diwensmns to sowethmg like 10' XS , or about 50 s uare
`inches.
`Roow wust he;availahle for dealer installahie
`options such as additional weuorg
`and wodew - possehily as
`piggy hack hoards.A
`f
`h tt Bower consuuptgontwust he 19: enoughhscttgaf the
`a ms in
`1e pro uc
`can provl e or a ou
`hours 0
`operation and about 2 weeks of data retention. These are
`excluslve, 1.e.
`8 hours of operatlon followed hp 2 weeks of
`data retention IS required. With this kind of power
`consuwption heat should not he a prohlew.
`
`that we
`Given our cost and power requirewents we expect
`,
`will he usong CHOS technology through out.
`He a so ex ect
`to use Surface Mounted Iechnoloqg(8 T).
`SHT allows a
`enser
`packing. wore shock resxstence. and lower Manufacturing
`The
`costs, althouoh inltlh1.60515 not he slothlu hluher.
`final decISLon on ShI will depen
`upon tie availability of
`
`l - Printed Circuit.Board
`
`_ The_hoard’s size. cost. power consuuption. and
`reliability are critical to tie sucess of our product.
`
`The hoard uust fit in our package, which liwits'its
`,
`dowensions to souething like 18‘ X 5 , or about 50 suuare
`inches. Roou uust he3availahle for dealer installah e
`options such as additional ueworg
`and wodew - po551hilg as
`P1999 hack hoards.)\\
`
`I;
`
`.
`1
`.
`,.
`D.
`,
`,
`.,...
`hattries in the product can provide for about 8 hours of
`operation and about 2 weeks of data retention.
`These are
`exclusove.
`;.e. 8 hours of operation followed by 2 weeks of
`data retention TS required. With this kind of power
`consuwption heat should not he a prohlew.
`
`'
`
`
`
`Given our cost and power requireuents we expect
`
`that we
`
`I - Printed Circuit Board
`
`. The.hoard's size, cost, power consuwption, and
`reliability are critical to tie sucess of our product.
`
`The hoard wust fit in our package, which liwits its
`.
`dlwenSIons to sowething like 10
`X 5 . or about 50 souare
`meMmmMuthMhMmmmwe
`options such as additional weworu
`and uodew - POSSlhlIQ as
`piggy hack boards. A.
`
`..--.-...:}Tl[ILLJ;..1.SZ-J£E.él:2...=é;.pasthvz..................... battrées in the pfigduc
`
`can provide tor about 8 hours of
`.
`opera ion and aka
`2 weeks of data retention. These are
`>exolu51ve,-1.e..8 hours of operation;followed hp 2 weeks of
`data retention is required. Hith tlis kind I
`‘
`COHSUHPtion heat should not he a Problew.
`o
`power
`
`Given our cost and power requireuents we expect
`
`that we
`
`_
`F 7
`___E ' [IA
`
`Fig . ”B
`
`.
`Fig a [/6
`
`

`

`US. Patent
`
`Nov. 20, 1990 .
`
`Sheet 9 of 15
`
`4,972,496
`
`I - Printed Circuit~Board
`
`_ The_hoard’s size, cost..Eower consuwption, and
`reliability are critical to t e sucess of our product.
`
`The board wust fit in our
`
`aoka e, which liwits its
`
`diwensions to sowething like 183 X 5%, or about 59 sTuare
`
`Roow wust be available for dealer installah e
`inches.
`options such as additional weworp
`and wodew - possxhxlp as
`P199! hack hoards. /\\
`
`.
`'
`I
`‘o'g'
`hattries in the product can provide for about 8 hours of
`operation and about 2 weeks of data-retention. These are
`exclu51ve,
`i.e._8 hours of operatxon_followed hp 2 weeks of
`data retention l5 required. With this kind of power
`consuwption heat should not he a problew.
`
`Given our cost and power requirewents we expect
`
`that we
`
`T’ig . IID
`
`
`
`I - Printed Circuit Board
`rower consuwption, and
`I; Ihe_hoard’s size, cost,
`reliability are critical to tie suoess of our product.
`
`upon tie availability of
`
`’lhe hoard wost fit in our package, which liwits its
`‘
`diwen51ons to sowethlng like 18‘ X 5 , or about 50 souare
`inches.
`.Roow wust he availahie for dealer installable
`options such as additional weworp
`and wodew - possahily as
`piggy hack hoard.
`How 15 the tiwe
`
`the
`‘Power consuwption wust he low enough so that
`hattries in the Product can provide for about 8 hours of
`operatgon and about 2 weeks of data retention.
`These are
`exclusxoe, p.e. 8 hours of operation_followed by 2 weeks of
`data retention IS required. with this kind of power
`consuwptxon heat should not he a prohlew.
`
`that we
`Cipen_our cost and power requirewents we expect
`‘
`will he using CHOS technology through out.
`He a so exgect
`to use Surface Haunted Iechnologp($ I).
`SH? allows a
`enser
`packing, wore shock reelstence, and lower wanuiacturing
`costs, although initial costs wag he Sll?ht19 higher.
`The
`final decision on SH! will depen
`
`I - Printed Circuit Board
`
`rower oonsuwption, and
`. The.hoard's size. cost,
`reliability-are critical.to tie suoess of our produch
`
`The board wust fit in our package, which liwits its
`_
`diwen51ons to sowethinq like 10' x 5 , or about 50 souare
`inches.
`Roow host he aua:lahle.for dealer installah e
`optlons such as additionai wewory
`and wodew - possibilu as
`p1ggp.haok hoard. —Now;as_ihe_t1ne
`the
`Power consuwption wust he low enough so that
`hattries in the product can provide for about 8 hours of
`operation and about 2 weeks of data retentlon. These are
`exclusxue,
`i.e._8 hours of operateon_£ollowed hp 2 weeks of
`data retention 15 required. With this kind of power
`consuwptlon heat should not he a problew.
`
`that we
`Given our cost and power requirewents we ex ect
`.
`H111 he usonq CHOS technology throu h out.
`He agso ex ect
`to use Surface Mounted Iechnology($ I).
`SHT allows a denser
`pmkmg,wwesMchrfiinmmm mdlmwrwmuhcmrmg
`costs, although 1n1t1a1_costs wa
`he sli htlp higher.
`The
`final dec231on on SHT will depen
`upon tfie arailahilitg of
`
`F 7 ’
`Ig ' I,
`
`F
`
`

`

`US. Patent
`
`Nov. 20, 1990 .
`
`Sheet 10 'of 15
`
`4,972,496
`
`l - Printed Circuit Beard
`
`. Theroard’s size, cost, FOHEF consuwption, and
`reliability are critical to tie suoess of our product.
`
`The board wust {it in our package, which liwits its
`.
`dihensrons to sowething like 19"X 5 , or about 50 suuare
`inches.
`Roow wust be available for dealer installab e
`options such as additional weworg
`and wodew - possibilg as
`piggy hack board.
`'
`
`the
`Bower consuwption Must be low enough so that
`.
`hattries 1n the product can provide for about 8 hours of
`operation aod about 2 weeks of data retention. These are
`excluSIpe,
`i.e._8 hours of operation followed by 2 weeks of
`data retention is required. Hlth this kind of power
`consuwption heat should not be a prohlew.
`
`.
`Given_our cost and power requirewents we ex ect
`that we
`will be usxng CHOS technology throu h out. We a so exgect
`to use Surface Hounted Technology(8 I).
`SHT allows a
`enser
`packing, wore shook reSistence, and lower Manufacturing
`costs, although initial costs wag he sli htlu higher,
`The
`final decrsron on ShI will depen
`upon tie availability of
`
`I - Printed Circuit Board
`
`
`
`_ The board’s size, cost, Power cons mption, and
`reliability are critical to tie suoess
`f our product.
`
`lo our package. which liwits its
`The board wust-fit
`_
`dlwenSIons to sowethinq like 18
`X 5 . or about 38 souare
`inches.
`Roow Must he;apailable for de ier installab e
`options such as additional weworg
`and wodew ~ possxbilg as
`piggy hack boardt
`
`Bower consuwption wust be low eno oh so that the
`hattrles 1n the product can prowide To
`about 8 hours of
`operation aod about 2 weeks of data re ention. These are
`exclusive,
`i.e. 8 hours of operation,£ llowed by 2 weeks of
`data retention IS required. Hlth this
`1nd of power
`oonsuwption heat should not he a proble .
`
`that we
`Given our cost and power requirewe ts we expect
`will be using CHOS technology through
`t.
`He a so exsect
`to use Surface Mounted Technologgts T)
`SHT allows a
`enser
`packing, wore shock resxstence, and lo er wanufacturing
`'wsn,auhwflimihakmskimabes[WWW mwmm 1m
`final decision on SHT will depen
`upon tie availability of
`
`I - Printed Circuit Board
`
`The board's size, oost, power
`consuwption, and reliability. are
`critical to the sucess of our product.
`
`_
`The board wust fit in our
`package
`thch.11HlfS its leEDSIOnS
`o sowethiog like 18“ X 5", or about
`50 s uare inches.
`Roow wust he
`aua; able for dealer installable
`and
`options such_as additional newer
`wodew - possahilg as P1999 back board.
`Power consuwption wust be low
`enough so that
`the
`battrles in the product can provide
`for about 8 hours of operation and
`about 2 weeks of_data_retention.
`These are exclusive
`i.e. 8 hours of
`operation followed b3 2 weeks of data
`retention is require . With this kind
`of power consuwption heat should not
`be a problew.
`
`Fig . [/6
`
`Fig . IIH
`
`Fig . III
`
`

`

`US. Patent
`
`Nov.20, 1990
`
`Sheet 11 of 15
`
`4,972,496
`
`Heme: ....................
`ID§
`PQTIEHT INFORHQTIOH
`Doctor: ............ .......
`
`Bed:
`
`Height:
`
`8.8 Height:
`
`8.8 Sex: Hale feHaIe
`
`B. 8 Sex: Hale fexale
`
`IDfi
`PATIENT INFORMATION
`
`OH:
`
`_
`
`Bed:
`
`Height:
`
`8. B Heig}1t:
`
`

`

`US. Patent
`
`Nov. 20,1990
`
`Sheet 12 of 15
`
`4,972,496
`
`N
`
`{pg ....
`g ..............r.....
`mum INFORHQTIOH
`we
`Doctor; ....................
`
`Height'
`
`8. 8 Height:
`
`8. 8 Sex: male fenale
`
`9 8 Sex: Male fenale
`
`Rage:
`
`...«..1.............
`
`IDfi ..... ROOM:
`
`ID§ ..... R00“:
`....................
`PATIENT INFORHRTIOH
`
`Doctor:
`
`Bed:
`
`Height:
`
`8. 8 Height:
`
`

`

`US. Patent
`
`Nov. 20, 1990
`
`Sheet 13 of 15
`
`4,972,496
`
`ID§--
`
`Hana:
`
`Doctor:
`
`-
`
`IDfl ----- Boon:
`PATIENT INFORHRTIOH
`..... ...............
`
`Bed;
`
`8.8 Sex: Male female
`
`Height:
`
`8:9 Height:
`
`8. 0
`
`Sex: MalE female
`
`Gamma:_-..--_--....;3
`
`B81l2
`
`IDfl ..... Rog“:
`Hang: ....................
`PATIENT INFORHRTIOH
`Doctop: ................
`
`Bed:
`
`Height:
`
`9.6 Height:
`
`

`

`US. Patent
`
`Nov. 20, 1990
`
`Sheet 14 of 15
`
`4,972,496
`
`IDg""'RDOH:
`Hana: ....................
`PQTIENT INFORHHTIOH
`Doctor; ................
`
`Bed:
`
`Height: 88 Height:
`
`98 Sex: male fenale
`
`

`

`US. Patent
`
`Nov. 20, 1990
`
`Sheet 15 of 15
`
`4,972,496
`
`229
`
`ACQUIRE POINTS &
`MIMIC ON SCREEN
`
`A CONVERT T0
`STROKE
`
`CHARACTERIZE
`STROKE
`
`LOOK UP/ RECOGNIZE
`
`EDITOR SOFTWARE TO
`INTERPRET &
`EXECUTE COMMAND
`
`CHARACTER SEND CHARACTER TO
`
`DETERMINE WHICH
`COMMAND & EXECUTE
`
`237
`
`DISPLAY
`CHARACTER
`
`

`

`mix
`
`1
`
`4,972,496
`
`HANDme KEYBOARDLESS ENTRY
`COMPUTER SYSTEM
`
`REFERENCE TO DISCLOSURE DOCUMENT
`
`Reference is hereby made to Disclosure Document
`No. 144,644, filed Jan. 14, 1986.
`REFERENCE TO RELATED CASE
`
`This is a continuation-in-part of US. patent applica-
`tion Ser. No. 06/889,513, filed July 25, 1986, now aban-
`cloned.
`
`FIELD OF THE INVENTION
`
`The present invention relates generally to a key-
`boardless input system to a computer, and when com-
`bined with a central processing unit, to a keyboardless
`entry computer system. More particularly, the present
`invention relates to an information storage, manipula-
`tion and transfer device on which text, data, computer
`commands and functions are entered by writing alpha-
`numeric or any other characters and symbols by hand
`with a penlike stylus on an Input/output (I/O) screen.
`In a preferred embodiment the I/O screen includes a
`transparent touch screen incorporated over a substan-
`tially flat output display. The present invention in its
`preferred embodiment
`is a self-contained computer
`system but can also function as a peripheral to a host
`computer.
`DESCRIPTION OF THE PRIOR ART
`
`Large amounts of information and sophisticated ap-
`plications software are now available on conventional
`keyboard computers. The utility of this information and
`p of application software could be greatly increased if text
`and data could be entered and applications software
`manipulated by writing in a normal fashion directly on
`a flat display. Thus, there is a need to allow the utility of
`computer technology to be extended for use by non-
`keyboard oriented individuals. There is also a need for
`a portable computer system that is lightweight, reliable,
`accurate, inexpensive and permits use while standing or
`walking. One way to reduce expense and size and in-
`crease utility is to employ a keyboardless entry system,
`such as a touch screen. However,
`this type of input
`device does not easily allow accurate detailed input
`within a real time framework with high resolution in a
`manner which is familiar and natural to the user.
`Many positioning technologies can be used to meet
`the requirements of the position sensing input technol-
`ogy. Essentially these requirements include accuracy,
`resolution and speed. The technologies include: me-
`chanical, electrostatic, electromagnetic, acoustic, opti—
`cal, and inertial. The desire in this system is to have its
`use as similar as possible to writing with pen or pencil
`on paper. One problem is proximity—a pen on paper
`only leaves a trail when actually in contact. Many of
`these technologies require an additional “pen down”
`sensor which is awkward to use in many commercial
`pens. Another problem is writing angle—a pen leaves
`the same trail independent of writing angle. Many of
`these technologies have the position detector displaced
`from the pen tip, so pen angle causes erroneous dis-
`placements. Beyond these general problems, each tech-
`nology has numerous advantages and disadvantages in
`(1) the pen: size, weight, shape and whether it needs to
`be powered and/or wired, and (2) the writing surface:
`transparency, smoothness, “feel”, and whether or not
`
`5
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`2
`physical contact is needed (as opposed to pressure trans-
`mitted through overlaying sheets of paper).
`A number of self-contained devices for viewing and
`processing large amounts of information are known.
`Most employ optical, magnetic or solid-state electronic
`storage means to store data. Illustrative of this body of
`art is US. Pat. No. 4,159,417 to Rubincam which dis-
`closes a portable electronic book configured to provide
`selective page by page call-up of large amounts of digi-
`tal data and displays it on a flat, solid-state screen. The
`preferred embodiment in the Rubincam patent uses an
`insertable holographic card, which may contain several
`hundred pages of text in digital form, as the main stor-
`age means. Rubincam’s device, however, does not
`allow new information or text to be entered or manipu-
`lated.
`In US. Pat. No. 4,016,542 to Azure an electronic data
`collection system is disclosed which employs a solid
`state Random Access Memory (RAM) for its primary
`memory. This patent, which discloses a conventional
`keyboard for data entry, is directed to a hand-held por-
`table data storage and transmission system, as well as an
`LED display and various Input/Output (I/O) connec-
`tors.
`»
`US. Pat. No. 3,487,731 to Frank discloses a means of
`converting handwriting into character data through the
`use of a computer system. The disclosed invention is
`based on matrix pattern matching and does not employ
`any coincident display technology.
`US. Pat. No. No. 4,491,960 to Brown shows a hand-
`written symbol recognition system in which an array of
`image Points, in the form of a raster line sampling, is
`converted into segment-oriented lists which are filtered
`and compressed to obtain topologic features which are
`then analyzed with a logic tree decision mechanism.
`US. Pat. No. No. 4,262,281 to Buckle et al. discloses
`a handwriting recognition device. The disclosed em-
`bodiment is for use with a host computer and does not
`employ coincident display technology.
`US. Pat. No. No. 4,475,239 to Van Raamsdonk dis-
`closes a text editing apparatus. The ’239 patent calls for
`the use of paper as a medium for the entering of editing
`functions and requires a conventional keyboard for the
`inputting of text.
`US. Pat. No. 4,521,909 to Wang shows a dual level
`pattern recognition system. The system is designed for
`use with an optical instrument.
`‘
`US. Pat. No. 4,520,357 to Castleberry et al. discloses
`an electroscopic information display and entry system
`with writing stylus. The disclosed embodiment does not
`claim to have the speed or accuracy to enable handwrit-
`ten character recognition.
`Additional prior art which discloses portable elec-
`tronic devices that provide large amounts of various
`types of stored information include US. Pat. No. Nos.
`4,218,760 to Levy; 4,4115,486 to Laine; and 3,932,859 to
`Kriakides et al. The Levy and Kriakides et a1. patents
`pertain to electronic dictionaries, while the Laine patent
`discloses a programmable television reminder system.
`None of these devices disclose the use of a handwritten
`input.
`In US. Pat. Nos. 4,071,691, 4,129,747, 4,198,539,
`4,293,734, 4,302,011, 4,353,552, 4,371,746 and 4,430,917
`to William Pepper, Jr. various methods or machine-
`human interfaces using finger touch are disclosed. The
`preferred embodiments in each of these inventions lack
`sufficient speed and resolution to allow handwritten
`
`

`

`3
`character recognition with a stylus and are designed for
`other purposes. US. Pat. No. 4,318,096 to Pepper
`teaches the use of a conductive stylus. The ’096 patent
`pertains to graphic design and allows line width and line
`intensity to vary by applying pressure on the stylus with
`the results displayed on a conventional CRT screen.
`US. Pat. No. 3,699,439 to Turner and US. Pat. No.
`4,055,726 to Turner et al. disclose two methods for
`electronic position sensing through the use of a probe.
`SUMMARY OF THE INVENTION
`
`The invention comprises a unique keyboardless com-
`puter system which has the ability to recognize and
`display Handwritten Symbols and cause the computer
`to display Font Symbols and, if desired,
`to execute
`editing functions pursuant to Editing Symbols, quickly,
`easily and at reasonable cost.
`The invention constitutes a computer housing with a
`flat display panel on which a user may “write” with a
`stylus, a capability to recognize Handwritten Symbols
`written on the panel with the Stylus and convert them
`to displayed Font Symbols and/or to execute Editing
`Functions with Editing Symbols, all with a minimum of
`technical complexity for the user.
`A further feature of this invention is that once the
`keyboardless, portable computer is loaded with the
`desired information and applications software, informa-
`tion and software can be used and responded to without
`requiring skills or knowledge related to state-of-the-art
`computers or other data source.
`The ease-of-use of the input technology of the present
`invention enhances the utility of the computer for key-
`board oriented individuals. The portability of the pres-
`ent invention also allows it to be used in applications
`and settings in which portable keyboard computers are
`awkward, difficult or impossible to use. For example, a
`multiplicity of blank, fully or partly completed forms
`may be stored in the portable computer memory. In a
`hospital, “sheets” of patient data can be stored in the
`memory of the portable computer, called up by a nurse
`as the nurse makes rounds and relevant data, such as
`blood pressure, temperature, etc., can then be entered
`manually with a stylus. These corrected or expanded
`forms can then be down-loaded into a central computer
`memory.
`The requirements of the position sensing input tech-
`nology are accuracy (Point to Point), resolution (abso-
`lute position) and speed (Points per unit time) to ade-
`quately definethe written Stroke for recognition analy-
`sis. For the recognition apparatus and methods pres-
`ently used, as described below, the present minimum
`requirements are: accuracy of 0.005 inch, resolution of
`0.015 inch, and speed of 150 Points per second. This
`accuracy allows a 41” high writing line with over 10 raw
`input Points along a Stroke of a small letter. The resolu-
`tion provides positioning of the symbol to within two
`pixels on a present display of 640 pixels to 9 inches. The
`speed permits about 50 raw input Points for a rapidly
`written single letter (5 second).
`One embodiment of the present invention comprises a
`transparent input screen. As the user writes alphanu-
`meric or other characters or symbols on the input
`screen, the character is represented as a stream of Points
`emulating written input with pen on paper. Once the
`discrete alphanumeric and other characters or symbols
`are complete, they are translated into computer text or
`computer commands that can be displayed on a display
`screen at a location that is preferably beneath the area
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`45
`
`50
`
`55
`
`65
`
`4,972,496
`
`4
`on the input screen where they were entered. The em—
`bodiment also comprises a pattern recognition algo-
`rithm which allows the translation of any written char-
`acter or symbol, such as ideographs and scientific sym-
`bols, into computer text.
`In a particular, presently preferred embodiment, a
`keyboardless computer according to the present inven—
`tion is configured as a manipulation and display device
`comprising a transparent touch screen and associated
`electronics placed over an 80 column by 25 line or
`larger display screen; a stylus for entry of data; a micro-
`processor and storage means; artificial intelligence/pat-
`tern recognition software and editing software; and a
`battery power system; and other I/O means.
`As used herein, “Handwritten Symbols” are any sym—
`bols capable of being handwritten and having communi-
`cative effect. By way of example, and not limitation,
`numbers, letters, “Kanji” (Japanese ideograms) or other
`language symbols, editing symbols and engineering,
`scientific, architectural and mathematical symbols are
`’ Handwritten Symbols. Other examples of Handwritten
`Symbols are free-hand drawings or signatures or any
`other such written information uniquely configured by
`a particular writer. Handwritten Symbols may also
`include Editing Symbols (defined below).
`As used herein, “Font Symbols” are computer-
`generated symbols which are displayed in a predeter—
`mined font format. By way of example and not limita-
`tion, alphanumeric symbols may be Font Symbols and
`displayed in numerous font formats. Japanese or Chin—
`ese “ideograms” may also be Font Symbols, as may be
`engineering, scientific, mathematical, architectural or
`other such characters. Other examples of Font Symbols
`include any form which can be stored and displayed by
`a computer, e.g., a drawing of a car or a house.
`As used herein, an “Editing Symbol” is a symbol
`' (such as a caret, horizontal line, short vertical line, long
`vertical line, etc.) which is intended, when recognized,
`to cause the computer to execute a particular Editing
`Function (defined below), such as insert text (caret),
`delete text (horizontal line), delete a letter (short verti-
`cal line) or move a margin (long vertical line), to list a
`few representative examples.
`“Editing Function” means any computer-generated
`text editing operation, such as by way of example and
`not limitation, insert text, delete text, move text and
`substitute text. Some primary Editing Functions are
`listed on paged 40 and 41 below.
`OBJECTS OF THE INVENTION
`
`It is therefore a primary object of the present inven—
`tion to provide improved methods and apparatus for
`providing a keyboardless computer on which usual
`computer functions are performed by writing in a nor-
`mal manner with a pen-like stylus on an input screen
`placed directly over a flat display.
`The keyboardless computer provided is ideally con—
`figured for use by non-keyboard oriented individuals,
`by keyboard individuals for Whom the utility of the
`computer is enhanced, and in various settings and appli—
`cations in which keyboard entry is awkward or impossi-
`ble.
`It is also an object of this invention to provide a
`means whereby computer-based information and appli-
`cations software can be loaded into a portable device
`for later viewing, manipulation of text and data, and
`adding new text and data in a normal handwriting
`mode. Thereafter the user may transmit this computer
`
`

`

`4,972,496
`
`6
`corresponding plurality of display signals to a display
`screen 20. Both input screen 18 and display screen 20
`are described in greater detail hereinbelow.
`Referring now to FIG. 2, there is shown a perspec-
`tive view of keyboardless computer system 10 embody-
`ing the features of the present invention. Keyboardless
`computer system 10 is contained in housing 12, which is
`a rectangular enclosed casing having a sloped top sur-
`face 22 with a multi-line, solid state display area 24.
`Input screen 18 is depicted in FIG. 2A as being posi-
`tioned over display screen 20. In this example, display
`screen 20 displays a plurality of horizontal lines 25 with
`the following indicia:
`
`Name
`Address
`
`20
`
`Handwritten entries are made above each line 25. The
`
`5
`text to another computer, a similar device, an external
`electronic storage device, a hard copy printer, or
`through a telecommunications system. Yet another ob-
`ject of thus invention is to provide a computer capable
`of recognizing Handwritten Symbols with a high de—
`gree of accuracy and of “learning” individual styles of
`handwriting.
`A further object of this invention is to provide a
`portable keyboardless computer in which data and com-
`mands are input with the use of a stylus.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`FIG. 1 is a schematic system block diagram of the
`present invention;
`FIG. 2 is a perspective View of the housing contain-
`ing the operating elements of the invention;
`FIG. 2A is an enlarged portion of FIG. 2, with parts
`removed to Show the positional relationship between
`the touch input screen and the display screen.
`FIG. 3ris a schematic diagrammatic view of the input
`screen, stylus and associated electronics;
`V
`FIG. 4 is an overall schematic system block diagram
`of the apparatus of a keyboardless entry computer sys-
`tem according to the present invention;
`FIG. 5 is a schematic block diagram depicting the
`movement of data within the system when modified by
`handwritten characters and commands;
`FIG. 6 is an overall system block diagram depicting
`the hierarchy of software used to operate the system;
`FIG. 7 is a generalized block diagram of the charac-
`ter and pattern recognition algorithm.
`FIGS. 8A and 8B together are a detailed block dia-
`gram of the pattern recognition algorithm.
`FIG. 9 is a schematic block diagram of the Stroke
`characterization subroutine.
`
`FIG. 10 is a top plan view of a screen illustrating the
`“initializing” of the database for Handwritten Symbols.
`FIGS. 11A through 111 are a series of top plan views
`of screens depicting the operation of a text editing sys-
`tern.
`
`FIGS. 12A through 12G are a series of top plan views
`of screens depicting the operation of a data entry sys-
`tem.
`'
`
`10
`
`15
`
`25
`
`3O
`
`35
`
`FIG. 13 is a generalized block diagram of the Linus
`Editor.
`
`45
`
`DETAILED DESCRIPTION OF THE
`PREFERRED EMBODIMENTS
`
`With reference now to the figures, wherein like nu-
`merals indicate like elements throughout the several
`views, and in particular with reference to FIG. 1, an
`overall block diagram of a portable handwritten, key-
`boardless entry computer system 10 is depicted. The
`complete computer system is encased in a housing 12.
`indicated graphically by the dashed line, and includes a
`conventional, general purpose digital microcomputer
`14, described in greater detail hereinbelow. Input infor-
`mation is provided to microcomputer 14 by stylus 16
`“writing” on writing or input screen 18. Stylus 16 (FIG.
`2) is connected to the computer of system 10 with wire
`17 (FIG. 2). As stylus 16 “writes” on input screen 18, a
`plurality of locating signals representative of a plurality
`of corresponding positional coordinates are transmitted
`to microcomputer 14. Microcomputer 14 has been pro-
`grammed in accordance with a computer program de-
`scribed hereinbelow, to recognize the stream of locating
`signals and to store these signals in a computer memory.
`The programmed microcomputer 14 also provides