United States Patent
`
`[191
`
`[1 1]
`
`4,202,041
`
`Kaplow et al.
`[45]
`May 6, 1980
`
`Flexibility in Computer Interaction", Control Engi-
`neering, Jul. I976, pp. 33-34.
`
`Primary Examir1er—Gareth D. Shaw
`Assistant Exam1‘ner—Thomas M. Heckler
`Attorney, Agent. or Firm—Arthur A. Smith, Jr.; Robert
`F. O’Connell
`
`[57]
`
`ABSTRACT
`
`A dynamically variable keyboard terminal system hav-
`ing a keyboard display unit using an input panel with a
`plurality of touch-sensitive locations thereon and a key-
`board display unit in alignment therewith. The system
`includes means for generating and storing information
`concerning symbols which make up one or more work-
`ing symbol sets, such symbol information being author
`definable in accordance with a predetermined terminal
`definition language. The symbol
`information can in-
`clude a symbol configuration for display, if applicable,
`or a symbol action to be performed, if applicable. Means
`are also provided for generating and storing informa-
`tion concerning a plurality of keyboard images which
`use such symbols and which keyboard information can
`be used for displaying a keyboard image at the keyboard
`display unit. A user can selectively touch one or more
`such locations so as to obtain symbol information for
`display on the keyboard display unit or on a monitor
`display unit, if applicable, or for performing an action
`associated with such symbol, if applicable.
`
`[54] DYNAMICALLY VARIABLE KEYBOARD
`TERMINAL
`
`[75]
`
`Inventors: Roy Kaplow, Newton; Michael K.
`Molnar, Cambridge, both of Mass.
`
`[73] Assignee: Massachusetts Institute of
`Technology, Cambridge, Mass.
`
`[21] Appl. No.: 814,723
`
`[22] Filed:
`
`Jul. 11, 1977
`
`G06F 3/14; G061? 13/06
`[51] 1m.c1.2
`.................................................... 364/900
`[52] us. c1.
`[58] Field of Search
`364/200 MS File, 900 MS File;
`l97/98; 340/365 R. 324 A
`
`[56]
`
`References Cited
`U.S. PATENT DOCUMENTS
`
`.................. .. 364/900
`Shashoua et al.
`6/ I967
`
`..... .. . 364/2(1)
`10/ 1970 Han et al.
`12/I971 Hamada etal
`.
`340/324A
`..
`5/1973 Bouchard et al.
`340/324 A
`9/1973
`Barkan et al.
`. . ... ..
`.. ... 340/365 C
`9/1973
`Reynolds et al.
`340/365 R X
`10/1973
`Ichida .................. ......... 364/9(X)
`ll/1973 Hacon
`340/365 R X
`l2/1973 Harris ......
`. 340/365 S X
`5/1914
`Saito et al.
`364/900
`..
`1/1915 Watson et al.
`......... 364/200
`6/I977
`Saich ...............
`340/324 A x
`l/1978
`Sukonick et al.
`.................... 364/9(1)
`
`
`
`3,325,786
`3,534,396
`3,631,457
`3,731,299
`3.757.322
`3,760,360
`3,766,528
`3,773,989
`3,777,222
`3,311,113
`3,859,635
`4,020,595
`4,070,7l0
`
`OTHER PUBLICATIONS
`
`Crook, Ken, “CRT Touch Panels Provide Maximum
`
`12Claims,l8DrawingFigures
`
`TWO 256K ~ Bl T
`
`soun-s-me
`
`PR2fi§5s°R
`uzuomzs
`INTERNAL
`vuoao SIGNAL
`MEMORY
`GENERATOR
`
`
`
`
`
`I 3
`
`KEYBOARD OVERLAY
`
`INTERRUPT
`
`KEYBOARD LOGIC
`CIRCUITRY AND
`INTERFACE
`
`
`    
`


`
`Samsung USP 7,793,773
`Exhibit 1015 Page 1
`
`

`
`
`
`U.S. Patent May, 6, 1980
`
`Sheet 1 of 16
`
`4,202,041
`
`2|
`
`
`
`
`-
`
`
`
`--'
`
`
`Pnocgsson
` SOLID-STATE
`AND
`
`
`.4
`INTERNAL
`MEMORY
`vmeo snemm.
`
`GENERATOR
`
`RE FRESH -MEMO RY
`CONTROLLER
`
`- I
`
`TWO 256K -BIT
`
`MEMORIES
`
`
`
`CLICK, BUZZ,ETC
`
`KEYBOARD OVERLAY
`
`F|G.|
`
` !"#$#%"&'()"*
`
`Exhibit 1015 Page 2
`
`INTERRUPT
`
`KEYBOARD LOGIC
`CIRCUITRY A ND
`INTERFACE
`
`

`
`U.S. Patent May, 6, 1980
`
`Sheet 2 of 16
`
`4,202,041
`
`»z_on_-mmomu
`
`wm.:.E
`
`z<umz<um
`
`._<zo_m44_m
`
`
`
`
`
`
`
`<~.o:s_oEA<N.o_om“:
`
`mmmmmmmmmmfi
`
`»z_on_-mmomu
`
`éasgg2IIIIIIIIII2;gs.
`
`mmooumo
`
`ZSDJOU
`
`muwznoo
`
`50.8
`
`z<um
`
`Au...uno.5“:
`
`¢~
`
`
`
`mmoo_EV5040248
`
`
`
`
`
`ZO_._.<U_u_._.ZuD_mmooumcmm._.ZDOU
`
`I...-uunnnn
`
`.<~.o:_ob
`
`_
`
`jm.o_n.
`
`+,-./.0123241567819
`
`Exhibit 1015 Page 3
`
`
`
`
`
`
`
`
`

`
`U.S. Patent May, 6, 1980
`
`Sheet 3 of 16
`
`4,202,041
`
`Enmmukz.E:mmm»z_Emma
`
`
`
`
`
`
`
`
`
`
`
`
`
`o»._<zo_m._<zo_mvm._<zu_m..|2.238:som
`
`.
`
`ll.I.|'Il||IIII-
`
`._.Z_On_Immomu|33EmmaD24—mDO_>mmn_:m.o:<Emu_mz<m»_
`
`
`:5mmfizaou_ommwop
`IIII..II.
`
`»z_om-mmomuwzuzmnu
`
`3:mukzaou
`
`z<omamouum5:;S5203:w:2mo"._._<zo_m_
`.||.I|l.|.|||.|l.mm
`
`omoumm
`
`..._<zo_mSu:o»u._m<zm
`..<zo_mmA4_omouum
`
`mmoozfi
`
`::um_u
`
`
`
`mabfimOKOUMI
`
`
`
`._.z.On_zmmomu
`
`mm:_D&
`
`AN.o_.._20?:
`
`
`
`
`
`JV.moo:omozu
`
`omoxo
`
`
`
`zowoos..
`
`omozo
`
`Ed;otzo_h.:n__2ou_mo:2oE
`._<Zo_mon_m»z_on_-mmomu
`Szutzm|.I|l.I.l_“.592
`:m_zEzqum>7m<nmmmm.uo%__9..5...
`
`
`amoumm...m»z_on_-mmomu
`mommuuomm_mommmuomm
`
`
`
`mommmooma.23.23__»z_on.-mmomo
`
`
`
`
`
`zumqmm.zmo..<zo_mI___.mn.:.m_omm
`
`zqumzumfimzo:.<oEFzmo_
`
`
`
`
`
`towoo:.zmo..<zo_mon<o»._<zo_m
`
`zmz;Sagomm
`
`
`
`oi<m_®E
`
`oAmmommmuoma
`
`
`
`
`
`DIOIUZUIZ,._.Dn_._.DOOIOIUn_ODZM
`
`:;<=>=?@ABAC@DEFG@H
`
`Exhi it 1015 Page 4
`
`
`
`
`
`
`
`
`

`
`
`
`US. Patent May, 6, 1980
`
`Sheet 4 of 16
`
`4,202,041
`
`
`
`INITIATE COUNTERS ( PREVIOUS COUNT¢ CURRENT COUNT
`BEGN "SEARCH"SCAN
`CURRENT COUNT<=0);
`
`OFDYNAMW KEYBOARD
`
`
`
`CROSSPOINT
`
` N0
`CLOSED ?
`
`YES
`
`ADD ITO CROSSPOINT
`COUNTER
`
`
` CROSSPOINT
`
`
`COU NT
`>PREV| OUS
`
`COUNT
`
`ygs
`
`6
`
`(sTART'hEcoRo"
`SCAN)
`
`SEARCHHSCAN
`
` \\
`NO
`
`ADVANCE SCAN TO
`NEXT CROSSPOINT
`
`SEARCH
`SCAN
`
`
`
`F|G.3
`. COMPLETE?
`
`
`
`NO
`
`YES
`
`IJKLMLNOPQPROSTUVOR
`
`Exhibit 1015 Page 5
`
`

`
`
`
`U.S. Patent May, 6, 1980
`
`Sheet 5 of 16
`
`4,202,041
`
` START “RECORD”
`
`
`
`
`SCAN OF KEYBOARD
`FR ESH CROSSPOINT
`BUFFER LIST
`
`
`
`I"s
`CROSSPO INT
`CLOSED
`
`
`
`ADD CROSSPOINT
`IDENTIFICATION TO
`CURRENT BUFFER LIST
`IN PROCESSOR MEMORY
`
`ADVANCE SCAN
`TO NEXT CROSSPOINT
`
`
`
`"RECORD"SCAN
`
`
`
`
`
`HAVE ALL
`CROSSPOINTS
`BEEN scum ED ?
`
`
`
`
`
`
`aeNAL"REcoRd'scAN
`FINISHED; PROCESSOR SAVES
`CURRENT CROSSPOINT LIST
`AND REPLACES WITH SYMBOL
`IF COMPLETE
`
`
`
`
`FIG.3A
`
`
`
`
`IsTART"sEARcH"
`SCAN)
`
`WXYZ[Z\]^_^`]abcd]e
`
`Exhibit 1015 Page 6
`
`

`
`
`
`U.S. Patent May, 6, 1980
`
`Sheet 6 of 16
`
`4,202,041
`
`
`
`cnoss POINT
`COUNT= 0?
`
`
`
`PREVIOUS
`COUNT> 0 _?
`
`YES
`
`
`
`SIGNAL
`END OF CHORD
`
`
`
`
`
`FlG.3B
`
`fghijiklmnmolpqrslt
`
`Exhibit 1015 Page 7
`
`

`
`ft
`
`m6,
`
`7aC..n..S
`
`4,202,041
`
`.mo<2_Uomqomfix
`
`
`Vsa\\W.n.._O<Q
`
`azo_EEommEMzo_E_mummooz<om»mx
`
`4om2>mm./Cmmoouwe
`
`3Jomzh...
`
`
`
`“$42.S.ozcéozEmom<omEg»mo5mE_oD...Em4om_2>m4J/
`
`
`
`A\.mm
`
`0omm93>tEmo_
`
`
`8\omqomfizom<om>mv_
`
`fmmqm<250n._h._.Es_ou
`
`6|_<UOI_
`
`vol
`
`mmz_.Som..mam
`
`.Cum._om2>m
`
`:53um:mou:
`
`uvwxyxz{|}|~{€‚{ƒ
`
`Exhibit 1015 Page 8
`
`
`
`
`

`
`U.S. Patent May, 6, 1980
`
`Sheet 8 Of16
`
`4,202,041
`
`CR°55"°°'“T'5
`LIST FROM
`KEYBOARD LOGIC
`
`"SYMBOL SEGMENT
`COMPLETE" SIGNAL
`FROM KEYBOARD
`L05“;
`
`KEYBOARD
`““°'°
`DEVICE
`
`ICURRENT I
`L SHIFT
`|
`
`CURRENT cROss=POmT
`LIST BUFFER
`
`CHORD MODE
`
`-——
`
`PREVIOUS CRO5S‘F'O|NT
`LIST BUFFER
`
`SEARCH CURRENT KEYBOARD
`DISCRIPTION FOR IDENTIFICATION
`OF KEY CORRESPONDING TO
`
`GENERATE
`"CLIcK"ON
`KEYBOARD
`
`NEW cRQ55-po|N-r
`
`.
`
`Auolo
`
`YES
`
`LEGITIMATE KEY
`
`SHIFT
`
`NOTATION
`CROSS-POINT
`SYMBOL
`TABLE
`CODE
`
`I I I I I |
`
`flj
`-—n%
`--||I-
`EIlI—-1
`=-:-1
`jI
`jI1
`
`CURRENT KEYBOARD I M AG E
`DESCRIPTION
`
`WORKING
`SYMBOL SET
`
`SYMBOL
`CO E
`
`SYMBOL
`DESCRIPTION
`
`""I
`I
`
`6|
`\ak
`
`I I I IMI L
`
`SYMBOL CODE x./I
`CORRESPONDING T0
`|
`CLOSED KEYICROSS-POINT)
`I
`
`FIG.5
`
`„…†‡ˆ‡‰Š‹Œ‹ŠŽ‘Š’
`
`Exhibit 1015 Page 9
`
`

`
`
`
`U.S. Patent
`
`May, 6, 1980
`
`Sheet 9 of 16
`
`4,202,041
`
`TO KEYB OARD DISPLAY
`
`VIDEO REFRESH
` INPUT SYMBOL BUFFER: LIST OF SYMBOL CODE
`M EMORY FOR
`
`{SAVE FOR EDITING OF INPUT LINE
`KEYBOARD
`AND SUBSEQUENT TRANSMISSION)
`
`
`
`
`_ ::J“I:
`__r_fl
`
`W ORKING SYM BOL SET
`
` EX E MPLARY
`SYMBOL DESCRIPTION
`
`
`
`SYNBOL
`CODE
`
`VISUAL DESCRIPTION
`
`
`
`
`
`
`OPTIONAL SYMBOL
`»;
`CODE SEQUENCE
`DELAYED AcTIoN,No
`— TERMINATE LINE fr‘ No
`TRANSMIT ? YES
`IMMEDIATE SEND _? NO
`
`
`NON -DELAYED AcTuoNs’
` ll
`
`
`
`F|G.6
`
`“”•–—–˜™š›šœ™žŸ ™š›
`
`Exhibit 1015 Page 10
`
`
`
`
`
`ADD SYMBOL
`CODE TO INPUT
`
`
`
`8 UF-‘FER’, THIS
`LINEIS INPUT
`STREAM
`
`DRA W SYM BOL IN
`"INPUT LINE"
`SPACE ON
`KEYBOA RD AND
`EXECUTE OTHER
`FRE-TRANSMIT
`SYMBOL ACTIONS
`
`

`
`
`
`U.S. Patent May, 6, 1980
`
`Sheet 10 of 16
`
`4,202,041
`
`
`DATA LIN E TO
`TO MONITOR
`
`DISPLAY
`COMPUTER OR
`
`SECONDARY STORAGE
`
`
`
`
`
`
`VI D EO REFRESH
`MEMORY FOR
`MONITOR
`
`INPUT BUFFER
`
`
`
`ACTIONS FOR ALL
`IFSYMBOLIS
`
`
`OF PROCESS
`SYMBOLS IN
`EXACTLY AS
`"TEFIMI N ATE”
`
`OPTIONALLY DISPLAY
`
`INPUT BUFFER
`TYPE
`IN FIGS. 5 8 6
`
`
`ALL SYMBOLS IN
`
`INPUT LINE ON
`MONITOR DISPLAY UNIT
`
`
`I
`
`
`
`
`
`I I I I
`
`DESCRIPTION or SYMBOLS
`SYMBOL LmT
`
`REPRESENTING
`ALL SYMBOLS
`ININPUT
`BUFFER
`
`I
`
`VISUAL DESCRIPTION
`
`¡¢£¤¥¤¦§¨©¨ª§«¬­®§¨¨
`
`Exhibit 1015 Page 11
`
`

`
`
`
`U.S. Patent May, 6, 1980
`
`Sheet 11 of 16
`
`4,202,041
`
`T KEYBOARD
`
`KEYBOARD
`DESCRIPTION
`
`DATA PAGES
`I DISK
`
`
`
`DIRECTORY
`INFORMATION
`
`
`
`
`LOCATE KEYBOARD
`PAGE IN KEYBOARD
`DIRECTORY INFORMA-
`TION STORE
`
`KEY BOARD
`
`IDENTITY
`CODE
`
`
`
`
`
`LOCATE KEYBOARD
`DESCRIPTION DATA
`PAGE IN DATA
`PAGE STO R E
`
`
`
`
`
`LABEL
`ouTLINE#
`MARKER #
`
`
`
`
`
`
`
`CROSS POINT IDENTITY a sHII=T#
`LEVEL
`
`
`END
`
`
`
`READ KEYBOARD
`DESCRIPTION DATA
`INTO PROCESSOR
`MEMORY
`
`cRoss
`POINTS
`E”|;$Y
`
`MEM_,
`
`6)
`
`DISPLAY
`KEYBOARD
`
`DISPLAY GLOBAL 8.
`STEP THROUGH
`CROSS-POINT ENTRY
`LIST 8 DISPLAY
`
`MARKERS, ouTLINEs,
`LABELS ON KEY-
`BOARD DISPLAY
`AT COMPUTE D
`KEY LOCATION
`
`
`
`Loox up
`__ su3Rou‘rIm-:5
`I
`FOR EACH
`
`
`
`
`MARKER
`MATRIX LABEL
`STRING
`
`
`
`
`
`\
`
`\
`
`\
`
`\
`
`
`
`OUTLIN E
`VECTOR
`SET
`
`
`
`¯°±²³²´µ¶·¶¸µ¹º»¼µ¶½
`
`Exhibit 1015 Page 12
`
`KEYBOARD
`DISPLAY
`UNIT
`
`

`
`
`
`U.S. Patent May, 6, 1980
`
`Sheet 12 of 16
`
`4,202,041
`
`
`
`"o1sRLAY"
`KEYBOARD
`
`
`
`EXAMINE NEXT
`CROSS POINT ENTRY
`
`YES
`
`PERFORM
`
`"INTERPRET
`
`DESCRIPTION"
`
`
`
`as
`MARKER
`SPECIFIED ?
`
`
`
`is
`OUTLINE
`
`SPECIFIED ?
`
`
`
`IS
`LABEL
`
`
`
`
`
`NO
`
`
`
`FIG.9
`
`¾¿ÀÁÂÁÃÄÅÆÅÇÄÈÉÊËÄÅÌ
`
`Exhibit 1015 Page 13
`
`
`
` MORE
`CROSSPOINT ?
`
`SPECIFIED ,?
`
`

`
`U.S. Patent May, 6, 1930
`
`Sheet 13 of 16
`
`4,202,041
`
`
` P ER FOR M
`INTERPRET
`DESCRIPTION
`
`
`
`
`
`
`ENTER NEXT
`GRAPHICAL
`OPERATOR FROM
`DESCRIPTION AND
`CLASSIFY AS TO
`TYPE
`
`DISPLAY
`EXECUTE
`DRAW
`DRAW
`
`
`
`VECTOR
`SUBROUTINE
`MATRIX
`STRING
`
`OF SYMBOLS
`
`
`
`
`
`MORE
`
`OPERATORS
`
`
`
`F|G.|O
`
`ÍÎÏÐÑÐÒÓÔÕÔÖÓ×ØÙÚÓÔÛ
`
`Exhibit 1015 Page 14
`
`

`
`U.S. Patent May, 6, 1980
`
`Sheet 14 of 16
`
`4,202,041
`
`GLOBAL
`OUTLINE
`CHOOSE A DEMONSTRANONJ
`
`KEYBOARD
`
`
`
`
`
`
`
`
`
`LEARNING TO READ
`
`STANDA RD TYPEWRITER
`
`ASSORTED FIGURES
`
`SPIRALS
`DRAWING on THE DISPLAY
`DRAWING on THE KEYBOARD
`
`
`
`MARKER
`
`LABEL
`
`
`
`
`
`
`DISPLAY
`DISPLAY
`HELP
`KEYBOARD
`
`
`KEYBOARD
`|
`GLOBAL
`
`CHANGE
`
`CLEAR
`
`PRINT
`
`OUTLINE
`
`CONSTRUCT A FIGURE ON THE KEYBOARD DISPLAY
`
`LABEL
`
`
`
`CHANGE
`
`KEYBOARD . HE”:
`
`CLEAR
`DISPLAY
`
`
`DISPLAY
`
`FIG. I2
`
`
`ÜÝÞßàßáâãäãåâæçèéâãå
`
`Exhibit 1015 Page 15
`
`

`
`U.S. Patent May, 6, 1980
`
`Sheet 15 of 16
`
`4,202,041
`
`NGE
`
`BOARD
`
`HELP
`
`CLEAR
`
`DISPLAY
`
`pR|N'r
`
`DISPLAY
`
`F|G.l3
`
`OUTLINE
`
`I
`
`MARKER
`
`CHANGE
`
`KEYBOARD
`
`HELP
`
`CLEAR
`
`DISPLAY
`
`PRINT
`
`DISPLAY
`
`F|G.l4
`
`êëìíîíïðñòñóðôõö÷ðñø
`
`Exhibit 1015 Page 16
`
`

`
`U.S. Patent May, 6, 1930
`
`Sheet 16 of 16
`
`4,202,041
`
`FISH
`
`FlG.|5
`
`ùúûüýüþ   
`
`Exhibit 1015 Page 17
`
`

`
`
`
`1
`
`DYNAMICALLY VARIABLE KEYBOARD
`TERMINAL
`
`4,202,041
`
`The Government has rights in this invention pursuant
`to Grant No. J-43163 and IPA-0010 awarded by the
`National Science Foundation.
`
`BACKGROUND OF THE INVENTION
`
`The field of this invention is the computer terminal
`and particularly interactive terminals which permit
`access and operation with respect to computers by op-
`erators using as an input device a keyboard or some
`such form of data entry arrangement by which the
`operator can utilize the power available in the computer
`and its associated memory and software.
`In the prior art, one standard form of computer termi-
`nal has been a keyboard device of the teletype or type-
`writer variety and at present such devices have the
`ability to provide the USASCII,
`I968 STANDARD
`SET OF NINETY-FIVE PRINTABLE CHARAC-
`TERS OR SOME PORTION THEREOF. This stan-
`dard set of symbols which has evolved from the type-
`writer with its keyboard and symbol set is generally
`associated with a video display output device which
`will display the selected characters in the form of the
`text which has been entered by the keyboard and con-
`ventional arrangements for editing, erasing and other
`operations.
`The limitations of prior art terminals may not have
`seemed unduly restrictive in many quarters. Indeed, the
`concept of defming the available symbols, or groups of
`symbols,
`to have special meanings under particular
`circumstances has allowed the standard symbol set to
`do more than double duty. Thus 1 and “ and .P. have
`all three been used to mean “raise to the power"; “O
`backspace -” may mean “the Greek letter theta"; “INT
`(a.b.f.x)" my mean
`
`0
`I fix)":
`(1
`
`and “A.SUB.i+j" may mean “A,-+,-”. In an environ-
`ment in which much of the serious communication from
`the user to the computer is still referred to as “writing
`code" such conventions are readily accepted. And yet,
`there are signs that the situation is increasingly less
`satisfactory, particularly as the user group becomes less
`dominated by computer professionals and demi-profes-
`sionals.
`The alternate forms in which the standard symbol
`sets are used are not desirable in instmctional applica-
`tions, for example, because they require the student to
`learn extraneous codes and because they stress nota-
`tional schemes different from those used in the pub-
`lished literature and in textbooks. They are also trouble-
`some because the particular notational conventions are
`relatively arbitrary;
`the convention adopted by one
`author is likely to be different from his colleagues’ and
`different from the conventions used in algorithmic pro-
`gramming languages, such as Fortran, Cobol, MAD,
`etc. These languages, in turn. use conventions which
`differ from one to another. Thus, the use of a limited,
`standard symbol set which does not include the natural
`symbols (or positioning) of particular applications para-
`doxically tends to lead to a diversity rather than a uni-
`
`5
`
`ll)
`
`2
`formity of notation. This is distracting and confusing.
`particularly in an educational situation.
`Another interesting type of example, which stresses
`the differences between display and input capabilities,
`can be drawn from a report on a sophisticated, multi-
`character cathode-ray-type display terminal developed
`at the MIT Electronic Systems Laboratory, for use in
`libraries, as part of the Intrex project (See “A Transla-
`tor Program for Displaying a Computer Stored Set of
`Special Characters" Mobyuki, Goto, Report ESL-R-
`429. Massachusetts Institute of Technology, 1970). The
`partial line of text illustrated below (in quotes) repre-
`sents a significant step beyond the display capability of
`run-of-the-mill terminals. The use of superscripts and
`subscripts and of nonstandard symbols allows the chem-
`ical fonnula in the line to be represented on the display
`in its standard form:
`
`". .
`
`. in the system Mn,,Fe3_,,«04+7for <1 . . ."
`
`The same report also explains how that information
`must be input:
`
`25
`
`. in the system Mn’sub x‘Fe‘sub 3—x"0‘sub 4+
`“. .
`‘gamma“ for x ‘less than‘ I . . ."
`
`The augmented display capability illustrated above is
`certainly valuable in its own right. However, there is
`less advantage in its use if the user must train himself to
`use the different inputting code. It is perhaps surprising
`that the latter form is as readable as it is; for a person
`unfamiliar with chemistry it might even be more read-
`able than the correctly displayed form. As an input
`scheme for "chemists”, however, spelling out the non-
`standard symbols and fomiat in English-like text is at
`best a tedious and error-prone option.
`Other schemes, deviating further and further from
`the standard symbol set, have been developed in re-
`sponse to the continuing pressure for more and different
`input symbols. One approach has been to replace the
`entire standard keyboard symbol set with another one,
`specially designed for a specific application. Perhaps
`the best known example of this is an APL terminal, for
`which the keyboard labels and type head are changed to
`provide a unique set of symbols, the symbol set of the
`APL programming language.
`Another approach has been to add more keys to the
`keyboard. Some of the extra keys may be assigned high-
`level operational meaning, such as log x or e*. Experi-
`mental terminals for engineering applications have been
`developed with as many as 250 “side” keys, alongside
`the standard keyboard. Nonetheless, such keyboards are
`still mainly limited to the fixed, though larger, symbol
`set which happens to be included. An additional disad-
`vantage to this, as a general approach,
`is simply the
`distraction that is presented by so formidable an array of
`keys, only a fraction of which are likely to be used in
`any brief period of time.
`A related scheme has been to assign multiple sets of
`meanings to a fixed number of keys. This is an extension
`of the ordinary “shift" concept which allows each key
`to represent
`two different symbols, depending on
`whether the shift key is depressed or not. With multiple
`meanings, the user may be required to type a special
`code—e.g., LEVEL III-—to indicate that he desires to
`switch to a particular set of meanings for the keys. It is
`difficult to provide many readable labels per key, how-
`ever, so the basic limitation remains.
`
`35
`
`45
`
`55
`
`65
`
`
`
`
`


`
`Exhibit 1015 Page 18
`
`

`
`4,202,041
`
`3
`An interesting attempt to provide a multiplicity of
`key labels for a multi-“shift”-level scheme has been
`made by the Philco Houston Operations of the Philco-
`Ford Corporation (See
`“New Conception Man-
`Machine Interface Devices", Developments Section,
`Computer Designs, 7, 32; 1968). Their device consists of
`a “keyboard” comprising two 4X4 arrays of push but-
`tons and an (exchangeable) ten page “book” of plastic
`sheet label overlays which have holes through which
`the buttons can protrude. The book spline locks into the
`center of the keyboard (between the two separate 4X4
`button arrays). When the book is open to a given pair of
`pages the overlays provide the key labels. Mechanical
`cams attached to the book spline and to the individual
`pages actuates switches under the keyboard to identify
`the particular book pad pages selected.
`In a related but more recent approach, workers at the
`Bell Telephone Laboratories have assembled a device
`in which key labels are projected onto a mechanical-key
`keyboard, while the use of partially-silvered mirrors
`allows the user to see his hand, the keys, and the label-
`s—all simultaneously (See Knowlton, K., “Virtual
`Pushbuttons as a Means of Person-Machine Interac-
`tion”, Proceedings Computer Graphics, Patterson Rec-
`ognition and Data Structure Conference, Beverly Hills,
`California, 1975).
`Another approach, used with CRT and other display
`terminals, has been to generate “soft-copy” input areas,
`in which special symbols or function names are dis-
`played on a reserved section of the CRT screen and
`selected by pointing at them with a light-pen, or by
`moving a cursor controlled by a “joy-stick" or track-
`ball, or by interrrupting a light beam in an array, or by
`a similar technique. The previously mentioned MIT
`Project-Intrex BRISC terminal extended this idea by
`displaying variable labels to identify a set of extra push
`buttons mounted along the bottom of the CRT screen.
`“Soft-copy" input modes implemented on the display in
`these various ways can provide an additional richness
`and flexibility in the input symbol set but are generally
`slower to work with than separate keyboards and are
`often not suitable for routine input. They are thus used
`in combination with a standard keyboard, which results
`in the user having to transfer back and forth from one
`type of input device to the other—he must drop the
`light pen to type, for example, and vice versa. More
`importantly, information input in that fashion is usually
`handled in a manner that is not homogeneous with nor-
`mal input. That is, with normal input-handling software
`a key push is simply converted into a binary-bit code,
`unique to the particular symbol selected, and appen-
`ded—if appropriate—to a symbol string input-strearn.
`Such a string is subsequently amenable to the usual
`range of “character-handling" operations, and, of
`course, it can be edited later or redisplayed. In contrast,
`with input schemes like those referred to immediately
`above, the controlling software routine is likely to be a
`specialized one, designed in conjunction with the spe-
`cific picture then being displayed, and directly carrying
`out one branch of the program or another, depending
`on the numerical x and y values of the screen locations
`selected.
`A second basic type of flaw in the usual input device
`is the lack of format flexibility. The user faces a fixed
`key layout, usually with fixed key labels; an instrument
`based on a design for putting English prose onto paper.
`As it happens, the standard symbol layout used is not
`ideally suited even for that purpose, but that quantita-
`
`5
`
`IO
`
`l5
`
`20
`
`25
`
`30
`
`35
`
`45
`
`55
`
`65
`
`4
`tive detail is almost irrelevant in the present context. It
`is important, however, to consider the possibility that
`touch-input communications to a computer may be
`qualitatively enhanced through the use of formats more
`closely related to the specific interactions underway.
`A third major flaw regarding standard, all-purpose
`keyboards, is that they are detrimental to the efficiency
`of human-computer interaction, especially in consider-
`ation of errors and the fallibility of human memory. In
`particular, it is not desirable to always present the user
`with a single keyboard through which he can send any
`communication which is, which has ever been, and
`which might ever be a valid statement. Such keyboard
`does not provide any clues as to what input is meaning-
`ful at a given instant in a given process, although the
`computer software is often very “knowledgeable” and
`sensitive about which is legitimate and what is not. At
`the same time, the very generality of the standard key-
`board provides unlimited opportunity for error.
`SUMMARY OF THE PRESENT INVENTION
`
`The enhanced-input-terminal of the present invention
`provides major new degrees of freedom for touch-type
`input, especially for on-line use of interactive computer
`programs and comprises an integrated system of hard-
`ware and software. The primary input device is a
`“cross-wire", touch-sensitive panel, overlaying a video
`display. A companion video unit is used for output
`display. These are supported by page-refresh memories,
`written (and read) directly by a suitable data processor
`which implements the software logic. The enhanced-
`input-terminal allows the user to define an essentially
`infinite set of symbols and an unlimited variety of "key-
`board" formats. In spite of the generality of input con-
`tent and format, the integrated system develops a stand-
`ard-type of binary-bit-coded input stream, in which the
`individual “symbols” are uniquely and canonically rep-
`resented. As symbol-strings or as a text-file. the input
`stream is amenable to all of the usual “character-ham
`dling” operations, such as substring, concatenate, etc.,
`and to operations of editing, transmission and output.
`The system includes a unique “Terminal Definition
`Language" and compiler, with which one can define
`new symbols, keys, chords (as later defined), and key-
`board images, all comprising an author working set of
`keyboard images and symbols for a user process. Thus,
`a specific set of symbols, keyboard formats, key labels,
`and local operational functions can be defined for each
`task and suitably stored to form a data base for the
`system.
`The objects and features of the invention are thus to
`provide a terminal having available:
`(1) Working symbol set stored in the data base, the‘
`symbols making up such set being author-defmable
`and the set being virtually unlimited in terms of the
`number of symbols or the visual aspect of the sym-
`bols;
`(2) Keyboard descriptions pennitting such keyboards
`to be displayed or drawn on a keyboard display
`unit, the keyboard formats being essentially com-
`pletely arbitrary, allowing an essentially unlimited
`number of possibilities ranging from the standard
`key layout of a typewriter, to drawings, templates,
`or any format oriented to inputting a problem and
`working through to a solution by interaction with
`the computer;
`(3) Descriptions for using keyboards, the keyboards
`being dynamically switchable, keeping pace with
`
` !"#$%&
`
`Exhibit 1015 Page 19
`
`

`
`
`
`4,202,041
`
`5
`the user‘s path through an interaction, being specif-
`ically relevant to the then current aspect of the
`task;
`(4) Sub-routines for use in the presentation of symbols
`from the symbol set. Input actions by the user gen-
`erate a binary coded input stream which is explic-
`itly meaningful in terms of the defined symbol set,
`and which has the attributes of an ordinary text
`that one applies to text files (or symbol strings)
`such as:
`(a) character-handling operations (e.g.,
`concatenate)
`(b) string comparison (e.g., for response analysis
`and interpretations)
`(c) editing
`((1) outputting (including display format control).
`Input is canonicalized, so that a given symbol is al-
`ways represented identically internally, regardless of
`the keyboard-image used to input it and regardless of
`the locations of the keys on different keyboards within 20
`a given working set. In addition, to the extent allowed
`by logical consistency, symbols may also be canonical-
`ized among two or more different working sets; indeed,
`the ASCII set is automatically canonicalized among all
`working sets.
`DESCRIPTION OF THE DRAWINGS
`
`6
`into the refresh memories, a video readout and signal
`generator 17, and a “keyboard" overlay 18 with associ-
`ated keyboard logic control and processor interface
`circuitry 19 are also included in the system. Controller
`16 and its interface circuitry is of a standard type and
`includes ten registers directly accessible by the proces-
`sor and through which information is transferred to and
`from the refresh memories. The keyboard overlay 18 as
`used in conjunction with the video display 13 makes up
`the keyboard display unit which operates as the termi-
`nal’s primary input device for use by the terminal opera-
`tor.
`
`The processor 11 provides its computational power,
`program and library storage media, and storage access
`mechanisms in a conventional manner. A PDP ll/40
`data storage media, for example, can include 28 k bytes
`of internal core memory and 4.8 megabytes of fast disk
`storage 21. The processor 11 is programmed in assem-
`bly language and utilizes a standard disk-operating sys-
`tem for disk storage 21. With the core memory and disk
`storage, the processor 11 provides most of the opera-
`tional logical power, the character and display process-
`ing, and the computational requirements of the termi-
`nal. Details of the operation of particular processors are
`normally well known to the art and that of the PDP
`11/40, for example, can be found in the operating in-
`struction manual and other information normally sup-
`plied to users thereof by the manufacturer. Accord-
`ingly. details thereof are not further needed here.
`The user interface comprises the two video display
`monitors 12 and 13. The monitor 12 serves the normal
`display functions of a terminal while the monitor 13 is
`the basis for the dynamically variable input keyboard.
`Each monitor is backed up by a 256 K-bit solid-state
`“refresh" memory 14, 15 and video-driver circuitry 17
`which develops a continuous black/white video input
`signal from the bit patterns in the respective memories
`14, 15. The video circuitry 17 generates a 454 line scan,
`each line being resolved into 576 points. The memories
`14, 15 are double-ported, and the memory access speeds
`are sufficiently fast that read-and-write access by the
`processor 11 does not interfere with the read activity of
`video circuitry. Thus,
`the displayed image on either
`monitor can be modified essentially at processor speeds.
`In addition, bit-map images stored on the fast disk mem-
`ory can be transferred directly to the video refresh
`memory using conventional computer interface hard-
`ware. In this fashion, the system can switch keyboard
`images in the order of 0.3 seconds.
`The memory word addressing unit associated with
`the display hardware allows an automatic multiple-
`word incrementing mode, which is extremely conve-
`nient for displaying dot matrix characteris one at a time.
`The memory control 16 allows software controlled
`switching among the memories for the computer read-
`write operations.
`While only the two aforementioned video displays
`are implemented in the current configuration, up to
`sixteen 256 K-bit memories can be addressed without
`requiring the normal memory extender unit, for exam-
`ple, on the PDP ll/40 itself, and without using any of
`the computer’s own memory address space. Thus, the
`exemplary system can be augmented to support as many
`as eight separate user-stations, each having the desired
`two video displays. Altemately, extra memories might
`be used to provide grey scales or multi-color signals, or
`various combinations for a smaller number of users.
`
`'()*+*,-./.0-1234-5/
`
`Exhibit 1015 Page 20
`
`length,
`
`10
`
`15
`
`25
`
`FIG. 1 is a system block diagram of a preferred em-
`bodiment of the invention;
`FIGS. 2 and 2A show in general a block diagram of
`the keyboard logic depicted in FIG. 1;
`FIGS. 3, 3A and 3B contain an overall flow chart
`showing the operation of the keyboard logic of FIG. 2
`in identifying cross-point closures at a keyboard display
`unit.
`
`FIG. 4 is a diagrammatic representation of the overall
`data base used in the invention;
`FIG. 5 is a diagrammatic representation of the opera-
`tion required to identify a symbol and to acquire a sym-
`bol description associated with a cross-point selected by
`a user;
`FIGS. 6 and 7 are diagrammatic representations of
`the operations required to display the visual appearance
`and to perform an action associated with symbols se-
`lected by a user and to transmit their symbol codes
`when required;
`FIG. 8 is a diagrammatic representation of the opera-
`tion required to change a keyboard image at the key-
`board display unit;
`FIGS. 9 and 10 are flow charts depicting the steps
`required in a portion of the operation shown in FIG. 8;
`FIGS. 11-15 depict exemplary keyboards useful in
`explaining a typical operation which could occur in
`using the system of the invention.
`DETAILED DESCRIPTION OF THE
`PREFERRED EMBODIMENT
`
`The system shown in FIG. 1 consists of three main
`parts: a processor 11 which in a particular embodiment
`may be, for example, a processor made and sold by
`Digital Equipment Corporation of Maynard, Massa-
`chusetts under the model designation PDP ll/40. Such
`processor has appropriate data storage media, includ-
`ing, for example, a peripheral disk storage memory unit
`21 as well as processor internal core memory. Two
`video monitor display units 12,13 with solid-state, page-
`refresh memories 14,15 and a memory control circuit 16
`for controlling the insertion of data from the data base
`
`30
`
`35
`
`40
`
`45
`
`55
`
`65
`
`

`
`7
`The third major component of the hardware is the
`keyboard display unit which includes overlay panel 18,
`with its associated scanning logic 19, discussed in con-
`nection with FIG. 2, and video display 13. The interface
`between such logic and the processor is standard and
`available from the processor manufacturer. In the case
`of a PDP 1 1/40 processor, for example, such interface is
`available as General Device Interface, DRll-C. The
`overlay panel 18 fits over the video display 13 which is
`oriented with its screen nearly horizontal, i.e., at a con-
`venient angle for operator use. The display 13 and over-
`lay panel 18 comprise the input key board. As disclosed,
`the overlay panel 18 is a cross-wire matrix, imbedded in
`plastic sheets. The user’s finger pressure at a cross-point
`causes a contact to be made and, an identification by
`logic circuit 19 to processor 11 of the appropriate row
`and column coordinates for all cross-points which have
`been closed. In the processor 11, an identified cross-
`point is checked against the stored list of the cross-
`points associated with the “keys” on the currently dis-
`played keyboard. If it is not a valid point (i.e., the cross-
`point is not used on the current keyboard), it is ignored.
`If it is valid, a key-click signal is echoed-back, the ap-
`propriate working-symbol code associated with the
`valid “key” is thereupon accessed from the stored cur-
`rent keyboard description and checked against the com-
`plete stored symbol set to select the symbol description
`associated with such symbol code. The symbol descrip-
`tion may include a pictorial representation of the se-
`lected symbol which will be thereupon presented to the
`appropriate location on the keyboard display monitor
`12. When a cross-point not associated with a key is
`pressed, the terminal processor also stores information
`concerning such cross-point for possible use in connec-
`tion with symbol related actions. An example, as shown
`in FIG. 13 discussed below, shows such cross-points
`being used in the construction of a drawing directly on
`the keyboard. When using the keyboard overlay for
`such graphical
`input,
`it
`is possible, using presently
`known expansion an