United States Patent [19J
`Agulnick et al.
`
`[75]
`
`[54] CONTROL OF A COMPUTER THROUGH A
`POSITTON-SENSED STYLUS
`Inventors: Todd Agulnick, Newton Centre,
`Mass.; Robert Carr, San Francisco,
`Calif.; Tony Roeber, Woodside,
`Calif.; S. Jerrold Kaplan, San
`Francisco, Calif.; David R. Low,
`Oakland, Calif.; Michael Ouye, Palo
`Alto, Calif.
`[73] Assignee: GO Corporation, Foster City, Calif.
`[21] Appl. No.: 610,231
`[22] Filed:
`Oct. 31, 1990
`Int. CI.5 •.•••.••.••••.••••.••.•..•••••.•.••••.•••.••..• G09G 5/00
`[51]
`[52] U.S. CI . ..................................... 345/156; 345/179
`[58] Field of Search ....................... 382/13, 14, 57, 59,
`382/40; 178/18, 19; 340/706, 707,709;
`345/156, 179
`
`[56]
`
`References Cited
`U.S. PATENT DOCUMENTS
`3,699,439 10/1972 Turner ............................... 324/71.1
`3,832,693 8/1974 Ishizaki et al ....................... 364/900
`4,016,542 4/1977 Azure .................................. 364/900
`4,055,726 10/1977 Turner et al .......................... 178/18
`4,071,691 1/1978 Pepper, Jr ............................. 178/19
`4,112,415 9/1978 Hilbrink ................................ 382/13
`4,129,747 12/1978 Pepper, Jr ............................. 178/18
`4,177,354 12/1979 Mathews ............................... 178/19
`4,184,147 1/1980 Seelbach ............................... 382/13
`4,198,539 4/1980 Pepper, Jr ............................. 178/18
`4,262,281 4/1981 Buckle et al .......................... 382/13
`4,293,734 10/1981 Pepper, Jr ............................. 178/18
`4,302,011 11/1981 Pepper, Jr ............................. 178/19
`4,318,096 3/1982 Thornburg et al ................. 360/706
`4,353,552 10/1982 Pepper, Jr ............................. 382/13
`4,365,235 12/1982 Greanias et al ....................... 382/13
`4,371,746 2/1983 Pepper, Jr ............................. 178/18
`4,456,787 6/1984 Schlosser et al ...................... 178/19
`4,475,239 10/1984 van Raamsdonk ................... 382/52
`4,520,357 5/1985 Castleberry et al ................ 340/703
`4,641,354 2/1987 Fukunaga et al ..................... 382/13
`4,672,677 6/1987 Yamakawa ............................ 382/13
`4,679,241 7/1987 Lukis ..................................... 382/13
`4,680,430 7/1987 Yoshikawa et al ................... 178/19
`4,680,804 7/1987 Kuzunuki et al ..................... 382/13
`4,764,885 8/1988 Greanias et al ..................... 364/571
`
`I 111111111111111111111111111111111111 IIIII IIIII 11111 lllll 111111111111111111
`US005347295A
`[11] Patent Number:
`[45] Date of Patent:
`
`5,347,295
`Sep. 13, 1994
`
`4,786,765 11/1988 Yamanami et al. ................... 178/19
`4,831,556 5/1989 Oono ................................... 364/521
`4,839,634 6/1989 More et al ..
`4,972,496 11/1990 Sklarew ................................ 382/13
`5,050,105 9/1991 Peters .................................. 340/734
`5,053,758 10/1991 Cornett et al ....................... 340/712
`5,151,950 9/1992 Hullender .
`
`FOREIGN PATENT DOCUMENTS
`0203324 3/1986 European Pat. Off ..
`0242598 10/1987 European Pat. Off ..
`0254561 1/1988 European Pat. Off ..
`2117154 10/1983 United Kingdom .
`2193023 1/1988 United Kingdom .................. 382/13
`
`OTHER PUBLICATIONS
`IBM Technical Disclosure Bulletin, "Object-Picking
`Method by Hierarchical Hand-Marking," vol. 30, No.
`9, pp. 348-350, Feb. 1988 (New York).
`GRiD Systems Corporation, "GRiDPAD Computer
`Owner's Guide" (1989-1990, California).
`Wacom Co., Ltd., "User Manual for SD-510 AS Type
`Digitizer" (Jul. 1988; place of publication unknown).
`Linus Technologies, Inc., LINUS Write-Top User's
`Guide (1987 Virginia).
`PC Magazine, "Digitizing Tablets/Pointing the Way to
`Easier Input" by Winn L. Rosch (1989 U.S.A.).
`Primary Examiner-Jeffery Brier
`Attorney, Agent, or Firm-Matthew C. Rainey; Amir H.
`Raubvogel
`ABSTRACT
`[57]
`A notebook computer which is controlled by a stylus
`executing gestures on the computer screen. The stylus
`and the computer include complementary electronic
`circuitry by which the proximity of the stylus tip to the
`computer is sensed. The proximity sensing is used to
`detect to approach of the stylus tip to the computer
`screen, and gestural commands are then entered on the
`screen by moving the stylus. The entry of a command is
`terminated by removing the stylus tip from proximity
`with the screen, which is detected by the computer,
`which then implements the command. Alternative
`methods of implementing the commands include time(cid:173)
`outs and command termination buttons.
`
`46 Claims, 73 Drawing Sheets
`
`Slation 00 S t 112
`
`Pert 116
`
`Bue
`Slalicn
`118
`
`FAX/Oala
`Modem
`114
`
`GOOGLE EXHIBIT 1017
`
`Page 1 of 85
`
`

`

`U.S. Patent
`
`Sep. 13, 1994
`
`Sheet 1 of 73
`
`5,347,295
`
`2--....
`
`Fig. 1
`
`GOOGLE EXHIBIT 1017
`
`Page 2 of 85
`
`

`

`U.S. Patent
`US. Patent
`
`Sep. 13, 1994
`Sep. 13, 1994
`
`Sheet 2 of 73
`Sheet 2 of 73
`
`5,347,295
`5,347,295
`
`0:
`
`0 -
`
`m.9“.
`
`9.
`
`C\I
`9
`LL
`
`Page 3 0f 85
`
`GOOGLE EXHIBIT 1017
`
`\
`
`¥
`C\I
`
`GOOGLE EXHIBIT 1017
`
`Page 3 of 85
`
`

`

`U.S. Patent
`
`Sep. 13, 1994
`
`Sheet 3 of 73
`
`5,347,295
`
`LCD
`
`Display -10
`
`Pen
`Sensor
`20
`
`Display
`Controller
`80
`
`Co-
`processor
`90
`I
`
`Custom
`ASIC
`104
`
`-
`
`SCSI
`100
`
`CPU
`50
`
`Static
`RAM
`70
`
`EEPROM
`60
`
`Serial
`Controller
`108
`
`I
`r:::::::1==::::1
`·
`::;i E
`II
`II
`xpansion II
`Slot 112
`
`I
`
`11
`
`Base
`Station
`Port 116
`
`Fig. 3
`
`Base
`Station
`118
`
`FAX/Data
`Modem
`114
`
`GOOGLE EXHIBIT 1017
`
`Page 4 of 85
`
`

`

`U.S. Patent
`
`Sep. 13, 1994
`
`Sheet 4 of 73
`
`5,347,295
`
`p Notebook: contents
`1
`·0o·· ument·····Edit·····create······View·····.show····.sort·······110/ ············
`
`:::::::a::::::::::::::::::::::::::::;:::::::::::::::::::::::::;:::::::::::::::::::::::::::::::::::::·::::::::::::,::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::····:
`
`0
`2 Name
`~
`340
`2
`I r - -
`D I
`· · · · · · · · · · · · · · · · · · · · t>
`First ExP2nence
`D Samples~·~·;~·················· t> 3
`D Application · · · · · · · · · · · · · · · · · · · · · · · · · · t> 6
`D Reports~ ·~20 · · · · · · · · · · · · · · · · · · · t> 1
`3001 t 290
`l 210 \.. 2eo
`
`1,()
`
`2
`[
`
`o
`~
`N
`
`n
`
`o ::, a:-::,
`
`iZ
`
`10
`
`190
`
`5
`_)
`
`0 -
`
`0
`
`N -
`
`GOOGLE EXHIBIT 1017
`
`Page 5 of 85
`
`

`

`U.S. Patent
`
`Sep. 13, 1994
`
`Sheet 5 of 73
`
`5,347,295
`
`.sample Notebook: con-tents
`-- 1 -
`....................................................................................................................................................................................
`Document Edit create View show sort
`:::::::::::::::::::::::::::::::::::::::::;::::::::::::::::;::::::::::::::::::;:::::::::::::::::::::;::::::::::::::::::::::::::: .. : .................................. ·····•··········.
`~~
`~
`· · · ·
`[) First Ex~rience · · · · · ·
`· · · · · t> 2
`[) samples · · · · · · · · · · · · · · · · · · · · · · · · · · · · · t> 3
`D Application · · · · · · · · · · · · · · · · · · · · · · · · · · t> 6
`D Reports · · · · · · · · · · · · · · · · · · · · · · · · · · · ·t> 1
`r ........................................... Help: Co~~·····························=···~···=···
`
`-()
`- ...(cid:173).,,
`
`0
`::,
`~
`::,
`c;-
`
`::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
`
`Document Edit create View Show sort
`10
`~~
`~
`D Using Help
`· · · · · · · · · · · · · · · t> 2
`D PenPolnt Help
`· · t> 10
`D Appfacations · · · · · · · · · · · · · · · · · t> 155
`
`~
`
`340
`
`•
`
`Fig. 5
`
`GOOGLE EXHIBIT 1017
`
`Page 6 of 85
`
`

`

`U.S. Patent
`
`Sep. 13, 1994
`
`Sheet 6 of 73
`
`5,347,295
`
`-1 -
`
`.::::::::::::::::::::::::::::::::::::::::::;:::::::::::::::::::::::::::::::::::::::::"···''
`
`.::::::::::·:·::··::::::::::::::·:::::::::::::::·:::::::::::::::::;:::::::::::
`
`sample Note.book: contents
`Document Edit create View Show Sort
`eage
`Name
`'-350
`Cl First ExP2rience · · · · · · · · · · · · · · · · · · · · · t> 2
`310--D .samples·· · · · · · · · · · · · · · · · · · · · · · · · · · · · · t> 3
`f D New Uses for Eggplant · · · · · · · · · · · · · · t> 4
`360lD Bottle Design Needed· -· · · · · · · · · · · · · t> 5
`D Application · · · · · · · · · · · · · · · · · · · · · · · · · · t> 6
`320-f-.-D Reports
`· · · · · · · · · · · · · · · · · - -· - · · -· · -t> 1
`D August 2nd · · · · · · · · · · · · · · · · · · · · · · t> 8
`c;70
`
`f proximityj
`
`,.;;."iiiiiiiiiiiiiiiiiiiiii~--------ia:zza-------......
`
`I
`
`Fig. 6
`
`GOOGLE EXHIBIT 1017
`
`Page 7 of 85
`
`

`

`U.S. Patent
`
`Sep. 13, 1994
`
`Sheet 7 of 73
`
`5,347,295
`
`.sample Notebook: contents
`-- 1 -
`....................................................................................................................................................................................
`Document Edit Create View Show sort
`:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: ... ::::::.;::::::::::::::::::::::::::::::::::::::.: .. : .. :.: ............................................... .
`~~
`~~
`· · · · · · · · · · ·
`[) First Exp?rience
`· · · t> 2
`[) samples· · · · · · · · · · · · · · · · · · · · · · · · · · · · · t> 3
`D Application · · · · · · · · · · · · · · · · · · · · · · · · · · · t> 6
`I) Reports · · · · · · · · · · · · · · · · · · · · · · · · · · · · t> 1
`r440
`390 ~----Ap_p_lic_atio_~ __ , ---········ .. ··-·· .. ·······~--
`Document Edit Insert case Format
`The present invent.ion describes
`·'=1.i"
`methods and apparatus for a stylus-driven
`,,
`computer. Said computer presents the
`11
`behavior of a multi-pa~e notebook indudin~ ll
`visible paRes, paRe-tum effects, tabbed
`\l
`ii
`bookmarks alonR one or mo~ edRe,
`ii
`•ttoatinA• paAes temporarily removed from
`380) their notebook location, a table-of-contents, ll
`l.. and contents of sub-sections. Each pa~e in II
`the notebook contains one document which ii
`ll
`may contain many physical paRes when
`printed. These paRes are analoRous to data li
`ii
`files in other computers and may contain
`II
`different types of documents such as text,
`drawinRS, FAX imaAes, or specialized forms. ll
`l!
`Some information C such as the •Help
`Information")_ is : :~ in separate
`
`#' ••
`
`3
`
`-.,..,, ... _...
`
`.-.--.•--h"
`
`..... .,.. .........
`
`--~····~··--E;· ~~=-=-:::::.=.-=>.-:=-===----=:::·••c·::::,._ .. _J"'-=:::::-=::"-':.:::>. .. -:_.~-:<:.--:,,:::-...::::::wn
`
`f proximity)
`1-'·••·--·•···-... · ... ·•··· ... _ .. ______ aazza ________ _.
`
`400
`
`~ D ~
`ti1~®~~
`
`Fig. 7
`
`GOOGLE EXHIBIT 1017
`
`Page 8 of 85
`
`

`

`U.S. Patent
`
`Sep. 13, 1994
`
`Sheet 8 of 73
`
`5,347,295
`
`390
`
`440
`
`6
`
`__
`
`Applicatr>n
`Document Edit Insert case Format
`·····················:····:···· ..... : ... ; ....................................................................... :; ............... :.:::;: ....... ::::;;;::::: ................................... .
`The present invention desaibes methods and
`I
`apparatus for a stylus-driven computer. Said computer
`presents the behavior of a multi-PSAe notebook inciudinq
`visible paQes, paQe-tum effects, tabbed bookmarks aJonq
`one or more edRe, llfloatinq• paQes temporarily removed
`from their notebook location. a table-of-contents, and
`contents of sub-sections. Each paQe in the notebook
`contains one document which may contain many physical
`PaRes when printed. These pai:ies are analoRous to date.
`files in other computers and may contain different types of
`documents such as text, drawinqs. FAX ime.Qes. or
`specialized forms. Some information ( such as the •Help
`Information") is contained in separate notebooks with all of
`the features listed above. The primary means of control of
`the computer is a rich set of qestures with the stylus over
`almost any object visible on the display. Gestures with a
`stylus offer a more concise vocabulary than tradition
`pointinR devices because they combine both selection (by
`their location) and meaninA (the specific Aesture) in one
`movement. A common set of Aestures can delete. move.
`copy, or edit diverse objects such as text fraqments.
`buttons. documents. or entire notebooks. Gestures can
`initiate the insertion of hand-written text, the editinq of
`..
`,,
`e><istinq text. Pe.Re-tuminQ, saollinq within a window.
`- zoominR or shrinkinq of entire paqes, selection. summoninq '!
`of windows conta.ininq operator quidance, or the creation of :\
`links (•Qoto• buttons) to other paQes.
`..
`
`380
`L...
`
`The proximity of the stylus to the displays front.
`surface is used to anticipate and terminate stylus input
`events. 8y sensinA both the proximity of the stylus tip to.
`and contact with, the display surface the user-interface
`software can more accurately discern the vertical
`movement of the $tylus, provide e. richer vocabulary of
`stylus movements for control of the computer. and offer
`better feedback to the user.
`
`When usinq the stylus for selection, movements into
`proximity can triqQer display events which qive the user e.
`preview of what display objects are understood to be
`tarqeted by the computer. For example the cornpu~er ~an
`Fig. 8
`
`,iJ..
`400J
`
`..
`
`··
`
`GOOGLE EXHIBIT 1017
`
`Page 9 of 85
`
`

`

`U.S. Patent
`
`Sep. 13, 1994
`
`Sheet 9 of 73
`
`5,347,295
`
`••••••••••••••••••••••••••••••••••• ...... u••••••••••nHoooo, .. ,,,oo••••••••••••••••H•oou,o••••••••u•••••••••••••••••••••••••••••••••••••••·•••·••••••·•• ••·•••·••••••··• •••••
`
`-
`
`1 -
`
`sample Nmebook: contents
`Document Edit create View Show Sort
`::;:::::::;::::::::::::::::;:;::::::::::::::::::::::::::::::::::::::::::::::::.:::.::::::::::::::::::::::::;::;:::::::::::::::::::::::::::: ..................... : .. :·················
`Name
`Page
`[j First Ex~rience · · · · · · · · · · · · · · · · · · · · · · t> 2
`CJ samples· · · · · · · · · · · · · · · · · · · · · · · · · · · · · · t> 3
`CJ Application · · · · · · · · · · · · · · · · · · · · · · · · · · · t> 6
`CJ Reports · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · t> 1
`
`390
`
`440
`
`6
`
`...:~~:-
`
`,380
`, __ ...................................................................... .
`"'f'plicatbn
`Document Edit Insert case Format
`I
`The present invention describes
`methods and apparatus for a stylus-driven
`,,
`computer. Said computer presents the
`11
`behavior of a multi-paAe notebook includinA II
`ii
`visible paAes, paAe-tum effects, tabbed
`bookmarks alonR one or more edAe,
`ii
`"floatinA• paAes temporarily removed from
`11
`their notebook location a table-of-contents, H
`and contents o su -sections. ach paAe in 1\
`)
`1! L 4 20
`the notebook contains one document which
`l!
`may contain many physical paAes when
`printed. These paAes are analoAous to data 11
`files in other computers and may contain
`11
`n
`different types of documents such as text,
`drawinRs, FAX imaRes. or specialized forms. H
`l!
`Some information ( such as the •Help
`
`,,,,,,~,:~:n:~~oni. i-~ :~teJ~l::::::.~::=~~~==~~=:~.~:~~'~''''
`·c·410
`\-------------1!:ZZ:ZJ ----------
`~ [~~t,'E,Xiffeq¥~ J!) a ~
`~ 0 rti.tl~t®~
`
`(proximity)
`
`Fig. 9
`
`GOOGLE EXHIBIT 1017
`
`Page 10 of 85
`
`

`

`U.S. Patent
`
`Sep. 131 1994
`
`Sheet 10 of 73
`
`5,347,295
`
`...... 1--
`
`::::::::::::::::::::::::::.:::::::::::m::::;:::::::::::::::::::::::::: •• :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::•····••u••···················· .. ···•······.···
`
`sample Notebook: eon-tents
`Document Edit Create View .show sort
`~~
`~~
`D First Experience · · · · · · · · · · · · · · · · · · · · · · t> 2
`D samples······················ · · · · · · · t> 3
`D Application · · · · · · · · · · · · · · · · · · · · · · · · · · · C> 6
`[) Reports · · · · · · · · · · · · · · · · · · · · · · · · · · · · · C> 7
`
`390
`
`440
`
`()
`0
`::::,
`(ti
`::::,
`~
`
`6
`
`Appllcatlon
`Document Edit Insert case Format
`The present invention desaibes methods and -1,=(cid:173)
`apparatus for a stylus-driven computer. Said
`computer presents the behavior of a multi-paAe
`notebook includinA visible paAes, paAe-tum effects,
`tabbed bookmarks alonA one or more edQe,
`"floatinR• paRes temporarily removed from their
`notebook location, a table-of-contents, and contents
`
`:::::::::-.:::::=:::=::::::::::::::::::-.::::: .. :.-::::::::::::::..,-:.::::::-•UHHO-••••:::::::::::::;:::::::::::::::::::::::.::,:.::-.u:,,: ............. ::::,,
`
`380
`
`...
`
`430
`
`-----------IZZZZJ----------
`(r:,rc»cimlt y)
`···•···-----
`~D~
`~ [Fj(stgXiie:ietj:ij
`[i)tl~~~
`
`Fig. 10
`
`GOOGLE EXHIBIT 1017
`
`Page 11 of 85
`
`

`

`U.S. Patent
`
`Sep. 13, 1994
`
`Sheet 11 of 73
`
`5,347,295
`
`•••••••••••oo,uoo•o•n•••••••••HoUo•••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••O•••••u••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••
`
`..- 1 -
`
`.:·:::··:·:·:::::::::::::::::::m::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::.::::: .......................................................... ·.·.·.··.·-
`
`5ample Notebook: Con1ents
`Document Edit Create View .show .sort
`~~
`~~
`D First Ex~rience
`· · · · · · · · · · · · · · · · · -· · · t> 2
`D samples······················ · · · · · · t> 3
`D Application · · · · · · · · · · · · · · · · · · · · · · · · · · · t> 6
`D P£ports · · · · · · · · · · · · · · · · · · · · · · · · · · ·t> 1
`440
`380
`Application
`
`6
`
`'iii'·
`,,
`11
`
`The present invention describes
`methods and apparatus for a stylus-driv n
`computer. Said computer presents the
`behavior of a multi-paAe notebook indu nA II
`ii
`visible paqes, paqe-tum effects, tabbed
`II
`bookmarks alonq one or more edqe,
`Sfloatinq• paqes temporarily removed fro
`their notebook location, a table-of-conte ts, il
`in ii
`and contents of sub-sedions. Each paA
`the notebook contains one document wh ch il
`ll
`may contain many physical paAes when
`printed. These paqes are analoAous to ata ii
`ii
`files in other computers and may contain
`ii
`different types of documents such as tex ,
`drawinqs, FAX imaAes, or specialized for s. ii
`Some information ( such as the 1H elp
`\I
`------~--------rt:z::1t::;a·-----·--...... -..
`)t.1
`Information') is contained in separate
`445
`
`11
`
`( proximity)
`
`~"-···--------------a:zza----------
`~ Ffi(s(Elpjri&fij J!] ~ ~
`~ ~ [Jtl~~~
`
`Fig. 11
`
`GOOGLE EXHIBIT 1017
`
`Page 12 of 85
`
`

`

`U.S. Patent
`
`Sep. 13, 1994
`
`Sheet 12 of 73
`
`5,347,295
`
`• • • • • • • • • . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . h . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
`
`-
`
`1 --
`
`.sample Notebook: con-tents
`Document Edit create View Show sort
`:::::::::::::::::::::::;;:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::;:::::::::::::::::;:::::::: ........ ··.·· ... ·.· .. ·
`M~
`~~
`D First Experience
`t> 2
`D samples·
`· t> 3
`D Application
`t> 6
`D Reports
`·
`t> 1
`
`·
`·
`
`· ·
`
`·
`
`·
`
`·
`
`·
`
`/'"440
`
`1 380
`
`j445
`
`r ···--·-·-· ... !. ........ _ Ap~~~~~············· .. ································~···
`Document Edit Insert case Format
`:::::::::::::::::::::::::~.::::::::::::::--.. :::::::::::::::::::::::::.-=::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::!!
`"'~;·;.
`The present invention describes
`methods and apparatus for a stylus-driven I
`computer. Said computer presents the
`behavior of a multi-paAe notebook includin~
`visible paQes, paQe-tum effects, tabbed
`bookmarks alonR one or more edQe,
`"floatinA• paAes temporarily removed from
`their notebook location, a table-of-contents,
`and contents of sub-sections. Each paAe in
`the notebook contains one document which
`..
`may contain many physical pf1Aes when
`printed. These paAes are analoAous to data 11
`ii
`files in other computers and may contain
`II
`different types of documents such as text,
`drawinRs, FAX imaReS, or specialized forms. II
`!!
`Some information (such as the •Help
`~!!J
`I nformationi is C!:Jn1.~~ in separate
`. ____ ... __ ,.
`···;.:.:···-~····=---==~===·~· ======-"'-"'··=-=-""""'···-:=-..:.-..:::::::=:~.--·-·"""····
`{proximity)
`~ ff~rsJ Eitti¥¢1
`
`___________ SZZZZZI ________ _.
`
`Fig. 12
`
`GOOGLE EXHIBIT 1017
`
`Page 13 of 85
`
`

`

`U.S. Patent
`
`Sep. 13, 1994
`
`Sheet 13 of 73
`
`5,347,295
`
`... 26•
`
`. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . u . . . . . . . . . . o . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
`
`Expense Report
`Insert Case
`
`·•··•• ••···--:•-•·••-•·-·····•·;-•... ·.• ·-·•·· •• -·· • ..•. · ....... - .. •
`
`. • - .• -. •
`
`• -·-.. •
`
`.
`
`X
`./
`0
`S
`P
`F
`
`ftl
`
`n
`0 :::s ,..
`:::s ... en
`0 -ftl
`0 .,
`
`n
`
`:::,
`Q.
`
`n -ftl
`
`:::, ,..
`CII
`
`-i
`0
`X
`"Tl
`
`0 .,
`3
`CII
`
`-0
`
`ftl .,
`
`V,
`0
`::,
`0
`
`•.•.• ~ ·•:--:--:.~•<-··•·•••··•:-:.:-:-•❖•
`
`Document Edit
`Undo
`Select All
`Move
`Copy
`Delete
`Options....
`
`Edit Pad
`Spe11.,.
`Proof....
`Find....
`
`450/
`
`Fig. 13
`
`GOOGLE EXHIBIT 1017
`
`Page 14 of 85
`
`

`

`U.S. Patent
`
`Sep. 13, 1994
`
`Sheet 14 of 73
`
`5,347,295
`
`Help: Title Line Gestures
`
`CJ)
`C
`3
`3
`0 ..,
`
`C Create new document
`F
`Find
`H
`Home
`p
`Print
`3
`Send
`✓ 0pt1ons
`
`460
`
`-- Next document
`-.... Previous document
`-... First document (notebook TOC)
`-
`-- Last document
`
`Fig. 14
`
`GOOGLE EXHIBIT 1017
`
`Page 15 of 85
`
`

`

`U.S. Patent
`
`Sep. 13, 1994
`
`Sheet 15 of 73
`
`5,347,295
`
`462
`-6--
`
`464 t
`
`490
`
`470
`
`480
`
`Application
`Document Edit Insert case Format
`::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::...........
`The present invention desaibes methods and
`·;;
`apparatus fore. stylus-driven computer. Said computer
`"
`presents the behavior of a multi-paRe notebook includinA ii
`ii
`visible paRes, PaRe-tum effects, tabbed bookmarks
`aJonA one or more edAe, 'floe.timt paAes temporarily
`11
`removed from their notebook location, e.
`"
`table-of-contents, and contents of sub-sections. Ee.ch
`"
`pe.Ae in the notebook contains one dorument which may 11
`ll C"'>
`contain many physical pe.Res when printed. These
`II g
`pe.Aes are e.ne.lOAOUs to date. files in other computers
`ii ~
`and may contain different types of documents such as
`text, drawinRs, FAX ime.Res, or specialized forms. Some 11 ~
`information (sud, e.s the •Help Information") is contained !I
`-ri
`ii ~
`in separate notebooks with all of the features listed
`above. The primary means of control of the computer is Ii
`!!..._._r-510
`a rich set of Restures with the
`er almost
`obiect visible on the display. Gestures wit a stylus offer ll
`a more concise vocabulary t e.n tra it1on pointinA
`11
`Ii
`devices because they combine both selection (by their
`II
`location) and meaninA (the specific Resture) in one
`l! 3
`movement. A common set of Aestures can delete,
`II ~
`move, copy, or edit diverse objects such as text
`II "'
`fraAments, buttons, documents, or entire notebooks.
`Gestures ce.n initiate the insertion of hand-written text.
`1\
`the editinA of e><istinA text, pe.Ae-tuminA, scrollinR within
`ii
`a window, zoomini;:a or shrinkinR of entire paQes,
`11
`seleaion. summoninA of windows containinA operator
`ii
`Quidance, or the creation of links c■Aoto• buttons) to
`H
`
`other ~~:~ximity of the stylus to the display's front I
`
`surface is used to anticipate and terminate stylus input
`/l
`events. By sensinA both the proximity of the stylus tip to, ii
`and conte.ct with, the display surface the user-interface
`\)
`~w.-ililiill.aliiWililiMililllllliililiiilai.iiilliliW&:al:i
`.
`,._:·_
`
`~ rrfsi:aw~ 4' D ~
`~ It] ti1~~~~
`
`Fig. 15
`
`500
`
`GOOGLE EXHIBIT 1017
`
`Page 16 of 85
`
`

`

`U.S. Patent
`
`Sep. 13, 1994
`
`Sheet 16 of 73
`
`5,347,295
`
`;::::.::::::;::::::;:.::::::1...::::::::::::::::::::::: ... ::.::.:::.:.::::: ....... ::::.:.::. .... ;:;::::::::;: .. u . . :;::oo:::::::::::::::::::::: .................. · .... ·.····· .. ········:··
`
`--6-
`
`520
`
`Application
`Document Edit Insert case Format
`object visible on the display. Gestures with a stylus offer .. ff"
`a more concise vocabulary than tradition pointinA
`11
`devices because they combine both selection (by their
`\I
`location) and mecninA (the specific Aesture) in one
`Ii
`movement. A common set of Qestures can delete,
`!!
`move, copy, or edit diverse objects such as text
`11
`il
`fraRments, buttons, doruments, or entire notebooks.
`II
`Gestures can initiate the insertion of hand-written te><t,
`11 o
`the editinR of existinR te>Ct, paAe-tuminA, scrollinR within
`H g
`a window, zoominA or shrinkinA of entire paAes,
`11 ~
`selection, summoninA of windows conteininA operator
`AUidance, or the creation of links c•Aoto• buttons) to
`11 ~
`II ~
`other paQes.
`The proximity of the stylus to the display's front
`" 5F
`!l v
`surface is used to anticipate and terminate stylus input
`events. By sensinA both the proximity of the stylus tip to, 11 ~-
`ll ~
`and contact with, the display sui:face the user~interface
`ii
`software can more accurately discern the vertical
`l<'
`II
`movement of the stylus, provide a richer vocabulary of
`ii 3
`stylus movements for control of the computer, and offer
`better feedback to the user.
`11 ~
`:: "'
`
`(.l
`
`470
`
`When usinR the stylus for selection, movements
`into proximity can triAAer display events which qive the
`user a preview of what display objects are understood to ,,
`be taf'Aeted by the computer. For example the computer H
`can expand or alter the appearance of a button in
`H
`anticipation of its selection. This solution overcomes the ii
`problems of selectinA displayed objects due to tiqht
`\l
`Ii
`spacinA and parallax effects in prior art.
`In the entry of multi-stroke A"'-Phical objects the
`\l
`sensinR of the departure of the stylus tip from the vicinity II
`.
`t
`.
`•::·.:-·
`
`~D~
`O!l~~~~
`
`Fig. 16
`
`GOOGLE EXHIBIT 1017
`
`Page 17 of 85
`
`

`

`U.S. Patent
`
`Sep. 13, 1994
`
`Sheet 17 of 73
`
`5,347,295
`
`:::::::::::::::::::::::::::::::::;:::::;:u::::::::::::::::::::::::::::::::::::::::::::::::z::~::~.:. .. ::z::.:::::::::::::::::::::::::::::::::::::;;:; ............ mom•n••Ho•o••
`
`--6-
`
`510
`530
`
`Application
`Document Edit Insert case Format
`·r
`The present invention desaibes methods and
`,,
`apparatus for a stylus-driven computer. Said computer
`presents the behavior of a multi-paRe notebook includinJ:1 ii
`visible paRes, PaRe-tum effects, tabbed bookmarks
`U
`alonR one or more edAe, 'floatinR1 paAes temporarily
`11
`ii
`removed from their notebook location, a
`table-of-contents, and contents of sub-sections. Each
`11
`paJ:1e in the notebook contains one document which may !!
`1! n
`contain many physical paRes when printed. These
`ii g
`paAes are analOAOUs to data files in other computers
`ii ~
`and may contain different types of documents such as
`te><t, drawinRS, FAX imaRes, or specialized forms. Some !! ~
`information (such as the 'Help Information") is contained ii .,,
`11 ~
`in separate notebooks with all of the features listed
`above. The primary means of control of the computer is
`11
`a rich set of Restures with the
`us over almos
`;;
`object visible on the display. Gestures w1 a stylus offer Ii
`a more concise vocabulary than tra it1on pointimi
`;;
`II
`devices because they combine both selection (by their
`location) and meaninA (the specific Aesture) in one
`movement. A common set of Restures can delete,
`move, copy, or edit diverse objects such as te><t
`fraAments, buttons, documents, or entire notebooks.
`Gestures can initiate the insertion of hand-written text,
`the editinA of e>eistinA te>et, paAe-tuminA, scrollinA within
`a window, zoominA or shrinkinA of entire pa~es,
`selection, summoninA of windows containin~ operator
`AUidance, or the creation of links c•Aoto• buttons) to
`
`11
`H 3
`ll ~
`Ii
`'""
`II
`11
`11
`II
`!!
`
`other ~:::ximity of the stylus to the display's front I
`
`!!
`surface is used to anticipate and terminate stylus input
`events. By sensinA both the proximity of the stylus tip to, 11
`and contact with, the display surface the user-interface
`II
`l,.i.Qiaill"--1.A!a..liliM.~~~aD~
`.
`"·''.,·
`
`~ [i(srt:#iffiJ ~ D ~
`~ a tiitl®~~
`
`Fig. 17
`
`GOOGLE EXHIBIT 1017
`
`Page 18 of 85
`
`

`

`U.S. Patent
`
`Sep. 13, 1994
`
`Sheet 18 of 73
`
`5,347,295
`
`•••:,••••:,:,: •• ;;.;;:;.:,;; •• ., ••••• _ouu••••-•-.. •• .. • .. •• .. •-••••••--••:;;.;;;:,;;.; .... ,.. .......... u ............... ouu•• .. ••••••••••• .. •• .. ••••• .. ••••• .. •••••'••••••••••• • ••
`
`--6--
`
`Application
`Document Edit Insert case Format
`1·
`The present invention describes methods and
`apparatus for a stylus-driven computer. Said computer
`"
`presents the behavior of a multi-pa~e notebook inciudin~ II
`II
`visible paAes, paRe-tum effects, tabbed bookmarks
`ii
`alonR one or more edRe, 8floatinR• paRes temporarily
`ll
`removed from their notebook location, a
`II
`table-of-contents, and contents of sub-sections. Each
`paRe in the notebook contains one document which may 11
`\\ n
`contain many physical paRes when printed. These
`II g
`paAes are m1aloROUs to data files in other computers
`11 ~
`end may contain different types of documents such as
`text, drawinRS, FAX imaRes, or speciaJized forms. Some ii ~
`information (such as the •Help Information") is contained ii
`..,,
`in separate notebooks with aU of the features listed
`\)
`[
`above. The primary means of control of the computer is ii se
`ii y
`a rich set of Restures with th
`n
`ffer II ~-
`object visible_ on the display. Gestures with a st lus
`ii :,
`a more conase vocabulary t an tra it1on pomtinA
`devices because they combine both selection (by their
`··
`location) and meeninR (the specific Aesture) in one
`" l{'
`movement. A common set of Restures can delete,
`" 3
`1.1. ~
`move, copy, or edit diverse objects such as text
`fraAments, buttons, documents, or entire notebooks.
`Gestures can initiate the insertion of hand-written text,
`the editinR of existinA text, paAe-tuminA, scrollinA within
`a window, zoominA or shrinkinq of entire paQes,
`selection, summoninR of windows containinA operator
`AUidence, or the aeation of links c•Aoto• buttons) to
`other paAes.
`
`540
`
`V"t
`
`The proximity of the stylus to the display's front
`\\
`surface is used to anticipate and terminate stylus input
`events. By sensinR both the proximity of the stylus tip to, \\
`Ii
`and contact with, the display surface the user-interface
`~-''.:-·
`~.w.lm.-'iillla■l~a.Al~illlal:~
`
`~ [FJ(s'j,ti~~si ~ a ~
`~ [Jtl~~~
`
`Fig. 18
`
`GOOGLE EXHIBIT 1017
`
`Page 19 of 85
`
`

`

`-~~~,,...,··········"'"'""'"""'""""""""'''"'''''"""''"''''""""'''""''"'""'"""""'""""""'"'""""'""""""""'•""'"'"'"''"""""'"""'"
`
`:::::::;:;:::;;;;;:;::: ........................................................... uoo::ooo•••••••••••••••••••u•o•••••U•••••••••,.•·
`
`11
`
`U.S. Patent
`
`Sep. 13, 1994
`
`Sheet 19 of 73
`
`5,347,295
`
`....- 6 ~
`
`Application
`.................. ~-
`Edit Insert case Format
`l
`545 = About ...
`nt invention desaibes methods and
`"
`stylus-driven computer. Said computer
`Print ...
`havior of a multi-paAe notebook includinA II
`Print setup"'
`II
`Ae-tum effects, tabbed bookmarks
`re edAe, 9floatini:t' paAes temporarily
`send ...
`II
`eir notebook location, a
`ll
`Ex~rt ...
`s, and contents of sub-sections. Each
`book contains one document which may 11
`Revert
`11 n
`ysical paAes when printed. These
`ii g
`ROUS to data files in other computers
`.
`11 ~
`and may contain different types of documents such as
`text, drawinAs, FAX imaAes, or specialized forms. Some II c:(cid:173)
`information (such as the 1Help Information") is contained ··
`.,,
`in separate notebooks with all of the features listed
`[
`above. The primary means of control of the computer is .. SP
`a rich set of Aestures with the stylus over almost any
`11 y
`obiect visible on the display. Gestures with a stylus offer Ii ~-
`II ~
`a more concise vocabulary than tradition pointinA
`ii B
`devices because they combine both selection (by their
`location) and meaninA (the specific Aesture) in one
`11 ~
`II 3
`movement. A common set of Aestures can delete,
`move, copy, or edit diverse obiects such as text
`ll ~
`ii ""
`fraAments, buttons, documents, or entire notebooks.
`II
`Gestures can initiate the insertion of hand-written text,
`II
`the editinA of existinA text, paAe-tuminA, scrollinA within
`l!
`a window, zoominA or shrinkinA of entire paQes,
`II
`selection, summoninA of windows containinA operator
`Quidance, or the creation of links c•Aoto• buttons) to
`\\
`II
`other paAes.
`·
`
`The proximity of the stylus to the display's front
`surface is u