(12> Ulllted States Patent
`Amro et al.
`
`US006278443B1
`(10) Patent No.:
`US 6,278,443 B1
`(45) Date of Patent:
`Aug. 21, 2001
`
`(54) TOUCH SCREEN WITH RANDOM FINGER
`PLACEMENT AND ROLLING ()N SCREEN
`T0 CONTROL THE MOVEMENT 0F
`_
`INFORMATION ON SCREEN
`
`7/1994 Logan et a1. ....................... .. 345/157
`5,327,161
`12/1994 Mikan
`.. 345/157
`5,376,946
`5,402,151 * 3/1995 DuWaer ............. ..
`.. 345/173
`5,406,307 * 4/1995 Hirayama et al.
`345/120
`5,432,531
`7/1995 Calder et al.
`345/173
`
`(75) Inventors: Hatim Yousef Amro, Austin; John
`_
`Paul D°ds°n> P?ugemne> both of TX
`(Us)
`
`345/173
`* 15/1995 Clark et a1‘ "
`345/173
`/1998 Allard et al. .
`,
`,
`345/173
`5,956,019 * 9/1999 Bang et a1. ..
`5,982,302 * 11/1999 Ure ....................................... .. 345/22
`
`(73) Assignee: International Business Machines
`Cor oration, Armonk, NY US
`p
`(
`)
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 0 days.
`
`( * ) Notice:
`
`(21) Appl. No.: 09/070,180
`
`Apr- 30’ 1998
`(22) Filed:
`(51) Int. c1.7 ..................................................... .. G09G 5/00
`(52) us. c1. ..................... .. 345/173; 345/157; 178/1801;
`178/18 05 178/18 06
`345/156 157
`
`(58) Field of Search
`
`_
`345/173’ 159’ 160’ 17281331186’ 118
`
`(56)
`
`References Clted
`US. PATENT DOCUMENTS
`
`4,550,221 * 10/1985 Mabusth ...... ..
`4,684,801 * 8/1987 Carroll et al.
`..
`4,736,191
`4/1988 MatZke et al.
`4,977,397
`12/1990 Kuo et al. .......................... .. 340/710
`
`* cited by examiner
`_
`_
`Primary Examiner—B1pm ShalWala
`Assistant Examiner—Mansou_r M- Said
`(74) Attorney, Agent, or Ftrm—Lesl1e Van LeeuWen; J
`Bruce schelkopf
`(57)
`
`ABSTRACT
`
`Acomputer controlled display system With a user interactive
`touch screen is provided With an on-screen mouse to Which
`user input may be aPP1i‘°jd by rolling of the/touch ?nger I0
`thereby move dlsplayed mformanoni the P0Inter or scrolled
`information on the screen. Means are provided Which are
`activated by the touching of the screen at any random
`position selected by the user for enabling the detection of
`any rolling of said placed ?ngertip in an orthogonal direc
`tion. Also provided are means responsive to the detection of
`said rolling of said placed ?ngertip for moving said dis
`played data in an orthogonal direction corresponding to the
`direction of said rolling. The data moved may be the cursor
`or pointer or, When scrolling, the Whole screen of data may
`be moved
`'
`
`21 Claims, 9 Drawing Sheets
`
`Location; IhlIp/Iwww mde state m uslschoollprofdev. html
`I What's New I What's Cool I Handbook I Net Search I Net DirectoryI Software_I
`
`N
`
`Michigan department of Education
`
`Main Menu
`
`Qffi?s
`
`MDE Gopher Phone Directog
`
`Professional Development Resources
`1? \-50d
`
`53
`
`1. Professio I'Ed-ucational rganizatio s
`[Michigan In‘, \
`I] National
`)(
`\kka'
`2. Educational Technology Institu}n/\ l?/A'
`3. Listservs & Discussion Lists
`4. News Groups
`5. Distance Education
`50 \
`56
`6. Institutes 8: Centers
`> A
`7. Michigan Calendar of Events
`TO MOVE POINTER,’EOLI. FINGER IN
`‘
`MOVEMENT DIRECTION ON MOUSE PAD
`
`\
`\
`\
`
`\ In,
`
`[I Professional Educational Organizations
`
`III III
`
`http://wwwmde state mi uslschoollprofdev. hh'nl
`
`X
`
`C |:] ?
`
`Microsoft Ex. 1016
`Microsoft v. Philips - IPR2018-00026
`Page 1 of 14
`
`

`

`1
`
`,0or
`
`DDF4*:1,«Fmmqum<o<
`
`ow><4mmHa><41mHo
`
`US. Patent
`
`10
`
`h
`
`f01
`
`Sno
`
`:mpm>mumzomh<qu===ouo\HA-
`
`
`
`mIn
`
`umpm<o<,mmnm[v>mozm£Oman—Ugmmhm<a<
`
`muH>mo
`
`mmh¢<a<
`
`9a¢<om>mx
`
`FFFm
`
`
`
`quFzHomIn
`
`mmhm<o<
`
`SN_
`
`m_‘9H“.
`
`mm
`
`onh<UH4mm<
`
`Hm:mpm»m
`
`w2Hp<mmmo
`
`Microsoft Ex. 1016
`
`Microsoft v. Philips -
`
`|PR2018—00026
`Page 2 of 14
`
`Microsoft Ex. 1016
`Microsoft v. Philips - IPR2018-00026
`Page 2 of 14
`
`
`
`
`
`

`

`US. Patent
`
`Aug. 21, 2001
`
`Sheet 2 0f 9
`
`US 6,278,443 B1
`
`mm
`
`
`
`
`
`328528macmoomo:gEms.gal:
`
`
`
`:oemoanmBEmEtmqmn$95.5
`
`
`
`
`
`mouhsomoz“cacao—gunE55335
`
`
`
`
`
`gasogemz5.82%€23ch503?;E33;—_EE52996233:_EmEmmuEézzsefi
`
`
`
`
`
`
`3:938:3—.Nommum:=288mEw9.53%..mmm«3.5:—zwflozzooh
`
`
`
`
`
`
`
` .2282D5522.2DEcumfizmfioHideous—om.3868.55.—
`
`
`
`
`
`
`
`
`
`
`
`
`
`cosmos—um853%:.m
`
`3.5.5952.v
`
`N.0.“—
`
`EE3205:8582E2%gagiiz
`
`
`
`
`
`8:93.moEVE—munay—6:2.5
`
`£356a8253:..o
`
`Microsoft Ex. 1016
`
`Microsoft v. Philips -
`
`|PR2018—00026
`Page 3 of 14
`
`Microsoft Ex. 1016
`Microsoft v. Philips - IPR2018-00026
`Page 3 of 14
`
`
`
`
`
`
`
`
`

`

`US. Patent
`
`Aug. 21, 2001
`
`Sheet 3 0f 9
`
`US 6,278,443 B1
`
`
`
`
`
`E50292@825€83ch503%;$2pan;—E;55.85%:E2%85:23.2
`
`
`mquOmoz“Sacco—gen_m=o_mm£o._n_
`
`
`
`Hosea89E.mzmoowas.g:82£32..
`
`
`
`noemoanmBEastmqmb$95.5
`
`
`
`
`
`
`
` 6:0szU$955D2.233.830_m=o.nmu=_om.mzommmfloi.—
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`awe—ozsuoh35.58::.Nma:5:835aEvan:.mmm8:.5:.
`
`
`
`
`
`Z.MuUZE.50.Jar—2.0mm>OZOh.
`
`
`
`
`
`D<n_”.520220ZOEUm—EDPZm—ZmSO—z
`
`
`
`:5;50202853?E22”gaggiz
`
`
`
`
`
`mEo>mhohaw—5.5=mw___u_2.5
`
`
`
`558:33535.m
`
`3:20932.v
`
`
`
`28:8a8:23.:.w
`
`Microsoft Ex. 1016
`
`Microsoft v. Philips -
`
`|PR2018—00026
`Page 4 of 14
`
`Microsoft Ex. 1016
`Microsoft v. Philips - IPR2018-00026
`Page 4 of 14
`
`

`

`US. Patent
`
`1,2
`
`26,SU
`
`1B3M
`
`M$9.65Dm.mzowmflzmwhoaccuses—om—_m=o_mm£oi.—
`
`
`
`
`329:.qu35.28%.”—.N.mSmm,m83%:.mM533:38.53.5.mM3:95352éM£5=o._mm=umEa9.533
`
`
`
`
`
`
`.n923:5a£53me
`mm,:5:.>mEe&_8;oma=_E23wmuevazéfifizU_U
`
`
`m89:5sz“Essa—Egon_m:o_mm£o.5
`
`
`
`E58562505832682%_ooow..2>>262£2;—E;$995232E2%was3:32.;
`
`
`
`
`
`
`
`
`
`
`
`
`22.3233EmEtmqmb2353
`
`
`
`a28...5glozuEzmoowas.magmalace::Ifii:
`
`
`
`
`
`n3.mmDOS.202955—5F2m2m>02
`
`
`
`
`
`
`
`Z.552—;‘50.Jar—2.0..m>O2C...
`
`3:0szU
`
`
`
`855mLBEns—mucmez—z
`
`Microsoft Ex. 1016
`
`Microsoft v. Philips -
`
`|PR2018—00026
`Page 5 of 14
`
`Microsoft Ex. 1016
`Microsoft v. Philips - IPR2018-00026
`Page 5 of 14
`
`

`

`US. Patent
`
`Aug. 21, 2001
`
`Sheet 5 0f 9
`
`US 6,278,443 B1
`
`532853:E2%8E52%.
`
`
`gob—83¢“cacao—26m.12.23035
`@5025?€222€895:58322,35%;;—E;
`
`
`
`
`
`
`
`
`
`aofilogomzoiEcmoowas.$253:22.:lfi:
`
`
`
`8:833.6Emstmqmu$953
`
`now
`
`533:385:5.m6)3:20362.v
`\AWom3m:559355a3.3::.m
`
`
`
`
`
`[you—5.5m:—zwfioczvoh3.53833.NkmmficozmzD5953B
`
`
`
`28:3.w8255:..w
`
`
`
`
`
`35$he3.2.0.553:82.5
`
`
`
`acct—3.5303:95.3353:23.958;.F
`
`
`
`
`
`
`
`
`
`
`
`2—mmUZE‘30..mthRumMSG—2Oh.
`
`
`
`D<mmmDOEZOZOEUMEDPZmS—msO—Z
`
`
`
`:5:o>muB._&_oocow\m:_E22mouE§§=HBEDD
`
`Microsoft Ex. 1016
`
`Microsoft v. Philips -
`
`|PR2018—00026
`Page 6 of 14
`
`Microsoft Ex. 1016
`Microsoft v. Philips - IPR2018-00026
`Page 6 of 14
`
`

`

`US. Patent
`
`A
`
`m,
`
`1
`
`B.umor.m,:5:5523.853?E2%”viii;DD
`
`1a“madammm__o_um~_=aw_o3:323:3—_m=o_mm£oi
`
`
`
`cosmsumfiw9.3::.mS-.
`m2235:.“quEcon—8.6m.3:23.93?—E
`
`
`.—mbmUUu=OmUMuEQEn—O_0>ODEzcmmmflchn—
`
`
`928:5a$33.55.wM5.38:38555.m63.5.5932.vm.5:
`
`
`
`
` $953a58552223202x082?.8032;252£2;—2558.92828?E2%85%;:
`
`
`
`
`
`
`
`
`
`238:3u5EmEtmqmn
`
`
`
`
`
`
`
`Hosea95:1.2300we:mmoEosac-o::._m.._2.
`
`
`
`
`
`3:93—moSUE—mo53:22.N.
`
`Microsoft Ex. 1016
`
`Microsoft v. Philips -
`
`|PR2018—00026
`Page 7 of 14
`
`Microsoft Ex. 1016
`Microsoft v. Philips - IPR2018-00026
`Page 7 of 14
`
`

`

`U.S. Patent
`
`Aug. 21, 2001
`
`Sheet 7 0f 9
`
`US 6,278,443 B1
`
`FIG. 7
`
`70
`ON TOUCH PANEL, SET UP ROUTINE A/
`FOR DISPLAYING FINGER MOUSE PAD
`IN RESPONSE TO FINGER TOUCH
`
`L
`
`SET ROUTINE RESPONSIVE TO SENSED FINGER
`71
`ROLLING FROM MOUSE PAD AREA FOR CALCULATING
`THE VECTOR OF FINGER MOVEMENT, N
`THE DIRECTION OF MOVEMENT AND TRAVERSED
`AREA OF TOUCH SENSITIVE PANEL PIXELS
`
`I
`
`72
`SET UP ROUTINE FOR MOVING CURSOR ON DISPLAY
`SCREEN FOR VECTOR KV WHERE V IS THE N
`VECTOR OF THE INCREMENT OF FINGER
`MOVEMENT ON TOUCH PANEL AND K
`IS A SELECTED CONSTANT SETTING
`CORRESPONDING CURSOR MOVEMENT
`
`I
`
`73
`SET UP ROUTINE FOR SCROLLING ALL DISPLAYED
`INFORMATION FOR A VECTOR CORRESPONDING /\/
`TO THE VECTOR OF FINGER MOVEMENT
`INCREMENT ON TOUCH PANEL
`74
`i
`CREATE ROUTINE FOR REMOVING MOUSE /\/
`PAD FROM DISPLAY AFTER COMPLETION
`OF INFORMATION MOVEMENT
`
`Microsoft Ex. 1016
`Microsoft v. Philips - IPR2018-00026
`Page 8 of 14
`
`

`

`U.S. Patent
`
`Aug. 21, 2001
`
`Sheet 8 0f 9
`
`US 6,278,443 B1
`
`80
`
`TOUCH PANEL
`PRESSED FOR
`MOUSE PAD?
`
`NO
`
`YES
`
`END
`
`SET UP MOUSE PAD
`IN PRESSED REGION
`
`81
`
`NO
`
`83
`
`YES
`—Y
`
`84
`
`CALCULATE V
`
`V
`85
`CALCULATE KV
`(
`V
`
`FIG. 8
`
`MOVE CURSOR KV
`86
`(
`ON DISPLAYSCREEN
`
`87
`
`88
`
`V
`
`NO
`
`REMOVE MOUSE PAD
`+ YES
`Q)
`
`Microsoft Ex. 1016
`Microsoft v. Philips - IPR2018-00026
`Page 9 of 14
`
`

`

`US. Patent
`
`n,
`
`Eco—“83m:o_mm£o.£.sI*\I_\<:mmEoSDm.mBEE—Ew—.N*/=28:szm\Cx/
`
`
`
`
`
`
`
`IESE—329.5%REDS—.3Exam—.35.w9.53%...mSAuxl
`.wmIHI8m:3:565.mw3:20asuz923:09amug—5.5m....mMzeta—.3
`
`
`
`
`mmuflsomom“cacao—gon—_m=o_mm£o.5
`
`
`1wmgBmR%Hm0—"—mP4M_mm,
`
`:5:52236283:_E23»ocEiEsESEDD.WPV2,m6PS.Um
`35:32:55
` cause:E220255252822xosucmz5822;253.2;—_EE,écazoocum‘ws_Emam—m
`
`
`
`
`
`
`
`
`DEmmDO—z202055—5him—Ego:m2—mmUZE.50.£5.25;m>CEOh.nlv
`
`
`$.53me26:25cumin:.5M
`
`
`
`
`
`
`
`22.6023koEmstmqmb
`
`
`
`969.1080:1.2300we:wluumgoIncas.Ems.
`
`0
`
`66412100:l10Oa.mm1e
`
`Microsoft Ex. 1016
`Microsoft v. Philips - IPR2018-00026
`Page 10 of 14
`
`

`

`US 6,278,443 B1
`
`1
`TOUCH SCREEN WITH RANDOM FINGER
`PLACEMENT AND ROLLING ON SCREEN
`TO CONTROL THE MOVEMENT OF
`INFORMATION ON-SCREEN
`
`TECHNICAL FIELD
`
`The present invention relates to user interactive computer
`supported display technology and particularly to such user
`interactive systems and methods Which are user friendly, i.e.
`provide even non-computer-literate users With an interface
`environment Which is easy to use and intuitive. More
`particularly, the present invention relates to touch screen
`technology, especially user intuitive touch screens through
`Which even novice computer users may access commercial,
`educational and entertainment information such as that
`Which may be accessed via the Internet.
`
`10
`
`15
`
`BACKGROUND OF RELATED ART
`
`The 1990’s decade has been marked by a technological
`revolution driven by the convergence of the data processing
`industry With the consumer electronics and communications
`industries. This advance has been even further accelerated
`by the eXtensive consumer and business involvement in the
`internet over the past feW years. As a result of these changes,
`it seems as if virtually all aspects of human endeavor in the
`industrialiZed World requires human computer interfaces.
`There is a need to make computer directed activities acces
`sible to a substantial portion of the World’s population
`Which, up to a feW years ago, Was computer illiterate, or at
`best computer indifferent. In order for the vast computer
`supported market places to continue and be commercially
`productive, it Will be necessary for a large segment of
`computer indifferent consumers to be involved in computer
`interfaces. Thus, the challenge of our technology is to create
`interfaces to computers Which are as close to the real World
`as possible.
`In this connection, there is the touch screen or touch
`panel. It is easy to use because it alloWs the user to point
`directly With his ?nger to make selections Which move a
`cursor or other element around a display screen. The touch
`panel in various forms has been in use for several years.
`Several different technologies have been involved in touch
`panels. Original touch panels used a series of infrared LEDs
`and light sensors, such as photodiodes, to provide loW
`resolution panels of up to 50 resolvable positions. The LEDs
`and sensors form a grid of invisible light beams Which the
`?nger breaks to, thus, indicate its position. The capacitively
`coupled touch panels Were able to develop a resolution of
`about 100 resolvable positions. Higher resolution touch
`screens have been developed using a variety of technologies
`from sound Waves re?ected off ?ngers to conductive/
`resistive layers separated by insulative material broken
`doWn by touch.
`While direct touch screens are simple to use, they do have
`some eye/hand coordination shortcomings. First, While the
`?nger is a natural pointer, because of its relatively Wide
`dimensions and difficulty in making precise movements in
`small increments, it has limitations in making movements
`and selections in high information density/high resolution
`screens. In this connection, the ?nger and like direct pointers
`are also hampered by parallax: if the conventional touch
`panel is 1/4“ to 1/2“ from the display, then the user’s ?ngers
`tend to touch the position on the panel that is aligned With
`their eyes and the selected point on the display screen and
`not the position on the touch panel directly perpendicular to
`the selected point on the display screen.
`
`25
`
`35
`
`45
`
`55
`
`65
`
`2
`SUMMARY OF THE INVENTION
`
`The present invention provides a solution to the above
`mentioned shortcomings of direct ?nger access touch
`screens While still preserving the ease-of-use and naturalness
`of such touch screens. The invention provides a user inter
`active touch screen in a computer controlled display system
`Which includes means for displaying data on said touch
`screen, means for sensing the placement of a user’s ?ngertip
`adjacent said screen, means responsive to said sensing
`means for enabling the detection of any rolling of said
`placed ?ngertip in an orthogonal direction and means acti
`vated by a detection of said rolling of said placed ?ngertip
`for moving said displayed data in an orthogonal direction
`corresponding to the direction of said rolling. In the most
`signi?cant embodiment, the displayed data Which is moved
`is the cursor. Also, the ?ngertip is preferably placed in a
`position remote from the cursor. Thus, the ?nger does not
`cover the cursor, Which may be small enough to make
`selections requiring small increments of movement in rela
`tively high information density screens. We have found that
`the rolling of a ?nger can precisely control the remote cursor
`movement. Eye/hand coordination to more precisely posi
`tion cursors is more easily done With ?nger rolling, Which
`can be ?ne tuned With the visual feedback of the resulting
`remote cursor movement. Also, parallaX is no problem
`because the pointer, i.e. the cursor is on the display screen
`rather than on the superimposed touch panel, and, thus, the
`thickness of the touch panel does not matter.
`For best results, When the placement of a ?nger is sensed,
`there are means responsive to the sensing means for dis
`playing an object indicating that said means for detecting the
`?nger rolling is enabled. These indicating means may simply
`be a pad image enclosing the ?ngertip. The screen may be
`a pressure sensitive screen, in Which case it may further
`include means responsive to a subsequent pressing of said
`placed ?ngertip for additional interactive user input.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`FIG. 1 is a block diagram of a data processing system
`including a central processing unit and touch panel display
`Which is capable of implementing the present invention;
`FIG. 2 is a diagrammatic vieW of a portion of a display
`screen on Which information interactively accessible
`through a cursor is displayed together With a transparent
`superimposed touch panel;
`FIG. 3 is the display screen structure of FIG. 2 after the
`panel has been pressed to bring up a displayed mouse pad;
`FIG. 4 is the display screen structure of FIG. 3 shoWing
`hoW ?nger movement Within the mouse pad results in a
`corresponding movement of the cursor,
`FIG. 5 is the display screen structure of FIG. 4 shoWing
`hoW a second iterative ?nger movement Within the mouse
`pad results in a further sequential corresponding cursor
`movement;
`FIG. 6 is the display screen structure of FIG. 5 after the
`cursor movement of FIG. 5 has been completed and after an
`information object on the screen has been interactively
`selected;
`FIG. 7 is a ?oWchart shoWing the development of a
`program for controlling movement of information on the
`display screen structures of FIGS. 2 through 6 through the
`touch panel on-screen mouse of the present invention;
`FIG. 8 is a ?oWchart shoWing the running of the program,
`the development of Which has been described With respect to
`FIG. 7; and
`
`Microsoft Ex. 1016
`Microsoft v. Philips - IPR2018-00026
`Page 11 of 14
`
`

`

`US 6,278,443 B1
`
`3
`FIG. 9 is the display screen of FIG. 4, but showing hoW
`the mouse pad of the invention may be used to move the
`cursor to the destination item by a continuous movement of
`increments at a selected rate.
`
`4
`coordinates and the corresponding input data value to the
`bus architecture 12.
`The touch panels or screens 10 may use any of the
`standard technologies discussed above. In the present
`embodiment, higher resolution panels using resistive/
`conductive composites should provide best results. Such
`structures use tWo slightly separated layers of transparent
`material, one coated With a thin layer of conductive material
`and the other With resistive material. The pressure of the
`?ngertip forces the layers to touch and the voltage drop
`across the resistive substrate is measured and used to deter
`mine the coordinates of the touched positions. There are
`many such conductive/resistive touch screen displays on the
`market Which may be used in the implementation of the
`present invention such as the IBM 2489 Model 600 and PGI
`Super Nightingale.
`There Will noW be described a simple illustration of the
`present invention With respect to the display screens of
`FIGS. 2 through 6. When the screen images are described,
`it Will be understood that these may be rendered by storing
`image and teXt creation programs such as those in any
`conventional WindoW operating system in the RAM 32 of
`the system of FIG. 1. The operating system is diagrammati
`cally shoWn in FIG. 1 as operating system 41. The program
`of this invention for creating and controlling the on-screen
`?nger mouse is shoWn as application 40. Display screen
`images are presented to the vieWer on-screen 19 of display
`monitor 17 of FIG. 1. In accordance With the standard touch
`screen techniques described above, the user may control the
`screen interactively through ?nger 18 touching touch screen
`10 Which operates through pointing device adapter 11 and
`bus 12 to call upon the routines in application program 40
`Which is loaded in system RAM 32 cooperating With the
`operating system 41 to create the images display adapter 14
`to control the display screen 19 on display monitor 38.
`The initial display screen of FIG. 2 shoWs a display screen
`51 Which for purposes of this eXample Was received over the
`Internet. The user is interested in professional development
`and has received a page from the Michigan State Depart
`ment of Education Which is shoWn on-screen 51. The cursor
`or pointer 50 is at the position shoWn. The user is interested
`in activating link 53 for further information on Educational
`Technology Institutes. Thus, he needs to move pointer 50 to
`link 53. Instruction 52 advises the user to press the screen for
`the mouse pad. In accordance With the present invention, the
`user may randomly press any point on-screen 51 With his
`?nger 18. He selects an area as shoWn Which does not
`contain any data. On-screen ?nger mouse pad 54 appears,
`FIG. 3, together With on-screen instruction 56 Which advises
`the user to roll his ?nger on the mouse pad 54 in the direction
`in Which he Wishes to move the pointer 50. It should be
`noted that the mouse pad 54 and the instruction 56 are on the
`screen 19 of display 17 of FIG. 1 together With cursor 50, as
`Well as link 53 and all of the other data. Touch screen 10 is,
`of course, transparent. As Will hereinafter be seen in greater
`detail, When the program is described, When the touch screen
`senses the touching of the screen by ?nger 18 Which is
`shoWn centered at point 55, the information is conveyed via
`adapter 11 and bus 12 to the system memory on Which the
`present program is loaded and the program sends appropri
`ate commands via display adapter 14 to provide on-screen
`mouse pad 54 around center point 55 Where the ?nger
`touched and provides instruction 56 to roll ?nger. The user
`folloWs this instruction, FIG. 4, in the direction toWard link
`53 Which he Wishes to move pointer 50. As ?nger 18 rolls,
`the portion of the tip of ?nger 18 Which is in contact With the
`screen 51 moves for a distance and direction de?ned by
`
`DETAILED DESCRIPTION OF THE
`PREFERRED EMBODIMENT
`Referring to FIG. 1, a data processing system is shoWn
`Which may function as the computer controlled display
`terminal With interactive touch screen used in implementing
`the on-screen randomly placeable ?nger mouse functions of
`the present invention. A central processing unit (CPU) 30
`such as one of the PC microprocessors available from
`International Business Machines Corporation is provided
`and interconnected to various other components by system
`bus 12. An operating system 41 runs on CPU 30 and
`provides control and is used to coordinate the functions of
`the various components of FIG. 1. Operating system 41 may
`be one of the commercially available operating systems such
`as the OS/2 operating system available from International
`Business Machines Corporation (OS/2 is a trademark of
`International Business Machines Corporation); Microsoft
`WindoWs 95TM or WindoWs NTTM, as Well as UNIX or AIX
`operating systems. Aprogramming application for operating
`the on-screen mouse interface of the present invention,
`application 40, to be subsequently described in detail, runs
`in conjunction With operating system 41 and provides output
`calls to the operating system 41 Which implement the
`various functions to be performed by the application 40. A
`read only memory (ROM) 31 is connected to CPU 30 via
`bus 12 and includes the basic input/output system (BIOS)
`that controls the basic computer functions. Random access
`memory (RAM) system 32, U0 adapter 16 and communi
`cations adapter 13 are also interconnected to system bus 12.
`It should be noted that softWare components, including the
`operating system 41 and the application 40, are loaded into
`memory system 32 Which is the computer system’s main
`memory. I/O adapter 16 conventionally communicates With
`the disk storage device 15, ie a hard drive. Communica
`tions adapter 34 interconnects bus 12 With outside netWorks
`such as the internet enabling the data processing system to
`communicate With other such systems. Keyboard 20 is
`connected to system bus 12 via keyboard adapter 21. The
`present invention is embodied in the softWare program and
`any conventional touch screen display may be used.
`Typically, in FIG. 1, there is a display 17 having surface 19
`upon Which the visual output from the computer is generated
`via display adapter 14. A touch sensitive display screen or
`panel 10 is superimposed upon display surface 19. This
`touch screen Which is about 1A“ to 1/2“ from surface 19 is
`responsive to a touch stimulus, e.g. ?nger 18, applied by the
`user to issue commands to the computer system. The touch
`screen 10 resolution is determined by digitiZing circuitry
`(not shoWn) in a pointing device adapter 11 to form a
`tWo-dimensional array of discrete coordinate points. Atouch
`stimulus applied to any of the coordinate points is detected
`by a sensor array (not shoWn) in the touch screen 10. The
`sensor array generates an analog signal responsive to the
`force imparted to the touch screen. This signal is digitiZed by
`a sampling A to D convertor circuit (not shoWn) in touch
`screen 10 to produce an input data value. This data value,
`together With the coordinates to Which it relates, are trans
`mitted from touch screen 10 to pointing device adapter 11.
`The input data value corresponding to each set of coordi
`nates is conventionally refreshed by the A to D converter
`circuit about 60 times a second. The pointing device adapter
`11 connected to the bus architecture 12 passes each set of
`
`10
`
`15
`
`25
`
`35
`
`45
`
`55
`
`65
`
`Microsoft Ex. 1016
`Microsoft v. Philips - IPR2018-00026
`Page 12 of 14
`
`

`

`US 6,278,443 B1
`
`10
`
`15
`
`25
`
`35
`
`5
`dashed lines 57 and 57a, i.e. these dashed lines de?ne the
`area in or adjacent to pad 54 Which ?ngertip 18 Will contact
`in its roll. This is, in effect, a short stroke across all of the
`touch sensitive pixels betWeen lines 57 and 57a. From this
`sensed information, the distance and direction of vector V
`may be calculated. As a result of this ?nger roll or stroke,
`pointer 50 Will be moved for a vector KV as shoWn to
`pointer position 50a. It should be noted that K is a constant
`preselected by the user to move the pointer for a convenient
`distance in the same direction and in proportion to the
`shorter ?nger roll or stroke. The constant K is adjustable by
`the user dependent upon the on-screen data need and dis
`tances.
`NoW With respect to FIG. 5, the cursor is noW moved to
`the neXt position as indicated in FIG. 4 but it still has not
`reached link 53. Another stroke or roll of ?nger 18 is needed.
`The ?nger 18 is noW rolled for a distance and direction
`de?ned by dashed lines 60 and 60a, i.e. these dashed lines
`de?ne the area in or adjacent to pad 54 Which ?ngertip 18
`Will contact in its roll Which, in effect, is a short stroke across
`all of the touch sensitive piXels betWeen lines 60 and 60a.
`From this sensed information, the distance and direction of
`vector V1 may be calculated. As a result of this ?nger roll
`or stroke, pointer 50 Will be moved for a vector KV1, as
`shoWn, to pointer position 50b. It should be noted that K is
`the constant preselected by the user as set forth above. As a
`result of this second movement, pointer 50 has reached the
`desired link 53, as shoWn in FIG. 6. Link 53 may be turned
`on and, thus, highlighted as shoWn by the convenient means
`of clicking, e.g. clicking on center point 55 With ?nger 18.
`In this connection, the program routine may be conveniently
`set up to sense a click Which could be a ?nger jab after a
`?nger roll increment has been completed. The system Will
`then go on to access the subject matter (not shoWn) linked
`to selected link 53.
`For convenience in illustration, the pointer 53 has been
`shoWn as reaching the desired target after just tWo incre
`mental movements. The movement may involve many
`increments. The key is that short incremental movement,
`e.g. rolls or strokes of the randomly placed ?nger mouse
`direct the pointer or cursor to eventually reach its target. It
`should further be noted that While the illustration involved
`the movement of a pointer, any form of data could be moved
`in any direction using the on-screen ?nger mouse of the
`present invention. For eXample, in scrolling of data, the
`mouse pad could be randomly placed at any point on the
`screen and then, through ?nger rolling, the Whole screen of
`data could be scrolled in any direction. This last embodiment
`could be applicable to provide a touch screen for direction
`?nder displays involving maps Wherein the pad could be set
`up at any point and then scrolled in any desired direction.
`The above description used as an eXample the movement
`of the cursor in a plurality of variable increments to reach a
`selected screen element. Movement in such variable incre
`ments may be effectively used in many circumstances, eg
`when moving or scrolling in geographical location displays.
`HoWever, in the situation described above Where the target
`is in a substantially straight line direction from the initial
`cursor position, it Would be advantageous to move the cursor
`to the target item by a continuous movement of increments
`at a selected rate. FIG. 9 shoWs such movement. In the
`eXample of FIG. 9, the initial mouse pad/?nger position is
`the same as that of FIG. 4 and ?nger 18 is rolled through
`vector VX. After the roll, the operator continues to depress
`?nger 18. As long as ?nger 18 is depressed, the cursor 50
`continues to move in repetitive increments KVX Where K is
`a constant as described above until the cursor at position 50d
`
`45
`
`55
`
`65
`
`6
`has reached its destination item 53. At this point, the
`operator Will lift ?nger 18 from the pad to indicate that the
`cursor has reached its destination. It should be noted that the
`rate of movement of increments KVX may be constant, or
`the rate may be made variable dependent upon the distance
`that ?nger 18 has rolled from its starting center point 18. For
`short rolls, the rate of movement may be sloWer so that the
`cursor movements may be ?nely tuned, While for longer
`?nger rolls the rate may be greater so that longer distances
`may be covered Within time frames.
`NoW With reference to FIG. 7, We Will describe a process
`implemented by a program according to the present inven
`tion for interactively generating an on-screen ?nger mouse
`pad and for controlling the movement of on-screen infor
`mation in response to ?nger rolling and strokes. The pro
`gram routines Which are created by the process of FIG. 7
`implement the operations described With respect to FIGS. 2
`through 6. In the ?oWchart of FIG. 7, a basic interface
`routine is set up, step 70, Whereby the operator may touch
`any convenient spot on the touch screen, and a ?nger mouse
`pad Will be set up as shoWn in FIG. 3. Then, step 71, a
`routine is set up Whereby the touch screen senses the touch
`screen piXels activated by a roll of the controlling ?nger in
`any direction and calculates the vector of ?nger movement
`from this information. This is the calculation of vector V,
`previously described With respect to FIG. 4. Then, step 72,
`a routine is set up for moving the cursor for the vector KV
`in response to this ?nger roll. This movement of the cursor
`or pointer is also shoWn and described With respect to FIG.
`4. The routines of steps 71 and 72 are, of course, repeated for
`each incremental ?nger roll or stroke until the desired
`pointer or cursor target destination is reached.
`Step 73 sets up a routine similar to that of step 72 but for
`the scrolling of a Whole body of screen information along
`any incremental vector resulting from the associated ?nger
`roll. Lastly, step 74, a routine is set up for removing the
`mouse pad from the display upon the completion of the
`pointer movement or scrolling movement as shoWn in FIG.
`6.
`NoW that the basic programs have been described and
`illustrated, there Will be described With respect to FIG. 8, a
`How of a simple operation shoWing hoW the program could
`be run to control a cursor or pointer movement. First, step
`80, a determination is made as to Whether the operator has
`touched the touch screen at any random point to set up a
`mouse pad. If No, a determination is made as to Whether the
`session is over, step 82. If Yes, then the routine and session
`are ended. If the decision in step 82 is No, then ?oW returns
`to step 80 and the system is set up to aWait the screen touch
`Which Will set up the mouse pad. When the decision from
`step 80 is Yes, i.e. a pad has been requested, a pad is set up,
`step 81, in the pressed region as shoWn in FIG. 3, and a
`determination is made, step 83, as to Whether there is any
`?nger rolling. If No, the Waiting continues. If Yes, then, step
`84, the vector V of the increment ?nger movement is
`calculated, FIG. 4. Then, step 85, the increment of pointer
`movement KV is calculated, FIG. 4, and, step 86, the pointer
`is moved for KV on-screen. At this point a determination is
`made, step 87, as to Whether the cursor or pointer has
`reached the end of its run, i.e. its target. If Yes, as in FIG. 6,
`the on-screen mouse pad is removed, step 88, and the How
`is returned via branch A to step 82 Where a determination is
`made as to Whether the session is over. If No, then the
`above-described procedure is continued until the end of the
`session.
`Although certain preferred embodiments have been
`shoWn and described, it Will be understood that many
`
`Microsoft Ex. 1016
`Microsoft v. Philips - IPR2018-00026
`Page 13 of 14
`
`

`

`US 6,278,443 B1
`
`7
`changes and modi?cations may be made therein Without
`departing from the scope and intent of the appended claims.
`What is claimed is:
`1. In a computer controlled display system With a user
`interactive touch display screen:
`means for displaying data on said screen,
`means for sensing the placement of a user’s ?ngertip
`adjacent said display screen,
`means activated by said sensing means for enabling the
`detection of any rolling of said placed ?ngertip in an
`orthogonal direction, and
`means responsive to the detection of said rolling of said
`placed ?ngertip for moving said displayed data in an
`orthogonal direction corresponding to the direction of
`said rolling.
`2. The computer controlled display system of claim 1
`Wherein said moved displayed data is a screen cursor.
`3. The computer controlled display system of claim 2
`Wherein said ?nger is placed at a display screen position
`remote from the position of said cursor on said display
`screen.
`4. The computer controlled display system of claim 1
`Wherein said moved displayed data is dat