I lllll llllllll Ill lllll lllll lllll lllll lllll 111111111111111111111111111111111
`US007450114B2
`
`c12) United States Patent
`Anwar
`
`(10) Patent No.:
`(45) Date of Patent:
`
`US 7,450,114 B2
`Nov. 11, 2008
`
`(54) USER INTERFACE SYSTEMS AND METHODS
`FOR MANIPULATING AND VIEWING
`DIGITAL DOCUMENTS
`
`(75)
`
`Inventor: Majid Anwar, Glasgow (GB)
`
`(73) Assignee: Picsel (Research) Limited, Glasgow
`(GB)
`
`( *) Notice:
`
`Subject to any disclaimer, the term ofthis
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 738 days.
`
`EP
`EP
`EP
`EP
`EP
`
`5,790,114 A
`
`8/ 1998 Geaghan et al.
`
`(Continued)
`
`FOREIGN PATENT DOCUMENTS
`
`0 438 194 A
`0 465 250
`0 479 496 A
`0 513 584 A
`529 121 A
`
`7 /1991
`111992
`4/1992
`1111992
`3/1993
`
`(Continued)
`
`OTHER PUBLICATIONS
`
`(21) Appl. No.: 09/835,458
`
`(22) Filed:
`
`Apr. 16, 2001
`
`(65)
`
`Prior Publication Data
`
`US 2002/0011990Al
`
`Jan.31,2002
`
`(30)
`
`Foreign Application Priority Data
`
`Blumberg, et. al .. , "Visual Realism and Interactivity for the Internet",
`Proceedings of IEEE Compcon, 1997, San Jose, Feb. 23-26, 1997,
`Los Alamitos, IEEE Comp. Soc. Press, Feb. 1997, pp. 269-273.
`
`(Continued)
`
`Primary Examiner-Sumati Lefkowitz
`Assistant Examiner-Srilakshmi K Kumar
`(74) Attorney, Agent, or Firm-Ropes & Gray LLP
`
`Apr. 14, 2000
`
`(GB)
`
`................................. 0009129.8
`
`(57)
`
`ABSTRACT
`
`(51)
`
`Int. Cl.
`G06F 31033
`(2006.01)
`G06F 31041
`(2006.01)
`(52) U.S. Cl. ..................... 345/179; 345/173; 178/18.01
`(58) Field of Classification Search ......... 345/156-184;
`178/18.01, 18.03, 19.01, 20.01
`See application file for complete search history.
`
`(56)
`
`References Cited
`
`Systems, including handheld computing devices that include
`system code stored within the memory and adapted to be
`executed by the processor. The system code can process an
`input byte stream that is representative of contents to be
`displayed on the touch sensitive display and can generate a
`content document file representative of an internal represen(cid:173)
`tation of the content. A tool document file may also be stored
`in the memory and may provide an internal representation of
`a document providing an image that is representative of a
`graphical tool. Associated with a tool document can be tool
`code that is capable of processing the content document file to
`create an internal representation of the contents that presents
`the content in a manner that achieves a display effect associ(cid:173)
`ated with the tool. The device will also include parsing code
`that processes the content document file, the tool document
`file, and the processed internal representation to generate a
`screen document for display on the touch sensitive display in
`a manner that portrays the display effect.
`
`5 Claims, 11 Drawing Sheets
`
`U.S. PATENT DOCUMENTS
`4,839,634 A * 6/1989 More et al.
`111992 Haff
`5,083,262 A
`5,369,735 A * 1111994 Thier et al.
`5,390,320 A
`2/1995 Smithline
`5,463,725 A * 10/1995 Henckel et al.
`5,530,865 A
`6/1996 Owens eta!.
`5,615,384 A
`3/1997 Allard et al.
`5,748,185 A
`5/1998 Stephan et al.
`5,754,348 A
`5/1998 Soohoo
`
`................. 345/173
`
`................. 345/423
`
`............. 715/776
`
`:50 Masnified thumbnail and main view
`
`26
`
`---------L--~44
`
`42
`
`Microsoft Ex. 1005
`Microsoft v. Philips - IPR2018-00025
`Page 1 of 21
`
`

`

`US 7,450,114 B2
`Page 2
`
`U.S. PATENT DOCUMENTS
`
`5,810,680 A
`5,867,166 A
`5,899,988 A
`5,909,207 A *
`5,910,805 A
`5,9ll,066 A
`5,945,979 A
`6,034,700 A
`6,121,960 A
`6,310,610 Bl*
`6,4ll,274 B2 *
`6,466,203 B2
`6,525,749 Bl *
`6,663,823 B2
`6,717,573 Bl*
`
`911998 Lobb et al.
`211999 Schick et al.
`511999 Depledge et al.
`6/1999 Ho ............................. 345/156
`611999 Haschart et al.
`6/1999 Atkinson et al.
`8/ 1999 Rutledge et al.
`3/2000 Nickell et al.
`912000 Carroll et al.
`10/2001 Beaton et al. ............... 345/173
`612002 Watanabe et al.
`........... 345/684
`10/2002 Van Ee
`212003 Moran et al. ................ 715/863
`12/2003 Kobayashi et al.
`412004 Shahoian et al ............. 345/161
`
`FOREIGN PATENT DOCUMENTS
`
`EP
`EP
`EP
`EP
`EP
`EP
`EP
`GB
`
`0 650 144 Al
`0 651 543
`0 753 832 A
`0764918 A
`0 860 769 A
`0 949 571 A2
`0 980 045
`2 313 277 A
`
`4/1995
`5/1995
`1/1997
`3/1997
`8/1998
`10/1999
`212000
`1111997
`
`WO
`WO
`WO
`WO
`WO
`WO
`WO
`
`WO 97/34240
`W0-97/36225
`WO 98/10356
`W098 37506 A
`W0-98/56151
`W09910840 A
`WOOO 10372 A
`
`9/1997
`10/1997
`3/1998
`8/1998
`12/1998
`3/1999
`3/2000
`
`OTHER PUBLICATIONS
`
`Rowe, J.H., "Metafiles and Computer Graphics", Computers and
`Graphics, Pergamon Press, Ltd. Oxford, Great Britain, vol. 10, No. 2,
`1986, pp. 103-106.
`Anon., "Electronic Magnifying Glasses," IBM Technical Disclosure
`Bulletin, vol. 37, No. 3, 353-54 (Mar. 1994).
`Bohmer, S. et al., "Word fur Windows 6.0", 1994, pp. 38-43, BHV(cid:173)
`Verlag, Korschenbroich, Germany.
`Office Action dated Jan. 23, 2008 U.S. Appl. No. 10/853,495.
`Final Office Action dated Feb. 27, 2008 for U.S. Appl. No.
`10/813,285.
`Preece et al. "Human Computer Interaction," pp. 2ll-236 and 714-
`715, ISBN 0-201-62769-8, Addison Wesley (1994).
`Kramer, A. "Classifying Two Dimensional Gestures in Interactive
`Systems," Proc. Oflntl. Gesture Workshop, 9/17-19/97, published as
`Lecture Notes In Artificial Intelligence vol. No. 1371, pp. 37-48,
`ISBN 3-540-64424-5, Springer-Verlag (1998).
`* cited by examiner
`
`Microsoft Ex. 1005
`Microsoft v. Philips - IPR2018-00025
`Page 2 of 21
`
`

`

`11
`I
`
`12
`/
`
`File Formati--Nativc Content ---1~
`Information
`
`11
`,----~/
`
`File Format
`Information -Native Content - - -1 . i
`
`8
`
`16
`,----~/
`
`Further processed
`e.g. compact reformat
`
`Internal Representation
`
`""' 14
`15
`
`Storage
`Local/Remote
`
`Parser/
`
`I
`18
`
`42
`
`22
`I
`
`Shape
`Processor
`
`44
`
`10
`
`N
`0
`0
`QO
`
`26
`D
`D
`D
`D
`D
`
`D
`
`Described in terms
`of generic library objects
`
`40
`
`30
`
`Figure 1
`
`Microsoft Ex. 1005
`Microsoft v. Philips - IPR2018-00025
`Page 3 of 21
`
`

`

`/46
`
`48
`
`50 Magnified thumbnail and main view
`
`26
`
`44
`
`42
`
`L-
`
`I
`
`Scaled thumbnail view
`generated from main
`document; therefore 'live'
`animations & magnification
`
`Figure 2
`
`~
`00
`•
`~
`~
`~
`
`~ = ~
`
`z 0
`~ ....
`....
`
`'"
`N
`0
`0
`QO
`
`('D
`('D
`
`1J1 =(cid:173)
`.....
`N
`0 .....
`....
`....
`
`d
`rJl
`-....l
`~
`tit
`'"=
`"'""
`"'"" ~ = N
`
`Microsoft Ex. 1005
`Microsoft v. Philips - IPR2018-00025
`Page 4 of 21
`
`

`

`Interaction still possible
`in thumbnail, main view,
`and magnification
`
`"bubble" thumbnails
`
`66
`
`64
`
`62
`
`60
`
`68
`
`70
`
`72
`
`26
`
`54
`
`44
`
`Figure 3
`
`Current view
`
`smaller views proportional
`to 'distance' from current page
`
`~
`00
`•
`~
`~
`~
`
`~ = ~
`
`z 0
`~ ....
`....
`
`'"
`N
`0
`0
`QO
`
`1J1 =(cid:173)
`.....
`
`('D
`('D
`
`(.H
`
`0 .....
`....
`....
`
`d
`rJl
`-....l
`~
`tit
`'"=
`"'""
`"'"" ~ = N
`
`Microsoft Ex. 1005
`Microsoft v. Philips - IPR2018-00025
`Page 5 of 21
`
`

`

`main view of map with 'trunk' roads
`
`48
`
`50
`
`.................... ~·························
`.....
`: .....
`
`.....
`....
`
`·······························f···················~···························
`
`potentially enlarged view
`and additional mapping
`information e.g. minor roads too
`
`Or. .. additional 'bubble' help information
`
`displayed; notelets, annotations etc •
`
`or
`
`~
`00
`•
`~
`~
`~
`
`~ = ~
`
`z 0
`~ ....
`.... .
`
`N
`0
`0
`QO
`
`1J1 =(cid:173)
`.....
`
`('D
`('D
`
`.i;...
`
`trapping
`information
`
`User interface
`information coloured
`
`Figure 4
`
`0 .....
`....
`....
`
`d
`rJl
`-....l
`~
`tit
`'"=
`"'""
`"'"" ~ = N
`
`Microsoft Ex. 1005
`Microsoft v. Philips - IPR2018-00025
`Page 6 of 21
`
`

`

`42
`
`'
`
`80
`
`/
`
`_,,. 26
`
`80
`
`84
`
`81
`
`I
`
`I
`
`I
`
`1-1 I
`•
`
`I~
`
`floating semi-transparent mler. .. shows
`scale adapting to zoom of underlying
`object -works with magnifier
`
`Figure 5
`
`floating semi-transparent
`query displays annotations
`for underlying objects
`
`Figure 6
`
`82
`
`e •
`
`00
`•
`~
`~
`~
`
`~ = ~
`
`88
`
`42
`
`z 0
`~ . ....
`....
`
`'"
`N
`0
`0
`QO
`
`('D
`
`1J1 =-('D
`.....
`Ul
`0 .....
`....
`....
`
`d
`rJl
`-....l
`~
`tit
`'"=
`"'""
`"'"" ~ = N
`
`Microsoft Ex. 1005
`Microsoft v. Philips - IPR2018-00025
`Page 7 of 21
`
`

`

`Click and drag from magnifier tool button,
`crates and drags magnifier onto/across screen
`
`26
`
`I
`
`90
`
`48
`
`...._
`,...
`
`)If
`
`... ··"
`
`.. ····
`
`Figure 7A
`
`Figure 7B
`
`'push' drag tool 'off screen
`in any direction removes object
`and restores 'icon'
`
`~
`00
`•
`~
`~
`~
`
`~ = ~
`
`z 0
`~ ....
`....
`
`'"
`N
`0
`0
`QO
`
`('D
`('D
`
`1J1 =(cid:173)
`.....
`O'I
`0 .....
`....
`....
`
`d
`rJl
`-....l
`~
`tit
`'"=
`"'""
`"'"" ~ = N
`
`Microsoft Ex. 1005
`Microsoft v. Philips - IPR2018-00025
`Page 8 of 21
`
`

`

`optionally "highlighted" text
`area (application specific tool)
`
`l
`
`100
`
`/
`
`108
`
`102
`
`http://www.domain ...
`
`I
`
`drag/click
`on
`enhanced UI
`control
`
`link, coloured
`if required
`
`I
`
`I
`
`~ 104
`
`i
`
`slider control (or button/switch)
`to engage enhance UI view
`
`Figure 8A
`
`highlights/URI
`stay strong colour
`
`i
`
`I
`
`I
`
`..
`
`...................
`.......•••••..•...•
`.••••••............
`
`http://www.domain ...
`
`..................•
`...................
`•••................
`........•••••••..••
`I ..•••••............
`I
`I ~ T ~
`
`I
`
`I
`
`I
`
`rest of the page (text, graphics)
`"fades" out using alpha
`transparency/contrast adjustment
`
`Figure 8B
`
`108
`
`---. 104
`
`~
`00
`•
`~
`~
`~
`
`~ = ~
`
`z
`0
`~
`....
`....
`
`'"
`N
`0
`0
`QO
`
`('D
`('D
`
`1J1 =-
`.....
`-....J
`0 .....
`....
`....
`
`d
`rJl
`-....l
`~
`tit
`'"=
`"'""
`"'"" ~ = N
`
`Microsoft Ex. 1005
`Microsoft v. Philips - IPR2018-00025
`Page 9 of 21
`
`

`

`simultaneous "collation" ofread ahead thumb{lails
`
`102
`
`http://www.domaiJ
`
`-......
`I:
`...... I
`
`I
`
`...... -
`-http://www.domain I -......
`
`I
`
`I
`
`100
`
`f§l~
`El
`E]~
`
`114
`
`114
`
`114
`
`~1-112
`t
`
`llO
`
`enhance
`UI button
`
`Floating thumbs of
`page view should link when
`'clicked' upon
`
`Figure 9
`
`~
`00
`•
`~
`~
`~
`
`~ = ~
`
`z 0
`~ ....
`....
`
`'"
`N
`0
`0
`QO
`
`1J1 =-('D
`.....
`
`('D
`
`QO
`
`0 .....
`....
`....
`
`d
`rJl
`-....l
`~
`tit
`'"=
`"'""
`"'"" ~ = N
`
`Microsoft Ex. 1005
`Microsoft v. Philips - IPR2018-00025
`Page 10 of 21
`
`

`

`~
`00
`•
`~
`~
`~
`
`~ = ~
`
`z 0
`~ ....
`.... .
`
`N
`0
`0
`QO
`
`('D
`('D
`
`1J1 =(cid:173)
`.....
`"° 0 .....
`....
`....
`
`d
`rJl
`-....l
`~
`tit
`'"=
`"'""
`"'"" ~ = N
`
`/124
`he'L [
`i
`I L"
`i
`u· ····
`ii
`... JI . ..
`
`he: I
`
`i
`
`-130
`
`'·
`
`: ··················i: ......... .
`
`he
`
`..__ 132
`
`··············· :···· ......... ,
`
`iu . i ~ 134
`I ..
`.
`
`1·
`
`········1
`
`122
`
`120
`
`124
`
`same as 65a, but sequence of
`characters entered, plate
`remains still, text scrolls
`underneath
`
`Figure 10
`
`hello
`
`......
`.....
`
`138
`
`Figure 11
`
`Microsoft Ex. 1005
`Microsoft v. Philips - IPR2018-00025
`Page 11 of 21
`
`

`

`G2J =OK
`
`Figure 12A
`
`~ = Delete
`
`Figure 12B
`
`[QJ = homepage
`
`or
`start over
`
`Figure 12C
`
`[2J =wipe
`
`or
`no
`
`Figure 12D
`
`[QJ
`
`Figure 12E
`
`B =next
`
`Figure 12F
`
`B
`
`Figure 12G
`
`=previous
`
`~
`00
`•
`~
`~
`~
`
`~ = ~
`
`z 0
`~ ....
`....
`
`'"
`N
`0
`0
`QO
`
`('D
`
`1J1 =-('D
`.....
`....
`0
`0 .....
`....
`....
`
`d
`rJl
`-....l
`~
`tit
`'"=
`"'""
`"'"" ~ = N
`
`Microsoft Ex. 1005
`Microsoft v. Philips - IPR2018-00025
`Page 12 of 21
`
`

`

`click
`
`Page stops (velocity= 0)
`
`..... . ...... -..
`
`... ..·····
`. . . . :
`. .. . ..
`...... ......
`
`: ~: ~! \
`:
`.:
`. ...... .
`
`. . .
`. . .
`.
`·· ...... ·· .....
`! \ : ~
`..
`
`Drag again
`
`..
`• __ ............ ..
`-......... · 0 .. -....... -...
`. ..
`.
`. . . .
`. .. . ..
`...... . ....
`
`...
`
`.. -...
`• . ... .,.•
`
`•
`•
`
`•
`•
`•
`
`Figure 13A
`
`Figure 13B
`
`click
`
`...-..
`.
`. .
`..
`......
`•
`
`•
`
`~
`00
`•
`~
`~
`~
`
`~ = ~
`
`z 0
`~ ....
`.... .
`
`N
`0
`0
`QO
`
`('D
`('D
`
`1J1 =(cid:173)
`.....
`....
`....
`0 .....
`....
`....
`
`d
`rJl.
`-....l
`~
`tit
`'"=
`"'""
`"'"" ~ = N
`
`Microsoft Ex. 1005
`Microsoft v. Philips - IPR2018-00025
`Page 13 of 21
`
`

`

`1
`USER INTERFACE SYSTEMS AND METHODS
`FOR MANIPULATING AND VIEWING
`DIGITAL DOCUMENTS
`
`REFERENCE TO RELATED APPLICATIONS
`
`The systems and methods described herein relate to earlier
`filed British patent application No. 0009129.8 and earlier
`filed U.S. patent application Ser. No. 09/703,502, as well as
`the U.S. patent application entitled, Systems and Methods for
`Digital Document Processing, filed even date herewith, all of
`which name Majid Anwar as an inventor, and the contents of
`which are herein incorporated by reference.
`
`FIELD OF THE INVENTION
`
`The systems and methods described herein relate to sys(cid:173)
`tems and methods for viewing and manipulating a display of
`digital documents, and more particularly to user interface
`systems and methods that allow a user to manipulate and view
`a digital document presented on a display, such as the display
`of a hand held electronic device, such as a computer, mobile
`communicator or phone; or a display device associated with a
`tactile commander.
`
`BACKGROUND OF THE INVENTION
`
`Today, there is a strong effort to build mobile and handheld
`computing devices that easily allow users to view documents,
`email, video presentations, and other forms of content. To
`achieve this convergence, engineers and scientists have devel(cid:173)
`oped systems including the systems described in the above(cid:173)
`referenced U.S. patent application entitled, Systems and
`Methods for Processing Digital Documents, the contents of
`which are hereby incorporated by reference. As described
`therein, digital content, whether a document, audio visual
`presentation, or some other type of content, is processed by a
`software system operating on a handheld device, mobile
`device, or some other platform, and translated into a uniform
`internal representation that can be processed and manipulated
`by the software system so that a display of different types of
`content can be generated by the system and presented on the
`screen display of the respective device.
`These systems, as well as other handheld and mobile com(cid:173)
`puting systems such as the Palm Pilot, Compaq Ipaq, and
`mobile phones, are therefore capable of providing a display of
`content to a user. However, these handheld and mobile sys(cid:173)
`tems are typically limited to simple input devices such as
`small and limited keyboards, commonly the keyboards
`present on a cellular phone, or by small touch screen systems,
`such as the touch screens provided with the Palm computing
`device. Therefore, although these systems are capable of pre(cid:173)
`senting content to a user wherein the content can be quite
`complex, these systems have limited capacity to allow a user
`to manipulate the display of that content, such as by paging
`through the different pages of a document or selecting differ(cid:173)
`ent portions of a document. Therefore, although these hand(cid:173)
`held and portable systems may be quite useful, there are
`limitations to their use based in part on the user interfaces
`available for supporting the manipulation and viewing of the
`content presented on these devices.
`Consequently, there is a need in the art for systems and
`methods that provide improved user interface tools that make
`it more facile to manipulate and view content presented by a
`handheld or portable device.
`Additionally, there is a need in the art for user interface 65
`tools that allow for the manipulation of content, when that
`content is separated from its native application program.
`
`US 7,450,114 B2
`
`2
`SUMMARY OF THE INVENTION
`
`The systems and methods described herein provide
`advanced user interface tools that allow a user to more easily
`manipulate and view content presented on a mobile device or
`handheld device. In one particular embodiment, the systems
`and methods described herein provide a graphical user inter(cid:173)
`face that exhibits a touch and feel user interface experience.
`More specifically, the systems and methods described herein
`10 include hand-held or mobile computer devices having a sys(cid:173)
`tem for simulating tactile control over a document which may
`be viewed on the device itself or through a remote instruction
`or remote instruction or remote display on another unit. These
`systems may include a housing which supports a processor,
`15 memory, and a touch-sensitive display (or display with
`remote touch-sensitive control), system code stored within
`the memory and adapted to be executed by the processor. The
`system code may generate or provide a digital representation
`of a document, wherein the digital representation may include
`20 data content and a page structure representative of the page
`layout of the document. Thus, in certain applications the
`rendered image can include the content of the document as
`well as the layout of the document, thereby providing an
`image of what the document physically looks like. The sys-
`25 tern may also include a rendering engine that may include a
`parser and a renderer for rendering at least a portion of the
`page layout of the digital representation on the touch-sensi(cid:173)
`tive display. A screen monitor canmonitorthe touch-sensitive
`screen for detecting movement across a surface of the touch
`30 sensitive screen, and an interface process can process the
`detected movement to detect a motion representative of a
`command to alter the page structure of the digital represen(cid:173)
`tation. A navigation module may be responsive to the inter(cid:173)
`face process and can change the rendered portion of the page
`35 layout. Thus, by altering the rendered portion of the page
`layout, the system allows a user to navigate through the digital
`representation of the document. Although the systems and
`methods of the invention will have applicability and value
`when used in other applications and on other types of sys-
`40 terns, for purposes of illustration, the invention will be
`described with reference to those applications where the sys(cid:173)
`tems facilitate the navigation of documents presented on a
`hand-held computing device.
`More particularly, the systems and methods described
`45 herein provide, among other things, computer devices having
`a system for simulating tactile control over a document. In
`one embodiment, these systems comprise a processor,
`memory, and a display; system code stored within the
`memory and adapted to be executed by the processor, the
`50 system code providing a digital representation of a document
`including data content and a page structure representative of
`a page layout of the document; a rendering engine for render(cid:173)
`ing at least a portion of the page layout of the digital repre(cid:173)
`sentation on the display; a screen monitor for monitoring the
`55 screen to detect movement of an object across an image
`presented on the display; an interface process for processing
`the detected movement to detect a motion representative of a
`command to alter the rendered page structure of the digital
`representation, and a navigation module responsive to the
`60 interface process for changing the rendered portion of the
`page layout, wherein altering the rendered portion of the page
`layout allows a user to navigate through the digital represen(cid:173)
`tation of the document.
`These computer devices can include touch-sensitive dis(cid:173)
`plays where the screen monitor monitors a touch-sensitive
`screen for detecting movement across a surface of the touch
`sensitive display, as well as computer displays capable of
`
`Microsoft Ex. 1005
`Microsoft v. Philips - IPR2018-00025
`Page 14 of 21
`
`

`

`US 7,450,114 B2
`
`3
`depicting a cursor moving across a screen of the display, and
`wherein the screen monitor detects movement of the cursor
`across a surface of the display. The processor, memory, screen
`monitor and a display may be arranged as a data processing
`platform useful with a plurality of applications and devices
`including hand-held computers, telephones, mobile data ter(cid:173)
`minal, a set top box, an embedded processor, a notebook
`computer, a computer workstation, a printer, a copier and a
`facsimile machine.
`In certain optional embodiments, the computer device may
`also include a velocity detector for determining a velocity
`vector associated with motion detected across the surface of
`the touch-sensitive display, as well as means for applying a
`velocity characteristic to a document within a display.
`Additionally, these computer devices can have an interface
`processes that make it more easy to navigate through a docu(cid:173)
`ment or a collection of documents and other content. These
`interface processes can include a page-flip detector for detect(cid:173)
`ing a motion across the surface of the touch-screen at a loca(cid:173)
`tion presenting a portion of the page layout graphically rep(cid:173)
`resentative of a comer of a document. The page-flip detector
`can render a portion of the page layout representative of a
`page adjacent a currently rendered page. Similarly, the device
`can include a page curl detector for rendering a portion of the
`page layout representative of a portion of a page adjacent a
`currently rendered page. Additionally, the interface process
`can include a gesturing process for detecting a predefined
`movement representative of a command for selecting a por(cid:173)
`tion of the page layout to be rendered, or for altering data
`content of the digital representation of the document. Still
`further interface controls include processes for controlling a
`transparency characteristic of a document presented on the
`display and for controlling a transparency characteristic of
`selected portions of the document for adjusting visibility of
`the selected portions relative to other portions of the docu(cid:173)
`ment. Other interface processes can provide tools, including
`tools representative of a magnifying tool, a ruler, a text entry
`cursor, a thumbnail navigation column, a thumbnail view of
`linked content and a query tool.
`In still other aspects, the invention provides computer
`devices, and related processes, having a context sensitive
`graphical interface tool. These devices may comprise a pro(cid:173)
`cessor, memory, and a touch-sensitive display; a content
`document file stored in the memory and being representative
`of an internal representation of the content; a tool document
`file stored in the memory and providing an internal represen(cid:173)
`tation of a document providing an image that is representative
`of the graphical interface tool; tool code capable of running of
`the processor and being associated with the tool document file
`and capable of processing the content document file to create
`an internal representation of the content that when rendered
`presents the content in a manner that achieves a display effect
`associated with the tool; parsing code that processes the con(cid:173)
`tent document file, the tool document file, and the processed
`internal representation to generate a screen document for 55
`display, and; interface code capable of running on the proces(cid:173)
`sor for allowing a user to arrange the image of the graphical
`interface tool into a selected contextual relationship over the
`rendered content and for directing the tool code to process a
`portion of the content document file associated with the
`selected position.
`The contextual relationship between the graphical inter(cid:173)
`face tool and the rendered content can vary depending upon
`the application, and may for example be selected from the
`group consisting of the relative position of the graphical inter(cid:173)
`face tool and the rendered content, the time at which the
`graphical interface tool acts on the rendered content, and the
`
`4
`state of the rendered content. These devices are flexible and
`may be embodied in different forms and devices, including,
`but not being limited to, a hand-held computer, a telephone,
`mobile data terminals, a set top box, an embedded processor,
`a notebook computer, a computer workstation, a printer, a
`copier, and a facsimile machines, as well as in car systems,
`and domestic devices such as audio players, microwaves,
`refrigerators, and washing machines.
`However, it will be understood by those of ordinary skill in
`10 the art that these interface tools may be employed in other
`applications including applications wherein content is dis(cid:173)
`played on a conventional computer workstation that includes
`typical input tools such as a standard keyboard and a mouse.
`Additionally, it will be understood that the systems and meth-
`15 ods described herein also provide useful tools for providing
`interfaces for embedded display systems, such as embedded
`visual displays employed as output devices. Examples of
`such embedded display systems can include cellular phones,
`copy machines that include a visual touch screen display that
`20 allows a user to select different options for performing a copy
`job and which may also present images to the user of the
`documents being copied. Other examples may include fax
`machines wherein visual displays are provided to a user to
`allow a user to view a depiction of an incoming fax. Other
`25 embodiments and applications of the user interface systems
`and methods described herein will be apparent to those of
`ordinary skill in the art.
`More particularly, the systems and methods described
`herein provide user interface tools that allow a user to
`30 manipulate content displayed on a screen. In particular, the
`systems and methods described herein provide software sys(cid:173)
`tems that create an abstraction layer for information that is to
`be presented on a display. This abstraction layer includes a
`document object wherein a document object contains infor-
`35 mation, or content, that is to be displayed on a screen. In one
`implementation, all information displayed on a screen is
`treated as one document. Thus, at the highest level, the entire
`contents of a screen is understood as one document object.
`Further to this embodiment, it will be understood that a docu-
`40 ment object may contain other document objects, each of
`which may contain a subset of the content displayed to the
`user. Thus, at the screen level, all information displayed will
`be understood as a single document wherein items, such as
`web pages, streamed video, and graphical icons, presented on
`45 the screen are each understood separately as document
`objects contained within the high level screen document
`object. Therefore, all content displayed on a screen is treated
`abstractly as a document, and this paradigm holds whether the
`content being displayed is information representative of a
`50 page of text or information representative of a user interface
`tool or window/desktop furniture.Accordingly, the user inter(cid:173)
`face systems and methods described herein provide user
`interface tools and functionality for allowing a user to
`manipulate document objects presented on a screen display.
`Additionally, the systems and methods described herein
`provide, in one embodiment, a handheld computing device
`that comprises a housing which supports a processor,
`memory, and a touch sensitive display. Further, the comput(cid:173)
`ing device may include system code stored within the
`60 memory and adapted to be executed by the processor. The
`system code can process an input byte stream that is repre(cid:173)
`sentative of content to be displayed on the touch sensitive
`display and can generate a content document file representa(cid:173)
`tive of an internal representation of the content. A tool docu-
`65 ment file may also be stored in the memory and may provide
`an internal representation of a document providing an image
`that is representative of a graphical tool. Associated with a
`
`Microsoft Ex. 1005
`Microsoft v. Philips - IPR2018-00025
`Page 15 of 21
`
`

`

`US 7,450,114 B2
`
`5
`tool document can be tool code that is capable of processing
`the content document file to create an internal representation
`of the content that presents the content in a manner that
`achieves a display effect associated with the tool. The device
`may also include parsing code that processes the content
`document file, the tool document file, and the processed inter(cid:173)
`nal representation to generate a screen document for display
`on the touch sensitive display in a manner that portrays the
`display effect.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`The foregoing and other objects and advantages of the
`invention will be appreciated more fully from the following
`further description thereof, with reference to the accompany(cid:173)
`ing drawings wherein;
`FIG. 1 provides a functional block diagram of one system
`according to the invention;
`FIG. 2 depicts one example ofa tool generated by a system
`such as the system depicted in FIG. 1;
`FIG. 3 depicts a graphical user interface tool that presents
`a plurality of thumbnail sketches for navigating through a
`document having a plurality of pages;
`FIG. 4 depicts a magnifying graphical user interface tool
`according to the invention which provides additional infor(cid:173)
`mation within a magnified area;
`FIG. 5 depicts a semi-transparent and adaptively sizeable
`ruler graphical tool;
`FIG. 6 depicts a transparent query marker graphical user
`interface tool;
`FIG. 7 depicts a user interface mechanism for activating
`and deactivating a graphical tool;
`FIGS. Sa and Sb depict a user interface tool for visually
`enhancing selected portions of a displayed document;
`FIG. 9 depicts a further user interface tool according to the
`invention;
`FIGS. 10 and 11 depict a text entry tool according to the
`invention;
`FIGS. 12a-12g depict a set of strokes for providing com(cid:173)
`mands to a hand-held system; and
`FIGS. 13A-13B depict a user interface tool for scrolling
`through a document by applying a velocity characteristic to
`the document being displayed.
`
`DETAILED DESCRIPTION OF THE
`ILLUSTRATED EMBODIMENTS
`
`The systems and methods described herein include sys(cid:173)
`tems and methods for manipulating and viewing documents
`displayed on a viewing surface such as a computer terminal,
`a display screen, a printer, plotter, or any other output device
`suitable for creating a visual representation of human read(cid:173)
`able information. For the purpose of illustration, the systems
`and methods will be described with reference to certain exem(cid:173)
`plary embodiments, including hand held computer systems
`that include touch screen displays and which are capable of
`displaying an integrated view of content produced in different
`formats. In particular, the systems and methods described
`herein include graphical user interface tools that are capable
`of presenting tools that can be presented as content that will
`be integrated along with other content being displayed on the
`screen.
`FIG. 1 depicts a system 10 according to the invention. The
`system 10 is shown as a functional block diagram of a com(cid:173)
`puter device of the type that commonly includes a processor,
`a memory and a display. However, the system 10 may also be
`realized, in whole or in part as a software system comprising
`
`6
`system code capable of executing on a processor to configure
`the processor as a system according to the invention. The
`depicted system 10 includes a computer process S, a plurality
`of source documents 11, a tool document file 30, a shape
`processor 22 and a video display 26. The computer process S
`includes a plurality of document agents 12, a library of
`generic data objects 16, an internal representation file 14, a
`memory buffer or file 15, and a parser/renderer engine lS.
`In the depicted embodiment, the display 26 can present the
`10 images of a plurality of different documents. Each of the
`representative outputs appearing on display 26 is termed a
`document, and each of the depicted documents can be asso(cid:173)
`ciated with one separate application program, such as Word,
`Netscape Navigator, Real Player, Adobe, Visio and other
`15 types of applications. It will be understood that the term
`document as used herein will encompass documents,
`streamed video, web pages, and any other form of data that
`can be processed and displayed by the computer process S.
`The computer process S generates a single output display
`20 that includes within that display one or more of the docu(cid:173)
`ments. The collection of displayed documents represent the
`content generated by the application programs and this con(cid:173)
`tent is displayed within the program window generated by the
`computer process S. The program window for the computer
`25 process S may also include a set of icons representative of
`tools provided with the graphical user interface and capable
`of allowing a user to control the operation, in this case the