(12) United States Patent
`Lerner et al.
`
`USOO6859909B1
`
`(10) Patent No.:
`(45) Date of Patent:
`
`US 6,859,909 B1
`Feb. 22, 2005
`
`(54) SYSTEM AND METHOD FOR ANNOTATING
`WEB-BASED DOCUMENTS
`
`(75) Inventors: Matthew Rubin Lerner, Berkeley, CA
`(US); Oliver Hurst-Hiller, San
`Francisco, CA (US); Jesse Gardner
`Kocher, San Francisco, CA (US);
`David Keel Peck, San Francisco, CA
`(US)
`(73) Assignee: Microsoft Corporation, Redmond, WA
`(US)
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 0 days.
`
`(*) Notice:
`
`(21) Appl. No.: 09/521,022
`(22) Filed:
`Mar. 7, 2000
`
`(51) Int. Cl. ................................................ G06F 15/00
`
`(52) U.S. Cl. ....................... 71.5/512; 71.5/530; 345/440;
`709/203
`
`(58) Field of Search .............................. 71.5/512, 501.1,
`71.5/530,526, 500; 345/440; 709/203, 217
`References Cited
`U.S. PATENT DOCUMENTS
`
`(56)
`
`5,822,539 A * 10/1998 van Hoff .................... 709/236
`5,826,025 A * 10/1998 Gramlich .................... 709/217
`6,081,829. A
`6/2000 Sidana ....................... 709/203
`6,262,728 B1 * 7/2001 Alexander ......
`... 345/440.1
`6,457,026 B1 * 9/2002 Graham et al. ............. 71.5/512
`6,687,878 B1 * 2/2004 Eintracht et al. ........... 71.5/512
`OTHER PUBLICATIONS
`Wilcox, Lynn D. et al. “Dynomite: A Dynamically Orga
`nized Ink and Audio Notebook', CHI 97, Mar. 22–27, 1997.
`pp. 186-193.
`Price, Morgan N, et al. "XLibris: The Active Reading
`Machine”. CHI 98, Apr. 18–23, 1998. Pp 22–23.
`
`
`
`Schilit, Bill N. et al. “Beyond Paper: Supporting Active
`Reading with Free Form Digital Ink Annotations”. CHI 98,
`Apr. 18–23, 1998. Pp. 249–256.
`Schilit, Bill N. et al., “Digital Library Information Appli
`ances” Digital Library 98; Pittsburgh, PA. Pp 217-226.
`Moran, Thomas P. et al. “Spatial Interpretation of Domain
`Objects Integrated into a Freeform Electronic Whiteboard”
`UIST '98, San Francisco, CA. Pp. 175-184.
`Kurtenbach, Gordon et al. “Issues in Combining Marking
`and Direct Manipulation Techniques' UIST 91, Nov.
`11–13, 1991. Pp 137–144.
`Long Jr., A. Chris, “Dissertation Proposal: The Design and
`Evaluation of Gestures for Pen-based User Interfaces”.
`Qualifying Exam Proposal References, pp 1-8.
`(List continued on next page.)
`Primary Examiner Joseph H. Feild
`Assistant Examiner Maikhanh Nguyen
`(74) Attorney, Agent, or Firm-Banner & Witcoff, Ltd.
`(57)
`ABSTRACT
`The present invention allows computer users to integrate any
`annotation, including ink, highlighter, text-based notes and
`audio, directly into a Web-based document (WBD) dis
`played by a Web browser. This integration enables others to
`view the personalized annotated WBD, which retains its
`original active links and properties, over the Internet without
`the need for Specialized Software. Annotations are integrated
`into WBDs by freezing the WBD, overlaying an image file
`containing the annotations onto the WBD, and enabling
`browser events to pass through the image layer. Annotations
`may also be integrated into WBDS by using component
`object technology. The present invention collects and orga
`nizes annotated WBDs, and provides users with an intuitive
`Web-based interface for accessing, Viewing and Searching
`the annotated WBDs. Users may annotate blank WBDs,
`effectively converting their Web browsers into online
`notebookS/Scrapbooks. The present invention also provides
`users with many novel interface techniques, Such as dog
`ears and its associated navigation tools, splitting pages,
`turning pages, Selecting and copying Various portions of a
`WBD (including Shaking out a copy), and marking menus
`Suited for right-handed or left-handed users.
`6 Claims, 23 Drawing Sheets
`
`sopropartle
`estYES
`888.8lemar
`
`American Airlines, Ex. 1004, p.1
`American Airlines v. Advanced Transactions
`IPR2023-01366
`
`

`

`US 6,859,909 B1
`Page 2
`
`OTHER PUBLICATIONS
`Gross, Mark D.,et al. “Ambiguous Intentions: a Paper-like
`Interface for Creative Design”, UIST '96, Seattle Washing
`ton. Pp. 183-192.
`Moran, Thomas, P. et al., “Pen-Based Interaction Tech
`niques for Organizing Material on an Electronic White
`board”. UIST '97, Banff, Alberta, Canada. Pp 45-54.
`Forsberg, Andrew et al., “The Music Notepad”. Brown
`University, Providence RI, 7 pages.
`Lerner, Matt A Survey of Pen-Based User Interfaces: When
`Will They Be as Good as Pencil and Paper? Brown Univer
`sity, Providence RI, 713 pages.
`Davis, Richard C., et al. “A Framework for Sharing Hand
`written Notes”. UIST 98, San Francisco, CA. Pp 119-120.
`Landay, James A. et al. “Sketching Storyboards to Illustrate
`Interface Behaviors”. HCI Institute, Carnegie Mellon Uni
`Versity, 2 pages.
`Landay, James A. et al. “Interactive Sketching for the Early
`Stages of User Interface Design”, Carnegie Mellon Univer
`sity, 8 pages.
`
`Long Jr., Allan Christian "Improving Gestures and Interac
`tion Techniques for Pen-Based User Interfaces”. CHI '98,
`Apr. 18–23, 1998. Pp 58–59.
`Rubine, Dean. “Combining Gestures and Direct Manipula
`tion” CHI '92, May 3–7, 1992. Pp. 659-660.
`Kurtenbach, Gordon et al. “Contextual Animation of Ges
`tural Commands’. Xerox Palo Alto Research Center, Palo
`Alto, CA, University of Toronto, 14 pages.
`Tapia, Mark A., et al., “Some Design Refinements and
`Principles on the Appearance and Behavior of Marking
`Menus” UIST '95, Nov. 14–17, 1995. Pp 189-195.
`Moran, Thomas P. et al. “Implicit Structures for Pen-Based
`Systems Within a Freeform Interaction Paradigm, Xerox
`Palo Alto Research Center, Palo Alto, CA, 11 pages.
`Bier, Eric A. et al. “Toolglass and Magic Lenses: The
`See-Through Interface' Xerox Palo Alto Research Center,
`Palo Alto, CA, University of Toronto, University of Wash
`ington, 8 pages.
`
`* cited by examiner
`
`American Airlines, Ex. 1004, p.2
`American Airlines v. Advanced Transactions
`IPR2023-01366
`
`

`

`U.S. Patent
`
`Feb. 22, 2005
`
`Sheet 1 of 23
`
`US 6,859,909 B1
`
`User Computer 100 -
`
`
`
`Processor
`110
`
`input Device
`120
`
`Temporary Storage
`Device
`140
`
`Web Browser
`160
`
`Communication
`Device
`18O
`
`Output Device
`130
`
`Resident Storage
`Device
`15O
`
`Annotation Software
`170
`
`FIG. 1
`
`American Airlines, Ex. 1004, p.3
`American Airlines v. Advanced Transactions
`IPR2023-01366
`
`

`

`U.S. Patent
`
`Feb. 22, 2005
`
`Sheet 2 of 23
`
`US 6,859,909 B1
`
`Annotation Repository 240
`
`Stroke
`Database
`290
`
`User
`information
`D
`alse
`
`saf
`3. ''',
`
`User Folder
`database
`270
`
`Network ink 21 Oc
`
`
`
`Network Link 210b
`
`internet
`Service
`Provider
`220
`
`Network ink 2 Oa
`
`User 200
`
`
`
`FG 2
`
`American Airlines, Ex. 1004, p.4
`American Airlines v. Advanced Transactions
`IPR2023-01366
`
`

`

`U.S. Patent
`
`Feb. 22, 2005
`
`Sheet 3 of 23
`
`US 6,859,909 B1
`
`
`
`User Record 300
`
`USet D 310
`
`User Name 320
`
`User E-mail 330
`
`List of Folder DS 350
`
`FIG. 3
`
`American Airlines, Ex. 1004, p.5
`American Airlines v. Advanced Transactions
`IPR2023-01366
`
`

`

`U.S. Patent
`
`Feb. 22, 2005
`
`Sheet 4 of 23
`
`US 6,859,909 B1
`
`
`
`Folder C 410
`
`Category 420
`
`List of Annotation DS 430
`
`Folder Record 400
`
`FIG. 4
`
`American Airlines, Ex. 1004, p.6
`American Airlines v. Advanced Transactions
`IPR2023-01366
`
`

`

`U.S. Patent
`
`Feb. 22, 2005
`
`Sheet 5 of 23
`
`US 6,859,909 B1
`
`
`
`Annotation Record 500
`
`Annotation ID 510
`
`Original WBD URL515
`
`URL Content Hash 520
`
`Annotation Size 525
`
`Time/Date Stamp 530
`
`Dog-Ear 535
`
`Pointer to Original WBD 540
`
`Pointer to Thumbnail File 548
`
`List of Stroke OS 550
`
`List of Note IDs 555
`
`FIG. 5
`
`American Airlines, Ex. 1004, p.7
`American Airlines v. Advanced Transactions
`IPR2023-01366
`
`

`

`U.S. Patent
`
`Feb. 22, 2005
`
`Sheet 6 of 23
`
`US 6,859,909 B1
`
`
`
`Stroke Record 600
`
`Stroke ID 610
`
`List of Points 670
`
`F.G. 6
`
`American Airlines, Ex. 1004, p.8
`American Airlines v. Advanced Transactions
`IPR2023-01366
`
`

`

`U.S. Patent
`
`Feb. 22, 2005
`
`Sheet 7 of 23
`
`US 6,859,909 B1
`
`
`
`Note Record 700
`
`Note ID 710
`
`Note Type 720
`
`Note Text 730
`
`Pointer to Audio File 740
`
`FIG. 7
`
`American Airlines, Ex. 1004, p.9
`American Airlines v. Advanced Transactions
`IPR2023-01366
`
`

`

`U.S. Patent
`
`Feb. 22, 2005
`
`Sheet 8 of 23
`
`US 6,859,909 B1
`
`
`
`invoking Legacy of Reagan, |
`Elizabeth Dole Endorses Bush
`estaurants
`Asserting that the Texas governor would
`Y -- restore trust to the White House. Elizabeth Dole
`endorsed George W. Bush for the Republican
`presidential nomination. Go to Article
`as being WIDEO Elizabeth Dole Endorses Bush
`
`
`
`}
`
`Network ?
`FoR students,
`Teach (Rs PAja EMTs
`-
`Text version
`-
`I Neediest
`
`afonia
`ES3
`2.
`
`840
`850
`860
`
`Estitu
`in Waley Citts uncoir Ariliitt ur kicto:
`asser are:
`
`Casic - Hicistricinct Exploic
`
`t
`
`
`
`
`
`&
`SS
`S&
`& staff
`3Sri
`is hip/amrylineatormtroplechwsn1/bclccheticle/08od.html
`3.
`Silicon Valley Cites Concern Amid Glee on
`Microsoft Case
`By JOHN MARKOFF with SEVELOR
`ALOALTO, Caff -- in Sion Wasley, the nation's high-
`technology capital, Judge homas Penfield Jackson's powerfs
`findings in the Microsof antitrust case have been recited with a senses
`windication but also ambivalence.
`
`-
`
`Inis Swing his finkins of fact a frid
`evening Jackson desvered two imph Thought you'd like to know
`Arti it's official: there is a
`industy
`&
`as
`aukla in
`Sf E. E. Microsoft problem
`surrend the software monopoli the industry.
`buily&actics buri consumers an
`Audio 9 P. He
`undermine innovation. He also found
`custoria
`there is nothing special about the
`t
`Psi; as
`software industry.
`purposes of
`antitrust, he is
`sighth soika Sri Sofia issy of
`sistentionary tickens. Edge First
`
`-
`
`800
`
`810
`
`815
`
`810
`
`805
`
`810
`820
`
`825
`
`:
`2.
`
`-
`
`i
`
`American Airlines, Ex. 1004, p.10
`American Airlines v. Advanced Transactions
`IPR2023-01366
`
`

`

`U.S. Patent
`
`Feb. 22, 2005
`
`Sheet 9 of 23
`
`US 6,859,909 B1
`
`Access contents of
`WBD
`905
`
`
`
`Wrap existing
`WBD layers into
`one fix-sized layer
`910
`
`Examine
`properties of WBO
`elements
`915
`
`
`
`Do properties
`exist that could
`unfreeze element?
`920
`
`Modify properties
`to freeze element
`925
`
`Do children
`elements exist?
`930
`
`
`
`Examine
`properties of each
`child element
`935
`
`FG. 9
`
`American Airlines, Ex. 1004, p.11
`American Airlines v. Advanced Transactions
`IPR2023-01366
`
`

`

`U.S. Patent
`
`Feb. 22, 2005
`
`Sheet 10 Of 23
`
`US 6,859,909 B1
`
`
`
`Image File
`1 OOO
`
`WBD 1010
`
`FIG 10
`
`American Airlines, Ex. 1004, p.12
`American Airlines v. Advanced Transactions
`IPR2023-01366
`
`

`

`U.S. Patent
`
`Feb. 22, 2005
`
`Sheet 11 Of 23
`
`US 6,859,909 B1
`
`Create Annotation
`Record
`1105
`
`Overlay blank
`imag file on top
`of WBD
`1110
`
`Current WBO
`still displayed?
`
`1115
`
`M
`
`
`
`
`
`
`
`
`
`
`
`
`
`Annotation
`event initiated
`by user?
`2O
`
`Access display
`mechanism for
`entire computer
`scree
`125
`
`Enable placement
`of annotation
`element on screen
`within browser
`boundaries
`1130
`
`Append annotation
`information into
`annotation record
`35
`
`ReloadWBO
`
`se
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`FIG. 11
`
`American Airlines, Ex. 1004, p.13
`American Airlines v. Advanced Transactions
`IPR2023-01366
`
`

`

`U.S. Patent
`
`Feb. 22, 2005
`
`Sheet 12 of 23
`
`US 6,859,909 B1
`
`1200a
`
`1210
`
`1200b
`
`insert image file
`multiple times or
`top of WBD
`1305
`
`Commence
`tracking of input
`device noverherts
`1310
`
`
`
`
`
`
`
`Wait until
`input device
`Wes
`315
`
`Dynamically clip
`irnage file around
`input device
`location
`1320
`
`F.G. 13
`
`American Airlines, Ex. 1004, p.14
`American Airlines v. Advanced Transactions
`IPR2023-01366
`
`

`

`U.S. Patent
`
`Feb. 22, 2005
`
`Sheet 13 of 23
`
`US 6,859,909 B1
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`Create annotation
`Record
`1405
`
`.
`
`s
`
`was
`t
`
`
`
`Current WBD
`still displayed?
`145
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`Annotation
`event initiated
`by user?
`1420
`
`Enable control
`object to capture
`input device
`movements
`1425
`
`Place annotation
`element on WBO
`in response to .
`input device
`rowerinents
`1430
`
`Disable control
`object from
`capturing input
`device novernents
`1435
`
`
`
`Append annotation
`information into
`annotation record
`144
`
`Reload W3D
`1445
`
`FIG. 14
`
`American Airlines, Ex. 1004, p.15
`American Airlines v. Advanced Transactions
`IPR2023-01366
`
`

`

`U.S. Patent
`
`Feb. 22, 2005
`
`Sheet 14 of 23
`
`US 6,859,909 B1
`
`
`
`s
`
`g 3.
`3.
`3.
`3.
`
`F.G. 15
`
`American Airlines, Ex. 1004, p.16
`American Airlines v. Advanced Transactions
`IPR2023-01366
`
`

`

`U.S. Patent
`
`Feb. 22, 2005
`
`Sheet 15 of 23
`
`US 6,859,909 B1
`
`
`
`is Netscape
`view go wide Bee, -
`as
`-
`S - 3
`figugi
`coad Home Search
`
`it
`
`-: - ---.
`
`2- :
`
`r
`
`-
`
`-.
`
`way .
`
`s
`:
`
`--rrent
`
`v.
`
`as (D.5cc (7)
`
`. Out Box (is) G 2Spa--1
`untiled si
`-ar
`"
`torpetition
`or
`Internet 7
`Music
`Research is
`
`800
`
`. . . .
`.
`. .
`.
`.
`. . . . . .
`. .
`.
`Thess are notes from the
`1007/89 meeting with Jesse.
`Kocher regarding our interface
`sign
`''':
`; ,
`da
`'.
`
`.
`
`t
`
`r succes 1.
`East
`
`FIG. 15A
`
`American Airlines, Ex. 1004, p.17
`American Airlines v. Advanced Transactions
`IPR2023-01366
`
`

`

`U.S. Patent
`
`Feb. 22, 2005
`
`Sheet 16 of 23
`
`US 6,859,909 B1
`
`
`
`afsikh
`http:Maqui comeroyo 0.04257257
`
`.
`
`.
`
`.
`
`.
`
`Finally, online trading
`gets personal.
`
`1610
`
`-
`
`,
`
`... '
`
`...txec.----
`pate: 11300. By mattiae-q.
`ate::1f1200. By mattae-quillcom
`t ---- a-------':
`ite crin.com Category. Research is nytimes.com Catego es
`
`-
`
`... r
`
`equili.com
`
`1600
`
`1610
`
`161O
`
`&
`
`:
`
`American Airlines, Ex. 1004, p.18
`American Airlines v. Advanced Transactions
`IPR2023-01366
`
`

`

`U.S. Patent
`
`Feb. 22, 2005
`
`Sheet 17 of 23
`
`US 6,859,909 B1
`
`
`
`
`
`
`
`
`
`Stroke
`completed?
`1705
`
`Dynamic stroke
`recognized?
`1710
`
`invoke appropriate
`corn Tand
`1715
`
`suild vector data
`matrix of stroke
`features
`1720
`
`Single
`stroke gesture
`recognized?
`1725
`
`
`
`
`
`initial
`stroke of multi-
`site.
`recognized?
`1735
`
`Wait for next
`t
`
`P.t 2
`stroke features
`1745
`
`Final
`stroke of multi
`stroke gesture
`recognized?
`1750
`
`invoke appropriate
`command
`1730
`
`
`
`FIG.
`17
`
`American Airlines, Ex. 1004, p.19
`American Airlines v. Advanced Transactions
`IPR2023-01366
`
`

`

`U.S. Patent
`
`Feb. 22, 2005
`
`Sheet 18 of 23
`
`US 6,859,909 B1
`
`1900
`
`1515
`
`
`
`
`
`1800
`
`
`
`T
`
`http://www.rytimes.com/library/tech/SS/i Aztech/articlesiosafhrd
`2. Silicon Valley Cites Concern Amic
`as
`Microsoft Case
`-
`
`800
`
`By JOHN RARSOFF with SEVELOHR
`ALOALTO, Calif -- In Silicon Wailey, the nation's
`technology capital, Judge Thomas Penfield Jackson'
`findings in the Microsoft antitrust case have been received
`windication but also ambivalence.
`
`
`
`in issuing his findings of fact on Friday
`evening. Jackson delivered two implicit
`
`F.G. 18
`
`American Airlines, Ex. 1004, p.20
`American Airlines v. Advanced Transactions
`IPR2023-01366
`
`

`

`U.S. Patent
`
`Feb. 22, 2005
`
`Sheet 19 of 23
`
`US 6,859,909 B1
`
`flip to bog-ear
`
`F.G. 21
`
`issue diagonal
`stroke starting in
`upper left-hand
`corner of desired
`WBd
`2010
`
`Navigate
`through Dog X
`Ears?
`
`Select desired
`WBO identified to
`small triangle
`2025
`
`issue simple
`vertical strokes to
`view adjacent dog
`eared WBDs
`2030
`
`F-PP
`
`issue diagonal
`stroke starting in
`upper left-hand
`corner of desired
`
`W3O
`
`O
`
`FIG. 20
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`American Airlines, Ex. 1004, p.21
`American Airlines v. Advanced Transactions
`IPR2023-01366
`
`

`

`U.S. Patent
`
`Feb. 22, 2005
`
`Sheet 20 Of 23
`
`US 6,859,909 B1
`
`Activate input
`device
`2510
`
`
`
`Move input device
`in circular pattern
`to turt to
`consecutive pages
`252O
`
`
`
`
`
`
`
`Turn pages
`faster or slower?
`25.30
`
`
`
`Vary speed of
`circling motion
`accordingly
`2540
`
`Deactivate input
`device when
`desired page is
`reached
`2550
`
`FIG. 25
`
`Turn Multiple Pages
`
`Q9.
`
`FIG. 24
`
`Page Turn
`
`FIG. 23
`
`Page split
`
`FIG. 22
`
`American Airlines, Ex. 1004, p.22
`American Airlines v. Advanced Transactions
`IPR2023-01366
`
`

`

`U.S. Patent
`
`Feb. 22, 2005
`
`Sheet 21 of 23
`
`US 6,859,909 B1
`
`Select Brodketed Areo select Rectangular Area
`
`-----
`
`FIG. 26
`
`American Airlines, Ex. 1004, p.23
`American Airlines v. Advanced Transactions
`IPR2023-01366
`
`

`

`U.S. Patent
`
`Feb. 22, 2005
`
`Sheet 22 of 23
`
`US 6,859,909 B1
`
`
`
`Use input device
`to select object to
`be copied
`27D
`
`Shake input
`device while
`retaining Selection
`to create copy of
`selection
`27.20
`
`Without
`deactivating input
`device, drag Copy
`of selection to
`desired location
`27.30
`
`Deactivate input
`device to leave
`copy of selection
`in desired location
`2740
`
`FIG. 27
`
`American Airlines, Ex. 1004, p.24
`American Airlines v. Advanced Transactions
`IPR2023-01366
`
`

`

`U.S. Patent
`
`Feb. 22, 2005
`
`Sheet 23 of 23
`
`US 6,859,909 B1
`
`
`
`284Ob
`
`284Od
`
`Regular Marking Menu
`280
`
`Right Handed Marking Menu
`282O
`
`Left Handed Marking Menu
`2830
`
`FIG. 28
`
`American Airlines, Ex. 1004, p.25
`American Airlines v. Advanced Transactions
`IPR2023-01366
`
`

`

`1
`SYSTEMAND METHOD FOR ANNOTATING
`WEB-BASED DOCUMENTS
`
`US 6,859,909 B1
`
`FIELD OF THE INVENTION
`The present invention relates to a System and method of
`marking, organizing, and Searching Web-based documents.
`
`BACKGROUND INFORMATION
`Standard desktop word processing applications enable
`computer users to create and edit text-based documents. To
`provide computer users with more flexibility in the editing
`and markup of these documents, Software utilities exist that
`integrate electronic Sticky notes and users on Screen draw
`ing directly into the document. This type of document
`annotation enables users to mark and View documents on
`Screen as if the documents were in front of them on paper.
`However, the current annotation System requires users to
`annotate documents compatible with Specific word proceSS
`ing applications that lack portability. For example, if a user
`wanted to Send another perSon an annotated document, the
`user would have to e-mail the document as an attachment
`and hope the recipient has the same application Software to
`view the document
`Accordingly, there is a need in the art for a simple method
`and System of annotating and collecting Web-based docu
`ments. There is a need for the integration of any type of
`annotation, including text, ink, highlighter and audio,
`directly into Web-based documents, Such that any perSon
`with a Web browser may view the annotations, with or
`without the Software used to create the annotations. There is
`also a need for a method and System of annotating and
`collecting blank Web-based documents, effectively convert
`ing a Web browser into an online notebook/scrapbook.
`
`SUMMARY OF THE INVENTION
`The present invention relates to the integration of any
`annotation, including ink, highlighter, text-based notes and
`audio, directly into a Web-based document (WBD) dis
`played by a Web browser. This integration enables others to
`view the personalized annotated WBD, which retains its
`original active links and properties, over the Internet without
`the need for Specialized Software. Annotations are integrated
`into WBDs by freezing the WBD, overlaying an image file
`containing the annotations onto the WBD, and enabling
`browser events to pass through the image layer. Annotations
`may also be integrated into WBDS by using component
`object technology. The present invention collects and orga
`nizes annotated WBDs, and provides users with an intuitive
`Web-based interface for accessing, Viewing and Searching
`the annotated WBDs. Users may annotate blank WBDs,
`effectively converting their Web browsers into online
`notebookS/Scrapbooks. The present invention also provides
`users with many novel interface techniques, Such as dog
`ears and its associated navigation tools, splitting pages,
`turning pages, Selecting and copying Various portions of a
`WBD (including shaking out a copy), and marking menus
`Suited for right-handed or left-handed users.
`BRIEF DESCRIPTION OF THE DRAWINGS
`FIG. 1 is a block diagram depicting the internal Structure
`of a user's computer in accordance with an exemplary
`embodiment of the present invention.
`FIG. 2 is a block diagram depicting a network architecture
`that facilitates the Storing, Searching and transfer of anno
`
`15
`
`25
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`2
`tated WBDs in accordance with an exemplary embodiment
`of the present invention.
`FIG.3 depicts a data structure of a user record that is used
`to Store information in a user information database in
`accordance with an exemplary embodiment of the present
`invention.
`FIG. 4 depicts a data structure of a folder record that is
`used to Store information in a user folder database in
`accordance with an exemplary embodiment of the present
`invention.
`FIG.5 depicts a data structure of an annotation record that
`is used to Store information in an annotation database in
`accordance with an exemplary embodiment of the present
`invention.
`FIG. 6 depicts a data structure of a stroke record that is
`used to Store information in a Stroke database in accordance
`with an exemplary embodiment of the present invention.
`FIG.7 depicts a data structure of a note record that is used
`to Store information in a note database in accordance with an
`exemplary embodiment of the present invention.
`FIG. 8 depicts an annotated WBD in accordance with an
`exemplary embodiment of the present invention.
`FIG. 8A depicts a pop-up menu from an annotation
`toolbar in accordance with an exemplary embodiment of the
`present invention.
`FIG. 9 is a flowchart of steps for freezing WBD content
`in accordance with an exemplary embodiment of the present
`invention.
`FIG. 10 depicts the imposition of an image file onto a
`WBD in accordance with an exemplary embodiment of the
`present invention.
`FIG. 11 is a flowchart of steps for annotating into a Web
`browser in accordance with an exemplary embodiment of
`the present invention.
`FIG. 12 is a block diagram depicting the Structure of
`WBD layers that enable Web browser events to pass through
`annotations in accordance with an exemplary embodiment
`of the present invention.
`FIG. 13 is a flowchart of steps for passing Web browser
`events through annotations in accordance with an exemplary
`embodiment of the present invention.
`FIG. 14 is a flowchart of steps for drawing into a Web
`browser via a control object in accordance with an exem
`plary embodiment of the present invention.
`FIG. 15 illustrates a thumbnail margin in accordance with
`an exemplary embodiment of the present invention.
`FIG. 15A illustrates an online notebook/scrapbook in
`accordance with an exemplary embodiment of the present
`invention.
`FIG. 16 illustrates a search interface in accordance with
`an exemplary embodiment of the present invention.
`FIG. 17 is a flowchart of Steps for recognizing gestures in
`accordance with an exemplary embodiment of the present
`invention.
`FIG. 18 depicts a dog-eared WBD in accordance with an
`exemplary embodiment of the present invention.
`FIG. 19 depicts a navigation widget in accordance with an
`exemplary embodiment of the present invention.
`FIG. 20 is a flowchart of steps for creating, deleting, and
`navigating through dog-ears in accordance with an exem
`plary embodiment of the present invention.
`FIG. 21 depicts a flip to dog-ear gesture representation in
`accordance with an exemplary embodiment of the present
`invention.
`
`American Airlines, Ex. 1004, p.26
`American Airlines v. Advanced Transactions
`IPR2023-01366
`
`

`

`US 6,859,909 B1
`
`3
`FIG. 22 depicts a page split gesture representation in
`accordance with an exemplary embodiment of the present
`invention.
`FIG. 23 depicts a page turn gesture representation in
`accordance with an exemplary embodiment of the present
`invention.
`FIG. 24 depicts a Spiral page-turning gesture representa
`tion in accordance with an exemplary embodiment of the
`present invention.
`FIG. 25 is a flowchart of steps for turning multiple pages
`with a dynamic gesture in accordance with an exemplary
`embodiment of the present invention.
`FIG. 26 depicts three Selection gesture representations in
`accordance with an exemplary embodiment of the present
`invention.
`FIG. 27 is a flowchart of steps for performing a shaking
`out a copy gesture in accordance with an exemplary embodi
`ment of the present invention.
`FIG. 28 illustrates a tap and dwell menu in accordance
`with a pen-based embodiment of the present invention.
`DETAILED DESCRIPTION
`
`15
`
`4
`tated WBDs in accordance with an exemplary embodiment
`of the present invention. According to one embodiment,
`after user 200 annotates a WBD through Web browser 160,
`annotation software 170 transmits it to annotation repository
`240 (i.e., server arrangement) via network link 210a, Inter
`net service provider 220, network link 210b, Internet 230,
`and network link 210c. Network link 210 may include
`telephone lines, DSL, cable networks, T1 lines, ATM/
`SONNET, wireless networks, or any other arrangement that
`allows for the transmission and reception of network Signals.
`It should be noted that, technically, user computer 100,
`Internet service provider 220 and Web server cluster 250 are
`also part of the Internet 230 because of their connectivity.
`In an exemplary embodiment, annotation repository 240
`is a Website that receives the annotated WBD through its
`Web server cluster 250, which may comprise a collection of
`Web server computers working in tandem to distribute the
`load of network traffic. These Web servers include proces
`Sors and memory for executing program instructions as well
`network interfaces. Annotation repository 240 also
`comprises, among other components, user information data
`base 260, user folder database 270, annotation database 280,
`stroke database 290 and note database 295. Those skilled in
`the art realize that these single-record databases may be
`represented in many different ways, Such as individual tables
`in one or more relational databases. Although all annotation
`related information is Sent to annotation repository 240, this
`information may also be Stored in temporary Storage device
`140 or resident storage device 150 for efficiency reasons.
`FIG.3 depicts a data structure of a user record that is used
`to store information in user information database 260 in
`accordance with an exemplary embodiment of the present
`invention. Each user record represents a single unique user
`of annotation repository 240, and a user record is created
`when a user 200 creates an account online. In particular, user
`record 300 includes a user ID 310 field, a user name 320
`field, a user e-mail 330 field, a user password 340 field, and
`a list of folder IDs 350 field. User ID 310 contains an integer
`value that identifies each user record 300 in user information
`database 260. User name 320 contains a unique ASCII string
`that represents each user 200 associated with annotation
`repository 240. User e-mail 330 contains a unique ASCII
`string that represents the e-mail address of each user 200
`asSociated with annotation repository 240. User password
`340 contains a unique ASCII string chosen for security
`purposes by each user 200 associated with annotation
`repository 240. The password may be encrypted using any
`standard UNIX utility, such as salt or crypt. And list of folder
`IDS 350 contains a linked list of references to records stored
`in user folder database 270.
`FIG. 4 depicts a data structure of a folder record that is
`used to store information in user folder database 270 in
`accordance with an exemplary embodiment of the present
`invention. Each folder record contains a list of annotated
`WBDs that relate to a certain category for each user 200. In
`particular, folder record 400 includes a folder ID 410 field,
`a category 320 field, and a list of annotation IDs 430 field.
`Folder ID 410 contains an integer value that identifies each
`folder record 400 in user folder database 270. Category 420
`contains an ASCII String that describes the category of
`associated annotated WBDs. And list of annotation IDS 430
`contains a linked list of references to records Stored in
`annotation database 280.
`FIG.5 depicts a data structure of an annotation record that
`is used to store information in annotation database 280 in
`accordance with an exemplary embodiment of the present
`invention. Each annotation record contains all pertinent
`
`35
`
`40
`
`45
`
`25
`
`Introduction
`The present invention fulfills this need in the art by
`allowing computer users to integrate any annotation, includ
`ing ink, highlighter, text-based notes and audio, directly into
`a Web-based document (WBD) displayed by a Web browser.
`Users may create annotations on preexisting WBDS or new
`blank WBDs, and can collect and organize annotated WBDs
`through an intuitive Web-based interface using many novel
`interface techniques.
`INFRASTRUCTURE
`FIG. 1 is a block diagram depicting the internal Structure
`of a user's computer in accordance with an exemplary
`embodiment of the present invention. User computer 100
`may be a personal computer, handheld personal digital
`assistant ("PDA"), or any other type of microprocessor
`based device. User computer 100 may include a processor
`110, input device 120, output device 130, temporary storage
`device 140, resident storage device 150, and communication
`device 180. Input device 120 may include a keyboard,
`mouse, pen-operated touch Screen, voice recognition device,
`and any other device that provides input from a user. Output
`device 130 may include a monitor, printer, disk drive,
`Speakers, or any other device that provides tangible output
`to user. Temporary storage device 140 may include RAM,
`caches, and any other Volatile Storage medium that tempo
`50
`rarily holds data while processing it. Resident Storage device
`150 may include a hard drive, CD-ROM drive, tape drive,
`removable Storage disk, or any other nonvolatile data Stor
`age medium. Annotation software 170 and software for a
`Web browser 160 may reside in resident storage device 150.
`Annotation software 170 may include, among others, a Web
`browser plug-in, independent Software program or a feature
`of an operating system. Web browser 160 may include,
`among others, MicroSoft's Internet Explorer or Netscape
`Navigator. Communication device 180 may include a
`modem, network interface card, or any other device able to
`transmit and receive Signals over a network. One skilled in
`the art would appreciate that the components of user com
`puter 100 may also be connected wirelessly, possibly
`through an infrared connection.
`FIG. 2 is a block diagram depicting a network architecture
`that facilitates the Storing, Searching and transfer of anno
`
`55
`
`60
`
`65
`
`American Airlines, Ex. 1004, p.27
`American Airlines v. Advanced Transactions
`IPR2023-01366
`
`

`

`US 6,859,909 B1
`
`15
`
`S
`information relating to the appearance and make-up of a
`Single annotated WBD associated with annotation repository
`240. In particular, annotation record 500 includes an anno
`tation ID 510 field, an original WBD URL515 field, a URL
`content hash 520 field, an annotation size 525 field, a
`time/data stamp 530 field, a dog-ear 535 field, a pointer to
`original WBD 540 field, a pointer to image file 545 field, a
`pointer to thumbnail file 548, a list of stroke IDs 550 field,
`and a list of note IDs 555 field.
`Annotation ID 510 contains an integer value that identi
`fies each annotation record 500 in annotation database 280.
`Original WBD URL 515 contains an ASCII string repre
`senting the Internet address of the unannotated WBD. URL
`content hash 520 contains an integer value that results from
`the execution of a Standard hash function on the contents of
`the WBD. Annotation repository 240 uses this value to
`generate an Internet address for the annotated WBD that can
`be mapped to the annotated WBDs location in annotation
`database 280. Annotation size 525 contains an integer value
`representing the size in bytes of the annotated WBD. Time/
`date stamp 530 contains an ASCII string representing the
`time and date that user 200 generated the annotated WBD.
`Dog-ear 535 stores a boolean value representing whether
`annotated WBD is marked as a dog-ear (“true") or not
`(“false"); the default value is “false.” Pointer to original
`WBD 540 stores the file location of the unannotated WBD,
`25
`pointer to image file 545 stores the file location of the image
`file containing the appearance of the annotations, and
`pointer to thumbnail file 548 stores the file location of the
`image file containing the miniaturized annotated WBD. List
`of stroke IDs 550 contains a linked list of references to
`records stored in