(12) United States Patent
`Dey et al.
`
`USOO6965890B1
`(10) Patent No.:
`US 6,965,890 B1
`(45) Date of Patent:
`Nov. 15, 2005
`
`(54) HYPER VIDEO: INFORMATION RETRIEVAL
`USING REALTIME BUFFERS
`
`(75) Inventors: Jayanta Kumar Dey, Cambridge, MA
`(US); Haiping Xu, Acton, MA (US)
`(73) Assignee: Verizon Laboratories Inc., Waltham,
`MA (US)
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 111 days.
`
`(*) Notice:
`
`(21) Appl. No.: 10/264,117
`(22) Filed:
`Oct. 3, 2002
`Related U.S. Application Data
`(63) Continuation of application No. 09/431,293, filed on Oct.
`29, 1999, now Pat. No. 6,493,707.
`(51) Int. Cl." ................................................ G06F 17/30
`(52)
`707/4; 707/3
`(58) Field of Search .......................... 707, 3.7is
`(56)
`References Cited
`U.S. PATENT DOCUMENTS
`5,504.675 A
`4/1996 Cragun et al. .............. 364/401
`5,515,098 A
`5/1996 Carles ........................... 348/8
`5,636,346 A
`6/1997 Saxe .......
`... 395/201
`5,708.845 A
`1/1998 Wistendahl et al. ........ 395/806
`SSS A 19. 19.
`McMilan - - - - - - - - - - - - - - - - - - - - - - 22.
`2 - 1 - 2
`I . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
`5933,822 A 8/1999 Braden-Harder et al. ...... 707/5
`5933832 A 8/1999 Suzuoka et all
`707/101
`so37422 A 8/1999 Nelson et al... 707,531
`5.941.944 A 8/1999 Messerly .................... 709/203
`5.987,457. A 11/1999 Ballard .......................... 707/5
`5.991,735 A 11/1999 Gerace ...
`... 705/10
`6,029,167 A
`2/2000 Evans ........................... 707/4
`E. A 3.2:
`s al. .............. 22,
`2Y- Y-2
`ISCIl C al. . . . . . . . . . . . . . . . . . .
`6,115,709 A 9/2000 Gilmour et al. ............... 707/9
`6,141,001. A 10/2000 Baleh............ so
`6,145,000 A 11/2000 Stuckman et al. .......... 709/219
`6,154,737 A 11/2000 Inaba et al. .................... 707/3
`
`6/1997 Wellner - - - - - - - - - - - - - - - - - - - - - - - - - 348/7
`
`5,640,193 A
`
`3/2001 Logan et al. ............... 707/501
`6,199.076 B1
`6,532.469 B1 * 3/2003 Feldman et al. ..
`... 707/102
`6,665,661 B1 * 12/2003 Crow et al. .................... 707/3
`OTHER PUBLICATIONS
`Yamuna et al. (Similarity-based retrieval of temporal docu
`ment), ACM Press, Year 2000, pp. 243-246.*
`Thorton et al. (Applying local search to temporal reasoning),
`Temporal representation and Reasoning, Date Jul. 7-9,
`2002.
`Chekuri et al., Web Search Using Automatic Classification,
`6" Intl. WWW conference, Apr. 7–11, 1997, http://www/
`scope.gmd.de/info/www6/posters/725/web search.html
`(11 Pages).
`U.S. Appl. No. 09/430,437, filed Oct. 29, 1999, Dey et al.
`U.S. Appl. No. 09/430,597, filed Oct. 29, 1999, Dey et al.
`U.S. Appl. No. 09/429,731, filed Oct. 29, 1999, Dey et al.
`U.S. Appl. No. 09/431,292, filed Oct. 29, 1999, Dey et al.
`* cited by examiner
`Primary Examiner Jean M. Corrielus
`Assistant Examiner-Baoquoc N. To
`(74) Attorney, Agent, or Firm-Leonard C. Suchyta, Esq.;
`Joel Wall, Esq.; Foley, Hoag & Eliot LLP
`(57)
`ABSTRACT
`Disclosed is a method and device for Selecting documents,
`Such as Web pages or Sites, for presentation to a user, in
`response to a user expression of interest, during the course
`of presentation to the user of a document, Such as a video or
`audio Selection, whose content varies with time. The method
`documents related to the portion of the temporal document
`in which the user has expressed interest. The method gen
`erates the Search query to use to select documents by
`reference to text associated with the portion of the temporal
`document in which the user has expressed interest, as by
`using the closed caption teSt associated with the video, or by
`using Speech recognition techniques. The method further
`uses a weighting function to weigh the terms used in the
`Search query, depending on their temporal relationship to the
`user expression of interest. The method uses a buffer to Save
`necessary information about the temporal occurrence of
`terms in the temporal document.
`38 Claims, 10 Drawing Sheets
`
`takes advantage of information retrieval techniques to Select
`
`ckoos REMANDER
`NESTEDTERM
`USS
`
`CHOOSE OF
`INVERTEDTERMSTS
`
`SS
`
`CH00SEM0ST
`SISNFANTTER NOT
`YETANAYZE)
`
`S10
`
`coxSULTINNERTED
`TER
`LSFOR
`MOST SIGNIFICANTTERM
`NOTYETANALY2ED
`
`65
`
`
`
`
`
`
`
`
`
`
`
`
`
`HAVE
`REANDER
`WERTED
`TESTS9EE
`SAS
`
`THERE
`NOOCUMENTS
`ONLSTWT
`syFFICIENT
`ses
`
`FaacD0CUENT
`(NHTSALCULATE
`FUSCOE
`
`RETURNCAENTS
`WITH HIGEST
`SCORES
`
`
`
`Page 1
`
`AMAZON EX. 1023
`Amazon v. CustomPlay
`US Patent No. 9,124,950
`
`

`

`U.S. Patent
`
`Nov. 15, 2005
`
`Sheet 1 of 10
`
`US 6,965,890 B1
`
`
`
`
`
`
`
`WORKSTATION
`
`LOCAL AREA
`NETWORK
`
`
`
`F.G. 1
`
`Page 2
`
`

`

`U.S. Patent
`
`Nov. 15, 2005
`
`Sheet 2 of 10
`
`US 6,965,890 B1
`
`O6
`
`O5
`
`O4
`
`O3
`
`.02
`
`O1
`
`O
`
`30
`
`10
`5
`20
`25
`TIME PRIOR TO SIGNAL
`(SECONDS)
`
`5
`
`O
`
`FIG. 2
`
`Page 3
`
`

`

`U.S. Patent
`
`Nov. 15, 2005
`
`Sheet 3 of 10
`
`US 6,965,890 B1
`
`06
`
`05
`
`.04
`
`O3
`
`O2
`
`O1
`
`O
`
`30
`
`10
`15
`2O
`25
`TIME PRIOR TO SGNAL
`(SECONDS)
`
`5
`
`O
`
`FIG. 3
`
`Page 4
`
`

`

`U.S. Patent
`
`Nov. 15, 2005
`
`Sheet 4 of 10
`
`US 6,965,890 B1
`
`O6
`
`O5
`
`.04
`
`O3
`
`.02
`
`O
`
`O
`
`30
`
`10
`5
`20
`25
`TIME PRIOR TO SIGNAL
`(SECONDS)
`
`5
`
`O
`
`FIG. 4
`
`Page 5
`
`

`

`U.S. Patent
`
`Nov. 15, 2005
`
`Sheet 5 of 10
`
`US 6,965,890 B1
`
`O6
`
`.05
`
`04
`
`03
`
`.02
`
`O1
`
`O
`
`30
`
`O
`15
`2O
`25
`TIME PROR TO SIGNAL
`(SECONDS)
`
`5
`
`O
`
`FIG. 5
`
`Page 6
`
`

`

`U.S. Patent
`
`Nov. 15, 2005
`
`Sheet 6 of 10
`
`US 6,965,890 B1
`
`ORIGINAL
`DOCUMENTS
`
`805
`
`815
`
`825
`
`DOCUMENT 1
`
`DOCUMENT 2
`
`
`
`TERM 1, TERM2,
`TERM 3, TERM 1,
`TERM 4, TERM 2,
`TERM 5,...,
`TERM m, ...
`
`TERM 3, TERM 1,
`TERM m, TERM
`m+ 1,
`
`DOCUMENT 3
`TERM m,
`TERM m+4,
`TERM 1, TERM 1,
`
`12 OCCURRENCE(S)
`2, 1 OCCURRENCE(S)
`3, 2 OCCURRENCE(S)
`
`TER M 3
`846
`
`OCCURRENCE(S)
`OCCURRENCE(S)
`
`835
`
`840
`
`845
`
`INVERTED
`TERM
`LISTS
`
`F.G. 6
`
`Page 7
`
`

`

`U.S. Patent
`
`Nov. 15, 2005
`
`Sheet 7 of 10
`
`US 6,965,890 B1
`
`LOOKUP TABLE
`
`1 OO
`
`DOCUMENT #
`
`DOCUMENT URL
`
`
`
`1
`2
`3
`
`URL
`UR 2
`URL3
`
`LOOKUP TABLE
`
`102 Y
`
`FG. 7
`
`
`
`
`
`TERM # TERM
`
`tenureau ADDRESS
`
`TL
`
`NUMBER OF DOCUMENTS
`
`CoNTAINING TERM
`
`MAX
`
`1
`2
`
`“TERM 1"
`"TERM 2"
`"TERM 3" |
`
`LOC. 1
`LOC. 2
`LOC. 3
`
`N
`N2
`N3
`
`TFM1
`TFM2
`TFM3
`
`
`
`
`
`Page 8
`
`

`

`U.S. Patent
`
`Nov. 15, 2005
`
`Sheet 8 of 10
`
`US 6,965,890 B1
`
`ORIGINAL
`DOCUMENTS
`
`805
`
`815
`
`825
`
`DOCUMENT
`
`DOCUMENT 2
`
`TERM 1, TERM2,
`TERM3, TERM 1,
`TERM 4, TERM2,
`TERM 5,...,
`TERM m, ...
`
`TERM3, TERM 1,
`TERMm, TERM
`m+1,
`
`DOCUMENT 3
`
`TERM m,
`TERM m+4,
`TERM 1, TERM 1,
`TERM m+2, ...
`
`DOCUMENT 1
`
`
`
`DOCUMENT 2
`
`DOCUMENT 3
`
`12 OCCURRENCE(S)
`266RRENCES
`1, 1 OCCURRENCE(S)
`1, 1 OCCURRENCE(S)
`11 OCCURRENCE(S)
`
`1, 1 OCCURRENCE(S)
`21 occuRRENCES
`
`1, 200CURRENCES)
`
`mn, 1 OCCURRENCES) in 95NCES)
`2, 1 OCCURRENCE(S)
`OCCURRENCES) || 3:AGENES
`1,
`
`mn, 1 OCCURRENCE(S)
`
`-
`
`N-N-1
`
`COMPRESSED
`DOCUMENT
`SURROGATES
`
`FG. 8
`
`Page 9
`
`

`

`U.S. Patent
`
`Nov. 15, 2005
`
`Sheet 9 of 10
`
`US 6,965,890 B1
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`CONSULT COMPRESSED
`DOCUMENT SURROGATE
`OUERY
`FOR TERM N SEARCH
`FOUND IN DOCUMENT
`
`
`
`
`
`525
`
`CONSULT INVERTED TERM
`LIST AND/OR LOOKUP TABLE
`FOR THAT TERM, AND
`SCORE
`ADJUST DOCUMENT
`ASAPPROPRIATE
`
`530
`
`
`
`ARE
`ANY TERMS
`FOUND
`N SEARCH OUERY
`IN COMPRESSED DOCUMENT
`OT
`SURROGATE N
`YET ANALYZEO
`?
`
`
`
`FG. 9
`
`PROCESS IS
`COMPLETED
`
`Page 10
`
`

`

`U.S. Patent
`
`Nov. 15, 2005
`
`Sheet 10 0f 10
`
`US 6,965,890 B1
`
`660
`
`REMAINDER
`CSE TERM
`LISTS
`
`START
`
`600
`
`1
`
`CHOOSETOP
`NVERTED TERM ISTS
`
`605
`
`CHOOSE MOST
`SIGNIFICANT TERM NOT
`YET ANALYZED
`
`CONSULT INVERTED TERM
`LST FOR
`MOST SIGNIFICANT TERM
`NOTYETANALYZED
`
`610
`
`615
`
`CALCULATE SCORE SDT FOREACH
`DOCUMENT ON INVERTED TERMLST,
`AND ADD SDT TO SCORE OF THAT
`DOCUMENT ON HIT LIST
`
`625
`
`CALCULATESMAX
`
`630
`
`N DOCUMENTS
`ON LST WITH
`SUFFICIENT
`SCORES
`
`ARE
`THERE
`ANY TERMS
`NOT YET
`ANALYZED
`
`HAVE
`REMAINDER
`NVERTED
`TERMLSTSBEEN
`ANALYZED
`
`655
`
`NO
`
`F.G. 10
`
`FOREACH DOCUMENT
`ONHIT LIST, CALCULATE
`FINAL SCORE
`
`ReturninDocuMENTS
`WITH HIGHEST
`SCORES
`
`645
`
`DONE
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`Page 11
`
`

`

`1
`HYPER VIDEO: INFORMATION RETREVAL
`USING REALTIME BUFFERS
`
`US 6,965,890 B1
`
`CROSS-REFERENCES TO RELATED
`APPLICATIONS
`The present application is a continuation of application
`Ser. No. 09/431,293, entitled Hypervideo: Information
`Retrieval Using Realtime Buffers, and filed on Oct. 29,
`1999, now U.S. Pat. No. 6,493,707 which application is
`hereby incorporated herein by reference.
`This application is also related to the following
`co-pending applications:
`
`Title
`
`Application
`Ser. No.
`
`Hypervideo: Information Retrieval Using 09/430,437
`Time-Related Multimedia
`Hypervideo: Information Retrieval Using 09/430,597
`Text from Multimedia
`Hypervideo: Information Retrieval Using 09/431,292
`Multimedia
`O9/429,731
`Hypervideo: Information Retrieval at User
`Request
`
`Filing Date
`
`Oct. 29, 1999
`
`Oct. 29, 1999
`
`Oct. 29, 1999
`Oct. 29, 1999
`
`All applications noted above are commonly owned by the
`assignee of the instant application.
`TECHNICAL FIELD
`This invention relates to techniques for retrieving material
`on the World WideWeb, and more particularly to methods
`of retrieving Web pages of interest to a user which relate to
`temporal material Such as Video programming.
`BACKGROUND OF THE INVENTION
`The Internet, of which the World Wide Web is a part,
`includes a Series of interlinked computer networks and
`servers around the world. Users of one server or network
`connected to the Internet may send information to, or acceSS
`information on, other networks or Servers connected to the
`Internet by the use of various computer programs which
`allow Such access, Such as Web browsers. The information
`is Sent to, or received from, a network or Server in the form
`of packets of data.
`The World Wide Web portion of the Internet comprises a
`Subset of interconnected Internet Sites which may be char
`acterized as including information in a format Suitable for
`graphical display on a computer Screen. Each Site may
`include one or more Separate pages. Pages, in turn, may
`include links to other pages within the Site, or to pages in
`other Web sites, facilitating the user's rapid movement from
`one page or site to another.
`A number of the Sites and pages accessed through the Web
`may consist entirely of “static” displays of text and/or
`images. These displayS may reside on one or more host
`Servers or networks, and may be accessed through the
`Internet for Storage and/or display on a remote Server or
`network. Other Sites or pages may have changing advertise
`ments or other Similar material as well as “static' displays of
`text and/or images.
`There are a number of techniques for permitting a user,
`while viewing one page or site on the Web, to request and
`be given access to other material that relates to the material
`being viewed, which can be applied when the material being
`Viewed contains Static text or image displays in whole or in
`part.
`
`1O
`
`15
`
`25
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`2
`In addition to accessing Static displays of text and/or
`images on the Web, it also may be possible to access
`material on the Web which is dynamic or changing. Such
`material will be referred to as “temporal documents” to
`reflect the fact that, unlike Static material, their content as
`made available to or perceived by a user may change with
`the passage of time. Examples of Such temporal documents
`are multimedia material Such as Video and audio
`programming, but there are other types of temporal docu
`ments as well. For example, the text of news bulletins, Stock
`quotations Such as would be seen on a "ticker tape', or Sports
`Scores may be made available; material Such as this by its
`nature also may be changing as it is viewed, either because
`the underlying information is changing, or because the
`information is "scrolled’ across the user's monitor, thus
`appearing as constantly changing with time. Other types of
`dynamic or changing material will also be apparent to one of
`ordinary skill in the art.
`Temporal documents may have been previously created
`and Stored on a server for later access (Such as a movie, or
`a recording of a previously-ocurring sports event) or a
`temporal document may reflect an event that is occurring
`“live” at the time the temporal document is transmitted over
`the Internet (Such as a live news broadcast or sports event,
`or a Stock ticker displaying real-time Stock transaction
`information).
`Whether the temporal document is previously-created or
`is being accessed live, it is useful to have a technique to
`facilitate a user obtaining material that relates to a portion of
`the temporal document he is viewing or listening to. Because
`the material is changing, however, Some of the techniques
`that may be used to provide access to material that is related
`to a Static page being Viewed, may not be readily applicable
`to temporal documents.
`Some previous methods of providing additional material
`related to changing content Such as Video programming have
`relied upon the prior manual choice of other Web
`documents, Such as pages or Sites, to be associated with
`particular portions of the Video content. Then, when a
`particular portion of the video programming is reached, the
`related Web page or document may automatically be pre
`Sented to the user, or the user may be informed of the
`availability of a link to the related material, and offered the
`choice of accessing it. Alternatively, no information about
`related material may be presented until or unless a user
`indicates interest during a particular Segment of the Video
`material (as by "clicking with a mouse, or pressing a
`button); upon an expression of interest, the particular other
`Web page previously chosen as related to this portion of the
`Video material may be presented to the user.
`This method of associating related material to a temporal
`document has drawbacks, however. Because it requires the
`preSelection of the associated material, it cannot be utilized
`with live material, or with material that has not been
`previously analyzed for the purpose. It also may be costly,
`in that it may require intensive manual manipulation of the
`multimedia material to choose other Web pages to associate
`with each portion of the Video or audio material, and to carry
`out the association. It also may be rigid, in that once the
`Selection is made it may remain unchanged regardless of
`whether other more appropriate related material becomes
`available. It may be both expensive and time-consuming to
`make changes once links have been established.
`Additionally, this method may offer a very limited choice to
`the user in that it may not be practical to offer a large number
`of links at each portion of the Video or audio material.
`Thus, there is a need for a method or device for permitting
`a user to obtain access to other material that is related to a
`
`Page 12
`
`

`

`3
`portion of a temporal document (Such as a video or audio
`program) being accessed on the Web, where the Selection of
`the related material offered to the user is not made in
`advance, but is done automatically at the time the user
`expresses an interest in obtaining Such material. Such a
`method or device makes “hyperVideo' a practical concept.
`One aspect of this need is a need for determining the
`portion of the temporal document about which the user
`would like to obtain additional information. In the case of a
`“static' display of material as might be presented to the user
`on a computer monitor, it may be possible to have the user
`indicate the material of interest by using a mouse or other
`Similar Selection device to maneuver a cursor on the monitor
`until it is Superimposed on the portion of the display of
`interest, and then to "click” on the material of interest. In the
`case of a changing display, Such as Video, that may not be
`practical. For example, because it may take a certain amount
`of time for the user to decide that he is interested in obtaining
`additional material, and a certain amount of time to maneu
`ver the mouse or other signaling device to indicate interest,
`the expression of interest may be delayed by a certain
`amount from the actual material in which the user is inter
`ested.
`Another aspect of this need is a need for determining what
`other material is related to the material in which the user has
`expressed an interest. In the case of a Static display which
`includes a display of text, it may be possible to have the user
`indicate the specific material in which he is interested (as by
`using a mouse to maneuver a cursor to the word or term
`displayed on the Screen), and then to use that specific text as
`the basis of a Search query using a conventional Web Search
`engine. But in the case of Video material, that may not be
`possible.
`BRIEF DESCRIPTION OF DRAWINGS
`The above-mentioned and other features of the invention
`will now become apparent by reference to the following
`description taken in connection with the accompanying
`drawings in which:
`FIG. 1 is a Schematic diagram of an embodiment of a
`computer System that may be operated according to the
`present invention.
`FIG. 2 is a diagram illustrating the weight to be assigned
`to different temporal portions of material Such as Video,
`based upon a user response at time to according to one
`embodiment of the present invention.
`FIG. 3 is a diagram illustrating the weight to be assigned
`to different temporal portions of material Such as Video,
`based upon a user response at time to, according to another
`embodiment of the present invention.
`FIG. 4 is a diagram illustrating the weight to be assigned
`to different temporal portions of material Such as Video,
`based upon a user response at time to, according to a further
`embodiment of the present invention.
`FIG. 5 is a diagram illustrating the weight to be assigned
`to different temporal portions of material Such as Video,
`based upon a user response at time to, according to a further
`embodiment of the present invention.
`FIG. 6 illustrates a conventional (prior art) relationship
`between documents and inverted term lists.
`FIG. 7 illustrates conventional (prior art) lookup tables
`which may be used in conjunction with inverted term lists.
`FIG. 8 illustrates a relationship between documents and
`compressed document Surrogates
`FIG. 9 is a flow chart which illustrates a process by which
`a document Score may be calculated, using compressed
`document Surrogates.
`
`15
`
`25
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`US 6,965,890 B1
`
`4
`FIG. 10 is a flow chart which illustrates a process by
`which a Search query may be carried out to identify material
`relating to a portion of a temporal document in which a user
`has expressed an interest, using compressed document Sur
`rogates according to the present invention.
`SUMMARY OF THE INVENTION
`According to the present invention, finding documents
`which relate to a portion of a temporal document includes (a)
`in response to a signal of interest at a particular time during
`the temporal document, identifying a portion of the temporal
`document for which related documents are to be found, (b)
`Selecting text associated with the portion of the temporal
`document identified, by reference to a buffer wherein is
`Stored each term in text associated with the temporal
`document, together with a time in the temporal document at
`which the term occurs, (c) weighting each term in the text
`Selected by a function according to the time at which the
`term occurs relative to the time at which the Signal of interest
`occurs, and (d) finding the related documents by use of
`information retrieval techniques applied to the text Selected.
`Each term in the text associated with the temporal document
`may be determined as that portion of the document is
`accessed, and may be Stored in a buffer together with the
`time in the temporal document at which the term occurs. The
`times at which the terms occur may be determined using the
`Synchronization information contained in the temporal
`document. The temporal document may use the SMIL
`Standard for Synchronization of multimedia elements in a
`document. The buffer may have a fixed size such that it has
`Sufficient capacity to hold all the terms which may occur in
`the temporal document during the time period that extends
`from when a signal of interest occurs to the earliest time
`prior to that for which the weighting function is nonzero.
`The fixed size of the buffer may be 8 kilobytes.
`Alternatively, the size of the buffer may be varied such that
`it has Sufficient capacity at any given time to hold all terms
`which have occurred in the temporal document during a time
`period that extends back from the given time to an earliest
`time for which the weighting function would be nonzero for
`a signal of interest occurring at the given time. Terms and
`associated times may be stored in the buffer as if the buffer
`were a continuous loop, Such that the buffer is filled in order,
`and when full a next entry is inserted at the beginning of the
`buffer. The temporal document may be video or audio
`material. The text stored in the buffer may be determined by
`the application of Speech recognition techniques to the audio
`component of the portion of the temporal document
`identified, or may be the closed-caption text associated with
`the portion of the temporal document identified. The tem
`poral document may include text, which text appearing to
`the user may vary with time, and the text Selected may be
`that portion of the temporal document identified. The text
`may include news bulletins, weather, Sports Scores or Stock
`transaction or pricing information. The temporal document
`may be being generated as it is being accessed. The weight
`ing function W(t) may be equal for all times between a time
`t before the Signal of interest is given and a time t before
`the Signal is given, and may be Zero for all other times. T
`may be 2 seconds and t 30 seconds. Alternatively, W(t) may
`be equal for all times between a time t before the signal of
`interest is given and a time t before the Signal is given, and
`may decrease from t until the time of the signal, and
`increase from a time t before the Signal is given to the time
`ta, and may be Zero for all other times. T, t and t may have
`specified values. W(t) may decrease linearly from the time
`t until the time of the Signal, and increase linearly from the
`
`Page 13
`
`

`

`15
`
`S
`time t before the signal is given to the time t. W(t) may be
`a double exponential function of time. W(t) may equal 0 for
`all times earlier than 30 seconds before the signal of interest
`is given, and later than the Signal of interest. The related
`documents may be accessed through the Internet. The
`related documents may be Selected from among a collection
`of documents which may be accessed through the Internet,
`by utilizing databases comprising information about the
`collection. The related documents may be selected from the
`collection according to the Scores achieved when evaluating
`documents in the collection according to a formula giving
`Scores to documents depending upon the occurrence in the
`documents of terms which occur in text associated with the
`portion of the temporal document identified, where each
`term is weighted by the weighting function according to the
`time at which the term occurs relative to the time at which
`the Signal of interest occurs. The terms which occur in the
`text which is associated with the portion of the temporal
`document identified, and the times at which the terms occur,
`may be obtained by reference to the buffer which contains
`the terms and the times in the Video or audio material at
`which the terms occur. A predetermined number of
`documents, 1000, may be selected. A score S, of a docu
`ment D in the collection may be determined by crediting the
`document D, for each term T in the temporal portion of the
`document identified which occurs in the document D, with
`an amount proportional to the weighting function, Robert
`Son's term frequency TF and to IDF. The determination
`of the documents in the collection which receive the highest
`Scores may be carried out using compressed document
`Surrogates. The determination of the documents in the
`collection which receive the highest scores may be carried
`out by a server which is distinct from the server which
`receives the Signal of interest.
`DETAILED DESCRIPTION OF THE
`PREFERRED EMBODIMENT(S)
`Referring to FIG. 1, a computer System 1 includes a
`WorkStation 2 having local Storage 3. The WorkStation may
`also be connected to a local area network 4 and may access
`to the Internet 5. The Internet 5 may include or be coupled
`to remote Storage 6. The WorkStation 2 may be any one of a
`variety of commercially available computers capable of
`providing the functionality described in more detail below.
`The local storage 3 may include ROM, RAM, a hard disk,
`a CD, or any other media capable of containing data and/or
`programs for the WorkStation 2 or other data. The local area
`network 4, which is coupled to and eXchanges data with the
`WorkStation, may also contain data and/or program infor
`mation for use by the workstation 2. The Internet 5 may be
`accessed in a conventional manner by the WorkStation 2.
`Alternatively, the WorkStation 2 may access the Internet 5
`through the local area network 4, as shown by the dotted line
`of FIG.1. The remote storage 6 may also contain data and/or
`program information for the WorkStation 2 or may contain
`other information, as will become apparent from the descrip
`tion below.
`The System described herein permits a user (utilizing the
`computer System 1 which includes the workstation 2) who
`has accessed the Internet 5, either directly or through the
`local area network 4, to be given access to other material that
`is related to a temporal document, Such as but not limited to
`Video or audio material, the user is accessing. In one
`embodiment, the System includes Software written in the
`Java language, running on a Hewlett Packard Server con
`65
`nected to the Internet, as well as Software written in the C
`language and in PERL running on an SGI O2 Server con
`
`45
`
`50
`
`55
`
`60
`
`US 6,965,890 B1
`
`25
`
`35
`
`40
`
`6
`nected to the Internet. Of courses, it will be appreciated by
`one of ordinary skill in the art that the system may be
`implemented using a variety of computers and programming
`languages.
`The System may be accessed by the user through the
`Internet 5 from his workstation 2 using a Web browser of
`conventional design, as would be familiar to one of ordinary
`skill in the art. The user then accesses a temporal document.
`In one embodiment, the temporal document is obtained from
`a collection of temporal documents previously prepared by
`the System and placed in a Video library made available
`through a video Server maintained in connection with the
`System. In this embodiment, the user may be permitted to
`choose the document in any one of a number of ways which
`will be known to one of ordinary skill in the art. The user
`may be given a list of documents which are available, and
`permitted to choose one, by clicking on it or indicating his
`interest in any one of a number of alternative ways which
`will be known to one of ordinary skill in the art.
`Alternatively, the user may be invited to Search by using
`Search engine or Search query techniqueS Such as will be
`familiar to one of ordinary skill in the art. Still other methods
`to permit the user to choose a document from among those
`in the library will be known to one of ordinary skill in the
`art. The user then may view (or listen to) the temporal
`document chosen through his work Station 2 connected to
`the Internet 5.
`In another embodiment, the temporal document may be
`obtained from another source on the Web. In this
`embodiment, the user may be permitted to employ a Search
`engine which is maintained as part of the System to find and
`retrieve a document to the System. The Search engine
`employed may be any one of a number of a type which will
`be familiar to one of ordinary skill in the art. The user then
`may view (or listen to) the temporal document chosen
`through his work station 2 connected to the Internet 5.
`In another embodiment, the temporal document may be
`obtained from another source on the Web. In this
`embodiment, the user may be permitted to employ a Search
`engine on his work Station 2 connected to the Internet 5 to
`retrieve and view (or listen to) the temporal document
`chosen. The Search engine employed may be any one of a
`number of a type which will be familiar to one of ordinary
`skill in the art. The user then may view (or listen to) the
`temporal document chosen through his work Station 2 con
`nected to the Internet 5.
`The system utilizes IR (information retrieval) techniques
`to Select the related material when interest in having acceSS
`to Such material is indicated. The System analyzes the
`content of a portion of the temporal document as to which
`the interest has been indicated, rather than pre-storing links
`to material which is determined to be related in advance.
`The System may be utilized in connection with any
`material which has a characteristic that, when accessed by a
`user or viewer through the computer System 1 which may
`include the WorkStation 2, it changes with time. This
`includes but is not limited to Video material and audio
`material, Such as movies, news programs, and Sports events.
`It may also include, for example, textual news bulletins that
`are displayed, either alone or Superimposed on other content,
`or Stock quotations or Sports Scores. These materials may be
`changing with time in that they are Scrolled acroSS the
`monitor for reading purposes, So that the portion of them
`accessed by the user changes with time.
`If the material accessed is video material, whether col
`lected into a Video library and previously Stored in a video
`
`Page 14
`
`

`

`US 6,965,890 B1
`
`7
`Server, or accessed from another location on the Internet, the
`Video material may have been previously broadcast, and
`each Video may have associated there with closed captions
`which contain text that accompanies the video. The closed
`caption material may include the text of dialogue, or Spoken
`words that accompany the Video and constitute the audio
`track.
`Included in the System is a technique that may be used to
`indicate when a portion of the temporal document in which
`there is interest has been reached. That a portion of the
`temporal document as to which additional, related material
`is desired has been reached, is indicated by means of a
`particular, preselected response being made after the portion
`of the document is displayed to the user. In one embodiment,
`a mouse is clicked, while in other embodiments Software
`which recognizes and responds to voice commands may be
`employed, a particular key (or any key) on a keyboard may
`be depressed, or a button on a joystick may be pressed. Other
`methods of providing a Signal to a computer System, known
`to one of ordinary skill in the art, may also be utilized.
`Further included in the System is a technique which may
`be used, when a Signal indicating interest in a portion of the
`document is given, to facilitate the determination of the
`portion of the temporal document in which the interest has
`been indicated, by utilizing the time at which the Signal
`indicating interest has been given.
`It is understood that a user may not be able to instanta
`neously think about the changing material that is being
`presented, make a decision that he is interested, and give the
`required Signal. Moreover, it is understood that while the
`user Sometimes may make a decision about interest based
`upon what appears or is heard at a particular instant, at other
`times the decision may be based upon a Sequence of material
`presented over a period of time, rather than based upon the
`material at a particular instant.
`For these reasons, the technique used in the System does
`not treat the content of the temporal document at the instant
`the Signal is given as that portion of the temporal document
`in which there is interest, and therefore as a basis for finding
`related material. Rather, it is assumed that there is a delay
`between the material of interest first being presented to the
`user, and the indication of interest, and it is further assumed
`that the user is interested in mate