`
`EXHIBIT 12
`
`DECLARATION OF MELODY DRUMMOND HANSEN IN SUPPORT OF DEFENDANT’S
`RESPONSIVE CLAIM CONSTRUCTION BRIEF
`
`Case No. 5:15-CV-02008-EJD
`
`
`
`
`
`
`
`
`
`
`
`Case 5:15-cv-02008-EJD Document 83-12 Filed 04/12/16 Page 2 of 13
`I lllll llllllll Ill lllll lllll lllll lllll lllll 111111111111111111111111111111111
`US005818441A
`[11] Patent Number:
`[45] Date of Patent:
`
`United States Patent [19J
`Throckmorton et al.
`
`5,818,441
`Oct. 6, 1998
`
`[54] SYSTEM AND METHOD FOR SIMULATING
`TWO-WAY CONNECTIVITY FOR ONE WAY
`DATA STREAMS
`
`[75]
`
`Inventors: John A. Throckmorton, Lake Oswego;
`Edward R. Harrison; Burt Perry,
`both of Beaverton, all of Oreg.
`
`[73] Assignee: Intel Corporation, Santa Clara, Calif.
`
`[21] Appl. No.: 490,822
`
`[22] Filed:
`
`Jun. 15, 1995
`
`Int. Cl.6
`[51]
`....................................................... H04N 7/10
`[52] U.S. Cl. ................................. 345/328; 348/13; 348/7;
`348/12
`[58] Field of Search ..................................... 345/327, 328;
`348/7, 12, 13, 473, 714
`
`[56]
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`5,347,304
`5,499,046
`5,513,180
`5,583,864
`5,594,491
`
`9/1994 Moura et al. ............................. 348/12
`3/1996 Schiller et al.
`........................... 348/12
`4/1996 Miyake et al. .............................. 348/7
`12/1996 Lightfoot et al. . ... ... ... .... ... ... ... ... . 348/7
`1/1997 Hodge ... ... ... ... .... ... ... ... ... ... .... ... ... 348/7
`
`5,613,192
`
`3/1997 Ikami et al.
`
`................................ 348/9
`
`Primary Examiner-Victor R. Kostak
`Attorney, Agent, or Firm-Blakely, Sokoloff, Taylor &
`Zafman
`
`[57]
`
`ABSTRACT
`
`A system supplying information associated with a broadcast
`television program to a consumer such that said consumer
`perceives the associated data as the result of two way
`interactivity with external sources of data such as online
`services or the Internet. The system includes equipment for
`inserting the associated data into the vertical blanking inter(cid:173)
`val of the television signal on the supplier side of the system.
`On the receiving side, the system includes a personal com(cid:173)
`puter capable of receiving the television program and storing
`the associated data locally. The consumer may then interact
`with the stored associated data in an apparently two way
`interactive manner. Additional interactivity may be achieved
`by adding an actual two way communication channel to the
`personal computer so that online services or the Internet may
`be accessed. This two way communication channel is made
`particularly effective if the associated data contains pointers
`to locations in the online services or the Internet which are
`particularly relevant to the television program.
`
`42 Claims, 5 Drawing Sheets
`
`IO
`Primary Data
`Stream
`Generation
`
`12
`
`20
`
`26
`
`Data
`Synchronizer
`
`Sequencer
`
`Encoder
`
`22
`
`17
`
`24
`
`Associated
`Data
`Generation
`16
`
`18
`
`42
`
`36
`
`Human
`Interface
`
`38
`
`Processor
`
`Printer
`
`44
`
`Receiver
`
`Memory
`
`40
`
`
`
`Case 5:15-cv-02008-EJD Document 83-12 Filed 04/12/16 Page 3 of 13
`
`FIG.1
`
`Data Stream
`Creation
`
`Data Stream
`-
`r Delivery
`
`-
`
`~
`
`Receiver
`
`2
`
`,
`
`3
`
`r
`
`/6
`
`,, ~4
`
`/5
`
`Consumer
`Interface
`
`-
`.
`
`Processor
`
`~
`
`-
`
`-
`.
`
`Memory
`
`,, /7
`
`Display
`
`d •
`\JJ.
`•
`~
`~ ......
`~ = ......
`
`0
`I")
`!"""
`~~
`'"""'
`\C
`\C
`00
`
`'Jl =(cid:173)~
`~ .....
`'"""' 0 .....,
`
`Ul
`
`Ul
`....
`00
`~
`00
`....
`~
`~
`~
`
`
`
`Case 5:15-cv-02008-EJD Document 83-12 Filed 04/12/16 Page 4 of 13
`
`FIG. 2
`
`12
`
`20
`
`14
`
`26
`
`30
`
`Sequencer
`
`Encoder
`
`1
`
`,
`
`.... Transmitter 1
`
`... ~
`
`10
`Primary Data
`Stream
`Generation
`19
`
`Data
`Synchronizer
`
`22
`
`•
`
`24
`
`17
`
`28
`
`Associated
`Data
`Generation
`16
`
`18
`
`r- -
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`- _,,._ -
`
`-
`
`42
`
`Human
`Interface
`
`38
`
`Processor
`
`36
`
`Receiver
`
`Printer
`
`34~1
`
`44
`
`Memory
`
`40
`
`d •
`\JJ.
`•
`~
`~ ......
`~ = ......
`
`0
`I")
`!"""
`~~
`'"""'
`\C
`\C
`00
`
`'Jl =(cid:173)~
`~ .....
`N
`0 .....,
`Ul
`
`Ul
`....
`00
`~
`00
`....
`~
`~
`~
`
`
`
`Case 5:15-cv-02008-EJD Document 83-12 Filed 04/12/16 Page 5 of 13
`
`FIG. 3
`
`-
`-
`
`87---
`
`76
`
`\
`
`Real-Time
`Trigger
`
`0
`
`78----
`
`66'--.
`
`Primary
`Data
`Rendering
`
`54
`
`j l
`
`v52
`
`56
`(
`
`-
`-
`
`58
`
`Receiver
`
`36
`
`62
`)
`
`Decoder
`
`Human
`Interface
`90 ---.'.. ,.
`
`-
`.._
`
`Local Data
`Manager
`~84
`
`0
`86_/
`
`83
`(
`
`Communications
`Manager
`••
`i.---- 64
`
`..
`
`~
`
`Associated Data
`Protocol
`Manager
`
`60
`
`88
`
`- Local Data
`.
`Storage
`
`80
`
`Al
`
`v82
`
`d •
`\JJ.
`•
`~
`~ ......
`~ = ......
`
`0
`I")
`!"""
`~~
`'"""'
`\C
`\C
`00
`
`'Jl =(cid:173)~
`~ .....
`
`~
`0 .....,
`Ul
`
`Ul
`....
`00
`~
`00
`....
`~
`~
`~
`
`
`
`Case 5:15-cv-02008-EJD Document 83-12 Filed 04/12/16 Page 6 of 13
`
`FIG. 4
`
`10
`Primary Data1
`Stream
`Generation
`
`r--12
`
`20
`
`14
`
`26
`
`30
`
`Data
`Synchronizer
`
`Sequencer
`
`Encoder
`
`1
`
`, •• Transmitter 1
`
`•:J
`
`32
`\
`
`22
`
`17
`
`24
`
`I
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`18
`
`Associated
`Data
`Generation
`16
`
`28
`- 38- - - - - - -36 - - - - _,_ - - -
`
`Processor
`
`Receiver
`
`Dis~ I I
`42
`
`34__r:
`
`I
`
`I Printer
`
`I
`I
`
`•I Memory
`'-40
`
`I Two-Way
`
`•: Comm Channel
`
`L46
`44
`L--------------------------
`
`d •
`\JJ.
`•
`~
`~ ......
`~ = ......
`
`0
`I")
`!"""
`O'I
`
`~
`
`'"""'
`\C
`\C
`00
`
`'Jl =(cid:173)~
`~ .....
`.i;;..
`0 .....,
`Ul
`
`Ul
`....
`00
`~
`00
`....
`~
`~
`~
`
`
`
`Case 5:15-cv-02008-EJD Document 83-12 Filed 04/12/16 Page 7 of 13
`
`FIG. 5
`
`-
`
`r-88
`
`Human Interface
`
`i.--
`
`Primary
`Data
`Rendering
`
`_ 54
`1
`I-/
`
`•
`
`87~ 76
`
`Real-Time
`Trigger
`
`78----.
`
`'
`
`90--J
`
`83
`
`Local Data
`Manager
`
`86~ 84
`
`Local Data
`Storage
`
`82
`
`80
`
`t-------- 9 4
`
`•
`
`Remote Data
`Manager
`
`92
`
`t------. 96
`
`..
`
`v52
`
`66'-._
`
`50
`
`56
`
`Receiver
`
`Decoder
`
`36_)
`
`53_/
`
`i--64
`62
`( I Associated Data
`41
`Protocol
`Manager
`60_)
`
`Communications Manager
`•
`
`• r7o
`
`('---- 68
`
`Network
`Protocol
`Manager
`~72
`Two-way
`Communication r---74
`Channel
`
`d •
`\JJ.
`•
`~
`~ ......
`~ = ......
`
`0
`I")
`!"""
`~~
`'"""'
`\C
`\C
`00
`
`'Jl =(cid:173)~
`~ .....
`Ul
`0 .....,
`Ul
`
`Ul
`....
`00
`~
`00
`....
`~
`~
`~
`
`
`
`Case 5:15-cv-02008-EJD Document 83-12 Filed 04/12/16 Page 8 of 13
`
`5,818,441
`
`1
`SYSTEM AND METHOD FOR SIMULATING
`TWO-WAY CONNECTIVITY FOR ONE WAY
`DATA STREAMS
`
`5
`
`15
`
`20
`
`BACKGROUND OF THE INVENTION
`1. Field of the Invention
`This invention relates to enhancing one-way broadcast
`data transmissions. More particularly, it relates to enhancing
`the quality and content of a primary information stream by
`creating and transmitting associated data which provides the
`appearance of an interactive connection to secondary
`sources of information.
`2. Background
`One of the limitations of nearly all forms of electronic
`mass media such as radio, television, audio CD's and video
`cassettes is that the communication of information or data is
`one way from the provider to the consumer. The character(cid:173)
`istics of one way sources of data are that the data is sent
`sequentially and is fleeting or at best can be recorded for
`later playback. And a consumer cannot interact with the data
`provided to seek additional information or services.
`There is a desire by electronic mass media consumers for
`additional services, and providers of television and radio
`broadcast services are looking for new sources of revenue. 25
`In particular, consumers are looking for information or data
`that is related to what they are viewing in the media. The
`additional information may be in the form of details on the
`content of the current program such as the recipe of a meal
`being demonstrated on a cooking show or biographies of 30
`actors in a drama or historical background information on
`events depicted in a program. Or, it could be program
`highlights such as key plays from the baseball game. In
`addition, consumers would like access to real time data such
`as stock price, updated baseball scores as they occur, traffic 35
`and weather conditions. Also, consumers would like to have
`access to special services associated with product advertis(cid:173)
`ing such as information on where to buy nationally adver(cid:173)
`tised products-possibly with a map to the nearest location,
`or the ability to instantly receive coupons.
`Recently, online services such as provided by the World
`Wide Web of the Internet have become available. Such
`services provide access to immense amounts of data on an
`interactive basis by linking digital computers together over
`sophisticated communications networks. At the same time,
`the cost of digital computing power is falling rapidly. For
`example, many homes now have one or more computers.
`And home computers have displays capable of showing
`television as well as audio capability. They also have
`memory and computing power.
`But while such services are generally available and com(cid:173)
`puters are becoming more widespread, finding relevant
`information is an arduous task that may take hours of
`searching even for an experienced user.
`In addition, these services require a 2-way connection 55
`from a consumer's location to the online network. In the
`consumer market, prolonged utilization of a single phone
`line for 2-way digital communications is expensive.
`Up until now, there has been no way for producers of mass
`market broadcast programming to deliver data associated by 60
`its relevancy to its subject matter that could be interactively
`displayed and manipulated by consumers on a real time
`basis. What is meant by real time is that the consumer
`receives and has access to the relevant data during the
`process of program reception. Therefore the data becomes 65
`an integral part of the experience desired by the program
`producers.
`
`2
`Although received during a program, the information is
`retained and may be traversed by the consumer at a later
`time, as if they were connected in a 2-way fashion to an
`online service.
`3. Prior Art
`Perhaps the earliest effort to address the foregoing needs
`is for program content providers to include a telephone
`number in either the radio or television broadcast. The
`consumer is invited to dial a number for additional infor-
`10 mation. However, this telephone number is part of the
`primary data stream which is transitory.
`A second attempt is the provision of closed caption
`programming for the hearing impaired on television. This
`takes the form of written text appearing somewhere on the
`television screen typically at the bottom and requires a
`special decoder or mode set on the television. The text is a
`written rendition of the audio portion of the television
`program. That is, it is a recapitulation of the information
`supplied by the primary data provider and is not stored or
`saved for later access by the consumer.
`Television networks also use a part of the unused band-
`width in the video signal to send the schedule of programs
`coming up on that network. One network is sending a
`comprehensive program listing in digital format over an
`unused section of the NTSC bandwidth. This is called an
`electronic program guide ("EPG"). A second network uses a
`portion of the unused NTSC bandwidth to send digitally
`encoded stock quotes and the subject matter of the topics
`covered in the news broadcast, and extended data services
`("XDS") sends the date, time, name of a scheduled program,
`type of program and how much of the program remains.
`Cable services provide additional detail about songs being
`played on subscriber digital audio services such as digital
`music express ("DMX(tm)"); The detail typically consists of
`the name of the artist, the name of the song and the album.
`A system called Gemstar(tm), provides information in digi(cid:173)
`tal format that enables consumers to record programs by
`referencing a number in a program guide.
`Many of the foregoing services are provided on broadcast
`television only over what is called the vertical blanking
`interval (the "VBI"). The VBI is a portion of the bandwidth
`defined for broadcast television and in the case of NTSC, for
`example, consists of the first 21 of the 525 raster lines that
`45 define a video frame. SECAM and PAL have a similar
`arrangement. Of the 21 lines, line 21 is divided into two
`frames of 60 hz bandwidth each. The closed caption is
`provided in the first frame of line 21.
`None of the foregoing allow a consumer to experience
`50 apparent interactivity with external data sources. That is,
`none provides data that persists for more than a few seconds
`and none allows the consumer to manipulate directly that
`data.
`
`40
`
`SUMMARY OF THE INVENTION
`The foregoing problems may be resolved by a system for
`supplying a primary data stream and associated data to a
`consumer such that the consumer perceives the associated
`data as the result of interactivity with external sources of
`information. This system includes a creation unit for creat(cid:173)
`ing the primary data stream and the associated data. The
`primary data stream and associated data are supplied to a one
`way delivery unit for delivering the primary data stream and
`the associated data to the consumer. At the consumer
`location, the system includes a receiver for receiving the
`primary data stream and a receiver for receiving the asso(cid:173)
`ciated data. These may be the same unit. The system further
`
`
`
`Case 5:15-cv-02008-EJD Document 83-12 Filed 04/12/16 Page 9 of 13
`
`5,818,441
`
`3
`includes computer memory storing the received associated
`data. A processor communicates with the memory for
`accessing the associated data. Finally, the system includes a
`user interface connected to the processor through which the
`consumer may interact with the system.
`The method of the present invention includes the steps of
`first generating a stream of primary data and of associated
`data. Next, both the primary data stream and associated data
`are delivered to the consumer. Next, both the primary data
`stream and the associated data are received at the location of
`the consumer. The primary data stream is rendered to the
`consumer as it arrives. The associated data is stored in local
`storage. Next the associated data is accessed by retrieving it
`from local memory.
`
`BRIEF DESCRIPTION OF IBE DRAWING
`
`The preferred embodiments of the invention will now be
`described in connection with the drawing wherein:
`FIG. 1 is a block diagram of the invention at the highest
`level of abstraction.
`FIG. 2 is a block diagram showing further detail of the
`entire system according to the present invention.
`FIG. 3 is a block diagram showing additional detail of the
`system located with the consumer.
`FIG. 4 is a block diagram of a second preferred embodi(cid:173)
`ment of the invention that includes interactive communica(cid:173)
`tions.
`FIG. 5 is a diagram showing additional detail of the
`system located with the consumer when interactive commu(cid:173)
`nications are included.
`
`DESCRIPTION OF THE PREFERRED
`EMBODIMENT
`
`An electronic mass media provider may be said to deliver
`a one-way stream of electronic data. This stream of data is
`herein referred to as a primary stream of data and includes
`live or pre-recorded information that is created by the
`broadcasting and entertainment industries for the purpose of
`delivery to consumers over delivery mediums such as broad(cid:173)
`cast television, video cassettes, radio and audio CD's. The
`content of the primary data stream is typically created by
`production studios for mass distribution to the consumer
`market. Often a distributor produces all or part of the
`programming content. For example, a local television news
`station obtains pre-produced video news and advertising
`content from national providers and inserts locally produced
`content and advertising for broadcast. Equipment used in the
`television industry are video cameras and video recorders. In
`radio, it is microphones and audio recorders. Typically, the
`primary data is organized into programs. A primary stream
`of data can be rendered intelligent to a consumer as either
`audio or video or a combination of the two.
`The term associated data as used herein refers to a stream
`of data generated separately from the primary data but
`having content that is relevant to the primary data in general
`and usually relevant to a particular program of primary data
`and is in this sense associated. Associated data is intended to
`enhance the utility of the primary data stream, but if it is not
`available, the primary data stream can stand on its own
`merits. The creation process typically uses commercially
`available software and hardware that output industry stan(cid:173)
`dard file formats such as Hypertext Markup Language
`(HTML) for text and graphic layout, Graphics Interchange
`Format (GIF) Joint Photographic Experts Group (JPEG)
`formats for still images, and so forth.
`
`20
`
`30
`
`4
`FIG. 1 is a block diagram of the invention at the highest
`level of abstraction. Referring now to FIG. 1, data stream
`creation unit 1 performs the function of generating both the
`primary data stream and the associated data stream. The two
`5 data streams may or may not be merged to create a combined
`stream of primary and associated data. The two data streams
`are supplied to non-interactive delivery unit 2 which per(cid:173)
`forms the function of delivering the two data streams to the
`consumer. The non-interactive delivery unit includes deliv-
`10 ery by broadcast, cable or a packaged media such as
`cassettes and audio CD's. Indeed, non-interactive delivery is
`intended to include all one-way electronic data delivery
`systems. The primary data stream and secondary data stream
`are typically transmitted to a consumer over the same
`15 delivery medium; however, the invention contemplates situ(cid:173)
`ations where this is not the case. For example, the primary
`data stream could be delivered by broadcast television and
`the associated data could be delivered over a high speed
`digital network, a FM sideband, a direct satellite broadcast,
`a cable network, a telephone, etc.
`When the delivered data reaches receiver 3, it is decoded
`from its delivery medium. That is, if it is sent by broadcast
`television, the receiver decodes the signal, separates the
`primary data from the associated data and passes the asso-
`25 ciated data on to processor 4. Processor 4 may further
`decode the associated data and in any event supplies the
`associated data to memory 5 where it is stored for future use.
`The consumer of the primary data stream may interact with
`the associated data via user interface 6 such as a keyboard,
`mouse or voice activation electronics through which the
`consumer gives commands to processor 4 and which in turn
`causes the associated data to be accessed and processed. The
`resulting data is then supplied to display 7.
`FIG. 2 provides additional system detail. Referring now to
`35 FIG. 2, reference numeral 10 refers to the primary data
`stream generating sub-system. The output of primary data
`stream generation sub-system 10 is typically an analog
`signal. However, it may be a digital signal as in the case of
`digital broadcast television. Data channel 12 connects pri-
`40 mary data stream generation sub-system 10 and sequencer
`14 and data channel 19 connects it to data synchronizer 20.
`Data channels 12 and 19 may be initiated by a live video
`feed, or analog or digital tape. Reference numeral 16 refers
`to an associated data generation sub-system. The output of
`45 associated data generation sub-module 16 is a digital signal
`that is supplied over digital data link 18 to sequencer 14 and
`over digital data link 17 to synchronizer 20. The digital data
`links of the system may be a conventional digital connection
`such as a serial or parallel or it may be a network link.
`50 Typical connecting media would be twisted pair, co-axial
`cable, fiber optic cable or a wireless media.
`Reference numeral 20 refers to a data synchronizing
`sub-system whose function is to synchronize the primary
`data stream generated by sub-system 10 with specific asso-
`55 ciated data. The input to data synchronizing sub-system 20
`is scene information from the primary data stream in the
`form of timecodes and time durations, and data from asso(cid:173)
`ciated data generator sub-system 16. It creates a so called
`script for the delivery and display of associated data at
`60 specific points in time. For example, data synchronizer 20
`creates a script that specifies that a detailed data sheet will
`be delivered to the consumer prior to a specific television
`product advertisement, and that the data sheet will be
`displayed on the consumer's personal computer display
`65 when a certain television advertisement starts.
`Data synchronizer 20 typically includes software which is
`patterned after common digital video editors such as Ado-
`
`
`
`Case 5:15-cv-02008-EJD Document 83-12 Filed 04/12/16 Page 10 of 13
`
`5,818,441
`
`5
`be's Premiere, which places parallel video and audio tracks
`on a standard time line such as SMPTE (Society of Motion
`Picture and Television Engineers) time code. This allows
`segments of each track to be manipulated independently but
`with reference to a common time line so that segments may 5
`later be reconstituted. The software of data synchronizer 20
`allows associated data to be laid out and manipulated on an
`additional data track. The size of particular associated data
`components is coordinated with the known bandwidth of the
`delivery medium. The user of data synchronizer 20 typically 10
`specifies that a unit of information should be displayed at a
`specific point in time. Data synchronizer 20 then calculates
`the time required to transfer the data and inserts a transfer
`specification into the script at the appropriate point prior to
`the display instruction. The output of data synchronizer 15
`sub-system 20 is a digital signal representing a script that
`synchronizes the output of primary data stream sub-system
`10 and associated data stream generator 16.
`The output of data synchronizer 20 is supplied over digital
`data link 22 to sequencer sub-system 14. Digital data link 22 20
`is a conventional digital link. Sequencer 14 combines the
`primary data stream with the associated data as specified by
`the script supplied by synchronizer sub-system 20.
`Sequencer 14 monitors the output from primary data stream
`sub-system 10 to obtain the information necessary to 25
`sequence transmission of associated data. For example,
`during the broadcast of a television program, data sequencer
`14 obtains a time code information such as SMPTE time
`code which is generated by the primary data stream. The
`information is used to coordinate mixing of associated data
`with primary data streams. In addition, sequencer 14 mixes
`other data feeds that are being transmitted over the same
`delivery medium. These may include external data services
`such as financial data, emergency broadcast information or
`weather information. Mixing of existing data streams is
`necessary only for delivery mediums that don't support
`separate, independent transmission of digital information.
`This function has the capacity to be used for the delivery of
`either live or pre-recorded programming, or a combination
`of both. For live programming, such as a local evening news 40
`program, the data stream sequencer takes input directly from
`the outputs of primary data stream generator 10, data syn(cid:173)
`chronizer 20, associated data generator 16, and time code
`from the primary data stream. For pre-recorded
`programming, this function processes the output of the data 45
`synchronizer 20 and sequences the broadcast of associated
`data with the primary data.
`The output of sequencer 14 is supplied over conventional
`digital data link 24 to encoder 26. Encoder 26 is a hardware
`component for insertion of digital information into the 50
`particular delivery medium that will be received by the
`consumer. If the delivery medium is broadcast television,
`encoder 26 inserts this information directly into the NTSC
`television signal, and thus delivers both the primary and
`associated data over a single delivery medium, broadcast 55
`television radio waves. Encoder 26 accepts input streams of
`both video and serialized digital information. It breaks the
`serialized digital information up into a series of packets, and
`modulates the packets into the ("VBI") of the video signal.
`Encoder 26 is commercially available hardware and soft- 60
`ware. A number of vendors market hardware for insertion
`and extraction of data into and from standard (NTSC/PAL)
`television signals. Norpak Corporation, EEG, Inc, and
`WavePhore, Inc. are examples.
`The output of encoder 26 is supplied via digital data link 65
`28 to transmitter sub-system 30 which performs the function
`of physically transmitting radio frequency waves into the
`
`6
`atmosphere. Such transmitters may be television or radio
`broadcast transmitters or a satellite broadcast system. Also,
`the data may be stored on a tape for later transmission.
`Reference numeral 34 generally indicates the equipment
`that must be present at the consumer's location. In general,
`system 34 is a computer. That is, it has digital computing
`capability as well as equipment to receive the primary and
`associated data streams. Receiver 36 could be a personal
`computer add-in adapter board, a television or radio broad(cid:173)
`cast receiver, a cable television converter box or it could be
`a satellite receiver for digital broadcast. In the preferred
`embodiment, it is a broadcast television receiver or tuner
`inside of a personal computer. Connected to receiver 36 is a
`microprocessor 38. Connected to microprocessor 38 is
`memory 40 which would typically be semiconductor RAM.
`Additional magnetic media such as a hard disk would be
`desirable. Microprocessor 38 is connected to human inter(cid:173)
`face 42, which is typically a CRT monitor, and to printer 44.
`Human interface 42 and receiver 36 could be part of a
`standard television.
`Reference numeral 32 refers to the delivery media which
`in this case are radio frequency electromagnetic waves
`passing through the atmosphere. If the delivery media is a
`video tape, laser disk or audio CD, then, transmitter 30
`becomes a video tape recorder, video or audio CD recorder,
`and receiver 36 becomes a VCR, laser disk player or an
`audio CD player respectively.
`FIG. 3 is a high level block diagram showing further
`details of system 34 of FIG. 2. Referring now to FIG. 3,
`30 receiver 36 receives radio frequency waves from input 50.
`Receiver 36 demodulates the input signal and supplies the
`primary data stream signal over data path 52 to primary data
`rendering sub-system 54 and the associated data signal over
`data path 56 to associated data decoder 58.
`Primary data rendering sub-system 54 performs the func(cid:173)
`tion of presenting the primary data stream to the consumer
`in the manner in which a typical consumer would expect to
`see the data presented. For example, in the case of television,
`the primary data rendering takes the form of a video image
`typically supplied by a cathode ray tube screen, or possibly
`a liquid crystal display screen and audio provided by an
`audio amplifier and speakers. In the preferred embodiment
`these components are in a personal computer. A second
`example is a broadcast radio demodulator, amplifier and
`speakers that renders radio waves received via radio broad(cid:173)
`casting audible to a consumer. Yet a third example is a stereo
`system that renders data encoded on compact disk or tape
`audible to the consumer.
`Decoder 58 performs the function of decoding the asso(cid:173)
`ciated data from the delivery media. In this case decoder 58
`consists of an analog to digital converter that converts
`analog encoded digital data back to digital format.
`Associated data protocol manager 60 is connected by
`one-way data path 62 to decoder 58 and by one way data
`path 64 to communications manager 66. Associated data
`protocol manager 60 performs the function of extracting the
`different forms of associated data from the incoming digital
`data steam and converting them to a form that can be used
`by communications manager 66. The types of associated
`data protocols include World Wide Web pages, closed
`captioning, stock quotes, sports scores, control commands
`for microprocessor 38 (of FIG. 2) to execute.
`Communications manager 66 performs the function of a
`common network interface by receiving data from several
`different types of communication devices using different
`data transmission protocols. Such devices include telephone
`
`35
`
`
`
`Case 5:15-cv-02008-EJD Document 83-12 Filed 04/12/16 Page 11 of 13
`
`5,818,441
`
`10
`
`35
`
`7
`modems, ISDN modems, cable modems, wireless modems,
`satellite modems, broadcast TV, radio and the like. Com(cid:173)
`munication manager 66 converts all data received, regard(cid:173)
`less of the source and protocol, into a standard format that
`can then be utilized by the rest of the system. This function
`may be implemented in one of two ways. Proprietary
`interfaces between the communication components may be
`designed, or standard industry interfaces such as Microsoft
`Windows' (a trademark of Microsoft Corporation) sockets
`may be used. The preferred embodiment uses Windows
`sockets. A Windows socket is a standard application inter(cid:173)
`face to access network data.
`Real time trigger 76 is connected to communications
`manager 66 by one-way data path 78 and to human interface
`88 by data path 87. Human interface 88 provides the
`consumer with input and output to the system. In the
`preferred embodiment, human interface 88 uses the key(cid:173)
`board and alternate input devices such as a mouse of a
`personal computer as input for requests and the display of
`the personal computer for displaying the data.
`Real time trigger 76 accepts commands sent as part of the
`associated data to display a page of information without the
`user asking for it. The output of realtime trigger is a
`command sent over data path 87 to human interface 88
`which causes the page of information to be displayed. For 25
`example, a broadcaster may want viewers to see a certain
`page of information as part of a program that is being
`viewed. Real time trigger allows data to be displayed using
`the same data protocols as consumer requested information.
`Local data storage 80 is connected to communications
`manager 66 by one-way data path 82 and by data path 83 to
`local data manager 84. The hardware implementation for
`local data storage 80 may be one or more of the following;
`RAM, disk, tape, recordable CD-ROM.
`Local data manager 84 is connected to communications
`manager 66 by data path 86 and to human interface 88 by
`two-way data path 90. Local data manager 84 receives
`commands from human interface 88 to retrieve associated
`data from local data storage 80 and sends that data to human
`interface 88 for presentation to the consumer. For example,
`a "Web Browser" may be used to display data pages from the
`World Wide Web (the "WWW"). Providers of WWW
`browsers include Netscape Communications Corp., America
`Online, Spyglass and others. Local data storage 80 will
`typically be of limited capacity. Thus, local data manager 84
`purges older and less used information. This is accomplished
`by assigning an expiration date and/or a priority to associ(cid:173)
`ated data files. The criteria for determining which data to
`purge is settable by the broadcaster and/or the consumer.
`Criteria include total amount of storage available, size of
`associated data files, expiration date and priority.
`The preferred embodiment illustrated in FIGS. 2 and 3
`operates in the following manner. A primary data stream is
`generated. Associated data is separately generated. A script 55
`is generated that synchronizes how the primary and associ(cid:173)
`ated data are linked together. The primary data, the associ(cid:173)
`ated data and the script are sent to sequencer 14 where they
`are combined. The primary data is transmitted in the con(cid:173)
`ventional fashion of its delivery medium. In the television 60
`model, this is broadcast over the air or on cable. The
`associated data is generally (but not always) sent by the
`same delivery medium as the primary data. In the case of
`television, the associated data is encoded in the VBI of the
`television signal. The signal is received by the consumer's 65
`equipment. The primary data stream is immediately ren(cid:173)
`dered and the associat