I 1111111111111111 11111 111111111111111 IIIII IIIII IIIII IIIII IIIIII IIII IIII IIII
`
`
`US007269338B2
`
`c12) United States Patent
`Janevski
`
`(IO) Patent No.:
`(45) Date of Patent:
`
`US 7,269,338 B2
`Sep.11,2007
`
`(54) APPARATUS AND METHOD FOR
`SYNCHRONIZING PRESENTATION FROM
`BIT STREAMS BASED ON THEIR CONTENT
`
`(75)
`
`Inventor: Angel Janevski, New York, NY (US)
`
`(73) Assignee: Koninklijke Philips Electronics N.V.,
`Eindhoven (NL)
`
`( *) Notice:
`
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 1435 days.
`
`(21) Appl. No.: 10/014,225
`
`(22) Filed:
`
`Dec. 11, 2001
`
`(65)
`
`Prior Publication Data
`
`US 2003/0156827 Al
`
`Aug. 21, 2003
`
`(51)
`
`Int. Cl.
`H04N 7100
`(2006.01)
`H04N 5191
`(2006.01)
`(52) U.S. Cl. ............................ 386/96; 386/80; 386/46;
`386/68
`(58) Field of Classification Search .................. 386/96,
`386/92, 113,124, 13,21,40,46,
`52,80,85,
`386/68
`See application file for complete search history.
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`7,072,575 B2 * 7/2006 Kang
`
`.........................
`
`386/125
`
`2003/0120685 Al*
`
`6/2003 Duncombe et al. ......... 707/200
`
`OTHER PUBLICATIONS
`
`William P. Lord, "Synchronized Personal Video Recorders", U.S.
`Appl. No. 09/894,060, filed Jun. 28, 2001.
`
`* cited by examiner
`
`Primary Examiner-Robert Chevalier
`(74) Attorney, Agent, or Firm-Paul
`Im
`
`(57)
`
`ABSTRACT
`
`Content of multiple digital bit streams with essentially the
`same renderable content, and situated at mutually remote
`locations, are played back in unison. First, time is synchro(cid:173)
`nized among the processors that participate in a synchro(cid:173)
`nized viewing session by playing back respective streams.
`Second, the playbacks are content-wise aligned, by rewind(cid:173)
`ing or fast forwarding, to effect overall a precisely synchro(cid:173)
`nized presentation. The content-wise alignment is achieved
`by means of a status message between processors
`that
`contains
`information characteristic of the sender's bit
`stream. If any participant, performs a control function ( e.g.
`rewind, fast forward, stop), all other participants follow
`synchronously. The processor that initiates the session is
`deemed the initiator, a role that is thereafter assumed by the
`participant that has last performed a control function. The
`initiator directs all participants, at session startup, upon
`execution of each control function, and periodically, to
`synchronize their playbacks to that of the initiator, whereby
`all playbacks are synchronized and maintain in synchroni(cid:173)
`zation.
`
`35 Claims, 9 Drawing Sheets
`
`113
`
`RECEIVER
`
`202
`
`212 ~ VIDEO TIMER I
`
`204
`
`REMOTE
`CONTROL
`SENSOR
`
`/206
`
`
`
`SYNCHRONIZATION REGISTER
`
`SIGNATURE PROCESSING
`WORKSPACE
`
`VIDEO BLOCK
`
`STATUS BUFFER
`
`
`
`MISCELLANEOUS STORAGE
`
`214
`
`216
`
`218
`
`220
`222
`
`\
`
`114a
`
`201
`
`
`PAGE 1 OF 20
`
`SONOS EXHIBIT 1007
`IPR of U.S. Pat. No. 8,942,252
`
`

`

`U.S. Patent
`
`Sep.11,2007
`
`Sheet 1 of 9
`
`US 7,269,338 B2
`
`112a
`
`112b
`
`114a
`
`113b 114b 116b
`
`..___, RC
`
`122a TELEVISION
`
`118
`
`122b
`TELEVISION
`
`HOUSE 1
`
`HOUSE 2
`
`FIG. 1
`
`PAGE 2 OF 20
`
`SONOS EXHIBIT 1007
`IPR of U.S. Pat. No. 8,942,252
`
`

`

`U.S. Patent
`
`Sep.11,2007
`
`Sheet 2 of 9
`
`US 7,269,338 B2
`
`RECEIVER
`., '
`
`' I"
`
`~
`
`'
`
`113
`
`0
`
`204
`(
`REMOTE
`CONTROL
`SENSOR
`
`;200
`
`202
`(
`208 ~ VIDEO PLAYER I f 210
`UNIT j / 3
`J STREAM CHARACTERISTICS
`I PERIOD
`
`212~
`
`VIDEO TIMER
`
`TIMER
`;r-,
`' ,.
`
`SYNCHRONIZATION REGISTER r---214
`SIGNATURE PROCESSING v216
`WORKSPACE
`
`VIDEO BLOCK
`
`STATUS BUFFER
`
`
`
`MISCELLANEOUS STORAGE
`
`--218
`
`--- 220
`---222
`
`~
`
`201
`
`FIG. 2
`
`\
`
`114a
`
`
`PAGE 3 OF 20
`
`SONOS EXHIBIT 1007
`IPR of U.S. Pat. No. 8,942,252
`
`

`

`U.S. Patent
`
`Sep.11,2007
`
`Sheet 3 of 9
`
`US 7,269,338 B2
`
`,---___,A
`I
`
`302
`________
`
`\I
`
`304
`A._ __
`
`\
`
`POINTER TIME STAMP POINTER TIME STAMP •••
`K+L K+l
`K
`K
`
`306
`
`__
`
`,...._______,A...._
`I
`
`\
`POINTER TIME STAMP
`214
`I"-
`N
`N
`
`SIGNATURE PROCESSING
`WORKSPACE
`
`1-r
`
`216
`
`••• FRAME ••• FRAME ••• FRAME
`• •• FRAME
`K
`N
`K+L
`N+M
`
`308
`
`VIDEO BLOCK
`
`f-..r-
`
`218
`
`STATUS BUFFER
`
`
`
`MISCELLANEOUS STORAGE
`
`FIG. 3
`
`v-,.-
`
`220
`
`222
`
`--
`~
`206
`
`
`PAGE 4 OF 20
`
`SONOS EXHIBIT 1007
`IPR of U.S. Pat. No. 8,942,252
`
`

`

`U.S. Patent
`
`Sep.11,2007
`
`Sheet 4 of 9
`
`US 7,269,338 B2
`
`INITIATOR r-- 114
`PVR
`
`114 ..--.__, PARTICIPANT
`PVR
`
`HR MIN SEC
`
`0
`
`0
`
`0
`
`0
`
`0
`
`0
`
`0
`
`0
`
`3
`
`4
`
`3
`
`4
`
`B }CASE
`C
`1
`
`B }CASE
`C
`2
`
`0
`
`0
`
`0
`
`0
`
`3
`
`4
`
`B }CASE
`C
`3
`
`HR
`
`MIN SEC
`
`A
`
`D
`
`0
`
`0
`
`A
`
`D
`
`0
`
`0
`
`0
`
`0
`
`0
`
`0
`
`2
`
`5
`
`2
`
`3
`
`A
`
`D
`
`0
`
`0
`
`0
`
`0
`
`2
`
`6
`
`f402
`I
`
`i( 40)
`
`}402
`I
`
`,( 40)
`
`£402
`I
`
`i( 40)
`
`(cid:141)
`
`(cid:141)
`
`(cid:141)
`
`FIG. 4
`
`
`PAGE 5 OF 20
`
`SONOS EXHIBIT 1007
`IPR of U.S. Pat. No. 8,942,252
`
`

`

`U.S. Patent
`
`Sep. 11, 2007
`
`Sheet 5 of 9
`
`US 7,269,338 B2
`
`402
`
`TIME STAMP
`A
`
`402
`
`f
`
`510
`
`TIME STAMP
`A
`510--,.~-------
`
`TIME STAMP V 520
`f
`B
`
`404
`
`TIME STAMP
`A
`
`TIME STAMP
`B
`
`(
`510
`
`560
`)
`PROGRAM
`INDENTIFIER
`
`520
`
`570
`~
`
`TIME STAMP
`C
`")
`530
`
`WATCHING
`MODE
`
`550
`/
`
`QUERY
`TIME
`MISREGISTRATION SIGNATURE
`)
`580
`
`540 5
`
`QUERY
`TIME STAMP
`)
`590
`
`FIG. 5
`
`
`PAGE 6 OF 20
`
`SONOS EXHIBIT 1007
`IPR of U.S. Pat. No. 8,942,252
`
`

`

`U.S. Patent
`
`Sep. 11, 2007
`
`Sheet 6 of 9
`
`US 7,269,338 B2
`
`,, ,
`INITIATOR AWAITS NOTIFICATION v- S602
`OF NEW PARTICIPANT
`',.
`INITIATOR SENDS ON ORIGINATING
`Lr'
`
`SYNCHRONIZATION MESSAGE TO
`THE NEW PARTICIPANT
`
`S604
`
`., ,..
`
`NEW PARTICIPANT ADVANCES ITS
`
`TIME COUNT TO MATCH TIME
`STAMP IN ORIGINATING v- S606
`
`SYNCHRONIZATION MESSAGE
`
`AND CORRESPONDINGLY FAST
`
`FORWARDS ITS COPY
`OF THE VIDEO
`
`',.
`NEW PARTICIPANT TRANSMITS if"'
`
`A JOIN REQUEST MESSAGE
`TO THE INITIATOR
`
`S608
`
`FIG. 6
`
`
`PAGE 7 OF 20
`
`SONOS EXHIBIT 1007
`IPR of U.S. Pat. No. 8,942,252
`
`

`

`U.S. Patent
`
`Sep.11,2007
`
`Sheet 7 of 9
`
`US 7,269,338 B2
`
`INITIATOR WAIS UNTIL:
`
`CD IT STARTS A SESSION, OR
`0 ITS PERIOD TIMER HAS EXPIRED, OR
`G) IT INITIATES A CONTROL FUNCTION, OR
`0 IT RECEIVES A RETRY REQUEST MESSAGE, OR
`0 IT RECEIVES A JOIN REQUEST MESSAGE
`
`S701
`
`S703
`
`NO THE INITIATOR RESETS
`ITS PERIOD TIMER
`
`THE INITIATOR SENDS AN ORIGINATING
`
`SYNCHRONIZATION MESSAGE TO THE PARTICIPANT
`
`S704
`
`THE PARTICIPANT SENDS A REPLY MESSAGE TO THE INITIATOR
`
`S705
`
`THE INITIATOR SELECTS A QUERY FRAME, CALCULATES ITS
`
`SIGNATURE AND CALCULATES TIME MISREGISTRATION
`
`S706
`
`THE INITIATOR BACKS TIME MISREGISTRATION,
`S708
`IF ANY, OUT OF THE QUERY TIME STAMP
`
`THE INITIATOR BUILDS AND TRANSMITS A STATUS MESSAGE
`
`CONTAINING THE PROGRAM INDENTIFIER, WATCHING MOOE, TIME
`
`MISREGISTRATION, QUERY SIGNATURE AND QUERY TIME STAMP
`
`S710
`
`FIG. 7A
`
`
`PAGE 8 OF 20
`
`SONOS EXHIBIT 1007
`IPR of U.S. Pat. No. 8,942,252
`
`

`

`U.S. Patent
`
`Sep.11,2007
`
`Sheet 8 of 9
`
`US 7,269,338 B2
`
`UPON RECEIPT OF STATUS MESSAGE, THE PARTICIPANT
`
`VALIDATES THE PROGRAM INDENTIFIER
`
`S712
`
`S716
`
`S714
`
`NO
`
`TRANSMIT ERROR
`MESSAGE TO INITIATOR
`
`S720
`
`THE PARTICIPANT
`ISSUES A CONTROL
`FUNCTION
`
`S724
`
`ADVANCE THE TIME
`COUNT BY THE TIME
`MISREGISTRTATION
`
`S728
`
`ROLL BACK THE TIME
`COUNT BY THE TIME
`MISREGISTRATION
`
`FIG. 78
`
`S726
`
`NO
`
`NO
`
`
`PAGE 9 OF 20
`
`SONOS EXHIBIT 1007
`IPR of U.S. Pat. No. 8,942,252
`
`

`

`U.S. Patent
`
`Sep.11,2007
`
`Sheet 9 of 9
`
`US 7,269,338 B2
`
`MIN OFF, ITERATION COUNT= 0; MINDIST = K,
`
`SEARCH RANGE= N; OFFSET= -N/2
`
`S802
`
`FIND PARTICIPANT FRAME WITH TIME STAMP S803
`
`CLOSEST TO QUERY TIME STAMP
`
`CURRENT FRAME IS SET TO POINT FRAME THAT IS N/2 I
`
`FRAMES PRIOR TO THE FOUND, CLOSEST FRAME
`
`S804
`
`HAMMING DISTANCE= HAMMING DISTANCE BETWEEN
`THE SIGNATURES OF THE CURRENT AND QUERY FRAMES
`
`S806
`
`S810
`
`OFFSET= OFFSET+ 1; CURRENT FRAME SET TO POINT TO NEXT S812
`
`
`I FRAME; ITERATION COUNT= ITERATION COUNT+ 1
`
`S814
`
`YES
`
`S820
`
`FAST FORWARD
`IMINOFFI
`I FRAMES
`
`S828
`
`STATUS PERIOD TIMER,
`
`
`WHEN IT EXPIRES, SEND
`RETRY REQUEST MESSAGE
`
`S824
`
`REVERSE IMINOFFl
`
`I FRAMES
`
`FIG. 8
`
`
`PAGE 10 OF 20
`
`SONOS EXHIBIT 1007
`IPR of U.S. Pat. No. 8,942,252
`
`

`

`US 7,269,338 B2
`
`1
`APPARATUS AND METHOD FOR
`SYNCHRONIZING PRESENTATION FROM
`BIT STREAMS BASED ON THEIR CONTENT
`
`BACKGROUND OF THE INVENTION
`
`25
`
`2
`forward, pause, etc.), and the time or frame into the program.
`The time or frame allows the recipient PVR to synchronize
`its replay with that of the sending PVR, by comparing the
`time or frame in the message with its own the current time
`5 or frame.
`In expanding on this concept of synchronizing a sending
`PVR with a recipient PVR by transmitting a time or frame
`from the sending PVR to the recipient PVR, it will be
`initially assumed, for purposes of illustrating the present
`10 invention, that both PVRs are playing back respective,
`identical copies of a video. The frame of the sending PVR
`is part of the sender's copy of the video, which resides in a
`bit stream that is stored in a storage medium. Similarly,
`frames of recipient PVR' s copy of the video reside in a bit
`15 stream that is stored in the recipient's storage medium.
`It will also be initially assumed that when the video timer
`of one PVR shows as its output the same time as does the
`other PVR's video timer, that the respective videos playing
`are at the same point content-wise in their respective play-
`20 backs. When any PVR fast forwards or rewinds, this corre(cid:173)
`spondingly and synchronously advances or rolls back the
`time count of its respective video timer.
`If, for example, the destination PVR's video timer reads
`1 hour, 1 minute and 1 second at a time when the destination
`PVR receives from the sending PVR a message having as its
`output time stamp 1 hour, 1 minute and 2 seconds (set
`according to the sending PVR's video timer), this might
`indicate the destination PVR's playback
`is one second
`30 behind that of the sending PVR. It might be the case, for
`example, that, according to the timing of a single reference
`clock, the destination PVR started its playback one second
`after the sending PVR started its playback. Based on that
`premise, the destination PVR can take corrective action to
`35 compensate for the one second time difference. Specifically,
`if the transmission time of the message was negligible, e.g.
`one millisecond while the time difference is one second, the
`full one second time difference can be relied on to take
`corrective action to synchronize the respective playbacks on
`the PVRs. The destination PVR would, for example, "fast
`forward" its local copy of the program by a full second and
`increment its video timer by a second. By this action, the
`destination's playback would catch up content-wise with
`that of the sender, and the respective video timers of the
`sender and destination PVRs would become synchronized.
`If, however, the transmission time was not negligible, it
`needs to be taken into account in comparing the output time
`stamp of the incoming message with the time the message is
`received at the recipient PVR so that the corrective com(cid:173)
`pensation applied appropriately reflects the extent to which
`the respective video timers out-of-synch and, correspond-
`ingly, the extent to which the respective playbacks are
`content-wise out-of-synch.
`However, the above technique alone will not always
`synchronize the video presentation, i.e. make concurrent the
`playback of corresponding frames in the respective play(cid:173)
`backs. The assumption made above, that the content being
`shown on either PVR at any arbitrarily-selected, common
`video timer time, is identical, does not strictly hold. The
`programs recorded on the two PVRs may, for example,
`begin at the same nominal video start time, but differ
`slightly, perhaps a second or so, as to the actual point in the
`video at which they respectively start. As a result, if both
`playbacks were to be viewed side-by-side, one would lag the
`other. Thus, even if the video timers of both respective PVRs
`were perfectly synchronous, the respective showings of the
`video might be out-of-synch.
`
`1. Field of the Invention
`The present invention relates generally to digital image
`playback, and more particularly to techniques for synchro(cid:173)
`nizing playback of two or more digital streams based on
`renderable content of those streams.
`2. Description of the Related Art
`Personal video recorders (PVRs) are video recording
`devices that may be used in conjunction with virtually every
`television broadcast system (e.g. cable, digital cable, satel(cid:173)
`lite, antenna, or combinations thereof), as well as to record
`from VHS, DVD, Internet sources, etc. PVRs may be
`programmed to automatically find and record a user's favor-
`ite television program or programs so that the user may
`watch what he or she wants, when he or she wants. Typi(cid:173)
`cally, PVRs contain a substantial amount of memory and are
`capable of recording more than thirty hours of program(cid:173)
`ming. The recorded prograniming may then be retrieved and
`viewed by the user as desired.
`Current PVR technology allows users to time shift the
`content of the program being recorded (typically television
`broadcasts). In other words, a user may record a television
`broadcast and watch it several hours, or even days, later.
`Alternatively, a user may decide to record a program and
`begin watching it a predetermined amount of time after the
`start of the program so that the user has the ability to skip
`any commercials
`that may be dispersed throughout
`the
`program. Thus, the user would be watching the program
`during virtually the same time period as people who are
`watching the live broadcast. However, given the capability
`to skip through the commercials, the user of the PVR will
`watch the same program as someone else in less time.
`Notwithstanding
`the above benefits of PVRs, there are
`disadvantages associated with their use. For example, many
`people enjoy watching various programs simultaneously 40
`(e.g., sporting events, talk shows or dramas) even though the
`people may be physically
`located in different locations.
`These people will often communicate with each other during
`the program by other communication means such as the
`telephone or Internet. Therefore, they are able to discuss the 45
`program as the events materialize. However, as people time
`shift content, they lose the ability to simultaneously watch
`shows "together" while at their respective locations. Inevi(cid:173)
`tably, the two users will be watching the same program out
`of synch and therefore one user will know the results of a 50
`dramatic scene or sporting event, for example, prior to the
`other user.
`Copending U.S. patent application Ser. No. 09/894,060,
`entitled "Synchronized Personal Video Recorders", filed
`Jun. 28, 2001, assigned to the assignee of the instant 55
`application, incorporated herein by reference, and not admit(cid:173)
`ted to be prior art by its mention in the background section,
`discloses a system in which one PVR synchronizes with
`another PVR by sending out a status message to the other
`PVR. The message issues when the user of the initiator PVR 60
`operates a PVR function such as start up, fast forward or
`rewind, to allow the recipient of the message to perform the
`counterpart function to keep the presentation on both PVRs
`synchronized. The message is also transmitted periodically,
`to update the synchronization. Within the message is an 65
`identifier of the program being watched or to be watched, an
`indicator of the mode of watching (e.g. normal play, fast
`
`
`PAGE 11 OF 20
`
`SONOS EXHIBIT 1007
`IPR of U.S. Pat. No. 8,942,252
`
`

`

`US 7,269,338 B2
`
`3
`Also, even if the two playbacks were to be in synch
`initially, the presentation may drift out-of-synch as it
`progresses. For example, the speed at which the respective
`PVRs play back their respective copies of the video may
`differ. These differences become more significant if the two 5
`PVRs have different actual speeds in the fast forward or
`rewind mode, and may cause the viewings to fall out-of-sync
`after one of the PVRs fast forwards or rewinds, commanding
`the other to follow concurrently and synchronously.
`Lack of synchronization may also occur from time to 10
`time, due, for example, to different commercials, and thus
`different commercial time periods, in the two playbacks.
`Both viewers, for instance, may be watching the same
`network, e.g. National Broadcasting Company (NBC), but
`through different cable or satellite providers, e.g. RCN or 15
`Time Warner.
`If, on the other hand, it is the current frame, rather than the
`current time, that is conveyed in the message, non-negligible
`transmission time still needs to be taken into account to
`synchronize presentation. If, for example, the source PVR 20
`sends the destination PVR a message that indicates that
`frame number "n" is currently playing on the source PVR,
`the destination PVR needs to know the transmission time, if
`non-negligible,
`in comparing the frame number that it is
`playing at the time of receipt of the message to the frame 25
`number indicated in the message.
`In the frame-based technique, even if the playbacks are in
`synch or transmission time is accounted for to bring the
`playbacks in synch, the PVRs may use different service
`providers that employ different compression schemes. One 30
`scheme might afford higher image quality than the other by
`including more frames; thus, the assumption above that the
`sender's copy of the video is the same as the recipient's copy
`carmot be strictly relied upon. In addition, adaptive tech(cid:173)
`niques are often used to vary the number of frames capturing 35
`a moving image based on the amount of movement in the
`image, time instant to time instant. The difference in the
`frames numbers for corresponding video content makes
`synchronization based on frame numbers problematic.
`For many situations, these synchronization errors are of 40
`such small magnitude that the viewers of respective play(cid:173)
`backs do not notice them.
`Yet, there exist viewing configurations in which "out-of(cid:173)
`synch" effects are significant and interfere with viewing
`enjoyment. Moreover,
`in some scenarios where,
`for
`instance, people at mutually remote locations are jointly
`executing a task simultaneously, e.g., using a manual pre(cid:173)
`recorded in video form to repair a large online system,
`precise synchronization of a telephone message, the presen(cid:173)
`tation and action based on the message and/or the presen(cid:173)
`tation may be necessary.
`To achieve precise synchronization, the present invention
`compares corresponding content or "landmarks" of pairs of
`video playbacks
`to be synchronized, determines video
`replay "distance" between the landmark pairs, and slows
`down or speeds up selected playbacks in accordance with
`these distances.
`U.S. Pat. No. 5,870,754 to Dimitrova et al. ("Dimitrova"),
`entitled "Video Retrieval of MPEG Compressed Sequences
`Using DC and Motion Signatures", and incorporated herein
`by reference, compares "DC+ M signatures" of a query video
`clip to DC+M signatures in a database to retrieve a video
`sequence whose content is similar to that of the query video
`clip, where a video clip is defined as a sequence of video
`frames.
`In one Dimitrova embodiment, DC coefficient informa(cid:173)
`tion from an I frame and motion vector information from the
`
`4
`following frame are combined to form a digital signature,
`hence the term "DC+M signature".
`An "I frame", under the MPEG (Moving Picture Experts
`Group) compression standard, is an intraframe coded frame,
`which is a coding of a single snapshot of an moving image.
`Interspersed between I frames are interframe coded frames
`comprised of information that represents merely a difference
`between
`the current state of the moving
`image and a
`reference state of the moving image as it existed at a
`previous moment.
`The signature embodies characteristics of the frames it
`represents, but uses less data. Signatures of respective I
`frames in a query video clip are compared to respective I
`frame signatures in a database video clip. The total Ham(cid:173)
`ming distance between the signatures of the query clip and
`the signatures of a database clip is calculated. (The total
`Hamming distance is the sum of the Hamming distances
`between respective signatures of query/database frame pairs
`of a current query clip and database clip, where the Ham(cid:173)
`ming distance between two signatures is based on a bit-by(cid:173)
`bit comparison between the signatures, as explained in
`Dimitrova).
`Specifically, the methodology shifts iteratively along the
`database to define a current database video clip, simulta(cid:173)
`neously adding, to the clip, database frames (I frames) and
`dropping database frames (I frames), with each iteration.
`The total Hamming distance is recalculated at each iteration,
`and the minimum Hamming distance over all iterations
`identifies the database video clip that most resembles the
`query video clip.
`An advantage to using Dimitrova signatures is that they
`can be derived without the overhead of fully decompressing
`the image bit stream. Thus, the Huffman or arithmetic
`coding can be decoded to leave the bit stream in partially
`decoded form so that, without further decompression, fre(cid:173)
`quency components such as DC coefficients of an image
`transform such as the discrete cosine transform (DCT) can
`be utilized in forming the signature, as explained more fully
`in Dimitrova.
`In the Dimitrova embodiment described above, I frames
`are used as the "representative frames", i.e. frames for which
`signatures are derived, if key frames have not been identified
`in the video sequence previous to Dimitrova's processing.
`Key frames are frames at shot boundaries, where a shot is a
`45 video sequence of a scene. Typically, there are a thousand or
`more shots in a movie. In another embodiment, Dimitrova
`uses all frames as representative frames.
`The present invention has a goal similar to that of Dim(cid:173)
`itrova, to compare characteristics of two video streams,
`50 except that the present invention uses the comparison to
`synchronize presentation of renderable content of the
`streams, whereas Dimitrova merely seeks a video clip simi(cid:173)
`lar to the query video clip. To adapt Dimitrova matching for
`the present invention, query frames are compared not against
`55 database frames, as in Dimitrova, but against frames in the
`participant's copy of the video, so that presentation of the
`video by the initiator and by the participant can, as a result,
`be made synchronous. Also, for the sake of processing
`speed, preferably a single query signature, corresponding
`60 primarily to a single frame, is transmitted to the participant
`for comparison, rather than transmitting all the signatures of
`a Dimitrova query video clip, which correspond primarily
`each to a separate frame. Accordingly, since a single query
`signature is compared, in each iteration, to a single candidate
`65 participant frame, the Hamming distance between the sig(cid:173)
`natures of that pair of frames is calculated. The overhead of
`a "total Haniming distance" calculation is thereby avoided.
`
`
`PAGE 12 OF 20
`
`SONOS EXHIBIT 1007
`IPR of U.S. Pat. No. 8,942,252
`
`

`

`US 7,269,338 B2
`
`5
`SUMMARY OF THE INVENTION
`
`6
`DETAILED DESCRIPTION OF THE
`PRESENTLY PREFERRED EMBODIMENTS
`
`5
`
`The present invention provides a system that allows two
`or more people with personal video recorders (PVRs) to
`precisely synchronize their time-shifted viewing.
`In one aspect, the present invention is directed to an
`apparatus for synchronizing presentation of renderable con(cid:173)
`tent of two digital bit streams that reside in respective
`storage media. Renderable content, as used herein, refers to 10
`content that is presentable in a form that a user can sense,
`e.g. visually or aurally. The apparatus includes a stream
`characteristic unit for deriving information characteristic of
`content of one of said streams and comparing the informa(cid:173)
`tion to information characteristic of content of the other 15
`stream. Content of the one stream is played back by pro(cid:173)
`gressing forward in the one stream. The presentation
`is
`synchronized by modifying
`that progress based on the
`comparison.
`In another aspect, the present invention is directed to
`synchronizing presentation of renderable content of two or
`more digital bit streams that reside in respective storage
`media. One of the streams is received, and information
`characteristic of content of the received stream is derived
`and incorporated into a signal which is outputted. When the
`signal is received, its characteristic information is compared
`to respective information characteristic of content of one or
`more of the other digital bit streams. Presentation of the two
`or more digital bit streams is synchronized by modifying the 30
`forward progress in the respective playbacks of the two or
`more streams based on the comparisons.
`Other objects and features of the present invention will
`become apparent from the following detailed description
`considered in conjunction with the accompanying drawings. 35
`It is to be understood, however, that the drawings are
`designed solely for purposes of illustration and not as a
`definition of the limits of the invention, for which reference
`should be made to the appended claims. It should be further
`understood that the drawings are not necessarily drawn to 40
`scale and that, unless otherwise indicated, they are merely
`intended to conceptually illustrate the structures and proce(cid:173)
`dures described herein.
`
`Referring to FIG. 1, an example of a synchronized PVR
`viewing system 110 in accordance with the present invention
`is illustrated. As illustrated in FIG. 1, broadcasts 112a, b of
`a television program from network A is made pursuant to
`any communication means known to one having ordinary
`skill in the art, such as cable, digital cable, satellite, antenna,
`over the Internet or combinations thereof. The same network
`production, e.g., specific baseball game, is transmitted to
`each of House 1 and House 2. As the broadcasts 112a, b
`enter House 1 and House 2, respectively, they are received
`by receivers 113a, b housed within each of the respective
`PVRs 114a, b. Hereinafter, the suffix "a" refers to the
`"initiator", and the suffix "b" refers to a "participant", in a
`synchronized viewing session. Initially, the "initiator" is the
`PVR that starts the session, although that role is handed off
`20 to any PVR that, as directed by its user, performs a control
`function (e.g., stop, pause, fast forward, reverse). All other
`PVRs participating
`in
`the session are "participants".
`Depending on context, the user of the initiator PVR is
`referred to as the "initiator", and the user of a participant
`25 PVR is referred to as a "participant". Since the broadcasts
`ll2a, b are received by the user, the user has the ability to
`record the broadcast 112a, b in the PVR 114a, b. The
`broadcast may be received in a receiver housed within a
`set-top box, DVD player, VHS player, personal computer,
`television, etc., and then routed to the PVR 114a, b. The user
`has the ability to record a variety of different programs in the
`PVR 114a, b along the bandwidth of the incoming broadcast
`signal. Additionally, since the broadcast signal is routed to
`each house separately, each house has the ability to sepa(cid:173)
`rately turn the broadcast signal on or off. The PVR 114a, b
`has an output that is connected to an input on a television
`122a, b so that the user may view the recorded programs on
`the television 122a, b.
`A remote control 116a, bis commonly associated with the
`personal video recorder 114a, b to allow the user to operate
`the personal video recorder 114a, b remotely. Typically, the
`remote control 116a, bis configured to transmit an infrared
`signal to the television 122a, b.
`The system 110 preferably has two communication net(cid:173)
`works associated therewith. The first is an Internet network
`118 that interconnects the PVRs 114a, b located at the two
`different locations (e.g. House 1 and House 2). The Internet
`network 118 supplies the means 119 for communicating
`information between the PVRs 114a, b such that synchro(cid:173)
`nization may be achieved. The second communication net-
`work is a telephone network 120, e.g., public switched
`telephone network (PSTN) or a private network, which
`provides a communication means for the two users to
`55 communicate while they are simultaneously viewing the
`recorded program 112a, b. Alternatively, either or both of the
`two communications networks may comprise Internet and/
`or telephone components.
`In order to implement the system 110, one of the users,
`e.g. user 1 from House 1, will call the other user, i.e. user 2
`from House 2, and the respective users will agree to simul(cid:173)
`taneously and synchronously watch a pre-recorded broad(cid:173)
`cast on their respective
`televisions. Here, pre-recorded
`broadcasts are intended to include live broadcasts that have
`65 been buffered for a delay period, which some receivers have
`the ability to do, so that the broadcast can be replayed
`continuously in a staggered time frame without commer-
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`In the drawings, in which like reference numerals identify
`similar or identical elements throughout the several views:
`FIG. 1 is an illustration of a synchronized viewing system
`with which the present invention is usable;
`FIG. 2 is a diagram of the structure of a personal video
`recorder (PVR) in accordance with the present invention;
`FIG. 3 is a diagram of a component of a PVR in
`accordance with the present invention;
`FIG. 4 is a message flow diagram demonstrating a syn(cid:173)
`chronization technique in accordance with the present inven(cid:173)
`tion;
`FIG. 5 is a message format diagram for messages shown
`in FIG. 4;
`FIG. 6 is a flow chart of the process of joining an ongoing
`session in accordance with the present invention;
`FIGS. 7 A and 7B are flow charts of time synchronization
`in accordance with the present invention; and
`FIG. 8 is a flow chart depicting frame synchronization in
`accordance with the present invention.
`
`45
`
`50
`
`60
`
`
`PAGE 13 OF 20
`
`SONOS EXHIBIT 1007
`IPR of U.S. Pat. No. 8,942,252
`
`

`

`US 7,269,338 B2
`
`5
`
`7
`cials. After the users agree to view a program in synchro(cid:173)
`nization, one of them would take the lead to initiate the
`system.
`Assuming that user 1 takes the lead as the "initiator", user
`1, via a remote control 116, via controls on the PVR 114a
`itself, or via control commands displayed on the television
`and activated by an input device such as a keyboard or
`remote controller, would activate a menu for synchronized
`viewing on the PVR 114a. User 1 would respond to and send
`the menu. Resulting other menus would be sent to user 2.
`The users would each indicate by their respective response
`to the menus they received whether or not they will be
`participating
`in the session. Based on the responses, a
`synchronized viewing session is established to begin at an
`agreed upon time. The agreed upon time may be a universal
`time, such as 9:00 P.M., or a relative time, such as in 5
`minutes. A series of menus for establishing a viewing
`session are discussed in U.S. patent application Ser. No.
`09/894,060, described above. A memory device and a pro(cid:173)
`cessor preferably reside in either the PVRs 114a, b or one of
`the other devices associated with system 110. Programming
`code associated with the system 110 preferably resides in the
`memory device and is processed by the processor.
`More than two users may participate in a synchronized
`viewing session; for example, users in three or more remote
`locations may arrange mutually by phone, by menus, by
`e-mail, by Internet chat, etc., to view a particular program
`simultaneously. During the session, viewers may drop out or
`be added to the session. When any session participant
`performs a control function ( e.g., start, pause, rewind, fast
`forward), that participant's PVR 114b broadcasts a com(cid:173)
`mand for that function that is immediately communicated
`and effected in the PVR 114b of each participant, to keep the
`presentation synchronized. To ensure that the PVRs 114a, b
`participating
`in a session remain synchronous, a status
`message is sent out periodically by the "initiator", i.e., the
`PVR 114a that initiated the session. The status message is
`also transmitted with each command that is broadcasted in 40
`response to a participant performing a control function.