Case 1:14-cv-02396-PGG-MHD Document 148-4 Filed 05/30/19 Page 1 of 28
`Case 1:14-cv-02396—PGG-MHD Document 148-4 Filed 05/30/19 Page 1 of 28
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`EXHIBIT 1
`
`EXHIBIT 1
`
`

`

`Case 1:14-cv-02396-PGG-MHD Document 148-4 Filed 05/30/19 Page 2 of 28
`
`
`
`
`I 1111111111111111 11111 1111111111 1111111111 111111111111111 IIIIII IIII IIII IIII
`US008010988B2
`
`02) United States Patent
`Cox
`
`(10) Patent No.:
`(45) Date of Patent:
`
`US 8,010,988 B2
`Aug. 30, 2011
`
`(54) USING FEATURES EXTRACTED FROM AN
`AUDIO AND/OR VIDEO WORK TO OBTAIN
`INFORMATION ABOUT THE WORK
`
`(76)
`
`Inventor:
`
`Ingemar J. Cox , London (GB)
`
`( <•) Notice:
`
`Subject to any disclaimer, the tem1 of this
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 797 days.
`
`(21) Appl. No.: 11/445,928
`
`(22) Filed:
`
`Jun.2,2 006
`
`(65)
`
`Prior Publication Data
`
`US 2007/0041667 Al
`
`Feb.22,2007
`
`Related U.S. Applicati on Data
`
`(63) Continuation-in-par t of application No. 09/950,972,
`filed on Sep. 13, 2001, now Pat. No. 7,058,223.
`
`(60) Provisional application No. 60/232,618, filed on Sep.
`14, 2000.
`
`(51)
`
`Int. Cl.
`H04N7 /173
`(20 11.01)
`(52) U.S. Cl. ....................................................... 725/110
`(58) Field of Class ification Se arch ........................ None
`See application file for complete search history.
`
`(56)
`
`Refe rences Cite d
`
`U.S. PATENT DOCUMENTS
`3,919,479 A
`11/ 1975 Moon et al.
`4,230 ,990 A
`10/ 1980 Lert, Jr. et al.
`5/ 1984 Kenyon et al.
`4,450,531 A
`4,495 ,526 A
`1/ 1985 Ba.ranoff-Rossine
`2/1985 Matthews
`4,499 ,601 A
`4.511 ,917 A
`4/ 1985 Kohler etal.
`4,547,804 A
`10/ 1985 Greenberg
`4,634 ,966 A
`1/ 1987 Nakatani et al.
`4,639,779 A
`1/ 1987 Greenberg
`
`4,677 ,455 A
`4,677,466 A
`4,682 ,370 A
`4,697,209 A
`4,739 ,398 A
`4,776,017 A
`4,805 ,020 A
`4,843 ,562 A
`4,9 18,730 A
`5,210 ,820 A
`5,283 ,819 A
`5,437 ,050 A
`5,481 ,294 A
`5,581 ,658 A
`5,594 ,934 A
`
`6/ 1987 Okajima
`6/ 1987 Le,t , Jr. et al.
`7/ 1987 Matthews
`9/ 1987 Kiewit et al.
`4/ 1988 Thomas et al.
`10/ 1988 Fujimoto
`2/ 1989 Greenberg
`6/ 1989 Kenyon el al.
`4/ I 990 Schulze
`5/ 1993 Kenyon
`2/1994 Glicketa l.
`7/ 1995 Lambetal.
`1/1996 Tho masetal.
`12/ 1996 O'Hagan etal.
`1/1997 Luetal.
`(Continued)
`
`OTHER PUBLICATIONS
`
`Peter N. Yianilos, Excluded Middle Vantage Point Forest for Nearest
`Neighbor Search, Aug. 1, 1999, pp. 1-12.*
`
`(Continued)
`
`Primary Examiner - Brian T Pendleton
`Assistant Examiner - Cai Chen
`(74) Attorney, Agent, or Firm - Amster, Rothstein &
`Ebenstein LLP
`
`ABSTR"-CT
`(57)
`Information about an audio or video file played on a device is
`provided by (a) extracting features from the audio or video
`file, (b) conummicating the features to a database, and (c)
`receiving the information about the audio or video file from
`the database. The infom1ation might include a song title, an
`album title, ancVor a perfonner name. TI1e information might
`include a title of a video work, a director of the video work,
`and/or names of perfonners in the video work. TI1e infonna(cid:173)
`tion might be rendered on an output of the device . The infor(cid:173)
`mation might be stored (e.g., persistently) locally on the
`device.
`
`52 Claim s, 10 Dra wing Sheets
`
`.,.
`
`

`

`Case 1:14-cv-02396-PGG-MHD Document 148-4 Filed 05/30/19 Page 3 of 28
`
`U.S. PATENT DOCUMENTS
`5,629,739 A
`5/ 1997 Dougherty
`5,692,213 A
`1111997 Godlberg et al.
`5,701,452 A
`12/1997 Siefert
`5,701,542 A
`12/1997 Seifert
`5,724,605 A
`3/ 1998 Wissner
`5,745,900 A
`4/ 1998 Burrows
`5,798,785 A *
`8/ 1998 Hendricks et al.
`5,850,490 A
`12/1998 Johnson
`5,918,223 A
`6/1999 Blum et al.
`5,953,415 A
`9/ 1999 Nielsen
`6,006,256 A
`12/1999 Zdepski et al.
`6,011,758 A
`1/2000 Dockes et al.
`6,026,439 A
`2/2000 Chowdhury et al.
`6,044,402 A
`3/2000 Jacobson et al.
`6,052,693 A
`4/2000 Smith et al.
`6,061,056 A
`5/2000 Menard et al.
`6,088,455 A
`7/2000 Logan et al.
`6,088,707 A
`7/2000 Bates et al.
`6,118,450 A
`9/2000 Proehl et al.
`6,119,124 A
`9/2000 Broder et al.
`6,169,986 Bl
`1/2001 Bowman
`6,173,406 Bl
`1/2001 Wang
`6,240,409 Bl
`5/2001 Aiken
`6,243,725 Bl
`6/2001 Hempleman et al.
`6,247,133 Bl
`6/2001 Palage et al.
`6,253,193 Bl
`6/2001 Ginter et al.
`6,263,348 Bl
`7/2001 Kathrow et al.
`6,330,593 Bl
`12/2001 Roberts et al.
`6,345,256 Bl
`2/2002 Milsted et al.
`6,349,296 Bl
`2/2002 Broder et al.
`6,360,215 Bl
`3/2002 Judd et al.
`6,363,377 Bl
`3/2002 Kravets et al.
`6,374,225 Bl
`4/2002 Hejna, Jr.
`6,381,601 Bl
`4/2002 Fujiwara et al.
`6,385,596 Bl
`5/2002 Wiser et al.
`6,408,128 Bl*
`6/2002 Abecassis
`6,418,421 Bl
`7 /2002 Hurtado et al.
`6,449,226 Bl
`9/2002 Kumagai
`6,452,874 Bl
`9/2002 Otsuka et al.
`6,477,704 Bl
`11/2002 Cremia
`6,496,802 Bl
`12/2002 Van Zoest et al.
`6,505,160 Bl
`1/2003 Levy et al.
`6,550,001 Bl
`4/2003 Corwin et al.
`6,550,011 Bl
`4/2003 Sims, III
`6,577,746 Bl
`6/2003 Evans et al.
`6,591,245 Bl
`7/2003 Klug
`6,598,228 B2
`7 /2003 Hejna, Jr.
`6,609,105 B2
`8/2003 Van Zoest et al.
`6,654,757 Bl
`11/2003 Stern
`6,665,661 Bl
`12/2003 Crow et al.
`6,675,174 Bl
`1/2004 Bolle et al.
`6,834,308 Bl *
`12/2004 Ikezoye et al.
`6,873,982 Bl
`3/2005 Bates et al.
`6,931,451 Bl*
`8/2005 Logan et al.
`6,941,275 Bl
`9/2005 Swierczek
`6,973,461 Bl
`12/2005 Fleming, III et al.
`6,978,419 Bl
`12/2005 Kantrowitz
`6,990,453 B2
`1/2006 Wang et al.
`7,013,301 B2
`3/2006 Holm et al.
`7,058,223 B2 *
`6/2006 Cox
`7,106,904 B2
`9/2006 Shima
`7,155,449 B2
`12/2006 Pingel et al.
`7,158,929 B2
`1/2007 Wouters et al.
`7,168,083 B2
`1/2007 Kalker et al.
`7,302,574 B2
`11/2007 Conwell et al.
`7,366,718 Bl
`4/2008 Pugh et al.
`7,421,723 B2
`9/2008 Harkness et al.
`7,477,739 B2
`1/2009 Haitsma et al.
`7,523,312 B2
`4/2009 Kalker et al.
`7,587,728 B2
`9/2009 Wheeler et al.
`7,647,604 B2
`1/2010 Ramaswamy
`7,650,616 B2
`1/2010 Lee
`7,757,248 B2
`7/2010 Harkness et al.
`2001/0001160 Al*
`5/2001 Shoff et al.
`2001/0003818 Al
`6/2001 Pingel et al.
`2002/0023020 Al
`2/2002 Kenyon et al.
`2002/0032698 Al
`3/2002 Cox
`2002/0120925 Al*
`8/2002 Logan
`
`US 8,010,988 B2
`Page 2
`
`345/811
`704/243
`
`2002/0156760 Al
`2003/0106017 Al
`2003/0146940 Al *
`2004/0199387 Al*
`2005/0160363 Al
`2006/0101069 Al
`2006/0206462 Al
`2007/0041667 Al
`2007/0083510 Al
`2007/0118375 Al
`2008/0091684 Al
`2008/0250241 Al
`
`10/2002 Lawrence et al.
`6/2003 Kanchirayappa et al.
`8/2003 Ellis et al.
`10/2004 Wang et al.
`7/2005 Bhogal et al.
`5/2006 Bell et al.
`9/2006 Barber
`2/2007 Cox
`4/2007 McArdle
`5/2007 Kenyon et al.
`4/2008 Ellis et al.
`10/2008 Ginter et al.
`
`OTHER PUBLICATIONS
`
`Baum, L., et al., "A Maximation Technique Occurring in the Statis(cid:173)
`tical Analysis of Probabilistic Functions of Markov Chains", The
`Annals of Mathematical Statistics, vol. 41, No. 1, pp. 164-171 (1970).
`Dempster, A. P., et al., "Maximum Likelihood from Incomplete Data
`via the $EM$ Algorithm", Journal of the Royal Statistical Society,
`Series B (Methodological), vol. 39, Issue 1, pp. 1-38 (1977).
`Reynolds, D., et al., "Robust Text-Independent Speaker Identifica(cid:173)
`tion Using Gaussian Mixture Speaker Models", IEEE Transactions
`on Speech and Audio Processing, vol. 3, No. 1, pp. 72-83 (Jan. 1995).
`Nievergelt, J., et al., "The Grid File: An Adaptable, Symmetric
`Multikey File Structure," ACM Transactions on Database Systems,
`vol. 9, No. 1, pp. 38-71 (Mar. 1984).
`Heintze, N., "Scalable Document Fingerprinting," Proc. USENIX
`Workshop on Electronic Commerce (1996).
`Wold, E., et al., "Content-Based Classification, Search, and Retrieval
`of Audio," IEEE Multimedia, vol. 3, Issue 3, pp. 27-63 (1996).
`Bhanu, B., eta!., "Learning Feature Relevance and Similarity Metrics
`in Image Databases", Proceedings of the IEEE Workshop on Con(cid:173)
`tent-Based Access of Image and Video Libraries, pp. 14-19 ( 1998).
`Del Bimbo, A., et al , "Using Weighted Spatial Relationships in
`Retrieval by Visual Contents", Image Description and Retrieval, pp.
`161-192 (1998).
`Indyk, P., and Motwani, R., "Approximate Nearest Neighbors:
`Towards Removing the Curse of Dimensionality," Proceeding of the
`Thirtieth Annual ACM Symposium on Theory of Computing, pp.
`604-613 (1998).
`La Cascia, M., et al , "Combining Textual and Visual Cues for
`Content-based Image Retrieval on the World Wide Web", Proceed(cid:173)
`ings of the IEEE Workshop on Content-Based Access of Image and
`Video Libraries, pp. 24-29 (1998).
`Yianilos, N. P., "Excluded Middle Vantage Point Forests for Nearest
`Neighbor
`Search," DIMACS
`Implementation
`Challenge,
`ALENEX'99 (1999).
`Yoshitaka, A., et al., "A Survey on Content-Based Retrieval for Mul(cid:173)
`timedia Databases", IEEE Transactions on Knowledge and Data
`Engineering, vol. 11, No. 1, pp. 81-93 (Jan./Feb. 1999).
`Lawrence, S., et al., "Digital Libraries and Automonous Citation
`Indexing," IEEE Computer, pp. 67-71 (Jun. 1999).
`Yianilos, N. P., "Locally Lifting the Curse of Dimensionality for
`Nearest Neighbor Search," Symposium on Discrete Algorithms, Pro(cid:173)
`ceeding of the Eleventh Annual ACM-SIAM symposium on Discrete
`Algorithms, pp. 361-370 (2000).
`Kimura, A., et al., "Very Quick Audio Searching: Introducing Global
`Pruning to the Time-Series Active Search," IEEE Conj on Acoustics,
`Speech and Signal Processing, (ICASSP '01), vol, 3, pp. 1429-1432
`(2001).
`Chavez, E., et al., "Searching in Metric Spaces", (Sep. 2001) ACM
`Computing Surveys, vol. 33, No. 3, pp. 273-321 (Sep. 2001).
`Haitsma, J., et al., "Robust Audio Hashing for Content Identification,
`Int." Workshop on Content Based Multimedia Indexing, Brescia, Italy
`(Sep. 19-21, 2001).
`Haitsma, J., and Walker, T., "A Highly Robust Audio Fingerprinting
`System," Journal of New Music Research, 1744-5027, vol. 32, Issue
`2, pp. 211-221 (2003).
`Schleimer, Saul, et al., "Winnowing: Local Algorithms for Document
`Fingerprinting ACM SIGMOD" (Jun. 9-12, 2003).
`"Searching Near-Replicas oflmages via Clustering" Edward Chang,
`Chen Li, James Wang, Peter Mork, Gio Wiederhold Proc. SPIE
`Symposium ofVoice, Video, and Data Communications, 1999.
`
`725/46
`
`386/68
`
`709/231
`
`709/231
`
`382/190
`
`725/51
`
`725/9
`
`

`

`Case 1:14-cv-02396-PGG-MHD Document 148-4 Filed 05/30/19 Page 4 of 28
`
`US 8,010,988 B2
`Page 3
`
`"Rime: A Replicated Image Detector for the World-Wide Web"
`Edward Y. Chang, James Ze Wang, Chen Li, and Gio Wiederhold,
`SPIE 1998.
`"Safeguarding and charging for information on the internet," H.
`Garcia-Molina, S. Ketchpel, and N. Shivakumar, Proceedings of
`ICDE, 1998.
`"Copy detection mechanisms for digital documents," S. Brin and H.
`Garcia-Molina, Proceedings of ACM SIG-MOD, May 1995.
`"The x-tree: An index structure for high-dimensional data," S.
`Berchtold, Proceedings of the 22nd VLDB, Aug. 1996.
`"The sr-tree: An index structure for high-dimensional nearest neigh(cid:173)
`bor queries," N. Katayama and S. Satoh, Proceedings of ACM
`SIGMOD, May 1997.
`"The k-d-b-tree: A search structure for large multidimensional
`dynamic indexes," J. T. Robinson, Proceedings of ACM SIGMOD,
`Apr. 1981.
`"Query by image and video content: the QBIC system," M. Flickner,
`H. Sawhney, W. Niblack, J. Ashley, Q. Huang, and et al, IEEE
`Computer 28(9), pp. 23{32, 1995.
`"Visual information retrieval," A. Gupta and R. Jain, Communica(cid:173)
`tions of the ACM 40(5), pp. 69-79, 1997.
`"Visualseek: A fully automated content-based image query system,"
`J. R. Smith and S.-F. Chang, ACM Multimedia Conference, 1996.
`"Similarity indexing: Algorithms and performance," D. A. White and
`R. Jain, Proc. SPIE vol. 2670, San Diego, 1996.
`"The r*-tree: an efficient and robust access method for points and
`rectangles," N. Beckmann, H.-P. Kriegel, R. Schneider, and B.
`Seeger, Proceedings of ACM Sigmod, May 1990.
`
`"R-trees: a dynamic index structure for spatial searching," A. Gutt(cid:173)
`man, Proceedings of ACM Sigmod, Jun. 1984.
`"Similarity
`indexing with the ss-tree," D. A. White and R. Jain,
`Proceedings of the 12th ICDE, Feb. 1996.
`"The tv-tree: an index structure for high-dimensional data," K.-L.
`Lin, H. V. Jagadish, and C. Faloutsos, VLDB Journal 3 (4), 1994.
`"M-tree: An efficient access method for similarity search in metric
`spaces," P. Ciaccia, M. Patella, and P. Zezula, Proceedings of the 23rd
`VLDB, Aug. 1997.
`"Nearest neighbor queries," N. Roussopoulos, S. Kelley, and F.
`Vincent, Proceedings of ACM Sigmod, May 1995.
`similarity
`"An extensible hashing
`index for high-dimensional
`search," C. Li, E. Chang, and J. Z. Wang, Stanford Technical
`Report, Aug. 1998. [NOT AVAILABLE].
`"Two algorithms for nearest-neighbor search in high dimensions" J.
`M. Kleinberg, Proc 29th STOC, 1997.
`"A Density-Based Algorithm for Discovering Clusters in Large Spa(cid:173)
`tial Databases with Noise" Martin Ester, Hans-Peter Kriegel, Jiirg
`Sander, Xiaowei Xu Proceedings of 2nd International Conference on
`Knowledge Discovery and Data Mining (KDD-96), 1996.
`"Adaptive Color Image Embeddings for Database Navigation"Yossi
`Rubner, Carlo Tomasi and Leonidas J. Guibas, Proceedings of the
`1998 IEEE Asian Conference on Computer Vision.
`A Quantitative Analysis and Performance Study for Similarity(cid:173)
`Search Methods in High-Dimensional Spaces R. Weber, H-J Schek,
`S. Blott Proc., 24th VLDB Conf. 1998.
`
`* cited by examiner
`
`

`

`Case 1:14-cv-02396-PGG-MHD Document 148-4 Filed 05/30/19 Page 5 of 28
`
`U.S. Patent
`
`Aug. 30, 2011
`
`Sheet 1 of 10
`
`US 8,010,988 B2
`
`WORK@t1
`
`WORK@t2
`
`FEATURE TO
`WORK ID
`TAGGING
`OPERATION($
`
`124
`
`WID
`INFORMATIO N
`
`11Q
`
`FEATURE (S) (VECTOR) WORK ID
`
`..........
`·•. ··.
`,._r-- 112
`
`138
`
`WID-ACT ION
`INFORMAT ION
`
`.............
`
`.. ............ .
`
`...
`...•. •··•;:;~
`WORK ID ASSOCIATED INFORMATION (e.g., ACT ION)
`
`.132~
`
`1.QQ
`FIGURE 1
`
`170
`
`

`

`Case 1:14-cv-02396-PGG-MHD Document 148-4 Filed 05/30/19 Page 6 of 28
`
`U.S. Patent
`
`Aug. 30, 2011
`
`Sheet 2 of 10
`
`US 8,010,988 B2
`
`SA TE LUTE, CABLE
`OR TERRE STRIAL
`TV BROADCA ST
`
`I -
`
`"
`L::::::::..~~::::::;:;
`
`"----~ tor_
`
`------,
`
`USER COMPUTER ,
`SET-TOP-BOX OR
`EQUIVALENT
`
`,=5:~~~
`~
`~cm
`
`110a
`
`WID
`DATABASE
`
`130a
`
`WIDAT
`DATABASE
`
`240
`
`_!:i__l
`
`MONITORING CE NTER
`
`r-
`I
`I
`I
`I
`I
`I
`I
`I
`L _____
`
`110a'
`
`WID
`DATABASE
`
`13oa· I
`I
`I
`I
`I l-
`_ J
`
`WIDA T
`DATAB ASE
`
`170a
`
`--;-----
`
`2 10
`
`FIGURE 2
`
`

`

`Case 1:14-cv-02396-PGG-MHD Document 148-4 Filed 05/30/19 Page 7 of 28
`
`U.S. Patent
`
`Aug. 30, 2011
`
`Sheet 3 of 10
`
`US 8,010,988 B2
`
`320
`
`SATELLITE , CABLE
`OR TERRESTRIAL
`1V BROADCAST
`
`330
`
`I
`I
`I
`I
`I
`f1
`310 I
`I
`L
`
`MONITORING CENTER
`
`' I
`
`I
`I 110b'
`I
`I
`I
`I
`
`WID
`DATABASE
`
`160b
`
`130b
`
`Wl DAT
`DATABASE
`
`-)
`
`340
`
`(cid:143)-
`
`.
`--+-
`i
`
`-
`
`-
`
`----1
`
`USER COMPUTER ,
`SET -TOP-BOX OR
`EQUIVALENT
`
`110b
`
`WID
`DATABASE
`
`-1
`
`-·-
`........
`) ........
`.... ,
`I
`I
`I
`I
`I
`
`FIGURE 3
`
`

`

`Case 1:14-cv-02396-PGG-MHD Document 148-4 Filed 05/30/19 Page 8 of 28
`
`U.S. Patent
`
`Aug. 30, 2011
`
`Sheet 4 of 10
`
`US 8,010,988 B2
`
`420
`
`4-----L--!;
`
`SATELLITE, CABLE
`OR TERRE STRIAL
`
`~(cid:143)-
`
`~t----
`
`I
`
`------,
`
`USER COMPUTER,
`SET-TOP-BOX OR
`EQUIVALENT
`
`~m
`
`170c
`
`__J
`
`110c
`
`WID
`DATABASE
`
`WIDAT
`DATABASE
`
`130c
`
`L ____
`
`_
`
`MONITOR ING CENTER
`
`FIGURE 4
`
`_
`
`___
`
`__
`
`__J
`
`

`

`Case 1:14-cv-02396-PGG-MHD Document 148-4 Filed 05/30/19 Page 9 of 28
`
`U.S. Patent
`
`Aug. 30, 2011
`
`Sheet 5 of 10
`
`US 8,010,988 B2
`
`WORK(WITH
`EXTRA-WORK
`INFOMRATION )
`
`.................... ..... .......... ········ ········ .. ····················· .................. ··········
`REAL-TIME (CENTRALIZED) MONITORING i
`140d
`:
`110d :
`
`WID
`INFORMA TION
`
`' .
`
`'
`'
`'
`'
`'
`
`INFORMATION
`RELATED TO WORK I
`:
`L ...... ~~~~~~~:r~~~·······!·~·
`
`
`594
`
`FEATURE(S) WORK
`
`1--- 0/_E_C_TO_R_) ___
`ID----t ~ ,
`112.d l
`·-···~· ~ ··-·-~·-------•c~ ··--···---···-····--~
`
`EXTRA-WORK
`
`WORK
`
`DAT ABASE
`GENERATION
`OPERATION(S
`
`580
`
`520
`
`510
`
`.. ... . ;• ~
`
`QWIO
`INFORMA TION
`lli
`51
`,,J ~
`EXTRA-WORK
`INFORMATION (VECTO R)
`
`..
`.....
`\ ~ ... ·• .......
`WORK ID
`
`550
`
`538
`
`ASSOCIA TED INFORMATION (e.g., AC TION )
`
`··• ...... ... .
`
`~-'
`FIGURE 5
`
`..
`WOR K ID
`
`570
`
`

`

`Case 1:14-cv-02396-PGG-MHD Document 148-4 Filed 05/30/19 Page 10 of 28
`
`U.S. Patent
`
`Aug. 30, 2011
`
`Sheet 6 of 10
`
`US 8,010,988 B2
`
`620
`
`SATELLITE, CAB LE
`OR TERRESTRIA L
`TV BROADCAST
`
`~~
`
`L:::::~~l===:::;;
`
`---- .
`
`USE R COMP UTER,
`SET-TOP-B OX OR
`EQU IVALEN T
`
`fil.Q
`
`I -
`I
`
`,------,
`
`l(cid:143)
`
`r_
`,---5=:====:e.__
`=
`
`I
`I
`I
`I
`QUE_~Y.-· ....
`.. -----·····
`I .. -··
`..
`-·
`
`______ .. --·r
`..
`..
`-·
`L-------
`,. ........ -·
`..
`~--
`-·
`-- -~ ~-~-
`
`,. ...... •"
`
`---
`
`--
`
`__ ........ -
`
`-
`
`-
`
`7
`
`.... ~ ,"
`..
`
`11oe
`
`520a/538a
`
`WIDAT
`DATA BASE
`
`MONITORING AND QUERY RESOLUTION CEN TER
`
`-_r-
`64°
`
`-
`
`-
`-
`-
`FIGURE 6
`
`-
`
`-
`
`-
`
`QW10
`DATAB ASE
`
`L __
`
`

`

`Case 1:14-cv-02396-PGG-MHD Document 148-4 Filed 05/30/19 Page 11 of 28
`
`U.S. Patent
`
`Aug. 30, 2011
`
`Sheet 7 of 10
`
`US 8,010,988 B2
`
`720
`
`SATELLITE, CABLE
`OR TERRESTRIAL
`TV BROADCAST
`
`I USER COMPUTE R, SET-TOP- BOX OR EQUIVALENT'
`I
`I
`
`-------
`
`.,,.___~ [(cid:143)
`
`
`
`~~=i:-:!......
`
`I
`I
`I
`I
`I
`I
`I
`I
`.
`-(cid:173)
`.
`
`. -::-__.::::-_ - - - -~-.. 1
`
`I
`I
`1,0 I
`l
`I
`I
`
`~08
`L-----===---'
`
`QUE RY
`
`-:-
`
`' I
`
`110f
`
`WID
`DATA BASE
`
`I
`I
`I
`I
`I
`I MONITORING CENTER
`----~
`FIGURE 7
`
`..
`
`OPTIONA L
`PRIVATE
`NETWORK
`
`J
`
`-1
`I
`I
`I
`I
`I
`
`QWID
`DATABASE
`
`560b
`
`74oa
`
`740b~
`
`\ .... 1
`
`J I
`I
`I
`I
`I
`DATABASE I
`I
`I
`I
`
`\MOA T
`
`QUERY RESOLUTION CENTER
`L-------
`
`

`

`Case 1:14-cv-02396-PGG-MHD Document 148-4 Filed 05/30/19 Page 12 of 28
`
`U.S. Patent
`
`Aug. 30, 2011
`
`Sheet 8 of 10
`
`US 8,010,988 B2
`
`820
`
`SATELLITE. CABLE
`OR TERRESTRIAL
`TV BROADCAST
`
`SATELLITE lV BROADCAST
`
`840a
`
`810
`
`MONITORING
`CENTER
`
`HOME VIEWERS
`
`INTERNET
`
`812
`
`840b
`
`DATABASE
`CENTER
`
`850
`
`E-COMMERCE
`OR OTHER WEB
`SITE
`
`FIGURE 8
`
`

`

`93
`V3
`:95
`e?
`5PF
`
`>:
`2°
`()3
`9
`gp—L
`
`"‘
`
`U)
`
`g 1
`
`2
`H
`I—A
`9
`
`O"
`
`Z8886‘010‘8Sfl
`
`~
`
`-
`
`
`
`
`
`Case 1:14-cv-02396-PGG-MHD Document 148-4 Filed 05/30/19 Page 13 of 28
`Case 1:14-Cv-02396-PGG-MHD Document 148-4 Filed 05/30/19 Page 13 of 28
`
`TIME
`TIME
`PLACE
`PLACE
`
`CHANNEL
`CHANNEL
`
`WORK IDENTIFIER
`WORK IDENTIFIER
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`- TIME: 8:00
`TIME: 8:00
`
`~
`
`PLACE: LA
`PLACE: LA
`
`CHANNEL: CBS
`CHANNEL: CBS
`
`WORK ID: 15642
`WORK ID: 15642
`
`~
`
`PLACE: NY
`PLACE: NY
`
`- TIME: 8:01
`
`TIME: 8:01
`
`
`
`CHANNEL: NBC
`CHANNEL: NBC
`
`WORK ID: 19023
`WORK ID: 19023
`
`~
`
`- TIME: 8:02
`
`TIME: 8:02
`
`
`
`PLACE: CHICAGO
`PLACE: CHICAGO
`
`CHANNEL: SCIFI
`CHANNEL: SCIFI
`
`WORK ID: 20564
`WORK ID: 20564
`
`
`
`
`
`V
`
`TIME: 8:01
`
`PLACE: NY
`
`CHANNEL: ABC
`
`WORK ID: 19625
`
`
`
`
`
`
`
`
`
`
`FIGURE 9
`FIGURE 9
`
`

`

`Case 1:14-cv-02396-PGG-MHD Document 148-4 Filed 05/30/19 Page 14 of 28
`
`U.S. Patent
`
`Aug. 30, 2011
`
`Sheet 10 of 10
`
`US 8,010,988 B2
`
`UNIQUE ID: 15642
`
`PRODUCT : COCA COLA
`
`CATEGORY : SODA
`
`MANUFACTURER : COCA COLA
`
`URL http://www .cocacola .com
`
`OTHER DATA
`
`1010
`
`1020
`
`1030
`
`1040
`
`1050
`
`1060
`
`FIGURE 10
`
`

`

`Case 1:14-cv-02396-PGG-MHD Document 148-4 Filed 05/30/19 Page 15 of 28
`
`US 8,010,988 B2
`
`1
`USING FEATURES EXTRACTED FROM AN
`AUDIO AND/OR VIDEO WORK TO OBTAIN
`INFORMATION ABOUT THE WORK
`
`§0. RELATED APPLICATIONS
`
`2
`interest in enhancing the television viewing experience. To
`this end, there have been a number of experiments with inter(cid:173)
`active television in which viewers can participate in a live
`broadcast. There are a variety of ways in which viewers can
`5 participate. For example, during game shows, users can
`answer the questions and their scores can be tabulated. In
`recent reality-based programming such as the ABC television
`game show, "Big Brother", viewers can vote on contestants
`who must leave the show, and be eliminated from the com-
`10 petition.
`
`§1.2.2 Embedding Work Identifying Code or Signals
`within Works
`
`The present application is a continuation-in-part of U.S.
`patent application Ser. No. 09/950,972 (incorporated herein
`by reference), titled "IDENTIFYING WORKS FOR INITI(cid:173)
`ATING A WORK-BASED ACTION, SUCH AS AN
`ACTION ON THE INTERNET," filed on Sep. 13, 2001 now
`U.S. Pat. No. 7,058,223, and listing Ingemar J. Cox as the
`inventor, which application claims benefit to the filing date of
`provisional patent application Ser. No. 60/232,618 (incorpo(cid:173)
`rated herein by reference), titled "Identifying and linking 15
`television, audio, print and other media to the Internet", filed
`on Sep. 14, 2000 and listing Ingemar J. Cox as the inventor.
`
`§1. BACKGROUND OF THE INVENTION
`
`§1.1 Field of the Invention
`
`The present invention concerns linking traditional media to
`new interactive media, such as that provided over the Internet
`for example. In particular, the present invention concerns
`identifying a work (e.g., content or an advertisement deliv(cid:173)
`ered via print media, or via a radio or television broadcast)
`without the need to modify the work.
`
`§ 1.2 Related Art
`
`§ 1.2 .1 Opportunities Arising from Linking Works
`Delivered Via Some Traditional Media Channel or
`Conduit to a More Interactive System
`
`The rapid adoption of the Internet and associated World
`Wide Web has recently spurred interest in linking works,
`delivered via traditional media channels or conduits, to a
`more interactive system, such as the Internet for example.
`Basically, such linking can be used to ( a) promote commerce,
`such as e-commerce, and/or (b) enhance interest in the work
`itself by facilitating audience interaction or participation.
`Commerce opportunities include, for example, facilitating
`the placement of direct orders for products, providing product
`coupons, providing further information related to a product, 45
`product placement, etc.
`In the context of e-commerce, viewers could request dis(cid:173)
`count vouchers or coupons for viewed products that are
`redeemable at the point of purchase. E-commerce applica(cid:173)
`tions also extend beyond advertisements. It is now common
`for television shows to include product placements. For
`example, an actor might drink a Coke rather than a Pepsi
`brand of soda, actors and actresses might wear designer(cid:173)
`labeled clothing such as Calvin Klein, etc. Viewers may wish
`to purchase similar clothing but may not necessarily be able to
`identify the designer or the particular style directly from the
`show. However, with an interactive capability, viewers would
`be able to discover this and other information by going to an
`associated Web site. The link to this Web site can be auto(cid:173)
`matically enabled using the invention described herein.
`In the context of facilitating audience interaction or par(cid:173)
`ticipation, there is much interest in the convergence of tele(cid:173)
`vision and computers. Convergence encompasses a very wide
`range of capabilities. Although a significant effort is being
`directed to video-on-demand applications, in which there is a
`unique video stream for each user of the service, as well as to
`transmitting video signals over the Internet, there is also
`
`Known techniques of linking works delivered via tradi(cid:173)
`tional media channels to a more interactive system typically
`require some type of code, used to identify the work, to be
`inserted into the work before it is delivered via such tradi-
`tional media channels. Some examples of such inserted code
`20 include (i) signals inserted into the vertical blanking interval
`("VBI") lines of a ( e.g., NTSC) television signal, (ii) water(cid:173)
`marks embedded into images, (iii) bar codes imposed on
`images, and (iv) tones embedded into music.
`The common technical theme of these proposed imple-
`25 mentations is the insertion of visible or invisible signals into
`the media that can be decoded by a computer. These signals
`can contain a variety of information. In its most direct form,
`the signal may directly encode the URL of the associated Web
`site. However, since the alphanumeric string has variable
`30 length and is not a particularly efficient coding, it is more
`common to encode a unique ID. The computer then accesses
`a database, which is usually proprietary, and matches the ID
`with the associated web address. This database can be con(cid:173)
`sidered a form of domain name server, similar to those
`35 already deployed for network addresses. However, in this
`case, the domain name server is proprietary and the addresses
`are unique ID's.
`There are two principal advantages to encoding a propri(cid:173)
`etary identifier into content. First, as previously mentioned, it
`40 is a more efficient use of the available bandwidth and second,
`by directing all traffic to a single Web site that contains the
`database, a company can maintain control over the technol(cid:173)
`ogy and gather useful statistics that may then be sold to
`advertisers and publishers.
`As an example of inserting signals into the vertical blank-
`ing interval lines of a television signal, RespondTV of San
`Francisco, Calif. embeds identification information into the
`vertical blanking interval of the television signal. The VBI is
`part of the analog video broadcast that is not visible to tele-
`50 vision viewers. For digital television, it may be possible to
`encode the information in, for example, the motion picture
`experts group ("MPEG") header. In the USA, the vertical
`blanking interval is currently used to transmit close-caption(cid:173)
`ing information as well as other information, while in the UK,
`55 the VBI is used to transmit teletext information. Although the
`close captioning information is guaranteed to be transmitted
`into the home in America, unfortunately, other information is
`not. This is because ownership of the vertical blanking inter(cid:173)
`val is disputed by content owners, broadcasters and local
`60 television operators.
`As an example of embedding watermarks into images,
`Digimarc of Tualatin, OR embeds watermarks in print media.
`Invisible watermarks are newer than VBI insertion, and have
`the advantage of being independent of the method ofbroad-
`65 cast. Thus, once the information is embedded, it should
`remain readable whether the video is transmitted in NTSC,
`PAL or SECAM analog formats or newer digital formats. It
`
`

`

`Case 1:14-cv-02396-PGG-MHD Document 148-4 Filed 05/30/19 Page 16 of 28
`
`US 8,010,988 B2
`
`4
`connection between the television and the PC, which may be
`expensive or at least inconvenient, and a sound card may have
`to be purchased.
`
`§ 1.2.3 Unmet Needs
`
`5
`
`3
`should be more reliable than using the vertical blanking inter(cid:173)
`val in television applications. Unfortunately, however, water(cid:173)
`marks still require modification of the broadcast signal which
`is problematic for a number of economic, logistical, legal
`(permission to alter the content is needed) and quality control
`(the content may be degraded by the addition of a watermark)
`reasons.
`As an example of imposing bar codes on images, print
`advertisers are currently testing a technology that allows an
`advertisement to be shown to a camera, scanner or bar code
`reader that is connected to a personal computer ("PC"). The
`captured image is then analyzed to determine an associated
`Web site that the PC's browser then accesses. For example,
`GoCode of Draper, UT embeds small two-dimensional bar 15
`codes for print advertisements. The latter signal is read by
`inexpensive barcode readers that can be connected to a PC.
`AirClic of Blue Bell, Pa. provides a combination ofbarcode
`and wireless communication
`to enable wireless shopping
`through print media. A so-called "CueCat" reads bar codes 20
`printed in conjunction with advertisements and articles in
`Forbes magazine. Similar capabilities are being tested for
`television and audio media.
`Machine-readable bar codes are one example of a visible
`signal. The advantage of this technology is that it is very 25
`mature. However, the fact that the signal is visible is often
`considered a disadvantage since it may detract from the aes(cid:173)
`thetic of the work delivered via a traditional media channel or
`conduit.
`As an example of embedding tones into music, Digital
`Convergence of Dallas, Tex. proposes to embed identification
`codes into audible music tones broadcast with television sig(cid:173)
`nals.
`All the foregoing techniques of inserting code into a work
`can be categorized as active techniques in that they must alter
`the existing signal, whether it is music, print, television or
`other media, such that an identification code is also present.
`There are several disadvantages that active systems share.
`First, there are aesthetic or fidelity issues associated with bar
`codes, audible tones and watermarks. More importantly, all
`media must be processed, before it is delivered to the end user,
`to contain these active signals. Even if a system is enthusias(cid:173)
`tically adopted, the logistics involved with inserting bar codes
`or watermarks into, say every printed advertisement, are for- 45
`midable.
`Further, even if the rate of adoption is very rapid, it never(cid:173)
`theless remains true that during the early deployment of the
`system, most works will not be tagged. Thus, consumers that
`are early-adopters will find that most media is not identified. 50
`At best, this is frustrating. At worst, the naive user may
`conclude that the system is not reliable or does not work at all.
`This erroneous conclusion might have a very adverse effect
`on the adoption rate.
`Further, not only must there be modification to the produc(cid:173)
`tion process, but modifications must also be made to the
`equipment in a user's home. Again, using the example of
`watermarking of print media, a PC must be fitted with a
`camera and watermark detection software must be installed.
`In the case of television, the detection of the identification
`signal is likely to occur at the set-top-box-this
`is the equip(cid:173)
`ment provided by the local cable television or satellite broad(cid:173)
`casting company. In many cases, this may require modifica(cid:173)
`tions to the hardware, which is likely to be prohibitively
`expensive. For example, the audible tone used by Digital 65
`Convergence to recognize television content, must be fed
`directly into a sound card in a PC. This requires a physical
`
`In view of the foregoing disadvantages of inserting an
`identification code into a work, thereby altering the existing
`signal, there is a need for techniques of identifying a work
`10 without the need of inserting an identification code into a
`work. Such an identification code can then be used to invoke
`a work-related action, such as work-related commerce meth(cid:173)
`ods and/or to increase audience interest by facilitating audi(cid:173)
`ence interaction and/or participation.
`
`§2. SUMMARY OF THE INVENTION
`
`This patent application describes an alternative solution
`that does not suffer from the problems outlined above. The
`solution is based on direct or indirect recognition of the media
`itself. Direct or indirect recognition refers to the fact that a
`number of possible configurations are possible, some of
`which directly recognize the work on the equipment in a
`user's home while other configurations perform this recogni(cid:173)
`tion indirectly by transmitting work-specific information to
`one or more remote sites. Neither technique requires the
`embedding of any form of active signal. Instead, when media
`in the form of music, print, television or multimedia is pre(cid:173)
`sented to a personal computer (PC), set-top-box or other
`30 device, such devices directly or indirectly recognize the
`media and initiate an action. The set of possible actions is
`potentially infinite and includes, for example, retrieving fur(cid:173)
`ther information, interacting with a live broadcast, registering
`the user for a service or product, purchasing a product or
`35 service and/or receiving discount coupons or certificates that
`can be used towards a purchase.
`Some embodiments consistent with the present invention
`provide a computer-implemented method, apparatus, or com(cid:173)
`puter-executable programs for linking a media work to an
`40 action. Such embodiments might (a) extract features from the
`media work, (b) determine an identification of the media
`work based on the features extracted, and ( c) determine an
`action based on the identification of the media work deter-
`mined. In some embodiments consistent with the present
`invention, the media work is an audio signal. The audio signal
`might be obtained from a broadcast, or an audio file format. In
`other embodiments consistent with the present invention, the
`media w