Case 1:14-cv-02396-PGG-MHD Document 153-2 Filed 06/28/19 Page 1 of 28
`Case 1:14-cv-02396—PGG-MHD Document 153-2 Filed 06/28/19 Page 1 of 28
`
`EXHIBIT A
`
`EXHIBIT A
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`

`

`Case 1:14-cv-02396-PGG-MHD Document 153-2 Filed 06/28/19 Page 2 of 28
`
`(12) Ulllted States Patent
`Cox
`
`(10) Patent N0.:
`(45) Date of Patent:
`
`US 8,010,988 B2
`Aug. 30, 2011
`
`US008010988B2
`
`a eWS
`
`a
`
`a
`
`6/1987 Okajima
`4,677,455 A
`er a1~
`2
`9/1987 Kiewit 6161.
`4,697,209 A
`4/1988 Thomas et al.
`4,739,398 A
`4,776,017 A 10/1933 pujimoto
`4,805,020 A
`2/1989 Greenberg
`4,843,562 A
`6/1989 Kenyon et a1.
`2
`lsichulze
`cnyon
`,
`,
`5,283,819 A
`2/1994 Glick 61:11.
`5,437,050 A
`7/1995 Lamb 6161.
`5,481,294 A
`1/1996 Thomas et al.
`5,581,658 A 12/1996 O’Hagan 6161.
`5,594,934 A
`1/1997 Lu 6161.
`(Continued)
`OTHER PUBLICATIONS
`Peter N. Yianilos, Excluded Middle Vantage Point Forest for Nearest
`Neighbor Search, Aug‘ 1, 1999, pp‘ 142,,
`(Continued)
`Primary Examiner * Brian T Pendleton
`Assislanl Examiner * Cai Chen
`74 All
`A I
`F' *Amt Rtht~ &
`(3 bgnStei;rI71J?/i) gen , 0r
`irm
`s er,
`0 s e1n
`
`(54) USING FEATURES EXTRACTED FROM AN
`AUDIO AND/0R VIDEO WORK TO OBTAIN
`INFORMATION ABOUT THE WORK
`(76) Inventor: Ingemar J. Cox, London (GB)
`( * ) Notice:
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`USC' 1549’) by 797 days'
`(21) Appl- No: 11/445,928
`(22) Filed;
`Jun. 2, 2006
`(65)
`Prior Publication Data
`Us 2007/0041667 A1
`Feb. 22, 2007
`
`Related U.S. ApplIcatIon Data
`(63) Continuation-in-part of application No. 09/950,972,
`?led on Sep. 13, 2001, noW Pat. No. 7,058,223.
`(60) Provisional application No. 60/232,618, ?led on Sep.
`14, 2000-
`(51) Int- Cl-
`(2011.01)
`H04N 7/173
`(52) US. Cl. ..................................................... .. 725/110
`(58) Field of Classi?cation Search ...................... .. None
`See application ?le for complete search history.
`References Cited
`US, PATENT DOCUMENTS
`3,919,479 A 1 H1975 Moon et a1‘
`4,230,990 A 10/ 1980 Len, Jr, et 31,
`4,450,531 A
`5/1984 Kenyon et a1.
`4,495,526 A
`1/ 1985 Bamnoff'Rossine
`2
`yslt?gfgsal‘
`4,547,804 A 10/1985 Greenberg
`4,634,966 A
`l/l987 Nakatani et al.
`4,639,779 A
`l/ 1987 Greenberg
`
`(56)
`
`ABSTRACT
`(57)
`Information about an audio or video ?le played on a device is
`_
`_
`_
`_
`provided by (a) extracting features from the audio or video
`?le, (b) communicating the features to a database, and (c)
`receiving the information about the audio or video ?le from
`the database. The information might include a song title, an
`album title, and/ or a performer name. The information might
`include a title of a video Work, a director of the video Work,
`and/or names of performers in the video Work. The informa
`tion might be rendered on an output of the device. The infor
`mation might be stored (e.g., persistently) locally on the
`devlce'
`
`52 Claims, 10 Drawing Sheets
`
`WORK @11
`
`WDRK @12
`
`12
`
`FEATURE
`EXTRACTION
`OPERATIONS
`
`124
`
`DATABASE
`GENERATION
`OPERATIDN(S)
`
`FEATURE
`(VECTOR) LOOKUP
`OPERATIONS)
`
`WlD
`
`INF MAYION OR
`N114 11g
`FEATUREISHVECTOR) WORK 1D ‘Fm
`
`WORK-ASSOClATED
`lNFORMATION LooKuP
`OPERATIONS)
`
`WlD-ACTlON
`INFORMATION __
`
`,-
`
`DATABASE
`GENERATlON
`oPERATloms)
`
`Wm
`
`ACTION
`lNlTlATlON
`OPERATlON(S)
`
`170
`
`

`

`Case 1:14-cv-02396-PGG-MHD Document 153-2 Filed 06/28/19 Page 3 of 28
`
`US 8,010,988 B2
`Page 2
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`,,
`
`m1 eta,
`
`,
`
`,
`
`US. PATENT DOCUMENTS
`2,2533%? 2 13133; goglggleny l
`’
`i
`.0
`erg 6‘ ‘1 ~
`5,701,452 A 12/1997 S1efert
`.
`5,701,542 A 12/1997 Se1fert
`.
`5,724,605 A
`3/1998 W1ssner
`5,745,900 A
`4/1998 Burrows .
`5,798,785 A *
`8/1998 Hendr1cks et al. ............ .. 725/46
`5,850,490 A 12/1998 Johnson
`5,918,223 A
`6/1999 Blum 6‘ 31'
`5,953,415 A
`9/1999 Nielsen
`6,006,256 A 12/1999 Zdepski et al.
`6,011,758 A
`1/2000 Dockes et al.
`6,026,439 A
`2/2000 Chowdhury et al.
`2
`31095315011 it 31~
`6,061,056 A
`50000 Menard et a1‘
`6,088,455 A
`7/2000 Logan et a1‘
`6,088,707 A
`7/ 2000 Bates et a1,
`6,118,450 A
`9/ 2000 Proehl et a1.
`6,119,124 A
`9/2000 Broder et a1~
`6,169,986 B1
`V2001 Bowman
`6’l73’406 Bl
`V2001 Wang
`6,240,409 B1
`5/2001 A1ken
`6,243,725 B1
`6/2001 Hempleman et al.
`6,247,133 B1
`6/2001 Palage et al.
`6,253,193 B 1
`60001 Ginter et a1‘
`6,263,348 B1
`7/2001 Kathrow et al.
`6,330,593 B1
`12/2001 Robens et a1,
`6,345,256 B1
`2/2002 Milsted et al.
`6,349,296 B1
`2/ 2002 Broder et al.
`6,360,215 B1
`3/2002 Judd et 31~
`6,363,377 B1
`3/2002 Kravets et a1~
`6,374,225 B1
`4/2002 Helna’ Jr‘
`638L601 Bl
`4/2002 Fullwara et 31'
`.
`6,385,596 B1
`5/2002 W1ser et al.
`6,408,128 B1* 6/2002 Abecass1s ..................... .. 386/68
`7/2002 Hunado et a1‘
`6,418,421 B1
`6,449,226 B1
`9/2002 Kumagai
`6,452,874 B1
`9/2002 Otsuka et a1‘
`6,477,704 B1
`1 1/ 2002 Cremia
`6,496,802 B1
`12/2002 Van Zoest et a1.
`6,505,160 B1
`1/2003 Levy et al.
`6,550,001 B1
`4/2003 C_0fWiIl et 31~
`6,550,011 B1
`4/2003 Slms, HI
`6,577,746 B1
`6/2003 Evans et 31'
`6,591,245 B1
`7/2003 Khlg
`6,598,228 B2
`7/2003 HeJna, Jr.
`6,609,105 B2
`8/2003 Van Zoest et al.
`6,654,757 B1
`110003 Stem
`6,665,661 B1
`12/2003 Crow 61:11.
`6,675,174 B1
`1/2004 Bone et a1‘
`6,834,308 B1* 12/2004 IkeZoye et a1, ,,,,,,,,,,,,,, ,, 709/231
`6,873,982 B1
`3/2005 Bates et a1.
`6,931,451 B1* 8/2005 Logan et a1. ................ .. 709/231
`6,941,275 B1
`9/2005 SWieEZek
`6,973,461 B1
`12/2005 F1en11ng,_III et a1~
`6,978,419 B1
`12/2005 Kantrowltz
`6’990’453 B2
`V2006 Wang et 31'
`7,013,301 B2
`3/2006 Holm et a1.
`*
`7,058,223 B2
`6/2006 Cox ............................ .. 382/190
`7 106 904 B2
`9/2006 Shima
`731553449 B2 12/2006 pingel et a1‘
`7,158,929 B2
`1/2007 Wouters et a1‘
`7,168,083 B2
`1/2007 Kalker et al.
`7,302,574 B2 11/2007 Conwell et al.
`7,366,718 B1
`4/2008 Pugh et al.
`7,421,723 B2
`9/2008 Hillklless et 31~
`7,477,739 B2
`1/2009 Haltsma et a1~
`7,523,312 B2
`4/2009 Kalker et al.
`7’5 87’728 B2
`9/2009 Wheeler et 31'
`7,647,604 B2
`1/2010 Ramaswamy
`7,650,616 B2
`1/2010 Lee
`7,757,248 B2
`7/2010 Harkness et a1.
`200 U000 l 160 A1 * 50001 Shoff et a1‘ ““““““““““ “ 72551
`2001/0003818 A1
`6/2001 Pingel et al.
`2002/0023020 A1
`2/ 2002 Kenyon e1 31,
`2002/0032698 A1
`3/2002 Cox
`2002/0120925 A1 *
`8/2002 Logan ............................. .. 725/9
`
`.
`
`2002/0156760 A1 10/2002 Lawrence et al.
`2003/0106017 A1
`6/2003 Kanchirayappa et al.
`2003/0146940 A1 *
`8/2003 Ellis et a1. ................... .. 345/811
`2004/0199387 A1 * 10/2004 Wang et al. ................. .. 704/243
`2005/0160363 A1
`7/2005 Bhogal et a1.
`2006/0101069 A1
`5/2006 Bell et al.
`2006/0206462 A1
`9/2006 Barber
`2007/0041667 A1
`2/2007 Cox
`2007/0083510 A1
`4/2007 McArdle
`2007/0118375 A1
`5/2007 Kenyon et a1.
`2008/0091684 A1
`4/2008 Ell'
`131
`2008/0250241 A1 10/2008 G' I: e 151
`“1 er 6
`'
`OTHER PUBLICATIONS
`Baum, L., et al., “A Maximation Technique Occurring in the Statis
`t1cal Analys1s of Probab1l1st1c Funct1ons of Markov Cha1ns , The
`Annals ofMathematicalStatistics,vol.41,No. 1,pp. 164-171 (1970).
`Dempster, A. P., et al., “Maximum Likelihood from Incomplete Data
`via the $EM$ Algorithm”, Journal of the Royal Statistical Society,
`Series B (Methodological), vol. 39, Issue 1, pp. 1-38 (1977).
`Reynolds, D., et al., “Robust Text-Independent Speaker Identi?ca
`tion Using Gaussian Mixture Speaker Models”, IEEE Transactions
`.
`.
`on Speech andAudio Processing, vol. 3, No. 1, pp. 72-83 (Jan. 1995).
`N.
`H J
`t
`1 “Th G .d Fl ' An Ad t bl S
`tr.
`‘Wrge a " e a " ,, e n ‘e' _
`aP ‘1 6’ Ymme 1°
`Mult1key F1le Structure, ACM Transactions on Database Systems,
`VOL 9, N<>~ 1’ PP~ 38-71 (Mar 1984)
`HeintZe, N., “Scalable Document Fingerprinting,” Proc. USENIX
`Workshop on Electronic Commerce (1996).
`Wold, E., et al., “Content-Based Classi?cation, Search, and Retrieval
`ofAudio,” IEEE Multimedia, vol. 3, Issue 3, pp. 27-63 (1996).
`Bhanu, B., et al., “Learning Feature Relevance and Similarity Metrics
`in Image Databases”, Proceedings of the IEEE Workshop on Con
`tentiBasedAccess ofImage and J/ideo Libraries, pp. 14-19 (1998).
`el B1mbo, A., et al , Us1ng We1ghted Spat1al Relat1onsh1ps 1n
`D -
`,,
`.
`.
`.
`.
`.
`.
`R .
`lb V. 1C
`,, I
`D _
`_
`dR _
`1
`etr1eva y 1sua
`ontents , mage escription an
`etrieva ,pp.
`161-192 (1998)
`_
`“
`_
`_
`Indyk, P., and Motwan1, R., Approx1mate Nearest Ne1ghbors:
`Towards Removing the Curse of Dimensionality,” Proceeding of the
`Thirtieth Annual ACM Symposium on Theory of Computing, pp.
`604-613 (1998).
`La Cascia, M., et al , “Combining Textual and Visual Cues for
`Content-based Image Retrieval on the World Wide Web”, Proceed
`ings of the IEEE Workshop on Content-Based Access of Image and
`J/ideo Libraries, pp. 24-29 (1998).
`Y1an1los, N. P., Excluded M1ddle Vantage Po1nt Forests for Nearest
`.
`.
`“
`.
`.
`N . hb
`S
`h,, DIMACS I l
`.
`Ch ”
`e1g or ,
`earc ,
`mp ementation
`a enge,
`ALENEX 99 (1999)
`_
`Yosh1taka, A., et al., “A Survey on Content-Based Retr1eval for Mul
`timedia Databases”, IEEE Transactions on Knowledge and Data
`Engineering, vol. 11, No. 1, pp. 81-93 (Jan/Feb. 1999).
`Lawrence, S., et al., “Digital Libraries and Automonous Citation
`Indexing,” IEEE Computer, pp. 67-71 (Jun. 1999).
`Yianilos, N. P., “Locally Lifting the Curse of Dimensionality for
`Nearest Neighbor Search,” Symposium on DiscreteAlgorithms, Pro
`ceeding of the Eleventh Annual ACM-SIAM symposium on Discrete
`.
`Algorithms, pp. 361-370 (2000).
`K1mura,A., et al., Very Qu1ckAud1o Search1ng: Introduc1ng Global
`Pruning to the Time-Series Active Search,”IEEE Conf onAcoustics,
`Speech and Signal Processing, (ICASSP ’01), vol, 3, pp. 1429- 1432
`(2001)
`ChaveZ, E., et al., “Searching in Metric Spaces”, (Sep. 2001) ACM
`Computing Surveys, vol. 33, No. 3, pp. 273-321 (Sep. 2001).
`Haitsma, J ., et al., “Robust Audio Hashing for Content Identi?cation,
`Int” Workshop on ContentBased Multimedia Indexing, Brescia, Italy
`(Sep‘ 19_21, 2001)‘
`Haitsma, J ., and Walker, T., “A Highly Robust Audio Fingerprinting
`»
`-
`System, Journal ofNeW Music Research, 1744-5027, vol. 32, Issue
`2
`211 221 2003
`’PP~_
`'
`(
`X, _
`_
`_
`S-chle1mer, Saul, et al., W1nnow,1ng: Local Algor1thms for Document
`Flngerprlntlng ACM SIGMOD (Jun 9-12, 2003)
`“Searching Near-Replicas of Images via Clustering” Edward Chang,
`Chen Li, James Wang, Peter Mork, Gio Wiederhold Proc. SPIE
`Symposium ofVoice, Video, and Data Communications, 1999.
`
`“
`
`.
`
`.
`
`.
`
`.
`
`

`

`Case 1:14-cv-02396-PGG-MHD Document 153-2 Filed 06/28/19 Page 4 of 28
`
`US 8,010,988 B2
`Page 3
`
`“Rime: A Replicated Image Detector for the World-Wide Web”
`EdWardY. Chang, James Ze Wang, Chen Li, and Gio Wiederhold,
`SPIE 1998.
`“Safeguarding and charging for information on the internet,” H.
`Garcia-Molina, S. Ketchpel, and N. Shivakumar, Proceedings of
`ICDE, 1998.
`“Copy detection mechanisms for digital documents,” S. Brin and H.
`Garcia-Molina, Proceedings of ACM SIG-MOD, May 1995.
`“The x-tree: An index structure for high-dimensional data,” S.
`Berchtold, Proceedings of the 22nd VLDB, Aug. 1996.
`“The sr-tree: An index structure for high-dimensional nearest neigh
`bor queries,” N. Katayama and S. Satoh, Proceedings of ACM
`SIGMOD, May 1997.
`“The k-d-b-tree: A search structure for large multidimensional
`dynamic indexes,” J. T. Robinson, Proceedings of ACM SIGMOD,
`Apr. 1981.
`“Query by image and video content: the QBIC system,” M. Flickner,
`H. SaWhney, W. Niblack, J. Ashley, Q. Huang, and et al, IEEE
`Computer 28(9), pp. 23{32, 1995.
`“Visual information retrieval,” A. Gupta and R. Jain, Communica
`tions ofthe ACM 40(5), pp. 69-79, 1997.
`“Visualseek: A fully automated content-based image query system,”
`J. R. Smith and S.-F. Chang, ACM Multimedia Conference, 1996.
`“Similarity indexing: Algorithms and performance,” D. A. White and
`R. Jain, Proc. SPIE vol. 2670, San Diego, 1996.
`“The r*-tree: an ef?cient and robust access method for points and
`rectangles,” N. Beckmann, H.-P. Kriegel, R. Schneider, and B.
`Seeger, Proceedings ofACM Sigmod, May 1990.
`
`“R-trees: a dynamic index structure for spatial searching,” A. Gutt
`man, Proceedings ofACM Sigmod, Jun. 1984.
`“Similarity indexing With the ss-tree,” D. A. White and R. Jain,
`Proceedings of the 12th ICDE, Feb. 1996.
`“The tv-tree: an index structure for high-dimensional data,” K.-L.
`Lin, H. V. Jagadish, and C. Faloutsos, VLDB Journal 3 (4), 1994.
`“M-tree: An ef?cient access method for similarity search in metric
`spaces,” P. Ciaccia, M. Patella, and P. ZeZula, Proceedings of the 23rd
`VLDB, Aug. 1997.
`“Nearest neighbor queries,” N. Roussopoulos, S. Kelley, and F.
`Vincent, Proceedings ofACM Sigmod , May 1995.
`“An extensible hashing index for high-dimensional similarity
`search,” C. Li, E. Chang, and J. Z. Wang, Stanford Technical
`Report , Aug. 1998. [NOT AVAILABLE].
`“Two algorithms for nearest-neighbor search in high dimensions” J.
`M. Kleinberg, Proc 29th STOC, 1997.
`“A Density-Based Algorithm for Discovering Clusters in Large Spa
`tial Databases With Noise” Martin Ester, Hans-Peter Kriegel, Jorg
`Sander, Xiaowei Xu Proceedings of 2nd International Conference on
`Knowledge Discovery and Data Mining (KDD-96), 1996.
`“Adaptive Color Image Embeddings for Database Navigation”Yossi
`Rubner, Carlo Tomasi and Leonidas J. Guibas, Proceedings of the
`1998 IEEE Asian Conference on Computer Vision.
`A Quantitative Analysis and Performance Study for Similarity
`Search Methods in High-Dimensional Spaces R. Weber, H-J Schek,
`S. Blott Proc., 24th VLDB Conf. 1998.
`* cited by examiner
`
`

`

`Case 1:14-cv-02396-PGG-MHD Document 153-2 Filed 06/28/19 Page 5 of 28
`
`US. Patent
`
`Aug. 30, 2011
`
`Sheet 1 or 10
`
`US 8,010,988 B2
`
`WORK @n
`
`WORK @t2
`
`FEATURE
`OPERATION(S
`
`FEATURE TO
`WORK '0
`OPERA-moms
`
`DATABASE
`
`(VECTOR) EXTRACTION
`OPERATION(S)
`
`140
`
`120
`
`150
`
`WID
`
`m
`
`FEATURE
`
`(VECTOR) LOOKUP OPERATION(S)
`
`,.
`
`l1__ L N \A ‘ -
`
`FEATURE(S) (VECTOR) WORK ID wJ\112
`
`160
`WORK-ASSOCIATED
`INFORMATION LOOKUP
`OPERATION(S)
`
`WID-ACTION
`INFORMATION
`
`138
`
`DATABASE
`GENERATION
`OPERATION(S)
`
`ACTION
`INITIATION
`OPERATION(S)
`
`170
`
`m
`FIGURE 1
`
`

`

`Case 1:14-cv-02396-PGG-MHD Document 153-2 Filed 06/28/19 Page 6 of 28
`
`US. Patent
`
`Aug. 30, 2011
`
`Sheet 2 0f 10
`
`US 8,010,988 B2
`
`SATELLITE CABLE
`OR TERRESTRIAL
`TV BROADCAST
`
`USER COMPUTER,
`SET-TOP-BOX OR
`EQUIVALENT
`
`FEATURE
`(VECTOR)
`EXTRACTION
`OPERATION(S
`
`FEATURE
`(VECTOR) LOOKUP
`OPERATION(S)
`
`120a/138a
`
`DATABASE
`GENERATION
`OPERATIONS
`
`WORK
`ASSOCIATED
`INFORMATION
`LOOKUP
`
`WI D
`DATA BAS E
`
`WI DAT
`DATABASE
`
`
`
`mm Wm
`
`A D
`
`E 8
`
`ACTION
`INITIATION
`OPERATION(S)
`
`WIDAT
`DATABASE
`
`FIGURE 2
`
`

`

`Case 1:14-cv-02396-PGG-MHD Document 153-2 Filed 06/28/19 Page 7 of 28
`
`US. Patent
`
`Aug. 30, 2011
`
`Sheet 3 0f 10
`
`US 8,010,988 B2
`
`320
`
`SATELLITE, CABLE
`OR TERRESTRIAL
`TV BROADCAST
`
`I’
`I
`
`_ I
`usER COMPUTER,
`SET-TOP-BOX oR
`EQUIVALENT
`
`FEATURE
`(VECTOR)
`EXTRACTION
`OPERATION(S
`
`FEATURE
`(VECTOR) LOOKUP
`OPERATION(S)
`
`ACTION
`INITIATION
`OPERATION(S)
`
`DATABASE
`GENERATION
`OPERAT|ON(S)
`
`WORK-ASSOCIATED
`INFORMATION LOOKUP
`OPERAT|ON(S)
`
`FIGURE 3
`
`

`

`Case 1:14-cv-02396-PGG-MHD Document 153-2 Filed 06/28/19 Page 8 of 28
`
`US. Patent
`
`Aug. 30, 2011
`
`Sheet 4 0f 10
`
`US 8,010,988 B2
`
`420
`
`_
`—'
`I _
`SATELLITE, CABLE
`I
`OR TERRESTRIAL
`USER COMPUTER.
`I
`W BROADCAST
`I
`SET-TOP-BOX OR
`, A EQUIVALENT
`I
`Q:
`
`430
`
`INTERNET
`
`l
`'
`I
`I
`
`140s
`
`1706
`
`AcnoN
`INITIATION
`QPERATIOMS)
`
`|
`I
`I
`I
`
`FEATURE
`(VECTOR)
`EXTRACTION
`OPERAT|ON(S
`--
`
`FEATURE
`(VECTOR) LOOKUP
`OPERAT|ON(S)
`
`WORK-ASSOCIATED
`INFORMATION LOOKUP
`OPERATION(S)
`
`DATABASE
`GENERATION
`OPERATION(S)
`
`110C
`
`WID
`DATABAS E
`
`VVIDAT
`DATABASE
`
`130C
`
`FIGURE 4
`
`

`

`Case 1:14-cv-02396-PGG-MHD Document 153-2 Filed 06/28/19 Page 9 of 28
`
`US. Patent
`
`Aug. 30, 2011
`
`Sheet 5 0f 10
`
`US 8,010,988 B2
`
`
`
`
`
`WORK (\MTH EXTRA-WORK INFOMRATION)
`
`
`
`
`
`
`
`FEATURE REAL-TIME (CENTRALIZED) MONITORING 140d (VECTOR) ExTRAcTION 110d 5
`
`
`
`
`
`
`
`OPERAT|ON(S) EXTRAWORK 5 INFORMATION
`
`_
`
`
`
`WID INFORMATION
`
`3
`
`=
`5
`5
`5
`INFORMATION
`RELATED To WORK :
`5
`5
`5
`
`FEATURE
`(VECTOR) LOOKUP
`
`"
`
`594
`\’\
`l
`EXTRA-WORK WORK
`INFORMATION
`ID 1
`
`(VECTOR)
`'
`-
`
`114d
`
`116d
`
`5
`5
`5
`5
`X’\ 5
`112 I
`5
`-
`
`‘-
`
`ID
`'
`-
`-
`
`' """""""""""""""""""""""""
`
`"""""""
`
`"""""""""" "
`
`EXTRA-WORK INFORMATION
`AGGREGATION OPERATION(S)
`540
`
`LOOKUP
`OPERATION(S)
`
`550
`
`560
`
`WORK-ASSOCIATED
`INFORMATION LOOKUP
`OPERATION(S)
`
`DATABASE
`GENERATION
`OPERAT‘OMS
`
`520
`
`580
`
`QWID
`INFORMATION
`J5
`EXTRA-WORK
`INFORMATION (VECTOR) WORK'D 05-2
`
`DATABAS E
`GENERATION
`OPERATION(S)
`
`WI DAT
`INFORMATION
`
`WORK ID ASSOCIATED INFORMATION (8.9., ACTION)
`
`ACTION
`INITIATION
`OPERATION(S)
`
`LE
`
`570
`
`§3_6J
`FIGURES
`
`

`

`Case 1:14-cv-02396-PGG-MHD Document 153-2 Filed 06/28/19 Page 10 of 28
`Case 1:14-cv-02396-PGG-MHD Document 153-2 Filed 06/28/19 Page 10 of 28
`
`US. Patent
`
`Aug. 30, 2011
`
`Sheet 6 0f 10
`
`US 8,010,988 B2
`
`620
`
`SATELLITE, CABLE
`OR TERRESTRIAL
`TV BROADCAST
`
`I — _ — — — — _ _ ‘1
`USER COMPUTER,
`SET~TOP-BOX OR
`EQUIVALENT
`
`
`
`
`
`
`6_1_0
`
`540a
`
`EXTRA-WORK INFORMATION
`AGGREGATION OPERATION(S)
`
`QUEF‘{Y.-—""
`
`5703
`
`ACTION
`
`
`INITIATION
`
`
`
`OPERATION(S)
`
`
`
`WORK-ASSOCIATED
`
`INFORMATION LOOKUP
`OPERATIONIS)
`
`WI D
`
`DATABAS E
`
`
`<1-ID
`
`520a/538a
`
`
`
`WI DAT
`
`
`DATABASE
`
`QWI D
`
`
`DATABASE
`
`GENERATION
`DATABAS E
`
`
`
`OPERATION(S
`
`
`MONITORING AND QUERY RESOLUTION CENTER
`
`5“
`
`HGURE6
`
`

`

`Case 1:14-cv-02396-PGG-MHD Document 153-2 Filed 06/28/19 Page 11 of 28
`
`US. Patent
`
`Aug. 30, 2011
`
`Sheet 7 0f 10
`
`US 8,010,988 B2
`
`720
`
`SATELLITE, CABLE
`OR TERRESTRIAL
`TV BROADCAST
`
`ALErF'
`| USER COMPEREE'T-TOP-BGT OR EQUIV
`
`EXTRA-WORK INFORMATION
`AGGREGATION OPERATION(S)
`
`QUERY
`
`ACTION
`INITIATION
`OPERATION(S)
`
`WWW
`wv
`OPN a
`NEW % Nmo n
`
`_ O D
`
`_ mm A w_
`P m
`K“ S R OA QAU E
`
`0 4 7
`
`WA TER
`m DEP
`My AMAIU E DB ?RM R
`NS E “Em 7
`T ANE U
`so nlu
`F
`
`_ \1 WW Dawn. 0
`
`N R
`
`6 I 0%
`
`_ AAR ATmN
`T mm MM
`_ @LN m m
`.. TO$I IA R \ AO(\ WT E
`ST||A ?mNC
`
`RRP AEMT
`000 TN U
`wmm N@ E_
`KME BMT ©_
`A EL
`
`P
`
`‘a T
`
`x _
`
`

`

`Case 1:14-cv-02396-PGG-MHD Document 153-2 Filed 06/28/19 Page 12 of 28
`
`US. Patent
`
`Aug. 30, 2011
`
`Sheet 8 0f 10
`
`US 8,010,988 B2
`
`860
`
`SATELLITE TV BROADCAST
`
`MONITORING
`
`CENTER I 840a
`\0 830
`
`820
`
`SATELLITE, CABLE
`
`OR TERRESTRIAL 7 TV BROADCAST
`
`810
`\J\ HOME VIEWERS
`
`812
`
`84Gb
`
`DATABASE
`CENTER
`
`850
`
`E-COMMERCE
`OR OTHER WEB
`SITE
`
`8 0
`FIGURE 8
`
`

`

`0
`Case 1:14-cv-02396-PGG-MHD Document 153-2 Filed 06/28/19 Page 13 of 28
`1
`b._H
`6
`R0U
`g
`
`cm
`
`mt
`
`wndzz<xo
`
`2mor.mmm
`
`MU
`
`d5992:
`
`%>zmo<d
`
`009S0m<udzz<zo
`
`mM,£029VEQS
`
`3w19e09
`
`m2
`
`,mBmMEDOE
`
`
`
`MmG0.5umA
`
`Dm..__..H2,mgmg::5m2:2;m2:
`
`D10391033;>2”mo/E5mafia
`
`mmv38no.V302,82:H9vac;«$2No.v30;
`Sm:28”fizz/Eu82”$2on$0£2on
`
`%mmifizmévac;
`
`CU
`
`mS.m2:
`
`LP
`
`4amain.
`
`
`
`

`

`Case 1:14-cv-02396-PGG-MHD Document 153-2 Filed 06/28/19 Page 14 of 28
`
`US. Patent
`
`Aug. 30, 2011
`
`Sheet 10 0f 10
`
`US 8,010,988 B2
`
`UNIQUE ID: 15642
`
`\.f\ 1010
`
`PRODUCT: COCA COLA
`
`\J\ 1020
`
`CATEGORY: SODA
`
`\r\ 1030
`
`MANUFACTURER; COCA COLA
`
`\r\ 1040
`
`URL http://www.cocacola.com
`
`\J\ 1050
`
`OTHER DATA
`
`\J\ 1060
`
`1000
`
`FIGURE 10
`
`

`

`Case 1:14-cv-02396-PGG-MHD Document 153-2 Filed 06/28/19 Page 15 of 28
`
`US 8,010,988 B2
`
`1
`USING FEATURES EXTRACTED FROM AN
`AUDIO AND/OR VIDEO WORK TO OBTAIN
`INFORMATION ABOUT THE WORK
`
`§0. RELATED APPLICATIONS
`The present application is a continuation-in-part of US.
`patent application Ser. No. 09/950,972 (incorporated herein
`by reference), titled “IDENTIFYING WORKS FOR INITI
`ATING A WORK-BASED ACTION, SUCH AS AN
`ACTION ON THE INTERNET,” ?led on Sep. 13, 2001 now
`US. Pat. No. 7,058,223, and listing Ingemar J. Cox as the
`inventor, Which application claims bene?t to the ?ling date of
`provisional patent application Ser. No. 60/232,618 (incorpo
`rated herein by reference), titled “Identifying and linking
`television, audio, print and other media to the Internet”, ?led
`on Sep. 14, 2000 and listing Ingemar J. Cox as the inventor.
`§1. BACKGROUND OF THE INVENTION
`§1.1 Field of the Invention
`The present invention concerns linking traditional media to
`neW interactive media, such as that provided over the Internet
`for example. In particular, the present invention concerns
`identifying a Work (e.g., content or an advertisement deliv
`ered via print media, or via a radio or television broadcast)
`Without the need to modify the Work.
`§1.2 Related Art
`§1.2.1 Opportunities Arising from Linking Works
`Delivered Via Some Traditional Media Channel or
`Conduit to a More Interactive System
`The rapid adoption of the Internet and associated World
`Wide Web has recently spurred interest in linking Works,
`delivered via traditional media channels or conduits, to a
`more interactive system, such as the Internet for example.
`Basically, such linking can be used to (a) promote commerce,
`such as e-commerce, and/ or (b) enhance interest in the Work
`itself by facilitating audience interaction or participation.
`Commerce opportunities include, for example, facilitating
`the placement of direct orders for products, providing product
`coupons, providing further information related to a product,
`product placement, etc.
`In the context of e-commerce, vieWers could request dis
`count vouchers or coupons for vieWed products that are
`redeemable at the point of purchase. E-commerce applica
`tions also extend beyond advertisements. It is noW common
`for television shoWs to include product placements. For
`example, an actor might drink a Coke rather than a Pepsi
`brand of soda, actors and actresses might Wear designer
`labeled clothing such as Calvin Klein, etc. Viewers may Wish
`to purchase similar clothing but may not necessarily be able to
`identify the designer or the particular style directly from the
`shoW. HoWever, With an interactive capability, vieWers Would
`be able to discover this and other information by going to an
`associated Web site. The link to this Web site can be auto
`matically enabled using the invention described herein.
`In the context of facilitating audience interaction or par
`ticipation, there is much interest in the convergence of tele
`vision and computers. Convergence encompasses a very Wide
`range of capabilities. Although a signi?cant effort is being
`directed to video-on-demand applications, in Which there is a
`unique video stream for each user of the service, as Well as to
`transmitting video signals over the Internet, there is also
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`2
`interest in enhancing the television vieWing experience. To
`this end, there have been a number of experiments With inter
`active television in Which vieWers can participate in a live
`broadcast. There are a variety of Ways in Which vieWers can
`participate. For example, during game shoWs, users can
`ansWer the questions and their scores can be tabulated. In
`recent reality-based programming such as the ABC television
`game shoW, “Big Brother”, vieWers can vote on contestants
`Who must leave the shoW, and be eliminated from the com
`petition.
`§1.2.2 Embedding Work Identifying Code or Signals
`Within Works
`Known techniques of linking Works delivered via tradi
`tional media channels to a more interactive system typically
`require some type of code, used to identify the Work, to be
`inserted into the Work before it is delivered via such tradi
`tional media channels. Some examples of such inserted code
`include (i) signals inserted into the vertical blanking interval
`(“VBI”) lines of a (e.g., NTSC) television signal, (ii) Water
`marks embedded into images, (iii) bar codes imposed on
`images, and (iv) tones embedded into music.
`The common technical theme of these proposed imple
`mentations is the insertion of visible or invisible signals into
`the media that can be decoded by a computer. These signals
`can contain a variety of information. In its most direct form,
`the signal may directly encode the URL of the associated Web
`site. HoWever, since the alphanumeric string has variable
`length and is not a particularly ef?cient coding, it is more
`common to encode a unique ID. The computer then accesses
`a database, Which is usually proprietary, and matches the ID
`With the associated Web address. This database can be con
`sidered a form of domain name server, similar to those
`already deployed for netWork addresses. HoWever, in this
`case, the domain name server is proprietary and the addresses
`are unique ID’s.
`There are tWo principal advantages to encoding a propri
`etary identi?er into content. First, as previously mentioned, it
`is a more e?icient use of the available bandWidth and second,
`by directing all tra?ic to a single Web site that contains the
`database, a company can maintain control over the technol
`ogy and gather useful statistics that may then be sold to
`advertisers and publishers.
`As an example of inserting signals into the vertical blank
`ing interval lines of a television signal, RespondTV of San
`Francisco, Calif. embeds identi?cation information into the
`vertical blanking interval of the television signal. The VBI is
`part of the analog video broadcast that is not visible to tele
`vision vieWers. For digital television, it may be possible to
`encode the information in, for example, the motion picture
`experts group (“MPEG”) header. In the USA, the vertical
`blanking interval is currently used to transmit close-caption
`ing information as Well as other information, While in the UK,
`the VBI is used to transmit teletext information. Although the
`close captioning information is guaranteed to be transmitted
`into the home in America, unfortunately, other information is
`not. This is because oWnership of the vertical blanking inter
`val is disputed by content oWners, broadcasters and local
`television operators.
`As an example of embedding Watermarks into images,
`Digimarc of Tualatin, OR embeds Watermarks in print media.
`Invisible Watermarks are neWer than VBI insertion, and have
`the advantage of being independent of the method of broad
`cast. Thus, once the information is embedded, it should
`remain readable Whether the video is transmitted in NTSC,
`PAL or SECAM analog formats or neWer digital formats. It
`
`

`

`Case 1:14-cv-02396-PGG-MHD Document 153-2 Filed 06/28/19 Page 16 of 28
`
`US 8,010,988 B2
`
`3
`should be more reliable than using the vertical blanking inter
`val in television applications. Unfortunately, however, Water
`marks still require modi?cation of the broadcast signal Which
`is problematic for a number of economic, logistical, legal
`(permission to alter the content is needed) and quality control
`(the content may be degraded by the addition of a Watermark)
`reasons.
`As an example of imposing bar codes on images, print
`advertisers are currently testing a technology that alloWs an
`advertisement to be shoWn to a camera, scanner or bar code
`reader that is connected to a personal computer (“PC”). The
`captured image is then analyZed to determine an associated
`Web site that the PC’s broWser then accesses. For example,
`GoCode of Draper, UT embeds small tWo-dimensional bar
`codes for print advertisements. The latter signal is read by
`inexpensive barcode readers that can be connected to a PC.
`AirClic of Blue Bell, Pa. provides a combination of barcode
`and Wireless communication to enable Wireless shopping
`through print media. A so-called “CueCat” reads bar codes
`printed in conjunction With advertisements and articles in
`Forbes magaZine. Similar capabilities are being tested for
`television and audio media.
`Machine-readable bar codes are one example of a visible
`signal. The advantage of this technology is that it is very
`mature. HoWever, the fact that the signal is visible is often
`considered a disadvantage since it may detract from the aes
`thetic of the Work delivered via a traditional media channel or
`conduit.
`As an example of embedding tones into music, Digital
`Convergence of Dallas, Tex. proposes to embed identi?cation
`codes into audible music tones broadcast With television sig
`nals.
`All the foregoing techniques of inserting code into a Work
`can be categorized as active techniques in that they must alter
`the existing signal, Whether it is music, print, television or
`other media, such that an identi?cation code is also present.
`There are several disadvantages that active systems share.
`First, there are aesthetic or ?delity issues associated With bar
`codes, audible tones and Watermarks. More importantly, all
`media must be processed, before it is delivered to the enduser,
`to contain these active signals. Even if a system is enthusias
`tically adopted, the logistics involved With inserting bar codes
`or Watermarks into, say every printed advertisement, are for
`midable.
`Further, even if the rate of adoption is very rapid, it never
`theless remains true that during the early deployment of the
`system, most Works Will not be tagged. Thus, consumers that
`are early-adopters Will ?nd that most media is not identi?ed.
`At best, this is frustrating. At Worst, the naive user may
`conclude that the system is not reliable or does not Work at all.
`This erroneous conclusion might have a very adverse effect
`on the adoption rate.
`Further, not only must there be modi?cation to the produc
`tion process, but modi?cations must also be made to the
`equipment in a user’s home. Again, using the example of
`Watermarking of print media, a PC must be ?tted With a
`camera and Watermark detection softWare must be installed.
`In the case of television, the detection of the identi?cation
`signal is likely to occur at the set-top-boxithis is the equip
`ment provided by the local cable television or satellite broad
`casting company. In many cases, this may require modi?ca
`tions to the hardWare, Which is likely to be prohibitively
`expensive. For example, the audible tone used by Digital
`Convergence to recogniZe television content, must be fed
`directly into a sound card in a PC. This requires a physical
`
`4
`connection betWeen the television and the PC, Which may be
`expensive or at least inconvenient, and a sound card may have
`to be purchased.
`
`§l.2.3 Unmet Needs
`In vieW of the foregoing disadvantages of inserting an
`identi?cation code into a Work, thereby altering the existing
`signal, there is a need for techniques of identifying a Work
`Without the need of inserting an identi?cation code into a
`Work. Such an identi?cation code can then be used to invoke
`a Work-related action, such as Work-related commerce meth
`ods and/or to increase audience interest by facilitating audi
`ence interaction and/ or participation.
`§2. SUMMARY OF THE INVENTION
`This patent application describes an alternative solution
`that does not suffer from the problems outlined above. The
`solution is based on direct or indirect recognition of the media
`itself. Direct or indirect recognition refers to the fact that a
`number of possible con?gurations are possible, some of
`Which directly recogniZe the Work on the equipment in a
`user’ s home While other con?gurations perform this recogni
`tion indirectly by transmitting Work-speci?c information to
`one or more remote sites. Neither technique requires the
`embedding of any form of active signal. Instead, When media
`in the form of music, print, television or multimedia is pre
`sented to a personal computer (PC), set-top-box or other
`device, such devices directly or indirectly recogniZe the
`media and initiate an action. The set of possible actions is
`potentially in?nite and includes, for example, retrieving fur
`ther information, interacting With a live broadcast, registering
`the user for a service or product, purchasing a product or
`service and/ or receiving discount coupons or certi?cates that
`can be used toWards a purchase.
`Some embodiments consistent With the present invention
`provide a computer-implemented method, apparatus, or com
`puter-executable programs for linking a media Work to an
`action. Such embodiments might (a) extract features from the
`media Work, (b) determine an identi?cation of the media
`Work based on the features extracted, and (c) determine an
`action based on the identi?cation of the media Work deter
`mined. In some embodiments consistent With the present
`invention, the media Work is an audio signal. The audio signal
`might be obtained from a broadcast, or an audio ?le format. In
`other embodiments consistent With the present invention, the
`media Work is a video signal. The video signal might be
`obtained from a broadcast, or a video ?le format.
`Some embodiments consist