Case 1:14-cv-02396-PGG-MHD Document 148-5 Filed 05/30/19 Page 1 of 30
`Case 1:14-cv-02396—PGG-MHD Document 148-5 Filed 05/30/19 Page 1 of 30
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`EXHIBIT 2
`
`EXHIBIT 2
`
`

`

`Case 1:14-cv-02396-PGG-MHD Document 148-5 Filed 05/30/19 Page 2 of 30
`
`
`
`I 1111111111111111 11111 1111111111 11111 111111111111111 IIIII IIIIII IIII IIII IIII
`US008205237B2
`
`02) United States Patent
`Cox
`
`(10) Patent No.:
`(45) Date of Patent:
`
`US 8,205,237 B2
`*Jun. 19,2 012
`
`(54)
`
`IDENTIFYING WORKS, USING A
`SUB-LI NEAR TIME SEARCH, SUCH AS AN
`APPROXIMATE NEAREST NEIGHBOR
`SEARCH, FOR INITIATING A WORK-BASED
`ACTION, SUCH AS AN ACTION ON THE
`INTERNET
`
`(76)
`
`Inventor :
`
`Ingemar J. Cox, London (GB)
`
`( "') No tice:
`
`Subject to any disclaimer , the term of this
`patent is extended or adjus ted under 35
`U.S.C . 154(b) by 594 days .
`
`This patent is subject to a terminal dis(cid:173)
`claimer.
`
`(21) Appl. No. : 11/977,202
`
`(22) Filed:
`
`Oct. 23, 2007
`
`(65)
`
`Prior Publication Data
`
`US 2008/00600 36 Al
`
`Mar. 6, 200 8
`
`Related U.S. Applicati on Data
`
`(63) Cont inuation of applicatio n No. 11/445,928, filed on
`Jun . 2, 2006, which is a continuation-i n-part of
`application No . 09/950,972, filed on Sep . 13, 200 1,
`now Pat. No . 7,058,223 .
`
`(60) Prov isional application No. 60/232 ,618, filed on Sep .
`14, 2000.
`
`(51)
`
`Int. CI.
`H04N 711 73
`(20 11.01)
`(52) U.S. Cl .
`....................................................... 725/110
`(58) Field of Classification Search ........................ None
`See application file for complete search history.
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCU M ENTS
`11/ 197 5 Moon et al.
`3,919 ,479 A
`JO/ 1980 Le1t. Jr. et al.
`4,230,990 A
`5/ 1984 Kenyon et al.
`4,450 ,531 A
`1/ 1985 Baranoff-Rossine
`4,495 ,526 A
`
`4,499 ,601 A
`4,511 ,917 A
`4,547,804 A
`4,634 ,966 A
`4,639 ,779 A
`4,677 ,455 A
`4,677,466 A
`4,682 ,370 A
`4,697 ,209 A
`4,739 ,398 A
`4,776 ,017 A
`4,805,020 A
`
`2/ I 985 Matthews
`4/ 1985 Kohler et al.
`10/ 1985 Greenberg
`1/1987 Nak.~tani et al.
`111987 Green berg
`6/ 1987 Okajima
`6/ 1987 Le,t, Jr. et al.
`7/ 1987 Matthews
`9/ 1987 Kiewit
`4/ 1988 Thomas et al.
`10/ 1988 Fujimoto
`2/ 1989 Greenberg
`(Continued )
`
`OTHER PUBLICATI ONS
`Peter N. Yianilos , Excluded Middle Vantage Point Forest for Neare st
`Neighbor Search , Aug . 1, 1999, pp . 1-12.*
`
`(Continued )
`
`Primary Examiner - Brian Pendleto n
`Assistant Exam iner - Cai Chen
`(74) Attorney, Ageni, or Pinn - Amster , Rothstein &
`Ebenste in LLP
`
`(57)
`
`ABSTRACT
`
`A media work may be associated with an action by(a) extract(cid:173)
`ing features from the media work, (b) determining an identi (cid:173)
`fication of the media work, based on the features extracted ,
`using a sub-linear time search , such as an approximate nearest
`neighb or search for examp le, and (c) determining an actio n
`based on the iden tification of the media work detemJ.ined. The
`media work may be an aud io work. 1l 1e features extracted
`from the work may include (A) a frequenc y decomposi tion of
`a signal of the audio work , (B) information san1ples of the
`audio work, (C) average intensi ties of sam pled windows of
`the audio work , and/or (D) informat ion from freque ncies of
`the audio work .
`
`40 Claims, 10 Drawing Sheets
`
`

`

`Case 1:14-cv-02396-PGG-MHD Document 148-5 Filed 05/30/19 Page 3 of 30
`
`US 8,205,237 B2
`Page 2
`
`U.S. PATENT DOCUMENTS
`4,843,526 A
`6/1989 Price, III
`4,843,562 A
`6/1989 Kenyon et al.
`4,918,730 A
`4/1990 Schulze
`5,210,820 A
`5/1993 Kenyon
`5,283,819 A
`2/1994 Glick et al.
`5,437,050 A
`7/1995 Lamb eta!.
`5,481,294 A
`1/1996 Thomas et al.
`12/1996 O'Hagan et al.
`5,581,658 A
`5,594,934 A
`1/1997 Lu et al.
`5,629,739 A
`5/1997 Dougherty
`5,692,213 A
`11/1997 Goldberg et al.
`5,701,452 A
`12/1997 Siefert
`5,701,542 A
`12/1997 Sasayama
`5,724,605 A
`3/1998 Wissner
`5,745,900 A
`4/1998 Burrows
`5,798,785 A
`8/1998 Hendricks et al.
`5,850,490 A
`12/1998 Johnson
`5,918,223 A
`6/1999 Blum et al.
`5,953,415 A
`9/1999 Nielsen
`6,006,256 A
`12/1999 Zdepski et al.
`6,011,758 A
`1/2000 Dockes eta!.
`6,026,439 A
`2/2000 Chowdhury et al.
`6,044,402 A
`3/2000 Jacobson et al.
`6,052,693 A
`4/2000 Smith et al.
`6,061,056 A *
`5/2000 Menard et al. ................ 715/704
`6,088,455 A
`7/2000 Logan et al.
`6,088,707 A
`7/2000 Bates et al.
`6,118,450 A
`9/2000 Proehl et al.
`6,119,124 A
`9/2000 Broder et al.
`6,169,986 Bl
`1/2001 Bowman et al.
`6,173,406 Bl
`1/2001 Wang et al.
`6,240,409 Bl
`5/2001 Aiken
`6,243,725 Bl
`6/2001 Hempleman et al.
`6,247,133 Bl
`6/2001 Palage et al.
`6,253,193 Bl
`6/2001 Ginter et al.
`6,263,348 Bl
`7/2001 Kathrow et al.
`6,330,593 Bl
`12/2001 Roberts et al.
`6,345,256 Bl
`2/2002 Milsted et al.
`6,349,296 Bl
`2/2002 Broder
`6,360,215 Bl
`3/2002 Judd et al.
`6,363,377 Bl
`3/2002 Kravets et al.
`6,374,225 Bl
`4/2002 Hejna, Jr.
`6,381,601 Bl
`4/2002 Fujiwara et al.
`6,385,596 Bl
`5/2002 Wiser et al.
`6,408,128 Bl
`6/2002 Abecassis
`6,418,421 Bl
`7/2002 Hurtado et al.
`6,446,068 Bl
`9/2002 Kortge
`6,449,226 Bl
`9/2002 Kumagai
`6,452,874 Bl
`9/2002 Otsuka et al.
`6,477,704 Bl
`11/2002 Cremia
`6,496,802 Bl
`12/2002 Van Zoe st et al.
`6,505,160 Bl
`1/2003 Levy
`6,550,001 Bl
`4/2003 Corwin et al.
`6,550,011 Bl
`4/2003 Sims, III
`6,577,746 Bl
`6/2003 Evans et al.
`6,591,245 Bl
`7/2003 Klug
`6,598,228 B2
`7/2003 Hejna, Jr.
`6,609,105 B2
`8/2003 Van Zoe st et al.
`6,654,757 Bl
`11/2003 Stern
`6,665,661 Bl
`12/2003 Crow et al.
`6,675,174 Bl
`1/2004 Bolle et al.
`6,834,308 Bl * 12/2004 Ikezoye et al. ................ 709/231
`6,873,982 Bl
`3/2005 Bates et al.
`6,931,451 Bl
`8/2005 Logan et al.
`6,941,275 Bl
`9/2005 Swierczek
`6,978,419 Bl
`12/2005 Kantrowitz
`6,978,461 B2
`12/2005 Shapiro et al.
`6,990,453 B2
`1/2006 Wang et al.
`7,013,301 B2
`3/2006 Holm et al.
`7,058,223 B2
`6/2006 Cox
`7,106,904 B2
`9/2006 Shima
`7,155,449 B2
`12/2006 Pingel et al.
`7,158,929 B2
`1/2007 Wouters et al.
`7,168,083 B2
`1/2007 Kalker et al.
`7,302,574 B2
`11/2007 Conwell et al.
`7,366,718 Bl
`4/2008 Pugh et al.
`7,421,723 B2
`9/2008 Harkness et al.
`7,477,739 B2
`1/2009 Haitsma et al.
`
`7,523,312 B2
`7,587,728 B2
`7,647,604 B2
`7,650,616 B2
`7,757,248 B2
`2001/0001160 Al *
`2001/0003818 Al
`2002/0023020 Al
`2002/0032698 Al
`2002/0120925 Al
`2002/0156760 Al
`2003/0106017 Al
`2003/0146940 Al
`2004/0199387 Al*
`2005/0160363 Al
`2006/0101069 Al
`2006/0206462 Al
`2007/0041667 Al
`2007/0083510 Al
`2007/0118375 Al
`2008/0091684 Al
`2008/0250241 Al
`
`4/2009 Kalker et al.
`9/2009 Wheeler et al.
`1/2010 Ramaswamy
`1/2010 Lee
`7/2010 Harkness et al.
`5/2001 Shoff et al.
`..................... 725/51
`6/2001 Pingel et al.
`2/2002 Kenyon et al.
`3/2002 Cox
`8/2002 Logan
`10/2002 Lawrence et al.
`6/2003 Kanchirayappa et al.
`8/2003 Ellis
`................... 704/243
`10/2004 Wang eta!.
`7/2005 Bhogal et al.
`5/2006 Bell et al.
`9/2006 Barber
`2/2007 Cox
`4/2007 McArdle
`5/2007 Kenyon et al.
`4/2008 Ellis et al.
`10/2008 Ginter et al.
`
`OTHER PUBLICATIONS
`
`Peter N. Yianlos, Excluded Middle Vantage Point Forest for Nearest
`Neighbor Search, Aug. 1, 1999, pp. 1-12.*
`P.N. Yianilos, "Locally Lifting the Curse of Dimensionality for Near(cid:173)
`est Neighbor Search" SODA 2000, pp. 361-370.
`Baum, L., et al., "A Maximation Technique Occurring in the Statis(cid:173)
`tical Analysis of Probabilistic Functions of Markov Chains", The
`AnnalsofMathematicalStatistics, vol.41,No.
`l,pp.164-171
`(1970).
`Dempster, A. P., et al., "Maximum Likelihood from Incomplete Data
`via the $EM$ Algorithm", Journal of the Royal Statistical Society,
`Series B (Methodological), vol. 39, Issue 1, pp. 1-38 (1977).
`Reynolds, D., et al., "Robust Text-Independent Speaker Identifica(cid:173)
`tion Using Gaussian Mixture Speaker Models", IEEE Transactions
`on Speech and Audio Processing, vol. 3, No. 1, pp. 72-83 (Jan. 1995).
`Nievergelt, J. et al., "The Grid File: An Adaptable, Symmetric
`Multikey File Structure," ACM Transactions on Database Systems,
`vol. 9, No. 1, pp. 38-71 (Mar. 1984).
`Heintze, N, "Scalable Document Fingerprinting," Proc. USENIX
`Workshop on Electronic Commerce (1996).
`Wold, E, et al., "Content-Based Classification, Search, and Retrieval
`of Audio," IEEE Multimedia, vol. 3, Issue 3, pp. 27-63 (1996).
`Bhanu, B., eta!., "Learning Feature Relevance and Similarity Metrics
`in Image Databases", Proceedings of the IEEE Workshop on Con(cid:173)
`tent-Based Access oflmage and Video Libraries, pp. 14-19 (1998).
`Del Bimbo, A., et al., "Using Weighted Spatial Relationships in
`Retrieval by Visual Contents", Image Description and Retrieval, pp.
`161-192 (1998).
`Indyk, P., and Motwani, R., "Approximate Nearest Neighbors:
`Towards Removing the Curse of Dimensionality," Proceeding of the
`Thirtieth Annual ACM Symposium on Theory of Computing, pp.
`604-613 (1998).
`La Cascia, M., et al., "Combining Textual and Visual Cues for Con(cid:173)
`tent-based Image Retrieval on the World Wide Web", Proceedings of
`the IEEE Workshop on Content-Based Access of Image and Video
`Libraries, pp. 24-29 ( 1998).
`Yoshitaka, A., et al., "A Survey on Content-Based Retrieval for Mul(cid:173)
`timedia Databases", IEEE Transactions on Knowledge and Data
`Engineering, vol. 11, No. 1, pp. 81-93 (Jan./Feb. 1999).
`Lawrence, S., et al., "Digital Libraries and Automonous Citation
`Indexing," IEEE Computer, pp, 67-71 (Jun. 1999).
`Kimura, A, et al., "Very Quick Audio Searching: Introducing Global
`Pruning to the Time-Series Active Search," IEEE Conf on Acoustics,
`Speech and Signal Processing, (ICASSP '01), vol. 3, pp, 1429-1432
`(2001).
`Chavez, E., et al., "Searching in Metric Spaces", ACM Computing
`Surveys, vol. 33, No. 3, pp. 273-321 (Sep. 2001).
`Haitsma, J., et al., "Robust Audio Hashing for Content Identification,
`Int" Workshop on Content Based Multimedia Indexing, Brescia, Italy
`(Sep. 19-21, 2001).
`
`

`

`Case 1:14-cv-02396-PGG-MHD Document 148-5 Filed 05/30/19 Page 4 of 30
`
`US 8,205,237 B2
`Page 3
`
`Haitsma, J., and Walker, T, "A Highly Robust Audio Fingerprinting
`System," Journal of New Music Research, 1744-5027, vol. 32, Issue
`2, pp. 211-221 (2003).
`Schleimer, Saul, et al., "Winnowing: Local Algorithms for Document
`Fingerprinting ACM SIGMOD" (Jun. 9-12, 2003).
`"Searching Near-Replicas oflmages via Clustering" Edward Chang,
`Chen Li, James Wang, Peter Mork, Gio Wiederhold Proc. SPIE
`Symposium ofVoice, Video, and Data Communications, 1999.
`"RIME: A Replicated Image Detector for the World-Wide Web"
`Edward Y. Chang, James Ze Wang, Chen Li, and Gio Wiederhold,
`SPIE 1998.
`"Safeguarding and charging for information on the internet," H.
`Garcia-Molina, S. Ketchpel, and N. Shivakumar, Proceedings of
`ICDE, 1998.
`for digital documents," S. Brin and H.
`"Detection mechanisms
`Garcia-Molina, Proceedings of ACM SIG-MOD, May 1995.
`"The x-tree: An index structure for high-dimensional data," S.
`Berchtold, Proceedings of the 22nd VLDB, Aug. 1996.
`"The sr-tree: An index structure for high-dimensional nearest neigh(cid:173)
`bor queries," N. Katayama and S. Satoh, Proceedings of ACM
`SIGMOD, May 1997.
`"The k-d-b-tree: A search structure for large multidimensional
`dynamic indexes," J. T. Robinson, Proceedings of ACM SIGMOD ,
`Apr. 1981.
`"Query by image and video content: The QBIC system," M. Flickner,
`H. Sawhney, W. Niblack, J. Ashley, Q. Huang, and et al, IEEE
`Computer 28(9), pp. 23{32, 1995.
`"Visual information retrieval," A. Gupta and R. Jain, Communica(cid:173)
`tions of the ACM 40(5), pp. 69-79, 1997.
`"Visualseek: A fully automated content-based image query system,"
`J. R. Smith and S.-F. Chang, ACM Multimedia Conference , 1996.
`"Similarity indexing: Algorithms and performance," D. A. White and
`R. Jain, Proc. SPIE vol. 2670, San Diego, 1996.
`"The r*-tree: an efficient and robust access method, for points and
`rectangles," N. Beckmann, H.-P. Kriegel, R. Schneider, and B.
`Seeger, Proceedings of ACM Sigmod, May 1990.
`"R-trees: a dynamic index structure for spatial searching," A. Gutt(cid:173)
`man, Proceedings of ACM Sigmod, Jun. 1984.
`"Similarity
`indexing with the ss-tree," D. A. White and R. Jain,
`Proceedings of the 12th ICDE, Feb. 1996.
`"The tv-tree: an index structure for high-dimensional data," K.-L.
`Lin, H. V. Jagadish, and C. Faloutsos, VLDB Journal 3 ( 4), 1994.
`"M-tree: An efficient access method for similarity search in metric
`spaces," P. Ciaccia, M. Patella, and P. Zezula, Proceedings of the 23rd
`VLDB , Aug. 1997.
`"Nearest neighbor queries," N. Roussopoulos, S. Kelley, and F.
`Vincent, Proceedings of ACM Sigmod, May 1995.
`similarity
`"An extensible hashing
`index for high-dimensional
`search," C. Li, E. Chang, and J. Z. Wang, Stanford Technical
`Report , Aug. 1998.
`"Two algorithms for nearest-neighbor search in high dimensions" J.
`M. Kleinberg, Proc 29th STOC, 1997.
`"A Density-Based Algorithm for Discovering Clusters in Large Spa(cid:173)
`tial Databases with Noise" Martin Ester, Hans-Peter Kriegel, Jorg
`Sander, Xiaowei Xu Proceedings of 2nd International Conference on
`Knowledge Discovery and Data Mining (KDD-96), 1996.
`"Adaptive Color Image Embeddings for Database Navigation"Yossi
`Rubner, Carlo Tomasi and Leonidas J. Guibas, Proceedings of the
`1998 IEEE Asian Conference on Computer Vision.
`A Quantitative Analysis and Performance Study for Similarity(cid:173)
`Search Methods in High-Dimensional Spaces R. Weber, H-J Schek,
`S. Blott Proc., 24th VLDB Conf. 1998.
`Bouktache, D, "A fast algorithm for the nearest neighbor classifier",
`IEEE Transactions on Pattern Analysis and Machine Intelligence,
`Mar. 1997, pp. 277-282.
`Nene et al., "A simple algorithm for nearest neighbor search in high
`dimensions", IEEE Transactions on Pattern Analysis and Machine
`Intelligence; Sep. 1997, pp. 989-1003.
`Arya et al. "Approximate nearest neighbor queries in fixed dimen(cid:173)
`sions", Proceedings of the 4th annual ACM-SIAM Symposium on
`Discrete algorithms, 1993; pp. 271-280.
`
`K. Fukunaga and P. M. Narendra. A branch and bound algorithm for
`computing
`k-nearest
`neighbors.
`IEEE
`Trans. Comput.,
`C-24:750{753, Jul. 1975.
`C.D. Feustel and L. G. Shapiro. The nearest neighbor problem in an
`abstract metric space. Pattern Recognition Letters, pp. 12 5 { 12 8, Dec.
`1982.
`Dennis Shasha and Tsong-Li Wang. New techniques for best-match
`retrieval. ACM Transactions on Information Systems, 8(2): 140{ 158,
`Apr. 1990.
`J. Uhlmann. Satisfying general proximity/similarity queries with
`metric trees. Information Processing Letters, 40 (4):175{9, Nov.
`1991.
`Sergey Brin, "Near Neighbor Search in Large Metric Spaces", Pro(cid:173)
`ceedings of the 2lstVLDB Conference, Zurich, Switzerland, Sep.
`1995.
`images using the hausdorff
`D. P. Huttenlocher, et al. Comparing
`distance. IEEE Transactions on Pattern Analysis and Machine Intel(cid:173)
`ligence, 15(3):850{63, Sep. 1993.
`Seidl et al. "Optimal multi-step k-nearest neighbor search", Proceed(cid:173)
`ings of ACM SIGMOD international conference on Managemet of
`data, 1998, pp. 154-165.
`W.A. Burkhard and R.M. Keller. Some Approaches to Best-Match
`File Searching. Communications of the ACM. vol. 16, No. 4, Apr.
`1973.
`Kushilevitz et al. "Efficient search for approximate nearest neighbor
`in high dimensional spaces", Proceedings of the 30th annual ACM
`Symposium on Theory of computing, 1998, pp. 614-623.* ;annual
`ACM Symposium on Theory of computing, 1998, pp. 614-623.
`Yianilos, P, "Data structures and algorithms for nearest neighbor
`search in general metric spaces", Proceedings of the ACM-SIAM
`Symposium on Discrete algorithms, 1993, pp. 311-321.
`Ardizzone, Edoardo et al., "Motion and Color-Based Video Indexing
`and Retrieval," Universita di palermo, Departimento di Ingegneria
`Elettrica, pp. 135-139, Viale delle Scienze, Palermo, Italy, IEEE
`1996.
`Deng, Yining et al., "Content-based Search of Video Using Color,
`Texture, and Motion," Dept. of Electrical and Computer Engineering,
`University of California, Santa Barbara, CA, pp. 534-537, IEEE
`1997.
`Fang, Min et al., "Computing Iceberg Queries Efficiently," Dept. of
`Computer Science, Stanford, CA, Paper No. 234, pp. 1-25.
`Flickner, Myron et al., "Query by Image and Video Content: The
`QBIC System," IBM Almaden Research Center, Sep. 1995, pp.
`23-32, IEEE 1995.
`Gargi, U et al., "Performance Characterization and Comparison of
`Video Indexing Algorithms," Dept. of Computer Science and Engi(cid:173)
`neering, The Pennsylvania State University, University Park, PA.
`Gionis, Aristides et al., "Similarity Search in High Demensions via
`Hashing," Dept. of Computer Science, Stanford University, Stanford,
`CA, pp. 518-529, Proceeding
`the 25th VLDB Conference,
`Edinburgh, Scotland, 1999.
`Indyk, Piotr et al., "Approximate Nearest Neighbors: Towards
`Removing the Curse of Dimensionality" (preliminary version) Dept.
`of Computer Science, Stanford University, Stanford, CA, pp. 1-13 &
`i-vii, Jul. 21, 1999.
`Iyengar, Giridharan et al., "Models for automatic classification of
`video sequences," MIT Media Laboratory, Cambridge, MA.
`Jain, Anil K., et al., "Image Retrieval using Color and Shape," Dept.
`of Computer Science, Michigan State University, Eas Lansing, MI,
`pp. 1-24, May 15, 1995.
`Ogle, Virginia E., et al., "Chabot: Retrieval from a Relational Data(cid:173)
`base of Images," University of California at Berkeley, Computer pp.
`40-48, IEEE 1995.
`Pentland, A. et al., "Photobook: Content-Based Manipulation of
`Image Databases," Perceptual Computing Section, The Media Labo(cid:173)
`ratory, Massachusetts
`Institute of Tech., International Jorunal of
`Computer Vision 18(3), pp. 233-254 (1996), 1996 Kluwer Academic
`Publishers. Manuf. in The Netherlands.
`Shivakumar, Narayanan et al., "SCAM: A Copy Detection Mecha(cid:173)
`nism for Digital Documents," Dept. of Computer Science, Stanford
`University, Stanford, CA, pp. 1-13.
`
`

`

`Case 1:14-cv-02396-PGG-MHD Document 148-5 Filed 05/30/19 Page 5 of 30
`
`US 8,205,237 B2
`Page 4
`
`Shivakumar, Narayanan et al., "Building a Scalable and Accurate
`Copy Detection Mechanism," Dept. of Computer Science, Stanford
`University, Stanford, CA.
`Indexing and Content-Based
`Srihari, Rohiini K., "Automatic
`Retrieval of Captioned Images," State University ofNewYork, Buf(cid:173)
`falo, Theme Feature, pp. 49-56, Sep. 1995, IEEE 1995.
`Swain, Michael and Ballard, Dana H., "Color Indexing," Interna(cid:173)
`tiona!Journal of Computer Vision 7: 1, p. 11-32 (1991), 1991 Kluwer
`Academic Publishers. Manuf. in The Netherlands.
`Wactlar, Howard D. et al., "Intelligence Access to Digital Video:
`Informedia Project," Carnegie Mellon University, Digital Library
`
`Initiative: Carnegie Mellon University, Computer, pp. 46-52, IEEE
`1996.
`Yeo, Boon-Lock et al., "Rapid Scene Analysis on Compressed
`Video," IEEE Transactions on Circuits and Systems for Video Tech(cid:173)
`nology, vol. 5, No. 6, pp. 533-544, Dec. 1995, Dept. of Electrical
`Engineering, Princeton University, Princeton, NJ, IEEE Log No.
`9415901.
`Indyk, Piotr et al., "Finding pirated video sequences on the Internet,"
`Dept. of Computer Science, Stanford University, Palo Alto, CA,
`Paper No. 199.
`
`* cited by examiner
`
`

`

`Case 1:14-cv-02396-PGG-MHD Document 148-5 Filed 05/30/19 Page 6 of 30
`
`U.S. Patent
`
`Jun. 19, 2012
`
`Sheet 1 of 10
`
`US 8,205,237 B2
`
`WO RK @t1
`
`WO RK @t2
`
`122
`
`124
`
`138
`
`.. --,_____
`. , ,. ..........
`
`~ ........
`114 11.§
`........
`
`.v·-
`FEATU RE(S) (VECTOR) WORK ID
`
`..
`..
`
`,,•'
`
`•. ··--.....•.......
`········••;:; ~ ···~~
`.
`..
`WORK ID ASSOC IATED INFORMATION (e.g., ACT ION)
`
`170
`
`FIGURE 1
`
`

`

`Case 1:14-cv-02396-PGG-MHD Document 148-5 Filed 05/30/19 Page 7 of 30
`
`U.S. Patent
`
`Jun. 19, 2012
`
`Sheet 2 of 10
`
`US 8,205,237 B2
`
`SATELLITE, CABLE
`OR TERRESTRIAL
`TV BROADCAST
`
`I
`
`------.-~~-
`
`I ~===::~!..-.
`
`~cm
`
`USER COMPUTER,
`SET-TOP-BOX OR
`EQUIVALENT
`
`110a
`
`WID
`DATABASE
`
`130a
`
`WIDAT
`DATABASE
`
`170a
`
`_ ) ____ _
`
`210
`
`FIGURE 2
`
`240
`
`__J -
`
`MONITORING CENTER
`
`-,
`
`120a/138a1
`
`110a'
`
`WID
`DATABASE
`
`L
`
`13oa· I
`I
`I
`I _
`I
`
`WIDAT
`DATABASE
`
`

`

`Case 1:14-cv-02396-PGG-MHD Document 148-5 Filed 05/30/19 Page 8 of 30
`
`U.S. Patent
`
`Jun. 19, 2012
`
`Sheet 3 of 10
`
`US 8,205,237 B2
`
`320
`
`SATELLITE, CABLE
`OR TERRESTRIAL
`TV BROADCAST
`
`I
`I
`~ I
`
`----
`
`-,
`
`USER COMPUTER,
`SET-TOP-BOX OR
`EQUIVALENT
`
`---
`
`(cid:143)-
`
`I
`I
`I
`I
`I
`f 1
`310 I
`I
`L
`
`MONITORING CENTER
`
`I
`I
`I
`I 110b'
`I
`I
`WID
`DATABASE
`I
`L __
`
`160b
`
`130b
`
`WIDAT
`DATABASE
`
`->
`
`340
`
`110b
`
`.
`. .
`.
`f-- 1-
`
`:
`
`.
`//
`
`I
`.1 ... /
`
`...- 1
`
`I
`I
`I
`I
`_I
`
`FIGURE 3
`
`

`

`Case 1:14-cv-02396-PGG-MHD Document 148-5 Filed 05/30/19 Page 9 of 30
`
`U.S. Patent
`
`Jun.19,2012
`
`Sheet 4 of 10
`
`US 8,205,237 B2
`
`420
`
`SATE LLITE . CABLE
`OR TER RESTRIA L
`TV BROADCAST
`
`,__~_
`c:=.~~:::::,
`
`r -
`I
`
`-
`
`-
`
`-
`
`-
`
`D
`
`- 1
`-
`-
`USER COMPUTER , I
`I
`
`SET-TOP -BOX OR
`EQUIVALE NT
`
`:
`I
`I
`_
`I
`.... •······
`I
`......, ...... .::.__
`.... •·
`-y-
`
`~m
`----
`
`410
`
`•. •····
`.... ., ......
`,.....
`.. ..
`.... •·············
`·-
`
`----j
`
`-----
`
`/
`•'
`...............
`
`110c
`
`W1D
`DATABASE
`
`WIDAT
`DATABASE
`
`130c
`
`MONITORING CENTER
`
`FIGURE 4
`
`1~
`
`___
`
`11_~
`
`:
`I
`I
`I
`I
`
`---
`·
`.
`:
`,'
`..
`. .
`
`,1'
`:'
`
`C- 440 ~--,
`
`I
`I
`I
`I
`I
`I
`I
`I
`
`

`

`Case 1:14-cv-02396-PGG-MHD Document 148-5 Filed 05/30/19 Page 10 of 30
`
`U.S. Patent
`
`Jun. 19, 2012
`
`Sheet 5 of 10
`
`US 8,205,237 B2
`
`WORK(WITH
`EXTRA-WORK
`INFOMRAT ION)
`
`·······••·••·· ....... ------------·--··· .......................................................
`
`...................... ..
`! WORK
`REAL•TIME (CENTRA LIZED) MONITORING 1
`------
`:
`140d
`:
`
`WlD
`INFORMATION
`
`110d1
`
`\
`: ........................ ................. ~~ ............... "" .................... ;
`
`INFORMATION
`RELATED TO VVORK :
`•
`:
`!
`
`-----.----
`.....--------.--
`EXTRA•WORK WORK
`INFORMATION
`ID
`
`FEATURE(S) WO RK
`(VECTOR)
`ID
`594 1---------
`·
`·
`
`·
`•
`
`'
`'
`~ 1
`112.st:
`:
`1
`
`DATABASE
`GENERATION
`OPERA TION(S
`
`580
`
`520
`
`510
`
`...... --·· ,,J 514
`EXTRA. WORK
`INFORMATION (VECTOR)
`
`-
`
`i'-1 ..
`...... ...
`
`WORK ID
`
`550
`
`538
`
`•••···········••······•···
`
`
`..
`WORK ID
`
`..
`ASSOC IATED INFORMAT ION (e.g., ACTIO N)
`
`~ J
`FIGURE 5
`
`570
`
`

`

`Case 1:14-cv-02396-PGG-MHD Document 148-5 Filed 05/30/19 Page 11 of 30
`
`U.S. Patent
`
`Jun.19,2012
`
`Sheet 6 of 10
`
`US 8,205,237 B2
`
`SATELLITE, CABLE
`OR TERRESTRIAL
`
`2
`
`
`
`L..-6-0---1r..;_~1--~-B..,,ROADCAST
`
`I
`
`I
`
`(cid:143).
`
`=
`
`~cm
`
`USER COMPUTER,
`SET-TOP-BOX OR
`EQUIVALENT
`
`filQ
`
`,--
`
`------!---
`------/----
`
`/
`
`/
`... ···•·
`
`.•
`
`MONITORING AND QUERY RESOLUTION CENTER
`
`-~
`
`640
`
`--------
`FIGURE 6
`
`__,
`
`

`

`Case 1:14-cv-02396-PGG-MHD Document 148-5 Filed 05/30/19 Page 12 of 30
`
`U.S. Patent
`
`Jun.19,2012
`
`Sheet 7 of 10
`
`US 8,205,237 B2
`
`SATELLITE, CABLE
`OR TER RESTRIA L
`
`720
`
`QUERY
`
`-
`
`-
`
`--,
`
`,--
`
`-l---
`
`!
`
`VI/ID
`DATABASE
`
`PRIVATE
`
`.,I
`110f I..
`I OPTIONAL I
`I NETWORK I
`I
`I
`I
`I
`I
`I
`I
`
`··~.
`.. -
`
`........
`
`--~
`--~
`
`560b
`
`..... -. I
`... \I
`JI
`I
`I
`I
`I
`I
`I
`I
`I
`
`I MONITOR ING CENTER
`
`L-----~
`FIGURE 7
`
`_J
`
`QWID
`DATABASE
`
`74oa
`
`740b"'1
`
`I
`I_ QUERY RESOLUTION CENTER
`
`

`

`Case 1:14-cv-02396-PGG-MHD Document 148-5 Filed 05/30/19 Page 13 of 30
`
`U.S. Patent
`
`Jun. 19, 2012
`
`Sheet 8 of 10
`
`US 8,205,237 B2
`
`820
`
`SATELLITE, CABLE
`OR TERRESTRIAL
`TV BROADCAST
`
`860
`
`840a
`
`810
`
`MONITORING
`CENTER
`
`HOME VIEWERS
`
`INTERNET
`
`812
`
`840b
`
`DATABASE
`CENTER
`
`850
`
`E-COMMERCE
`OR OTHER WEB
`SITE
`
`FIGURE 8
`
`

`

`Case 1:14-cv-02396-PGG-MHD Document 148-5 Filed 05/30/19 Page 14 of 30
`
`TIME
`
`PLACE
`
`CHANNEL
`
`WORK IDENTIFIER
`
`~
`
`- TIME: 8:00
`PLACE: LA
`
`CHANNE L: CBS
`
`WORK ID: 15642
`
`- TIME: 8:01
`
`PLACE: NY
`
`CHANNEL: NBC
`
`WORK ID: 19023
`
`-
`
`~
`
`- TIME: 8:02
`
`~
`
`PLACE : CHICAGO
`
`CHANNEL: SCIFI
`
`WORK ID: 20564
`
`TIME: 8:01
`
`PLACE: NY
`
`CHANNEL: ABC
`
`WORK ID: 19625
`
`FIGURE 9
`
`~
`r,J.
`•
`--= ~
`~ = t""I'-
`
`t""I'(cid:173)
`
`~
`?
`
`~
`~\C
`N
`Q
`~
`N
`
`r.n
`:r
`
`~
`
`~ ...
`¢ ....
`
`\C
`
`~
`Q
`
`d
`rJJ.
`00
`N = !JI
`N
`vJ
`-.J
`t=:1
`N
`
`

`

`Case 1:14-cv-02396-PGG-MHD Document 148-5 Filed 05/30/19 Page 15 of 30
`
`U.S. Patent
`
`Jun . 19, 2012
`
`Sheet 10 of 10
`
`US 8,205,237 B2
`
`1010
`
`1020
`
`1030
`
`1040
`
`1050
`
`1060
`
`UNIQUE ID: 15642
`
`PRODUCT : COCA COLA
`
`CATEGORY: SODA
`
`MANUFACTURER : COCA COLA
`
`URL http://www.cocacola.com
`
`OTHER DATA
`
`1000
`
`FIGURE 10
`
`

`

`Case 1:14-cv-02396-PGG-MHD Document 148-5 Filed 05/30/19 Page 16 of 30
`
`US 8,205,237 B2
`
`1
`IDENTIFYING WORKS, USING A
`SUB-LINEAR TIME SEARCH, SUCH AS AN
`APPROXIMATE NEAREST NEIGHBOR
`SEARCH, FOR INITIATING A WORK-BASED
`ACTION, SUCH AS AN ACTION ON THE
`INTERNET
`
`§0. RELATED APPLICATIONS
`
`5
`
`2
`vision and computers. Convergence encompasses a very wide
`range of capabilities. Although a significant effort is being
`directed to video-on-demand applications, in which there is a
`unique video stream for each user of the service, as well as to
`transmitting video signals over the Internet, there is also
`interest in enhancing the television viewing experience. To
`this end, there have been a number of experiments with inter(cid:173)
`active television in which viewers can participate in a live
`broadcast. There are a variety of ways in which viewers can
`10 participate. For example, during game shows, users can
`answer the questions and their scores can be tabulated. In
`recent reality-based prograniming such as the ABC television
`game show, "Big Brother", viewers can vote on contestants
`who must leave the show, and be eliminated from the com-
`15 petition.
`§1.2.2 Embedding Work Identifying Code or Signals
`Within Works
`Known techniques of linking works delivered via tradi(cid:173)
`tional media channels to a more interactive system typically
`require some type of code, used to identify the work, to be
`inserted into the work before it is delivered via such tradi-
`tional media channels. Some examples of such inserted code
`include (i) signals inserted into the vertical blanking interval
`("VBI") lines of a ( e.g., NTSC) television signal, (ii) water(cid:173)
`marks embedded into images, (iii) bar codes imposed on
`images, and (iv) tones embedded into music.
`The common technical theme of these proposed imple(cid:173)
`mentations is the insertion of visible or invisible signals into
`the media that can be decoded by a computer. These signals
`30 can contain a variety of information. In its most direct form,
`the signal may directly encode the URL of the associated Web
`site. However, since the alphanumeric string has variable
`length and is not a particularly efficient coding, it is more
`common to encode a unique ID. The computer then accesses
`35 a database, which is usually proprietary, and matches the ID
`with the associated web address. This database can be con(cid:173)
`sidered a form of domain name server, similar to those
`already deployed for network addresses. However, in this
`case, the domain name server is proprietary and the addresses
`40 are unique ID's.
`There are two principal advantages to encoding a propri(cid:173)
`etary identifier into content. First, as previously mentioned, it
`is a more efficient use of the available bandwidth and second,
`by directing all traffic to a single Web site that contains the
`45 database, a company can maintain control over the technol(cid:173)
`ogy and gather useful statistics that may then be sold to
`advertisers and publishers.
`As an example of inserting signals into the vertical blank(cid:173)
`ing interval lines of a television signal, RespondTV of San
`50 Francisco, Calif. embeds identification information into the
`vertical blanking interval of the television signal. The VBI is
`part of the analog video broadcast that is not visible to tele(cid:173)
`vision viewers. For digital television, it may be possible to
`encode the information in, for example, the motion picture
`55 experts group ("MPEG") header. In the USA, the vertical
`blanking interval is currently used to transmit close-caption(cid:173)
`ing information as well as other information, while in the UK,
`the VBI is used to transmit teletext information. Although the
`close captioning information is guaranteed to be transmitted
`60 into the home in America, unfortunately, other information is
`not. This is because ownership of the vertical blanking inter(cid:173)
`val is disputed by content owners, broadcasters and local
`television operators.
`As an example of embedding watermarks into images,
`65 Digimarc of Tualatin, Oreg. embeds watermarks in print
`media. Invisible watermarks are newer than VBI insertion,
`and have the advantage ofbeing independent of the method of
`
`The present application is a continuation of U.S. patent
`application Ser. No. 11/445,928 (incorporated herein by ref(cid:173)
`erence), titled "USING FEATURES EXTRACTED FROM
`ANAUDIOAND/ORVIDEOWORKTOOBTAININFOR(cid:173)
`MATION ABOUT THE WORK," filed on Jun. 2, 2006, and
`listing Ingemar J. Cox as the inventor, which is a continua(cid:173)
`tion-in-part of U.S. patent application Ser. No. 09/950,972
`(incorporated herein by reference, issued as U.S. Pat. No.
`7,058,223 on Jun. 6, 2006), titled "IDENTIFYING WORKS
`FOR INITIATING A WORK-BASED ACTION, SUCH AS
`AN ACTION ON THE INTERNET," filed on Sep. 13, 2001, 20
`now U.S. Pat. No. 7,058,223 and listing Ingemar J. Cox as the
`inventor, which application claims benefit to the filing date of
`provisional patent application Ser. No. 60/232,618 (incorpo(cid:173)
`rated herein by reference), titled "Identifying and linking
`television, audio, print and other media to the Internet", filed 25
`on Sep. 14, 2000 and listing Ingemar J. Cox as the inventor.
`
`§1. BACKGROUND OF THE INVENTION
`
`§1.1 Field of the Invention
`The present invention concerns linking traditional media to
`new interactive media, such as that provided over the Internet
`for example. In particular, the present invention concerns
`identifying a work (e.g., content or an advertisement deliv(cid:173)
`ered via print media, or via a radio or television broadcast)
`without the need to modify the work.
`§ 1.2 Related Art
`§1.2.1 Opportunities Arising from Linking Works Deliv(cid:173)
`ered Via Some Traditional Media Channel or Conduit to a
`More Interactive System
`The rapid adoption of the Internet and associated World
`Wide Web has recently spurred interest in linking works,
`delivered via traditional media channels or conduits, to a
`more interactive system, such as the Internet for example.
`Basically, such linking can be used to ( a) promote commerce,
`such as e-commerce, and/or (b) enhance interest in the work
`itself by facilitating audience interaction or participation.
`Commerce opportunities include, for example, facilitating
`the placement of direct orders for products, providing product
`coupons, providing further information related to a product,
`product placement, etc.
`In the context of e-commerce, viewers could request dis(cid:173)
`count vouchers or coupons for viewed products that are
`redeemable at the point of purchase. E-commerce applica(cid:173)
`tions also extend beyond advertisements. It is now common
`for television shows to include product placements. For
`example, an actor might drink a Coke rather than a Pepsi
`brand of soda, actors and actresses might wear designer(cid:173)
`labeled clothing such as Calvin Klein, etc. Viewers may wish
`to purchase similar clothing but may not necessarily be able to
`identify the designer or the particular style directly from the
`show. However, with an interactive capability, viewers would
`be able to discover this and other information by going to an
`associated Web site. The link to this Web site can be auto(cid:173)
`matically enabled using the invention described herein.
`In the context of facilitating audience interaction or par(cid:173)
`ticipation, there is much interest in the convergence of tele-
`
`

`

`Case 1:14-cv-02396-PGG-MHD Document 148-5 Filed 05/30/19 Page 17 of 30
`
`US 8,205,237 B2
`
`5
`
`4
`directly into a sound card in a PC. This requires a physical
`connection between the television and the PC, which may be
`expensive or at least inconvenient, and a sound card may have
`to be purchased.
`§ 1.2.3 Unmet Needs
`In view of the foregoing disadvantages of inserting an
`identification code into a work, thereby altering the existing
`signal, there is a need for techniques of identifying a work
`without the need of inserting an identification code into a
`10 work. Such an identification code can then be used to invoke
`a work-related action, such as work-related commerce meth(cid:173)
`ods and/or to increase audience interest by facilitating audi(cid:173)
`ence interaction