`
`
`
`
`
`
` Exhibit 2
`
`
`
`
`
`Case 1:14-cv-02396-PGG-SN Document 234-4 Filed 11/11/20 Page 2 of 28
`
`USOO8.010988B2
`
`(12) United States Patent
`Cox
`
`(10) Patent No.:
`(45) Date of Patent:
`
`US 8,010,988 B2
`Aug. 30, 2011
`
`(54) USING FEATURES EXTRACTED FROMAN
`AUDIO AND/OR VIDEO WORK TO OBTAIN
`INFORMATION ABOUT THE WORK
`
`(76) Inventor: Ingemar J. Cox, London (GB)
`(*) Notice:
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 797 days.
`
`(21) Appl. No.: 11/445,928
`
`(22) Filed:
`
`Jun. 2, 2006
`
`(65)
`
`Prior Publication Data
`US 2007/0041667 A1
`Feb. 22, 2007
`O
`O
`Related U.S. Application Data
`(63) Continuation-in-part of application No. 09/950,972,
`filed on Sep. 13, 2001, now Pat. No. 7,058,223.
`(60) Provisional application No. 60/232,618, filed on Sep.
`14, 2000.
`
`(51) Int. Cl.
`(2011.01)
`HO)4N 7/173
`(52) U.S. Cl. ....................................................... 725/110
`(58) Field of Classification Search ........................ None
`See application file for complete search history.
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`11, 1975
`Moon et al.
`3,919,479
`Lert, Jr. et al.
`10, 1980
`4,230,990
`Kenyon et al.
`5, 1984
`4.450,531
`1, 1985
`Baranoff-Rossine
`4.495,526
`2, 1985
`Matthews
`4,499,601
`4, 1985
`Kohler et al.
`4,511.917
`Greenberg
`10, 1985
`4,547,804
`1, 1987
`Nakatani et al.
`4,634,966
`Greenberg
`1, 1987
`4,639,779
`
`6/1987 Okajima
`4,677,455 A
`g 3. it. et al.
`33.3% A
`aWS
`9, 1987 Kiewit et al.
`4,697,209 A
`4, 1988 Th
`tal.
`4,739,398 A
`4,776,017 A 10/1988 E. al
`4,805,020 A
`2/1989 Greenberg
`4,843,562 A
`6/1989 Kenyonet al.
`23: A
`$32, Sh
`5,283,819 A
`2/1994 Glicket al.
`5.437,050 A
`7/1995 Lamb et al.
`5,481.294 A
`1/1996 Thomas et al.
`5,581,658 A 12/1996 O'Hagan et al.
`5,594,934 A
`1/1997 Lu et al.
`(Continued)
`
`enyon
`
`4 kW
`
`OTHER PUBLICATIONS
`Peter N.Yianilos, Excluded Middle Vantage Point Forest for Nearest
`Neighbor Search, Aug. 1, 1999, pp. 1-12.*
`
`(Continued)
`Primary Examiner — Brian T Pendleton
`Assistant Examiner — Cai Chen
`(74) Attorney, Agent, or Firm — Amster, Rothstein &
`Ebenstein LLP
`
`ABSTRACT
`(57)
`s
`s
`s
`Information about an audio or video file played on a device is
`provided by (a) extracting features from the audio or video
`file, (b) communicating the features to a database, and (c)
`receiving the information about the audio or video file from
`the database. The information might include a song title, an
`album title, and/or a performer name. The information might
`include a title of a video work, a director of the video work,
`and/or names of performers in the video work. The informa
`tion might be rendered on an output of the device. The infor
`mation might be stored (e.g., persistently) locally on the
`device.
`
`52 Claims, 10 Drawing Sheets
`
`WORKg
`
`WoRK 82 i
`
`12
`
`FEATURE
`EXTRACTON
`oPERATIONis
`
`FATURET
`WORK
`AS&ING
`gPERATIONS
`
`124
`
`FEATUR
`(WECTOR) EXTRACTION
`oPERATIONs)
`
`AABASE
`GENERATION
`oPERATIONS
`
`FATUR
`(WECTOR) LookuP
`oPERATIONs)
`
`-
`
`Wid
`NFRMATM
`114 11B:
`
`WORK-ASSOCATED
`INFORMATION LOOKSF
`OPERATIONis)
`
`DATABASE
`GENERATION
`oPERATIon(s)
`
`
`
`ActN
`INITIATION
`oPERATIONS
`
`100
`
`
`
`US 8,010,988 B2
`Page 2
`
`U.S. PATENT DOCUMENTS
`36:33 A RE Ea
`W-1 4
`odlberg et al.
`5,701,452 A 12, 1997 Siefert
`5,701,542 A 12, 1997 Seifert
`5,724,605 A
`3, 1998 Wissner
`5,745,900 A
`4, 1998 Burrows
`5,798,785 A * 8/1998 Hendricks et al. .............. T25/46
`5,850,490 A 12/1998 Johnson
`5,918,223. A
`6/1999 Blum et al.
`5,953,415 A
`9/1999 Nielsen
`6,006.256 A 12/1999 Zdepski et al.
`6,011,758 A
`1/2000 Dockes et al.
`6,026,439 A
`2/2000 Chowdhury et al.
`8: A
`3.2.99
`st al.
`mith et al.
`6,061,056 A
`5, 2000 Menard et al.
`6,088,455 A
`7/2000 Logan et al.
`6,088,707 A
`7/2000 Bates et al.
`6,118,450 A
`9, 2000 Proehlet al.
`6,119,124. A
`9, 2000 Broder et al.
`6,169,986 B1
`1/2001 Bowman
`6,173,406 B1
`1/2001 Wang
`6,240,409 B1
`5, 2001 Aiken
`6,243,725 B1
`6/2001 Hempleman et al.
`6,247,133 B1
`6/2001 Palage et al.
`6.253,193 B1
`6, 2001 Ginter et al.
`6,263,348 B1
`7/2001 Kathrow et al.
`6,330,593 B1
`12/2001 Roberts et al.
`6,345,256 B1
`2/2002 Milsted et al.
`6,349,296 B1
`2/2002 Broder et al.
`6,360,215 B1
`3, 2002 Judd et al.
`6,363,377 B1
`3/2002 Kravets et al.
`6,374,225 B1
`4/2002 Hejna, Jr.
`6,381,601 B1
`4/2002 Fujiwara et al.
`6,385,596 B1
`5, 2002 Wiser et al.
`ck
`6,408,128 B1
`6/2002 Abecassis ....................... 386.68
`6,418,421 B1
`7/2002 Hurtado et al.
`6,449.226 B1
`9/2002 Kumagai
`6,452,874 B1
`9, 2002 Otsuka et al.
`6,477,704 B1
`1 1/2002 Cremia
`6,496,802 B1
`12/2002 Van Zoest et al.
`6,505,160 B1
`1/2003 Levy et al.
`6,550,001 B1
`4/2003 Corwin et al.
`6,550,011 B1
`4/2003 Sims, III
`6,577,746 B1
`6, 2003 Evans et al.
`6,591,245 B
`72003 Klug
`6,598,228 B2
`7/2003 Hejna, Jr.
`6,609, 105 B2
`8, 2003 Van Zoest et al.
`6,654,757 B1
`11/2003 Stern
`6,665,661 B1
`12/2003 Crow et al.
`6,675,174 B1
`1/2004 Bolle et al.
`6,834.308 B1* 12/2004 Ikezoye et al. ................ TO9,231
`6,873,982 B1
`3/2005 Bates et al.
`6,931,451 B1* 8/2005 Logan et al. .................. TO9,231
`6,941,275 B1
`9/2005 Swierczek
`6,973,461 B1
`12/2005 Fleming, III et al.
`6,978,419 B1
`12/2005 Kantrowitz
`6,990,453 B2
`1/2006 Wang et al.
`7,013,301 B2
`3, 2006 Holm et al.
`ck
`7,058,223 B2
`6/2006 Cox .............................. 382,190
`7,106.904 B2
`9, 2006 Shima
`7. 15 5.449 B2 12/2006 Pingel et al.
`7,158,929 B2
`1/2007 Wouters et al.
`7,168,083 B2
`1/2007 Kalker et al.
`7,302,574 B2 11/2007 Conwell et al.
`7.366,718 B1
`4/2008 Pugh et al.
`7,421,723 B2
`9/2008 Harkness et al.
`7,477,739 B2
`1/2009 HaitSma et al.
`7,523,312 B2
`4/2009 Kalker et al.
`7,587,728 B2
`9, 2009 Wheeler et al.
`7,647,604 B2
`1/2010 Ramaswamy
`7,650,616 B2
`1/2010 Lee
`7.757,248 B2
`7/2010 Harkness et al.
`2001/0001160 A1* 5, 2001 Shoffet al. ..................... 7.25/51
`2001/0003818 A1
`6/2001 Pingel et al.
`2002fOO23020 A1
`2/2002 Kenyon et al.
`2002fOO32698 A1
`3, 2002 Cox
`2002/0120925 A1* 8/2002 Logan ............................... 725/9
`
`2002/0156760 A1 10, 2002 Lawrence et al.
`2003/0106017 A1
`6/2003 Kanchirayappa et al.
`2003/0146940 A1* 8, 2003 Ellis et al. ..................... 345,811
`2004/0199387 A1* 10/2004 Wang et al. ................... TO4,243
`2005, 0160363 A1
`7/2005 Bhogal et al.
`2006/0101069 A1
`5, 2006 Bell et al.
`2006/0206462 A1
`9, 2006 Barber
`2007/0041667 A1
`2, 2007 COX
`2007/0O8351.0 A1
`4/2007 McArdle
`2007/01 18375 A1
`5/2007 Kenyonet al.
`2008/0091684 A1
`4, 2008 Ellis et all
`5688:03: A
`1658. Nal
`ea.
`OTHER PUBLICATIONS
`Baum, L., et al., “A Maximation Technique Occurring in the Statis
`-- %d
`tical Analysis of Probabilistic Functions of Markov Chains”. The
`Annals of Mathematical Statistics, vol. 41, No. 1, pp. 164-171 (1970).
`Dempster, A. P. et al., “Maximum Likelihood from Incomplete Data
`via the SEMS Algorithm”, Journal of the Royal Statistical Society,
`Series B (Methodological), vol.39, Issue 1. pp. 1-38 (1977).
`Reynolds, D., et al., “Robust Text-Independent Speaker Identifica
`tion Using Gaussian Mixture Speaker Models”, IEEE Transactions
`on Speech and Audio Processing, vol. 3, No. 1, pp. 72-83 (Jan. 1995).
`Ni
`lt. J.. et al... “The Grid File: An Adaptable. S
`tri
`eyergette a
`negll File:
`aptable, Symmetric
`Multikey File Structure.” ACM Transactions on Database Systems,
`vol. 9, No. 1, pp. 38-71 (Mar. 1984).
`Heintze, N., "Scalable Document Fingerprinting.” Proc. USENIX
`Workshop on Electronic Commerce (1996).
`Wold, E., et al., “Content-Based Classification, Search, and Retrieval
`of Audio.” IEEE Multimedia, vol. 3, Issue 3, pp. 27-63 (1996).
`Bhanu, B., et al., “Learning Feature Relevance and Similarity Metrics
`in Image Databases”. Proceedings of the IEEE Workshop on Con
`tent Based Access of Image and Video Libraries, pp. 14-19 (1998).
`D
`& &
`el Bimbo, A., et al. “Using Weighted Spatial Relationships in
`Retrieval by Visual C
`... I
`Descripti
`d Retrieval
`etrieval by Visual Contents”. Image Description and Retrieval, pp.
`161-192 (1998).
`& 8
`Indyk, P., and Motwani, R., "Approximate Nearest Neighbors:
`Towards Removing the Curse of Dimensionality.” Proceeding of the
`Thirtieth Annual ACM Symposium On Theory of Computing, pp.
`604-613 (1998).
`La Cascia, M., et al., “Combining Textual and Visual Cues for
`Content-based Image Retrieval on the World WideWeb. Proceed
`ings of the IEEE Workshop on Content-Based Access of Image and
`Video Libraries, pp. 24-29 (1998).
`& 8
`Yianilos, N. P. "Excluded Middle Vantage Point Forests for Nearest
`Neighb
`Search.
`DIMACS Impl
`Chall
`eignbor
`earch,
`mplementation
`allenge,
`ALENEX'99 (1999).
`Yoshitaka, A., et al., “A Survey on Content-Based Retrieval for Mul
`timedia Databases', IEEE Transactions on Knowledge and Data
`Engineering, vol. 11, No. 1, pp. 81-93 (Jan./Feb. 1999).
`Lawrence, S., et al., “Digital Libraries and Automonous Citation
`Indexing.” IEEE Computer, pp. 67-71 (Jun. 1999).
`Yianilos, N. P. “Locally Lifting the Curse of Dimensionality for
`Nearest Neighbor Search.” Symposium On Discrete Algorithms, Pro
`ceeding of the Eleventh Annual ACM-SIAM symposium On Discrete
`Algorithms, pp. 361-370 (2000).
`& 8
`Kimura, A., et al., “Very Quick Audio Searching: Introducing Global
`Pruning to the Time-Series Active Search.” IEEE Confon Acoustics,
`Speech and Signal Processing, (ICASSP 01), vol. 3, pp. 1429-1432
`(2001).
`Chavez, E., et al., “Searching in Metric Spaces'. (Sep. 2001) ACM
`Computing Surveys, vol. 33, No. 3, pp. 273-321 (Sep. 2001).
`Haitsma, J., et al., “Robust Audio Hashing for ContentIdentification,
`Int.” Workshop on Content Based Multimedia Indexing, Brescia, Italy
`(Sep. 19-21, 2001).
`Haitsma, J., and Walker, T., "A Highly Robust Audio Fingerprinting
`s
`System.” Journal of New Music Research, 1744-5027, vol. 32, Issue
`211-221 (2003
`2
`pp. 21-221 (2003),
`Schleimer, Saul, et al., Winnowing: Local Algorithms for Document
`Fingerprinting ACM SIGMOD” (Jun. 9-12, 2003).
`"Searching Near-Replicas of Images via Clustering Edward Chang,
`Chen Li, James Wang, Peter Mork, Gio Wiederhold Proc. SPIE
`Symposium of Voice, Video, and Data Communications, 1999.
`
`Case 1:14-cv-02396-PGG-SN Document 234-4 Filed 11/11/20 Page 3 of 28
`
`
`
`US 8,010,988 B2
`Page 3
`
`“Rime: A Replicated Image Detector for the World-Wide Web”
`Edward Y. Chang, James Ze Wang, Chen Li, and Gio Wiederhold,
`SPIE 1998.
`“Safeguarding and charging for information on the internet. H.
`Garcia-Molina, S. Ketchpel, and N. Shivakumar, Proceedings of
`ICDE, 1998.
`“Copy detection mechanisms for digital documents.” S. Brin and H.
`Garcia-Molina, Proceedings of ACM SIG-MOD, May 1995.
`“The X-tree: An index structure for high-dimensional data.” S.
`Berchtold, Proceedings of the 22nd VLDB, Aug. 1996.
`“The Sr-tree: An index structure for high-dimensional nearest neigh
`bor queries.” N. Katayama and S. Satoh, Proceedings of ACM
`SIGMOD, May 1997.
`“The k-d-b-tree: A search structure for large multidimensional
`dynamic indexes,” J. T. Robinson, Proceedings of ACM SIGMOD,
`Apr. 1981.
`“Query by image and video content: the QBIC system.” M. Flickner,
`H. Sawhney, W. Niblack, J. Ashley, Q. Huang, and et al. IEEE
`Computer 28(9), pp. 23:32, 1995.
`“Visual information retrieval.” A. Gupta and R. Jain, Communica
`tions of the ACM 40(5), pp. 69-79, 1997.
`"VisualSeek: A fully automated content-based image query system.”
`J. R. Smith and S.-F. Chang, ACM Multimedia Conference, 1996.
`“Similarity indexing: Algorithms and performance.” D. A. White and
`R. Jain, Proc. SPIE vol. 2670, San Diego, 1996.
`“The r-tree: an efficient and robust access method for points and
`rectangles.” N. Beckmann, H.-P. Kriegel, R. Schneider, and B.
`Seeger, Proceedings of ACM Sigmod, May 1990.
`
`“R-trees: a dynamic index structure for spatial searching.” A. Gutt
`man, Proceedings of ACM Sigmod, Jun. 1984.
`“Similarity indexing with the ss-tree.” D. A. White and R. Jain,
`Proceedings of the 12th ICDE, Feb. 1996.
`“The tv-tree: an index structure for high-dimensional data.” K.-L.
`Lin, H. V. Jagadish, and C. Faloutsos, VLDB Journal 3 (4), 1994.
`“M-tree: An efficient access method for similarity search in metric
`spaces.” P. Ciaccia, M. Patella, and P. Zezula, Proceedings of the 23rd
`VLDB, Aug. 1997.
`"Nearest neighbor queries.” N. Roussopoulos, S. Kelley, and F.
`Vincent, Proceedings of ACM Sigmod, May 1995.
`“An extensible hashing index for high-dimensional similarity
`search.” C. Li, E. Chang, and J. Z. Wang, Stanford Technical
`Report, Aug. 1998. NOT AVAILABLE).
`“Two algorithms for nearest-neighbor search in high dimensions' J.
`M. Kleinberg, Proc 29th STOC, 1997.
`"A Density-Based Algorithm for Discovering Clusters in Large Spa
`tial Databases with Noise' Martin Ester, Hans-Peter Kriegel, Jörg
`Sander, Xiaowei Xu Proceedings of 2nd International Conference on
`Knowledge Discovery and Data Mining (KDD-96), 1996.
`“Adaptive Color Image Embeddings for Database Navigation'Yossi
`Rubner, Carlo Tomasi and Leonidas J. Guibas, Proceedings of the
`1998 IEEE Asian Conference on Computer Vision.
`A Quantitative Analysis and Performance Study for Similarity
`Search Methods in High-Dimensional Spaces R. Weber, H-J Schek,
`S. Blott Proc., 24th VLDB Conf. 1998.
`* cited by examiner
`
`Case 1:14-cv-02396-PGG-SN Document 234-4 Filed 11/11/20 Page 4 of 28
`
`
`
`U.S. Patent
`
`Aug. 30, 2011
`
`Sheet 1 of 10
`
`US 8,010,988 B2
`
`WORK Gt 1
`
`WORK Gt2
`
`
`
`
`
`122
`
`FEATURE
`EXTRACTION
`OPERATION(S
`
`FEATURE TO
`WORK D
`TAGGING
`OPERATION(S
`
`
`
`124
`
`DATABASE
`GENERATION
`OPERATION(S)
`
`WD
`
`m
`
`FEATURE
`(VECTOR) EXTRACTION
`OPERATION(S)
`
`140
`
`120
`
`
`
`150
`
`
`
`
`
`
`
`FEATURE
`(VECTOR) LOOKUP
`OPERATION(S)
`
`
`
`160
`
`
`
`WORK-ASSOCIAED
`INFORMATION LOOKUP
`OPERATION(S)
`
`C C GENERATION
`OPERATION(S)
`
`DATABASE
`
`
`
`WD-ACTION
`INFORMATION
`
`
`
`
`
`Case 1:14-cv-02396-PGG-SN Document 234-4 Filed 11/11/20 Page 5 of 28
`
`
`
`ACTION
`NITATION
`OPERATION(S)
`
`170
`
`1 OO
`FIGURE 1
`
`
`
`U.S. Patent
`
`Aug. 30, 2011
`
`Sheet 2 of 10
`
`US 8,010,988 B2
`
`
`
`
`
`
`
`Case 1:14-cv-02396-PGG-SN Document 234-4 Filed 11/11/20 Page 6 of 28
`
`SATELLITE, CABLE |-
`OR TERRESTRIAL
`TV BROADCAST
`
`a
`
`-
`
`USER COMPUTER,
`SET-TOP-BOX OR
`EOUIVALENT
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`FEAURE
`(VECTOR)
`EXTRACTION
`OPERATION(S
`
`
`
`FEATURE
`(VECTOR) LOOKUP
`OPERATION(S)
`
`DATABASE
`GENERATION
`OPERATIONS
`
`160a
`
`
`
`
`
`
`
`WORK-
`ASSOCATED
`INFORMATION
`LOOKUP
`goPERATION(S
`
`
`
`
`
`ACTION
`INITIATION
`OPERATION(S)
`
`17Oa
`
`FIGURE 2
`
`MONTORING CENTER
`20a138a
`
`
`
`
`
`
`
`
`
`U.S. Patent
`
`Aug. 30, 2011
`
`Sheet 3 of 10
`
`US 8,010,988 B2
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`320
`
`SATELLITE, CABLE
`OR TERRESTRIAL
`TV BROADCAST
`
`a-
`
`l
`
`USER COMPUTER,
`SET-TOP-BOXOR
`EOUIVALENT
`
`14Ob
`
`11 Ob
`
`FEATURE
`(VECTOR)
`EXTRACTION
`OPERATION(S
`
`FEATURE
`(VECTOR) LOOKUP
`OPERATION(S)
`
`ACTION
`NITIATION
`OPERATION(S)
`
`WORK-ASSOCATED
`NFORMATION LOOKUP
`OPERATION(S)
`
`DATABASE
`GENERATION
`OPERATION(S)
`
`MoNITORING CENTER
`12Obf 138b.
`
`
`
`
`
`
`
`
`
`Case 1:14-cv-02396-PGG-SN Document 234-4 Filed 11/11/20 Page 7 of 28
`
`C d
`CD
`
`
`
`
`
`WID
`DATABASE
`
`WOA
`DATABASE
`
`FIGURE 3
`
`
`
`U.S. Patent
`
`Aug. 30, 2011
`
`Sheet 4 of 10
`
`US 8,010,988 B2
`
`420
`
`SATELLITE, CABLE
`OR TERRESTRAL
`W BROADCAST
`
`
`
`
`
`
`
`
`
`
`
`
`
`- - - - - - -
`
`USER COMPUTER,
`SE-TOP-BOX OR
`EQUIVALENT
`
`17Oc
`
`140c
`
`
`
`ACTION
`NITATION
`OPERATION(S)
`
`FEATURE
`(VECTOR)
`EXTRACTION
`OPERATION(S
`
`
`
`
`
`
`
`FEATURE
`(VECTOR) LOOKUP
`OPERATION(S)
`
`WORK-ASSOCATED
`NFORMATION LOOKUP
`OPERATION(S)
`
`
`
`
`
`Case 1:14-cv-02396-PGG-SN Document 234-4 Filed 11/11/20 Page 8 of 28
`
`11 Oc
`
`
`
`
`
`
`
`CC
`
`DATABASE
`GENERATION
`OPERATION(S)
`
`WID
`DATABASE
`
`WOAT
`DATABASE
`
`130C
`
`MONITORING CENTER
`
`FIGURE 4
`
`
`
`U.S. Patent
`
`Aug. 30, 2011
`
`Sheet 5 of 10
`
`US 8,010,988 B2
`
`WORK (WITH
`EXTRA-WORK
`INFOMRATION)
`
`
`
`
`
`REAL-TIME (CENTRALIZED) MONITORING:
`WORK
`FEATURE
`140d
`:
`(VECTOR) EXTRACTION
`110d
`OPERATION(S)
`
`
`
`
`
`WD
`INFORMATION
`
`
`
`EXTRA-WORK
`INFORMATION
`
`150C
`
`
`
`
`
`NFORMATION
`RELATED TO WORK : 5
`
`FEATURE
`(VECTOR) LOOKUP
`OPERATION(S)
`
`* 114g 116d,
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`EXTRA-WORK INFORMATION
`AGGREGATION OPERATION(S)
`
`
`
`
`
`
`
`DATABASE
`GENERATION
`OPERATION(S
`
`520
`
`OUERY
`
`
`
`
`
`LOOKUP
`OPERATION(S)
`
`550
`
`560
`
`
`
`WORK-ASSOCATED
`NFORMATION LOOKUP
`OPERATION(S)
`
`540
`
`
`
`
`
`C d 510
`OWD
`INFORMATION
`
`
`
`538
`
`DATABASE
`GENERATION
`OPERATION(S)
`
`NFORMATION
`
`Case 1:14-cv-02396-PGG-SN Document 234-4 Filed 11/11/20 Page 9 of 28
`
`WORK ID | ASSOCATED INFORMATION (e.g., ACTION
`
`-
`
`
`
`
`
`ACTION
`NITIATION
`OPERATION(S)
`
`34
`
`570
`
`535
`FIGURE 5
`
`
`
`U.S. Patent
`
`Aug. 30, 2011
`
`Sheet 6 of 10
`
`US 8,010,988 B2
`
`620
`
`SATELLITE, CABLE
`OR TERRESTRAL
`TV BROADCAST
`
`rema -a - m
`
`- - - -
`
`USER COMPUTER,
`SET-TOP-BOX OR
`EQUIVALENT
`
`610
`
`630
`
`
`
`540a
`
`EXTRA-WORK iNFORMATION
`AGGREGATION OPERATION(S)
`QUERY--
`
`-
`
`570a
`
`-
`
`
`
`ACTION
`INITIATION
`OPERATION(S)
`
`- - - - - - - - -
`- - - - - - - - - - - - - - - -
`
`WORK-ASSOCIATED
`INFORMATION LOOKUP
`OPERATION(S)
`
`
`
`
`
`WD
`DATABASE
`
`DATABASE
`
`520a1538a
`
`DATABASE
`GENERATION
`OPERATION(S
`
`WIDAT
`DATABASE
`
`
`
`
`
`
`
`
`
`
`
`
`
`Case 1:14-cv-02396-PGG-SN Document 234-4 Filed 11/11/20 Page 10 of 28
`
`L-
`
`MONTORING AND OUERY RESOLUTION CENTER
`
`
`
`U.S. Patent
`
`Aug. 30, 2011
`
`Sheet 7 of 10
`
`US 8,010,988 B2
`
`SATELLITE, CABLE
`OR TERRESTRAL
`TV BROADCAST
`
`720
`
`USER coMPUTER SET-Top-Box OREGUIVALENT
`
`730
`
`
`
`
`
`
`
`
`
`
`
`EXTRA-WORK INFORMATION
`AGGREGATION OPERATION(S)
`
`ACTION
`INTATION
`OPERATION(S)
`
`- - - - - - - - -
`
`
`
`11 Of
`
`WD
`DATABASE
`
`
`
`
`
`LOOKUP
`OPERATION(S)
`
`OPTIONAL
`PRVATE
`NETWORK
`
`550b
`
`
`
`WORK-ASSOCATED
`NFORMATION LOOKUP
`OPERATION(S)
`
`DATABASE
`GENERATION
`OPERATION(S
`
`Case 1:14-cv-02396-PGG-SN Document 234-4 Filed 11/11/20 Page 11 of 28
`
`MONITORING CENTER
`
`FIGURE 7
`
`V.
`740a
`
`OWD
`DATABASE
`
`DATABASE
`
`
`
`
`
`740b
`
`
`
`
`
`DATABASE
`GENERATION
`OPERATION(S
`
`
`
`U.S. Patent
`
`Aug. 30, 2011
`
`Sheet 8 of 10
`
`US 8,010,988 B2
`
`
`
`
`
`
`
`SATELLITE, CABLE
`OR TERRESTRAL
`TV BROADCAST
`
`SATELLTE TV BROADCAS
`
`
`
`MONTORING
`CENTER
`
`HOME VIEWERS
`
`Case 1:14-cv-02396-PGG-SN Document 234-4 Filed 11/11/20 Page 12 of 28
`
`
`
`
`
`DATABASE
`CENTER
`
`E-COMMERCE
`OR OTHER WEB
`STE
`
`FIGURE 8
`
`
`
`Case 1:14-cv-02396-PGG-SN Document 234-4 Filed 11/11/20 Page 13 of 28
`Case 1:14-cv-02396-PGG-SN Document 234-4 Filed 11/11/20 Page 13 of 28
`
`U.S. Patent
`
`Aug. 30, 2011
`
`Sheet 9 of 10
`
`US 8,010,988 B2
`
`
`
`_n:OmJMZZ<IO
`
`
`
`Omzszz<IO
`
`
`
`mmo”._m_ZZ<IO
`
`
`
`OO<0_IOHwodfia>2nm_o<._n_
`
`<4Umoajn.
`
`
`
`NounHm2;SumHm=>=._.commM=>=._.
`
`
`
`vwmomun:VEO>>mwomrH:me>>NvmmwUn:me>>
`
`
`
`
`
`mMKDOE
`
`mmomwHn:¥m0>>
`
`Um<H._m_ZZ<IO
`
`>2Hmay/WE
` Sum”m2:
`
`
`
`
`
`
`
`
`
`U.S. Patent
`
`Aug. 30, 2011
`
`Sheet 10 of 10
`
`US 8,010,988 B2
`
`
`
`
`
`
`
`
`
`
`
`UNOUE ID: 15642
`
`PRODUCT: COCA COLA
`
`CATEGORY: SODA
`
`MANUFACTURER COCA COLA
`
`URL http://www.cocacola.com
`
`OTHER DATA
`
`1010
`
`1020
`
`1030
`
`1040
`
`1050
`
`1060
`
`Case 1:14-cv-02396-PGG-SN Document 234-4 Filed 11/11/20 Page 14 of 28
`
`FIGURE 10
`
`
`
`US 8,010,988 B2
`
`1.
`USING FEATURES EXTRACTED FROMAN
`AUDIO AND/OR VIDEO WORK TO OBTAIN
`INFORMATION ABOUT THE WORK
`
`SO, RELATED APPLICATIONS
`The present application is a continuation-in-part of U.S.
`patent application Ser. No. 09/950,972 (incorporated herein
`by reference), titled “IDENTIFYINGWORKS FOR INITI
`ATING A WORK-BASED ACTION, SUCH AS AN
`10
`ACTION ON THE INTERNET filed on Sep. 13, 2001 now
`U.S. Pat. No. 7,058,223, and listing Ingemar J. Cox as the
`inventor, which application claims benefit to the filing date of
`provisional patent application Ser. No. 60/232,618 (incorpo
`rated herein by reference), titled “Identifying and linking
`television, audio, print and other media to the Internet’, filed
`on Sep. 14, 2000 and listing Ingemar J. Cox as the inventor.
`
`15
`
`2
`interest in enhancing the television viewing experience. To
`this end, there have been a number of experiments with inter
`active television in which viewers can participate in a live
`broadcast. There are a variety of ways in which viewers can
`participate. For example, during game shows, users can
`answer the questions and their scores can be tabulated. In
`recent reality-based programming Such as the ABC television
`game show, "Big Brother, viewers can Vote on contestants
`who must leave the show, and be eliminated from the com
`petition.
`
`S1.2.2 Embedding Work Identifying Code or Signals
`within Works
`
`Known techniques of linking works delivered via tradi
`tional media channels to a more interactive system typically
`require some type of code, used to identify the work, to be
`inserted into the work before it is delivered via such tradi
`tional media channels. Some examples of Such inserted code
`include (i) signals inserted into the vertical blanking interval
`(“VBI) lines of a (e.g., NTSC) television signal, (ii) water
`marks embedded into images, (iii) bar codes imposed on
`images, and (iv) tones embedded into music.
`The common technical theme of these proposed imple
`mentations is the insertion of visible or invisible signals into
`the media that can be decoded by a computer. These signals
`can contain a variety of information. In its most direct form,
`the signal may directly encode the URL of the associated Web
`site. However, since the alphanumeric string has variable
`length and is not a particularly efficient coding, it is more
`common to encode a unique ID. The computer then accesses
`a database, which is usually proprietary, and matches the ID
`with the associated web address. This database can be con
`sidered a form of domain name server, similar to those
`already deployed for network addresses. However, in this
`case, the domain name server is proprietary and the addresses
`are unique ID's.
`There are two principal advantages to encoding a propri
`etary identifier into content. First, as previously mentioned, it
`is a more efficient use of the available bandwidth and second,
`by directing all traffic to a single Web site that contains the
`database, a company can maintain control over the technol
`ogy and gather useful statistics that may then be sold to
`advertisers and publishers.
`As an example of inserting signals into the vertical blank
`ing interval lines of a television signal, RespondTV of San
`Francisco, Calif. embeds identification information into the
`vertical blanking interval of the television signal. The VBI is
`part of the analog video broadcast that is not visible to tele
`vision viewers. For digital television, it may be possible to
`encode the information in, for example, the motion picture
`experts group (“MPEG) header. In the USA, the vertical
`blanking interval is currently used to transmit close-caption
`ing information as well as other information, while in the UK,
`the VBI is used to transmitteletext information. Although the
`close captioning information is guaranteed to be transmitted
`into the home in America, unfortunately, other information is
`not. This is because ownership of the vertical blanking inter
`Val is disputed by content owners, broadcasters and local
`television operators.
`As an example of embedding watermarks into images,
`Digimarc of Tualatin, OR embeds watermarks in print media.
`Invisible watermarks are newer than VBI insertion, and have
`the advantage of being independent of the method of broad
`cast. Thus, once the information is embedded, it should
`remain readable whether the video is transmitted in NTSC,
`PAL or SECAM analog formats or newer digital formats. It
`
`S1. BACKGROUND OF THE INVENTION
`
`S1.1 Field of the Invention
`
`The present invention concerns linking traditional mediato
`new interactive media, such as that provided over the Internet
`for example. In particular, the present invention concerns
`identifying a work (e.g., content or an advertisement deliv
`ered via print media, or via a radio or television broadcast)
`without the need to modify the work.
`S 1.2 Related Art
`S1.2.1 Opportunities Arising from Linking Works
`Delivered Via Some Traditional Media Channel or
`Conduit to a More Interactive System
`
`25
`
`30
`
`35
`
`40
`
`45
`
`55
`
`60
`
`65
`
`The rapid adoption of the Internet and associated World
`Wide Web has recently spurred interest in linking works,
`delivered via traditional media channels or conduits, to a
`more interactive system, such as the Internet for example.
`Basically, such linking can be used to (a) promote commerce,
`Such as e-commerce, and/or (b) enhance interest in the work
`itself by facilitating audience interaction or participation.
`Commerce opportunities include, for example, facilitating
`the placement of direct orders for products, providing product
`coupons, providing further information related to a product,
`product placement, etc.
`In the context of e-commerce, viewers could request dis
`count Vouchers or coupons for viewed products that are
`redeemable at the point of purchase. E-commerce applica
`tions also extend beyond advertisements. It is now common
`50
`for television shows to include product placements. For
`example, an actor might drink a Coke rather than a Pepsi
`brand of Soda, actors and actresses might wear designer
`labeled clothing such as Calvin Klein, etc. Viewers may wish
`to purchase similar clothing but may not necessarily be able to
`identify the designer or the particular style directly from the
`show. However, with an interactive capability, viewers would
`be able to discover this and other information by going to an
`associated Web site. The link to this Web site can be auto
`matically enabled using the invention described herein.
`In the context of facilitating audience interaction or par
`ticipation, there is much interest in the convergence of tele
`vision and computers. Convergence encompasses a very wide
`range of capabilities. Although a significant effort is being
`directed to Video-on-demand applications, in which there is a
`unique video stream for each user of the service, as well as to
`transmitting video signals over the Internet, there is also
`
`Case 1:14-cv-02396-PGG-SN Document 234-4 Filed 11/11/20 Page 15 of 28
`
`
`
`US 8,010,988 B2
`
`Case 1:14-cv-02396-PGG-SN Document 234-4 Filed 11/11/20 Page 16 of 28
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`3
`should be more reliable than using the vertical blanking inter
`Val in television applications. Unfortunately, however, water
`marks still require modification of the broadcast signal which
`is problematic for a number of economic, logistical, legal
`(permission to alter the content is needed) and quality control
`(the content may be degraded by the addition of a watermark)
`CaSOS.
`As an example of imposing bar codes on images, print
`advertisers are currently testing a technology that allows an
`advertisement to be shown to a camera, Scanner or bar code
`reader that is connected to a personal computer (“PC”). The
`captured image is then analyzed to determine an associated
`Web site that the PC’s browser then accesses. For example,
`GoCode of Draper, UT embeds small two-dimensional bar
`codes for print advertisements. The latter signal is read by
`inexpensive barcode readers that can be connected to a PC.
`AirClic of Blue Bell, Pa.. provides a combination of barcode
`and wireless communication to enable wireless shopping
`through print media. A so-called “CueCat' reads bar codes
`printed in conjunction with advertisements and articles in
`Forbes magazine. Similar capabilities are being tested for
`television and audio media.
`Machine-readable bar codes are one example of a visible
`signal. The advantage of this technology is that it is very
`mature. However, the fact that the signal is visible is often
`considered a disadvantage since it may detract from the aes
`thetic of the work delivered via a traditional media channel or
`conduit.
`As an example of embedding tones into music, Digital
`Convergence of Dallas, Tex. proposes to embed identification
`codes into audible music tones broadcast with television sig
`nals.
`All the foregoing techniques of inserting code into a work
`can be categorized as active techniques in that they must alter
`the existing signal, whether it is music, print, television or
`other media, Such that an identification code is also present.
`There are several disadvantages that active systems share.
`First, there are aesthetic or fidelity issues associated with bar
`codes, audible tones and watermarks. More importantly, all
`media must be processed, before it is delivered to the end user,
`to contain these active signals. Even if a system is enthusias
`tically adopted, the logistics involved withinserting barcodes
`or watermarks into, say every printed advertisement, are for
`midable.
`Further, even if the rate of adoption is very rapid, it never
`theless remains true that during the early deployment of the
`system, most works will not be tagged. Thus, consumers that
`are early-adopters will find that most media is not identified.
`At best, this is frustrating. At worst, the naive user may
`conclude that the system is not reliable or does not work at all.
`This erroneous conclusion might have a very adverse effect
`on the adoption rate.
`Further, not only must there be modification to the produc
`tion process, but modifications must also be made to the
`equipment in a user's home. Again, using the example of
`watermarking of print media, a PC must be fitted with a
`camera and watermark detection Software must be installed.
`In the case of television, the detection of the identification
`signal is likely to occur at the set-top-box—this is the equip
`ment provided by the local cable television or satellite broad
`casting company. In many cases, this may require modifica
`tions to the hardware, which is likely to be prohibitively
`expensive. For example, the audible tone used by Digital
`Convergence to recognize television content, must be fed
`directly into a Sound card in a PC. This requires a physical
`
`10
`
`15
`
`25
`
`30
`
`35
`
`4
`connection between the television and the PC, which may be
`expensive or at least inconvenient, and a sound card may have
`to be purchased.
`
`S 1.2.3 Unmet Needs
`In view of the foregoing disadvantages of inserting an
`identification code into a work, thereby altering the existing
`signal, there is a need for techniques of identifying a work
`without the need of inserting an identification code into a
`work. Such an identification code can then be used to invoke
`a work-related action, such as work-related commerce meth
`ods and/or to increase audience interest by facilitating audi
`ence interaction and/or participation.
`
`S2. SUMMARY OF THE INVENTION
`
`This patent application describes an alternative solution
`that does not suffer from the problems outlined above. The
`Solution is based on director indirect recognition of the media
`itself. Direct or indirect recognition refers to the fact that a
`number of possible configurations are possible. Some of
`which directly recognize the work on the equipment in a
`user's home while other configurations perform this recogni
`tion indirectly by transmitting work-specific information to
`one or more remote sites. Neither technique requires the
`embedding of any form of active signal. Instead, when media
`in the form of music, print, television or multimedia is pre
`sented to a personal computer (PC), set-top-box or other
`device. Such devices directly or indirectly recognize the
`media and initiate an action. The set of possible actions is
`potentially infinite and includes, for example, retrieving fur
`ther information, interacting with a live broadcast, registering
`the user for a service or product, purchasing a product or
`service and/or receiving discount coupons or certificates that
`can be used towards a purchase.
`Some embodiments consistent with the present invention
`provide a computer-implemented method, apparatus, or com
`puter-executable programs for linking a media work to an
`action. Such embodiments might (a) extract features from the
`media work, (b) determine an identification of the media
`work based on the features extracted, and (c) determine an
`action based on the identification of the media work deter
`mined. In some embodiments consistent with the present
`invention, the media work is an audio signal. The audio signal
`might be obtained from abroadcast, oran audio f