Case 1:14-cv-02396-PGG-SN Document 239-4 Filed 11/12/20 Page 1 of 17
`
`
`
`Exhibit 22
`
`
`
`
`
`
`
`
`

`

`(12) United States Patent
`Wold et al.
`
`(10) Patent No.:
`(45) Date of Patent:
`
`US 7.562,012 B1
`Jul. 14, 2009
`
`US0075620 12B1
`
`(54) METHOD AND APPARATUS FOR CREATING
`A UNIQUE AUDIO SIGNATURE
`(75) Inventors: Erling H. Wold, El Cerrito, CA (US);
`Thomas L. Blum, San Francisco, CA
`(US); Douglas F. Keislar, Berkeley, CA
`(US); James A. Wheaton, Fairfax, CA
`(US)
`(73) Assignee: Audible Magic Corporation, Los Gatos,
`CA (US)
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 748 days.
`(21) Appl. No.: 09/706,227
`
`(*) Notice:
`
`Nov. 3, 2000
`
`(22) Filed:
`(51) Int. Cl.
`(2006.01)
`G06F 5/00
`(52) U.S. Cl. .................................... 704/200; 704/200.1
`(58) Field of Classification Search .............. 704/210.1,
`704/270, 500-504, 200, 200.1
`See application file for complete search history.
`References Cited
`
`(56)
`
`U.S. PATENT DOCUMENTS
`
`3,919,479 A 11/1975 Moon et al. ............... 179f 1 SB
`4,230,990 A 10, 1980 Lert, Jr. et al. ................ 455,67
`4,449.249 A
`5, 1984 Price
`4.450,531 A
`5/1984 Kenyon et al. .............. 364f604
`4,677,455 A
`6/1987 Okajima .........
`... 357/38
`4,677,466 A
`6, 1987 Lert, Jr. et al. ................ 358/84
`4,739,398 A
`4, 1988 Thomas et al. ................ 358/84
`4,843,562 A
`6/1989 Kenyon et al. ..
`... 364,487
`4,918,730 A
`4, 1990 Schulze .......
`... 381/43
`5,210,820 A
`5/1993 Kenyon ......................... 395/2
`5,247,688 A
`9/1993 Ishigami
`5,283,819 A
`2/1994 Glicket al. ................... 379/90
`5,327,521 A * 7/1994 Savic et al.
`... 704, 272
`5.437,050 A
`7/1995 Lamb et al. .................... 455.2
`5,442,645 A
`8/1995 Ugon et al.
`5,504,518 A
`4, 1996 Ellis et al.
`
`Case 1:14-cv-02396-PGG-SN Document 239-4 Filed 11/12/20 Page 2 of 17
`
`
`
`300
`
`302
`
`5,581,658 A 12/1996 O'Hagan et al. .............. 395.22
`5,588,119 A 12/1996 Vincent et al.
`5,612,974 A
`3, 1997 Astrachan
`5,613,004 A
`3/1997 Cooperman et al.
`
`(Continued)
`FOREIGN PATENT DOCUMENTS
`
`EP
`
`O349106 A1
`
`1, 1990
`
`(Continued)
`OTHER PUBLICATIONS
`
`L. Baum et al., A Maximization Technique Occurring in the Statisti
`cal Analysis of Probabilistic Functions of Markov Chaims, The
`Annals of Mathematical Statistics., vol. 41, No. 1 pp. 164-171, 1970.
`(Continued)
`Primary Examiner Michael NOpsasnick
`(74) Attorney, Agent, or Firm Blakely Sokoloff Taylor &
`Zafman LLP
`
`(57)
`
`ABSTRACT
`
`A method and apparatus for creating a signature of a sampled
`work in real-time is disclosed herein. Unique signatures of an
`unknown audio work are created by segmenting a file into
`segments having predetermined segment and hop sizes. The
`signature then may be compared against reference signatures.
`One aspect may be characterized in that the hop size of the
`sampled work signature is less than the hop size of reference
`signatures. A method for identifying an unknown audio work
`is also disclosed.
`
`8 Claims, 7 Drawing Sheets
`
`308
`
`Present Invention
`
`

`

`US 7.562,012 B1
`Page 2
`
`U.S. PATENT DOCUMENTS
`
`6, 1997 Stefket al.
`5,638,443 A
`5,692.213 A 1 1/1997 Goldberg et al.
`5,701,452. A 12/1997 Siefert
`5,710,916 A
`1/1998 Barbara et al.
`5,724,605 A
`3, 1998 Wissner
`5,732, 193 A
`3, 1998 Aberson
`5,850,388 A 12/1998 Anderson et al.
`5,918,223 A * 6/1999 Blum et al. .................... 707/1
`5,924,071 A
`7/1999 Morgan et al.
`5,930,369 A
`7, 1999 Cox et al.
`5,949,885 A
`9, 1999 Leighton
`5,959,659 A
`9, 1999 Dokic
`5,983,176 A 11/1999 Hoffert et al. ............... TO4,233
`6,006,183. A 12/1999 Lai et al.
`6,006,256 A 12/1999 Zdepski et al. .............. 709/217
`6,011,758 A
`1/2000 Dockes et al. ................. 369,30
`6,026,439 A
`2/2000 Chowdhury et al. ........ TO9,233
`6,044,402 A
`3/2000 Jacobson et al. ............ 709,225
`6,067,369 A
`5, 2000 Kamei
`6,088,455 A
`7/2000 Logan et al.
`6,092,040 A
`7/2000 Voran ......................... TO4,228
`6,096.961 A
`8, 2000 Bruti et al. ...
`... 84.609
`6,118,450 A
`9, 2000 Proehlet al. ...
`... 345,349
`6, 192,340 B1
`2/2001 Abecassis ................... 7041270
`6,195,693 B1
`2/2001 Berry et al.
`6,229,922 B1
`5/2001 Sasakawa et al.
`6.243,615 B1
`6/2001 Neway et al.
`6.243,725 B1
`6/2001 Hempleman et al.
`... 705/57
`6.253,193 B1
`6/2001 Ginter et al. ...
`6.253,337 B1
`6/2001 Maloney et al. ............... T14? 38
`6,279,010 B1
`8/2001 Anderson
`6.279,124 B1
`8/2001 Brouwer et al.
`6,285,596 B1
`9/2001 Miura et al.
`6,330,593 B1 12.2001 Roberts et al.
`6,345,256 B1
`2/2002 Milsted et al.
`6,374,260 B1
`42002 Hoffert et al.
`6,385,596 B1
`5/2002 Wiser et al.
`6,418,421 B1
`7/2002 Hurtado et al.
`6,422,061 B1
`72002 Sunshine et al.
`6,438,556 B1
`8/2002 Malik et al.
`6,449.226 B1
`9/2002 Kumagai
`6,452,874 B1
`9/2002 Otsuka et al.
`6,453,252 B1
`9/2002 Laroche
`6,460,050 B1
`10/2002 Pace et al.
`
`
`
`Case 1:14-cv-02396-PGG-SN Document 239-4 Filed 11/12/20 Page 3 of 17
`
`11/2002 Cremia
`6,477,704 B1
`1 1/2002 Cusson et al.
`6,487,641 B1
`12/2002 Chen et al.
`6,490,279 B1
`12/2002 van Zoest et al.
`6,496,802 B1
`2/2003 Ward
`6,526,411 B1
`4/2003 Foote ......................... TO4,500
`6,542,869 B1
`4/2003 Corwin et al.
`6,550,001 B1
`4/2003 Sims, III
`6,550,011 B1
`7/2003 Klug
`6,591.245 B1
`8/2003 Gopinath et al.
`6,609,093 B1
`8/2003 Story et al.
`6,609,105 B2
`6,628,737 B1 * 9/2003 Timus ........................ 375,368
`6,636,965 B1
`10/2003 Beyda et al.
`6,654,757 B1 1 1/2003 Stern
`6,732,180 B1
`5/2004 Hale et al.
`6,771,885 B1
`8/2004 Agnihotri et al.
`6,834.308 B1
`12/2004 Ikezoye et al.
`6,947,909 B1
`9/2005 Hoke, Jr.
`6,968,337 B2 * 1 1/2005 Wold .......................... 7O7/1OO
`7,043,536 B1
`5/2006 Philyaw et al.
`7,047.241 B1
`5/2006 Erickson
`7,058,223 B2
`6/2006 Cox
`7,181,398 B2
`2/2007 Thong et al.
`7,269,556 B2
`9/2007 Kiss et al.
`7,281,272 B1 10/2007 Rubinet al.
`7,349,552 B2
`3/2008 Levy et al.
`
`4/2008 Schmelzer et al.
`7,363,278 B2
`8/2001 DeMartin et al.
`2001/00 13061 A1
`2001/0027522 A1 10, 2001 Saito
`2001/00342.19 A1 10, 2001 Hewitt et al.
`2001 0037304 A1 11, 2001 Paiz
`2001/0056430 A1 12/2001 Yankowski
`2002.0049760 A1
`4, 2002 Scott et al.
`2002fOO64149 A1
`5, 2002 Elliott et al.
`2002fOO82999 A1
`6, 2002 Lee et al.
`2002fOO87885 A1
`7/2002 Peled et al.
`2002/O123990 A1
`9, 2002 Abe et al.
`2002/0133494 A1
`9, 2002 Goedken
`2002/0152262 A1 10, 2002 Arkin et al.
`2002fO156737 A1 10, 2002 Kahn et al.
`2002/0158737 A1 10/2002 Yokoyama
`2002fO186887 A1 12/2002 Rhoads
`2002fO198789 A1 12/2002 Waldman
`2003, OO14530 A1
`1/2003 Bodin et al.
`2003, OO18709 A1
`1/2003 Schrempp et al.
`2003, OO23852 A1
`1, 2003 Wold
`2003/0033321 A1
`2/2003 Schrempp et al.
`2003/003701.0 A1
`2/2003 Schmelzer
`2003, OO61352 A1
`3/2003 Bohrer et al.
`2003, OO61490 A1
`3/2003 Abajian
`2003/0095660 A1
`5/2003 Lee et al.
`2003. O135623 A1
`7/2003 Schrempp et al.
`2003,0191719 A1 10, 2003 Ginter et al.
`2003/0195852 A1 10/2003 Campbell et al.
`2004/0008864 A1
`1/2004 Watson et al.
`2004/0010495 A1
`1/2004 Kramer et al.
`2004.0053654 A1
`3/2004 Kokumai et al.
`2004/OO73513 A1
`4/2004 Stefik et al.
`2004/0089142 A1
`5/2004 Georges et al.
`2004O133797 A1
`7/2004 Arnold
`2004/0148191 A1
`7/2004 Hoke, Jr.
`2004/01631.06 A1
`8/2004 Schrempp et al.
`2004/0201784 A9 10/2004 Dagtas et al.
`2005, 0021783 A1
`1/2005 Ishii
`2005/0044189 A1
`2/2005 Ikezoye et al.
`2005. O154678 A1
`7, 2005 Schmelzer
`2005. O154680 A1
`7, 2005 Schmelzer
`2005. O154681 A1
`7, 2005 Schmelzer
`2005/0216433 A1
`9, 2005 Bland et al.
`2005/0267945 A1 12/2005 Cohen et al.
`2006/0034.177 A1
`2/2006 Schrempp
`2006/0062426 A1
`3/2006 Levy et al.
`
`2007/OO78769 A1
`2008/0008173 A1
`2008. O154730 A1
`
`4/2007 Way
`1/2008 Kanevsky et al.
`6/2008 Schmelzer et al.
`
`FOREIGN PATENT DOCUMENTS
`
`EP
`EP
`EP
`EP
`EP
`EP
`EP
`WO
`WO
`WO
`WO
`WO
`WO
`WO
`WO
`WO
`WO
`WO
`WO
`WO
`
`6, 1990
`0402210 A1
`5, 1992
`O517405 A2
`O6893.16 A2 12/1995
`O731446
`9, 1996
`O859503
`8, 1998
`O459046
`4f1999
`1354276 B1
`12/2007
`WO96,36163 A3 11, 1996
`WO98,20672 A3
`5, 1998
`WOOO,05650 A1
`2/2000
`WOOOf 39954 A1
`T 2000
`WOOOf 63800 A1 10, 2000
`WOO1?23981 A1
`4, 2001
`WOO1? 62004
`8, 2001
`WOO2,03203
`1, 2002
`WO O2, 15035 A2
`2?2002
`WOO2,373.16 A2
`5, 2002
`WO O2/O82271 A1 10, 2002
`WOO3,OO7235 A1
`1/2003
`WO 03/009149 A1
`1/2003
`
`

`

`US 7.562,012 B1
`Page 3
`
`5, 2003
`WO 03/036496 A1
`WO
`8, 2003
`WOO3,067459 A1
`WO
`WO 03/09 1990 A1 11, 2003
`WO
`WO WO 2004/044820 A1
`5, 2004
`WO WO 2004/070558 A2
`8, 2004
`WO WO 2006/O15168 A2
`2, 2006
`
`OTHER PUBLICATIONS
`A. P. Dempster et al. “Maximum Likelihood from Incomplete Data
`via the SEMS Algorithm”, Journal of the Royal Statistical Society,
`Series B (Methodological), vol.39, Issue 1. pp. 1-38, 1977.
`D. Reynolds et al., “Robust Text-Independent Speaker Identification
`Using Gaussian Mixture Speaker Models', IEEE Transactions on
`Speech and Audio Processing, vol. 3, No. 1, pp. 72-83, Jan. 1995.
`PCT International Search Report, PCT/US 01/50295, mailed May
`14, 2003, 5 pages.
`Beritelli. F., et al., “Multilayer Chaotic Encryption for Secure Com
`munications in packet switching Networks.” IEEE, vol. Aug. 2, 2000,
`pp. 1575-1582.
`Blum, T., Keislar D., Wheaton, J., and Wold, E., "Audio Databases
`with Content-Based Retrieval.” Prodeedings of the 1995 Interna
`tional Joint Conference on Artificial Intelligence (IJCAI) Workshop
`on Intelligent Multimedia Information Retrieval, 1995.
`Breslin, Pat, et al., Relatable Website, “Emusic uses Relatable's open
`Source audio recongnition solution, TRM, to signature its music
`catabblog for MusicBrainz database.” http://www.relatable.com/
`news/pressrelease/001017 release.html, Oct. 17, 2000.
`Cosi, P., De Poli, G., Prandoni, P., “Timbre Characterization with
`Mel-Cepstrum and Neural Nets.” Proceedings of the 1994 Interna
`tional Computer Music Conference, pp. 42-45, San Francisco, No
`date.
`Feiten, B. and Gunzel, S., “Automatic Indexing of a Sound Database
`Using Self-Organizing Neural Nets.” Computer Music Journal, 18:3,
`pp. 52-65, Fall 1994.
`Fischer, S. Leinhart, R., and Effelsberg, W., “Automatic Recognition
`of Film Genres.” Reihe Informatik, Jun. 1995, Universitat Man
`nheim, Praktische Informatik IV, L15, 16, D-68131 Mannheim.
`Foote, J., “Similarity Measure for Automatic Audio Classification.”
`Institute of Systems Science, National University of Singapore,
`1977. Singapore.
`Gonzalez, R. and Melih, K., "Content Based Retrieval of Audio.' The
`Institute for Telecommunication Research, University of Wol
`longong, Australia, No date.
`Haitsma, J., et al., “Robust Audio Hashing for Content Identifica
`tion'. CBMI 2001, Second International Workshop on Content Based
`Multimedia and Indexing, Brescia, Italy, Sep. 19-21, 2001.
`Kanth, K.V. etal. "Dimensionality Reduction or Similarity Searching
`in Databases.” Computer Vision and Image understanding, vol. 75.
`Nos. 1/2 Jul/Aug. 1999, pp. 59-72, Academic Press. Santa Barbara,
`CA, USA.
`Keislar, D., Blum, T. Wheaton, J., and Wold, E., "Audio Analysis for
`Content-Based Retrieval” Proceedings of the 1995 International
`Computer Music Conference.
`Ohtsuki, K., et al., “Topic extraction based on continuos speech
`recognition in broadcase-news speech.” Proceedings IEEE Work
`shop on Automated Speech Recognition and Understanding, 1997.
`pp. 527-534, N.Y., N.Y., USA.
`www.palisdesys.com/products/
`Packethound
`Tech
`Specs,
`packethount/tck specs/prod Phtechspecs.shtml, 2002.
`“How does PacketHound work?”. www.palisdesys.com/products/
`packethound how does it work prod Pghhow.shtml 2002.
`Pellom, B. et al., “Fast Likelihood Computation Techniques in Near
`est-Neighbor search for Continuous Speech Recognition.”. IEEE
`Signal Processing Letters, vol. 8, pp. 221-224 Aug. 2001.
`Ken C. Pohlmann, “Principles of Digital Audio'. SAMS/A Division
`of Prentice Hall Computer Publishing.
`Scheirer, E., Slaney, M.. “Construction and Evaluation of a Robust
`Multifeature Speech/Music Discriminator.” PP 1-4, Proceedings of
`ICASSP-97, Apr. 2-24, Munich, Germany.
`Scheirer, E.D., “Tempo and Beat Analysis of Acoustic Musical Sig
`nals.” Machine Listening Group, E15-401D MIT Media Laboratory,
`pp. 1-21, Aug. 8, 1997, Cambridge, MA.
`
`Schneier, Bruce Applied Cryptography, Protocols, Algorithms and
`Source Code in C, Chapter 2 Protocol Building Blocks, 1996, pp.
`30-31.
`Smith, Alan J., "Cache Memories.” Computer Surveys, Sep. 1982,
`University of California, Berkeley, California, vol. 14, No. 3, pp.
`1-61.
`Vertegaal, R. and Bonis, E., “ISEE: An Intuitive Sound Editing Envi
`ronment.” Computer Music Journal, 18:2, pp. 21-22, Summer 1994.
`Wang, Yao, et al., “Multimedia Content Analysis.” IEEE Signal Pro
`cessing Magazine, pp. 12-36, Nov. 2000, IEEE Service Center,
`Piscataway, N.J., USA.
`Wold, Erling, et al., “Content Based Classification, Search and
`Retrieval of Audio.” IEEE Multimedia, vol. 3, No. 3, pp. 27-36, 1996
`IEEE Service Center, Piscataway, N.J., USA.
`Zawodny, Jeremy, D., “A C Program to Compute CDDB discids on
`Linus and FreeBSD.” internet]http://jeremy.Zawodny.com/c/discid
`linux-1.3 targZ, 1 page, Apr. 14, 2001, retrieved July, 17, 2007.
`European Patent Application No. 0275234731, Supplementary Euro
`pean Search Report Dated May 8, 2006, 4 pages. (5219P004EP).
`European Patent Application No. 02756525.8, Supplementary Euro
`pean Search Report Dated June 28, 2006, 4 pages. (5219P005EP).
`European Patent Application No. 02782170, Supplementary Euro
`pean Search Report Dated Feb. 7, 2007, 4 pages. (5219P005XEP).
`European Patent Application No. 02725522.3, Supplementary Euro
`pean Search Report Dated May 12, 2006, 2 pages (5219P007EP).
`PCT Search Report PCT/US02/10615, International Search Report
`dated Aug. 7, 2002, 2 pages. (5219P007PCT).
`PCT Search Report PCT/US02/33186, International Search Report
`dated Dec. 16, 2002, pp. 1-4. (5219P005XPCT).
`PCT Search Report PCT/US04/02748, International Search Report
`and Written Opinion dated Aug. 20, 2007, 6 pages. (5219P008PCT).
`PCT Search Report PCT/US05/26887, International Search Report
`dated May 3, 2006, 2 pages. (5219P009PCT).
`PCT Search Report PCT/US08/09127. International Search Report
`dated Oct. 30, 2008, 8 pages. (5219PO11 PCT).
`Office Action for U.S. Appl. No. 09/51 1,632 () mailed Dec. 4, 2002.
`Office Action for U.S. Appl. No. 09/51 1,632 (P001) mailed May 13,
`2003.
`Office Action for U.S. Appl. No. 09/51 1,632 mailed Aug. 27, 2003.
`Office Action for U.S. Appl. No. 09/51 1,632 mailed Feb. 5, 2004.
`Notice of Allowance for U.S. Appl. No. 09/51 1,632 mailed Aug. 10,
`2004.
`Notice of Allowance for U.S. Appl. No. 10/955,841 mailed Sep. 26,
`2006.
`Notice of Allowance for U.S. Appl. No. 10/955,841 mailed Mar. 23.
`2007.
`Notice of Allowance for U.S. Appl. No. 10/955,841 mailed Sep. 11,
`2007.
`Notice of Allowance for U.S. Appl. No. 10/955,841 mailed Feb. 25,
`2008.
`Office Action for U.S. Appl. No. 08/897,662 mailed Aug. 13, 1998.
`Notice of Allowance for U.S. Appl. No. 08/897,662 mailed Jan. 29.
`1999.
`Office Action for U.S. Appl. No. 10/192,783 mailed Dec. 13, 2004.
`Notice of Allowance for U.S. Appl. No. 10/192,783 mailed Jun. 7.
`2005.
`Notice of Allowance for U.S. Appl. No. 1 1/239,543 (P004C) mailed
`Apr. 23, 2008.
`Office Action for U.S. Appl. No. 09/910,680 mailed Nov. 17, 2004.
`Office Action for U.S. Appl. No. 09/910,680 mailed May 16, 2005.
`Office Action for U.S. Appl. No. 09/910,680 mailed Sep. 29, 2005.
`Office Action for U.S. Appl. No. 09/910,680 mailed Jun. 23, 2006.
`Office Action for U.S. Appl. No. 09/910,680 mailed Aug. 8, 2006.
`Office Action for U.S. Appl. No. 09/910,680 mailed Jan. 25, 2007.
`Office Action for U.S. Appl. No. 09/910,680 mailed Dec. 5, 2007.
`Office Action for U.S. Appl. No. 09/999,763 mailed Apr. 6, 2005.
`Office Action for U.S. Appl. No. 09/999,763 mailed Oct. 6, 2005.
`Office Action for U.S. Appl. No. 09/999,763 mailed Aug. 7, 2006.
`Office Action for U.S. Appl. No. 09/999,763 mailed Oct. 6, 2006.
`Office Action for U.S. Appl. No. 09/999,763 mailed Mar. 7, 2007.
`Office Action for U.S. Appl. No. 09/999,763 mailed Aug. 20, 2007.
`Office Action for U.S. Appl. No. 09/999,763 mailed Jan. 7, 2008.
`Office Action for U.S. Appl. No. 09/999,763 mailed Jun. 27, 2008.
`
`Case 1:14-cv-02396-PGG-SN Document 239-4 Filed 11/12/20 Page 4 of 17
`
`

`

`US 7.562,012 B1
`Page 4
`
`Audible Magic Office Action for U.S.
`Apr. 18, 2007.
`Audible Magic Office Action for U.S.
`Oct. 11, 2007.
`Audible Magic Office Action for U.S.
`Jan. 14, 2008.
`Audible Magic Office Action for U.S.
`Jul. 9, 2008.
`Audible Magic Office Action for U.S.
`Jan. 7, 2009.
`Audible Magic Office Action for U.S.
`Nov. 17, 2008.
`Audible Magic Office Action for U.S.
`Dec. 29, 2008.
`Audible Magic Office Action for U.S.
`May 24, 2006.
`Audible Magic Office Action for U.S.
`Nov. 2, 2006.
`Audible Magic Office Action for U.S.
`Apr. 11, 2007.
`Audible Magic Office Action for U.S.
`Nov. 1, 2007.
`Audible Magic Office Action for U.S.
`May 9, 2008.
`Audible Magic Office Action for U.S.
`Jan. 6, 2009.
`Audible Magic Office Action for U.S.
`Jul. 17, 2008.
`Audible Magic Office Action for U.S.
`Jan. 9, 2009.
`* cited by examiner
`
`11/048.338 mailed
`
`11/048.338 mailed
`
`11/048.338 mailed
`
`11/048.338 mailed
`
`11/048.338 mailed
`
`12/035,599 mailed
`
`12/035,609 mailed
`
`. 10/356,318 mailed
`
`. 10/356,318 mailed
`
`. 10/356,318 mailed
`
`. 10/356,318 mailed
`
`. 10/356,318 mailed
`
`. 10/356,318 mailed
`
`11/191,493 mailed
`
`11/191,493 mailed
`
`. 10/072,238 mailed
`
`11/116,710 mailed
`
`11/116,710 mailed
`
`11/116,710 mailed
`
`11/116,710 mailed
`
`11/116,710 mailed
`
`11/116,710 mailed
`
`Office Action for U.S. Appl. No
`09/999,763 mailed Dec. 22, 2008.
`Office Action for U.S. Appl. No
`... 10/072,238 mailed May 3, 2005.
`Office Action for U.S. Appl. No
`. 10/072,238 mailed Oct. 25, 2005.
`Office Action for U.S. Appl. No
`... 10/072,238 mailed Apr. 25, 2006.
`Appl.
`Audible Magic Office Action for U.
`. 10/072,238 mailed
`S.
`No
`Sep. 19, 2007.
`Appl.
`Audible Magic Office Action for U.
`. 10/072,238 mailed
`S.
`No
`Apr. 7, 2008.
`Appl.
`Audible Magic Office Action for U.
`S.
`Oct. 1, 2008.
`Audible Magic Office Action for U.
`Dec. 13, 2004.
`Audible Magic Office Action for U.
`Apr. 8, 2005.
`Audible Magic Office Action for U.
`Oct. 7, 2005.
`Audible Magic Office Action for U.
`Apr. 20, 2006.
`Appl.
`Audible Magic Office Action for U.
`S.
`Jul. 31, 2006.
`Appl.
`Audible Magic Office Action for U.
`S.
`Jan. 16, 2007.
`Appl. No. 12/042,023
`Audible Magic Notice of Allowance for U.S.
`mailed Dec. 29, 2008.
`Audible Magic Office Action for U
`.S. Appl. No. 11/048.307 mailed
`Aug. 22, 2007.
`Audible Magic Office Action for U
`.S. Appl. No. 11/048.307 mailed
`May 16, 2008.
`Audible Magic Office Action for U
`.S. Appl. No. 11/048.308 mailed
`Feb. 25, 2008.
`
`. Appl.
`
`. Appl.
`
`. Appl.
`
`. Appl.
`
`Case 1:14-cv-02396-PGG-SN Document 239-4 Filed 11/12/20 Page 5 of 17
`
`

`

`U.S. Patent
`
`Jul. 14, 2009
`
`Sheet 1 of 7
`
`US 7.562,012 B1
`
`
`
`UOQUQAU I Qu@SQJA I "OIH
`
`-
`
`-
`
`- -
`
`90 I
`
`Case 1:14-cv-02396-PGG-SN Document 239-4 Filed 11/12/20 Page 6 of 17
`
`

`

`U.S. Patent
`
`Jul. 14, 2009
`
`Sheet 2 of 7
`
`US 7.562,012 B1
`
`9 IZ
`
`† IZ
`
`ZIZ
`
`90Z <No. od ZOZ
`
`
`
`Case 1:14-cv-02396-PGG-SN Document 239-4 Filed 11/12/20 Page 7 of 17
`
`

`

`U.S. Patent
`
`Jul. 14, 2009
`
`Sheet 3 of 7
`
`US 7.562,012 B1
`
`de
`C
`er
`
`
`
`Case 1:14-cv-02396-PGG-SN Document 239-4 Filed 11/12/20 Page 8 of 17
`
`S
`
`

`

`U.S. Patent
`
`Jul. 14, 2009
`
`Sheet 4 of 7
`
`US 7.562,012 B1
`
`
`
`on
`C
`r
`
`5
`
`Case 1:14-cv-02396-PGG-SN Document 239-4 Filed 11/12/20 Page 9 of 17
`
`

`

`Case 1:14-cv-02396-PGG-SN Document 239-4 Filed 11/12/20 Page 10 of 17
`69320wmy41.LeSaC
`1
`2Z1U1db
`
`
`
` 74Mmmmmdfi-mmwSagoNmmIvSfimfiHeSowSu><DOMEm:EoEwomm/0%D90Nma64:G1.mm
`
`e6wmPSOU
`
`7m
`
`
`
`,mn20:52:€3onmm,m6E
`
`U.S. Patent
`
`N:88E832:
`
`
`
`
`._UUUUUue/AlHimY/Al/Sm
`
`EE3:
`
`
`
`83.50on$30M
`
`
`
`Emmi063w
`
`/com
`
`
`

`

`U.S. Patent
`
`
`
`US 7.562,012 B1
`
`
`
`UOQUQAUI Qu@SQIA
`
`Case 1:14-cv-02396-PGG-SN Document 239-4 Filed 11/12/20 Page 11 of 17
`
`

`

`U.S. Patent
`
`US 7.562,012 B1
`
`
`
`L 'OIH
`
`Case 1:14-cv-02396-PGG-SN Document 239-4 Filed 11/12/20 Page 12 of 17
`
`

`

`1.
`METHOD AND APPARATUS FOR CREATING
`A UNIQUE AUDIO SIGNATURE
`
`2
`Hence, there exists a need to provide a system which over
`comes the disadvantages of the prior art.
`
`US 7,562,012 B1
`
`BACKGROUND OF THE INVENTION
`
`BRIEF DESCRIPTION OF THE INVENTION
`
`10
`
`The present invention relates to data communications. In
`particular, the present invention relates to creating a unique
`audio signature.
`A method for creating a signature of a sampled work in
`real-time is disclosed herein. One aspect of the present inven
`tion comprises: receiving a sampled work; segmenting the
`sampled work into a plurality of segments, the segments
`having predetermined segment and hop sizes; creating a sig
`nature of the sampled work based upon the plurality of seg
`ments; and storing the sampled work signature. Additional
`aspects include providing a plurality of reference signatures
`having a segment size and a hop size. An additional aspect
`may be characterized in that the hop size of the sampled work
`signature is less than the hop size of the reference signatures.
`An apparatus for creating a signature of a sampled work in
`real-time is also disclosed. In a preferred aspect, the apparatus
`comprises: means for receiving a sampled work; means for
`segmenting the sampled workinto a plurality of segments, the
`segments having predetermined segment and hop sizes;
`means for creating a signature of the sampled work based
`upon the plurality of segments; and storing the sampled work
`signature. Additional aspects include means for providing a
`plurality of reference signatures having a segment size and a
`hop size. An additional aspect may be characterized in that the
`hop size of the sampled work signature is less than the hop
`size of the reference signatures.
`A method for identifying an unknown audio work is also
`disclosed. In another aspect of the present invention, the
`method comprises: providing a plurality of reference signa
`tures each having a segment size and a hop size; receiving a
`sampled work; creating a signature of the sampled work, the
`sampled work signature having a segment size and a hop size;
`storing the sampled work signature; comparing the sampled
`work signature to the plurality of reference signatures to
`determine whether there is a match; and wherein the method
`is characterized in that the hop size of the sampled work
`signature is less than the hop size of the reference signatures.
`Further aspects of the present invention include creating a
`signature of the sampled work by calculating segment feature
`vectors for each segment of the sampled work. The segment
`feature vectors may include MFCCs calculated for each seg
`ment.
`
`BRIEF DESCRIPTION OF THE DRAWING
`FIGURES
`
`FIG. 1 is a flowchart of a method according to the present
`invention.
`FIG. 2 is a diagram of a system suitable for use with the
`present invention.
`FIG. 3 is a diagram of segmenting according to the present
`invention.
`FIG. 4 is a detailed diagram of segmenting according to the
`present invention showing hop size.
`FIG. 5 is a graphical flowchart showing the creating of a
`segment feature vector according to the present invention.
`FIG. 6 is a diagram of a signature according to the present
`invention.
`FIG. 7 is a functional diagram of a comparison process
`according to the present invention.
`
`1. Field of the Invention
`The present invention relates to data communications. In
`particular, the present invention relates to creating a unique
`audio signature.
`2. The Prior Art
`Background
`Digital audio technology has greatly changed the land
`scape of music and entertainment. Rapid increases in com
`15
`puting power coupled with decreases in cost have made it
`possible individuals to generate finished products having a
`quality once available only in a major studio. Once conse
`quence of modern technology is that legacy media storage
`standards, such as reel-to-reel tapes, are being rapidly
`replaced by digital storage media, such as the Digital Versatile
`Disk (DVD), and Digital Audio Tape (DAT). Additionally,
`with higher capacity hard drives standard on most personal
`computers, home users may now store digital files Such as
`audio or video tracks on their home computers.
`Furthermore, the Internet has generated much excitement,
`particularly among those who see the Internet as an opportu
`nity to develop new avenues for artistic expression and com
`30
`munication. The Internet has become a virtual gallery, where
`artists may post their works on a Web page. Once posted, the
`works may be viewed by anyone having access to the Internet.
`One application of the Internet that has received consider
`able attention is the ability to transmit recorded music over
`the Internet. Once music has been digitally encoded into a file,
`the file may be both downloaded by users for play, or broad
`cast (“streamed) over the Internet. When files are streamed,
`they may be listened to by Internet users in a manner much
`like traditional radio stations.
`Given the widespread use of digital media, digital audio
`files, or digital video files containing audio information, may
`need to be identified. The need for identification of digital
`45
`files may arise in a variety of situations. For example, an artist
`may wish to Verify royalty payments or generate their own
`Arbitron(R)-like ratings by identifying how often their works
`are being streamed or downloaded. Additionally, users may
`wish to identify a particular work. The prior art has made
`efforts to create methods for identifying digital audio works.
`However, systems of the prior art suffer from certain dis
`advantages. For example, prior art systems typically create a
`reference signature by examining the copyrighted work as a
`whole, and then creating a signature based upon the audio
`characteristics of the entire work. However, examining a
`work in total can result in a signature may not accurately
`represent the original work. Often, a work may have distinc
`tive passages which may not be reflected in a signature based
`upon the total work. Furthermore, often works are electroni
`cally processed prior to being streamed or downloaded, in a
`manner that may affect details of the work's audio character
`istics, which may result in prior art systems missing the
`identification of such works. Examples of such electronic
`processing include data compression and various sorts of
`audio signal processing Such as equalization.
`
`25
`
`35
`
`40
`
`50
`
`55
`
`60
`
`65
`
`Case 1:14-cv-02396-PGG-SN Document 239-4 Filed 11/12/20 Page 13 of 17
`
`

`

`3
`DETAILED DESCRIPTION OF PREFERRED
`EMBODIMENTS
`
`US 7,562,012 B1
`
`4
`seconds. It is contemplated that if one were searching for very
`short Sounds (e.g., Sound effects such as gunshots), segments
`as Small as 0.01 seconds may be used in the present invention.
`Since humans don’t resolve audio changes below about 0.018
`seconds, segment lengths less than 0.018 seconds may not be
`useful. On the other hand, segment lengths as high as 30-60
`seconds may be used in the present invention. The inventors
`have found that beyond 30-60 seconds may not be useful,
`since most details in the signal tend to average out.
`Generating Signatures
`Next, in act 104, each segment is analyzed to produce a
`signature, known herein as a segment feature vector. It is
`contemplated that a wide variety of methods known in the art
`may be used to analyze the segments and generate segment
`feature vectors. In an exemplary non-limiting embodiment of
`the present invention, the segment feature vectors may be
`created using the method described in U.S. Pat. No. 5,918.223
`to Blum, et al., which is incorporated by reference as though
`set forth fully herein.
`Storing the Signatures
`In act 106, the segment feature vectors are stored to create
`a representative signature of the sampled work.
`Each above-listed step will now be shown and described in
`detail.
`Referring now to FIG. 2, a diagram of a system suitable for
`use with the present invention is shown. FIG. 2 includes a
`client system 200. It is contemplated that client system 200
`may comprise a personal computer 202 including hardware
`and Software standard in the art to run an operating system
`such as Microsoft Windows.(R), MAC OSR), or other operating
`systems standard in the art. Client system 200 may further
`include a database 204 for storing and retrieving embodi
`ments of the present invention. It is contemplated that data
`base 204 may comprise hardware and software standard in the
`art and may be operatively coupled to PC 202. Database 204
`may also be used to store and retrieve the works and segments
`utilized by the present invention.
`Client system 200 may further include an audio/video
`(A/V) input device 208. A/V device 208 is operatively
`coupled to PC 202 and is configured to provide works to the
`present invention which may be stored in traditional audio or
`video formats. It is contemplated that A/V device 208 may
`comprise hardware and Software standard in the art config
`ured to receive and sample audio works (including video
`containing audio information), and provide the sampled
`works to the present invention as digital audio files. Typically,
`the A/V input device 208 would supply raw audio samples in
`a format such as 16-bit stereo PCM format. A/V input device
`208 provides an example of means for receiving a sampled
`work.
`It is contemplated that sampled works may be obtained
`over the Internet, also. Typically, streaming media over the
`Internet is provided by a provider, such as provider 218 of
`FIG. 2. Provider 218 includes a streaming application server
`220, configured to retrieve works from database 222 and
`stream the works in a formats standard in the art, Such as
`Real R, Windows MediaR), or QuickTime(R). The server then
`provides the streamed works to a web server 224, which then
`provides the streamed work to the Internet 214 through a
`gateway 216. Internet 214 may be any packet-based network
`standard in the art, such as IP. Frame Relay, or ATM.
`To reach the provider 218, the present invention may utilize
`a cable or DSL head end 212 standard in the art operatively,
`which is coupled to a cable modem or DSL modem 210 which
`is in turn coupled to the system's network 206. The network
`
`Case 1:14-cv-02396-PGG-SN Document 239-4 Filed 11/12/20 Page 14 of 17
`
`Persons of ordinary skill in the art will realize that the
`following description of the present invention is illustrative
`only and not in any way limiting. Other embodiments of the
`invention will readily Suggest themselves to Such skilled per
`sons having the benefit of this disclosure.
`It is contemplated that the present invention may be
`embodied in various computer and machine-readable data
`structures. Furthermore, it is contemplated that data struc
`tures embodying the present invention will be transmitted
`across computer and machine-readable media, and through
`communications systems by use of standard protocols such as
`those used to enable the Internet and other computer network
`ing standards.
`The invention further relates to machine-readable media on
`which are stored embodiments of the present invention. It is
`contemplated that any media Suitable for storing instructions
`related to the present invention is within the scope of the
`present invention. By way of example, such media may take
`the form of magnetic, optical, or semiconductor media.
`The present invention may be described through the use of
`flowcharts. Often, a single instance of an embodiment of the
`present invention will be shown. As is appreciated by those of
`25
`ordinary skill in the art, however, the protocols, processes,
`and procedures described herein may be repeated continu
`ously or as often as necessary to satisfy the needs described
`herein. Accordingly, the representation of the present inven
`tion through the use of flowcharts should not be used to limit
`the scope of the present invention.
`The present invention may also be described through the
`use of web pages in which embodiments of the present inven
`tion may be viewed and manipulated. It is contemplated that
`Such web pages may be programmed with web page creation
`35
`programs using languages standard in the art Such as HTML
`or XML. It is also contemplated that the web pages described
`herein may be viewed and manipulated with web browsers
`running on operating systems standard in the art. Such as the
`Microsoft Windows(R and