(12)
`
`United States Patent
`Levy et al.
`
`(10) Patent N0.:
`(45) Date of Patent:
`
`US 6,505,160 B1
`*J an. 7, 2003
`
`US006505160B1
`
`(54) CONNECTED AUDIO AND OTHER MEDIA
`OBJECTS
`
`1/1978 Barger, Jr. et al. ........ .. 179/2 R
`4,071,698 A
`4,230,990 A 10/1980 Lert, Jr. et al. ............. .. 455/67
`4,284,846 A
`8/1981 Marley ......... ..
`179/1 SE
`
`
`(75) Inventors: Kenneth L. Levy, Stevenson, WA _ ~
`
`gURsj’lgfo?rey B‘ Rhoads’ West Lmn’
`
`2
`,
`,
`4,450,531 A
`
`
`2111156 ~ ~ ~ ~ ~ ~ 0 man
`
`~ .
`5/1984 Kenyon et al. ........... .. 364/728
`
`(73) Assignee: Digimarc Corporation, Tualatin, OR
`(Us)
`
`(List continued on neXt page.)
`
`FOREIGN PATENT DOCUMENTS
`
`( * ) Notice:
`
`.
`
`.
`
`.
`
`.
`
`.
`
`Subject‘ to any disclaimer, the term of this
`patent 15 extended or adjusted under
`U-S-C- 154(b) by 0 days-
`
`This patent is subject to a terminal dis-
`Claimen
`
`(21) Appl. No.: 09/563,664
`
`(22) Filed:
`
`May 2, 2000
`
`Related U_S_ Application Data
`
`(63) Continuation-in-part of application No. 09/476,686, ?led on
`Dec. 30, 1999, which is a continuation-in-part of application
`No. 08/746,613, ?led on Nov. 12, 1996, now Pat. No.
`6,122,403, which is a continuation-in-part of application No.
`08/649,419, ?led on May 16, 1996, now Pat. No. 5,862,260,
`and a continuation-in-part of application No. PCT/US96/
`06618, ?led on May 7, 1996, and a continuation-in-part of
`application No. 08/508,083, ?led on Jul. 27, 1995, now Pat.
`No. 5,841,978.
`(60) lljgggisional application No. 60/134,782, ?led on May 19,
`
`(51) Int. Cl.7 .............................................. .. G10L 11/00
`(52) us. Cl. .................... .. 704/270; 455/306; 709/219;
`725/112
`(58) Field of Search ............................... .. 382/100 317'
`348/460 552. 455/306 201 305. 706/217’
`219. %07/5’01. 7250i 109’ 110 ’112 113’
`’ 133. 764/243 2,70. 3’81/5’6. 713/176’
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`3,919,479 A 11/1975 Moon et al.
`
`EP
`
`EP
`EP
`EP
`EP
`JP
`JP
`
`11 1985
`
`0161512
`
`12992
`0464328 A2
`2/1994
`0 581 317 A2
`4/1995
`0 649 074 A1
`3/1999
`0 901 282 A2
`9/1999
`11-265396 A
`11-272287 A 10/1999
`
`........ .. H04N 17 00
`
`H04IG/5j60
`G07D/7/00
`. 6036/21/02
`........ .. H04N/5/913
`......... .. G06F/17/30
`........... .. G10L/3/00
`
`(List continued on neXt page.)
`
`OTHER PUBLICATIONS
`_
`_
`_
`_
`Depovere, et al., “The VIVA Project: D1g1talWatermark1ng
`for Broadcast Monitoring,” 1999 IEEE, pp. 202—205.
`VoyatZis et al., “The Use of Watermarks in the Protection of
`Digital Multimedia Products,” PrOC. Of the‘ IEEE, vol. 87,
`NO_ 7, Jn1_ 1999, pp, 1197—1207_
`
`(List continued on next page.)
`
`Primary Examiner—Andrew W. Johns
`(74) Attorney Agent Or Firm_JOe1 R_ Meyer Digimarc
`C
`-
`)
`)
`’
`orporanon
`(57)
`
`ABSTRACT
`
`Media objects are transformed into active, connected objects
`via identi?ers embedded into them or their containers. In the
`eenteXt Of a user’s playback eXperienee, a deeeding process
`extracts the identi?er from a media object and possibly
`additional Context information and forwards it to a Server‘
`The server, in turn, maps the identi?er to an action, such as
`returning metadata, re-directing the request to one or more
`other servers, requesting information from another server to
`identify the media object, etc. The linking process applies to
`broadcast objects as well as objects transmitted over net
`works in streaming and compressed ?le formats.
`
`27 Claims, 2 Drawing Sheets
`
`i
`j
`j
`MeAtadaia/
`l
`<j:> We mapping ,j
`if’ H ciion
`I
`‘j
`lD-meiadaia, i‘
`I
`Association
`actiori(s) mapping ‘
`|
`
`: Registration Process |
`
`' Derivation of
`ID from
`Content
`
`Embedding Process
`File Header
`Embedder
`‘D
`(compressed
`iile disiribuior, NJ
`ripper, etc.)
`Stegaricgraphic
`Embedder (reai
`‘line or baich
`watermark
`el'iwdii'ig)
`
`Dynamic
`Derivation
`from Coriierii
`
`Mew/a6. mogiaiwrsi
`
`Google Ex. 1013
`
`

`

`US 6,505,160 B1
`Page 2
`
`US. PATENT DOCUMENTS
`
`4,495,526 A
`
`1/1985 Baranoff-Rossine ....... .. 360/15
`
`- 455/166
`2/1985 Matthews ------- --
`4,499,601 A
`358/84
`4/1985 Kéhler et al. ..
`4,511,917 A
`358/142
`4,547,804 A 10/1985 Greenberg
`324/778
`4,634,966 A
`1/1987 Nakatani et al.
`358/142
`4,639,779 A
`1/1987 Greenberg
`358/84
`4,677,466 A
`6/1987 Lert, Jr. et al.
`_ 455/166
`4,682,370 A
`7/1987 Matthews
`4,697,209 A
`9/1987 Kiewit et al. ............... .. 358/84
`4,739,398 A
`4/1988 Thomas et al. ............. .. 358/84
`4,776,017 A 10/1988 FujimOtO ____ u
`__ 381/43
`4,805,020 A * 2/1989 Greenberg
`358/147
`4,843,562 A
`6/1989 Kenyon et al.
`.. 364/487
`4,918,730 A
`4/1990 SchulZe . . . . .
`. . . .. 381/43
`4,931,871 A * 6/1990 Kramer __
`_358/142
`4,945,412 A
`7/1990 Kramer .................... .. 358/142
`4,967,273 A 10/1990 Greenberg ................ .. 358/142
`5,019,899 A
`5/1991 Boles et aL
`358/84
`
`. . . . . .. 455/2
`6/1991 Call . . . . . . . . . .
`5,023,929 A
`. 455/154.1
`7/1992 MankovitZ
`5,134,719 A
`358/142
`5,200,822 A * 4/1993 Bron?n et a1
`395/2
`5,210,820 A
`5/1993 Kenyon
`5,214,792 A
`5/1993 Alwadish ................... .. 455/45
`5,251,301 A * 10/1993 Cook ....................... .. 395/200
`5,276,629 A
`1/1994 Reynolds
`364/487
`5,303,393 A
`4/1994 Noreen et al.
`455/3.2
`5,319,453 A * 6/1994 CopriviZa et al.
`..... .. 348/6
`5,371,551 A 12/1994 Logan et al.
`348/571
`2132’???
`3832 5515285."; ............ ..'.'.'.'.?6Z‘r’5“58/Z
`5:437:050 A
`7/1995 Lamb et al. ................. .. 455/2
`5,491,838 A
`2/1996 Takahisa et aL
`455/66
`5,504,518 A
`4/1996 Ellis etal. ................... .. 348/2
`5,539,635 A
`7/1996 Larson, Jr. ............ .. 364/401 R
`5,564,073 A 10/1996 Talfahisa
`455/66
`5572246 A 11/1996 E111? et al' "
`348/2
`5,577,249 A 11/1996 Califano
`. 395/611
`5,577,266 A 11/1996 Takahisa eta.
`455/66
`5,579,537 A 11/1996 Takahisa _________ __
`__ 455/66
`5,581,658 A 12/1996 O’Hagan et al. ........... .. 395/22
`5,612,729 A
`3/1997 Ellis et al. ................... .. 348/2
`348/2
`5,621,454 A
`4/1997 Ellis etal- -
`5,640,193 A * 6/1997 Wellnef ~~
`----- ~~ 348/7
`5,661,787 A
`8/1997 Peeeek
`~~ 379/10101
`5,708,478 A
`1/1998 TognaZZini
`..... .. 348/552
`5,721,827 A
`2/1998 Logan etal. ..
`.. 395/200.47
`5,732,216 A
`3/1998 Logan et aL ________ __ 39500033
`5,761,606 A * 6/1998 WolZien ................... .. 455/6.2
`5,774,452 A
`6/1998 WoloseWicZ ..
`370/212
`5,774,664 A * 6/1998 Hidary et a1- --
`395/200-48
`5,806,031 A
`9/1998 Fineberg
`- 704/254
`2 *
`gigs? a1‘
`533/
`5,842,162 A 11/1998 Fineberguiiii ............... .. 704/233
`5:875:249 A
`2/1999 MintZeret al. ............. .. 380/54
`5,892,536 A
`4/1999 Logan etal. ..
`348/3
`5,893,095 A
`4/1999 Jain et al.
`707/6
`5,918,223 A
`6/1999 Blum et al. .
`707/1
`5974548 A 10/1999 Adams ~~~~ ~~
`- 713/200
`5,986,692 A 11/1999 Logan et al.
`348/13
`5,987,509 A * 11/1999 Portuesi
`.709/217
`5,991,737 A 11/1999 Chen _ _ _ _ _ _ _ _ _
`_ _ _ __ 705/26
`5,999,224 A * 12/1999 Maeda et al.
`348/563
`6,002,443 A 12/1999 Iggulden ................... .. 348/553
`6,021,432 A * 2/2000 Sizer, 11 et al. ........... .. 709/217
`6,058,430 A * 5/2000 Ka lan ........... ..
`709/245
`6,061,719 A * 50000 Begdineni et a1‘
`709018
`6,075,568 A * 6/2000 Matsuura .... ..
`348/478
`6,081,830 A * 6/2000 Schindler
`709/204
`6,088,455 A
`7/2000 Logan et al. ............. .. 380/200
`
`6,098,106 A * 8/2000 Philyaw et al. ........... .. 709/238
`6,169,541 B1 * 1/2001 Smith ........... ..
`345/327
`6,181,817 B1
`1/2001 Zabih et aL _
`u 382/170
`
`6,182,018 B1
`6,199,076 B1
`6,286,036 B1
`
`. . . .. 702/66
`1/2001 Tran et al. . . . . . .
`707/501
`3/2001 Logan et aL
`9/2001 Rhoads ..................... .. 709/217
`
`FOREIGN PATENT DOCUMENTS
`
`1/1994
`W094/00842 A1
`9/1997
`W097/33273 A1
`W097/41683 A1 11/1997
`W0 97/43736 A1 11/1997
`W098/20675 A1
`5/1998
`W098/36372 A1
`8/1998
`W098/43237 A1 10/1998
`W0 99/18723 A1
`4/1999
`W099/35809 A1
`7/1999
`W0 99/41900 A1
`8/1999
`W099/48099 A1
`9/1999
`
`W0 99/57623 A2 11/1999
`W0 00/58940 A2 10/2000
`W000/79709 A1 12/2000
`W001/20483 A2
`3/2001
`W001/20609 A2
`3/2001
`W001/35676 A1
`5/2001
`WO 02/11123 A2
`2/2002
`W0 02/27600 A2
`4/2002
`
`......... .. G11B/17/22
`GlOL/5/O6
`-
`-------- -- H04N/5/262
`.......... .. G06K/9/36
`H04N/5/44
`G06K/9/62
`GlOL/3/00
`.. H04N/5/913
`HO4M/3/5O
`-
`---------- -- H04N/1/32
`....... .. G11B/23/087
`
`G06F/l/OO
`GOlH/1/36
`H04B/17/00
`G0617/17/00
`6113/27/00
`-
`--------- -- H04N/11/20
`......... .. G01L/17/00
`--------- -- G06F/17/60
`
`W0
`W0
`W0
`W0
`W0
`W0
`W0
`W0
`W0
`W0
`W0
`
`W0
`W0
`W0
`W0
`W0
`W0
`W0
`W0
`
`OTHER PUBLICATIONS
`Zheef‘A WWW Seeviee te Emeee eee Peeve Digiel
`copynght Watermarks? P’OC- Of the Euro- Conf 0” Mum‘
`mediaApplications, Services and Techniques, May, 1996, 15
`pages
`Kim, W_G et al., “A Watermarking Scheme for Both Spatial
`and Frequency Domain to Extract te Seal Image Without the
`Original Image,” Proc. 5”1 Int. Symp. on Signal Processing
`and its Applications) Aug' 1999, pp‘ 293_296~
`.
`.
`“
`.
`.
`.
`Omshr et al., A Method of Watermakmg With Mult1reso
`llltion Analysis and Pseudo Noise Sequences,” Systems and
`Computers in Japan, vol- 29, NO- 5, May 1998,1111' 11—19
`MintZer et al., “If One Watermark is Good, Are More
`Better?,” Proc. IEEE Int. Conf on Acoustics, Speech and
`Signal Processing, Mar. 1999. pp. 2067—2069.
`U .
`L t
`1
`“H. hl R b t I
`Wt ak.
`‘1 e a"
`1g y 0.“ ,,mage nda erm mg mg
`Complementary Modulations, Proc. 2 Information Secu
`"Ty Works/‘0P; LNCS V01- 1729, NW 1999, PP- 136—153
`Ohbuchi et al., “AShape—Preserving Data Embedding Algo
`rithm for NURBS Curves and Surfaces,” Proc. Computer
`Graphics International (CGI ’99), Jun. 1999, pp. 180—187.
`Yi et al.‘, “Agent—Based Copyright Protection Architecture
`for Onhne Electron“: Pubhshmg? PVOC' SPIE V01‘ 3657:
`Security and Watermarking Of Multimedia Contents) Jan
`1999,PP- 484—493
`US. patent application Ser. No. 60/134,782, Rhoads, ?led
`May 19, 1999.
`US. patent application Ser. No. 60/178,028, Rhoads et al.,
`?led Jan' 26 2000
`.
`’
`. '.
`US. patent application Ser. No. 09/343,104, Rodriguez et
`211-, ?led Jun- 29, 1999
`US, patent application Ser. No. 09/476,686, Rhoads et al.,
`?led Dec_ 30, 1999_
`-
`-
`-
`US. patent application Ser. No. 09/504,239, Davis et al.,
`?led Feb 15> 2009
`US. patent application Ser. No. 09/531,076, Rhoads et al.,
`?led Mar. 18, 2000.
`
`Google Ex. 1013
`
`

`

`US 6,505,160 B1
`Page 3
`
`US. patent application Ser. No. 09/547,664, Rhoads et al.,
`?led Apr. 12, 2000.
`US. patent application Ser. No. 09/574,726, Rhoads et al.,
`?led May 18, 2000.
`US. patent application Ser. No. 09/858,189, Rhoads et al.,
`?led May 14, 2001.
`US. patent application Ser. No. 09/578,551, ConWell et al.,
`?led May 25, 2000.
`US. patent application Ser. No. 09/888,339, ConWell et al.,
`?led Jun. 21, 2001.
`
`US. patent application Ser. No
`. 09/420,945, Kenyon, ?led
`Oct. 19, 1999.
`US. patent application Ser. No
`. 60/218,824, Kenyon, ?led
`Jul. 18, 2000.
`US. patent application Ser. No
`. 60/155,064, Kenyon, ?led
`Sep. 21, 1999.
`US. patent application Ser. No
`. 2002/0023020, Kenyon et
`al., ?led Feb. 21, 2002.
`
`* cited by examiner
`
`Google Ex. 1013
`
`

`

`US. Patent
`
`Jan. 7, 2003
`
`Sheet 1 0f 2
`
`US 6,505,160 B1
`
`_§HMmlco._mo/Mfilmumu§II_E
`
`A/m
`
`mvEEmPa.Emuflmfi
`
`
`
`
`
`
`
`.ondEflmwe:.25“meoo.9
`
`AmEQUEEmuSwE
`
`.8533:50,9
`
`toqom
`
`co=$mcm0
`
`of9:
`
`IIIIIIII__95ng3:88
`
`go:9
`
`EmEoO
`
`\Smbmon
`
`8:04,
`
`co:m_oomm<
`
`.mmemEE-D_
`
`
`
`mcaamE.85_92:82-9
`
`cozmbwamm
`
`$me
`
`cozEcmo
`
`0:;m3/
`
`_IIII||_
`
`.25.meoo,9
`
`w._030w
`
`.25“3:50.9
`
`#588Q
`
`
`
`.55.:c983E:93
`
`93me
`
`$2206
`
`I
`
`
`
`AmvEmhmoa.mfiuflmé$25606.EmuSwE
`
`
`wwwooi5:32:65
`$8805
`mEEmmbm
`_____£82_$0.550_______
`
`$692
`
`$622
`
`IIIIIIIJ
`
`Q_
`
`BE).
`
`528
`
`
`
`Bug:m__n_
`
`LmnuwnEm
`
`88¢anan
`
`.LoSQEwE9:
`
`A90.503:
`
`o_cqm50cmm9m
`
`
`
`aw:LmUUmQEm
`
`:83LowE:
`
`meLQmB
`
`aEvoocm
`
`2:595
`
`cozm>cmo
`
`EmEooE0:
`
`Google Ex. 1013
`
`Google Ex. 1013
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`

`

`U.S. Patent
`
`Jan. 7, 2003
`
`Sheet 2 0f 2
`
`US 6,505,160 B1
`
`MP3 Distributor \
`/ Embed OID —>l Played on user’s PC —>
`
`Link exist?
`
`MP3 Ripper
`¢
`Escient’s CDDB
`
`NO
`Players
`' Real / V Yes
`' WMP
`*
`Web a e dis la ed
`- WinAmp
`O63: ""kftm .51“?
`of? ugser's gcy
`
`_ MusicMatCh
`
`(wu cons ram 5
`
`Fig. 2
`
`Google Ex. 1013
`
`

`

`US 6,505,160 B1
`
`1
`CONNECTED AUDIO AND OTHER MEDIA
`OBJECTS
`
`RELATED APPLICATION DATA
`This patent application is a continuation in part of US.
`patent application Ser. No. 09/476,686, ?led Dec. 30, 1999,
`Which claims priority to US. Provisional Application No.
`60/134,782, ?led May 19, 1999.
`This patent application is also a continuation in part of
`US. patent application Ser. No. 08/746,613 ?led Nov. 12,
`1996 (now US. Pat. No. 6,122,403) Which is a continuation
`in part of US. patent application Ser. No. 08/649,419, ?led
`May 16, 1996 (now US. Pat. No. 5,862,260), and a CIP of
`PCT Application PCT/US96/06618, ?led May 7, 1996, and
`a CIP of US. patent application Ser. No. 08/508,083, ?led
`Jul. 27, 1995 (now US. Pat. No. 5,841,978).
`The subject matter of the present application is related to
`that disclosed in US. Pat. No. 5,862,260, and in co-pending
`applications 08/746,613, ?led Nov. 12, 1996 (US. Pat. No.
`6,122,403); 09/343,104, ?led Jun. 29, 1999; 60/164,619,
`?led Nov. 10, 1999; 09/476,686, ?led Dec. 30, 1999; 09/503,
`881, ?led Feb. 14, 2000; 09/525,865, ?led Mar. 15, 2000;
`60/191,778 ?led Mar. 24; and 09/547,664, ?led Apr. 12,
`2000, Which are hereby incorporated by reference.
`
`TECHNICAL FIELD
`The invention relates to linking audio and other multime
`dia data objects With metadata and actions via a communi
`cation netWork, e.g., computer, broadcast, Wireless, etc.
`BACKGROUND AND SUMMARY
`Advances in computer and Wireless networking, multi
`media coding, and higher bandWidth communication links
`are creating many neW Ways to distribute and enjoy multi
`media content, such as music and movies. Coding formats
`for audio like MPEG 1 Layer 3 (MP3) have already caused
`signi?cant changes in music delivery to consumers. Despite
`the advances in technology, content distributors and broad
`casters still need to address hoW to effectively promote and
`sell content.
`This disclosure describes systems and processes for link
`ing audio and other multimedia data objects With metadata
`and actions via a communication netWork, e.g., computer,
`broadcast, Wireless, etc. Media objects are transformed into
`active, connected objects via identi?ers embedded into them
`or their containers. These identi?ers can be embedded by the
`oWner or distributor of the media object, or automatically
`created from the media object. In the context of a user’s
`playback experience, a decoding process extracts the iden
`ti?er from a media object and possibly additional context
`information and forWards it to a server. The server, in turn,
`maps the identi?er to an action, such as returning metadata,
`re-directing the request to one or more other servers,
`requesting information from another server to identify the
`media object, etc. If the identi?er has no de?ned action, the
`server can respond With an option for the user to buy the link
`and control the resulting action for all objects With the
`current identi?er. The linking process applies to broadcast
`objects as Well as objects transmitted over netWorks in
`streaming and compressed ?le formats.
`Further features Will become apparent With reference to
`the folloWing detailed description and accompanying draW
`1ngs.
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`55
`
`60
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`FIG. 1 is a diagram illustrating examples of media object
`linking processes and systems.
`
`65
`
`2
`FIG. 2 is a diagram illustrating media object linking
`applications.
`
`DETAILED DESCRIPTION
`Linking Audio and Other Media Objects via Identi?ers
`The folloWing sections describe systems and processes for
`linking audio and other media objects to metadata and
`actions via an identi?er. For the sake of illustration, the
`disclosure focuses on a speci?c media type, namely audio
`signals (e.g., music, sound tracks of audio visual Works,
`voice recordings, etc.). HoWever, these systems, their com
`ponents and processes apply to other types of media signals
`as Well, including video, still images, graphical models, etc.
`As described further beloW, an identi?er attached to an audio
`signal is used to connect that signal With metadata and/or
`programmatic or device actions. In the context of this
`document, the terms “media object” and “audio object” refer
`to an electronic form of a media signal and audio signal,
`respectively. The linking of media signals applies to objects
`that are transmitted over Wire netWorks (such as a computer
`netWork), Wireless netWorks (such as a Wireless telephone
`netWork), and broadcast (AM, FM, digital broadcast, etc.).
`There are a number of Ways to associate an identi?er With
`an audio object. One Way to associate the identi?er is to
`insert it in the form of a numeric or alphanumeric code (e. g.,
`binary or M-ary code) in the electronic ?le in Which the
`audio is stored. Another Way to associate the identi?er is to
`embed it as auxiliary data in the audio signal using stega
`nographic methods, such as digital Watermarking or other
`data hiding techniques. Yet another Way is to derive the
`identi?er from the audio signal, the table of contents, the ?le
`system structure, or its container (e.g., an electronic ?le or
`physical package for data like ?ash memory, Digital Versa
`tile Disk (DVD), minidisk, or compact disk (CD). The
`physical media may have identifying characteristics, such as
`a unique identi?er or encoded metadata, or other attributes
`from Which an identi?er can be derived (e.g., CD disk
`Wobble).
`When the identi?er is associated With metadata or actions,
`it transforms the media object into a “linked” object. The
`identi?er travels With the object through distribution, includ
`ing in some cases, through physical distribution in packaged
`media and through electronic distribution (broadcast or
`netWork communication). The identi?er may travel Within
`the same band as the audio object, such as a Watermark, or
`via a separate band, such as a ?le header or footer or separate
`broadcast band. A decoding device or programmatic process
`extracts the identi?er from the object and uses it to retrieve
`related data or actions (“metadata”). In the case of an audio
`object, like a song, the metadata typically includes the title,
`artist, lyrics, copyright oWner, sound recording oWner, infor
`mation about buying or sampling opportunities and URLs to
`this type of data as Well as Web sites and other programs and
`devices. Linked actions include device or programmatic
`processes for electronically establishing a license, transfer
`ring content (either streaming or doWnload), sending an
`email, recording marketing data about a transaction, etc. The
`identi?er alloWs a fan of a particular type of music or artist
`to get more information about the music and to buy more
`music. From the perspective of the artists and record labels,
`the identi?er provides an additional opportunity to promote
`their music and sell content, concert tickets, etc.
`In addition, in some implementations Where identi?er
`linking transactions are monitored, it enables the vendors of
`music to gather data about electronic transactions triggered
`by the link. For example, users of information may choose
`to provide information about themselves When they register
`their decoding device or softWare With the system. Auser ID
`
`Google Ex. 1013
`
`

`

`US 6,505,160 B1
`
`3
`or other context information may then be recorded When the
`identi?er is extracted and used to trigger a transaction. Many
`entities involved in the distribution of media signals can
`bene?t from the linking capability. Artists can link their
`music to information about themselves and provide elec
`tronic buying opportunities for music, concert tickets,
`clothing, etc. Rights holding organiZations can use the link
`to inform users about itself and licensing opportunities. In
`some cases, the link may also be used to monitor playing and
`distribution of copies of the music. Record labels can link
`their music to information about the artist, the label, elec
`tronic buying opportunities, etc. Electronic retailers can
`increase sales by linking users to opportunities to sample
`and buy additional music (via doWnload or streaming deliv
`ery over a Wire or Wireless netWork). Conventional brick and
`mortar retailers can use linking to provide information about
`the music and to provide buying opportunities. Radio sta
`tions and other broadcasters can use the linking capability to
`bring users to their Web sites, creating advertising revenue,
`to provide electronic buying opportunities for music, concert
`tickets, clothing items, etc. These and other forms of linked
`metadata and actions may be implemented in various com
`binations in different application scenarios.
`Depending on the application, the identi?er may identify
`the media object in Which it is embedded, or entities, things
`or actions other than that particular media object. One type
`of identi?er is an object ID that identi?es an audio object.
`This identi?er may be a number associated With the object,
`such as its International Standard Recording Code (ISRC).
`Another type of identi?er is distributor ID that identi?es the
`distributor of the audio object. Another type of identi?er is
`a broadcaster ID that identi?ers the broadcaster of the audio
`object. Of course, more than one identi?er may be encoded
`into an audio object or its container. In the event that an
`object ID is not encoded With an audio object, but instead,
`a distributor or broadcaster identi?er is encoded With the
`object, other context information, such as the time of play
`back or distribution, location of distribution, etc. may be
`used to identify the audio object as part of the linking
`process. An example is a radio station that marks its broad
`casts With a station ID and maintains a playlist database With
`the air times of each audio object. At decoding time, the
`station ID is extracted and used along With context infor
`mation such as the air time of the audio object to look up the
`audio object or its corresponding metadata and actions. This
`approach enables the linking system to provide audio object
`speci?c metadata or actions even Without requiring a unique
`object identi?er in every audio object.
`System Implementation
`FIG. 1 is a diagram of a system con?guration of linked
`media objects. In this con?guration, an identi?er links audio
`objects to metadata via an electronic netWork, such as the
`Internet, a Wireless netWork, or a broadcast netWork. As
`depicted in FIG. 1, an embedding process may be used to
`encode an identi?er in an audio object or its container. In
`some cases, an embedding process encodes the identi?er in
`the audio ?le (e.g., a tag in a ?le header or footer), in the
`audio signal (a digital Watermark), or in the physical pack
`aging. The identi?er may also be derived as a function of the
`audio signal or other information in the ?le or physical
`packaging (e.g., track information on a CD). In the case of
`dynamically derived identi?ers, an embedding process is not
`necessary because the identi?er can be derived from the
`content at decoding time.
`In some application scenarios, the embedding process
`interacts With a registration process to get an identi?er. The
`embedding process provides information about the object
`
`10
`
`15
`
`25
`
`35
`
`45
`
`55
`
`65
`
`4
`(e.g., a title and artist name, an ISRC, name of distributor,
`etc.). In response, the registration process provides an iden
`ti?er and stores a database record of the association betWeen
`identi?er and the object or other information used in decod
`ing to identify the object, such as its distributor or broad
`caster. The registration process may be used to assign an
`identi?er to an audio object and to distributors or broadcast
`ers of audio objects. The embedding and registration pro
`cesses may occur before the audio object is distributed to
`consumers, or sometime thereafter, such as When a user
`transfers (e.g., “rips”) an a media object from one format to
`another (e.g., a packaged format to an electronic ?le format
`such as a compressed ?le format).
`Once registered, an interactive or automated mapping
`process associates the identi?er With data or actions. The
`registration process creates a database of identi?ers and
`associates the identi?ers With corresponding media objects,
`distributors, broadcasters, etc. The mapping process associ
`ates the identi?ers With corresponding metadata or actions.
`Once associated With an audio object and metadata, the
`identi?er transforms the audio object into a linked object.
`The identi?er remains With the object through distribution,
`although some embedding processes are more robust than
`others to intentional or unintentional distortion/removal of
`the identi?er. There a variety of different distribution sce
`narios. Some examples depicted in FIG. 1 include transfer
`ring an audio object over a computer netWork, streaming the
`object over a computer netWork, or broadcasting it (e.g.,
`AM/FM broadcasting, digital broadcasting, broadcasting
`over Wireless carriers, etc.). Whatever the distribution
`process, a user ultimately receives the linked object in a
`player, tuner, or capture device.
`To activate the linked object, a decoding process extracts
`the identi?er and uses it to access associated data or actions.
`The decoding process may be implemented as a separate
`program or device, or integrated into a player, tuner, or some
`other capture device, such as a listening devices that con
`verts ambient audio Waves to an electronic signal and then
`extracts the identi?er from the signal.
`In the con?guration shoWn in FIG. 1, the decoding
`process forWards the extracted identi?er to a communication
`application, Which in turn, forWards it in a message to a
`server. The decoding process or the communication appli
`cation may add additional context information to the mes
`sage sent to the to a server. The context information may
`relate to the user, the user’s device, the attributes of the
`session (time of playback, format of playback, type of
`distribution (e.g., broadcast or transmitted audio ?le), etc.)
`Based on identi?er and optional context information, the
`server determines an associated action to perform, such as
`re-directing an identi?er or context data to another server,
`returning metadata (including programs, content, etc.),
`doWnloading content, logging a transaction record. To ?nd
`the associated action or actions, the server maps the iden
`ti?er to actions based on the information established in the
`mapping process. The server may: 1) look up the data and
`actions in a local database stored in its memory subsystem;
`2) route the identi?er to one or more other servers via the
`netWork, Which in turn look up related actions and data
`associated With the identi?er; or 3) perform some combina
`tion of actions 1 and 2.
`In the ?rst case, server 1 returns data or actions associated
`With the identi?er. The server may look up related data based
`on the identi?er alone, or based on the identi?er and other
`context information. Context information may be informa
`tion provided by the user, by the user’s computer or device,
`or by some other process or device. In the second case, the
`
`Google Ex. 1013
`
`

`

`US 6,505,160 B1
`
`5
`server looks up one or more addresses associated With the
`identi?er and forWards the identi?er and/or possibly other
`context data to secondary servers at these addresses via
`conventional networking protocols. Again, this context data
`may include data from the user, the user’s computer, some
`other device or database. For example, server 1 might query
`a remote database for instructions about hoW to process an
`identi?er. These instruction may specify data to return to the
`communication application or to forWard to another server,
`Which in turn, looks up associated data and returns it to the
`communication application. A server may return data that an
`audio player displays to the user or uses to control rendering
`of the content. For example, the server can tell the player
`that the object contains inappropriate content for children.
`The player or user can make decisions about Whether or hoW
`to play the material based on this information.
`Both the server and the player can adopt a set of rules. The
`server rules may be used to control What the server returns
`in response to an identi?er and context data. The player rules
`may be used to control What the player displays to the user
`or hoW it renders the content based on data returned from a
`server.
`Either the ?rst server, or a server one or more levels of
`indirection from the identi?er may return data and program
`matic actions to a player via the communication application.
`Each server in these levels of indirection receives a database
`key, such as an identi?er or context information, from the
`previous server, and uses it to look up corresponding actions.
`These actions may include returning data or programs to the
`communication application or to previous servers in the
`routing path of the message from the communication appli
`cation. Also, the servers may route requests for information
`or actions to other servers. The server or servers may return
`data or perform actions in response to the identi?er (or other
`context data) that do not directly impact the decoding
`process, or the device in Which it operates.
`The system depicted in FIG. 1 alloWs several different
`interested parties to establish services linked via the identi
`?er. For example, server 1 can be con?gured to provide
`generic promotional and/or licensing information associated
`With an identi?er. If the content oWner, distributor, retailer,
`artist or other related party Wishes to provide information or
`services for a connected object, then server 1 may also route
`the identi?er for that object, and possibly context
`information, the address of the communication application,
`and instructions, to servers maintained by these entities.
`These servers, in turn, provide promotional, sales, or licens
`ing information, and electronic buying or licensing oppor
`tunities speci?c to that entity back to the consumer over the
`netWork via the communication application.
`In the context of a netWork con?guration, Internet proto
`cols may be used to return data to the communication
`application or to the device or system in Which it operates.
`The communication application may be implemented in a
`Web broWser, such as Internet Explorer or Netscape Navi
`gator. Examples of Ways of exchanging information betWeen
`a client player and a server include returning a Web page
`With metadata and program scripts designed to run on the
`end user’s system. The metadata itself may include active
`links, such as URLs to other netWork resources, such as a
`Web site or some other netWork service. The path of the
`identi?er from the decoding process, and the return path
`from a server to the communication application may include
`one or more hops through a Wire or Wireless connection
`using standard Wire and Wireless communication protocols
`like TCP/IP, HTTP, XML, WAP, Bluetooth, etc. In addition,
`data returned to the user may be routed through one or more
`
`10
`
`15
`
`25
`
`35
`
`45
`
`55
`
`65
`
`6
`servers that may forWard the data, and in some cases,
`augment the data or modify it in some fashion.
`FIG. 2 is a diagram illustrating applications of the system
`depicted in FIG. 1. In the application scenarios depicted in
`FIG. 2, an embedding process encodes an object identi?er
`(OID) into an audio ?le, such as an ID3 tag in the header of
`an MP3 ?le or audio frame headers in the MP3 ?le. FIG. 2
`shoWs tWo embedding scenarios. The ?rst is an MP3 dis
`tributor that embeds OIDs in MP3 ?les before transmitting
`them over a netWork, such as the Internet, typically via a
`Web site interface. The second is a ?le ripping process Where
`a programmed computer or other device extracts an audio
`object from packaged media such as a CD and converts it
`into a coded ?le format like MP3. In the latter case, the
`ripping process may extract metadata from the CD, such as
`the table of contents, and use this metadata as a key to a
`database (CDDB) to get information about the songs on the
`CD, such as title, artists, etc. The table of contents or other
`metadata from a package medium, such as optical or mag
`netic storage or ?ash memory, may be hashed into an index
`to a database entry that stores information about the media
`signal stored on the medium. The ripping process uses the
`information returned from