(12) INTERNATIONAL APPLICATION PUBLISHED UNDER THE PATENT COOPERATION TREATY(PCT)
`
`(19) World Intellectual Property Organization
`International Bureau
`
` QO AAAA
`
`(10) International Publication Number
`(43) International Publication Date
`
`
`
` 9 August 2001 (09.08.2001) PCT WO 0 1/58063 AZ
`
`(51) International Patent Classification’:
`
`H04H
`
`(21) International Application Number:
`
`PCT/GBO1/00413
`
`Panos Ashlar Court Ravenscourt Gardens London W6
`OTU [GB/GB}; Ashlar Court, Ravenscourt Gardens, Lon-
`don W6 OTU (GB). VOUKELATOS,Stathis [GB/GB], 7
`ComerCrescent, Southall, Middlesex UB2 4XD (GB).
`
`(22) International Filing Date: 2 February 2001 (02.02.2001)
`
`(25) Filing Language:
`
`(26) Publication Language:
`
`English
`
`English
`
`(74) Agent: SHARP,Alan, Cooper; QED LP. Services Lim-
`ited, Dawley Road, Hayes, Middlesex UB3 1HH (GB).
`
`(81) Designated States (national): JP, KR, US.
`
`(30) Priority Data:
`0002259.0
`
`2 February 2000 (02.02.2000)
`
`GB
`
`(84) Designated States (regional): European patent (AT, BE,
`CH, CY, DE, DK, ES, FI, FR, GB, GR,IE, IT, LU, MC,
`NL,PT, SE, TR).
`
`(71) Applicant (for all designated States except US): CEN-
`TRAL RESEARCH LABORATORIES LIMITED
`[GB/GB]; Dawley Road, Hayes, Middlesex UB3 1HH
`(GB).
`
`Published:
`
`without international search report and to be republished
`upon receipt ofthat report
`
`(72) Inventors; and
`(75) Inventors/Applicants (for US only): KUDUMAKIS,
`
`For two-letter codes and other abbreviations, refer to the "Guid-
`ance Notes on Codes andAbbreviations” appearing at the begin-
`ning ofeach regular issue of the PCT Gazette.
`
`(54) Title: A METHOD OF LABELLING A MULTI-FREQUENCY SIGNAL
`
`ENCODER
`
`Decoder
`
`
`
`
`
`
`
`
`Coded
`IR Filter
`Notch
`Signal
`[Frame
`
`
`‘ost
`regen||
`
`
`
`
`
`
`Codes ICE
`
`Extracted
`
`DECODER
`
`(57) Abstract: A system for labelling and subsequently identifying a multi-frequency signal, includes meansfor inserting a code sig-
`nal into a multi-frequency signal, signal distribution means, signal receiving means, code extractions means, and monitoring means
`to determine which parts of the frequency spectrum will at least partly mask the code signal at a given time using predetermined
`criteria. The meansforinserting a code signal includes means for eliminating one or more frequency ranges being locatedin a part
`of the frequency spectrum that will at least partly mask the codesignal, the location of the frequency ranges being eliminated from
`said multi-frequency signal varies with the frequency content of said multi-frequency signal.
`
`Sony Exhibit 1007
`Sony Exhibit 1007
`Sony v. MZ Audio
`Sony v. MZ Audio
`
`Encoder
`
`
`Input
`
`
`
`Signal["Frame Frequency WR Filter
`
`
`
`
`
`Formation
`Selection
`Design
`
`
`
`
`
`
`Coded
`
`Signal ICE
`
`MACACAA
`
`01/58063A2
`
`S
`
`

`

`WO01/58063
`
`PCT/GB01/00413
`
`A METHOD OF LABELLING A MULTI-FREQUENCY SIGNAL
`
`This invention relates to a methodof labelling a multi-frequency signal,
`
`and particularly, though not exclusively, to a methodoflabelling an audio or
`
`video signal prior to broadcast or distribution to provide an audit trail. It also
`
`relates to a system for labelling such a signal and a system for controlling replay
`
`of suchasignal.
`
`A known methodoflabelling or watermarking a plural channel audio
`
`10
`
`signalis disclosed in WO96/21290. Although the technology wasinitially
`
`targeted at the broadcast monitoring field, there are a numberof other
`
`application areas whereit can be employed. Theseinclude: digital television
`
`systems, streaming audio overthe Internet, and digital audio distribution. The
`
`system provides a methodoflabelling an audio signal by embedding an
`
`15
`
`identifying code inaudibly within the signal. The code can be used for identifying
`
`copyright ownership,fingerprinting and access control to digital audio data.
`
`Twonotchesare inserted in the audio band to provide frequencies at which the
`
`code maybeinserted. The codesignalis inserted as a series of pulses at the centre
`
`frequencies of the notches, and insertion is initiated when the program content
`
`20
`
`provides sufficient masking conditions for the code to be inserted inaudibly. A
`
`maskingfilter is employed to determine the masking level of the incomingsignal
`
`at the chosen code frequencies. The level of unwanted signal breakthroughin the
`
`notch frequencies is also monitored as it can prevent correct extraction of the
`
`code. Whilst this processis in progress, if either level falls below a pre-
`
`25
`
`determined value the code generation is abandoned. Thus, the codes are inserted
`
`as often as the input signal conditions allow.
`
`The technique can be applied to both mono andstereophonic signals. The
`
`codeis inserted in both channels simultaneously in a way that gives monophonic
`
`compatibility for coded stereo signals. The system, however, has a potential
`
`30
`
`security problem as an attacker can filter out the code by the use of narrow notch
`
`filters operating at the same frequencies as usedat the original encoding process.
`
`To enhancethe security of the system, US 5,113,437 discloses implementing
`
`1
`
`

`

`WO 01/58063
`
`PCT/GBO01/00413
`
`frequency hopping, by allowing the encoder to switch randomly between three
`
`predetermined notch frequencypairs. In order to decode thesignal it is necessary
`
`to provide three decoders connected in parallel, each decoder being responsive to
`
`one of the three notch frequency pairs. Another methodofinserting a code in one
`
`or more frequency componentsof an audio signal is disclosed in US 5,450,490.
`
`Accordingto a first aspect of the invention there is provided a method as
`
`specified in claims 1 — 10.
`
`According to a second aspectof the invention there is provided a system
`
`as specified in claim 11,
`
`According to a third aspect of the invention there is provided a system as
`
`specified in claim 12.
`
`According to a further aspect of the invention there is provided a signal as
`
`specified in claim 13.
`
`15
`
`20
`
`Embodiments ofthe invention will now be described, by way of example
`
`only, with reference to the accompanying schematic drawings in which:-
`
`Figure 1 showsa schematic diagram of an embodimentof the invention,
`
`Figure 2 showsa flow diagram of an embodimentof the invention, and
`
`Figure 3 showsa schematic diagram of a second embodimentof the
`
`invention.
`
`The present invention includes a method for appropriately selecting the
`
`part of the frequency spectrum where each watermark codeis inserted, providing
`
`25
`
`improved audio quality and extra security in the form of frequency hopping. The
`
`method described may be been implementedin software.
`
`The present invention differs from prior art systemsin that the selection of
`
`the location of the notch or notchesin the frequency spectrum of a signal (and
`
`hence the frequency of the embedded code)is chosen adaptively with regard to
`
`30
`
`the frequency content of the signal (with the possible addition of a random
`
`2
`
`

`

`WO 01/58063
`
`PCT/GB01/00413
`
`offset). Moreover, in generalit does not require the existence of a decoder array
`
`for all the possible notch frequency values in order to extract the codes, although
`
`use of such an array is not precluded.
`
`The placementof the notch frequencies plays a significant role to the
`
`subjective quality of the coded signals. The codes are more perceptibleif the
`
`notch frequencies coincide with the main frequency componentof the signal. On
`
`the other hand, they haveto be placed in a part of the spectrum with sufficient
`
`energy so that frequent masking conditions can be met. Therefore, a criterion that
`
`satisfies these requirements is neededforthe selection of the code frequencies.
`
`In one embodiment, the method comprises the following elements:
`
`e Segmentation of the input signals into frames and transformation into the
`
`frequency domain (unless the inputsignalis already in this form).
`
`e Selecting the appropriate notch frequency location for each frame accordingto
`
`a predeterminedcriterion.
`
`15
`
`e Adapting the encoder and decoderfilter parameters to the selected notch
`
`frequencies.
`
`e Adding a degree of randomness or unpredictability in determining the
`
`precise location of the notch frequencies.
`
`20
`
`The integration of these main elements of the invention to the encoder and
`
`decoder of WO96/21290is illustrated in the block diagram of Figure 1.
`
`The input signalis digitized and processed in frames. Once a frame of
`
`samples has been assembled, the notch frequencyselection criterion is applied to
`
`determinethe position of the notch frequencies. The function of the criterion is
`
`illustrated in Figure 2. A frequency analysis technique, e.g. FFT, is applied to
`
`generate a set of spectral coefficients. The spectral coefficients are grouped to
`
`form frequency bands of approximate width 0.6 - 0.7 kHz. The energy content of
`
`each bandis calculated from the corresponding spectral coefficients. The band
`
`with the maximum energy content is found. This process up to here can use part
`
`30
`
`of the psycho-acoustic modeling performed by an MPEG encoder. The notch
`
`3
`
`

`

`WO 01/58063
`
`PCT/GB01/00413
`
`frequencies are placed in one of the two neighbouring bands,asillustrated in the
`flow diagram of Figure 2. This Figure shows that when the band with maximum
`the codeis either placed in the nearest neighbour
`
`max’
`
`energy in it is determined (B
`if the energy peak is narrower than somethreshold value,or placed in
`maxt+l
`
`band B
`
`10
`
`the second nearest neighbour band B
`
`max+2
`
`if the energy peakis broad.
`
`Changingthe position of the notch frequencies during the encoding
`process involves the employment of a new filter set that will be responsive to the
`new frequencyvalues. Since the set of possible values that the notch frequencies
`can take is large and depends uponthesignal content, using a pre-computedset
`
`of filters for each possible notch frequency valueis not practical and would
`increase significantly the memory requirements of the system. Therefore,it is
`moreefficient to design the newfilter set in real time every time the position of
`the notches is changed. The band-pass and band-stopfilters are designed by
`applying a frequency transformation to a prototype low-passfilter, as described
`for examplein the book “Introduction to Digital Signal Processing”, by J. G.
`Proakis and D. G. Manolakis, Maxwell Macmillan International Editions (1989).
`By applying the appropriate frequency transformation to a 4"-orderIIR prototype
`low-passfilter 8"-order band-pass and band-stopfilters are generated. Thus, only
`onefilter set correspondingto the current notch frequency values needsto be
`
`20
`
`stored at any given time.
`The notch frequencyselection andfilter design process are applied in an
`identical fashion during the decoding ofa signal, as shownin Figure 1(b). The
`
`decoderis able to reproduce the same sequence of notch frequencies with the
`encoder and extract the codes from the signal, unless significant distortion has
`
`25
`
`been introduced to the signal spectrum.
`
`A second waytolocate the bestplace to insert the notchfilters will now be
`described. For each input block, a search is performed for the fundamental and
`harmonicsof the input audio stream. Methodssuchas Fast Fourier Transform,
`Cepstrum, Correlogram or the Gold-Rabiner algorithm can be used to find both
`the fundamental and its harmonics. The notch filters can be inserted in the upper
`
`30
`
`or lower edges of these harmonics (with the possible addition of a random offset).
`
`4
`
`

`

`WO 01/58063
`
`PCT/GB01/00413
`
`Care must be taken to ensur2 the insertion is not audible. This can be achieved,
`
`for example, using the psyclio-acoustic model.
`
`In addition to improved audio quality, security against malicious attacks,
`
`that would attempt to removethe codesbyinserting notchesto the codedsignal,
`is enhanced. If the code frequencies change frequently, then it becomes more
`
`difficult for an attacker to remove all the codes withoutintroducingsignificant
`
`distortion to the original signal content by removing many notchesall the time.
`
`The security of the system can be enhanced by adding some randomnessto the
`
`selection of the notch frequencies. Thisis illustrated in Figure 3, in the context of
`
`an access control application. Detection of the codesis possible only if the
`decoderis provided with the (secret) key. The access control mechanism will
`
`allow playbackof the audio signal onlyif the correct codes are extracted.
`
`The present invention can provide the following advantages:-
`
`a) Improved audio quality through adapting the notch frequencyselection to the
`
`15
`
`input signal content.
`
`b) Enhanced security against maliciousattacks.
`
`Whilst the high decoding performanceofthe original audio watermarking
`
`algorithm is maintained.
`
`Of course, the codeis not inserted continuously into the signal — the signal
`
`20
`
`is constantly monitored to check that the frequency content of the signal can
`
`mask the code, and insertion is not performed if the program content changes so
`
`that the code would become moreeasily audible. This can be done using the
`
`psycho-acoustic model as used by the MPEG encodingprocess, or the
`
`fundamental and harmonics methodas described aboveor by the frequency
`
`analysis described in WO96/21290.If there is not a long enough “window of
`
`opportunity” to insert the entire code sequencein a single step, it is of course
`
`possible to cut the code up into shorter lengths and insert each part in succession,
`
`preferably sending a few bits of data at either end of the portion ofcodetelling
`
`the decoder how much codeis being sent or where the next lengthis to start in
`
`5
`
`

`

`WO 01/58063
`
`PCT/GB01/00413
`
`terms of the entire code sequence. Amplitude or phase modulation of the code
`
`signal can be employed.
`
`As a further refinement, data telling the decoder where notches are going
`
`to be inserted, orthefilter coefficients corresponding to these notches (and/or
`
`data telling the decoder how muchcodeis being sent or where the next length is
`
`to start in terms of the entire code sequence) can be sentvia a different channel.
`
`For example, the MPEG-4 IPMP framework includes an IPMP data stream which
`
`can be used for the transmission of any private data (such as the notch
`
`frequencies) from the encoderto the decoder(for a full description ofthis see for
`
`example “MPEG-4Intellectual Property Management and Protection (IPMP)
`
`Overview & Applications Document MPEG/N2614, Rome December 1998,
`
`http:/ /www.cselt.it/mpeg/public/w2614.zip).
`
`The data, such as the notch frequenciesorthe filter coefficients
`
`corresponding to these notch frequencies, transmitted using the IPMP data
`
`15
`
`stream from the encoderto the decoder, may be encrypted,in orderto further
`
`improve security of an MPEG-4 terminal. A decryption key can be sent using the
`
`IPMPdata stream,or using a different communication channel.
`
`In this case, where the position of the notchesis sent via a channel such as
`
`the IPMPdata stream, the decoder does not need to run a psycho-acoustic model
`
`20
`
`or other similar analysis to calculate the positions of the notch frequenciesor the
`
`correspondingfilter coefficients. Thus, this embodimentis more robustto signal
`
`processing which can alter the apparent frequency contentof the signal between
`
`encoder and decoder, and can result in lower decoder complexity and cost.
`
`Finally, GB 0002259.0, from which the present application claimspriority,
`
`25
`
`especially the diagrams, is incorporated herein by reference.
`
`6
`
`

`

`WO 01/58063
`
`PCT/GB01/00413
`
`CLAIMS
`
`A methodoflabelling a multi-frequency signal including or consisting of
`
`a) eliminating one or more frequency ranges from said signal, b) inserting
`
`a codesignal into said multi-frequency signalin the said one or more
`
`frequency ranges, the multi-frequency signal being monitored to
`
`determine which parts of the frequency spectrum will at least partly mask
`
`the codesignalat a given time using one or more predeterminedcriteria,
`
`10
`
`and that the said one or more frequency ranges being eliminated is/are
`
`located in a part of the frequency spectrum thatwill at least partly mask
`
`the codesignal, characterised in that the location of the frequency ranges
`
`being eliminated from the said multi-frequency signal varies with the
`
`frequency contentof said multi-frequency signal.
`
`A methodas claimed in claim 1 in which an optimum frequencyfor
`
`insertion of the code signal is determined, and the corresponding
`
`frequency range being eliminatedis offset from said optimum frequency
`
`by a given amount, the given amount having an apparently random or
`
`pseudo-random or quasi-random nature and being capable of being
`
`predicted by a decoder having a suitable key.
`
`A methodas claimed in any preceding claim in which the multi-frequency
`
`signal comprises a video or audio signal.
`
`A methodas claimed in any preceding claim in which the multi-frequency
`
`signal comprises a compressed digital data stream.
`
`A method as claimed in any preceding claim in which the methodis
`
`performed in the frequency domain on a frame by framebasis.
`
`A methodof labelling a first multi-frequency signal, including or
`
`consisting of a) eliminating one or more frequency ranges from said first
`
`signal, b) inserting a code signal into saidfirst signal in the said one or
`
`7
`
`

`

`WO01/58063
`
`PCT/GB01/00413
`
`more frequency ranges, the said first signal being monitored successively
`to determine whichparts of the frequency spectrum will at least partly
`
`mask the codesignal at a given time using one or more predetermined
`
`criteria, the said one or more frequency ranges being eliminated being
`
`located in a part of the frequency spectrum thatwill at least partly mask
`the codesignal, characterised in that the method is performedin the
`
`frequency domain on a frame by framebasis.
`
`A methodas claimed in any preceding claim in which the multi-frequency
`signal comprises a plural channel audio signal, and the codeis inserted
`
`10
`
`into a plurality of said channels simultaneously.
`
`A methodas claimed in any preceding claim in which the signal comprises
`an MPEG-4 data stream,and data specifying the position of the frequency
`ranges being eliminated is sent from the encoderto the decoder using the
`
`IPMP data channel.
`
`15
`
`A methodas claimedin claim 2 in which the signal comprises an MPEG-4
`
`data stream, and the said key is sent from the encoderto the decoder using
`
`the IPMPdata channel.
`
`10.
`
`A methodas claimedin claim 8 in which the data specifying the position
`
`of the frequency ranges being eliminated is encrypted.
`
`20
`
`11.
`
`A system for labelling and subsequently identifying a multi-frequency
`
`signal, consisting of or including a meansfor inserting a code signal into a
`
`multi-frequency signal, signal distribution means, signal receiving means,
`
`code extraction means, and monitoring means to determine which parts of
`
`the frequency spectrum will at least partly mask the code signalat a given
`
`25
`
`time using one or more predetermined criteria, the means for inserting a
`
`code signal including meansfor eliminating one or more frequency ranges
`
`being located in a part of the frequency spectrum thatwill at least partly
`
`maskthe codesignal, the location of the frequency ranges being
`
`8
`
`

`

`WO 01/58063
`
`PCT/GB01/00413
`
`eliminated from the seid multi-frequency signal varying with the
`
`frequency content of said multi-frequencysignal.
`
`12.
`
`A system for controlling replay of a signal consisting of or including a
`
`meansfor inserting a code signal into a multi-frequency signal, signal
`
`distribution means, signal receiving means, code extraction means, and
`
`monitoring means to determine which partsof the frequency spectrum
`
`will at least partly mask the codesignal at a given time using one or more
`
`predeterminedcriteria, the meansfor inserting a code signal including
`
`meansfor eliminating one or more frequency ranges being located in a
`
`part of the frequency spectrum that will at least partly mask the code
`
`signal, the location of the frequency ranges being eliminated from the said
`
`multi-frequency signal varying with the frequency contentof said multi-
`
`frequencysignal.
`
`13.
`
`A signal labelled using a methodas specified in claims 1 - 10.
`
`9
`
`

`

`WO 01/58063
`
`PCT/GBO1/00413
`
`1/3
`
`Fig.1(a).
`
`Encoder
`
`Input
`Signal
`
`
`
`
`Notch
`
`
`Frame
`
`IIR Filter
`Frequency
`
`Formation
`
`Design
`
`
`Selection
`
`
`
`
`
`
`ICE
`
`ENCODER
`
`
`
`Fig.1(b).
`
`Decoder
`
`Coded
`ToIch
`
`
`
`
`
`Signal|Frame ole IIR Filter
`
`Seleatinn
`Design
`
`
`
`
`
`
`ICE
`DECODER
`
`
`
`10
`10
`SUBSTITUTE SHEET (RULE26)
`
`

`

`WO 01/58063
`
`PCT/GB01/00413
`
`2/3
`
`Fig.2.
`
`Group spectral coefficients into bands B; of bandwidth
`0.6-0.7 KHz approximately
`
`Calculate the energy E; of each subband B;, from the
`corresponding spectral coefficients
`
`Find the band B,,,, with maximum energy
`
`YES
` c=max+1
`
`c=max+2
`
`Insert notches in band Bo
`
`on
`11
`SUBSTITUTE SHEET (RULE26)
`
`

`

`WO 01/58063
`
`PCT/GB01/00413
`
`3/3
`
`Fig.3(a).
`
`Encoder
`
`
`Frame
`
`Solon
`Input
`
`
`otc
`Signal
`
`
`‘Selection
`
`
`
`
`
`IIR Filter
`
`Design
`
`
`Random Number
`Generator
`
`
`KEY
`
`ICE
`ENCODER
`
`Coded
`Signal
`
`Coded
`
`
`Notch
`Signal|Frame
`
`Frequency
`
`Formation
`Selection
`
`
`IIR Filter
` KEY
`Design
`
`
`
`
`ICE
`DECODER
`
`Fig.3(b).
`
`Decoder
`
`Random Number
`Generator
`
`NA
`
`
`
`
`
`
`
`
`
`Extracted
`Codes
`
`
`ACCESS
`CONTROL
`MECHANISM
`
`
`Output
`
`
`
`12
`12
`SUBSTITUTE SHEET (RULE26)
`
`