111111
`
`(19) United States
`(12) Patent Application Publication
`Alexander et al.
`
`11111111111111111111111111111111111111111111111111111111111111
`US 20010045153Al
`
`(10) Pub. No.: US 2001/0045153 A1
`Nov. 29, 2001
`(43) Pub. Date:
`
`(54) APPARATUS FOR DETECTING THE
`FUNDAMENTAL FREQUENCIES PRESENT
`IN POLYPHONIC MUSIC
`
`(75)
`
`Inventors: JohnS. Alexander, Voorhees, NJ (US);
`Kristopher A. Daniel, Philadelphia, PA
`(US); Themis G. Katsianos, Longmont,
`CO (US)
`
`Correspondence Address:
`AKIN, GUMP, STRAUSS, HAUER & FELD,
`L.L.P.
`ONE COMMERCE SQUARE
`2005 MARKET STREET, SUITE 2200
`PHILADELPHIA, PA 19103 (US)
`
`(73) Assignee: Lyrrus Inc. d/b/a GVOX
`
`(21) Appl. No.:
`
`09/797,893
`
`(22) Filed:
`
`Mar. 2, 2001
`
`Related U.S. Application Data
`
`(63) Non-provisional of provisional application No.
`60/188,057, filed on Mar. 9, 2000.
`
`Publication Classification
`
`(51)
`Int. Cl? ....................................................... G10H 7/00
`(52) U.S. Cl. ................................................. 84/609; 84/622
`
`ABSTRACT
`(57)
`A method for determining a fundamental frequency of a note
`in a musical signal comprising the steps of receiving the
`musical signal; determining a frequency of each of a plu(cid:173)
`rality of tracks of the musical signal; generating a plurality
`of subharmonics for each track; and computing the funda(cid:173)
`mental frequency based on a frequency of at least one
`subharmonic selected from the plurality of subharmonics,
`the frequency of the at least one selected sub harmonic being
`within a neighborhood of the frequency of the track having
`a lowest frequency.
`
`,--10
`
`12
`
`PROGRAMMED COMPUTER
`
`PROCESSOR
`
`32
`
`30
`
`18 22
`
`STORAGE
`DEVICE
`
`MODEM
`
`SYNTHESIZER/
`LOUD
`SPEAKER
`
`VIDEO
`DISPLAY
`
`20
`
`34
`
`28
`TO OTHER
`MUSICIANS,
`ETC.
`
`Verizon Wireless
`Exhibit 1029-0001
`
`

`

`Patent Application Publication Nov. 29, 2001 Sheet 1 of 23
`
`US 2001/0045153 A1
`
`,......_
`Ul
`-o
`c
`0
`0
`(J)
`...__,.
`Ul
`w
`::2:
`t-
`
`t0
`0
`N
`
`0
`00
`..-
`
`00
`L()
`..-
`
`c.o
`t0
`..--
`
`t0
`..-
`..--
`
`0
`(J)
`0
`
`1'-c.o
`0
`
`L()
`~
`0
`
`t0
`N
`0
`
`0
`
`ro
`0
`I
`
`Verizon Wireless
`Exhibit 1 029-0002
`
`c.o
`0
`
`0
`
`N
`0
`I
`
`~
`0
`I
`
`<.0
`0
`I
`
`w o,..--..
`:::)Cfl
`t-t-
`- . . J
`..Jo
`0..>
`::2:---
`<(
`
`

`

`Patent Application Publication Nov. 29, 2001 Sheet 2 of 23
`
`US 2001/0045153 A1
`
`,..--....
`0
`Q)
`(I)
`
`'-.-./
`
`E
`w
`~
`1-
`
`-
`
`0
`0
`L()
`""¢
`
`0
`0
`0
`""¢
`
`0
`0
`L{)
`1'0
`
`0
`0
`0
`1'0
`
`0
`0
`l{")
`C\.1
`
`0
`0
`0
`N
`
`0
`0
`L{)
`...-
`
`0
`0
`0
`
`"
`
`I
`0
`0
`L{)
`
`L I,
`
`0
`0
`L{)
`~
`
`I
`0
`0
`0
`...--
`
`>-
`0
`Z,--...
`WN
`::J:r:
`0'-.-./
`w
`0::
`lL..
`
`Verizon Wireless
`Exhibit 1 029-0003
`
`I
`r
`0
`0
`L{)
`N
`
`0
`0
`0
`N
`
`~
`•
`-~
`~
`
`

`

`Fig. 3
`
`,-10
`
`12
`
`PROGRAMMED COMPUTER
`
`INPUT
`DEVICE
`
`PROCESSOR
`,----------..,
`: I DIGITAL
`
`I
`
`I
`
`OUTPUT
`PORT
`
`36
`
`BUS 1 .. I •I
`
`MODEM
`
`20
`
`34
`
`I .. TO NOTATION
`ENGINE
`
`28
`TO OTHER
`MUSICIANS,
`ETC.
`
`18 22
`
`STORAGE
`DEVICE
`
`SYNTHESIZER/
`LOUD
`SPEAKER
`
`VIDEO
`DISPLAY
`I
`L __________ .J
`
`26
`
`24
`
`15
`
`m<
`9-co
`o: ~­
`-·N
`....... 0
`--"::J
`
`I
`
`2~ <D -·
`> oro
`oro
`ocn
`.t:>.cn
`
`I")
`
`I")
`
`""C
`~ .....
`~ = .....
`~ 't:l -....
`~ ..... .... 0 =
`~ -a -....
`~ ..... .... 0 =
`
`z
`0
`~
`N
`~~
`
`N c c
`'"""'
`'JJ. =(cid:173)~
`~ .....
`
`~
`0 ......,
`N
`~
`
`Cj
`'JJ.
`
`N c c
`'"""' -c c
`
`""" Ul
`'"""' Ul
`~ >
`'"""'
`
`

`

`Patent Application Publication Nov. 29, 2001 Sheet 4 of 23
`
`US 2001/0045153 A1
`
`START
`
`/"50
`
`100 ' -
`
`GENERATE
`TRACKS
`
`200 ' -
`
`CLUSTER
`TRACKS
`
`300 '-
`
`400 '-
`
`DETECT
`INSIDE
`NOTES
`
`I
`
`CREATE
`EVENT
`LIST
`
`500 ' -
`
`GENERATE
`MIDI
`OUTPUT
`
`END
`
`Fig. 4
`
`Verizon Wireless
`Exhibit 1 029-0005
`
`

`

`Patent Application Publication Nov. 29, 2001 Sheet 5 of 23
`
`US 2001!0045153 Al
`
`START
`
`/" 100
`
`RECEIVE
`SIGNAL
`
`v
`
`100.1
`
`GENERATE v
`
`SPECTRUM
`
`100.2
`
`INTERPOLATE
`AMP /FREQUENCY J
`OF SPECTRUM
`PEAKS
`
`100.3
`
`ASSOCIATE
`
`REMAINING v
`
`PEAKS WITH
`TRACKS
`
`100.4
`
`UPDATE
`TRACK
`FREQUENCY
`
`v
`
`100.5
`
`TO 200
`
`Fig. 5
`
`Verizon Wireless
`Exhibit 1 029-0006
`
`

`

`Patent Application Publication Nov. 29, 2001 Sheet 6 of 23
`
`US 2001!0045153 Al
`
`FROM 100.3
`
`COMPUTE
`DISTANCE OF
`EACH PEAK TO
`EACH TRACK
`
`100.4.1
`_r
`
`DISCARD
`PEAKS WITH
`DISTANCE > D1
`
`v
`
`100.4.2
`
`FOR EACH PEAK
`GENERATE A
`SORTED LIST OF
`TRACK DISTANCE
`
`v
`
`100.4.3
`
`I
`
`TO 100.5
`
`Fig. 6
`
`Verizon Wireless
`Exhibit 1 029-0007
`
`

`

`Patent Application Publication Nov. 29, 2001 Sheet 7 of 23
`
`US 2001!0045153 Al
`
`FROM 100.4
`
`; - 100.5
`
`UPDATE
`TRACK WITH
`PARAMETERS
`OF NEAREST
`PEAK
`
`.f 100.5.1
`
`COAST TRACKS v
`
`NOT ASSOCI A TED
`WITH A PEAK
`
`100.5.2
`
`INSTANTIATE
`
`A NEW TRACK v 100.5.3
`
`FOR EACH
`UNASSOCIATED
`PEAK
`
`TO 200
`
`Fig. 7
`
`Verizon Wireless
`Exhibit 1 029-0008
`
`

`

`Patent Application Publication Nov. 29, 2001 Sheet 8 of 23
`
`US 2001/0045153 A1
`
`FROM 1 00
`
`j
`
`200
`
`200.1
`
`200.2
`
`200.3
`
`200.4
`
`200.5
`
`FILTER
`TRACKS
`
`COMBINE
`TRACKS
`
`CLASSIFY /
`RECLASSIFY
`TRACKS
`
`UPDATE
`CENTROIDS
`
`MERGE
`CENTROIDS
`
`200.6
`
`YES
`
`NO
`
`TO 300
`
`Fig. 8
`
`Verizon Wireless
`Exhibit 1 029-0009
`
`

`

`Patent Application Publication Nov. 29, 2001 Sheet 9 of 23
`
`US 2001!0045153 Al
`
`Fig. 9A
`
`/200.3
`
`FROM 200.2
`
`ASSIGN CENTROID
`TO EACH TRACK
`
`SET i = 0
`
`ZERO MEMBER LISTS
`
`i = i + 1
`
`SET j
`
`- 1
`
`COMPARE FREQUENCY
`OF CENTROID i
`TO TRACK j
`
`GENERATE
`SUBHARMONIC
`SERIES FOR THE
`CENTROID /TRACK
`HAVING THE HIGHER
`FREQUENCY
`
`SET k = 1
`
`COMPUTE DISTANCE
`BETWEEN THE
`CENTROID /TRACK
`HAVING THE LOWER
`FREQUENCY AND
`SUBHARMONIC k
`
`TO 200.3.10
`
`200.3.1
`
`200.3.2
`
`200.3.3
`
`200.3.4
`
`200.3.5
`
`200.3.6
`
`200.3.7
`
`200.3.8
`
`200.3.9
`
`FROM 200.3.1 0
`
`FROM 200.3.14
`FROM 200.3.15
`
`Verizon Wireless
`Exhibit 1029-0010
`
`

`

`Patent Application Publication Nov. 29, 2001 Sheet 10 of 23
`
`US 2001!0045153 A1
`
`FROM 200.3.9
`
`TO 200.3.16
`
`TO 200.3.18
`
`TO 200.3.17
`
`200.3.10
`
`YES
`
`SELECT
`SUBHARMONIC
`THAT IS NEAREST IN
`FREQUENCY TO
`CENTROID /TRACK
`
`200.3.11
`
`200.3.12
`
`NO
`
`ADD TRACK TO
`CENTROID LIST
`
`200.3.13
`
`200.3.14
`
`YES
`
`200.3.15
`
`YES
`
`SET CHANGE FLAG
`IF ANY TRACK HAS
`CHANGED CENTROID
`
`200.3.20
`
`TO 200.4
`
`Fig. 9B
`
`Verizon Wireless
`Exhibit 1029-0011
`
`

`

`Patent Application Publication Nov. 29, 2001 Sheet 11 of 23
`
`US 2001/0045153 A1
`
`I<~
`
`l
`
`1
`
`0
`0
`L{)
`N
`
`0
`0
`0
`N
`
`0
`0
`L{)
`
`0
`0
`0
`
`0
`0
`L{)
`
`>-()
`z_...
`WN
`::JI
`o.._...
`w
`0::::
`L1...
`
`_...
`
`(.)
`
`Q) en
`E
`.._...
`w
`~
`1-
`
`0
`0
`1.0
`...,..
`
`0
`0
`0
`"¢
`
`0
`0
`1.0
`I"')
`
`0
`0
`0
`I"')
`
`0
`0
`I()
`N
`
`0
`0
`0
`C\1
`
`0
`0
`I()
`...-
`
`0
`0
`0
`..--
`
`0
`
`Verizon Wireless
`Exhibit 1029-0012
`
`

`

`Patent Application Publication Nov. 29, 2001 Sheet 12 of 23
`
`US 2001/0045153 A1
`
`/ " 200.4
`
`200.4.1 \._
`
`FROM 200.3
`
`CALCULATE A
`NEW TIME
`FOR EACH
`CENTROID
`
`200.4.2 \._
`
`CALCULATE A
`NEW FREQUENCY
`FOR EACH
`CENTROID
`
`1
`TO 200 .. 5
`
`Fig.ll
`
`Verizon Wireless
`Exhibit 1029-0013
`
`

`

`Patent Application Publication Nov. 29, 2001 Sheet 13 of 23
`
`US 2001!0045153 A1
`
`/200.4.2
`
`200.4.2.1
`
`200.4.2.2
`
`200.4.2.3
`
`FROM 200.4.1
`
`FOR EACH CENTROID,
`CONSTRUCT
`SUBHARMONIC
`SERIES FOR EACH
`MEMBER TRACK
`
`FOR EACH CENTROID,
`CREATE AN
`UPDATE LIST OF
`NEIGHBORING
`SUBHARMONICS
`
`AVERAGE THE
`FREQUENCIES OF
`THE SUBHARMONICS
`IN THE RESPECTIVE
`MEMBER LISTS
`
`Fig. 12
`
`Verizon Wireless
`Exhibit 1029-0014
`
`

`

`Patent Application Publication Nov. 29, 2001 Sheet 14 of 23
`
`US 2001/0045153 A1
`
`Fig.13A
`
`~200.5
`
`FROM 200.4
`
`SET i = 1
`
`SET j = 1
`
`COMPARE FREQUENCY
`OF CENTROID i
`TO FREQUENCY
`OF CENTROID j
`
`GENERATE
`SUBHARMONIC
`SERIES FOR THE
`CENTROID
`HAVING THE HIGHER
`FREQUENCY
`
`SET k = 1
`
`200.5.1
`
`200.5.2
`
`200.5.3
`
`200.5.4
`
`200.5.5
`
`COMPUTE DISTANCE
`BETWEEN THE
`CENTROID
`HAVING THE LOWER
`FREQUENCY AND
`SUBHARMONIC k
`
`200.5.6
`
`200.5.7
`
`YES
`
`TO 200.5.8
`
`FROM 200.5.12
`
`FROM 200.5.13
`
`Verizon Wireless
`Exhibit 1029-0015
`
`

`

`Patent Application Publication Nov. 29, 2001 Sheet 15 of 23
`
`US 2001/0045153 A1
`
`FROM 200.5. 7
`
`TO 200.5.15
`
`TO 200.5.16
`
`SELECT
`SUBHARMONIC
`THAT IS NEAREST IN
`FREQUENCY TO
`CENTROID
`
`200.5.8
`
`200.5.9
`
`NO
`
`MERGE CENTROIDS
`AND MEMBER
`LISTS
`
`UPDATE
`CENTROID
`TIME/FREQUENCY
`
`200.5.10
`
`200.5.11
`
`200.5.12
`
`YES
`
`200.5.13
`
`YES
`
`200.5.20
`
`SET CHANGE FLAG
`IF MEMBER
`LIST CHANGES
`
`TO 200.6
`
`Fig.13B
`
`Verizon Wireless
`Exhibit 1029-0016
`
`

`

`Patent Application Publication Nov. 29, 2001 Sheet 16 of 23
`
`US 2001/0045153 A1
`
`..........
`0
`(I)
`fJ)
`E
`...._....
`w
`:::;?!
`} -
`
`0
`0
`L[)
`.q-
`
`0
`0
`0 .q-
`
`0
`0
`1.0
`I")
`
`0
`0
`0
`I"")
`
`0
`0
`L[)
`N
`
`0
`0
`0
`N
`
`0
`0
`L()
`....-
`
`0
`0
`0 a..-
`
`I<!>
`
`Verizon Wireless
`Exhibit 1029-0017
`
`I
`1
`
`0
`0
`0
`N
`
`0
`0
`L[)
`N
`
`0
`0
`L[)
`,......
`
`0
`0
`0 ,......
`
`0
`0
`L[)
`
`>-() z,......,
`
`I.J.JN
`::JI
`o..._
`I.J.J
`0::::
`LL.
`
`

`

`Patent Application Publication Nov. 29, 2001 Sheet 17 of 23
`
`US 2001/0045153 A1
`
`0
`0
`1.{)
`.q-
`
`0
`0
`0 .q-
`
`0
`0
`tO
`I"')
`
`0
`0
`0
`I"'")
`
`0
`0
`tO
`C\1
`
`0
`0
`0
`C\1
`
`........... .
`
`(,)
`Q)
`CIJ
`E
`' - '
`w
`~
`I-
`
`I<~
`
`<~
`~
`
`~
`
`J
`~------~------~~------;--------;--------T 8
`
`0
`0
`I!)
`C\1
`
`0
`0
`0
`N
`
`0
`0
`L()
`
`I
`
`0
`0
`0
`
`0
`0
`lO
`or-
`
`..,
`
`0
`
`O-r-
`
`...... r-..
`
`['...
`~
`~
`
`0
`0
`L()
`
`>-u
`
`z~
`WN
`:::J:::c
`0 "'-../
`w
`~
`LL.
`
`Verizon Wireless
`Exhibit 1029-0018
`
`

`

`Patent Application Publication Nov. 29, 2001 Sheet 18 of 23
`
`US 2001/0045153 A1
`
`v 300.4
`
`UPDATE
`CENTROID
`LIST
`
`1
`
`TO 400
`
`Fig.16
`
`Verizon Wireless
`Exhibit 1029-0019
`
`

`

`Patent Application Publication Nov. 29, 2001 Sheet 19 of 23
`
`US 2001/0045153 A1
`
`; - 300.3
`
`FROM 300.2
`
`300.3.1
`
`NORMALIZE
`DOCKET ELEMENT
`
`300.3.2
`
`PROCESS THROUGH
`NEURAL NETWORK
`
`300.3.3
`
`UNNORMALIZE
`DOCKET ELEMENT
`
`300.3.5
`
`YES
`
`ADD NEW
`DOCKET ENTRY
`TO DOCKET
`
`TO 300.4
`
`Fig. 17
`
`Verizon Wireless
`Exhibit 1 029-0020
`
`

`

`WEIGHT (2,1)
`
`WEIGHTS (N ... 24),(1)
`1-------------
`BIAS (1, 1)
`
`WEIGHTS (N ... 24),(2)
`1-------------
`
`WEIGHTS (N .•. 24),(3)
`1-------------
`
`24 DATA
`INPUT
`VALUES
`FROM
`
`DOCKET L WEIGHTS_(N . ..:.:24),(4L
`
`Fig. 18
`
`,--38
`
`I
`
`I
`
`OUTPUT
`.. , PROBA-
`BILITY
`
`WEIGHTS (N ... 24),(5)
`1-------------
`
`WEIGHTS (N ... 24),(6)
`1-------------
`BIAS (1 ,6)
`
`WEIGHT (2,6)
`
`m<
`9-co
`fr :::J.
`-·N
`....... 0
`--"::J
`
`2~ <D -· oro
`
`oro
`1\Jcn
`--"en
`
`""C
`~ .....
`~ = .....
`~ 't:l -....
`.... 0 =
`~ -a -....
`.... 0 =
`
`I")
`
`~ .....
`
`I")
`
`~ .....
`
`z
`0
`~
`N
`~~
`
`N c c
`'"""'
`'JJ. =(cid:173)~
`~ .....
`N c
`0 ......,
`N
`~
`
`Cj
`'JJ.
`
`N c c
`'"""' -c c
`
`""" Ul
`'"""' Ul
`~
`>
`'"""'
`
`

`

`Patent Application Publication Nov. 29, 2001 Sheet 21 of 23
`
`US 2001/0045153 A1
`
`I
`I
`
`I
`I
`
`I
`I
`
`I
`I
`
`0
`0
`0
`
`-I-
`
`~
`
`I
`
`I
`
`-~
`
`- - - -
`
`~
`
`- -
`
`...
`
`- -
`
`I-
`
`0
`0
`0')
`
`0
`0
`(X)
`
`0
`0
`I"
`
`0
`0
`tO
`
`0
`0
`Lf)
`
`r-....
`u
`Q) en
`E
`'-"'
`w
`~
`1-
`
`- - -
`
`-I-
`
`-I-
`
`-I-
`
`0
`0 ..q-
`
`0
`0 n
`
`0
`0
`N
`
`0
`0
`
`~
`
`Verizon Wireless
`Exhibit 1 029-0022
`
`1-
`
`- -
`
`I-
`
`I
`T
`0
`N
`
`I
`I
`
`0
`0
`
`~
`
`~
`
`I
`
`•
`0
`(X)
`
`I
`
`0
`tO
`
`I
`
`0
`..q-
`
`I
`
`0
`N
`
`0\
`~
`. ~
`•
`~
`
`w
`1-0:::
`ow
`zco
`~
`-~ oz
`~
`
`

`

`Patent Application Publication Nov. 29, 2001 Sheet 22 of 23
`
`US 2001!0045153 Al
`
`START
`
`RECEIVE
`SIGNAL
`
`REJECT
`NOISE
`
`I 100'
`
`1 00.1'
`
`f
`
`v
`
`1 00.6'
`
`GENERATE
`SPECTRUM
`
`1 00.2'
`
`f
`
`INTERPOLATE
`AMP /FREQUENCY
`OF SPECTRUM
`PEAKS
`
`v
`
`ASSOCIATE
`REMAINING
`PEAKS WITH
`TRACKS
`
`1 00.4'
`
`f
`
`TO 200'
`
`Fig. 20
`
`Verizon Wireless
`Exhibit 1 029-0023
`
`

`

`Patent Application Publication Nov. 29, 2001 Sheet 23 of 23
`
`US 2001!0045153 Al
`
`~ 200'
`
`FROM 100'
`
`CLASSIFY I
`RECLASSIFY
`TRACKS
`
`UPDATE
`CENTROIDS
`
`MERGE
`CENTROIDS
`
`DETERMINE
`STATE
`OF NOTE
`
`200.3'
`
`200.4'
`
`200.5'
`
`200. 7'
`
`200.6'
`
`YES
`
`NO
`
`TO 300
`
`Fig. 21
`
`Verizon Wireless
`Exhibit 1 029-0024
`
`

`

`US 2001/0045153 A1
`
`Nov. 29, 2001
`
`1
`
`APPARATUS FOR DETECTING THE
`FUNDAMENTAL FREQUENCIES PRESENT IN
`POLYPHONIC MUSIC
`
`CROSS REFERENCE TO RELATED
`APPLICATIONS
`[0001] This application claims the benefit of U.S. Provi(cid:173)
`sional Application No. 60!188,057, filed Mar. 9, 2000,
`entitled "Polyphonic Pitch Detecting Method".
`
`BACKGROUND OF THE INVENTION
`[0002] The present invention relates to a method and
`apparatus for analyzing audio signals, and more particularly,
`to a method and apparatus for determining the fundamental
`frequencies present in a polyphonic musical signal.
`
`[0003] Many persons, including musicians, own or have
`access to a personal computer. Further, many musicians have
`a need for transcribing a musical score or for comparing the
`performance of a musical score with the transcribed score.
`
`[0004] An apparatus, such as a personal computer, capable
`of detecting the notes of a live musical performance would
`be useful for rendering the notes of the performance into a
`score or for comparison of the notes with the notes of an
`existing score.
`
`[0005] The sound generated by a musical instrument fre(cid:173)
`quently comprises a plurality of simultaneously sounded
`notes, i.e. a chord. Music comprising a plurality of simul(cid:173)
`taneous notes is referred to as polyphonic music, whereas
`music comprising only a single note at a time, such as vocal
`music, is referred to as monophonic music. Regardless of
`whether the music is polyphonic or monophonic, the sound
`wave produced by a note generally comprises a plurality of
`harmonically related frequencies (harmonics), the harmonic
`of the note having the lowest frequency being called the
`fundamental frequency of the sound wave.
`
`[0006]
`It is well known that a musical sound can be
`synthesized for a particular musical instrument or vocalist if
`the frequency, amplitude and time boundaries of the funda(cid:173)
`mental frequency component(s) of the musical sound can be
`determined. However, it is a non-trivial problem to isolate
`and measure the actual frequency, amplitude and the time
`boundaries of the fundamental frequency component(s)
`because of the harmonic characteristic of the musical sound
`wave. The problem is further exacerbated when a chord is
`sounded, because there is a plurality of fundamental fre(cid:173)
`quencies to be determined and the harmonics of the funda(cid:173)
`mental frequencies are generally interleaved. The complex(cid:173)
`ity of a polyphonic musical signal is clearly shown in FIGS.
`1 and 2 in which FIG. 1 shows a time domain representation
`of a four note chord comprising the notes C3, E3, G3 and
`C4, and FIG. 2. shows a frequency domain representation of
`the same chord. In addition to the innate complexity of a
`polyphonic signal, a chord may be characterized by notes
`which are harmonically related, i.e. C below middle C and
`middle C, as depicted in FIGS. 1 and 2. In this case all of
`the harmonics of the fundamental frequency of the higher
`note are masked by the harmonics of the lower note. The
`process for detection of the fundamental frequencies of the
`aforementioned type of chord is referred to in this applica(cid:173)
`tion as the "inside note" detection process.
`
`[0007] Some of the problems and the drawbacks of vari(cid:173)
`ous approaches for detecting the fundamental frequencies
`
`characteristic of polyphonic music are cited in U.S. Pat. No.
`6,140,568 which is hereby incorporated by reference and
`need not be repeated here. Accordingly, there is a need for
`a method, suitable for execution in a personal computer, for
`reliably detecting the fundamental frequencies present in
`polyphonic music. It is well known that fast Fourier trans(cid:173)
`form (FFT) techniques are the most efficient means for
`determining the spectral composition of a complex wave(cid:173)
`form where there is no a priori knowledge of the spectrum
`of the complex waveform. However, it is categorically stated
`in U.S. Pat. No. 6,140,568 that the FFT is not a practical
`technique for extracting the fundamental frequencies of the
`musical signal in a personal computer because of the neces(cid:173)
`sity for performing the extraction process with an FFT
`having an integration period that is a multiple of the aggre(cid:173)
`gate wavelength of the sound wave, creating unrealistically
`massive demands on system resources.
`
`[0008] The present invention overcomes the problems of
`the prior art by utilizing a combination of short period FFT
`processing and computationally efficient time domain pro(cid:173)
`cessing to extract the fundamental frequencies of the musi(cid:173)
`cal sound wave, thereby eliminating the need to perform an
`FFT having an integration period that is a multiple of the
`aggregate wavelength of the sound wave and in so doing
`making it feasible to utilize a personal computer for detect(cid:173)
`ing the fundamental frequencies of a polyphonic musical
`signal.
`
`BRIEF SUMMARY OF THE INVENTION
`
`[0009] Briefly stated, the present invention comprises a
`method for determining a fundamental frequency of a note
`in a musical signal comprising the steps of receiving the
`musical signal; determining a frequency of each of a plu(cid:173)
`rality of tracks of the musical signal; generating a plurality
`of subharmonics for each track; and computing the funda(cid:173)
`mental frequency of the note based on a frequency of one or
`more subharmonics selected from the plurality of subhar(cid:173)
`monics, where the frequency of the selected subharmonic(s)
`are within a neighborhood of the frequency of the track
`having a lowest frequency.
`
`[0010] The present invention includes a further method for
`determining a fundamental frequency of a note in a musical
`signal. The method comprises the steps of receiving the
`musical signal; determining a frequency of each of a plu(cid:173)
`rality of tracks of the musical signal; classifying each track
`to one and only one of a plurality of centroids, where each
`centroid is characterized by a frequency; updating the fre(cid:173)
`quency of each one of the centroids based on an average of
`the frequency of each track classified to the respective
`centroid and a frequency of one or more subharmonics of
`each centroid for which the frequency of one or more of the
`subharmonics falls within a neighborhood of the respective
`centroid; and merging the plurality of centroids such that a
`remaining centroid represents the fundamental frequency of
`the note.
`
`[0011] The present invention also includes a method for
`detecting the presence of an inside note comprising the steps
`of creating a list of a plurality of docket entries meeting a
`predetermined criteria in which each docket entry represents
`a separate note; applying each docket entry to a previously
`trained neural network; and outputting a signal for each
`docket entry proportional to a probability of the docket entry
`
`Verizon Wireless
`Exhibit 1 029-0025
`
`

`

`US 2001/0045153 Al
`
`Nov. 29, 2001
`
`2
`
`being a note having a fundamental frequency within a
`neighborhood of a harmonic of a different note.
`[0012] The present invention also includes a computer
`readable medium on which is stored a computer executable
`program code for determining at one or more fundamental
`frequencies of a musical signal. The program comprises
`code for receiving a musical signal; code for determining a
`frequency of each of a plurality of tracks in the musical
`signal; code for generating a plurality of subharmonics for
`each track; and code for computing one or more fundamen(cid:173)
`tal frequencies based one or more subharmonics selected
`from the plurality of sub harmonics where a frequency of the
`selected subharmonics are within a neighborhood of the
`frequency of the track having a lowest frequency.
`
`[0013] The present invention also includes a programmed
`computer for determining a fundamental frequency of a
`musical signal. The computer comprises an input device for
`receiving a musical signal and for converting the musical
`signal into a digital signal; a storage device having a portion
`for storing computer executable program code; a processor
`for receiving the digital signal and the computer program,
`where the computer program operates on the digital signal to
`determine a frequency of each of a plurality of tracks in the
`musical signal to generate a plurality of subharmonics for
`each track and computes the fundamental frequency based
`on a frequency of one or more subharmonics selected from
`the plurality of subharmonics and a frequency of the track
`having a lowest frequency; and an output device for out(cid:173)
`putting the at least one fundamental frequency.
`BRIEF DESCRIPTION OF THE SEVERAL
`VIEWS OF THE DRAWINGS
`[0014] The foregoing summary as well as the following
`detailed description of preferred embodiments of the inven(cid:173)
`tion, will be better understood when read in conjunction with
`the appended drawings. For the purpose of illustrating the
`invention, there are shown in the drawings embodiments
`which are presently preferred. It should be understood,
`however, that the invention is not limited to the precise
`arrangements and instrumentalities shown.
`[0015]
`In the drawings:
`
`[0016] FIG. 1 is a time domain representation of a four
`note chord comprising notes C3, E3, G3 and C4;
`[0017] FIG. 2 is a frequency domain representation of the
`chord shown in FIG. 1, illustrating a plurality of tracks
`formed from a spectral analysis of the four note chord;
`[0018] FIG. 3 is a functional block diagram of an appa(cid:173)
`ratus for detecting the fundamental frequencies present in a
`polyphonic musical signal in accordance with a preferred
`embodiment of the present invention;
`[0019] FIG. 4 is a flow diagram of a method for deter(cid:173)
`mining the fundamental frequencies present in the poly(cid:173)
`phonic musical signal in accordance with a first preferred
`embodiment of the present invention;
`[0020] FIG. 5 is a flow diagram of the steps for generating
`the plurality of tracks in accordance with the first preferred
`embodiment of the present invention;
`[0021] FIG. 6 is a flow diagram depicting the steps for
`associating a spectral peak with one of the plurality of tracks
`in accordance with the first preferred embodiment of the
`present invention;
`
`[0022] FIG. 7 is a flow diagram of the steps for updating
`a frequency of each of the tracks in accordance with the first
`preferred embodiment of the present invention;
`
`[0023] FIG. 8 is a flow diagram of the steps for clustering
`the plurality of tracks in accordance with the first preferred
`embodiment of the present invention;
`
`[0024] FIG. 9A is a flow diagram of the steps for classi(cid:173)
`fying/reclassifying the plurality of tracks in accordance with
`the first preferred embodiment of the present invention;
`
`[0025] FIG. 9B is a continuation of FIG. 9A;
`
`[0026] FIG.10 is frequency domain depiction of the notes
`of FIG. 2, showing the plurality of tracks and the initial
`assignment of a centroid to each of the plurality of tracks in
`accordance with the first preferred embodiment of the
`present invention;
`
`[0027] FIG. 11 is a flow diagram of the steps for updating
`a time and a frequency of each of the centroids in accordance
`with the first preferred embodiment of the present invention;
`
`[0028] FIG. 12 is a detailed flow diagram of the steps for
`updating the frequency of each of the centroids in accor(cid:173)
`dance with the first preferred embodiment of the present
`invention;
`
`[0029] FIG. 13A is a flow diagram of the steps for
`merging the centroids in accordance with the first preferred
`embodiment of the present invention;
`
`[0030] FIG. 13B is a continuation of FIG. 13A;
`
`[0031] FIG. 14 is a frequency domain depiction of the
`tracks of FIG. 2 showing the reclassification of the tracks to
`the centroids after a first iteration of a classification/reclas(cid:173)
`sification loop;
`
`[0032] FIG. 15 is a frequency domain depiction of the
`tracks of FIG. 2 showing the reclassification of the tracks to
`the centroids after a second iteration of the classification/
`reclassification loop;
`
`[0033] FIG.16 is a flow diagram of the steps for detecting
`an inside note in accordance with the first preferred embodi(cid:173)
`ment of the present invention;
`
`[0034] FIG. 17 is a flow diagram of the steps for com(cid:173)
`puting the probability of a potential note being an inside note
`in accordance with the first preferred embodiment of the
`present invention;
`
`[0035] FIG. 18 is a schematic block diagram of a neural
`network in accordance with the first preferred embodiment
`of the present invention;
`
`[0036] FIG. 19 is a frequency domain depiction of the
`tracks of FIG. 2 showing the final determination of a start
`time, a stop time and a fundamental frequency of each of the
`notes comprising the four note chord after processing by the
`neural network;
`
`[0037] FIG. 20 is a flow diagram of the steps for gener(cid:173)
`ating the plurality of tracks according to the second pre(cid:173)
`ferred embodiment of the present invention; and
`
`[0038] FIG. 21 is a flow diagram of the steps for cluster(cid:173)
`ing the plurality of tracks according to the second preferred
`embodiment of the present invention.
`
`Verizon Wireless
`Exhibit 1 029-0026
`
`

`

`US 2001/0045153 Al
`
`Nov. 29, 2001
`
`3
`
`DETAILED DESCRIPTION OF 1HE
`INVENTION
`
`[0039] Referring to the drawings, wherein like numerals
`are used to indicate like elements throughout the several
`figures and the use of the indefinite article "a" may indicate
`a quantity of one or more than one of an element, there is
`shown in FIG. 3 a block diagram of an apparatus 10 for
`detecting a single note of a monophonic musical signal and
`multiple simultaneous notes of a polyphonic musical signal
`according to a preferred embodiment of the present inven(cid:173)
`tion. The apparatus 10 includes a programmed computer 12
`comprising an input device 14 for receiving a representation
`of the musical signal, a storage device 22 having a portion
`for storing a computer executable program code (computer
`program), a processor 20 for executing the computer pro(cid:173)
`gram stored in the storage device 22 and an output device 15
`for outputting a signal representing the notes of the poly(cid:173)
`phonic musical signal, all of which are connected by a bus
`36.
`
`[0040] Desirably, the programmed computer 12 is a type
`of open architecture computer called a personal computer
`(PC). In the first preferred embodiment, the programmed
`computer 12 operates under the Windows® operating sys(cid:173)
`tem manufactured by Microsoft® Corporation and employs
`a Pentium® III microprocessor chip manufactured by Intel®
`Corporation as the processor 20. However, as will be appre(cid:173)
`ciated by those skilled in the art, other operating systems and
`microprocessor chips may be used. Further, it is not neces(cid:173)
`sary to use a PC architecture. Other types of computers, such
`as the Apple® Macintosh® computer manufactured by
`Apple Inc., or a special purpose or other general purpose
`computer may be used within the spirit and scope of the
`invention.
`
`In the first preferred embodiment, the input device
`[0041]
`14 is operative with a microphone 16 for receiving electrical
`signals over a microphone input line 32 representative of the
`sound waves from a musical instrument such as a recorder,
`clarinet, saxophone, violin or a trumpet (not shown), or from
`the voice tract of a human 17. The input device 14 also
`accepts electrical signals representative of the vibrations of
`the strings of a guitar 19 from a transducer 18 shown
`attached to the guitar 19 over a transducer input line 30.
`Preferably, the input device 14 is a conventional sound card
`available from numerous vendors and adapted to conven(cid:173)
`tional installation in the computer 12. Typically, the sound
`card provides an audio amplifier, bandpass filter and an
`analog-to-digital converter, each of a kind well known to
`those skilled in the art, for converting an analog electrical
`signal from the microphone 16 and the analog electrical
`signal from the transducer 18 into a digital audio signal
`compatible with the components of the programmed com(cid:173)
`puter 12. In the first preferred embodiment, the analog
`microphone 16 and transducer 18 signals are sampled at a
`rate of 44.1 KHz., each sample being represented by a 16 bit
`word. Preferably, the digital audio signal is stored in 1024
`word buffers in a portion of the storage device 22 in either
`a .WAY or a .AIFF format. As would be clear to those skilled
`in the art, the present invention is not limited to accepting the
`input signals from a transducer 18 or microphone 32 or to
`the aforementioned sample rate, buffer size or sample size
`and data format. Other devices for converting sound waves
`
`to electrical signals and other sample rates, sample sizes,
`buffer sizes and data formats could be used within the spirit
`and scope of the invention.
`
`[0042] The programmed computer 12 also includes the
`storage device 22. Desirably, the storage device 22 includes
`a random access memory (RAM), a read only memory
`(ROM), and a hard disk memory connected within the
`programmed computer 12 in an architecture well known to
`those skilled in the art. In addition to storing the computer
`program, the storage device 22 also stores the information
`representing the notes in the musical signal. The computer
`12 also includes a floppy disk drive and/or a CD-ROM drive
`for entering computer programs and other information into
`the programmed computer 12.
`
`[0043] Preferably, the output device 15 includes a digital
`output port 34 for providing a digital output signal conform(cid:173)
`ing to the Musical Interface Device Interface (MIDI) speci(cid:173)
`fication. Preferably, the MIDI output is provided to a device
`for transforming the MIDI signal into musical notation or
`alternatively, to a synthesizer, for reproducing the musical
`signal. The output device further includes a modem 28 for
`exchanging information with computers used by other musi(cid:173)
`cians, instructors etc. The connection of the modem 28 to
`other musicians may be via a point-to-point telephone line,
`a local area network, the Internet etc. The output device 15
`also includes a video display 24 where for instance, the notes
`played on a musical instrument or sung or stored in the
`storage device 22 could be displayed on, for instance, a
`musical staff; and a synthesizer/loudspeaker 26, for listening
`to the notes stored in the computer 12.
`
`In the first preferred embodiment the executable
`[0044]
`program code for determining the fundamental frequencies
`of the musical signal is stored in the ROM. However, as will
`be appreciated by those skilled in the art, the program code
`could be stored on any computer readable medium such as
`the hard disk, a floppy disk or a CD-ROM and still be within
`the spirit and scope of the invention. Further, the computer
`program could be implemented as a driver that is accessed
`by the operating system and application software; as part of
`an application; as part of a browser plug-in; or as part of the
`operating system.
`
`In the present invention, a track represents an
`[0045]
`ongoing frequency component of a note. The present inven(cid:173)
`tion provides a method for determining a fundamental
`frequency of a note in a musical signal wherein the method
`comprises receiving the musical signal, determining a fre(cid:173)
`quency of each of a plurality of tracks in the musical signal,
`generating a plurality of subharmonics for each track, and
`computing the fundamental frequency based on the fre(cid:173)
`quency of one or more subharmonics selected from the
`plurality of subharmonics, where the frequency of each of
`the selected subharmonics is within a neighborhood of the
`frequency of the track having the lowest frequency. The
`method also includes classifying each track to one and only
`one of a plurality of centroids where each centroid is
`characterized by a frequency, updating the frequency of each
`centroid based on an average of the frequency of each track
`classified to the respective centroid and a frequency of every
`subharmonic of each centroid whose frequency falls with a
`neighborhood of the frequency of the centroid, and merging
`the plurality of centroids such that at least one remaining
`centroid represents the frequency of the note.
`
`Verizon Wireless
`Exhibit 1 029-0027
`
`

`

`US 2001/0045153 Al
`
`Nov. 29, 2001
`
`4
`
`[0046] Referring now to FIG. 4 there is shown a method
`50 for determining the fundamental frequencies present in a
`polyphonic musical signal received by the apparatus 10
`according to the first preferred embodiment. The method 50
`comprises the steps of generating tracks 100, clustering the
`tracks 200, detecting inside notes 300, creating an event list
`400 and generating a M