United States
`(19)
`(12) Patent Application Publication (10) Pub. N0.2 US 201 1/0003638 A1
`Lee et al.
`(43) Pub. Date:
`Jan. 6, 2011
`
`US 20110003638A1
`
`(54) MUSIC INSTRUCTION SYSTEM
`
`Publication Classi?cation
`
`(75) Inventors:
`
`Harold Lee, Sherman Oaks, CA
`(Us); Roger B- Dannenberg,
`Pmsburgh, PA (Us)
`
`Correspondence Address:
`SNYDER, CLARK, LESCH & CHUNG, LLP
`950 Herndon Parkway, Suite 365
`HERNDON, VA 20170 (US)
`
`(73) Assigneez
`
`THE WAY OF H INC Sherman
`Oaks CA (Us) ’
`’
`’
`
`(21) Appl' N05
`
`12/775,665
`
`(22) Filed:
`
`May 7, 2010
`
`Related US Application Data
`
`(60) Provisional application No. 61/222,909, ?led on Jul. 2,
`2009.
`
`(51) Int. Cl.
`(200601)
`A63F 13/00
`(2006.01)
`A63F 9/24
`(52) us. Cl. ........................................... .. 463/35; 463/43
`
`ABSTRACT
`(57)
`A method includes receiving a user selection of a musical
`piece; providing performance cues to a user to perform musi
`cal events on a musical instrument, wherein the performance
`cues are synchronized to expert performance data of the musi
`cal piece; receiving audio data corresponding to musical
`events performed by the user on the musical instrument;
`detecting fundamental frequencies associated With the user
`perforrned musical events; determining an extent to Which the
`user-performed musical events have been correctly or incor
`rectly performed; providing real-time or near real -time audio
`feedback and/ or visual feedback indicating the extent to
`Which the user-performed musical events have been correctly
`or incorrectly performed; and using the expert performance
`data as real-time or near real-time audio and/or video feed
`back by controlling an output level of the expert performance
`data output to the user during a session.
`
`100 \
`
`USER
`105
`
`1 15
`
`/120&
`
`140-1
`
`Petitioners Ex. 1003 Page 1
`
`

`

`Patent Application Publication
`
`Jan. 6, 2011 Sheet 1 0f 29
`
`US 2011/0003638 A1
`
`100
`
`USER
`105
`
`Fig. 1
`
`Petitioners Ex. 1003 Page 2
`
`

`

`Patent Application Publication
`
`Jan. 6, 2011 Sheet 2 0f 29
`
`US 2011/0003638 A1
`
`COMMUNICATION
`INTERFACE
`220
`
`INPUT
`225
`
`PROCESSING
`SYSTEM
`205
`
`OUTPUT
`230
`
`MEMORY/STORAGE
`210
`
`APPLICATIONS
`215
`
`Fig. 2
`
`Petitioners Ex. 1003 Page 3
`
`

`

`Patent Application Publication
`
`Jan. 6, 2011 Sheet 3 0f 29
`
`US 2011/0003638 A1
`
`
`
`
`
`WAVEFORM MANAGER 305 ‘
`
`PERFORMANCE CUE ‘i
`MANAGER
`310
`
`MUSICAL PIECE DATA
`MANAGER
`315
`
`PERFORMANCE
`EVALUATION
`MANAGER
`320
`
`-
`
`I‘
`
`FEEDBACK MANAGER
`325
`
`SETTINGS MANAGER
`330
`
`Fig. 3
`
`Petitioners Ex. 1003 Page 4
`
`

`

`Patent Application Publication
`
`Jan. 6, 2011 Sheet 4 0f 29
`
`US 2011/0003638 A1
`
`EXPERT
`PERFORMANCE
`AUDlO TRACK
`
`ACCOMPANIMENT
`AUDIO TRACK
`
`PERFORMANCE CUE
`DATA
`
`405
`
`410
`
`415
`
`PERFORMANCE CUE
`DATA
`
`;
`5
`
`'"
`
`~'
`
`..
`
`‘I.
`
`'
`
`~
`
`'
`
`0
`420
`
`Petitioners Ex. 1003 Page 5
`
`

`

`Patent Application Publication
`
`Jan. 6, 2011 Sheet 5 0f 29
`
`US 2011/0003638 A1
`
`5051
`1
`;
`EXPERT swims-é»
`505-2
`5
`v-
`
`z
`
`2
`
`:
`
`v
`
`2
`
`:
`
`s
`
`1
`
`E
`
`=
`
`2
`
`
`
`YOU 515: t
`
`:" i
`
`:
`
`. 5 K?
`‘ _ _ :70372554:
`
`
`
`
`%/M 525
`COACH 5054
`505-5
`5
`1;
`5
`3
`;_
`THEORY C3
`ARTIST Dawkry
`0
`TITLE
`Over
`BAR
`52
`BEAT
`4
`SECTION Solo
`CURRENT G7
`NEXT
`C
`555
`
`"
`
`T 540
`
`
`
`_ i
`
`
`
`E Y
`
`
`
`I E
`
`F
`. E
`I 5
`
`
`: i
`: P
`
`
`SCORE 505-5
`555\
`SCORE LEVEL ACCURACY
`55,000 HARD
`75%
`STREAK MULTIPLIER
`7NOTES
`2X
`
`5056
`5 -
`5 0‘
`
`565
`
`NECK 5057
`
`1
`E
`J
`
`Y
`545
`
`WC <3) G <0 G (mm F
`lg. 5
`
`INPUT TUNE
`
`FX
`
`MIX
`
`BAND OUTPUT
`
`Petitioners Ex. 1003 Page 6
`
`

`

`Patent Application Publication
`
`Jan. 6, 2011 Sheet 6 0f 29
`
`US 2011/0003638 A1
`
`620
`
`625
`
`615
`
`610
`
`Petitioners Ex. 1003 Page 7
`
`

`

`Patent Application Publication
`
`Jan. 6, 2011 Sheet 7 0f 29
`
`US 2011/0003638 A1
`
`705
`
`740
`
`735 ~
`
`730 \
`
`725
`
`750
`
`Petitioners Ex. 1003 Page 8
`
`

`

`Patent Application Publication
`
`Jan. 6, 2011 Sheet 8 0f 29
`
`US 2011/0003638 A1
`
`.............. -‘
`
`760
`
`Petitioners Ex. 1003 Page 9
`
`

`

`Patent Application Publication
`
`Jan. 6, 2011 Sheet 9 0f 29
`
`US 2011/0003638 A1
`
`830
`
`825
`
`FIG. 8
`
`Petitioners Ex. 1003 Page 10
`
`

`

`Patent Application Publication
`
`Jan. 6, 2011 Sheet 10 0f 29
`
`US 2011/0003638 A1
`
`/
`905
`
`.
`
`=
`
`FIG. 9
`
`Petitioners Ex. 1003 Page 11
`
`

`

`Patent Application Publication
`
`Jan. 6, 2011 Sheet 11 0f 29
`
`US 2011/0003638 A1
`
`LEFT
`FOOT\
`1010
`
`LEFT
`HAND
`
`RIGHT
`HAND
`
`RlGHT
`FOOT
`
`FIG. 10
`
`Petitioners Ex. 1003 Page 12
`
`

`

`Patent Application Publication
`
`Jan. 6, 2011 Sheet 12 0f 29
`
`US 2011/0003638 A1
`
`in
`
`is
`
`‘go
`
`12
`
`s.»
`
`is
`
`‘
`
`1;:
`
`25¢
`
`a;
`
`‘ m
`
`:g:
`
`am
`
`we _
`
`1
`
`FIG. 11
`
`Petitioners Ex. 1003 Page 13
`
`

`

`Patent Application Publication
`
`Jan. 6, 2011 Sheet 13 0f 29
`
`US 2011/0003638 A1
`
`a
`
`a
`
`a
`
`a
`
`4/ 88
`
`Petitioners Ex. 1003 Page 14
`
`

`

`Patent Application Publication
`
`Jan. 6, 2011 Sheet 14 0f 29
`
`US 2011/0003638 A1
`
`
`
`
`
`ZOFOmE-MQ kmwZO EMOHEMQ
`
`owmv
`
`>
`
`>
`
`F
`
`
`
`.En. EMOHEmE
`
`mmwr
`
`F
`
`mmw .GE
`
`Petitioners Ex. 1003 Page 15
`
`

`

`Patent Application Publication
`
`US 2011/0003638 Al
`
` m5? 3 23m nmFzom>> $530.20
`
`a
`
`
`
`
`
`mm; 02
`
`02
`
`O
`
`5
`
`/ 88
`
`mm;
`
`ONF .UE
`
`Petitioners Ex. 1003 Page 16
`
`

`

`Patent Application Publication
`
`Jan. 6, 2011 Sheet 16 0f 29
`
`US 2011/0003638 A1
`
`CUE
`START
`
`CUE
`END
`
`E M 1.0
`_' 8
`§ m
`0 \
`O
`(D
`
`‘
`
`"4,0
`A 447 Hz
`
`5
`Z
`Lu LU LO
`3 Z :7:
`E3 _. r
`E ‘
`‘
`
`440 Hz
`
`33.875
`
`34.125
`
`H
`
`34.875
`
`35.125
`
`H
`
`34.0
`
`GRAPH1
`
`35.0
`
`TIMELINEV
`1300
`
`USER
`PERFORMANCE
`FREQUENCY
`13 0
`
`34.0
`
`GRAPH 2
`
`/_‘,_
`35.0
`
`\
`/
`
`442 Hz
`TIMELINE
`1300
`438 Hz
`
`"433 Hz
`
`FIG. 13
`
`Petitioners Ex. 1003 Page 17
`
`

`

`Patent Application Publication
`
`Jan. 6, 2011 Sheet 17 0f 29
`
`US 2011/0003638 A1
`
`FREQ
`
`FREQ
`
`FREQ
`
`Expert Gesture
`
`1405
`
`time
`
`Performed Gestu re /' 1410
`time
`
`Expert - Performed
`
`1420
`
`Petitioners Ex. 1003 Page 18
`
`

`

`Patent Application Publication
`
`Jan. 6, 2011 Sheet 18 0f 29
`
`US 2011/0003638 A1
`
`owmw a
`
`r
`
`
`
`m_. .GE
`
`Petitioners Ex. 1003 Page 19
`
`

`

`Patent Application Publication
`
`Jan. 6, 2011 Sheet 19 0f 29
`
`US 2011/0003638 A1
`
`INCREASE DIFFICULTY
`LEVEL OF SESSION
`1615
`
`USER
`
`‘I
`‘
`
`EVALUATE
`AND
`COMPARE
`1605
`
`DETERMINE
`EXTENT OF
`CORRECTNESS OR
`INCORRECTNESS
`
`4-71610
`
`110
`
`115
`
`\ PLAY
`EXPERT /
`PERFORMANCE
`AUDIO TRACK
`1620
`
`FIG. 16
`
`Petitioners Ex. 1003 Page 20
`
`

`

`Patent Application Publication
`
`Jan. 6, 2011 Sheet 20 0f 29
`
`US 2011/0003638 A1
`
`POSITIVE
`FEEDBACK
`
`115
`
`NEGATIVE
`FEEDBACK
`
`LIGHT—UP A DIM PERFORMANCE
`
`CUE
`1750
`
`PERFORMANCE CUE
`1705
`
`120
`
`145
`
`CHANGE COLOR
`1710
`
`MODIFY MUSICAL
`EVENT IMAGE
`1715
`
`DISPLAY POSITIVE
`WORDS
`1720
`
`
`
`INCREMENT USER’S
`SCORE
`1725
`
`CHANGE COLOR
`1755
`
`MODIFY MUSICAL
`EVENT IMAGE
`1760
`
`
`
`DISPLAY NEGATIVE
`WORDS
`1765
`
`DECREMENT USER‘S
`SCORE
`1770
`
`PLAY EXPERT TRACK
`1730
`
`MUTE EXPERT TRACK
`1775
`
`Petitioners Ex. 1003 Page 21
`
`Petitioners Ex. 1003 Page 21
`
`

`

`Patent Application Publication
`
`Jan. 6, 2011 Sheet 21 of 29
`
`US 2011/0003638 A1
`
`
`
`1805
`
`1810_>
`
`Your Score:
`
`128,924
`
`Correct Streaks:
`88 notes 36.29 seconds
`
`Error Streaks:
`
`8 notes 1.03 seconds
`
`1815—>
`
`late timing <—1825
`
`Accuracy:
`<— 1820
`78%
`
`Error Tendency:
`
`flat pitch
`
`Your Level:
`
`Novice
`Skilled
`Advanced
`Prodi o <— 1830
`;
`
`FIG. 18
`
`Petitioners Ex. 1003 Page 22
`
`Petitioners Ex. 1003 Page 22
`
`

`

`Patent Application Publication
`
`Jan. 6, 2011 Sheet 22 of 29
`
`US 2011/0003638 A1
`
`
`
`
`
`Your Score:
`
`128,924
`
`Congratulations!
`You scored over 100,000 points
`
`You have unlocked another song!
`Choose one:
`
`C)
`
`”Moonlight Sonata”
`for guitar
`Beethoven
`
`
`
`C] “Choro No. 1”Heitor Villa—Lobos C) ”Roundabout"
`
`Yes
`
`
`
`FIG. 19
`
`
`
`Petitioners Ex. 1003 Page 23
`
`Petitioners Ex. 1003 Page 23
`
`

`

`Patent Application Publication
`
`Jan. 6, 2011 Sheet 23 0f 29
`
`US 2011/0003638 A1
`
`Session’s absolute difficulty factor
`
`Current session difficulty factor
`
`# observed performance events
`
`# target performance events
`
`etc.
`
`harmonic content factor
`
`performance mechanics factor
`
`FIG. 20
`
`Petitioners Ex. 1003 Page 24
`
`Petitioners Ex. 1003 Page 24
`
`

`

`Patent Application Publication
`
`Jan. 6, 2011 Sheet 24 of 29
`
`US 2011/0003638 A1
`
`2105
`
`2110
`
`2115
`
`2120
`
`2125
`
`2130
`
`2135
`
`hours
`
`sessions sessions
`started complete
`
`high
`score
`
`session
`.
`.
`difficulty
`
`average accuracy
`score
`average
`
`last
`month
`
`720
`
`180,000
`
`7.00
`
`70,000
`
`0
`80 /0
`
`
`
`1,070
`
`12,840
`
`9,084
`
`220,000
`
`61,513
`
`75.79%
`
`FIG. 21
`
`Petitioners Ex. 1003 Page 25
`
`Petitioners Ex. 1003 Page 25
`
`

`

`Patent Application Publication
`
`Jan. 6, 2011 Sheet 25 of 29
`
`US 2011/0003638 A1
`
`,VVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVV
`mrage
`accuracy
`swarm 530:9
`
`COUNTRY
`
`
`
`535 m9
`averagem Mr”racy
`
`
`wage
`ROCK
`
`
`
`
`
`
`
`
`
`
`
`200,050
`
`7 M
`
`56.0%
`
`85%
`
`
`
`
`c LASs l CAL
`
`22.3.1156
`
`E ‘36
`
`55.800
`
`90%
`
`2210
`
`2205
`
`
`
`:asz ma
`
`inam?)
`
`41§W0k
`
`is best
`
`'13:»:
`
`
`
`11w:
`
`929
`
`5;“-
`
`2‘2
`
`':C
`
`1079
`
`sassims
`
`1‘. 520
`
`.J.“
`
`249
`
`12?
`
`12.550
`
`:33:
`
`3:034
`
`729
`
`‘92
`
`103
`
`3.085
`
`
`
`
`
`mm: score
`
`new)
`
`120960
`
`200.000
`
`220.090
`
`mm
`
`
`
`
`
`353;,
`
`GM
`
`7 W
`
`3.50
`
`8.93
`
`613
`
`
`
`average we ii:
`
`501139
`
`mm
`
`80,0{6
`
`853033
`
`61513
`
`75%
`
`3%
`
`85“:
`
`95-52
`
`75.7%
`
`
`
`
`
`
`FIG. 22
`
`
`
`
`
`
`
`2215
`
`Petitioners Ex. 1003 Page 26
`
`Petitioners Ex. 1003 Page 26
`
`

`

`Patent Application Publication
`
`Jan. 6, 2011 Sheet 26 of 29
`
`US 2011/0003638 A1
`
`~. 3'35
`
`Petitioners Ex. 1003 Page 27
`
`Petitioners Ex. 1003 Page 27
`
`

`

`Patent Application Publication
`
`Jan. 6, 2011 Sheet 27 of 29
`
`US 2011/0003638 A1
`
`
`
`Keyboard1'“ H! H nu; "I (mm m:
`
`2410
`
`
`
`
`
`
`
`(aft fact
`
`left hand
`
`right. Hand
`
`righ't foot
`
`FIG. 24
`
`Petitioners Ex. 1003 Page 28
`
`Petitioners Ex. 1003 Page 28
`
`

`

`Patent Application Publication
`
`Jan. 6, 2011 Sheet 28 of 29
`
`US 2011/0003638 A1
`
`
`
`
`
`wOZ_._.._.m_w«.mem_>_m_0m_w_
`
`momm
`
`
`
`ZO_mmm_w<m0“.mm<mmma
`
`owmm
`
`
`
`
`
`
`
`DZ<wMDOMOZ<SEOHEMQmo_>Omn_
`
`._.Zm_S=Z<&_>_OOU<
`
`mme
`
`ONmN
`
`mme<“—OMOZ<EmOmme<HOMFmD
`
`
`
`mZO_._.<3._<>m_moz<§m0mmma
`
`mNmN
`
`
`
`comm
`
`
`
`
`
`xo<m0mmmm>F<Omzm_n__>Omn_¥O<mommum>:._wO&mn=>Omn_
`
`
`
`
`
`
`
`owmmommN
`
`
`
`wmfimfiQEOOZO_mwm_wAwhmijOOZO_mmm_m
`
`
`
`
`
`
`
`
`
`O._.MDZFZOOO...MDZFZOO
`
`
`
`XOOJm.mmNGEx0015.m—mN.OE
`
`
`
`
`
`
`
`ommm<mN.G—H—comm
`
`Petitioners Ex. 1003 Page 29
`
`Petitioners Ex. 1003 Page 29
`
`
`
`
`
`
`
`
`

`

`Patent Application Publication
`
`Jan. 6, 2011 Sheet 29 of 29
`
`US 2011/0003638 A1
`
`
`
`
`
`mmOUwwhémzmw
`
`ommm
`
`
`
`MDZFZOO
`
`.<mN.OE_>_Omn_
`
`
`
`mwmmx0015
`
`mDZFZOO
`
`(mm.OE.201“.
`
`
`
`mmmmx0015
`
`oomN
`
`
`
`m._,m_._n=>_OoZO_mwmm
`
`mmmw
`
`mmN.0."—
`
`Petitioners Ex. 1003 Page 30
`
`Petitioners Ex. 1003 Page 30
`
`
`
`
`
`

`

`US 2011/0003638 A1
`
`Jan. 6, 2011
`
`MUSIC INSTRUCTION SYSTEM
`
`CROSS REFERENCE TO RELATED
`APPLICATION
`
`[0001] This application claims priority under 35 U.S.C.
`§1 19 based on US. provisional application No. 61/222,909,
`filed Jul. 2, 2009, the disclosure of which is hereby incorpo-
`rated by reference herein in its entirety.
`
`BACKGROUND
`
`Interactive games with musical themes have been
`[0002]
`introduced in which a user utilizes a game controller to simu-
`late a performance of a song. For example, these interactive
`games may display a scrolling time-line that indicates when
`to press a button on the game controller, sing into the game
`controller, or strike the game controller (e.g., a drum). While
`these interactive games may be entertaining, such interactive
`games do not teach the user to play a real musical instrument.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`FIG. 1 is a diagram illustrating an exemplary envi-
`[0003]
`ronment in which an embodiment of an exemplary music
`instruction system may be implemented;
`[0004]
`FIG. 2 is a diagram illustrating exemplary compo-
`nents of an exemplary music instruction system;
`[0005]
`FIG. 3 is a diagram illustrating exemplary func-
`tional components associated with an exemplary music
`instruction system;
`[0006]
`FIG. 4 is a diagram illustrating exemplary musical
`data that may be associated with musical pieces;
`[0007]
`FIG. 5 is a diagram illustrating an exemplary user
`interface of the music instruction system that includes exem-
`plary visuals for providing music instruction;
`[0008]
`FIGS. 6-7B are diagrams illustrating exemplary per-
`formance cues when the musical instrument corresponds to a
`guitar;
`FIG. 8 is a diagram illustrating exemplary perfor-
`[0009]
`mance cues when the musical instrument corresponds to a
`human voice;
`[0010]
`FIG. 9 is a diagram illustrating exemplary perfor-
`mance cues when the musical instrument corresponds to a
`keyboard;
`[0011]
`FIG. 10 is a diagram illustrating exemplary perfor-
`mance cues when the musical instrument corresponds to
`drum kit;
`FIG. 11 is a diagram illustrating an exemplary user
`[0012]
`interface of the music instruction system that includes syn-
`chronized performance cues to musical data;
`[0013]
`FIGS. 12A-12C are flow diagrams illustrating an
`exemplary process for detecting a fundamental frequency
`associated with a musical event;
`[0014]
`FIG. 13 is a diagram illustrating an exemplary pro-
`cess in which the music instruction system may evaluate a
`musical event and score the musical event;
`[0015]
`FIG. 14 is a diagram illustrating an exemplary pro-
`cess in which the music instruction system may evaluate a
`continuous musical gesture based on the fundamental fre-
`quency(s) associated with the continuous musical gesture;
`[0016]
`FIG. 15 is a diagram illustrating an exemplary pro-
`cess in which the music instruction system may calculate a
`score;
`
`FIG. 16 is a diagram illustrating an exemplary pro-
`[0017]
`cess in which the music instruction system may provide feed-
`back to a user based on the user’s performance;
`[0018]
`FIG. 17 is a diagram illustrating exemplary positive
`feedback and negative feedback that may be provided to a
`user;
`FIG. 18 is a diagram illustrating an exemplary user
`[0019]
`interface ofthe music instruction system that includes scoring
`information;
`[0020]
`FIG. 19 is a diagram illustrating an exemplary user
`interface of the music instruction system that allows a user to
`select a musical piece based on a user’s overall score;
`[0021]
`FIG. 20 is a diagram illustrating an exemplary pro-
`cess in which the music instruction system may calculate an
`overall score for a session;
`[0022]
`FIG. 21 is a diagram illustrating an exemplary user
`interface ofthe music instruction system that may summarize
`a user’s progress of music instruction over time;
`[0023]
`FIG. 22 is a diagram illustrating an exemplary pro-
`cess in which the music instruction system may track the skill
`of a user according to musical genres;
`[0024]
`FIG. 23 is a diagram illustrating an exemplary con-
`version chart that may be used by the music instruction sys-
`tem;
`FIG. 24 is a diagram illustrating an exemplary user
`[0025]
`interface of the music instruction system that includes exem-
`plary visuals for multiple users; and
`[0026]
`FIGS. 25A and 25B are diagrams illustrating an
`exemplary process in which the music instruction system may
`provide a session to a user.
`
`DETAILED DESCRIPTION OF PREFERRED
`EMBODIMENTS
`
`[0027] The following detailed description refers to the
`accompanying drawings. The same reference numbers in dif-
`ferent drawings may identify the same or similar elements.
`Also, the following detailed description does not limit the
`invention.
`
`[0028] The term “music instruction system,” as used
`herein,
`is intended to be broadly interpreted to include a
`device or a system capable of providing musical instruction
`for playing a musical instrument. By way of example, but not
`limited thereto, the music instruction system may correspond
`to a computer, a game system or game console, a communi-
`cation device, a client and a server arrangement via a network,
`a peer-to-peer arrangement, an application service provider
`(ASP) arrangement, network device residing in the Internet,
`etc. The music instruction system may be implemented in a
`distributed manner utilizing one or more devices or systems
`or in a centralized manner utilizing one or more devices or
`systems.
`[0029] The term “musical instrument,” as used herein, is
`intended to be broadly interpreted to include an instrument
`capable of producing sound. By way of example, but not
`limited thereto, a musical instrument may correspond to a
`wind instrument, a single reed instrument, a double reed
`instrument, a brass instrument, a string instrument, an elec-
`tronic instrument, a keyboard instrument, drum(s), a percus-
`sion instrument, or a human voice. The term “musical instru-
`ment,” is not intended to be interpreted to include a virtual
`musical instrument. By way of example, but not limited
`thereto, a virtual musical instrument may correspond to a
`game controller, such as a guitar controller utilized in Guitar
`Hero® or Rock Band® video games.
`
`Petitioners Ex. 1003 Page 31
`
`Petitioners Ex. 1003 Page 31
`
`

`

`US 2011/0003638 A1
`
`Jan. 6, 2011
`
`is
`[0030] The term “musical piece,” as used herein,
`intended to be broadly interpreted to include an assembly of
`musical events. By way of example, but not limited thereto, a
`musical piece may correspond to a song (an instrumental with
`or without lyrics), a musical composition (e.g., a sonata, a
`concerto, etc.), a vocal piece., a fingering exercise, a musical
`scale, a beat or a rhythm, chord fingerings, a harmonic pro-
`gression, or the like.
`is
`[0031] The term “musical event,” as used herein,
`intended to be broadly interpreted to include a sound produc-
`ing event. By way of example, but not limited thereto, a
`musical event may correspond to a note, a vocal utterance
`(e.g., speech, etc.), or a percussive sound.
`[0032] The term “musical data,” as used herein, is intended
`to be broadly interpreted to include data used by the music
`instruction system. By way of example, but not limited
`thereto, musical data may include audio data (e.g., accompa-
`niment audio tracks, expert performance audio tracks, audio
`performance cue data, audio feedback data, audio waveform
`data, etc.), visual data (e.g., expert performer video tracks,
`user video tracks, visual performance cue data, visual feed-
`back data, etc.), musical piece data (e.g., artist, chords,
`melody, music theory, etc.), as well as other types of data
`described in this description.
`[0033] According to exemplary embodiments described
`herein, a music instruction system may provide musical
`instruction to a user for playing a musical instrument. As
`previously described, according to an exemplary embodi-
`ment, the music instruction system may include a computer.
`By way of example, but not limited thereto, the computer may
`correspond to a desktop computer, a laptop computer, a hand-
`held computer, or the like. According to another exemplary
`embodiment, the music instruction system may include a
`game system. By way of example, but not limited thereto, the
`game system may correspond to a Microsoft® Xbox, a
`Sony® PlayStation (II, III, etc.), a Nintendo® Wii, or the like.
`According to yet another exemplary embodiment, the music
`instruction system may include a communication device. By
`way of example, but not limited thereto, the communication
`device may correspond to an Apple® iPhone, an Apple®
`iPod, a Motorola® Droid, or the like. In still other embodi-
`ments, the music instruction system may include a client/
`server architecture. By way of example, but not limited
`thereto, the client/server architecture may correspond to the
`computer, the game system, or the communication device
`communicatively coupled to a server or another type of net-
`work device on the Internet. The music instruction system
`may also include combinations of the above device-types, as
`well as other arrangements as described herein.
`[0034] According to an exemplary embodiment described
`herein, the music instruction system may provide a user with
`various visual and auditory instructional guides. For example,
`the music instruction system may include scrolling wave-
`forms to give a user a visualization of the sound associated
`with a musical piece. According to an exemplary implemen-
`tation, the scrolling waveforms may correspond to an expert’s
`performance of the musical piece and the user’s performance
`of the musical piece or performance cues. The music instruc-
`tion system may provide performance cues that are synchro-
`nized to musical data (e.g., pre-recorded tracks, etc.). The
`performance cues may be specific to the user’s musical instru-
`ment. Additionally, the music instructions system may dis-
`play a video image of the user to permit the user to see
`
`himself/herself while performing. The music instruction sys-
`tem may also provide the user with a score corresponding to
`the user’s performance.
`[0035]
`Further, according to an exemplary embodiment,
`the music instruction system may use one or multiple funda-
`mental frequency detection algorithms to detect frequencies
`associated with musical events (e.g., notes) played by the
`user. For example, the music instruction system may select
`one or multiple fundamental frequency detection algorithms
`depending on the musical instrument, the musical piece, a
`particular passage in the musical piece, etc. The music
`instruction system may provide to the user various types of
`feedback (e.g., visual feedback, auditory feedback, etc.) to
`the user. The music instruction system may provide a user
`with a particular type of feedback (e.g., positive feedback,
`negative feedback, etc.) based on, among other things,
`whether a musical event is played correctly or not, and/or how
`correctly or incorrectly the musical event is played (i.e., an
`extent with which a musical event is correctly/incorrectly
`played in terms of pitch, amplitude, onset time, etc.).
`[0036]
`FIG. 1 is a diagram illustrating an exemplary envi-
`ronment 100 in which an embodiment of an exemplary music
`instruction system may be implemented. As illustrated in
`FIG. 1, environment 100 may include a user 105, a musical
`instrument 110, and a music instruction system 115.
`[0037] The number of devices and configuration in envi-
`ronment 100 is exemplary and provided for simplicity. In
`practice, environment 100 may include more devices, fewer
`devices, different devices, and/or differently arranged devices
`than those illustrated in FIG. 1. Also, some functions
`described as being performed by a particular device may be
`performed by a different device or a combination of devices.
`Additionally, or alternatively, in other implementations, a
`device may include a combination of devices.
`[0038] User 105 may be a person that performs with musi-
`cal instrument 110. In this example, musical instrument 110
`may correspond to a string instrument, such as a guitar. Fur-
`ther, in this example, music instruction system 115 may cor-
`respond to a computer system. For example, as illustrated in
`FIG. 1, the computer system may include a display 120, a
`computer 125, a keyboard 130, a mouse 135, speakers 140-1
`and 140-2 (generally referred to as speakers 140), and a
`camera 145.
`
`[0039] Musical instrument 110 may be connected to music
`instruction system 115 to provide music instruction system
`115, among other things, its audio output. By way ofexample,
`but not limited thereto, musical instrument 110 may be con-
`nected to music instruction system 115 via a cable, a micro-
`phone, or a pickup, depending on the type of musical instru-
`ment 110. In this example, musical instrument 110 (i.e., the
`guitar) may be connected to an input of music instruction
`system 115 via a cable (not illustrated).
`[0040] Display 120 may include a device capable of pro-
`viding visual output. For example, display 120 may corre-
`spond to a computer monitor or a television. According to an
`exemplary implementation, display 120 may include a touch
`screen. According to other exemplary implementations, dis-
`play 120 may not include a touch screen and/or may not be
`utilized as an input device. Computer 125 may include a
`computational device. According to an exemplary implemen-
`tation, computer 125 may operate according to an operating
`system (e.g., Windows, Macintosh, Linux, etc.). In this
`example, computer 125 may correspond to a desktop com-
`puter. Keyboard 130 and mouse 135 may include devices that
`
`Petitioners Ex. 1003 Page 32
`
`Petitioners Ex. 1003 Page 32
`
`

`

`US 2011/0003638 A1
`
`Jan. 6, 2011
`
`permit user 105 to input information into and navigate on
`computer 125. Speakers 140 may include a device that pro-
`vides auditory output. According to other implementations,
`for example, speakers 140 may be associated with an audio
`system (e.g., a stereo system, etc.) or incorporated within
`display 120. Camera 145 may include a device capable of
`capturing Visual images, such as pictures and/or Video. In this
`example, camera 145 may correspond to a web cam. How-
`ever, according to other implementations, for example, cam-
`era 145 may correspond to a camcorder or some other type of
`video camera. The visual images may be displayed on display
`120. Music instruction system 115 may send and/or receive
`visual images to/from other persons (e.g., other users, a
`remote instructor, etc.) via a network (e.g., the Internet, etc.).
`[0041] According to an exemplary operation, music
`instruction system 115 may provide user 105 musical instruc-
`tion for playing musical instrument 110. As will be described
`herein, music instruction system 115 may provide user 105
`with various user interfaces to assist user 105 in his/her per-
`formance of a musical piece. Music instruction system 115
`may detect musical events performed by user 105 based on
`one or more frequency detection algorithms, which are
`described further below.
`
`FIG. 2 is a diagram illustrating exemplary compo-
`[0042]
`nents of music instruction system 115. As illustrated, accord-
`ing to an exemplary implementation of music instruction
`system 115, music instruction system 115 may include a
`processing system 205, memory/storage 210 including appli-
`cations 215, a communication interface 220, an input 225,
`and an output 230. According to other implementations,
`music instruction system 115 may include fewer components,
`additional components, different components, and/or a dif-
`ferent arrangement of components than those illustrated in
`FIG. 2 and described herein.
`
`Processing system 205 may include a processor, a
`[0043]
`microprocessor, a co-processor, an application specific inte-
`grated circuit (ASIC), a controller, a programmable logic
`device, a chipset, a field programmable gate array (FPGA), or
`some other component that may interpret and/or execute
`instructions and/or data. Processing system 205 may control
`the overall operation, or a portion thereof, of music instruc-
`tion system 115, based on, for example, an operating system
`and one or more applications (e.g., applications 215).
`[0044] Memory/storage 210 may include a memory and/or
`a secondary storage. For example, memory/storage 210 may
`include a random access memory (RAM), a dynamic random
`access memory (DRAM), a read only memory (ROM), a
`programmable read only memory (PROM), a flash memory,
`and/or some other type ofmemory. Memory/storage 210 may
`include a hard disk (e.g., a magnetic disk, an optical disk, a
`magneto-optic disk, a solid state disk, etc.) or some other type
`of computer-readable medium, along with a corresponding
`drive. Memory/storage 210 may include a memory, a storage
`device, or storage component that is external to and/or remov-
`able from music instruction system 115, such as, for example,
`a Universal Serial Bus (USB) memory stick, a hard disk, mass
`storage, off-line storage, cloud computing, etc.
`[0045] The term “computer-readable medium,” as used
`herein, is intended to be broadly interpreted to include, for
`example, a memory, a storage device (e.g., a hard disk and
`corresponding drive), a compact disc (CD), a digital versatile
`disc (DVD), or the like. The computer-readable medium may
`be implemented in a single device, in multiple devices, in a
`centralized manner, or in a distributed manner. Memory/stor-
`
`age 210 may store data, applications 215, and/or instructions
`related to the operation of music instruction system 115.
`[0046] Applications 215 may include software that pro-
`vides various services, functions, user interfaces, or the like.
`According to an exemplary implementation, applications 215
`may include a music instruction application that provides one
`or more of the processes related to instructing a user to play a
`musical
`instrument. For example,
`theses processes may
`include providing user interfaces, detecting fundamental fre-
`quencies, scoring, providing feedback to the user, and/or
`other functions associated with music instruction system 115,
`as described herein. Applications 215 may be stored in
`memory/storage 210.
`[0047] Communication interface 220 may permit music
`instruction system 115 to communicate with other devices,
`networks, systems and/or the like. Communication interface
`220 may include a wireless interface and/or a wired interface.
`Communication interface 220 may operate according to one
`or more protocols, standards, and/or the like. Communication
`interface 220 may include a receiver, a transmitter, and/or a
`transceiver.
`
`Input 225 may permit an input into music instru-
`[0048]
`ment system 115. For example, input 225 may include a
`button, a keypad, a knob, a touchpad, keyboard 130, an input
`port, display 120, a microphone, mouse 135, voice recogni-
`tion logic, fingerprint recognition logic, a web cam (e.g.,
`camera 145), and/or some other type of input component.
`[0049] Output 230 may permit music instrument system
`115 to provide an output. For example, output 230 may
`include display 120, light emitting diodes (LEDs), an output
`port, speakers 140, a vibratory mechanism, and/or some type
`of output component.
`[0050] As described herein, according to an exemplary
`embodiment, music instrument system 115 may perform one
`or more processes in response to processing system 205
`executing software instructions contained in a computer-
`readable medium, such as memory/storage 210. The software
`instructions may be read from memory/storage 210 or
`received from another device via communication interface
`
`220. The software instructions may cause processing system
`205 to perform processes described herein. Alternatively,
`music instruction system 115 may perform processes in
`response to processing system 205, or other combinations
`with processing system 205 (e.g., hardware and firmware,
`hardware, software and firmware).
`[0051]
`FIG. 3 is a diagram illustrating exemplary func-
`tional components associated with music instruction system
`115. As illustrated, music instruction system 115 may include
`a waveform manager 305, a performance cue manager 310, a
`musical piece data manager 315, a performance evaluation
`manager 320, a feedback manager 325, and a settings man-
`ager 330. Waveform manager 305, performance cue manager
`310, musical piece data manager 315, performance evalua-
`tion manager 320, feedback manager 325, and/or settings
`manager 330 may be implemented as a combination of hard-
`ware (e.g., processing system 205, memory/storage 210) and
`software (e.g., applications 215) based on the components
`illustrated and described with respect to FIG. 2.Altematively,
`waveform manager 305, performance cue manager 310,
`musical piece data manager 315, performance evaluation
`manager 320, feedback manager 325, and/or settings man-
`ager 330 may be implemented as hardware and firmware or
`hardware, software, and firmware.
`
`Petitioners Ex. 1003 Page 33
`
`Petitioners Ex. 1003 Page 33
`
`

`

`US 2011/0003638 A1
`
`Jan. 6, 2011
`
`[0052] Waveform manager 305 may manage audio wave-
`form data. According to an exemplary implementation, wave-
`form manager 305 may display a waveform corresponding to
`an expert performance of the musical piece. As described
`further below, music instruction system 115 may use musical
`data, which includes pre-recorded audio tracks, during a ses-
`sion. The pre-recorded audio tracks may include, among
`other things, audio tracks corresponding to expert perfor-
`mances ofmusical pieces. Waveform manager 3 05 may select
`an expert performance audio track from the musical data
`according to the musical piece and/or the musical instrument
`to be performed during the session. Waveform manager 305
`may display a waveform representative of the expert perfor-
`mance audio track. According to an exemplary implementa-
`tion, the waveform may scroll and include other indicators
`(e. g., a cue line to indicate a current time position ofthe expert
`performance, etc.), similarity scores, etc., as will be described
`further below.
`
`[0053] Additionally, or alternatively, according to an exem-
`plary implementation, waveform manager 305 may display a
`waveform representative of a user’s performance ofthe musi-
`cal piece. For example, waveform manager 305 may process
`an input from the user’s musical instrument to generate and
`display a scrolling waveform representative of the user’s per-
`formance of th