SERIAL NUMBER
`
`0 8'/ 0 55! 1
`
`5 0
`
`F'ILING 0',
`07/
`93
`
`bLASS
`
`-36.4
`
`SUBCLASS
`
`GROUP ART UNIT
`
`00 L a Lirw2l
`EXAMINFA
`
`-AS
`01
`-3,
`
`BOLA-X,l PALI-i 0, L'T f,"
`LCF'j ANf3iEJ ES CPj.
`
`CA,- IAN E.
`
`MC1.)OVIALL, PALO ALTO CA-, MICHAEL U.
`
`5513129
`
`4, 4, CEIN-g 1 NLJI 1-16
`VERIFIED
`
`T,4A-F A* 4: -f'. *4'
`
`1!
`
`4:
`
`1--
`
`GF.,'A1,4TED
`
`,0/1i8/C13'
`
`**FORE IGN/PCT
`VER T F I ETi
`
`F 10
`
`1 IG!
`
`1. 1 11 q G
`
`Foreign PricritY clairned
`35 USC 119 candittonsmet
`
`yes
`
`13 &,40
`
`Verified and Acknowledgeci
`Examiner-$ jnjt[,&rs
`ALFRED - A 77=41
`
`no
`
`AS
`
`4_ FIL
`
`STATE OR SHEETI
`COUNTRY ORW13S
`
`R-MALL E-1,41-1-ITY
`
`2-3
`
`4S IRECEIVZD
`
`$601.06 FAKE00200
`
`DMKWr NO.
`
`FERRY EA-DG.
`FRA!1K:T'- iCl
`- I -D
`
`CA '44 I I I
`
`I
`
`METHOD AND -:YSTEM FOR CON k
`! 0-ITN -
`I a fompu
`ENVIRONMENT IN RESPONSE TO AUDIO SIW-4A6
`YXr-1-GENERATED VIRTUAL
`
`Il!"
`
`U.%CW.atCOMMA%t.&TMOtOoPW43OL(fMIO-7811
`
`PARTS OF APPLICATION
`FILED SEPARATELY
`
`NOTICE OF ALLOWANCE MAILED
`
`ISSUE FEE
`
`Amount Due
`
`I I
`
`Date Paid
`
`v, _, o,o
`
`I ) , ; e"-q
`
`I
`
`Label
`Area
`
`aim PTO-436A
`lev. 8/92) gr
`
`Assistant Examiner
`
`CLAIMS
`Toial Claims
`
`Claim
`
`,,
`
`ELLIS a.
`PRIMARY E
`GRoup 2460
`
`MVIR
`ER
`
`Pdmary Ex
`PREPARED FOR ISSUE,-
`
`Sheeis Drwg. I Figs.
`
`I 1-T
`
`V
`
`A CH
`TCH
`iner NUMBER
`
`3q
`
`WARNING: The information
`
`discia
`by the Uniled Stat;wl
`Patent & Trademark
`
`hereih may be restricted.
`CICKitle 35, Sections 122, 181 and 368.
`Unauthorized disclosure may be prohibited
`he
`de Ti
`Office is restricted to authodzed
`Possession outside the U.S.
`employees and contractors
`only.
`
`(FACE)
`
`Page 1 of 140
`
`PETITIONERS' EXHIBIT 1009
`
`

`
`p :J
`
`Date
`Entered
`or
`Counted
`
`CONTENTS
`
`1. Application
`
`papers.
`
`26AII'Ttkf
`
`f
`
`&' a
`
`R
`
`17
`
`/J' eAJ
`
`Sillllf
`
`z,
`
`8'.
`
`6.
`
`17.
`
`26.
`
`31.
`
`32.
`
`A,PP.ROVED FOR LIMSE
`
`INITIALS4,
`
`;4
`
`Date
`Received
`or
`Mailed
`
`RECE I P D
`
`OCT, 2 7 1993
`
`3,00
`
`9-1.3- 9,3
`
`/-0 -3
`
`I
`
`(FRONT)
`
`Page 2 of 140
`
`PETITIONERS' EXHIBIT 1009
`
`

`
`Staple Issue Slip Here
`
`Posmou
`
`
`
`
`
`-—
`
`I _
`
`-31
`
`j—
`
`INDEX OF CLAIMS
`
`C
`
`
`
`9
`
`‘I
`
`O2H
`
`
`
`
`
`- 7
`------H-—
`
`-H---KHZ
`------—
`I
`
`-E! ----HH—
`-El-------—
`
`-IE-H---—HZ
`
`-El-----HH—
`
`-El-----—-— «
`
`-IE-----HH—
`-IE-------—
`
`------H
`
`
`
`
`
`
`
`
`IDon
`
`‘\
`
`‘/5’
`.1
`E]EIE———--
`7
`El - .EE“:]-
`':'
`HSIE'< HE“IIIIIII
`
`i?I!~IIIII!IIIIIIIIII
`
`t/‘
`
`EaHIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII
`
`I V V
`E--
`flmmunann
`H33.-
`.a.
`|/- \/
`
`flEE——
`[51Ifl———
`ma /I..-
`-E:lE——-——
`
`
`
`
`
`ILez:»1-I-I----
`
`
`—EE———————Z
`-E:3I———————]
`4» -EI3I_——————Z
`-E5————_——-
`-EI!———————C
`
`I?3]E2lEEE————
`
`SYMBOLS
`./ ................................. Rejected
`........ Allowed
`ral) canceled
`. Restrict
`. Non-el
`. lmerfamnco
`. Appeal
`................................. Oblocted
`
`
`
`(TN
`
`+N A0
`
`
`
`
`
`
`
`\
`
`
`
`
`
`------_—]
`-lE————H——C
`
`A\l
`
`
`
`Page 3 of 140
`
`(LEFT INSIDE)
`
`PETIT|,ONERS' EXHIBIT 10 I
`
`
`
`
`
`Page 3 of 140
`
`PETITIONERS' EXHIBIT 1009
`
`

`
`SEARCHED
`
`Class
`
`Sub.
`
`Date
`
`Exmr.
`
`i I i)-15-As- 6z
`
`SEARCH NOTES
`
`Date
`
`I
`
`VO -712-
`
`6gqi 635
`
`A dl
`
`/57
`15 J///
`
`I;v-z> I
`
`INTERFERENCE SEARCHED_
`
`Class
`
`Sub.
`
`Date
`
`.
`
`,5(a
`9
`
`Exmr.
`
`-7
`
`(RIGHT OUTSIDE)
`
`Page 4 of 140
`
`PETITIONERS' EXHIBIT 1009
`
`

`
`PATENT APPLICATION SERIAL NO.
`
`OV391650
`
`U.S. DEPARTMENT OF COMMERCE
`PATENT AND TRADEMARK OFFICE
`FEE RECORD SHEET
`
`060 TL-0-8/02/93, 08091650
`
`1, 201
`
`536L.(cid:173)00 CK FA-K.E-00200
`
`PTO-1556
`(5/87)
`
`00
`
`Page 5 of 140
`
`PETITIONERS' EXHIBIT 1009
`
`

`
`m
`
`BAR CODE LABEL
`
`U.S. PATENT APPLICATION
`
`SERIAL NUMBER
`
`FILING DATE
`
`CLASS
`
`GROUP ART UNIT
`
`08/091,650
`
`07/14/93
`
`364
`
`2304
`
`z
`
`(L
`(L
`
`MARK BOLAS, PALO ALTO, CA; IAN E. MCDOWALL, PALO ALTO, CA; MICHAEL N.
`BOLAS, LOS ANGELES, CA.
`
`"CONTINUING DATA*********************
`VERIFIED
`
`**FOREIGN/PCT APPLICATIONS************
`VERIFIED
`
`FOREIGN FILING LICENSE GRANTED 10/18/93
`
`SMALL ENTITY
`
`STATE OR
`COUNTRY
`
`SHEETS
`DRAWING
`
`TOTAL
`CLAIMS
`
`INDEPENDENT
`CLAIMS
`
`FILING FEE
`RECEIVED
`
`ATTORNEY DOCKET No.
`
`CA
`
`7
`
`23
`
`7
`
`$601.00
`
`FAKE00200
`
`cn
`(n
`III
`
`LU
`
`ALFRED A. EQUITZ
`LIMBACH & LIMBACH
`2001 FERRY BLDG.
`SAN FRANCISCO, CA 94111
`
`METHOD AND SYSTEM FOR CONTROLLING C014PUTERGENERATED VIRTUAL
`ENVIRONMENT IN RESPONSE TO AUDIO SIGNALS
`
`This is to certify that annexed hereto is a true cou fr9m the records of the United States
`ch is identified above.
`Patent and Trademark Office of the application
`By authority of the
`COMMISSIONER OF PATENTS AND TRADEMARKS
`
`Date
`
`Certifying Officer
`
`Page 6 of 140
`
`PETITIONERS' EXHIBIT 1009
`
`

`
`L Rai,
`
`5 0
`
`JUL
`
`og
`
`1993
`
`Honor
`
`er
`s
`Washiiigton, D.C. 20231
`
`a n
`
`4IMBACH & LIXBACH
`2001 Perry Bull-laing, LqAn-ZrA=iSC0j CA 94111
`15) 433-4150
`
`-
`
`of Patents
`
`Attorney's Docket No.
`
`PAKE-00200
`
`SIR:
`
`(a
`Transmitted Lewith for filing is the patent application of Inventor(s)
`Title:
`
`NEW APPLICATION TRANSMITTAL
`
`CERTIFICATION UNDER 37 CFR 51.10
`I hereby certify that this New Application and the documents referred to as enclosed herein are being
`deposited with the United States Postal Service on this date JULY 14 , 1993, in an envelope bearing "Express
`Mail Post Office To Addressee" Mailing Label Number TB388605756US addressed to: Patent Application,
`Honorable Commissioner of Patents and Trademarks, Washington, D.C.
`20231.
`L=--) 2--
`LANA T BRENNER
`- Signatlire
`(Name of person mailing paper)
`
`Enclmsed are:
`The papers required for filing date under CFR 51.53(b):
`_2_Pages of specification (including 4-page "Appendix A" and claims);
`
`7 Sheets of drawings.
`formal
`informal
`
`x
`
`X Declaration or oath (Signed by one inventor only)
`Power of Attorney
`Assignment of the invention to
`Fee Calculation
`Amendment changing number of claims or deleting multiple dependencies is enclosed.
`
`CLAIMS AS FILED
`
`*If less than zero, enter "0".
`
`x
`
`Filing Fee Calculation .
`Small Entity Statement - verified statement enclosed (tmsigned)
`50X Filing Fee Reduction (if applicable).
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`$1,072.00
`
`$
`
`536.00
`
`Other Fees
`Recording Assigrunent ($40.001
`Other fees
`Specify
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`. $,
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`Total Fees Enclosed
`
`.
`
`.
`
`.
`
`.
`
`. $536.00
`
`x
`
`15ayment of Fees
`x Check(s) in the amount of
`enclosed.
`536.00
`Charge Account No. 12-1420 in the amount of $(cid:173) A duplicate of this transmittal is attached.
`No fee accompanies this application per 37 CFR S 1.53(b) and (d).
`Authorization to Charge Additional Fees
`The Commissioner is hereby authorized to charge any additional fees (or credit any overpayment) associated with
`§1.17 to Account No. 12-1420, except wheme
`thi: communication and which may be required under 37 CFR 51.16 or
`eat is attached.
`thi
`application is filed under 37 CFR §1.53. A duplicate
`Information Disclosure Statement
`x Return Receipt Postcard
`x REFERENCE TO DISCLOSURE DOCUMENTS:
`The present application is related to the tecbnology described in the following Disclosure DocuDents: Document No.
`286735, received (emd date-stamped) July 15, 1991 by the USPTO; amd Document No. 286752, received (and date-
`stamped) July 1-%, 1991 by the USPTO. The USPTO is requested to retain both these Disclosure Documents in its files.
`
`Dated
`
`By:_
`
`Name:
`Alfred A. Ecruitz
`Registration No.
`30,922
`
`Page 7 of 140
`
`PETITIONERS' EXHIBIT 1009
`
`

`
`x_LRnt
`
`'JUL
`14
`1993
`
`0
`
`PATENT
`
`-1-
`
`5
`
`10
`
`15
`
`20
`
`25
`
`30
`
`VIRTUAL ENVIRONMENT IN RESPONSE TO AUDIO SIGNALS
`
`Field of the Invention
`The invention pertains to methods and apparatus
`for controlling a computer system in response to
`music signals, or in response to prerecorded control
`tracks corresponding to audio signals (such as music
`signals).
`In preferred embodiments, invention
`pertains to methods and apparatus for creating and
`modifying, or otherwise controlling, computer-
`generated virtual environments (or displayed virtual
`objects in virtual environments) in response to music
`signals or in response to prerecorded control tracks
`corresponding to audio signals.
`
`-
`
`Background of the Invention
`The terms "virtual environment," "virtual
`world," and "virtual reality" are used
`interchangeably to describe a computer---samu-1ated
`environment (intended to be immersive) which includes
`a graphic display (from a user's first person
`perspective, in a form intended to be immersive to
`the user), and
`opti-onally also sounds which simulate
`environmental sounds. The abbreviation IIVRII will
`sometimes be used herein to denote "virtual reality,"
`"virtual environment," or "virtual world."
`A
`computer system programmed with software, and
`including peripheral devices, for producing a virtual
`environment will sometimes be referred to herein as a
`VR system or VR processor.
`The graphic display generated by a VR system can
`be a two-dimensional (2D) or a three-dimensional (3D)
`display. Typically, a VR system includes an input
`device and user interface software which enable a
`user to interact with the scene being displayed,
`
`Page 8 of 140
`
`PETITIONERS' EXHIBIT 1009
`
`

`
`0
`
`PATENT
`
`-2-
`
`typically to simulate motion in the virtual
`environment or manipulation of displayed
`represen-tations of objects ("virtual objects") in the
`virtual environment. Typically, the illusion of
`immersion in a virtual reality system is strengthened
`by the use of head-tracking or some other such system
`which directs the computer to generate images along
`the area of viewing interest of the user.
`The present invention is a method and apparatus
`particularly useful for creating and/or controlling
`virtual environments. A VR system which embodies the
`invention can rapidly and inexpensively create,
`animate, or otherwise control a wide variety of
`entertaining virtual environments and virtual objects
`in response to music or in response to prerecorded
`"control tracks" which correspond to audio signals
`(such as music).
`While currently being used in the research and
`scientific communities, VR systems are becoming less
`expensive and are poised to reach the consumer
`electronics market as entertainment devices.
`VR systems must generate a much greater amount
`of content data (image data and audio data simulating
`environmental appearance and sounds) than must be
`Whereas
`generated in most other olectronic media.
`video game systems require complex scenes to be
`generated and themes to be programmed, such systems
`can easily limit the scope of the game content
`because they can easily constrain the player to move
`in a few simple directions (e.g., left and right) and
`need only produce images to be presented on flat
`screen.monitors or on simple 3D field-sequential type
`monitors.
`In contrast, by their very nature, VR systems
`allow the user to look around and fly around in many
`
`5
`
`10
`
`15
`
`20
`
`25
`
`CM
`
`35
`
`Page 9 of 140
`
`PETITIONERS' EXHIBIT 1009
`
`

`
`a
`
`5
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`m
`
`-3-
`
`PATENT
`
`different directions and positions. Even where the
`user is constrained to look only toward the left or
`the right, VR systems must construct complete
`This complexity has
`representations of 3D worlds.
`made it very difficult to generate.virtual worlds for
`the consumer entertainment market in a quick fashion.
`In addition to the complexity of creating static
`3D models for virtual worlds, it has also been
`difficult to control the dynamics of virtual worlds.
`VR systems to date are notorious for providing only
`The few
`very boring and nearly static environments.
`VR systems that include dynamic motions of the
`virtual world either base such motions on physical
`laws (such as gravity) or base the motions on
`corresponding motions produced by human users (such
`as motion of the fingers of a user wearing a
`conventional "glove" input device).
`The present invention overcomes the limitations
`of conventional VR systems by providing an efficient
`way to generate content data (i.e., animated image
`data and audio data) to'fill or populate a virtual
`environment in a choreographed response to input
`music signals.
`There has long been an interest in the virtual
`reality field with respect to the possibility of
`virtual musical instruments and the creation of new
`The
`and novel instruments within a virtual world.
`present invention is a radical shift from previous
`attempts to combine music and virtual environments.
`Conventional efforts to integrate music with
`virtual environments have, to date, all been directed
`toward-creation of music from a virtual environment.
`The musical expression of the user has been tteated
`as an urge seeking to be brought forth, and virtual
`environments have been seen as vehicles for the user
`
`Page 10 of 140
`
`PETITIONERS' EXHIBIT 1009
`
`

`
`-4-
`
`PATENT
`
`to perform music or dance without having to learn
`special physical skills.
`Much effort has been made
`to make sounds appear to be coming from virtual
`This has been
`objects in the virtual environment.
`done by running audio into the VR system and then
`convolving the audio in such a way as to make it
`appear to come from a certain place in the virtual
`environment.
`
`For example, at the NASA Ames View Lab, Scott
`Fisher, Rick Jacoby, and others explored virtual
`one aspect of the research was the
`environments.
`integration of audio into the virtual experience.
`This included the use of audio cues for such purposes
`as telling one if one bumped into a virtual object,
`but there was no tactile feedback for such events.
`The research pushed into the more artistic realm of
`creation of music in the context of a virtual world.
`Mark Bolas and Phil Stone created the Virtual
`In this system, the
`Theremin and virtual drum kit.
`user wore a glove and a hand tracker and moved the
`gloved hand to manipulate virtual objects which were
`in turn linked to various synthesizer parameters.
`,Thus, by manipulating virtual objects (as taught, for
`example, by U.S. Patent 4,988,981, issued January 29,
`1991), sounds of different qualities could be
`A skilled user could create modern sounding
`created.
`These ideas have been carried
`musical interludes.
`forth by people such as Jaron Lanier who has given a
`number of public performances in which he manipulates
`virtual objects to create a musical performance.
`Research and exploration along these lines is
`expected to continue (the virtual "air guitar" and
`In all VR
`the like will probably be developed).
`systems of this type, manipulation of a virtual
`object causes the sound or music to change.
`
`5
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`I
`
`Page 11 of 140
`
`PETITIONERS' EXHIBIT 1009
`
`

`
`11
`
`-5-
`
`PATENT
`
`Currently, virtual worlds are created by
`describing a simulation and a number of objects.
`The
`interaction of the objects is described in some form
`of simulation la'nguage or graphical description.
`Traditionally, the control and creation of the
`objects is driven by "world building" software.
`Once
`a virtual world has been created, a limited number of
`its parameters may be manipulated by the user from
`"inside" the virtual world.
`One example of how these
`databases are created is described in PCT
`International Patent Application WO 92/09948, by VPL
`As is evident from WO 92/09948, it has
`Research Inc.
`been very difficult in VR systems to create and
`define animation for all or even some of the virtual
`objects in a virtual world.
`Until the present
`
`inventio-,,.-it had not.,.been
`interface.-to
`'n
`nodes in a database defining a virtual environment,
`and to manipulate such nodes, on the basis of music.
`Conventional VR systems and music have thusfar
`been uged together in ways which have the following
`disadvantages:
`a VR system has been used as a virtual
`musical instrument, so that the user must "play" the
`virtual instrument (by manipulating an input device)
`This means that the system creates
`to hear anything.
`music, and that the system's musical output is
`limited by the user's ability to "play" the
`"instrument;"
`VR systems that have given sounds to
`virtual objects (e.g., the system displays a virtual
`kitchen sink and produces a "drip.-drip" sound which
`seems to come from the sink's location) have required
`that the sounds are generated by signals produced
`within the VR system in response to user manipulation
`of,an input device or internal programs, which
`
`5
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`Page 12 of 140
`
`PETITIONERS' EXHIBIT 1009
`
`

`
`m
`
`-6-
`
`PATENT
`
`The
`signals are then interpreted by a synthesizer.
`sounds produced by the synthesizer are thus cued from
`the VR system in response to manipulation of an input
`device (which manipulation may, for example, to cause
`a user to "move" into a position to view or otherwise
`interact with a virtual kitchen sink from which
`Thus, these VR
`sounds will then seem to emanate).
`systems have depended on user manipulation of an
`input device to control the appearance or activities
`of objects in a virtual environment, to cause the VR
`system to cue production of sound events; and
`(c) VR systems have played musical scores as
`background music for the virtual environment.
`Basically, the paradigm to date has been to
`create systems that have (virtual) object-driven
`sounds. This invention reverses the paradigm to
`create a system which has musically-driven objects.
`One VR system has been developed in which a VR
`processor is programmed to perform simple operations
`to modify a virtual environment in response to voice
`commands. This VR system, developed at the NASA Ames
`View Lab during the years 1988-1989, was capable of
`displaying a virtual object, or terminating the
`display of a virtual object, in response to a voice
`However, the system did
`command from a human user.
`not produce, modify, or otherwise control a virtual
`environment in response to music, or in response to a
`prerecorded control track corresponding to an audio
`signal.
`Outside the VR field, many attempts have been
`made to produce devices which provide users with
`visual light effects based on an audio signal, such
`as music. However, these systems have been
`disappointing to watch (principally because the light
`shbws are two-dimensional and are not obviously
`
`5
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`1)
`
`Page 13 of 140
`
`PETITIONERS' EXHIBIT 1009
`
`

`
`11
`
`-7-
`
`PATENT
`
`correlated with the audio input), and have typically
`met with disappointment when marketed.
`An example of a conventional apparatus for
`producing visual light effects based on audio signals
`is described in U.S. Patent 4,081,829 (issued March
`This apparatus controls the display of
`28, 1978).
`two-dimensional rings or solid shapes on the screen
`of a television receiver, in response to audio input
`However, only a limited set of two-
`signals.
`dimensional shapes can be displayed and only limited
`changes in their shape or color can be accomplished
`in response to the audio input.
`Another example of a conventional apparatus for
`producing visual light effects in response to audio
`signals is described in U.S. Patent 4,257,062 (issued
`March 17, 1981). This apparatus controls a set of
`lamps which are mounted in eyewear to be worn by -the
`user, by switching individual ones of the lamps on
`
`and of f in response to....Imusic..(cid:173) Peak levels of specif ic
`frequency bands of the music are detected and
`employed to switch on or off different ones of the
`
`1 amps.
`Another system for producing visual effects in
`response to audio signals has been described in the
`January 1993 issue of NewMedia magazine (at page 18)
`as a system which includes a Silicon Graphics Iris
`Indigo workstation, and which alters the appearance
`of colored visual representations of sound waves
`(displayed on a large screen in a concert hall) in
`response to crowd noise (picked up by a microphone
`during a concert) and live music in MIDI format
`(generated by musicians during the concert) supplied
`to the workstation.
`It is believed that prerecorded control tracks
`(which correspond to prerecorded audio such as music)
`
`10
`
`I v15
`
`20
`
`25
`
`30
`
`35
`
`Page 14 of 140
`
`PETITIONERS' EXHIBIT 1009
`
`

`
`-8-
`
`PATENT
`
`have not been employed to control operation of a
`computer system, such as to control generation of a
`virtual environment by a VR computer system. It is
`also believed that control signals have not been
`extracted from music for use in controlling
`generation of a virtual environment by a VR system
`(e.g., by populating the virtual environment with
`animated virtual objects which move in response to
`the music).
`
`Summary of the Invention
`In a preferred embodiment, the invention is a
`computer system and computer-implemented method for
`the creation and control of a virtual world in
`response to music signals and/or prerecorded control
`tracks corresponding to the music signals.
`The
`system includes means for interfacing between the
`computer software which controls production of the
`virtual world, and live or prerecorded music (and/or
`prerecorded control tracks).
`The invention
`transcends traditional use of VR as a musical
`instrument, and enables a VR system to be employed as
`a virtual stage driven by music.
`In another class of embodiments, the invention
`controls operation of a computer system (which need
`not be a VR system) in response to one or luore
`prerecorded control tracks corresponding to audio
`signals, or in response to both music signals and one
`or more such prerecorded control tracks.
`The component of the inventive system which
`generates control signals from input music (and/or
`prerecorded control tracks and/or human generated
`input signals), or which sends prerecorded cohtrol
`tracks in appropriate format to a VR system or other
`
`5
`
`10
`
`15
`
`20
`
`25
`
`30
`
`Page 15 of 140
`
`PETITIONERS' EXHIBIT 1009
`
`

`
`11
`
`PATENT
`
`-9-
`processor, will sometimes be referred to herein as an
`"Acoustic Etch" system or an "Acoustic Etch."
`In preferred embodiments, the invention employs
`music to manipdlate or control a virtual environment.
`This can be accomplished in several ways.
`Since
`music cannot directly interact with the virtual
`environment, the Acoustic Etch receives music (in
`some electronic, acoustic, or optical form) and
`generates control signals therefrom which are used by
`a VR system to influence activity in the virtual
`world.
`The control signals derived from the music may
`be extracted from the music directly. For example,
`the Acoustic Etch can employ a simple algorithm (of
`the same type used by well known graphic equalizers)
`to extract a rhythm signal indicative of the beat of
`some frequency band of the music (e.g. a band
`representing drums), or of some other parameter of a
`frequency band of the music. The rhythm signal is
`sent to the VR system which in turn generates control
`signalstherefrom to control the rhythm of a virtual
`dancer (or some other moving virtual object).
`As an alternative (or in addition) to extracting
`signals from music itself for processing by a VR
`system, the invention can supply to the VR system one
`or more prerecorded control tracks corresponding to
`the inusic, or can generate control signals from
`prerecorded control tracks and then supply such
`control signals to the VR system for processing. For
`example, control tracks can be prerecorded along with
`left and right tracks of a stereo music signal.
`The
`prerecorded control tracks, left stereo track, and
`right stereo track, can then be played back
`(simultaneously or with selected delays between them)
`and received in parallel by the VR system.
`The
`
`5
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`Page 16 of 140
`
`PETITIONERS' EXHIBIT 1009
`
`

`
`11
`
`-10-
`
`PATENT
`
`control tracks can be generated automatically (e.g.,
`by electronic signal processing circuitry) in
`response to a music signal and then recorded, or can
`be generated in.response to manually asserted
`commands from a person (while the person listens to
`such music signal) and then recorded.
`Prerecorded control tracks can be indicative of
`more sophisticated analysis of a corresponding music
`signal than could be conveniently performed by some
`contemplated (e.g., inexpensive) VR system
`The'placement and
`embodiments of the invention.
`rhythm of dancers could be encoded in prerecorded
`control tracks, for example.
`The use of prerecorded control tracks has
`several advantages and features, including the
`following:
`
`an entire song can be choreographed and
`prerecorded with a control track (for example,
`indicative of placement and rhythm of dancers), so
`that the control track forms part of the prerecorded
`choreographed musical work;
`the control track can include higher level
`information, such as pictures of a dancer or other
`performer, which can be used as source data by the VR
`system to display images of the performer in the
`virtual environment;
`the medium for the control track need not
`the same as that of the music.
`For example, the
`music may be recorded on a compact disk (CD) while
`the control track is recorded on a computer game
`cartridge or other medium;
`synchronization of the control track and
`the music can be accomplished under control of the VR
`system, which could use the control track to
`synchronize with the music, or vice versa;
`
`5
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`I
`
`I
`
`Page 17 of 140
`
`PETITIONERS' EXHIBIT 1009
`
`

`
`-11-
`
`PATENT
`
`the control track can be encoded (or
`(e)
`processed) in a way which accounts for the "delay
`time" required ior the VR system to use the
`information coming from the control track.
`This will
`improve the apparent synchronization between the
`music and the graphics data output from the VR
`system, even when the VR system requires a long time
`to "draw" a particular frame of an animated virtual
`world; and
`
`a prerecorded control track can eliminate
`(f)
`theneed for some embodiments of the invention to
`include means for automatically decoding musical
`expression (the automatic decoding of musical
`expression is poorly understood).
`For example, an operator can record a control
`track which is emotionally linked with a song.
`The
`VR system could then easily convert the control track
`into a variety of control signals, and can produce
`more repeatable and interesting results than could be
`achieved by processing the music directly (in the
`absence of the control track).
`The major disadvantage of using a prerecorded
`control track is that the control track must be
`generated and recorded in advance, and then played
`It must be delivered in
`back in some way.
`conjunction with the music, and the easiest way to do
`this is on the same physical recording medium.
`An advantage of embodiments of the invention
`which directly process music (rather than processing
`a prerecorded control track) is that the music and
`the VR control signals generated therefrom are more
`independent,than are a control track and the VR
`control signals generated therefrom (and can be
`related in any of a variety of ways). In embodiments
`which directly process music, the visual experience
`
`5
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`Page 18 of 140
`
`PETITIONERS' EXHIBIT 1009
`
`

`
`-12-
`
`PATENT
`
`and emotional coupling between the VR and the music
`is looser, since the interpretation is generically
`related to musical signals and their processing.
`However, specific processing algorithms can be used
`by the VR system for specific songs- thus tailoring
`the algorithm to the music.
`In essence, preferred embodiments of the
`invention use music to create a "track" of distilled
`music which is in a form usable by a VR system.
`The
`interpretation of the information is still dependent
`on the VR system, or the particular VR software being
`run by a computer system.
`The same "raw" music or
`control track can be interpreted differently by
`different VR systems (or VR software programs) in the
`sense that different VR systems (or programs) can
`generate different sets of control.signals in
`response to a single raw input signal. Alternatively,
`the same VR system (or program) can interpret the
`same "raw" music or control track differently at
`different times. The control track can be used to
`program the VR system's response and thus tailor the
`system to a specific song.
`
`Obiects and Advantages
`Accordingly, several objects and advantages of
`various embodiments of the present invention are:
`to provide an apparatus which extracts
`information from music (or other audio) for the
`control and manipulation of objects within a virtual
`environment;
`to provide an apparatus which uses a control
`track prerecorded along with audio (music, in
`preferred embodiments) for the control and
`manipulation of objects within a virtual environment;
`
`5
`
`10
`
`15
`
`20
`
`25
`
`30
`
`n
`
`Page 19 of 140
`
`PETITIONERS' EXHIBIT 1009
`
`

`
`PATENT
`
`to provide a VR system which delays audio (in
`response to which control signals are generated) in
`order to compensate for the lag introduced by other
`components of the VR system;
`to provide a virtual experience in which music
`
`effectively drives the display of an animated
`graphical scene;
`
`to provide a mechanism by which music is used to
`control and influence a virtual environment in such a
`way as to relieve the database which describes the
`
`virtual environment from having to define all the
`motions of the objects in the virtual environment;
`
`to provide a control track for the influence and
`
`control of a virtual environment in which the control
`track is created during or following the music
`recording and production process when individual
`tracks (of a multi-track musical work) that are used
`for a particular mix are available before being mixed
`down; and
`
`to provide a control track which can contain
`information (such as images of a performer's face,
`
`for example) other than information extracted from
`corresponding music;
`Further objects and advantages are to provide
`for the rapid creation-and animation of a virtual
`
`environment from music which already has a high level
`
`of production quality.
`
`Brief Description of the DrawiDcLs
`Figure 1 is a diagram of a preferred embodiment
`
`of the inventive system, in which a music source is
`interfaced to a VR system by an Acoustic Etch system.
`
`The blocks may or may not represent physically
`distinct objects (several of the blocks could be
`
`implemented in a single device).
`
`5
`
`10
`
`15
`
`20
`
`25
`
`30
`
`I
`
`I
`
`Page 20 of 140
`
`PETITIONERS' EXHIBIT 1009
`
`

`
`11
`
`0
`
`PATENT
`
`5
`
`10
`
`15
`
`20
`
`25
`
`30
`
`-14-
`Figure 2 is a diagram of a variation on the Fig.
`1 embodiment, in which the Acoustic Etch system
`receives or contains prerecorded control tracks, and
`music corresponding to the control tracks is used to
`cue output of the stored control tracks to the 'R
`processor.
`Figure 3 is a graph of a control track and a
`corresponding music signal, where the control track
`is phase shifted relative to the music signal by a
`degree adequate to compensate for delays that are
`expected to be introduced, in other parts of the
`system, during processing initiated in response to
`the control track.
`Figure 4 is a block diagram of a variation on
`the Acoustic Etch apparatus employed in the Fig. 1
`system.
`Figure 5 is a block diagram of a system for
`creating an audio tape with control tracks for
`playback by the system shown in Figure 6.
`Figure 6 is a block diagram of a system for
`playback of the audio tape produced by the Figure 5
`system.
`Figure 7 is a schematic diagram of a circuit
`suitable for implementing any of signal conditioning
`blocks 120A and 120B.
`Figure 8 is a schematic diagram of a circuit
`suitable for implementing either of tape IF
`convertors 140X or 140Y (of Fig. 5).
`Figure 9 is a schematic diagram of a circuit
`suitable for implementing either of tape IF
`convertors 220X or 220Y (of Fig. 6).
`Figure 10 is a block level description of the
`software which is preferably run on VR system 250 of
`Fig. 6.
`
`Page 21 of 140
`
`PETITIONERS' EXHIBIT 1009
`
`

`
`-15-
`
`PATENT
`
`Figure 11 is a representation of a typical
`single eye image as displayed on display 260 of Fig.
`
`6.
`
`Detailed Description of the Preferred Embodiments
`The term "audio signal" is used herein in a
`broad sense to include not only sound waves but also
`electrical, optical, or other signals representing
`sound waves (such as the electrical output of a
`transducer in response to sound waves). The terms
`"music signal" and "music" are used interchangeably
`herein in a broad sense to include not only sound
`waves that are recognizable by a human listener as
`music, but also electrical, optical, or other signals
`representing such sound waves (such as the electrical
`output of a transducer in response to the sound
`waves). Typically, a system embodying the invention
`will receive and process music signals in the form of
`digitized electrical signals.
`Figure 1 is a diagram of a preferred embodiment
`In Fig. 1, music source 1
`of the.inventive system.
`is interfaced to VR processor 7 by Acoustic Etch
`system 3. VR processor 7 is a computer programmed
`with software for implementing a virtual environment.
`Specifically, VR processor 7 can cause image data
`representing a virtual environment to be displayed on
`display device 8 and can cause left and right
`channels of audio signals (simulating sounds in the
`virtual environment) to be played back to a user
`wearing headphones 10 (which include left and right
`speakers). Display device 8 can be any of a variety
`of.devices, such as a device which mounts on the head
`of a human user (preferably including left and right
`monitors for providing a stereoscopic display to the
`user), or a single flat screen display which outputs
`
`5
`
`10
`
`15
`
`20
`
`2