United States Patent
`
`[19]
`
`[11]
`
`Patent Number:
`
`5,513,129
`
`Bolas et al.
`
`[45] Date of Patent:
`
`Apr. 30, 1996
`
`I|||||||||||||IllIIIIIIIIIIIIIII|l||IIllll||||IIIIIIIIIIIIIIIIIIIIIIIIIIIII
`US005513129A
`
`[54] METHOD AND SYSTEM FOR
`CONTROLLING COMPUTER-GENERATED
`VIRTUAL ENVIRONMENT IN RESPONSE
`To AUDIO SIGNALS
`
`[75]
`
`Inventors: Mark Bolas; Ian E. McDowall, both of
`Palo Alto; Michael N. Bolas, Los
`Angeles’ an of Calif.
`
`[73] Assignee: Fakespace, Inc., Menlo Park, Calif.
`
`5,307,456
`5,319,452
`
`. . . .. 395/154
`4/1994 MacKay . .. .... . . .. .. . . . .
`6/1994 Funahashi
`........................... .. 84/464 R
`
`FOREIGN PATENT DOCUMENTS
`
`..... .. 84/610
`2142461
`1/1985 United Kingdom ..
`
`6/1992 WIPO ..: .......................... .. G06F 3/03
`W092/09948
`
`OTHER PUBLICATIONS
`Jacobson, et 231., “Time for Technojuju,” NewMedia, p. 18,
`jam 1993_
`
`firimary ?aminer_1~"E'mS BL‘Riamiiie1§; L‘ b h Alfr d A
`ttomey,
`gent, or
`zrm— im ac
`im ac ;
`e
`Equitz
`
`.
`
`[57]
`
`ABSTRACT
`
`_
`_
`A “}5“‘°d and, apparatus f0‘ the °°¥“’°1 a“d_mamP“1a“,°“ 91°
`3. V1I'[U8l e1'1V11'0IlII1CIllL (Sl.1Cll ZS V11T11a.l objects T.l'1CI'C1I1) 1Il
`response to a music signal. The music is either interpreted
`directly to eflect the control and manipulation. Alternatively,
`a control traclccorresponding to an audio signal (such as'a
`,
`i processe
`0 con 0 an
`
`manipulate the virtual world (or to control some other
`process of a computer system) as the audio signal is playing.
`In preferred embodiments, a computer creating a virtual
`world interprets the music, the control track, or both, and
`“S95 the “"_5“1lin:‘%_ im°°TmaFi°“ 10 m°_difYv “real? and °r
`control objects
`in the virtual environment. Preferred
`embodiments of the inventive system include apparatus for
`delaying input music to compensate for lag introduced by
`the system components, such as delay required to implement
`processing of control
`tracks corresponding to the input
`mum
`
`23 Claims, 7 Drawing Sheets
`
`[21] App1'N°': 91’650
`[22] Filed:
`Jul. 14, 1993
`
`Int. Cl.5 ...................................................... G06F 17/00
`[51]
`
`. ... .. .......
`[52] U.S. Cl.
`.. .. ... ... .. 364/578
`[58] Field of Search ..................................... 364/578, 514,
`364/DIG_ 1’
`2;
`358/81’ 82; 345/156; 84/645, 610’ 464 R:
`601, 602, 609, 611’ 634’ 635, 641’ 642,
`DIG 1, DIG 2’ DIG. 29; 395/152’ 153,
`20°
`
`[56]
`
`References Cited
`
`U-S- PATENT DOCUMENTS
`84/464
`1/1970 King
`9/1971 Tuber-‘:1:""""""""""""""":::"'369/13.5
`11/1971 Mm, et a1_
`__
`359/135
`3/1975 Coyle _________ __
`353/32
`3/1978 Brown ... ... ....
`.... .... 358/82
`1/1980 Wakeruan
`84/DIG. 29
`3/1981 Mefedlth -- - - --
`- ------- 353/31
`5/1981 Kaipinsky
`358/82
`8/1988 Paist ......... ..'........ ..
`358/81
`1/1991 Zimmerman et al.
`...... 340/709
`9/1992 MacKay et al.
`...................... .. 395/155
`
`
`
`3 490 328
`3:609:019
`3,517,647
`3,900,335
`4,081,829
`4,182,214
`4357352
`4,267,561
`4,768,086
`4,988,981
`5,148,154
`
`11
`
`HEAD-
`
`TRACKING
`MEANS
`
`8
`
`(DISPLAY)
`
`3
`
`
`
`
`
`
`
`MUSIC
`SOURCE
`
`ANALYZER
`:
`
`
`vn PROCESSOR
`
`INTERFACE
`
`9
`
` NPUT DEVICE)
`10
`
`HEADPHONES
`
`)
`
`(
`
`
`
`AUDIO SIGNAL
`(OPTIONALLY WITH
`CONTROL mAcK(3))
`
`AUDIO SIGNAL WITH CONTROL TRACK(S)
`(DR CONTROL TRACK(S) ALONE)
`
`AUDIO
`OUTPUT
`SIGNAL
`
`Page 1 of 23
`
`PET|T|ONERS' EXHIBIT 1001
`
`Page 1 of 23
`
`PETITIONERS' EXHIBIT 1001
`
`

`
`U.S. Patent
`
`Apr. 30, 1996
`
`Sheet 1 of 7
`
`5,513,129
`
`Ennis
`
`ém:
`
`m_,_ca<E
`
`mzfiz
`
`I
`
`ow
`
`5mainHzmzg
`
`m
`
`.$_,_oIn_9a£
`5&8
`
`o_%<
`
`._<zm._m
`
`mommmoommm>
`
`E
`
`m_o<n_mm._.2_
`
`
`
`
`
`Am§o<E.SE28:._._>>3,55o_%<
`
`
`
`
`
`m__,_S<@v_o<ESE28me
`
`m
`
`><._n_ma
`
`9
`
`5%..
`
`mafia
`
`ém:
`
`$205
`
`m>
`
`momwmoomm
`
`émz
`
`,..z_va<E
`
`mz<m_s_
`
`:
`
`_..9".
`
`_m
`
`-mu<Em:z_EN
`>._<2<-E-
`
`
`o
`
`m
`
`v._
`
`ézuaoE:<
`
`::>>>._._<2o_ES
`
`
`
`2eva<ESE28
`
`SE28
`
`v_o<E
`
`momzom
`
`<_.
`
`mas:
`
`H538
`
`Page 2 of 23
`
`PET|T|ONERS' EXHIBIT 1001
`
`Page 2 of 23
`
`PETITIONERS' EXHIBIT 1001
`
`
`
`
`
`
`
`
`

`
`9
`
`U.S. Patent
`
`%U
`
`S
`
`a
`
`m__mE.m.9“.m.22%
`
`ms:A..||%_
`_o_m=_>_
`
`M.
`
`>mo_Es_
`
`mommmuommuISE28
`E,EMwmaofimo'm_ov,_‘M..._,.w_mmm2_
`
`sEm>m5%.
`
`m_o_>ma
`
`
`
`.__>=Eo..$8._<2w_wv_o<E45
`
`0
`
`_oo_.5%:E.
`
`._.I'_99
`
`fieu_o
`
`fl,
`
`
`
` azozéfizB2Emma:335,mo_>m5>mézoio5m>opo_%<m_o<uEm:2_
`
`._E_wa
`
`_,__gags.
`
`wo._<z<
`
`vast
`
`._<2o_.Eo
`
`SE28
`
`e.0_u_
`
`Page 3 of 23
`
`PET|T|ONERS' EXHIBIT 1001
`
`Page 3 of 23
`
`PETITIONERS' EXHIBIT 1001
`
`
`

`
`U.S. Patent
`
`Apr. 30, 1996
`
`Sheet 3 of 7
`
`5,513,129
`
` 101A
`121A
`MULTITRACK
`SIGNAL
`TAPE PLAYER
`CONDITIONING
`
`
`
`
`
`
`
`SIGNAL
`CONDITIONING
`
`; 120:3
`
`
`
`MICROPROCESSDR 13”‘
`
`1 30X
`
`MICROPROCESSOR 13”
`w/A _, D
`+ DIGITAL I/O
`
`130v
`
`TAPEIF
`CONVERTER
`
`14”‘
`
`1 40X
`
`14”
`
`TAPEIF
`CONVERTER
`14Qy
`
`SWITCHES
`
`NW 150
`
`POT
`
`165
`
`DISK
`DRIVE
`
`(OR OTHER
`DATA MEDIA)
`
`4 TRACK
`TAPE RECORDER
`
`2 TRACK
`TAPE PLAYER
`
`170L
`
`170v
`
`Page 4 of 23
`
`PET|T|ONERS' EXHIBIT 1001
`
`Page 4 of 23
`
`PETITIONERS' EXHIBIT 1001
`
`

`
`U.S. Patent
`
`Apr. 30, 1995
`
`Sheet 4 of 7
`
`5,513,129
`
`205 4 TRACK
`205FI
`TAPE PLAYER
`PROCESSOR
`
`
`
`
`
`210
`
`220X
`
`TAPE IF
`CONVERTER
`
`
`
`TAPE IF
`CONVERTER
`
`221Y
`
`
`
`210R MULTICHANNEL AUDIO
`
`
`
`DIGITAL WITH
`MICROPROCESSOR
`SERIAL OUTPUT
`
` 240
`
` VR SYSTEM
`
`FIG. 6
`
`Page 5 of 23
`
`PET|T|ONERS' EXHIBIT 1001
`
`Page 5 of 23
`
`PETITIONERS' EXHIBIT 1001
`
`

`
`U.S. Patent
`
`Apr. 30, 1996
`
`Sheet 5 of 7
`
`5,513,129
`
`FIG. 7
`
`
`
`15K
`
`1
`5 09
`TO
`MICRO-
`
`PROCESSOR
`
`0.1
`0.1
`FROM
`‘———"'”'—“|
`TAPE
`ANALOG '1.
`1
`K
`IN +5
`-
`
`
`
`-5
`
`Page 6 of 23
`
`PET|T|ONERS' EXHIBIT 1001
`
`Page 6 of 23
`
`PETITIONERS' EXHIBIT 1001
`
`

`
`U.S. Patent
`
`Apr. 30, 1996
`
`Sheet 6 of 7
`
`5,513,129
`
`OPEN AND INITIALIZE DEVICES
`
`4
`
`00
`
`INITIALIZE VARIABLES
`
`410
`
`A
`
`DEFINE AND CREATE
`STANDARD OBJECTS
`
`READ HEAD POSITION
`
`READ CONTROL
`TRACK INFORMATION
`
`420
`
`43D
`
`440
`
`READ DIGITZED AUDIO AND
`INPUT INFORMATION
`
`450
`
`CREATE; DISTROY; MOVE AND
`MODIFY OBJECTS
`
`460
`
`CALL DRAWING ROUTINE
`
`470
`
`IS ESCAPE KEY DEPRESSED 2
`
`480
`
`FIG. 10
`
`49°
`
`
`
`Page 7 of 23
`
`I
`
`PET|T|ONERS' EXHIBIT 1001
`
`Page 7 of 23
`
`PETITIONERS' EXHIBIT 1001
`
`

`
`U.S. Patent
`
`Apr. 30, 1996
`
`Sheet 7 of 7
`
`5,513,129
`
`FIG. 11
`
`Page 8 of 23
`
`PET|T|ONERS' EXHIBIT 1001
`
`Page 8 of 23
`
`PETITIONERS' EXHIBIT 1001
`
`

`
`1
`METHOD AND SYSTEM FOR
`CONTROLLING COMPUTER-GENERATED
`VIRTUAL ENVIRONMENT IN RESPONSE
`TO AUDIO SIGNALS
`
`Field of the Invention
`
`The invention pertains to methods and apparatus for
`controlling a computer system in response to music signals,
`or in response to prerecorded control tracks corresponding to
`audio signals (such as music signals). In preferred embodi-
`ments,
`invention pertains to methods and apparatus for
`creating and modifying, or otherwise controlling, computer-
`generated virtual environments (or displayed virtual objects
`in virtual environments) in response to music signals or in
`response to prerecorded control
`tracks corresponding to
`audio signals.
`
`BACKGROUND OF THE INVENTION
`
`The terms “virtual environment,” “virtual world,” and
`“virtual reality” are used interchangeably to describe a
`computer-simulated environment (intended to be immer-
`sive) which includes a graphic display (from a user’s first
`person perspective, in a form intended to be immersive to
`the user), and optionally also sounds which simulate envi-
`ronmental sounds. The abbreviation
`will sometimes
`be used herein to denote “virtual reality,” “virtual environ-
`ment,” or “virtual world”. A computer system progrannned
`with software, and including peripheral devices, for produc-
`ing a virtual environment will sometimes be referred to
`herein as a VR system or VR processor.
`The graphic display generated by a VR system can be a
`two-dimensional (2D) or a three-dimensional (3D) display.
`Typically, a VR system includes an input device and user
`interface software which enable a user to interact with the
`scene being displayed, typically to simulate motion in the
`virtual environment or manipulation of displayed represen-
`tations of objects (“virtual objects”) in the virtual environ-
`ment. Typically, the illusion of immersion in a virtual reality
`system is strengthened by the use of head-tracking or some
`other such system which directs the computer to generate
`images along the area of viewing interest of the user.
`The present invention is a method and apparatus particu-
`larly useful for creating and/or controlling virtual environ-
`ments. A VR system which embodies the invention can
`rapidly and inexpensively create, animate, or otherwise
`control a wide variety of entertaining virtual environments
`and virtual objects in response to music or in response to
`prerecorded “control
`tracks” which correspond to audio
`signals (such as music).
`While currently being used in the research and scientific
`communities, VR systems are becoming less expensive and
`are poised to reach the consumer electronics market as
`entertainment devices.
`
`VR systems must generate a much greater amount of
`content data (image data and audio data simulating envi-
`ronmental appearance and sounds) than must be generated in
`most other electronic media. Whereas video game systems
`require complex scenes to be generated and themes to be
`programmed, such systems can easily limit the scope of the
`game content because they can easily constrain the player to
`move in a few simple directions (e.g., left and right) and
`need only produce images to be presented on flat screen
`monitors or on simple 3D field-sequential type monitors.
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`5,513,129
`
`2
`
`In contrast, by their very nature, VR systems allow the
`user to look around and fly around in many different
`directions and positions. Even where the user is constrained
`to look only toward the left or the right, VR systems must
`construct complete representations of 3D worlds. This com-
`plexity has made it very diflicult to generate virtual worlds
`for the consumer entertainment market in a quick fashion.
`In addition to the complexity of creating static 3D models
`for virtual worlds, it has also been diflicult to control the
`dynamics of virtual worlds. VR systems to date are notori-
`ous for providing only very boring and nearly static envi-
`ronments. The few VR systems that
`include dynamic
`motions of the virtual world either base such motions on
`physical laws (such as gravity) or base the motions on
`corresponding motions produced by human users (such as
`motion of the fingers of a user wearing a conventional
`“glove” input device).
`The present invention overcomes the limitations of con-
`ventional VR systems by providing an efiicient way to
`generate content data (i.e., animated image data and audio
`data) to fill or populate a virtual environment in a choreo-
`graphed response to input music signals.
`There has long been an interest in the virtual reality field
`with respect to the possibility of virtual musical instruments
`and the creation of new and novel instruments within a
`virtual world. The present invention is a radical shift from
`previous attempts to combine music and virtual environ-
`ments.
`
`Conventional efforts to integrate music with virtual envi-
`ronments have, to date, all been directed toward creation of
`music from a virtual environment. The musical expression
`of the user has been treated as an urge seeking to be brought
`forth, and virtual environments have been seen as vehicles
`for the user to perform music or dance without having to
`learn special physical skills. Much effort has been made to
`make sounds appear to be coming from virtual objects in the
`virtual environment. This has been done by running audio
`into the VR system and then convolving the audio in such a
`way as to make it appear to come from a certain place in the
`virtual environment.
`
`For example, at the NASA Ames View Lab, Scott Fisher,
`Rick Jacoby, and others explored virtual environments. One
`aspect of the research was the integration of audio into the
`virtual experience. This included the use of audio cues for
`such purposes as telling one if one bumped into a virtual
`object, but there was no tactile feedback for such events. The
`research pushed into the more artistic realm of creation of
`music in the context of a virtual world.
`Mark Bolas and Phil Stone created the Virtual Theremin
`and virtual drum kit. In this system, the user wore a glove
`and a hand tracker and moved the gloved hand to manipulate
`virtual objects which were in turn linked to various synthe-
`sizer parameters. Thus, by manipulating virtual objects (as
`taught, for example, by U.S. Pat. 4,988,981, issued Jan. 29,
`1991), sounds of different qualities could be created. A
`skilled user could create modern sounding musical inter-
`ludes. These ideas have been carried forth by people such as
`Jaron Lanier who has givenja number of public perfor-
`mances in which he manipulates virtual objects to create a
`musical performance. Research and exploration along these
`lines is expected to continue (the virtual “air guitar” and the
`like will probably be developed). In all VR systems of this
`type, manipulation of a virtual object causes the sound or
`music to change.
`Currently, virtual worlds are created by describing a
`simulation and a number of objects. The interaction of the
`
`Page 9 of 23
`
`PET|T|ONERS' EXHIBIT 1001
`
`Page 9 of 23
`
`PETITIONERS' EXHIBIT 1001
`
`

`
`3
`
`4
`
`5,513,129
`
`objects is described in some form of simulation language or
`graphical description. Traditionally, the control and creation
`of the objects is driven by “world building” software. Once
`a virtual world has been created, a lirrrited number of its
`parameters may be manipulated by the user from “inside”
`the virtual world. One example of how these databases are
`created is described in PCT International Patent Application
`WO 92109948, by VPL Research Inc. As is evident from WO
`92/09948, it has define animation for all or even some of the
`virtual objects in a virtual world. Until the present invention,
`it had not been proposed to interface to nodes in a database
`defining a virtual environment, and to manipulate such
`nodes, on the basis of music.
`Conventional VR systems and music have thusfar been
`used together in ways which have the following disadvan-
`tages:
`(a) a VR system has been used as a virtual musical
`instrument, so that the user must “play” the virtual
`instrument (by manipulating an input device) to hear
`anything. This means that the system creates music,
`and that the system’s musical output is limited by the
`user’s ability to “play” the “instrument;”
`(b) VR systems that have given sounds to virtual objects
`(e.g., the system displays a virtual kitchen sink and
`produces a “drip-drip” sound which seems to come
`from the sink’s location) have required that the sounds
`are generated by signals produced within the VR sys-
`tem in response to user manipulation of an input device
`or internal programs, which signals are then interpreted
`by a synthesizer. The sounds produced by the synthe-
`sizer are thus cued from the VR system in response to
`manipulation of an input device (which manipulation
`may, for example, to cause a user to “move” into a
`position to view or otherwise interact with a virtual
`kitchen sink from which sounds will then seem to
`emanate). Thus, these VR systems have depended on
`user manipulation of an input device to control the
`appearance or activities of objects in a virtual environ-
`ment, to cause the VR system to cue production of
`sound events; and
`
`(c) VR systems have played musical scores as background
`music for the virtual environment.
`
`Basically, the paradigm to date has been to create systems
`that have (virtual) object-driven sounds. This invention
`reverses the paradigm to create a system which has musi-
`cally-driven objects.
`One VR system has been developed in which a VR
`processor is programmed to perform simple operations to
`modify a virtual environment in response to voice com-
`mands. This VR system, developed at the NASAAmes View
`Lab during the years 1988-1989, was capable of displaying
`a virtual object, or terminating the display of a virtual object,
`in response to a voice command from a human user. How-
`ever,
`the system did not produce, modify, or otherwise
`control a virtual environment in response to music, or in
`response to a prerecorded control track corresponding to an
`audio signal.
`Outside the VR field, many attempts have been made to
`produce devices which provide users with visual light effects
`based on an audio signal, such as music. However, these
`systems have been disappointing to watch (principally
`because the light shows are two-dimensional and are not
`obviously correlated with the audio input), and have typi-
`cally met with disappointment when marketed.
`An example of a conventional apparatus for producing
`visual light effects based on audio signals is described in
`U.S. Pat. No. 4,081,829 (issued Mar. 28, 1978). This appa-
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`ratus controls the display of two-dimensional rings or solid
`shapes on the screen of a television receiver, in response to
`audio input signals. However, only a linrited set of two-
`dimensional shapes can be displayed and only limited
`changes in their shape or color can be accomplished in
`response to the audio input.
`Another example of a conventional apparatus for produc-
`ing visual
`light efl“ects in response to audio signals is
`described in U.S. Pat. No. 4,257,062 (issued Mar. 17, 1981).
`This apparatus controls a set of lamps which are mounted in
`eyewear to be worn by the user, by switching individual ones
`of the lamps on and off in response to music. Peak levels of
`specific frequency bands of the music are detected and
`employed to switch on or ofi different ones of the lamps.
`Another system for producing visual efiects in response to
`audio signals has been described in the Jan. 1993 issue of
`NewMedia magazine (at page 18) as a system which
`includes a Silicon Graphics Iris Indigo workstation, and
`which alters the appearance of colored visual representations
`of sound waves (displayed on a large screen in a concert
`hall) in response to crowd noise (picked up by a microphone
`during a concert) and live music in MIDI format (generated
`by musicians during the concert) supplied to the worksta-
`tion.
`
`tracks (which
`It is believed that prerecorded control
`correspond to prerecorded audio such as music) have not
`been employed to control operation of a computer system,
`such as to control generation of a virtual environment by a
`VR computer system. It is also believed that control signals
`have not been extracted from music for use in controlling
`generation of a virtual environment by a VR system (e.g., by
`populating the virtual environment with animated virtual
`objects which move in response to the music).
`
`SUMMARY OF THE INVENTION
`
`In a preferred embodiment, the invention is a computer
`system and computer-implemented method for the creation
`and control of a virtual world in response to music signals
`and/or prerecorded control
`tracks corresponding to the
`music signals. The system includes means for interfacing
`between the computer software which controls production of
`the virtual world, and live or prerecorded music (and/or
`prerecorded control tracks). The invention transcends tradi-
`tional use of VR as a musical instrument, and enables a VR
`system to be employed as a virtual stage driven by music.
`In another class of embodiments, the invention controls
`operation of a computer system (which need not be a VR
`system) in response to one or more prerecorded control
`tracks corresponding to audio signals, or in response to both
`music signals and one or more such prerecorded control
`tracks.
`
`The component of the inventive system which generates
`control signals from input music (and/or prerecorded control
`tracks and/or human generated input signals), or which
`sends prerecorded control tracks in appropriate format to a
`VR system or other processor, will sometimes be referred to
`herein as an “Acoustic Etch” system or an “Acoustic Etch.”
`In preferred embodiments, the invention employs music
`to manipulate or control a virtual environment. This can be
`accomplished in several ways. Since music cannot directly
`interact with the virtual environment, the Acoustic Etch
`receives music (in some electronic, acoustic, or optical
`form) and generates control signals therefrom which are
`used by a VR system to influence activity in the virtual
`world.
`
`Page 10 of 23
`
`PET|T|ONERS' EXHIBIT 1001
`
`Page 10 of 23
`
`PETITIONERS' EXHIBIT 1001
`
`

`
`5
`
`6
`
`5,513,129
`
`The control signals derived from the music may be
`extracted from the music directly. For example, the Acoustic
`Etch can employ a simple algorithm (of the same type used
`by well known graphic equalizers) to extract a rhythm signal
`indicative of the beat of some frequency band of the music
`(e.g. a band representing drums), or of some other parameter
`of a frequency band of the music. The rhythm signal is sent
`to the VR system which in turn generates control signals
`therefrom to control the rhythm of a virtual dancer (or some
`other moving virtual object).
`As an alternative (or in addition) to extracting signals
`from music itself for processing by a VR system,
`the
`invention can supply to the VR system one or more prere-
`corded control tracks corresponding to the music, or can
`generate control signals from prerecorded control tracks and
`then supply such control signals to the VR system for
`processing. For example, control tracks can be prerecorded
`along with left and right tracks of a stereo music signal. The
`prerecorded control tracks, left stereo track, and right stereo
`track, can then be played back (simultaneously or with
`selected delays between them) and received in parallel by
`the VR system. The control tracks can be generated auto-
`matically (e.g., by electronic signal processing circuitry) in
`response to a music signal and then recorded, or can be
`generated in response to manually asserted commands from
`a person (while the person listens to such music signal) and
`then recorded.
`tracks can be indicative of more
`Prerecorded control
`sophisticated analysis of a corresponding music signal than
`could be conveniently performed by some contemplated
`(e.g., inexpensive) VR system embodiments of the inven-
`tion. The placement and rhythm of dancers could be encoded
`in prerecorded control tracks, for example.
`The use of prerecorded control tracks has several advan-
`tages and features, including the following:
`(a) an entire song can be choreographed and prerecorded
`with a control track (for example, indicative of place-
`ment and rhythm of dancers), so that the control track
`forms part of the prerecorded choreographed musical
`work;
`
`(b) the control track can include higher level information,
`such as pictures of a dancer or other performer, which
`can be used as source data by the VR system to display
`images of the performer in the virtual environment;
`(c) the medium for the control track need not the same as
`that of the music. For example,
`the music may be
`recorded on a compact disk (CD) while the control
`track is recorded on a computer game cartridge or other
`medium;
`(d) synchronization of the control track and the music can
`be accomplished under control of the VR system,
`which could use the control track to synchronize with
`the music, or vice versa;
`
`(e) the control track can be encoded (or processed) in a
`way which accounts for the “delay time” required for
`the VR system to use the information coming from the
`control track. This will improve the apparent synchro-
`nization between the music and the graphics data
`output from the VR system, even when the VR system
`requires a long time to “draw” a particular frame of an
`animated virtual world; and
`(f) a prerecorded control track can eliminate the need for
`some embodiments of the invention to include means
`for automatically decoding musical expression (the
`automatic decoding of musical expression is poorly
`understood).
`
`10
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`For example, an operator can record a control track which
`is emotionally linked with a song. The VR system could then
`easily convert the control track into a variety of control
`signals, and can produce more repeatable and interesting
`results than could be achieved by processing the music
`directly (in the absence of the control track).
`The major disadvantage of using a prerecorded control
`track is that the control track must be generated and recorded
`in advance, and then played back in some way. It must be
`delivered in conjunction with the music, and the easiest way
`to do this is on the same physical recording medium.
`An advantage of embodiments of the invention which
`directly process music (rather than processing a prerecorded
`control track) is that the music and the VR control signals
`generated therefrom are more independent than are a control
`track and the VR control signals generated therefrom (and
`can be related in any of a variety of ways). In embodiments
`which directly process music,
`the visual experience and
`emotional coupling between the VR and the music is looser,
`since the interpretation is generically related to musical
`signals and their processing. However, specific processing
`algorithms can be used by the VR system for specific songs-
`thus tailoring the algorithm to the music.
`In essence, preferred embodiments of the invention use
`music to create a “trac ” of distilled music which is in a form
`
`usable by a VR system. The interpretation of the information
`is still dependent on the VR system, or the particular VR
`software being run by a computer system. The same “raw”
`music or control track can be interpreted differently by
`different VR systems (or VR software programs) in the sense
`that diiferent VR systems (or programs) can generate dif-
`ferent sets of control signals in response to a single raw input
`signal. Alternatively, the same VR system (or program) can
`interpret the same “raw” music or control track differently at
`different times. The control track can be used to program the
`VR system’s response and thus tailor the system to a specific
`song.
`
`OBJECTS AND ADVANTAGES
`
`Accordingly, several objects and advantages of various
`embodiments of the present invention are:
`to provide an apparatus which extracts information from
`music (or other audio) for the control and manipulation
`of objects within a virtual environment;
`to provide an apparatus which uses a control track pre-
`recorded along with audio (music, in preferred embodi-
`ments) for the control and manipulation of objects
`within a virtual environment;
`
`to provide a VR system which delays audio (in response
`to which control signals are generated) in order to
`compensate for the lag introduced by other components
`of the VR system;
`to provide a virtual experience in which music effectively
`drives the display of an animated graphical scene;
`to provide a mechanism by which music is used to control
`and influence a virtual environment in such a way as to
`relieve the database which describes the virtual envi-
`ronment from having to define all the motions of the
`objects in the virtual environment;
`to provide a control track for the influence and control of
`a virtual environment in which the control track is
`
`created during or following the music recording and
`production process when individual tracks (of a multi-
`track musical work) that are used for a particular mix
`are available before being mixed down; and
`
`Page 11 of 23
`
`PET|T|ONERS' EXHIBIT 1001
`
`Page 11 of 23
`
`PETITIONERS' EXHIBIT 1001
`
`

`
`5,513,129
`
`7
`
`to provide a control track which can contain information
`(such as images of a perforrner’s face, for example)
`other than information extracted from corresponding
`music;
`Further objects and advantages are to provide for the rapid
`creation and animation of a virtual environment from music
`
`which already has a high level of production quality.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`FIG. 1 is a diagram of a preferred embodiment of the
`inventive system, in which a music source is interfaced to a
`VR system by an Acoustic Etch system. The blocks may or
`may not represent physically distinct objects (several of the
`blocks could be implemented in a single device).
`FIG. 2 is a diagram of a variation on the Fig. 1 embodi-
`ment, in which the Acoustic Etch system receives or con-
`tains prerecorded control tracks, and music corresponding to
`the control tracks is used to cue output of the stored control
`tracks to the VR processor.
`FIG. 3 is a graph of a control track and a corresponding
`music signal, where the control track is phase shifted relative
`to the music signal by a degree adequate to compensate for
`delays that are expected to be introduced, in other parts of
`the system, during processing initiated in response to the
`control track.
`
`FIG. 4 is a block diagram of a variation on the Acoustic
`Etch apparatus employed in the FIG. 1 system.
`FIG. 5 is a block diagram of a system for creating an audio
`tape with control tracks for playback by the system shown
`in FIG. 6.
`
`FIG. 6 is a block diagram of a system for playback of the
`audio tape produced by the FIG. 5 system.
`FIG. 7 is a schematic diagram of a circuit suitable for
`implementing any of signal conditioning blocks 120A and
`120B.
`
`FIG. 8 is a schematic diagram of a circuit suitable for
`implementing either of tape IF convertors 140X or 140Y (of
`FIG. 5).
`FIG. 9 is a schematic diagram of a circuit suitable for
`implementing either of tape IF convertors 220X or 220Y (of
`FIG. 6).
`FIG. 10 is a block level description of the software which
`is preferably run on VR system 250 of FIG. 6.
`FIG. 11 is a representation of a typical single eye image
`as displayed on display 260 of Fig. 6.
`
`DETAILED DESCRIPTION OF THE
`PREFERRED EMBODIMENTS
`
`The term “audio signal” is used herein in a broad sense to
`include not only sound waves but also electrical, optical, or
`other signals representing sound waves (such as the electri-
`cal output of a transducer in response to sound waves). The
`terms “music signal” and “music” are used interchangeably
`herein in a broad sense to include not only sound waves that
`are recognizable by a human listener as music, but also
`electrical, optical, or other signals representing such sound
`waves (such as the electrical output of a transducer in
`response to the sound waves). Typically, a system embody-
`ing the invention will receive and process music signals in
`the form of digitized electrical signals.
`FIG. 1 is a diagram of a preferred embodiment of the
`inventive system. In FIG. 1, music source 1 is interfaced to
`VR processor 7» by Acoustic Etch system 3. VR processor 7
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`8
`is a computer programmed with software for implementing
`a virtual environment. Specifically, VR processor 7 can
`cause image data representing a virtual environment to be
`displayed on display device 8 and can cause left and right
`channels of audio signals (simulating sounds in the virtual
`environment) to be played back to a user wearing head-
`phones 10 (which include left and right speakers). Display
`device 8 can be any of a variety of devices, such as a device
`which mounts on the head of a human user (preferably
`including left and right monitors for providing a stereo-
`scopic display to the user), or a single flat screen display
`which outputs either a non-stereoscopic display or a stereo-
`scopic display. Head-tracking means 11 (included in both the
`FIG. 1 and FIG. 2 embodiments) is provided for optionally
`providing input (to processor 7) indicative of the position of
`the head of a human user wearing a head-mounted embodi-
`ment of display device 8.
`Processor 7 is a computer programmed with software
`enabling a human user to interact with the virtual environ-
`ment by manipulating input device 9, whose output
`is
`supplied to processor 7. In one embodiment, input device 9
`includes a glove and sensors mounted to the glove for
`detecting movements of a user’s hand within the glove. In
`another embodiment, input device 9 includes a frame and
`sensors for producing output signals indicative of forces or
`torques exerted on the frame by a user. The frame is
`preferably mounted to display device 8 (or to a base sup-
`porting the display device) symmetrically with respect to an
`axis of symmetry the display device, with limited freedom
`to move relative thereto, and the sensors are preferably
`mounted at the ends of the limited range of motion of the
`frame.
`
`An analog-to-digital conversion circuit within Acoustic
`Etch unit 3 receives and digitizes a music signal from source
`1. The music signal is optionally accompanied by one or
`more prerecorded control tracks corresponding to the music
`signal, which control tracks are played back with the music
`signal. Analyzer 5 within Acoustic Etch unit 3 receives the
`digitized output of circuit 4, and generates control signals by
`processing the music signal (or both the music signal and the
`control tracks). The control signals output from analyzer 5
`are supplied through interface 6 to VR processor 7, for use
`within processor 7 for controlling generation of the virtual
`environment. One or more of the control tracks (or both the
`music signal and