I lllll llllllll Ill lllll lllll lllll lllll lllll 111111111111111111111111111111111
`US007188186Bl
`
`c12) United States Patent
`Meyer et al.
`
`(10) Patent No.:
`(45) Date of Patent:
`
`US 7,188,186 Bl
`Mar.6,2007
`
`(54) PROCESS OF AND SYSTEM FOR
`SEAMLESSLY EMBEDDING EXECUTABLE
`PROGRAM CODE INTO MEDIA FILE
`FORMATS SUCH AS MP3 AND THE LIKE
`FOR EXECUTION BY DIGITAL MEDIA
`PLAYER AND VIEWING SYSTEMS
`
`(76)
`
`Inventors: Thomas W. Meyer, P.O. Box 230345,
`Encinitas, CA (US) 92023-0345;
`Josslyn Motha Meyer, P.O. Box
`230345, Encinitas, CA (US) 92023-0345
`
`( *) Notice:
`
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 0 days.
`
`(21) Appl. No.: 09/389,942
`
`(22) Filed:
`
`Sep. 3, 1999
`
`(51)
`
`Int. Cl.
`G06F 15116
`(2006.01)
`(52) U.S. Cl. ....................................... 709/231; 717/167
`(58) Field of Classification Search ................ 717/167;
`705/14, 15; 709/206, 231
`See application file for complete search history.
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`5,668,599 A *
`5,802,481 A *
`5,838,906 A *
`5,929,836 A *
`5,930,765 A *
`5,974,181 A *
`5,983,005 A *
`6,014,651 A *
`6,038,591 A *
`6,061,403 A *
`6,157,740 A *
`
`9/1997 Cheney et al. ......... 375/240.15
`9/1998 Prieto ......................... 702/190
`11/1998 Doyle et al ................. 709/202
`7I1999 Der et al.
`................... 345/629
`7/1999 Martin ........................ 705/14
`10/1999 Prieto ......................... 382/232
`11/ 1999 Monteiro et al.
`........... 709/231
`1/2000 Crawford .................... 705/400
`3/2000 Wolfe et al ................. 709/206
`512000 Marino ....................... 375/240
`12/2000 Buerkle et al .............. 382/233
`
`6,185,683 Bl*
`6,216,152 Bl*
`6,351,736 Bl*
`6,357,047 Bl*
`6,411,725 Bl*
`6,449,380 Bl *
`6,508,709 Bl *
`6,526,097 Bl *
`6,539,544 B2 *
`6,632,992 B2 *
`6,748,362 Bl *
`
`............... 713/176
`2/2001 Ginter et al.
`4/2001 Wong et al ................. 709/203
`212002 Weisberg et al.
`............. 705/14
`3/2002 Kurtze et al ................ 725/151
`612002 Rhoads ....................... 382/100
`912002 Acharya et al.
`............ 382/100
`1/2003 Karmarkar .. ... .. ... ... ... ... 463/42
`212003 Sethuraman et al. ..... 375/240.2
`3/2003 Ebisawa ..................... 717 /177
`10/2003 Hasegawa ... ... .. ... ... ... ... 84/609
`............... 704/500
`6/2004 Meyer et al.
`
`OTHER PUBLICATIONS
`
`Documentation for ID3Lib 3.05 by Dirk Mahoney, pp. 1-18, Nov.
`22, 1998.*
`"What Makes People Click Advertising on the Web", Jim Sterne,
`pp. 1-12, published Aug. 22, 1997.*
`Nilsson, Martin, "ID3v2 Made Easy" http://www.id3.org/easy.html,
`print date May 12, 2003.
`
`(Continued)
`
`Primary Examiner-Larry D. Donaghue
`(74) Attorney, Agent, or Firm-Rines and Rines
`
`(57)
`
`ABSTRACT
`
`A novel process and system for flexibly adding supplemental
`digital program content such as, for example, transactional
`advertising content, games, polls, contests, interactive music
`videos, and e-commerce content generally and the like, into
`pre-prepared digital media files, such as an MP3 audio file
`or the like, for playback by digital playback apparatus,
`wherein the pre--prepared media file is modified by embed(cid:173)
`ding therein executable code representing such supplemen(cid:173)
`tal program content, and enabling the playback apparatus to
`decode and execute the presentation of the supplemental
`program material as an addition to the playback of the
`pre-prepared media file content.
`
`28 Claims, 10 Drawing Sheets
`(3 of 10 Drawing Sheet(s) Filed in Color)
`
`MPEGAudio
`File
`
`Exe<:utoble
`Cod•
`
`1 - - - - - - I
`I
`
`1 I~----,
`PossibleAdditioocJ
`Compression lo
`Ensure Room for
`Executable Code at
`End of Frames
`
`I
`
`I
`
`I
`
`I
`
`I
`
`I
`
`lnsertionofExeculobleCodeal
`Endolhld'ioDotoinEoch
`Audia Frame
`
`Medi<lfilewith
`Embedded
`Executable Code
`
`I
`_J
`
`MPEG Audio with Ancillary Bits Encoding
`
`Page 1 of 18
`
`BMW EXHIBIT 1006
`
`

`

`US 7,188,186 Bl
`Page 2
`
`OTHER PUBLICATIONS
`
`Nilsson, Martin, "ID3v2-Informal Standard" http://www.id3.org/
`id3v2.3.0.html, print date May 12, 2003.
`
`Freed N. et al., "Multipurpose Internet Mail Extensions (MIME)
`Part One", RFC 2045, (cont.), pp. 1-26, published Nov. 1996.
`* cited by examiner
`
`Page 2 of 18
`
`

`

`U.S. Patent
`
`Mar.6,2007
`
`Sheet 1 of 10
`
`US 7,188,186 Bl
`
`Media File
`
`Executable
`Code
`
`Encoding ·Process
`
`Media File with
`Embedded
`Executable Code
`
`Encoding System Overview
`Figure 1
`
`Page 3 of 18
`
`

`

`U.S. Patent
`
`Mar.6,2007
`
`Sheet 2 of 10
`
`US 7,188,186 Bl
`
`Figure la
`
`Page 4 of 18
`
`

`

`U.S. Patent
`
`Mar.6,2007
`
`Sheet 3 of 10
`
`US 7,188,186 Bl
`
`Figure lb
`
`Page 5 of 18
`
`

`

`U.S. Patent
`
`Mar.6,2007
`
`Sheet 4of10
`
`US 7,188,186 Bl
`
`Figure le
`
`Page 6 of 18
`
`

`

`U.S. Patent
`
`Mar.6,2007
`
`Sheet 5 of 10
`
`US 7,188,186 Bl
`
`I Media File with
`
`Embedded
`Executable Code
`
`/\
`
`\J
`
`Decoding Process
`
`Media File
`
`~
`
`\
`
`Executable
`Code
`
`/\
`
`\)
`
`Media Player or
`Viewer
`
`Verification Process
`(Optional)
`
`4
`I
`I
`I
`I
`I
`
`L ___ 51ricb~~~]~~-------
`
`Playback Environment
`
`Execution
`Environment
`
`L
`
`Decoding System Overview
`Figure 2
`
`Page 7 of 18
`
`

`

`U.S. Patent
`
`Mar.6,2007
`
`Sheet 6of10
`
`US 7,188,186 Bl
`
`MPEG Audio
`File
`
`I\
`
`\
`
`I
`
`Executable
`Code
`
`/\
`
`\)
`
`Unsynchronization of
`Executable Code
`
`Encapsulation in
`id3v2 Format
`
`~
`
`Insertion at beginning of mp3
`Audio Stream
`
`~odinq Process
`
`J
`
`/ MPEG Audio File with /\
`Embedded Executable
`Code
`
`\J
`
`MPEG Audio with id3v2 Encoding
`Figure 3
`
`Page 8 of 18
`
`

`

`U.S. Patent
`
`Mar.6,2007
`
`Sheet 7of10
`
`US 7,188,186 Bl
`
`MPEG Audio
`File
`
`Executable
`Code
`
`Possible Additional
`Compression to
`Ensure Room for
`Executable Code at
`End of Frames
`
`Insertion of Executable Code at
`End of Audio Data in Each
`Audio Frame
`
`~oding Process
`
`_J
`
`Medta File with
`Embedded
`Executable Code
`
`MPEG Audio with Ancillary Bits Encoding
`Figure 4
`
`Page 9 of 18
`
`

`

`U.S. Patent
`
`Mar.6,2007
`
`Sheet 8 of 10
`
`US 7,188,186 Bl
`
`MPEG File
`
`Executable
`Code
`
`Encode Executable
`Code in One or
`More Private Dato
`Packets
`
`lnserlio.n of New Private Doto
`Pockets in the Existing MPEG
`File
`
`I Encoding Process
`
`_J
`
`Media File with
`Embedded
`Executable Code
`
`MPEG with Private Stream Encoding
`Figure 5
`
`Page 10 of 18
`
`

`

`U.S. Patent
`
`Mar.6,2007
`
`Sheet 9 of 10
`
`US 7,188,186 Bl
`
`r-
`
`Media File
`
`I\
`
`\)
`
`Selection of
`·Appropriate
`Locations in Media
`to Embed Data
`
`Executable
`Code
`
`I
`
`\)
`
`Transformation of
`Executable Code
`into a Bit Stream
`
`Insertion of Executable Code by
`Directly Modifying the Media
`
`I SteganO<Jrophic Encoding Process
`
`- -
`
`I
`
`Media File with
`Embedded
`Executable Code
`
`/\
`
`\J
`
`Encoding Using Steganographic Techniques
`Figure 6
`
`11
`
`_J
`
`Page 11 of 18
`
`

`

`U.S. Patent
`
`Mar.6,2007
`
`Sheet 10 of 10
`
`US 7,188,186 Bl
`
`Media File
`
`Executable
`Code
`
`Watermarking
`Process
`
`Encoding Process
`
`Watermarked Media
`File with
`Embedded
`Executable Code
`
`Using Watermarking with Encoding
`
`Figure 7
`
`Page 12 of 18
`
`

`

`US 7,188,186 Bl
`
`1
`PROCESS OF AND SYSTEM FOR
`SEAMLESSLY EMBEDDING EXECUTABLE
`PROGRAM CODE INTO MEDIA FILE
`FORMATS SUCH AS MP3 AND THE LIKE
`FOR EXECUTION BY DIGITAL MEDIA
`PLAYER AND VIEWING SYSTEMS
`
`FIELD OF INVENTION
`
`The present invention relates to the field of digital com- 10
`puter systems as embodied in digital media playback appa(cid:173)
`ratus such as players for playing or permitting the viewing
`by playback of one or more of audio, video, still image, 3-D
`or other media formats of a wide variety of types, and, more
`particularly, to imbuing such systems with an extended 15
`capability to supplement their pre-prepared presentations
`with graphic, interactive and/or e-commerce content such as
`transactional advertising, interactive music videos and the
`like, and e-commerce generally, through novel techniques
`for seamlessly embedding executable program code repre- 20
`senting such supplementary content into the various types of
`pre-prepared media files that the various players or viewers
`may execute-the invention being particularly, though by no
`means exclusively, highly useful with "MP3" media formats
`and the like, later discussed.
`This is a Request for Continued Examination (RCE).
`
`BACKGROUND
`
`There has been extensive prior use of the general concept
`of embedding various types of information in video, audio,
`sound, text, and other multimedia formats. There are two
`main approaches that have heretofore been employed to
`achieve these ends:
`1) extending the format or creating a new format which 35
`contains that type of data; or
`
`2
`in empiricial data), U.S. Pat. No. 5,850,481 (embedded
`documents, but not for arbitrary data or computer code);
`U.S. Pat. No. 5,889,868 (digital watermarks in digital data),
`and U.S. Pat. No. 5,893,067 (echo data hiding in audio
`signals),
`Prior publications describing such techniques include
`Bender, W. D. Gruhl, M Morimoto and A. Lu, "Tech(cid:173)
`niques for data hiding", IBM Systems Journal, Vol. 35,
`Nos. 3 & 4, 1996, p. 313-336;
`MPEG Spec-ISO/IEC 11172, parts 1-3, Information
`Technology-Coding of moving pictures and associ(cid:173)
`ated audio for digital storage media at up to about 1.5
`Mbit/s, Copyright 1993, ISO/IEC, and
`ID3v2 spec
`A survey of techniques for multimedia data labeling and
`particularly for copyright labeling using watermarking
`through encoding low bit-rate information is presented by
`Langelaar, G. C. et al in "Copy Protection for Multimedia
`Data based on Labeling Techniques".
`Underlying the present invention, however, is a novel
`technique for embedding a set of executable program
`instructions at high bit rates into a media file, without
`substantially affecting the user's playback experience of the
`25 media, and wherein, unlike prior art techniques, sequences
`of executable code are embedded in audio, video, image or
`sound formats, such as, for example, entertainment music or
`video programs or the like, which have not been specifically
`pre-designed as a container for, or to contain, such execut-
`30 able codes.
`This supplemental embedding is done seamlessly and
`facilely, enabling supplementary graphic, interactive and/or
`e-commerce program content, such as the before-mentioned
`transactional and other advertising, interactive music videos,
`and e-commerce content, to be incorporated into the enter(cid:173)
`tainment or other media files for execution by players and
`viewers while playing back the original entertainment or
`other media file material.
`
`2) embedding the data, using techniques which allow the
`data to be recovered, but which do not affect the backwards
`compatibility of the format.
`Examples of the second type of approach arise frequently
`in communication and entertainment media; for example,
`the backwards compatibility of color-television broadcast,
`or the transmission of subtitles and other information
`embedded in a video signal. Another example of a very 45
`popular application is the encoding of identification infor(cid:173)
`mation in a media file so that it is robust to degradation, and
`transformation of the media file generally for purposes such
`as intellectual-property protection-often referred to as
`"watermarking". These techniques and others of similar 50
`character are directed, however, towards the embedding of
`relatively low bit-rate data, roughly on the order of22 binary
`digits (bits) of data per second. Such data typically consists
`of short and simple ASCII text or other unique identifiers. In
`another application, a control code is used for a computer 55
`system, to provide a very short signal control code as for
`preventing the computer from copying a copy-protected data
`file.
`Among prior patents illustrative of such and related
`techniques and uses are U.S. Pat. Nos. 4,379,947 (dealing 60
`with the transmitting of data simultaneously with audio);
`U.S. Pat. No. 5,185,800 (using bit allocation for transformed
`digital audio broadcasting signals with adaptive quantization
`based on psychoauditive criteria); U.S. Pat. No. 5,687,236
`(steganographic techniques), U.S. Pat. No. 5,710,834 (code 65
`signals conveyed through graphic images); U.S. Pat. No.
`5,832,119 (controlling systems by control signals embedded
`
`40 This technique has four main advantages:
`1) executable code may be placed directly in the media file,
`simplifying content distribution and permitting the data
`and the executable code to be tightly integrated;
`2) augmented viewers can transparently access the execut(cid:173)
`able code;
`3) existing viewers are backwards compatible and can still
`view the media file; and
`4) large amounts of supplemental data may be facilely
`embedded in the media file
`Using stegonographic techniques, moreover, the inven(cid:173)
`tion can embed data at very high bit rates. In one embodi(cid:173)
`ment later discussed, for example, more than 3000 bits of
`executable code data per second are embedded in an MP3
`audio file encoded at a bit-rate of 128,000 bits/sec. (later
`discussed in connection with Table 1 herein).
`
`OBJECTS OF INVENTION
`
`A principal object of the invention, accordingly, is to
`provide a new and improved process and system for seam(cid:173)
`lessly embedding executable program code into pre-pre(cid:173)
`pared media file formats for execution by digital media
`player and viewing apparatus or systems and the like, to
`provide supplementary content such as the before-men(cid:173)
`tioned transactional advertising, games, interactive music
`videos, e-commerce and the like, for execution and presen-
`
`Page 13 of 18
`
`

`

`US 7,188,186 Bl
`
`3
`tation by the digital player and viewing apparatus or sys(cid:173)
`tems, while continuing to present the original pre-prepared
`media file programs.
`A further object is to provide such a novel technique that
`is particularly useful for "MP3" applications and the like.
`Another object is to open up new methods of conducting
`business through enabling consumer and related digital
`media file or disc players and viewers and the like to present
`also supplemental transactional advertising and e-commerce
`functions and the like, as later more fully explained.
`Other and further objects will be explained hereinafter
`and are more particularly delineated in the appended claims.
`
`10
`
`4
`FIGS. 4 and 5 are similar diagrams directed respectively
`to using ancillary bits encoding and private stream encoding
`with the MPEG.
`FIG. 6 is a diagram showing the incorporation of the
`previously (and later) discussed steganographic techniques
`into the encoding, and
`FIG. 7 illustrates the use of the before-mentioned water(cid:173)
`marking processes with the encoding technique of the inven(cid:173)
`tion.
`
`DESCRIPTION OF PREFERRED
`EMBODIMENTS OF THE INVENTION
`
`SUMMARY
`
`In summary, however, from one of its broader points of
`view, the invention embraces a process for supplementing
`pre-prepared media digital file content to be performed by a
`digital playback apparatus, with supplemental digital pro(cid:173)
`gram content, that comprises, preparing such supplemental
`digital program content in the form of executable code; and
`embedding the executable code into the pre-prepared media
`file for execution by the playback apparatus supplementary
`to the playback of the pre-prepared media file content.
`From another viewpoint, the invention provides a system
`for flexibly adding supplemental digital program content to
`the playback of a pre-prepared media file by digital playback
`apparatus, comprising, means for modifying the pre-pre(cid:173)
`pared media digital file to embed sequences of executable
`code therein representing such supplemental program con(cid:173)
`tent; means provided in the digital playback apparatus for
`decoding the embedded code during playback of the modi(cid:173)
`fied media file at the digital playback apparatus; and, in
`addition to means for playing back of the pre-prepared
`content of the media file, means provided at the digital
`playback apparatus responsive to the decoding for also
`presenting thereat the supplemental program content.
`Preferred and best mode embodiments, designs, applica(cid:173)
`tions and implementations will be later fully described.
`
`DRAWINGS
`
`40
`
`Before describing the preferred implementations of the
`15 novel process and systems of the invention, it is believed
`necessary to define and provide illustrative examples of the
`various terms and system components involved.
`As earlier stated, and underlying the invention, is the
`overall concept of providing and novelly embedding in
`20 multi-media formats, executable code representing content
`supplemental to the present-day pre-prepared content of
`multi-media files (audio, video, still image, 3-D, combina(cid:173)
`tions thereof, or other media formats) that are provided to be
`played or viewed by digital computer systems or apparatus
`25 (such as portable music players, PDAs or personal digital
`assistants, digital televisions, car stereos, home audio sys(cid:173)
`tems, video walls, Web TV, console and portable digital
`game devices and the like). By seamlessly embedding such
`sequences of supplemental content executable code in the
`30 pre-prepared media file, the player or viewer apparatus, in
`decoding the code, can also present such supplemental or
`additional content (for example, ads, interactive music vid(cid:173)
`eos, e-commerce, games, polls, and contests etc., as earlier
`mentioned), while presenting the pre-prepared program of
`35 the media file-adding immeasurable additional information
`and facilities and significantly also increasing business and
`customer opportunities as a result thereof-in fact, a new
`method of doing or conducting such business at the player
`or viewer apparatus.
`Referring to FIGS. la, lb and le, for example, the
`addition of graphical, interactive and e-commerce-enabled
`content through the sequenced executable code embedding
`in the media file in accordance with present invention, is
`illustrated as presented upon exemplary well-known
`45 present-day widely distributed "MP3" audio players. The
`abbreviation "MP3" describes an MPEG layer III audio
`file-a standard audio file format in wide use today through(cid:173)
`out the world, as described in the published standard ISO/
`IEC 11172-3. The abbreviation MPEG stands for the audio
`and video compressed formats developed by the Motion
`Picture Experts Group, and formalized by ISO/IEC JTCl, as
`the before-mentioned International Standard ISO/IEC
`11172, parts 1-3. Part 1 addresses the System, Part 2 is
`Video; and Part 3 is Audio Encoding.
`In FIG. la, the embedding of a 468x60 pixel interactive
`bamier ad [the "United Colors Of Benetton"] into the MP3
`stream is shown displayed within a "Free Amp" type MP3
`player-adding media presentation to the player music [the
`"Paint It Black" song]. This enables advertiser-sponsored
`60 music files, using the music to sell products. The facility for
`promoting and marketing directly to whatever demographic
`may be desired is thus opened up by the invention, also
`enabling the subsidizing of music distribution as well.
`Similar opening-up of new avenues is provided in FIG.
`lb, illustrating the embedding of executable code (in this
`case, an interactive music video which responds to and is
`synchronized with the user's mouse movement and the beat
`
`The file of this patent contains at least one drawing
`executed in color. Copies of this patent with color
`drawing(s) will be provided by the Patent and Trademark
`Office upon request and payment of the necessary fee.
`The invention will now be described with reference to the
`accompanying drawings,
`FIGS. la, lb and le of which are pictorial diagrams 50
`illustrative of novel supplementary presentations achievable
`by the use of sequences of embedded executable code of the
`present invention, with consumer and other digital "MP3"
`players;
`FIG. 1 is a block and flow diagram showing an overview
`of the encoding process and system operating in accordance
`with a preferred embodiment of the invention,
`FIG. 2 is a similar diagram presenting an overview of the
`decoding of the media file embedded with the executable
`code created in FIG. 1, as executed by the media player or
`viewer;
`FIG. 3 is again a diagram similar to FIG. 1, but detailed
`to show encoding for the illustrative specific example of
`MPEG audio (discussed in the earlier-referenced "MPEG
`Spec" publication), with id3v2 encoding (discussed in the
`previously cited "ID3v2 spec");
`
`55
`
`65
`
`Page 14 of 18
`
`

`

`US 7,188,186 Bl
`
`5
`of the music, as illustrated by the dancing figures), within a
`well-known commercial "Sonique" MP3 player.
`As a third example of the new business solicitation
`opportunities afforded by the present invention, FIG. le
`illustrates the embedding of executable code (in this case, an
`e-commerce application with a graphical user interface),
`allowing the user to buy tickets (so-labeled) or a CD, within
`another well-known commercial version of the Sonique
`MP3 player. This opens up a new and previously unavailable
`avenue of embedded point-of-purchase sales, enabling fully 10
`transactional ads and merchandising directly to the listener
`at the playback apparatus.
`Further in accordance with the novel technique of the
`invention, the embedding of this executable code in the
`media file is seamlessly effected in such a way, as later 15
`explained, that compatible players/viewers are able to
`extract the executable code and perform operations based on
`it, while incompatible players/viewers are still able to play
`the media file as if there were no additional embedded
`information. Currently, none of these players of the type
`shown in FIGS. la, lb and le support displays such as
`illustrated; they only allow for the extraction oflimited data
`such as song titles and lyrics information from the MP3
`stream, and for separately downloaded code ("plugins")
`which can be executed while the MP3 is playing. They do
`not allow for the embedding of sequences of executable
`code in the MP3 stream as is effected by the present
`invention.
`Examples of the types of media files susceptible to
`operating with the embedded executable code sequence
`programs of the invention include, but are not limited to, the
`previously enumerated audio, video, still image, 3-D, or a
`combination of these or other media formats. Among these
`are MP3, SDMI, CD audio, AIFF, AU, WAY, equipment
`offered under the trademarks RealAudio, and Quicktime, 35
`MPEG, AVI, JPEG, JFIF, GIF, PNG, TIFF, DXF, or VRML.
`Among the types of executable code programs which may
`be embedded into such media files are Java files. Equip(cid:173)
`ments marketed under the trademarks Macromedia Director,
`Shockwave or Flash, Perl. VRML, TCL, Visual Basic, 40
`machine code, byte codes, any archive format such as cab,
`jar or zip; or any combination of any of these programs with
`any non-executable media, including but not limited to
`image, audio, 3-D, or text. The content may be, but is not
`limited to, advertising, as previously mentioned, entertain- 45
`ment, utilities, applications, education, design, interactive
`advertising, transactional merchandising, or interactive con(cid:173)
`tent such as music, video games, polls and contests, and the
`like.
`It is now in order to explain how the encoding and code
`embedding in the pre-prepared media file may be imple(cid:173)
`mented in accordance with the invention. Reference is
`accordingly made to FIG. 1 wherein a pre-prepared media
`file, so-labeled, (audio, image, video, 3-D, database, or other
`multimedia data as before mentioned), and a predetermined
`prepared executable code sequence (such as the previously
`mentioned computer programs marketed under the trade(cid:173)
`marks Java class files, Macromedia Shockwave and Flash,
`binary executable, byte codes, Visual Basic, Java Script,
`etc.) are shown fed to a encoding processor for embedding 60
`the code sequences, such as, for example, of the encoding
`types later described. Well-known encoding processes may
`be used depending on the media file format and any well(cid:173)
`known encoding processes may be used depending on the
`media file format and any well-known encoding processes
`may be used depending on the media file format and any
`well-known compression techniques for the media file to be
`
`6
`created, as later discussed. There then results a modified
`media file with embedded executable code without affecting
`its backwards compatibility with existing file formats, and
`without substantially affecting the user's experience of play(cid:173)
`back of the pre-prepared media file content.
`It is to be noted, moreover, that the invention embeds
`sequences of executable code representing content supple(cid:173)
`mental to the pre-prepared content of the media file, into
`media formats that have not been specifically predesigned as
`a container for, or to contain any such executable code. The
`code is placed directly into the media file, simplifying
`content distribution and permitting the data and executable
`code to be tightly integrated as earlier noted.
`At the player or viewer playback apparatus, FIG. 2, the
`media file with the embedded executable code is shown fed
`to a decoding processor, such as later described. The original
`pre-prepared media file content [the "Paint It Black" song of
`FIG. la, for example]is usually unchanged in the decoding
`process because it may often be unfeasible to remove the
`20 executable code and because doing so would not typically
`improve the user's playback experience, and is shown
`communicating on the left-hand side of FIG. 2 with the
`media player apparatus of the playback environment. The
`decoded executable code [representing the "United Colors
`25 Of Benetton" display content of FIG. la, for example],
`bitwise identical to the original executable code encoded in
`FIG. 1, is fed (right-hand side in FIG. 2) to the execution part
`of the playback environment. As an optional feature, a
`well-known checksum or digital signature is shown usable at
`30 "Verification Process", to verify that the code is unchanged
`before it is executed.
`Where desired, moreover, the execution of the executable
`code [the "United Colors Of Benetton" display above dis(cid:173)
`cussed, for example] may be synchronized by well-known
`communication with the player playback of the media file
`[the song "Paint It Black" or a selected portion thereof], as
`schematically shown by the legend "SYNC".
`As a more specific example, FIG. 3 diagrams the use of
`the before-described MPEG audio file into which the execut(cid:173)
`able code is to be embedded. Reference is again made to the
`earlier cited "MPEG Spec" and "ID3v2 Spec" publica(cid:173)
`tions--the MPEG audio file being (though not limited to)
`an MPEG-I, MPEG-2, or MPEG-2.5 file, to be encoded
`using the Layer I, II, or III encodings. As in FIG. 1, the
`executable code, so-labeled in FIG. 3, may be in any kind of
`computer programs such as Java trademark) class files,
`Macromedia Shock wave and Flash, (trademark) binary
`executables, byte codes, Visual Basic, Java Script trade(cid:173)
`mark) etc. The executable code is first shown unsynchro-
`50 nized by modifying any consecutive bytes of form %1
`11111111 lllxxxxx so that they do not resemble a synchro(cid:173)
`nization byte in the MPEG audio stream. The code is then
`encapsulated in ID3v2 format, and inserted in the encoding
`process, as shown, at the beginning of the MP3 audio stream
`55 as an ID3v2 tag. This results in an MPEG audio file with the
`embedded executable code, backwards compatible but
`slightly larger to accommodate the embedded code, and with
`the audio data unaffected, and any occurrences of the MPEG
`sync signal taken care of by the unsynchronization scheme.
`More specifically, a preferred encoding system will now
`be detailed for such an MPEG audio stream, particularly
`with reference to the before cited publicly available
`ID3v2.3.0 specification ("ID3v2"). There are a number of
`existing content content types described in the ID3v2 spec,
`65 there named "frames". Frames are defined primarily for the
`ASCII text data such as song titles and lyrics, or for a still
`image to be embedded in the MPEG audio file. In this
`
`Page 15 of 18
`
`

`

`US 7,188,186 Bl
`
`8
`The execution environment is then invoked to begin
`execution of the executable code simultaneously with the
`playback of the audio file. Additional Application Program(cid:173)
`ming Interfaces (APis) may be defined with reference to the
`execution environment to control the exact behavior of the
`execution environment relative to the audio file while play(cid:173)
`back is occurring.
`Specific references for fuller details of the above-ex(cid:173)
`plained techniques usable in the encoding and decoding
`process components of the invention, are:
`
`[IS0-8859-1] ISO/IEC DIS 8859-1.
`
`8-bit single-byte coded character sets, Part 1: Latin alphabet
`No. 1. Technical committee/subcommittee JTC 1/SC 2;
`
`[MIME] Freed, N. and N. Borenstein, "Multipurpose Inter-
`net Mail Extensions (MIME)
`
`45
`
`7
`description, we create and define a new type of 1D3 frame,
`named "EXEC", which is designed as a container for
`executable content, generally intended to be executed while
`the audio is playing.
`In the following description, the notation $xx is used to 5
`refer to a hexadecimal-encoded octet, e.g., $00 represents
`the eight binary digits 00000000. The first four octets ofthis
`frame are the IS0-8859-1 (ASCI 1) characters "EXEC".
`This, in tum, is followed by a four-octet Size header and a
`two-octet Flags header, as described in the ID3v2 specifi- 10
`cation. This is followed by $00, to represent the use of
`IS0-8859-1 text encoding within the frame, or by $01, to
`represent the use ofISO/IEC 10646-1 (Unicode) text encod(cid:173)
`ing, later referenced. The next element is the ISO 8859-1
`encoded MIME type of the executable content, as described 15
`in IETF RFC 2045, also later referenced, followed by $00.
`For example, the MIME type of the before-mentioned
`Part One: Format of Internet Message Bodies", RFC 2045,
`Macromedia Flash files is "application/x-shockwave-flash".
`November 1996; and
`This is followed by a description of the executable code in
`the text encoding defined for that frame, terminated by $00 20 [UNICODE] ISO/IEC 10646-1:1993.
`if the encoding is ASCI 1, or $00 $00 if the encoding is
`Unicode. This is followed by a single octet, which is $00 if
`there is no checksum, or $01 if there is a checksum, as
`described in connection with FIG. 2. If there is a checksum,
`this is followed by a checksum created by summing the 25
`octets of the executable code together, and taking the result
`modulo 256. This is useful because this lets the executable
`code be examined before executing, to reduce the possibility
`that there have been transmission errors that might cause
`erroneous instructions to be executed.
`The final element is the binary or ASC I 1 data of the
`executable code to be embedded in the MPEG data stream.
`As earlier explained, in FIG. 3, it is necessary to perform the
`"unsynchronization" step within the executable code, where
`any occurrences of the bit sequence 11111111 lllxxxxx,
`where "x" represents a bit which may either be 0 or 1, must
`be replaced. This is because MPEG players use such a bit
`sequence to identify the beginning of the audio data, and
`would otherwise begin interpreting the executable code as
`audio data. In the unsynchronization technique defined in
`the ID3v2 specification, all such sequences are replaced with
`the sequence: 11111111 00000000 lllxxxxx. Additionally,
`all sequences of the form $FF $00 are replaced with the
`sequence $FF $00 $00.
`The EXEC frame of the invention just described is then
`embedded in the ID3v2 tag as described in the ID3v2
`specification. The resulting ID3v2 tag is then placed at the
`beginning of the MPEG audio stream, FIG. 3, using the
`standard techniques described in the ID3v2 specification.
`In the preferred embodiment of the invention, the decod(cid:173)
`ing process is a simple reversal of the encoding process
`above-detailed. The ID3v2 tag is extracted from the begin(cid:173)
`ning of the MPEG audio stream, and the relevant data is
`retrieved from the EXEC frame. The unsynchronization step 55
`in the executable code is then reversed, replacing all occur(cid:173)
`rences of $FF 00 XX by the sequence $FF XX, where XX
`represents any binary octets. If there is a checksum encoded
`in the EXEC frame, the octets of the executable code are
`sUlllilled, the result taken modulo 256, and co