(12) United States Patent
`Chang et al.
`
`USOO684.8004B1
`(10) Patent No.:
`US 6,848,004 B1
`(45) Date of Patent:
`Jan. 25, 2005
`
`(54) SYSTEM AND METHOD FOR ADAPTIVE
`DELIVERY OF RICH MEDIA CONTENT TO
`A USER IN A NETWORK BASED ON REAL
`TIME BANDWIDTH MEASUREMENT &
`SESSNERDING TO AVAILABLE
`
`(75) Inventors: shit styleEppan,
`Chappaqua, NY (US); Keeranoor G.
`Kumar, Randolph, NJ (US); James S.
`Lipscomb, Yorktown Heights, NY
`(US); Jai Menon, Croton-on-Hudson,
`NY (US); Liang-Jie Zhang, White
`Plains, NY (US)
`
`(73) Assignee: International Business Machines
`Corporation, Armonk, NY (US)
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 0 days.
`
`(*) Notice:
`
`(21) Appl. No.: 09/447,585
`(22) Filed:
`Nov. 23, 1999
`7
`(51) Int. Cl
`(52) U.S. Cl. ....................... 709,232,709,224,709:21;
`370/232
`(58) Field of Search ................................. 709,231, 226,
`709/225, 224, 228, 219, 223, 230, 232,
`233,234, 235; 370/229, 230, 231, 477,
`431, 232, 233,234; 719/315, 316, 332
`References Cited
`
`G06F 15/16
`
`(56)
`
`U.S. PATENT DOCUMENTS
`
`5,737,619 A 4/1998 Judson
`5.841,978 A 11/1998 Rhoads
`5,884,037 A
`3/1999 Aras et al. .................. 709/226
`5,918,012 A 6/1999 Astiz et al.
`6,272.539 B1 * 8/2001 Cuomo et al. .............. 709/223
`6,292.834 B1
`9/2001 Ravi et al. .................. 709/233
`6,363,056 B1 * 3/2002 Beigi et al. ...........
`... 370/252
`6,377,989 B1 * 4/2002 Fukasawa et al. .......... 709/224
`2003/0101263 A1
`5/2003 Bouillet et al. ............. 709/225
`
`* cited by examiner
`
`Primary Examiner William C. Vaughn, Jr.
`74) Attorney, Agent, or Firm-David M. Shofi
`y, Ag
`(57)
`ABSTRACT
`A network implementing HotMedia architecture provides
`real time measurement and prediction of bandwidth for
`adaptive content delivery of rich media according to avail
`able user bandwidth. The rich media file created at the
`creation Station is transmitted by the delivery Station to the
`client Station in Steaming Java applets having a framework
`including a thumbnail frame and a media frame. In real
`includi
`humbnail f
`d
`dia f
`I
`1
`E. R lists R. t ity R
`ava appel, predicts une available bandwidin Ior next media
`delivery based on the detected bandwidth and fetches the
`corresponding Sensitive bandwidth media fame from a web
`Server according to a linear or non-linear prediction algo
`rithm resulting in bandwidth Sensitive rich media content
`elivery from the delivery Station to the client Station for
`deli
`f
`he deli
`he cli
`ion f
`display and user interaction.
`
`5,548,761. A * 8/1996 Balasundaram et al. ... 717/159
`
`17 Claims, 3 Drawing Sheets
`
`THEANAILABLE
`BANDWIDTH
`st (p).(23)
`
`
`
`
`
`SEND
`B TO
`DECISION
`MAKER
`
`NO
`
`D
`ON SERVER
`
`300 /
`
`307
`
`AVAILABLE
`BANDDT
`PREDICATION 1
`(=3)
`
`B
`
`35
`
`B
`
`325
`AVAILABLE
`BANOWIOT
`PREDCATION 2
`
`529
`
`531
`
`30
`
`START
`
`APPLE IN
`
`LOAD THUMBNAIL SET i = 0
`THE DETECTED BANDWIDTH B1 (=1)
`RICH MEDLA IMPRESSION
`
`OAD hmnaster.class
`THE DETECTED BANDWIDTH B2 (-2)
`himmaster RUNNING
`333
`
`327
`
`DECISION MAKER
`- Hotelio MASTER
`- ACON HANDER
`SEND HTTP REQUEST TO SERVER
`RECEIVE THE APPROPRIATE
`RICH MEDIA CONTEN
`THE DETECTED BANDWIDTH B (x3)
`PLAYBACKRCH MEDIA CONTENT
`
`YES
`SEND B TO DECISION MAKER
`
`DECISION MAKER ON THE SERVER
`
`303
`
`305
`
`309
`
`31
`
`33
`
`37
`
`319
`
`32
`
`Amazon / WAG Acquisition
`Exhibit 1020
`Page 1
`
`

`

`U.S. Patent
`
`Jan. 25, 2005
`
`Sheet 1 of 3
`
`US 6,848,004 B1
`
`-------------------------;;--------
`
`
`
`
`
`- - - - - - -
`
`-a
`
`a
`
`m or
`
`e
`
`m to
`
`up or
`
`- - - - - -
`
`- - - - - - - - - - - - - - - - - - -NOIIVIS NOll?}}) INHINO)
`
`| | | + | } } | | | | | ZZ-| | | | | | } | | } | |
`
`
`
`
`
`+------ - - - - - - ------------------ • • • --------+
`
`r- - - - - ~- - - - - - - - - - - - - - - - - - - - - - - - - - - - ~~~~ -.
`
`
`
`- - - - - - - - - -_________NOIIVIS_A}}\T?0 INHINO) ;
`
`T??
`
`Amazon / WAG Acquisition
`Exhibit 1020
`Page 2
`
`

`

`U.S. Patent
`
`Jan. 25, 2005
`
`Sheet 2 of 3
`
`US 6,848,004 B1
`
`3
`
`
`
`
`
`
`
`
`
`
`
`2 (f) I „H
`
`3000
`[!?
`
`Amazon / WAG Acquisition
`Exhibit 1020
`Page 3
`
`

`

`U.S. Patent
`
`Jan. 25, 2005
`
`Sheet 3 of 3
`
`US 6,848,004 B1
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`Amazon / WAG Acquisition
`Exhibit 1020
`Page 4
`
`

`

`US 6,848,004 B1
`
`1
`SYSTEMAND METHOD FOR ADAPTIVE
`DELIVERY OF RICH MEDIA CONTENT TO
`A USER IN A NETWORK BASED ON REAL
`TIME BANDWDTH MEASUREMENT &
`PREDICTION ACCORDING TO AVAILABLE
`USER BANDWIDTH
`
`2
`ing in multiple media type contents. At the core of HotMedia
`client is a Smart content algorithm that inferS media types
`from the incoming data Stream and fetches the media
`renderer components, user-interface components and hyper
`linked action components, all just in time, resulting in
`progressive and context driven enrichment of the user expe
`rience. Further details related to HotMedia architecture are
`decribed in U.S. Pat. No. 6,611,812 entitled “Framework For
`Progressive Hierarchical and Adaptive Delivery Rich Media
`Presentation and ASSociated Meta Data”, issued Aug. 26,
`2003 (SE9-98-033), supra.
`Often instances of rich media incorporate links to other
`presentations to expand a user experience. The process of
`clicking on a link in a media for transfer to the other
`presentation is referred to as "hot linking * or “hyper
`linking” which is further described in U.S. Pat. No. 5,841,
`978 entitled “Networking Using Steganographically Embed
`ded Data Objects” issued Nov. 24, 1998 and U.S. Pat. No.
`5,918,012 entitled “Hyper Linking Time Based Data' issued
`Jan. 29, 1999. The real power of “HotMedia” is in how the
`experience is delivered to the user. A dynamic content
`delivery System to a user should take into account the user's
`connection Speed to enhance the user experience. For
`example, when a user has experienced reduced bandwidth
`because of having heavy CPU usage, phone line noise or
`other web browser instances, the user could receive Seam
`lessly adjusted web content corresponding to the reduced
`bandwidth. LAN users might get higher quality Video clips
`or audio clips than T1 users, with 56K modem connection
`user's Surfing higher quality images than 28.8K modem
`users. An adaptive content delivery System according to user
`connection speed or bandwidth should have the following
`design goals:
`1. An adaptive content delivery framework that expands
`the Scope of information that can be adaptively delivered.
`2. A content creation phase that enables individual speci
`fication and customizability of information to be adapted to
`the delivery for every bandwidth sensitive content context.
`3. Adaptive content delivery Servers requiring neither
`change to the Server code nor changes to content for Seam
`leSS interfacing.
`4. A cascading delivery Structure for efficient content
`delivery as used in the HotMedia architecture.
`5. A content delivery system which provides for adaptive
`delivery disablement for the content.
`Prior art related to multimedia delivery systems include
`the following:
`U.S. Pat. No. 5,737,619 entitled “World-Wide Web
`Browsing with Content Delivery Over An Idle Connection
`& Interstitial Content Display” by D. H. Judson, discloses a
`personal computer Supporting a graphical user interface and
`an Internet browser. The computer locally Stores retrieves
`and outputs information objects to reduce waiting time
`normally associated with downloaded hypertext documents
`having high resolution graphics. In one embodiment, a web
`page is displayed on the graphical user interface and has at
`least one link to a hypertext document, preferably located at
`a remote server. When the user clicks on the link, the link is
`activated by the browser to request downloading of the
`hypertext documents from the remote Server to the graphical
`user interface. While the user waits for a reply and/or as the
`hypertext document is being downloaded, the browser dis
`plays a previously cached information object.
`An Internet program “Speed Select” from Match Logic of
`Excite (http://www.matchlogic.com) makes use of a user's
`historical profile Saved over time to predict a user's con
`
`25
`
`35
`
`40
`
`15
`
`RELATED APPLICATIONS
`1. U.S. Pat. No. 6,356,921, issued Mar. 21, 2002, entitled
`“Framework for Progressive Hierarchical and Adaptive
`Delivery Rich Media Presentations and Associated Meta
`Data” (SE9-98-030V).
`2. U.S. Pat. No. 6,448,980, issued Sep. 10, 2002, entitled
`“Personalizing Rich Media Presentations Based On User
`Response to the Presentation” (SE9-98-028V).
`3. U.S. Ser. No. 09/100,419, filed Jun. 20, 1998, entitled
`“Progressive Interleaved Delivery of Interactive Descrip
`tions and Renderers for Electronic Publishing of Merchan
`dise” (SE9-98-004).
`4. U.S. Ser. No. 10/392,055, filed Mar. 19, 2003, entitled
`“System and Method for Tracking User Interactions and
`Navigation During Rich Media Presentations” (SE999-006/
`1963-7334US1).
`5. U.S. Ser. No. 09/438,493, filed Nov. 12, 1999, entitled
`“A System and Method of Enriching Non-Linkable Media
`Representations In A Network By Enabling An Overlying
`Hotlink Canvas” (SE9-99-012/1963-7341).
`All of the above applications have been assigned to the
`ASSignee of the present invention and are fully incorporated
`herein by reference.
`BACKGROUND OF INVENTION
`1. Field of Invention
`This invention relates to multimedia network, Systems
`and methods of operation. More particularly, the invention
`relates to Systems and methods for adaptive delivery of rich
`media content to a user in a network based on real time
`bandwidth measurement and prediction, according to avail
`able user bandwidth.
`2. Description of Prior Art
`In recent years, there has been a sizeable growth in the use
`of rich media over the World Wide Web as more and more
`individuals and institutions are beginning to realize the
`web's potential in a broad range of applications including
`electronic commerce, education training, news, etc.
`Examples of rich media include animation, audio, 3-D,
`panoramas, and Videos. There are two apparent clusters of
`rich media technology. One at the low end and the other at
`the high end, which hamperS effective deployment of Some
`rich media in Internet based applications. The “low end”
`cluster comprises Static images and Simple non-interactive
`animations (typically animated GIFs) which are easy to
`deploy and therefore have widespread usage. The "high end’
`clusters compriseS richer and more natural experiences with
`larger interactivities, Such as panoramas, 3-D, Streaming
`audio/video, and composite media (e.g., MPEG-4), but the
`difficulty of deployment has limited widespread usage. A
`novel scaleable architecture called HotMedia bridges the
`60
`gap between the two clusters thereby achieving widespread
`web penetration. A key feature of the technology is a Suitable
`delivery file format that can contain heterogeneous compo
`Sitions of media bit Streams as well as meta data that defines
`behavior, composition and interaction Semantics. The deliv
`ery file format enables the creation of lightweight Single file
`representation of interactive, multistage presentations result
`
`45
`
`50
`
`55
`
`65
`
`Amazon / WAG Acquisition
`Exhibit 1020
`Page 5
`
`

`

`US 6,848,004 B1
`
`3
`nection Speed at any time, based on measurements taken at
`Similar times of the day.
`None of the prior art discloses a generalized adaptive rich
`media content delivery framework based on real time band
`width measurements, prediction and dynamic content deliv
`ery at the user So that advertisers or content providers can
`ensure that all times users receive bandwidth-Sensitive con
`tent.
`
`4
`coupled to a network implementing HotMedia architecture
`and incorporating the principles of the present invention.
`FIG. 2 is a representation of a HotMedia file format
`comprising a sequence of frames for transmitting rich media
`in the system of FIG. 1.
`FIG. 3 is a flow diagram for real time measurement and
`prediction of bandwidth to the client Station according to the
`available client bandwidth.
`
`SUMMARY OF THE INVENTION
`An object of the invention is a multimedia network
`adaptively delivering rich media to a client Station according
`to available user bandwidth in the network.
`Another object is an adaptive delivery System for rich
`media content to a client Station in a multimedia network
`based on bandwidth measurement and prediction according
`to available user bandwidth.
`Another object is an adaptive delivery System for rich
`media content in a network which enables a content creator
`to specify presentation events or user interactions that trig
`ger adaptive content delivery during presentation of rich
`media content to a client Station.
`Another object is an adaptive delivery System for rich
`media content to a client Station in a network that takes
`advantage of dynamic Java class loading mechanism and
`progressive rich media delivery mechanism to perform real
`time bandwidth measurement and prediction in the network.
`Another object is an adaptive delivery System and method
`of operation for rich media content to a client Station in a
`network which provides Seamless interfacing with multiple
`adaptive content delivery Servers requiring neither changes
`to Server code nor changes to content.
`These and other objects, features and advantages are
`achieved in a network implementing HotMedia architecture
`including a content creation Station and a content delivery
`Station coupled to a client Station via the network for
`delivery of rich media to the client station for display and
`interaction with a user. The rich media file created at the
`creation Station is transmitted by the delivery Station to the
`client Station in Streaming Java applets having a framework
`including a header frame, a thumbnail frame, a media frame,
`a meta frame, and an end-of-Stream frame. The content
`creation Station enables a creator to Specify presentation
`events or user interactions that trigger an adaptive content
`delivery during presentation of the rich media content. The
`client Station takes advantage of the dynamic Java class
`loading mechanism and progressive rich media mechanism
`to do real time bandwidth measurement and prediction. The
`delivery station monitors available user bandwidth derived
`from the client Station and calibrates a rich media delivery,
`fine-tuned to each user capability. In real-time, the client
`station detects the bandwidth used by the Java applet;
`predicts the bandwidth for next media delivery based on the
`detected bandwidth and fetches the corresponding Sensitive
`bandwidth media frame from a web server according to an
`algorithm resulting in bandwidth Sensitive rich media con
`tent delivery from the delivery station to the client station for
`display and user interaction.
`DESCRIPTION OF DRAWING
`The invention will be further understood from the fol
`lowing detailed description of a preferred embodiment taken
`in conjunction with an appended drawing, in which:
`FIG. 1 is a System diagram of a content creator Station,
`content delivery Station, client Station for user interaction,
`
`15
`
`25
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`DESCRIPTION OF PREFERRED EMBODIMENT
`In FIG. 1, a system 10 for real time bandwidth measure
`ment and prediction according to a user's available band
`width includes a content creation Station 20 and a content
`delivery station 30 coupled to a network 40, typically a
`distributed information System, e.g., the Internet capable of
`handling rich media. The network further includes a client
`station 50 which enables a user 45 to interact with the rich
`media to provide inputs to the client Station for real time
`bandwidth measurement and prediction. The delivery station
`30 delivers appropriate rich media content to the client
`station 50. A tracking server 42 is coupled to the network 40
`for tracking the user interactions with rich media file
`received by the client Station.
`The content creation station 20 includes a standard CPU
`21 coupled to a hard disk 22; a display 23, an input/output
`terminal 24 and a Standard random acceSS memory 25.
`Contained in the disk 22 are Software for Station operation
`26, e.g. Windows 95 and an authoring tool for a creator (not
`shown) to create rich media in a HotMedia file format
`supported by the HotMedia Architecture as will be described
`in conjunction with FIG. 2. Authoring tools are well known
`in the art as evidenced by U.S. Pat. No. 5,801,687 entitled
`“Authoring Tool Comprising Nested State Machines For
`Use In A Computer System”, issued Sep. 1, 1998. Content
`creation tables enable a creator to specify presentation
`events or user interaction to trigger an adaptive content
`delivery during presentation of rich media contact. For
`example, when the creator is creating a hot link to different
`rich media file from within one, the creator will be presented
`a check box to indicate whether the execution of the link
`should be adaptively delivered adapting to the predicted
`available bandwidth and Specify the detailed mapping infor
`mation. Accordingly in the offering phase, the content
`creator can choose which delivery is preferred. In general,
`the content creator can create different versions of rich
`media content corresponding to different network connec
`tions. The different versions can be done via multiple files or
`one Single file. All dynamic content delivery options are
`saved in a HotMedia file.
`The content delivery station 30 similarly contains a stan
`dard CPU 31 coupled to a hard disk 32, an input/output
`terminal 33, and a Standard random access memory 34.
`Contained in the disk 32 are a Station operating System 35,
`e.g. Windows NT, web server software 36 for delivering
`over the network 40, a HotMedia file(s) 37 created by the
`authoring tool through Java applets. An adaptive delivery
`CGI (Common Gateway Interface) file 38 including a deliv
`ery logic (not shown) and decision maker software 39 for
`delivering bandwidth-sensitive media content from the web
`Server 36.
`The client station includes a standard CPU 51 coupled to
`a hard disk 52; a display 53, an input/output terminal 54, and
`a standard random access memory 55. Further details on the
`client station are described in U.S. Ser. No. 10/392,055, filed
`Jun. 20, 1998, entitled “Progressive Interleaved Delivery of
`Interactive Descriptions and Renderers for Electronic Pub
`
`Amazon / WAG Acquisition
`Exhibit 1020
`Page 6
`
`

`

`US 6,848,004 B1
`
`15
`
`S
`lishing of Merchandise” (SE9-98-004), Supra. Contained in
`the disk 52 are software for station operation 56, e.g.
`Windows 95, a web browser 57; Software 58 for content
`presentation including Hot Media kernel code and decision
`maker Software 59 for choosing bandwidth-sensitive media
`content from the web server 36.
`The client Station takes full advantage of dynamic Java
`class loading mechanism and progressive rich media deliv
`ery to perform real time bandwidth measurement and pre
`diction. The client station monitors available user bandwidth
`and calibrates itself to provide a web browsing experience
`fine-tuned to each user's capability.
`Having described the network and System of the present
`invention, a description of the HotMedia file format is
`believed appropriate for delivery of a rich media presenta
`tion to a user(s) in which the file format contributes to the
`real time measurement and prediction of bandwidth of the
`rich media presentation according to available user band
`width.
`FIG. 2 shows a HotMedia presentation file 200 in a
`framework 201 which is essentially a sequence of frames
`types comprising header 202, thumbnail 204, meta 206,
`media 208, and an end of stream 212. The first frame is the
`header frame 202. The header frame is actually followed by
`25
`the thumbnail frame 204. After these frames a sequence of
`other frames occurs and in no pre-mandated order or num
`ber. The header frame 202 is the only one whose presence is
`mandatory in the format. The most degenerate yet useful
`version would have a header frame followed by a thumbnail
`frame 204. Media frames 208 appear in all other instances.
`Meta frames 206 are present only in cases where non-default
`behavior and configuration are desired on cases where hyper
`linked action Semantics are to be incorporated. The infor
`mation for adaptive delivery option is Stored in Meta frames
`206. All frames have a similar 12 bit initial section 213 that
`enables a uniform procedure for their identification to a type
`and frame label as well as the determination of their size.
`The HotMedia file format is created to have minimum
`overhead and maximum modularity. The format makes it
`suitable for optimal delivery of a low bandwidth as well as
`for rich experiences over high bandwidth.
`The header frame 202 includes a frame header 213 which
`provides information about frame size 214, types and flags
`216 and frame labels 218. The header frame 202 also
`includes media information 220 containing definition on
`different media tracks that may be contained in the file.
`General information 224 in the header frame 202 provides
`the class name of the code that renders the media type on the
`client station. When HotMedia kernel code in the content
`presntation software in CPU 51 parses the media track
`information on the file header and if it encounters the
`Specification of a class name, the media type field is ignored
`and the code Specified by the class name is Sent from the
`delivery server 30 in order to render the data of the track in
`question. The class name permits the introduction of whole
`new media types into the presentation. No changes or
`additions to the client kernel code are needed So long as the
`media objects implements the interfaces Specified in the
`HotMedia framework 201 for first class members of a media
`object Set.
`The thumbnail frames 204 carry a minimalistic represen
`tation of the information shared by the HotMedia file
`framework 201. For example, if the information carried in
`the file were a description of merchandise, the thumbnail
`would carry a single image. Thumbnail frames make poS
`Sible providing the user a quick introduction to the Subject
`
`35
`
`6
`with minimal code and data transfer. Since users pay atten
`tion to detail only when the Subject is of Some interest, it
`would be wasteful for Such data and codes to enable any
`thing beyond the initial experience unless an interest is
`indicated. In addition to a frame header, the thumbnail
`frames also carry parameters for producing image pan and
`animation effects on the single image (all not shown).
`Meta frames 206 carry specification of non-media specific
`information pertaining to enablement of hyper linked actions
`from media context, tracking of interaction and adaptive
`delivery of rich media content. Action Semantics associated
`with media Segments can be expressed in HotMedia files
`200 using range 205 and action Subtypes 207. Range frames
`205 are made up of range records that identify and label
`Spatio-temporal Segments of media. A Spatial range can be
`described by a rectangular or elliptical contour. Temporal
`ranges can refer to a time range in milliseconds for a real
`time media-like audio or a count range for non-real-time
`media like animation. The range specification can also
`express a combination of Space and time.
`Action records 207 may be pre-programmed into the
`HotMedia file 200 for the execution of many types of action
`in the context of a presentation. Action frames can carry
`multiple action records all pertaining to a Single actionable
`context, namely the presentation of media Segment as Speci
`fied by a range. Action records are the Specification mecha
`nism for all types of actions that can be triggered during the
`context of presenting a certain media Segment while in a
`certain State of the media object. Actions can belong to a
`variety of types that include LINK to specify hyperlinking,
`DISPLAY used to specify the display of contours, cursor
`changes, etc; PLAY SOUND used to create sound effects in
`various contexts, TRACK used to Specify the tracking of
`interactions and events associated with a user experience;
`and ADAPT DELIVERY used to specify the dynamic deliv
`ery of rich media content adapting to the available user
`bandwidth. Also contained in an action record is the Speci
`fication of an action trigger. The triggerS for actions can be
`user interaction, presentation State updates or player State
`updates. An example of would be the coming in to view of
`a certain Segment of a presentation, the movement of mouse
`into a Spatio-temporal range of a presentation, the click of
`the left mouse button within the present temporal range of
`the presentation, the completion of loading of a Segment of
`the media data and So forth. Although the range record
`Specifying a media Segment associated with most action
`triggers are of type spatial, temporal or both there are Some
`Situations where the range type is inconsequential. A range
`type DONTCARE is designed for such situations. An
`example is where the action is determined entirely by the
`internal State of the media object producing the completion
`of a data loading trigger. The final piece of information
`contained in an action record is the data for action. The size
`of this data and its interpretation is different for each action
`type and Subtype. The data information is where for
`example, the target URL String is found when the action type
`is LINK and the color information when the action type is
`DISPLAY.
`The media frame 208 contains media bit stream data 209,
`initialization data 210, behavior data 215 and code 217. The
`behavior data or initialization data define the page and
`configuration of a corresponding media player respectively.
`Media bit Stream data belonging to a particular media track
`and can be carried over multiple frames and these frames can
`be interlinked with meta frames and frames belonging to
`other media tracks. Media frames are identified by frame
`type and track identifier as carried in their header 213. The
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`Amazon / WAG Acquisition
`Exhibit 1020
`Page 7
`
`

`

`15
`
`7
`frame type and track identifier enable true multiplexed
`delivery and hence ideal for Static or dynamic proportioning
`of various media tracks over limited bandwidth. Media
`frames can also often contain codes 217 that render the
`media type on the client Station. The code in turn can be
`physically present or could be a URL reference. When
`present, the code is separated from the file on the Server Side
`30 (see FIG. 1), and thereafter the code or data can be made
`available for on demand delivery. In the case of a URL
`reference to code, the code is fetched from a remote HTTP
`1O
`Server to be similarly made available on demand.
`The end-of-Stream frame 212 is a marker that signals the
`end of the entire presentation associated with the HotMedia
`file 200. Frames belonging to multiple media tracks and
`meta data can be multiplexed in the HotMedia file. The end
`of Stream frame is valuable to the client Station in recog
`nizing the end of each HotMedia file data stream which may
`otherwise be impossible when Stream initialization data for
`each Stream do not announce their respective lengths.
`Further details on the HotMedia file format are described
`in IBM Research Report RC 21519.97069 entitled “The
`HotMedia Architecture: Progressive and Interactive Media
`for the Internet”, dated July 1999, available from IBM
`Research, Yorktown, N.Y. 10598, and fully incorporated
`herein by reference.
`After creation, the files 200 are transmitted by the delivery
`station 30 to the client station 50 through Java applets. Java
`applets are Java programs meant to be embedded in and
`controlled by a larger application, Such as a web browser or
`an applet viewer. To include an applet as executable content,
`a special tag is required in the document. This tag points to
`an applet and provides configuration information about the
`applet. The Java applet is composed of classes. While a
`Single applet may consist of only a Single class, most large
`applets are broken into many classes. Each class is Stored in
`a separate class file. Conveniently, Java facilitates the load
`ing of class files in the processor. Further details on Java and
`loading class files are described in the text, “Exploring
`Java-2nd Edition” by P. Niemeyer et al., published by
`O'Reilly & Associates, 11011 Morris Street, Sebastopol,
`Calif. 95472 (1997).
`Turning to FIG. 3, a process 300 is shown for real time
`bandwidth measurement and prediction of the HotMedia file
`200 according to a user's available bandwidth. Briefly, the
`basic idea is for the client station to (a) detect real time
`bandwidth used by the Java applet file, (b) predict the
`available bandwidth for the next media content delivery
`based on the detected bandwidth, then (c) fetch the corre
`sponding bandwidth-Sensitive track content from the content
`delivery station 30 or served by the content delivery station.
`The process results in an adaptive bandwidth-Sensitive rich
`media content delivery System.
`In FIG. 3, the delivery process 300 is initiated in step 301.
`In step 303, a HotMedia Java class (hm.class) is loaded first
`when an HTML page with the HotMedia applet is loaded.
`The Java applet initializes itself, reads the thumbnail frame
`204, and starts a running thread in step 305. An index (i) is
`set to zero. Abandwidth (B1) for the applet can be calculated
`by dividing the thumbnail size in bits by the total time (T1)
`required by the thumbnail as provided by a client Station
`clock. (B1=Thumbnail size/T1). The detected bandwidth B1
`for (i=1) is provided to an index block 307 for Subsequent
`calculation of the next available bandwidth for rich media
`content delivery.
`In Step 309 a rich media file 200' which may be in one
`or more files is accessed in the content delivery Station.
`
`45
`
`50
`
`55
`
`60
`
`65
`
`US 6,848,004 B1
`
`25
`
`35
`
`40
`
`8
`Much of the interesting phenomena occur only when the
`user becomes interested in probing beyond the thumbnail
`experience. The rich media impression means that the user
`triggers a rich media type that requires special code for its
`decoding and rendering, e.g. Panorama, animation and
`Video. This impression can be achieved automatically or
`through user's mouse click.
`In Step 311, the Java applet will load the master class
`(hmmaster.class). The load time T2 for the master class is
`provided by the client Station clock. T2 is used to calculate
`a bandwidth B2 for the hmmaster.class where the size of the
`himmaster.class is a known file size. B2 is calculated as
`B2=Size of hmmaster.class/T2 while setting i-2. The
`detected bandwidth B2 is provided to the index block 307
`for prediction of the next available bandwidth.
`In Step 313, the hmmaster.class continues to run but is not
`used in predicting the next available bandwidth. Instead, B1
`and B2 can be used by the client station to predict the next
`available bandwidth B3 for rich media content delivery to
`the client by the relationship B3*=f(B1, B2), where f is a
`mapping operator (estimated). For instance, a linear pre
`diction algorithm B3*=2*B2-B1 (Equation 1) can be used
`to predict the next available bandwidth for the file 200". For
`input Sequences B(i), B(i-1) . . . B(2), B(1), equation 1 can
`be re-written B*(i+1)=2*B(i)-B(i-1)(i=2,3,) (Equation 2).
`Equation 2 can be re-written B*(i+1)=B(i)+B(i) B-(i-1)).
`(Equation 3). A more simpler algorithm is B*(i+1)=B(i).
`(Equation 4) which means the last monitored user bandwidth
`can be regarded as the next predicted available user band
`width.
`In step 315, the predicted algorithm for (i=3) is B(i)=B3*
`else Bi=Bi is provided to the index block 327 for deciding
`where the predicted available user bandwidth Bi should be
`Sent. There are two options for the decision maker to take
`advantage of this predicted available user bandwidth. The
`default Setting is that the decision maker is located on the
`client Station. This option can be changed by using the
`parameter “DM” in the HotMedia applet tag shown as
`follows:
`< Param Name="DM Value="Server'>
`That means the decision maker is located on the web
`server if the above applet tag exists. This flexibility allows
`the author to customize the decision maker on the web
`Server. Obviously, it provides Seamless interfacing with
`multiple adaptive content delivery Servers requiring neither
`changes to Server code nor changes to content.
`So if the answer from index block 327 is “No”, the
`predicted available user bandwidth Bi will be sent to
`decision maker 317 on the client station through in