`(10) Patent No:
`a2) United States Patent
`Changetal.
`(45) Date of Patent:
`Jan. 25, 2005
`
`
`US006848004B1
`
`4/1998 Judson
`5,737,619 A
`(54) SYSTEM AND METHOD FOR ADAPTIVE
`11/1998 Rhoads
`5,841,978 A
`DELIVERY OF RICH MEDIA CONTENT TO
`3/1999 Aras et al. oo. 709/226
`5,884,037 A *
`A USER IN A NETWORK BASED ON REAL
`6/1999 Astiz, et al.
`5,918,012 A
`TIME BANDWIDTH MEASUREMENT &
`8/2001 Cuomoet al. vce 709/223
`6,272,539 Bl *
`PREDICTION ACCORDING TO AVAILABLE
`9/2001 Ravi et al. veces 709/233
`6,292,834 BL *
`USER BANDWIDTH
`3/2002 Beigi et al.
`......ceessssee: 370/252
`6,363,056 BI ‘
`Inventors: Sih-Pin Subrina Chang, Old Tappan,
`(75)
`
`
`NI (US). Shu-Chen Jeane Chen 6,377,989 B1*4/2002 Fukasawa etal. ........ 709/224
`,
`;
`2003/0101263 Al *
`5/2003 Bouillet et al.
`............. 709/225
`Chappaqua, NY (US); Keeranoor G.
`Kumar, Randolph, NJ (US); JamesS.
`Lipscomb, Yorktown Heights, NY
`(US); Jai Menon, Croton-on-Hudson,
`NY (US); Liang-Jie Zhang, White
`Plains, NY (US)
`
`* cited by examiner
`
`Primary Examiner—William C. Vaughn, Jr.
`(74) Attorney, Agent, or Firm—David M. Shofi
`
`(73) Assignee:
`
`International Business Machines
`Corporation, Armonk, NY (US)
`
`(57)
`
`ABSTRACT
`
`(*) Notice:
`
`Subject to any disclaimer, the term ofthis
`patent is extended or adjusted under 35
`US.C. 154(b) by 0 days.
`
`A network implementing HotMedia architecture provides
`real
`time measurement and prediction of bandwidth for
`adaptive content delivery of rich media according to avail-
`able user bandwidth. The rich media file created at
`the
`creationstation is transmitted by the delivery station to the
`(21) Appl. No.: 09/447,585
`client station in steaming Java applets having a framework
`(22)
`Filed:
`Nov. 23, 1999
`including
`a thumbnail frame and a media frame. In real-
`51) Ute C0? oecceccsssscssssseessssesssseeessseesee GO6F 15/16 Os
`
`709/232: 709/224: 709ae
`(5, US. Cl
`370/232
`(58) Field of Search .......c.ccsssscseesseeee 709/231, 226,
`709/225, 224, 228, 219, 223, 230, 232,
`233, 234, 235; 370/229, 230, 231, 477,
`431, 232, 233, 234; 719/315, 316, 332
`
`(56)
`
`References Cited
`U.S. PATENT DOCUMENTS
`
`time, the client station detects the bandwidth used by the
`Java applet; predicts the available bandwidth for next media
`delivery based on the detected bandwidth and fetches the
`corresponding sensitive bandwidth media fame from a web
`server according to a linear or non-linear prediction algo-
`rithm resulting in bandwidth sensitive rich media content
`delivery from the delivery station to the client station for
`.
`.
`.
`display and user interaction.
`
`5,548,761 A *
`
`8/1996 Balasundaram et al.
`
`.... 717/159
`
`17 Claims, 3 Drawing Sheets
`
`301
`
`START
`
`AVAILABLE
`BANDWIDTH
`Oe '
`af
`THE AVAILABLE
`BANDWIDTH
`w
`333
`(it iss
`BF (23)
`sen
`
`AVATLABLE
`
`
`Bo|wo, NE) then
`BANDWIDTH
`itp 3e
`i
`
`
`PREDICATION 2
`DERSION
`Bit=B3
`OW SERVER
`else
`(isi)
`
`MAKER
`Bi=Bit
`
`DECISION MAKER
`~ HotMedia MASTER
`
`— ACTION HANDLER
`
`SEND HTTP REQUEST TO SERVER
`
`APPLET INIT
`LOAD THUMBNAIL SET i = 0
`THE DETECTED BANDWIDTH Bt
`RICH MEDIA IMPRESSION
`LOAD hmmaster.class
`THE DETECTED BANDWIDTH 82 {i=2)
`hmmaster RUNNING
`
`(i=1)
`
`303
`305
`
`309
`AoE
`33
`
`37
`
`J
`
`
`
`
`SEND BF TO DECISION MAKER
`
`
`
`j~329
`
`31
`
`By
`
`RICH MEDIA CONTENT
`THE DETECTED BANDWIDTH Bi (3)
`PLAYBACK RICH MEDIA CONTENT
`
`321}
`
`DEay
`
`IPR2022-01227
`IPR2022-01227
`EXHIBIT 1020 - PAGE 0001
`EXHIBIT 1020 - PAGE 0001
`
`
`
`U.S. Patent
`
`Jan. 25, 2005
`
`Sheet 1 of 3
`
`US 6,848,004 B1
`
`rr ren
`
`g¢CG\G7
`
`1
`
`
`
`sousINSTT)
`
`ro-cccc
`
`1|I411J
`
`
`
`
`
`
`
`NOILYISNOLIV38)INSINOD|
`
`SIG eS
`4asnONDIOVEL
`r1t
`[31g||uaau3s!Wlavay||vIn||3M
`ysId|t|4jI|1I|1il)0/]ANONGNnd)|1g#61¢!boNOUSATESNIN
`NOEi_o_--eepoeeeeeeeeeee\JVMLIOS
`92c%Gc\C
`190AvaAnac]
`tei
`
`8S
`
`IALVOV
`
`J1901Ad3AI130
`
`iw[=LS9S
`YaSMOUE
`
`oF
`
`OF
`
`lpY3AN3S
`
`ig9¢
`
`IPR2022-01227
`IPR2022-01227
`EXHIBIT 1020 - PAGE 0002
`EXHIBIT 1020 - PAGE 0002
`
`ONINOHLAY
`
`|
`
`]
`
`4 a
`
`i
`
`
`
`
`
`
`
`U.S. Patent
`
`Jan. 25, 2005
`
`Sheet 2 of 3
`
`US 6,848,004 B1
`
`cle
`
`3
`
`
`
`Y30V3HINVY3
`
`SOV1d%3dAL
`
`912
`
`
`
`WadNnd9VIGIK
`
`
`
`NOLLVAYOJNINOLLYNYOINI
`
`¥22022
`
`IW
`
`802cle
`
`NOLLOY
`
`LOC
`
`JONVY
`
`$02
`
`5009
`
`Lie
`
`
`
`
`
`MOIAVH3d§=NOLLYZTVILUINIWV3Y1S18
`
`Gie012602
`
`IPR2022-01227
`IPR2022-01227
`EXHIBIT 1020 - PAGE 0003
`EXHIBIT 1020 - PAGE 0003
`
`
`
`
`
`
`U.S. Patent
`
`Jan. 25, 2005
`
`Sheet 3 of 3
`
`US 6,848,004 B1
`
`ou
`
`G7Git
`
`TIBVIIVAY
`
`HLOIMGNYa
`
`(1+!=!)ZNOLYOIOId
`
`AY30
`
`UNA
`
`HIGIMONYE
`
`sfO=4!8
`
`!0=4!4
`
`as|o
`
`
`
`
`
`
`
`(=!)19HLCIMGNYG031931303HL
`
`LNTL3ldd¥
`
`
`
`
`
`(Z=!)78HLOIMONYS031931903HL
`
`Ol38
`
`NOISIN30
`
`uD
`
`
`
`YaH¥ANOISIOIG
`
`
`
`UTIONVHNOLLOY-
`
`
`
`UVTdOUddYIHLIAL3I34
`
`
`
`IN3INODYIG3NHOLY
`
`
`
`INJINODVIC3NHOWXOVEAYTd
`
`1E¢~4USAYISFHLNOMDANOISDIO)yaqy3sofIS3NDIYdLLHGNIS
`
`|NOLLVOIGRNdssojoaeysouulyGO}
`
`TvIVAYNOISSIUdMIYIOINHOY
`
`
`fgHLOIMONYSgee$TIAVIIVAYJHLONINNNYJ2ysowwY
`
`LOEQ=!13STIYNSNNHLCv¥01
`
`
`
`62¢YSNWNNOISIDIGOL¢@GNISYIUISWAOLpeWOH—
`
`—t—NaS
`($<!)4ite
`©-‘OQIqLuvsLog
`
`
`
`
`
`(¢<!)18HLCIMONYO319a1303HL
`
`£0¢
`
`GOt
`
`60£
`
`Ie
`
`ele
`
`LI¢
`
`ote
`
`bee
`
`IPR2022-01227
`IPR2022-01227
`EXHIBIT 1020 - PAGE 0004
`EXHIBIT 1020 - PAGE 0004
`
`
`
`
`
`
`
`
`US 6,848,004 B1
`
`1
`SYSTEM AND METHOD FOR ADAPTIVE
`DELIVERY OF RICH MEDIA CONTENT TO
`A USER IN A NETWORK BASED ON REAL
`TIME BANDWIDTH MEASUREMENT &
`PREDICTION ACCORDING TO AVAILABLE
`USER BANDWIDTH
`
`RELATED APPLICATIONS
`
`1. U.S. Pat. No. 6,356,921, issued Mar. 21, 2002, entitled
`“Framework for Progressive Hierarchical and Adaptive
`Delivery Rich Media Presentations and Associated Meta
`Data” (SE9-98-030V).
`2. US. Pat. No. 6,448,980, issued Sep. 10, 2002, entitled
`“Personalizing Rich Media Presentations Based On User
`Response to the Presentation” (SE9-98-028V).
`3. US. Ser. No. 09/100,419, filed Jun. 20, 1998, entitled
`“Progressive Interleaved Delivery of Interactive Descrip-
`tions and Renderers for Electronic Publishing of Merchan-
`dise” (SE9-98-004).
`4. US. Ser. No. 10/392,055, filed Mar. 19, 2003, entitled
`“System and Method for Tracking User Interactions and
`Navigation During Rich Media Presentations” (SE999-006/
`1963-7334US1).
`5. U.S. Ser. No. 09/438,493, filed Nov. 12, 1999, entitled
`“A System and Method of Enriching Non-Linkable Media
`Representations In A Network By Enabling An Overlying
`Hotlink Canvas” (SE9-99-012/1963-7341).
`All of the above applications have been assigned to the
`Assignee of the present invention and are fully incorporated
`herein by reference.
`BACKGROUNDOF INVENTION
`
`1. Field of Invention
`
`This invention relates to multimedia network, systems
`and methods of operation. More particularly, the invention
`relates to systems and methodsfor adaptive delivery of rich
`media content to a user in a network based on real time
`
`bandwidth measurement and prediction, according to avail-
`able user bandwidth.
`
`2. Description of Prior Art
`In recent years, there has been a sizeable growth in the use
`of rich media over the World Wide Web as more and more
`individuals and institutions are beginning to realize the
`web’s potential in a broad range of applications including
`electronic commerce, education training, news, etc.
`Examples of rich media include animation, audio, 3-D,
`panoramas, and videos. There are two apparent clusters of
`rich media technology. One at the low end and the other at
`the high end, which hamperseffective deployment of some
`rich media in Internet based applications. The “low end”
`cluster comprises static images and simple non-interactive
`animations (typically animated GIFs) which are easy to
`deploy and therefore have widespread usage. The “high end”
`clusters comprises richer and more natural experiences with
`larger interactivities, such as panoramas, 3-D, streaming
`audio/video, and composite media (e.g., MPEG-4), but the
`difficulty of deployment has limited widespread usage. A
`novel scaleable architecture called HotMedia bridges the
`gap between the two clusters thereby achieving widespread
`web penetration. Akey feature of the technologyis a suitable
`delivery file format that can contain heterogeneous compo-
`sitions of mediabit streams as well as meta data that defines
`
`behavior, composition and interaction semantics. The deliv-
`ery file format enables the creation of lightweightsingle file
`representation ofinteractive, multistage presentationsresult-
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`2
`ing in multiple media type contents. At the core of HotMedia
`client is a smart content algorithm that infers media types
`from the incoming data stream and fetches the media
`renderer components,user-interface components and hyper-
`linked action components, all just
`in time, resulting in
`progressive and context driven enrichmentof the user expe-
`rience. Further details related to HotMedia architecture are
`decribed in U.S. Pat. No. 6,611,812 entitled “Framework For
`Progressive Hierarchical and Adaptive Delivery Rich Media
`Presentation and Associated Meta Data”, issued Aug. 26,
`2003 (SE9-98-033), supra.
`Often instances of rich media incorporate links to other
`presentations to expand a user experience. The process of
`clicking on a link in a media for transfer to the other
`presentation is referred to as “hot
`linking ” or “hyper
`linking” which is further described in U:S. Pat. No. 5,841,
`978 entitled “Networking Using Steganographically Embed-
`ded Data Objects” issued Nov. 24, 1998 and US. Pat. No.
`5,918,012 entitled “Hyper Linking Time Based Data”issued
`Jan. 29, 1999. The real power of “HotMedia”is in how the
`experience is delivered to the user. A dynamic content
`delivery system to a user should take into account the user’s
`connection speed to enhance the user experience. For
`example, when a user has experienced reduced bandwidth
`because of having heavy CPU usage, phoneline noise or
`other web browser instances, the user could receive seam-
`lessly adjusted web content corresponding to the reduced
`bandwidth. LAN users might get higher quality video clips
`or audio clips than T1 users, with 56K modem connection
`user’s surfing higher quality images than 28.8K modem
`users. An adaptive content delivery system according to user
`connection speed or bandwidth should have the following
`design goals:
`1. An adaptive content delivery framework that expands
`the scope of information that can be adaptively delivered.
`2. Acontent creation phase that enables individual speci-
`fication and customizability of information to be adapted to
`the delivery for every bandwidth sensitive content context.
`3. Adaptive content delivery servers requiring neither
`change to the server code nor changes to content for seam-
`less interfacing.
`4. A cascading delivery structure for efficient content
`delivery as used in the HotMediaarchitecture.
`5. Acontent delivery system which provides for adaptive
`delivery disablement for the content.
`Prior art related to multimedia delivery systems include
`the following:
`U.S. Pat. No. 5,737,619 entitled “World-Wide Web
`Browsing with Content Delivery Over An Idle Connection
`& Interstitial Content Display” by D. H. Judson,discloses a
`personal computer supporting a graphical user interface and
`an Internet browser. The computer locally stores retrieves
`and outputs information objects to reduce waiting time
`normally associated with downloaded hypertext documents
`having high resolution graphics. In one embodiment, a web
`page is displayed on the graphical user interface and has at
`least one link to a hypertext document, preferably located at
`a remote server. Whenthe user clicks on the link, the link is
`activated by the browser to request downloading of the
`hypertext documents from the remote server to the graphical
`user interface. While the user waits for a reply and/or as the
`hypertext document is being downloaded, the browser dis-
`plays a previously cached information object.
`An Internet program “Speed Select” from Match Logic of
`Excite (http:/Awww.matchlogic.com) makes use of a user’s
`historical profile saved over time to predict a user’s con-
`
`IPR2022-01227
`IPR2022-01227
`EXHIBIT 1020 - PAGE 0005
`EXHIBIT 1020 - PAGE 0005
`
`
`
`US 6,848,004 B1
`
`3
`nection speed at any time, based on measurements taken at
`similar times of the day.
`Noneofthe prior art discloses a generalized adaptive rich
`media content delivery framework based on real time band-
`width measurements, prediction and dynamic content deliv-
`ery at the user so that advertisers or content providers can
`ensure that all times users receive bandwidth-sensitive con-
`tent.
`
`SUMMARYOF THE INVENTION
`
`An object of the invention is a multimedia network
`adaptively delivering rich mediato a client station according
`to available user bandwidth in the network.
`
`Another object is an adaptive delivery system for rich
`media content to a client station in a multimedia network
`
`based on bandwidth measurementand prediction according
`to available user bandwidth.
`
`Another object is an adaptive delivery system for rich
`media content in a network which enables a content creator
`to specify presentation events or user interactions that trig-
`ger adaptive content delivery during presentation of rich
`media content to a client station.
`
`Another object is an adaptive delivery system for rich
`media content to a client station in a network that takes
`advantage of dynamic Java class loading mechanism and
`progressive rich media delivery mechanism to perform real
`time bandwidth measurementand prediction in the network.
`Another object is an adaptive delivery system and method
`of operation for rich media content to a client station in a
`network which provides seamless interfacing with multiple
`adaptive content delivery servers requiring neither changes
`to server code nor changes to content.
`These and other objects, features and advantages are
`achieved in a network implementing HotMedia architecture
`including a content creation station and a content delivery
`station coupled to a client station via the network for
`delivery of rich media to the client station for display and
`interaction with a user. The rich media file created at the
`
`creation station is transmitted by the delivery station to the
`client station in streaming Java applets having a framework
`including a header frame, a thumbnail frame, a media frame,
`a meta frame, and an end-of-stream frame. The content
`creation station enables a creator to specify presentation
`events or user interactions that trigger an adaptive content
`delivery during presentation of the rich media content. The
`client station takes advantage of the dynamic Java class
`loading mechanism and progressive rich media mechanism
`to do real time bandwidth measurementand prediction. The
`delivery station monitors available user bandwidth derived
`from the client station and calibrates a rich media delivery,
`fine-tuned to each user capability. In real-time,
`the client
`station detects the bandwidth used by the Java applet;
`predicts the bandwidth for next media delivery based on the
`detected bandwidth and fetches the correspondingsensitive
`bandwidth media frame from a web server according to an
`algorithm resulting in bandwidth sensitive rich media con-
`tent delivery from the delivery station to the client station for
`display and user interaction.
`
`DESCRIPTION OF DRAWING
`
`The invention will be further understood from the fol-
`
`lowing detailed description of a preferred embodimenttaken
`in conjunction with an appended drawing, in which:
`FIG. 1 is a system diagram of a content creator station,
`content delivery station, client station for user interaction,
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`4
`coupled to a network implementing HotMedia architecture
`and incorporating the principles of the present invention.
`FIG. 2 is a representation of a HotMedia file format
`comprising a sequence of framesfor transmitting rich media
`in the system of FIG. 1.
`FIG. 3 is a flow diagram for real time measurement and
`prediction of bandwidth to the client station according to the
`available client bandwidth.
`
`DESCRIPTION OF PREFERRED EMBODIMENT
`
`In FIG. 1, a system 10 for real time bandwidth measure-
`ment and prediction according to a user’s available band-
`width includes a content creation station 20 and a content
`delivery station 30 coupled to a network 40, typically a
`distributed information system, e.g., the Internet capable of
`handling rich media. The network further includes a client
`station 50 which enables a user 45 to interact with the rich
`media to provide inputs to the client station for real time
`bandwidth measurementandprediction. The delivery station
`30 delivers appropriate rich media content
`to the client
`station 50. A tracking server 42 is coupledto the network 40
`for
`tracking the user interactions with rich media file
`received by the client station.
`The content creation station 20 includes a standard CPU
`
`21 coupled to a hard disk 22; a display 23, an input/output
`terminal 24 and a standard random access memory 25.
`Contained in the disk 22 are software for station operation
`26, e.g. Windows 95 and an authoring tool for a creator (not
`shown) to create rich media in a HotMedia file format
`supported by the HotMedia Architecture as will be described
`in conjunction with FIG. 2. Authoring tools are well known
`in the art as evidenced by U.S. Pat. No. 5,801,687 entitled
`“Authoring Tool Comprising Nested State Machines For
`Use In A Computer System”, issued Sep. 1, 1998. Content
`creation tables enable a creator to specify presentation
`events or user interaction to trigger an adaptive content
`delivery during presentation of rich media contact. For
`example, when the creator is creating a hotlink to different
`rich media file from within one, the creator will be presented
`a check box to indicate whether the execution of the link
`
`should be adaptively delivered adapting to the predicted
`available bandwidth and specify the detailed mapping infor-
`mation. Accordingly in the offering phase,
`the content
`creator can choose which delivery is preferred. In general,
`the content creator can create different versions of rich
`media content corresponding to different network connec-
`tions. The different versions can be donevia multiple files or
`one single file. All dynamic content delivery options are
`saved in a HotMediafile.
`
`The content delivery station 30 similarly contains a stan-
`dard CPU 31 coupled to a hard disk 32, an input/output
`terminal 33, and a standard random access memory 34.
`Contained in the disk 32 are a station operating system 35,
`e.g. Windows NT, web server software 36 for delivering
`over the network 40, a HotMediafile(s) 37 created by the
`authoring tool through Java applets. An adaptive delivery
`CGI (Common GatewayInterface) file 38 including a deliv-
`ery logic (not shown) and decision maker software 39 for
`delivering bandwidth-sensitive media content from the web
`server 36.
`
`The client station includes a standard CPU 51 coupled to
`a hard disk 52; a display 53, an input/output terminal 54, and
`a standard random access memory 55. Further details on the
`client station are described in US. Ser. No. 10/392,055, filed
`Jun. 20, 1998, entitled “Progressive Interleaved Delivery of
`Interactive Descriptions and Renderers for Electronic Pub-
`
`IPR2022-01227
`IPR2022-01227
`EXHIBIT 1020 - PAGE 0006
`EXHIBIT 1020 - PAGE 0006
`
`
`
`US 6,848,004 B1
`
`5
`lishing of Merchandise” (SE9-98-004), supra. Contained in
`the disk 52 are software for station operation 56, e.g.
`Windows 95, a web browser 57; software 58 for content
`presentation including Hot Media kernel code and decision
`maker software 59 for choosing bandwidth-sensitive media
`content from the web server 36.
`
`The client station takes full advantage of dynamic Java
`class loading mechanism and progressive rich media deliv-
`ery to perform real time bandwidth measurement and pre-
`diction. The client station monitors available user bandwidth
`
`and calibrates itself to provide a web browsing experience
`fine-tuned to each user’s capability.
`Having described the network and system of the present
`invention, a description of the IlotMedia file format
`is
`believed appropriate for delivery of a rich media presenta-
`tion to a user(s) in which the file format contributes to the
`real time measurementand prediction of bandwidth of the
`rich media presentation according to available user band-
`width.
`
`FIG. 2 shows a HotMedia presentation file 200 in a
`framework 201 which is essentially a sequence of frames
`types comprising header 202,
`thumbnail 204, meta 206,
`media 208, and an end of stream 212. Thefirst frame is the
`header frame 202. The header frame is actually followed by
`the thumbnail frame 204. After these frames a sequence of
`other frames occurs and in no pre-mandated order or num-
`ber. The header frame 202 is the only one whosepresenceis
`mandatory in the format. The most degenerate yet useful
`version would have a header frame followed by a thumbnail
`frame 204. Media frames 208 appear in all other instances.
`Meta frames 206 are present only in cases where non-default
`behavior and configuration are desired on cases where hyper
`linked action semantics are to be incorporated. The infor-
`mation for adaptive delivery option is stored in Meta frames
`206. All frames have a similar 12 bit initial section 213 that
`
`enables a uniform procedurefor their identification to a type
`and frame label as well as the determination of their size.
`The HotMedia file format
`is created to have minimum
`
`overhead and maximum modularity. The format makes it
`suitable for optimal delivery of a low bandwidth as well as
`for rich experiences over high bandwidth.
`The header frame 202 includes a frame header 213 which
`provides information about frame size 214, types and flags
`216 and frame labels 218. The header frame 202 also
`includes media information 220 containing definition on
`different media tracks that may be contained in the file.
`General information 224 in the header frame 202 provides
`the class nameof the code that renders the media type on the
`client station. When HotMedia kernel code in the content
`presntation software in CPU 51 parses the media track
`information on the file header and if it encounters the
`specification of a class name, the media typefield is ignored
`and the code specified by the class name is sent from the
`delivery server 30 in order to render the data of the track in
`question. The class name permits the introduction of whole
`new media types into the presentation. No changes or
`additions to the client kernel code are needed so long as the
`media objects implements the interfaces specified in the
`HotMedia framework 201forfirst class members of a media
`
`objectset.
`The thumbnail frames 204 carry a minimalistic represen-
`tation of the information shared by the HotMedia file
`framework 201. For example, if the information carried in
`the file were a description of merchandise, the thumbnail
`would carry a single image. Thumbnail frames make pos-
`sible providing the user a quick introduction to the subject
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`6
`with minimal code and data transfer. Since users pay atten-
`tion to detail only when the subject is of some interest, it
`would be wasteful for such data and codes to enable any-
`thing beyond the initial experience unless an interest is
`indicated. In addition to a frame header,
`the thumbnail
`frames also carry parameters for producing image pan and
`animation effects on the single image (all not shown).
`Meta frames 206 carry specification of non-media specific
`information pertaining to enablementof hyper linked actions
`from media context, tracking of interaction and adaptive
`delivery of rich media content. Action semantics associated
`with media segments can be expressed in HotMediafiles
`200 using range 205 and action subtypes 207. Range frames
`205 are made up of range records that identify and label
`spatio-temporal segments of media. A spatial range can be
`described by a rectangular or elliptical contour. Temporal
`ranges can refer to a time range in milliseconds for a real
`time media-like audio or a count range for non-real-time
`media like animation. The range specification can also
`express a combination of space and time.
`Action records 207 may be pre-programmed into the
`HotMedia file 200 for the execution of many types of action
`in the context of a presentation. Action frames can carry
`multiple action records all pertaining to a single actionable
`context, namely the presentation of media segmentas speci-
`fied by a range. Action records are the specification mecha-
`nism for all types of actions that can be triggered during the
`context of presenting a certain media segment while in a
`certain state of the media object. Actions can belong to a
`variety of types that include LINKto specify hyperlinking;
`DISPLAY used to specify the display of contours, cursor
`changes, etc; PLAY SOUNDusedto create sound effects in
`various contexts; TRACK used to specify the tracking of
`interactions and events associated with a user experience;
`and ADAPT DELIVERYused to specify the dynamic deliv-
`ery of rich media content adapting to the available user
`bandwidth. Also contained in an action record is the speci-
`fication of an action trigger. The triggers for actions can be
`user interaction, presentation state updates or player state
`updates. An example of would be the coming in to view of
`a certain segmentof a presentation, the movement of mouse
`into a spatio-temporal range of a presentation, the click of
`the left mouse button within the present temporal range of
`the presentation, the completion of loading of a segment of
`the media data and so forth. Although the range record
`specifying a media segment associated with most action
`triggers are of type spatial, temporal or both there are some
`situations where the range type is inconsequential. A range
`type DONTCAREis designed for such situations. An
`example is where the action is determined entirely by the
`internal state of the media object producing the completion
`of a data loading trigger. The final piece of information
`contained in an action record is the data for action. The size
`of this data andits interpretation is different for each action
`type and subtype. The data information is where for
`example, the target URL string is found whenthe action type
`is LINK andthe color information when the action type is
`DISPLAY.
`
`The media frame 208 contains media bit stream data 209,
`initialization data 210, behavior data 215 and code 217. The
`behavior data or initialization data define the page and
`configuration of a corresponding media player respectively.
`Mediabit stream data belonging to a particular media track
`and can be carried over multiple frames and these frames can
`be interlinked with meta frames and frames belonging to
`other media tracks. Media frames are identified by frame
`type and track identifier as carried in their header 213. The
`
`IPR2022-01227
`IPR2022-01227
`EXHIBIT 1020 - PAGE 0007
`EXHIBIT 1020 - PAGE 0007
`
`
`
`US 6,848,004 B1
`
`7
`frame type and track identifier enable true multiplexed
`delivery and henceideal for static or dynamic proportioning
`of various media tracks over limited bandwidth. Media
`frames can also often contain codes 217 that render the
`media type on the client station. The code in turn can be
`physically present or could be a URL reference. When
`present, the code is separated from thefile on the server side
`30 (see FIG. 1), and thereafter the code or data can be made
`available for on demand delivery. In the case of a URL
`reference to code, the code is fetched from a remote HTTP
`server to be similarly made available on demand.
`The end-of-stream frame 212 is a marker that signals the
`end of the entire presentation associated with the HotMedia
`file 200. Frames belonging to multiple media tracks and
`meta data can be multiplexed in the HotMediafile. The end
`of stream frame is valuable to the client station in recog-
`nizing the end of each HotMediafile data stream which may
`otherwise be impossible when stream initialization data for
`each stream do not announcetheir respective lengths.
`Further details on the HotMediafile format are described
`in IBM Research Report RC 21519[97069] entitled “The
`HotMedia Architecture: Progressive and Interactive Media
`for the Internet”, dated July 1999, available from IBM
`Research, Yorktown, N.Y. 10598, and fully incorporated
`herein by reference.
`After creation, the files 200 are transmitted by the delivery
`station 30 to the client station 50 through Java applets. Java
`applets are Java programs meant to be embedded in and
`controlled by a larger application, such as a web browseror
`an applet viewer. To include an applet as executable content,
`a special tag is required in the document. This tag points to
`an applet and provides configuration information about the
`applet. The Java applet is composed of classes. While a
`single applet may consist of only a single class, most large
`applets are broken into manyclasses. Each class is stored in
`a separate class file. Conveniently, Java facilitates the load-
`ing ofclass files in the processor. Further details on Java and
`loading class files are described in the text, “Exploring
`Java—2nd Edition” by P. Niemeyer et al., published by
`O’Reilly & Associates, 11011 Morris Street, Sebastopol,
`Calif. 95472 (1997).
`Turning to FIG. 3, a process 300 is shown for real time
`bandwidth measurementandprediction of the HotMediafile
`200 according to a user’s available bandwidth. Briefly, the
`basic idea is for the client station to (a) detect real time
`bandwidth used by the Java applet file, (b) predict
`the
`available bandwidth for the next media content delivery
`based on the detected bandwidth, then (c) fetch the corre-
`sponding bandwidth-sensitive track content from the content
`delivery station 30 or served by the content delivery station.
`The process results in an adaptive bandwidth-sensitive rich
`media content delivery system.
`In FIG. 3, the delivery process 300 is initiated in step 301.
`In step 303, a HotMedia Java class (hm.class) is loadedfirst
`when an HTMLpage with the HotMedia applet is loaded.
`The Java applet initializes itself, reads the thumbnail frame
`204, and starts a running thread in step 305. An index(i) is
`set to zero. A bandwidth (B1)for the applet can be calculated
`by dividing the thumbnailsize in bits by the total time (T1)
`required by the thumbnail as provided by a client station
`clock. (B1=Thumbnail size/T1). The detected bandwidth B1
`for (i=1) is provided to an index block 307 for subsequent
`calculation of the next available bandwidth for rich media
`content delivery.
`In Step 309 a rich mediafile 200° which may be in one
`or more files is accessed in the content delivery station.
`
`10
`
`15
`
`20
`
`25
`
`45
`
`50
`
`55
`
`60
`
`65
`
`8
`Much of the interesting phenomena occur only when the
`user becomes interested in probing beyond the thumbnail
`experience. The rich media impression meansthat the user
`triggers a rich media type that requires special code forits
`decoding and rendering, e.g. Panorama, animation and
`video. This impression can be achieved automatically or
`through user’s mouseclick.
`In Step 311, the Java applet will load the master class
`(hmmaster.class). The load time T2 for the master class is
`provided by the client station clock. T2 is used to calculate
`a bandwidth B2 for the hmmaster.class where the size of the
`hmmaster.class is a known file size. B2 is calculated as
`
`B2=Size of hmmaster.class/T2 while setting i=2. The
`detected bandwidth B2 is provided to the index block 307
`for prediction of the next available bandwidth.
`In step 313, the hmmaster.class continues to run but is not
`used in predicting the next available bandwidth. Instead, B1
`and B2 can be used bythe client station to predict the next
`available bandwidth B3* for rich media content delivery to
`the client by the relationship B3*=f (B1, B2), where f is a
`mapping operator (*estimated). For instance, a linear pre-
`diction algorithm B3*=2*B2-B1 (Equation 1) can be used
`to predict the next available bandwidth for the file 2007. For
`input sequences B(i), B(i-1) .
`.
`. B(2), B(1), equation 1 can
`be re-written B*(i+1)=2* BG)-BG-1)G=2, 3,) (Equation 2).
`Equation 2 can be re-written B*(i+1)=B(@)+[B(i) B-G-1)].
`(Equation 3). A more simpler algorithm is B*(i+1)=B().
`(Equation 4) which meansthe last monitored user bandwidth
`can be regarded as the next predicted available user band-
`width.
`
`In step 315, the predicted algorithm for (i=3) is BG@)=B3*
`else Bi*=Bi* is provided to the index block 327 for deciding
`wherethe predicted available user bandwidth Bi* should be
`sent. There are two options for the decision maker to take
`advantage of this predicted available user bandwidth. The
`default setting is that the decision maker is located on the
`client station. This option can be changed by using the
`parameter “DM” in the HotMedia applet
`tag shown as
`follows:
`<Param Name=“DM” Value=“Server”’>
`That means the decision maker is located on the web
`
`server if the above applet tag exists. This flexibility allows
`the author to customize the decision maker on the web
`
`it provides seamless interfacing with
`server. Obviously,
`multip