`JonathanC.Soo(cid:3)
`Telemedia,NetworksandSystemsGroup
`MITLaboratoryforComputerScience
`Cambridge,MA
`Abstract
`TheWorldWideWebiscurrentlynotwellorientedtowardsdistributingstream-orientedmediasuch
`asaudioandvideo.ThelimitationisnotinHTTP[]itself,butincurrentlyexistingbrowsers.After
`openinganHTTPconnectiontoaserver,mostbrowserswritealldatatoalocal(cid:12)lebeforepassingit
`toanexternalviewer.Whilethisworkswellfortextandgraphics,itmakesviewingstream-oriented
`mediaimpracticalbecauseofthelongdelaybeforestartofplayback,andbecausetheentire(cid:12)lemustbe
`storedonthelocalhost.Inaddition,itisnotpossibletosend\live"streamsofdata.
`Thispaperdescribesaprototypebrowserdesignedtosolvesomeoftheseproblems.Itsupportsasubset
`ofHTTP . .
`
`Introduction
`TheWorldWideWebisapopularmechanismfordistributingmanytypesofdata.Inthelastfew
`years,ithasbecomeamajorsourceoftra(cid:14)conInternetandanimportantpartoftheacademicand
`commercialinformationinfrastructure.
`Whileitcurrentlyworksverywellfortextandgraphics,isisnotaswellsuitedfordistributingstream-
`orientedmediasuchasaudioandvideo.EvenforusersonalocalEthernet,thetimeneededtodownload
`acompleteaudioorvideosegmentdiscouragesbrowsing.
`Inaddition,itiscurrentlynotpossibletoaccess\live"mediastreamsthroughWebbrowsers.Live
`mediaaremediaofpossiblyindeterminiateduration,wherecapture,transmission,andplaybackare
`overlapped,andlatencyisnearlyconstantandrelativelyshort;forinstance,alivenewscastmighthave
`latencyofafewseconds.Incurrentbrowsers,suchasNCSAMosiacandtclwww,itisnotpossibleto
`overlaptransmissionandplayback,resultinginadelayproportionaltothelengthofthesegmentbeing
`transmitted.
`Thispaperdescribesascriptable,extensible,andembeddablebrowserthatallowslivemultimediato
`betransmittedusingHTTP.Examplesofliveaudioandvideoarepresented,andapproachestolive
`multimediaarediscussed.Thepaperalsodescribestheperformanceofthebrowseronprerecorded
`multimedia,andpresentssomeapplicationsusingthebrowserasaplatformfordistributedcomputing.
`(cid:3)Theauthorcanbereachedat:MITLaboratoryforComputerScience,Room ,TechnologySquare,Cambridge,
`MA ;Tel:( ) - ;Email:jcsoo@mit.edu
`
`Petitioners' Exhibit 1042
`Page 0001
`
`
`
`Approach
`. CurrentWebBrowserImplementations
`Mostbrowserstodayaredesignedprimarilyforbrowsingthroughtextandgraphicsoverfastnetwork
`connections,usinga"storeandforward"approach.
`Inatypicaltransaction,abrowser(cid:12)rstgeneratesanHTTPrequestonbehalfofauser.TheURL
`providedisparsedto(cid:12)ndthehostandportofaserver,andaTCPconnectionisopened.Therequest
`headerissentovertheconnection,andthebrowserthenwaitsfortheHTTPreplyheader.
`Ontheserverside,theheaderisparsed,theappropriatedataislocated,aHTTPreplyheaderwiththe
`datatypeandlengthisgeneratedandsent,andthedataiswrittendirectlytothenetworkconnection.
`ThebrowserthenreadsandparsestheHTTPreplyheader,determiningthelengthandtypeofthedata
`beingsent.Ifthedataisofatypethatissupportednativelybythebrowser,itisreadandprocessed.
`Ifitisnot,thedataiscopiedfromtheTCPconnectiontoatemporary(cid:12)leonlocalstorage,andthe
`nameofthis(cid:12)leisthenpassedtoanuser-de(cid:12)nedapplicationthoughacommandlineparameter.
`Thisbasicmodelisverye(cid:11)ective.Mostimportantly,itisextensible;addingnewdatatypesisfairly
`simple,usuallyrequiringasmallmodi(cid:12)cationtoacon(cid:12)guration(cid:12)leandnotrequiringcompilationof
`thebrowser.Sub-applicationscanbedesignedandtestedcompletelyseperatelyfromthebrowser,and
`thereisanestablishedproceduretopromoteexperimentaldatatypestogenerallyacceptedones.
`However,therearesomeproblemswiththedetailsoftheimplementation.Themostsigni(cid:12)cantisthe
`stepwherethedataiscopiedfromthenetworkconnectionintoatemporary(cid:12)lebeforeasub-application
`isspawned.Forsmalltextandgraphic(cid:12)leswheretransmissiontimeisshortcomparedtotheconnection
`setuptime,thereislittlee(cid:11)ectonresponsetime.
`Forlargeaudioandvideo(cid:12)les,however,thisextracopymeansthattheentire(cid:12)lemustbetransferred
`beforethesub-applicationcanbespawned.Severalproblemsresultdirectlyfromthis.Themost
`importantisthatitisnotpossibletosendlivemediastreams;playbackcannotbeoverlappedwith
`transmissionorcapture.
`Evenforprerecordedmedia,responsetimesforlongsegmentsmaybemeasuredinminutes,andthe
`clientmusthaveenoughlocalstoragetostoretheentire(cid:12)lewhetheritwillbeplayedcompletelyor
`not.Anotherdisadvantageofthisapproachisthatitisine(cid:14)cient;notonlyisthedatabeingcopiedone
`moretimethannecessary,itprobablyisbeingcopiedtoarelativelyslowmechanicalstoragedevice.
`Asnetworkconnectionsbecomefaster,thismaybecomeaseriousbottleneck.
`.ProposedChanges
`Theapproachtakentosolvetheseproblemswastorestructuretheclienttoavoidthecopytodisk.
`Insteadofimmediatelyreadingthedatastreamandwritingittoa(cid:12)le,theconnectionispasseddirectly
`tothesub-application.Thesub-applicationcanthenreaddatafromtheserverasitisneeded,and
`overlapprocessingwithI/O.Infact,thesub-applicationhasaTCPconnectiontotheHTTPserver
`process,andcansendandreceivedatasimultaneously.
`Ontheserverside,nomodi(cid:12)cationsarenecessarytotakeadvantageofthisapproachtoplayingmedia
`(cid:12)les.TheserversendsdatatotheclientasfastastheTCPconnectionwillaccept,andblockswhen
`theTCPconnectiondoes.
`
`Petitioners' Exhibit 1042
`Page 0002
`
`
`
`Forlivemultimedia,someserver-sidescriptsarenecessarytocreatelivemediasourcesfrommedia
`captureapplications.Fortunately,boththeNCSAandCERNHTTPservershaveagatewayinterface
`thatmakeiteasytodothisbypassingTCPsocketsdirectlytosub-applications.
`. Goals
`Thedesigngoalofthisprojectwastocreateaclientthatwouldimplementtheaboveapproach,aswell
`asformaplatformforfutureresearchinrelatedareas.Itwasdesiredtohaveaclientthatwouldbe
`easilyunderstandable,portable,extensible,andembeddable.Althoughanexistingbrowsercouldhave
`beenmodi(cid:12)ed,thiswasnotdoneforseveralreasons.
`(cid:15)MostbrowsershavealargefractionofcodededicatedtoHTMLpresentationanduserinterfaces,
`andsupportofalternateprotocolssuchasFTPandNNTP.Excludingthiscodesimpli(cid:12)esdevel-
`opment,reducesbugs,andreducesthenumberofconstraintsputontheHTTPimplementation.
`(cid:15)MostbrowsersarewritteninC.Whilethismayprovidegoodperformance,itmakesexperimen-
`tationmoredi(cid:14)cultcomparedtousinganinterpretedlanguage.
`(cid:15)Mostbrowsersaredesignedasstand-aloneprograms,controlledonlybyhumanusers.Modifying
`onetobeembeddedinanotherprogramortobeotherwisecontrolledbyotherprogramsorscripts
`wouldhavebeendi(cid:14)cult.
`
`Implementation
`Theimplementationwasdividedintotwoparts;anHTTPlibraryandasimplebrowser.TheHTTP
`libraryimplementsasubsetofHTTP,andthebrowsersupportsseveralofthecommondatatypes
`foundontheWeb,andseveralcommonsub-applicationsusedtoviewthosetypes.
`ThelanguageplatformthatwaschosenwasTCL. b[],withtheTCL-DPnetworkingextensions
`andsomeCextensionstohandleHTTPheaderparsing.ThecomputingplatformwasaDECAlpha
`runningOSF/ ..
`Thesizeofthe(cid:12)nalcodewasapproximately linesofTCL,and linesofC.
` . HTTPLibrary
`TheHTTPlibraryiswrittenalmostcompletelyinTCL.Itusesthesocketmanagementfunctionsof
`TCL-DPtoestablishTCPconnectionstoHTTPservers,andgeneratesanHTTPheader.Because
`TCLhasdi(cid:14)cultyusingnon-UNIXlinedelimiters,italsousesafunctionwritteninCtoreadthe
`HTTPreplyheader.
`WhencalledwithanURLandmethod,thehttpfunction(cid:12)rstparsestheURLtodeterminethename
`oftheservertoaccess,andtheportnumbertouse.ThelibrarythenopensaTCPconnectionand
`sendsandreceivestheappropriateHTTPheaders.
`TheTCPconnectionisreadonlyuntiltheendoftheHTTPheader,sothatthenextbytereadwould
`bethestartofthedatarequested.Atthatpoint,theHTTPlibraryreturnsthehandleoftheTCP
`connectionandpassestheparsedlistofMIMEheaderstothecallingfunction.
`
`Petitioners' Exhibit 1042
`Page 0003
`
`
`
` .Browser
`Thebrowseracceptsuserinputanddetermineshowtopresentthedatareceivedfromtheservertothe
`user.AftercallingtheHTTPlibrary,adispatchfunctiondeterminesthetypeofdatabeingreceivedby
`examiningtheContent-typeentryoftheMIMEheader.Foreachtype,thereisauser-de(cid:12)nedhandler
`procedurethatthedispatchercalls.
`Forsometypesofdata,suchas"text/plain"and"app/x-tcl-script",thedatamaybereadandprocessed
`withinthebrowser.Forothers,asub-applicationisstarted.
`Here,insteadofwritingthedatatoa(cid:12)lebeforestartingthesub-application,thesub-applicationis
`startedasachildprocess,andtheexisitingTCPconnectionthatwasreturnedtothebrowseris
`redirectedtobethesub-application'sstandardinputandoutput.
`Atthispoint,thebrowser'shandletotheTCPconnectionisclosed,andthereisnofurtherinteraction
`withtheserverbythebrowser.
`MostapplicationsintheUNIXenvironmentcaneasilybecon(cid:12)guredtoreadfromstandardinput.In
`manycases,thisisthedefaultbehavior.InTCL,startingtheseapplicationswithasocketredirected
`standardinputandoutputcanbehandledinasinglecommand.
`Thebrowsercurrentlysupportsmostofthecommoncontent-typestypicallyfoundontheWeb,including
`text/plainandtext/html,image/ti(cid:11),image/jpegandimage/gif,audio/basic,andvideo/mpeg.
`In
`addition,itsupportsVuSystem[ ]streams,whichareinterleavedaudio,video,andtextstreamsused
`intheMITTNSVuSystemproject.
` . Server
`Attheserverside,nomodi(cid:12)cationsareneededforsendingexisting(cid:12)les.Forlivemedia,however,some
`scriptsareneccesarytorunthelivemediacaptureapplications.
` . . TheCommonGatewayInterface
`BoththeNCSAandCERNHTTPservershaveasub-applicationinterfaceknownastheCommon
`GatewayInterface(CGI)[ ].ThisinterfaceprovidesastandardwayofpassingdetailsofHTTP
`transactionstosub-applications.Twoslightlydi(cid:11)erentvariantsexistoneachserver.AregularCGI
`scriptneedsonlytowriteashortcontent-typeheaderbeforesendingdatatotheclient.Thedatafrom
`theCGIscriptis(cid:12)rstreadbytheserver,whichcalcluatesit'slengthbeforesendingittotheclient.
`ThedataisnotsenttotheclientuntiltheCGIscriptsignalstheendofthedatabyterminatingitself.
`LivemediasourcesarenotimplementableusingregularCGIscripts.
` . .NPH-HeaderScripts
`AnNPH-HeaderscriptissimilartotheregularCGIscriptexceptthatthesub-applicationisrequired
`togenerateacompleteHTTPheaderitself,andthesub-applicationisgiventhedirectTCPconnection
`totheclientratherthanapipetotheserver.Thismakeslivemediapossible.
`TheNPH-Headerscriptsusedtoinvokethemediacaptureapplicationsareverysimilartothoseused
`inthebrowser.MostUNIXapplicationsareeasilycon(cid:12)guredthroughcommandlineoptionstowrite
`theiroutputtostandardoutputratherthana(cid:12)le.
`
`Petitioners' Exhibit 1042
`Page 0004
`
`
`
` . . Content-length
`Itisimportanttonotethatinmanycases,nouseismadeofthecontent-lengthvaluereturnedbythe
`server.Infact,noneofthesub-applicationsusedhaveaprovisionforalengthparameter.
`SomedataformatssuchasthevariousimageformatssuchasGIFandJPEGhavethecontentlength
`essentiallyencodedintheheader,ormayhavesomeotherwayofdeterminingend-of-(cid:12)le.Someother
`applicationssuchasaudiostreamplayersmaysimplyacceptdatauntiltheconnectionisclosed,which
`happenswhentheHTTPserverhasnomoredatatosend.
`Forcompleteness,aconventionwasadoptedwhereanegativecontent-lengthsigni(cid:12)edaninde(cid:12)nite-
`lengthdatastream.
` .DistributedApplications
`Twosimpledistributedapplicationswerealsocreated.BothoftheseusedthedirectTCPconnection
`totransferlivedata.
` .. Asimpleaccesslogmonitor
`The(cid:12)rstapplicationwrittenwasasimpleaccesslogmonitor.Asimpleserver-sidenph-headerscript
`waswrittentocopydatawrittentotheserver'saccesslogtostandardoutput.Ontheclientside,the
`datawaswrittentotheuserdisplayasitwasreceived.
` ..Acomplexaccesslogmonitor
`Thesecondapplicationwassimilartotheaccesslogmonitor,butwasimplementedusingRPCandwas
`interactive.Aserver-sidenph-scriptwaswrittenwithanewoutputcontent-typeofx-tcl-rpc.Onthe
`clientside,ahandlerwaswrittenthatreadfromtheconnection,andevaluatedtheTCLcode.
`Forthisapplication,thebrowserwasrunwiththeTKextensions,apopularwindowingsystembuilt
`aroundTCL.Theserverapplicationtransferredaprogramtotheclientthatcreatedtheuserinterface.
`Then,foreverylogaccess,itsentaTCLcommandtotheclient,modifyingtheclient'sstateand
`updatingtheclient'sdisplay.
`Theserverapplicationalsomonitoredtheconnectionforcommandssentbytheclient.Forinstance,the
`clientcouldspecifytotheserverwhetheritwasinterestedinallconnectionslogged,oronlyconnections
`madebypreviouslyunseenhosts.
`PreliminaryResults
`Thebrowserwastestedwithavarietyofsub-applications,(cid:12)letypes,andserversoveralocalEthernet
`andtheInternet.Duetotimerestrictions,detailedmeasurementscouldnotbemade;however,the
`subjectiveresultsarefairlyclearandreportedhere.
`
`Petitioners' Exhibit 1042
`Page 0005
`
`
`
`. ServerCompatibility
`ThebrowserwastestedwithboththeNCSAandCERNHTTPservers.Initially,thereweresomeprob-
`lemsbecausetheserversusedi(cid:11)erentlinedelimiters;theNCSAserverusesMS-DOSdelimiters,while
`theCERNserverusesUNIXconventions.BecauseTCLhasdi(cid:14)cultywithMS-DOSlinedelimiters,a
`shortCfunctionwaswrittenthatappearstohavesolvedthisproblem.
`.AudioTransmission
`TheaudioapplicationaplayprovidedwithDECOSF/ wasusedforaudioplayback,andallaudio
`sampleswereoftypeaudio/basic,whichhasbitsamplesandksamplespersecond.
`WhenstartedfromtheTNSbrowser,runningoveralocalEthernet,therewasnonoticabledelaybefore
`startofplaybackandthe(cid:12)leplayedwithoutaudibleglitches.Incontrast,whenstartedfromNCSA
`Mosaic,therewasapproximatelyasixsecondpausebeforestartofplaybackwhenplayinga MB
`sound(cid:12)le.
`Forliveaudio,thearecordapplicationprovidedwithDECOSF/ wasusedforsampling.Liveaudio
`performancewasidenticaltopre-recordedaudioperformance;therewasnonoticabledelaybeforeoutput
`started,andnoglitches.
`AudioperformanceovertheInternetwashighlyvariable.Ingeneral,thedelaybetweenthestartof
`therequestandthestartofoutputwaslessthanasecond,comparedtonearlyaminuteforasimilar
`( MB)audio(cid:12)le.However,thequalityofoutputvaried.Onsomeoccasions,severalminutesofaudio
`couldbeplayedwithoutaudibleglitches.Atothertimes,longperiodsofuninterruptedaudioareoften
`followedbylongperiodsofsilence,withoccasionalfragmentsofsound.Itispossiblethatthisiscaused
`inpartbyrigidratecontrolintheaudioplayer;iftheincomingdatastreamfallsbehindtheaudio
`output,outputisstopped,andalldatathatarrives"late"isdiscarded.Outputcontinuesonlywhen
`theincomingdatarateincreasesenoughsothattheindividualsamplesare"ontime".
`LiveaudiotransmissionsovertheInternetwerenotattempted.However,itislikelythatperformance
`willbesimilartotheperformanceofpre-recordedaudio.
`. VideoTransmission
`ThevideoapplicationsusedfortestingweretheBerkeleympegplayapplicationandtheTNSVuSystem
`applicationsvsrecordandvsplay.mpegplayacceptsaMPEGencodedvideoonlystreamwhilevsplay
`acceptamultiplexedaudio/video/textstream.Forthesetests,vsplaywasusedwithoutcompression.
`. . MPEGvideotransmission
`WhenstartedfromMosaic,mpegplaytookapproximately secondstodownloadandplayan KB
`video(cid:12)leoveralocalEthernet.WhenstartedfromtheTNSbrowser,playbackstartedapproximately
`(cid:12)vesecondsaftertheinitialrequest.Therewasnovisibledi(cid:11)erenceinplaybacktimebetweenthetwo
`invocations.
`mpegplaywasalsorunwithavideo(cid:12)lelocatedonanNCSAserver.ThedownloadtimefromNCSA
`wasapproximately secondsfora KBvideo(cid:12)le.WhenplayeddirectlyovertheInternet,usingthe
`TNSbrowser,MPEGvideoplaybackperformancewashighlyvariable.Theplaybackratewasbursty;
`severalsecondsofvideowouldplayatafastratefollowedbyapausesofabouthalfasecond.
`
`Petitioners' Exhibit 1042
`Page 0006
`
`
`
`Itispossiblethatthisbehaviorisaresultofalackofratecontrolinmpegplayandthevariable
`bandwidthcompressionofMPEGvideostreams.mpegplaywillplaybackframesassoonastheyare
`received,keepinganyTCPreceivebu(cid:11)ersempty.Also,MPEGvideostreamsareofvariablebitrate;
`lowbandwidthinterframedataissentbetweenhigherbandwidthframes.
`BecausetheonlyMPEGencodersavailablewerebatchencoders,liveMPEGtransmissionwasnot
`attempted.
`. .VuSystemvideotransmission
`WhenrunonalocalEthernetonpre-recordedvideo(cid:12)les,vsplayperformedsimilarlytoaplayand
`mpegplay;therewasasigni(cid:12)cantreductioninresponsetimewhenusingtheTNSbrowser,andthere
`wasnovisibledi(cid:11)erencebetweenthetwoapproaches.
`LiveVuSystemmediastreamswerealsotransmittedoverthelocalEthernet.Eightbit, by
`pixelframesweresentatsustainedratesof(cid:12)vefps,alongwithasinglebasicaudiostream.Atthis
`relativelylowrate,networkplaybackwasnotdistinguishablefromlocalplayback.Atthistime,testing
`athigherbitrateswasnotattempted.
`Conclusions
`LiveMultimediaispossiblethroughHTTPwithoutmuchmodi(cid:12)cationtoclientsandservers.Even
`averysimpleimplementationofanHTTPclientallowsforadequateperformanceoveralocalarea
`network.
`Inaddition,manyofthetechniquesusedresultinimprovedperformanceforprerecorded
`media.
`Thereismoredi(cid:14)cultyinusingthesetechniquesoverslowernetworks.Foranygivenmediatype,there
`willbesomenetworksthatsimplydonothavethenecessarybandwidthforlivetransmission.Forthese
`networks,theonlychoiceistouselocalstoragetocachetheincomingdata.
`ThemostinterestingareaistransmissionovernetworkssuchastheInternetthatmayhaveadequate
`averagebandwidth,buthighvariabilityofbandwidth.Sub-applicationsmustbedevelopedthatcan
`toleratethis.
`Onesolutionmightinvolvebu(cid:11)eringatthereceivingend,sothatoccasionalpausesintransmissioncan
`betolerated.AnotherapproachmightbetotakeadvantageofthefullduplexTCPconnectionanduse
`feedbacktocontrolthetransmissionrate,possiblya(cid:11)ectingthe(cid:12)naloutputquality.
`Inallofthesecases,anewperspectiveonHTTPisneeded.Ratherthanbeingviewedasabrowser
`protocol,MIMEtypescanbelookedatasnamingprotocolsinadditiontodatatypes.HTTPcanthen
`beseenasapowerful,widelyusedprotocolnegotiationtool.
`Acknowledgements
`TheauthorwouldliketothankallofthemembersoftheTNSgroup,especiallyProfessorDavid
`Tennenhousefortheinitialmotivation,andDavidWeatherall,ChrisLindblad,andHenryHouhfor
`theirfeedbackandsupport.
`
`Petitioners' Exhibit 1042
`Page 0007
`
`
`
`References
`[ ]RobMcCool,\TheCommonGatewayInterface",NationalCenterforSupercomputerApplica-
`tions.
`[]TimBerners-Lee,\HyperTextTransferProtocolRequirements",EuropeanLaboratoryforPar-
`ticlePhysics.
`[ ]Tennenhouse,DavidL.etal.,\ASoftwareOrientedApproachtotheDesignofMediaPro-
`cessingEnvironments",ProceedingsoftheInternationalConferenceonMultimediaComputingand
`Systems,pp. -,May .
`[]J.K.Ousterhout,\Tcl:AnEmbeddedCommandLanguage,",ComputerScienceDivision
`(EECS),UniversityofCalifornia,Berkeley,CA,January .
`AbouttheAuthor
`JonathanSooisaM.Eng.studentintheTelemedia,Networks,andSystemsGroupattheMIT
`LaboratoryforComputerScience.Heisinterestedintheapplicationsofdistributedsystems,andthe
`ofthetoolsthatenablethem.HewasanundergraduateatMIT,andisavicepresidentandfounder
`ofAgoraTechnologyGroup,Inc.Heisreachableatjcsoo@mit.edu
`
`Petitioners' Exhibit 1042
`Page 0008
`
`