Multimedia
`Systems:·
`An Overview
`
`~
`Advances in
`distributed
`multimedia
`systems have
`begun to
`significantly affect
`the development
`of on-demand
`multimedia
`services.
`Researchers are
`working within
`established
`computer areas to
`transform existing
`technolog ies and
`develop new ones.
`The big picture
`shows multimedia
`as the merging of
`computing,
`communications,
`and broadcast ing.
`
`Borko Furht
`Florida Atlantic Universi ty
`
`M ultimedia systems combine a
`
`variety of information sources,
`such as voice, graphics, anima(cid:173)
`tion, images, audio, and fuli-
`motion video, into a wide range of applications.
`The big picture shows multimedia as the mergi ng
`of three industries: computing, communication,
`and broadcasting.
`Research and development efforts in multime(cid:173)
`dia computing fall into two groups. One group
`centers its efforts on the stand-alone mul timedia
`workstation and associated software systems and
`tools, such as music composition, computer-aided
`learning, and interactive video. The other com(cid:173)
`bines multimedia computing with distributed sys(cid:173)
`tems. This offers even greater promise. Potential
`new applications based on distributed multime(cid:173)
`dia systems include multimedia information sys(cid:173)
`tems, collaboration and conferencing systems,
`on-demand multimedia services, and distance
`learning.
`The defining charalteristic of multimedia sys(cid:173)
`tems is the incorporation of continuous media
`such as voice, video, and animation. Distributed
`multimedia systems require continuous data
`transfer over relatively long periods of time (for
`example, playout of a video stream from a remote
`camera), media synchronization, very large stor(cid:173)
`age, and special indexing and retrieval techniques
`adapted to multimedia data types. The sidebar
`lists a number of acronyms relevant to multime(cid:173)
`dia systems.
`
`Technical demands
`A multimedia system can either store audio and
`video information and use it later in an applica(cid:173)
`tion such as training, or t ransmit it live in real
`
`107Q.9UX/94/ $4.00© 1994 IEEE
`
`time. Live audio and video can be interactive, such
`as multimedia conferencing, or noninteractive, as
`in TV broadcasting. Similarly, stored still images
`can be used in an interactive mode (browsing and
`retrieval) or in a noninteractive mode (slide show).
`The complexity of multimedia applications
`stresses all th e components of a computer system.
`Multimedia requires great processing power to
`implement software codecs, multimedia file sys(cid:173)
`tems, and corresponding file formats. The archi·
`tecture m ust provide high bus bandwidth and
`efficient 1/0. A multimedia operating system
`should support new data types, real-time schedul(cid:173)
`in g, and fast-interrupt processing. Storage and
`memory requirements include very high capacity,
`fast access times, and high transfer rates. New net(cid:173)
`works and protocols are necessary to provide the
`high bandwidth, low latency, and low jitter
`required for multimedia. We also need new object(cid:173)
`oriented, user-friendly software development
`tools, as well as tools for retrieval and data man(cid:173)
`agement-important for large, heterogeneous, net·
`worked and distributed multimedia systems.
`
`AD PCM
`
`Abbreviations
`adaptive differential pulse code
`modulation
`asynchronous transfer mode
`bit error rate
`broadband integrated service digital
`network
`International Telegraph and
`Telephone Consultative Committee
`coder/decoder
`Codec
`discrete cosine transform
`OCT
`differential pulse code modulation
`DPCM
`digital signal processor
`DSP
`digital video interactive
`DVI
`forward discrete cosine transform
`FDCT
`fibre distributed data interface
`FDDI
`inverse discrete cosine transform
`IDCT
`Joint Photographic Expert Group
`!PEG
`MMOS multimedia operating system
`Moving Pictures Expert Group
`MPEG
`NTSC
`National Television Systems
`Committee
`phase alternating line
`packet error rate
`priority token ring
`quality of service
`reduced instruction set computer
`synchronous transfer mode
`
`ATM
`BER
`B-ISDN
`
`CCITI
`
`PAL
`PER
`PTR
`QOS
`RISC
`STM
`
`IPR2016-01710
`UNIFIED EX1014
`
`

`
`Table 1. Storage requirements for various dqta types.
`
`Object type
`
`Image ·
`Text
`Bitmapped graphics
`ASCII
`EBCDIC Still photos
`Faxes
`
`Size and
`bandwidth
`
`Sample:
`2 KB
`per page 64 KB per image
`Detailed (color)
`7.5 MB ~r image
`KB= Kbytes MB=Mbytes
`
`Audio
`Noncoded stream of
`digitized audio or voice
`
`Animation
`Synched image and
`stream at 15-19 frames/s
`
`Voice/phone BKHz/
`8 bits (mono) 6-44 KB/s
`Audio CD DA 44.1 kHz/
`16bit 176 KB/s
`
`2.5 MB/s for 320 x 640 x 16
`pixels/frame (16 bit color)
`16 frames/s
`
`Video
`TV analog or digital image
`with synched streams
`at 24.30 frames/s
`
`27.7 MB/s for
`640 x 480 x 24 pixels
`per frame (24·bit color)
`30 frames/s
`
`Storage
`requireme~nts __ _ ~
`MB Mbyte
`1 CB
`GB Gbyte
`
`100MB
`
`10MB
`
`1MB
`
`500MB
`
`100MB
`
`1 GB
`
`550MB
`
`Fig11re 1. Storage
`requirements for a
`typical mtlltimedia
`t1pplicatio11 w i t/I
`compressed images
`and video.
`
`100 fax
`100 color
`line images
`images
`(uncompr.) (compress.
`15:1)
`
`1 hour of
`10 min of 10 min of
`animation
`di9itital
`digitized
`video
`(compress.
`video
`15:1)
`(compress. (comgress.
`20 :1 )
`30:1)
`
`Researchers are working within established
`computer areas to transform existing technolo(cid:173)
`gies, or develop new technologies. for multime(cid:173)
`dia. This research involves fast processors,
`high-speed networks,
`large-capacity storage
`devices, new algorithms and data structures, video
`and audio compression algorithms, graphics sys(cid:173)
`tems, human-computer interfaces, real-time oper(cid:173)
`ating systems, object-oriented programming,
`information storage and retrieval, hypertext and
`hypermedia, languages for scripting, parallel pro(cid:173)
`cessing methods, and complex architectures for
`distributed systems.
`
`Multimedia compression
`Compression techniques clearly play a crucial
`role in digital multimedia applications. Audio,
`image, and video signals produce a vast amount
`of data. Table 1 illustrates the m ass storage
`req uirements for various media types.
`Present multimedia systems require data com(cid:173)
`pression for three reasons: the large storage
`requirements of multimedia data, relatively slow
`storage devices that cannot play multimedia data
`(specifically video) in real time, and network
`
`bandwidth that does not allow real-time video
`data transmission.
`For example, a single frame of a color video,
`with 620- x 560-pixel frames at 24 bits per pixel,
`would take up about l Mbyte. At a real-time rate
`of 30 frames per second, that equals 30 Mbytes for
`one second of video. A typical multimedia appli(cid:173)
`cation might store more than 30 minutes of
`video, 2,000 images, and 40 minutes of stereo
`sound on each side of a laser disc. That applica(cid:173)
`tion would require about SO Gbytes of storage for
`video, 15 Gbytes for images, and 0.4 Gbytes for
`audio. That means a total of 65.4 Gbytes of stor(cid:173)
`age on the whole disc.
`Even if we had enough storage available, we
`wouldn't be able to play back t he video in real
`time due to the insufficient bit rate of storage
`devices. The speed of a real-time storage dev ice
`would need to be 30 Mbytes/s. However, today's
`CD-ROM technology provides a transfer rate of
`about 300 Kbytes/s. At the present state of storage
`device technology, the only solution is to com(cid:173)
`press the data before storage and decompress it
`before playback.
`Modern image and video compression tech(cid:173)
`n iq ues reduce these tremendous storage require(cid:173)
`ments. Advanced techniques can compress a
`typical image at a ratio ranging from 10:1 to 50:1,
`achieving video compression up to 2,000:1.
`Figure 1 illustrates storage requirements for a
`multimedia application consisting of various
`media types. compressing the images by a ratio of
`15:1 and the video by factors of 30:1 and 200:1.
`The total storage requirement for this application
`becomes a little over 2 Gbytes, much more feasi(cid:173)
`ble than 225.5 Gbytes uncompressed.
`Digital data compression relies on vacious com(cid:173)
`putational algorithms, implemented either in soft(cid:173)
`ware or in hardware. We can classify compression
`
`

`
`Shortn-
`JPEG
`
`H.261 px64
`
`MPEG
`
`Digital compression and coding of
`continuous-tone still images
`Video coder/decoder for audio-visual
`~es at px64 Kbps
`
`Coding of moving pictures and
`associated audio
`
`Stancbrds group
`Joint Photographk Experts Group
`
`Specialist Group on Coding for
`Visual Telephony
`
`Moving Pictures Experts Group
`
`Compression ntdos
`15:1 (full color still-frame
`applications)
`100:1 to 2000:1
`(video-based tele(cid:173)
`communications
`200: 1 Motion-intensive
`applications
`
`techniques into lossless and lossy approaches.'
`Lossless techniques can recover the original rep(cid:173)
`resentation perfectly. Lossy techniques recover the
`presentation with some loss of accuracy. The lossy
`techniques provide higher compression ratios,
`though, and therefore are applied more often in
`image and video compression than lossless
`techniques.
`We can further divide the lossy techniques into
`prediction-, frequency-, and importance-based
`techniques (such as
`techniques. Predictive
`ADPCM) predict subsequent values by observing
`previous values. Frequency-oriented techniques
`apply the discrete cosine tra nsform {DCf), relat(cid:173)
`ed to fast Fourier transform. Importance-oriented
`techn iques use other characteristics or images as
`the basis for compression; for example, the OVI
`technique employs color lookup tables and data
`filtering.
`Hybrid compression techniques combine sev(cid:173)
`eral approaches, such as OCT and vector quanti(cid:173)
`zation or differential pulse code modulation.
`Various groups have established standards for dig(cid:173)
`ital multimedia compression based on the exist(cid:173)
`ing JPEG, MPEG, and px64 standards, as Table 2
`shows.
`
`JPEG
`Originally, JPEG targeted full-color still frame
`applications. achieving a 15:1 average compres(cid:173)
`sion ratlo.2•3 However, some real-time, full-motion
`video applications also use JPEG. The JPEG stan(cid:173)
`da rd offers four modes of operation:
`
`1. sequential OCT-based encoding, which
`encodes each image component in a single
`left-to-right, top-to-bottom scan;
`
`2. progressive OCT-based encoding, which
`encodes the image in multiple scans in
`order to produce a quick, rough, decoded
`Image when the transmission time is long;
`
`3. lossless encoding, which encodes the image
`to guarantee an exact reproduction; and
`
`4. hierarchical encoding, which encodes the
`image at multiple resolutions.
`
`Th e JPEG algorithm decomposes the inpu t
`image into 8 x 8 source blocks. It shifts the pixels,
`originally in the range 0 to 51 1, to the range of
`- 128to +128, then t ransforms them into the fre(cid:173)
`quency domain using forwa rd discrete cosine
`transform (FOCT). The transformed 64-point dis(cid:173)
`crete signal is a function of two spatial dimen(cid:173)
`sions, x and y. Its components are called spatial
`frequencies, or OCT coefficients.
`For a typical 8 x 8 image block, most spatial fre(cid:173)
`quencies have zero or near-zero values and need
`not be encoded. This is the foundation for data
`compression. In the next step, all 64 OCT coeffi(cid:173)
`cients are quantized with the 64-elemcnt quanti(cid:173)
`zation
`table specified by the application.
`Quantization reduces the amplitude of the coeffi(cid:173)
`cients that contribute little or nothing to the qual(cid:173)
`ity of the image, thus increasing the number of
`zero-value coefficients.
`After quantizatio n, the OCT coefficients are
`ordered into the "zig-zag" sequence shown in
`Figure 2a (see next page), becau se the low-fre(cid:173)
`quency coefficients are more likely to be nonzero
`than the high-frequency coefficients (Figure 2b).
`Finally, the last stage of J PEG is entropy cod(cid:173)
`ing. The JPEG standard specifies two entropy cod(cid:173)
`ing methods: Huffman coding a nd arithmetic
`coding. The technique p rovides additional com(cid:173)
`pression by encoding the quantized OCT coeffi(cid:173)
`cients into a more compact form.3
`
`MPEG
`The MPEG standard is intended for compress(cid:173)
`ing full-motion video.• It uses interframe com(cid:173)
`pression, achieving compression ratios of up to
`200:1 by stori ng only the differences between sue-
`
`-··--·
`
`

`
`Figure 2. (a) Zig-zag
`ordering of DCT
`coefficients. (b) T11e
`proba/Jility oftlie
`coefficients being
`nonzero!
`
`Vertical frequency
`
`Probability of being nonzero
`
`8
`
`16
`
`40
`32
`24
`Zig-zag index
`
`48
`
`56
`
`b
`
`Horizontal frequency
`
`0 - 1
`
`5 - 6 14- 15 27- 28
`
`2 / 4 / 7 /13/16/26/29/42
`
`i / 8 /12/17/25/30/41/~3
`
`9/11/18/24/31/40/44/53
`
`0.5
`
`14
`
`/19/23/32/39/45/52/5
`
`10
`
`1
`
`
`
`20/22/33/38/' 46/51/55/60
`211/34/37/47/50/56/59/~1
`35~36/48~49/57~58/62~63
`
`a
`
`cessive frames . MPEG specifications also include
`an algorithm for compressing audio data at ratios
`from 5:1to10:1.
`MPEG codes frames in a sequence using three
`different algorithms, as Figure 3 shows. A OCT·
`based algorithm similar to JPEG first codes
`intraframes (!). To exploit temporal redundancy
`between frames, MPEG codes the remaining
`frames using two prediction techniques. One
`codes predicted frames (P) with forward predictive
`coding, where the actual frame is coded with
`reference to a past frame. The other codes inter(cid:173)
`polated, or bidirectional, frames (B) with bidirec(cid:173)
`tionally predicted, interpolated coding, also caUed
`motion-compensated interpolation. Bidirectional
`prediction uses a past and a future frame to code
`current frames, providing the highest amount of
`compression.
`The coding process for P and B frames includes
`a motion estimator that finds the best matching
`block common to the reference frames. The
`motion vector then specifies the distance between
`predicted and actual blocks. The difference, called
`the error term, is then encoded using the DCT(cid:173)
`based transform coding.
`The present standard, called MPEG-1, com(cid:173)
`presses 320 x 240 full-motion video in applica(cid:173)
`tions such as
`interactive multimedia and
`broadcast television. The minimum data rate
`required is 1.5 Mbps.
`MPEG-2 will compress 720 x 480 fu ll-motion
`video in broadcast television and video-on(cid:173)
`demand applications. It will require a data rate in
`the range of 4 to 10 Mbps and will provide VCR(cid:173)
`quality video.
`
`Future broadcast television and video-on(cid:173)
`demand services will use MPEG-3 to compress
`full-motion, HDTV-quality video. The projected
`required data rate is 5 to 20 Mbps.
`Full-motion video applications, such as inter(cid:173)
`active multimedia and video telephony, that con(cid:173)
`sist of small frames and require slow refreshing
`will use MPEG-4. Such applications will need a
`data rate of 9 to 40 Kbps.
`
`px64
`The H.261 standard, commonly called px64,
`achieves very high compression ratios for full(cid:173)
`color, real-time motion video transmission. The
`algorithm combines intraframe and interframe
`coding to provide fast processing for on-the-fly
`video compression and decompression, optimized
`for applications such as video-based telecommu(cid:173)
`nications. Because its applications usually are not
`motion-intensive, the algorithm uses limited
`motion search and estimation strategies to
`achieve higher compression ratios. For standard
`video communication images, px64 can achieve
`compression ratios of 100:1 to more than 2,000:1.
`The standard covers the entire ISON channel
`capacity (px64 Kbps, p = l, 2, ... , 30). This increas(cid:173)
`es the ISON channel capacity from 64 Kbps to
`2.048 Mbps. The video coding algorithm is
`intended for real-time communications requiring
`minimum delays. For p = 1 or 2, due to limited
`available bandwidth, this algorithm only imple(cid:173)
`ments desktop face-to-face visual communica(cid:173)
`tions (videophone). However, for p of 6 or higher,
`more complex pictures are transmitted, suiting
`the algorithm for videoconferencing.
`
`

`
`Forward prediction
`
`FiKfire 3. The MPEG
`compression algorithm,
`showing (a) a group of
`frames and (b) the
`MPEG coding procedure.
`
`a
`
`I frame
`
`Color
`space
`converter
`
`(RGB-+-YUV)
`
`+
`
`Color
`space
`converter
`
`(RGB-+-YUV) -
`
`b
`
`Reference
`!Tame
`
`Bidirectional pred iction
`
`Quantization
`
`Entropy
`encoder
`
`Compressed image data
`100111001 ...
`
`Entropy
`encoder
`
`Compressed image data
`100111001 ...
`
`Motion
`estimator
`
`px64 operates with two picture formats adopt(cid:173)
`ed by the CCIIT: the conunon intermediate for(cid:173)
`mat (CIF), and the quarter-CIF (QCIF).~ The
`standard consists of a OCT-based compression
`algorithm, similar to JPEG, and a differential pulse
`code modulation (!)PCM) algorithm with motion
`estimation.
`lntraframe mode codes and quantizes frames
`using the ocr transform coding, then sends each
`to the video f!! Ultiplex coder. The inverse quan(cid:173)
`tizer and mer decompress the frames, then store
`them in the picture memory for interframe cod(cid:173)
`ing.
`lnterframe coding uses DPCM-based prediction
`to compare every macro block of the actual frame
`with the available macro blocks of th e previous
`frame. The algorithm then creates the difference
`as error terms that are OCT-coded, quantized, and
`sent to the video multiplex coder along with the
`motion vector. The final stage uses variable word(cid:173)
`length en tropy coding (such as the Huffman
`coder) to produce more compact code.
`
`Implementing compression algorithms
`When implementi ng a compression/decom-
`
`pression algorithm, the key question is how to
`partition between hardware and software in order
`to maximize performance and minimize cost.
`Most implementations use specialized video
`processors and programmable digital signal
`processors (DSPs). However, powerful RISC proces(cid:173)
`sors are making software-only solutions feasible.
`We can classify implementations of compres·
`sion algorithms into three categories: (1) a hard(cid:173)
`wired approach that maximizes performance (for
`example, C cube), (2) a software solution that
`emphasizes flexibility with a general-purpose
`processor, and (3) a hybrid approach that uses spe(cid:173)
`cialized video processors.
`AT&T took the hybrid approach, creating the
`AVP 4310E encoder and the AVP 42200 decoder
`for the px64 and MPEG standards. 6 The encoder
`accepts video Input at 30 frames/s and outputs
`data at a selectable data rate from 40 Kbytes/s to 4
`Mbytes/s. The ha rdware implements computa·
`tionally intensive functions, such as motion esti(cid:173)
`mation and Huffman coding. The user can
`program key parameters, such as frame rate, delay,
`bit-rate, and resolution. A programmable RISC
`processor implements less stable functions.
`
`_____ __._
`
`

`
`characteristks
`Data rate
`T raffle pattern
`Reliability requirements
`Latency requirements
`Mode of communication
`Temporal relationship
`
`No loss
`None
`Point-to-point
`None
`
`Stream"Oriented highly bursty
`Some km
`Low, for example, 20 m s
`Multipoint
`Synchronized transmission
`
`Bandwidth
`managem ent
`softWare
`
`Issues regular
`priority tokens
`
`Priority token ring
`
`Regular
`client
`
`Priority
`multimedia
`client
`
`Figilte 4. Priority token
`ring suits multimedia
`applicatio11s.
`
`Researchers at UC Berkeley implernented a
`software MPEG encoder in C using X Windows
`and analyzed its performance on different com(cid:173)
`puter platforms.7 The results showed that current
`RISC workstations, such as the HP750, can decode
`a 320 x 240 video sequence at 10 to 15 frames/s,
`about half the rate of real-time performance. The
`group anticipated a new generation of worksta(cid:173)
`tions capable of real-time, software-only decoding
`of video signals.
`
`Multimedia networking
`Many applications, such as video mail, video
`conferencing, and collaborative work systems,
`require networked multimedia. In these applica(cid:173)
`tions, the multimedia objects are stored at a serv(cid:173)
`er and played back at the clients' sites. Such
`applications might require broadcasting multi(cid:173)
`media data to various remote locations or access(cid:173)
`ing large depositories of multimedia sources.
`Traditional LAN environments, in which data
`sources are locally available, cannot support access
`to remote multimedia data sources for a number
`of reasons. Table 3 contrasts traditional data trans(cid:173)
`fer and multimedia transfer.
`Multimedia networks require a very high trans(cid:173)
`fer rate or bandwidth, even when the data is com(cid:173)
`pressed. For example, an MPEG- 1 session requires
`a bandwidth of about 1.5 Mbps. MPEG-2 through
`-4 will take 4 to 10 Mbps, while the projected
`
`required bandwidth for HDTV is 5 to 20 Mbps.
`Besides being h igh, the transfer rate must also be
`predictable.
`Traditional networks are used to provide error(cid:173)
`free transmission. However, most multimedia
`applications can tolerate errors in transmission
`due to corruption or packet loss without retrans(cid:173)
`mission or correction. In some cases, to meet real(cid:173)
`time delivery requiremen ts or to achieve
`synch ronization, some packets are even discard(cid:173)
`ed. As a result, we can apply lightweight trans(cid:173)
`mission protocols to multimedia networks. These
`protocols cannot accept retransmission, since that
`might introduce unacceptable delays.
`Multimedia networks must provide the low
`latency required for interactive operation. Since
`multimedia data must be synchronized when it
`arrives at the destination site, networks should
`provide synchronized transmission with low jitter.
`In multimedia networks, most communica(cid:173)
`tions are mu ltipoint, as opposed to traditional
`point-to-point communication. For example, con(cid:173)
`ferences involving more than two participants
`need to distribute information in different media
`to each participant. Confere11ce 11etworks use mul(cid:173)
`ticasting and bridging distribution methods.
`Multicasting replicates a single input signal and
`delivers it to multiple destinations. Bridging com(cid:173)
`bines multiple input signals into one or more
`output signals, which it then delivers to the
`participants.a
`Traditional networks do not suit multimedia.
`Ethernet provides only 10 Mbps, its access time is
`not bounded, and its latency and jitter are unpre(cid:173)
`dictable.Token-ring networks provide 16 Mbps
`and are deterministic; from this point of view,
`they can handle multimedia. However, the pre(cid:173)
`dictable worst-case access latency can be very
`high.
`An FDDl n etwork provides 100 Mbps band(cid:173)
`width, sufficient for multimedia. In the synchro(cid:173)
`nized mode, FDDI has low access latency and low
`jitter. FDDI also guarantees a bounded access delay
`and a predictable average bandwidth for synchro(cid:173)
`nous traffic. However, due to the high cost, FDDI
`networks are used primarily for backbone net(cid:173)
`works, rather than networks of workstations.
`Less expensive alternatives include enhanced
`traditional networks. Fast Ethernet, for example,
`provides up to 100 Mbps bandwidth. Priority
`token ring is another system.
`In priority token ring networks, the multime(cid:173)
`dia traffic is separated from regular traffic by pri(cid:173)
`ority (see Figure 4). The bandwidth manager plays
`
`

`
`a crucial role by tracking sessions, determining
`ratio priority, and registering multimedia sessions.
`Priority token ring (PTR) works on existing net(cid:173)
`works and does not require configuration control.
`The admission control in PTR guarantees band(cid:173)
`width to multimedia sessions; however, regular
`traffic can experience delays. For example, let's
`assume a priority token ring network at 16 Mbps
`that connects 32 nodes. When no priority scheme
`is set, each node gets an average of 0.5 Mbps of
`bandwidth. When half the bandwidth (8 Mbps) is
`dedicated to multimedia, the network can handle
`about 5 MPEG sessions (at 1.S Mbps). In that case,
`the remaining 27 nodes can expect about 8 Mbps
`divided by 27, or 0.296 Mbps, about half of what
`they would get without priority enabled.
`Crimmlns9 evaluated
`three priority ring
`schemes for their applicability to video confer(cid:173)
`encing applications: (1) equal priority for video
`and asynchronous packets, (2) permanent h igh
`priority for video packets and permanent low pri(cid:173)
`ority for asynchronous packets, and (3) time(cid:173)
`adjusted high-priority for video packets (based on
`their ages) and permanent low priority for asyn(cid:173)
`chronous packets.
`The first scheme, which en tails direct compe(cid:173)
`tition between video conference and asynchro(cid:173)
`nous stations, achieves the lowest network delay
`for asynchronous traffic. However, it reduces the
`video conference quality. The second scheme, in
`which video conference stations have the perma(cid:173)
`nent high priority, produces no degradation in
`conference quality, but increases the asynchro(cid:173)
`nous network delay. Finally, the time-adjusted pri(cid:173)
`ority system provides a trade-off between first two
`schemes. The quality of video conferencing is bet(cid:173)
`ter than in the first scheme, while the asynchro(cid:173)
`n ous network delays are shorter than in the
`second scheme.9
`Present optical network technology can sup(cid:173)
`port the Broadband Integrated Services Digital
`Network (B-ISDN) standard, expected to become
`the key network for multimedia applications.10
`B-ISDN access can be basic or primary. Basic ISDN
`access supports 28 + D channels, where the trans(cid:173)
`fer rate of a B channel is 64 Kbps, and that of a D
`chan nel is 16 Kbps. Primary CSDN access supports
`238 + D in the US and 30B + D in Europe.
`The two B channels of the ISON basic access
`provide 2 x 64 Kbps, or 128 Kbps of composite
`bandwidth. Conferences can use part of this
`capacity for wideband speech, saving the remain(cid:173)
`der for purposes such as control, meeting data,
`and compressed video. 11
`
`Multimedia object composition
`
`Spatial composition
`
`Temporal composition
`
`Point
`synchronization
`
`Continuous
`synchronization
`
`EJ ~
`1
`[ Image I
`
`a
`
`t,
`
`b
`
`t,
`
`Time
`
`Figu re S. Multimedia
`objecrs composition.
`
`Proposed B-ISDN networks are in either syn(cid:173)
`chronous transfer mode (STM) or asynchronous
`transfer mode (ATM), to handle both constant
`and variable bit-rate traffic applications. STM pro(cid:173)
`vides fixed bandwidth channels, and therefore is
`not flexible enough to handle the different types
`of traffic typical in multimedia applications. On
`the other hand, ATM is suitable for multimedia
`traffic; it provides great flexibility in bandwidth
`allocation by assigning fixed length packets, called
`cells, to virtual connection. ATM can also increase
`the bandwidth efficiency by buffering and statis(cid:173)
`tically multiplexing bursty traffic at the expense
`of cell delay and loss.10
`
`Multimedia synchronization
`Multimedia systems include multiple sou rces
`of various media either spatially or temporally to
`create composite multimedia documents. Spatial
`composition links various multimedia objects into
`a single entity (Figure Sa), dealing with object size,
`rotation, and placement with in the entity.
`Temporal composition creates a multimedia pre(cid:173)
`sen tation by arranging the multimedia objects
`according to temporal relationship (Figure Sb).12
`We can divide temporal composition, or syn(cid:173)
`chronization, into continuous and poin t synchro(cid:173)
`nization. Continuous synchronization requires
`constant synchronization of lengthy events. An
`example of contin uous syn chronizat ion is video
`telephony, where audio and video signals are cre(cid:173)
`ated at a remote site, transmitted over the net (cid:173)
`work, then synchronized continuously at the
`
`

`
`Source
`a
`
`Destinations
`
`b
`
`Source Source
`
`Destinations
`
`Source
`c
`
`Destinations
`
`Source
`d
`
`Source
`
`Destinations
`
`Flgi1re 6. Locati<m
`models for multimedia
`data: (a) local single
`source, (b) local
`multiple source, (c)
`distributed single
`source, and (d)
`distributed multiple
`source.
`
`receiver site for playback. In point synchroniza(cid:173)
`tion, a single point of one media block coincides
`with a single point of another media block. An
`example of point synchronization is a slide show
`with blocks of audio allotted to each slide.
`Two further classes of synchronization are
`serial and parallel synchronization. Serial syn(cid:173)
`chronization determines the rate at which events
`must occur within a single data stream (intrame(cid:173)
`dia synchronization), while parallel synchroniza(cid:173)
`tion determines the relative schedule of separate
`synchronization streams (intermedia synchro·
`nization).
`
`Data location models
`Responsibility for maintaining intermedia syn(cid:173)
`chronization falls onto both the sources and des-
`
`tinations of data, but most techniques rely more
`on the destinations. One topology classification
`that can determine the required synchronization
`builds on data location models. '2•13 Figure 6 shows
`four data location models:
`
`1. Local single source. A single source, such as
`a CD-ROM, distributes the media streams to
`the playback devices. As long as the devices
`maintain their playback speed, no synchro(cid:173)
`nization technique is required.
`
`2. Local multiple sources. More than one source
`distributes media streams to the playback
`devices. An example is a slide show played
`with music or an audio tape. Synchroniza(cid:173)
`tion is required within the workstation.
`
`3. Distributed single source. One source, such
`as a videotape, distributes media streams
`across a network to one or more nodes'
`playback devices; an example is cable TV.
`The technique requires no synchronization
`other than maintaining the speeds of the
`playback devices.
`
`4. Distributed multiple sources. This is the most
`complex case, where more than one source
`distributes media streams to multiple
`playback devices on multiple nodes. This
`group further breaks down into multiple
`sources from one node distributed to another
`
`Figure 7. Various causes
`of asynchrony in a video
`telephony system.11
`
`Video
`source
`
`Video
`encoder
`
`Audio
`source
`
`Audio
`encoder
`
`l
`
`Different encoding
`times for video
`and audio
`
`<II :s QI
`~ ·.:;
`"5
`~
`w
`w
`~
`
`c:
`0
`'iii '-
`"§~
`c: u
`~
`
`Network
`8-ISDN
`
`c:
`·~ w
`.~~ EO
`££ ~
`.,"O
`,:
`
`Video
`encoder
`
`Audio
`decoder
`
`l
`
`1
`
`Variations in fjacket
`arrival times jitter)
`introduced by:
`• independent routing
`paths,
`- packet loss
`· packet buffering
`
`c:
`0 o·.=
`., "'
`"O c:
`.,
`5·t;
`"O
`
`c:
`0
`·-"'
`0'..J
`"O c:
`~·~ .,
`
`"O
`
`+
`
`Frequencies of
`Start-up delay due
`physica l display
`to decompression
`aevices do not
`pipeline can cause
`match; audio
`video to trail audio
`where decompression and video may
`not stay in synch
`dela~ is generally
`sma er; skew in
`over long periods
`display
`
`

`
`Figure 9. Nomitral and
`a"ctllal streams, s11owing
`skew, jitter, u tilization,
`and speed.
`
`tTrimmit
`
`Figure 8 . Definition of
`tlte end-to-end delay.
`
`'iencode
`
`'tautter
`
`'toepacketi:e
`
`'toe<:odre
`
`'tPresent
`
`Stream A
`(Nominal)
`
`Stream A
`(Actual)
`
`Normal
`
`litter
`
`Skew
`
`Skew correctio
`
`to
`
`tl
`
`Time
`
`t2
`
`node (for example, a video call); mult iple
`sources from two or more nodes distributed
`to another node; multiple sources from one
`node distributed to two or more nodes (for
`example, HDTV); and multiple sources from
`two or more nodes distributed to two or more
`nodes (for example, a group teleconference).
`
`For the first two location models, local syn(cid:173)
`chronization within the workstation suffices.
`However, the two cases with distributed sources
`require more complex synchronization algo(cid:173)
`rithms to eliminate the various causes of asyn(cid:173)
`chrony. '2 Figure 7 shows an example of video
`telephony, noting various places within the sys(cid:173)
`tems that contribute to asynchrony. The task of
`synchronization, whether implemented in the
`network or in th e receiver, is to eliminate all the
`variations and delays incurred during the trans(cid:173)
`mission of multiple media streams and to main(cid:173)
`tain synchronization among the media streams.
`The end-to-end delay of a distributed multi(cid:173)
`media system consists of all the delays created at
`the source site, network, and receiver site.12 It d if(cid:173)
`fers slightly for real-time video and audio and for
`stored multimedia objects, as shown in Figure 8.
`
`Quality of service
`Implementing a synchronization algorithm for
`a specific application requires specifying the qua