PROCEEDINGS OF SPIE
`
`SPIEDigitalLibrary.org/conference-proceedings-of-spie
`
`(cid:40)(cid:85)(cid:85)(cid:82)(cid:85)(cid:3)(cid:85)(cid:72)(cid:86)(cid:76)(cid:79)(cid:76)(cid:72)(cid:81)(cid:87)(cid:3)(cid:89)(cid:76)(cid:71)(cid:72)(cid:82)(cid:3)(cid:70)(cid:82)(cid:71)(cid:76)(cid:81)(cid:74)(cid:3)(cid:88)(cid:86)(cid:76)(cid:81)(cid:74)
`(cid:89)(cid:76)(cid:85)(cid:87)(cid:88)(cid:68)(cid:79)(cid:3)(cid:85)(cid:72)(cid:73)(cid:72)(cid:85)(cid:72)(cid:81)(cid:70)(cid:72)(cid:3)(cid:83)(cid:76)(cid:70)(cid:87)(cid:88)(cid:85)(cid:72)
`
`(cid:61)(cid:75)(cid:68)(cid:81)(cid:74)(cid:15)(cid:3)(cid:42)(cid:88)(cid:68)(cid:81)(cid:77)(cid:88)(cid:81)(cid:15)(cid:3)(cid:54)(cid:87)(cid:72)(cid:89)(cid:72)(cid:81)(cid:86)(cid:82)(cid:81)(cid:15)(cid:3)(cid:53)(cid:82)(cid:69)(cid:72)(cid:85)(cid:87)
`
`(cid:42)(cid:88)(cid:68)(cid:81)(cid:77)(cid:88)(cid:81)(cid:3)(cid:61)(cid:75)(cid:68)(cid:81)(cid:74)(cid:15)(cid:3)(cid:53)(cid:82)(cid:69)(cid:72)(cid:85)(cid:87)(cid:3)(cid:47)(cid:17)(cid:3)(cid:54)(cid:87)(cid:72)(cid:89)(cid:72)(cid:81)(cid:86)(cid:82)(cid:81)(cid:15)(cid:3)(cid:5)(cid:40)(cid:85)(cid:85)(cid:82)(cid:85)(cid:3)(cid:85)(cid:72)(cid:86)(cid:76)(cid:79)(cid:76)(cid:72)(cid:81)(cid:87)(cid:3)(cid:89)(cid:76)(cid:71)(cid:72)(cid:82)(cid:3)(cid:70)(cid:82)(cid:71)(cid:76)(cid:81)(cid:74)(cid:3)(cid:88)(cid:86)(cid:76)(cid:81)(cid:74)
`(cid:89)(cid:76)(cid:85)(cid:87)(cid:88)(cid:68)(cid:79)(cid:3)(cid:85)(cid:72)(cid:73)(cid:72)(cid:85)(cid:72)(cid:81)(cid:70)(cid:72)(cid:3)(cid:83)(cid:76)(cid:70)(cid:87)(cid:88)(cid:85)(cid:72)(cid:15)(cid:5)(cid:3)(cid:51)(cid:85)(cid:82)(cid:70)(cid:17)(cid:3)(cid:54)(cid:51)(cid:44)(cid:40)(cid:3)(cid:24)(cid:25)(cid:27)(cid:24)(cid:15)(cid:3)(cid:44)(cid:80)(cid:68)(cid:74)(cid:72)(cid:3)(cid:68)(cid:81)(cid:71)(cid:3)(cid:57)(cid:76)(cid:71)(cid:72)(cid:82)(cid:3)(cid:38)(cid:82)(cid:80)(cid:80)(cid:88)(cid:81)(cid:76)(cid:70)(cid:68)(cid:87)(cid:76)(cid:82)(cid:81)(cid:86)
`(cid:68)(cid:81)(cid:71)(cid:3)(cid:51)(cid:85)(cid:82)(cid:70)(cid:72)(cid:86)(cid:86)(cid:76)(cid:81)(cid:74)(cid:3)(cid:21)(cid:19)(cid:19)(cid:24)(cid:15)(cid:3)(cid:3)(cid:11)(cid:20)(cid:23)(cid:3)(cid:48)(cid:68)(cid:85)(cid:70)(cid:75)(cid:3)(cid:21)(cid:19)(cid:19)(cid:24)(cid:12)(cid:30)(cid:3)(cid:71)(cid:82)(cid:76)(cid:29)(cid:3)(cid:20)(cid:19)(cid:17)(cid:20)(cid:20)(cid:20)(cid:26)(cid:18)(cid:20)(cid:21)(cid:17)(cid:24)(cid:27)(cid:26)(cid:22)(cid:19)(cid:25)
`(cid:40)(cid:89)(cid:72)(cid:81)(cid:87)(cid:29)(cid:3)(cid:40)(cid:79)(cid:72)(cid:70)(cid:87)(cid:85)(cid:82)(cid:81)(cid:76)(cid:70)(cid:3)(cid:44)(cid:80)(cid:68)(cid:74)(cid:76)(cid:81)(cid:74)(cid:3)(cid:21)(cid:19)(cid:19)(cid:24)(cid:15)(cid:3)(cid:21)(cid:19)(cid:19)(cid:24)(cid:15)(cid:3)(cid:54)(cid:68)(cid:81)(cid:3)(cid:45)(cid:82)(cid:86)(cid:72)(cid:15)(cid:3)(cid:38)(cid:68)(cid:79)(cid:76)(cid:73)(cid:82)(cid:85)(cid:81)(cid:76)(cid:68)(cid:15)(cid:3)(cid:56)(cid:81)(cid:76)(cid:87)(cid:72)(cid:71)(cid:3)(cid:54)(cid:87)(cid:68)(cid:87)(cid:72)(cid:86)
`
`(cid:39)(cid:82)(cid:90)(cid:81)(cid:79)(cid:82)(cid:68)(cid:71)(cid:72)(cid:71)(cid:3)(cid:41)(cid:85)(cid:82)(cid:80)(cid:29)(cid:3)(cid:75)(cid:87)(cid:87)(cid:83)(cid:86)(cid:29)(cid:18)(cid:18)(cid:90)(cid:90)(cid:90)(cid:17)(cid:86)(cid:83)(cid:76)(cid:72)(cid:71)(cid:76)(cid:74)(cid:76)(cid:87)(cid:68)(cid:79)(cid:79)(cid:76)(cid:69)(cid:85)(cid:68)(cid:85)(cid:92)(cid:17)(cid:82)(cid:85)(cid:74)(cid:18)(cid:70)(cid:82)(cid:81)(cid:73)(cid:72)(cid:85)(cid:72)(cid:81)(cid:70)(cid:72)(cid:16)(cid:83)(cid:85)(cid:82)(cid:70)(cid:72)(cid:72)(cid:71)(cid:76)(cid:81)(cid:74)(cid:86)(cid:16)(cid:82)(cid:73)(cid:16)(cid:86)(cid:83)(cid:76)(cid:72)(cid:3)(cid:82)(cid:81)(cid:3)(cid:20)(cid:24)(cid:3)(cid:45)(cid:88)(cid:81)(cid:3)(cid:21)(cid:19)(cid:21)(cid:19)(cid:3)(cid:3)(cid:55)(cid:72)(cid:85)(cid:80)(cid:86)(cid:3)(cid:82)(cid:73)(cid:3)(cid:56)(cid:86)(cid:72)(cid:29)(cid:3)(cid:75)(cid:87)(cid:87)(cid:83)(cid:86)(cid:29)(cid:18)(cid:18)(cid:90)(cid:90)(cid:90)(cid:17)(cid:86)(cid:83)(cid:76)(cid:72)(cid:71)(cid:76)(cid:74)(cid:76)(cid:87)(cid:68)(cid:79)(cid:79)(cid:76)(cid:69)(cid:85)(cid:68)(cid:85)(cid:92)(cid:17)(cid:82)(cid:85)(cid:74)(cid:18)(cid:87)(cid:72)(cid:85)(cid:80)(cid:86)(cid:16)(cid:82)(cid:73)(cid:16)(cid:88)(cid:86)(cid:72)
`
`UNIFIED 1008
`
`

`

`Error Resilient Video Coding Using Virtual Reference Picture
`
`Guanjun Zhang and Robert L. Stevenson
`Department of Electrical Engineering
`University of Notre Dame
`Notre Dame, IN 46556, USA
`
`ABSTRACT
`Due to widely used motion-compensated prediction coding, errors propagate along decoded video sequence and may result
`in severe quality degradation. Various methods have been reported to address this problem based on the common idea of
`diversifying prediction references. In this paper, we present an alternative way of concealing the references pictures errors.
`A generated virtual picture is used as a reference instead of an actual sequence picture in the temporal prediction. The
`virtual reference picture is generated in a way to filter damaged parts of previously decoded pictures so that the decoder
`can still get a clean reference picture in case of errors. Coding efficiency is effected due to the fact that the virtual reference
`is less correlated to the currently encoded picture. The simulations on H.264 codec have shown quality improvement of
`the proposed method over intra-coded macroblock refreshment. It can be used on any motion-compensated video codec to
`combat channel errors.
`
`1. INTRODUCTION
`Error resilience become an important feature of video transmission over error-prone networks. Generating error resilient
`video bit streams has become an intensive research area and many solutions has been reported to address this problem.
`In conventional motion-compensated prediction video coding, errors in a single picture may cause mismatch between
`encoder and decoder so that the effect of errors may propagate along decoded video sequence and thus result in severe
`quality degradation. A common way is using independently decodable video coding structures in different levels from
`macroblock to sequence. The picture level refreshment is commonly found as I frame in all coding standards. In sequence
`level, there is multiple description coding, such as Video Redundancy Coding(VRC) in H.263+. 1 The refreshment using
`intra coded macroblocks has been proven to be an effective way to combat errors due to packet loss or bit errors. The side
`effect of using intra coded macroblocks is loss in coding efficiency.
`A another basic idea of concealing error effect in inter coded video stream is to use robust prediction reference so
`that the visual quality variation is alleviated. Girod et. al. diversify the references of motion-compensated prediction to
`improve the coding efficiency 2,3 while Wang et. al. found out it can also function against errors, 4 as long as a balance
`between efficiency and robustness is achieved. Multiple references of motion-compensated prediction has been adopted
`into H.263 and H.26L as an important tool for robustness as well as higher coding efficiency 5.6 Reference of current
`pictures are distributed in several previously coded pictures. If each currently coded block use one reference block, the
`situation is not different from conventional prediction since a block is damaged when its reference has errors. If more than
`one reference blocks are searched, it may cost a large amount of additional time and power. In 7 the authors find out that,
`although multiple reference are used, the actual references are unevenly located in the temporally previous picture and a
`fast motion estimation algorithm is then developed.
`In error concealment capable video codecs, the decoders always first try to conceal errors in current decoded pictures
`before using them as references. If back channels are not available or timely inapplicable, the decoders usually directly
`copy the same area from another picture. If more time and complexity is affordable, a space interpolation or motion-
`compensated interpolation can be applied to yield more accurate reconstructions.
`To further improve error concealment, methods of flexible slice structures are standardized in MPEG-4 AVC/H.264.
`It basically distributes macroblocks from different areas of the pictures into a slice structure. When a slice is crushed,
`the benefits are two-fold: visual quality is less effected since the macroblocks are scattered in the picture; and space
`Tel: (574)631-8308 Fax: (574)631-4393 Email: {gzhang, rls}@nd.edu
`
`896
`
`Image and Video Communications and Processing 2005, edited by Amir Said,
`John G. Apostolopoulos, Proc. of SPIE-IS&T Electronic Imaging,
`SPIE Vol. 5685 © 2005 SPIE and IS&T · 0277-786X/05/$15
`
`(cid:39)(cid:82)(cid:90)(cid:81)(cid:79)(cid:82)(cid:68)(cid:71)(cid:72)(cid:71)(cid:3)(cid:41)(cid:85)(cid:82)(cid:80)(cid:29)(cid:3)(cid:75)(cid:87)(cid:87)(cid:83)(cid:86)(cid:29)(cid:18)(cid:18)(cid:90)(cid:90)(cid:90)(cid:17)(cid:86)(cid:83)(cid:76)(cid:72)(cid:71)(cid:76)(cid:74)(cid:76)(cid:87)(cid:68)(cid:79)(cid:79)(cid:76)(cid:69)(cid:85)(cid:68)(cid:85)(cid:92)(cid:17)(cid:82)(cid:85)(cid:74)(cid:18)(cid:70)(cid:82)(cid:81)(cid:73)(cid:72)(cid:85)(cid:72)(cid:81)(cid:70)(cid:72)(cid:16)(cid:83)(cid:85)(cid:82)(cid:70)(cid:72)(cid:72)(cid:71)(cid:76)(cid:81)(cid:74)(cid:86)(cid:16)(cid:82)(cid:73)(cid:16)(cid:86)(cid:83)(cid:76)(cid:72)(cid:3)(cid:82)(cid:81)(cid:3)(cid:20)(cid:24)(cid:3)(cid:45)(cid:88)(cid:81)(cid:3)(cid:21)(cid:19)(cid:21)(cid:19)
`(cid:55)(cid:72)(cid:85)(cid:80)(cid:86)(cid:3)(cid:82)(cid:73)(cid:3)(cid:56)(cid:86)(cid:72)(cid:29)(cid:3)(cid:75)(cid:87)(cid:87)(cid:83)(cid:86)(cid:29)(cid:18)(cid:18)(cid:90)(cid:90)(cid:90)(cid:17)(cid:86)(cid:83)(cid:76)(cid:72)(cid:71)(cid:76)(cid:74)(cid:76)(cid:87)(cid:68)(cid:79)(cid:79)(cid:76)(cid:69)(cid:85)(cid:68)(cid:85)(cid:92)(cid:17)(cid:82)(cid:85)(cid:74)(cid:18)(cid:87)(cid:72)(cid:85)(cid:80)(cid:86)(cid:16)(cid:82)(cid:73)(cid:16)(cid:88)(cid:86)(cid:72)
`
`

`

`interpolation is more efficient because damaged macroblocks have more opportunities to be surrounded by error-free
`macroblocks from other slice. Although these concealment steps utilize clean content from earlier decoded pictures, the
`damaged areas can only be partially recovered in most cases. The mismatch problem still exists, if the concealed areas do
`not happen to completely match the original ones.
`In this paper, we present a video coding system which has the capability to combat the error propagation in a damaged
`video sequence. A virtual reference picture(VRP) is generated by using a non-linear filter on several real pictures in both
`encoder and decoder, and then used to predict the incoming picture. It become an novel solution to the mismatch problem
`in terms of more chances to completely recover the damaged areas than the above systems.
`The rest of this paper is organized as follows. In section 2, we will introduce the video coding system using virtual
`reference pictures and discuss its error resilience capability. In section 3, we will describe the simulation environment.
`Simulation results will be presented and discussed in section 4, which is followed by the conclusions.
`
`2. VIRTUAL REFERENCE PICTURE
`In today’s popular video codecs, motion-compensated prediction uses earlier decoded pictures as references, which are
`actually displayed after decoding. If a back channel or retransmission is not available to the system, a damaged picture
`becomes a damaged reference at the decoder. We argue that it is not necessary to use a real video frame as a prediction
`reference. A virtual picture generated from real pictures can also be use as the reference, as long as it has appropriate error
`resilience properties and acceptable price in the sense of efficiency loss.
`We propose a video coding system, as illustrated in figure 1. The encoder generates a virtual picture by applying a
`nonlinear filter on several previously coded pictures. It then uses the virtual picture to predict the incoming picture. A
`same filter is used in the decoder to generate matched reference picture as in the encoder. We assume that there is no back
`channel available.
`
`. . .
`
`DCT
`
`Q
`
`IQ
`
`IDCT
`
`Frame
`Memory
`
`Motion Vectors
`
`Motion
`Compensation
`
`Virtual
`Reference
`Picture
`Generation
`
`Motion
`Estimation
`
`Entropy
`Encoding
`
`1010011...
`
`Entropy
`Decoding
`
`IQ
`
`IDCT
`
`1010011 . . .
`
`Motion Vectors
`
`Motion
`Compensation
`
`. .
`
`Frame
`Memory
`
`Virtual
`Reference
`Picture
`Generation
`
`(a) Encoder
`
`(b) Decoder
`
`Figure 1. A video coding system using virtual reference pictures.
`
`In an error free environment, both encoder and decoder use identical virtual reference pictures. In case of errors, the
`decoder first conceals the errors in a picture by spatial interpolation or simple temporal interpolation, such as a copy from
`the same areas in a previously decoded picture. Then the virtual reference picture is yielded by using the concealed picture
`and several other decoded pictures.
`If we consider the concealed picture as a copy of the encoded picture which is damaged by additive noise, the filter
`shall have the capability to conceal the noise from this picture and generate a virtual picture mostly using information from
`
`SPIE-IS&T/ Vol. 5685 897
`
`(cid:39)(cid:82)(cid:90)(cid:81)(cid:79)(cid:82)(cid:68)(cid:71)(cid:72)(cid:71)(cid:3)(cid:41)(cid:85)(cid:82)(cid:80)(cid:29)(cid:3)(cid:75)(cid:87)(cid:87)(cid:83)(cid:86)(cid:29)(cid:18)(cid:18)(cid:90)(cid:90)(cid:90)(cid:17)(cid:86)(cid:83)(cid:76)(cid:72)(cid:71)(cid:76)(cid:74)(cid:76)(cid:87)(cid:68)(cid:79)(cid:79)(cid:76)(cid:69)(cid:85)(cid:68)(cid:85)(cid:92)(cid:17)(cid:82)(cid:85)(cid:74)(cid:18)(cid:70)(cid:82)(cid:81)(cid:73)(cid:72)(cid:85)(cid:72)(cid:81)(cid:70)(cid:72)(cid:16)(cid:83)(cid:85)(cid:82)(cid:70)(cid:72)(cid:72)(cid:71)(cid:76)(cid:81)(cid:74)(cid:86)(cid:16)(cid:82)(cid:73)(cid:16)(cid:86)(cid:83)(cid:76)(cid:72)(cid:3)(cid:82)(cid:81)(cid:3)(cid:20)(cid:24)(cid:3)(cid:45)(cid:88)(cid:81)(cid:3)(cid:21)(cid:19)(cid:21)(cid:19)
`(cid:55)(cid:72)(cid:85)(cid:80)(cid:86)(cid:3)(cid:82)(cid:73)(cid:3)(cid:56)(cid:86)(cid:72)(cid:29)(cid:3)(cid:75)(cid:87)(cid:87)(cid:83)(cid:86)(cid:29)(cid:18)(cid:18)(cid:90)(cid:90)(cid:90)(cid:17)(cid:86)(cid:83)(cid:76)(cid:72)(cid:71)(cid:76)(cid:74)(cid:76)(cid:87)(cid:68)(cid:79)(cid:79)(cid:76)(cid:69)(cid:85)(cid:68)(cid:85)(cid:92)(cid:17)(cid:82)(cid:85)(cid:74)(cid:18)(cid:87)(cid:72)(cid:85)(cid:80)(cid:86)(cid:16)(cid:82)(cid:73)(cid:16)(cid:88)(cid:86)(cid:72)
`
`

`

`other clean pictures. When this virtual picture is used to predict the next incoming picture, the effect of errors in the next
`picture is further constrained since its reference has more correct data compared to the damaged picture.
`In this work, we assume the difference between the concealed picture and the error-free picture is equivalent to additive
`white noise for simplicity. We will only use and test the nonlinear median filter to generate the virtual reference pictures
`due to its capability of conceal white noise. The virtual reference picture is generated from three previously decoded
`pictures. Every pixel value in the virtual reference picture is obtained by median filtering the three pixels, which are from
`the same position of the three decoded pictures, respectively. If we only use forward prediction, the dependence of the
`real sequence pictures and the virtual reference pictures are illustrated in figure 2. the solid arrows represent the prediction
`directions and the dash arrows represent contributions of real pictures to the virtual reference pictures. Note that the first
`two pictures of a video sequence still use the previous real picture as their reference, respectively.
`
`VRP
`
`0
`
`VRP
`
`2
`
`I
`
`0
`
`P
`
`1
`
`P
`
`2
`
`P
`
`3
`
`P
`4
`
`5P
`
`P
`
`6
`
`VRP
`
`1
`
`VRP3
`
`Figure 2. Dependence of pictures in the virtual reference video coding system.
`
`Due to the lower correlation between the virtual reference picture and the incoming picture, coding efficiency in the
`proposed video coding system will be lower than the conventional coding system. In section 4, we will show that the bit
`rate increment is acceptable and comparable to that of the macroblock refreshment in most tests.
`
`3. SIMULATION SETUP
`The proposed video coding system is built based on MPEG-4 AVC/H.264 reference software JM9.0. 8 Simulations are
`run on both Virtual Reference Picture(VRP) codec and the original H.264 codec, and their results are compared. The error
`concealment of a damaged picture involves spatial and temporal interpolations as developed in. 9 The concealed picture
`along with two more previously decoded pictures are used to generate the virtual reference picture.
`All pictures of a test sequence are coded in forward prediction frame mode except the first I picture. All valid types of
`motion estimations and predictions are allowed as in H.264 baseline. At this stage, only intra macroblock refreshment is
`used in both codecs as the error control method in order to compare their performance in error-prone environments. Test
`video sequences are encoded by both VRP and H.264 encoder using the same H.264 baseline parameters except settings
`for how many intra macroblock per picture. The VRP encoder use the same or half number of intra macroblocks than the
`H.264 encoder. In H.264, only one reference picture is use.
`The noisy channel is assumed to be the binary symmetric channel as in many wireless applications. Random bit errors
`are added to the coded bit streams to simulate the situations when lower level protection, i.e., channel coding, fails. These
`errors may damage header or data information and cause termination of decoding a slice. Information in the slice after the
`
`898 SPIE-IS&T/ Vol. 5685
`
`(cid:39)(cid:82)(cid:90)(cid:81)(cid:79)(cid:82)(cid:68)(cid:71)(cid:72)(cid:71)(cid:3)(cid:41)(cid:85)(cid:82)(cid:80)(cid:29)(cid:3)(cid:75)(cid:87)(cid:87)(cid:83)(cid:86)(cid:29)(cid:18)(cid:18)(cid:90)(cid:90)(cid:90)(cid:17)(cid:86)(cid:83)(cid:76)(cid:72)(cid:71)(cid:76)(cid:74)(cid:76)(cid:87)(cid:68)(cid:79)(cid:79)(cid:76)(cid:69)(cid:85)(cid:68)(cid:85)(cid:92)(cid:17)(cid:82)(cid:85)(cid:74)(cid:18)(cid:70)(cid:82)(cid:81)(cid:73)(cid:72)(cid:85)(cid:72)(cid:81)(cid:70)(cid:72)(cid:16)(cid:83)(cid:85)(cid:82)(cid:70)(cid:72)(cid:72)(cid:71)(cid:76)(cid:81)(cid:74)(cid:86)(cid:16)(cid:82)(cid:73)(cid:16)(cid:86)(cid:83)(cid:76)(cid:72)(cid:3)(cid:82)(cid:81)(cid:3)(cid:20)(cid:24)(cid:3)(cid:45)(cid:88)(cid:81)(cid:3)(cid:21)(cid:19)(cid:21)(cid:19)
`(cid:55)(cid:72)(cid:85)(cid:80)(cid:86)(cid:3)(cid:82)(cid:73)(cid:3)(cid:56)(cid:86)(cid:72)(cid:29)(cid:3)(cid:75)(cid:87)(cid:87)(cid:83)(cid:86)(cid:29)(cid:18)(cid:18)(cid:90)(cid:90)(cid:90)(cid:17)(cid:86)(cid:83)(cid:76)(cid:72)(cid:71)(cid:76)(cid:74)(cid:76)(cid:87)(cid:68)(cid:79)(cid:79)(cid:76)(cid:69)(cid:85)(cid:68)(cid:85)(cid:92)(cid:17)(cid:82)(cid:85)(cid:74)(cid:18)(cid:87)(cid:72)(cid:85)(cid:80)(cid:86)(cid:16)(cid:82)(cid:73)(cid:16)(cid:88)(cid:86)(cid:72)
`
`

`

`errors are considered lost. For simplicity, we set each slice to have the same number of macroblocks in order to synchronize
`displaying of next slice.
`The first test is to demonstrate that the VRP system is able to conceal errors of a decoded picture and constrain the
`propagation of errors. Uniformly distributed random bit errors are generated and added to both bit streams. PSNR values
`of each decoded or concealed picture in the sequences are recorded and compared. Rate distortion optimization is set to
`work so that compared sequences have the same PSNR. Coding efficiency is compared in terms of bit stream length.
`The second test is to statistically show that the VRP system has more protection to the video streams than the intra
`macroblock refreshment. This test collects PSNR values of decoded sequences at different bit error rates(BER) and error
`patterns. At each interested BER point, the first test is repeated one hundred times under different errors and yield an
`averaged PSNR value. Then a trend of video quality drop along with increased BER can be drawn. Comparing curves of
`two system will bring the conclusion of error resilience capability.
`
`4. RESULTS AND DISCUSSION
`Error resilient capability of the VRP video coding system is first illustrated. The test sequence mobile contains 30 pictures
`and is in 4:2:0 SIF format(352 × 240). Quantization parameters are set to 28 for all pictures in both encoders. In both
`sequences, there are 10 intra macroblocks in each H.264 inter picture and 5 in each VRP inter picture. Figure 3 shows
`a typical example of the PSNR values of the damaged sequences. Between picture No.7 and No.9, The proposed VRP
`sequence only experiences a short period of quality drop and recover quickly, while the H.264 sequence needs a very long
`time to recover from errors in the single picture No.8. The same thing happens to pictures between No.12 and No.16.
`This results in a higher average PSNR of VRP sequence(27.77dB) than H.264 sequence(25.05dB). Pictures from No.7 to
`No.9 of VRP sequence are displayed in figure 4, which shows that the part in the middle of the calendar is damaged and
`concealed. The PSNR drops that happen earlier in both sequences are due to some errors that can not be concealed by the
`current algorithm implementations in the reference software JM9.0.
`
`PSNR of decoded Frame of mobile when the BER is 1E−5
`
`VRP
`Error Free
`H264
`
`0
`
`5
`
`10
`
`15
`Frame number
`
`20
`
`25
`
`30
`
`Figure 3. PSNR value comparison of sequence mobile.
`
`38
`
`36
`
`34
`
`32
`
`30
`
`28
`
`26
`
`24
`
`22
`
`20
`
`18
`
`16
`
`PSNR (dB)
`
`At a frame rate of 30 frame per second, the average bit rate of each P frame in mobile is 10.9 KByte/s from the H.264
`encoder and 16.2 KByte/s from the VRP encoder. The overhead of 48.7% largely dues to the fact that complicated context
`and details of a real picture is filtered by the median filter. Because the VRP bit stream is longer than the H.264 bit stream,
`it also encounters more errors. This makes the VRP curve have more pits than the H.264 curve.
`
`SPIE-IS&T/ Vol. 5685 899
`
`(cid:39)(cid:82)(cid:90)(cid:81)(cid:79)(cid:82)(cid:68)(cid:71)(cid:72)(cid:71)(cid:3)(cid:41)(cid:85)(cid:82)(cid:80)(cid:29)(cid:3)(cid:75)(cid:87)(cid:87)(cid:83)(cid:86)(cid:29)(cid:18)(cid:18)(cid:90)(cid:90)(cid:90)(cid:17)(cid:86)(cid:83)(cid:76)(cid:72)(cid:71)(cid:76)(cid:74)(cid:76)(cid:87)(cid:68)(cid:79)(cid:79)(cid:76)(cid:69)(cid:85)(cid:68)(cid:85)(cid:92)(cid:17)(cid:82)(cid:85)(cid:74)(cid:18)(cid:70)(cid:82)(cid:81)(cid:73)(cid:72)(cid:85)(cid:72)(cid:81)(cid:70)(cid:72)(cid:16)(cid:83)(cid:85)(cid:82)(cid:70)(cid:72)(cid:72)(cid:71)(cid:76)(cid:81)(cid:74)(cid:86)(cid:16)(cid:82)(cid:73)(cid:16)(cid:86)(cid:83)(cid:76)(cid:72)(cid:3)(cid:82)(cid:81)(cid:3)(cid:20)(cid:24)(cid:3)(cid:45)(cid:88)(cid:81)(cid:3)(cid:21)(cid:19)(cid:21)(cid:19)
`(cid:55)(cid:72)(cid:85)(cid:80)(cid:86)(cid:3)(cid:82)(cid:73)(cid:3)(cid:56)(cid:86)(cid:72)(cid:29)(cid:3)(cid:75)(cid:87)(cid:87)(cid:83)(cid:86)(cid:29)(cid:18)(cid:18)(cid:90)(cid:90)(cid:90)(cid:17)(cid:86)(cid:83)(cid:76)(cid:72)(cid:71)(cid:76)(cid:74)(cid:76)(cid:87)(cid:68)(cid:79)(cid:79)(cid:76)(cid:69)(cid:85)(cid:68)(cid:85)(cid:92)(cid:17)(cid:82)(cid:85)(cid:74)(cid:18)(cid:87)(cid:72)(cid:85)(cid:80)(cid:86)(cid:16)(cid:82)(cid:73)(cid:16)(cid:88)(cid:86)(cid:72)
`
`

`

`(a)
`
`(d)
`
`(b)
`
`(e)
`
`(c)
`
`(f)
`
`Figure 4. Pictures from mobile: a) - c) picture No.7 to No.9 of the error free sequence; d) - f) picture No.7 to No.9 of the damaged and
`concealed VRP sequence.
`
`Sequence
`
`foreman
`container
`silent
`
`Table 1. Overhead of VRP pictures to the H.264 pictures.
`VRP
`H.264
`Intra MB/picture
`Intra MB/picture
`bitrate(Byte/f)
`5
`10
`854
`5
`5
`366
`5
`5
`557
`
`bitrate(Byte/f)
`996
`433
`665
`
`Overhead
`
`16.6%
`18.3%
`19.4%
`
`Following results demonstrate that the VRP codec outperforms the H.264 codec in sense of error resilience. Sequence
`foreman, container and silent are all in 4:2:0 QCIF format(176× 144). With quantization parameter set to 28, one hundred
`pictures of each sequence are coded in IPPP··· order with only one intra picture at the begin of the sequence.
`Table 1 shows that the overhead of the VRP pictures to the H.264 pictures is lower than 20%. These numbers are
`much smaller than the one from mobile because their context is homogeneous and correlation between pictures are higher.
`Considering that the VRP pictures use only half of intra macroblocks as H.264 pictures, this number is acceptable given the
`PSNR vs. BER performance shown in figure 5, 6, and 7. Basically, VRP sequences lead H.264 at least 1.5dB at the BER
`1.0E-5. The difference of two curves get larger as BER goes up. In figure 5, the simulations run over a large range of BER
`so that a complete picture of the quality difference is shown. Others only run in smaller BER ranges since they have clearly
`illustrated their performance. And in fact, the channel coding usually does not behave that bad to have source decoders
`face a very high BER. From figure 6 and figure 7, we can see that the VRP sequences are almost flat, which implies that
`the VRP has very good error resilience capability in this area. In figure 8, the PSNR vs. BER curves of sequence mobile
`mentioned above is also given. The difference of VRP and H.264 at 1.0E-5 is 2.28dB.
`Note that all these results are achieved when we only use a very basic median filter and assume a simple error model.
`In the future, the work will focus on developing more practical error models and finding more effective filters.
`
`900 SPIE-IS&T/ Vol. 5685
`
`(cid:39)(cid:82)(cid:90)(cid:81)(cid:79)(cid:82)(cid:68)(cid:71)(cid:72)(cid:71)(cid:3)(cid:41)(cid:85)(cid:82)(cid:80)(cid:29)(cid:3)(cid:75)(cid:87)(cid:87)(cid:83)(cid:86)(cid:29)(cid:18)(cid:18)(cid:90)(cid:90)(cid:90)(cid:17)(cid:86)(cid:83)(cid:76)(cid:72)(cid:71)(cid:76)(cid:74)(cid:76)(cid:87)(cid:68)(cid:79)(cid:79)(cid:76)(cid:69)(cid:85)(cid:68)(cid:85)(cid:92)(cid:17)(cid:82)(cid:85)(cid:74)(cid:18)(cid:70)(cid:82)(cid:81)(cid:73)(cid:72)(cid:85)(cid:72)(cid:81)(cid:70)(cid:72)(cid:16)(cid:83)(cid:85)(cid:82)(cid:70)(cid:72)(cid:72)(cid:71)(cid:76)(cid:81)(cid:74)(cid:86)(cid:16)(cid:82)(cid:73)(cid:16)(cid:86)(cid:83)(cid:76)(cid:72)(cid:3)(cid:82)(cid:81)(cid:3)(cid:20)(cid:24)(cid:3)(cid:45)(cid:88)(cid:81)(cid:3)(cid:21)(cid:19)(cid:21)(cid:19)
`(cid:55)(cid:72)(cid:85)(cid:80)(cid:86)(cid:3)(cid:82)(cid:73)(cid:3)(cid:56)(cid:86)(cid:72)(cid:29)(cid:3)(cid:75)(cid:87)(cid:87)(cid:83)(cid:86)(cid:29)(cid:18)(cid:18)(cid:90)(cid:90)(cid:90)(cid:17)(cid:86)(cid:83)(cid:76)(cid:72)(cid:71)(cid:76)(cid:74)(cid:76)(cid:87)(cid:68)(cid:79)(cid:79)(cid:76)(cid:69)(cid:85)(cid:68)(cid:85)(cid:92)(cid:17)(cid:82)(cid:85)(cid:74)(cid:18)(cid:87)(cid:72)(cid:85)(cid:80)(cid:86)(cid:16)(cid:82)(cid:73)(cid:16)(cid:88)(cid:86)(cid:72)
`
`

`

`PSNR vs. BER for sequence foreman
`
`VRP
`H264
`
`2e−6
`
`4e−6
`
`6e−6
`
`8e−6
`
`1e−5
`BER
`
`3e−5
`
`5e−5
`
`7e−5
`
`9e−5
`
`2e−4
`
`Figure 5. PSNR vs. bit error rate curves of sequence foreman, QCIF.
`
`40
`
`35
`
`30
`
`25
`
`20
`
`15
`
`PSNR (dB)
`
`5. CONCLUSIONS
`We propose a novel solution to alleviate the error propagation in the motion-compensated video coding. The video coding
`system uses a virtual reference picture instead of a real sequence picture for inter prediction. An identical nonlinear filter
`is used to generate the VRP at both encoder and decoder from three previously decoded pictures. Errors from the damaged
`picture are filtered and clean information from other pictures are used.
`Simulations demonstrate that the proposed video coding system can achieve smaller period of video quality degradation
`and higher PSNR than standard H.264 video in error environments when only macroblock refreshment is applied. The
`proposed method need slightly higher bitrate to maintain the same video quality in error free environment due to less
`correlation between VRP and incoming pictures. The proposed approach is suitable for error resilient hybrid motion-
`compensated video coding such as MPEG-4 AVC/H.264.
`
`REFERENCES
`1. S. Wenger, G.D. Knorr, J. Ott, and F. Kossentini. Error resilience support in h.263+. IEEE Trans. Circuits Syst. Video
`Technol., 8(7):867–877, Nov. 1998.
`2. Bernd Girod. Efficiency analysis of multihypothesis motion-compensated prediction for video coding. IEEE Tran. on
`Image Processing., 9(12):173–183, Feb. 2000.
`3. Markus Flierl, Thomas Wiegand, and Bernd Girod. Rate-constrained multihypothesis prediction for motion-
`compensated video compression. IEEE Trans. on Circuit and Systems for Video Tech., 12(11), November 2002.
`4. Huisheng Wang and Antonio Ortega. Robust video communication by combining scalability and multiple description
`coding techniques. In Conf. Proc. IS&T/SPIE 15th Annual Symp. on Electronic Imaging 2003(EI03), Jan. 2003.
`5. Study group 16 ITU-T. Annexes u, v, and w to recommendation h.263. Draft for ”H.263++”, November 2000.
`6. ITU-T. Recommendation h.264 / iso/iec 11496-10 final committee draft. JVT-E022, September 2002.
`7. Yu-Wen Huang et. al. Analysis and reduction of reference frames for motion estimation in mpeg-4 avc/jvt/h.264. In
`IEEE Int. Conf. on Acoustics Speech and Signal Processing(ICASSP03), volume 3, pages 145–148, 2003.
`8. Joint Video Team(JVT). Reference software jm9.0. http://iphome.hhi.de/suehring/tml/download/, Nov. 2004.
`
`SPIE-IS&T/ Vol. 5685 901
`
`(cid:39)(cid:82)(cid:90)(cid:81)(cid:79)(cid:82)(cid:68)(cid:71)(cid:72)(cid:71)(cid:3)(cid:41)(cid:85)(cid:82)(cid:80)(cid:29)(cid:3)(cid:75)(cid:87)(cid:87)(cid:83)(cid:86)(cid:29)(cid:18)(cid:18)(cid:90)(cid:90)(cid:90)(cid:17)(cid:86)(cid:83)(cid:76)(cid:72)(cid:71)(cid:76)(cid:74)(cid:76)(cid:87)(cid:68)(cid:79)(cid:79)(cid:76)(cid:69)(cid:85)(cid:68)(cid:85)(cid:92)(cid:17)(cid:82)(cid:85)(cid:74)(cid:18)(cid:70)(cid:82)(cid:81)(cid:73)(cid:72)(cid:85)(cid:72)(cid:81)(cid:70)(cid:72)(cid:16)(cid:83)(cid:85)(cid:82)(cid:70)(cid:72)(cid:72)(cid:71)(cid:76)(cid:81)(cid:74)(cid:86)(cid:16)(cid:82)(cid:73)(cid:16)(cid:86)(cid:83)(cid:76)(cid:72)(cid:3)(cid:82)(cid:81)(cid:3)(cid:20)(cid:24)(cid:3)(cid:45)(cid:88)(cid:81)(cid:3)(cid:21)(cid:19)(cid:21)(cid:19)
`(cid:55)(cid:72)(cid:85)(cid:80)(cid:86)(cid:3)(cid:82)(cid:73)(cid:3)(cid:56)(cid:86)(cid:72)(cid:29)(cid:3)(cid:75)(cid:87)(cid:87)(cid:83)(cid:86)(cid:29)(cid:18)(cid:18)(cid:90)(cid:90)(cid:90)(cid:17)(cid:86)(cid:83)(cid:76)(cid:72)(cid:71)(cid:76)(cid:74)(cid:76)(cid:87)(cid:68)(cid:79)(cid:79)(cid:76)(cid:69)(cid:85)(cid:68)(cid:85)(cid:92)(cid:17)(cid:82)(cid:85)(cid:74)(cid:18)(cid:87)(cid:72)(cid:85)(cid:80)(cid:86)(cid:16)(cid:82)(cid:73)(cid:16)(cid:88)(cid:86)(cid:72)
`
`

`

`PSNR vs. BER for sequence container
`
`VRP
`H264
`
`40
`
`39
`
`38
`
`37
`
`36
`
`35
`
`34
`
`33
`
`32
`
`31
`
`PSNR (dB)
`
`30
`1e−6
`
`2e−6
`
`3e−6
`
`4e−6
`
`5e−6
`
`6e−6
`
`7e−6
`
`8e−6
`
`9e−6
`
`1e−5
`
`BER
`
`Figure 6. PSNR vs. bit error rate curves of sequence container, QCIF.
`
`9. Ye-Kui Wang et. al. The error concealment feature in the h.26l test model. In IEEE Inter. Conf. on Image Process-
`ing(ICIP), 2002.
`
`902 SPIE-IS&T/ Vol. 5685
`
`(cid:39)(cid:82)(cid:90)(cid:81)(cid:79)(cid:82)(cid:68)(cid:71)(cid:72)(cid:71)(cid:3)(cid:41)(cid:85)(cid:82)(cid:80)(cid:29)(cid:3)(cid:75)(cid:87)(cid:87)(cid:83)(cid:86)(cid:29)(cid:18)(cid:18)(cid:90)(cid:90)(cid:90)(cid:17)(cid:86)(cid:83)(cid:76)(cid:72)(cid:71)(cid:76)(cid:74)(cid:76)(cid:87)(cid:68)(cid:79)(cid:79)(cid:76)(cid:69)(cid:85)(cid:68)(cid:85)(cid:92)(cid:17)(cid:82)(cid:85)(cid:74)(cid:18)(cid:70)(cid:82)(cid:81)(cid:73)(cid:72)(cid:85)(cid:72)(cid:81)(cid:70)(cid:72)(cid:16)(cid:83)(cid:85)(cid:82)(cid:70)(cid:72)(cid:72)(cid:71)(cid:76)(cid:81)(cid:74)(cid:86)(cid:16)(cid:82)(cid:73)(cid:16)(cid:86)(cid:83)(cid:76)(cid:72)(cid:3)(cid:82)(cid:81)(cid:3)(cid:20)(cid:24)(cid:3)(cid:45)(cid:88)(cid:81)(cid:3)(cid:21)(cid:19)(cid:21)(cid:19)
`(cid:55)(cid:72)(cid:85)(cid:80)(cid:86)(cid:3)(cid:82)(cid:73)(cid:3)(cid:56)(cid:86)(cid:72)(cid:29)(cid:3)(cid:75)(cid:87)(cid:87)(cid:83)(cid:86)(cid:29)(cid:18)(cid:18)(cid:90)(cid:90)(cid:90)(cid:17)(cid:86)(cid:83)(cid:76)(cid:72)(cid:71)(cid:76)(cid:74)(cid:76)(cid:87)(cid:68)(cid:79)(cid:79)(cid:76)(cid:69)(cid:85)(cid:68)(cid:85)(cid:92)(cid:17)(cid:82)(cid:85)(cid:74)(cid:18)(cid:87)(cid:72)(cid:85)(cid:80)(cid:86)(cid:16)(cid:82)(cid:73)(cid:16)(cid:88)(cid:86)(cid:72)
`
`

`

`PSNR vs. BER for sequence silent
`
`40
`
`35
`
`30
`
`PSNR (dB)
`
`VRP
`H264
`
`25
`1e−6
`
`2e−6
`
`3e−6
`
`4e−6
`
`5e−6
`
`6e−6
`BER
`
`7e−6
`
`8e−6
`
`9e−6
`
`1e−5
`
`2e−5
`
`Figure 7. PSNR vs. bit error rate curves of sequence silent, QCIF.
`
`PSNR vs. BER for sequence mobile
`
`VRP
`H264
`
`40
`
`35
`
`30
`
`25
`
`20
`
`PSNR (dB)
`
`15
`1e−6
`
`3e−6
`
`5e−6
`
`7e−6
`BER
`
`9e−6
`
`2e−5
`
`Figure 8. PSNR vs. bit error rate curves of sequence mobile, SIF.
`
`SPIE-IS&T/ Vol. 5685 903
`
`(cid:39)(cid:82)(cid:90)(cid:81)(cid:79)(cid:82)(cid:68)(cid:71)(cid:72)(cid:71)(cid:3)(cid:41)(cid:85)(cid:82)(cid:80)(cid:29)(cid:3)(cid:75)(cid:87)(cid:87)(cid:83)(cid:86)(cid:29)(cid:18)(