(19) United States
`(12) Patent Application Publication (10) Pub. No.: US 2005/0201633 A1
`Mo0n et al.
`(43) Pub. Date:
`Sep. 15, 2005
`
`US 2005O2O1633A1
`
`(54) METHOD, MEDIUM, AND FILTER
`REMOVING A BLOCKING EFFECT
`(75) Inventors: Joo-hee Moon, Seoul (KR); Sun-young
`Park, Seoul (KR)
`
`Correspondence Address:
`STAAS & HALSEY LLP
`SUTE 700
`1201 NEW YORKAVENUE, N.W.
`WASHINGTON, DC 20005 (US)
`
`(73) Assignees: Daeyang Foundation, Seoul (KR); Sam-
`sung Electronics Co., Ltd., Suwon-si
`(KR)
`
`(21) Appl. No.:
`(22) Filed:
`
`11/077,332
`Mar. 11, 2005
`
`Foreign Application Priority Data
`(30)
`Mar. 11, 2004 (KR)............................ 10-2004-OO16619
`
`Publication Classification
`
`(51) Int. Cl. ............................ G06K 9/40; G06K 9/36
`(52) U.S. Cl. ............................................ 382/268; 382/236
`
`(57)
`
`ABSTRACT
`
`Provided are a method, medium, and filter for removing
`discontinuity of an image. The filtering method includes
`determining the direction or gradient on a boundary of a
`block of an image divided into blocks of a predetermined
`size, based on pixel distribution between adjacent blockS
`and filtering the blocks based on the determined direction or
`gradient.
`
`CURRENT
`PICTURE F
`
`
`
`
`
`
`
`FRAME
`MEMORY
`
`
`
`RESTORED
`PICTURE F.
`
`106
`
`INTRA
`PREDICTOR
`
`108
`
`110
`
`TRANSFORMER
`
`QUANTIZER
`
`112
`RE
`ARRANGER
`
`114
`
`ENTROPY
`CODER
`
`116
`
`INVERSE
`TRANSFORMER
`
`OE
`QUANTIZER
`
`IPR2021-00827
`Unified EX1004 Page 1
`
`

`

`Patent Application Publication Sep. 15, 2005 Sheet 1 of 8
`
`US 2005/0201633 A1
`
`ÅdOHINE
`
`HECHOO
`
`- 20 || O:
`
`I “?IJI
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`HO1OICE!bjd
`
`WHINI
`
`-EOESHEAN||
`
`
`
`
`HEZIINWñ?)HE WHO-|SNWHLHEITI-' CEHOISEB
`
`IPR2021-00827
`Unified EX1004 Page 2
`
`

`

`Patent Application Publication Sep. 15, 2005 Sheet 2 of 8
`
`US 2005/0201633 A1
`
`FIG. 2
`
`
`
`16
`
`6
`
`IPR2021-00827
`Unified EX1004 Page 3
`
`

`

`Patent Application Publication Sep. 15, 2005 Sheet 3 of 8
`
`US 2005/0201633 A1
`
`FIG. 4
`
`REFERENCE PICTURE
`
`CURRENT PICTURE
`
`FIG. 5A
`
`
`
`
`
`
`
`6
`
`
`
`16
`
`ZZZZZZZZZZZZZZZZZZZZ
`
`ZZZZZZZZZZZZZZZZZZZZZ
`
`16
`
`t-1-1-1-
`
`ZZZZZZZZZZZZZZZZZZZ
`
`
`
`ZZZZZZZZZZZZZZZZZZZZ
`
`
`
`
`
`
`
`
`
`8
`ZZZ2ZZZZZ
`
`8
`
`ZZZZZZZZZZ
`
`IPR2021-00827
`Unified EX1004 Page 4
`
`

`

`Patent Application Publication Sep. 15, 2005 Sheet 4 of 8
`
`US 2005/0201633 A1
`
`FIG. 6A
`
`
`
`FIG, 6B
`
`2.
`R
`s
`al 4.
`
`IPR2021-00827
`Unified EX1004 Page 5
`
`

`

`Patent Application Publication Sep. 15, 2005 Sheet 5 of 8
`
`US 2005/0201633 A1
`
`FIG. 7
`
`f k-p (x, N- )
`
`?
`
`Jaer--r--
`
`
`No. No.
`§ §§
`§§25 ºf
`/*, ,
`
`fk-1 (N-1,y)
`
`
`
`
`
`
`
`
`
`IPR2021-00827
`Unified EX1004 Page 6
`
`

`

`Patent Application Publication Sep. 15, 2005 Sheet 6 of 8
`
`US 2005/0201633 A1
`
`FIG. 8A
`
`FIG. 8B
`
`
`
`IPR2021-00827
`Unified EX1004 Page 7
`
`

`

`Patent Application Publication Sep. 15, 2005 Sheet 7 of 8
`
`US 2005/0201633 A1
`
`FIG. 9
`
`
`
`FIG, 10
`
`1010
`GRADENT
`DETERMINING UNIT
`
`1020
`FILTERING
`UNIT
`
`IPR2021-00827
`Unified EX1004 Page 8
`
`

`

`Patent Application Publication Sep. 15, 2005 Sheet 8 of 8
`
`US 2005/0201633 A1
`
`FIG.
`
`11
`
`
`
`IPR2021-00827
`Unified EX1004 Page 9
`
`

`

`US 2005/0201633 A1
`
`Sep. 15, 2005
`
`METHOD, MEDIUM, AND FILTER REMOVING A
`BLOCKING EFFECT
`
`BACKGROUND OF THE INVENTION
`0001. This application claims the benefit of Korean
`Patent Application No. 10-2004-0016619, filed on Mar. 11,
`2004, in the Korean Intellectual Property Office, the disclo
`Sure of which is incorporated herein in its entirety by
`reference.
`0002) 1. Field of the Invention
`0.003 Embodiments of the present invention relate to
`encoding and decoding of motion picture data, and, more
`particularly, to a method, medium, and filter for removing a
`blocking effect.
`0004 2. Description of the Related Art
`0005 Encoding picture data is necessary for transmitting
`images via a network having a fixed bandwidth or for Storing
`images in Storage media. A great amount of research has
`been conducted for the effective transmission and Storage of
`images. Among various image encoding methods, trans
`form-based encoding is most widely used, while discrete
`cosine transform (DCT) is widely used in the field of
`transform-based image encoding.
`0006 Among a variety of image encoding Standards,
`H.264 AVC standards apply integer DCT to intraprediction
`and interprediction to obtain a high compression rate and
`encode a difference between a predicted image and an
`original image. Since information of less importance among
`DCT coefficients is discarded after the completion of DCT
`and quantization, the quality of an image decoded through
`an inverse transform is degraded. In other words, while a
`transmission bit rate for image data is reduced due to
`compression, image quality is degraded. DCT is carried out
`in block units of a predetermined size into which an image
`is divided. Since transform coding is performed in block
`units, a blocking effect arises where discontinuity occurs at
`boundaries between blocks.
`0007 Also, motion compensation in block units causes a
`blocking effect. Motion information of a current block,
`which can be used for image decoding, is limited to one
`motion vector per block of a predetermined size within a
`frame, e.g., per macroblock. A predictive motion vector
`(PMV) is subtracted from an actual motion vector, and then
`the actual motion vector is encoded. The PMV is obtained
`using a motion vector of the current block and a motion
`vector of a block adjacent to the current block.
`0008 Motion-compensated blocks are created by copy
`ing interpolated pixel values from blocks of different loca
`tions in previous reference frames. AS a result, pixel values
`of blocks are Significantly different and a discontinuity
`occurs on the boundaries between blockS. Moreover, during
`copying, a discontinuity between blocks in a reference frame
`is intactly delivered to a block to be compensated for. Thus,
`even when a 4x4 block is used in H.264 AVC, filtering
`should be performed on a decoded image to remove any
`discontinuity acroSS block boundaries.
`0009. As described above, a blocking effect arises due to
`an error caused during transform and quantization on a block
`basis and is a type of image quality degradation, where
`discontinuity on the block boundary occurs regularly like
`
`laid tiles as a compression rate increases. To remove Such
`discontinuity, filters are used. The filters are classified into
`post filters and loop filters.
`0010 Post filters are located on the rear portions of
`encoderS and can be designed independently of decoderS.
`On the other hand, loop filters are located inside encoders
`and perform filtering during the encoding process. In other
`words, filtered frames are used as reference frames for
`motion compensation of frames to be encoded next.
`0011 Various methods have been studied to reduce the
`blocking effect and post filtering methods, as one of them,
`include the following Schemes. One is to overlap adjacent
`blocks, So that they can have a proper degree of correlation
`when encoded. Another is to perform low pass filtering on
`pixels located on the block boundary based on the fact that
`the visibility of the blocking effect is caused by a high spatial
`frequency of a discontinuous portion of a block.
`0012 Filtering by loop filters inside encoders is advan
`tageous over post filters in Some respects. First, by including
`loop filters inside of encoders, a proper degree of image
`quality can be guaranteed. In other words, it is possible to
`ensure Superior image quality in the manufacturing of con
`tents by removing the blocking effect. Secondly, there is no
`need for an extra frame buffer in decoderS. Namely, Since
`filtering is performed in macroblock units during decoding
`and filtered frames are directly Stored in a reference frame
`buffer, an extra frame buffer is not required. Thirdly, when
`using a post filter, a structure of a decoder is simpler, and
`Subjective and objective results of Video streams are Supe
`O.
`0013 However, conventional loop filters cannot com
`pletely remove the blocking effect because they are not
`based on the direction between blocks.
`
`SUMMARY OF THE INVENTION
`0014 Embodiments of the present invention provide a
`method, medium, and filter for removing any discontinuity
`based on the direction or gradient between blockS during the
`encoding and decoding of images.
`0015 Additional aspects and/or advantages of the inven
`tion will be set forth in part in the description which follows
`and, in part, will be obvious from the description, or may be
`learned by practice of the invention.
`0016. According to an aspect of the present invention,
`there is provided a filtering method including: determining
`a direction or a gradient on a boundary of a block of an
`image divided into blocks of a predetermined size, based on
`pixel distribution between adjacent blocks, and filtering the
`blocks based on the determined direction or gradient or
`discretion.
`0017 According to another aspect of the present inven
`tion, there is provided a filtering method which removes any
`discontinuity on boundaries between blocks of a predeter
`mined size in an image composed of the blockS. The filtering
`method includes: determining a direction of a discontinuity
`on a boundary of a block based on a difference in pixel
`values between a pixel on the boundary of the block and a
`pixel on a boundary of an adjacent block of the block, and
`filtering the block using different Selected pixels, based on
`the determined direction or gradient.
`
`IPR2021-00827
`Unified EX1004 Page 10
`
`

`

`US 2005/0201633 A1
`
`Sep. 15, 2005
`
`0.018. According to an aspect of the present invention, the
`adjacent block is located to the left-side and upside from the
`block.
`Preferably, the determining comprises calculating a
`0.019
`sum of differences in pixel value between the pixel on the
`boundary of the block to be filtered and the pixel on the
`boundary of the adjacent block, in the horizontal, the Ver
`tical, and the diagonal directions and determining a direction
`to be the direction of discontinuity on the boundary of the
`block to be filtered.
`0020. According to an aspect of the present invention, 4
`pixels of an adjacent block and 4 pixels of the block are
`Selected according to the determined direction in the hori
`Zontal, the vertical, or the diagonal direction to filter the
`block.
`0021 According to yet another aspect of the present
`invention, there is provided a filter which removes any
`discontinuity on boundaries between blocks of a predeter
`mined size in an image composed of the blocks. The filter
`includes a direction determining unit that determines the
`direction of a discontinuity on a boundary of a block of an
`image divided into blocks of a predetermined size, based on
`pixel distribution between adjacent blockS and a filtering
`unit that filters the blocks based on the determined direction.
`0022. According to an aspect of the present invention, the
`direction determining unit calculates a Sum of differences in
`pixel value between the pixel on the boundary of the block
`and the pixel on the boundary of the adjacent block, in the
`horizontal, the Vertical, and the diagonal directions and
`determines a direction to be the direction of discontinuity on
`the boundary of the block.
`0023. According to an aspect of the present invention, the
`filtering unit Selects 4 pixels of adjacent block and 4 pixels
`of the block to be filtered according to the determined
`direction in the horizontal, the vertical, or the diagonal
`direction to filter the block.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`0024. These and/or other aspects and advantages of the
`invention will become apparent and more readily appreci
`ated from the following description of the embodiments,
`taken in conjunction with the accompanying drawings of
`which:
`FIG. 1 is a block diagram of an encoder according
`0.025
`to a preferred embodiment of the present invention;
`0.026
`FIG. 2 illustrates directions of 9 prediction modes
`in an intra 4x4 mode,
`0027 FIG. 3 illustrates variable blocks that can be
`owned by a macroblock in interprediction;
`0028 FIG. 4 illustrates multiple reference pictures used
`for motion estimation;
`0029 FIG. 5A shows boundary pixels filtered with
`respect to a luminance block and a filtering order;
`0030 FIG. 5B shows boundary pixels filtered with
`respect to a chrominance block and a filtering order;
`0031 FIGS. 6A and 6B show pixels used for filtering;
`
`0032 FIG. 7 shows boundary pixels of blocks adjacent
`to a current block for explaining directivity-based filtering
`according to the present invention;
`0033 FIGS. 8A and 8B are views for explaining calcu
`lation of a difference between pixel values of two pixels,
`0034 FIG. 9 shows pixel values used when filtering is
`performed based on the directivity;
`0035 FIG. 10 is a block diagram of a filter for removing
`a blocking effect according to the present invention; and
`0036 FIG. 11 shows a boundary portion between blocks.
`
`DETAILED DESCRIPTION OF THE
`INVENTION
`0037 Reference will now be made in detail to the
`embodiments of the present invention, examples of which
`are illustrated in the accompanying drawings, wherein like
`reference numerals refer to the like elements throughout.
`The embodiments are described below to explain the present
`invention by referring to the figures.
`0038 FIG. 1 is a block diagram of an encoder according
`to a preferred embodiment of the present invention.
`0039 The encoder includes a motion estimator unit 102,
`a motion compensator 104, an intra predictor 106, a trans
`former 108, a quantizer 110, a re-arranger 112, an entropy
`coder 114, a de-quantizer 116, an inverse transformer 118, a
`filter 120, and a frame memory 122.
`0040. The encoder encodes macroblocks of a current
`block in an encoding mode Selected among various encoding
`modes. To encode Video, a picture is divided into Several
`macroblocks. After encoding the macroblocks in all the
`encoding modes of interprediction and all the encoding
`modes of intraprediction, the encoder Selects one encoding
`mode according to a bit rate required for encoding of the
`macroblocks and the degree of distortion between the origi
`nal macroblockS and decoded macroblockS and performs
`encoding in the Selected encoding mode.
`0041
`Inter mode is used in interprediction where a
`difference between the motion vector information indicating
`a location of one macroblock Selected from a reference
`picture or locations of a plurality of macroblockS Selected
`from a reference picture and a pixel value is encoded in
`order to encode macroblocks of a current picture. Since
`H.264 offers a maximum of 5 reference pictures, a reference
`picture to be referred to by a current macroblock is Searched
`in a frame memory that Stores reference pictures. The
`reference pictures Stored in the frame memory may be
`previously encoded pictures or pictures to be used.
`0042 Intra mode is used in intraprediction where a
`predicted value of a macroblock to be encoded is calculated
`using a pixel value of a pixel that is spatially adjacent to the
`macroblock to be encoded and a difference between the
`predicted value and the pixel value is encoded, instead of
`referring to reference pictures, in order to encode the mac
`roblocks of the current picture.
`0043. There exist a large number of modes depending on
`how to divide an image in inter mode. Similarly, there exist
`numerous modes depending on the direction of the predic
`tion in intra mode. Thus, Selecting the optimal mode among
`these modes is a very important task that affects the perfor
`
`IPR2021-00827
`Unified EX1004 Page 11
`
`

`

`US 2005/0201633 A1
`
`Sep. 15, 2005
`
`mance of image encoding. To this end, generally, rate
`distortion (RD) costs in all the possible modes are calcu
`lated, a mode having the Smallest RD costS is Selected as the
`optimal mode, and encoding is performed in the Selected
`mode. As a result, a lot of time and costs are required for
`image encoding.
`0044) The encoder according to an embodiment of the
`present invention performs encoding in all the modes inter
`prediction and intraprediction can have, calculates RD costs,
`Selects a mode having the Smallest RD costs as the optimal
`mode, and performs encoding in the Selected mode.
`0.045
`For interprediction, the motion compensator 102
`Searches for a predicted value of a macroblock of a current
`picture in reference pictures. If a reference block is found in
`/2 or 4 pixel units, the motion compensator 104 calculates
`an intermediate pixel value of the reference block to deter
`mine a reference block data value. AS Such, interprediction
`is performed by the motion estimator 102 and the motion
`compensator 104.
`0046) Also, the intra predictor 106 performs intrapredic
`tion where the predicted value of the macroblock of the
`current picture is Searched within the current picture. A
`decision whether to perform interprediction or intrapredic
`tion on a current macroblock is made by calculating RD
`costs in all the encoding modes and Selecting a mode having
`the Smallest RD cost as an encoding mode of the current
`macroblock. Encoding is then performed on the current
`macroblock in the Selected encoding mode.
`0047 As described above, if predicted data to be referred
`to by a macroblock of a current frame is obtained through an
`interprediction or intraprediction, the predicted data is Sub
`tracted from the macroblock of the current picture. The
`transformer 108 performs transform on the resulting mac
`roblock of the current picture and the quantizer 110 quan
`tizes the transform macroblock. The macroblock of the
`current picture that undergoes a Subtraction of a motion
`estimated reference block is called a residual that is encoded
`to reduce the amount of data in encoding. A quantized
`residual is processed by the re-arranger 112 for encoding by
`the entropy coder 114.
`0.048. To obtain a reference picture to be used in inter
`prediction, the current picture is restored by processing a
`quantized picture by the de-quantizer 116 and the inverse
`transformer 118. The restored current picture is stored in the
`frame memory 122, and is then used to perform an inter
`prediction on a picture that follows the current picture. If the
`restored picture passes through the filter 120, it becomes the
`original picture that additionally includes Several encoding
`COS.
`0049 FIG. 2 illustrates directions of 9 prediction modes
`in intra 4x4 mode.
`0050. It can be seen from FIG. 2 that a prediction is
`performed on blockS in the Vertical, horizontal, and diagonal
`directions, each of which is represented by a mode name. In
`other words, intra 4x4 mode includes a vertical mode, a
`horizontal mode, a DC mode, a diagonal down left mode,
`a diagonal down right mode, a vertical right mode, a hori
`Zontal down mode, a vertical left mode, and a horizonta
`up mode.
`
`0051. In addition to the intra 4x4 mode, there exists an
`intra 16x16 mode. The intra 16x16 mode is used in the case
`of a uniform image and there are four modes in the intra
`16x16 mode.
`0.052 FIG. 3 illustrates variable blocks that can be
`owned by a macroblock in an interprediction.
`0053. In an interprediction according to H.264, one
`16x16 macroblock may be divided into 16x16, 16x8, 8x16,
`or 8x8 blocks. Each 8x8 block may be divided into 8x4,
`4x8, or 4x4 Sub-blockS. Motion estimations and compensa
`tions are performed on each Sub-block, and thus a motion
`vector is determined. By performing an interprediction using
`various kinds of variable blocks, it is possible to effectively
`perform an encoding according to the properties and motion
`of an image.
`0054 FIG. 4 illustrates multiple reference pictures used
`for motion estimation.
`0055 H.264 AVC performs a motion prediction using
`multiple reference pictures. In other words, at least one
`reference picture that is previously encoded can be used as
`a reference picture for motion prediction. Referring to FIG.
`4, to find a macroblock that is most similar to a macroblock
`of a current picture, a maximum of 5 pictures are Searched.
`These reference pictures all should be stored in both an
`encoder and a decoder.
`0056. Hereinafter, filtering performed by the filter 120 of
`FIG. 1 will be described in detail.
`0057 The filter 120 is a deblocking filter and can perform
`filtering on boundary pixels of MxN blocks. Hereinafter, it
`is assumed that MxN blocks are 4x4 blocks. Filtering is
`performed in macroblock units, and all the macroblockS
`within a picture are Sequentially processed. To perform
`filtering with respect to each macroblock, pixel values of
`upper and left filtered blockS adjacent to a current macrob
`lock are used. Filtering is performed Separately for lumi
`nance and chrominance components.
`0.058 FIG. 5A shows boundary pixels filtered with
`respect to a luminance block and a filtering order.
`0059. In each macroblock, filtering is first performed on
`the vertical boundary pixels of a macroblock. The vertical
`boundary pixels are filtered from left to right as indicated by
`an arrow in the left side of FIG. 5A. Then, filtering is
`performed on the horizontal boundary pixels based on a
`result of filtering the vertical boundary pixels. The horizon
`tal boundary pixels are filtered in an up to down direction as
`indicated by an arrow in the right side of FIG. 5A. Since
`filtering is performed in macroblock units, filtering for
`removing any discontinuity of luminance is performed on 4
`lines composed of 16 pixels.
`0060 FIG. 5B shows boundary pixels filtered with
`respect to a chrominance block and a filtering order.
`0061 Since the chrominance block has a size of 4x4 that
`is /4 of the luminance block, filtering of chrominance
`components is performed on 2 lines composed of 8 pixels.
`0062 FIGS. 6A and 6B show pixels used for filtering.
`0063 Pixels are determined based on a 4x4 block bound
`ary, changed pixel values are calculated using filtering
`equations indicated below, and pixel values p0, p1, p.2, q0,
`
`IPR2021-00827
`Unified EX1004 Page 12
`
`

`

`US 2005/0201633 A1
`
`Sep. 15, 2005
`
`q1, and q2 are mainly changed. Filtering of not only lumi
`nance components but also chrominance components is
`performed in an order Similar to that used in the luminance
`block.
`FIG. 7 shows boundary pixels of blocks adjacent
`0.064
`to a current block for explaining direction or gradient-based
`filtering according to an aspect of the present invention.
`0065 Direction-based filtering according to an aspect of
`the present invention is performed on pixels located on all
`the 4x4 block boundaries, using pixel values in a picture that
`is already decoded in macroblock units, in a method similar
`to deblocking filtering of H.264 AVC. However, unlike
`deblocking filtering of H.264 AVC that is performed on each
`block boundary only in the vertical and/or horizontal direc
`tions, direction-based filtering according to an aspect of the
`present invention Searches for direction in the diagonal
`direction as well as in the Vertical and/or horizontal direc
`tions of each 4x4 block and is performed in the found
`direction. A Search for direction of a 4x4 block is done using
`pixels located on the boundaries of upper and left two blockS
`that are adjacent to a current block in a Spatial domain. If a
`block size is NxN, a boundary pixel of a k" current block is
`represented by f (x, y), right boundary pixels of a left-side
`adjacent block of the k" current block are represented by fi
`(N-1, y), and lower boundary pixels of an upper adjacent
`block of the kth current block are represented by f (x, y).
`Here, p denotes one period. For example, if a 176x144
`image is divided into 16x16 blocks, there are 11 blocks in a
`row and 9 blocks in a column. In this case, p is equal to 11.
`Then, f(x, y) becomes an immediately upper pixel of f.
`(x, y).
`Here, X and y move pixel by pixel, and pixels used
`0.066
`in filtering pixels located on the boundaries are marked with
`hatched lines. To detect the diagonal direction three pixel
`values of an adjacent block are used. For example, adjacent
`pixels (720) are used to detect direction of a pixel 1 (710).
`0067 Referring to FIG. 7, the denominations of detected
`direction are three: a vertical/horizontal direction a diagonal
`right-up direction; and a diagonal right-down direction.
`0068 FIGS. 8A and 8B are views for explaining the
`calculation of a difference between pixel values of two
`pixels.
`0069 FIG. 8A is a view for explaining a detection of the
`directivity of vertical boundary pixels with respect to the
`vertical direction and FIG. 8B is a view for explaining the
`detection of the directivity of horizontal boundary pixels
`with respect to the horizontal direction. To calculate a
`difference between the pixel values of two pixels, let a
`Square be one pixel and let an arrow be the directivity. In the
`present invention, the diagonal direction is added to the
`vertical/horizontal directions used in H.264 AVC. By per
`forming filtering using pixel values that are similar to those
`of a current block when the discontinuities between blocks
`have diagonal directivity, it is possible to prevent averaging
`in comparison to when filtering is performed using different
`pixel values. That is, it is possible to have Smooth bound
`CS.
`
`0070 Directivity detection includes the following stages:
`0.071) CD Calculating a Difference Between Pixels:
`0072 Pixel values located on a vertical boundary of a
`block are Sequentially filtered using 4x4 blocks that are
`located to the left side of a current block. V, RDV, and
`RUV, which denote the three directions from an origin, i.e.,
`a top-left point of a k" block, are calculated as follows.
`
`0073. An image that is decoded and input to a filter is
`represented by a function f(x, y). To know the direction or
`gradient, absolute values of the differences between pixel
`values that are located on boundaries between adjacent
`blocks in respective directions or gradients are calculated. A
`block size is NXN. In this embodiment, N is 4.
`0.074
`Also, when pixels on the horizontal boundary of a
`block are filtered vertically using 4x4 blockS located up
`from the current block, a difference between the pixel values
`is calculated as follows. Like the calculation of a difference
`between pixels located on the vertical boundary, a difference
`between the pixels located on the horizontal boundary is
`calculated on a pixel-by-pixel basis from an origin, i.e., a
`top-left point of the kth block.
`
`W
`
`0075) (2) Calculating the Minimum Value:
`0076 After a difference between the pixel values is
`calculated in each direction in operation CD, the minimum
`value among the three differences is Searched as follows:
`DV=min(V, RDV RUV) or
`(3)
`DH =min(H RDH RUH)
`0077. The direction of the minimum value is determined
`to be the direction of the pixels located on boundaries
`between adjacent blocks. Pixels located on the vertical
`boundary and pixels located on the horizontal boundary are
`respectively filtered in the determined direction. Hereinafter,
`filtering will be described.
`
`IPR2021-00827
`Unified EX1004 Page 13
`
`

`

`US 2005/0201633 A1
`
`Sep. 15, 2005
`
`0078 (3) Filtering:
`0079. Once the direction is determined on the vertical/
`horizontal boundaries of a current block, filtering is per
`formed based on the determined direction.
`0080 FIG. 9 shows pixel values used when filtering is
`performed based on the directivity or gradient.
`0.081
`Pixels used for filtering a boundary of a block can
`be seen from FIG. 9. In other words, it can be seen that when
`Vertical boundary pixels in the horizontal direction are
`filtered, not only pixels in the horizontal direction but also
`pixels in the diagonal direction can be Selected and filtered
`according to the determined direction.
`0082 FIG. 10 is a block diagram of a filter for removing
`a blocking effect.
`0.083. A directivity or gradient determining unit 1010
`calculates the direction of a discontinuity on the boundary
`between a current block and an adjacent block based on a
`difference in the pixel value between the current block and
`the adjacent block. A filtering unit 1020Selects pixels having
`the calculated direction and performs filtering on the
`Selected pixels. A direction determination was described
`above and filtering will be described later in detail.
`0084.
`Hereinafter, pixel value calculation by filtering will
`be described in detail.
`0085 For filtering, information about the necessity of
`filtering and information about a filtering Strength are deter
`mined. The filtering Strength differS depending on a bound
`ary Strength called a BS parameter. The BS parameter differs
`depending on prediction modes of two blocks, a motion
`difference between the two blocks, and presence of encoded
`residuals of the two blocks.
`
`Condition
`
`Any one of two blocks is in intra mode and any one of
`the two blocks is located on a boundary of a macroblock
`Any one of two blocks is in intra mode
`Any one of two blocks has a residual signal
`MV >= one-sample interval and motion compensation is
`performed using difference reference frames
`Others
`
`BS
`
`4
`
`3
`2
`1.
`
`O
`
`0.086. In Table 1, a determination is sequentially made in
`the order of top-down as to whether any one of the condi
`tions is satisfied. When any one of the conditions is first
`Satisfied, a value corresponding to the condition is deter
`mined to be a Bs parameter. For example, if the boundary of
`a block is the boundary of a macroblock and any one of the
`adjacent two blockS is encoded in intraprediction mode, the
`BS parameter is 4.
`0.087
`If a block is not located on the boundary of a
`macroblock and any one of two blockS is in an intrapredic
`tion mode, the Bs parameter is 3. If any one of two blocks
`is in an interprediction mode and has a nonzero transform
`coefficient, the Bs parameter is 2. If any one of two blocks
`does not have a nonzero transform coefficient, a motion
`difference between the two blockS is equal to or greater than
`1 pixel of luminance, and motion compensation is per
`formed using other reference frames, the BS parameter is 1.
`
`If any condition is not satisfied, the Bs parameter is 0. The
`BS parameter of 0 indicates that there is no need for filtering.
`0088. After the Bs parameter is determined, pixels
`located on the boundary of a block are searched. In a filter
`that removes discontinuity, it is important to distinguish the
`actual discontinuity that expresses objects of an image from
`discontinuity caused by quantization of transform coeffi
`cients. In order to preserve quality of an image, the actual
`discontinuity should be filtered as little as possible. On the
`other hand, discontinuity caused by quantization should be
`filtered as much as possible.
`FIG. 11 shows a boundary portion between blocks.
`0089)
`0090 Pixel values of a line having actual discontinuity as
`shown in FIG. 11 inside two adjacent blocks will be
`explained as an example. Since filtering is not performed
`when the Bs parameter is 0, the Bs parameter is not 0, and
`parameters C. and B are used to determine whether to
`perform filtering on each pixel. These parameters have
`correlations with a quantization parameter (QP) and differ
`depending on local activity around a boundary. Selected
`pixels are filtered when conditions of following Equation 4
`are Satisfied.
`
`(4)
`41-qo<?
`0091. When two pixels that are closest to a boundary are
`less than C. and p1, p0, q1, and q0 are less than B that is less
`than C, discontinuity around a boundary is determined to be
`caused by quantization. C. and B are determined according to
`a table prescribed by H.264 AVC and differ depending on the
`OP.
`
`Index=min(max(0, QPA+OffsetA), 51)
`(5),
`Index=min (max(0, QPA+Offset), 51)
`0092 where QA is an average of QPs of two adjacent
`blocks. By controlling an indeX within a range of a QP, i.e.,
`0, 51, using Equation 5, C. and f are obtained. According
`to the table prescribed by H.264 AVC, when IndexA-16 or
`IndexBC16, both C. and B are or one of C. and B is 0, which
`means that filtering is not performed. This is because it is
`inefficient to perform filtering when the QP is very small.
`0093. Also, an offset value that controls C. and B can be
`Set by an encoder and its range is -6, +6). The amount of
`filtering can be controlled using the offset value. By con
`trolling a property of a filter for removing discontinuity
`using a nonzero offset value, it is possible to improve the
`Subjective quality of a decoded image.
`0094) For example, when a difference between the pixel
`values of adjacent blockS is Sm