USOO8681867B2
`
`(12) United States Patent
`Teng et al.
`
`(10) Patent No.:
`(45) Date of Patent:
`
`US 8,681,867 B2
`Mar. 25, 2014
`
`(54) SELECTIVE DEBLOCK FILTERING
`TECHNIQUES FOR VIDEO CODING BASED
`ON MOTION COMPENSATION RESULTING
`IN A CODED BLOCK PATTERN VALUE
`
`FOREIGN PATENT DOCUMENTS
`
`JP
`
`2005O2O771 A
`
`1, 2005
`
`OTHER PUBLICATIONS
`
`(75) Inventors: Chia-Yuan Teng, San Diego, CA (US);
`Sharath Manjunath, San Diego, CA
`(US); Yan Ye, San Diego, CA (US)
`(73) Assignee: QUALCOMM Incorporated, San
`Diego, CA (US)
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 1617 days.
`Appl. No.: 11/350,666
`
`(*)
`
`Notice:
`
`(21)
`(22)
`(65)
`
`(60)
`
`(51)
`
`(52)
`
`(58)
`
`(56)
`
`Filed:
`
`Feb. 8, 2006
`
`Prior Publication Data
`US 2011 FO1 10427 A1
`May 12, 2011
`
`Related U.S. Application Data
`Provisional application No. 60/728,361, filed on Oct.
`18, 2005.
`
`(2006.01)
`
`Int. Cl.
`H04N II/02
`U.S. C.
`USPC ..................................................... 375/24O16
`Field of Classification Search
`USPC ........................................ 375/24O.O1 240.29
`See application file for complete search history.
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`5,173,773 A * 12/1992 Ueda et al. ............... 375,240.18
`5,892,518 A * 4, 1999 Mizobata et al. ............. 345/474
`2005.0053145 A1* 3, 2005 HSu et al. .........
`... 375,240.16
`2005, 0175091 A1* 8, 2005 Puri et al. .........
`... 375,240.03
`2006/025 1174 A1* 11/2006 Caviedes et al. ......... 375,24024
`
`
`
`Srinivasean Set al., “Windows Media Video 9: overview and appli
`cations.” Signal Processing. Image Communication Elsevier Science
`Publishers, Amsterdam, NL, vol. 19, No. 9, Oct. 2004, pp. 851-875.
`Xiaoyan Sun et al., “In-loop dfblocking filtfr for block-based video
`coding Signal Processing, 2002 6th International Conference on
`Aug. 26-30, 2002, Piscataway, NJ,USA, IEEE, vol. 1, Aug. 26, 2002,
`pp. 33-36.
`Puri A et al. “Video coding using the H.264/MPEG-4 AVC compres
`sion standard” Signal Processing. Image Communication, elsevier
`Science Publishers, Amsterdam, NL, vol. 19, No. 9, Oct. 2004, pp.
`T93-849.
`Srinivasan S, Signal Processing: Image Communication, NL,
`Elsevier Science Publishers, Oct. 2004, V 19 N9, pp. 851-875.
`(Continued)
`
`Primary Examiner — Leron Beck
`(74) Attorney, Agent, or Firm — Timothy F. Loomis; Brent
`Boyd
`
`ABSTRACT
`(57)
`This disclosure describes selective deblock filtering tech
`niques that are particularly useful with coding standards that
`do not specify in-loop deblock filtering for standard compli
`ance. In accordance with this disclosure, deblock filtering
`may be selectively performed with respect to block bound
`aries of a given video block being coded, a motion compen
`sation process can be performed using a filtered version of the
`prediction video frame used to code the given video block, or
`both. This disclosure also provides selection rules that can be
`applied to determine what type of deblocking techniques to
`apply in various scenarios. The selection rules may improve
`the video coding and may also ensure that mismatch between
`video blocks at an encoder and a decoder is not introduced by
`the deblock filtering.
`
`31 Claims, 5 Drawing Sheets
`
`COOING DEVICE
`10
`
`VIDEO CODNG UNIT
`14
`
`OB FILTERING UNIT
`18
`
`MEMORY
`12
`
`IPR2021-00827
`Unified EX1005 Page 1
`
`

`

`US 8,681,867 B2
`Page 2
`
`(56)
`
`References Cited
`
`OTHER PUBLICATIONS
`
`Translation of Office Action in Japan application 2008-536779 cor-
`responding to U.S. App. No. 1 1/350,666, citing Srinivasan S Sig-
`nal Processing pp. 851 875 year 2004 and JP2005020771
`dated Feb. 22, 2011.
`
`“Windows Media Video 9: overview and applications”.
`Xiaoyan Sun et al: “In-loop deblocking filter for block-based video
`coding Signal Processing, 2002 6th International Conference'on
`Aug. 26-30, 2002, Piscataway, NJ,USA, IEEE, vol. 1, Aug. 26, 2002,
`pp. 33-36, XP010627917.
`International Search Report and Written Opinion—PCT/US2006/
`040761 ISA/EPO-Mar. 23, 2007.
`
`* cited by examiner
`
`IPR2021-00827
`Unified EX1005 Page 2
`
`

`

`U.S. Patent
`
`Mar. 25, 2014
`
`Sheet 1 of 5
`
`US 8,681,867 B2
`
`
`
`CODNG DEVICE
`10
`
`MEMORY
`12
`
`VIDEO CODNG UNIT
`14
`
`OB FILTERING UNIT
`18
`
`F.G. 1
`
`IPR2021-00827
`Unified EX1005 Page 3
`
`

`

`U.S. Patent
`
`Mar. 25, 2014
`
`Sheet 2 of 5
`
`US 8,681,867 B2
`
`\.
`
`reference frame
`
`DEBLOCKING UNIT
`24
`
`
`
`
`
`NEIGHBOR'S cbp = 0
`AN
`NEIGHBOR'S MV = CURRENT MV
`
`reference frame
`
`FG. 2
`
`IPR2021-00827
`Unified EX1005 Page 4
`
`

`

`U.S. Patent
`
`Mar. 25, 2014
`
`Sheet 3 of 5
`
`US 8,681,867 B2
`
`30
`
`reference frame 1
`
`reference frame 2
`
`NEIGHBOR'S cbp = 0
`AND
`NEIGHBOR'S MV = CURRENT MV
`
`
`
`
`
`
`
`
`
`33
`
`36
`
`QP D Threshold?
`
`37
`
`FIG. 3
`
`IPR2021-00827
`Unified EX1005 Page 5
`
`

`

`U.S. Patent
`
`Mar. 25, 2014
`
`Sheet 4 of 5
`
`US 8,681,867 B2
`
`40
`
`reference frame 1
`
`reference frame 2
`
`FILTERNG
`DECSION
`
`
`
`FILTERING
`
`FIG. 4
`
`IPR2021-00827
`Unified EX1005 Page 6
`
`

`

`U.S. Patent
`
`Mar. 25, 2014
`
`Sheet 5 of 5
`
`US 8,681,867 B2
`
`CODE FRAMES USING PREDICTION-BASED CODNG
`TECHNICUES
`
`51
`
`YES
`
`52
`
`TEXTURE COOING
`USED FOR A GIVENVIDEO
`BLOCK2
`
`NO
`
`MOTION COMPENSATION BASED
`ON FILTERED REFERENCE FRAME
`
`
`
`
`
`
`
`
`
`
`
`TEXTURE CODING
`USED FOR A NEIGHBOR TO THE GIVEN
`VIDEO EBLOCK?
`
`
`
`55
`
`56
`
`BLOCK AND THE NEIGHBOR VIDEO
`BLOCK HAVE SIMLARMOTION
`VECTORS )
`
`YES
`
`DEBLOCK FILTER A BLOCK BOUNDARY
`OF THE GIVEN VIDEO BLOCK
`
`53
`
`FIG. 5
`
`57
`DO NOT DEBLOCK FILTER THE BLOCK
`BOUNDARY OF THE GIVEN VIDEO
`BLOCK
`
`IPR2021-00827
`Unified EX1005 Page 7
`
`

`

`US 8,681,867 B2
`
`1.
`SELECTIVE DEBLOCK FILTERING
`TECHNIQUES FOR VIDEO CODING BASED
`ON MOTION COMPENSATION RESULTING
`INA CODED BLOCK PATTERN VALUE
`
`RELATED APPLICATIONS
`
`This application claims the benefit of provisional U.S.
`Application Ser. No. 60/728,361, entitled “SELECTIVE
`10
`MC-DB: A DEBLOCKING POST-FILTER BASED ON
`MOTION COMPENSATION, filed Oct. 18, 2005, assigned
`to the assignee of the present application, and incorporated
`herein by reference in its entirety for all purposes.
`
`TECHNICAL FIELD
`
`15
`
`This disclosure relates to digital video processing and,
`more particularly, filtering techniques that can reduce blocki
`ness artifacts in coded video frames.
`
`BACKGROUND
`
`Digital video capabilities can be incorporated into a wide
`range of devices, including digital televisions, digital direct
`broadcast systems, wireless communication devices, per
`Sonal digital assistants (PDAs), laptop computers, desktop
`computers, digital cameras, digital recording devices, cellu
`lar or satellite radio telephones, direct two-way communica
`tion devices (sometimes referred to as “walkie-talkies'), and
`the like. Digital video devices can provide significant
`improvements over conventional analog video systems in
`creating, modifying, transmitting, storing, recording and
`playing full motion video sequences.
`A number of different video coding standards have been
`established for coding digital video sequences. The Moving
`Picture Experts Group (MPEG), for example, has developed
`a number of standards including MPEG-1, MPEG-2 and
`MPEG-4. Other standards include the International Telecom
`munications Union (ITU) H.263 standards, QuickTimeTM
`technology developed by Apple Computer of Cupertino
`Calif., Video for WindowsTM developed by Microsoft Corpo
`ration of Redmond, Wash., IndeoTM developed by Intel Cor
`poration, RealVideoTM from RealNetworks, Inc. of Seattle,
`Wash., and CinepakTM developed by SuperMac, Inc. Further
`more, new standards continue to emerge and evolve, includ
`ing the ITU H.264 standards and a number of proprietary
`standards.
`Many video coding standards allow for improved transmis
`sion rates of video sequences by coding data in a compressed
`fashion. Compression can reduce the overall amount of data
`that needs to be transmitted for effective transmission of
`Video frames. Most video coding standards, for example,
`utilize graphics and video compression techniques designed
`to facilitate video and image transmission over a narrower
`bandwidth than can be achieved without compression. Many
`of the MPEG standards and the ITU H.263 and ITU H.264
`standards, for example, Support video coding techniques that
`utilize similarities between successive video frames, referred
`to as temporal or inter-frame correlation, to provide inter
`frame compression. Such inter-frame compression is typi
`cally achieved via motion estimation and motion compensa
`tion coding techniques. In addition, Some video coding
`techniques utilize similarities within frames, referred to as
`spatial or intra-frame correlation, to compress the video
`frames. Intra-frame compression is typically achieved via
`spatial estimation and intra-prediction coding techniques.
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`2
`The use of discrete video blocks in inter-frame and/or
`intra-frame compression can cause artifacts in the video
`sequence between adjacent video blocks. In particular, when
`a video frame is divided into video blocks for video coding,
`the edge of one coded video block may appear discontinuous
`with the adjacent edge of another coded video block. When
`this occurs, the decoded video frame may appear “blocky.”
`which is highly undesirable. Transforms and quantization of
`video blocks can compound this undesirable blockiness
`effect in coded video frames.
`In order to remove such “blockiness, filtering can be per
`formed on the block boundaries of the video blocks to
`“smooth the transitions between adjacent video blocks.
`Deblocking filters generally refer to filters that are used to
`smooth the transitions between adjacent video blocks to
`reduce or eliminate blockiness artifacts. The ITU-T H.264
`standard, for example, requires a deblocking filter as part of
`the in-loop coding. In this case, when filtering is part of the
`in-loop video coding, the previously coded frames used in
`motion estimation and motion compensation are filtered ver
`sions of such frames. For other standards that do not mandate
`a deblocking filter as part of the coding loop, post deblock
`filtering may still improve the quality of the video coding by
`removing blockiness artifacts after the coding is performed.
`
`SUMMARY
`
`This disclosure describes deblock filtering techniques that
`address both originated blockiness and inherited blockiness
`that can manifestas a result of prediction-based video coding.
`Originated blockiness refers to the blockiness introduced by
`poor texture coding. Originated blockiness occurs at the
`video block boundaries of a coded video frame. Inherited
`blockiness, on the other hand, refers to the blockiness that can
`be inherited from the prediction video frame that is used by
`the prediction-based video coding techniques in order to code
`a current video block. Inherited blockiness can occur at any
`location in the coded video frame, and is not restricted to
`block boundaries. Therefore, traditional deblock filtering
`applied around block boundaries may not remove inherited
`blockiness.
`This disclosure provides for selective deblock filtering,
`which is particularly useful with coding standards that do not
`specify in-loop deblock filtering for standard compliance,
`such as MPEG-4 and ITU H.263 P0 (profile 0 of the ITU
`H.263 Standard). In this case, post filtering can be applied in
`a manner that accounts for originated blockiness and inher
`ited blockiness. In particular, deblock filtering may be selec
`tively performed with respect to block boundaries of a given
`Video block being coded, a motion compensation process can
`be performed using a filtered version of the prediction video
`frame used to code the given video block, or both. This
`disclosure also provides selection rules that can be applied to
`determine what type of deblocking techniques to apply in
`various scenarios. The selection rules can ensure that mis
`match between video blocks at an encoder and a decoder is
`not introduced by the filtering.
`In one embodiment, this disclosure provides a video cod
`ing device comprising a video coding unit that codes frames
`of a video sequence using prediction-based coding tech
`niques on video blocks of the frames, and a deblock filtering
`unit. The prediction-based coding techniques, for example,
`may include a motion compensation process that uses a non
`filtered version of a prediction frame in order to provide
`motion-based coding. The deblock filtering unit (a) deblock
`filters a block boundary of a given video block if the given
`Video block is texture coded; (b) performs a motion compen
`
`IPR2021-00827
`Unified EX1005 Page 8
`
`

`

`3
`sation process using a filtered version of a prediction video
`frame used to code the given video block if the given video
`block is not texture coded, a neighbor video block to the given
`video block is not texture coded, and the given video block
`and the neighbor video block have substantially similar
`motion vectors; and (c) deblock filters the block boundary of
`the given video block and performs the motion compensation
`process using the filter version of the prediction video frame
`if neither (a) nor (b) is satisfied.
`In another embodiment, this disclosure provides a method
`comprising: (a) coding frames of a video sequence using
`prediction-based coding techniques on video blocks of the
`frames; (b) deblock filtering a block boundary of a given
`video block if the given video block is texture coded; (c)
`performing a motion compensation process using a filtered
`version of a prediction video frame used to code the given
`video block if the given video block is not texture coded, a
`neighbor video block to the given video block is not texture
`coded, and the given video block and the neighbor video
`block have substantially similar motion vectors; and (d)
`deblock filtering the block boundary of the given video block
`and performing the motion compensation process using the
`filter version of the prediction video frame if neither (b) nor
`(c) is satisfied.
`In another embodiment, this disclosure provides a deblock
`filtering unit for video coding. In this case, the deblock filter
`ing unit (a) deblock filters a block boundary of a given video
`block if the given video block is texture coded; (b) performs
`a motion compensation process using a filtered version of a
`prediction video frame used to code the given video block if
`the given video block is not texture coded, a neighbor video
`block to the given video block is not texture coded, and the
`given video block and the neighbor video block have substan
`tially similar motion vectors; and (c) deblock filters the block
`boundary of the given video block and performs the motion
`compensation process using the filter version of the predic
`tion video frame if neither (a) nor (b) is satisfied.
`In another embodiment, this disclosure provides a com
`puter readable medium comprising instructions that upon
`execution during a video coding process: (a) deblock filter a
`block boundary of a given video block if the given video
`block is texture coded; (b) perform a motion compensation
`process using a filtered version of a prediction video frame
`used to code the given video block if the given video block is
`not texture coded, a neighbor video block to the given video
`block is not texture coded, and the given video block and the
`neighbor video block have substantially similar motion vec
`tors; and (c) deblock filter the block boundary of the given
`Video block and perform the motion compensation process
`using the filter version of the prediction video frame if neither
`(a) nor (b) is satisfied.
`In another embodiment, this disclosure provides a video
`coding device comprising means for coding frames of a video
`sequence using prediction-based coding techniques on video
`blocks of the frames; means for deblock filtering a block
`boundary of a given video block if the given video block is
`texture coded; means for performing a motion compensation
`process using a filtered version of a prediction video frame
`used to code the given video block if the given video block is
`not texture coded, a neighbor video block to the given video
`block is not texture coded, and the given video block and the
`neighbor video block have substantially similar motion vec
`tors; and means for deblock filtering the block boundary of
`the given video block and performing the motion compensa
`tion process using the filter version of the prediction video
`frame if the video block is not texture coded and one or more
`of the following conditions are satisfied: the neighbor video
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`US 8,681,867 B2
`
`5
`
`10
`
`15
`
`4
`block to the given video block is texture coded, and the given
`video block and the neighbor video block have substantially
`different motion vectors.
`In another embodiment, this disclosure provides a video
`coding device comprising a video coding unit that codes
`frames of a video sequence using prediction-based coding
`techniques on video blocks of the frames. In this case, the
`Video coding unit performs deblock filtering including: (a)
`deblock filtering a block boundary of a given video block if
`the given video block is texture coded; (b) performing a
`motion compensation process using a filtered version of a
`prediction video frame used to code the given video block if
`the given video block is not texture coded, a neighbor video
`block to the given video block is not texture coded, and the
`given video block and the neighbor video block have substan
`tially similar motion vectors; and (c) deblock filtering the
`block boundary of the given video block and performing the
`motion compensation process using the filter version of the
`prediction video frame if neither (a) nor (b) is satisfied.
`In an added embodiment, this disclosure contemplates the
`techniques herein implemented with respect to intra-predic
`tion coding rather than motion based coding. In this case, a
`method may comprise: (a) deblock filtering a block boundary
`of a given video block if the given video block is texture
`coded; (b) performing an intra prediction process using a
`filtered version of a reference frame used to code the given
`video block if the given video block is not texture coded, a
`neighbor video block to the given video block is not texture
`coded, and the given video block and the neighbor video
`block have substantially similar motion vectors; and (c)
`deblock filtering the block boundary of the given video block
`and performing the spatial estimation process using the filter
`version of the prediction video frame if neither (a) nor (b) is
`satisfied. Devices and/or deblocking filters that implement
`this added technique are also contemplated.
`These and other techniques and embodiments described
`herein may be implemented in a digital video device in hard
`ware, software, firmware, or any combination thereof. If
`implemented in Software, the Software may be executed in a
`digital signal processor (DSP). In that case, the software that
`executes the techniques may be initially stored in a computer
`readable medium and loaded and executed in the DSP for
`effective deblock filtering in a digital video device. Imple
`mentations using hardware, one or more processors, or com
`binations of hardware, Software, firmware and processors are
`also contemplated.
`Additional details of various embodiments are set forth in
`the accompanying drawings and the description below. Other
`features, objects and advantages will become apparent from
`the description and drawings, and from the claims.
`
`BRIEF DESCRIPTION OF DRAWINGS
`
`FIG. 1 is an exemplary block diagram of a video coding
`device according to an embodiment of this disclosure.
`FIG. 2 is a flow diagram illustrating an exemplary archi
`tecture of a video decoder unit and a deblocking unit accord
`ing to one embodiment of this disclosure.
`FIG. 3 is a flow diagram illustrating an exemplary archi
`tecture of a video decoder unit that includes integrated
`deblocking functionality according to one embodiment of
`this disclosure.
`FIG. 4 is a flow diagram illustrating an exemplary archi
`tecture of a video decoder unit that includes deblocking func
`tionality using an ITU H.264 deblocking filter.
`
`IPR2021-00827
`Unified EX1005 Page 9
`
`

`

`US 8,681,867 B2
`
`5
`FIG. 5 is a flowchart illustrating a deblocking technique
`consistent with this disclosure.
`
`DETAILED DESCRIPTION
`
`6
`Although these components are illustrated separately, in
`Some cases, deblock filtering functionality may be integrated
`into the video coding unit. In general, coding unit 14 performs
`prediction-based coding techniques, such as inter-frame cod
`ing via motion estimation and motion compensation tech
`niques, or intra-frame coding via spatial estimation and intra
`prediction techniques.
`Coding unit 14 may be referred to as a “CODEC, but
`generally refers to an encoder, a decoder, or an integrated
`encoder/decoder. The techniques described herein are most
`applicable to decoding, but may also apply during encoding
`particularly if decoding techniques are used as part of the
`encoding steps. Coding device 14 may be implemented
`within hardware, Software, firmware, one or more digital
`signal processors (DSPs), microprocessors, application spe
`cific integrated circuits (ASICs), field programmable gate
`arrays (FPGAs), discrete hardware components, or various
`combinations thereof.
`Deblocking unit 18 performs selective deblock filtering
`according to the techniques of this disclosure. In particular,
`deblocking unit 18 selectively performs deblock filtering
`with respect to block boundaries of a given video block being
`coded, performs an additional motion compensation process
`using a filtered version of the prediction video frame used to
`code the given video block, or both. Selection criteria applied
`by deblocking unit 18 determine the level and extent of
`deblock filtering in a manner that accounts for originated
`blockiness and inherited blockiness. The techniques can
`allow deblock filtering of block boundaries, but may avoid
`Such filtering in the motion compensation process in cases
`where such filtering could introduce a mismatch between
`blocks used in encoding and blocks used in decoding. In other
`words, for blocks that are texture coded, motion compensa
`tion using filtered versions of the prediction blocks can intro
`duce mismatch. Thus, for blocks that are texture coded,
`motion compensation may use non filtered-versions of the
`prediction blocks
`Various architectures are also described, which can sim
`plify implementation of the techniques of this disclosure. In
`most cases, coding device 10 may include many other com
`ponents, which are not illustrated in FIG. 1 for simplicity.
`These additional components, for example, may include
`transmitters and receivers to send and receive the video
`sequences, video capture devices, such as digital cameras, to
`capture the video sequences, or any other component com
`monly used in a device that codes video.
`Most of the popularly used video coding technologies,
`such as ITU H.264/AVC, ITU H.263, ITU H.261, MPEG-1,
`MPEG-2, MPEG-4, Windows Media Video 9.0 (WMV9) and
`Real Video 9.0 (RV9), exploit block-based transform and
`Scalar quantization as a basis of texture coding. Although the
`block based texture coding is a relatively simple and effective
`scheme to compress video, one drawback of texture coding is
`the manifestation of blocking artifacts. Blocking artifacts,
`also referred to as blockiness, are especially noticeable when
`coarse quantization is used to code the video content, which is
`typical for video transmission at low bit-rates.
`As noted above, blockiness can be classified into two types:
`originated blockiness and inherited blockiness. The origi
`nated blockiness is generally the blockiness introduced by
`poor texture coding. It is more severe when the video blocks
`are coded with higher quantization parameters. Such origi
`nated blockiness occurs at pixels around the block boundaries
`and has the same length as the block size (e.g., 8 pixels for
`MPEG-4). Since it is visible around the block boundaries, a
`low pass filter applied on those areas is usually sufficient to
`resolve the problem.
`
`This disclosure describes selective deblock filtering tech
`niques that are particularly useful with coding standards that
`do not specify in-loop deblock filtering for standard compli
`ance, such as MPEG-4 and ITU H.263 P0. In accordance with
`this disclosure, deblock filtering may be selectively per
`formed with respect to block boundaries of a given video
`block being coded, a motion compensation process can be
`performed using a filtered version of the prediction video
`frame used to code the given video block, or both. This
`disclosure also provides selection rules that can be applied to
`determine what type of deblocking techniques to apply in
`various scenarios. The selection rules may improve the video
`coding and may also ensure that mismatch between video
`blocks at an encoder and a decoder is not introduced by the
`filtering.
`The described techniques can account for both originated
`blockiness and inherited blockiness that can manifest as a
`result of prediction-based video coding. Originated blocki
`ness refers to the blockiness introduced by poor texture cod
`ing. Originated blockiness occurs at the video block bound
`aries of a coded video frame. Inherited blockiness, on the
`other hand, refers to the blockiness that can be inherited from
`the prediction video frame used by the prediction-based video
`coding. The prediction video frame refers to the reference
`frame used to perform the coding, and for motion-based
`coding, is typically a previous (or Subsequent) frame to a
`current frame of a video sequence being coded. In any case,
`inherited blockiness can occur at any location in the coded
`video frame, and is not restricted to block boundaries.
`FIG. 1 is a block diagram of a video coding device 10
`according to this disclosure. Coding device 10 may comprise
`any of a wide variety of devices that can be used to encode or
`decode video sequences. Examples of coding device 10 gen
`erally include any computer. Such as a server, a workstation or
`any other desktop computing device, or a mobile computing
`device Such as a laptop computer or a personal digital assis
`tant (PDA). Other examples include digital television broad
`casting satellites and receiving devices such as digital televi
`sions, digital cameras, digital video cameras or other digital
`recording devices. Still other examples include digital video
`telephones Such as mobile telephones having video capabili
`ties, direct two-way communication devices with video capa
`bilities, other wireless video devices, and the like. The tech
`niques may be particularly useful for Small hand-held devices
`in which size and battery consumption are more relevant
`insofar as the techniques may reduce filtering complexity.
`Coding device 10 includes a memory 12, which may com
`prise any Volatile or non-volatile storage elements. Memory
`12 generally stores video sequences, e.g., during the encoding
`and decoding process. In some cases, memory 12 may
`include both on-chip and off-chip memory. For example,
`memory 12 may include a relatively large off-chip memory
`space that stores a video sequence, and a smaller and faster
`local on-chip memory used in the coding process. In that case,
`the off-chip memory may comprise dynamic random access
`memory (DRAM), or FLASH memory, and a local on-chip
`memory may comprise synchronous random access memory
`(SRAM). For simplicity, however, a single memory 12 is
`illustrated to represent any number of memory elements that
`can be used to facilitate video coding.
`In the example shown in FIG. 1, coding device 10 includes
`a video coding unit 14 and a deblock (DB) filtering unit 18.
`
`10
`
`15
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`IPR2021-00827
`Unified EX1005 Page 10
`
`

`

`US 8,681,867 B2
`
`10
`
`15
`
`25
`
`30
`
`7
`On the other hand, inherited blockiness is the blocking
`artifact inherited from the reference frame for inter-predicted
`(or intra-predicted) video blocks. In other words, inherited
`blockiness is blockiness that is originally in the reference
`video block identified within the reference frame by the pre
`diction-based coding. Since this reference video block is used
`to construct the coded block, and is typically identified by a
`motion vector, the blockiness of the reference video block can
`be inherited by a current video block that is coded using the
`reference. Inherited blockiness artifacts may be visible at any
`location and are not restricted to block boundaries. In addi
`tion, inherited blockiness can have any length (e.g., not just 8
`pixels for MPEG-4). The traditional deblocking filter applied
`around block boundaries will typically not remove such
`inherited blockiness.
`Inherited blockiness is less of a problem in video coding
`systems that Support in-loop deblock filtering. In this case,
`filtered frames are used as reference frames for the prediction
`based coding. Therefore, in Such systems, inherited blocki
`ness is generally removed, at least to a large extent, because
`the reference frame has already been deblock filtered. After
`motion compensation, texture coding can even further reduce
`the difference between the predicted (motion compensated)
`block and the original block, which can help to reduce inher
`ited blockiness even more. Without texture coding, however,
`blockiness from the reference frame will be directly passed
`on to blocks of an encoded frame.
`For ITU H.264 and ITU H.263 P3 (Profile 3 of the ITU
`H.263 standard), which support in-loop deblocking filter,
`inherited blockiness may not be a significant problem
`because blockiness in the reference frame is generally
`reduced to acceptable levels by the in-loop deblock filtering.
`On the other hand, inherited blockiness is a more significant
`problem whena in-loop deblocking filtering is not exploited,
`e.g., in MPEG-4 and ITU H.263 P0. In this case, since the
`reference frame used in decoding the current frame is the
`reference frame before deblock filtering is applied, if there is
`not sufficient texture coding (e.g., when the quantization
`parameter is high) in the current frame, the blockiness in the
`reference frame will be passed on to the current frame. One
`40
`focus of this disclosure is a description of a post deblocking
`filter solution that can resolve the inherited blockiness prob
`lem for MPEG4 and ITU H.263 PO.
`The severity of inherited blockiness may depend on the
`block types (coded block pattern and motion vectors) and the
`type of deblocking filter used (in-loop or post-loop). Inherited
`blockiness of four different block types and two different
`filter types (loop filter and post filter) is shown in Table 1.
`
`8
`cbp=0 and post-loop filter is used. This disclosure proposes a
`solution referred to as Selective Motion Compensation based
`De-Blocking (SMCDB) in order to remove inherited blocki
`ness, particularly for those blocks with cbp=0. When cbpz0
`(cbp not equal to 0), indicating that texture coding was used
`for the video block, motion compensation based on a filtered
`reference frame may be avoided so that mismatch is not
`introduced between video blocks at the encoder and the
`decoder.
`Since in-loop deblock filtering reduces inherited blocki
`ness, one solution to remove the blocking artifacts is to
`replace the unfiltered reference frame with the filtered one at
`the decoder. However, this can introduce mismatch problems
`between the encoder and the decoder because the encoder
`uses non-filtered frames as reference frames while the
`decoder uses the filtered frames as reference frames. The
`mismatch problem may also become more severe due to error
`propagation until the next “intra’ frame arrives.
`Another approach is to apply a deblocking filter on all pixel
`locations, instead of only applying deblock filtering on block
`boundaries. Since blockiness inherited from the reference
`frame can appear anywhere, application of deblock filtering
`to all locations can remove inherited blockiness in many
`cases. This technique, however, is overly complex from an
`implementation standpoint, and often causes blurred images
`due to over-filtering.
`Yet another solution is based on history tracking. Inherited
`blockiness typically originates from an I-frame (intra-coded
`frame) and is passed from one frame to the next through
`temporal prediction. By tracking the movement of each block
`(hence the block