US008326075B2
`
`(12) United States Patent
`Wilkins et al.
`
`(10) Patent No.:
`(45) Date of Patent:
`
`US 8,326,075 B2
`Dec. 4, 2012
`
`(54) SYSTEMAND METHOD FOR VIDEO
`ENCOOING USINGADAPTIVE LOOP FILTER
`(75) Inventors: Paul Wilkins, Cambridge (GB); James
`Bankoski, North Greenbush, NY (US);
`Yaowu Xu, Clifton Park, NY (US)
`(73) Assignee: Google Inc., Mountain View, CA (US)
`
`(*) Notice:
`
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 1033 days.
`(21) Appl. No.: 12/329,070
`(22) Filed:
`Dec. 5, 2008
`
`(65)
`
`O
`O
`Prior Publication Data
`US 201O/OO61645 A1
`Mar. 11, 2010
`
`Related U.S. Application Data
`(60) Provisional application No. 61/096,147, filed on Sep.
`11, 2008.
`(51) Int. Cl
`(2006.01)
`G06K9/40
`(2006.01)
`H04N L/32
`2006.O1
`H4N I/4I
`(
`ions. 358/426.O1
`(52) U.S. Cl
`58 Fi id f c - - - - - ificati- - - - - -s - - - - - - - h
`s
`No
`(58)
`O li assi site eas - - - - - - - - - -hhi - - - - - - - - O
`ee application file for complete search history.
`References Cited
`
`(56)
`
`U.S. PATENT DOCUMENTS
`3,825,832 A
`7, 1974 Frei et al.
`4,719,642 A
`1, 1988 Lucas
`4,729,127 A
`3, 1988 Chan et al.
`4,736,446 A
`4/1988 Reynolds et al.
`4,868,764 A
`9, 1989 Richards
`
`1/1990 Mann
`4,891,748 A
`5,068,724. A 1 1/1991 Krause et al.
`5,083,214 A
`1/1992 Knowles
`93. A 38: St.i.
`5,136,376 A
`8/1992 Yagasaki et al.
`5,164,819 A 1 1/1992 Music
`5,270,812 A 12/1993 Richards
`5,274.442 A 12/1993 Murakami et al.
`(Continued)
`FOREIGN PATENT DOCUMENTS
`1555832
`7/2005
`(Continued)
`OTHER PUBLICATIONS
`“Series H. Audiovisual and Multimedia Systems; Infrastructure of
`audiovisual services Coding of moving video; Advanced video
`coding for generic audiovisual services'. H.264. Version 1. Interna
`tional Telecommunication Union. Dated May 2003.
`(Continued)
`O1
`
`EP
`
`Primary Examiner — Randolph I Chu
`(74) Attorney, Agent, or Firm — Young Basile Hanlon &
`MacFarlane, P.C.
`ABSTRACT
`(57)
`Methods for decoding frames of compressed video informa
`tion are disclosed. Each frame includes a plurality of blocks
`having a prediction stage parameter and a residual error
`attribute. One method includes categorizing each of the
`OCKS 1ntO CategOrles based. On at least One Of the ored1ct1On
`blocks i
`gories based
`1
`f the predicti
`stage parameter or the residual error attribute, determining a
`loop filter strength value to each of the categories, and filter
`ing the boundary adjacent to at least one of the blocks using
`the loop filter strength assigned to the category in which that
`block is categorized.
`
`24 Claims, 6 Drawing Sheets
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`BASELNELOOPFTERSTRENGTH
`(f) SPECIFIED FOR THE FRAME
`
`INTERREFERENCE
`FRAMESUSED
`
`GOLDEN FRAME
`
`ALTERNATEFRAME
`S5
`
`NONZEROMOTION
`WECTORORSFLIT
`
`NONEROMOTION
`WECTORORSPLT
`
`NONZEROMOTION
`WECTORORSPLT
`
`NONZERO
`+57
`
`NONZERO
`+7
`
`SPLT
`+8
`
`
`
`NONZERO
`57
`
`SPLT
`8
`
`IPR2021-00827
`Unified EX1006 Page 1
`
`

`

`US 8,326,075 B2
`Page 2
`
`U.S. PATENT DOCUMENTS
`Hingorani et al.
`1, 1994
`5,278,647
`5, 1994
`Kuban et al.
`5,313,306
`8, 1994
`Earl et al.
`5,341,440
`7, 1995
`Schwartz
`5.432,870
`10, 1995
`Shaw et al.
`5,457,780
`Agarwal
`1, 1996
`5.488,570
`Eyuboglu et al.
`T. 1996
`5,537.440
`10, 1996
`Polit
`5,561477
`11, 1996
`Lee et al.
`5,576,767
`11, 1996
`Walker et al. ................. 375/355
`5,579.348
`12, 1996
`Abecassis
`5,589,945
`4, 1997
`Civanlar et al.
`5,623.308
`5, 1997
`Haskell et al.
`5,629,736
`Fujinami
`6, 1997
`5,640,208
`8, 1997
`Porter et al.
`5,659,539
`12, 1997
`Abecassis
`5,696,869
`Agarwal
`2, 1998
`5,721,822
`3, 1998
`Wittenstein et al.
`5,734,744
`4, 1998
`Hall et al.
`5,737,020
`5, 1998
`5,748,247
`Hu
`6, 1998
`Zicket al.
`5,774,593
`Hageniers et al.
`8, 1998
`5,793,647
`10, 1998
`Astle
`5,818,969
`10, 1998
`Moeller et al.
`5,828,370
`5, 1999
`Moeller et al.
`5,903,264
`7, 1999
`Jeannin
`5,929,940
`7, 1999
`Ottesen et al.
`5,930,493
`Goldberg et al.
`10, 1999
`5,963,203
`12, 1999
`Miller et al.
`5.999,641
`1, 2000
`Cannon et al.
`6,014,706
`Hayashi et al.
`3, 2000
`6,041,145
`Ogura
`5, 2000
`6,061,397
`Chiang et al.
`T/2000
`6,084,908
`8, 2000
`Dieterich
`6,100,940
`8, 2000
`Miller et al.
`6,108,383
`Leiper
`8, 2000
`6,112,234
`9, 2000
`Chun et al.
`6,115,501
`9, 2000
`Weaver et al.
`6,119,154
`9, 2000
`Matsumura et al.
`6,125,144
`Sugiyama
`10, 2000
`6,141,381
`12, 2000
`6,167, 164
`Lee
`1, 2001
`Miller et al.
`6,181,822
`2, 2001
`Dimitrova et al.
`6,185,363
`2, 2001
`Tan et al.
`6,188,799
`5, 2001
`Kim
`6,240,135
`9, 2001
`Miller et al.
`6,292,837
`12, 2001
`Miller et al.
`6,327,304
`4, 2002
`Miller et al.
`6,370,267
`6, 2002
`6,400,763
`Wee
`T/2002
`Okumura et al.
`6,414.995
`T/2002
`Wu et al.
`6,418,166
`Wang et al.
`8, 2002
`6,434, 197
`2, 2003
`Zlotnick
`6,522,784
`3, 2003
`6,529,638
`Westerman
`5/2003
`Wilkins
`6,560,366
`2, 2004
`Ishihara
`6,687,303
`2, 2004
`Wee et al.
`6,697,061
`3, 2004
`Tan et al.
`6,707,952
`T/2004
`Conklin
`6,765,964
`Uenoyama et al.
`9, 2004
`6,798,837
`11, 2004
`Kawashima et al.
`6,826,229
`6, 2005
`Gomila et al.
`6,907,079
`8, 2005
`Zlotnick
`6,934.419
`1, 2006
`Bottreau et al.
`6,985,526
`1, 2006
`6,987,866
`Hu
`4, 2006
`Ameres et al.
`7,027,654
`Mukerjee et al.
`10, 2006
`7,116,831
`1/2007
`Zhou
`7,170,937
`3, 2007
`Melanson ..................... 375,247
`7,194,036
`6, 2007
`Yoshimura et al.
`7,226,150
`6, 2007
`Sun et al.
`7,236,524
`2, 2008
`Lu et al.
`7,330,509
`4, 2008
`Melanson ..................... 341,144
`7,358,881
`3, 2009
`Ameres et al.
`7,499.492
`T/2010
`Tsuie et al. ................... 375/350
`7,751,514
`5, 2002
`Sethuraman et al.
`2002fOO64228
`11, 2002
`Valente
`2002fO168114
`1, 2003
`Lee et al.
`2003/0023982
`
`B2
`B1
`B2
`B2
`B2
`B2
`B1
`B2
`A1
`A1
`A1
`
`2003. O189982 A1 10, 2003 MacInnis
`2004/O120400 A1
`6/2004 Linzer
`2004/017961.0 A1
`9, 2004 Lu et al.
`2004/O181564 A1* 9, 2004 MacInnis et al. ............. TO8,322
`2004/022841.0 A1 11/2004 Ameres et al.
`2004/0240556 Al 12/2004 Winger et al.
`2005, 0013494 A1
`1/2005 Srinivasan et al.
`2005.00532.94 A1
`3/2005 Mukerjee et al.
`2005/01 17653 A1
`6/2005 Sankaran
`2005, 0147165 A1
`7, 2005 Yoo et al.
`2005, 01693.74 A1
`8/2005 Marpe et al.
`2005/0196063 A1* 9/2005 Guangxi et al. .............. 382.261
`2005/0265447 A1 12/2005 Park
`2005/0276323 Al 12/2005 Martemyanov et al.
`2005/0276,327 A1 12/2005 Lee et al.
`2006.0062311 A1
`3/2006 Sun et al.
`2006/0098.737 A1
`5/2006 Sethuraman et al.
`2006/O126962 A1* 6/2006 Sun ............................... 382,268
`2006, O153301 A1
`7/2006 Guleryuz
`2006,0182181 A1
`8, 2006 Lee et al.
`2006/0215758 A1
`9/2006 Kawashima
`2006/0239345 A1 10, 2006 Taubman et al.
`2007,0009044 A1
`1/2007 Tourapis et al.
`2007/0025448 A1
`2/2007 Cha et al.
`2007/0047648 A1
`3/2007 Tourapis et al.
`2007, 0140342 A1
`6/2007 Karczewicz et al.
`2007/0177673 A1
`8/2007 Yang
`2007/O189735 A1
`8/2007 Kawashima et al.
`2007/02O1559 A1
`8, 2007 He
`2007/0237241 A1 10, 2007 Ha et al.
`2007,0253490 A1 11/2007 Makino
`2007,0253491 A1 11/2007 to et al.
`2007/0274385 A1 11/2007 He
`2008.0025398 A1
`1/2008 Molloy et al.
`2008.0025411 A1
`1/2008 Chen et al.
`2008. O170629 A1
`7/2008 Shim et al.
`20080212678 A1
`9, 2008 Booth et al.
`2009/0287493 A1* 11/2009 Janssen et al. ................ TO4,500
`
`FOREIGN PATENT DOCUMENTS
`1838.108 A1
`9, 2007
`10-2005-0018730
`2, 2005
`10-2008-0056083
`6, 2008
`2002089487 A2 11/2002
`2006062377 A1
`6, 2006
`2006083614 A2
`8, 2006
`20070523O3 A2
`5/2007
`2008.005124 A2
`1, 2008
`
`EP
`KR
`KR
`WO
`WO
`WO
`WO
`WO
`
`OTHER PUBLICATIONS
`
`“Series H. Audiovisual and Multimedia Systems; Infrastructure of
`audiovisual services Coding of moving video; Advanced video
`coding for generic audiovisual services'. H.264. Version 3. Interna
`tional Telecommunication Union. Dated Mar. 2005.
`“Overview; VP7 Data Format and Decoder'. Version 1.5. On2 Tech
`nologies, Inc. Dated Mar. 28, 2005.
`“Series H. Audiovisual and Multimedia Systems; Infrastructure of
`audiovisual services Coding of moving video; Advanced video
`coding for generic audiovisual services'. H.264. Amendment 1: Sup
`port of additional colour spaces and removal of the High 4:4:4 Profile.
`International Telecommunication Union. Dated Jun. 2006.
`“VP6 Bitstream & Decoder Specification'. Version 1.02. On2 Tech
`nologies, Inc. Dated Aug. 17, 2006.
`“Series H. Audiovisual and Multimedia Systems; Infrastructure of
`audiovisual services Coding of moving video'. H.264. Amend
`ment 2: New profiles for professional applications. International
`Telecommunication Union. Dated Apr. 2007.
`“VP6 Bitstream & Decoder Specification”. Version 1.03. On2 Tech
`nologies, Inc. Dated Oct. 29, 2007.
`“Series H. Audiovisual and Multimedia Systems; Infrastructure of
`audiovisual services Coding of moving video'. H.264. Advanced
`video coding for generic audiovisual services. Version 8. Interna
`tional Telecommunication Union. Dated Nov. 2007.
`Architectures for Efficient Partitioning of Video Coding Algo
`rithms—H.264 decoder, Nov. 2006.
`
`IPR2021-00827
`Unified EX1006 Page 2
`
`

`

`US 8,326,075 B2
`Page 3
`
`Wenger et al.; RTP Payload Format for H.264 Video: The Internet
`Society; 2005.
`A High Efficient Method for Parallelizing Reconstructor & Loop
`Deblocking Filter on Multi-core Processor Platform, Feb. 2008.
`Lihua Zhu, Guangfei Zhu, Charles Wang; Implementation of video
`deblocking filter on GPU, Apr. 2008.
`Method for unloading YUV-filtered pixels from a deblocking filter
`for a video decoder, Oct. 2006.
`An Optimized In-Loop H.264 De-Blocking Filter on Multi-Core
`Engines, Feb. 2007.
`U.S. Appl. No. 12/102,694, Video Compression and Encoding
`Method, filed Apr. 14, 2008.
`U.S. Appl. No. 11/932,870, Video Compression and Encoding
`Method, filed Oct. 31, 2007.
`
`ON2 Technologies, Inc., White Paper TrueMotionVP7Video Codec,
`Jan. 10, 2005, 13 pages, Document Version: 1.0, Clifton Park, New
`York.
`ON2 Technologies, Inc., White Paper On2's TrueMotion VP7 Video
`Codec, Jul. 11, 2008, 7 pages, DocumentVersion: 1.0, Clifton Park,
`New York.
`Sikora, T. et al. Shape-adaptive DCT for generic coding of video,
`Circuits and Systems for Video Technology, IEEE Transactions on
`vol. 5, Issue 1, p. 59-62, Feb. 1, 1995.
`Wiegand et al. “Overview of the H264/AVC Video Coding Stan
`dard.” IEEE Transactions on Circuits and Systems for Video Tech
`nology, vol. 13, No. 7, pp. 568, 569, Jul. 1, 2003.
`PCT International Search Report and Written Opinion (Date of Mail
`ing: Apr. 21, 2010); PCT/US2009/056462, filed Sep. 10, 2009.
`* cited by examiner
`
`IPR2021-00827
`Unified EX1006 Page 3
`
`

`

`U.S. Patent
`
`Dec. 4, 2012
`
`Sheet 1 of 6
`
`9US 8
`
`326,075 B2
`
`dassdddNOo
`
`WVAUISIG
`
`AdOUWLNY
`
`ONIGOONA
`
`0¢~\-#
`WYOASNVAL
`HSUAANI
`
`CE“\—w|NOLONULSNOOTY
`
`8Z-\_»|NOILVZLINVNOAG
`
`NOILVZILNVNOWwOASNVLL
`eeONLI|dOO1
`
`veJ
`
`NOLLDIGHYd
`
`WVAULS
`
`IPR2021-00827
`Unified EX1006 Page 4
`
`WALNI/AVLINI
`
`OdCIALNdNI
`
`IPR2021-00827
`Unified EX1006 Page 4
`
`
`
`
`
`
`
`

`

`U.S. Patent
`
`Dec. 4, 2012
`
`Sheet 2 of 6
`
`US 8,326,075 B2
`
`?NIXIOOT{[HCl
`
`YIHITIH
`
`NOILOIOIHYIAI
`
`/--
`
`
`
`WYIORISNWYIL
`
`HSYIHANI
`
`
`
`
`
`NOIIVZI INVITÒRGI
`
`
`
`KJO HLNH
`
`?NIGIOONH
`
`([HSSHYRIWOO
`
`
`
`WW.HYIJ SLI!!
`
`
`
`
`
`
`
`
`
`IPR2021-00827
`Unified EX1006 Page 5
`
`

`

`U.S. Patent
`
`Dec. 4, 2012
`
`Sheet 3 of 6
`
`US 8,326,075 B2
`
`44
`
`
`
`46-?as
`
`PREDICTIONUSING
`A PREVIOUSLY
`--- CODED FRAME
`
`PREDICTION
`WITHIN CURRENT
`NN FRAME
`-N-
`-38"
`
`38
`
`--- PREDICTIONUSING
`A PREVIOUSLY
`CODED FRAME
`CURRENT FRAME
`
`61
`
`34
`
`60
`
`STRENGTH
`MODIFIER
`
`
`
`LOOP
`FILTER
`CONTROL
`
`FIG. 4
`
`PREVIOUSLY
`CODED FRAMES
`
`65
`
`62
`
`REFERENCE
`64 FRAMETYPE
`N PREDICTION
`66
`MODE |
`- - - - - -
`
`RESIDUAL
`ERROR
`ATTRIBUTE
`
`IPR2021-00827
`Unified EX1006 Page 6
`
`

`

`U.S. Patent
`
`Dec. 4, 2012
`
`Sheet 4 of 6
`
`US 8,326,075 B2
`
`TWOld6d84Ld94Sdbdededld
`
`89+LQO+
`
`89+LQ+
`
`LT1dSOWdZ-NON
`
`
`LITdSOWdZ-NON
`
`OWAZ-NON
`
`LO+
`
`FAVALVNUELTV
`
`GO+3
`
`AWVUdNACIOO
`
`PO+}
`
`OLL
`
`
`
`HINVYLSV1
`
`EO+F
`
`
`
`“HOIDIANOLLOW
`
`
`
`‘IOLOAANOLLOW
`
`
`
`“HOLDIANOLLON
`
`NOILOWOWHZ-NON
`
`
`NOILOWOW4Z-NON
`
`NOLLOWOWXZ-NON
`
`L1dSYOYOLOAA
`
`LT1d$YOWOLOLA
`
`LTTd$WOWOLOAA
`
`HONAMATAWAIN
`
`
`
`dasnSIHWVad
`
`
`
`AAVFHLWOdAaLdIOgdS(3)
`
`
`
`d4d09VuLNIdN
`
`CP+IO+3
`
`LI1dS
`
`diSnYaO0IN
`
`
`
`
`
`HLONAULSWALTdOO'TANITASVA
`
`IPR2021-00827
`Unified EX1006 Page 7
`
`IPR2021-00827
`Unified EX1006 Page 7
`
`
`
`
`
`
`
`
`
`
`

`

`U.S. Patent
`
`Dec. 4, 2012
`
`Sheet 5 of 6
`
`US 8,326,075 B2
`
`SETREFERENCE
`FRAME LOOPFILTER
`MODIFIERSTO
`DEFAULT
`
`SET PREDICTION
`MODE LOOPFILTER
`MODIFIERSTO
`DEFAULT
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`132
`
`?
`IS
`CURRENT FRAME
`AKEYFRAME
`
`
`
`LOOP
`FILTERMODIFIERS
`ENABLED FORCURRENT
`FRAME
`
`UPDATE
`LOOPFILTER
`MODIFIERS
`
`UPDATEREFERENCE
`FRAME LOOPFILTER 1\140
`MODIFIERS
`
`UPDATE PREDICTION
`MODELOOPFILTER 1\142
`MODIFIERS
`
`FIG. 6
`
`IPR2021-00827
`Unified EX1006 Page 8
`
`

`

`U.S. Patent
`
`Dec. 4, 2012
`
`Sheet 6 of 6
`
`US 8,326,075 B2
`
`
`
`
`
`NO
`
`
`
`
`
`LOOP
`FILTERMODIFIERS
`ENABLED FORCURRENT
`FRAME
`
`UPDATE
`LOOPFILTER
`MODIFIERS
`
`
`
`
`
`UPDATEREFERENCE
`FRAMELOOPFILTER 1\140
`MODIFIERS
`
`UPDATE PREDICTION
`MODELOOPFILTER 1\142
`MODIFIERS
`
`FIG.7
`
`IPR2021-00827
`Unified EX1006 Page 9
`
`

`

`1.
`SYSTEMAND METHOD FOR VIDEO
`ENCOOING USINGADAPTIVE LOOP FILTER
`
`US 8,326,075 B2
`
`2
`stage parameter with respect to at least one of the blocks. The
`method includes reconstructing the at least one block based
`on the prediction stage parameter, computing a residual error
`attribute from the reconstructed block, computing a filter
`strength value based on a baseline filter strength and at least
`one incremental value. The incremental value is selected from
`a plurality of preset values based at least on one of the pre
`diction stage parameter and residual error attribute associated
`with the at least one block. The boundary adjacent to the at
`least one block is filtered using the selected filter strength
`value.
`Other embodiments of the invention are described in addi
`tional detail hereinafter.
`
`15
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`CROSS-REFERENCE TO RELATED
`APPLICATIONS
`
`This application claims priority to U.S. provisional patent
`application No. 61/096,147, filed Sep. 11, 2008, which is
`incorporated herein in its entirety by reference.
`
`10
`
`TECHNICAL FIELD
`
`The present invention relates in general to video encoding
`and more particularly, video encoding using a loop filter.
`
`BACKGROUND
`
`An increasing number of applications today make use of
`digital video for various purposes including, for example,
`remote business meetings via video conferencing, high defi
`nition video entertainment, video advertisements, and shar
`ing of user-generated videos. AS technology is evolving,
`people have higher expectations for video quality and expect
`high resolution video with Smooth playback at a high frame
`rate.
`There can be many factors to consider when selecting a
`Video coder for viewing digital video. Some applications may
`require excellent video quality where others may need to
`comply with various constraints including, for example,
`bandwidth or storage requirements. To permit higher quality
`transmission of video while limiting bandwidth consump
`tion, a number of video compression schemes are noted
`including proprietary formats such as VPX (promulgated by
`On2 Technologies, Inc. of Clifton Park, N.Y.), H.264 standard
`promulgated by ITU-TVideo Coding Experts Group (VCEG)
`and the ISO/IEC Moving Picture Experts Group (MPEG),
`including present and future versions thereof. H.264 is also
`known as MPEG-4 Part 10 or MPEG-4 AVC (formally, ISO/
`IEC 14496-10).
`Many video coding techniques use block based prediction
`and quantized block transforms. With block based prediction,
`a reconstructed frame buffer can be used to predict subse
`quent frames. The use of block based prediction and quan
`tized block transforms can give rise to discontinuities along
`block boundaries. These discontinuities (commonly referred
`to as blocking artifacts) can be visually disturbing and can
`reduce the effectiveness of the reference frame as a predictor
`for Subsequent frames. These discontinuities can be reduced
`by the application of a loop filter. The loop filter can be
`applied to the reconstructed frame buffers. Some conven
`tional loop filters apply different filtering strengths to differ
`ent block boundaries. For example, Some compression sys
`tems vary the strength of the loop filter based on, for example,
`whether the block has been inter-coded or intra-coded. Other
`compression systems apply a filter strength based on, for
`example, the difference between the extent of the discontinu
`ity and threshold level. Further, for example, some compres
`sion systems may vary the strength of the loop filter by com
`puting a difference value illumination change of a block
`compared to its neighboring block.
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`SUMMARY
`
`One embodiment of the invention is disclosed as a method
`for reducing blocking artifacts at the boundary between adja
`cent blocks reconstructed from a frame of compressed video
`information. The video information includes a prediction
`
`65
`
`The description herein makes reference to the accompany
`ing drawings wherein like reference numerals refer to like
`parts throughout the several views.
`FIG. 1 is a block diagram of a video compression system in
`accordance with one embodiment of the present invention.
`FIG. 2 is a block diagram of a video decompression system
`in accordance with one embodiment of the present invention.
`FIG. 3 is a schematic diagram of intra-prediction and inter
`prediction modes used in the video compression and decom
`pression systems of FIGS. 1 and 2.
`FIG. 4 is a block diagram of a loop filter control used to
`compute a strength modifier used in the video compression
`system of FIG. 1.
`FIG. 5 is a flowchart diagram of a method of selecting the
`strength modifier of FIG. 4.
`FIG. 6 is a flowchart diagram of a method of updating loop
`filtering video data used in the video compression system of
`FIG 1.
`FIG. 7 is a flowchart diagram of another method of updat
`ing loop filtering video data used in the video compression
`system of FIG. 1.
`
`DETAILED DESCRIPTION
`
`Disclosed herein are embodiments of an adaptive loop
`filter that remove or reduce blocking artifacts. Further, dis
`closed herein are embodiments of an adaptive loop filter that
`either remove or reduce blocking artifacts usingless overhead
`data and/or reduce computational complexity.
`FIG. 1 is a block diagram of a video encoder 14 using an
`adaptive loop filter 34 according to one embodiment of the
`present invention.
`In the disclosed embodiments, block-based video com
`pression operates on fixed-shaped groups of neighboring pix
`els, called a macroblock. In general, each frame of video can
`be divided into macroblocks, where each macroblock con
`sists of a plurality of smaller-sized blocks. These pixel groups
`within the macroblocks and blocks can be compared with
`either data found in the current frame or in other frames in
`order to formulate motion data and error signals. In this
`embodiment, each macroblock can be a group of 16x16 pix
`els. In other embodiments, macroblocks can also be any other
`suitable size.
`Although the description of embodiments of the adaptive
`loop filter innovations are described in the context of the VP8
`Video coding format, alternative embodiments of the present
`invention can be implemented in the context of other video
`coding formats. Further, the embodiments are not limited to
`any specific video coding standard or format.
`To remove discontinuities at block boundaries, loop filter
`ing can be applied to reconstructed frames during a recon
`
`IPR2021-00827
`Unified EX1006 Page 10
`
`

`

`3
`struction path. As explained in more detail below, the choice
`of loop filter and the strength of the loop filter can have a
`significant effect on image quality. A filter that is too strong
`may cause blurring and loss of detail. A filter that it is too
`weak may not adequately suppress discontinuities between
`adjacent blocks.
`Referring to FIG. 1, to encode an input video stream 16,
`encoder 14 performs the following functions in a forward
`path (shown by the Solid connection lines) to produce an
`encoded bitstream 26: intra/interprediction 18, transform 19,
`quantization 22 and entropy encoding 24. Encoder 14 also
`includes a reconstruction path (shown by the dotted connec
`tion lines) to reconstruct a frame for encoding of further
`macroblocks. Encoder 14 performs the following functions in
`the reconstruction path: dequantization 28, inverse transfor
`mation 30, reconstruction 32 and loop filtering 34. Other
`structural variations of encoder 14 can be used to encode
`bitstream 26.
`Referring to FIG. 1, when input video stream 16 is pre
`sented for encoding, each frame within input video stream 16
`can be processed in units of macroblocks. At intra/interpre
`diction stage 18, each macroblock can be encoded using
`either intra prediction or interprediction mode. In either case,
`a prediction macroblock can be formed based on a recon
`structed frame. In the case of intra-prediction, for example, a
`prediction macroblock can be formed from samples in the
`current frame that have been previously encoded and recon
`structed. In the case of inter-prediction, for example, a pre
`diction macroblock can beformed from one or more previous
`or future frames (i.e. reference frames) that have already been
`encoded and reconstructed. Further, alternate embodiments
`can encode a macroblock by using some combination of both
`intra prediction and interprediction.
`Next, still referring to FIG. 1, the prediction macroblock
`can be subtracted from the current macroblock to produce a
`residual macroblock (residual). Transform stage 19 transform
`codes the residual and quantization stage 22 quantizes the
`residual to provide a set of quantized transformed coeffi
`cients. The quantized transformed coefficients can be then
`entropy coded by entropy encoding stage 24. The entropy
`coded coefficients, together with the information required to
`decode the macroblock, Such as the type of prediction mode
`used, motion vectors and quantizer value, can be outputted to
`compressed bitstream 26.
`The reconstruction path in FIG. 1 can be present to permit
`that both the encoder and the decoder use the same reference
`frames required to decode the macroblocks. The reconstruc
`tion path, similar to functions that take place during the
`decoding process, which are discussed in more detail below,
`includes dequantizing the transformed coefficients by
`dequantization stage 28 and inverse transforming the coeffi
`cients by inverse transform stage 30 to produce a derivative
`residual macroblock (derivative residual). At the reconstruc
`tion stage 32, the prediction macroblock can be added to the
`derivative residual to create a reconstructed macroblock. The
`adaptive loop filter 34 can be applied to the reconstructed
`macroblock to reduce blocking artifacts.
`Referring to FIG. 2, in accordance with one embodiment,
`to decode compressed bitstream 26, a decoder 21 having a
`structure similar to the reconstruction path of the encoder 14
`discussed previously performs the following functions to pro
`duce an output video stream 35: entropy decoding 25,
`dequantization 27, inverse transformation 29, intra/interpre
`diction 23, reconstruction 31, adaptive loop filter 34 and
`deblocking filtering 33. Other structural variations of decoder
`21 can be used to decode compressed bitstream 26.
`
`10
`
`15
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`US 8,326,075 B2
`
`4
`When compressed bitstream 26 is presented for decoding,
`the data elements can be entropy decoded by entropy decod
`ing stage 25 to produce a set of quantized coefficients.
`Dequantization stage 27 dequantizes and inverse transform
`stage 29 inverse transforms the coefficients to produce a
`derivative residual that is identical to that created by the
`reconstruction stage in the encoder 14. Using header infor
`mation decoded from the compressed bitstream 26, at intra/
`interprediction stage 23, decoder 21 creates the same predic
`tion macroblock as was created in encoder 14. At the
`reconstruction stage 31, the prediction macroblock can be
`added to the derivative residual to create a reconstructed
`macroblock. The adaptive loop filter 34 can be applied to the
`reconstructed macroblock to reduce blocking artifacts. A
`deblocking filter 33 can be applied to the reconstructed mac
`roblock to further reduce blocking distortion and the result
`can be outputted to output video stream 35.
`Although the description of embodiments of the adaptive
`loop filter innovations are described with reference to adap
`tive loop filter 34 in the encoder, the described filtering tech
`niques are also implemented in adaptive loop filter 34 in the
`decoder. Reference to adaptive loop filter 34 in the decoder
`has been omitted throughout the disclosure only to aid in
`understanding of the invention. However, the filtering inno
`vations are not limited to adaptive loop filter 34 in the encoder
`and can be applied to adaptive loop filter 34 in the decoder or
`any other unit incorporating filtering techniques.
`FIG. 3 shows reference frames 44, 48 and a current frame
`36 that is currently being encoded or decoded. As discussed
`previously, each frame can be processed in units of macrob
`locks and at intra/inter prediction stage 18, each macroblock
`can be encoded using either intra prediction, interprediction
`Some combination of interprediction and intraprediction. For
`example, a current macroblock 38 is being encoded or
`decoded using inter prediction from a macroblock 46 from
`previously coded reference frame 44. Similarly, a current
`macroblock 38 is being encoded or decoded using interpre
`diction from a macroblock 50 from previously encoded ref
`erence frame 48. Also, for example, a current macroblock 38"
`is being encoded or decoded using intra prediction from a
`macroblock 52 within current frame 36.
`Blocking artifacts can be created during the encoding pro
`cess and can originate from, for example, intra/interpredic
`tion stage 18, transform stage 19 or quantization stage 22.
`Since some conventional filters make filter strength depen
`dent on block boundaries, computational processing can be
`complex and time-consuming.
`FIG. 4 is a block diagram illustrating a loop filter control 61
`of adaptive loop filter 34 in one embodiment of the present
`invention. According to one embodiment, loop filter control
`61 determines strength modifier 60 based on block attributes.
`Block attributes are based on existing encoded information
`about a block or information that is passed to the decoder to
`assist in properly decoding the bitstream.
`Block attributes can include a prediction stage parameter
`65 and a residual error attribute 66. Prediction stage param
`eter 65 can include a reference frame type 62 and a type of
`prediction mode 64. As discussed in more detail below,
`strength modifier 60 alters the levels of thresholds in adaptive
`loop filter 34.
`Reference frame type 62 can be determined by, similar to
`the illustration in FIG. 3, whether intra mode or inter frame
`mode coding is used when constructing prediction blocks. If
`intra mode predictive coding is used, reference frame type 62
`can be intra-frame (i.e. the current frame). When using an
`intra-frame, the prediction block can be formed, as discussed
`
`IPR2021-00827
`Unified EX1006 Page 11
`
`

`

`5
`previously, from samples in the current frame that have been
`previously encoded and reconstructed.
`If inter mode predictive coding is used, inter-frames can be
`used as a basis for formulating the prediction block. When
`using inter-frames, the prediction block can be formed, for
`example, from one or more previous frames, future frames or
`some combination thereof that have already been encoded
`and reconstructed. Accordingly, when using inter-frames, ref
`erence frame type 62 may include, for example, a last frame,
`a golden frame oran alternate reference frame. The last frame
`can be the previously encoded frame before the current frame.
`The golden frame can be a past frame chosen arbitrarily from
`the distant past to use as a predictor for Subsequent frames.
`The alternate reference frame may include any frame that is
`not the last frame or the golden frame. For example, the
`alternate reference can be a past frame, a future frame, or a
`constructed reference frame. Further, for example, the con
`structed reference may be the reference frame as disclosed in
`patent application titled “System and Method for Video
`Encoding. Using Constructed Reference Frame' that is
`assigned to the assignee of the present invention, is filed
`concurrently herewith and which is hereby incorporated by
`reference in its entirety.
`Type of prediction mode 64 can be determined, similar to
`reference frame type 62, by whether intra mode or interframe
`mode coding is used when constructing prediction blocks (as
`illustrated in FIG.3). If intra mode predictive coding is used,
`two types of intra-coding can be supported, which are
`denoted as non-split mode and split mode. If inter mode
`predictive coding is used, two types of inter-coding can be
`supported, which are denoted as non-split mode and split
`mode.
`If inter mode predictive coding is used with non-split
`mode, residual error attribute 66 can be determined by
`whether the resulting motion vector is null or is non-zero.
`As discussed previously, a macroblock can be an array of
`16x16 luminance pixels. In intra-coding, each macroblock
`can be further split into, for example, 4x4 luminance samples
`referred to as 4x4 Sub-blocks. Accordingly, a macroblock can
`be made of 16 4x4 sub-blocks. This means that a prediction
`block may be formed for either a macroblock (i.e. non-split
`mode) or each of the 16 4x4 sub-blocks (i.e. split mode).
`Other sub-block sizes are also available such as 16x8, 8x16,
`and 8x8. Although the description of embodiments for intra
`coding arc is described with reference to 4x4 sub-block split
`mode, any other Sub-block size can be used with split mode,
`and the description of the embodiments is not limited to a 4x4
`sub-block.
`In intra-coding, non-split mode results in prediction of the
`whole 16x16 macroblock whereas split mode leads to sepa
`rately predicting each 4x4 Sub-block.
`For intra-coding non-split mode, for example, one of four
`prediction modes can be utilized to reference neighboring
`pixel samples of previously-coded blocks which are to the left
`and/or above the 16x16 block to be predicted. The four select
`able prediction modes may be vertical prediction, horizontal
`prediction, DC prediction and plane prediction.
`For intra-coding split mode, for example, one of nine pre
`diction modes can be utilized to reference neighboring pixel
`samples of previously-coded blocks which are to the left
`and/or above the 4x4 sub-block to be predicted. The nine
`selectable prediction modes may be vertical prediction, hori
`Zontal prediction, DC prediction, diagonal down-left predic
`tion, diagonal down-right prediction, Vertical-right predic
`tion, horizontal-down prediction, Vertical-left prediction and
`horizontal-up prediction.
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`US 8,326,075 B2
`
`10
`
`15
`
`6
`In inter-coding, non-split mode results in calculating one or
`motion vectors based on displacing an area of a correspond
`ing reference frame for prediction of the whole 16x16 mac
`roblock. Alternatively, split mode results in calculating a
`motion vector based on displacing an area of a corresponding
`reference frame for prediction of a partition of the 16x16
`macroblock. The 16x16 macroblock may be split into parti
`tions of 16x8, 8x16, 8x8 or 4x4 each with its own motion
`vector. Other partition sizes are also available.
`A motion vector can be calculated for each whole macrob
`lock or each separate partition. In particular, motion compen
`sation predicts pixel values of the macroblock (or the corre
`sponding partition within the macroblock) from a translation
`of the reference frame. The motion vector for each macrob
`lock or partition may either be null, which indicates there has
`been no change in motion, or non-zero, which indicates there
`has been a change in motion.
`Although the description of embodiments describes how
`adaptive loop filter 34 applies a different strength modifier 60
`based on the prediction stage parameter 65 and residual error
`attribute 64, any other loop filter attribute may be varied such
`as the filter type, filter coefficients, and filter taps, and the
`description of th