|||||||||||||||||||||||||||||||||l|||||
`
`U8008363724B2
`
`(12) United States Patent
`Su et a].
`
`(10) Patent N0.:
`
`(45; Date of Patent:
`
`US 8,363,724 32
`Jan. 29, 2013
`
`(54) METHODS AND APPARATUS USING
`VIRTUAL REFERENCE PICTURES
`
`[75)
`
`Inventors: Yeping Su. Vancouver. WA (US); Peng
`Yin. West Windsor, NJ (US): Pun-1n
`Bihhas Pandlt. Franklin Park. NJ (US):
`Cristina Gomila. Princeton. NJ (US)
`
`[73) Assignee: Thomson Licensing, Boulogne.
`Billancourt (FR)
`
`[ "‘ ) Notice:
`
`Subject to any disclaimer, the term ofthis
`patent is extended or aidinsted under 35
`USC‘. l54(b) by 728 days.
`
`(21) App1.No.:
`
`123309.066
`
`(22) PCT Filed:
`
`Jul. 10. 2007
`
`[86) PCT No.:
`
`PCTIUSZBOTJO'15719
`
`§371 (CHI).
`(2). (4) Date:
`
`Jan. 5. 2009
`
`(87) PCT Pub. No; W020081008331
`
`PCT Pub. Date: Jan. 17, 2008
`
`(65)
`
`Prior Publication Data
`US 201 1102413224 A]
`Oct. 6. 20]]
`
`325324001
`
`(56)
`
`References Cited
`
`US. PATENT DOCUMENTS
`7.650.030 132
`l-"20l'0 Lei et a].
`2.929.605 B2
`432011 Kimata el al.
`8.179.960 B2 "'
`532012 McGowan el al.
`2002-0114392 Al
`8.52002 Sekiguchi at a].
`2006:“0l46l43 Al
`1-2006 Xin et 31.
`
`JP
`JP
`JP
`JP
`JP
`W0
`WO
`W0
`
`FOR] EIGN PA'l'l 5N1“ DOCUMENTS
`2000-209588 A}.
`732000
`2004-088801 A
`3-"2004
`2005-124300 A
`5-"2005
`2006-12917? AP.
`5-".2006
`12007-159111 A
`6:200?
`2005.-'0|1285 Al
`2-"2005
`2005007253 A1
`“"2005
`20063064210 AI
`6.52006
`
`O'l‘lll'iR PUBLICA'TIONS
`
`Budagavi: “Video Compression using Blur Compensation." Image
`ProceSsing. IEEE International Conference in Geneva. Italy. Sep.
`11-14. 2005. Piscatnwny. Nipp.882-885.}(P-010851195.
`Del‘anx er 0].: "Background mosaicking for low bit rate video cod-
`ing.“ IEEE Proceedings of the International Conference on Image
`Processing. Lansanne. Sap.
`l6— l‘).
`1996, NY. pp. 673676,
`XPOIUZUEISS.
`
`(Continued)
`
`Primary Examiner — Donald Mills
`(74) glimmer. Agent, or Firm — Robert D. Shedd: Guy [-l.
`Eriksen
`
`Related US. Application Data
`
`[57)
`
`ABSTRAC'I'
`
`(60) Provisional application No. 600330.195. filed on Ju].
`l l. 2006.
`
`(51) 00.0.
`(2000.01)
`1104.0! 2m
`375040.15; 375040.01: 37504012
`(52) us. (:1.
`(58) Field of Classification Search
`None
`See application file for complete search history.
`
`There are provided methods and apparatus using virtual ref—
`erence pictures. An apparatus includes an encoder for encod-
`ing at least one picture, using at least one virtual reference
`picture. to form a resultant bitstream. The at least one virtual
`relererice picture is dill‘erent than a previously decoded pic-
`lure.
`
`87 Claims. 5 Drawing Sheets
`
`200
`
`/-
`
`REFERENCE
`PlCTURE
`
`
`
`DECODEJ
`OUTPUT
`
`PICTURE
`
`BUFFER
`
`2508
`290
`250
`
`UNIFIED 1 00 1
`
`UNIFIED 1001
`
`

`

`US 8,363,724 B2
`
`Page 2
`
`OTHER PUBLICATIONS
`ISO-“115C JTCl.-'SC29.-WGll NT53‘): “Requirements on Multi-view
`Video Coding v.5.” Intemalional Organization for Standardisation.
`Coding of Moving Pictures and Audio. Nice. France. Oct. 2005. 5
`I)?!»
`ISOIIEC HCI.-"SC29.I'WG11 NSDIR: 'Teehnologies under study for
`reference piclure management and high-level syntax for multiview
`video coding." Montreux, Switzerland. Apr. 2006. XP-UO30014SIU.
`ISOIIEC JTCi.-'SC29IWG1] N8019: “Description 01' Core Experi-
`ments in MV’C." Montreux, Switzerland. Apr. 2006.
`ITU-T Recommendation H.264. “Advanced video coding forgeneric
`audiovisual services." Series 1-1: Audiovisual and Multimedia Sys-
`
`tems. Infraeruclure of Audiovisual Services—Coding of moving
`video. Mar. 2005.
`Kim 0131.: “Comments on I-Iigh- Level syntax for MVC Contribution
`to the 75m MPEG meeting,” International Organisation for
`Slandardisalien. ISOJIEC J'TC l.-’SC29.-'WG1 MPEGZOOS.-"m133l9.
`Moulrem. Apr. 2006. lip—003004 1988.
`Martinian, 13. ct a].: Results of Core Experiment IE on Multivicw
`Coding.
`ISOH'IEC J'l"Clx'SC29.-’WGIL M13122. Apr.
`2006.
`Moniremi. Switzerland. 5 pgs.
`International Search Report. dated Jul.
`
`l l. 2003.
`
`* cited by examiner
`
`

`

`US. Patent
`
`Jan. 29, 2013
`
`Sheet 1 of 5
`
`US 8,363,724 32
`
`
`
`DECODED
`PICTURE
`BUFFER
`
`
`
`180
`
`
`REFERENCE
`PICTURE
`
`BUFFER
`
`BH’STREAM
`
`
`
`
`VIRTUAL
`REFERENCE
`
`PICTURE
`
`
`
`DECODED
`OUTPUT
`PICTURE
`
`BUFFER
`
`2503
`250
`290
`
`FIG. 2
`
`

`

`US. Patent
`
`Jan. 29, 2013
`
`Sheet 2 of 5
`
`US 8,363,724 32
`
`w 305 f
`
`300
`
`SET VRP_PHESENT_FLAG = 0
`
`31 U
`
`315
`
`NO
`
`IS VFIP ENABLED
`?
`
`YES
`
`SET VFIP_PRESENT_FLAG = 1
`
`SET NUM_VRF'S AND VRP
`PARAMETER SYNTAXES
`
`VFIP GENERATION
`
`37°
`
`INSERT VRP IN DPB AND SET
`FRAM E_NUM./POC OR
`LONG_TERM_FRAME_IDX OR
`VRP_ID
`
`REFERENCE LIST
`CONSTRUCTION
`WITHOUT VRP
`
`INCLUDE VRP IN REFERENCE
`
`warn; LOW-LEVEL
`SYNTAXES INTO
`BITSTHEAM
`
`365
`
`LIST REORDEFIING
`
`
`
`m 399
`
`WRITE HIGH-LEVEL SYNTAXES
`
`INTO BITSTREAM
`
`350
`
`FIG. 3
`
`
`
`
`
`TO VRP BY
`
`
`ENCODE CURRENT
`PICTURE, REFER
`
`355
`
`FIEFJDX IF VFIP
`PRESENT
`
`REMOVE VFIP
`FROM DPB
`
`36-0
`
`INCLUDE VRP IN DEFAULT
`REFERENCE LIST CONSTRUCTION
`
`

`

`US. Patent
`
`Jan. 29, 2013
`
`Sheet 3 of 5
`
`US 8,363,724 32
`
`m 405
`
`r 400
`
`READ HIGH -LEVEL SYNTAXES FROM BITSTREAM.
`INCLUDING VRP_PRESENT_FLAG, NUM_VRPS
`AND OTHER VRP PARAMETER SYNTAXES
`
`410
`
`VRP_PRESENT_FLAG
`==17
`
`YES
`
`DECODE VRP PARAMETERS
`
`VRP GENERATION
`
`415
`
`N0
`
`
`
`
`
`
`INSERT VRP IN DPB AND SET
`FRAME_NUM/POC OR
`
`LONG_TERM_,FRAME_IDX 0R
`
`VRP'ID
`REFERENCE LIST
`
`CONSTRUCTION
`
`WTTHOUT VRP
`
`
`INCLUDE VRP IN DEFAULT
`
`REFERENCE LIST CONSTRUCTION
`
`4‘50
`
`INCLUDE VRP IN REFERENCE
`
`LIST REORDERING
`
`
`
`READ LOW-LEVEL SYNTAXES
`FROM BITSTREAM
`
`445
`
`DECODE CURRENT PICTURE.
`REFER TO VRP BY REFJDX F
`VRP PRESENT
`
`450
`
`m 499
`
`FIG- 4
`
`

`

`US. Patent
`
`Jan. 29, 2013
`
`Sheet 4 Of 5
`
`US 8,363,724 32
`
`m 505
`
`[— 500
`
`SET VRPHPRESENLFLAG = 0
`
`(J1 _L0-‘|
`
`510
`
`IS VRP ENABLED?
`
`NO
`
`YES
`
`SET VRP_PRESENT_FLAG = 1
`
`520
`
`SET NUM_VRPS AND VRP
`PARAMETER SYNTAXES
`
`525
`
`VRP GENERATION
`
`530
`
`STORE VRP IN LOCAL MEMORY
`AND GET VRPJD
`
`535
`
`REFERENCE LIST CONSTRUCTION
`
`WITHOUT VRP
`
`WRITE HIGH-LEVEL SYNTAXES
`INTO BITSTREAM
`
`ENCODE CURRENT PICTURE,
`REFER TO VRP BY VRP_IO IF VRP
`PRESENT
`
`540
`
`545
`
`550
`
`55° FIG. 5
`
`RELEASE MEMORY FOR VRP
`
`555
`
`WRITE LOW-LEVEL SYNTAXES
`INTO BITSTREAM
`
`

`

`US. Patent
`
`Jan. 29, 2013
`
`Sheet 5 of 5
`
`US 8,363,724 32
`
`
`
`605
`
`(500
`
`READ HIGH-LEVEL SYNTAXES FROM BITSTREAM,
`INCLUDING VRP_PRESENT_FLAG, NUM‘VRPS
`AND OTHER VRP PARAMETER SYNTAXES
`
`610
`
`
`
`
`
`VRP_PRESENT_FLAG ==1 .
`
`
`
`
`
`YES
`
`DECODE VRP PARAMETERS
`
`VRP GENERATION
`
`STORE VRP IN LOCAL MEMORY
`
`AND SET VRP‘ID
`
`
`
`REFERENCE LIST CONSTRUCTION
`
`WITHOUT VRP
`
`READ LOW- LEVEL SYNTAXES
`FROM BITSTREAM
`
`550
`
`660
`
`
`DECODE CURRENT PICTURE,
`REFER TO VRP BY VRP_ID IF VRP
`PRESENT
`
`
`
`
`
`
`
`
`
`RELEASE MEMORY FOR VRP
`
`665
`
`REMOVE VRP FROM DPB
`
`699
`
`670
`
`FIG. 6
`
`

`

`US 8.363324 B2
`
`1
`METHODS AND APPARATUS USING
`VIRTUAL REFERENCE PICTURES
`
`CROSS-REFERENC E TO RELATED
`APPLICATIONS
`
`This application claims the benefit. under 35 U .S.(T. §365
`of International Application PCTfU82007t015719. filed Jul .
`10. 2007 which was published in accordance with PCT
`Article 21(2) on Jan. 17. 2008 in English and which claims the
`benefit ol'U.S. provisional patent application No. 601830.195
`filed Jnl. II. 2006.
`
`10
`
`'I‘IiCIINICAl. FIELD
`
`The present principles relate generally to video encoding
`and decoding and. more particularly. to methods and appara-
`ttts using virtual reference pictures.
`
`BACKGROUND
`
`In existing video compression systems and standards such
`as the international Organization for Standardizatiotv'lnter-
`national Electrotechnical Commission (lSOr‘IEC) Moving
`Picture Experts Group-4 (MPEG-4) Part lOAdvanced Video
`Coding (AVC) standardflntemationa] 'Ielecomluunication
`Union. Teleconimunicalion Sector (ITU-T) 11.264 recom-
`mendation (hereinafter the “MPEGJI AVC standard“), previ—
`ously reconstntctedfdecoded pictures are used as references
`for fitture pictures. while motion estimation and compensa-
`tion is employed in order to compensate for any motion
`activity between those pictures. In Multi-view Video Coding
`(MVC). reconstructedfdecoded pictures from neighboring
`views can also be used as source of prediction. where dispar-
`ity estimation and compensation is involved.
`There are cases where prediction can be enhanced if certain
`processing is applied to the decoded pictures such as. for
`example. weighted prediction in the MPEG-4 AVC standard
`and a prior art technique of adaptive reference generation. By
`processing decoded pictures. the quality ofprediction signals
`could be enhanced and therefore the coding efiiciency can be
`improved. In the case ofweighted prediction. the global illu-
`mination mismatch is addressed between the current picture
`and the decoded pictttres. In the case of adaptive reference
`generation. different types of noises can be suppressed by
`adaptive reference generation in order to provide more related
`reference signals.
`
`SUMMARY
`
`These and other drawbacks and disadvantages of the prior
`art are addressed by the present principles, which are directed
`to methods and apparatus using virtual reference pictures.
`According to an aspect of the present principles. there is
`provided an apparatus. The apparatus includes an encoder for
`encoding at least one picture. using at least one virtual refer-
`ence picture. to form a resultant bitstreani. The at least one
`virtual
`reference picture is different
`than a previously
`decoded picture.
`According to another aSpeet of the present principles. there
`is provided a method. The method includes encoding at least
`one picture. using at least one virtual reference picture. to
`form a resultant bitstream. The at least one virtual reference
`
`picture is difl'erent than a previously decoded picture.
`According to yet another aspect of the present principles.
`there is provided an apparatus. The apparatus includes a
`decoder for decoding at least one picture. using at least one
`
`3t]
`
`4t]
`
`45
`
`50
`
`55
`
`60
`
`65
`
`2
`
`virtual reference picture. from a bitstream. The at least one
`virtual
`reference picture is different
`than a previously
`decoded picture.
`According to a further aspect of the present principles.
`there is provided a method. The method includes decoding at
`least one picture. ttsing at least one virtual reference picture.
`from a bitstt‘eam. The at least one virtual reference picture is
`different than a previously decoded picture.
`These and other aspects. features and advantages of the
`present principles will become apparent from the following
`detailed description of exemplary embodiments. which is to
`be read in connection with the accompanying drawings.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`The present principles may be better understood in accor-
`dance with the following exemplary l'lgures. in which:
`FIG. 1 is a block diagram for an exemplary video encoder
`supporting virtual reference pictures to which the present
`principles may be applied. in accordance with an embodiment
`ofthe present principles;
`FIG. 2 is a block diagram for an exemplary video decoder
`supporting virtual reference pictures to which the present
`principles may be applied. in accordance with an embodiment
`ofthe present principles;
`FIG. 3 is a flow diagram for an exemplary method for
`encoding video content using Virtual Relbrcnce Picture
`(VPR) trianagement in a Decoded Picture Buffer (DPB). in
`accordance with an embodiment of the present principles:
`FIG. 4 is a flow diagram for an exemplary method for
`decoding video content using Virtual Reference Picture
`NPR) management in a Decoded Picture Buffer (DPB). in
`accordance with an embodiment of the present principles:
`FIG. 5 is a flow diagram for an exemplary method for
`encoding video content using Virtual Reference Picture
`(VPR) management in local memory. in accordance with an
`embodiment of the present principles; and
`FIG. 6 is a flow diagram for an exemplary method for
`decoding video content
`rising Virtual Reference Picture
`(VPR) management in a Decoded Picture Buffer (DPB). in
`accordance with an embodiment of the present principles.
`
`DETAILED DESCRIPTION
`
`The present principles are directed to methods and appa-
`ratus using vinual reference pictures.
`The present description illustrates the present principles. It
`will thus be appreciated that those skilled in the art will be
`able to devise various arrangements that. although not explic—
`itly described or shown herein. embody the present principles
`and are included within its spirit and scope.
`All examples and conditional language recited herein are
`intended for pedagogical purposes to aid the reader in tinder-
`standing the present principles and the concepts contributed
`by the inventor(s] to furthering the art. and are to be construed
`as being without
`limitation to such specifically recited
`examples and conditions.
`Moreover.
`all
`statements herein reciting principles,
`aspects. and embodiments ofthe present principles. as well as
`specific examples thereof. are intended to encompass both
`structural and functional equivalents thereof. Additionally. it
`is intended that such equivalents include both currently
`known equivalents as well as equivalents developed in the
`future. i.e.. any elements developed that perform the same
`function. regardless of structure.
`Thus. for example. it will be appreciated by those skilled in
`the art that the block diagrams presented herein represent
`
`

`

`3
`
`4
`
`US 8,363,T24 B2
`
`conceptual views of illustrative circuitry embodying the
`present principles. Similarly, it will be appreciated that any
`flow charts.
`flow diagrams.
`state transition diagrams.
`pseudocode. and the like represent various processes which
`ntay be substantial ly represented in computer readable media
`and so eatecuted by a computer or processor, whether or not
`such computer or processor is explicitly shown.
`The functions of the various elements shown in the figures
`may be provided through the use of dedicated hardware as
`well as hardware capable ofexecuting software in association
`with appropriate software. When provided by a processor. the
`functions may be provided by a single dedicated processor. by
`a single shared processor, or by a plurality of individual
`processors. some ofwhich may be shared. Moreover, explicit
`use of the term “processor“ or “controller” should not be
`construed to refer exclusively to hardware capable of execut-
`ing sollware. and may implicitly include. without limitation.
`digital
`signal processor
`[“DSP") hardware.
`read-only
`memory (“ROM”)
`for storing software.
`random access
`memory (“RAM”), and non-volatile storage.
`Other hardware, conventional andtor custom, may also be
`included. Similarly. any switches shown in the figures are
`conceptual only. Their function may be carried out through
`the operation of program logic, through dedicated logic.
`through the interaction of program control and dedicated
`logic. or even manually. the particular tccluiique being select-
`able by the implementer as more specifically understood from
`the context.
`
`In the claims hereof. any element expressed as a means for
`performing a specified function is intended to encompaSs any
`way of performing that function including, for example. a] a
`combination ofcircuit elements that performs that function or
`b] software in any form.
`including.
`therefore. firmware,
`microcode or the like. combined with appropriate circuitry
`for executing that software to perform the function. The
`present principles as defined by such claims reside in the fact
`that the functionalities provided by the various recited means
`are combined and brought together in the manner which the
`claims call for. It is thus regarded that any means that can
`provide those functionalities are equivalent to those shown
`herein.
`Reference in the specification to “one embodiment" or “an
`embodiment" of the present principles means that a particular
`feature, structure. characteristic. and so forth described in
`connection with the embodiment is included in at least one
`
`embodiment of the present principles. Thus. the appearances
`of the phrase “in one embodiment" or “in an embodiment”
`appearing in various places throughout the specification are
`not necessarily all referring to the same embodiment.
`As used herein. “high level syntax" refers to syntax present
`in the bitstream that resides hierarchically above the macrob«
`lock layer. For example, high level syntax, as used herein,
`may refer to. but is not limited to. syntax at the slice header
`level. Supplemental Enhancement Information (SE1) level.
`picture parameter set level. sequence parameter set level and
`NAL unit header level.
`
`Turning to FIG. 1_. an exemplary video encoder supporting
`virtual reference pictures to which the present principles may
`be applied is indicated generally by the reference numeral
`100.
`
`An input to the video encoder 100 is connected in signal
`communication with a non-inverting input ofa combiner 1 10
`and a first input of a motion estimator (ME) 180. The output
`of the combiner 110 is connected in signal communication
`with an input ofa discrete cosine transformer 12!). The output
`of the discrete cosine transformer 120 is connected in signal
`communication with an input of a quantizer 130. An output of
`
`10
`
`3t]
`
`4t]
`
`45
`
`50
`
`55
`
`60
`
`65
`
`the quantizer 130 is connected in signal conununication with
`an input of a variable length coder (VII?) 140 and an input of
`an inverse quantizer 150. An output of the variable length
`coder (LVC) 140 is available as an output of the encoder 100.
`An output of the inverse quantizer 150 is connected in
`signal communication with an input of all inverse discrete
`cosine transformer 155. An output ofthe inverse cosine traits—
`former 155 is connected in signal communication with a first
`non-inverting input of a combiner 165. An output of the
`combiner 165 is connected in signal communication with an
`input of a loop filer 160. An output of the loop filer 160 is
`connected in signal communication with an input of a
`decoded picture buffer 175. A first output of the decoded
`picture buffer 175 is connected in signal communication with
`an input ofa virtual reierence pictures buffer 1'70.
`An output of switch 195 is connected in signal communi-
`cation with a second input of the motion estimator 188 and a
`second input ofa motion compensator 190.
`An input of the switch 195 is connected in signal commu-
`nication with a second output of the decoded picture buffer
`175 or an output of the virtual reference pictures buffer 170.
`An output of the motion estimator 180 is connected in signal
`communication with a first input of the motion compensator
`190. An output ofthe motion compensator 190 is connected in
`signal connnunication with a second non-inverting input of
`the combiner 165 and an inverting input 0 l‘tlte combiner 110.
`Turning to FIG. 2. an exemplary video decoder supporting
`virtual reference pictures to which the present principles may
`be applied is indicated generally by the reference numeral
`200.
`
`The video decoder 200 includes a variable length decoder
`210 for receiving a bitstream. A first output of the variable
`length decoder 210 is connected in signal communication
`with a first input of a motion compensator 260. A second
`output of the variable length decoder 210 is connected in
`signal communication with an input ofan inverse quantizer
`220. An output of the inverse quantizer 220 is connected in
`signal communication with an input of an inverse discrete
`cosine transformer 23!). An output of the discrete cosine
`transformer is connected in signal communication with a first
`non-inverting input of a combiner 240. An output of the
`combiner 240 is connected in signal communication with an
`input ofa loop filter 290. An output of the loop filter 290 is
`connected in signal connnunication with an input of a
`decoded picture buffer 250. A first output of the decoded
`picture buffer 250 is connected in signal communication with
`an input of a virtual reference pictures buffer 255. A second
`output of the decoded picture buffer 250 is available as an
`output ofthe decoder 200.
`An output of a switch 265 is connected in signal continu—
`nication with a second input of the motion compensator 260.
`An output of the motion compensator 260 is connected in
`signal connnunication with a second non-inverting input of
`the combiner 240.
`
`An input of the switch 265 is connected in signal commu-
`nication with an output ofthe virtual reference pictures buffer
`255 or the second output of the decoded picture buffer 250.
`The decoded picture butter 250 includes a long—term
`memory portion 250A and a short-term memory portion
`25013.
`
`In accordance with the present principles. methods and
`apparatus are provided for the management of virtual refer-
`ence pictures (VRP) in a video encoder andfor video decoder.
`In an embodiment. virtual reference pictures are created from
`already decoded pictures. and are not required for display
`purposes. In an embodiment, virtual reference pictures can be
`utilized for prediction. but are not required for display pur-
`
`

`

`5
`
`6
`
`US 8,363,?24 B2
`
`poses. In an embodiment, the use ofVRP in a video encoder
`andfor decoder involves one or more of the following ele-
`ments for implementation: syntax support: a storagefmemory
`model for VRP: and a normative procedure of including VRP
`in the prediction loop.
`By Way of illustration and not limitation, some potential
`applications ofthe proposed management methods and appa—
`ratus include filtered reference prediction. reference warping.
`view interpolated prediction in Multi-view Video Coding
`(MVC) and other methods involving generated virtual refer-
`ences. Depending on the type of processing applied on
`decoded pictures. it might be impossible or inconvenient to
`obtain a processed reference signal on a local basis. Infinite
`Impulse Response (IIR) filtering and picture warping are two
`such examples. where it is better to apply the processing on a
`frame basis instead of on a block basis. Another example is
`view synthesis prediction. In mttlti-view video coding. the
`redtmdancy between adjacent camera views can be exploited
`through view synthesis. A view synthesized picture can be
`created by warping and blending neighboring view pictures
`using camera parameters andfor depth information. The syn—
`thesized picture can be used as a reference picture for the
`prediction of the current view picture, which could provide
`better prediction sources comparing to disparity compensated
`prediction.
`It should be noted that the concept of virtttal
`reference pictures [VRP] can also be applied to scenarios
`where the processing can be done locally such as.
`for
`example, in the cases of sample weighting or Finite Impulse
`Response (FIR) filtering.
`It is to be appreciated that while the concept of adaptive
`reference generation has been previously proposed and
`weighted prediction as a special case of filtered reference
`picture has been adopted in the MPEG-4 AVC‘ standard no
`known prior art exists for managing generated virtual refer-
`ence pictures. In previously proposed methods. the manage—
`ment of generated references can be handled by simply pro—
`cessingffiltering the encodedfdecoded pictures in the process
`of compensation. In other words. a complete generated ref-
`erence picture is not needed for previous approaches. There is
`no prior art in how to manage fully generated reference pic-
`lures.
`
`Embodiments incorporating the present principles may
`include one or more of the following aspects.
`further
`described in detail herein below: signaling of virtual refer"
`ence pictures; memory management for virtual reference pic-
`tures; and usage of virtually generated frames in prediction
`Signaling ofVErtual Reference Pictures
`It is desirable to support a flexible way to signal generated
`reference pictures which subsequently can be used in the
`management of those pictures in terms of storage and refer—
`ences. In order to do so, in one embodiment, we propose the
`following syntaxes.
`In the embodiment, we signal at the sequence level whether
`VRP is enabled. This can be implemented. e.g.. by introduc-
`ing a high-level syntax vrp_present_fiag.
`In the embodiment. we signal at the picture level how many
`virtual references are present and the method to create them.
`For example. in one implementation of the embodiment. the
`use and the parameters required by generation of virtual ref-
`erence pictures will be present in the syntax for each coded
`picture. For a possible MPEG-4 AVC standard based imple-
`mentation. slice header syntaxes num_vrps could be intro-
`duced. Each virtual reference picture will be then assigned a
`vrp_id. which corresponds to the order in which virtual ref—
`erence picture parameter syntaxes appear in the slice header.
`The parameters for the generation of each virtual reference
`picture will depend on the exact method of generation. In the
`
`10
`
`3t]
`
`4t]
`
`45
`
`50
`
`55
`
`60
`
`65
`
`example of view synthesized prediction in Multinview Video
`Coding, camera parameters and depth information may be
`included. In the example of a warped reference in regular
`video coding, warping parameters (such as. for example,
`hoinograph matrix elements) may be signaled. A more con-
`crete example in the case of adaptive reference filtering is to
`signal FIR filter coefficients, where n2 coefficients will need
`to be quantized and transmitted if non-separable two-dimen-
`sional filters are to be applied.
`Memory Management for Virtual Reference Pictures
`Since the virtual reference pictures need to be generated
`and stored at both the encoder and decoder. the associated
`storage memory should be considered.
`There are several approaches to providing a memory man—
`agement tnodel for virtual reference pictures: [1) in a first
`approach. store generated virtual reference pictures in the
`decoded picture buffer: and (2) in a second approach. store
`virtually generated frames in a temporary generated picture
`bufl‘er which is only valid during the encoding/decoding of
`the current frame.
`With respect to providing a memory management model in
`accordance with the first approach mentioned above. since
`virtual reference pictures are only needed for the encoding(
`decoding ofthe current picture, decoded picture buffer inser-
`tion and deletion processes should be properly defined. in one
`possible implementation. generated reference pictures will be
`inserted in the decoded picture buffer before reference lists
`are constructed. and will be removed right alter the encoding!
`decoding of the current frame is finished.
`When virtual reference pictures are stored in the decoded
`picture buffer. they will need to be differentiated from non-
`virtttal decoded pictures, there are several options how this
`can be done in an MPEG-4 AVC-bascd implementation.
`Some exemplary options for differentiating virtual reference
`pictures stored in the decoded picture buffer from non—virtual
`reference pictures including. for example: (I) store virtual
`reference pictures as short-term reference pictures and use
`unused frame_numfpicture_ordcr_count: (2) store virtual
`reference pictures as [orig-term reference pictures and use
`unused longterm_id’s in the long term memory; and (3) since
`a virtual
`reference picture is different
`from previously
`decoded pictures in nature, dedicated memory slots can be
`allocated in the decoded picture buffer lor the storage of
`virtual reference pictures. In that VRP memory. virtual refer~
`ence pictures will be identified by their vrp_id_. which is
`unique for each virtual reference picture.
`With respect to providing a memory management model in
`accordance with the second approach mentioned above, by
`sore virtually generated frames in a temporary generated
`picture buffer which is only valid during the encodingfdecod—
`ing of the current frame. this temporary generated picture
`buffer will be able to store all virtually generated pictures.
`Virtual reference pictures will be identified by their vrp_id.
`which is unique for each virtual reference picture.
`Usage of Virtually Generated Frames in Prediction
`In orderto allow virtually generated pictures to be referable
`for the prediction of the current picture. we need to allow a
`flexible way of including vinual reference pictures in the
`prediction process. [it accordance with the present principles,
`two exemplary embodiments are proposed for accomplishing
`this. although it is to be appreciated that given the teachings of
`the present principles provided herein. one ofordi nary skill in
`this and related arts will contemplate these and other ways to
`include virtual reference pictures in the prediction process,
`while maintaining the scope of the present principles.
`In one embodiment for including virtual reference pictures
`in the prediction process, we refer virtual reference pictures
`
`

`

`7
`
`8
`
`US 8,363,?24 B2
`
`directly. In this method. reference_vrp_id syntax is used to
`signal which virtual reference picture is used in the prediction
`process. By doing this. the conventional reference lists will
`stay untouched. and the syntax reticrence_vrp_id is only
`present when VRP is involved in the prediction.
`In another embodiment for including virtual reference pic-
`tures in the prediction process. we refer virtual reference
`pictures through reference lists. Accordingly. the default ref-
`erence list marking process may be modified to include vir-
`tual reference pictures if they are present. and we can use
`general RPLR commands to support construction of refer-
`ence lists or V’RP-specilic Reference Picture List Reordering
`(RPI..R) commands can be defined to support construction of
`reference lists. Specifically. if VRP—dedicated slots are allo—
`cated in the decoded picture buffer or a temporary VRP buffer
`is used, a VRP-specific RPIR command will be introduced to
`place the virtual
`reference picture indicated by refer-
`ence_vrp_id into the reference list under construction. More-
`over. we can introduce additional syntax elements in the slice
`header that will signal for each reference index whether it
`refers to a Virtual Reference Picture or not. If it does. then we
`can signal the same using the associated reference_vrp_id. In
`this method. we do not need to introduce a new VRP-speciflc
`liPl..R command. because a general RPIR conmiand can be
`used to allow a different reference index to refer to the same
`
`10
`
`decoded picture in the Decoded Picture Buffer. Then, through
`the reference_vrp_id, we can signal iflhis reference picture is
`an original decoded reference picture or if it is a virtual
`reference picture generated from the decoded reference pic-
`ture.
`
`3t]
`
`Turning to FIG. 3. an exemplary method for encoding
`video content using Virtual Reference Picture (V’PR) man-
`agement in a Decoded Picture Buffer [DPB] is indicated
`generally by the reference numeral 300. The method 300
`includes a start block 305 that passes control to a function
`block 310. The filnction block 310 sets vrp_present_flag
`equal to zero. and passes control to a decision block 315. The
`decision block 315 determines whether or not VRP is
`
`enabled. If so. then control is passed to a function block 320.
`Otherwise. control is passed to a function block 370.
`The function block 320 sets vrp_present_flag equal to one.
`and passes control to a function block 325. The function block
`325 sets num_vrps and VRP parameter syntaxes. and passes
`control to a function block 330. The function block 330 per-
`forms VRP generation to generate one or more VRl’s (here-
`inafter “VRP"). and passes control to a ftutction block 335.
`The function block 335 inserts the VRP in the decoded picture
`bull‘er
`(DPB).
`sets
`fraine_nurru’Picture Order Count or
`long_terninframeridx or vrp_id= and passes control to a func—
`tion block 340. The function block 340 includes the VRP in
`
`reference list construction. and passes control to a function
`block 345. The function block 345 includes the VRP in ref-
`
`erence list reordering, and passes control to a function block
`350. The function block 350 writes high-level syntaxes into
`the bitstream. and passes control to a function block 355. The
`function block 355 encodes the current picture. refers to the
`VRP by ref_idx if the VRP is present, and passes control to a
`function block 360. The function block 360 retrieves theVRP
`from the OPE. and passes control to a function block 3 65. The
`function block 365 writes the low-level syntaxes into the
`bilstrcant. and passes control to an end block 399.
`'lhe function block 3'70 performs reference list construc-
`tion with the VRP. and passes control to the function block
`350.
`
`Turning to FIG. 4, an exemplary method for decoding
`video content using Virtual Reference Picture (VPR) man-
`agement in a Decoded Picture Buffer (DPB) is indicated
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`generally by the reference numeral 400. The method 400
`includes a start block 405 that passes control to a function
`block 410. The function block 410 reads high-level syntaxes
`from the bitstream including.
`e.g..
`vrp_present_flag_.
`nuni_vrps. and other V'RP parameter syntaxes, and passes
`conLrol to a decision block 415. The decision block 415 deter-
`mines whether or not vrp_present_flag is equal to one. If so,
`then control is passed to a function block 420. Otherwise.
`control is passed to a function block 460.
`The functi