(19)
`
`(12)
`
`*EP002278816B1*
`EP 2 278 816 B1
`
`(11)
`
`EUROPEAN PATENT SPECIFICATION
`
`(45) Date of publication and mention
`of the grant of the patent:
`24.04.2013 Bulletin 2013/17
`
`(21) Application number: 10179945.0
`
`(22) Date of filing: 07.07.2003
`
`(51) Int Cl.:
`H04N 7/26 (2006.01)
`H04N 21/44 (2011.01)
`H04N 7/24 (2011.01)
`H04N 7/36 (2006.01)
`
`H04N 7/50 (2006.01)
`H04N 21/234 (2011.01)
`H04N 7/46 (2006.01)
`
`(54) Post-decoder buffer management for an H.264-SVC MPEG bitstream.
`
`Post Dekodierer Puffer Verfahren für einen H.264-SVC MPEG Bitstrom.
`
`Méthode de gestion d’ un buffer post-décodeur pour un flux MPEG H.264-SVC.
`
`(84) Designated Contracting States:
`AT BE BG CH CY CZ DE DK EE ES FI FR GB GR
`HU IE IT LI LU MC NL PT RO SE SI SK TR
`
`(74) Representative: Eisenführ, Speiser & Partner
`Postfach 10 60 78
`28060 Bremen (DE)
`
`(30) Priority: 11.07.2002 JP 2002202781
`17.07.2002 JP 2002207681
`14.01.2003 JP 2003006198
`
`(56) References cited:
`EP-A2- 0 729 276
`JP-A- 10 229 563
`
`EP-A2- 1 011 270
`US-A- 5 909 224
`
`(43) Date of publication of application:
`26.01.2011 Bulletin 2011/04
`
`(62) Document number(s) of the earlier application(s) in
`accordance with Art. 76 EPC:
`09152597.2 / 2 053 863
`06122279.0 / 1 742 479
`03741225.1 / 1 406 451
`
`(73) Proprietor: Panasonic Corporation
`Kadoma-shi
`Osaka 571-8501 (JP)
`
`(72) Inventors:
`• Lim, Chong Soon
`Osaka 540-6207 (JP)
`• Foo, Teck Wee
`Osaka 540-6207 (JP)
`• Shen, Sheng Mei
`Osaka 540-6207 (JP)
`• Kadono, Shinya
`Osaka 540-6207 (JP)
`• Kondo, Satoshi
`Osaka 540-6207 (JP)
`• Hagai, Makoto
`Osaka 540-6207 (JP)
`• Abe, Kiyofumi
`Osaka 540-6207 (JP)
`
`• SULLIVAN G: "DRAFT FOR H.263++ ANNEXES U,
`V, AND W TO RECOMMENDATION H.263", ITU-T
`H.263++ DRAFT FOR H.263++ ANNEXES U, V AND
`W TORECOMMENDATION H.263, XX, XX, 1
`November 2000 (2000-11-01), pages 1-46,
`XP000986538,
`• MISKA M HANNUKSELA ET AL: "Random Access
`and Time Information", 2. JVT MEETING;
`29-01-2002 - 01-02-2002; GENEVA, CH; (JOINT
`VIDEO TEAMOF ISO/IEC JTC1/SC29/WG11 AND
`ITU-T SG.16 ),, no. JVT-B109, 1 February 2002
`(2002-02-01) , XP030005105, ISSN: 0000-0443
`• MISKA M HANNUKSELA ET AL: "Random Access
`and Time Information: Appendix A. Proposed
`Changes to Joint Model", 2. JVT MEETING;
`29-01-2002 - 01-02-2002; GENEVA, CH; (JOINT
`VIDEO TEAMOF ISO/IEC JTC1/SC29/WG11 AND
`ITU-T SG.16 ),, no. JVT-B109_Appendix_A, 1
`February 2002 (2002-02-01), XP030005106, ISSN:
`0000-0443
`• SCHWARZ H ET AL: "Overview of the Scalable
`Video Coding Extension of the H.264/AVC
`Standard", IEEE TRANSACTIONS ON CIRCUITS
`AND SYSTEMS FOR VIDEO TECHNOLOGY, IEEE
`SERVICE CENTER, PISCATAWAY, NJ, US, vol.
`17, no. 9, 1 September 2007 (2007-09-01), pages
`1103-1120, XP011193019, ISSN: 1051-8215, DOI:
`10.1109/TCSVT.2007.905532
`
`Note: Within nine months of the publication of the mention of the grant of the European patent in the European Patent
`Bulletin, any person may give notice to the European Patent Office of opposition to that patent, in accordance with the
`Implementing Regulations. Notice of opposition shall not be deemed to have been filed until the opposition fee has been
`paid. (Art. 99(1) European Patent Convention).
`
`Printed by Jouve, 75001 PARIS (FR)
`
`EP2 278 816B1
`
`UNIFIED 1006
`
`

`

`EP 2 278 816 B1
`
`Description
`
`Technical Field
`
`5
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`[0001] The present invention relates to a picture decoding method for decoding correctly the encoded picture and
`displaying it.
`
`Background Art
`
`[0002] Recently, with an arrival of the age of multimedia which handles integrally audio, video and pixel values of
`others, existing information media, i.e., newspapers, journals, TVs, radios and telephones and other means through
`which information is conveyed to people, has come under the scope of multimedia. Generally speaking, multimedia
`refers to something that is represented by associating not only with characters but also with graphics, audio and especially
`pictures and the like together. However, In order to include the aforementioned existing information media in the scope
`of multimedia, It appears as a prerequisite to represent such information in digital form.
`[0003] However, when calculating the amount of information contained In each of the aforementioned information
`media as the amount of digital Information, the information amount per character requires 1(cid:97)2 bytes whereas the audio
`requires more than 64 Kbits (telephone quality) per second and when it comes to the moving picture, it requires, more
`than 100Mbits (present television reception quality) per second. Therefore, it is not realistic to handle the vast information
`directly in the digital format via the information media mentioned above. For example, a videophone has already been
`put into practical use via Integrated Services Digital Network (ISDN) with a transmission rate of 64 Kbit/s (cid:97) 1.5 Mbit/s,
`however, it is not practical to transmit video captured on the TV screen or shot by a TV camera. This therefore requires
`information compression techniques, and for instance, in the case of the videophone, video compression techniques
`compliant with H.261 and H.263 standards internationally standardized by ITU-T (International Telecommunication
`Union-Telecommunication Standardization Sector) are employed. According to information compression techniques
`compliant with the MPEG-1 standard, picture information as well as music information can be stored in an ordinary music
`CD (Compact Disc).
`[0004] Here, MPEG (Moving Picture Experts Group) is an international standard for compression of moving picture
`signals and MPEG-1 is a standard that compresses video signals down to 1.5 Mbit/s, that is, to compress information
`of TV signals approximately down to a hundredth. The transmission rate within the scope of the MPEG-1 standard is
`limited primarily to about 1.5 Mbit/s, therefore, MPEG-2 which was standardized with the view to meet the requirements
`of high-quality picture allows data transmission of moving picture signals at a rate of 2(cid:97)15 Mbit/s. In the present circum-
`stances, a working group (ISO/IEC JTC1/SC29/WG11) in the charge of the standardization of the MPEG-1 and the
`MPEG-2 has achieved a compression rate which goes beyond what the MPEG-1 and the MPEG-2 have achieved,
`realized encoding/decoding operations on a per-object basis and standardized MPEG-4 in order to realize a new function
`required by the era of multi media. In the process of the standardization of the MPEG-4, the standardization of encoding
`method for a low bit rate was aimed, however, the aim is presently extended to a more versatile encoding of moving
`pictures at a high bit rate including interlaced pictures.
`[0005] Recently, a new picture encoding as a next generation encoding of the MPEG-4 called SVC is under the process
`of the standardization jointly worked by the ITU-T and the ISO/IEC.
`[0006] Fig. 24 is a diagram showing a prediction structure, a decoding order and a display order of pictures. "Picture"
`is a term indicating either a frame or a field and the term "picture" here is used in stead of frame or field in the present
`specification. The hatched pictures in Fig. 24 present the pictures to be stored in the memory for reference when other
`pictures are encoded/decoded. 10 is an intra coded picture and P3, P6 and P9 are predictive coded pictures (P-picture).
`The predictive encoding in the scheme of the JVT standard differs from that of the conventional MPEG-1/2/4. An arbitrary
`picture is selected out of a plurality of encoded pictures as a reference picture and a predictive image can be generated
`from the reference picture. For example, a picture P9 may select an arbitrary picture out of three pictures of I0, P3 and
`P6 and generate a predictive image using the selected picture. Consequently, it heightens a possibility to select the
`more applicable predictive image than the conventional case of applying MPEG-1/2/4 and thereby improves a compres-
`sion rate. B1, B2, B4, B5, B7 and B8 are bi-directionally predictive coded pictures (B-picture), differing from inter-picture
`prediction, wherein a plurality of pictures (two pictures) are selected and a predictive image is generated using the
`selected pictures and then encoded. It is especially known that the accuracy of the predictive image can be greatly
`improved and so can be the compression rate by performing interpolation prediction using an average value of two
`pictures temporally previous and subsequent for generating a predictive image. The marks of "I" for an intra coded
`picture, "P" for a predictive coded picture and "B" for a bi-directionally predictive coded picture are used in order to
`differentiate encoding method of each picture.
`[0007]
`In order to refer to the temporally previous and subsequent pictures for the B-pictures, the temporally previous
`pictures shall be coded/decoded at first. This is called reordering of pictures and often takes place in the conventional
`
`2
`
`

`

`EP 2 278 816 B1
`
`MPEG-1/2/4. Therefore, in contrast with an encoding order (Stream Order), an order of displaying the pictures which
`are decoded (Display Order) is reordered as shown in Fig. 24 showing a prediction structure, a decoding order and a
`display order of pictures. B-pictures in the example of Fig. 24 are displayed at the moment when the stream is decoded,
`therefore, there is no need to store them when they are not referred to by other pictures. However, I-pictures and P-
`pictures have to be stored in a memory since they are displayed after being decoded when the decoding of the following
`B-picture is terminated.
`[0008] The terms and the meanings of the hatched pictures in the diagram showing the prediction structure, the
`decoding order and the display order of the pictures are the same as those used in Fig. 24.
`[0009] Fig. 26 is a block diagram showing a picture encoding apparatus for realizing a conventional picture encoding
`method. The following illustrates an operation of the picture encoding apparatus for realizing the conventional picture
`encoding method in Fig. 26.
`[0010] A picture structure determination unit PicStruct determines an encoding type (I-picture, P-picture and B-picture)
`for each picture, notifies a reference picture control unit RefPicCtrl of the encoding type and the pictures that can be
`referred to in the encoding and informs also a reordering unit ReOrder of the encoding order of the pictures. The reordering
`unit ReOrder reorders the order of an input picture PicIn into an encoding order and outputs the reordered pictures to
`a motion estimation unit ME and a subtraction unit Sub. The motion estimation unit ME refers to the reference pictures
`stored in a picture memory PicMem1, determines an applicable reference picture and detects a motion vector indicating
`a pixel position of the reference picture and sends them to a variable length coding unit VLC, the picture memory PicMem1
`and a motion compensation unit MC. The picture memory PicMem1 outputs the pixels of the reference picture according
`to the motion vector MV to the motion compensation unit MC whereas the motion compensation unit MC generates a
`predictive image using the pixels in the reference picture gained from the picture memory PicMem1 and the motion
`vector MV.
`[0011] The subtraction unit Sub calculates a difference between the picture reordered by the reordering unit ReOrder
`and the predictive image. The difference is converted to frequency coefficients by an orthogonal transformation unit T
`and then the frequency coefficients are quantized by the quantization unit Q and outputted as quantized values Coef.
`[0012] An inverse quantization unit IQ inverse quantizes the quantized values Coef and restores them as frequency
`coefficients. The inverse orthogonal transformation unit IT performs inverse frequency conversion for the frequency
`coefficients to be outputted as pixel differential values. An addition unit Add adds the predictive image to the pixel
`differential values and obtains a decoded picture.
`[0013] The reference picture control unit RefPicCtrl, according to the encoding type of the picture, judges whether or
`not the decoded picture is to be stored in the picture memory PicMem1 to be referred to as a reference picture and
`whether or not the decoded picture is to be removed from the picture memory PicMem1 (no longer referred to as a
`reference picture) and notifies of the operation using a memory control command MMCO.
`[0014] A switch SW is turned ON when the memory control command MMCO ordered a storage and thereby the
`decoded picture is stored in the picture memory PicMem1 as a reference picture. The picture memory PicMem1 releases
`the area where the decoded picture is stored so that other decoded pictures can be stored when the picture memory
`PicMem1 instructs that the decoded picture shall be removed from the picture memory PicMem1.
`[0015] The variable length coding unit VLC encodes the quantized values Coef, the motion vector MV and the memory
`control command MMCO and outputs an encoded stream Str.
`[0016] The case in which the encoding includes the frequency conversion and the quantization is shown, however,
`the encoding may be the one without them such as DPCM, ADPCM, and linear predictive encoding. The encoding may
`be the one in which the frequency conversion and the quantization are integrated or the one that is not accompanied
`by the quantization after the frequency conversion as in a bit-plane encoding.
`[0017] Fig. 27 shows bit streams of the memory control command MMCO. The variable length coding unit VLC encodes
`"000" which means a release of a whole memory area so that the picture memory is initialized at the beginning of the
`encoding/decoding or in the head of the GOP (Group Of Picture). Also, the variable length coding unit VLC encodes
`"01" when the decoded picture is stored in the picture memory. When a picture stored in the picture memory is released
`at the same time, the variable length coding unit VLC encodes a picture number following the "001" since the picture
`number to be released has to be indicated. When a plurality of pictures are released, the command to release a picture
`needs to be encoded for a plural number of times, therefore, a command to store a picture is encoded in addition to the
`command to release a picture. The variable length coding unit VLC encodes sequentially a plurality of memory control
`commands MMCO and encodes lastly "1" indicating that the memory control command MMCO is complete. In this way,
`the memory control command MMCO is encoded as an encoded stream Str.
`[0018] Fig. 28 is a block diagram showing a picture decoding apparatus for realizing a conventional picture decoding
`method. The same numbers are put for the devices that operate in the same manner as the picture encoding apparatus
`for realizing the conventional picture encoding method shown in Fig. 26.
`[0019] A variable length decoding unit VLD decodes an encoded stream Str and outputs a memory control command
`MMCO, a motion vector MV and quantized values Coef. A picture time Time is inputted from outside and is a signal for
`
`5
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`3
`
`

`

`EP 2 278 816 B1
`
`specifying a picture to be displayed. When a picture to be displayed is a decoded picture, an output from the adding unit
`Add is selected at a selector Sel and sent out to a display unit Disp. When a picture to be displayed is a picture stored
`in the picture memory PicMem1, it is read out from the picture memory PicMem1, selected at the selector Sel and
`outputted to a display unit Disp.
`[0020] As described above, the picture memory PicMem1 outputs, to the motion compensation unit MC, pixels ac-
`cording to the motion vector MV whereas the motion compensation unit MC generates a predictive image according to
`the pixels obtained from the picture memory PicMem1 together with the motion vector MV.
`[0021] The inverse quantization unit IQ inverse quantizes the quantized values Coef and restores them as frequency
`coefficients. Furthermore, the inverse orthogonal transformation IT performs inverse frequency conversion for the fre-
`quency coefficients to be outputted as pixel differential values. The addition unit Add adds the predictive image to the
`pixel differential values to generate a decoded picture.
`[0022] The picture memory PicMem1 releases the area in which the decoded picture is stored so that other decoded
`picture can be stored.
`[0023] The example of the decoding including the inverse frequency conversion and the inverse quantization is de-
`scribed above, however, the decoding may be the one without them such as DPCM, ADPCM and a linear predictive
`encoding. The decoding may be the one in which the inverse frequency conversion and the inverse quantization are
`integrated or the one that is not accompanied by the inverse quantization after the frequency conversion as in a bit-plane
`encoding.
`[0024] With the use of the picture decoding apparatus for realizing the conventional picture decoding method shown
`in Fig. 28, it is obvious that the combination of the conventional picture encoding types shown in Figs. 24 and 25 allows
`a correct decoding of the encoded stream Str encoded by the picture encoding apparatus for realizing the conventional
`picture encoding method shown in Fig. 26.
`[0025] The more flexible combination is considered here as a picture encoding type.
`[0026] Fig. 1 is a diagram showing a prediction structure, a decoding order and a display order of the pictures, which
`do not exist in the related art. The prediction structure with respect to B-picture differs in the vicinity of Picture 4 in Fig.
`1. Namely, Picture 2 that is a B-picture is stored in the picture memory to be referred to as a predictive image of Picture
`1 and Picture 3. Consequently, the encoding order and the display order of each picture are as shown in Fig. 1.
`[0027] Pictures B5 and B6 are B-pictures that are not stored since they are not referred to in a predictive coding.
`However, differing from Fig. 24, the display time for the pictures B5 and B6 has not yet come at the time when they are
`decoded since it is the time for other picture to be displayed. That is, at the time of decoding the picture B5, the picture
`P4 shall be displayed and at the time of decoding the picture B6, the picture B5 shall be displayed. Since the pictures
`B5 and B6 are not stored, they cannot be taken out from the picture memory at the display time. Therefore, the pictures
`which are not referred to for predictive encoding are not stored in the picture memory, therefore, the pictures B5 and B6
`cannot be displayed after being decoded with the use of the conventional encoding/decoding method. Namely in the
`case of not storing the pictures that are not referred to in predictive encoding as in the example shown in Fig. 24, only
`Pictures 1, 2, 4, and 7 can be displayed.
`[0028] Thus, considering the more flexible combination as a picture encoding type, it is a problem that the pictures
`which cannot be displayed after being decoded occur. It is conceivable to add another picture memory for display and
`store the pictures that are not stored in the picture memory PicMem1 in this picture memory for display so that they can
`be displayed; however, the weak point is that this picture memory requires a huge memory in this case.
`[0029] Furthermore, there rises a new problem in the reproduction of a picture in the middle of the stream even if
`another picture memory for display is introduced. Fig. 2 is a diagram showing a prediction structure, a decoding order
`and a display order of pictures. The difference comparing with Fig. 25 is that the prediction structure in the vicinity of
`Picture 7 becomes completely independent. The pictures following a picture I7 are not referred to when the pictures with
`display time preceding the picture I7 are encoded/decoded. Therefore, the pictures following the picture I7 can be
`encoded correctly if the decoding starts from the picture I7 and the picture I7 can be reproduced independently. In this
`way, the insertion of I picture while streaming often takes place. This system to reproduce a picture in the middle of the
`stream, which complies with MPEG-2, is called GOP (Group Of Picture).
`[0030] The correspondence of a reproduced picture of the picture decoding apparatus and that of the picture encoding
`apparatus in the case of reproducing the picture in the middle of the stream has to be assured, and the easy method is
`to initialize the whole area of the picture memory, However, picture 6 Is not yet displayed and stored In the picture
`memory when Picture 7 is decoded, Picture 6 therefore cannot be displayed from the picture memory at its display time
`if the entire picture memory is initialized before the display of Picture 6 takes place.
`[0031] US5909224 defines a 4 frame buffer. Obsolete frames are removed after having been defined as no longer
`needed (for display or for reference). It does not define to judge beforehand whether a non reference frame is needed
`for display.
`[0032] EP0729276 defines a decoder buffer for an H.262 bitstream. The buffer is split into several layers corresponding
`to reference picture buffer, display picture buffer, temporary coded data buffer. For respective input picture sizes, re-
`
`5
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`4
`
`

`

`EP 2 278 816 B1
`
`spective display buffer capacity pattern are defined. furthermore, frames are divided into sub-segments to fit into the
`limited 16 Mbits capacity buffers.
`[0033] XP030005105 and XP030005106, ITU-H.264 standards, define an Instantaneous Decoder Refresh and long
`term/short term reference picture buffers.
`[0034] P and B pictures may use this buffer as well. Picture size may not change except when MMCO = reset. However,
`XP030005106 is not concerned with "display time". XP030005105 is concerned with pre-decoder buffer and with pres-
`entation time stamps for transmission.
`[0035] The object of the present invention therefore is to allow the display of the pictures that cannot be displayed
`after being decoded by taking the memory amount necessary for encoding/decoding of the picture into consideration.
`
`5
`
`10
`
`Disclosure of Invention
`
`[0036] The invention is defined by the appended claim.
`
`15
`
`Brief Description of Drawings
`
`[0037]
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`Fig. 1 is a diagram showing a prediction structure, a decoding order and a display order of pictures.
`Fig. 2 is a diagram showing a prediction structure, a decoding order and a display order of the pictures.
`Fig. 3 is a block diagram showing a picture encoding apparatus for realizing a picture encoding method described
`In a first embodiment.
`Fig. 4 is a flowchart showing an operation of a reference picture control unit described in the first embodiment.
`Fig, 5A, 5B and 5C are state diagrams showing a storage status of the pictures in the memory.
`Fig. 6 is a flowchart showing an operation of the picture encoding apparatus described in a second embodiment.
`Fig. 7 is a flowchart showing an operation of the picture encoding apparatus described in a third embodiment.
`Fig. 8 is a flowchart showing an operation of the picture encoding apparatus described in a fourth embodiment.
`Fig. 9 is a block diagram showing a picture decoding apparatus for realizing a picture decoding method of the present
`invention described in a fifth embodiment.
`Fig. 10 is a flowchart showing an operation of the picture decoding apparatus of the present invention described in
`the fifth embodiment.
`Fig. 11 is a flowchart showing another operation of the picture decoding apparatus of the present invention described
`in the fifth embodiment.
`Fig. 12 is a flowchart showing yet another operation of the picture decoding apparatus of the present invention
`described in the fifth embodiment.
`Fig. 13 is a flowchart showing another operation of the picture decoding apparatus of the present invention described
`in the fifth embodiment.
`Fig. 14 is a block diagram showing a usage of a virtual display delay buffer of a picture encoding apparatus.
`Fig. 15 is a block diagram showing a processing of post decoder buffer operation for encoding according to the
`present invention.
`Fig. 16 is a block diagram showing a processing of post decoder buffer operation for decoding according to the
`present invention.
`Fig. 17 is an example of using the virtual display delay buffer of the picture encoding apparatus for limiting the
`maximum number of the reference pictures.
`Fig. 18 is an example of using the virtual display delay buffer for deciding the time to display a first picture.
`Fig. 19 is an illustration of a storage medium in order to store a program for realizing the picture encoding method
`and the picture decoding method of each embodiment in a computing system, described in a seventh embodiment.
`Fig. 20 is a block diagram showing an overall structure of a content supply system described in a eighth embodiment.
`Fig. 21 is an outline view showing an example of a cell phone using the picture encoding/decoding method of the
`present invention described in the eighth embodiment.
`Fig. 22 is a block diagram of the cell phone.
`Fig. 23 is a block diagram showing an example of digital broadcasting system described in the eighth embodiment.
`Fig. 24 is a diagram showing a prediction structure, a decoding order and a display order of the pictures.
`Fig. 25 is a diagram showing a prediction structure, a decoding order and a display order of the pictures.
`Fig. 26 is a block diagram of the picture decoding apparatus for realizing the conventional picture encoding method.
`Fig. 27 is a mapping diagram showing examples of codes for a memory control commands MMCO.
`Fig. 28 is a block diagram of the picture decoding apparatus for realizing the conventional picture decoding method.
`
`5
`
`

`

`Best Mode for Carrying Out the Invention
`
`EP 2 278 816 B1
`
`[0038] The following describes a first embodiment useful for understanding the present invention.
`
`5
`
`(First Embodiment) (not covered by the claim)
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`[0039] Fig. 3 is a block diagram showing a picture encoding apparatus for realizing a picture encoding method. The
`same referential numbers are put for the devices that operate in the same manner as described In the block showing a
`picture encoding apparatus for realizing a conventional encoding method shown in Fig. 26 and the explanation is thereby
`abbreviated.
`[0040] Differences between the block diagram in Fig. 26 showing the picture encoding apparatus for realising the
`conventional picture encoding method and the block diagram in Fig. 3 showing the picture encoding apparatus for
`realizing the picture encoding method are that a display picture control unit DisPicCtrl is added to Fig. 3 and that
`instructions sent from the display picture control unit DisPicCtrl are outputted to a reference picture control unit RefPicCtrl
`and a picture memory PicMem2.
`[0041]
`In the picture encoding apparatus 100 shown in Fig. 3, a picture size modification unit PicSize for obtaining a
`picture size modified by external operations as well as an encoding type of each picture (I-picture, P-picture and B-
`picture) from the picture structure determination unit PicStruct and outputting information indicating the picture size to
`be modified to a reference picture control unit RefPicCtrl is newly set.
`[0042] The operation of the picture memory PicMem2 is almost same as that of the picture memory PicMem1, therefore,
`only different operations will be explained.
`[0043] The display picture control unit DispPicCtrl obtains a picture time Time and judges whether a picture, which is
`not stored as it is not for reference, can be displayed immediately or not (whether it is necessary to store the picture in
`the picture memory until its display time). The picture time Time, a signal for specifying a picture to be displayed, is
`inputted from outside. The picture time can be obtained in the following ways; from the time information outputted from
`the system for transmitting pictures via a transmission line such as a packet, from the time information In process of
`formatting a video stream and audio stream for multiplexing them; or from the time information In process of formatting
`a video stream. The picture time may be either an absolute time which Informs of the time for each picture or a relative
`time which informs of the order of the pictures. Moreover, the Intervals of displaying pictures is normally fixed, therefore,
`the order of displaying pictures may be considered as the display time.
`[0044] Now, the case in which the picture is immediately displayable is a case in which the picture gained after the
`calculation in the adder Add corresponds with the picture to be displayed indicated by the picture time Time. In this case,
`a picture to be displayed before the picture that is not yet displayed and outputted for encoding is not found in the picture
`memory PicMem2. When the picture is not immediately displayable, the display picture control unit DispPicCtrl instructs
`the reference picture control unit RefPicCtrl to store the picture, even though it is not for reference, In the picture memory
`PicMem2. Consequently, the picture which is not displayed immediately is stored in the picture memory PicMem2 without
`fall regardless of whether it is for reference or not and can be displayed out of the picture memory PicMem2 in the
`decoding apparatus.
`[0045] Fig. 4 is a flowchart showing an operation of the reference picture control unit RefPicCtrl.
`[0046] The reference picture control unit RefPicCtrl judges whether or not a decoded picture (picture) is to be stored
`for reference for a predictive image (Step 10). When the decoded picture is to be used for reference, the operation
`proceeds to Step 12, otherwise to Step 11.
`[0047]
`In Step 11, the reference picture control unit RefPicCtrl judges whether or not the decoded picture is immediately
`displayable. "Immediately displayable" here means that the decoded picture can be displayed at the time of being
`decoded (see for example Picture 1 in Fig. 1). The decoded picture which is not immediately displayable means that it
`needs to be displayed later (for instance, B5 shown in Fig. 1). When the picture is immediately displayable, the operation
`is terminated, otherwise, goes on to Step 12.
`[0048]
`In Step 12, the reference picture control unit RefPicCtrl obtains an area capable of storing a picture in the
`picture memory PicMem2 and instructs to store the decoded picture in the area obtained in the picture memory PicMem2
`using a memory control command MMCO in Step 13.
`[0049]
`In this way, the picture which is not displayed immediately is stored in the picture memory PicMem2 and can
`be outputted for display from the picture memory PicMem2 when the time has come for its display. This does not require
`an unnecessary picture memory assigned for a picture for display and a picture that needs to be stored for display can
`be stored in the picture memory assigned for a picture for reference.
`[0050] The picture memory PicMem2 includes an area for reference in which a reference picture is stored for generating
`a predictive image and an area for display in which a picture for display is stored.
`[0051] Meanwhile, a picture size can be modified for each GOP (Group Of Picture) mentioned before. The modification
`of the picture size takes place only when a whole area for reference in the memory storing an unnecessary reference
`
`6
`
`

`

`EP 2 278 816 B1
`
`picture is released (make the status of the memory reusable).
`[0052] However, when the modification of the picture size takes place as described above, the picture for display that
`is not yet displayed is stored in the memory area, and it is necessary to determine explicitly a coping strategy of how to
`handle this picture for display but not yet displayed (whether to delete it or to store it until it is displayed).
`[0053] Here, a storing state of pictures in the memory when the change of the picture size takes place is explained In
`stages.
`[0054] Figs. 5A, 5B and 5C are state diagrams showing the storing status of the pictures in the memory in stages.
`[0055]
`In 5A, pictures 200a, 200b and 200c are the pictures for reference (the pictures to be used for reference in
`order to generate a predictive image) whereas pictures 201a, 201b, 201c, 201d, and 201e are the pictures for display
`(pictures to be displayed and not displayed yet).
`[0056] The pictures 201a, 201b, 201c, 201d and 201e will be displayed in the numeric order as shown in Fig. 5A.
`[0057] Fig. 5A illustrates the status in which the whole memory areas assigned for the reference pictures 200a, 200b
`and 200c are released for the reusability.
`[0058] Fig. 5B shows that the picture size is modified following the status shown in Fig. 5A. A reference picture 202a
`being modified to a bigger size is stor