`
`
`
`
`
`
`
`
`
`
`
`INTERNATIONAL TELECOMMUNICATION UNION
`
`
`
`
`
`ITU-T
`
`TELECOMMUNICATION
`STANDARDIZATION SECTOR
`OF ITU
`
`H.264
`
`(05/2003)
`
`SERIES H: AUDIOVISUAL AND MULTIMEDIA SYSTEMS
`Infrastructure of audiovisual services – Coding of moving
`video
`
`Advanced video coding for generic audiovisual
`services
`
`ITU-T Recommendation H.264
`
`
`Unified Patents, Ex. 1021
`
`000001
`
`

`

`ITU-T H-SERIES RECOMMENDATIONS
`AUDIOVISUAL AND MULTIMEDIA SYSTEMS
`
`
`CHARACTERISTICS OF VISUAL TELEPHONE SYSTEMS
`INFRASTRUCTURE OF AUDIOVISUAL SERVICES
`General
`Transmission multiplexing and synchronization
`Systems aspects
`Communication procedures
`Coding of moving video
`Related systems aspects
`Systems and terminal equipment for audiovisual services
`Directory services architecture for audiovisual and multimedia services
`Quality of service architecture for audiovisual and multimedia services
`Supplementary services for multimedia
`MOBILITY AND COLLABORATION PROCEDURES
`Overview of Mobility and Collaboration, definitions, protocols and procedures
`Mobility for H-Series multimedia systems and services
`Mobile multimedia collaboration applications and services
`Security for mobile multimedia systems and services
`Security for mobile multimedia collaboration applications and services
`Mobility interworking procedures
`Mobile multimedia collaboration inter-working procedures
`BROADBAND AND TRIPLE-PLAY MULTIMEDIA SERVICES
`Broadband multimedia services over VDSL
`
`For further details, please refer to the list of ITU-T Recommendations.
`
`
`H.100–H.199
`
`H.200–H.219
`H.220–H.229
`H.230–H.239
`H.240–H.259
`H.260–H.279
`H.280–H.299
`H.300–H.349
`H.350–H.359
`H.360–H.369
`H.450–H.499
`
`H.500–H.509
`H.510–H.519
`H.520–H.529
`H.530–H.539
`H.540–H.549
`H.550–H.559
`H.560–H.569
`
`H.610–H.619
`
`
`Unified Patents, Ex. 1021
`
`000002
`
`

`

`
`
`ITU-T Recommendation H.264
`
`Advanced video coding for generic audiovisual services
`
`
`
`
`
`
`
`
`Summary
`As the costs for both processing power and memory have reduced, network support for coded video data has diversified,
`and advances in video coding technology have progressed, the need has arisen for an industry standard for compressed
`video representation with substantially increased coding efficiency and enhanced robustness to network environments.
`This Recommendation represents an evolution of the existing video coding standards (H.261, H.262 and H.263) and it
`was developed in response to the growing need for higher compression of moving pictures for various applications such
`as videoconferencing, digital storage media, television broadcasting, Internet streaming and communication. It is also
`designed to enable the use of the coded video representation in a flexible manner for a wide variety of network
`environments. The use of this Recommendation allows motion video to be manipulated as a form of computer data and
`to be stored on various storage media, transmitted and received over existing and future networks and distributed on
`existing and future broadcasting channels
`
`
`Source
`ITU-T Recommendation H.264 was approved by ITU-T Study Group 16 (2001-2004) under
`Recommendation A.8 procedure on 30 May 2003.
`This edition includes the modifications introduced by ITU-T Rec. H.264 (2003) Corrigendum 1 approved by ITU-T
`Study Group 16 (2001-2004) under the ITU-T Recommendation A.8 procedure on 7 May 2004.
`
`
`
`
`
`the ITU-T
`
`
`
`
`
`
`
`ITU-T Rec. H.264 (05/2003)
`
`i
`
`Unified Patents, Ex. 1021
`
`000003
`
`

`

`
`
`FOREWORD
`The International Telecommunication Union (ITU) is the United Nations specialized agency in the field of
`telecommunications. The ITU Telecommunication Standardization Sector (ITU-T) is a permanent organ of
`ITU. ITU-T is responsible for studying technical, operating and tariff questions and issuing Recommendations
`on them with a view to standardizing telecommunications on a worldwide basis.
`The World Telecommunication Standardization Assembly (WTSA), which meets every four years,
`establishes the topics for study by the ITU-T study groups which, in turn, produce Recommendations on these
`topics.
`The approval of ITU-T Recommendations is covered by the procedure laid down in WTSA Resolution 1.
`In some areas of information technology which fall within ITU-T's purview, the necessary standards are
`prepared on a collaborative basis with ISO and IEC.
`
`
`
`
`NOTE
`In this Recommendation, the expression "Administration" is used for conciseness to indicate both a
`telecommunication administration and a recognized operating agency.
`
`Compliance with this Recommendation is voluntary. However, the Recommendation may contain certain
`mandatory provisions (to ensure e.g.
`interoperability or applicability) and compliance with
`the
`Recommendation is achieved when all of these mandatory provisions are met. The words "shall" or some
`other obligatory language such as "must" and the negative equivalents are used to express requirements. The
`use of such words does not suggest that compliance with the Recommendation is required of any party.
`
`
`
`
`INTELLECTUAL PROPERTY RIGHTS
`ITU draws attention to the possibility that the practice or implementation of this Recommendation may
`involve the use of a claimed Intellectual Property Right. ITU takes no position concerning the evidence,
`validity or applicability of claimed Intellectual Property Rights, whether asserted by ITU members or others
`outside of the Recommendation development process.
`As of the date of approval of this Recommendation, ITU [had/had not] received notice of intellectual
`property, protected by patents, which may be required to implement this Recommendation. However,
`implementors are cautioned that this may not represent the latest information and are therefore strongly urged
`to consult the TSB patent database.
`
`
`
`
` ITU 2004
`All rights reserved. No part of this publication may be reproduced, by any means whatsoever, without the
`prior written permission of ITU.
`
`ii
`
`ITU-T Rec. H.264 (05/2003)
`
`Unified Patents, Ex. 1021
`
`000004
`
`

`

`
`
`CONTENTS
`
`Page
`
`
`
`Foreword ..........................................................................................................................................................................xiii
`0
`Introduction .............................................................................................................................................................xiv
`0.1
`Prologue ............................................................................................................................................................ xiv
`0.2
`Purpose.............................................................................................................................................................. xiv
`0.3
`Applications....................................................................................................................................................... xiv
`0.4
`Profiles and levels.............................................................................................................................................. xiv
`0.5 Overview of the design characteristics............................................................................................................... xv
`0.5.1
`Predictive coding ........................................................................................................................................xv
`0.5.2
`Coding of progressive and interlaced video................................................................................................xv
`0.5.3
`Picture partitioning into macroblocks and smaller partitions......................................................................xv
`0.5.4
`Spatial redundancy reduction.....................................................................................................................xvi
`0.6 How to read this specification ........................................................................................................................... xvi
`Scope ............................................................................................................................................................................1
`1
`2 Normative references..................................................................................................................................................1
`3 Definitions ...................................................................................................................................................................1
`4 Abbreviations..............................................................................................................................................................8
`5 Conventions.................................................................................................................................................................9
`5.1
`Arithmetic operators.............................................................................................................................................9
`5.2
`Logical operators .................................................................................................................................................9
`5.3
`Relational operators ...........................................................................................................................................10
`5.4
`Bit-wise operators...............................................................................................................................................10
`5.5
`Assignment operators .........................................................................................................................................10
`5.6
`Range notation....................................................................................................................................................10
`5.7 Mathematical functions ......................................................................................................................................10
`5.8
`Variables, syntax elements, and tables ...............................................................................................................11
`5.9
`Text description of logical operations ................................................................................................................12
`5.10 Processes ............................................................................................................................................................13
`Source, coded, decoded and output data formats, scanning processes, and neighbouring relationships .........13
`6.1
`Bitstream formats ...............................................................................................................................................13
`6.2
`Source, decoded, and output picture formats .....................................................................................................13
`6.3
`Spatial subdivision of pictures and slices ...........................................................................................................15
`6.4
`Inverse scanning processes and derivation processes for neighbours ...............................................................16
`6.4.1
`Inverse macroblock scanning process.........................................................................................................16
`6.4.2
`Inverse macroblock partition and sub-macroblock partition scanning process...........................................17
`6.4.2.1
`Inverse macroblock partition scanning process ......................................................................................17
`6.4.2.2
`Inverse sub-macroblock partition scanning process ...............................................................................18
`6.4.3
`Inverse 4x4 luma block scanning process...................................................................................................18
`6.4.4
`Derivation process of the availability for macroblock addresses................................................................18
`6.4.5
`Derivation process for neighbouring macroblock addresses and their availability.....................................19
`6.4.6
`Derivation process for neighbouring macroblock addresses and their availability in MBAFF frames ......19
`6.4.7
`Derivation processes for neighbouring macroblocks, blocks, and partitions..............................................20
`6.4.7.1 Derivation process for neighbouring macroblocks .................................................................................21
`6.4.7.2 Derivation process for neighbouring 8x8 luma block.............................................................................21
`6.4.7.3 Derivation process for neighbouring 4x4 luma blocks ...........................................................................22
`6.4.7.4 Derivation process for neighbouring 4x4 chroma blocks .......................................................................22
`6.4.7.5 Derivation process for neighbouring partitions.......................................................................................22
`6.4.8
`Derivation process for neighbouring locations ...........................................................................................24
`6.4.8.1 Specification for neighbouring luma locations in fields and non-MBAFF frames .................................24
`6.4.8.2 Specification for neighbouring luma locations in MBAFF frames.........................................................24
`Syntax and semantics ...............................................................................................................................................27
`7.1 Method of describing syntax in tabular form......................................................................................................27
`7.2
`Specification of syntax functions, categories, and descriptors ...........................................................................28
`7.3
`Syntax in tabular form ........................................................................................................................................30
`7.3.1
`NAL unit syntax .........................................................................................................................................30
`7.3.2
`Raw byte sequence payloads and RBSP trailing bits syntax ......................................................................31
`7.3.2.1 Sequence parameter set RBSP syntax.....................................................................................................31
`7.3.2.2 Picture parameter set RBSP syntax.........................................................................................................32
`
`6
`
`7
`
`
`
`
`
`
`
`ITU-T Rec. H.264 (05/2003)
`
`iii
`
`Unified Patents, Ex. 1021
`
`000005
`
`

`

`
`
`7.3.2.3 Supplemental enhancement information RBSP syntax...........................................................................33
`7.3.2.3.1 Supplemental enhancement information message syntax ................................................................33
`7.3.2.4 Access unit delimiter RBSP syntax ........................................................................................................33
`7.3.2.5 End of sequence RBSP syntax................................................................................................................33
`7.3.2.6 End of stream RBSP syntax....................................................................................................................34
`7.3.2.7 Filler data RBSP syntax..........................................................................................................................34
`7.3.2.8 Slice layer without partitioning RBSP syntax.........................................................................................34
`7.3.2.9 Slice data partition RBSP syntax ............................................................................................................34
`7.3.2.9.1 Slice data partition A RBSP syntax..................................................................................................34
`7.3.2.9.2 Slice data partition B RBSP syntax..................................................................................................34
`7.3.2.9.3 Slice data partition C RBSP syntax..................................................................................................35
`7.3.2.10 RBSP slice trailing bits syntax..............................................................................................................35
`7.3.2.11 RBSP trailing bits syntax ......................................................................................................................35
`7.3.3
`Slice header syntax .....................................................................................................................................36
`7.3.3.1 Reference picture list reordering syntax .................................................................................................37
`7.3.3.2 Prediction weight table syntax................................................................................................................38
`7.3.3.3 Decoded reference picture marking syntax.............................................................................................39
`7.3.4
`Slice data syntax .........................................................................................................................................40
`7.3.5 Macroblock layer syntax.............................................................................................................................41
`7.3.5.1 Macroblock prediction syntax.................................................................................................................42
`7.3.5.2 Sub-macroblock prediction syntax .........................................................................................................43
`7.3.5.3 Residual data syntax ...............................................................................................................................44
`7.3.5.3.1 Residual block CAVLC syntax........................................................................................................45
`7.3.5.3.2 Residual block CABAC syntax........................................................................................................46
`7.4
`Semantics............................................................................................................................................................47
`7.4.1
`NAL unit semantics ....................................................................................................................................47
`7.4.1.1 Encapsulation of an SODB within an RBSP (informative) ....................................................................49
`7.4.1.2 Order of NAL units and association to coded pictures, access units, and video sequences....................50
`7.4.1.2.1 Order of sequence and picture parameter set RBSPs and their activation .......................................50
`7.4.1.2.2 Order of access units and association to coded video sequences .....................................................51
`7.4.1.2.3 Order of NAL units and coded pictures and association to access units..........................................51
`7.4.1.2.4 Detection of the first VCL NAL unit of a primary coded picture ....................................................52
`7.4.1.2.5 Order of VCL NAL units and association to coded pictures............................................................53
`7.4.2
`Raw byte sequence payloads and RBSP trailing bits semantics .................................................................53
`7.4.2.1 Sequence parameter set RBSP semantics ...............................................................................................53
`7.4.2.2 Picture parameter set RBSP semantics ...................................................................................................56
`7.4.2.3 Supplemental enhancement information RBSP semantics .....................................................................58
`7.4.2.3.1 Supplemental enhancement information message semantics...........................................................58
`7.4.2.4 Access unit delimiter RBSP semantics ...................................................................................................58
`7.4.2.5 End of sequence RBSP semantics...........................................................................................................58
`7.4.2.6 End of stream RBSP semantics...............................................................................................................58
`7.4.2.7 Filler data RBSP semantics.....................................................................................................................59
`7.4.2.8 Slice layer without partitioning RBSP semantics ...................................................................................59
`7.4.2.9 Slice data partition RBSP semantics.......................................................................................................59
`7.4.2.9.1 Slice data partition A RBSP semantics ............................................................................................59
`7.4.2.9.2 Slice data partition B RBSP semantics ............................................................................................59
`7.4.2.9.3 Slice data partition C RBSP semantics ............................................................................................59
`7.4.2.10 RBSP slice trailing bits semantics.........................................................................................................60
`7.4.2.11 RBSP trailing bits semantics.................................................................................................................60
`7.4.3
`Slice header semantics................................................................................................................................60
`7.4.3.1 Reference picture list reordering semantics ............................................................................................65
`7.4.3.2 Prediction weight table semantics...........................................................................................................66
`7.4.3.3 Decoded reference picture marking semantics .......................................................................................67
`7.4.4
`Slice data semantics....................................................................................................................................69
`7.4.5 Macroblock layer semantics .......................................................................................................................70
`7.4.5.1 Macroblock prediction semantics ...........................................................................................................76
`7.4.5.2 Sub-macroblock prediction semantics ....................................................................................................76
`7.4.5.3 Residual data semantics ..........................................................................................................................79
`7.4.5.3.1 Residual block CAVLC semantics...................................................................................................79
`7.4.5.3.2 Residual block CABAC semantics ..................................................................................................80
`8 Decoding process.......................................................................................................................................................80
`8.1
`NAL unit decoding process.................................................................................................................................81
`8.2
`Slice decoding process........................................................................................................................................82
`
`iv
`
`ITU-T Rec. H.264 (05/2003)
`
`Unified Patents, Ex. 1021
`
`000006
`
`

`

`
`
`Decoding process for picture order count...................................................................................................82
`8.2.1
`8.2.1.1 Decoding process for picture order count type 0 ....................................................................................83
`8.2.1.2 Decoding process for picture order count type 1 ....................................................................................84
`8.2.1.3 Decoding process for picture order count type 2 ....................................................................................85
`8.2.2
`Decoding process for macroblock to slice group map ................................................................................86
`8.2.2.1 Specification for interleaved slice group map type.................................................................................87
`8.2.2.2 Specification for dispersed slice group map type ...................................................................................87
`8.2.2.3 Specification for foreground with left-over slice group map type ..........................................................87
`8.2.2.4 Specification for box-out slice group map types.....................................................................................88
`8.2.2.5 Specification for raster scan slice group map types ................................................................................88
`8.2.2.6 Specification for wipe slice group map types .........................................................................................88
`8.2.2.7 Specification for explicit slice group map type.......................................................................................89
`8.2.2.8 Specification for conversion of map unit to slice group map to macroblock to slice group map ...........89
`8.2.3
`Decoding process for slice data partitioning...............................................................................................89
`8.2.4
`Decoding process for reference picture lists construction ..........................................................................90
`8.2.4.1 Decoding process for picture numbers....................................................................................................90
`8.2.4.2
`Initialisation process for reference picture lists ......................................................................................91
`8.2.4.2.1
`Initialisation process for the reference picture list for P and SP slices in frames.............................91
`8.2.4.2.2
`Initialisation process for the reference picture list for P and SP slices in fields...............................92
`8.2.4.2.3
`Initialisation process for reference picture lists for B slices in frames.............................................92
`8.2.4.2.4
`Initialisation process for reference picture lists for B slices in fields...............................................93
`8.2.4.2.5
`Initialisation process for reference picture lists in fields..................................................................94
`8.2.4.3 Reordering process for reference picture lists.........................................................................................94
`8.2.4.3.1 Reordering process of reference picture lists for short-term pictures ..............................................95
`8.2.4.3.2 Reordering process of reference picture lists for long-term pictures ...............................................96
`8.2.5
`Decoded reference picture marking process ...............................................................................................96
`8.2.5.1 Sequence of operations for decoded reference picture marking process ................................................97
`8.2.5.2 Decoding process for gaps in frame_num...............................................................................................97
`8.2.5.3 Sliding window decoded reference picture marking process..................................................................98
`8.2.5.4 Adaptive memory control decoded reference picture marking process ..................................................98
`8.2.5.4.1 Marking process of a short-term picture as “unused for reference”.................................................98
`8.2.5.4.2 Marking process of a long-term picture as “unused for reference”..................................................98
`8.2.5.4.3 Assignment process of a LongTermFrameIdx to a short-term reference picture.............................99
`8.2.5.4.4 Decoding process for MaxLongTermFrameIdx...............................................................................99
`8.2.5.4.5 Marking process of all reference pictures as “unused for reference” and setting
`MaxLongTermFrameIdx to “no long-term frame indices”.................................................................................99
`8.2.5.4.6 Process for assigning a long-term frame index to the current picture ..............................................99
`8.3
`Intra prediction process....................................................................................................................................100
`8.3.1
`Intra_4x4 prediction process for luma samples ........................................................................................100
`8.3.1.1 Derivation process for the Intra4x4PredMode......................................................................................101
`8.3.1.2
`Intra_4x4 sample prediction .................................................................................................................102
`8.3.1.2.1 Specification of Intra_4x4_Vertical prediction mode ....................................................................103
`8.3.1.2.2 Specification of Intra_4x4_Horizontal prediction mode................................................................103
`8.3.1.2.3 Specification of Intra_4x4_DC prediction mode ...........................................................................103
`8.3.1.2.4 Specification of Intra_4x4_Diagonal_Down_Left prediction mode ..............................................103
`8.3.1.2.5 Specification of Intra_4x4_Diagonal_Down_Right prediction mode............................................104
`8.3.1.2.6 Specification of Intra_4x4_Vertical_Right prediction mode .........................................................104
`8.3.1.2.7 Specification of Intra_4x4_Horizontal_Down prediction mode ....................................................104
`8.3.1.2.8 Specification of Intra_4x4_Vertical_Left prediction mode ...........................................................105
`8.3.1.2.9 Specification of Intra_4x4_Horizontal_Up prediction mode .........................................................105
`8.3.2
`Intra_16x16 prediction process for luma samples ....................................................................................105
`8.3.2.1 Specification of Intra_16x16_Vertical prediction mode.......................................................................106
`8.3.2.2 Specification of Intra_16x16_Horizontal prediction mode...................................................................106
`8.3.2.3 Specification of Intra_16x16_DC prediction mode ..............................................................................106
`8.3.2.4 Specification of Intra_16x16_Plane prediction mode...........................................................................107
`8.3.3
`Intra prediction process for chroma samples ............................................................................................107
`8.3.3.1 Specification of Intra_Chroma_DC prediction mode ...........................................................................108
`8.3.3.2 Specification of Intra_Chroma_Horizontal prediction mode................................................................109
`8.3.3.3 Specification of Intra_Chroma_Vertical prediction mode....................................................................110
`8.3.3.4 Specification of Intra_Chroma_Plane prediction mode........................................................................110
`8.3.4
`Sample construction process for I_PCM macroblocks ............................................................