Signals and Communication Technology
`
`Mathias Wien
`
`High
`Efficiency
`Video Coding
`
`Coding Tools and Specification
`
`GE Video Exhibit 2001
`Unified Patents v. GE Video, IPR2019-00726
`
`

`

`Signals and Communication Technology
`
`aesciascia@mintz.com
`
`GE Video Exhibit 2001
`
`

`

`More information about this series at http://www.springer.com/series/4748
`
`aesciascia@mintz.com
`
`GE Video Exhibit 2001
`
`

`

`Mathias Wien
`
`High Efficiency Video
`Coding
`Coding Tools and Specification
`
`123
`
`aesciascia@mintz.com
`
`GE Video Exhibit 2001
`
`

`

`Mathias Wien
`Institut für Nachrichtentechnik
`RWTH Aachen University
`Aachen
`Germany
`
`ISSN 1860-4862
`ISBN 978-3-662-44275-3
`DOI 10.1007/978-3-662-44276-0
`
`ISSN 1860-4870 (electronic)
`ISBN 978-3-662-44276-0
`(eBook)
`
`Library of Congress Control Number: 2014944714
`
`Springer Heidelberg New York Dordrecht London
`
`Ó Springer-Verlag Berlin Heidelberg 2015
`This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part of
`the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations,
`recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission or
`information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar
`methodology now known or hereafter developed. Exempted from this legal reservation are brief
`excerpts in connection with reviews or scholarly analysis or material supplied specifically for the
`purpose of being entered and executed on a computer system, for exclusive use by the purchaser of the
`work. Duplication of this publication or parts thereof is permitted only under the provisions of
`the Copyright Law of the Publisher’s location, in its current version, and permission for use must
`always be obtained from Springer. Permissions for use may be obtained through RightsLink at the
`Copyright Clearance Center. Violations are liable to prosecution under the respective Copyright Law.
`The use of general descriptive names, registered names, trademarks, service marks, etc. in this
`publication does not imply, even in the absence of a specific statement, that such names are exempt
`from the relevant protective laws and regulations and therefore free for general use.
`While the advice and information in this book are believed to be true and accurate at the date of
`publication, neither the authors nor the editors nor the publisher can accept any legal responsibility for
`any errors or omissions that may be made. The publisher makes no warranty, express or implied, with
`respect to the material contained herein.
`
`Printed on acid-free paper
`
`Springer is part of Springer Science+Business Media (www.springer.com)
`
`aesciascia@mintz.com
`
`GE Video Exhibit 2001
`
`

`

`Preface
`
`Since the first publication of the video coding standard H.264 | AVC in 2003, the
`video world has changed. Video content has become a major fraction of digital
`network traffic worldwide and is still growing. The demand for HD and Ultra HD
`video (with picture resolutions of 4K  2K and more) increases, inducing even
`higher bandwidth needs. The established standard H.264 | AVC is used for HD
`content while having been developed mainly for resolutions around standard
`definition television. The standardization effort for ‘‘High Efficiency Video
`Coding’’ (HEVC) answers the demand for improved compression performance at
`resolutions of HD and beyond. The design of the included coding tools enables
`video compression to the desired degree in conjunction with implementation
`friendliness and options for parallelization at multiple levels. It is expected to be
`widely adopted for video services at HD and Ultra HD quality, providing Ultra HD
`video at similar bitrates as used for HD video today. Based on the known concepts
`of the well-established hybrid coding scheme, new coding structures and better
`coding tools have been developed and specified for HEVC. The new standard is
`expected to be taken up easily by established industry as well as new endeavors,
`answering the needs of today’s connected and ever-evolving online world.
`This book presents the standard and strives to explain it in a clear and coherent
`language. It provides a comprehensive and consistently written description of the
`applied concepts and coding tools. For synchronization of beginners in the field, a
`chapter on the fundamentals of video coding is included. It provides a general
`overview on the elements of video coding systems, the representation of color
`video, and an introduction of the building blocks of the hybrid coding scheme.
`Another chapter is dedicated to the topic of specification by itself; it deals with
`requirements on specification text and with imposed fundamental technological
`guidelines. The understanding of these principles is utile for assessment of the
`
`aesciascia@mintz.com
`
`v
`
`GE Video Exhibit 2001
`
`

`

`vi
`
`Preface
`
`algorithmic design at present as well as for future development of extensions and
`potential additional coding tools. The following chapters follow the structure of
`the HEVC specification, giving insight to the design and concepts of the coding
`tools in the specification.
`The book shall help readers to understand the state-of-the-art concepts of video
`coding with HEVC as their latest instantiation. It shall contribute to the promul-
`gation of the HEVC standard and provide support in adopting it for new and
`fascinating applications.
`Examples in the chapters make use of bitstreams and test sequences according
`to the common testing conditions of the Joint Collaborative Team on Video
`Coding (JCT-VC). Bitstreams according to these conditions were frequently made
`public on the JCT-VC experts mailing list by the reference software coordinators.
`Different versions of these bitstreams have been used as anchors in numerous tests
`and extensive experiments throughout the HEVC development. Coding examples
`in this book are given for the test sequence BasketballDrive, provided courtesy of
`NTT DOCOMO, Inc., Japan, and ParkScene, provided courtesy of Tokyo Institute
`of Technology, Nakajima Laboratory, Japan. Examples for the sequences of
`the JCT-VC common testing conditions further include BlowingBubbles and
`RaceHorses, provided courtesy of NTT DOCOMO, Inc., Japan; ChinaSpeed,
`provided courtesy of Shanghai Shulong Computer Technology Co., Ltd., China;
`FourPeople, provided courtesy of Vidyo Inc., USA; and PeopleOnStreet, provided
`courtesy of Samsung Electronics Co., Ltd., Korea.
`Hundreds of experts in the JCT-VC have worked hard, spending time in never-
`ending meetings with tremendous numbers of input contributions to achieve the
`goal of a stable high-quality high-performance specification. Their contribution
`and commitment are highly appreciated. The standardization work has been car-
`ried out under the prudent and stimulating lead of the JCT-VC chairs Gary J.
`Sullivan and Jens-Rainer Ohm. Their thoughtful and precise guidance is specifi-
`cally appreciated. It formed and coined this collaborative team.
`This book would have not been possible without help and support from several
`people. I want to thank an uncounted number of JCT-VC experts for numerous
`conversations and debates on various aspects of the specification. Special thanks
`go to T. K. Tan and Andrew Segall for manifold and insight-full discussions over
`the years; and to Benjamin Bross who knew all answers to questions on the
`specification. Very special thanks go to Rickard Sjöberg for his expertise and
`helpful comments on picture types and reference picture management; and spe-
`cifically to Peter Amon for his comprehensive and remarkably careful text review.
`Peter, I owe you more than a beer. Special thanks for review and support also go to
`people at Institut für Nachrichtentechnik of RWTH Aachen University: Julian
`Becker, Max Bläser, Christopher Bulla, Olena Chubach, Christian Feldmann, Iris
`
`aesciascia@mintz.com
`
`GE Video Exhibit 2001
`
`

`

`Preface
`
`vii
`
`Heisterklaus, Cordula Heithausen, Fabian Jäger, Ningqing Qian, Inge Reissel,
`Christian Rohlfing, and Uday Thakur; very special thanks to Bastian Cellarius who
`helped generating the example figures.
`Most special
`thanks go to Prof. Jens-Rainer Ohm for providing distinct
`leadership, kind mentoring, and encouraging guidance. Thank you for providing
`me room to write this book. It is an outstanding experience and honor to work in
`your team. The warmest thanks go to my wonderful family: Sabine and our
`children Frederic, Leonard, and Marlene.
`
`Aachen, May 2014
`
`Mathias Wien
`
`aesciascia@mintz.com
`
`GE Video Exhibit 2001
`
`

`

`Contents
`
`1
`
`2
`
`1.3
`
`Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
`1.1
`How to Read This Book . . . . . . . . . . . . . . . . . . . . . . . . . . .
`1.2
`A Brief Walk Through the History of Video Coding
`Standards. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
`1.2.1
`Advanced Video Coding . . . . . . . . . . . . . . . . . . . . .
`1.2.2
`High Efficiency Video Coding . . . . . . . . . . . . . . . . .
`Evolution of a Specification. . . . . . . . . . . . . . . . . . . . . . . . .
`1.3.1
`Formal Procedure for a Standard in ISO/IEC . . . . . . .
`1.3.2
`Formal Procedure for a Recommendation
`in the ITU-T . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
`The Joint Collaborative Team on Video Coding. . . . . . . . . . .
`1.4.1
`Approval Process . . . . . . . . . . . . . . . . . . . . . . . . . .
`1.4.2 Method of Working . . . . . . . . . . . . . . . . . . . . . . . .
`1.4.3
`Deliverables. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
`1.4.4
`Structure of the JCT-VC . . . . . . . . . . . . . . . . . . . . .
`References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
`
`1.4
`
`Video Coding Fundamentals . . . . . . . . . . . . . . . . . . . . . . . . . . . .
`2.1
`Video Coding Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . .
`2.1.1
`Video Acquisition. . . . . . . . . . . . . . . . . . . . . . . . . .
`2.1.2
`Pre-processing . . . . . . . . . . . . . . . . . . . . . . . . . . . .
`2.1.3
`Encoding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
`2.1.4
`Transmission . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
`2.1.5
`Decoding. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
`2.1.6
`Post-processing . . . . . . . . . . . . . . . . . . . . . . . . . . .
`2.1.7
`Display . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
`Structure of a Video Sequence . . . . . . . . . . . . . . . . . . . . . . .
`2.2.1
`Pictures, Frames, and Fields. . . . . . . . . . . . . . . . . . .
`2.2.2
`Sample Shape . . . . . . . . . . . . . . . . . . . . . . . . . . . .
`Representation of Color. . . . . . . . . . . . . . . . . . . . . . . . . . . .
`
`2.2
`
`2.3
`
`aesciascia@mintz.com
`
`1
`1
`
`2
`5
`5
`6
`8
`
`10
`12
`13
`14
`15
`16
`19
`
`23
`23
`24
`25
`25
`26
`26
`27
`27
`28
`28
`29
`30
`
`ix
`
`GE Video Exhibit 2001
`
`

`

`Contents
`
`2.4
`
`The CIE Standard Observer . . . . . . . . . . . . . . . . . . .
`2.3.1
`Color Primaries . . . . . . . . . . . . . . . . . . . . . . . . . . .
`2.3.2
`Display Transfer Characteristics . . . . . . . . . . . . . . . .
`2.3.3
`Color Conversion . . . . . . . . . . . . . . . . . . . . . . . . . .
`2.3.4
`Chroma Sub-sampling . . . . . . . . . . . . . . . . . . . . . . .
`2.3.5
`The Hybrid Video Coding Scheme . . . . . . . . . . . . . . . . . . . .
`2.4.1
`Picture Partitioning . . . . . . . . . . . . . . . . . . . . . . . . .
`2.4.2
`Intra Prediction . . . . . . . . . . . . . . . . . . . . . . . . . . .
`2.4.3
`Inter Prediction . . . . . . . . . . . . . . . . . . . . . . . . . . .
`2.4.4 Motion Estimation . . . . . . . . . . . . . . . . . . . . . . . . .
`2.4.5
`Residual Coding . . . . . . . . . . . . . . . . . . . . . . . . . . .
`2.4.6
`In-loop Filtering . . . . . . . . . . . . . . . . . . . . . . . . . . .
`2.4.7
`The Decoded Picture Buffer . . . . . . . . . . . . . . . . . .
`2.4.8
`Entropy Coding . . . . . . . . . . . . . . . . . . . . . . . . . . .
`Encoder Control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
`2.5.1
`Distortion Measures . . . . . . . . . . . . . . . . . . . . . . . .
`2.5.2
`Rate-Distortion Optimization . . . . . . . . . . . . . . . . . .
`Compression Artifacts . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
`2.6
`References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
`
`2.5
`
`x
`
`3
`
`31
`32
`32
`33
`35
`37
`39
`40
`41
`43
`43
`53
`55
`55
`62
`63
`65
`66
`69
`
`73
`74
`74
`75
`75
`76
`77
`77
`78
`78
`79
`79
`80
`81
`81
`82
`83
`83
`84
`86
`88
`
`3.2
`
`3.3
`
`Design and Specification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
`3.1
`Specification Fundamentals . . . . . . . . . . . . . . . . . . . . . . . . .
`3.1.1
`Interoperability. . . . . . . . . . . . . . . . . . . . . . . . . . . .
`3.1.2
`Specification Scope . . . . . . . . . . . . . . . . . . . . . . . .
`3.1.3
`Text Classification . . . . . . . . . . . . . . . . . . . . . . . . .
`3.1.4
`Editing Perspective . . . . . . . . . . . . . . . . . . . . . . . . .
`Specification Elements . . . . . . . . . . . . . . . . . . . . . . . . . . . .
`3.2.1
`Syntax and Semantics . . . . . . . . . . . . . . . . . . . . . . .
`3.2.2
`Decoding Process . . . . . . . . . . . . . . . . . . . . . . . . . .
`3.2.3
`Parsing Process . . . . . . . . . . . . . . . . . . . . . . . . . . .
`Specification Principles . . . . . . . . . . . . . . . . . . . . . . . . . . . .
`3.3.1
`Loss Robustness . . . . . . . . . . . . . . . . . . . . . . . . . . .
`3.3.2
`Independent Parsing . . . . . . . . . . . . . . . . . . . . . . . .
`3.3.3
`Bit-Exact Specification . . . . . . . . . . . . . . . . . . . . . .
`3.3.4
`Parallelization . . . . . . . . . . . . . . . . . . . . . . . . . . . .
`3.3.5
`Dynamic Range . . . . . . . . . . . . . . . . . . . . . . . . . . .
`3.3.6
`Timing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
`Conformance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
`How to Read the Specification Text . . . . . . . . . . . . . . . . . . .
`3.5.1
`Terminology . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
`3.5.2
`Conventions and Geometric Relations . . . . . . . . . . . .
`
`3.4
`3.5
`
`aesciascia@mintz.com
`
`GE Video Exhibit 2001
`
`

`

`Contents
`
`3.6
`
`Drafting Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
`3.6.1 Measuring the Compression Performance . . . . . . . . .
`3.6.2
`Proposal Assessment . . . . . . . . . . . . . . . . . . . . . . . .
`References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
`
`4
`
`4.2
`
`Coding Structures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
`4.1
`Temporal Coding Structures. . . . . . . . . . . . . . . . . . . . . . . . .
`4.1.1
`Temporal Layers . . . . . . . . . . . . . . . . . . . . . . . . . .
`4.1.2
`Picture Types . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
`4.1.3
`Splicing of Video Sequences . . . . . . . . . . . . . . . . . .
`Comparison to H.264 j AVC. . . . . . . . . . . . . . . . . . .
`4.1.4
`Spatial Coding Structures . . . . . . . . . . . . . . . . . . . . . . . . . .
`4.2.1
`Blocks and Units . . . . . . . . . . . . . . . . . . . . . . . . . .
`4.2.2
`Slices and Slice Segments . . . . . . . . . . . . . . . . . . . .
`4.2.3
`Tiles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
`4.2.4
`Coding Tree Block and Coding Block. . . . . . . . . . . .
`4.2.5
`Prediction Block. . . . . . . . . . . . . . . . . . . . . . . . . . .
`4.2.6
`Transform Tree and Transform Block . . . . . . . . . . . .
`Comparison to H.264 j AVC. . . . . . . . . . . . . . . . . . .
`4.2.7
`Reference Pictures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
`4.3.1
`Reference Picture Sets . . . . . . . . . . . . . . . . . . . . . .
`4.3.2
`Reference Picture Lists . . . . . . . . . . . . . . . . . . . . . .
`4.3.3
`Short-Term Reference Picture Set Signaling . . . . . . .
`4.3.4
`Long-Term Reference Picture Set Signaling . . . . . . .
`Comparison to H.264 j AVC. . . . . . . . . . . . . . . . . . .
`4.3.5
`References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
`
`4.3
`
`5.3
`
`5 High-Level Syntax . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
`5.1
`Byte Stream Format . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
`5.2
`Network Abstraction Layer . . . . . . . . . . . . . . . . . . . . . . . . .
`5.2.1
`NAL Unit Structure . . . . . . . . . . . . . . . . . . . . . . . .
`5.2.2
`NAL Unit Types . . . . . . . . . . . . . . . . . . . . . . . . . .
`5.2.3
`Access Units . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
`5.2.4
`Decoding Units . . . . . . . . . . . . . . . . . . . . . . . . . . .
`Parameter Sets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
`5.3.1
`Video Parameter Set . . . . . . . . . . . . . . . . . . . . . . . .
`5.3.2
`Sequence Parameter Set . . . . . . . . . . . . . . . . . . . . .
`5.3.3
`Picture Parameter Set . . . . . . . . . . . . . . . . . . . . . . .
`Slice Segment Header . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
`5.4.1
`Picture Order Count . . . . . . . . . . . . . . . . . . . . . . . .
`Supplemental Enhancement Information . . . . . . . . . . . . . . . .
`
`5.4
`
`5.5
`
`xi
`
`91
`92
`97
`99
`
`101
`102
`105
`106
`112
`113
`114
`114
`115
`118
`120
`121
`122
`123
`124
`125
`128
`130
`130
`131
`132
`
`133
`133
`134
`134
`136
`139
`140
`140
`143
`144
`144
`145
`145
`147
`
`aesciascia@mintz.com
`
`GE Video Exhibit 2001
`
`

`

`xii
`
`6
`
`Contents
`
`149
`150
`152
`152
`153
`153
`154
`154
`154
`155
`155
`156
`156
`157
`158
`159
`
`161
`162
`163
`163
`165
`
`166
`166
`167
`168
`168
`169
`170
`172
`172
`174
`175
`176
`177
`
`179
`179
`180
`181
`183
`
`5.6
`
`5.7
`
`Hypothetical Reference Decoder. . . . . . . . . . . . . . . . . . . . . .
`5.6.1
`Coded Picture Buffer . . . . . . . . . . . . . . . . . . . . . . .
`5.6.2
`Decoded Picture Buffer . . . . . . . . . . . . . . . . . . . . . .
`5.6.3
`Sub-picture Operation . . . . . . . . . . . . . . . . . . . . . . .
`5.6.4
`Operation Points. . . . . . . . . . . . . . . . . . . . . . . . . . .
`5.6.5
`Conformance Points . . . . . . . . . . . . . . . . . . . . . . . .
`5.6.6
`Signaling HRD Parameters in the VPS and SPS. . . . .
`Video Usability Information. . . . . . . . . . . . . . . . . . . . . . . . .
`5.7.1
`Geometrical Relations . . . . . . . . . . . . . . . . . . . . . . .
`5.7.2
`Video Signal Type and Color Information . . . . . . . . .
`5.7.3
`Frame/Field Indication . . . . . . . . . . . . . . . . . . . . . .
`5.7.4
`Default Display Window . . . . . . . . . . . . . . . . . . . . .
`5.7.5
`Timing Information . . . . . . . . . . . . . . . . . . . . . . . .
`5.7.6
`Bitstream Restrictions . . . . . . . . . . . . . . . . . . . . . . .
`Comparison to H.264 j AVC. . . . . . . . . . . . . . . . . . . . . . . . .
`5.8
`References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
`
`6.3
`6.4
`6.5
`
`Intra Prediction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
`6.1
`Prediction Mode and Prediction Block . . . . . . . . . . . . . . . . .
`6.2
`Reference Samples for Intra Prediction . . . . . . . . . . . . . . . . .
`6.2.1
`Reference Construction . . . . . . . . . . . . . . . . . . . . . .
`6.2.2
`Lowpass Smoothing . . . . . . . . . . . . . . . . . . . . . . . .
`Strong Smoothing for 32  32 Luma
`6.2.3
`Reference Samples . . . . . . . . . . . . . . . . . . . . . . . . .
`Planar Intra Prediction. . . . . . . . . . . . . . . . . . . . . . . . . . . . .
`DC Intra Prediction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
`Angular Intra Prediction . . . . . . . . . . . . . . . . . . . . . . . . . . .
`6.5.1
`One-Dimensional Prediction Reference . . . . . . . . . . .
`6.5.2
`Interpolated Prediction . . . . . . . . . . . . . . . . . . . . . .
`6.5.3
`Horizontal and Vertical Intra Prediction . . . . . . . . . .
`Signaling and Predictive Coding of Intra Prediction Modes . . .
`6.6.1
`Luma Intra Prediction Mode . . . . . . . . . . . . . . . . . .
`6.6.2
`Derivation of the Chroma Intra Prediction Mode . . . .
`Intra Coding Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
`6.7
`Comparison to H.264 j AVC. . . . . . . . . . . . . . . . . . . . . . . . .
`6.8
`References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
`
`6.6
`
`7
`
`Inter Prediction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
`7.1 Motion Compensated Prediction . . . . . . . . . . . . . . . . . . . . . .
`7.1.1
`Uni-prediction and Bi-prediction . . . . . . . . . . . . . . .
`7.1.2
`Coding Block Partitioning into Prediction Blocks . . . .
`7.1.3 Weighted Prediction . . . . . . . . . . . . . . . . . . . . . . . .
`
`aesciascia@mintz.com
`
`GE Video Exhibit 2001
`
`

`

`Contents
`
`7.3
`
`7.2 Motion Vector Representation . . . . . . . . . . . . . . . . . . . . . . .
`7.2.1 Motion Data Storage Reduction . . . . . . . . . . . . . . . .
`7.2.2 Merging Motion Vectors . . . . . . . . . . . . . . . . . . . . .
`7.2.3
`Predictive Motion Vector Coding . . . . . . . . . . . . . . .
`7.2.4
`Signaling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
`Sub-Sample Interpolation. . . . . . . . . . . . . . . . . . . . . . . . . . .
`7.3.1
`Luma Sub-Sample Interpolation Filtering . . . . . . . . .
`7.3.2
`Chroma Sub-Sample Interpolation Filtering . . . . . . . .
`7.3.3
`Derivation of the Interpolation Filter Coefficients. . . .
`Inter Coding Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . .
`Comparison to H.264 j AVC. . . . . . . . . . . . . . . . . . . . . . . . .
`7.5.1 Motion Vector Representation . . . . . . . . . . . . . . . . .
`7.5.2
`Sub-Sample Interpolation . . . . . . . . . . . . . . . . . . . .
`References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
`
`7.4
`7.5
`
`8
`
`9
`
`Residual Coding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
`8.1
`Transforms and Quantization . . . . . . . . . . . . . . . . . . . . . . . .
`8.1.1
`Integer DCTs . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
`Integer 4  4 DST . . . . . . . . . . . . . . . . . . . . . . . . .
`8.1.2
`8.1.3
`Dynamic Range and Transform Normalization. . . . . .
`8.1.4
`Quantizer Design . . . . . . . . . . . . . . . . . . . . . . . . . .
`8.1.5
`Quantizer Weighting Matrix . . . . . . . . . . . . . . . . . .
`8.1.6
`Decoder-Side Weighting and Level
`Scaling Operation . . . . . . . . . . . . . . . . . . . . . . . . . .
`Signaling of the Quantization Parameter . . . . . . . . . .
`8.1.7
`Coded Representation of Transform Blocks . . . . . . . . . . . . . .
`8.2.1
`Transform Sub-Blocks. . . . . . . . . . . . . . . . . . . . . . .
`8.2.2
`Last Significant Coefficient Position . . . . . . . . . . . . .
`8.2.3
`Transform Block Coefficient Coding . . . . . . . . . . . .
`8.2.4
`Sign Data Hiding . . . . . . . . . . . . . . . . . . . . . . . . . .
`Transform Skip . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
`8.3
`Transform and Quantization Bypass . . . . . . . . . . . . . . . . . . .
`8.4
`PCM Coding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
`8.5
`Comparison to H.264 j AVC. . . . . . . . . . . . . . . . . . . . . . . . .
`8.6
`References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
`
`8.2
`
`In-Loop Filtering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
`9.1
`Deblocking Filter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
`9.1.1
`Determination of Edges. . . . . . . . . . . . . . . . . . . . . .
`9.1.2
`Determination of the Deblocking Filter
`Strength Parameter . . . . . . . . . . . . . . . . . . . . . . . . .
`Deblocking Filtering . . . . . . . . . . . . . . . . . . . . . . . .
`Deblocking Filter Example . . . . . . . . . . . . . . . . . . .
`
`9.1.3
`9.1.4
`
`aesciascia@mintz.com
`
`xiii
`
`184
`184
`185
`188
`190
`191
`191
`192
`194
`197
`198
`198
`200
`202
`
`205
`206
`206
`210
`211
`213
`216
`
`218
`218
`219
`219
`220
`221
`223
`224
`224
`225
`225
`226
`
`229
`229
`230
`
`231
`233
`240
`
`GE Video Exhibit 2001
`
`

`

`xiv
`
`Contents
`
`9.2
`
`. . . . . . . . . . . . . . . . . . . . . . . . . . .
`Sample Adaptive Offset
`9.2.1
`Edge Offset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
`9.2.2
`Band Offset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
`9.2.3
`Signaling of SAO Parameters. . . . . . . . . . . . . . . . . .
`9.2.4
`SAO Filter Example . . . . . . . . . . . . . . . . . . . . . . . .
`9.2.5
`Encoder-Side Derivation of Sample Adaptive
`Offset Parameters . . . . . . . . . . . . . . . . . . . . . . . . . .
`Comparison to H.264 j AVC. . . . . . . . . . . . . . . . . . . . . . . . .
`9.3
`References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
`
`10 Entropy Coding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
`10.1 Fixed- and Variable-Length Coding . . . . . . . . . . . . . . . . . . .
`10.1.1 Fixed-Length Codes . . . . . . . . . . . . . . . . . . . . . . . .
`10.1.2 Exp-Golomb Codes. . . . . . . . . . . . . . . . . . . . . . . . .
`10.2 CABAC—Context-Based Adaptive Binary
`Arithmetic Coding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
`10.2.1 Process Overview . . . . . . . . . . . . . . . . . . . . . . . . . .
`10.2.2 Binary Arithmetic Coding . . . . . . . . . . . . . . . . . . . .
`10.2.3 Binarization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
`10.2.4 Context Initialization. . . . . . . . . . . . . . . . . . . . . . . .
`10.2.5 Context Selection . . . . . . . . . . . . . . . . . . . . . . . . . .
`10.3 Comparison to H.264 j AVC. . . . . . . . . . . . . . . . . . . . . . . . .
`References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
`
`11 Profiles, Tiers, and Levels . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
`11.1 Profiles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
`11.1.1 Main Profile . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
`11.1.2 Main 10 Profile . . . . . . . . . . . . . . . . . . . . . . . . . . .
`11.1.3 Main Still Picture Profile. . . . . . . . . . . . . . . . . . . . .
`11.2 Tiers and Levels. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
`11.3 Syntax Structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
`References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
`
`12 Extensions to HEVC. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
`12.1 Range Extensions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
`12.1.1 Proposed RExt Profiles . . . . . . . . . . . . . . . . . . . . . .
`12.1.2 Proposed RExt Tools . . . . . . . . . . . . . . . . . . . . . . .
`12.1.3 Comparison to H.264 j AVC. . . . . . . . . . . . . . . . . . .
`12.2 Common Specification Structure for Multi-Layer
`Video Coding Extensions . . . . . . . . . . . . . . . . . . . . . . . . . .
`12.2.1 Definition of Layers . . . . . . . . . . . . . . . . . . . . . . . .
`12.2.2 Proposed Joint Tools. . . . . . . . . . . . . . . . . . . . . . . .
`
`241
`242
`243
`244
`245
`
`245
`249
`250
`
`251
`252
`252
`253
`
`253
`254
`255
`266
`274
`275
`281
`282
`
`283
`284
`285
`285
`286
`286
`287
`289
`
`291
`292
`292
`293
`295
`
`295
`296
`297
`
`aesciascia@mintz.com
`
`GE Video Exhibit 2001
`
`

`

`Contents
`
`12.3 Multiview Coding. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
`12.3.1 Proposed Multiview Profile . . . . . . . . . . . . . . . . . . .
`12.3.2 Comparison to H.264 j AVC. . . . . . . . . . . . . . . . . . .
`12.4 Scalable Extension . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
`12.4.1 Proposed SHVC Profile. . . . . . . . . . . . . . . . . . . . . .
`12.4.2 Proposed SHVC Tools . . . . . . . . . . . . . . . . . . . . . .
`12.4.3 Comparison to H.264 j AVC. . . . . . . . . . . . . . . . . . .
`12.5 3D Video Coding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
`References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
`
`xv
`
`299
`300
`300
`301
`301
`301
`304
`305
`306
`
`Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
`
`309
`
`aesciascia@mintz.com
`
`GE Video Exhibit 2001
`
`

`

`Symbols and Operators
`
`Mathematical Functions and Operators
` j j
`Absolute value
`b c
`Round to lower integer value
`d e
`Round to higher integer value
`if v\a
`a;
`Clip{a,b,v} ¼
`if a  v  b
`v;
`if b\v
`b;
`
`Modulo operator
`¼ sgnðvÞ
`vj j þ 1
`2
`if v [ 0
`1;
`¼
`if v ¼ 0
`0;
`1
`if v\0
`
`mod
`round{v}
`
`sgn(v)
`
`
`
`8< :
`
`8< :
`
`Bit-Wise Operators
`
`&
`Bit-wise ‘and’ operation
`|
`Bit-wise ‘or’ operation
`x  y Bit-shift to the right of binary representation of x by y bits.
`Integer division by 2y, y C 0
`x  y Bit-shift to the left of binary representation of x by y bits.
`Integer multiplication with 2y, y C 0
`
`Variables
`
`Bd
`1id
`
`Bit depth
`Layer identifier in the NAL unit header
`
`xvii
`
`aesciascia@mintz.com
`
`GE Video Exhibit 2001
`
`

`

`xviii
`
`Symbols and Operators
`
`NL Number of lines in picture
`NP Number of pixels (samples) per line in picture
`tid
`Temporal sub-layer identifier in the NAL unit header
`
`Codes and Binarization
`Btp;nðvÞ Binarization of value v with type ‘tp’ and parameter n
`Ctp;nðvÞ VLC for value v of type ‘tp’ and parameter n
`tp;nðcÞ Return symbol value for code word c of VLC type ‘tp’ and parameter n
`Ci
`
`aesciascia@mintz.com
`
`GE Video Exhibit 2001
`
`

`

`Acronyms
`
`This list collects the acronyms used in this book. It further includes some acro-
`nyms which have been established during the work of the JCT-VC on HEVC, and
`others which are commonly used in the context of video coding. Marked acronyms
`(*) indicate deprecated naming conventions or refer to tools which have not been
`adopted into the HEVC specification. If acronyms relate to the H.264 | AVC
`specification, a corresponding indication is given. For acronyms that are used in
`this book, a reference to the relevant chapter or section is provided to facilitate
`access.
`
`3D-HEVC
`AAC
`AAP
`AhG
`AI
`AIF*
`ALF*
`AMP
`AMVP
`ANG*
`APS
`ASO
`AU
`AUD
`AVC
`BLA
`BD
`BL
`BO
`BoG
`BP
`
`3D high efficiency video coding (Sect. 12.5)
`Advanced audio coding
`Alternative Approval Process (Sect. 1.3.2)
`Ad hoc group
`All intra (JCT-VC CTC, Sect. 3.6.1.1)
`Adaptive interpolation filter
`Adaptive loop filter
`Asymmetric motion partitioning (Sect. 7.1.2)
`Advanced motion vector prediction (Sect. 7.2.3)
`Angular intra prediction (see Sect. 6.5)
`Adaptation parameter set (Sect. 5.3.1)
`Arbitrary slice ordering (H.264 | AVC, Sect. 4.2.7)
`Access unit (Sect. 5.2.3)
`Access unit delimiter (Sect. 5.2.2)
`Advanced video coding
`Broken link access [picture] (Sect. 4.1.2.1)
`Bjøntegaard delta measurement (Sect. 3.6.1.3)
`Base layer (Sect. 12.4)
`SAO band offset (Sect. 9.2)
`Break-out group (Sect. 1.4.4)
`Bandpass [filter]
`
`xix
`
`aesciascia@mintz.com
`
`GE Video Exhibit 2001
`
`

`

`xx
`
`BPB
`CABAC
`CAVLC
`CB
`CBF*
`CBP
`CBR
`CD
`CD
`CE
`CfE
`CfP
`CIE
`CIF
`CPB
`CRA
`CRFB*
`CRT
`CS
`CSS
`CT
`CTB
`CTC
`CTU
`CTX
`CU
`CVS
`DAM
`DC
`DCT
`DCTIF
`DIF*
`DIF*
`DIS
`DLP*
`DMVD
`DPB
`DPCM
`DST
`DU
`DUT*
`DVD
`EG
`EL
`EO
`
`Acronyms
`
`Bitstream partition buffer (Sect. 12.2)
`Context-based adaptive binary arithmetic coding (Sect. 10.2)
`Context adaptive variable length coding (Sect. 2.4.8)
`Coding block (Sect. 4.2.4)
`Coded block flag (see Sect. 8.2.1)
`Coded block pattern (H.264 | AVC, Sect. 4.2.7)
`Constant bitrate (Sect. 5.6.1)
`Committee draft (Sect. 1.3.1)
`Compact disk
`Core experiment (Sect. 1.4.4)
`Call for evidence (Sect. 1.3.1)
`Call for proposals (Sect. 1.3.1)
`Commission internationale de l’éclairage (Sect. 2.3.1)
`Common interchange format 352 9 288 (Sect. 11.2)
`Coded picture buffer (Sect. 5.6.1)
`Clean random access point (Sect. 4.1.2.1)
`Compressed reference frame buffer (see Sect. 7.2.1)
`Cathode ray tube
`Constraint set (in CfP)
`Coded slice segment (Sect. 5.2.3)
`Collaborative team (Sect. 1.4.1)
`Coding tree block (Sect. 4.2.4)
`Common testing conditions (Sect. 3.6.1)
`Coding tree unit (Sect. 4.2.4)
`CABAC context (Sect. 10.2)
`Coding unit (Sect. 4.2.4)
`Coded video sequence (Sect. 4.1)
`Draft amendment (Sect. 1.3.1)
`Direct current
`Discrete cosine transform (Sect. 2.4.5.2)
`DCT interpolation filter (Sect. 7.3.3)
`DCT interpolation filter, syn. DCTIF (Sect. 7.3.3)
`Directional interpolation filter
`Draft international standard (Sect. 1.3.1)
`Decodable leading picture, syn. RADL (Sect. 4.1.2.2)
`Decoder side motion vector derivation
`Decoded picture buffer (Sect. 5.6.2)
`Differential pulse code modulation
`Discrete sine transform (Sect. 2.4.5.3)
`Decoding unit (Sect. 5.2.4)
`Directional unified transform (see Sect. 8.1.2)
`Digital versatile disk
`Exp-Golomb code (Sect. 2.4.8.4)
`Enhancement layer (Sect. 12.4)
`SAO edge offset (Sect. 9.2)
`
`aesciascia@mintz.com
`
`GE Video Exhibit 2001
`
`

`

`Acronyms
`
`EOB
`EOS
`EOTF
`FCD
`FD
`FDAM
`FDIS
`FLC
`FMO
`fps
`FRExt
`GRD
`GOP
`HD
`HDTV
`HEVC
`HM
`HP
`HRD
`HSS
`HTM
`HVC*
`
`HVS
`IBDI*
`IDCT
`IDR
`IEC
`IEEE
`IRAP
`IS
`ISDN
`ISO
`ITU
`JCT
`JCT-VC
`JCT-3V
`JM
`JPEG
`JTC
`JVT
`KLT
`KTA
`LCTB*
`LCTB*
`
`xxi
`
`End of bitstream (Sect. 5.2.2)
`End of sequence (Sect. 5.2.2)
`Electro-optical transfer function (Sect. 2.3.3)
`Final committee draft (Sect. 1.3.1)
`Filler data (Sect. 5.2.2)