rearanne
`0 MULTIMEDIA.
`ENGINEERING
`Fundamentals,
`Algorithms, and ates
`
`Huitang Sun © IPR2018-01413
`
`Yun Q. Shi
`
`.
`
`
`
`Sony EX1008 P
`
`1
`
`IPR2018-01413
`Sony EX1008 Page 1
`
`

`

`Sony EX1008 Page 2
`
`IPR2018-01413
`
`IPR2018-01413
`Sony EX1008 Page 2
`
`

`

`Sony EX1008 Page 3
`
`IPR2018-01413
`
`IPR2018-01413
`Sony EX1008 Page 3
`
`

`

`Sony EX1008 Page 4
`
`IPR2018-01413
`
`IPR2018-01413
`Sony EX1008 Page 4
`
`

`

`IMAGEand VIDEO
`COMPRESSION
`for MULTIMEDIA
`ENGINEERING
`Fundamentals,
`Algorithms, and Standards
`
`IPR2018-01413
`Sony EX1008 Page 5
`
`IPR2018-01413
`Sony EX1008 Page 5
`
`

`

`IMAGE PROCESSING SERIES
`Series Editor: Phillip A. Laplante
`
`Forthcoming Titles
`
`Adaptive Image Processing: A ComputationalIntelligence
`Perspective
`Ling Guan, Hau-San Wong, and Stuart William Perry
`Shape Analysis and Classification: Theory and Practice
`Luciano da Fontoura Costa and Roberto Marcondes Cesar, Jr.
`
`IPR2018-01413
`Sony EX1008 Page 6
`
`IPR2018-01413
`Sony EX1008 Page 6
`
`

`

`[MAGE and VIDEO
`COMPRESSION
`for MULTIMEDIA
`ENGINEERING
`Fundamentals,
`Algorithms, and Standards
`
`Yun Q. Shi
`NewJersey Institute of Technology
`Newark, NJ
`Huitang Sun
`Mitsubishi Electric Information Technology Center
`America Advanced Television Laboratory
`New Providence, NJ
`
`CRC Press
`Boca Raton London NewYork Washington, D.C.
`
`IPR2018-01413
`Sony EX1008 Page 7
`
`IPR2018-01413
`Sony EX1008 Page 7
`
`

`

`Library of Congress Cataloging-in-Publication Data
`
`3, Video compression
`
`Shi, Yun Q.
`Image and video compression for multimedia engineering : fundamentals, algonthms, and standards /
`Yun Q. Shi, Huifang Sun.
`p.
`cm.
`Includes bibliographical references and index.
`ISBN 0-8493-3491-8 (alk. paper)
`1. Multimedia systems.
`2. Image compression,
`QA76.575.5555 1999
`006.7—dc21
`
`—‘|. Sun, Huifang.
`
`II. Title.
`
`99-047137
`
`This book contains information obtained from authentic and highly regarded sources. Reprinted material is quoted with
`permission, and sourcesare indicated. A wide varietyofreferencesare listed. Reasonableefforts have been made to publish
`reliable data and information, but the author and the publisher cannot assume responsibility for the validity ofall materials
`or for the consequencesof their use.
`Neither this book nor any part may be reproduced or transmitted in any form or by any means, electronic or inechanical,
`including photocopying, microfilming, and recording, or by any information storage or retrieval system, without pnor
`permission in wnting from the publisher,
`The consent of CRC Press LLC does not extend to copying for general distribution, for promotion, for creating new
`works, or for resale. Specific permission must be obtained in wniting from CRC Press LLC for such copying.
`Direct all inquiries to CRC Press LLC, 2000 Corporate Blvd., N.W., Boca Raton, Florida 33431.
`
`Trademark Notice: Product or corporate names may be trademarksor registered trademarks, and are only used for
`identification and explanation, without intent to infringe.
`
`© 2000 by CRC Press LLC
`
`No claim to onginal U.S. Government works
`International Standard Book Number 0-8493-3491-8
`
`Library of Congress Card Number 99-047137
`Printed in the United States of Amenca | 234567890
`Printed on acid-free paper
`
`IPR2018-01413
`Sony EX1008 Page 8
`
`IPR2018-01413
`Sony EX1008 Page 8
`
`

`

`SSSSSSa
`
`Preface
`
`[tis well knownthat in the 1960s the advent of the semiconductor computerand the space program
`swiftly brought the field ofdigital image processing into public focus. Since then the field has
`experienced rapid growth and has entered into every aspect of modern technology. Since the early
`1980s, digital
`image sequence processing has been an attractive research area because an image
`sequence, as a collection of images, may provide more information than a single image frame. The
`increased computational complexity and memory space required for image sequence processing
`are becoming more attainable, This is due to more advanced, achievable computational capability
`resulting from the continuing progress made in technologies, especially those associated with the
`VLSIindustry and information processing.
`In addition to image and image sequence processing in the digitized domain, facsimile trans-
`mission has switched from analog to digital since the 1970s. However, the conceptofhigh definition
`television (HDTV) whenproposed in the late 1970s and early 1980s continued to be analog. This
`has since changed, In the U.S., the first digital system proposal for HDTV appeared in 1990. The
`Advanced Television Standards Committee (ATSC), formed by the television industry, recom-
`mended the digital HDTY system developed jointly by the seven Grand Alliance members as the
`standard, which was approved by the Federal Communication Commission (FCC) in 1997. Today's
`worldwide prevailing concept of HDTV 1s digital. Digital television (DTV) provides the signal that
`can be used in computers. Consequently,
`the marriage of TV and computers has begun, Direct
`broadcasting by satellite (DBS), digital video disks (DVD), video-on-demand (VOD), video games,
`and other digital video related media and services are available now, or soon will be.
`As in the case of image and video transmission andstorage, audio transmission and storage
`through some media have changed from analog to digital. Examples include entertainment audio
`on compact disks (CD) and telephone transmission over long and medium distances. Digital TV
`signals, mentioned above, provide another example since they include audio signals, ‘Transmission
`and storage of audio signals through some other media are about to change to digital, Examples
`ofthis include telephone transmission through local area and cable TY.
`Although most signals generated from various sensors are analog in nature, the switching from
`analog to digital is motivated by the superiority of digital signal processing and transmission over
`their analog counterparts, The principal advantage ofthe digital signal
`is ils robustness against
`various noises. Clearly, this results from the fact that only binary digits exist in digital format and
`it 1s mucheasier to distinguish one state from the other than to handle analog signals.
`Anotheradvantageofbeing digital is ease of signal manipulation. In addition to the development
`ofa variety ofdigital signal processing techniques (including image, video, andaudio) andspecially
`designed software and hardware that may be well known, the following development Is an example
`of this advantage. The digitized information format,
`i-e.,
`the bitstream, often in a compressed
`version,
`is a revolutionary change in the video industry that enables many manipulations which
`are either impossible or very complicated to execute in analog format. Forinstance, video, audio,
`and other data can befirst compressed to separate bitstreams and then combined to form a signal
`bitstream, thus providing a multimedia solution for many practical applications. Information from
`different sources and to different devices can be multiplexed and demultiplexed in terms ofthe
`bitstream. Bitstream conversion in terms ofbit rate conversion, resoluuion conversion, and syntax
`conversion becomes feasible. In digital video, content-based coding, retrieval, and manipulation
`and the ability to edit video in the compressed domain becomefeasible. All system-timing signals
`
`IPR2018-01413
`Sony EX1008 Page 9
`
`IPR2018-01413
`Sony EX1008 Page 9
`
`

`

`in the digital systems can be included in the bitstream instead of being transmitted separately as
`in traditional analog systems.
`The digital format is well suited to the recent development of modern telecommunication
`structures as exemplified by the Internet and World Wide Web (WWW), Therefore, we can seethat
`digital computers, consumerelectronics (including television and video games), and telecommu-
`nications networks are combined to produce an information revolution, By combining audio, video,
`and other data, multimedia becomesanindispensable element of modern life, While the pace and
`the future of this revolution cannot be predicted, one thing is certain:
`this process is going to
`drastically change many aspects of our world in the next several decades.
`One of the enabling technologies in the information revolution is digital data compression,
`since the digitization of analog signals causes data expansion.
`In other words, storage and/or
`transmission ofdigitized signals require more storage space and/or bandwidth than the original
`analog signals.
`The focus of this book is on image and video compression encountered in multimedia engi-
`neering. Fundamentals, algorithms, and standardsare the three emphases ofthe book, It is intended
`lo serve as a senior/graduate-level text, Its material is sufficient for a one-semester or one-quarter
`graduate course on digital image and video coding. For this purpose, at the end of each chapter
`there is a section of exercises containing problems and projects for practice, and a section of
`references for further reading.
`Based onthis book, a short course entitled “Image and Video Compression for Multimedia,”
`was conducted at Nanyang Technological University, Singapore in March and April, 1999. The
`response (o the short course was overwhelminglypositive.
`
`IPR2018-01413
`Sony EX1008 Page 10
`
`IPR2018-01413
`Sony EX1008 Page 10
`
`

`

`
`
`Authors
`
`Dr. Yun Q.Shi has been a professor with the Department of Electrical and Computer Engineering
`al the New Jersey Institute of Technology, Newark, NJ since 1987. Before that he obtained his B.S.
`degree in Electronic Engineering and M.S. degree in Precision Instrumentation fromthe Shanghai
`Jiao Tong University, Shanghai, China and his Ph.D. in Electrical Engineering from the University
`of Pittsburgh. His research interests include motion analysis from image sequences, video coding
`and transmission, digital
`inyage watermarking, computer vision, applications of digital
`image
`processing and pattern recognition to industrial automation and biomedical engineering, robust
`stability, spectral factorization, multidimensional systems andsignal processing. Prior to entering
`graduate school, he worked in a radio factory as a design and test engineer in digital control
`manufacturing and in electronics.
`He is the author or coauthor of about 90 journal and conference proceedings papers in his
`research areas and has been a formal reviewer of the Mathematical Reviews since 1987, an IEEE
`senior member since 1993, and the chairman of Signal Processing Chapter of IEEE North Jersey
`Section since 1996. He was an associate editor for /EEE Transactions on Signal Processing
`responsible for Multidimensional Signal Processing from 1994 to 1999, the guest editor of the
`special issue on Image Sequence Processing for the /nternarional Journal of Imaging Systems ane
`Technology, published as Volumes 9,4 and 9.5 in 1998, one of the contributing authors in the area
`of Signal and Image Processing to the Comprehensive Dictionary of Electrical Engineering, pub-
`lished by the CRC Press LLC in 1998. His biography has been selected by Marquis Who's Who
`for inclusion in the 2000 edition of Who's Whe in Science and Engineering.
`
`Dr. Huifang Sun received the B.S. degree in Electrical Engineering from Harbin Engineering
`Institute, Harbin, China, and the Ph.D.in Electrical Engineering from University of Ottawa, Ottawa,
`Canada. In 1986 he jointed Fairleigh Dickinson University, Teaneck, NJ as an assistant professor
`and was promoted to an associate professor in electrical engineering. From 1990 to 1995, he was
`with the David Sarnoff Research Center (Sarnoff Corp.) in Princeton as a memberof technical
`staff and later promoted to technology leader of Digital Video Technology where his activities
`included MPEGvideo coding, AD-HDTY, and Grand Alliance HDTV development. He joinedthe
`Advanced Television Laboratory, Mitsubishi Electric Information Technology Center America
`(ITA), New Providence, NJ in 1995 as a senior principal technical staff and was promotedto deputy
`director in 1997 working in advancedtelevision development and digital video processing. He has
`been active in MPEG video standards for many years and holds 10 U.S. patents with several
`pending. He has authored or coauthored more than 80 journal and conference papers and obtained
`the 1993 best paper award of IEEE Transactions on Consumer Electronics, and 1997 best paper
`award of International Conference on Consumer Electronics. For his contributions to HDTV
`development, he obtained the 1994 Sarnoff (echnical achievement award. He ts currently the
`associate editor of JEEE Transactions on Circuits and Systemsfor Video Technology.
`
`IPR2018-01413
`Sony EX1008 Page 11
`
`IPR2018-01413
`Sony EX1008 Page 11
`
`

`

`
`
`Acknowledgments
`
`Weare pleased to express our gratitude here for the support and help we received in the course of
`writing this book.
`The first author thankshis friend and former colleague, Dr. C. Q. Shu, for fruitful technical
`discussions related to some contents of the book. Sincere thanks also are directed to several of his
`friends and former students, Drs. J. N. Pan, X. Xia, S. Lin, and Y. Shi, for their technical contri-
`butions and computer simulations related to some subjects of the book. He is grateful to Ms. L.
`Fitton for her English editing of 11 chapters, and to Dr. Z. F. Chen for her help in preparing many
`graphics.
`The second author expresses his appreciation to his colleagues, Anthony Vetro and Ajay
`Divakaran,for fruitful technical discussion related to some contents ofthe book and for proofreading
`nine chapters. He also extends his appreciation to Dr. Xiaobing Lee for his helpin providing some
`useful references, and to manyfriends and colleagues of the MPEGers who provided wonderful
`MPEG documents and tutorial materials that are cited in some chapters ofthis book. He also would
`like to thank Drs. Tommy Poor, Jim Foley, and Toshiaki Sakaguchi for their continuing support
`and encouragement.
`Both authors would liketo expresstheir deep appreciation to Dr, Z. F, Chen for her great help
`in formatting all the chapters ofthe book. They also thank Dr. F. Chichesterfor his help in preparing
`the book.
`Special thanks go to the editor-in-chief of the Image Processing book series of CRC Press,
`Dr. P. Laplante, for his constant encouragement and guidance. Help from the editors at CRC Press,
`N, Konopka, M. Mogck,and otherstaff, is appreciated,
`The first author acknowledges the Support he received associated with writing this book from
`the Electrical and Computer Engineering Department at the New Jersey Institute of Technology.
`In particular, thanks are directed to the departmentchairman, Professor R. Haddad, and the associate
`chairman, Professor K. Sohn. Heis also grateful to the Division of Information Engineering and
`the Electrical and Electronic Engineering School at Nanyang Technological University (NTU),
`Singapore for the support he received during his sabbatical
`leave.
`It was in Singapore that he
`finished writing the manuscript. In particular, thanks go to the dean of the school, Professor Er
`Meng Hwa,andthedivision head, Professor A. C. Kot. With pleasure, he expresses his appreciation
`to many ofhis colleagues at the NTU for their encouragement and help. In particular, his thanks
`go to Drs. G. Li and J. S. Li, and Dr. G. A. Bi. Thanks are also directed to many colleagues,
`graduate students, and sometechnicalstaff from industrial companies in Singapore who attended
`the short course which was based on this book in March/April 1999 and contributed their enthu-
`Siastic support and somefruitful discussion.
`Last butnotleast, both authors thank their families for their patient support during the course
`of the writing. Without their understanding and support we would not have been able to complete
`this book.
`
`Yun Q. Shi
`Huifang Sun
`
`IPR2018-01413
`Sony EX1008 Page 12
`
`IPR2018-01413
`Sony EX1008 Page 12
`
`

`

`
`
`Content and Organization
`of the Book
`
`The entire book consists of 20 chapters which can be grouped into four sections:
`
`I. Fundamentals,
`Il. Still Image Compression,
`III. Motion Estimation and Compensation, and
`IV. Video Compression.
`
`In the following, we summarize the aim and content of each chapter and each part, and the
`relationships between some chapters and between the four parts.
`Section I includes the first six chapters. It provides readers with a solid basis for understanding
`the remaining three parts of the book.
`In Chapter 1,
`the practical needs for image and video
`compression is demonstrated. The feasibility of image and video compression is analyzed. Specif-
`ically, both statisucal and psychovisual redundancies are analyzed and the removal ofthese redun-
`dancies leads to image and video compression. In the course of the analysis, some fundamental
`characteristics of the human visual system are discussed. Visual quality measurement as another
`important concept in the compression is addressed in both subjective and objective quality measures.
`The new trend in combining the virtues of the two measures also is presented, Some information
`theory results are presented as the final subject of the chapter.
`Quantization, as a crucial step in lossy compression, is discussed in Chapter 2. It is known that
`quantization has a direct impact on both the coding bit rate and quality of reconstructed frames.
`Both uniform and nonuniform quantization are covered. The issues of quantization distortion,
`optimum quantization, and adaptive quantization are addressed, The final subject discussed in the
`chapteris pulse code modulation (PCM) which, as the earliest, best-established, and most frequently
`applied coding system normally serves as a standard against which other coding techniques are
`compared.
`Twoefficient coding schemes, differential coding and transform coding (TC), are discussed in
`Chapters 3 and 4, respectively. Both techniques utilize the redundancies discussed in Chapter|.
`ihus achieving data compression, In Chapter 3, the formulation of general differential pulse code
`modulation (DPCM) systems is describedfirst, followed by discussions of optimum linear predic-
`lion and several implementationissues. Then, delta modulation (DM), an important, simple, special
`case of DPCM, is presented. Finally, application of the differential coding technique to interframe
`coding and information-preserving differential coding are covered.
`Chapter 4 begins with the introduction of the Hotelling transform, the discrete version ofthe
`optimum Karhunen and Loevetransform. Throughstatistical, geometrical, and basis vector (image)
`interpretations, this introduction providesa solid understanding ofthe transform coding technique.
`Several linear unitary transformsare thenpresented, followed by performance comparisons between
`these transforms in terms of energy compactness, mean square reconstruction error, and compula-
`tional complexity. It is demonstrated that the discrete cosine transform (DCT) performs better than
`others,
`in general. In the discussion of bit allocation, anefficient adaptive scheme is presented
`using thresholding coding devised by Chen and Pratt in 1984, which established a basis for the
`internationalstill image coding standard, Joint Photographic (image) Experts Group (JPEG). The
`
`IPR2018-01413
`Sony EX1008 Page 13
`
`IPR2018-01413
`Sony EX1008 Page 13
`
`

`

`comparison between DPCM and TCis given. The combination of these two techniques (hybrid
`transform/waveform coding), and ils application in image and video coding also are described.
`The last two chapters in the first part cover some coding (codeword assignment) techniques.
`In Chapter 5,
`two types of variable-length coding techniques, Huffman coding and arithmetic
`coding, are discussed. First, an introduction to some basic coding theory is presented, which can
`be viewed as a continuation of the information theory results presented in Chapter |. Then the
`Huffmancode,as an optimum andinstantaneous code, and a modified version are covered. Huffman
`coding is a systematic procedure for encoding a source alphabet with each source symbol having
`an occurrence probability. As a block code (a fixed codeword having an integer number ofbits ts
`assigned to a source symbol), it is optimum in the sense that it produces minimumcoding redun-
`dancy. Some limitations of Huffman coding are analyzed. As a stream-based coding technique,
`arithmetic codingis distinct fromand is gaining more popularity than Huffman coding. It maps a
`string of source symbols into a string of code symbols, Free of the integer-bits-per-source-symbo]
`restriction, arithmetic coding is moreefficient. The principle of arithmetic coding and someol its
`implementation issues are addressed.
`While the two types of variable-length coding techniques introduced in Chapter 5 can be
`classified as fixed-length to variable-length coding techniques, both run-length coding (RLC) and
`dictionary coding, discussed in Chapter6, can be classified as variable-length to fixed-length coding
`techniques. The discrete Markoy source model(another portion ofthe information theory results)
`that can be used to characterize 1-D RLC, is introduced at the beginning of Chapter 6. Both |-D
`RLC and 2-D RLC are then introduced. The comparison between 1-D and 2-D RLC is made in
`terms of coding efficiency and transmission error effect. The digital facsimile coding standards
`based on 1-D and 2-D RLCare introduced. Another focus of Chapter 6 is on dictionary coding.
`Two groupsof adaptive dictionary coding techniques, the LZ77 and LZ78algorithms, are presented
`and their applications are discussed. Atthe end ofthe chapter, a discussionofinternational standards
`for lossless still image compression is given. For both lossless bilevel and multilevel still image
`compression,the respective standardalgorithms and their performance comparisons are provided.
`Section IT of the book (Chapters7, 8, and 9) is devotedto still image compression. In Chapter7.
`the international still image coding standard, JPEG, is introduced, Two classes of encoding: lossy
`and lossless; and four modes of operation: sequential DCT-based mode, progressive DCT-based
`mode, lossless mode, and hierarchical mode are covered. The discussion in the first part of the
`book is very useful in understanding whatis introduced here for JPEG.
`Dueto its higher codingefficiency and superior spatial and quality scalability features over the
`DCTcoding technique, the discrete wavelet transform (DWT) coding has been adopted by JPEG-
`2000still image coding standards as the core technology. Chapter 8 begins with an introduction to
`wavelet transform (WT), which includes a comparison between WT and the short-time Fourier
`transform (STFT), and presents WT asa unification of several existing techniques known as filter
`bank analysis, pyramid coding, and subband coding. Then the DWT for still
`image coding is
`discussed. In particular, the embedded zerotree wavelet (EZW) technique and set partitioning in
`hierarchical trees (SPIHT) are discussed. The updated JPEG-2000 standard activity is presented.
`Chapter 9 presents three nonstandard still image coding techniques: vector quantization (VQ),
`fractal, and model-based image coding. All three techniques have several important features such
`as very high compressionratios for certain kinds of images, and very simple decoding procedures.
`Due to some limitations, however, they have not been adopted by thestill image coding standards.
`On the other hand, the facial model and face animation technique have been adopted by the MPEG-4
`video standard.
`Section III, consisting of Chapters 10 through 14, addresses the motion estimation and motion
`compensation — keyissues in modern video compression.In this sense, Section III is a prerequisite
`to Section IV, which discusses various video coding standards. The first chapter in Section III,
`Chapter 10, introduces motion analysis and compensation in general. The chapter begins with the
`concept of imaging space, which characterizes all images andall image sequences in temporal and
`
`IPR2018-01413
`Sony EX1008 Page 14
`
`IPR2018-01413
`Sony EX1008 Page 14
`
`

`

`image sequences are special proper subsets of the
`spatial domains. Both temporal and spatial
`imaging space. A single image becomes merely a specific cross section of the imaging space. Two
`techniques in video compression utilizing interframe correlation, both developed in the late !960s
`and early 1970s, are presented. Frame replenishment is relatively simpler in modeling and imple-
`mentation. However, motion compensated coding achieves higher coding efficiency and better
`quality in reconstructed frames with a 2-D displacement model. Motion analysis is then viewed
`from the signal processing perspective. Three techniques in motion analysis are briefly discussed,
`They are block matching, pel
`recursion, and optical
`flow, which are presented in detail
`in
`Chapters 1], 12, and 13, respectively. Finally, other applications of motion compensation to image
`sequence processing are discussed.
`Chapter 11 addresses the block matching technique, which presently is the most frequently
`used motion estimation technique, The chapterfirst presents the original block matching technique
`proposed by Jain and Jain. Several different matching criteria and search strategies are then
`discussed. A thresholding multiresolution block matching algorithm ts described in some detail so
`as lo provide an insight into the technique. Then, the limitations of block matching techniques are
`analyzed, from which several new improvements are presented. They include hierarchical block
`matching, multigrid block matching, predictive motion field segmentation, and overlapped block
`matching. All of these techniques modify the nonoverlapped, equally spaced,
`fix-sized, small
`rectangular block model proposed by Jain and Jain in some way so that the motion estimation is
`more accurate and has fewer block artifacts and less overhead side information.
`The pel recursive techniqueis discussed in Chapter 12. First, determination of 2-D displacement
`vectors is converted via the use ofthe displaced frame difference (DFD) concept to a minimization
`problem, Second, descent methods tn optimization theory are discussed. In particular, the steepest
`descent method and Newton-Raphson method are addressed in terms of algorithm, convergence,
`and implementation issues such as selection of step-size and initial value. Third,
`the first pel
`recursive techniques proposed by Netravali and Robbins are presented. Finally, several improvement
`algorithms are described.
`Optical flow,the third technique in motion estimationfor video coding, is covered in Chapter 13.
`First, some fundamental issues in motion esumation are addressed, They include the difference
`and relationships between 2-D motion and optical flow,
`the aperture problem, and the ill-posed
`nature of motion estimation. The gradient-based and correlation-based approaches to optical flow
`determination are then discussed in detail. For the former,
`the Horn and Schunck algorithm is
`illustrated as a representative technique and some other algorithms are briefly introduced. For the
`latter, the Singh method is introduced as a representative technique. In particular, the concepts of
`conservation information and neighborhood information are emphasized. A correlation-feedback
`algorithm is presented in detail (o provide an insight into the correlation technique. Finally, multiple
`attributes for conservation information are discussed.
`Chapter 14, the last chapter in Section III, provides a further discussion and summary of 2-D
`motion estimation. First, a few features common to all
`three major techniques discussed in
`Chapters 11, 12, and 13 are addressed. They are the aperture and 1ll-posed inverse problems,
`conservation and neighborhood information, occlusion and disocclusion, rigid and nonrigid motion.
`Second,a variety ofdifferent classifications of motion estimation techniques is presented. Frequency
`domain methods are discussed as well. Third, a performance comparison between the three major
`techniques in motionestimation is made, Finally, the new trends in motion estimationare presented.
`Section IV, discussing various video coding standards, is covered in Chapters 15 through 20.
`Chapter 15 presents fundamentals of video coding,First, digital video representation is discussed.
`Second, the rate distortion function of the video signal
`is covered — the fourth portion of the
`information theory results presented in this book. Third, various digital video formats are discussed.
`Finally,
`the current digital image/video coding standards are summarized. The full names and
`abbreviations of some organizations,
`the completion time, and the major features of various
`image/video coding standards are listed in two tables.
`
`IPR2018-01413
`Sony EX1008 Page 15
`
`IPR2018-01413
`Sony EX1008 Page 15
`
`

`

`Chapter 16 is devoted to video coding standards MPEG-1/2, which are the most widely used
`video coding standardsat the present. The basic technique of MPEG-1/2 is a full-motion-compen-
`sated DCT and DPCM hybrid coding algorithm. The features of MPEG-1 (including layered data
`structure) and the MPEG-2 enhancements(including field/frame modes for supportingthe interlaced
`video input and scalability extension) are described. Issues ofrate control, optimum modedecision,
`and multiplexing are discussed.
`Chapter 17 presents several application examples of MPEG-1/2 video standards. They are the
`ATSC DTV standard approved by the FCCin the U.S., transcoding, the down-conversion decoder,
`and error concealment. Discussion of these applications can enhance the understanding and mas-
`tering of MPEG-1/2 standards. Some research work is reported that may be helpful for graduate
`students to broaden their knowledgeof digital video processing — an active research field.
`Chapter 18 presents the MPEG-4 video standard. The predominant feature of MPEG-4, content-
`based manipulation,
`is emphasized. The underlying concept of audio/visual objects (AVOs) Is
`introduced. The important functionalities of MPEG-4: content-based interactivity (including bit-
`stream editing, synthetic and natural hybrid coding [SNHC]), content-based coding efficiency, and
`universal access (including content-based scalability), are discussed. Since neither MPEG-! nor
`MPEG-2 includes synthetic video and content-based coding,
`the most important application of
`MPEG-4 is in a multimedia environment.
`Chapter 19 introduces ITU-T video coding standards H.261 and H.263, which are utilized
`mainly for videophony and videoconferencing. The basic technical details of H.261, the earliest
`video coding standard, are presented. The technical improvements by which H.263 achieves high
`coding efficiency are discussed, Features of H.263+, H.263++, and H.26L are presented.
`Chapter 20 covers the systems part of MPEG — multiplexing/demultiplexing and synchronizing
`the coded audio and video as well as other data, Specifically, MPEG-2 systems and MPEG-4
`systems are introduced. In MPEG-2 systems, two forms: Program Stream and Transport Stream,
`are described. In MPEG-4 systems, some multimedia application related issues are discussed.
`
`IPR2018-01413
`Sony EX1008 Page 16
`
`IPR2018-01413
`Sony EX1008 Page 16
`
`

`

`
`
`Contents
`
`Section | Fundamentals
`
`Introduction
`Chapter |
`1.1
`Practical Needs for Image and Video COMPLeSsiON,........ccccecsecssensecnsetccuesseneenteessteneeeenesentesa
`1.2
`Feasibility of Image and Video Compression ......c.ccccsssssesscsetsesenetecersenesecsteenerseeceeteeeeeatenses4
`Teoh
`Statistical MedWndaney ys acocsxeeecmtse-vesyscertavse eigiracsanennensansaeaeter<@arersae*accaat onceTike Jets4
`1.222
`Psychovisual\ Redundancy <i.cccccsese:sccsctassessscastecsecnysunechncsesatsoyceverectapsyentzsensevsssetasseetisoas9
`P53) Wistial) Quality, Measurements cc.csta cee ccacseuusessecusvercnsasevessocesaccsteiecavsbesuekennp atvearansieebaleuti crs duranasys 18
`1.3.1
`Subjective Quality Measurement.....scscsscsrscasseessersesseresssnsnsstenseneseseesesseenspueueaneanearseer 1D
`1.3.2 Objective Quality Measurement..i-.c0...sseoccuvsesseerecdscoseentesnrserreseeeecansdecdneaneaaneennanseterin2)
`‘Information Theory Results. cccc.cccjcccsseccstasecesssstosostccecvessnauacesdesterpsanstenevsensareydh-searcaites begs edo
`TASTY
`SIESTA Opyi reccesteses erro rude tesesens va ksaelstanpicasststeacson- ey cy mecuetveoesmeae cmvne venta
`Sogia easter eee
`1.4.2
`Shannon’s Noiseless Source Coding Theorem...........:ccccscssunienertsensseestesnereesensenees 25
`1.4.3
`Shannon’s Noisy Channel Coding Theorem.......-2--..:::ccescseceseseneeerenserentseresssneceeene26
`1.4.4
`Shannon’s Source Coding Theorem.......c::.-c:cceccccsessseseneessesenessrensenrerereanserrensrsensenses27
`1.4.5
`Information Transmission TheOrem.....cccsceceesseccereesssessecsecencesecesessusnseeeecareeresensenreness27
`SUIMMALY...-ceccsecsesessscvensnsenscersersenssresesnsnencansssesensuns