(12) United States Patent
`Dattani et al.
`
`(10) Patent No.:
`(45) Date of Patent:
`
`US 7,010,044 B2
`Mar. 7, 2006
`
`USOO701 004.4B2
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`2003/0223495 A1 12/2003 Sun et al. .............. 375/240.12
`
`* cited b
`cited by examiner
`
`(54) INTRA 4x4 MODES3, 7 AND 8
`AVAILABILITY DETERMINATION INTRA
`ESTMATION AND COMPENSATION
`(75) Inventors: Doni S. Dattani, Waterloo (CA);
`Lowell L. Winger, Waterloo (CA);
`Simon Booth, Kitchener (CA)
`(73) Assignee: LSI Logic Corporation, Milpitas, CA
`Primary Examiner Young Lee
`(US)
`Subject to any disclaimer, the term of this Sp Attorney, Agent, or Firm-Christopher P. Maiorana,
`patent is extended or adjusted under 35
`--
`U.S.C. 154(b) by 268 days.
`(21) Appl. No.: 10/623,329
`(22) Filed:
`Jul. 18, 2003
`
`(*) Notice:
`
`(65)
`
`Prior Publication Data
`US 2005/0013376 A1 Jan. 20, 2005
`(51) Int. Cl.
`(2006.01)
`H04N 7/12
`(52) U.S. Cl. ................................................. 375,240.24
`(58) Field of Classification Search ........... 375/24001,
`375/240.08, 240.12, 240.24; H04N 7/12
`See application file for complete Search history.
`
`ABSTRACT
`
`57
`(57)
`An apparatus including a first processing circuit and a
`Second precessing circuit. The first processing circuit may be
`configured to generate a plurality of reconstructed Samples
`in response to one or more macroblocks of an input signal.
`The Second processing circuit may be configured to deter
`mine availability of intra 4x4 prediction modes for each
`luma Sub-block of a current macroblock in response to
`available reconstructed Samples adjacent to the current mac
`roblock.
`
`22 Claims, 10 Drawing Sheets
`
`NT1
`--- 1992).------------------------------- -
`
`OFFSET
`
`PCTURE MEMORY ACCESS
`
`
`
`
`
`A B C D E F G H
`
`J K L M
`
`NPU
`
`BLOCK
`LOCATION
`
`
`
`MODES
`
`ENABLE
`MODES
`3 AND 7
`
`ENABLE
`MODE 8
`
`MODES
`
`Unified Patents, LLC v. Elects. & Telecomm. Res. Inst., et al.
`
`Ex. 1020, p. 1
`
`

`

`U.S. Patent
`
`Mar. 7, 2006
`
`Sheet 1 of 10
`
`US 7,010,044 B2
`
`
`
`
`
`SE&Jºn LOId (CJELOrmae LSNOOH ? HO) BOHnOS
`
`
`
`
`
`
`
`>HEGIOONE
`
`
`
`>HECIO OECD
`
`Unified Patents, LLC v. Elects. & Telecomm. Res. Inst., et al.
`
`Ex. 1020, p. 2
`
`

`

`U.S. Patent
`
`Mar. 7, 2006
`
`Sheet 2 of 10
`
`US 7,010,044 B2
`
`
`
`SXAOOTE 8X8
`
`9 ECJOW
`
`Z EICJOW
`
`A ECJOW
`
`9 ECJOW
`
`9 ECJOW
`
`
`
`
`
`
`
`10/
`
`SXAOOTBOX-HOV/W
`
`Unified Patents, LLC v. Elects. & Telecomm. Res. Inst., et al.
`
`Ex. 1020, p. 3
`
`

`

`U.S. Patent
`
`Mar. 7
`9
`
`2006
`
`Sheet 3 of 10
`
`010,044 B2
`US 7
`9
`
`
`
`96
`
`Unified Patents, LLC v. Elects. & Telecomm. Res. Inst., et al.
`
`Ex. 1020, p. 4
`
`

`

`U.S. Patent
`
`Mar. 7, 2006
`
`Sheet 4 of 10
`
`US 7,010,044 B2
`
`
`
`
`
`(dn-TVLNOZINJOH) 8
`
`
`
`
`
`Unified Patents, LLC v. Elects. & Telecomm. Res. Inst., et al.
`
`Ex. 1020, p. 5
`
`

`

`U.S. Patent
`
`US 7,010,044 B2
`
`
`
`
`
`
`
`
`
`~ 00||
`
`Unified Patents, LLC v. Elects. & Telecomm. Res. Inst., et al.
`
`Ex. 1020, p. 6
`
`

`

`U.S. Patent
`
`Mar. 7, 2006
`
`Sheet 6 of 10
`
`US 7,010,044 B2
`
`106 132
`
`136
`
`INPUT
`
`GENERAL
`1402 PROCESSING
`
`
`
`ENCODING
`
`COMPRESSED
`
`146
`
`INT2
`
`
`
`
`
`
`
`
`
`
`
`158 -
`
`160/T 150 / 156
`PRED INT1
`152 -
`154
`INTRA
`PREDICTION
`LUMA
`PROCESSING
`134
`
`126
`190
`
`192
`
`COMPRESSED
`196
`
`
`
`
`
`DECODING
`
`
`
`GENERAL
`PROCESSING
`
`UNCOMPRESSED
`204
`
`
`
`
`
`INTRA
`PREDICTION
`LUMA
`PROCESSING
`194
`
`Unified Patents, LLC v. Elects. & Telecomm. Res. Inst., et al.
`
`Ex. 1020, p. 7
`
`

`

`U.S. Patent
`
`Mar. 7, 2006
`
`Sheet 7 of 10
`
`US 7,010,044 B2
`
`80
`
`ENCOOING
`
`146
`
`COMPRESSED
`
`
`
`PICTURE
`ELEMENT
`LUMA
`PROCESSING
`
`PREDICTION
`PROCESSING
`
`FILTER
`
`
`
`
`
`
`
`
`
`
`
`
`
`162
`
`Unified Patents, LLC v. Elects. & Telecomm. Res. Inst., et al.
`
`Ex. 1020, p. 8
`
`

`

`U.S. Patent
`
`Mar. 7, 2006
`
`Sheet 8 of 10
`
`US 7,010,044 B2
`
`190
`
`DECODING
`
`
`
`
`
`
`
`
`
`
`
`INTER
`PREDICTION
`PROCESSING
`
`
`
`PICTURE
`ELEMENT
`LUMA
`PROCESSING
`
`Unified Patents, LLC v. Elects. & Telecomm. Res. Inst., et al.
`
`Ex. 1020, p. 9
`
`

`

`Unified Patents, LLC v. Elects. & Telecomm. Res. Inst., et al.
`
`Ex. 1020, p. 10
`
`

`

`U.S. Patent
`
`Mar. 7, 2006
`
`Of 10
`Sheet 10
`
`US 7,010,044 B2
`
`300
`V
`
`302
`NTRA PREDICTION SELECTED
`
`
`
`
`
`
`
`
`
`IS INTRA 4x4
`PREDICTION
`SELECTED?
`
`YES
`
`NO
`
`30
`6
`PERFORM 16x16
`INTRA PREDICTION
`
`
`
`
`
`
`
`ARE SAMPLES ADJACENT TO
`TOP AND LEFT EDGE OF
`CURRENT SUB-BLOCK (E.G.,
`P(X,-1);X=0-7 AND P(-1,Y);
`Y=0-3) AVAILABLE?
`
`310
`YES MODES 3, 7 AND
`8 ARE VALID
`
`
`
`
`
`
`
`
`
`
`
`NO
`
`ARE SAMPLES
`ADJACENT TO TOP
`EDGE OF CURRENT
`SUB-BLOCKAVAILABLE
`
`
`
`4 31
`
`YES
`
`MENP7
`
`NO
`
`ARE SAMPLES
`ADJACENT TO LEFT
`EDGE OF CURRENT
`SUB-ELOCKAVAILABLE
`
`318
`YES
`
`MODE 8 SWALID
`
`NO
`
`320
`MODES 3, 7, AND 8
`ARE NOT AVAILABLE
`
`F.G. 11
`
`Unified Patents, LLC v. Elects. & Telecomm. Res. Inst., et al.
`
`Ex. 1020, p. 11
`
`

`

`US 7,010,044 B2
`
`1
`INTRA 4x4 MODES 3, 7 AND 8
`AVAILABILITY DETERMINATION INTRA
`ESTIMATION AND COMPENSATION
`
`FIELD OF THE INVENTION
`
`The present invention relates to processing digital Video
`generally and, more particularly, to a method and/or circuit
`for determining availability of intra estimation and/or intra
`compensation of intra 4x4 Sample prediction modes 3, 7 and
`8 as specified in subclause 8.3.1.2 of ISO/IEC 14496-10
`AVC and ITU-T Rec. H.264.
`
`BACKGROUND OF THE INVENTION
`
`When a current block (or macroblock) is encoded/de
`coded in intra mode, a prediction block is formed based on
`adjacent Samples from previously encoded/decoded and
`reconstructed blocks. The prediction block is subtracted
`from the current block prior to encoding. When the current
`block is decoded in intra mode, a prediction block is formed
`based upon Samples from previously decoded and recon
`structed blocks. The prediction block is added to the current
`block following decoding.
`A prediction block for encoding/decoding luminance can
`be formed for each 4x4 Sub-block of a macroblock or for the
`entire 16x16 macroblock. Conventional approaches restrict
`the availability of Some modes when encoding 4x4 Sub
`blockS. Having alternative modes available for comparison
`when making an estimation decision can improve compres
`Sion accuracy and efficiency.
`It would be desirable to have a Solution that would allow
`as many modes for intra prediction as possible for the
`number of Samples available.
`
`SUMMARY OF THE INVENTION
`
`15
`
`25
`
`35
`
`40
`
`The present invention concerns an apparatus comprising
`a first processing circuit and a Second precessing circuit. The
`first processing circuit may be configured to generate a
`plurality of reconstructed Samples in response to one or
`more macroblocks of an input Signal. The Second processing
`circuit may be configured to determine availability of intra
`4x4 prediction modes for each luma Sub-block of a current
`macroblock in response to available reconstructed Samples
`adjacent to the current macroblock.
`The objects, features and advantages of the present inven
`tion include providing a method and/or circuit for determin
`ing availability of intra estimation and/or intra compensation
`of intra 4x4 Sample prediction modes 3, 7 and 8 that may (i)
`allow modes 3, 7 and 8 to be valid more frequently than in
`conventional approaches, (ii) provide alternative modes for
`comparison when making estimation decisions, (iii) obtain
`more accurate compression, (iv) obtain more efficient com
`pression and/or (v) base mode validation on Samples used by
`55
`each mode.
`
`45
`
`50
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`These and other objects, features and advantages of the
`present invention will be apparent from the following
`detailed description and the appended claims and drawings
`in which:
`FIG. 1 is a block diagram illustrating encoding and
`decoding operations,
`FIG. 2 is a block diagram illustrating partitions or Seg
`ments of pictures,
`
`60
`
`65
`
`2
`FIG. 3 is a diagram generally illustrating an example intra
`4x4 prediction operation;
`FIG. 4 is a diagram illustrating various intra 4x4 predic
`tion modes;
`FIG. 5 is a block diagram illustrating various components
`of a compressed Video System;
`FIG. 6 is a block diagram illustrating an encoder in
`accordance with a preferred embodiment of the present
`invention;
`FIG. 7 is a more detailed diagram of the encoder of FIG.
`6;
`FIG. 8 is a block diagram illustrating a decoder in
`accordance with a preferred embodiment of the present
`invention;
`FIG. 9 is a more detailed diagram of the decoder of FIG.
`8;
`FIG. 10 is a more detailed block diagram illustrating an
`example control circuit of FIGS. 7 and 9; and
`FIG. 11 is a flow diagram illustrating a mode enablement
`process in accordance with a preferred embodiment of the
`present invention.
`
`DETAILED DESCRIPTION OF THE
`PREFERRED EMBODIMENTS
`
`Referring to FIG. 1, a block diagram is shown illustrating
`encoding and decoding operations. In general, a data Stream
`(e.g., a video stream) may comprise a series of Source
`pictures 70a-n. The source pictures may also be referred to
`as images, frames, a group-of-pictures (GOP) or a sequence.
`The pictures generally comprise contiguous rectangular
`arrays of pixels (i.e., picture elements). Compression of
`digital Video without significant quality degradation is usu
`ally possible because video Sequences contain a high degree
`of: 1) spatial redundancy, due to the correlation between
`neighboring pixels, 2) spectral redundancy, due to correla
`tion among the color components, 3) temporal redundancy,
`due to correlation between Video frames, and 4) psycho
`Visual redundancy, due to properties of the human visual
`system (HVS). Video frames generally comprise three rect
`angular matrices of pixel data representing a luminance
`Signal (e.g., luma Y) and two chrominance signals (e.g.,
`chroma Cb and Cr) that correspond to a decomposed rep
`resentation of the three primary colors (e.g., Red, Green and
`Blue) associated with each picture element. The most com
`mon format used in Video compression Standards is eight
`bits and 4:2:0 Sub-Sampling (e.g., the two chroma compo
`nents are reduced to one-half the vertical and horizontal
`resolution of the luma component). However, other formats
`may be implemented to meet the design criteria of a par
`ticular application.
`Each picture may comprise a complete frame of Video
`(e.g., a frame picture) or one of two interlaced fields from an
`interlaced Source (e.g., a field picture). The field picture
`generally does not have any blank lines between the active
`lines of pixels. For example, if the field picture is viewed on
`a normal display, the field picture would appear short and
`fat. For interlaced Sequences, the two fields may be encoded
`together as a frame picture. Alternatively, the two fields may
`be encoded Separately as two field pictures. Both frame
`pictures and field pictures may be used together in a single
`interlaced Sequence. High detail and limited motion gener
`ally favors frame picture encoding. In general, field pictures
`occur in pairs (e.g., top/bottom, odd/even, field1/field2). The
`output of a decoding process for an interlaced Sequence is
`generally a Series of reconstructed fields. For progressive
`Scanned Sequences, all pictures in the Sequence are frame
`
`Unified Patents, LLC v. Elects. & Telecomm. Res. Inst., et al.
`
`Ex. 1020, p. 12
`
`

`

`US 7,010,044 B2
`
`15
`
`3
`pictures. The output of a decoding process for a progressive
`Sequence is generally a Series of reconstructed frames.
`The source pictures 70a-n may be presented to an
`encoder 72. The encoder 72 may be configured to generate
`a Series of encoded pictures 74a–n in response to the Source
`pictures 70a-n, respectively. For example, the encoder 72
`may be configured to generate the encoded pictures 74a–n
`using a compression standard (e.g., MPEG-2, MPEG-4,
`H.264, etc.). In general, encoded pictures may be classified
`as intra coded pictures (I), predicted pictures (P) and bi
`predictive pictures (B). Intra coded pictures are generally
`coded without temporal prediction. Rather, intra coded pic
`tures use Spatial prediction within the same picture. For
`example, an intra coded picture is generally coded using
`information within the corresponding Source picture (e.g.,
`compression using spatial redundancy). An intra coded
`picture is generally used to provide a receiver with a starting
`point or reference for prediction. In one example, intra
`coded pictures may be used after a channel change and to
`recover from errors.
`Predicted pictures (e.g., P-pictures or P-frames) and bi
`predictive pictures (e.g., B-pictures or B-frames) may be
`referred to as inter coded. Inter coding techniques are
`generally applied for motion estimation and/or motion com
`pensation (e.g., compression using temporal redundancy).
`25
`P-pictures and B-pictures may be coded with forward pre
`diction from references comprising previous I and P pic
`tures. For example, the B-picture 74b and the P-picture 74c
`may be predicted using the I-picture 74a (e.g., as indicated
`by the arrows 76 and 78, respectively). The B-pictures may
`also be coded with (i) backward prediction from a next I or
`P-reference picture (e.g., the arrow 80) or (ii) interpolated
`prediction from both past and future I or P-references (e.g.,
`the arrows 82a and 82b, respectively). However, portions of
`P and B-pictures may also be intra coded or skipped (e.g.,
`not sent at all). When a portion of a picture is skipped, the
`decoder generally uses the associated reference picture to
`reconstruct the Skipped portion with no error.
`However, the concept of what particular pictures may
`reference what other particular pictures may be generalized
`in a particular compression Standard (e.g., H.264). For
`example, P-pictures may reference temporally forward or
`backward. B-pictures may have similar forward or backward
`references. The restriction is generally not time, but rather
`how many frames are stored in a buffer so that the frames
`may be decoded in a different order than the frames are
`displayed. In one example, the frames may be referenced
`forward in time. In another example, the frames may be
`referenced backward in time (e.g., re-ordering the frames).
`In one example, a B-frame may differ from a P-frame in
`that a B-frame may do interpolated prediction from any two
`reference frames. Both reference frames may be (i) forward
`in time, (ii) backward in time, or (iii) one in each direction.
`B-pictures can be, and are expected to often be, used as
`prediction references in H.264. In many cases an important
`distinction is between reference and non-reference frames.
`The encoded pictures 74a-n may be presented to a
`decoder 84. The decoder 84 is generally configured to
`generate a Series of reconstructed pictures corresponding to
`the Source pictures 70a-n (e.g., images, frames, fields, etc.)
`in response to the encoded pictures. In one example, the
`decoder 84 may be implemented within the encoder 72 and
`the reconstructed pictures may be used in the prediction
`operations of the encoding process.
`Referring to FIG. 2, a block diagram is shown generally
`illustrating partitions or Segments of pictures. In general, a
`picture (e.g., an image, a frame, a field, etc.) 70i may be
`
`35
`
`4
`divided (e.g., Segmented, partitioned, etc.) into a number of
`macroblockS 86. The macroblockS generally comprise an
`array of pixels having vertical and horizontal dimensions of
`equal size (e.g., 32x32, 16 x 16, etc). The macroblocks
`generally comprise luminance data (e.g., luma Y) and
`chrominance data (e.g., blue chroma Cb and red chroma Cr).
`In one example, the luminance data may have a resolution
`that is twice that of the chrominance data (e.g., a 4:2:0
`format).
`The macroblocks 86 may be grouped in a number of slices
`90. The slices 90 may comprise an arbitrary number of
`macroblocks 86. The slices 90 generally run from left to
`right and may comprise an entire row of the picture 70i.
`However, a slice 90 may comprise less than or more than an
`entire row of macroblocks 86 (e.g., H.264 compliant). In one
`example, a Slice 90 may be defined as a particular number
`of macroblockS 86 grouped together. For broadcast profiles,
`the macroblocks 86 in a slice 90 are generally consecutive
`macroblocks in raster Scan order. However, for Streaming
`and/or Video-conferencing applications, a map may be sent
`identifying which Scattered macroblocks are grouped
`together in a slice. A compression Standard (e.g., H.264) may
`also provide an option of using macroblockS or macroblock
`pairs. A macroblock pair comprises two macroblockS
`located one above the other. When macroblock pairs are
`used, a slice or row generally comprises macroblock pairs
`rather than macroblockS.
`In one example, the macroblock 86 may be implemented
`as a 16x16 block. The macroblock 86 may be encoded in an
`inter prediction mode (e.g., compression based upon tem
`poral redundancy) or an intra prediction mode (e.g., com
`pression based upon spatial redundancy). In the inter pre
`diction mode, each 16x16 macroblock 86 may be predicted
`with a single 16x16 vector (e.g., mode 1). Alternatively, the
`macroblock 86 may be segmented into two 16x8 blocks
`(e.g., mode 2) or two 8x16 blocks (e.g., mode 3), in which
`case two motion vectors may be generated for predicting the
`macroblock 86. The macroblock 86 may also be segmented
`into four 8x8 blocks (e.g., mode 4), in which case four
`motion vectors may be generated for the macroblock 86.
`When the macroblock 86 is segmented into the four 8x8
`blocks (e.g., mode 4), each 8x8 block may be optionally
`further segmented into two 4x8 Sub-blocks (e.g., mode 5),
`two 8x4 sub-blocks (e.g., mode 6) or four 4x4 sub-blocks
`(e.g., mode 7). An encoder generally decides which “mode”
`to use for encoding each macroblock 86. For example, an
`error Score may be computed based on a closeness of match
`determination for each mode, with the modes that use more
`vectors being penalized (e.g., by increasing the respective
`error score) because of the additional bits that it will take to
`encode the motion vectors.
`When a block or macroblock is to be encoded in the intra
`prediction mode, a prediction block is generally formed
`based upon previously decoded and reconstructed blockS. In
`an encoder, the prediction block is generally Subtracted from
`the current block prior to encoding. In a decoder, the
`prediction block is generally added to the current block prior
`to filtering. For luminance (or luma) Samples, the prediction
`block may be formed for either each 4x4 Sub-block in the
`macroblock or for the entire 16x16 macroblock. When each
`4x4 luma block is to be predicted, any available one of nine
`prediction modes may be used for each 4x4 luma block.
`When the entire macroblock (e.g., a 16x16 luma block) is to
`be encoded, any of four available prediction modes may be
`used.
`Referring to FIG. 3, a diagram illustrating an intra pre
`diction operation for a 4x4 luma block is shown. For each
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`Unified Patents, LLC v. Elects. & Telecomm. Res. Inst., et al.
`
`Ex. 1020, p. 13
`
`

`

`US 7,010,044 B2
`
`15
`
`25
`
`35
`
`40
`
`S
`4x4 luma block 91 to be predicted in a current (or source)
`slice 92, a top edge 93 and a left edge 94 are generally
`determined. The top edge 93 and the left edge 94 of the 4x4
`luma block are used to determine whether reconstructed
`samples in a reconstructed slice 95 that are above and to the
`left of the 4x4 luma block (e.g., samples A-M) have been
`encoded and reconstructed. If the reconstructed Samples
`A-M are available in the encoder and decoder, a prediction
`block 96 may be generated using the reconstructed Samples
`A-M. However, not all of the reconstructed samples A-M
`may be available within the current Slice. In general, only
`previously encoded/decoded Samples within a current Slice
`are considered available for intra prediction in order for
`Slices to be independently decoded.
`Specifically, not all of the samples A-M may be available
`within the current reconstructed slice 95. In general, only
`previously encoded/decoded Samples within a current recon
`Structed Slice are considered available for intra prediction in
`order for Slices to be independently decoded. In addition,
`Some modes of operation (e.g., a constrained intra mode)
`may consider only macroblockS coded in an intra prediction
`mode within a slice to be available (e.g., for the constrained
`intra mode only other intra macroblocks within the Slice are
`considered available, inter coded macroblocks are consid
`ered unavailable). Also, in H.264, a slice may not always be
`independently decoded from other slices. For example, a
`loop (or deblocking) filter may operate between slices.
`However, the decoding proceSS for pixels, up to but not
`including the deblocking filter portion of the decoding
`process, may be independently decoded in the various slices.
`In general, intra prediction is performed on the decoded
`samples prior to the deblocking filter process.
`In general, adjacent (or neighboring) samples refers to
`reconstructed Samples in a line directly above or to the left
`of the current block. For field coded pictures and frame
`coded pictures the meaning of neighboring/adjacent is very
`Simple: Vertically adjacent Samples are in the line above in
`the picture (which may be either a frame or a field), and
`horizontally adjacent Samples are the line to the left in the
`picture (which may be either a frame or a field). However,
`with macroblock adaptive field/frame (MB-AFF) coded
`pictures (e.g., particularly when using constrained intra
`prediction), the samples considered to be adjacent for intra
`prediction may depend on the mode of the current macrob
`lock. For example, when processing a frame macroblock,
`the adjacent Samples generally comprise Samples that are
`adjacent to the current block with the picture Samples
`arranged as a frame. When processing a field macroblock,
`the adjacent Samples generally comprise the Samples that are
`adjacent to the current block with the picture Samples
`arranged as the same parity field as the current macroblock.
`In one example, with MB-AFF coding and constrained
`intra prediction, if a left adjacent macroblock pair are coded
`with one FRAME macroblock intra predicted and the other
`FRAME macroblock not intra predicted (e.g., inter pre
`dicted), the neighboring samples I-L may or may not be
`available for prediction for the FIELD macroblocks in the
`current macroblock pair. In another example, when a field
`macroblock pair is to the left of a current frame macroblock
`pair, if one of the left macroblocks is not available (e.g., due
`to being non-intra predicted), all of the samples I-L are
`generally not available for both macroblocks in the current
`macroblock pair. In general, the present invention provides
`for separately determining the availability of the individual
`adjacent samples (e.g., A-L).
`Referring to FIG. 4, a diagram illustrating various intra
`4x4 prediction modes is shown. When intra 4x4 prediction
`
`45
`
`50
`
`55
`
`60
`
`65
`
`6
`is available, the prediction block for each 4x4 sub-block
`may be formed using one of the nine prediction modes
`depending upon which previously encoded blocks are avail
`able. In a conventional process, the particular modes used
`for generating the prediction blocks are determined based
`upon all Samples from adjacent encoded blocks being avail
`able. In general, nine optional intra 4x4 prediction modes
`(e.g., modes 0-8) may be used to form the luma 4x4
`prediction block 96. The prediction block 96 is generally
`formed by copying particular ones of the Samples A-M into
`positions in the prediction block (e.g., a-p) based upon each
`particular intra prediction mode. The arrows generally indi
`cate the direction in which the Samples A-M are copied in
`each prediction mode.
`The encoder generally Selects the prediction mode for
`each 4x4 luma block that produces a prediction block 96 that
`most closely resembles the current block 91. For example,
`the encoder may select the mode that minimizes a difference
`(or residual) between the predicted block 96 and the block
`91 to be encoded. In one example, a measurement (e.g., Sum
`of absolute differences (SAD)) may be determined to indi
`cate the prediction error.
`Referring to FIG. 5, a block diagram of a system 100 is
`shown. In general, a content provider 102 presents video
`image, audio or other data 104 to be compressed and
`transmitted to an input of an encoder 106. The compressed
`data 108 from the encoder 106 may be presented to an
`encoder transport System 110. An output of the encoder
`transport System 110 generally presents a signal 112 to a
`transmitter 114. The transmitter 114 transmits the com
`pressed data via a transmission medium 116. The content
`provider 102 may comprise a video broadcast, DVD, or any
`other Source of Video data Stream. The transmission medium
`116 may comprise a broadcast, cable, Satellite, network,
`DVD, hard drive, or any other medium implemented to
`carry, transfer, and/or Store a compressed bitstream.
`On a receiving side of the system 100, a receiver 118
`generally receives the compressed data bitstream from the
`transmission medium 116. The receiver 118 presents a
`bitstream 120 to a decoder transport system 122. The
`decoder transport System 122 generally presents the bit
`stream via a link 124 to a decoder 126. The decoder 126
`generally decompresses the data bitstream and presents the
`data via a link 128 to an end user 130. The end user 130 may
`comprise a television, monitor, computer, projector, hard
`drive, or any other medium implemented to carry, transfer,
`present, display and/or Store an uncompressed bitstream.
`Referring to FIG. 6, a block diagram illustrating an
`encoder 106 in accordance with a preferred embodiment of
`the present invention is shown. The encoder 106 may be
`implemented, in one example, as an H.264 compliant
`encoder. The encoder 106 generally comprises a processing
`block 132 and a processing block 134. The encoder 106 may
`also comprise an encoding block 136. The processing block
`132 may be implemented as a general processing block. The
`processing block 134 may be implemented as an intra
`prediction luma processing block.
`The general processing block 132 may have an input 140
`that may receive a signal (e.g., INPUT). The signal INPUT
`generally comprises an uncompressed digital video signal
`comprising a series of pictures (e.g., frames, fields, etc.).
`Each picture generally comprises a representation of a
`digital Video Signal at a particular time. The general pro
`cessing block 132 may be configured to generate a plurality
`of macroblocks from each picture. The general processing
`
`Unified Patents, LLC v. Elects. & Telecomm. Res. Inst., et al.
`
`Ex. 1020, p. 14
`
`

`

`7
`block 132 may also have an output 142 that may present one
`or more signals (e.g., CTR1) to an input 144 of the encoding
`circuit 136.
`The encoding circuit 136 may have an output 146 that
`may present a signal (e.g., COMPRESSED). The signal
`COMPRESSED may be a compressed and/or encoded bit
`Stream, Such as an H.264 compliant digital Video bitstream.
`In one example, the encoding circuit 136 may be configured
`to perform entropy coding. The circuit 136 may be further
`configured to provide Serialization (e.g., Zig-Zag Scan) and
`re-ordering of the transformed and quantized pictures.
`The general processing circuit 132 may have an output
`150 that may present one or more signals (e.g., INT1) to an
`input 152 of the intra prediction luma processing block 134.
`Similarly, the intra prediction luma processing block 134
`may have an output 154 that may present a signal (e.g.,
`INT2) to an input 156 of the general processing block 132,
`an output 158 that may present a signal (e.g., PRED) to an
`input 160 of the general processing block 132 and an input
`162 that may receive the signal INPUT. The signal INT1
`may comprise, in one example, previously encoded/decoded
`and reconstructed Samples of the pictures in the Signal
`INPUT. The signal INT2 may comprises, in one example,
`mode information regarding prediction Samples generated
`by the block 134. The signal PRED generally comprises one
`or more prediction Samples related to each picture.
`Referring to FIG. 7, a more detailed diagram of the
`encoder 106 of FIG. 6 is shown. The intra prediction
`processing block 134 generally comprises a block (or cir
`cuit) 164 and a block (or circuit) 166. The circuit 164 may
`be implemented, in one example, as a control circuit. The
`circuit 166 may be implemented as a picture element luma
`processing block. The circuit 164 may be configured to
`determine available intra prediction modes in response to the
`signals INPUT and INT1. In particular, the circuit 164 may
`be configured to determine availability of reconstructed
`samples A-M for each 4x4 luma block to be intra predicted.
`The circuit 164 may be configured to generate a signal (e.g.,
`MODES) in response to the signals INPUT and INT1. In one
`example, the signal MODES may be implemented as one or
`more individual control signals. Alternatively, the Signal
`MODES may be implemented as a multibit signal, where
`each bit may be used as a control Signal. In one example, the
`signal MODES may be configured to indicate availability of
`intra 4x4 prediction Sample modes 3, 7 and 8.
`The circuit 166 may be configured to generate prediction
`blocks for each 4x4 luma block to be encoded. The circuit
`166 may be configured to receive the signals INPUT, INT1
`and MODES. The circuit 166 may be configured to generate
`the signals INT2 and PRED in response to the signals
`INPUT, MODES and INT1.
`The circuit 132 generally comprises a block (or circuit)
`170, a block (or circuit) 172, a block (or circuit) 173, a block
`(or circuit) 174, a block (or circuit) 176, a block (or circuit)
`177, a block (or circuit) 178, a block (or circuit) 180, a block
`(or circuit) 182, a block (or circuit) 184, a block (or circuit)
`186 and a block (or circuit) 188. The circuit 170 may be
`implemented as an inter prediction processing circuit. The
`circuit 172 may be implemented as a motion estimation
`circuit. The circuit 173 may be implemented as a deblocking
`(or loop) filter. The circuit 174 may be implemented as a
`picture memory circuit. The circuit 176 may be implemented
`as a Selection circuit, Such as a 2:1 multiplexer. The circuit
`177 may be implemented as a summing circuit. The circuit
`178 may be implemented as a transform circuit. In one
`example, the circuit 178 may be configured to perform an
`4x4 integer transform or a discrete cosine transform (DCT).
`
`15
`
`25
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`US 7,010,044 B2
`
`8
`The circuit 180 may be implemented as a control circuit. The
`circuit 182 may be implemented as a quantization circuit.
`The circuit 184 may be implemented as an inverse quanti
`zation circuit. The circuit 186 may be implemented as an
`inverse transform circuit. The circuit 188 may be imple
`mented as a Summing circuit.
`An output of the quantization circuit 182, an output of the
`motion estimation circuit 172, an output of the inter pro
`cessing circuit 170 and the signal INT2 may be presented as
`the signal CTR1 at the output 142. The inverse quantization
`circuit 184 is generally configured to reverse the quantiza
`tion process performed by the quantization circuit 182. The
`inverse transform circuit 186 is generally configured to
`reverse the transformation process (e.g., DCT or 4x4 inte
`ger) performed by the circuit 178. The inverse transform
`circuit 186 may also be referred to as an inverse DCT block
`or an IDCT block.
`The signal INPUT may be presented to the inter predic
`tion processing block 170, the motion estimation block 172
`and the Summing block 177. The summing block 177 may
`mathematically combine the signal INPUT with either (i) an
`output of the interprediction processing block 170 or (ii) the
`signal PRED from the block 134. The selection may respond
`to a signal provided by the control circuit 180. The signal
`INPUT may be compressed with the transform circuit 178.
`The transform circuit 178 may translate the macroblocks in
`the signal INPUT from time domain frames to frequency
`domain frames. The quantization block 182 may reduce the
`number of bits in a number of coefficients representing the
`signal INPUT. The encoding block 136 may provide entropy
`coding (e.g., Huffman coding, binary arithmetic coding,
`context adaptive binary arithmetic coding or CABAC, etc.)
`to implement a loSSleSS compression having frequent values
`represented in fewer bits.
`The inverse quantization circuit 184 and the inverse
`transform circuit