United States Patent (19)
`Malladi et al.
`
`54 METHOD FOR PARTITIONING HARDWARE
`AND FIRMWARE TASKS IN DIGITAL
`AUDIO/VIDEO DECODING
`
`75 Inventors: Srinivasa R. Malladi, San Jose; Marc
`A. Miller, Palo Alto, Kwok K. Chau,
`Los Altos, all of Calif.
`73 Assignee: LSI Logic Corporation, Milpitas,
`Calif.
`
`Appl. No.: 643,185
`21
`22 Filed:
`May 3, 1996
`51
`Int. Cl. ................................ H04N 7/36; H04N 7/50
`52 U.S. Cl. .......................... 348/390; 348/423: 348/845;
`381/2; 704/504
`58 Field of Search ..................................... 348/423, 845,
`348/390; 381/2; 704/504; H04N 7/36, 7/50
`
`56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`5,394,473 2/1995 Davidson ................................ 704/211
`5,506,832 4/1996 Arshi ......................................... 348/15
`5,598,352 1/1997 Rosenau .................................. 348/423
`OTHER PUBLICATIONS
`Martin Boliek, “Real-Time Discrete Cosine Transform Chip
`Using Generalized Chen Transforms Technology,” pp.
`428-431, Ricoh California Research Center, Menlo Park,
`CA.
`Unknown, “Digital Audio Compression (AC-3).” T3 review
`copy of draft ATSC audio standard, Aug. 12, 1994, Doc.
`T3/251.
`
`USOO5815206A
`Patent Number:
`11
`(45) Date of Patent:
`
`5,815,206
`Sep. 29, 1998
`
`Unknown, “Information Technology-Generic Coding of
`Moving Pictures and Associated Audio Information. Video, ”
`ISO/IEC 13818–2, Draft International Standard, Nov. 9,
`1994.
`Unknown, “Coding of Moving Pictures and Associated
`Audio,” ISO/IEC 13818–3, International Standard, Nov. 11,
`1994.
`Dave Bursky, “Single Chip Performs Both Audio and Video
`Decoding,” Electronic Design, Apr. 3, 1995.
`Unknown, “Coding of Moving Pictures and Associated
`Audio for Digital Storage Media At Up To About 1.5
`MBITIs," 3–11171 rev. 1, (Part 3 Audio), May 30, 1995.
`Primary Examiner-Howard Britton
`Attorney, Agent, or Firm-Hickman & Martine, LLP
`57
`ABSTRACT
`Disclosed is a partitioning procedure for designing MPEG
`decoders, AC-3 decoders, and decoderS for other audio/
`Video Standards. The procedure provides that Some Specified
`decoding functionality be implemented exclusively in the
`form of hardware and certain other Specified decoding
`functionality be provided exclusively as firmware or soft
`ware. A Video decoder designed according to this procedure
`includes the following elements: (a) firmware or Software
`for implementing, in conjunction with a CPU, video header
`processing functions; and (b) hardware for implementing
`preparsing assist, macroblock reconstruction, and Video dis
`play control functions. An audio decoder designed according
`to this procedure includes the following elements: (a) firm
`ware or Software for implementing, in conjunction with a
`CPU, decoding fields containing parameters for processing
`the audio data; and (b) hardware for implementing matrixing
`and windowing functions on the audio data.
`24 Claims, 6 Drawing Sheets
`
`FIRWARSOFTWARE
`
`CHANNEL EFFERAOIC
`
`200
`
`1202
`
`yes
`
`ANYMORE
`ECODING
`TASKS
`
`AUDO HEADER
`
`DECODING OFBT ALLOCATION
`
`DECODNG OF SCALEFACTORS
`
`WARIABLE ENGTHOECOOING OF
`SAMPLES
`
`ReQANTIZATION OF SAMPLES
`
`SUB-BANDSYNTHESS
`ONE DIMENSIONALict
`
`WINOWING
`
`ALL CHANNELS
`CO)?
`
`
`
`
`
`
`
`Unified Patents, LLC v. Elects. & Telecomm. Res. Inst., et al.
`
`Ex. 1028, p. 1
`
`

`

`Unified Patents, LLC v. Elects. & Telecomm. Res. Inst., et al.
`
`Ex. 1028, p. 2
`
`

`

`U.S. Patent
`
`Sep. 29, 1998
`
`Sheet 2 of 6
`
`5,815,206
`
`TRANSPORT DEMULTIPLEXER RECEIVES BIT STREAM 8, GENERATES
`PACKETIZED ELEMENTARY STREAMS FOR A SELECTED CHANNEL
`
`101
`
`102
`
`100
`
`PACKET DEMULTIPLEXER RECEIVES A PACKETIZED ELEMENTARY STREAM HAVINGA
`VIDEO BIT STREAM, AN AUDIO BIT STREAMS AND A SYNCHRONIZATION BIT STREAM
`AUDIO
`106
`VIDEO STREAM DEFINED
`104 SYNCHRONIZATION
`STREAM DEFINED
`STREAM
`DEFINED
`
`DECODE SEQUENCE HEADER
`
`110
`
`108
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`LOAD OUANT MATRICES
`
`111
`
`DECODE SEQUENCE EXTENSION
`
`112
`
`DECODE GROUP OF PICTURES HEADER
`
`114
`
`DECODEGROUP OF PICTURES EXTENSION 115
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`DECODE PICTURE HEADER
`
`DECODE PICTURE EXTENSION
`
`LOADSLCE PARAMETERS
`
`PROC FSS)
`MACROBLOCK
`(fig. 1C)
`
`116
`
`117
`
`120
`
`121
`
`41
`
`O
`
`-16 NEXT
`
`PORTION OF BIT STREAM
`A START CODE
`
`SNEXT
`BIT STREAM A
`SLICE 2
`
`142
`
`yes
`
`144
`
`SNEXT
`BIT STREAM A
`PICTUREP
`
`SNEXT
`BIT STREAM A
`GOPs?
`
`146
`
`150
`
`DONE UNTIL NEXT WIDEO
`START CODE IS DETECTED
`
`O
`
`
`
`SNEXT
`BIT STREAMA
`SEQUENCE 2
`
`148
`
`Unified Patents, LLC v. Elects. & Telecomm. Res. Inst., et al.
`
`Ex. 1028, p. 3
`
`

`

`U.S. Patent
`
`Sep. 29, 1998
`
`Sheet 3 of 6
`
`5,815,206
`
`120
`
`M 125
`
`MACROBLOCK
`PROCESSINGLOGIC
`
`122
`
`123
`
`DECODE MACROBLOCK
`
`DETERMINE MACROBLOCK
`TYPE
`
`
`
`124
`
`
`
`
`
`DECODE DISCRETE
`COSINE TRANSFORM
`COEFFICIENT
`(DCT COEFFICIENTS)
`
`PERFORMINVERSE
`SCAN OPERATION
`
`PERFORMINVERSE 1
`QUANTIZATION (IQ)
`
`PERFORM INVERSE DISCRETE
`COSINE TRANSFORM
`(IDCT)
`
`
`
`126
`
`FETCH MOTION
`REFERENCE
`
`VIDEO CORE
`(VCORE)
`----4-
`
`- - - - - - - -
`
`MACROBLOCK
`- - - - - PROCESSINGLOGIC
`
`142
`(fig. 1B)
`
`FIG. 1C
`
`Unified Patents, LLC v. Elects. & Telecomm. Res. Inst., et al.
`
`Ex. 1028, p. 4
`
`

`

`U.S. Patent
`
`Sep. 29, 1998
`
`Sheet 4 of 6
`
`5,815,206
`
`Gar) FIRMWARESOFTWARE
`
`CHANNEL BUFFER AUDIO
`
`2OO
`
`|
`
`AUDO HEADER
`
`DECODING OF BIT ALLOCATION
`
`204
`
`202
`
`
`
`yes
`
`220
`
`ANYMORE
`DECODING
`TASKS
`
`O
`
`DECODING OF SCALEFACTORS
`
`VARIABLE LENGTH DECODING OF
`SAMPLES
`
`
`
`REGUANTIZATION OF SAMPLES
`
`SUB-BAND SYNTHESS
`ONE DIMENSIONAL IDCT
`
`
`
`
`
`ALL CHANNELS
`DECODEDP
`
`
`
`FG. 2
`
`Unified Patents, LLC v. Elects. & Telecomm. Res. Inst., et al.
`
`Ex. 1028, p. 5
`
`

`

`U.S. Patent
`
`Sep. 29, 1998
`
`Sheet S of 6
`
`5,815,206
`
`CSAR) FIRMWAREISOFTWARE
`- - - - - - - - - - - - - - -
`CHANNEL BUFFER
`AUDIO
`
`3OO
`
`3O4
`
`
`
`|
`
`nO
`
`ANYMORE
`DECODING
`TASKSP
`
`
`
`
`
`ALL CHANNELS
`DECODED
`
`
`
`ACORE
`
`FIG. 3
`
`Unified Patents, LLC v. Elects. & Telecomm. Res. Inst., et al.
`
`Ex. 1028, p. 6
`
`

`

`Unified Patents, LLC v. Elects. & Telecomm. Res. Inst., et al.
`
`Ex. 1028, p. 7
`
`

`

`5,815,206
`
`15
`
`2
`AS can be appreciated, digital Video and audio data for
`high quality playback requires quite significant computa
`tional resources. MPEG-2 video decoding, for example,
`requires a processor capable of decoding 720 by 480
`(NTSC) pixel frames for display at 30 Hz, and 720 by 576
`(PAL) pixel frames for display at 25 Hz. Each frame
`includes color and intensity pixel data that is encoded
`according to a two-dimensional discrete cosine transform,
`quantization, variable length encoding, half pel averaging,
`etc. An MPEG-2 video decoder must therefore reverse each
`of these encoding functions fast enough for display at 30 Hz.
`The decoder will also be called upon to concurrently decode
`audio data via an inverse discrete cosine transform (in the
`case of MPEG) or an inverse fast fourier transform (in the
`case of AC-3) and other techniques.
`In View of these requirements, audio/video decoderS must
`be carefully designed to ensure Sufficient Speed and band
`width. On the one hand, the decoder can be implemented as
`Software or firmware. In this case, a generic CPUSupplies all
`computational power for decoding. Unfortunately, most
`general purpose CPUs available today, Simply cannot per
`form the decoding tasks. Even if they could (and they may
`be able to in the near future), they would have to be
`dedicated to the decoding process and would therefore be
`unavailable for processing other CPU related tasks not
`directly related to the decoding process. Even the highest
`performance CPUs will, for the foreseeable future, be unable
`to handle all decoding tasks without experiencing Serious
`processing interruptions.
`On the other hand, audio and Video decoding tasks may be
`entirely implemented as hardware. However, all the hard
`ware dedicated to performing the numerous decoding tasks
`requires Substantial space on an integrated circuit (IC) chip.
`AS more functions are implemented as hardware, there is
`less room on the chip available for other functions and the
`overall cost of the chip goes up. Therefore, hardware only
`decoderS do not necessarily make efficiently use of Silicon.
`Further, designing new decoder hardware for decoding is
`a quite expensive process. The design is first provided as a
`Boolean description in a hardware design language Such as
`Verilog available from Cadence Design Systems, Inc. of
`Santa Clara, Calif. Then the code for the processor design
`model is used to create a net list, which is, in turn, used to
`create a physical layout for the integrated circuit. The
`physical layout must then be converted to reticles (or masks)
`for fabricating the ultimate Silicon Version of the integrated
`circuit. At each Stage in the process, from a hardware design
`language description through Silicon hardware, the inte
`grated circuit must be extensively tested for bugs and to
`improve performance. To the extent that generic hardware
`Such as CPUs can be employed, the decoder design might be
`Significantly Streamlined.
`In view of the above, various tradeoffs must be considered
`in designing a decoder as Software, hardware, or Some
`combination of the two. Thus, it would be desirable to have
`a procedure for intelligently partitioning decoding tasks
`between Software and hardware.
`
`25
`
`1
`METHOD FOR PARTITIONING HARDWARE
`AND FIRMWARE TASKS IN DIGITAL
`AUDIO/VIDEO DECODING
`CROSS REFERENCE TO RELATED
`APPLICATIONS
`This application is related to the following U.S. patent
`applications: (1) U.S. patent application Ser. No. 08/642,396
`filed on the same day as the instant application, and naming
`Srinivasa R. Malladi and Venkat Mattela as inventors, and
`entitled “Microarchitecture of Video Core for MPEG-2
`Decoder,” (2) to U.S. patent application Ser. No. 08/642,520
`filed on the same day as the instant application, and naming
`Srinivasa R. Malladi and Mahadev S. Kolluru as inventors,
`and entitled “Microarchitecture for Audio Core for an
`MPEG-2 and AC-3 Decoder,” and (3) U.S. patent applica
`tion Ser. No. 08/642,393 filed on the same day as the instant
`application, and naming Srinivasa R. Malladi as inventor,
`and entitled "Method and Apparatus for Designing
`Re-useable Core Interface Shells.” All three applications are
`incorporated herein by reference in their entireties and for all
`purposes.
`BACKGROUND OF THE INVENTION
`The present invention relates to Systems for decoding
`coded Video and audio information. More specifically, the
`invention relates to partitioning decoding tasks between
`hardware and Software/firmware in order to increase decod
`ing efficiency and performance.
`Various standards have been developed for the purpose of
`providing digitally encoded Video and audio data that can be
`reconstructed to provide good quality playback. In the late
`1980s, a digital audio/video reconstruction Standard known
`as “MPEG” (for Motion Pictures Experts Group) was pro
`mulgated by the International Standards Organization (ISO).
`MPEG syntax provides an efficient way to represent audio
`and Video Sequences in the form of compact coded data.
`MPEG unambiguously defines the form of a compressed bit
`Stream generated for digital audio/video data. Given knowl
`edge of the MPEG rules, one can thus create a decoder
`40
`which reconstructs an audio/video Sequence from the com
`pressed bit Stream.
`The MPEG-2 video standard is described in a document
`entitled “Generic Coding of Moving Pictures and Associated
`Audio Information: Video,” ISO/IEC 13818-2 (hereinafter
`“the MPEG/Video Document”), and is hereby incorporated
`by reference for all purposes. The MPEG audio standards
`are described in: (1) the MPEG-1 audio document entitled
`“Coding of Moving Pictures and Associated Audio for
`Digital Storage Media at up to about 1.5 MBit/s” (Part 3
`Audio) 3-11171 rev 1 (1995); and (2) the MPEG-2 audio
`document entitled “Generic Coding of Moving Pictures and
`Associated Audio Information” ISO/IEC 13818-3,
`(hereinafter “the MPEG/Audio Documents”). Both MPEG
`audio Standards documents are hereby incorporated by ref
`55
`erence for all purposes. All above-referenced MPEG stan
`dards documents may be obtained form ISO/IEC Case
`Postale 56, CH- 1211, Geneva 20, Switzerland.
`A competing audio Standard employing Dolby(E) process
`ing and known as “AC-3” has also been developed by the
`United States Advanced Television Systems Committee for
`digital encoding and decoding of audio data. This Standard
`is described in the “Digital Audio Compression (AC-3)”
`draft ATSC STANDARD” AC3STD68. DOC (1994)
`(hereinafter “the AC-3 Document”) which is available from
`Dolby Laboratories, Inc. located in San Francisco, Calif. and
`is hereby incorporated by reference for all purposes.
`
`35
`
`45
`
`50
`
`60
`
`65
`
`SUMMARY OF THE INVENTION
`The present invention provides Such intelligent partition
`ing procedure for designing MPEG decoders, AC-3
`decoders, and decoderS for other audio/video Standards. The
`invention provides that Some specified decoding function
`ality be implemented exclusively in the form of hardware
`and certain other Specified decoding functionality be pro
`vided exclusively as firmware or software. Still other decod
`
`Unified Patents, LLC v. Elects. & Telecomm. Res. Inst., et al.
`
`Ex. 1028, p. 8
`
`

`

`5,815,206
`
`5
`
`15
`
`25
`
`3
`ing functionality might be provided as either hardware or
`firmware, depending upon the requirements of the designer
`and based on System implementation issues. Whether to
`provide the decoding functionality as exclusively hardware,
`exclusively firmware, or either hardware or firmware is
`based upon criteria Such as (1) the computational intensity of
`the decoding functionality, (2) the flexibility that the asso
`ciated standard (e.g., MPEG or AC-3) allows in implement
`ing the functionality, (3) the amount of CPU interruptions
`that would be required to implement the functionality as
`firmware and (4) the size constraints of hardware.
`Those functions that are most computationally intensive
`are implemented as hardware. Also, those functions that
`would cause frequent CPU interrupts are preferably imple
`mented as hardware. Such hardware is designed to perform
`its intended function more efficiently than a general purpose
`CPU. This allows the system CPU to execute “system' tasks
`without devoting processing resources to the functions
`implemented on the hardware.
`The hardware may be either a “core” or custom designed
`hardware. A core design specifies the transistor-by-transistor
`layout and associated Signal interfaces of a particular hard
`ware block. Thus, a significant advantage of core-based
`designing derives from the core's availability for repeated
`use in many different chip designs for different applications.
`In each Such chip design, the decoding functions Specified
`by the core can be employed without redesign. Thus, the
`core may be used on a first integrated circuit having a first
`integrated circuit design and on a Second integrated circuit
`having a Second integrated circuit design, with the first and
`Second integrated circuit designs having at least Some fea
`tures not in common. If a “system chip having multiple
`cores is employed, the first integrated circuit design may
`include a first collection of cores, while the Second inte
`grated circuit may include a Second collection of cores,
`etc.-even though the first and Second collections of cores
`have at least one core not in common.
`Cores are obviously most appropriate for implementing
`those Standard decoding functions that are rather rigid (i.e.,
`those functions that a Standard indicates must be imple
`mented in a rather specific manner). Often these rigidly
`defined functions are also computationally intensive, So a
`hardware core implementation is the logical choice.
`For flexibly defined functions of a decoding Standard, a
`designer is free to design hardware or Software blocks,
`optimized for the designer's purposes, to perform those
`functions. Note that if a decoding task is rigidly established
`by Standard, a designer will generally have no interest in
`customizing those tasks, and a core provides the best result.
`On the other hand, decoding tasks that are less rigidly
`defined by Standard may be implemented as customized
`hardware or Software to Serve the designer's Specific needs.
`AS noted, firmware implements. Some functions of a
`decoding Standard. Such functions are typically those that
`are not particularly computationally intensive and/or those
`55
`that may be performed by CPU polling as opposed to CPU
`interrupts.
`In one aspect, the present invention provides a video
`decoder for decoding Video data contained in a bitstream.
`Such video decoder may be characterized as including the
`following elements: (a) firmware or software for
`implementing, in conjunction with a CPU, video header
`processing functions; and (b) hardware for implementing
`preparsing assist, macroblock reconstruction, and Video dis
`play control functions.
`In preferred embodiments, the Video header processing
`functions implemented by the firmware include program
`
`4
`ming hardware control registers with control information
`obtained from processing one or more video headers (e.g.,
`MPEG video headers including Sequence headers, group of
`picture headers, picture headers, slice headers, Sequence
`extensions, and picture extensions). In further preferred
`embodiments, the hardware includes a Video core for imple
`menting at least the following operations: inverse Scan,
`inverse quantization, IDCT, half pel averaging, and merge
`functions. In this embodiment, the hardware may also
`include macroblock processing logic for implementing mac
`roblock header processing, motion vector decoding, variable
`length decoding, and run length decoding.
`Preferably, the firmware also implements audio and video
`Synchronization, and preparsing functions. Examples of
`audio and Video synchronization functions include deter
`mining a starting point for decoding, Video display control
`register programming, and audio playback control register
`programming. Examples of preparsing functions include
`extraction of time Stamp information from the bitstream,
`issuing commands to demultiplex the bitstream to at least a
`channel Video memory and a channel audio memory, and
`programming direct memory access to transfer the bitstream
`from a stream buffer to at least the channel video memory or
`the channel audio memory.
`Preferably the hardware also implements preparsing assist
`functions and Video display control functions. Examples of
`the preparsing assist functions implemented as hardware
`include byte alignment and Start code detection, writing to a
`Start code table, and buffer management for the bitstream.
`The above Video processing functions are described in the
`MPEG video document previously incorporated by refer
`CCC.
`In another aspect, the present invention provides an audio
`decoder for decoding audio data contained in a bitstream.
`This audio decoder may be characterized as including the
`following elements: (a) firmware or software for
`implementing, in conjunction with a CPU, decoding fields
`containing parameters for processing the audio data; and (b)
`hardware for implementing Sub-band Synthesis and window
`ing functions on the audio data. Preferably, the audio
`decoder hardware includes an audio core having a data path,
`a ROM, an input RAM interface, and an output RAM
`interface.
`The firmware or Software may also implement one or
`more of the following functions: (1) loading audio Samples
`contained in the audio data to a storage region before the
`audio samples are processed by the hardware, (2) providing
`control information from the bitstream to one or more
`control registers in the hardware, (3) requantizing the audio
`Samples, (4) processing headers contained in the audio data,
`and (5) error checking. Preferably, it can process both
`MPEG data and AC-3 data. In the case of MPEG data, the
`firmware may decode bit allocation parameters, decode
`Scale factors, and perform variable length decoding of
`Samples, for example. In the case of AC-3 data, the firmware
`may perform bit allocation, decode exponents, perform
`decoupling, perform rematrixing, and perform dynamic
`range compression. The above audio processing functions
`are described in the AC-3 and MPEG audio documents
`previously incorporated by reference.
`These and other features and advantages of the invention
`will be described in more detail below with reference to the
`drawings.
`BRIEF DESCRIPTION OF THE DRAWINGS
`FIG. 1A is a diagram of bit Stream containing video data
`encoded according to the MPEG-2 standard.
`
`35
`
`40
`
`45
`
`50
`
`60
`
`65
`
`Unified Patents, LLC v. Elects. & Telecomm. Res. Inst., et al.
`
`Ex. 1028, p. 9
`
`

`

`5,815,206
`
`15
`
`S
`FIG. 1B is a process flow diagram of the video decoding
`Steps performed in firmware in accordance with a preferred
`embodiment of this invention.
`FIG. 1C is a process flow diagram detailing the video
`decoding Steps performed in hardware in accordance with a
`preferred embodiment of this invention.
`FIG. 2 is a process flow diagram illustrating the MPEG
`audio decoding Steps performed in firmware in accordance
`with a preferred embodiment of this invention.
`FIG. 3 is a process flow diagram illustrating the AC-3
`audio decoding Steps performed in firmware in accordance
`with a preferred embodiment of this invention.
`FIG. 4 is a block diagram illustrating the hardware blocks
`used to decoding video and audio data in accordance with a
`preferred embodiment of this invention.
`DETAILED DESCRIPTION OF THE
`PREFERRED EMBODIMENTS
`A. General Items
`The following discussion and associated figures provide
`one preferred implementation of the partitioning procedure
`of the present invention. This procedure is employed to
`design a decoder which can decode MPEG-2 audio and
`Video data as well as AC-3 audio data. Hereinafter, except
`where distinctions between the two versions of the MPEG
`25
`standard exist, the terms “MPEG” and “MPEG-2 will be
`used interchangeably to reference those Video decoding
`algorithms promulgated in the original MPEG-1 Document
`as well as in the MPEG-2 Document, and any future
`versions of MPEG decoding. Further, the term AC-3 is
`intended to cover current, past, and future versions of that
`audio decoding Standard.
`AS noted, Some hardware decoding tasks will be per
`formed on hardware “cores.” AS used herein, a core is the
`hardware layout of a Substantially Self-contained integrated
`circuit module such as a CPU, an Asynchronous Transfer
`Mode (ATM) unit, a memory unit, a network interface, an
`audio decoder, and a video decoder. The physical core has
`asSociated therewith a core design which Specifies a collec
`tion of mask layouts used to generate the reticles for
`photolithography Steps employed during integrated circuit
`fabrication. The core design also includes certain processing
`parameters associated with masks, Such as ion implant
`energy and dose, etch conditions, oxidation conditions,
`chemical vapor deposition conditions, etc. Still further, the
`core design includes information specifying the core inter
`face parameterS Such as the types of input and output signals
`required, the locations of the various interface connections,
`etc.
`AS noted, hardware cores are most useful for functions
`that are rigidly defined by Standard. Customized hardware
`may be employed for other computationally intensive and/or
`highly repetitive functions that are not So rigidly defined by
`Standard.
`LeSS repetitive and computationally inexpensive tasks
`may be implemented as Software or more preferably firm
`ware. If software is employed, it should be provided with an
`efficient compiler and preferably as assembly language. Of
`course, it remains within the Scope of this invention to
`implement Some functions in higher level languages Such as
`Fortran, Pascal, C++, C, etc. Throughout the remainder of
`the specification, references to “firmware” or “software” are
`intended to cover both possibilities.
`In one preferred embodiment, the entire Software and
`hardware decoder along with other System functions are
`provided on a Single "System on a chip.” System on a chip
`designs typically include multiple cores integrated on Single
`
`35
`
`6
`Semiconductor chip that might include a microprocessor
`core, a Video decoder core, and an audio decoder core: all
`taken from library of core designs. A System designer is left
`with the task of integrating the various cores on a Single chip
`and providing any processing functionality not Specified in
`the core designs. A part of this invention defines what
`functionality must be defined as cores and what functionality
`should be defined by the designer. This allows the designer
`to customize Some parts of the System chip and thereby
`differentiate its product from that of other designers. For a
`more detailed discussion on Video and audio cores and core
`microarchitectures, reference may be made to related U.S.
`patent application Ser. No. 08/642,396 and U.S. patent
`application Ser. No. 08/642,520 both of which were previ
`ously incorporated herein by reference.
`A designs and partitioning associated with this invention
`may take Several forms. For example, the design for the
`hardware itself is preferably Stored on a machine readable
`media Such as a magnetic or optical Storage unit. The
`information content of the design preferably includes a
`Series of hardware layouts Specifying the locations and
`features of various circuit elements comprising the hardware
`architecture. Ultimately, the design is implemented as hard
`ware on one or more chips. Thus, the design exists as both
`an intangible description of hardware and as the actual
`hardware itself. In general, chips and their designs that
`partition decoder functions in the manner described herein
`fall within the scope of this invention.
`The following flow diagrams illustrate how various func
`tions of MPEG and AC-3 standards are implemented are
`partitioned in accordance with one preferred embodiment of
`the present invention.
`B. Partitioning Video Decoding Tasks
`As the present invention preferably implements portions
`of the MPEG video decoding algorithm, the general MPEG
`video decoding algorithm will now be described with ref
`erence to FIGS. 1A, 1B, and 1C. For purposes of the
`following discussion, it will be assumed that the reader has
`available a copy of the MPEG-2/Video Document, previ
`ously incorporated herein by reference.
`Compressed data Such as Video data is generally trans
`mitted as a bit Streams of “ones' and “Zeros' representing
`coded data. In MPEG, bit streams are typically divided into
`packets of data, and each packet is configured to have a start
`code indicative of either Video, audio or Synchronization
`data. By way of example, a Video packet may begin with
`twenty three consecutive Zeros “0, a one “1”, followed by
`a “1,” and then followed by an identifying “byte” (i.e., a total
`of 32 bits) which may designate the current packet as either
`Video, audio or Synchronization data.
`The following is a brief description of the architectural
`hierarchy associated with a packetized video bit stream as
`described in the MPEG/Video Document. While this
`description presents the example of an NTSC video standard
`(employing a resolution of 720 by 480 pixels), the invention
`also covers other standards Such as the PAL video standard
`(employing a resolution of 720 by 576 pixels).
`When a decoder receives a bit Stream, it receives a
`packetized elementary stream (PES) header and an associ
`ated Video payload. The Video payload begins with a Video
`start code. From there, the outer most level of the hierarchy
`begins at a “sequence’ level, which includes one or more
`coded frames that have Some pixel data commonality. By
`way of example, pixel data commonality between frames
`may be represented as an identical blue Sky background
`asSociated with filming an air show.
`The next level in the video bit stream hierarchy is a
`“group of pictures” (GOP) level. The GOP level typically
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`Unified Patents, LLC v. Elects. & Telecomm. Res. Inst., et al.
`
`Ex. 1028, p. 10
`
`

`

`5,815,206
`
`15
`
`25
`
`35
`
`40
`
`7
`includes a group of about 4 to 5 frames (an amount Sufficient
`to allow humans to perceive a change in image) also having
`Some pixel data commonality. Following the GOP level is a
`“picture” level. As noted, a picture in MPEG-2 is defined as
`a frame having a grid of 720-by-480 pixels (or 720 by-240
`pixels for a field). At a level below the picture level is a
`"slice' level. A slice level is defined as series of one or more
`groups of 16-by-16 pixels that are aligned in horizontal rows
`about a frame. In general, Slices are used to identify Specific
`regions of pixel data that are common between Successive
`frames pictures. As described in FIG. 6-8 of the MPEG/
`Video Document, a picture may be carved into Several
`slices. Below the slice level is a "macroblock' level which
`identifies a Square block of 16-by-16 pixels. Thus, a Single
`720-by-480 MPEG-2 frame includes 1350 macroblocks
`arranged as 45 to a row over 30 rows. For a 4:2:0 chromi
`nance Structure, each macroblock includes four 8-by-8 pixel
`“luminance' blocks, and two 8-by-8 “chrominance' blocks
`(denoted chroma red and chroma blue).
`FIGS. 1A through 1C illustrate the method by which an
`MPEG-2 decoder receives a compressed digital bit stream
`and decompresses that bit stream using a combination of
`firmware, and hardware cores in order to increase decoding
`efficiency. FIG. 1A shows an exemplary MPEG-2 video bit
`stream that must be decoded, and FIGS. 1B and 1C present
`the method steps employed in decoding the Video bit Stream
`data. In a preferred embodiment, Video decoding Steps of
`FIG. 1B are performed exclusively by firmware, while the
`steps described in FIG. 1C are performed exclusively by
`hardware, including at least one core.
`The steps of FIG. 1B begin at a step 100 where a digital
`bit Stream is received. The method then proceeds to a step
`101 where a transport demultiplexer receives the bit stream.
`The transport demultiplexer functions as a Selection mecha
`nism which allows the identification and Selection of a
`particular channel. The channel Selection proceSS is neces
`sitated since the bit stream received at step 100 may include
`bit stream data for a number of different channels, each of
`which may contain independent content. Once a particular
`channel is Selected, the transport demultiplexer routes a
`packetized elementary stream (PES) for the selected chan
`nel. Consequently, the bit stream data information associated
`with unselected channels is simply disregarded during the
`decoding process.
`The method then proceeds to a step 102 where a packet
`demultiplexer receives the PES generated at step 101. The
`packet demultiplexer is generally designed to identify and
`Sort the PES into audio packets, Video packets, or Synchro
`nization (i.e., time stamp) packets. Once Sorted, the audio
`data will be diverted through an audio bit stream path 106,
`the video data will be diverted through a video bit stream
`path 104 and the synchronization data will be diverted
`through a synchronization bit stream path 108.
`For illustration purposes, a hypothetical packetized video
`bit stream is shown in FIG. 1A. The video bit stream begins
`with a video start code (VSC) 10. Following VSC 10 is a
`Sequence Start code 12 indicating the beginning of a new
`Sequence. Sequence Start code 12 is then followed by a
`Sequence header 14. AS is well known in the art, headers
`provide decoders with additional identifying characteristics
`about particular video data pertaining to the sequence (or
`other video unit) associated with the header.
`Returning now to FIG. 1B, the method continues from
`step 104 to a step 110 where sequence header 14 is decoded.
`As described in the MPEG/Video Document, a sequence
`header may contain a variety of identifying information Such
`as horizontal picture size, Vertical picture size, frame rate
`
`45
`
`50
`
`55
`
`60
`
`65
`
`8
`code, the bit rate code, quantization matrix information, etc.
`Once sequence header 14 is decoded, the method will
`proceed to a step 111 where a quantization matrix identified
`in Sequence header 14 is loaded into a predefined memory
`location on a video core (VCORE). For detailed discussion
`of the Video core's microarchitecture, reference may be
`made to a related pending U.S. patent application Ser. No.
`08/642,396 and prev