United States Patent [19J
`Stearns et al.
`
`111111
`
`1111111111111111111111111111111111111111111111111111111111111
`US005774676A
`[11] Patent Number:
`[45] Date of Patent:
`
`5,774,676
`*Jun.30, 1998
`
`[54] METHOD AND APPARATUS FOR
`DECOMPRESSION OF MPEG COMPRESSED
`DATA IN A COMPUTER SYSTEM
`
`95 32578 11/1995 WIPO .
`
`OTHER PUBLICATIONS
`
`[75]
`
`Inventors: Charles C. Stearns, San Jose;
`Stephanie W. Ti, Milpitas, both of
`Calif.
`
`[73]
`
`Assignee: S3, Incorporated, Santa Clara, Calif.
`
`[ * l
`
`Notice:
`
`The term of this patent shall not extend
`beyond the expiration date of Pat. No.
`5,774,676.
`
`[21]
`
`Appl. No.: 538,887
`
`[22]
`
`Filed:
`
`Oct. 3, 1995
`
`Int. Cl.6
`...................................................... G06F 17/00
`[51]
`[52] U.S. Cl. ......................................................... 395/200.77
`[58] Field of Search ............................ 364/514 R, 514 A,
`364/715.02; 395/200.77, 412, 439; 382/233,
`246, 234
`
`[56]
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`5,212,742
`5,335,321
`5,379,356
`5,394,534
`5,493,339
`5,557,538
`5,642,139
`
`....................... 382/233
`5/1993 Normille et a!.
`8/1994 Harney et a!. .......................... 395/162
`1!1995 Purcell et a!.
`. ... ... ... ... .... ... ... ... 382/233
`2/1995 Kulakowski et a!.
`.................. 395/425
`2/1996 Birch et a!. ............................. 348/461
`9/1996 Retter et a!. ... ... ... ... ... .... ... .. 364/514 R
`6/1997 Eglit et a!. .............................. 382/233
`
`FOREIGN PATENT DOCUMENTS
`
`0 503 956
`
`9/1992 European Pat. Off ..
`
`Foley, P.: "The MPACT Media Processor Redefines the
`Multimedia PC", IEEE Comput. Soc. Press, USA, pp.
`311-318, XP000577494, proceedings of Compean '96.
`Gruger, K., et al.: MPEG-1 Low Cost Encoder Solution,
`Proceedings of the SPIE-The International Society for
`Optical Engineering, 1995, USA, pp. 41-51, XP000577418,
`Advanced Image and Video Communications and Storage
`Technologies, Amsterdam, Netherlands, 20-23 Mar. 1995.
`
`Primary Examiner-Ellis B. Ramirez
`Attorney, Agent, or Firm---Skjerven, Morrill, MacPherson,
`Franklin & Friel LLP; Norman R. Klivans
`
`[57]
`
`ABSTRACT
`
`MPEG compressed data is decompressed in a computer
`system by sharing computational decompression tasks
`between the computer system host microprocessor, the
`graphics accelerator, and a dedicated MPEG processor in
`order to make best use of resources in the computer system.
`Thus the dedicated MPEG processor is of minimum capa(cid:173)
`bility and hence advantageously minimum cost. The host
`microprocessor is used to decompress the MPEG upper data
`layers. The more powerful the host microprocessor, the more
`upper data layers it decompresses. The remainder of the
`decompression (lower data layers) is performed by the
`MPEG dedicated processor and/or the graphics accelerator.
`
`26 Claims, 14 Drawing Sheets
`
`Microfiche Appendix Included
`(1 Microfiche, 51 Pages)
`
`I
`
`I
`
`Frame
`38~ Buffer
`
`50
`\
`Sound
`System ~---~---
`
`52
`\
`CD-ROM
`
`Graphics Ace.
`
`System
`Memory 1---36
`
`MPEG Ace.
`\
`408
`
`I
`
`I
`Peripheral Bus
`
`\
`42
`
`34
`\
`
`30
`\
`
`Micro
`
`Chip Set - Processor
`
`Lower layer
`decompression
`
`Upper layer
`decompression
`
`Page 1 of 24
`
` ZTE EXHIBIT 1007
`
`

`
`U.S. Patent
`
`Jun.30, 1998
`
`Sheet 1 of 14
`
`5,774,676
`
`Vv'hite /Green Book Layer
`MPEG System Layer
`I
`
`Video Layer
`Sequence
`Group of Pictures
`Picture
`Slice
`MacroBiock
`Block
`
`Audio Layer
`
`Layer II
`
`Private
`Memory
`
`\
`44
`
`~;_
`
`MPEG
`Accelerator
`
`_; __
`
`t \
`46
`
`I
`I
`I
`I
`
`Lower layer
`decompression
`
`FIG. 1
`
`Frame
`Buffer
`
`\
`38
`
`Graphics
`Accelerator
`\
`40
`
`42
`\
`Peripheral Bus
`
`FIG. 2
`
`Sound
`System
`\
`50
`
`52
`\
`
`CD-ROM
`
`System
`Memory
`
`\
`36
`
`Chip Set 1---
`
`\
`34
`Upper layer
`decompression
`
`Micro
`Processor
`\
`30
`
`Page 2 of 24
`
`

`
`U.S. Patent
`
`Jun.30, 1998
`
`Sheet 2 of 14
`
`5,774,676
`
`- - - - - - - - - - - - - - - - - - - - -
`
`Frame
`Buffer
`
`38
`
`Sound
`System
`
`MPEG
`Accelerator
`
`Graphics
`Accelerator
`
`. ,J--34
`
`36
`
`30
`
`Micro
`Processor
`
`.
`i 46
`52 t--: - _j_ _____ ~-Pe-r-iph_eL__ra-1 -Bu_s_.,__: .. _,, Chip Set
`!CD-ROM I :
`
`\
`
`'
`
`.
`
`42
`
`Lower layer
`decompression
`
`Upper layer
`decompression
`
`FIG. 3
`
`Frame
`~ Buffer
`
`38
`
`Graphics Ace.
`
`MPEG Ace.
`
`\ '
`
`408
`
`36
`
`34
`
`30
`
`Chip Set
`
`Micro
`Processor
`
`Lower layer
`decompression
`
`Upper layer
`decompression
`
`FIG. 4
`
`50
`\
`Sound
`System
`
`1--..;-
`
`52
`\
`CD-ROM
`
`Page 3 of 24
`
`

`
`~ 42
`
`Audio
`~ Decompression
`Module
`
`Memory
`Controller
`
`---
`Private
`Memory
`\
`44
`
`I
`
`~
`
`1--- 64
`
`68
`
`70
`
`I
`I
`I PCI
`I ! Master
`I and
`Slave
`I
`
`72
`
`I
`I
`
`- - - - - - - - - - - - - - - - - - - - - - - - - -
`
`d
`•
`\Jl
`•
`
`46
`\
`
`I
`-
`Video
`~ Decompression f---~
`Module ~ 6o
`1 Arbiter
`I
`I
`I
`
`Synchronization
`Module
`
`t--- 62
`
`Audio Display
`Module
`
`1-- 66
`
`Speakers
`
`50
`
`FIG. 5
`
`Page 4 of 24
`
`

`
`U.S. Patent
`
`Jun.30, 1998
`
`Sheet 4 of 14
`
`5,774,676
`
`Start
`Frame
`
`Start
`Frame
`
`Start
`Frame
`
`Start
`Frame
`
`Start
`Frame
`
`Start
`Frame
`
`Display
`
`VDE
`
`1
`
`decode
`IO frame
`r--
`Host ~I
`
`P3
`Software
`( Driver ) Program
`
`display
`IO frame
`-----1
`
`decode
`81 frame
`
`1
`
`i
`
`1 n~ecode
`
`~frame 1 I
`1
`
`display
`81 frame
`----1
`I
`
`1
`
`decode
`~frame
`
`1
`
`:
`
`~ram
`
`FIG. 6
`
`decode
`P6 frame
`
`1
`
`1
`
`display
`display
`P3 frame
`82 frame
`------l ~--i
`I
`I ~~ecode
`Ui frame
`I
`
`~ram
`
`Start
`Frame
`
`Start
`Frame
`
`Start
`Frame
`
`Start
`Frame
`
`Start
`Frame
`
`Start
`Frame
`
`Display
`
`VOE
`
`decode
`IO frame
`
`decode
`P3 frame
`
`I display
`I !0 fram~
`
`no display
`81 frame
`
`display
`82 frame
`--!
`
`display
`P3 frame
`
`decode
`decode
`82 frame
`81 frame
`~~----~~~~
`
`decode
`P6 frame
`
`decode
`84 frame
`
`I
`
`abandon
`81 frame
`
`Host
`( Sof~ware) P3
`Program
`\ Dnver
`
`81
`
`P6
`
`Program
`
`Program
`
`Program
`
`84
`Program
`
`FIG. 7
`
`Page 5 of 24
`
`

`
`U.S. Patent
`
`Jun.30, 1998
`
`Sheet 5 of 14
`
`5,774,676
`
`84
`
`96
`... ~
`VLD r-- Buffer
`A
`
`--·
`
`!
`i
`
`Q
`Matrix
`
`1--104
`
`1----
`
`IO/ IZZ/
`IDCT
`
`\
`88
`
`L__._
`
`1\0
`
`Buffer f--
`B
`
`92
`\
`FR 1-----
`
`~
`
`To
`Memory
`
`Buffer
`I
`c
`102 l _____ _j
`
`l
`
`I
`
`Master Controller
`
`1---- 82 FIG 8
`
`0
`8
`16
`24
`32
`40
`48
`56
`
`0
`2
`3
`9
`10
`20
`21
`35
`
`9
`17
`25
`33
`41
`49
`57
`
`4
`8
`11
`19
`22
`34
`36
`
`2
`10
`18
`26
`34
`42
`50
`58
`
`5
`7
`12
`18
`23
`33
`37
`48
`
`5
`13
`21
`29
`37
`45
`53
`61
`
`15
`26
`30
`40
`45
`51
`56
`58
`
`3
`11
`19
`27
`35
`43
`51
`59
`
`6
`13
`17
`24
`32
`38
`47
`49
`
`4
`12
`20
`28
`36
`44
`52
`60
`
`j
`
`14
`16
`25
`31
`39
`46
`50
`57
`
`FIG. 9
`
`6
`14
`22
`30
`38
`46
`54
`62
`
`27
`29
`41
`44
`52
`55
`59
`62
`
`7
`15
`23
`31
`39
`47
`55
`63
`
`28
`42
`43
`53
`54
`60
`61
`63
`
`Page 6 of 24
`
`

`
`U.S. Patent
`
`Jun.30, 1998
`
`Sheet 6 of 14
`
`5,774,676
`
`OJ [D . B UJ ITJ OJ lTI OJ [iJ lTI
`
`:1
`
`'
`I
`
`2
`
`I
`I
`
`3
`
`!
`
`4
`
`5
`
`6
`
`7
`
`8
`
`~
`
`FIG. 10
`
`1 begin
`
`end 2 j 3 begin
`
`end 1 j2 begin
`end 3 j 4 begin
`
`--
`
`[------- -----------~---
`
`r-
`~-=---=
`~ .-
`sj9 begin
`
`i
`!
`r-------------,~--
`
`end 7j8
`
`end 415 begin
`
`end 6!7 begin
`end 9j1o begin
`
`end 5
`
`end 10
`
`FIG. 11
`
`Page 7 of 24
`
`

`
`U.S. Patent
`
`Jun.30, 1998
`
`Sheet 7 of 14
`
`5,774,676
`
`DATA_IN
`
`140
`
`142
`
`DOWN_HALF
`
`150
`
`Page 8 of 24
`
`

`
`U.S. Patent
`
`Jun.30, 1998
`
`Sheet 8 of 14
`
`5,774,676
`
`FORWARD REF. FRAME
`
`BACKWARD REF. FRAME
`
`HORIZ. INTERP.
`
`VERTICAL INTERP.
`
`118
`
`122
`
`HORIZ. INTERP.
`
`VERTICAL INTERP.
`
`118A
`
`122A
`
`POST PROCESSING
`
`124
`
`WR_DATA
`FIG. 13A
`
`DATAJN
`
`r-178
`
`IDCT MEMORY
`
`1--- 180
`
`HORIZ. INTERP.
`
`VERTICAL INTERP.
`
`f---118
`
`--
`
`-122
`
`POST PROCESSING
`
`1--- 124
`
`WR_DATA
`
`FIG. 13B
`
`Page 9 of 24
`
`

`
`U.S. Patent
`
`Jun.30, 1998
`
`Sheet 9 of 14
`
`5,774,676
`
`MPEG Driver
`
`~
`
`MPG_Init
`Initialize drivers.
`
`~
`
`- -
`
`MPG_Open
`Open MPEG file, prepare
`to read and parse.
`~
`Read MPEG file data during initialization.
`Send Video packets to VDE driver.
`Send Audio packets to ADE driver.
`
`I
`I
`
`~
`
`MPG_Oecode
`Start audio and video
`decode and playback.
`
`r ___________ L _________
`
`VDE_Init
`Allocate system memory buffers.
`
`VDE_Open
`Initialize pointers and variables.
`
`VDE_AddPacket
`Parse video packet data into
`Header buffer and Picture buffer.
`
`VDE_Decode
`Program CP2 to start VDE decoding.
`
`- ·
`
`[ Read MPEG file data as needed to keep
`audio buffers filled. Send any video
`packets encountered to VDE driver.
`
`1----t....,
`
`VDE_AddPacket
`Parse video packet data into
`Header buffer and Picture buffer.
`
`MPG_Ciose
`Close MPEG file, terminate decode.
`
`VDE_Ciose
`Make sure VDE is stopped.
`
`MPG_Exit
`Deinitialize drivers.
`
`VDE_Exit
`Free system memory buffers.
`
`FIG. 14A
`
`Page 10 of 24
`
`

`
`U.S. Patent
`
`Jun.30, 1998
`
`Sheet 10 of 14
`
`5,774,676
`
`[
`
`VDE_Init
`
`~
`
`I
`
`Allocate system memory buffers.
`
`~
`
`I
`
`Raw Buffer
`is used to hold raw video packet
`data until it can be parsed.
`~
`
`Header Buffer
`is used to hold parameters extracted from
`Sequence, Group, and Picture headers.
`Used to program CP2 registers.
`
`1
`
`Picture Buffer
`is used to hold picture layer data, to be
`copied later into CP2 private memory.
`··~
`
`I
`Target Buffers
`are two buffers where CP2 will transfer
`decompressed frames using PCI bus master.
`
`l
`
`End
`
`1
`
`I
`
`Initialize variables for new MPEG file.
`Prepare to receive video packets.
`
`FIG. 14B
`
`Page 11 of 24
`
`

`
`U.S. Patent
`
`Jun.30, 1998
`
`Sheet 11 of 14
`
`5,774,676
`
`I VDE_AddPacket J
`~
`
`Extract video PTS, if any.
`
`~
`
`Copy rest of video packet into Raw
`buffer temporarily, appending it to any
`leftover data from previous packet.
`
`~
`
`Parse packet data in Raw buffer.
`
`~.
`
`If Sequence header found, extract
`image size, quantizer matrices, etc.
`
`1
`
`If Group header found,
`extract time code, etc.
`
`1
`
`If Picture header found, extract
`temporal reference, picture type, etc.
`Calculate PTS if none was given.
`
`!
`
`Locate end of picture,
`and pad with picture end code.
`
`~
`
`If End of Sequence found,
`mark end of video sequence.
`
`~
`
`End
`
`I
`
`J
`
`FIG. 14C
`
`Copy into Header buffer.
`
`Copy into Header buffer.
`
`Copy into Header buffer.
`
`Copy into Picture buffer.
`
`Copy into Header buffer.
`
`Page 12 of 24
`
`

`
`U.S. Patent
`
`Jun.30, 1998
`
`Sheet 12 of 14
`
`5,774,676
`
`Program CP2 to partition private memory:
`VDE Input buffers Ping and Pong,
`3 VDE Reference Frame buffers.
`
`Fill VDE Input Ping and Pong
`buffers with picture data.
`
`Get from Picture buffer.
`
`Program CP2 with Sequence information:
`Image size, quantization matrices.
`
`Get from Header buffer.
`
`I
`[ Initialize STC to a reasonable value.
`
`J
`
`Program CP2 to decode first picture:
`VPTS, Picture Offset, Picture Type, etc.
`
`l
`
`Start VDE.
`
`.~
`
`Get from Header buffer.
`
`Program CP2 to decode second picture:
`VPTS, Picture Offset, Picture Type, etc.
`
`Get from Header buffer.
`
`~
`
`End
`
`l
`
`I
`
`FIG. 14D
`
`Page 13 of 24
`
`

`
`U.S. Patent
`
`Jun.30, 1998
`
`Sheet 13 of 14
`
`5,774,676
`
`DV-IRQ Handler
`~
`Called when SCR=VPTS. Indicates start
`of PCI master transfer of picture PN.
`and start of decoding of picture PNt1·
`~
`Check next entry in Header buffer.
`if next entry is End of Sequence, stop.
`~
`If next entry is Sequence Header, program
`CP2 with new quantization matrices.
`~
`If next entry is Group Header, reset
`some counters to start the next group.
`
`I '
`
`If next entry is Picture Header, program
`CP2 for next picture P N+2:
`VPTS, Picture Offset, Picture Type, etc.
`~
`Send finished picture PN_1 from system
`memory buffer to 868 pixel formatter.
`
`~
`
`End
`
`~
`
`Get from Header buffer.
`
`Get from Header buffer.
`
`-
`
`--
`Get from Header buffer.
`
`Get from Header buffer.
`
`FIG. 14E
`
`Page 14 of 24
`
`

`
`U.S. Patent
`
`Jun.30, 1998
`
`Sheet 14 of 14
`
`5,774,676
`
`I CV _IRQ Handler I
`1
`
`Called when CP2 detects that one of
`the VDE Input Ping or Pong buffer
`has been consumed.
`
`T
`
`Fill Ping or Pong buffer with next
`block of picture data.
`T
`I
`
`~
`
`VDE_Ciose
`~
`Make sure VDE and timers are stopped.
`
`1
`
`End
`
`~~
`
`f
`
`VDE_Exit
`
`~
`Free system memory buffers
`allocated by VDE_Init.
`~
`End
`
`I
`
`I
`
`Get from Picture buffer.
`
`FIG. 14F
`
`Page 15 of 24
`
`

`
`5,774,676
`
`5
`
`2
`microcontroller. The book layer and entire MPEG system
`layer parsed to the last pixel of the compressed data are
`decompressed using the C-Cube Microsystems products.
`Thus these chips accomplish the entire decompression on
`their own, because these chips are intended for use in
`consumer type devices (not computers). Thus these chips
`include a system memory, a CD-ROM controller and any
`necessary processing power to perform complete MPEG
`decompression.
`Similar products are commercially available from a vari(cid:173)
`ety of companies. While these products perform the decom(cid:173)
`pression task fully in a functional manner, they are relatively
`expensive due to their inclusion of the large number of
`functions dedicated to MPEG decompression. Thus their
`15 commercial success has been limited by high cost.
`
`10
`
`1
`METHOD AND APPARATUS FOR
`DECOMPRESSION OF MPEG COMPRESSED
`DATA IN A COMPUTER SYSTEM
`
`CROSS-REFERENCE TO RELATED
`APPLICATIONS
`
`This application is related to copending and commonly
`owned U.S. patent applications Ser. No. 08/489,488, filed
`Jun. 12, 1995, entitled "Decompression of MPEG Com-
`pressed Data in a Computer System", Charles C. Stearns,
`Ser. No. 08/490,322, filed Jun. 12, 1995, entitled "Video
`Decoder Engine", Soma Bhattacharjee et al., Ser. No.
`08/489,489, filed Jun. 12, 1995, entitled "Audio Decoder
`Engine," Charlene S. Ku et al., and Ser. No. 08/508,636,
`filed Jul. 28, 1995, entitled "Frame Reconstruction For
`Video Data Compression", Stephanie W. Ti et al., all incor(cid:173)
`porated by reference.
`
`MICROFICHE APPENDIX
`
`A microfiche appendix including 1 fiche and a total of 51
`frames is a part of this disclosure.
`
`BACKGROUND OF THE INVENTION
`
`SUMMARY
`It has been recognized by the present inventors that in a
`computer (i.e., personal computer or workstation)
`20 environment, that already available elements are capable of
`performing a large portion of the MPEG decompression
`task. Thus in this environment use of a dedicated fully
`functional MPEG decompression integrated circuit is not
`necessary, and instead a substantial portion of the decom-
`25 pression can be off-loaded onto other conventional computer
`system elements. Thus only a relatively small portion of the
`actual data decompression must be performed by dedicated
`circuitry, if any. In accordance with the invention, the MPEG
`decompression task is allocated amongst various already
`30 existing elements of a typical computer system and if
`necessary, depending on the capabilities of these other
`elements, an additional relatively small (hence inexpensive)
`dedicated MPEG decompression circuit is provided.
`Thus advantageously in accordance with the present
`invention the MPEG (compressed using layers) content of
`data is decompressed in a computer system typically already
`including a microprocessor, graphics accelerator, frame
`buffer, peripheral bus and system memory. A shared com-
`putational approach between the microprocessor (host
`processor), graphics accelerator and a dedicated device
`makes best use of the computer system existing resources.
`This is a significant advantage over the prior art where the
`MPEG decompression is performed entirely by a dedicated
`processor. Thus in accordance with the invention by parti(cid:173)
`tioning of the decompression process amongst the major
`available elements in a personal computer, decompression is
`provided inexpensively.
`Further, in accordance with the present invention frame
`reconstruction is carried out by logic circuitry including
`50 three main elements which are a horizontal interpolation
`element, a vertical interpolation element, and a post pro(cid:173)
`cessing element. The horizontal interpolation element inter(cid:173)
`polates two adjacent (horizontally adjacent) pixels in one
`particular MPEG block of pixels. That is, this is a digital
`averaging filter. The output of the horizontal interpolation
`element is then provided to the vertical interpolation element
`which similarly interpolates (averages) two pixels which are
`vertically adjacent in that same MPEG block. (In another
`embodiment, the vertical interpolation is before the hori(cid:173)
`zontal interpolation.) In the post-processing element (which
`is a selector) the vertically and horizontally interpolated data
`is processed in conjunction with externally provided IDCT
`MPEG decompressed data to generate the final output data.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`FIG. 1 shows conventional content layering for MPEG
`compression.
`
`1. Field of the Invention
`This invention relates to data decompression, and specifi(cid:173)
`cally to decompression of MPEG compressed data in a
`computer system.
`2. Description of Prior Art
`The well-known MPEG (Motion Picture Experts Group)
`data standard defines two compression/decompression
`processes, called conventionally MPEG 1 and MPEG 2. For
`purposes of this disclosure, MPEG 1 and MPEG 2 are
`similar. The MPEG 1 standard is described in the ISO 35
`publication No. ISO/IEC 11172: 1993(E), "Coding for mov(cid:173)
`ing pictures and associated audio ... ", and the MPEG 2
`standard is defined in the ISO publication No. ISO!IEC
`13818-2, both incorporated by reference herein in their
`entirety. The MPEG standard defines the format of com- 40
`pressed audio and video data especially adapted for e.g.,
`motion pictures or other live video. MPEG compression is
`also suitable for other types of data including still pictures,
`text, etc. The MPEG standard in brief (the above-mentioned
`publications are more complete) defines the data format 45
`structure shown in FIG. 1 for CD-ROM content. The top
`required layer is the MPEG system layer having underneath
`it, in parallel, the video layer and audio layer. The MPEG
`system layer contains control data describing the video and
`audio layers.
`Above (wrapped around) the MPEG system layer is
`another (optional) layer called the White book ("video CD")
`or the Green book ("CDI") that includes more information
`about the particular program (movie). For instance, the book
`layer could include Karaoke type information, high resolu- 55
`tion still images, or other data about how the program
`content should appear on the screen. The video layer
`includes sequence (video), picture (frame), slice (horizontal
`portions of a frame), macroblock (64 pixels by 64 pixels)
`and block (8 pixels by 8 pixels) layers, the format of each of 60
`which is described in detail by the MPEG standard.
`There are commercially available integrated circuits
`(chips) for MPEG decompression. Examples are those sold
`by C-Cube Microsystems and called the CL-450 and
`CL-480 products. In these products the MPEG audio and 65
`visual decompression (of all layers) is accomplished com(cid:173)
`pletely in dedicated circuitry in an internally programmable
`
`Page 16 of 24
`
`

`
`5,774,676
`
`3
`FIG. 2 shows one embodiment of the invention with
`partitioning of decompression including a dedicated MPEG
`processor with associated private memory, in a computer.
`FIG. 3 shows a second embodiment of the invention also
`with a dedicated MPEG processor in a computer.
`FIG. 4 shows a third embodiment of the invention with
`partitioning of MPEG compression in a computer system
`using a high performance graphics accelerator.
`FIG. 5 shows a block diagram of a chip including MPEG
`video and audio decompression in accordance with the
`invention.
`FIG. 6 shows host processorNDE partitioning of video
`decompression.
`FIG. 7 shows graceful degradation of video decompres(cid:173)
`sion by abandoning frames.
`FIG. 8 shows in a block diagram three stage pipelining in
`the VDE.
`FIG. 9 shows a transparent IZZ process.
`FIG. 10 shows a group of pictures in display order in
`accordance with MPEG compression for frame reconstruc(cid:173)
`tion.
`FIG. 11 shows an arrangement of slices in a picture in
`accordance with MPEG compression.
`FIG. 12 shows a frame reconstruction circuit in accor(cid:173)
`dance with the present invention.
`FIGS. 13A and 13B show respectively parallel and serial
`processing in a frame reconstruction circuit as used for
`reconstruction of a B-type frame in accordance with the
`present invention.
`FIGS. 14A to 14F show a flowchart for a computer
`program for performing higher level video decompression in
`a host processor.
`Identical reference numbers in different figures refer to
`similar or identical structures.
`
`DETAILED DESCRIPTION OF 1HE
`PREFERRED EMBODIMENTS
`
`40
`
`As well known, each element in a computer system (e.g.,
`personal computer or workstation) has particular strength
`and weaknesses. For instance, the microprocessor (host
`processor) is typically the single most capable and expensive
`circuit in a computer system. It is intended to execute a 45
`single instruction stream with control flow and conditional
`branching in minimum time. Due to its internal arithmetic
`units, the microprocessor has high capability for data pars(cid:173)
`ing and data dependent program execution. However, the
`microprocessor is less capable at transferring large quanti- 50
`ties of data, especially data originating from peripheral
`elements of the computer.
`The core logic chip set of a computer interfaces the
`microprocessor to the peripherals, manages the memory
`subsystem, arbitrates usage and maintains coherency. 55
`However, it has no computational capabilities of its own.
`The graphics subsystem manages and generates the data
`which is local to the frame buffer for storing video and
`graphics data. The graphics subsystem has a capability to
`transfer large amounts of data but is not optimized for 60
`control flow conditional branching operation.
`The present inventors have recognized that in MPEG
`compressed content (video data) having the various layers,
`each layer has certain characteristics requiring particular
`hardware (circuit) properties to parse that level of informa- 65
`tion. For example, it has been determined that in the book
`and system layers of MPEG, which are the top most layers
`
`5
`
`10
`
`20
`
`35
`
`4
`in the video data stream, the information resembles a pro(cid:173)
`gram data/code data stream and in fact may contain execut(cid:173)
`able code (software). The information at that level is thus
`like a program code stream containing control flow
`information, variable assignments and data structures.
`Hence it has been recognized that the microprocessor is
`suited for parsing such information. (The term "parsing"
`herein indicates the steps necessary to decompress data each
`layer of the type defined by the MPEG standard.)
`The video layer, under the system layer, includes the
`compressed video content. There are as described above an
`additional six layers under the video layer as shown in FIG.
`1. These layers are the sequence layer, group of pictures
`layer, picture layer, slice layer, macroblock layer, and block
`15 layer. All but the macroblock and block layers contain
`additional control and variable information similar to the
`type of information in the system layer. Thus again the
`microprocessor is best suited for parsing the information
`down to but not including the macroblock layer.
`Within the macroblock and block layers are compressed
`pixel data that requires, according to MPEG decompression,
`steps including 1) variable length decoding (VLD), 2)
`inverse zig-zagging (IZZ), 3) inverse quantization (IQ), 4)
`inverse discrete cosine transformation (IDCT), and 5)
`25 motion vector compensation (MVC), in that order. The
`VLD, IZZ, IQ, and especially IDCT are computationally
`intensive operations, and suitable for a peripheral processor
`or the microprocessor capabilities, assuming adequate pro(cid:173)
`cessing capability being available in the microprocessor.
`30 However, in some cases depending on the microprocessor
`capabilities, the microprocessor itself may be insufficient in
`power or completely utilized already for parsing the upper
`layers.
`The remaining task for video decompression is motion
`vector compensation (MVC) also referred to as frame recon(cid:173)
`struction (FR). MVC requires retrieving large quantities of
`data from previously decompressed frames to reconstruct
`new frames. This process requires transferring large
`amounts of video data and hence is suited for the graphics
`accelerator conventionally present in a computer system. An
`example of such a graphics accelerator is the Trident
`TVP9512, or S3 Inc. Trio 64V.
`The audio stream layer under the system layer includes
`the compressed audio content. Audio decompression
`requires 1) variable length decoding, 2) windowing, and 3)
`filtering. Since audio sampling rates are lower than pixel
`(video) sampling rates, computational power and data band(cid:173)
`width requirements for audio decompression are relatively
`low. Therefore, a microprocessor may be capable of accom(cid:173)
`plishing this task completely, assuming it has sufficient
`computational power available.
`Thus in accordance with the invention the MPEG decom(cid:173)
`pression process is partitioned between the various hardware
`components in a computer system according to the compu(cid:173)
`tational and data bandwidth requirements of the MPEG
`decompression. Thus the system partitioning depends on the
`processing power of the microprocessor.
`Therefore, while the present invention is applicable to
`computers including various microprocessors of the types
`now commercially and to be available, the following
`description is of a computer systems having a particular
`class of microprocessor (the 486DX2 class microprocessors
`commercially available from e.g., Intel and Advanced Micro
`Devices.) Thus this description is illustrative and the prin(cid:173)
`ciples disclosed herein are applicable to other types of
`computer systems including other microprocessors of all
`
`Page 17 of 24
`
`

`
`5,774,676
`
`5
`types. As a general rule, it has been found empirically that
`no more than 30% of the microprocessor's computing
`capability should be used for MPEG decompression in order
`to preserve the remaining portion for other tasks. It has to be
`understood that this rule of thumb subjective and somewhat
`arbitrary; it is not to be is construed as limiting.
`Moreover, the actual steps of MPEG decompression and
`apparatus to perform same are well known; see e.g. U.S. Pat.
`No. 5,196,946 issued Mar. 23, 1993 to Balkanski et al.; U.S.
`Pat. No. 5,379,356 issued Jan. 3, 1995 to Purcell et al., and
`European Patent Application publication 93304152-7, pub(cid:173)
`lished Jan. 12, 1993, applicant C-Cube Microsystems, Inc.
`Therefore one skilled in the art will understand how to
`implement these well-known functions, which may be car(cid:173)
`ried out in a variety of ways, all of which are contemplated 15
`in accordance with the invention.
`In accordance with the first embodiment of the present
`invention shown in FIG. 2, microprocessor 30 (the host
`processor) has been found only to have computational power
`sufficient to decompress the MPEG book layer and system 20
`layer. Also, in this computer system the graphics accelerator
`40 e.g., the Trio 64V chip from S3 Inc. has insufficient
`computing power to accomplish the motion vector compen(cid:173)
`sation (MVC) decompression. Therefore, a dedicated pro(cid:173)
`cessor called the MPEG accelerator 46 is provided to 25
`perform the remainder of the MPEG decompression tasks. It
`is to be understood that the MPEG accelerator 46 may be
`any suitable processor or dedicated logic circuit adapted for
`performing the required functions. The private memory 44 is
`e.g. one half megabyte of random access memory used to 30
`accomplish the MVC and is distinct from the frame buffer in
`the FIG. 1 embodiment.
`The other elements shown herein including the system
`memory 36, chip set 34, sound system 50, CD-ROM player
`52, and the peripheral bus 42, are conventional. In one 35
`version of the FIG. 2 embodiment as shown by the dotted
`line connecting MPEG accelerator 46 to PCI (peripheral)
`bus 42, the MPEG accelerator 46 is connected to PCI bus 42
`for video and audio decompression and typically would be
`a chip on an add-in card. The type of microprocessor 30, 40
`how the sound system 50 and other elements are connected,
`and the particular interconnection between the MPEG accel(cid:173)
`erator 46 and the peripheral bus 42 are not critical to the
`present invention. Further, the particular partitioning
`described herein is not critical to the present invention but is 45
`intended to be illustrative.
`In a second version of the FIG. 2 embodiment, MPEG
`accelerator connects (see dotted lines) directly to graphics
`accelerator 40 for video decompression and to sound system 50
`50 for audio decompression, not via peripheral bus 42. This
`version would be typical where MPEG accelerator 46 is
`located on the motherboard of the computer.
`In FIG. 2, the lower layer MPEG decompression includes
`the functions performed by the private memory 44 and the 55
`MPEG accelerator 46. The upper layer decompression is that
`performed by microprocessor 30.
`It is to be understood that typically the source of the
`MPEG program material is a CD-ROM to be played on
`CD-ROM player 52. However, this is not limiting and the 60
`program material may be provided from other means such as
`an external source.
`A second embodiment is shown in FIG. 3. Again, here the
`486 class microprocessor 30 has sufficient computational
`power only to decompress the book layer and the system 65
`layer. In this embodiment a more capable graphics accel(cid:173)
`erator 40A has the capability to perform the MPEG decom-
`
`6
`pression motion vector compensation (MVC). Therefore,
`the memory requirement for accomplishing MVC, which
`was accomplished by the private memory 44 in FIG. 2, here
`takes place either in the frame buffer 38 or the system
`5 memory 36. Therefore, in this case the lower layer decom(cid:173)
`pression includes the functions performed by the graphics
`accelerator 40A, unlike the case with FIG. 2.
`The FIG. 3 embodiment, like that of FIG. 12, has two
`versions as shown by the dotted lines. In the first version,
`10 MPEG accelerator 46 communicates via peripheral bus 42.
`In the second version, MPEG accelerator 46 is directly
`connected to sound system 50 for audio decompression and
`to graphics accelerator 40A for video decompression.
`A third embodiment is shown in FIG. 4. In this case the
`MPEG accelerator functionality is included in a yet more
`powerful graphics accelerator 40B (a graphics controller).
`As in the embodiment of FIG. 3, the memory storage
`requirements for motion vector compensation (MVC) are
`satisfied by the off-screen memory in the frame buffer 38 or
`a non-cacheable portion of the system memory 36. The
`decompression of the audio layer is performed by either the
`sound system 50, the graphics accelerator 40A, or the
`microprocessor 30.
`Also, in accordance with the invention there may be a
`partitioning of the audio decompression between the micro(cid:173)
`processor 30 and a dedicated audio decompression processor
`which may be part of the MPEG accelerator. A system of this
`type for audio decompression is disclosed in the above
`mentioned U.S. patent application Ser. No. 08/489,489, filed
`Jun. 12, 1995, entitled "Audio Decoder Engine", Charlene
`Ku et al.
`Thus in accordance with the invention the MPEG decom(cid:173)
`pression process is partitioned between various elements of
`a computer system. The more powerful the host
`microprocessor, the more upper layer decompression tasks it
`handles. The remainder of the decompression tasks are
`off-loaded to a dedicated MPEG accelerator (processor)
`circuit, or to a graphics accelerator already conventionally
`present in a computer system on a layer-by-layer basis. Thus
`the need for dedicated circuitry for MPEG decompression is
`minimized in accordance with the capabilities of the other
`elements of the computer system, hence reducing total
`computer system cost and making MPEG decompression
`more widely available even in low cost computer systems.
`The various elements of FIGS. 2, 3, and 4 are
`conventional, as is their interconnection, except for the
`MPEG accelerator and the decompression software in the
`microprocessor.
`The following describes a system as shown in present
`FIG. 2 for video decompression. This particular embodiment
`of the invention is illustrative and is for MPEG 1 decom(cid:173)
`pression. The two chief elements disclosed herein are (1) the
`software driver (program) executed by the microprocessor
`which performs the upper layer video decompression, and
`(2) the MPEG accelerator circui