/
`
`PATENT DATE
`
`, . 99
`FEB O ~ i~
`
`PATENT
`NUMBER
`
`8EAIAL NUMBER
`
`ALING DATE Cl.ASS
`
`J •
`
`,i
`
`. ,,
`
`..
`
`FILING FEE
`RECEIVED
`
`ATTORNEY'S
`DOCKETNO.
`
`U.S. DEPT. OF COMM./ PAT. & TM-P'J'0..438t. (Rev.1 !
`
`PARTS OF APPLICATION
`ALEO SEPARATELY
`
`NOTICE OF ALLOWANCE MAILED
`
`ISSUE FEE
`Amount Due
`
`r·
`·) I . , )
`J \)0 'J'r.,
`
`Label
`Area
`
`Form PT0~436A
`(Rev. 8/92)
`
`/
`Q,Li~ N
`/
`Assistant Examiner
`
`Vo
`
`Total Claims/
`
`:);6·
`
`/
`
`I
`
`Sheet$. Drwg. u.
`?H'J K NGUYEN
`PfllMARY EXAMINER
`,,_
`L/.
`ISSUE
`GROIJP 2400
`· / ~ (0 3
`,:j,v.; tll'<fr BATCH
`Primary Examiner NUMBER
`PREPARED FOR ISSUE
`
`WARNING: The information disclosed herein may be restricted,
`by the United States Code Title 35, Sections 1
`Paten! & Trademark Office is restricted to au
`
`uthorized disclosure may be prohibited
`81 and 368. Possession outside the U,S,
`'zed ernpioyees and contractors only
`
`AMAZON 1002
`Page 1 of 148
`
`

`

`08J74S269
`
`PATENT APPLiCATION
`.~PPROVED FOR LICENSE I.I .
`!Jt1, •• l 9 L;:r
`· 1111m111~1m1m11111-
`6 3 ~
`.:S
`INITIALS
`I
`•
`·. ~· <·
`
`1
`
`08748269
`
`Date
`Entered
`or
`Counted
`
`CONTENTS
`
`Date
`Received
`or
`Malled
`
`papers.
`
`7.
`
`8.
`
`9.
`
`10.
`
`11.
`
`12.
`
`13.
`
`14.
`
`15.
`
`16.
`
`17.
`
`18.
`
`19.
`
`20.
`
`21.
`
`22.
`
`23.
`
`24.
`
`25.
`
`26.
`
`27.
`
`28.
`
`29.
`
`30.
`
`31.
`
`32.
`
`(FRONT}
`
`AMAZON 1002
`Page 2 of 148
`
`

`

`..
`Staple Issue 9"
`
`,
`
`IONO.
`
`-7
`I/ v·
`
`J •
`C/t.L)
`27',
`
`.
`
`DATE
`.
`. '?i'.;-
`,::/ f/
`i I 14191-
`/-i7-·Cfi(
`, - !J
`
`POSITION
`CLASSIFIER
`EXAMINER
`TYPIST
`VERIFIER
`
`. CORPS CORR .
`
`~-
`
`SPEC.HANO
`FILE MAINT.
`DRAFTING
`
`INDEX OF CLAIMS
`
`1 Claim
`
`•
`• C
`~ :gi
`0
`51
`52
`, 53
`54
`155
`i 56,
`57
`58
`
`'
`I
`i
`
`r= 59
`
`'
`I
`i
`I
`
`Date
`
`I ! I
`-+-+
`
`I
`!
`
`I
`
`i
`
`l
`!
`'
`
`I
`
`I
`
`I
`
`'
`
`i
`
`i
`
`;
`
`!
`
`i
`
`'
`
`i
`
`I
`i
`
`I
`;
`
`I
`.
`
`i
`
`-
`
`-
`
`i
`
`I
`
`i
`I
`i
`'
`'
`I I i
`
`'
`
`I
`!
`
`'
`
`00 I
`61
`62
`. 63
`64.
`;es
`66
`I 67
`
`i
`
`68
`69
`70
`. 71
`72
`i 73
`
`'
`
`i
`
`~.
`
`I
`
`'
`
`'
`I
`'
`
`tn:i t-
`
`I
`
`·q, . 8
`
`'
`
`i
`I
`:
`
`. .
`
`I
`
`'
`
`i
`
`...
`
`i
`
`'
`
`'
`
`L-I
`
`!
`
`!
`
`f-
`
`'
`
`25
`26
`. 27
`28
`
`I
`
`..
`
`'
`
`'
`--+
`i
`
`·-
`·-
`
`'
`
`·-, -----L-
`I
`
`-->·--+-~·---;
`
`I
`··+-·+--·+· -+·-+-·--1
`
`SYMBOLS
`Rejected
`v ...
`AHowed
`"" ,, .
`• \Hunugh num!;f:ra!) Ctmctltd
`+ ..
`RestrieteO
`N ....
`""" Noo-e~ted
`. .. lnterterenoo
`f .•
`A ...
`. ...• A;>peal
`0
`Obiecmd
`
`;
`
`i
`
`76
`77
`78
`79
`80
`; 81
`i 82
`83
`'84
`85
`86
`87
`88'
`. 89
`90 I
`! 91
`
`~ 92'
`
`I
`i
`
`93
`94 1
`95
`96
`97
`98
`99
`too
`
`I I
`
`I
`
`i
`I
`!
`-'--
`I
`I
`
`'
`
`I
`!
`i
`'
`
`I
`I
`!
`I
`
`I
`I
`I
`
`'
`'
`
`i
`I
`
`I
`
`I
`
`'
`
`i
`!
`I
`I
`I
`
`.
`i
`
`I
`I
`
`i
`I
`
`I
`I
`
`I
`I
`
`i
`
`'
`
`'-j
`
`·-
`
`i
`
`I
`
`i
`i
`
`' 1-1-_
`
`i
`
`'
`
`:
`
`_J
`
`+-H
`
`'
`
`'
`
`i
`
`'
`
`I
`
`'
`
`I
`
`i
`
`i
`
`I
`I
`
`(LEFT INSIDE\
`
`~.
`30
`31
`32 1
`~~---~
`, 34
`35 ,
`36
`SI
`38
`39
`40
`41
`42
`143
`I 44 •
`45
`46
`47
`48
`149
`50,
`
`AMAZON 1002
`Page 3 of 148
`
`

`

`Date
`
`I · Exrnr.
`
`f /t? · ·· v
`
`~-'-
`
`k"<...ivl.l<:..V~
`
`t/J-(i' 1 K tJ
`
`SEARCH NOTES,/·
`
`Exmr.
`
`,I
`
`INTERFERENCE SEARCHED ,
`Sub.
`Class
`Date
`Exmr/
`Ii
`~.::>,. e1i 1 Jig c/
`
`').,.
`/
`'\'. \
`
`AMAZON 1002
`Page 4 of 148
`
`

`

`PATENT NUMBER
`
`' .
`
`Ct.ASS
`
`'
`ORIGINAL CI.ASSIFICATION
`SUBCLASS
`14(
`'2; 0 ;;;)--
`
`APPUCAT!ON SERIAL NUMBER
`
`O( 74\.J.(q
`
`APPUCA1 : I 'S NAME (Pt:EASE PRINT!
`
`\.,t\. A 0
`
`lF "REISSUE, ORJGINAL PATENT NUMBER
`
`INTERNATIONAL CLASSIFICATION
`
`c'.1 ()
`
`b T
`
`PT0270
`\AEV.5.Q:1)
`
`11/00
`I
`/
`I
`
`CLASS
`
`CROSS REFERENCE(S)
`SU6CI.ASS
`{ONE SUBC1.ASS PER BLOCK!
`_,,,./'
`
`j
`
`-V
`
`/
`
`.,. _,.
`
`/
`
`/
`
`./
`GROUP
`ART UNIT
`
`ASS1ST ANT EXAMINER jPLEASE ST AMP OR PRINT FULL NAME)
`
`\Jo
`~Li PF
`'J. 77J PRIMARY EXAMINER (PLEASE STAMP OR PRINT FULL NAMEl
`Pt-i\.l t.
`"'I/
`II YGrJ
`OMMERCE
`U.S. DE!'ARTMENT OF C
`ISSUE CLASSIFICATION SUP
`PA TENT A.ND TRADt:MARK OfFICE
`
`AMAZON 1002
`Page 5 of 148
`
`

`

`United States Patent [19J
`Chau
`
`I IIIII IIIIIIII Ill lllll lllll lllll lllll lllll lllll lllll lllll 111111111111111111
`US005870087 A
`[11] Patent Number:
`[45] Date of Patent:
`
`5,870,087
`Feb. 9, 1999
`
`[54] MPEG DECODER SYSTEM AND METHOD
`HAVING A UNIFIED MEMORY FOR
`TRANSPORT DECODE AND SYSTEM
`CONTROLLER FUNCTIONS
`
`[75]
`
`Inventor: Kwok Kit Chau, Los Altos, Calif.
`
`[73] Assignee: LSI Logic Corporation, Milpitas,
`Calif.
`
`[21] Appl. No.: 748,269
`
`[22] Filed:
`
`Nov. 13, 1996
`
`Int. Cl.6
`G06T 13/00
`[51]
`.....................................................
`[52] U.S. Cl. .............................................................. 345/302
`[58] Field of Search ..................................... 345/302, 418;
`707 /101, 102, 103, 104
`
`[56]
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`5,675,511 10/1997 Prasad et al. ........................... 345/302
`5,692,213 11/1997 Goldberg et al. ....................... 345/302
`5,767,846
`6/1998 Nakamura et al. ..................... 345/302
`
`Primary Examiner-Phu K. Nguyen
`Assistant Examiner-Cliff N. Vo
`Attorney, Agent, or Firm-Conley, Rose & Tayan; Jeffrey C.
`Hood
`
`[57]
`
`ABSTRACT
`
`An MPEG decoder system and method for performing video
`decoding or decompression which
`includes a unified
`memory for multiple functions according to the present
`invention. The video decoding system includes transport
`logic, a system controller, and MPEG decoder logic. The
`video decoding system of the present invention includes a
`single unified memory which stores code and data for the
`transport, system controller and MPEG decoder functions.
`The single unified memory is preferably a 16 Mbit memory.
`The MPEG decoder logic includes a memory controller
`which couples to the single unified memory, and each of the
`transport logic, system controller and MPEG decoder logic
`access the single unified memory
`through the memory
`controller. The video decoding system implements various
`frame memory saving schemes, such as compression or
`dynamic allocation, to more efficiently use the memory. In
`one embodiment,
`the memory
`is not required
`to store
`reconstructed frame data during B-frame reconstruction,
`thus considerably reducing the required amount of memory
`for this function. Alternatively, the memory is only required
`to store a portion of the reconstructed
`frame data. In
`addition, these savings in memory allow portions of the
`memory to also be used for transport and system controller
`functions. The present
`invention
`thus provides a video
`decoding system with reduced memory requirements.
`
`20 Claims, 16 Drawing Sheets
`
`1:
`224 :ui---------,
`
`1---------,
`: Video Encoder :
`226
`I
`L--------~
`
`I
`
`: Audio DAC 1
`:
`228
`I --------
`
`I
`I
`
`Video
`
`Audio
`
`----;
`Transport
`MP~LJ
`+MIPS
`211
`204
`I Decoder 1
`~~
`L_________
`210
`
`Coded
`Stream
`
`Channel
`Receiver
`202
`
`1----
`:
`
`: 1
`
`I
`I
`
`16Mbit
`SDRAM
`
`-1
`
`AMAZON 1002
`Page 6 of 148
`
`

`

`U.S. Patent
`
`Feb. 9, 1999
`
`Sheet 1 of 16
`
`5,870,087
`
`(9
`u.
`
`(l)
`0)
`~
`o+-'
`·- N
`+-'
`(/) C CO
`-~ ::::)
`-0
`(l)
`~
`
`AMAZON 1002
`Page 7 of 148
`
`

`

`U.S. Patent
`
`Feb. 9, 1999
`
`Sheet 2 of 16
`
`5,870,087
`
`CO'\
`
`E~
`Q) 0
`1n E
`>- Q)
`Cl) ~
`
`~'\
`
`+-'
`Q)
`u,
`a.
`.c
`(.)
`
`~'\
`
`--L...
`0
`::, ~
`a.. Q)
`(.) g
`L... a.. -
`
`~'\
`
`0
`Q)
`"'C
`5
`
`-
`
`0
`
`__.,,.-r--a>"'\
`
`>---
`
`"'C Q)
`L... >
`co ·c
`IO
`
`~'\
`
`u,
`:::J
`co
`- -
`(.) a..
`
`L...
`(9 Q)
`w "'C
`a.. 8
`~c w
`
`co
`~ ;:!\
`
`N
`(!)
`u.
`
`L...
`
`(9 Q) w "'C
`--- a.. 8
`
`~ Q)
`0
`
`,L,,--"
`
`....
`
`AMAZON 1002
`Page 8 of 148
`
`

`

`U.S. Patent
`
`Feb. 9, 1999
`
`Sheet 3 of 16
`
`5,870,087
`
`0 =c
`
`::::,
`<(
`
`.Q~
`II
`""C
`::::,
`I
`I <(
`
`I
`I
`I
`
`~_?:2 ~--:
`--=1-~
`I (9 :? ID c=:!il I
`O
`I ~<-gN
`I N
`u
`I~
`I
`,------1---1-
`I
`I
`a.>
`Cl
`: ,~11 :
`~I I _______
`
`--1
`
`I
`I
`I
`
`,---
`ID
`I
`I
`""C
`I 8
`I c <.01 I
`I W~
`I O
`a>
`I
`:2
`I
`
`I
`I
`I
`
`,---
`I
`I O
`I <(
`
`--1
`
`I
`I
`I
`
`I Cl col I
`
`------.
`
`1
`
`I
`
`_J
`
`""C E
`a> ca
`"'O Q)
`0
`t...
`(.) u5
`
`AMAZON 1002
`Page 9 of 148
`
`

`

`Coded
`Data
`
`-.1
`
`Variable Length
`Decoding
`302
`
`Inverse Scan
`304
`
`Inverse
`Quantisation
`306
`
`Inverse OCT
`308
`
`I I
`
`QFS[n]
`
`QF[v][u]
`
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`L- F[v][u]
`:
`1------------------------------------------------------------J
`
`r:JJ.
`
`d .
`.
`~
`~ .....
`~ = .....
`
`"'!"j
`~
`?'
`~~
`"'"" ~
`
`~
`~
`
`'JJ. =(cid:173)~
`~ ....
`0 ....,
`"'""
`O'I
`
`,i;;..
`
`~
`
`Frame-store
`Memory
`212
`
`! 316 !! 314 i
`Motion
`Compensation
`310
`
`1---r-~---
`
`Decoded
`Samples
`
`L-
`
`f[y][x]
`
`L- d[y][x]
`
`1
`
`FIG. 4
`
`Ul
`~
`
`.....:a = = 00
`
`.....:a
`
`AMAZON 1002
`Page 10 of 148
`
`

`

`without field
`
`with field
`repeat
`
`dynamic allocation
`
`compression
`
`40-segment* xi
`2.556
`
`3.22-segment*
`x/2.801
`
`compressed
`bitstream
`
`local
`compression
`
`b-frame
`anchor
`anchor &
`2.3/2.25 2.067/2.056
`b-frame
`+1vbv
`1.367 /1.306
`+1vbv
`
`.....
`.....
`.....
`.....
`.,,,..,.. ............
`
`anchor
`
`b-frame anchor &
`b-frame
`
`3.27/40-segment*,
`compress anchor & partial b-frame
`2.117/1.806
`+1vbv
`
`FIG. 5
`
`*Chroma "Line Repeat" Mode Only
`
`r:JJ.
`
`d .
`.
`~
`~ .....
`~ = .....
`
`"'!"j
`~
`?'
`~~
`'""" ~
`
`~
`~
`
`'JJ. =(cid:173)~
`~ ....
`Ul
`0 ....,
`'"""
`O'I
`
`Ul
`~
`
`.....:a = = 00
`
`.....:a
`
`AMAZON 1002
`Page 11 of 148
`
`

`

`Table 1
`MPEG Decoder: Memory Partition
`
`L64005
`NTSC
`12,441,600
`(3.000)
`
`PAL
`12,718,080
`(2.556)
`
`Compress Displayed Anchor
`NTSC
`PAL
`9,538,560
`11,197,440
`(2.300)
`(2.250)
`
`1,805,008
`430,765
`500,000
`28,672
`19,200
`12,800
`4,096
`4,096
`60,000
`2,864,637
`
`1,805,008
`520,575
`600,000
`28,672
`23,040
`15,360
`4,096
`4,096
`60,000
`3,060,847
`
`3,610,016
`
`3,610,016
`
`4,669,645
`
`4,865,855
`
`frame store
`(number of frames)
`
`vbv buffer
`video non-instantaneous decode
`video-display sync
`audio buffer
`audio-video latency difference
`audio-display sync
`video transport buffer
`audio transport buffer
`4ms jitter at 15Mbps
`sub-total for channel buffer
`
`packetheaderoverhead
`
`20,000
`
`20,000
`
`OSD
`
`675,840
`
`808,192
`
`sub-total for software & data
`
`775,139
`
`170,097
`
`Grand Total
`
`16,777,216
`
`16,777,216
`
`1,873,171
`
`(114,271)
`
`frame store + channel buffer
`
`15,306,237
`
`15,778,927
`
`14,208,205
`
`16,063,295
`
`FIG.SA
`
`r:JJ.
`
`d .
`.
`~
`~ .....
`~ = .....
`
`"'!"j
`~
`?'
`~~
`
`'"""' ~
`~
`~
`
`'JJ. =(cid:173)~
`~ ....
`0 ....,
`'"""'
`O'I
`
`O'I
`
`Ul
`~
`
`.....:a = = 00
`
`.....:a
`
`AMAZON 1002
`Page 12 of 148
`
`

`

`Table 1
`MPEG Decoder: Memory Partition
`
`Compress Anchor & Segment B
`NTSC: 3.267 segments PAL: 40 segments
`8,778,240
`8,985,600
`(2.117)
`(1.806)
`
`Compress B-Frame
`NTSC
`PAL
`10,229,760
`8,570,880
`(2.067)
`(2.056)
`
`Compress Anchor & 8-Frame
`NTSC
`PAL
`5,667,840
`6,497,280
`(1.367)
`(1.306)
`
`3,610,016
`
`3,610,016
`
`3,610,016
`
`3,610,016
`
`4,669,645
`
`4,865,855
`
`4,669,645
`
`4,865,855
`
`2,633,491
`
`2,097,569
`
`4,645,859
`
`2,658,417
`
`5,743,891
`
`4,585,889
`
`13,447,885
`
`13,851,455
`
`11,435,517
`
`13,290,607
`
`10,337,485
`
`11,363,135
`
`FIG. 68
`
`r:JJ.
`
`d .
`.
`~
`~ .....
`~ = .....
`
`"'!"j
`~
`?'
`~~
`'""" ~
`
`~
`~
`
`'JJ. =(cid:173)~
`~ ....
`-..J
`0 ....,
`'"""
`O'I
`
`Ul
`~
`
`.....:a = = 00
`
`.....:a
`
`AMAZON 1002
`Page 13 of 148
`
`

`

`Memory Bandwidth vs Memory Size (Frame &Channel)
`NTSC
`
`800.0E+6
`
`700.0E+6
`
`600.0E+6
`
`u
`~ 500.0E+6
`
`~ 400.0E+6
`
`-2 >-
`-"O ·:;:
`
`"O
`C: co
`a:i
`
`300.0E+6
`
`200.0E+6
`
`100.0E+6
`
`000.0E+O
`
`15,306,237
`
`13,447,885
`11,435,517
`Memory Size (bit)
`
`10,337,485
`
`FIG. 7
`
`180
`
`160
`
`140 --~
`120 Q) t .c
`
`100 .._..
`c'."'
`0
`E
`80 ~
`<J -..c:
`60 j
`
`"O
`C:
`40 ~
`<J
`
`20
`
`0
`
`D100%
`D96%
`~87%
`12'23 72%
`-B-100%
`-A- 96%
`-e- 87%
`472%
`
`r:JJ.
`
`d .
`.
`~
`~ .....
`~ = .....
`
`"'!"j
`~
`?'
`~~
`"'"" ~
`
`~
`~
`
`'JJ. =(cid:173)~
`~ ....
`0 ....,
`"'""
`O'I
`
`00
`
`Ul
`~
`
`.....:a = = 00
`
`.....:a
`
`AMAZON 1002
`Page 14 of 148
`
`

`

`Memory Bandwidth vs Memory Size (Frame &Channel)
`PAL
`
`600
`
`500
`
`~
`
`:..0 u
`
`400 Q) t e
`
`~
`300 ~
`Q)
`~
`
`800.0E+6
`
`700.0E+6
`
`600.0E+6
`
`500.0E+6
`
`400.0E+6
`
`300.0E+6
`
`200.0E+6
`
`100.0E+6
`
`0
`Q)
`~
`en
`Q) >.
`e
`..c: :s -~
`
`"C
`C
`CCI
`CD
`
`000.0E+O
`
`15778927
`
`13851455
`13290607
`Memory Size (bit)
`
`11363135
`
`FIG. 8
`
`r:JJ.
`
`d .
`.
`~
`~ .....
`~ = .....
`
`"'!"j
`~
`?'
`~~
`"""" ~
`
`~
`~
`
`'JJ. =(cid:173)~
`~ ....
`0 ....,
`""""
`O'I
`
`~
`
`D100%
`D96%
`~87%
`EZ2l 72%
`-B-100%
`-ts- 96%
`-e- 87%
`4-72%
`
`Ul
`~
`
`.....:a = = 00
`
`.....:a
`
`<l -..c: __.
`
`"C -~
`
`"C
`C
`CCI
`CD
`<l
`
`200
`
`100
`
`0
`
`AMAZON 1002
`Page 15 of 148
`
`

`

`bit
`
`NTSC
`
`PAL
`
`Video Frames (incl. compressed) 10,583,248
`10,790,608
`VBV Buffer
`1,805,008
`1,805,008
`Video Decode Buffer
`430,765
`520,575
`Video-Display Sync Buffer
`600,000
`500,000
`Audio Buffer
`28,672
`28,672
`Audio-Video Latency Buffer
`19,200
`23,040
`Audio-Display Sync Buffer
`12,800
`15,360
`Transport Buffer
`8,192
`8,192
`Jitter Buffer ( 4ms)
`60,000
`60,000
`PES Header Overhead
`20,000
`20,000
`OSD
`808,192
`675,840
`MIPS Code and Data
`2,097,569
`2,633,491
`
`FIG. 9
`
`r:JJ.
`
`d .
`.
`~
`~ .....
`~ = .....
`
`"'!"j
`~
`?'
`~~
`"'"" ~
`
`~
`~
`
`'JJ. =(cid:173)~
`~ ....
`"'""
`C
`0 ....,
`"'""
`O'I
`
`Ul
`~
`
`.....:a = = 00
`
`.....:a
`
`AMAZON 1002
`Page 16 of 148
`
`

`

`Bandwidth
`(Mbyte/sec)
`PAL
`NTSC
`
`18.82
`
`72.70
`
`38.29
`
`30.38
`
`15.98
`
`72.59
`
`38.23
`
`30.38
`
`160.19
`
`157.18
`
`Channel In/Out
`
`Motion Compensation
`
`Store
`
`Display & OSD
`
`TOTAL
`
`•With 16x1M SDRAM running at 100 MHz, Bandwidth Available
`is 200Mbyte/sec
`• Expecting 20Mbyte/sec for MIPS Code and Data
`
`FIG. 10
`
`r:JJ.
`
`d .
`.
`~
`~ .....
`~ = .....
`
`"'!"j
`~
`?'
`~~
`"'"" ~
`
`~
`~
`
`'JJ. =(cid:173)~
`~ ....
`"'""
`"'""
`0 ....,
`"'""
`O'I
`
`Ul
`~
`
`....:a = = 00
`
`....:a
`
`AMAZON 1002
`Page 17 of 148
`
`

`

`Processing Power vs Memory (Frame & Channel) Size
`NTSC
`
`~ I
`
`f //
`
`I
`
`6,000
`
`I
`5,000
`
`4,000
`
`3,000
`
`~
`u
`0
`..0
`.... u
`0
`
`ctJ
`~
`(])
`u
`>,
`()
`
`2,000
`
`l"'7"7"1
`
`1,000 I
`
`~f'//1
`
`~~
`
`'.//,I
`
`I
`
`D 1-cycle/coeff
`&'S.'Sl 2-cycle/coeff
`EZZl 4-cycle/coeff
`-B-27MHz
`-fr 54MHz
`-6-81MHz
`+ 100MHz
`+ 108MHz
`• 135MHz
`-.-162MHz
`-e-189MHz
`-+-216MHz
`
`0 ~~~~~-~'-----'-
`15,306,237
`
`....,___.............._.............._ __ __.__.............__.............._ _ ___,
`
`
`
`.............. ............_ __
`
`
`11,435,517
`10,337,485
`13,447,885
`Memory Size (bit)
`
`FIG. 11
`
`r:JJ.
`
`d .
`.
`~
`~ .....
`~ = .....
`
`"'!"j
`~
`?'
`~~
`'"""
`
`~
`~
`~
`
`'JJ.
`
`~
`~
`
`=-
`....
`'"""
`N
`0 ....,
`'"""
`O'I
`
`Ul
`~
`
`....:a =
`= 00
`
`....:a
`
`AMAZON 1002
`Page 18 of 148
`
`

`

`U.S. Patent
`
`Feb. 9, 1999
`
`Sheet 13 of 16
`
`5,870,087
`
`N
`I
`~
`"'"
`N
`
`L..
`Q) Q)
`0 "O
`L.. 0
`::::I CJ
`0 Q)
`Cl) 0
`
`t g_ Cl)
`U) + Cl.
`C
`-
`~ ~
`I-
`
`AMAZON 1002
`Page 19 of 148
`
`

`

`U.S. Patent
`
`Feb. 9, 1999
`
`Sheet 14 of 16
`
`5,870,087
`
`t::
`0 a.
`Cl)
`en + a..
`C:
`-
`~ ~
`I-
`
`AMAZON 1002
`Page 20 of 148
`
`

`

`Transport
`+
`System
`Controller
`
`16
`
`1•
`
`1
`
`•1TD
`• nTDOE
`-------NPW
`-------1
`------
`
`Source
`Decoder
`
`DCH
`TROY
`
`Signal
`
`Description
`
`Type
`
`TD[15:0]
`TDOE
`NPW
`DCH
`TROY
`
`Transmission Data
`Transmission Data Output Enable (active low)
`New Packet Word (reset if TD[15: 13]=000)
`Data Chain Hold (i.e. wait cycle request)
`Transmission Ready (for new packet
`
`Bidirectional
`Input
`Input
`Output
`Output
`
`FIG. 14
`
`r:JJ.
`
`d .
`.
`~
`~ .....
`~ = .....
`
`"'!"j
`~
`?'
`~~
`"'"" ~
`
`~
`~
`
`'JJ. =(cid:173)~
`~ ....
`"'""
`Ul
`0 ....,
`"'""
`O'I
`
`Ul
`~
`
`.....:a = = 00
`
`.....:a
`
`AMAZON 1002
`Page 21 of 148
`
`

`

`1~1 l*I I I I I I l*I I I l*I I I I I I I I I I I I I I 1~1
`
`Reset (to idle) IO I oj o I
`I
`
`r:JJ.
`
`d .
`.
`~
`~ .....
`~ = .....
`
`"'!"j
`~
`?'
`~~
`
`General Data Read I O I O j 1 I word count•
`
`I
`
`General Data Write: I O j 1 I O I word count•
`
`I
`
`address
`
`address
`
`Video Stream Write: I 1Jo I 1 i
`
`Audio Stream Write: i 1 I 1 I O I
`*256 words if word count=O
`
`I
`
`I
`
`FIG. 15
`
`""" ~
`
`~
`~
`
`'JJ. =(cid:173)~
`~ ....
`"""
`0 ....,
`"""
`
`O'I
`
`O'I
`
`Ul
`~
`
`.....:a = = 00
`
`.....:a
`
`AMAZON 1002
`Page 22 of 148
`
`

`

`5,870,087
`
`1
`MPEG DECODER SYSTEM AND METHOD
`HAVING A UNIFIED MEMORY FOR
`TRANSPORT DECODE AND SYSTEM
`CONTROLLER FUNCTIONS
`
`INCORPORATION BY REFERENCE
`The following references are hereby incorporated by
`reference.
`The ISO/IEC MPEG specification referred to as ISO/IEC
`13818 is hereby incorporated by reference in its entirety.
`U.S. patent application Ser. No. 08/654,321
`titled
`"Method and Apparatus for Segmenting Memory to Reduce
`the Memory Required for Bidirectionally Predictive-Coded
`Frames" and filed May 28, 1996 is hereby incorporated by
`reference in its entirety as though fully and completely set
`forth herein.
`titled
`U.S. patent application Ser. No. 08/653,845
`"Method and Apparatus for Reducing the Memory Required
`for Decoding Bidirectionally Predictive-Coded Frames Dur(cid:173)
`ing Pull-Down" and filed May 28, 1996 is hereby incorpo(cid:173)
`rated by reference in its entirety as though fully and com(cid:173)
`pletely set forth herein.
`titled
`U.S. patent application Ser. No. 08/689,300
`"Method and Apparatus for Decoding B Frames in Video
`Codecs with Minimal Memory" and filed Aug. 8, 1996 now 25
`U.S. Pat. No. 5,818,533, whose inventors are David R. Auld
`and Kwok Chau, is hereby incorporated by reference in its
`entirety as though fully and completely set forth herein.
`1. Field of the Invention
`The present invention relates generally to digital video
`compression, and more particularly to an MPEG decoder
`system which includes a single unified memory for MPEG
`transport, decode and system controller functions.
`2. Description of the Related Art
`Full-motion digital video requires a large amount of
`storage and data transfer bandwidth. Thus, video systems
`use various types of video compression algorithms to reduce
`the amount of necessary storage and transfer bandwidth. In
`general, different video compression methods exist for still 40
`graphic images and for full-motion video. Intraframe com(cid:173)
`pression methods are used to compress data within a still
`image or single frame using spatial redundancies within the
`frame. Interframe compression methods are used to com(cid:173)
`press multiple frames, i.e., motion video, using the temporal 45
`redundancy between the frames. Interframe compression
`methods are used exclusively for motion video, either alone
`or in conjunction with intraframe compression methods.
`Intraframe or still image compression techniques gener(cid:173)
`ally use frequency domain techniques, such as the discrete so
`cosine transform (DCT). Intraframe compression typically
`uses the frequency characteristics of a picture frame to
`efficiently encode a frame and remove spatial redundancy.
`Examples of video data compression for still graphic images
`are JPEG (Joint Photographic Experts Group) compression ss
`and RLE (run-length encoding). JPEG compression
`is a
`group of related standards that provide either lossless (no
`image quality degradation) or lossy (imperceptible to severe
`degradation) compression. Although JPEG compression was
`originally designed for the compression of still images rather 60
`than video, JPEG compression is used in some motion video
`applications. The RLE compression method operates by
`testing for duplicated pixels in a single line of the bit map
`and storing the number of consecutive duplicate pixels
`rather than the data for the pixels themselves.
`In contrast to compression algorithms for still images,
`most video compression algorithms are designed to com-
`
`35
`
`s
`
`2
`press full motion video. As mentioned above, video com(cid:173)
`pression algorithms for motion video use a concept referred
`to as interframe compression to remove temporal redundan(cid:173)
`cies between frames. Interframe compression involves star-
`ing only the differences between successive frames in the
`data file. Interframe compression stores the entire image of
`a key frame or reference frame, generally in a moderately
`compressed format. Successive frames are compared with
`the key frame, and only the differences between the key
`10 frame and the successive frames are stored. Periodically,
`such as when new scenes are displayed, new key frames are
`stored, and subsequent comparisons begin from this new
`reference point. It is noted that the interframe compression
`ratio may be kept constant while varying the video quality.
`15 Alternatively,
`interframe
`compression
`ratios may be
`content-dependent,
`i.e., if the video clip being compressed
`includes many abrupt scene transitions from one image to
`another, the compression is less efficient. Examples of video
`compression which use an interframe compression
`tech-
`20 nique are MPEG, DVI and Indeo, among others.
`
`MPEG BACKGROUND
`
`A compression standard referred to as MPEG (Moving
`Pictures Experts Group) compression is a set of methods for
`compression and decompression of full motion video images
`which uses the interframe and intraframe compression tech-
`niques described above. MPEG compression uses both
`motion compensation and discrete cosine transform (DCT)
`processes, among others, and can yield compression ratios
`30 of more than 30: 1.
`The two predominant MPEG standards are referred to as
`MPEG-1 and MPEG-2. The MPEG-1 standard generally
`concerns frame data reduction using block-based motion
`compensation prediction (MCP), which generally uses tem(cid:173)
`poral differential pulse code modulation
`(DPCM). The
`MPEG-2 standard is similar to the MPEG-1 standard, but
`includes extensions to cover a wider range of applications,
`including interlaced digital video such as high definition
`television (HDTV).
`Interframe compression methods such as MPEG are based
`on the fact that, in most video sequences, the background
`remains relatively stable while action takes place in the
`foreground. The background may move, but large portions
`of successive frames in a video sequence are redundant.
`MPEG compression uses this inherent redundancy to encode
`or compress frames in the sequence.
`An MPEG stream
`includes
`three types of pictures,
`referred to as the Intra (I) frame, the Predicted (P) frame, and
`the Bi-directional
`Interpolated
`(B) frame. The I or
`Intraframes contain the video data for the entire frame of
`video and are typically placed every 10 to 15 frames.
`Intraframes provide entry points into the file for random
`access, and are generally only moderately compressed.
`Predicted frames are encoded with reference to a past frame,
`i.e., a prior Intraframe or Predicted frame. Thus P frames
`only include changes relative to prior I or P frames. In
`general, Predicted frames receive a fairly high amount of
`compression and are used as references for future Predicted
`frames. Thus, both I and P frames are used as references for
`subsequent frames. Bi-directional pictures include the great-
`est amount of compression and require both a past and a
`future reference
`in order to be encoded. Bi-directional
`frames are never used as references for other frames.
`In general, for the frame(s) following a reference frame,
`i.e., P and B frames that follow a reference I or P frame, only
`small portions of these frames are different from the corre-
`
`65
`
`AMAZON 1002
`Page 23 of 148
`
`

`

`5,870,087
`
`3
`sponding portions of the respective reference frame. Thus,
`for these frames, only the differences are captured, com(cid:173)
`pressed and stored. The differences between these frames are
`typically generated using motion vector estimation logic, as
`discussed below.
`When an MPEG encoder receives a video file, the MPEG
`encoder generally first creates the I frames. The MPEG
`encoder may compress the I frame using an intraframe
`compression technique. The MPEG encoder divides respec(cid:173)
`tive frames into a grid of 16x16 pixel squares called mac- 10
`roblocks
`in order
`to perform motion
`estimation/
`compensation. Thus, for a respective target picture or frame,
`i.e., a frame being encoded, the encoder searches for an
`exact, or near exact, match between
`the target picture
`macroblock and a block in a neighboring picture referred to
`as a search frame. For a target P frame the encoder searches 15
`in a prior I or P frame. For a target B frame, the encoder
`searches in a prior or subsequent I or P frame. When a match
`is found, the encoder transmits a vector movement code or
`motion vector. The vector movement code or motion vector
`only includes information on the difference between the 20
`search frame and the respective target picture. The blocks in
`target pictures that have no change relative to the block in
`the reference picture or I frame are ignored. Thus the amount
`of data that is actually stored for these frames is significantly
`reduced.
`After motion vectors have been generated, the encoder
`then encodes the changes using spatial redundancy. Thus,
`after finding the changes in location of the macro blocks, the
`MPEG algorithm further calculates and encodes the differ(cid:173)
`ence between corresponding macroblocks. Encoding the 30
`difference is accomplished through a math process referred
`to as the discrete cosine transform or DCT. This process
`divides the macroblock into four sub blocks, seeking out
`changes in color and brightness. Human perception is more
`sensitive to brightness changes than color changes. Thus the 35
`MPEG algorithm devotes more effort to reducing color data
`than brightness.
`Therefore, MPEG compression is based on two types of
`redundancies in video sequences, these being spatial, which
`is the redundancy in an individual frame, and temporal, 40
`which is the redundancy between consecutive frames. Spa(cid:173)
`tial compression is achieved by considering the frequency
`characteristics of a picture frame. Each frame is divided into
`non-overlapping blocks, and each block is transformed via
`the discrete cosine transform (DCT). After the transformed 45
`blocks are converted to the "DCT domain", each entry in the
`transformed block is quantized with respect to a set of
`quantization tables. The quantization step for each entry can
`vary, taking into account the sensitivity of the human visual
`system (HVS)} to the frequency. Since the HVS is more 50
`sensitive to low frequencies, most of the high frequency
`entries are quantized to zero. In this step where the entries
`are quantized, information is lost and errors are introduced
`to the reconstructed image. Run length encoding is used to
`transmit
`the quantized
`values. To further
`enhance
`compression, the blocks are scanned in a zig-zag ordering
`that scans the lower frequency entries first, and the non-zero
`quantized values, along with the zero run lengths, are
`entropy encoded.
`When an MPEG decoder receives an encoded stream, the 60
`MPEG decoder reverses the above operations. Thus the
`MPEG decoder performs inverse scanning to remove the zig
`zag ordering, inverse quantization to de-quantize the data,
`and the inverse DCT to convert the data from the frequency
`domain back to the pixel domain. The MPEG decoder also 65
`performs motion compensation using the transmitted motion
`vectors to recreate the temporally compressed frames.
`
`55
`
`4
`When frames are received which are used as references
`for other frames, such as I or P frames, these frames are
`decoded and stored in memory. When a temporally com(cid:173)
`pressed or encoded frame is received, such as a P or B frame,
`5 motion compensation is performed on the frame using the
`prior decoded I or P reference frames. The temporally
`compressed or encoded frame, referred to as a target frame,
`will include motion vectors which reference blocks in prior
`decoded I or P frames stored in the memory. The MPEG
`decoder examines the motion vector, determines the respec(cid:173)
`tive reference block in the reference frame, and accesses the
`reference block pointed to by the motion vector from the
`memory.
`A typical MPEG decoder includes motion compensation
`logic which includes local or on-chip memory. The MPEG
`decoder also includes an external memory which stores prior
`decoded reference frames. The MPEG decoder accesses the
`reference frames or anchor frames stored in the external
`memory
`in order to reconstruct
`temporally compressed
`frames. The MPEG decoder also typically stores the frame
`being reconstructed in the external memory.
`An MPEG decoder system also typically includes trans(cid:173)
`port logic which operates to demultiplex received data into
`a plurality of individual multimedia streams. An MPEG
`25 decoder system also generally includes a system controller
`which controls operations in the system and executes pro(cid:173)
`grams or applets.
`Prior art MPEG video decoder systems have generally
`used a frame store memory for the MPEG decoder motion
`compensation
`logic which stores the reference frames or
`anchor frames as well as the frame being reconstructed.
`Prior art MPEG video decoder systems have also generally
`included a separate memory for the transport and system
`controller functions. It has generally not been possible to
`combine
`these memories, due to size limitations. For
`example, current memory devices are fabricated on an 4
`Mbit granularity. In prior art systems, the memory require(cid:173)
`ments for the transport and system controller functions as
`well as the decoder motion compensation
`logic would
`exceed 16 Mbits of memory, thus requiring 20 or 24 Mbits
`of memory. This additional memory adds considerable cost
`to the system.
`The amount of memory is a major cost item in the
`production of video decoders. Thus, it is desired to reduce
`the memory requirements of the decoder system as much as
`possible to reduce its size and cost. Since practical memory
`devices are implemented using particular convenient dis(cid:173)
`crete sizes, it is important to stay within a particular size if
`possible for commercial reasons. For example, it is desired
`to keep the memory requirements below a particular size of
`memory, such as 16 Mb, since otherwise a memory device
`of 20 or 24 Mb would have to be used, resulting in greater
`cost and extraneous storage area. As mentioned above, it has
`heretofore not been possible
`to combine
`the memory
`required for the transport and system controller functions
`with the memory required for the MPEG decoder logic due
`to the memory size requirements.
`Therefore, a new video decoder system and method is
`desired which efficiently uses memory and combines the
`memory subsystem for reduced memory requirements and
`hence reduced cost.
`
`SUMMARY OF THE INVENTION
`The present invention comprises an MPEG decoder sys(cid:173)
`tem and method for performing video decoding or decom(cid:173)
`pression which includes a unified memory for multiple
`
`AMAZON 1002
`Page 24 of 148
`
`

`

`5,870,087
`
`6
`FIG. 8 illustrates the relationship of memory bandwidth
`vs. memory size in the PAL encoding scheme;
`FIG. 9 illustrates the memory partitions according to the
`preferred embodiment of the invention;
`FIG. 10 illustrates the estimated memory bandwidth dis(cid:173)
`tribution in the preferred embodiment of the invention;
`FIG. 11 illustrates the "worst case" relationship of pro(cid:173)
`cessing power vs. memory size in the NTSC decoding
`scheme;
`FIG. 12 illustrates the clock domains in the system;
`FIG. 13 illustrates clock operating frequencies according
`to the preferred embodiment of the invention;
`FIG. 14 illustrates an example of the packet data interface
`between the transport controller and the source decoder; and
`FIG. 15 illustrates packet header formats used in the
`preferred embodiment.
`
`10
`
`5
`functions according to the present invention. The video
`decoding
`system
`includes
`transport
`logic, a system
`controller, and MPEG decoder logic. The video decoding
`system of the present invention includes a single unified
`memory which stores code and data for the transport logic, 5
`system controller and MPEG decoder functions. The single
`unified memory
`is preferably a 16 Mbit memory. The
`present invention thus requires only a single memory, and
`thus has reduced memory requirements compared to prior
`art designs.
`The video decoding system includes transport logic which
`operates to demultiplex received data into a plurality of
`individual multimedia streams. The video decoding system
`also includes a system controller which controls operations
`in the system and executes programs or applets. The video 15
`decoding system further includes decoding logic, preferably
`MPEG decoder logic, which performs motion compensation
`between temporally compressed frames of a video sequence
`during video decoding or video decompression. The
`memory includes a plurality of memory portions, including 20
`a video frame portion for storing video frames, a system
`controller portion for storing code and data executable by
`the system controller, and a transport buffer for storing d