FILE HISTORY
`US 5,870,087
`
`5,870,087
`PATENT:
`INVENTORS: Chau, Kwok Kit
`
`TITLE:
`
`MPEG decoder system and method
`having a unified memory for transport
`decode and system controller functions
`
`APPLICATION
`NO:
`FILED:
`ISSUED:
`
`US1996748269A
`
`13 NOV 1996
`09 FEB 1999
`
`COMPILED:
`
`17 NOV 2016
`
`SONY EX. 1002
`Page 1
`
`

`
`I
`
`Now.
`
`5870087
`
`C.I" -I .
`
`-I I
`
`I
`
`J'I I
`
`~Lj &~f
`
`/
`
`p./
`
`,*1
`
`BES COPY
`
`y
`Iority dlained
`019condlkornstlEys
`EMin'nIUiWg
`,kndAok*vv1@d9vd
`
`Ue
`
`A
`FILED
`i
`
`I STATE OR I 1SHti
`INWlE
`TOTAL
`ICO)UNTRY DWJBCLAIMS I CLAIMS
`I
`f 'II
`
`FILING FE
`ATTORNES'
`DOCKErT NO.
`RECEIVED
`I
`,~'i I iI 01I'l
`
`U.S. DEPT OFCOMM.1 PAT. TM-PM!L f.124
`
`*OF APPLICATION
`SEPARATELY
`,E OF ALLOWANCE MAILEDCLISAOWD
`LI
`Aselatant Examiner
`
`Vo
`
`Total Cim
`
`Print Cidm
`
`ISSUE PEE
`ai Due
`D i Paid
`.J
`0
`
`Label
`Area
`
`*" - -43A
`
`'Shelq,15rwg.
`
`~PHO1 K. NGUYEN
`PRIMARY EXAMINER
`GROUP 24M0
`
`ISSUETC
`
`Prima EOr NUMIBER
`PREPARED FOR ISSUE
`
`DRAWING
`Flgsl. Pj*g.
`
`Prin 0a1g.
`
`WARNING: The Informalon disclosed herein may be restlicted.
`ulhiorIzed disclosure may be prohibited
`by the United States Code Title 35. Sections 1 2 8and36 oeslnotieteUS
`Palant & Trademark Office Is restrIcted to au
`zed employees and contractors only.
`1" jfj Aj
`
`11, 11- 4
`
`h
`
`(FACE)
`
`.
`
`.V..-
`
`SONY EX. 1002
`Page 2
`
`

`
`5,870,087
`
`MPEG DECODER SYSTEM AND METHOD HAVING A UNIFIED
`MEMORY FOR TRANSPORT DECODE AND SYSTEM CONTROLLER
`FUNCTIONS
`
`Transaction History
`
`
`Transaction Description
`Date
`Initial Exam Team nn
`11-29-1996
`02-06-1997 Application Captured on Microfilm
`09-18-1997
`Information Disclosure Statement (IDS) Filed
`09-18-1997
`Information Disclosure Statement (IDS) Filed
`02-04-1998 Case Docketed to Examiner in GAU
`04-27-1998 Case Docketed to Examiner in GAU
`05-11-1998 Mail Non-Final Rejection
`05-11-1998 Non-Final Rejection
`06-08-1998 Response after Non-Final Action
`06-16-1998 Date Forwarded to Examiner
`09-02-1998 Mail Notice of Allowance
`09-02-1998 Notice of Allowance Data Verification Completed
`10-26-1998
`Issue Fee Payment Verified
`10-26-1998 Change in Power of Attorney (May Include Associate POA)
`10-26-1998 Mailroom Date of Drawing(s)
`10-28-1998 Drawing(s) Received at Publications
`11-09-1998 Drawing(s) Processing Completed
`11-09-1998 Drawing(s) Matched to Application
`11-25-1998 Communication - Re: Power of Attorney (PTOL-308)
`01-11-1999
`Issue Notification Mailed
`02-09-1999 Recordation of Patent Grant Mailed
`02-22-2016 Petition Requesting Trial
`06-29-2016 File Marked Found
`08-22-2016 Request for Trial Granted in Part
` 
`
`SONY EX. 1002
`Page 3
`
`

`
`jVjjckjj
`
`, .
`
`",j
`
`00/748269
`
`Date
`Entered
`or
`Counted
`
`P IT P
`
`TO
`
`APPROVED FOR LICENSE E-
`
`08748269
`
`Date
`CONTENTSReceived
`Mailed
`
`-. PPPers..
`
`SE 0C
`
`2 1998
`
`-~1*A
`
`6.
`
`PTO Grant f'El 0 15
`
`27.
`
`29.
`
`__
`
`32.
`
`(FRONT)
`
`_____ _____
`
`____10.
`
`________________11.
`
`_____ _____
`
`____12.
`
`_____ _____
`
`____13.
`
`_____ ____
`
`____15.
`
`_____ _____
`
`____19.
`
`_____ ____
`
`____21.
`
`_____ _____
`
`____22.
`
`____ ___
`
`____
`
`___23.
`
`____ ___
`
`___
`
`___24.
`
`____ ___
`
`____
`
`___25.
`
`_____ _____
`
`_____26.
`
`____ ___
`
`____
`
`___28.
`
`____ ___
`
`___
`
`___30.
`
`SONY EX. 1002
`Page 4
`
`

`
`~)
`
`ArA
`
`SYMBOLS
`
`
`....... ... ...
`.A owid
`................
`-(t1woughnumiboiW) Crcols4
`.Rsrldo
`.. .. ........
`.Won-glocled
`...............
`
`A . ............ pe
`0 .... .
`objecled
`-.
`
`(LEFT INSIDE)
`
`SONY EX. 1002
`Page 5
`
`

`
`(RIGHT OUTSIDE)
`
`SEARCHED
`
`Class
`
`Sub.
`
`Date
`
`(Exmr.
`
`Va
`
`,
`
`091-U
`
`~LF<
`
`7b)
`
`of
`
`lot~
`
`SONY EX. 1002
`Page 6
`
`

`
`United States Patent [19]
`Chau
`
`[54] MPEG DECODER SYSTEM AND METHOD
`HAVING A UNIFIED MEMORY FOR
`TRANSPORT DECODE AND SYSTEM
`CONTROLLER FUNCTIONS
`
`[75]
`
`Inventor: Kwok Kit Chau, Los Altos, Calif.
`
`[73] Assignee: LSI Logic Corporation, Milpitas,
`Calif.
`
`[21] Appl. No.: 748,269
`
`Nov. 13, 1996
`[22] Filed:
`Int. Cl.6 ................. . . . . . . . . . . . . . . . . . . . . . G06T 13/00
`[51]
`345/302
`[52] U.S. ClI
`............................................
`345/302, 418;
`[58] Field of Search ...........................
`707/101, 102, 103, 104
`
`[56]
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`10/1997 Prasad et al..................... 345/302
`345/302
`11/1997 Goldberg et al ...............
`345/302
`6/1998 Nakamura et al ..............
`
`,675,5 11
`5,692,213
`5,767,846
`
`Primary Exaniner-Phu K. Nguyen
`Assistant Exaniner-Cliff N. Vo
`Attorney, Agent, or Firin-Conley, Rose & Tayon; Jeffrey C.
`H-ood
`
`Coded
`Strea
`
`Channel
`Receiver
`
`Transport
`+MIPS
`204 __
`
`111111INi
`
`
`11111 11111 111111111
`
`
`11111 11111 11111
`
`11111 11111
`11I111I1I1II11111111
`
`US005870087A
`[11] Patent Number:
`[45] Date of Patent:
`
`5,870,087
`Feb. 9, 1999
`
`[57]
`
`ABSTRACT
`
`An MPEG decoder system and method for performing Video
`decoding or decompression which includes a unified
`memory for multiple functions according to the present
`invention. The video decoding system includes transport
`logic, a system controller, and MPEG decoder logic. The
`video decoding system of the present invention includes a
`single unified -memory which stores code and data for the
`transport, system controller and MPEG decoder functions.
`The single unified memory is preferably a 16 Mbit memory.
`The MPEG decoder logic includes a memory controller
`which couples to the single unified memory, and each of the
`transport logic, system controller and MPEG decoder logic
`access the single unified memory through- the memory
`controller. The video decoding system implements various
`frame' memory. saving schemes, such as compression or
`dynamic allocation, to more efficiently use the memory. In
`to store
`is not required
`the memory
`one embodiment,
`reconstructed frame data during B-frame reconstruction,
`thus considerably reducing the required amount of memory
`for this function. Alternatively, the memory is only required
`to store a portion of the reconstructed frame data. In
`addition, these savings in memory allow portions of the
`memory to also be used for transport and system controller
`thus provides a video
`functions. The present invention
`decoding system with reduced memory requirements.
`
`20 Clainis, 16 Drawing Sheets
`
`Video
`
`Audio
`
`~/212
`
`16Mbit
`SDRAM
`
`SONY EX. 1002
`Page 7
`
`

`
`'U. S. Patent
`
`Feb. 9, 1999
`
`Sheet 1 of 16
`
`5,870,087
`
`C%4
`
`4-8
`
`CN
`(D-
`
`Mo.
`
`. N7
`
`SONY EX. 1002
`Page 8
`
`

`
`U.S. -Patent'
`
`Feb. 9, 1999
`
`ISheet 2 of 16
`
`5,870,087
`
`C%4
`
`U.
`
`SONY EX. 1002
`Page 9
`
`

`
`U.S. Patent
`
`Feb. 9, 1999
`
`Sheet 3 of 16
`
`5,870,087
`
`coQ
`
`(Y)
`0
`
`.1
`
`i
`
`CN
`
`II
`
`lCo
`
`ujcOI
`
`0.
`
`C%
`
`c"II
`
`CN2+~f
`
`cu
`
`>CN
`W 3c)
`N
`C'*4
`
`7S E
`
`SONY EX. 1002
`Page 10
`
`

`
`U.S. Patent
`
`Feb. 9, 1999
`
`Sheet 4 of 16
`
`5,870,087
`
`-0U)
`"C)
`
`8E
`
`0C
`
`rr
`
`c) =
`
`- V-
`
`a CN
`
`LE
`
`LL
`
`'00
`
`o
`
`CC
`
`. 0
`
`SONY EX. 1002
`Page 11
`
`

`
`U.S. Patent
`
`Feb. 9, 1999
`
`Sheet 5 of 16
`
`5,870,087
`
`0 0
`
`'
`cu
`
`CL,
`(D
`
`cuE
`
`0.
`
`c c
`
`u
`
`L- LO
`C*4%~
`
`C CU)
`CO >
`
`a)
`
`E00
`
`-4,Co.
`
`10
`
`EC)
`
`Ccn
`
`C'J
`
`C4
`
`ECO
`
`0 C
`
`L E00
`
`0
`
`0 Ec
`
`u,
`
`CDu
`
`SONY EX. 1002
`Page 12
`
`

`
`U.S. 'Patent
`
`Feb. 9, 1999
`
`Sheet 6 of 16
`
`5,870,087
`
`LO
`
`toJ
`CD,
`
`U')
`CD
`
`C)
`
`U.
`
`C
`
`oZc
`
`0c
`
`n
`
`CC
`0m
`
`__
`
`__
`
`0
`
`cNi
`
`c C
`
`=)
`C)
`
`~0.
`
`o
`
`w
`
`SONY EX. 1002
`Page 13
`
`

`
`U.S. Patent
`
`Feb. 9, 1999
`
`Sheet 7 of 16
`
`5,870,087
`
`CDJ
`
`LO
`
`ci
`
`L0
`00
`00
`r-
`9t
`cl
`
`T-
`0 )
`4,
`
`co
`li
`
`CD
`LO
`
`U-*
`
`zC
`
`C)
`
`a)
`
`E
`
`C 2
`
`-
`
`0E
`
`-0..CU
`
`0L
`
`U
`
`LO
`LO
`00
`LO
`co
`occ
`
`U)
`s
`mC-i)
`(0
`
`0
`
`C0
`
`-0c
`
`c C:>
`
`(D
`
`4-
`
`CD
`
`-.
`a
`cn o
`
`E(00C
`(D
`li
`
`0U
`
`)
`H
`
`SONY EX. 1002
`Page 14
`
`

`
`U. S. Patent
`
`Feb. 9, 1999
`
`Sheet 8 of 16
`
`5,870,087
`
`-00-
`
`C:o
`C) m co IQ
`
`C:
`
`m 00 r-,
`
`(j!qo9s/,qAq) /oeq/qpmug
`C)
`CD
`C>
`04C
`
`C)
`00
`
`CD
`CD
`
`C)
`qlqr
`
`C)
`CN
`
`LL
`
`L-
`
`Ld)
`cle)
`
`U)
`co
`
`cy,
`N-
`
`-D
`a)
`N
`C,,
`
`0Ea)
`
`C)D
`co
`
`C)
`to
`
`C)
`
`c
`C-)
`a1)E
`cu
`U-
`ai)
`N
`
`CO
`
`C
`
`cam
`0E
`
`wi
`C)
`C=;
`CD
`
`CD C0
`w
`CD
`
`CD
`r.0C
`
`co
`C.D
`C
`to
`Lw
`wui w
`C)
`CD
`C)
`C: 6
`.6
`=6
`'CD
`C:)
`CD
`m')VC
`LO)
`(oas/selAq) qlp!mpueg
`
`CD
`Lw
`C:)
`C6
`CD
`
`Lw
`C:
`6=
`CD
`
`SONY EX. 1002
`Page 15
`
`

`
`U.S. Patent
`
`Fe'b. 9, 1999
`
`Sheet 9 of 16-
`
`5,870,087
`
`
`
`--o -O-cz -o-,o 0-0, -0-0- -C-e -o-,o -a-0,
`
`
`
`
`
`
`
`
`
`to I-- C*4 C) co rl*_ CN
`cr m w P-- C) 0') m rl_
`
`(j!qo,s/9jAq) Aow8lvj/qflp!MPUeGV
`CD
`C>
`C.
`DCD.
`CNV
`
`)
`
`C
`C:)
`
`>CD
`>
`>
`
`)cr)
`
`c
`
`rl-.,
`C)
`UDC)
`
`Co
`
`E
`
`CY)
`
`n-
`
`V-
`
`C0
`
`C)
`C)
`cy)
`
`co
`
`C)
`C)
`
`(oas/selAq) qlp!MPUBG
`
`c Co
`
`d
`a-)
`Eco
`
`U-
`
`cn
`
`0E
`
`C
`
`0E
`
`SONY EX. 1002
`Page 16
`
`

`
`U.S. Patent
`
`Feb. 9, 1999
`
`Sheet 10 of 16
`
`5,870,087
`
`0 WLO C) CN C> C>) CN C> C) 04
`
`<
`
`CL
`
`C:)"CC" CN -
`r-. 00L) to
`
`C C4C)OM
`00 C)
`
`
`
`~ ~C:>
`
`-
`
`CDO C C) C CD
`CC 00U)CDCN
`Co DC) IN- (C) C>)C0)C) C>)lrlr0m
`qrdl
`C N C) N*- C) to (ND0 V- (>CD C00Cqq-
`C (
`CO NNr_- C)
`04C)M ')
`to
`co
`Lo Co -rrULoco
`
`-
`
`cn~
`(1)
`
`E
`
`a)
`4
`
`C
`4
`
`cu~
`
`)n
`
`0)
`
`Lcn
`
`Q9
`
`>Q,
`
`C
`
`)0
`
`I
`m
`0 0 8 .
`
`0
`660mz
`.(5 (5 co) -L = C O
`
`0)
`(9
`U-
`
`SONY EX. 1002
`Page 17
`
`

`
`U.S. Patent
`
`Feb. 9, 1999
`
`Sheet 11 of 16
`
`5,870,087
`
`CF)
`00O
`M) U")
`LOc) N
`~-
`r-
`
`C)
`
`c6
`C)
`
`00
`MCr
`6
`Cf)
`
`co
`
`r-
`1LO
`
`ca
`
`~cl)
`
`C,O
`
`_C5r
`F-
`
`C*4
`00
`
`C)
`r--
`-
`
`CO
`0)
`0M
`66V-
`CoJ
`Mr) CO
`(D
`
`cu
`m m-
`
`cu
`
`C
`
`CO _
`
`0 (
`
`9
`
`C,0
`
`C)
`
`m
`
`C
`
`0
`
`CL
`
`4 0
`
`C
`
`CD
`
`L.
`
`)
`
`(3),
`
`a),CN
`
`X
`
`C=
`
`0
`
`E
`
`0
`
`C C
`
`0
`
`>C/)
`
`SONY EX. 1002
`Page 18
`
`

`
`U.S. Patent
`
`Feb. 9, 1999
`
`Sheet 12 of 16
`
`5,870,087
`
`(D
`
`V-
`
`LO00
`co
`lqq-
`
`Nr
`
`P-C)
`
`0C
`
`=)
`cr'
`
`0 00
`
`>% >~
`Cq-W)Cc)
`
`r- CN
`
`W0
`rr
`
`tQ C-4,0M
`rD
`
`(D
`Cm
`
`CNV144-
`LO
`
`cu
`
`cn)
`
`0m c
`
`n-c
`
`0 L
`
`Q-
`
`-
`
`0>0D
`0)
`C6
`
`ff
`
`I,
`
`S4
`
`U
`
`00:
`C)
`ci
`
`Nfojqojoe~j/9joAo
`
`SONY EX. 1002
`Page 19
`
`

`
`U.S. Patent
`
`Feb. 9, 1999
`
`Sheet 13 of 16
`
`5,870,087
`
`CD
`Ir-c/w
`
`C*
`
`0.
`
`t
`
`0 C L
`
`-F-
`
`SONY EX. 1002
`Page 20
`
`

`
`U.S. Patent
`
`'Feb. 9, 1999
`
`Sheet 14 of 16
`
`5,870,087
`
`U)0
`
`0 C
`
`L
`(U
`
`F-
`
`SONY EX. 1002
`Page 21
`
`

`
`U.S. Patent
`
`Feb. 9, 1999
`
`Sheet 15 of 16
`
`5,870,087
`
`L- o
`
`C)
`
`0
`
`o
`
`CL C) 0 o
`
`cn 0
`
`000
`:E5~
`m
`
`Q
`
`0D
`
`L)
`
`1)
`
`C,Cr
`
`lq;
`
`ci
`
`01
`
`Cu
`
`CL
`C)
`(1)
`Cl cuc
`
`0 2Qi
`
`C:
`L-
`
`SONY EX. 1002
`Page 22
`
`

`
`U.S. Patent
`
`Fcb. 9, 1999
`
`Shecet 16 of 16
`
`5,870,087
`
`LO
`
`LL
`
`*ci5
`
`E U
`
`)
`I....
`
`0
`
`Ec
`
`u
`
`0C
`
`D
`
`C)
`
`-40
`cu
`
`0
`
`C)
`C)
`
`cu
`cu
`
`0
`
`-9
`
`U)
`C,)
`U)
`
`CO~
`
`CN C)
`
`C)"
`
`SONY EX. 1002
`Page 23
`
`

`
`5,870,087
`
`10
`
`MPEC DECODER SYSTEM AND METHOD
`HAVING A UNIFIED MEMORY FOR
`TRANSPORT DECOIDE AND SYSTEM
`CONTROLLER FUNCTIONS
`INCORPORATION BY REFERENCE5
`The following references are hereby incorporated by
`reference.
`The ISOAlEC MPEG specification referred to as ISO/AEC
`13818 is hereby incorporated by reference in its entirety.
`U.S. patent application Ser. No. 08/654,321
`titled
`"Method and Apparatus for Segmenting Memory to Reduce
`the Memory Required for Bidirectionally Predictive- Coded
`Frames" and filed May 28, 1996 is hereby incorporated by
`reference in its entirety as though fully and completely set 15
`forth herein.
`U.S. patent -application Ser. No. 08/653,845
`titled
`"Method and Apparatus for Reducing the Memory Required
`for Decoding Bidirectionally Predictive -Coded Frames Dur-
`ing Pull-Down" and filed May 28, 1996 is hereby incorpo- 20
`rated by reference in its entirety as though fully and com-
`pletely set forth herein.
`U.S. patent application Ser. No. 08/689,300
`titled
`"Method and Apparatus for Decoding B Frames in Video 2
`Codecs with Minimal Memory" and filed Aug. 8, 1996 now 2
`U.S. Pat. No. 5,818,533, whose inventors are David R. Auld
`and Kwok Chau, is hereby incorporated by reference in its
`entirety as though fully and completely set forth herein.
`1. Field of the Invention
`3
`Thbe present invention relates generally to digital video 3
`compression, and more particularly to an MPEG decoder
`system which includes a single unified memory for MPEG
`transport, decode and system controller functions.
`2. Description of the Related Art
`Full-motion digital video requires a large amount of
`storage and data transfer bandwidth., Thus, video systems
`use various types of video compression algorithms to reduce
`the amount of necessary storage and transfer bandwidth. In
`general, different video compression methods exist for Still 40
`graphic images and for full-motion video. Intraframe com-
`pression methods are used to compress data within a still
`image or single frame using spatial redundancies within the
`frame. Interframe compression methods are used to com-
`press multiple frames, i.e., motion video, using the temporal '4
`redundancy between the frames. Interframe compression
`methods are used exclusively for motion video, either alone
`or in conjunction with intraframe compression methods.
`Intraframe or still image compression techniques gener--
`ally use frequency domain techniques, such as the discrete '50
`cosine transform (DCI). Intraframe compression typically
`uses the frequency characteristics of a picture frame to
`efficiently encode a frame and remove spatial redundancy.
`Examples of video data compression for still graphic images
`are JPEG (Joint Photographic Experts Group) compression 55
`and RLE (run-length encoding). JPEG compression is. a
`groupof related standards that provide either lossless (no
`image quality degradation) or lossy (imperceptible to severe
`degradation) compression. Although JPEG compression was
`originally designed for the compression of still images rather 60
`than video, JPEG compression is used in some motion video
`applications. .The RLE compression method operates by
`testing for duplicated pixels in a single line of the bit map
`the number of consecutive duplicate pixels
`and storing
`rather than the data for the pixels themselves.
`In contrast to compression algorithms for still images,
`most video compression algorithms are designed to com-
`
`press full motion video. As mentioned above,'video com-
`pre ssion algorithms for. motion video use a concept referred
`to as interframe compression to remove temporal redundan-
`cies between frames. Interframe, compression involves stor-
`ing only the differences between successive frames in the
`data file. Interframe compression stores the entire image of
`a key frame or reference frame, generally in a moderately'
`compressed format. Successive frames are compared with
`the, key frame, and only the differences between the key
`iframe and the successive frames Are stored.. Periodically,
`such as when new scenes are displayed, new key frames are
`stored, and subsequent comparisons begin from this new
`reference point. It is noted that the interframe compression
`ratio may be kept constant while varying the video quality.
`interframe compression ratios may be
`Alternatively,
`content-dependent, i.e., if the video clip being compressed
`includes many abrupt.scene transitions from one image to,
`another, the compression is less efficient. Examples of video
`compression which use an interframe compression tech-
`nique are MPEG,.DVI and Indeo, among others.
`
`MPEG BACKGROUND
`A compression standard referred to as MPEG (Movin g
`Pictures Experts Group) compression is a set of methods for
`compression and decompression of full motion video images
`which uses the interframe and intraframe compression tech-
`niques described above. MPEG compression uses both
`motion compensation and discrete cosine transform (DCT)
`processes, among others, and can yield compression ratios
`of more than,30:1.
`The two predominant MPEG standards are referred to as
`MPEG-i and MPEG-2. The MPEG-i standard generally
`concerns frame data reduction using block-based motion
`compensation prediction (MCP), which generally uses tem-
`poral differential pulse code modulation (DPCM). The
`MPEG-2 standard is similar to the MPEG-i standard, but
`includes extensions to cover a wider range of applications,
`including interlaced digital.video such as high definition
`television (HDTV)..
`Interframe compression methods such as MPEG are based
`on the fact that, in most video sequences, the background
`remains relatively stable while action takes place in the
`foreground. The background may move, but large portions
`of successive frames in a video sequence are redundant.
`MPEG compression uses this inherent redundancy to encode
`or compress frames in the sequence.
`An MPEG stream
`types of pictures,
`includes
`three
`referred to as the Intra (1) frame, the Predicted (P) frame, and
`)the Bi-directional Interpolated (B) frame . The I or
`Intraframes contain the video data for the entire frame of
`video and are typically placed every 10 to 15 frames.
`Intraframes provide entry points into the, file for random
`access, and are generally only moderately compressed.
`5Predicted frames are encoded with reference to a past frame,
`i.e., a prior Intraframe or Predicted frame. Thus P frames
`only include changes relative to prior I or P frames. In
`general, Predicted frames receive a fairly high amount of
`compression and are used as references for future Predicted
`frames. Thus, both I and P frames are used as references for
`subsequent frames. Bi-directional pictures include the great-
`est amount of compression. and require both a past and a
`in order to be encoded. Bi-directional
`future reference
`frames are never used as references for other frames.
`5 In general, for the frame(s) following a reference frame,
`i.e., P and B frames that follow a reference I or P frame, only
`small portions of these frames are different from the corre-
`
`35
`
`65
`
`SONY EX. 1002
`Page 24
`
`

`
`3
`sponding portions of the respective reference frame. Thus,
`for these frames, only the differences are captured, com-
`pressed and stored. The differences between these frames are
`typically generated using motion vector estimation logic, as
`discussed below.
`* When an MPEG encoder receives a video file, the MPEG
`encoder generally first creates the I frames. The MPEG
`the I frame. using an intraframe
`encoder may compress
`compression technique. The MPEG encoder divides respec-
`tive frames into a grid of 16x16 pixel squares called mac-
`to perform motion estimation!
`in order
`roblocks
`compensation. Thus, for a respective target picture or frame,
`i.e., a frame being encoded, the encoder searches for an
`target picture
`the
`exact, or near exact, match between
`macroblock and a block in a neighboring picture referred to
`as a search frame. For a target P frame the encoder searches
`in a prior 1 or P frame. For a target B frame, the encoder
`searches in a prior or subsequent I or P frame. When a match
`is found, the encoder transmits a vector movement code or
`motion vector. The vector movement code or motion vector
`only includes information on the difference between the
`search frame and the respective target picture. The blocks in
`target pictures that have no change relative to the block in
`the reference picture or I frame are ignored. Thus the amount
`*of data that is actually stored for these frames is significantly
`reduced.
`After motion vectors have been generated, the encoder
`then encodes the changes using spatial redundancy. Thus,
`after finding the changes in location of the macroblocks, the
`MPEG algorithm further calculates and encodes the differ-
`ence between corresponding macroblocks. Encoding the
`difference is accomplished through a math process referred
`to as the discrete cosine transform or DCT. Thbis process
`divides the macroblock into four sub blocks, seeking out
`changes in color and brightness. H-uman perception is more
`sensitive to brightness changes than color changes. Thus the
`MPEG algorithm devotes more effort to reducing color data
`than brigh tness.
`Therefore, MPEG compression is based on two types of
`redundancies in video sequences, these being spatial, which
`is the redundancy in an individual frame, and temporal,
`which is the redundancy between consecutive frames. Spa-
`tial compression is achieved by considering the frequency
`characteristics of a picture frame. Each frame is divided into
`non-overlapping'blocks, and cach block is transformcd via
`the discrete cosine transform (DCT). After the transformed
`blocks are converted to the "DCT domain", each entry in the
`transformed block is quantized with respect to a set of
`quantization tables. The quantization step for each entry can
`vary, taking into a ccount the sensitivity of the human visual
`system (HVS)} to the frequency. Since the H-VS is more
`sensitive to low frequencies, most of the high frequency
`*entries are quantized to zero. In this step where the entries
`are quantized, information is lost and errors are introduced
`to the reconstructed image. Run length encoding is used to
`the quantized. values. To further enhance
`transmit
`compression, the blocks are scanned in a zig-zag ordering
`that scans the lower frequency entries first, and the non-zero
`quantized values, along with the zero run lengths, are
`entropy encoded.
`When an MPEG d ecoder receives an encoded stream, the
`MPEG decoder reverses the above. operations. Thus the
`MPEG decoder performs inverse scanning to remove the zig
`zag ordering, inverse quantization to de-quantize the data,
`and the inverse DCT1 to convert the data from the frequency
`domain back to the pixel domain. The MPEG decoder also
`* performs motion compensation using the transmitted motion
`vectors to recreate the temporally compressed frames.
`
`65
`
`SUMMARY OF THJE INVENTION
`The present invention comp)rises an MPEG decoder sys-
`tem and method for performing video decoding or decom-
`pression which includes a unified memory for multiple
`
`5,870,087
`
`15
`
`.4
`When frames are received which are used as references
`for other frames, such as I or P frames, these frames are
`decoded and stored in memory. When a temporally COM-
`pressed or encoded frame is received, such as 'a P or B frame,
`5 motion compensation is performed on the frame using the
`prior decoded I or P reference frames. The temporally
`compressed or encoded frame, referred to as a target frame,
`will include motion vectors which reference blocks in prior
`decoded I or P frames stored in the memory. The MPEG
`10 decoder examines the motion vector, determines the respec-
`tive reference block in the reference frame, and accesses the
`reference block pointed'to by the motion vector from the
`.-
`memory.
`A typical MPEG decoder includes motion compensation
`logic which includes local or on-chip memory. The MPEG
`decoder also includes an external memory which stores prior
`decoded reference frames. The MPEG decoder accesses the
`reference frames or anchor frames stored in the external
`temporally compressed
`in order. to reconstruct
`memory
`20 frames. The MPEG decoder also -typically stores the frame
`being reconstructed in the external memory.
`An MPEG decoder system also typically includes trans-
`port logic which operates to demultiplex received data into
`25a plurality' of individual multimedia st reams. An MPEG
`25decoder system also generally includes a system controller
`which controls operations in the system and executes pro-
`grams or applets.
`Prior art MPEG video~ decoder systems have generally
`30 used a frame store memory for the MPEG decoder motion
`compensation logic which stores the reference frames or
`anchor frames as' well as the frame being reconstructed.
`Prior art MPEG video decoder systems have also generally
`included a separate memory for the transport and system
`35controller functions. It has generally. not been possible to
`to size limitations. For
`combine
`these memories, due
`ex ample, current memory devices are fabricated on an 4
`Mbit granularity. In prior art systems, the memory require-
`ments for the transport and system controller functions as
`logic would'
`the decoder motion compensation
`40 well as
`exceed 16 Mbits of memory, thus requiring 20 or 24 Mbits
`of memory. This additional memory adds considerable cost
`to the system.
`in the
`The amount of memory is a major cost item
`45 production of video decoders. Thus, it is desired to reduce
`the memory requirements of the decoder -system as much as
`possible to reduce its size and cost. Since practical memory
`devices are implemented using particular convenient dis-
`crete sizes, it is important to stay within a particular size if
`50 possible for commercial reasons. For example, it is desired
`to keep the memory requirements below a particular size of
`memory, such as 16 Mb, since otherwise a memory device
`of 20 or 24 Mb would have to be used, resulting in greater,
`cost and extraneous storage area. As mentioned above, it has
`the memory
`to combine
`55 heretofore not been possible
`required for the transport and system controller functions
`with the memory required for the MPE G decoder logic due
`to the memory size requirements.
`Therefore, a new video decoder system and method is
`60 desired which efficiently uses memory. and combines the
`memory subsystem for reduced memory requirements and
`hence reduced cost.
`
`SONY EX. 1002
`Page 25
`
`

`
`5,870,087
`
`5
`to the present invention. The video
`functions according
`includes transport logic, a system
`decoding. system
`controller, and MPEG decoder logic. The video'decoding
`system of the present invention includes* a single unified
`memory which stores code- and data for the transport logic,
`system controller and MPEG decoder functions. The single
`unified memory is preferably a 16 Mbit memory. The
`present invention thus requires only-a single memory, and
`thus has reduced memory requirements compared to prior
`art designs.
`The video decoding system includes transport logic which
`operates to demultiplex received data into;.a plurality of
`individual multimedia streams. The video decoding system
`also includes a system controller which controls operations
`in the system and executes programs or-applets. The video
`decoding system further includes decoding logic, preferably
`MPEG decoder logic, which performs motion compensation
`between temporally compressed frames of a video sequence
`during video decoding or video decompression. The
`memory includes a plurality of memory portions, including2
`a video frame portion for storing video frames, a system
`controller portion for storing code and data executable by
`the system controller, and a transport buffer for storing data
`the transport logic. The MPEG decoder
`used by
`logic
`preferably includes a memory controller which couples to2
`the single unified memory. Each of the transport logic,
`system controller, and MPEG decoder logic accesses the
`single unified memory through the memory controller.
`Thbe video decoding system implements various frame3
`memory saving schemes, such as compression or dynamic
`allocation,. to reduce the 'required amount of frame store
`is not
`the memory
`memory. Also, in one embodiment,
`required to store reconstructed frame data during motion
`compensation, thus considerably reducing the required3
`the
`amount of memory for this function. Alternatively,
`memory is only required to store a portion of the. recon-
`structed frame data. These savings in memory allow portions
`of the memory to also be used for transport and system
`controller functions.
`The present invention thus provides a video decoding4
`system with reduced memory requirements.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`A better understanding of the present invention can be4
`obtained when the following detailed description of the
`preferred embodiment is considered in conjunction with the
`following drawings, in which:
`FIG. 1 illustrates a computer system which performs
`video decoding and which includes a motion compensationf
`logic having a frame memory which stores reference block
`data according to the present invention;
`FIG. 2 is a block diag ram illustrating the computer system
`of FIG. 1;
`FIG. 3 isa block diagram illustrating an MPEG decoder
`system including a unified memory for MPEG transport,
`system controller, and decode functions according to the
`present invention;
`FIG. 4 is a block diagram illustrating the MPEG decoder
`logic in the system of FIG. 3;
`FIG. 5 illustrates,various frame memory saving.schemes
`used in various embodiments of the invention;
`FIGS. 6a and' 6b illustrate a table listing the memory
`partitions under different display schemes;
`FIG. 7 illustrates the relationship of memory bandwidth
`vs. memory size in the NTSC decoding scheme;
`
`FIG. 8 illustrates the relationship of: memory bandwidth
`vs. memory size in the PAL encoding scheme;
`FIG. 9 illustrates the memory partitions. according to the
`preferred embodiment of the invention;.
`5 FIG. 10 illustrates the estimated memory bandwidth dis-
`tribution in the preferred embodiment of the invention;
`FIG. 11 illustrates the "worst case" relationship of pro-
`cessing power vs. memory size in the NTSC decoding
`10 scheme;
`FIG. 12,illustrates the clock domains in the system;
`FIG. 13 illustrates clock operating frequencies according
`to the preferred embodiment of the invention;
`FIG. 14 illustrates.an example of the packet data interface
`15 between the transport controller.and the source decoder;'and
`FIG. 15 illustrates packet header formats used in the
`preferred embodiment.
`
`20
`
`DETAILED DESCRIPTION OF THE
`PREFERRED EMBODIMENT'
`Video Compression System
`Referring now to FIG.J., a system for performing video
`decoding or decompression and including a-unified memory
`to the present invention is shown. The video
`according
`25 decoding system of the present invention includes a single
`unified memory which stores code and data for the transport,
`system controller and MPEG decoder functions. This sim-
`plifies the design and reduces the memory requirements in
`the system.
`As shown, in one embodiment the video decoding or
`decompression system is comprised in a general purpose
`computer system 60. The video decoding'system may com-
`prise any of various types of systems, including a computer
`system, set-top box, television, or other device.
`. The computer system 60 is preferably coupled to a media
`storage unit 62 which stores digital video files which are to
`be decompressed or decoded. by the computer system 60.
`The media storage unit 62 may also store the resultant
`decoded or decompressed video file.
`In the preferred
`40 embodiment, the computer system 60 receives a compressed
`video file or bitstream and generates a normal uncompressed
`digital video file. In the present disclosure, the term "com-
`pressed video file" refers to a video file which has been
`compressed according to any of various video compression
`45 algorithms which use motion estimation techniques, includ-
`ing the MPEG standard, among others, and the
`term
`"uncompressed digital video file" refers to a stream of
`decoded or uncompressed video.
`As shown, the computer system 60 preferably includes a
`50 video decoder 74 which performs video decoding or decom-
`pression ope 'rations. The video decoder 74 is preferably an
`MPEG decoder. The computer system 60 optionally may
`also include an MPEG encoder 76. The MPEG decoder 74
`and MPEG encoder 76 are preferably adapter cards coupled
`to a bus in the computer system, but are 'shown external to
`the computer system 60 for illustrative purposes. The com-
`puter system 60 also includes software, represented by
`floppy disks 72, which may perform portions of the video
`decompression or decoding. operation and/or may perform
`60 other operations, as desired.
`The computer system 60 preferably
`includes various
`standard components, including one or more processors, one
`or more buses, a hard drive and memory. Referring now to
`FIG. 2, a block diagram illustrating the components com-
`65 p)rised in the computer system of FIG. 1 is shown. It is noted
`that FIG. 2 is illustrative only, and other computer architec-
`tures may be used, as desired. As shown, the computer
`
`30
`
`35
`
`55
`
`SONY EX. 1002
`Page 26
`
`

`
`5,870,087
`
`7
`system includes at least one processor 80 coupled through
`chipset logic 82 to a system memory 84. The chipset 82
`preferably includes a. PCI (Peripheral Component
`Interconnect) bridge for interfacing to PCI bus 86, or another
`type of bus bridge for interfacing to another type of expan-
`sion bus. In FIG. 2, MPEG decoder 74 and MPEG encoder
`76 are shown connected to PCI bus 86. Various other
`components may be comprised in the computer system, such
`as video 88 and hard drive 90.
`:As mentioned above, in the preferred embodiment of FIG.
`1 the computer system 60 includes or is coupled to one or
`more digital storage or media storage devices.Jor example,
`the computer system 60
`in the embodiment'of FIG. 1,
`couples to media storage unit 62 through cable 64. The
`media storage unit 62 preferably comprises a RAID
`(Redundant Array of Inexpensive Disks) disk array, or
`includes one or more CD-ROM drives and/or one or more
`Digital Video Disk (DVD) storage units; or other media, for
`storing digital video to be decompressed and/or for storing
`the resultant decoded video data. The computer system may
`also include one or .more internal RAID arrays, CD-ROM
`drives and/or may couple to one or more separate Digital
`Video Disk (DVD) storage units. The computer system 60
`also may connect to other types of digital or analog storage
`devices or media, as desired.
`Alternatively, the compressed digital video file may be
`received from an external source, such as a remote storage
`device or remote computer system. In this embodiment, the
`computer system preferably includes an input devic