`Diaz et al.
`
`[19]
`
`[11] Patent Number:
`
`5,812,789
`
`[45] Date of Patent:
`
`Sep. 22, 1998
`
`US005812789/\
`
`[54] VIDEO AND/OR AUDIO DECOMPRESSION
`AND/OR COMPRESSION DEVICE THAT
`SHARES A MEMORY INTERFACE
`
`[75]
`
`Inventors; Raul Zegers Diaz, Palo Alto; Jelferson
`Eugene Owen, Freernont, both of Calif.
`
`[73] Assignee: STMicruelectrunics, Inc., Carrollton,
`Tex.
`
`[21] Appl. No.: 702,911
`
`[22]
`
`Filed:
`
`Aug. 26,1996
`
`G06F 17/00
`Int. Cl.6.
`[51]
`395/200.77
`[52] U.S.Cl.
`[58] Field of Search ......................... 395/200.77, 200.82,
`395/507, 890, 729; 348/402, 407, 10
`
`[56]
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`Galbi, D. et al., “An MPEG-1 Audio/Video Decoder With
`Run-Length Compressed Antialiased Video Overlays,”
`1995 IEEE International Solid-State Circuits Conference,
`pp. 286-287, 381.
`Maturi, G., “Single Chip MPEG Audio Decoder,"" IEEE
`Transactioiis on Consumer Electronics, vol. 38, N o. 3, Aug.
`1992, PP. 348-356.
`Butler, B. and T. Mace, “The Great Leap Forward,” PC
`Magazine, Oct. 11, 1994, pp. 241-244, 246, 248, 250,
`253-254, 256, 260-261, 264, 266-268, 273-275, 278.
`Doquilo, J., “Symmetric Multiprocessing Servers: Scaling
`the Performance Wall,” Infoworld, Mar. 27, 1995, pp.
`82-85, 88-92.
`Video Electronics Standards Association, “VESA Unified
`Memory Architechture Hardware Specilications Proposal,”
`Version: 1.0p, Oct. 31, 1995, pp. 1-38.
`Video Electronics Standards Association, “VESA Unified
`Memory Architecture VESABIOS Extensions VUMA-SBE
`Proposal,” Version: 1.0p, Nov. 1, 1995, pp. 1-26.
`King, A., Inside Windows 95, Microsoft Press, Redmond,
`Washington, 1994, pp. 85-90.
`“MPEG Video Overview,” SGS-Thomson A/Iicroelectronics
`Technical Note, Apr. 1992, pp. 1-4.
`“On the Bus Arbitration for MPEG 2 Video Decoder” ; VLSI
`Tech, System and Application, 1995 Symposium.
`“A Low Cost Graphics and Multimedia Workstation Chip
`Set”; IEEI: Micro, 1994.
`
`4,774,660
`4,894,565
`5,027,400
`5,371,893
`5,459,519
`5.522,080
`5,557,538
`5,598,525
`5.621,893
`5,623,672
`
`..............
`9/1988 Conforti
`1/1990 Marquardt
`6/1991 Baji et al.
`12/1994 Price et al.
`10/1995 Scalise et al.
`5/1996 Harney et al.
`9/1996 Retter et al.
`1/1997 Nally et al.
`4/1997 Joli
`..... ..
`4/1997 Poppal
`
`
`
`....... .. 395/729
`. 395/729
`348/10
`. 395/729
`. 348/431
`. 395/729
`. 348/402
`. 395/507
`395/200.82
`....... .. 395/729
`
`
`
`..
`
`.
`
`FOREIGN PMENT DOCUMENTS
`
`O 673 171 A2
`
`9/1995
`
`European Fat. 011'.
`
`.
`
`Primary Examiner—Ellis B. Ramirez
`Attorney, Agent, or Firm—David V. Carlson; Theodore E.
`Galanthay; Lisa K. Jorgenson
`
`OTHER PUBLICATIONS
`
`[57]
`
`ABSTRACT
`
`Bheda, H. and P. Srinivasan, “A High-Performance Cross-
`Platform MPEG Decoder,” Digital Wdeo Compression on
`Personal Computers: Algorithms and Technologies. SPIE
`Proceedings, Feb. 7-8, 1994, V0. 2187, pp. 241-248.
`Bursky, D., “Highly Integrated Controller Eases MPEG-2
`Adoption,” Electronic Design, Aug. 21, 1995, vol. 43, No.
`17, pp. 141-142.
`
`A11 electronic system that contains a first device that requires
`a memory interface and video and/or audio decompression
`and/or compression device that shares a memory interface
`and memory with the first device While still permitting the
`video and/or audio decompression and/or compression
`device to operate in real time is disclosed.
`
`33 Claims, 5 Drawing Sheets
`
` FlRS1 DEV1CE
`
`
`
`
`EMA [HOWE
`
`
`REFRES1 LOGIC
`
`NA LNGJN1:
`
`v
`I
`MBITER
`
`MEMORY CONTROLLER
`MEMORY [NTRFACE
`
`Page 1 of 13
`
`Samsung Exhibit 1001
`Petitioners HTC & LG - Exhibit 1001, p. l
`
`Petitioners HTC & LG - Exhibit 1001, p. 1
`
`
`
`U.S. Patent
`
`Sep. 22, 1998
`
`Sheet 1 0f5
`
`5,812,789
`
`
`
`
`
`MEMORY
`INTERFACE
`
`VIDEO DECODING
`CIRCUIT
`
`CIRCUIT
`
`M
`
`
`I 2 I
`I AUDIO DECODING I
`
`
`
`
`
`
`
`
`FIG.
`1 a
`(PRIOR ART)
`
`VIDEO DECODING
`CIRCUIT
`
`
`
`I 12 I
`l AUDIO DECODING I
`
`
`
`CIRCUIT
`
`M
`
`FIG.
`
`7 b
`
`(PRIOR ART)
`
`Page 2 of 13
`
`Petitioners HTC & LG - Exhibit 1001, p. 2
`
`Petitioners HTC & LG - Exhibit 1001, p. 2
`
`
`
`+LnuE+LaP&U
`
`Sep. 22, 1998
`
`Sheet 2 of 5
`
`5,812,789
`
`Emacs
`
`2:
`
`mam<m_
`
`wa_
`
`N?
`
`_o¢
`
`moo_mm
`
`_u¢
`
`Hmm¢_Io
`
`z_<2
`
`>mozmz
`
`we
`
`mam_o¢
`
`o~_
`
`M:6.3
`
`:3memo:
`
`¢m_mama
`
`
`
`xm_oom<I
`
`
`
`mo_mmw_
`
`of
`
`o>a
`
`20¢no
`
`N4
`
`m:o<oN©_
`
`2:
`
`mmooomo
`
`o_
`
`»mo:m2
`
`8
`
`Xl
`
`\
`
`Em5mi
`
`o_o=<
`
`omooo
`
`aw
`
`_N9
`
`m2<m_
`
`mm__=m
`
`»<4mm_o
`
`
`¢EE>z8maém$98
`92mssm82>:53
`
`mo_Ia<mu
`
`moH<mm4moo<
`
`me
`
`mo<¢mm_z_
`
`_
`V5
`
`mm42omHzoo
`
`253
`
`Page 3 of 13
`
`LI.6P
`
`GL&ppTHST.Cn.m.H.
`
`_
`
`.H.mhxE
`
`AU1
`
`AU
`
`3D1,
`
`Petitioners HTC & LG - Exhibit 1001, p. 3
`
`
`
`
`U.S. Patent
`
`Sep. 22, 1998
`
`Sheet 3 0f5
`
`5,812,789
`
`222
`
`mzaoma
`
`E55
`
`%__m_8m
`
`8§Ez_
`
`elm
`
`mSE55:
`
`8
`
`mzazH._<3
`
`N65%
`
`Page 4 of 13
`
`Petitioners HTC & LG - Exhibit 1001, p. 4
`
`Petitioners HTC & LG - Exhibit 1001, p. 4
`
`
`
`tHEtaP&U
`
`Sep. 22, 1998
`
`Sheet 4 of 5
`
`5,812,789
`
`mi0:
`
`EE
`
`mm:
`
`32mm:
`
`£5:
`
`
`
`fita3&5
`
`5§H.:z_
`
`§_
`
`of922
`
`058
`
`oz<~_m_._<om82>
`
`1:5
`
`9E$>z8Himx38
`
`8.TE<mo
`
`~_o:$:8<
`
`mm:
`
`$3528
`
`25S_
`
`m.b~_._,~
`
`mamGa
`
`2_
`
`X:
`
`xma9::
`
`mag
`
`of
`
`as3
`
`£|_
`
`mm
`
`E5.59
`
`%<
`
`ole3E003E083
`Q69E8
`_on_HE
`
`3
`
`3%
`
`on_EOE:we/.22:
`
`m._
`
`m=._o<o
`
`N3
`
`Page 5 of 13
`
`LI.6P
`
`GL&CTHST.Cn.m.H.
`
`_
`
`.H.mhxE
`
`AU1
`
`AU
`
`5D1,
`
`Petitioners HTC & LG - Exhibit 1001, p. 5
`
`
`
`
`
`
`
`
`U.S. Patent
`
`Sep. 22, 1998
`
`Sheet 5 0f5
`
`5,812,789
`
`9:
`
`$22,I
`
`m
`
`..§
`
`E
`
`<._
`
`.._.._3mE,02
`
`B§Ez_E05:
`
`93
`
`$2
`
`m:
`
`Emom
`H.5003052T23
`5§Ez_mamQ59H
`E089mE<$d8<R
`%:Ed8<on
`
`Wmoouzm
`
`fl
`
`SN
`
`EN
`
`EN
`
`o_N
`
`8§Ez_
`
`_
`:
`
`Ejofizoo
`
`25N:
`
`oi
`
`9:
`
`20m8ea
`
`m
`
`8<2
`
`9:
`
`NE
`
`_on_
`
`52%
`
`O2
`
`3
`
`SmGa
`
`51222R:
`
`amazeE052we
`
`
`
`X:H._>_m_o
`
`xma92:
`
`New
`
`mm:
`
`NVGEN
`
`Page 6 of 13
`
`LI.6P
`
`GL&CTHST.Cn.m.H.
`
`_
`
`.H.mhxE
`
`AU1
`
`AU
`
`/0D1,
`
`Petitioners HTC & LG - Exhibit 1001, p. 6
`
`
`
`
`
`5,812,789
`
`1
`VIDEO AND/OR AUDIO DECOMPRESSION
`AND/OR COMPRESSION DEVICE THAT
`SHARES A MEMORY INTERFACE
`
`CROSS-REFERENCE TO RELATED
`APPLICATIONS
`
`This application contains some text and drawings in
`common with pending U.S. patent applications entitled:
`“Video and/or Audio Decompression and/or Compression
`Device that Shares a Memory” by Jefferson E. Owen, Raul
`Z. Diaz, and Osvaldo Colavin Ser. No. 08/702,910 filed on
`Aug. 26, 1996, and has the same effective filing date and
`ownership as the present application, and to that extent is
`related to the present application, which is incorporated
`herein by reference.
`
`BACKGRO UND
`
`The present invention relates to the field of electronic
`systems having a video and/or audio decompression and/or
`compression device, and is more specifically directed to
`sharing a memory interface between a video and/or audio
`decompression and/or compression device and another
`device contained in the electronic system.
`The size of a digital representation of uncompressed video
`images is dependent on the resolution, and color depth of the
`image. Amovie composed of a sequence of such images, and
`the audio signals that go along with them, quickly becomes
`large enough so that uncompressed such a movie typically
`cannot fit entirely onto conventional recording medium,
`such as a CD. It is also typically now prohibitively expen-
`sive to transmit such a movie uncompressed.
`It is therefore advantageous to compress video and audio
`sequences before they are transmitted or stored. A great deal
`of effort is being expanded to develop systems to compress
`these sequences. There are several coding standards cur-
`rently used that are based on the discrete cosine transfer
`algorithm including MPEG-1, MPEG-2, H.261, and H.263.
`(MPEG stands for “Motion Picture Expert Group”, a com-
`mittee of the International Organization for Standardization,
`ISO.) The MPEG-1 , MPEG-2, H.261 , and H.263 standards
`are decompression protocols that describe how an encoded
`bitstream is to be decoded. The encoding can be done in any
`manner, as long as the resulting bitstream complies with the
`standard.
`
`Video and/or audio compression devices (hereinafter
`encoders) are used to encode the video and/or audio
`sequence before it is transmitted or stored. The resulting
`bitstream is decoded by a video and/or audio decompression
`device (hereinafter decoder) before the video and/or audio
`sequence is displayed. However, a bitstream can only be
`decoded by a decoder if it complies to the standard used by
`the decoder. To be able to decode the bitstream on a large
`number of systems it is advantageous to encode the video
`and/or audio sequences to comply to a well accepted decom-
`pression standard. The MPEG standards are currently well
`accepted standards for one way communication. H.261, and
`H.263 are currently well accepted standards for video tele-
`phony.
`Once decoded the images can be displayed on an elec-
`tronic system dedicated to displaying video and audio, such
`as television or digital video disk (DVD) player, or on
`electronic systems where image display is just one feature of
`the system, such as a computer. A decoder needs to be added
`to these systems to allow them to display compressed
`sequences, such as received images and associated audio, or
`ones taken from a storage device. An encoder needs to be
`
`10
`
`15
`
`20
`
`25
`
`35
`
`40
`
`45
`
`55
`
`60
`
`65
`
`2
`added to allow the system to compress video and/or audio
`sequences, to be transmitted or stored. Both need to be added
`for two way communication such as video telephony.
`A typical decoder, such as an MPEG decoder 10 shown in
`FIG. 1a, contains video decoding circuitry 12, audio decod-
`ing circuitry 14, a microcontroller 16, and a memory inter-
`face 18. The decoder can also contain other circuitry
`depending on the electronic system the decoder is designed
`to operate in. For example, when the decoder is designed to
`operate in a typical television the decoder will also contain
`an on screen display (OSD) circuit.
`FIG. 1b shows a better decoder architecture, used in the
`STi3520 and STi3520A MPEG Audio/MPEG-2 Video Inte-
`grated Decoder manufactured by SGS-THOMSON Micro-
`electronics. The decoder has a register interface 20 instead
`of a microcontroller. The register interface 20 is coupled to
`an external microcontroller 24. The use of a register inter-
`face 20 makes it possible to tailor the decoder 10 to the
`specific hardware the decoder 10 interfaces with or change
`its operation without having to replace the decoder by just
`reprogramming the register interface. It also allows the user
`to replace the microcontroller 24, to upgrade or tailor the
`microcontroller 24 to a specific use, by just replacing the
`microcontroller and reprogramming the register interface
`20, without having to replace the decoder 10.
`The memory interface 18 is coupled to a memory 22. A
`typical MPEG decoder 10 requires 16 Mbits of memory to
`operate in the main profile at main level mode (MP at ML).
`This typically means that the decoder requires a 2 Mbyte
`memory. Memory 22 is dedicated to the MPEG decoder 10
`and increases the price of adding a decoder 10 to the
`electronic system. In current technology the cost of this
`additional dedicated memory 22 can be a significant per-
`centage of the cost of the decoder.
`An encoder also requires a memory interface 18 and
`dedicated memory. Adding the encoder to an electronic
`system again increases the price of the system by both the
`price of the encoder and its dedicated memory.
`A goal in the semiconductor industry is to reduce the die
`area of an integrated circuit device for a given functionality.
`Some advantages of reducing the die area is the increase in
`the number of the die that can be manufactured on same size
`
`silicon wafer, and the reduction in price per die resulting
`therefrom. This results in both an increase in volume and
`reduction in price of the device.
`Many of the functional circuits described above for FIG.
`1a and FIG. 1b take up a lot of die space. However, each of
`them is needed to make the respective decoder operate.
`FIG. 1c shows a computer 25 containing a decoder 10, a
`main memory 168 and other typical components such as a
`modem 199, and graphics accelerator 188. The decoder 10
`and the rest of the components are coupled to the core logic
`chipset 190 through a bus 170. The bus is typically a PCI
`(peripheral component interface) or ISA (industry standard
`architecture) bus, and each component contains an appro-
`priate interface for interfacing with the bus.
`When any component needs access to the memory 168
`either to read from or write to the main memory 168, it
`generates a request which is placed on the bus 26. When the
`request is a write the data to be written is also placed on the
`bus 26. The request is processed in the core logic chipset I90
`and the data is then either written to or read from the main
`memory 168. When data is read from the main memory 168
`the data is now placed on the bus and goes to the component
`that requested the read.
`There are typically many components in the computer
`systems that may require access to the main memory 168,
`
`Page 7 of 13
`
`Petitioners HTC & LG - Exhibit 1001, p. 7
`
`Petitioners HTC & LG - Exhibit 1001, p. 7
`
`
`
`5,812,789
`
`3
`and they are typically all coupled to the same bus 174, or
`possibly several buses 170, 198 connected together by a PCI
`bridge 192, if there are not enough connectors on one bus to
`accommodate all of the peripherals. However, the addition
`of each bus is very expensive. Each request is typically
`processed according to a priority scheme. The priority
`scheme is typically based on the priority given to the device
`and the order in which the requests are received. Typically,
`the priority scheme is set up so no device monopolizes the
`bus, starving all of the other devices. Good practice suggests
`that no device on the bus require more than approximately
`50% of the bus’s bandwidth.
`
`10
`
`The minimum bandwidth required for the decoder 10 can
`be calculated based on the characteristics and desired opera-
`tion of the decoder. These characteristics include the stan-
`
`15
`
`dard to which the bitstream is encoded to comply with,
`whether the decoder is to operate in real time, to what extent
`frames are dropped, and how the images are stored.
`Additionally, the latency of the bus that couples the decoder
`to the memory should be considered.
`If the decoder does not operate in real time the decoded
`movie would stop periodically between images until the
`decoder can get access to the memory to process the next
`image. The movie may stop quite often between images and
`wait.
`
`To reduce the minimum required bandwidth and still
`operate in real
`time,
`the decoder 10 may need to drop
`frames. If the decoder 10 regularly does not decode every
`frame then it may not need to stop between images.
`However, this produces very poor continuity in the images.
`This is problematic with an image encoded to the MPEG-1
`or MPEG-2 standard, or any standards that uses temporal
`compression. In temporal (interpicture) compression some
`of the images are decoded based on previous images and
`some based on previous and future images. Dropping an
`image on which the decoding of other images is based is
`unacceptable and will result in many poor or even com-
`pletely unrecognizable images.
`The computer can also contain both a decoder and
`encoder to allow for video telephony, as described above. In
`this case not operating in real time would mean that the
`length of time between the occurrence of an event, such as
`speaking, at one end of the conversation until the event is
`displayed at the other end of the conversation is increased by
`the time both the encoder and then the decoder must wait to
`
`get access to the bus and the main memory. Not being able
`to operate in real time means that there would be gaps in the
`conversation until
`the equipment can catch up. This
`increases the time needed to have a video conference, and
`makes the conference uncomfortable for the participants.
`One widely used solution to allow a component in a
`computer system to operate in real
`time is to give the
`component its own dedicated memory. Thus, as shown in
`FIG. 1C,
`the decoder 10 can be given its own dedicated
`memory 22, with a dedicated bus 26 to connect the decoder
`10 to its memory 22. The dedicated memory 22, its con-
`troller and the pins to control this memory significantly
`increase the cost of adding a decoder 10 to the computer.
`SUMMARY OF THE INVENTION
`
`The present application discloses an electronic system
`that contains a first device and video and/or audio decom-
`pression and/or compression device capable of operating in
`real time. Both the first device and the video and/or audio
`decompression and/or compression device require a
`memory interface. The video and/or audio decompression
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`4
`and/or compression device shares a memory interface and
`the memory with the first device. In the preferred embodi-
`ment of the invention the shared memory interface contains
`an arbiter. The arbiter and DMA engines of the video and/or
`audio decompression and/or compression device and of the
`first device are configured to arbitrate between the two
`devices when one of them is requesting access to the
`memory. This allows the use of one memory interface to
`control the access of both the video and/or audio decom-
`pression and/or compression device and the first device to
`the memory.
`When the video and/or audio decompression and/or com-
`pression device used in an electronic system, such as a
`computer, already containing a device that has a memory
`interface the video and/or audio decompression and/or com-
`pression device can share that memory interface and the
`memory of the device and the memory interface and
`memory of the video and/or audio decompression and/or
`compression device can be eliminated. Eliminating this
`memory interface reduces the die area without changing the
`critical dimensions of the device. Therefore increasing the
`volume and reducing the cost of the decoder or encoder.
`Eliminating the memory greatly reduces the cost of adding
`the video and/or audio decompression and/or compression
`device to the electronic system while not requiring the video
`and/or audio decompression and/or compression device to
`be connected to the system bus, allowing the video and/or
`audio decompression and/or compression device to operate
`in real time.
`
`An advantage of the present invention is significant cost
`reduction due to the fact that the video and/or audio decom-
`pression and/or compression device does not need its own
`dedicated memory but can share a memory with another
`device and still operate in real time.
`Another significant advantage of the present invention is
`that the die space needed for the video and/or audio decom-
`pression and/or compression device is smaller because the
`memory interface on the video and/or audio decompression
`and/or compression device is eliminated.
`A further advantage of the present invention is that the
`video and/or audio decompression and/or compression
`device can share the memory of the device with which it is
`sharing the memory interface more efficiently.
`Another advantage of the present invention is that the cost
`of producing a video and/or audio decompression and/or
`compression device is reduced because the memory inter-
`face on the video and/or audio decompression and/or com-
`pression device is eliminated.
`Another advantage of the present invention is that the
`video and/or audio decompression and/or compression
`device can be monolithically integrated into the first device
`and no extra packaging or pins are needed for the video
`and/or audio decompression and/or compression device, and
`no pins are needed for the first device to connect to the video
`and/or audio decompression and/or compression device,
`saving pins on both devices and producing a better connec-
`tion between the two devices.
`
`Other advantages and objects of the invention will be
`apparent to those of ordinary skill in the art having reference
`to the following specification together with the drawings.
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`FIGS. la and 1b are electrical diagrams, in block form, of
`prior art decoders.
`FIG. 1c is an electrical diagram, in block form, of a
`computer system containing a decoder according to the prior
`art.
`
`Page 8 of 13
`
`Petitioners HTC & LG - Exhibit 1001, p. 8
`
`Petitioners HTC & LG - Exhibit 1001, p. 8
`
`
`
`5,812,789
`
`5
`in block form, of an
`FIG. 2 is an electrical diagram,
`electronic system containing a device having a memory
`interface and an encoder and decoder.
`
`in block form, of a
`FIG. 3 is an electrical diagram,
`computer system containing a core logic chipset designed
`for the CPU to share a memory interface with an encoder
`and decoder.
`
`in block form, of a
`FIG. 4 is an electrical diagram,
`computer system containing a graphics accelerator designed
`to share a memory interface with an encoder and/or decoder.
`
`DETAILED DESCRIPTION OF THE
`PREFERRED EMBODIMENT
`
`FIG. 2 shows an electronic system 40 containing a first
`device 42 having access to a memory 50 through a memory
`interface 48, and a decoder 44 and encoder 46, having access
`to the same memory 50 through the same memory interface
`48. First device 42 can be a processor, a core logic chipset,
`a graphics accelerator, or any other device that requires
`access to the memory 50, and either contains or is coupled
`to a memory interface. Any parts common to FIGS. 1
`through 4 are indicated using the same numbering system. In
`the preferred embodiment of the invention, electronic sys-
`tem 40 contains a first device 42, a decoder 44, an encoder
`46, a memory interface 48, and a memory 50. Although,
`either the decoder 44 or encoder 46 can be used in the
`decoder/encoder 45 without the other. For ease of reference,
`a video and/or audio decompression and/or compression
`device 45 will hereinafter be referred to as decoder/encoder
`45. The decoder/encoder 45 may be a single device, or cell
`on an integrated circuit, or may be two separate devices, or
`cells in an integrated circuit. In the preferred embodiment of
`the invention, the first device 42, decoder/encoder 45, and
`memory interface 48 are on one integrated circuit, however,
`they can be on separate integrated circuits in any combina-
`tion.
`
`10
`
`15
`
`20
`
`25
`
`35
`
`The decoder 44 includes a video decoding circuit 12 and
`an audio decoding circuit 14, both coupled to a register
`interface 20. The decoder 44 can be either a video and audio
`
`40
`
`decoder, just a video, or just an audio decoder. If the decoder
`44 is just a video decoder it does not contain the audio
`decoding circuitry 14. The audio decoding can be performed
`by a separate audio codec coupled to the first device 42, or
`through software.
`In the preferred embodiment of the
`invention, when the decoder/encoder 45 is in a system
`containing a processor and is coupled to the processor, the
`audio decoding is performed in software. This frees up space
`on the die without causing significant delay in the decoding.
`If the audio decoding is performed in software, the processor
`should preferably operate at a speed to allow the audio
`decoding to be performed in real time without starving other
`components of the system that may need to utilize the
`processor. For example, currently software to perform AC-3
`audio decoding takes up approximately 40% of the band-
`width of a 133 MHZ Pentium. The encoder 46 includes a
`
`video encoding circuit 62 and an audio encoding circuit 64,
`both coupled to a register interface 20. The encoder 46 can
`be either a video and audio encoder, just a video, or just an
`audio encoder. If the encoder 46 is just a video encoder, it
`does not contain the audio encoding circuitry 64. The audio
`encoding can be performed by a separate audio codec
`coupled to the first device 42, or through software. In the
`preferred embodiment of the invention, when the decoder/
`encoder 45 is in a system containing a processor and is
`coupled to the processor, the audio encoding is performed in
`software presenting the same advantages of freeing up space
`
`45
`
`55
`
`60
`
`65
`
`6
`on the die without causing significant delay in the encoding.
`The register interfaces 20 of the decoder 44 and encoder 46
`are coupled to a processor.
`The decoder 44 and encoder 46 are coupled to the direct
`memory access (DMA) engine 52. The decoder and encoder
`can be coupled to the same DMA engine as shown in FIG.
`2, or each can have its own DMA engine, or share a DMA
`engine with another device. When the decoder/encoder 45
`are two separate devices or cells, decoder 44 and encoder 46
`can still be coupled to one DMA engine 52. When the
`decoder/encoder is one device or is one cell on an integrated
`circuit, the DMA engine 52 can be part of the decoder/
`encoder 45, as shown in FIG. 2. The DMA engine 52 is
`coupled to the arbiter 54 of the memory interface 48.
`The first device 42 also contains a DMA engine 60. The
`DMA engine 60 of the first device 42 is coupled to the arbiter
`54 of the memory interface 48. The arbiter is also coupled
`to the refresh logic 58 and the memory controller 56. The
`memory interface 48 is coupled to a memory 50. The
`memory controller 56 is the control logic that generates the
`address the memory interface 48 accesses in the memory 50
`and the timing of the burst cycles.
`In current technology, memory 50 is typically a DRAM.
`However, other types of memory can be used. The refresh
`logic 58 is needed to refresh the DRAM. However, as is
`known in the art, if a different memory is used, the refresh
`logic 58 may not be needed and can be eliminated.
`The decoder/encoder 45 is coupled to the memory 50
`through devices, typically a bus 70, that have a bandwidth
`greater than the bandwidth required for the decoder/encoder
`45 to operate in real time. The minimum bandwidth required
`for the decoder/encoder 45 can be calculated based on the
`characteristics and desired operation of the decoder, includ-
`ing the standard to which the bitstream is encoded to comply
`with, whether the decoder/encoder 45 is to operate in real
`time, to what extent frames are dropped, and which images
`are stored. Additionally,
`the latency of the bus 70 that
`couples the decoder/encoder 45 to the memory 50 should be
`considered.
`
`A goal is to have the decoder/encoder 45 operate in real
`time without dropping so many frames that
`it becomes
`noticeable to the human viewer of the movie. To operate in
`real time the decoder/encoder 45 should decoder and/or
`encode images fast enough so that any delay in decoding
`and/or encoding cannot be detected by a human viewer. This
`means that the decoder/encoder 45 has a required bandwidth
`that allows the decoder/encoder 45 to operate fast enough to
`decode the entire image in the time between screen
`refreshes, which is typically 1/30 of a second, with the human
`viewer not being able to detect any delay in the decoding
`and/or encoding. To operate in real time the required band-
`width should be lower than the bandwidth of the bus. In
`
`order not to starve the other components on the bus, i.e. deny
`these components access to the memory for an amount of
`time that would interfere with their operation, this required
`bandwidth should be less the entire bandwidth of the bus.
`Therefore a fast bus 70 should be used. Afast bus 70 is any
`bus whose bandwidth is equal to or greater that the required
`bandwidth. There are busses in current technology, includ-
`ing the ISA bus, whose bandwidth is significantly below the
`bandwidth required for this.
`In the preferred embodiment of the invention the decoder/
`encoder 45 is coupled to the memory 50 through a fast bus
`70 that has a bandwidth of at least the bandwidth required
`for the decoder/encoder 45 to operate in real time, a thresh-
`old bandwidth. Preferably the fast bus 70 has a bandwidth of
`
`Page 9 of 13
`
`Petitioners HTC & LG - Exhibit 1001, p. 9
`
`Petitioners HTC & LG - Exhibit 1001, p. 9
`
`
`
`5,812,789
`
`7
`at least approximately twice the bandwidth required for the
`decoder/encoder 45 to operate in real time. In the preferred
`embodiment the fast bus 70 is a memory bus, however any
`bus having the required bandwidth can be used.
`The decoder/encoder 45 only requires access to the
`memory during operation. Therefore, when there is no need
`to decode or encode,
`the first device 42, and any other
`devices sharing the memory 50 have exclusive access to the
`memory and can use the entire bandwidth of the fast bus 70.
`In the preferred embodiment, even during decoding and
`encoding the decoder/encoder 45 does not always use the
`entire required bandwidth. Since the fast bus 70 has a
`bandwidth a little less than twice the required bandwidth the
`decoder/encoder 45 uses at most 60% of the bandwidth of
`the fast bus 70.
`
`The required bandwidth is determined based on the size
`and resolution of the image, and the type of frame (I, P, or
`B). In the preferred embodiment the decoder/encoder typi-
`cally will be using less than 40% of the bandwidth of the fast
`bus 70. This frees up the remaining bandwidth to be used by
`the other devices the decoder/encoder 45 is sharing the
`memory 50 with.
`The decoder/encoder 45 can decode a bitstream formatted
`according to any one or a combination of standards. In the
`preferred embodiment of the invention the decoder/encoder
`45 is a multi-standard decoder/encoder capable of decoding
`and encoding sequences formatted to comply to several well
`accepted standards. This allows the decoder/encoder 45 to
`be able to decode a large number of video and/or audio
`sequences. The choice of which standards the decoder/
`encoder 45 is capable of decoding bitstreams formatted to
`and of encoding sequences to comply to is based on the
`desired cost, efficiency, and application of the decoder/
`encoder 45.
`
`In the preferred embodiment, these standards are capable
`of both intrapicture compression and interpicture compres-
`sion. In intrapicture compression the redundancy within the
`image is eliminated. In interpicture compression the redun-
`dancy between two images are eliminated and only the
`difference information is transferred. This requires the
`decoder/encoder 45 to have access to the previous or future
`image that contains information needed to decode or encode
`the current image. These precious and/or future images need
`to be stored then used to decode the current image. This is
`one of the reasons the decoder/encoder 45 requires access to
`the memory, and requires a large bandwidth. The MPEG-1
`and MPEG-2 standards allow for decoding based on both
`previous images and/or future images. Therefore for a
`decoder/encoder 45 capable of operating in real time to be
`able to comply with the MPEG-1 and MPEG-2 standards it
`should be able to access two images, a previous and a future
`image, fast enough to decode the current image in the 1/30 of
`a second between screen refreshes.
`
`An MPEG environment is asymmetrical; there are much
`fewer encoders than decoders. The encoders are very diffi-
`cult and expensive to manufacture and the decoders are
`comparatively easy and cheap. This encourages many more
`decoders than encoders, with the encoders in centralized
`locations, and decoders available such that every end user
`can have a decoder. Therefore, there are many receivers but
`few transmitters.
`
`For video telephony and teleconferencing each end user
`has to be able to both receive and transmit. H.261, and H.263
`are currently well accepted standards for video telephony.
`An encoder that can encode sequences to comply to the
`H.261 and H.263 standards is less complicated, having a
`
`10
`
`15
`
`20
`
`25
`
`35
`
`40
`
`45
`
`55
`
`60
`
`65
`
`8
`lower resolution and lower frame rate than an encoder that
`
`complies to the MPEG-1 or MPEG-2 standards, possibly
`making the quality of the decoded images somewhat lower
`than those from an encoder that complies with the MPEG-1
`or MPEG-2 standards. Such an encoder, since it should be
`inexpensive and operate in real time, is also less efficient
`than an encoder to encode sequences to comply to the
`MPEG-1 or MPEG-2 standards. This means that the com-
`
`pression factor, which is the ratio between the source data
`rate and the encoded bitstream data rate, of such an encoder
`is lower for a given image quality than the compression
`factor of an MPEG encoder. However, because such an
`encoder is less complicated, it is much cheaper and faster
`than an encoder capable of complying with the MPEG-1
`and/or MPEG-2 standards. This makes video telephony
`possible, since both a long delay in encoding the signal and
`a cost that is prohibitively expensive for many users is
`unacceptable in video telephony.
`In the preferred embodiment, the decoder/encoder 45 is
`capable of decoding a bitstream formatted to comply to the
`MPEG-1, MPEG-2, H.261, and H.263 standards, and encod-
`ing a sequence to produce a bitstream to comply to the
`H.261, and H.263 standards. This allows the decoder/
`encoder 45 to be used for video telephony. Having the
`encoding comply to the H.261 and H.263 standards but not
`the MPEG-1 and MPEG-2 balances the desire to reduce the
`
`cost of transmission and storage by encoding to produce the
`highest compression factor and the desire to keep cost low
`enough to be able to mass market the device.
`The decoder/encoder 45 is preferably monolithically inte-
`grated into the first device as shown in FIG. 3 and FIG. 4.
`I