United States Patent (19)
`Guetz et al.
`
`US006091777A
`Patent Number:
`11
`(45) Date of Patent:
`
`6,091,777
`Jul.18, 2000
`
`54 CONTINUOUSLY ADAPTIVE DIGITAL
`VIDEO COMPRESSION SYSTEMAND
`METHOD FOR AWEB STREAMER
`
`5,680,129 10/1997 Weinberger et al. ..................... 341/65
`5,745,758 4/1998 Shaw et al. ............................. 395/672
`5,903,673 5/1999 Wang et al. ............................ 348/416
`
`75 Inventors: William N. Guetz; Keshoolal Bakhru;
`Fredric W. Ritchey, all of San Diego,
`Calif.
`73 Assignee: Cubic Video Technologies, Inc., San
`Diego, Calif.
`
`21 Appl. No.: 09/084,820
`22 Filed:
`May 26, 1998
`9
`Related U.S. Application Data
`
`63 Continuation-in-part of application No. 08/976,127, Sep. 18,
`1997.
`H04B 1/66; HO4N 7/12
`51 Int. Cl."
`52 U.S. Cl. .......................... 375/240; 348/384; 348/398;
`348/413; 382/242
`58 Field of Search ..................................... 348/384, 385,
`348/415, 413, 402, 390, 10, 14, 15, 17,
`12, 398, 416; 382/232, 244, 239, 246, 250,
`242: 395/200.77, 672; 345/336,337; 341/65;
`370/354, 468; 375/240
`
`56)
`
`References Cited
`U.S. PATENT DOCUMENTS
`
`4,903,317 2/1990 Nishihara et al. ...................... 382/244
`5,115,426 5/1992 Spanke .................
`... 370/468
`5,546,477 8/1996 Knowles et al. ........................ 382/242
`5,557,320 9/1996 Krebs ........................................ 348/12
`5,611,038 3/1997 Shaw et al. ...
`... 348/390
`5,617,142 4/1997 Hamilton ................................ 348/384
`
`Primary Examiner Richard Lee
`Assistant Examiner-Gims Philippe
`Attorney, Agent, or Firm-Frank D. Gilliam
`57
`ABSTRACT
`A cost effective continuously adaptive digital Video System
`and method for compressing color Video data for moving
`images. The method describes capturing an analog video
`frame and digitizing the image into a preferred Source input
`format for compression using a combination of unique lossy
`and lossleSS digital compression techniques including Sub
`band coding, wavelet transforms, motion detection, run
`length coding and variable length coding. The system
`includes encoder and decoder (CODEC) sections for com
`pression and decompression of visual images to provide
`high compression with good to excellent Video quality. The
`compressed video data provides a base Video layer and
`additional layers of video data that are multiplexed with
`compressed digital audio to provide a data Stream that can be
`packetized for distribution over inter or intranets, including
`wireless networks over local or wide areas. The (CODEC)
`System continuously adjusts the compression of the digital
`images frame by frame in response to comparing the avail
`able bandwidth on the data channel to the available band
`width on the channel for the previous frame to provide an
`output data Stream commensurate with the available band
`width of the network transmission channel and with the
`receiver resource capabilities of the client users. The com
`pression may be further adjusted by adjustment of the frame
`rate of the output data Stream.
`
`34 Claims, 12 Drawing Sheets
`
`CURRENT
`FRAMEATA
`
`
`
`TARGE
`BTRATE
`
`TS3
`MOON
`WS
`PREVIOUS
`
`MINIMUM
`CUAY
`TheSHODS
`
`NEAR
`SEEC
`MOVING
`"ETHRESHolds Ecks
`ONAL
`OOK-UP
`HIGHBANDS
`
`REVIOUS
`BrATE
`
`CoMPRESSE
`AA
`
`CURREN
`BTRAE
`
`FRAME
`ATE
`
`MNIMUM
`output
`FRAMERAE
`
`IPR2018-01413
`Sony EX1025 Page 1
`
`

`

`US. Patent
`
`Jul. 18, 2000
`
`Sheet 1 0f 12
`
`6,091,777
`
`w.OE
`
`
`
`I._.02m:OWEN
`
`
`
`m0“.OZEOOZM
`
`Qz<mIQ:
`
`
`
`<._.<D.._.Z<DO
`
`z<s_u_u_DI
`
`mooozw
`
`meo/E
`
`0:H3210”.
`
`
`
`57$meZOFOS.<._.<n_>><m>D.>
`
`
`
`
`
`
`
`mi.N._._
`
`m._._
`
`20.5.2
`
`zofiommo
`
`
`
`m2”.:m59::
`
`
`
`5528moz<m>-2:9:o,
`oz<m>-F>mo“.
`oz<m:-F82$
`
`ZO_._.OS_
`
`zO_._.Om._.mo
`
`m<._<om
`
`
`
`
`
`.95”—0w..m>w._Mao—2N
`
`Omo_>O>0
`
`ow._.z
`
`.2412.
`
`<._.<D>><m34>.NjD.m._._>
`
`>oz<3m0“.Dz<m
`
`woz<m:9:mzO_._.<N_._.z<DO
`.95m04m>m._mmOEF>mo“.Dz<m
`.DmmmDPm<O
`
`“.04m>m3_.
`
`024$-me
`
`+529
`
`|PR2018—O1413
`
`Sony EX1025 Page 2
`
`IPR2018-01413
`Sony EX1025 Page 2
`
`
`
`
`
`
`
`
`
`

`

`U.S. Patent
`
`Jul.18, 2000
`
`Sheet 2 of 12
`
`6,091,777
`
`Z "SDI
`
`
`
`IPR2018-01413
`Sony EX1025 Page 3
`
`

`

`U.S. Patent
`
`Jul.18, 2000
`
`Sheet 3 of 12
`
`6,091,777
`
`
`
`
`
`VULVO EWVH-I
`
`
`
`JUNEB HTIO
`
`STOIABHd
`
`IPR2018-01413
`Sony EX1025 Page 4
`
`

`

`U.S. Patent
`
`Jul.18, 2000
`
`Sheet 4 of 12
`
`6,091,777
`
`READ AV FILE
`(POSSIBLY SKIPPING FRAMES
`UNTIL THE CORRECT TIME
`INDEX)
`
`NO
`
`SUCCESSFU
`READ?
`
`YES
`
`SUB-BAND ENCODE ENTRE
`VIDEO FRAME
`
`TEST LOW-LOWLEVEL 3 DATA
`FOR MOTION (AGAINST
`REFERENCE)
`
`SET MOTION DETECTION
`THRESHOLDS FOR LEVELS
`2 AND 1
`
`
`
`
`
`AV FILE
`(YUV DATA)
`
`---
`
`DONE ENCODING
`
`THRESHOLD
`TABLE
`
`
`
`
`
`TEST LEVEL 3, LEVEL 2, AND - T
`LEVEL HIGHBANDSFOR-
`---
`MOTION (AGAINST REF)
`
`REFERENCE
`
`COMPRESS AND PACKETIZE
`(WRITE) MOVING BLOCKS
`
`-
`
`
`
`
`
`
`
`CALCULATE TIME OF NEXT
`FRAME (BASED ON
`BANDWIDTH OF CURRENT
`FRAME)
`
`FIG. 4
`
`IPR2018-01413
`Sony EX1025 Page 5
`
`

`

`U.S. Patent
`
`Jul.18, 2000
`
`Sheet 5 of 12
`
`6,091,777
`
`E||WITTOA TE
`
`H LQINA
`NEAIS) HLINA NI8
`
`
`
`w
`
`EZIS NIE
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`LIGZT99+ STEAET NOLLWZILNVnO
`
`
`
`:SE||}}\/CINTOR BAISTATONI-NON
`
`NOI_L\/ZILN\/[^?O HO
`
`
`
`
`
`H_LCHINA EZIS CHELLS
`
`IPR2018-01413
`Sony EX1025 Page 6
`
`

`

`U.S. Patent
`
`Jul.18, 2000
`
`Sheet 6 of 12
`
`6,091,777
`
`
`
`IPR2018-01413
`Sony EX1025 Page 7
`
`

`

`U.S. Patent
`
`Jul.18, 2000
`
`Sheet 7 of 12
`
`6,091,777
`
`
`
`WT & Inverse WT, Level:1, Filter. Cubic 6-tap
`
`"To
`
`50
`
`100
`
`150
`
`200
`
`250
`
`300
`
`350
`
`pixels
`
`Fig. 7
`
`IPR2018-01413
`Sony EX1025 Page 8
`
`

`

`U.S. Patent
`
`Jul.18, 2000
`
`Sheet 8 of 12
`
`6,091,777
`
`
`
`WT & Inverse WT, Level:2, Filter: Cubic 6-tap
`
`0.4
`
`-
`
`IPR2018-01413
`Sony EX1025 Page 9
`
`

`

`U.S. Patent
`
`Jul.18, 2000
`
`Sheet 9 of 12
`
`6,091,777
`
`
`
`WT & Inverse WT, Level:3, Filter: Cubic 6-tap
`
`error
`
`-0.2 O
`
`50
`
`100
`
`150
`
`200
`
`250
`
`300
`
`350
`
`pixels
`
`Fig. 9
`
`IPR2018-01413
`Sony EX1025 Page 10
`
`

`

`U.S. Patent
`
`Jul.18, 2000
`
`Sheet 10 of 12
`
`6,091,777
`
`COef-Diff>- Coef-Diff>- Coef-Diffe
`BWRatio
`COef-Diffa 2/3* Size 3 Size 3/24
`SizeL3/24 COef-Diff>
`EB
`=TARGET -2/3'Size And COef-
`And
`And
`2/3"
`BW/
`LL3
`Diff-- Size COef-Diffa
`COef-Diffa
`SizeLL3
`PREVIOUS
`LL3/24
`*SizeLL3/
`M2/3
`BW
`24
`SizeLL3
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`Fig.11
`
`IPR2018-01413
`Sony EX1025 Page 11
`
`

`

`US. Patent
`
`Jul. 18, 2000
`
`Sheet 11 0f 12
`
`6,091,777
`
`Dz<m10....>a
`
`
`
`
`
`<._.<n_zO_._.O_2omooomo
`
`Ema:N<55>oz<3.moz<mm:m>3026:
`
`woz<mNd>oz<Ni362a:
`
`w._m>m:m<20>mooomooz<mm3m
`
`
`MII>.NII>.wII>
`
`Eda“14>.23>
`
`m:.<n_mD
`
`mozmmmu—mm
`
`moz<m
`
`m<._<om.
`
`OmwNwZOF<NFZ<DGND
`IHOij
`
`mooowo
`
`woz<mIQ...>m
`
`
`
`<._.<n_zO_._.O_>_
`
`oz<m3-._
`
`oz<m>-._
`
`moz<m>-3”82%>oz<.2.>9820%k
`ZO_wmm>zOom.0.m0...>DZ<3.>
`mozmmmiwm
`
`woz<m
`
`
`
`O...._.Dn_._.DO
`
`
`
`mm>_mn_><._n_m_o
`
`NF.90
`
`
`
`Z.<._.<QZO_._.O_2>DZ<D.>
`
`|PR2018—O1413
`
`Sony EX1025 Page 12
`
`N._>m._3mj>
`.__.—.
`
`m._.<n_n5
`
`
`
`IPR2018-01413
`Sony EX1025 Page 12
`
`
`
`
`
`
`
`
`
`
`
`

`

`US. Patent
`
`Jul. 18, 2000
`
`Sheet 12 0f 12
`
`6,091,777
`
`z<>>111.moEmEZ
`2358552
`
`
`
`805mm?omkz
`
`
`
`<25-mmsstow-Sm;H89>mtwOazoo
`
`EmOu—bfin.On_
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`OFommmngxxmhmmo_>omn_
`
`
`
`2.xmozfimzzo_mmmmn_s_oo
`
`
`
`$5.9362500oz<89>09<z<H55.
`
`z_:3IonomzkNae>3>OFowkzEm>zoo
`onmt:ma:8EEzmou.
`
`2.32022303:
`
`
`.mgmiowofimMPZmz_m_>_oo-
`
`wnE9.._.<._.<_>_m0n_onmuv>D>
`
`
`
`m._.<s=om_oQz<mm._..__u_oz<mm3m-
`
`
`
`
`
`
`
`>Dz<3mon—Io<mszDZ<Emoz<m>Ozw30mmm~50“.O._.z_
`
`
`
`DMIQZE«50“.O._.n5mmOHm-
`
`awn—“SmOu=u_z.mus—(Eu—
`
`
`
`mamO._.m0<mmm._.z_-
`
`|PR2018—O1413
`
`Sony EX1025 Page 13
`
`IPR2018-01413
`Sony EX1025 Page 13
`
`
`

`

`1
`CONTINUOUSLY ADAPTIVE DIGITAL
`VIDEO COMPRESSION SYSTEMAND
`METHOD FOR AWEB STREAMER
`
`RELATED APPLICATIONS
`The present application is a continuation-in-part of U.S.
`patent application, Ser. No. 08/976,127 entitled NETWORK
`ADAPTIVE DIGITAL VIDEO COMPRESSION SYSTEM
`AND METHOD filed Sep. 18, 1997.
`
`TECHNICAL FIELD
`The present invention relates generally to an improvement
`in the field of digital television Systems and more
`particularly, but not by way of limitation, to a continuously
`adaptive digital Video compression System and method for a
`Web Streamer.
`
`15
`
`BACKGROUND OF THE INVENTION
`Multimedia communications is a rapidly developing field.
`Recent advances in both the computer industry and tele
`communications field has made digital Video and audio
`economically viable for Visual communications. This
`progreSS has been Supported by the availability of digital
`channels. Such as the narrow band Integrated Services Digital
`Network (ISDN) and its successor the broadband ISDN and
`its progress to local area networks (LANs), wide area
`networks (WANs), digital satellite and wireless networks,
`digital terrestrial broadcasting channels and corporate intra
`net networks and will lead to communication-based appli
`cations Such as Video phone, Video conference Systems,
`digital broadcast TV/HDTV, remote sensing, and Surveil
`lance. Digital Storage-based audio visual applications
`include Server-client based data bases, education, training,
`Video-on-demand type entertainment, advertising, and docu
`ment Storage and transfer.
`Specific examples of actual and anticipated applications
`include a web streamer which provides video data Streams
`from stored video clips at a server. Desirably, the video data
`would be delivered Such that there is no need to store the
`data at the client before displaying. Such an application
`would provide training on demand, advertising promotions,
`product demonstrations, product promotions on the internet,
`music videos and communications between executives of
`companies and other Such uses. Other applications would
`include Sending video data one way for telesales, product
`Support, tourism promotions, road/traffic conditions, Secu
`rity and Surveillance, Video e-mail and the like. Another
`exemplary application would be video conferencing Such as
`used for corporate work groups, medical diagnostics and
`conferencing, distance education and training, customer
`Support and professional conferences. Other contemplated
`applications of the invention would include further expan
`Sion into areas Such as live Video Streamer multicast and
`multipoint conferencing.
`A cost effective digital compression and decompression
`arrangement for Video and audio data Streams while deliv
`ering high quality Video and audio Streams is essential for
`the introduction and widespread use of Visual communica
`tions. To reduce transmission and Storage costs, improved bit
`rate compression Schemes are needed. Image, Video, and
`audio signals are amenable to compression due to consid
`erable Statistical redundancy in the Signals. Within a Single
`image or a Single video frame there exists Significant cor
`relation among neighboring Samples, giving rise to what is
`generally termed “spatial correlation.” Also in moving
`
`25
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`6,091,777
`
`2
`images, Such as full motion video, there is significant
`correlation among Samples in different Segments of time
`Such as Successive frames. This correlation is generally
`referred to as “temporal correlation.” However, a difficulty
`arises in the provision of a digital compression arrangement
`for a web streamer in that the bandwidth of the transmission
`channel is Subject to change during transmission and clients
`with varying receiver resources may join or leave the
`network as well during transmission.
`Accordingly, there is a present need in the art for an
`improved cost effective system and method that uses both
`Spatial and temporal correlation to remove the redundancy in
`the Video to achieve high compression in transmission and
`to maintain good to excellent image quality while continu
`ally adapting to change in the available bandwidth of the
`transmission channel and to the limitations of the receiving
`resources of the clients. The purpose of the present invention
`is then to provide a next-generation cost effective video
`compression/decompression (CODEC) System for Storage
`and distribution of high quality multimedia information on
`information networks using personal computers (PCS) that
`is continuously adaptive to changing conditions.
`In reviewing the prior art it is found that a known
`technique for taking advantage of the limited variation
`between frames of a television broadcast is known as
`motion-compensated image coding. In Such coding, the
`current frame is predicted from the previously encoded
`frame using motion estimation and compensation, and the
`difference between the actual current frame and the pre
`dicted current frame is coded. By coding only the difference,
`or residual, rather than the image frame itself, it is possible
`to improve image quality, for the residual tends to have
`lower amplitude than the image, and can thus be coded with
`greater accuracy. Motion estimation and compensation are
`discussed in Lim, J. S. Two-Dimensional Signal and Image
`Processing, Prentice Hall, pp. 497-507 (1990). A frame of
`estimated motion vectors is produced by comparing the
`current and previous frames. Typically, each motion vector
`is simply a pair of X and y values representing estimates of
`the horizontal and Vertical displacement of the image from
`one frame to the next at a particular location. The motion
`vectors are coded as Side information. In the decoder, the
`current image frame is computed by Summing the decoded
`residual with a motion-compensated version of the prior
`image frame. Motion compensation is typically performed
`on each pixel of the prior frame using bilinear interpolation
`between nearest motion vectors.
`A review of the patent literature has uncovered Some
`patents that are of interest. U. S. Pat. No. 5,218,435 dated
`Jun. 8, 1993 and issued to J. S. Lim et all for DIGITAL
`ADVANCED TELEVISION SYSTEMS teaches image
`quality improvement in high definition television using
`multi-Scale representation of motion compensated residuals.
`The bandwidth of the Subband filters vary with the fre
`quency band and the total number of coefficients in the
`multi-Scale represented frames is equal to the number of
`values in the residual. Image initialization in the receivers is
`achieved using original image leakage, but the leakage
`factor is varied for different frequency Subbands. To free up
`channel capacity at Scene changes, a frame-wide decision is
`made as to whether to motion compensate a particular frame.
`Chrominance resolution employs encoding all of the Sub
`bands of the chroma residuals, instead of just the low
`Subbands.
`U.S. Pat. No. 5,043,808 issued on Aug. 27, 1991 to S. C.
`Knauer et all for HIGH DEFINITION TELEVISION
`ARRANGEMENT EMPLOYING MOTION COMPEN
`
`IPR2018-01413
`Sony EX1025 Page 14
`
`

`

`6,091,777
`
`5
`
`25
`
`3
`SATED PREDICTION ERROR SIGNALS teaches a high
`definition television System where the television signal is
`encoded by developing motion vectors that describe the best
`motion estimate of the image to be transmitted, by devel
`oping motion estimation error Signals, by encoding these
`error Signals within the same bandwidth as occupied by
`standard NTSC signals and by transmitting the encoded
`error Signals during periods that correspond to the active
`scan intervals of the NTSC TV signal. The motion vectors
`themselves, together with Video and control Signals, are
`transmitted during the NTSC retrace period.
`U.S. Pat. No. 5,043,810 dated Aug. 17, 1991 issued to F.
`W. P. Vreeswijk et al for a METHOD AND APPARATUS
`FOR TEMPORALLY AND SPATIALLY PROCESSINGA
`VIDEO SIGNAL. This patent teaches a system having a
`15
`transmitting Section that has transmitting Section signal
`paths for at least three classes of motion, each with a
`preprocessing circuit which is provided with means for
`individually Sampling in accordance with Separate Sampling
`patterns So that each preprocessing circuit Supplies a Video
`Signal which is Suitable for a display with an optimum
`distribution of temporal and/or Spatial resolution for the
`asSociated class of motion. Dependent on the class of motion
`determined, one of the preprocessing circuits is coupled to
`a channel So that Video Signal Supplied to the channel is
`suitable for a display with an optimum distribution of
`temporal and/or Spatial resolution for the given class of
`motion. The associated receiver also has three receiving
`Section signal paths comprising a postprocessing circuit
`which decodes a received Video signal according to a
`method which Selects the correct postprocessing circuit in
`accordance with the class of motion So that a display with an
`increasing Spatial resolution can be achieved in the case of
`a decreasing motion.
`U.S. Pat. No. 4,943,855 dated Jul. 24, 1990 issued to H.
`35
`Bheda et all for PROGRESSIVE SUB-BAND IMAGE
`CODING SYSTEM. This patent teaches reducing image
`data redundancies through progressive Subband coding. The
`image is separated into a Selected plurality of Subbands, and
`the Sub-band with the largest non-redundant data content is
`chosen and used to predict the data in the other Subbands.
`Only prediction error information of the predicted Sub-bands
`is encoded and transmitted together with the encoded chosen
`Sub-band. An Overall performance error Signal can also be
`evaluated at the encoder end and used to further improve
`performance.
`U.S. Pat. No. 5,272.529 issued on Dec. 21, 1993 to J. E.
`Frederikesen for ADAPTIVE HIERARCHICAL SUB
`BAND VECTOR OUANTIZATION ENCODER. This
`patent teaches a System for data reduction digital Video
`Signals based on vector quantization of vectors formed from
`coefficients of a discrete cosine transform of pixel blockS.
`The coefficients are grouped into Subbands and both Scaler
`and vector quantization are used. Vector quantization is
`implemented either directly on the vectors or on vectors
`formed from inter-frame differences between the trans
`formed vectors. The vector quantization Searching routine is
`in accordance with the Voronoi regions resulting from an
`off-line codeword clustering method using a minimum dis
`tance criterion.
`60
`U.S. Pat. No. 4,817,182 dated Mar. 28, 1989 for TRUN
`CATED SUBBAND CODING IMAGES teaches analyzing
`image data in a number of iterated analysis procedures,
`using two-dimensional quadrature mirror filters to Separate
`low-pass Spatial filter response component and three differ
`ently oriented high-pass Spatial filter response components,
`which filter response components are decimated in both
`
`4
`dimensions. The high pass filter response components are
`coded "as is and the low-pass filter response component is
`coded “as is only in the last iteration. In the earlier analysis
`procedures the low-pass filter response component provides
`the input data for the Succeeding analysis procedure. No
`motion compensation is taught.
`U.S. Pat. No. 4,969,040 dated Nov. 6, 1990 and issued to
`H. Gharavi for APPARATUS AND METHOD FOR DIF
`FERENTIAL SUB-BANDING CODING OF VIDEO SIG
`NALS teaches an arrangement for achieving a high com
`pression of a video signal. The PEL-by PEL difference
`between an input Signal consisting of digital PEL values of
`a Scanned Video signal and a motion compensated interframe
`prediction signal is decomposed into Several narrow bands
`using Separable two-dimensional quadrature mirror filtering.
`Each Sub-band is quantized by a Symmetric uniform quan
`tizer with a center dead Zone. Entropy coderS code the
`quantized values by variable word-length coding the non
`Zero quantized values and transmitting that information with
`the corresponding run-length coded positional information.
`The outputs of the coders are combined into a constant rate
`bit Stream. AS required, the dead Zones and Step sizes of the
`quantizers are adjusted to force more Zero value quantized
`levels thereby reducing the amount of data.
`In Signal Processing: Image Communication 2 (1990) pg.
`81-94, K. S.Thyagarajan and Harry Sanchez discussed the
`ENCODING OF VIDEOCONFERENCING SIGNALS
`USING VDPCM. The techniques of motion detection, inter
`frame linear block prediction and Vector quantization were
`incorporated in an arrangement for encoding monochrome
`image Sequences for Video conferencing application. Data
`compression rate reduction is accomplished by identifying
`and processing only those regions that exhibit noticeable
`changes between Successive frames, by estimating the mag
`nitude of the change through linear block or vector predic
`tion and quantizing the residual vectors through a vector
`quantizer. The motion detector uses a modified block match
`ing algorithm to detect the moving blockS. Perceptually
`based edged detectors are used to design vector quantizer
`(VQ) codebooks for different classes of image blocks to
`achieve better visual quality. Encoding rates under 60 kbps
`are achieved with acceptable visual quality at nominal
`computational complexity.
`OBJECTS OF THE INVENTION
`Accordingly, one object of the present invention is to
`provide an improved digital Video compression arrangement
`having a continuously adaptive Video signal compression/
`decompression (CODEC) system for storage and distribu
`tion of good to excellent multimedia information on infor
`mation networks using personal computers (PCs).
`Yet another object of the present invention is to provide a
`digital Video compression arrangement wherein the encoder
`continuously adapts the compression to the a designated
`output data rate commensurate with the presently available
`bandwidth on the transmission channel and/or to the accept
`able data rate for the decoder at the receiver of a client.
`Still another object of the present invention is to provide
`continuously flexible CODEC wherein resolution, image
`size, and frame rate of the digital color Video data Stream can
`be adjusted at the encoder and/or the decoder at client user.
`Another object of the present invention is to provide a
`digital video compression arrangement which provides a
`plurality of layered compressed Video signal data Streams
`representative of a base level Visual image and additional
`layers of added Visual details.
`
`40
`
`45
`
`50
`
`55
`
`65
`
`IPR2018-01413
`Sony EX1025 Page 15
`
`

`

`6,091,777
`
`S
`Another object of the present invention is to provide a
`digital Video compression arrangement which provides lay
`ers of compressed digital Video data Steams that permit
`adaption of the arrangement to available information net
`work capacity.
`Another object of the present invention is to provide a
`digital video compression System that provides layers of
`multicast/multimedia information over an information net
`work to a number of customers that may individually choose
`the level of Visual image they wish to receive.
`Another object of the present invention is to provide a
`CODEC design which provides for the decoder portion to be
`providable in Software only to a recipient personal computer
`user whereby real time Video streams may be decoded
`without Storage of the received data Stream at the decoder of
`a receiver
`Yet another object of the present invention is to imple
`ment the method of the invention by providing the CODEC
`design in the form of Software that is provided to data
`processing means for providing the encoder and decoder
`Sections of Such CODEC.
`The foregoing has outlined Some of the more pertinent
`objects of the invention. These objects should be construed
`to be merely illustrative of some of the more prominent
`features and application of the intended invention. Many
`other beneficial results can be attained by applying the
`disclosed invention in a different manner or by modifying
`the invention within the Scope of the disclosure.
`Accordingly, other objects and a fuller understanding of the
`invention and the detailed description of the preferred
`embodiment in addition to the scope of the invention defined
`by the claims taken in conjunction with the accompanying
`drawings.
`
`15
`
`25
`
`6
`eters such as the presently available bandwidth of the
`transmission channel and by receiving feedback from
`receiving clients or by interrogating the transmission chan
`nel during the call setup mode. The CODEC of the system
`includes a number of unique features to achieve high com
`pression of color Video with good-to-excellent Video quality.
`The novel method of the present invention contemplates
`compressing input color Video data for moving imageS for a
`Web Stream to be transmitted over a transmission channel
`having a variable bandwidth to client users having receivers
`that may receive data input at different acceptable data rates.
`This novel method includes the Steps of compressing an
`input color Video data Stream for moving images on the basis
`of a target bandwidth of a data transmission channel to
`provide a compressed output video data Stream and trans
`mitting the compressed output color Video data Stream to at
`least one receiver over the transmission channel. The actual
`bandwidth of the transmission channel is then determined as
`the compressed output video data Stream was transmitted to
`at least one receiver. The compression of the input color
`Video data Stream is then continuously changed on a frame
`by frame basis in response to Such determination whereby
`the transmission of a Subsequent output color Video data
`stream is commensurate with the available bandwidth of the
`transmission channel.
`This novel method contemplates compressing the input
`color video data Stream by detecting the amount of motion
`that is Sensed in the input color Video data Stream and
`changing the rate of compression of Said input color Video
`data Stream to provide an output color Video data Stream
`having a data rate commensurate with the available band
`width of the transmission channel. This data rate by also be
`adjusted by adjusting the frame rate of the output com
`pressed data Stream So that the output data Stream becomes
`commensurate with the available bandwidth of the trans
`mission channel and to the receiver resources of the least
`capable client user of the Web Stream.
`More specifically, the novel method of the present inven
`tion contemplates continually determining the actual band
`width of the transmission channel as the compressed output
`Video data Stream is transmitted to a receiver and continually
`changing the compression of the Video data Stream to adapt
`the bit rate of the data stream to be commensurate with the
`available variable bandwidth of the transmission channel
`and/or to the receiver resources of the least capable receiver
`that receives a web stream of the layered Video and audio
`data. The compression of the input color Video data Stream
`is determined by detecting the amount of motion that is
`occurring between Successive frames. This detection of
`motion is accomplished by determining whether a plurality
`of variable thresholds have been exceeded. By continually
`monitoring the available bandwidth of the transmission
`channel and the receiver resources of the clients, the thresh
`olds are continually updated on a frame-by-frame basis to
`thereby adjust the bit rate of the output data stream to be
`commensurate with the immediately available bandwidth.
`The novel invention also provides that in addition to varying
`the thresholds to determine motion between frames by
`comparing pixel blocks of Successive frames, the frame rate
`of the output color Video data Stream may also be adjusted
`to further vary the data rate of Such data Stream to an
`optimum rate.
`The method of the present invention also preferably
`contemplates converting the input RGB color Video signal to
`a preferred YUV format and then spectrally decomposing
`the image frames of such YUV signal to a plurality of
`multifrequency Subbands to a plurality of levels. These
`
`35
`
`40
`
`45
`
`SUMMARY OF THE INVENTION
`Briefly Stated, the present invention contemplates a cost
`effective digital Video compression arrangement that is
`continuously adaptive to available bandwidth capacity on
`the transmission channel and to the receiver resources of
`client users for distribution of color video multimedia/
`multicast information over both packet Switched networks
`and point-to-point communication circuits that use personal
`computers (PCs). The digital video compression arrange
`ment uses a novel compression/decompression arrangement
`(CODEC) having encoder and decoder sections to provide
`continuously adaptive compression and decompression of
`multicast/multimedia information that includes color Video
`and accompanying audio over packet Switched networks and
`point-to-point communication circuits by means of com
`50
`pressed Video and audio Signal data Streams that are layered
`to provide a base layer representing a base visual level and
`progressive additional data Streams to provide higher levels
`of visual resolution. Preferably, the CODEC will be pro
`vided as Software that is supplied to PC's so that both the
`encoder and decoder may be provided by the processing of
`digital data streams by the PC's.
`Since the decoder may exist as Software that has been
`transmitted to a receiver PC, the received data Streams may
`be decoded in real time, without Storage, to provide a desired
`visual display. The decoder portion of the CODEC may be
`provided in the form of software that is transmitted to a
`recipient user prior to receipt of Video and audio Signal data
`Streams. The recipient user may also determine the level of
`Visual image desired and Select the layers of data Streams to
`be utilized. The encoder portion of the CODEC may receive
`control Signals relating to measurement of network param
`
`55
`
`60
`
`65
`
`IPR2018-01413
`Sony EX1025 Page 16
`
`

`

`7
`Subbands are then quantized to provide a predetermined
`number of Subbands for the Y, U and V components whereby
`a Selection of data may be made to provide a base layer Video
`data Stream and additional layers of Video data So that a user
`may determine the resolution and Visual quality to be
`displayed by Such user.
`The foregoing has outlined rather broadly the more per
`tinent and important features of the present invention in
`order that the detailed description of the invention that
`follows may be better understood so that the present con
`tributions to the art may be more fully appreciated. Addi
`tional features of the invention will be described hereinafter
`which form the subject of the claims of the invention. It
`should be appreciated by those skilled in the art that the
`conception and the disclosed specific embodiment may be
`readily utilized as a basis for modifying or designing other
`Structures and methods for carrying out the Same purposes of
`the present invention. It should also be realized by those
`skilled in the art that Such equivalent constructions and
`methods do not depart from the Spirit and Scope of the
`invention as Set forth in the appended claims.
`
`15
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`For a fuller understanding of the nature and objects of the
`invention, reference should be made to the following
`detailed description taken in conjunction with the accom
`pany drawings in which:
`FIG. 1 is a functional block diagram of the operation of
`the encoder of a CODEC operated accordance with a
`preferred embodiment of the present invention.
`FIG. 2 is a pictorial representation illustrating the con
`version of a color video signal from a RGB format into a
`YUV format and Subsequent spectral decomposition into
`three multifrequency Subbands.
`FIG. 3 is a representation of the motion detection algo
`rithm used in the encoder of FIG. 1.
`FIG. 4 is a motion detection flow diagram of the operation
`of the algorithm of FIG. 3.
`FIG. 5A is a representation of a table used in the quan
`tization Step of the present invention.
`FIG. 5B is a representation of a typical histogram of an
`image and Several parameters used for Scalar quantization.
`FIG. 6 is a representation of the wavelet 6-tap FIR filter
`used in the quantization Step of the invention.
`FIG. 7 illustrates the original and reconstructed intensity
`of a line chosen in an image, with the corresponding error
`shown at the bottom of each curve, having used a 6-tap FIR
`filter with a first level Subband coding.
`FIG. 8 illustrates the original and reconstructed intensity
`of a line chosen in the same image of FIG. 7, with the
`corresponding error shown at the bottom of each curve,
`having used a 6-tap FIR filter with a second level of Subband
`coding.
`FIG. 9 illustrates the original and reconstructed intensity
`of a line chosen in the same image of FIG. 7, with the
`corresponding error shown at the bottom of each curve,
`having used a 6-tap FIR filter with a third level of Subband
`coding.
`FIG. 10 is a table illustrating a Delta Bw Index table for
`determining changes to be made in thresho