`Guetz et al.
`
`US006091777A
`Patent Number:
`11
`(45) Date of Patent:
`
`6,091,777
`Jul.18, 2000
`
`54 CONTINUOUSLY ADAPTIVE DIGITAL
`VIDEO COMPRESSION SYSTEMAND
`METHOD FOR AWEB STREAMER
`
`5,680,129 10/1997 Weinberger et al. ..................... 341/65
`5,745,758 4/1998 Shaw et al. ............................. 395/672
`5,903,673 5/1999 Wang et al. ............................ 348/416
`
`75 Inventors: William N. Guetz; Keshoolal Bakhru;
`Fredric W. Ritchey, all of San Diego,
`Calif.
`73 Assignee: Cubic Video Technologies, Inc., San
`Diego, Calif.
`
`21 Appl. No.: 09/084,820
`22 Filed:
`May 26, 1998
`9
`Related U.S. Application Data
`
`63 Continuation-in-part of application No. 08/976,127, Sep. 18,
`1997.
`H04B 1/66; HO4N 7/12
`51 Int. Cl."
`52 U.S. Cl. .......................... 375/240; 348/384; 348/398;
`348/413; 382/242
`58 Field of Search ..................................... 348/384, 385,
`348/415, 413, 402, 390, 10, 14, 15, 17,
`12, 398, 416; 382/232, 244, 239, 246, 250,
`242: 395/200.77, 672; 345/336,337; 341/65;
`370/354, 468; 375/240
`
`56)
`
`References Cited
`U.S. PATENT DOCUMENTS
`
`4,903,317 2/1990 Nishihara et al. ...................... 382/244
`5,115,426 5/1992 Spanke .................
`... 370/468
`5,546,477 8/1996 Knowles et al. ........................ 382/242
`5,557,320 9/1996 Krebs ........................................ 348/12
`5,611,038 3/1997 Shaw et al. ...
`... 348/390
`5,617,142 4/1997 Hamilton ................................ 348/384
`
`Primary Examiner Richard Lee
`Assistant Examiner-Gims Philippe
`Attorney, Agent, or Firm-Frank D. Gilliam
`57
`ABSTRACT
`A cost effective continuously adaptive digital Video System
`and method for compressing color Video data for moving
`images. The method describes capturing an analog video
`frame and digitizing the image into a preferred Source input
`format for compression using a combination of unique lossy
`and lossleSS digital compression techniques including Sub
`band coding, wavelet transforms, motion detection, run
`length coding and variable length coding. The system
`includes encoder and decoder (CODEC) sections for com
`pression and decompression of visual images to provide
`high compression with good to excellent Video quality. The
`compressed video data provides a base Video layer and
`additional layers of video data that are multiplexed with
`compressed digital audio to provide a data Stream that can be
`packetized for distribution over inter or intranets, including
`wireless networks over local or wide areas. The (CODEC)
`System continuously adjusts the compression of the digital
`images frame by frame in response to comparing the avail
`able bandwidth on the data channel to the available band
`width on the channel for the previous frame to provide an
`output data Stream commensurate with the available band
`width of the network transmission channel and with the
`receiver resource capabilities of the client users. The com
`pression may be further adjusted by adjustment of the frame
`rate of the output data Stream.
`
`34 Claims, 12 Drawing Sheets
`
`CURRENT
`FRAMEATA
`
`
`
`TARGE
`BTRATE
`
`TS3
`MOON
`WS
`PREVIOUS
`
`MINIMUM
`CUAY
`TheSHODS
`
`NEAR
`SEEC
`MOVING
`"ETHRESHolds Ecks
`ONAL
`OOK-UP
`HIGHBANDS
`
`REVIOUS
`BrATE
`
`CoMPRESSE
`AA
`
`CURREN
`BTRAE
`
`FRAME
`ATE
`
`MNIMUM
`output
`FRAMERAE
`
`IPR2018-01413
`Sony EX1025 Page 1
`
`
`
`U.S. Patent
`
`Jul. 18, 2000
`
`Sheet 1 of 12
`
`6,091,777
`
`NOLLOW
`
`NOILOALIG
`
`JIVYLidLADYVL
`
`
`
`TOYULNODSGNV@A-OLHOIH6
`aNvan-1}SANVd
`GNVdA-1AYOSs
`
`OSLN
`
`LNdNl
`
`NOILOALAC
`NOILLOW
`SOGNV4HOIH6NOILVZILNYWNO
`
`-€NSdO1SA51SHOW|AYOdGNVE
`AQGNVNY¥O4GNVd
`
`4OTSAN31|‘adaSAYNLdVO
`GNVa-aNsS
`
`YVIVOS
`
`
`
`-GNS40STSAA1SNOW¢
`
`OAdIASAD
`
`WLvaMWeauneTl,ev
`
`
`
`WVAYLSNOILOWViVaMVeANn’oA
`
`|Sls
`
`
`
`HLONATONAZ
`
`YOdONIGOONS
`
`NVWasSNH
`
`LAMOVd
`
`
`
`Viva“LNWNO
`
`ONVdHOIH
`
`AQOONA
`
`O/lLYWYOS
`
`LANOL
`
`
`
`C11.211
`
`e711
`
`IPR2018-01413
`Sony EX1025 Page 2
`
`IPR2018-01413
`Sony EX1025 Page 2
`
`
`
`
`
`
`
`
`
`
`
`U.S. Patent
`
`Jul.18, 2000
`
`Sheet 2 of 12
`
`6,091,777
`
`Z "SDI
`
`
`
`IPR2018-01413
`Sony EX1025 Page 3
`
`
`
`U.S. Patent
`
`Jul.18, 2000
`
`Sheet 3 of 12
`
`6,091,777
`
`
`
`
`
`VULVO EWVH-I
`
`
`
`JUNEB HTIO
`
`STOIABHd
`
`IPR2018-01413
`Sony EX1025 Page 4
`
`
`
`U.S. Patent
`
`Jul.18, 2000
`
`Sheet 4 of 12
`
`6,091,777
`
`READ AV FILE
`(POSSIBLY SKIPPING FRAMES
`UNTIL THE CORRECT TIME
`INDEX)
`
`NO
`
`SUCCESSFU
`READ?
`
`YES
`
`SUB-BAND ENCODE ENTRE
`VIDEO FRAME
`
`TEST LOW-LOWLEVEL 3 DATA
`FOR MOTION (AGAINST
`REFERENCE)
`
`SET MOTION DETECTION
`THRESHOLDS FOR LEVELS
`2 AND 1
`
`
`
`
`
`AV FILE
`(YUV DATA)
`
`---
`
`DONE ENCODING
`
`THRESHOLD
`TABLE
`
`
`
`
`
`TEST LEVEL 3, LEVEL 2, AND - T
`LEVEL HIGHBANDSFOR-
`---
`MOTION (AGAINST REF)
`
`REFERENCE
`
`COMPRESS AND PACKETIZE
`(WRITE) MOVING BLOCKS
`
`-
`
`
`
`
`
`
`
`CALCULATE TIME OF NEXT
`FRAME (BASED ON
`BANDWIDTH OF CURRENT
`FRAME)
`
`FIG. 4
`
`IPR2018-01413
`Sony EX1025 Page 5
`
`
`
`U.S. Patent
`
`Jul.18, 2000
`
`Sheet 5 of 12
`
`6,091,777
`
`E||WITTOA TE
`
`H LQINA
`NEAIS) HLINA NI8
`
`
`
`w
`
`EZIS NIE
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`LIGZT99+ STEAET NOLLWZILNVnO
`
`
`
`:SE||}}\/CINTOR BAISTATONI-NON
`
`NOI_L\/ZILN\/[^?O HO
`
`
`
`
`
`H_LCHINA EZIS CHELLS
`
`IPR2018-01413
`Sony EX1025 Page 6
`
`
`
`U.S. Patent
`
`Jul.18, 2000
`
`Sheet 6 of 12
`
`6,091,777
`
`
`
`IPR2018-01413
`Sony EX1025 Page 7
`
`
`
`U.S. Patent
`
`Jul.18, 2000
`
`Sheet 7 of 12
`
`6,091,777
`
`
`
`WT & Inverse WT, Level:1, Filter. Cubic 6-tap
`
`"To
`
`50
`
`100
`
`150
`
`200
`
`250
`
`300
`
`350
`
`pixels
`
`Fig. 7
`
`IPR2018-01413
`Sony EX1025 Page 8
`
`
`
`U.S. Patent
`
`Jul.18, 2000
`
`Sheet 8 of 12
`
`6,091,777
`
`
`
`WT & Inverse WT, Level:2, Filter: Cubic 6-tap
`
`0.4
`
`-
`
`IPR2018-01413
`Sony EX1025 Page 9
`
`
`
`U.S. Patent
`
`Jul.18, 2000
`
`Sheet 9 of 12
`
`6,091,777
`
`
`
`WT & Inverse WT, Level:3, Filter: Cubic 6-tap
`
`error
`
`-0.2 O
`
`50
`
`100
`
`150
`
`200
`
`250
`
`300
`
`350
`
`pixels
`
`Fig. 9
`
`IPR2018-01413
`Sony EX1025 Page 10
`
`
`
`U.S. Patent
`
`Jul.18, 2000
`
`Sheet 10 of 12
`
`6,091,777
`
`COef-Diff>- Coef-Diff>- Coef-Diffe
`BWRatio
`COef-Diffa 2/3* Size 3 Size 3/24
`SizeL3/24 COef-Diff>
`EB
`=TARGET -2/3'Size And COef-
`And
`And
`2/3"
`BW/
`LL3
`Diff-- Size COef-Diffa
`COef-Diffa
`SizeLL3
`PREVIOUS
`LL3/24
`*SizeLL3/
`M2/3
`BW
`24
`SizeLL3
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`Fig.11
`
`IPR2018-01413
`Sony EX1025 Page 11
`
`
`
`U.S. Patent
`
`Jul. 18, 2000
`
`Sheet 11 of 12
`
`6,091,777
`
`
`STSA31
`
`€VLVOA300030GNVasNns
`
`
`
`HH,ZHH,'HHH,HLONATO8SZ
`
`
`
`
`
`ViVdNOILLOWda3qd003d
`
`GNV4@HOIHA6
`
`CHT,ZH!LAT)
`
`aivddn
`
`YVIVOS
`
`8NOILVZILNVNOAG
`
`JONSYSas44ayAONISN4q093d
`
` STAAI1ZWLWdAGNWNSaNva
`SANVvaaNs
`
`SONVdHOIHA6
`
`
`
`VLVdNOILOW
`
`anv211nonisnaivddn
`sanva2TAAA
`
`Z11,.©TY:E11
`
`
`
`‘SGNVEAGNVN'AZ2iGaLvadn
`
`JONSYSAaY
`
`SGNVd
`
`SGNVdA-970
`LAGNVN'A
`
`
`
`GNvVdn-7OLINdLNO
`
`NOISHSANO9D
`
`GNV¢dA-1g'9'YYAAINGAV1dSI0
`
`ZL“bis
`
`
`
`NIV.LVdNOILLOW
`
`IPR2018-01413
`Sony EX1025 Page 12
`
`IPR2018-01413
`Sony EX1025 Page 12
`
`
`
`
`
`
`
`
`
`
`U.S. Patent
`
`Jul. 18, 2000
`
`Sheet 12 of 12
`
`6,091,777
`
`WYO4LV1dOd
`
`
`
`
`
`
`
`INWIYYOMLAN
`NVMa:3OV4NSLNI
`
`
`
`C3a0V1NS.LNIOS.LN
`
`
`
`OAGIAS3LISOdWOD
`
`
`
`
`
`Viva-FYVMLAIOS-LNdNI
`
`
`
`OLQAYYSISNVYLSadiAOwd
`
`
`
`NIMYOMLANNOISSSYdWOD
`
`
`
`
`
`SLAWOVdONIGOOGNVOJCIADONNYSZLISIG-
`NINddHONOYMHLZ:Z:¢ANNAOLOSLNLYSANOD
`OdSHLSd0€LVLVWHOS
`
`
`
`0:2:7ANAe,‘bisoidywioaONYwad
`
`
`
`
`
`‘$Q1sl4dGAOVTYALNIANISINOD-
`
`SddStLVLVWHOS
`
`
`
`ALVWIOSGGNVY3L1dGNVaSNs-
`
`
`
`(A)SGNVdAONSNDAYAYNOSOLNI
`
`
`
`
`
`AGNVNYO4HOWSANOONV
`
`
`
`QSHSINIAYNOAOLdNAYOLS-
`
`Yd34ingOldNISAWVYSs
`
`sndOLSOVAYALNI-
`
`IPR2018-01413
`Sony EX1025 Page 13
`
`
`
`
`
`IPR2018-01413
`Sony EX1025 Page 13
`
`
`
`
`1
`CONTINUOUSLY ADAPTIVE DIGITAL
`VIDEO COMPRESSION SYSTEMAND
`METHOD FOR AWEB STREAMER
`
`RELATED APPLICATIONS
`The present application is a continuation-in-part of U.S.
`patent application, Ser. No. 08/976,127 entitled NETWORK
`ADAPTIVE DIGITAL VIDEO COMPRESSION SYSTEM
`AND METHOD filed Sep. 18, 1997.
`
`TECHNICAL FIELD
`The present invention relates generally to an improvement
`in the field of digital television Systems and more
`particularly, but not by way of limitation, to a continuously
`adaptive digital Video compression System and method for a
`Web Streamer.
`
`15
`
`BACKGROUND OF THE INVENTION
`Multimedia communications is a rapidly developing field.
`Recent advances in both the computer industry and tele
`communications field has made digital Video and audio
`economically viable for Visual communications. This
`progreSS has been Supported by the availability of digital
`channels. Such as the narrow band Integrated Services Digital
`Network (ISDN) and its successor the broadband ISDN and
`its progress to local area networks (LANs), wide area
`networks (WANs), digital satellite and wireless networks,
`digital terrestrial broadcasting channels and corporate intra
`net networks and will lead to communication-based appli
`cations Such as Video phone, Video conference Systems,
`digital broadcast TV/HDTV, remote sensing, and Surveil
`lance. Digital Storage-based audio visual applications
`include Server-client based data bases, education, training,
`Video-on-demand type entertainment, advertising, and docu
`ment Storage and transfer.
`Specific examples of actual and anticipated applications
`include a web streamer which provides video data Streams
`from stored video clips at a server. Desirably, the video data
`would be delivered Such that there is no need to store the
`data at the client before displaying. Such an application
`would provide training on demand, advertising promotions,
`product demonstrations, product promotions on the internet,
`music videos and communications between executives of
`companies and other Such uses. Other applications would
`include Sending video data one way for telesales, product
`Support, tourism promotions, road/traffic conditions, Secu
`rity and Surveillance, Video e-mail and the like. Another
`exemplary application would be video conferencing Such as
`used for corporate work groups, medical diagnostics and
`conferencing, distance education and training, customer
`Support and professional conferences. Other contemplated
`applications of the invention would include further expan
`Sion into areas Such as live Video Streamer multicast and
`multipoint conferencing.
`A cost effective digital compression and decompression
`arrangement for Video and audio data Streams while deliv
`ering high quality Video and audio Streams is essential for
`the introduction and widespread use of Visual communica
`tions. To reduce transmission and Storage costs, improved bit
`rate compression Schemes are needed. Image, Video, and
`audio signals are amenable to compression due to consid
`erable Statistical redundancy in the Signals. Within a Single
`image or a Single video frame there exists Significant cor
`relation among neighboring Samples, giving rise to what is
`generally termed “spatial correlation.” Also in moving
`
`25
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`6,091,777
`
`2
`images, Such as full motion video, there is significant
`correlation among Samples in different Segments of time
`Such as Successive frames. This correlation is generally
`referred to as “temporal correlation.” However, a difficulty
`arises in the provision of a digital compression arrangement
`for a web streamer in that the bandwidth of the transmission
`channel is Subject to change during transmission and clients
`with varying receiver resources may join or leave the
`network as well during transmission.
`Accordingly, there is a present need in the art for an
`improved cost effective system and method that uses both
`Spatial and temporal correlation to remove the redundancy in
`the Video to achieve high compression in transmission and
`to maintain good to excellent image quality while continu
`ally adapting to change in the available bandwidth of the
`transmission channel and to the limitations of the receiving
`resources of the clients. The purpose of the present invention
`is then to provide a next-generation cost effective video
`compression/decompression (CODEC) System for Storage
`and distribution of high quality multimedia information on
`information networks using personal computers (PCS) that
`is continuously adaptive to changing conditions.
`In reviewing the prior art it is found that a known
`technique for taking advantage of the limited variation
`between frames of a television broadcast is known as
`motion-compensated image coding. In Such coding, the
`current frame is predicted from the previously encoded
`frame using motion estimation and compensation, and the
`difference between the actual current frame and the pre
`dicted current frame is coded. By coding only the difference,
`or residual, rather than the image frame itself, it is possible
`to improve image quality, for the residual tends to have
`lower amplitude than the image, and can thus be coded with
`greater accuracy. Motion estimation and compensation are
`discussed in Lim, J. S. Two-Dimensional Signal and Image
`Processing, Prentice Hall, pp. 497-507 (1990). A frame of
`estimated motion vectors is produced by comparing the
`current and previous frames. Typically, each motion vector
`is simply a pair of X and y values representing estimates of
`the horizontal and Vertical displacement of the image from
`one frame to the next at a particular location. The motion
`vectors are coded as Side information. In the decoder, the
`current image frame is computed by Summing the decoded
`residual with a motion-compensated version of the prior
`image frame. Motion compensation is typically performed
`on each pixel of the prior frame using bilinear interpolation
`between nearest motion vectors.
`A review of the patent literature has uncovered Some
`patents that are of interest. U. S. Pat. No. 5,218,435 dated
`Jun. 8, 1993 and issued to J. S. Lim et all for DIGITAL
`ADVANCED TELEVISION SYSTEMS teaches image
`quality improvement in high definition television using
`multi-Scale representation of motion compensated residuals.
`The bandwidth of the Subband filters vary with the fre
`quency band and the total number of coefficients in the
`multi-Scale represented frames is equal to the number of
`values in the residual. Image initialization in the receivers is
`achieved using original image leakage, but the leakage
`factor is varied for different frequency Subbands. To free up
`channel capacity at Scene changes, a frame-wide decision is
`made as to whether to motion compensate a particular frame.
`Chrominance resolution employs encoding all of the Sub
`bands of the chroma residuals, instead of just the low
`Subbands.
`U.S. Pat. No. 5,043,808 issued on Aug. 27, 1991 to S. C.
`Knauer et all for HIGH DEFINITION TELEVISION
`ARRANGEMENT EMPLOYING MOTION COMPEN
`
`IPR2018-01413
`Sony EX1025 Page 14
`
`
`
`6,091,777
`
`5
`
`25
`
`3
`SATED PREDICTION ERROR SIGNALS teaches a high
`definition television System where the television signal is
`encoded by developing motion vectors that describe the best
`motion estimate of the image to be transmitted, by devel
`oping motion estimation error Signals, by encoding these
`error Signals within the same bandwidth as occupied by
`standard NTSC signals and by transmitting the encoded
`error Signals during periods that correspond to the active
`scan intervals of the NTSC TV signal. The motion vectors
`themselves, together with Video and control Signals, are
`transmitted during the NTSC retrace period.
`U.S. Pat. No. 5,043,810 dated Aug. 17, 1991 issued to F.
`W. P. Vreeswijk et al for a METHOD AND APPARATUS
`FOR TEMPORALLY AND SPATIALLY PROCESSINGA
`VIDEO SIGNAL. This patent teaches a system having a
`15
`transmitting Section that has transmitting Section signal
`paths for at least three classes of motion, each with a
`preprocessing circuit which is provided with means for
`individually Sampling in accordance with Separate Sampling
`patterns So that each preprocessing circuit Supplies a Video
`Signal which is Suitable for a display with an optimum
`distribution of temporal and/or Spatial resolution for the
`asSociated class of motion. Dependent on the class of motion
`determined, one of the preprocessing circuits is coupled to
`a channel So that Video Signal Supplied to the channel is
`suitable for a display with an optimum distribution of
`temporal and/or Spatial resolution for the given class of
`motion. The associated receiver also has three receiving
`Section signal paths comprising a postprocessing circuit
`which decodes a received Video signal according to a
`method which Selects the correct postprocessing circuit in
`accordance with the class of motion So that a display with an
`increasing Spatial resolution can be achieved in the case of
`a decreasing motion.
`U.S. Pat. No. 4,943,855 dated Jul. 24, 1990 issued to H.
`35
`Bheda et all for PROGRESSIVE SUB-BAND IMAGE
`CODING SYSTEM. This patent teaches reducing image
`data redundancies through progressive Subband coding. The
`image is separated into a Selected plurality of Subbands, and
`the Sub-band with the largest non-redundant data content is
`chosen and used to predict the data in the other Subbands.
`Only prediction error information of the predicted Sub-bands
`is encoded and transmitted together with the encoded chosen
`Sub-band. An Overall performance error Signal can also be
`evaluated at the encoder end and used to further improve
`performance.
`U.S. Pat. No. 5,272.529 issued on Dec. 21, 1993 to J. E.
`Frederikesen for ADAPTIVE HIERARCHICAL SUB
`BAND VECTOR OUANTIZATION ENCODER. This
`patent teaches a System for data reduction digital Video
`Signals based on vector quantization of vectors formed from
`coefficients of a discrete cosine transform of pixel blockS.
`The coefficients are grouped into Subbands and both Scaler
`and vector quantization are used. Vector quantization is
`implemented either directly on the vectors or on vectors
`formed from inter-frame differences between the trans
`formed vectors. The vector quantization Searching routine is
`in accordance with the Voronoi regions resulting from an
`off-line codeword clustering method using a minimum dis
`tance criterion.
`60
`U.S. Pat. No. 4,817,182 dated Mar. 28, 1989 for TRUN
`CATED SUBBAND CODING IMAGES teaches analyzing
`image data in a number of iterated analysis procedures,
`using two-dimensional quadrature mirror filters to Separate
`low-pass Spatial filter response component and three differ
`ently oriented high-pass Spatial filter response components,
`which filter response components are decimated in both
`
`4
`dimensions. The high pass filter response components are
`coded "as is and the low-pass filter response component is
`coded “as is only in the last iteration. In the earlier analysis
`procedures the low-pass filter response component provides
`the input data for the Succeeding analysis procedure. No
`motion compensation is taught.
`U.S. Pat. No. 4,969,040 dated Nov. 6, 1990 and issued to
`H. Gharavi for APPARATUS AND METHOD FOR DIF
`FERENTIAL SUB-BANDING CODING OF VIDEO SIG
`NALS teaches an arrangement for achieving a high com
`pression of a video signal. The PEL-by PEL difference
`between an input Signal consisting of digital PEL values of
`a Scanned Video signal and a motion compensated interframe
`prediction signal is decomposed into Several narrow bands
`using Separable two-dimensional quadrature mirror filtering.
`Each Sub-band is quantized by a Symmetric uniform quan
`tizer with a center dead Zone. Entropy coderS code the
`quantized values by variable word-length coding the non
`Zero quantized values and transmitting that information with
`the corresponding run-length coded positional information.
`The outputs of the coders are combined into a constant rate
`bit Stream. AS required, the dead Zones and Step sizes of the
`quantizers are adjusted to force more Zero value quantized
`levels thereby reducing the amount of data.
`In Signal Processing: Image Communication 2 (1990) pg.
`81-94, K. S.Thyagarajan and Harry Sanchez discussed the
`ENCODING OF VIDEOCONFERENCING SIGNALS
`USING VDPCM. The techniques of motion detection, inter
`frame linear block prediction and Vector quantization were
`incorporated in an arrangement for encoding monochrome
`image Sequences for Video conferencing application. Data
`compression rate reduction is accomplished by identifying
`and processing only those regions that exhibit noticeable
`changes between Successive frames, by estimating the mag
`nitude of the change through linear block or vector predic
`tion and quantizing the residual vectors through a vector
`quantizer. The motion detector uses a modified block match
`ing algorithm to detect the moving blockS. Perceptually
`based edged detectors are used to design vector quantizer
`(VQ) codebooks for different classes of image blocks to
`achieve better visual quality. Encoding rates under 60 kbps
`are achieved with acceptable visual quality at nominal
`computational complexity.
`OBJECTS OF THE INVENTION
`Accordingly, one object of the present invention is to
`provide an improved digital Video compression arrangement
`having a continuously adaptive Video signal compression/
`decompression (CODEC) system for storage and distribu
`tion of good to excellent multimedia information on infor
`mation networks using personal computers (PCs).
`Yet another object of the present invention is to provide a
`digital Video compression arrangement wherein the encoder
`continuously adapts the compression to the a designated
`output data rate commensurate with the presently available
`bandwidth on the transmission channel and/or to the accept
`able data rate for the decoder at the receiver of a client.
`Still another object of the present invention is to provide
`continuously flexible CODEC wherein resolution, image
`size, and frame rate of the digital color Video data Stream can
`be adjusted at the encoder and/or the decoder at client user.
`Another object of the present invention is to provide a
`digital video compression arrangement which provides a
`plurality of layered compressed Video signal data Streams
`representative of a base level Visual image and additional
`layers of added Visual details.
`
`40
`
`45
`
`50
`
`55
`
`65
`
`IPR2018-01413
`Sony EX1025 Page 15
`
`
`
`6,091,777
`
`S
`Another object of the present invention is to provide a
`digital Video compression arrangement which provides lay
`ers of compressed digital Video data Steams that permit
`adaption of the arrangement to available information net
`work capacity.
`Another object of the present invention is to provide a
`digital video compression System that provides layers of
`multicast/multimedia information over an information net
`work to a number of customers that may individually choose
`the level of Visual image they wish to receive.
`Another object of the present invention is to provide a
`CODEC design which provides for the decoder portion to be
`providable in Software only to a recipient personal computer
`user whereby real time Video streams may be decoded
`without Storage of the received data Stream at the decoder of
`a receiver
`Yet another object of the present invention is to imple
`ment the method of the invention by providing the CODEC
`design in the form of Software that is provided to data
`processing means for providing the encoder and decoder
`Sections of Such CODEC.
`The foregoing has outlined Some of the more pertinent
`objects of the invention. These objects should be construed
`to be merely illustrative of some of the more prominent
`features and application of the intended invention. Many
`other beneficial results can be attained by applying the
`disclosed invention in a different manner or by modifying
`the invention within the Scope of the disclosure.
`Accordingly, other objects and a fuller understanding of the
`invention and the detailed description of the preferred
`embodiment in addition to the scope of the invention defined
`by the claims taken in conjunction with the accompanying
`drawings.
`
`15
`
`25
`
`6
`eters such as the presently available bandwidth of the
`transmission channel and by receiving feedback from
`receiving clients or by interrogating the transmission chan
`nel during the call setup mode. The CODEC of the system
`includes a number of unique features to achieve high com
`pression of color Video with good-to-excellent Video quality.
`The novel method of the present invention contemplates
`compressing input color Video data for moving imageS for a
`Web Stream to be transmitted over a transmission channel
`having a variable bandwidth to client users having receivers
`that may receive data input at different acceptable data rates.
`This novel method includes the Steps of compressing an
`input color Video data Stream for moving images on the basis
`of a target bandwidth of a data transmission channel to
`provide a compressed output video data Stream and trans
`mitting the compressed output color Video data Stream to at
`least one receiver over the transmission channel. The actual
`bandwidth of the transmission channel is then determined as
`the compressed output video data Stream was transmitted to
`at least one receiver. The compression of the input color
`Video data Stream is then continuously changed on a frame
`by frame basis in response to Such determination whereby
`the transmission of a Subsequent output color Video data
`stream is commensurate with the available bandwidth of the
`transmission channel.
`This novel method contemplates compressing the input
`color video data Stream by detecting the amount of motion
`that is Sensed in the input color Video data Stream and
`changing the rate of compression of Said input color Video
`data Stream to provide an output color Video data Stream
`having a data rate commensurate with the available band
`width of the transmission channel. This data rate by also be
`adjusted by adjusting the frame rate of the output com
`pressed data Stream So that the output data Stream becomes
`commensurate with the available bandwidth of the trans
`mission channel and to the receiver resources of the least
`capable client user of the Web Stream.
`More specifically, the novel method of the present inven
`tion contemplates continually determining the actual band
`width of the transmission channel as the compressed output
`Video data Stream is transmitted to a receiver and continually
`changing the compression of the Video data Stream to adapt
`the bit rate of the data stream to be commensurate with the
`available variable bandwidth of the transmission channel
`and/or to the receiver resources of the least capable receiver
`that receives a web stream of the layered Video and audio
`data. The compression of the input color Video data Stream
`is determined by detecting the amount of motion that is
`occurring between Successive frames. This detection of
`motion is accomplished by determining whether a plurality
`of variable thresholds have been exceeded. By continually
`monitoring the available bandwidth of the transmission
`channel and the receiver resources of the clients, the thresh
`olds are continually updated on a frame-by-frame basis to
`thereby adjust the bit rate of the output data stream to be
`commensurate with the immediately available bandwidth.
`The novel invention also provides that in addition to varying
`the thresholds to determine motion between frames by
`comparing pixel blocks of Successive frames, the frame rate
`of the output color Video data Stream may also be adjusted
`to further vary the data rate of Such data Stream to an
`optimum rate.
`The method of the present invention also preferably
`contemplates converting the input RGB color Video signal to
`a preferred YUV format and then spectrally decomposing
`the image frames of such YUV signal to a plurality of
`multifrequency Subbands to a plurality of levels. These
`
`35
`
`40
`
`45
`
`SUMMARY OF THE INVENTION
`Briefly Stated, the present invention contemplates a cost
`effective digital Video compression arrangement that is
`continuously adaptive to available bandwidth capacity on
`the transmission channel and to the receiver resources of
`client users for distribution of color video multimedia/
`multicast information over both packet Switched networks
`and point-to-point communication circuits that use personal
`computers (PCs). The digital video compression arrange
`ment uses a novel compression/decompression arrangement
`(CODEC) having encoder and decoder sections to provide
`continuously adaptive compression and decompression of
`multicast/multimedia information that includes color Video
`and accompanying audio over packet Switched networks and
`point-to-point communication circuits by means of com
`50
`pressed Video and audio Signal data Streams that are layered
`to provide a base layer representing a base visual level and
`progressive additional data Streams to provide higher levels
`of visual resolution. Preferably, the CODEC will be pro
`vided as Software that is supplied to PC's so that both the
`encoder and decoder may be provided by the processing of
`digital data streams by the PC's.
`Since the decoder may exist as Software that has been
`transmitted to a receiver PC, the received data Streams may
`be decoded in real time, without Storage, to provide a desired
`visual display. The decoder portion of the CODEC may be
`provided in the form of software that is transmitted to a
`recipient user prior to receipt of Video and audio Signal data
`Streams. The recipient user may also determine the level of
`Visual image desired and Select the layers of data Streams to
`be utilized. The encoder portion of the CODEC may receive
`control Signals relating to measurement of network param
`
`55
`
`60
`
`65
`
`IPR2018-01413
`Sony EX1025 Page 16
`
`
`
`7
`Subbands are then quantized to provide a predetermined
`number of Subbands for the Y, U and V components whereby
`a Selection of data may be made to provide a base layer Video
`data Stream and additional layers of Video data So that a user
`may determine the resolution and Visual quality to be
`displayed by Such user.
`The foregoing has outlined rather broadly the more per
`tinent and important features of the present invention in
`order that the detailed description of the invention that
`follows may be better understood so that the present con
`tributions to the art may be more fully appreciated. Addi
`tional features of the invention will be described hereinafter
`which form the subject of the claims of the invention. It
`should be appreciated by those skilled in the art that the
`conception and the disclosed specific embodiment may be
`readily utilized as a basis for modifying or designing other
`Structures and methods for carrying out the Same purposes of
`the present invention. It should also be realized by those
`skilled in the art that Such equivalent constructions and
`methods do not depart from the Spirit and Scope of the
`invention as Set forth in the appended claims.
`
`15
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`For a fuller understanding of the nature and objects of the
`invention, reference should be made to the following
`detailed description taken in conjunction with the accom
`pany drawings in which:
`FIG. 1 is a functional block diagram of the operation of
`the encoder of a CODEC operated accordance with a
`preferred embodiment of the present invention.
`FIG. 2 is a pictorial representation illustrating the con
`version of a color video signal from a RGB format into a
`YUV format and Subsequent spectral decomposition into
`three multifrequency Subbands.
`FIG. 3 is a representation of the motion detection algo
`rithm used in the encoder of FIG. 1.
`FIG. 4 is a motion detection flow diagram of the operation
`of the algorithm of FIG. 3.
`FIG. 5A is a representation of a table used in the quan
`tization Step of the present invention.
`FIG. 5B is a representation of a typical histogram of an
`image and Several parameters used for Scalar quantization.
`FIG. 6 is a representation of the wavelet 6-tap FIR filter
`used in the quantization Step of the invention.
`FIG. 7 illustrates the original and reconstructed intensity
`of a line chosen in an image, with the corresponding error
`shown at the bottom of each curve, having used a 6-tap FIR
`filter with a first level Subband coding.
`FIG. 8 illustrates the original and reconstructed intensity
`of a line chosen in the same image of FIG. 7, with the
`corresponding error shown at the bottom of each curve,
`having used a 6-tap FIR filter with a second level of Subband
`coding.
`FIG. 9 illustrates the original and reconstructed intensity
`of a line chosen in the same image of FIG. 7, with the
`corresponding error shown at the bottom of each curve,
`having used a 6-tap FIR filter with a third level of Subband
`coding.
`FIG. 10 is a table illustrating a Delta Bw Index table for
`determining changes to be made in thresholds for detecting
`motion be