(12) United States Patent
`Linzer et al.
`
`USOO622985OB1
`(10) Patent No.:
`US 6,229,850 B1
`(45) Date of Patent:
`May 8, 2001
`
`(54) MULTIPLE RESOLUTION VIDEO
`COMPRESSION
`
`(75) Inventors: Elliot N. Linzer, Bronx; Aaron Wells,
`New Rochelle, both of NY (US)
`(73) Assignee: C-Cube Semiconductor II, Inc.,
`Milpitas, CA (US)
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 0 days.
`
`(*) Notice:
`
`(21) Appl. No.: 09/421,178
`(22) Filed:
`Oct. 19, 1999
`Related U.S. Application Data
`(62) Division of application No. 08/999,763, filed on Jul. 22,
`1997, now Pat. No. 6,005,621.
`(51) Int. Cl." ................................................... H04N 7/12
`(52) U.S. Cl. .................................... 375/240.11; 375/240.1
`(58) Field of Search ..................................... 348/397-399,
`348/385, 387-389, 437-438, 426, 441,
`445-446, 448, 458–459, 408, 437.1, 438.1;
`382/240; 386/111; 375/240.1, 240.11, 240.19,
`240.25, 240
`
`(56)
`
`References Cited
`U.S. PATENT DOCUMENTS
`
`4,780,760 * 10/1988 Waldman et al. ................... 348/397
`5,048,111
`9/1991 Jones et al. .......................... 348/384
`5,173,773
`12/1992 Ueda et al. .......................... 348/407
`5,270,813
`12/1993 Puri et al. ..
`... 348/415
`5,469,212
`11/1995 Lee ....................................... 348/412
`(List continued on next page.)
`OTHER PUBLICATIONS
`“Switching Facilities in MPEG-2: Necessary But Not suf
`ficient”, S. Merrill Weiss, SMPTE Journal, Dec. 1995, pp.
`788-8O2.
`
`“Balancing act: Digital ad insertion over multipe Zones', R.
`James Kelso, Communications Technology, May 1995, pp.
`68-76.
`“Advantages of digital ad insertion”, Chris Brechin, Com
`munications Technology, May 1995, pp. 50-54.
`“Ad insertion system architecture”, Thomas A. Walsh, Com
`munications Technology, May 1995, pp. 56-66.
`
`(List continued on next page.)
`
`Primary Examiner Vu Le
`(74) Attorney, Agent, or Firm-Proskauer Rose, LLP
`(57)
`ABSTRACT
`An apparatus and method for compressing multiple resolu
`tion versions of a Video Signal are disclosed. A first resolu
`tion version of a Video signal is applied to an input of a first
`Video compressor and to an input of a Video Scaler. The first
`Video compressor encodes the first resolution version of the
`Video signal to generate a first compressed Video bit Stream.
`The Video Scaler generates a reduced resolution version of
`the video signal from the first resolution version. The
`reduced resolution version is Supplied to a Second Video
`compressor and to the first Video compressor. The first video
`compressor utilizes the reduced resolution version of the
`Video signal in performing a hierarchical motion estimation
`(ME) Search as part of the encoding process for the first
`resolution version. The Second Video compressor encodes
`the reduced resolution version to generate a Second com
`pressed bit Stream. The Second Video compressor receives
`motion vectors or other results of the hierarchical ME Search
`performed in the first Video compressor, and uses these
`results to facilitate the encoding of the reduced resolution
`version. The apparatus and method may be used in a
`non-linear Video editor, a Video Server or other Video pro
`cessing System. The Video Scaler and first and Second Video
`compressors may share memory, a transform unit and other
`processing hardware Such that System cost and complexity
`are reduced.
`
`24 Claims, 3 Drawing Sheets
`
`12
`
`COMPUTER
`
`69
`
`WIDEO
`DECOMPRESSOR
`
`
`
`
`
`----4----
`WIDEO SCALAR
`
`
`
`LOWOUALITY
`WIDEO
`COMPRESSOR
`
`NON-LINEAR
`STORAGE DEVICE
`FOR LOW
`OUAITY COPY
`OF THE WIDEO
`
`- HIGHOUALITY
`
`WIDEO
`COMPRESSOR
`
`NON-LINEAR
`SORAGE DENICE
`FOR HIGH
`CUALITY COPY
`OF THE WIDEO
`
`
`
`
`
`68
`
`SS'
`
`IPR2018-01413
`Sony EX1030 Page 1
`
`

`

`US 6,229,850 B1
`Page 2
`
`U.S. PATENT DOCUMENTS
`
`5,606,369
`5,737,023
`5,742,343
`5,742,892
`5,821,986
`5,832,234
`5,841,470
`5,845,088
`5,886,736
`5,973,739
`6,005,621
`6,023.299
`6,031,572
`
`: 2/1997
`4/1998
`4/1998
`4/1998
`10/1998
`11/1998
`11/1998
`12/1998
`3/1999
`10/1999
`12/1999
`2/2000
`2/2000
`
`... 348/385
`Keesman et al. ..
`Linzer ............
`... 348/416
`... 348/415
`Haskell et al. .
`Chaddha ....
`... 348/398
`Yuan et al. ............................. 348/17
`Iverson et al. ....................... 348/398
`Welsh ................................... 348/397
`Lewis .
`... 709/247
`Chen ...................................... 348/43
`Nilsson ................................ 348/397
`Linzer et al. ..
`... 348/398
`Katata et al. ......
`... 348/415
`Christopoulos ...................... 348/397
`
`
`
`OTHER PUBLICATIONS
`“Architectures for MPEG Compressed Bitstream Scaling”
`Huifang Sun, Wilson Kwok and Joel W. Zdepski, IEEE
`Transactions on Circuits and Systems for Video Technology,
`vol. 6., No. 2, Apr., 1996, pp. 191-199.
`Vincent et al., "Spatial prediction in Scalable video coding,
`Internation Broadcasting Convention, 1995, pp. 244-249.*
`Bayrakeri et al., “Temporally Scalable video coding using
`nonlinear deinterlacing, Proceedings Data Compression
`Conf., Mar. 1997, pp. 423.*
`Bayrakeri et al., “MPEG-2 nonlinear temporally scalable
`coding and hybrid quantization', IEEE ICASSP-97, Apr.
`1997, vol. 4, pp. 2629–2634.*
`* cited by examiner
`
`IPR2018-01413
`Sony EX1030 Page 2
`
`

`

`U.S. Patent
`
`May 8, 2001
`
`Sheet 1 of 3
`
`US 6,229,850 B1
`
`10
`N
`
`
`
`FIG. 1
`(PRIOR ART)
`
`COMPUTER
`
`
`
`14
`
`
`
`B
`1.
`
`19
`
`16
`
`VIDEO
`COMPRESSOR
`
`VIDEO
`DECOMPRESSOR
`
`NON-LINEAR STORAGE DEVICE
`
`FIG. 2
`(PRIOR ART)
`
`50
`
`12
`
`
`
`
`
`12
`
`WIDEO
`COMPRESSOR
`
`WIDEO
`DECOMPRESSOR
`
`NON-LINEAR STORAGE DEVICE
`
`
`
`16
`
`IPR2018-01413
`Sony EX1030 Page 3
`
`

`

`U.S. Patent
`
`May 8, 2001
`
`Sheet 2 of 3
`
`US 6,229,850 B1
`
`FIG. 3
`(PRIOR ART)
`
`2-N
`
`12
`
`
`
`WIDEO
`SOURCE
`
`22
`
`VIDEO
`SCALER
`
`4.
`
`WIDEO
`COMPRESSOR
`
`16
`
`STORAGE
`DEVICE
`
`FIG. 4
`
`34
`
`FIRST
`VIDEO
`COMPRESSOR
`
`DATA FOR MOTION
`ESTIMATION
`SEARCH
`
`
`
`STORAGE
`DEVICE
`
`40
`
`MOTION
`ESTIMATION
`RESULTS
`46
`
`SECOND
`VIDEO
`COMPRESSOR
`
`
`
`38
`
`30
`
`N
`
`
`
`32
`
`VIDEO
`SOURCE
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`WIDEO
`SCALER
`
`36
`
`IPR2018-01413
`Sony EX1030 Page 4
`
`

`

`U.S. Patent
`
`May 8, 2001
`
`Sheet 3 of 3
`
`US 6,229,850 B1
`
`
`
`FIG. 5
`
`FULL RES
`
`FULL RESOLUTION SEARCH -o- USE FOR FIRST COMPRESSOR
`
`1/4 RES
`
`QUARTER r SEARCH -- USE FOR SECOND COMPRESSOR
`
`1116
`RES
`
`BASE SEARCH
`
`SO
`\
`
`- - - --- - -
`
`-
`
`-
`
`VIDEO SCALAR
`
`LOWOUALITY
`VIDEO
`COMPRESSOR
`
`NON-LINEAR
`STORAGE DEVICE
`FOR LOW
`OUALITY COPY
`OF THE WIDEO
`
`
`
`
`
`
`
`
`
`
`
`S8
`
`SS'
`
`FIG. 6
`
`62
`
`
`
`HIGH QUALITY
`VIDEO
`COMPRESSOR
`
`NON-LINEAR
`STORAGE DEVICE
`FOR HIGH
`QUALITY COPY
`OF THE WIDEO
`
`
`
`72
`
`
`
`
`
`WIDEO
`DECOMPRESSOR
`
`69
`
`IPR2018-01413
`Sony EX1030 Page 5
`
`

`

`US 6,229,850 B1
`
`1
`MULTIPLE RESOLUTION VIDEO
`COMPRESSION
`
`2
`operator. Each edit decision on the edit decision list may, for
`example, indicate the kind of edit transition/operation to use
`(cut, fade, wipe dissolve, over-dub, etc.), which Video clips
`are operated on by the edit operation and the edit points (e.g.,
`Specific pictures or audio frames at which the editing opera
`tion begins and ends) within those clips. The operator can
`preview a “facsimile” of the final event as formed by
`performing the edit operations indicated on the edit decision
`list on the compressed Video signals Stored in the non-linear
`Storage device 16. After the operator has approved the edit
`decisions, the computer 18 accesses the original video
`Signals Stored on the linear Storage devices 12 and performs
`edit operations on these video signals according to the edit
`decision list.
`Typically, an off-line video editor 50 produces a final
`edited video event with better video and audio quality than
`an on-line video editor 10. This is because the video event
`produced by on-line Video editing contains compression
`artifacts produced by the lossy or imperfect compression and
`decompression of the Video. Nevertheless, an on-line video
`editor 10 is less expensive than an off-line video editor 50
`because the on-line Video editor 10 needs only a Single linear
`Storage device 12. Furthermore, the linear Storage device 12
`used in the on-line video editor 10 need not be as Sophisti
`cated as the linear Storage device 12 used in the off-line
`video editor 50 since it does not have to advance to multiple
`Specific edit points (as would be necessary in the off-line
`video editor in forming the final edited video event from the
`edit list). The on-line video editor 10 is easier to use because
`the physical media (video tapes) on which the original video
`information is Stored need not be resident in the linear
`storage device 12 to create the final edited video event This
`is because once the Video signals are transferred to the
`non-linear Storage device 16, the Video signals need not be
`retrieved from the linear Storage device again.
`In order to reduce the level of compression artifacts
`present in the final edited video event in the on-line video
`editor 10, the on-line video editor 10 may typically use high
`compression data rates, high resolution and/or inter-frame
`coding. Video Signals from Source 12 in System 10 are
`compressed before being Stored in the Storage device 16 in
`order to Save disk space and because the throughput of
`inexpensive disk drives is generally insufficient to Support
`uncompressed high resolution video. To insure Sufficiently
`high quality edited Video, a non-lineir editor typically uses
`high resolutions such as 720x480 pixels at 30 frames/sec
`and high compressed bit rates such as 18-50 Mbits/sec.
`The use of high resolutions, high bit rates and/or inter
`frame encoding in the compressor 14 can increase the
`difficulty of processing functions Such as accessing Stored
`compressed Video streams, playing back more than one bit
`Stream at the same time, and decoding/decompressing with
`trick modes Such as fast forward and fast reverse. A com
`pression System which utilizes compressed Video bit Streams
`having low resolution, low bit rate and/or only intra-frame
`encoding does not Suffer these drawbacks. It is therefore
`desirable in many applications to provide a System in which
`multiple resolution and/or multiple bit rate versions of a
`given Video signal can be compressed and Stored. The high
`resolutions, high bit rates and inter-frame encoding can then
`be utilized when necessary, while the advantages of low
`resolution, low bit rates and intra-frame encoding can also
`be provided in appropriate applications.
`Video Servers represent another application in which
`Storage of multiple versions of compressed Video bit streams
`is desirable. Such video servers are used to deliver video bit
`Streams to end users over data communication networks. For
`
`15
`
`35
`
`40
`
`25
`
`This application is a divisional of Ser. No. 08/999,763,
`filed Jul 22, 1997, now U.S. Pat. No. 6,005,621.
`FIELD OF THE INVENTION
`The present invention relates generally to digital Video
`compression and more particularly to Video compression
`methods and apparatus for producing different resolution
`compressed versions of a given video signal.
`BACKGROUND OF THE INVENTION
`FIG. 1 shows an “on-line” “non-linear video editing
`system 10. One or more linear video sources 12, such as
`Video tape recorders (VTRs), each output an analog or
`digital Video signal to a compressor 14. The compressor 14
`compresses each Video Signal and Stores the digital com
`pressed Video signals in a non-linear Storage device 16, Such
`as a magnetic disk. Unlike the linear Video signal Source 12,
`which accesses Video signals Sequentially, the non-linear
`Storage device 16 Supports random access of Video signals.
`Compression of the video signals in the compressor 14 may
`be achieved according to the MPEG, MPEG-2, Dolby TM
`AC-3, etc., Standards for compressing video and associated
`audio. See ISO/IEC IS 13818-1, 2, 3: Generic Coding of
`Moving Pictures and Associated Audio: Systems, Video and
`Audio. The compressed Video signals are then edited using
`a computer 18 which accesses the compressed Video signals
`for display and editing. In displaying a compressed video
`Signal, the non-linear Storage device 16 outputs a com
`pressed video signal to a Video decompressor 19, which
`decompresses the Video signal for display on a display
`monitor. After editing, the edited compressed Video signal
`may be retrieved from the non-linear Storage device 16 and
`decompressed by decompreSSor 19 for Storage on a linear
`Storage device (Such as a VTR) or transmitted.
`The non-linear video editing system 10 enables the opera
`tor to access the Video in a random fashion instead of a
`Sequential fashion. This facilitates the editing of the Video
`and provides for rather precise editing. The editing is said to
`be “on-line' in that the final edited video event is incremen
`tally constructed as the edits are made. For example, if the
`operator chooses to splice a first Video clip to a Second Video
`clip, then the compressed form of the first Video clip is
`Spliced to the compressed form of the Second Video clip, at
`the time the operator makes Such a choice to produce an
`edited compressed Video signal. The edited video event is
`formed by decompressing the edited compressed video
`Signal.
`FIG. 2 shows a an “off-line” “non-linear video editing
`system 50. Multiple linear video sources 12 are used to input
`Video signals to a Video compressor 14. The Video compres
`Sor 14 compresses each Video signal and Stores the com
`55
`pressed Video Signal on a non-linear Storage device 16. AS
`before, a computer 18 can access the compressed Video
`Signals on the non-linear Storage device 16 for viewing and
`editing. The compressed Video signals are decompressed in
`decompressor 19 prior to viewing on a display monitor (not
`shown).
`Unlike the video editing system 10, the video editing
`system 50 does not incrementally form the final edited video
`event as each edit decision is made. Rather, the formation of
`the final edited video event is deferred until the editing of the
`Video signals is complete. The computer 18 generates a "edit
`decision list’ as each editing operation is performed by the
`
`45
`
`50
`
`60
`
`65
`
`IPR2018-01413
`Sony EX1030 Page 6
`
`

`

`US 6,229,850 B1
`
`15
`
`35
`
`40
`
`3
`example, a World Wide Web server may be used to deliver
`video bit streams to different end users over different types
`of lines, including plain old telephone service (POTS) lines,
`integrated services digital network (ISDN) lines, T1 lines
`and the like. A version of a given compressed bit Stream that
`may be suitable for a POTS user would be considered poor
`quality by a T1 user, and a bit stream suitable for a T1 user
`would be at too high a bit rate for a POTS user. It is therefore
`desirable for the video server to store a given video bit
`stream at multiple bit rates. The “optimal' resolution for a
`compressed video bit stream is the one that yields the best
`Subjective Video quality after decompression. This optimal
`resolution generally decreases with bit rate, Such that it is
`desirable for the video server to compress the different bit
`rate Streams at different resolutions.
`FIG. 3 shows a conventional video compression system
`20 which generates and Stores multiple resolution versions
`of a given bit stream. The system 20 includes a video source
`12, video compressor 14 and storage device 16 which
`operate in the manner previously described in to conjunction
`with FIGS. 1 and 2. The system 20 also includes a video
`Scaler 22 which receives a given Video signal from the
`Source 12 and generates a number of reduced resolution
`versions thereof These reduced resolution versions are Sup
`plied to the Video compressor 14, which generates a com
`25
`pressed Video bit Stream at an appropriate bit rate for each
`of the reduced resolution versions of the Video signal, and
`Stores the resulting compressed Streams on the Storage
`device 16.
`The system 20 Suffers from a number of significant
`problems. For example, each of the reduced resolution
`versions of a given video signal are separately and indepen
`dently compressed. The information used to encode the
`Video signal at one resolution is generally not used to
`facilitate the encoding proceSS for the other reduced reso
`lution versions. In addition, the Video Scaler 22 and Video
`compressor 14 are not configured in a manner which enables
`them to share at least a portion of a common memory. The
`System 20 therefore requires relatively large amounts of
`memory. These and other drawbacks of the system 20
`unduly increase its cost and complexity, and limit its use
`fulness in non-linear editor applications, Video Server appli
`cations and numerous other important Video processing
`applications.
`AS is apparent from the above, there is a need for an
`improved multiple resolution video compression System in
`which hardware and processing resources can be shared to
`thereby significantly reduce the cost and complexity of the
`System.
`
`45
`
`50
`
`SUMMARY OF THE INVENTION
`The present invention provides an apparatus and method
`for compressing multiple resolution versions of a Video
`Signal. Each of the compressed version of the Video signal
`may be independently decompressed, i.e., without resort to
`information contained in another version of the Video signal.
`Illustratively, Such multiple compressed versions of the
`Video signal may be used in a non-linear Video editing
`System. One compressed version of the Video signal may be
`used to determine how to edit the Video to produce a final
`edited Video event. A Second compressed version of the
`Video signal may be used to create the final edited Video
`event. For example, a lower quality compressed version of
`the Video signal may be viewed and used as a basis to form
`an edit decision list. After the edit decision list is complete,
`the actual editing operations may be performed on a higher
`
`55
`
`60
`
`65
`
`4
`quality compressed version of the video signal to produce a
`final edited high quality Video event.
`The invention permits the sharing of memory between a
`Video Scaler and Video compressor in a Video compression
`System, and the Sharing of motion estimation, discrete cosine
`transform and other compression hardware between mul
`tiple Video compressors in the Video compression System.
`The invention thereby considerably reduces the cost and
`complexity of a Video compression System. The invention
`also significantly improves the performance and capabilities
`of non-linear Video editors, Video Servers and other video
`processing applications which utilize multiple resolution
`compression Systems.
`An exemplary embodiment of the invention includes first
`and Second Video compressors and a Video Scaler. A first
`resolution version of a Video Signal is applied to an input of
`a first video compressor and to an input of a Video Scaler.
`The first resolution version may be a CCIR601 image
`Sequence or other high resolution unscaled Video image
`Sequence. The first video compressor encodes the first
`resolution version of the Video Signal to generate a first
`compressed video bit stream in accordance with MPEG-2 or
`another Suitable video encoding Standard. The Video Scaler
`generates a reduced resolution version of the Video signal
`from the first resolution version. The reduced resolution
`version may be a QQCIF image Sequence, a QCIF image
`Sequence or a CIF image Sequence, the images of which are
`/64 size, /16 size and 4 size, respectively, relative to a full
`resolution CCIR601 image. Other types of images and
`image resolutions may also be used, and the Video Scaler
`may generate multiple reduced resolution versions of the
`Video signal. The reduced resolution version or versions are
`Supplied to a Second Video compressor and to the first video
`compressor. The first Video compressor utilizes the reduced
`resolution versions of the Video Signal in performing a
`hierarchical motion estimation (ME) search as part of the
`encoding process for the first resolution version. The Second
`Video compressor encodes the reduced resolution version to
`generate a Second compressed bit stream. The Second Video
`compressor receives motion vectors or other results of the
`hierarchical ME Search performed in the first video
`compressor, and uses these results to facilitate the encoding
`of hie reduced resolution version. Although Some sharing of
`encoding information may occur during the compression,
`each Video signal is independently decompressible without
`resort to information contained in the other Video signal.
`In accordance with other aspects of the invention, the
`Video Scaler and first Video compressor may share a common
`memory for temporary Storage of reduced resolution images,
`and the first and Second compressors may be configured to
`utilize common motion estimation and discrete cosine trans
`form hardware. For example, a Single discrete cosine trans
`form unit may be shared in a time multiplexed manner
`between the first and Second Video compressors.
`According to another embodiment, a nonlinear editor is
`provided with first and second video compressors. The first
`Video compressor is for compressing a first version of an
`input video Signal to produce a low quality compressed
`Video signal. The Second Video compressor is for compress
`ing a Second version of the Same input video signal to
`produce a high quality compressed Video Signal, having a
`higher quality than the low quality compressed video signal.
`Each of the low quality and high quality compressed video
`Signals produced by the first and Second Video compressors,
`respectively, are independently decompressible.
`Illustratively, the low and high quality compressed Video
`Signals can be achieved a number of ways Such as providing
`
`IPR2018-01413
`Sony EX1030 Page 7
`
`

`

`US 6,229,850 B1
`
`S
`different quality first and Second versions of the input video
`Signal for input to the first and Second Video compressors.
`For example, the first version may be one or more of the
`following: Spatially Scaled, temporally Scaled, non
`interlaced (progressive Scanned or de-interlaced), or bit rate
`reduced, whereas the Second version may be one or more of
`the following: full Spatial resolution, full temporal
`resolution, interlaced, or full bit-rate. Alternatively, or in
`addition, the first and Second Video compressors can com
`preSS the first and Second versions of the input video signal
`differently. For example, the first compressor nay use one or
`more of the following: a less efficient compression Standard,
`Such as MPEG-1, intra-coding only, frame prediction only,
`or low bit rate coding whereas the Second compressor may
`use one or more of the following: a highly efficient com
`15
`pression Standard, Such as MPEG-2, inter and intra-coding,
`field and frame prediction, or high bit rate coding.
`Illustratively, the non-linear Video editor includes a
`decoder which can Simultaneously present more low quality
`compressed Video signals than high quality compressed
`Video signals in real time. Alternatively, or in addition, the
`non-linear Video editor includes a computer that can perform
`more edits/effects (e.g., cuts, wipes, fades, dissolves, trick
`modes, etc.) on the low quality compressed Video signals
`than on the high quality compressed Video signals.
`These and other features and advantages of the present
`invention will become more apparent from the accompany
`ing drawings and the following detailed description.
`
`25
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`FIG. 1 shows a conventional off-line video editing sys
`tem.
`FIG.2 shows a conventional on-line Video editing System.
`FIG. 3 shows a conventional Video compression System.
`FIG. 4 shows an exemplary video compression System
`Suitable for providing multiple resolution compression in
`accordance with the present invention.
`FIG. 5 shows a motion estimation Strategy according to an
`embodiment of the present invention.
`FIG. 6 shows a non-linear Video editor according to an
`embodiment of the present invention.
`DETAILED DESCRIPTION OF THE
`INVENTION
`The present invention is illustrated herein using an exem
`plary implementation of a multiple resolution video com
`pression System particularly well-Suited for use in applica
`tions Such as “non-linear video editors and video servers. It
`should be understood that the techniques of the present
`invention are more broadly applicable to any application in
`which it is desirable to generate multiple resolution versions
`of a given video signal. The invention is Suitable for use with
`MPEG-2 encoding as well as other video encoding stan
`dards. The term “reduced resolution' as used herein is
`intended to include any version of a given video signal
`which has a lower resolution than an unscaled version of the
`given Signal Supplied from a video Source. The reduced
`resolution may be a reduced luminance resolution or a
`reduced chrominance resolution or both.
`FIG. 4 shows an exemplary embodiment of a multiple
`resolution video compression System 30 in accordance with
`the present invention. A video Source 32 Supplies a video
`Signal to a first video compressor 34 and a video Scaler 36.
`The Video Source 32 may be a Single video Signal Source
`Such as a video camera or Video cassette recorder or a
`
`6
`multiple-signal Source made up of a combination of various
`Sources. The Video Scaler 36 Scales the Video Signal to at
`least one reduced resolution and Sends the resulting reduced
`resolution version or versions to a Second Video compressor
`38. The first video compressor 34 compresses the full
`resolution unscaled version of the Video signal from Source
`32 and Stores the resulting compressed Video bit Stream in
`the storage device 40. The storage device 40 may be an
`optical or magnetic disk drive or disk array, an electronic
`memory or other Suitable compressed Video Storage device.
`The Second Video compressor 38 compresses at least one
`reduced resolution version of the Video signal to generate at
`least one corresponding reduced resolution compressed
`Video bit Stream. The compressed Stream or Streams from the
`Second Video compressor 38 are also Stored in the Storage
`device 40. The compressed bit streams generated by the first
`and second video compressors 34, 38 may differ in bit rate,
`group of picture (GOP) structure and/or other parameters.
`The video scaler 36 and the video compressors 34, 38 may
`be combined into a single device to facilitate Sharing of
`hardware resources as will be described in greater detail
`below.
`At least one reduced resolution version of the Video signal
`is supplied from the video scaler 36 via line 44 to the first
`video compressor 34. The reduced resolution version is used
`in a hierarchical motion estimation (ME) search performed
`on frames of the unscaled full resolution video signal in the
`first video compressor 34. The scaler 36 and first video
`compressor 34 can therefore share at least a portion of a
`common memory. The first video compressor 34 performs
`the hierarchical ME Search and generates ME search results
`which are supplied via line 46 to the second video com
`pressor 38. The second video compressor 38 uses the ME
`Search results from the first video compressor 34 in encoding
`the reduced resolution version Supplied from the Video
`scaler 36. For example, the second video compressor 38 may
`utilize an average of motion vectorS Supplied from the first
`Video compressor 34 or the actual motion vectors generated
`by a particular hierarchical ME Search Stage. One way in
`which the second video compressor 38 can use the results of
`one stage of the motion estimation is shown in FIG. 5. For
`this example, it is assumed that the Second Video compressor
`38 compresses video at 4 the resolution of the first video
`compressor 34. The full resolution video signal is decimated
`to /4 and /16 resolutions. Hierarchical motion estimation is
`then used in compressing the Video signals. In Such a motion
`estimation, a Zero centered Search is conducted on the /16
`resolution image. The results of the first Search are used as
`a basis to perform a Second Search on the 4 resolution image
`(e.g., in the local vicinity of one or more candidate motion
`vectors identified in the first search). The results of the
`Second Search may then be used as a basis for performing a
`Search on the full resolution image. The results of the Second
`Search may also be, used directly to generate motion vectors
`in the second video compressor 38. The results of the third
`Search on the full resolution image are used to generate
`motion vectors for the first video compressor 34. The first
`and Second compressors 34, 38 may share common Video
`compression hardware. For example, a single discrete cosine
`transform (DCT) unit could be time multiplexed between the
`first and second compressors 34, 38. Other types of video
`compression hardware which could be time multiplexed or
`otherwise shared between first and Second compressors 34,
`38 include the inverse DCT, quantizer, inverse quantizer,
`motion compensator, run-level encoder, variable length
`encoder and pre-processing filters.
`An exemplary hierarchical ME technique Suitable for use
`in the system 30 of FIG. 4 is described in U.S. patent
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`IPR2018-01413
`Sony EX1030 Page 8
`
`

`

`US 6,229,850 B1
`
`5
`
`7
`application Ser. No. 08/652,231 of John Ju filed on May 23,
`1996 and entitled “Video Encoding With Multi-Stage Pro
`jection Motion Estimation,” which is assigned to the present
`assignee and incorporated by reference herein. In accor
`dance with that technique, the first video compressor 34 of
`system 30 may receive a video signal from source 32 in the
`form of a sequence of CCIR601 video images. For each
`CCIR601 image, the video scaler 36 generates first, second
`and third reduced resolution images which may be a QQCIF
`image, a QCIF image and a CIF image, respectively, which
`are /64 size, /16 size and 4 size, respectively, relative to the
`CCIR601 image. These reduced resolution images are Sup
`plied via line 44 to the first video compressor 34 as previ
`ously described. Alternatively, the video scaler 36 may
`generate only a Subset of the reduced resolution versions of
`the Signal from Source 12, with the remaining reduced
`15
`resolution versions generated within the first video com
`pressor 34.
`The Video compressor 34 performs a first Stage motion
`vector search on the /64 size QQCIF image using a (0,0)
`motion vector Starting point and a first Search range Suitable
`for detecting global motion within the original CCIR601
`image. Global motion refers generally to motion involving
`many objects in a given image, Such as the motion produced
`by a panning or Zooming operation. The first Video com
`preSSor 34 then performs a Second Stage motion vector
`25
`Search on the /16 size QCIF image using the (0,0) starting
`point and a Second Smaller Search range Suitable for detect
`ing local motion within the original CCIR601 image. Local
`motion refers generally to motion involving Small and/or
`irregular movements of a particular object or objects in a
`given image. The first Video compressor 34 then performs a
`third Stage motion vector Search on the 4 size CIF image
`using a starting point based on Scaled versions of the motion
`vectors identified in the first and Second Stage Searches, and
`a Search range Smaller than the first and Second Search
`ranges. The first Video compressor 34 finally performs a
`fourth Stage Search on the original CCIR601 image or a
`reconstruction thereof using the motion vectors identified in
`the third Search Stage. A motion compensation type and a
`suitable set of motion vectors for the original CCIR601
`image is determined based on the results of the fourth Stage
`Search. This motion compensation type and Set of motion
`vectors may be Supplied to the Second Video compressor 38
`via line 46 and used in the encoding of one or more reduced
`resolution imageS. It should be noted that the present inven
`tion may utilize any of a number of other types of hierar
`chical ME Search techniques.
`The video compression system 30 of FIG. 4 is well-suited
`for use in Video Server applications. For example, a Video
`Server may utilize the first video compressor 34 to generlte
`a high resolution, high bit rate, inter-frame encoded version
`of a given Video signal for delivery over relatively high
`bandwidth ISDN or T1 connections, while the second video
`compressor 38 is used to generate a low resolution, low bit
`rate, intra-frame encoded version of the given video signal
`for