(12) United States Patent
`Washino
`
`USOO6370198B1
`(10) Patent No.:
`US 6,370,198 B1
`(45) Date of Patent:
`Apr. 9, 2002
`
`(54) WIDE-BAND MULTI-FORMAT
`AUDIO/VIDEO PRODUCTION SYSTEM
`WITH FRAME-RATE CONVERSION
`
`(76)
`
`(21)
`(22)
`
`(63)
`(60)
`
`(51)
`
`(52)
`
`(58)
`
`(56)
`
`Inventor: Kinya Washino, 80 Hamilton St.,
`Dumont, NJ (US) 07624
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 0 days.
`
`Notice:
`
`Appl. No.: 09/305,953
`Filed:
`May 6, 1999
`Related U.S. Application Data
`
`Continuation-in-part of application No. 08/834,912, filed on
`Apr. 7, 1997, now Pat. No. 5,999.220.
`Provisional application No. 60/084.522, filed on May 7,
`1998.
`Int. Cl............................. H04N 7/12: H04N 5/46;
`HO4N 7/01
`U.S. Cl. .................. 375/240.26; 348/441; 348/445;
`348/555; 348/556; 348/722
`Field of Search ................................. 348/441, 445,
`348/448, 426, 432, 454, 558, 568, 555-556,
`722,911, 575; 358/527, 524; 375/240.26,
`240.29
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`5,327,235 A 7/1994 Richards ..................... 348/441
`5,444,491. A * 8/1995 Lim ..........
`... 348/441
`5,532,749 A * 7/1996 Hong ...........
`... 348/449
`5,537,157 A
`7/1996 Washino et al.
`... 348/722
`5,608,464 A 3/1997 Woodham .....
`... 348/578
`5,724,101 A * 3/1998 Haskin ......
`... 348/441
`5,754.248 A * 5/1998 Faroudja .......
`... 348/474
`5,812.204 A
`9/1998 Baker et al. ................ 348/453
`
`5,838,381. A 11/1998 Kasahara et al. ........... 348/458
`5.999.220 A * 12/1999 Washino ..................... 348/441
`* cited by examiner
`Primary Examiner Vu Le
`(74) Attorney, Agent, or Firm-Gifford, Krass, Groh,
`Sprinkle, Anderson & Citkowski, PC
`(57)
`ABSTRACT
`A multi-format digital Video production System enables a
`user to process an input video program to produce an output
`version of the program in a final format which may have a
`different frame rate, pixel dimensions, or both. An internal
`production format of 24 fps is preferably chosen to provide
`the greatest compatibility with existing and planned formats
`associated with HDTV standard 4:3 or widescreen 16:9
`high-definition television, and film. Images are re-sized
`horizontally and vertically by pixel interpolation, thereby
`producing larger or Smaller image dimensions So as to fill the
`particular needs of individual applications. Frame rates are
`adapted by inter-frame interpolation or by traditional
`schemes, including “3:2 pull-down” for 24-to-30 fps con
`versions. Simple speed-up (for 24-to-25 conversions) or
`slow-down (for 25-to-24 conversions) for playback, or by
`manipulating the frame rate itself using a program Storage
`facility with asynchronous reading and writing capabilities.
`The step of converting the signal to a HDTV format is
`preferably performed using a modified upconversion pro
`cess for wideband signals (utilizing a higher sampling clock
`frequency) and a resizing to HDTV format frame dimen
`Sions in pixels. The present invention thus encourages
`production at relatively low pixel dimensions to make use of
`lower-cost general-purpose technology and to maintain high
`Signal-to-noise ratio, and then Subsequently expands the
`resultant image into a So-called up-converted program. This
`is in contrast to alternative approaches, which recommend
`operating at HDTV-type resolution, then down-converting,
`as necessary, to Smaller image formats.
`
`29 Claims, 14 Drawing Sheets
`
`
`
`822 -
`
`- -
`Audio
`
`Audio
`Outputs
`
`Graphics
`Processor
`
`830
`?
`
`Digital
`--- Outputs
`idescreer E. agent
`Standard
`vil? --> video outpu
`– it Analog YA
`- Outputs
`B Digital
`
`Outputs
`
`How
`Video AF
`es
`
`
`
`
`
`controller
`
`User
`interface
`
`836
`
`\-
`
`Analog/Corponent
`-- Wideo Outputs
`a -------
`Digital
`Fi in Out
`Video /F
`Outputs
`
`

`

`U.S. Patent
`
`Apr. 9, 2002
`
`Sheet 1 of 14
`
`US 6,370,198 B1
`
`FILM
`
`184
`
`FILM
`
`
`
`194
`
`INTERMEDIATE
`
`2160
`
`3072
`
`Figure 1b
`
`

`

`U.S. Patent
`
`Apr. 9, 2002
`
`Sheet 2 of 14
`
`US 6,370,198 B1
`
`FILM
`
`204
`
`
`
`
`
`200 - 960 -
`N
`
`720
`
`216O
`
`-- 1200 --
`
`-- 3840 -e-
`Figure 1C
`
`
`
`FILM
`
`209
`
`2160
`
`-- 3840 --
`
`Figure 1d
`
`

`

`U.S. Patent
`U.S. Patent
`
`Apr. 9, 2002
`
`Sheet 3 of 14
`
`US 6,370,198 B1
`US 6,370,198 B1
`
`wOSS700ed1SAIC|99|Adv|“ODI7881SIGGUY|vosssodud[7§‘TTEVAONSE:SorYYO‘L78
`
`
`
`
`
`
`
`
`
`
`
`C CO
`
`Zainbi4ve
`
`HOSSE00àld
`
`Motorola v. Stellar
`
`Motorola Exhibit 1043
`Page 004
`
`
`

`

`U.S. Patent
`
`Apr. 9, 2002
`
`Sheet 4 of 14
`
`US 6,370,198 B1
`
`88 ||
`
`984
`
`W08
`
`90),
`
`
`
`
`
`
`
`
`
`
`
`• • • • • • • • •,
`
`39 WW||
`&#ENN? OS
`
`

`

`U.S. Patent
`
`Apr. 9, 2002
`
`Sheet 5 of 14
`
`US 6,370,198 B1
`
`Figure 4
`
`AWALOG
`SIGNAL
`//WPLJT
`
`DIGITAL
`SIGMAI
`/WPUT
`
`42
`
`406
`
`A/D
`COW/ERJOR
`
`404
`
`4B
`
`SIGWAI
`COMPRESSOR
`
`410
`
`DATABUS
`
`
`
`412S STORAGF
`MEAWS
`A
`
`SIORAGF
`MEAM/S
`B
`
`4f6
`SIGWA ?
`PROCESSOR
`422
`
`4 f6
`
`424
`
`COWWERTOR
`
`42D
`
`L/SER
`INTERFACE
`
`42
`
`DIGITAL
`SIGWAI
`OUTPUT
`
`AWALOG
`SIGNAL
`OUTPUI
`
`

`

`U.S. Patent
`
`Apr. 9, 2002
`
`Sheet 6 of 14
`
`US 6,370,198 B1
`
`Figure 5
`
`160
`
`FILM
`24FPS
`
`
`
`--------
`
`HDry
`30/60 FPS
`
`- 164
`-/
`
`166
`NTSC/WIDESCREEN-1
`30 FPS
`
`168
`FILMRECORDER /
`24 FPS
`
`PASECNIWDESCREx
`
`25 FPS
`
`170
`
`--
`
`HDTV
`25/50 FPS
`
`------ COLOR PRINTER
`
`172
`
`- 174
`
`

`

`U.S. Patent
`U.S. Patent
`
`Apr. 9, 2002
`Apr. 9, 2002
`
`Sheet 7 of 14
`Sheet 7 of 14
`
`US 6,370,198 B1
`US 6,370,198 B1
`
`9 € Inôl-ff
`
`ganbi-+
`
`
`
`SLNALNOO5CIZLSOdW09/SOTNT
`
`MSIO.CUVH
`
`9E2
`
`Sng¥L¥
`
`/NOISSAYdNO
`
`NOISSaedNO03d
`
`YOSSAIONd
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`Motorola v. Stellar
`
`Motorola Exhibit 1043
`Page 008
`
`
`
`

`

`U.S. Patent
`
`Apr. 9, 2002
`
`Sheet 8 of 14
`
`US 6,370,198 B1
`
`“‘ssaid01gq
`sueywih@)dsirz
`aoe1eqUya@)d84Fbz
`ame]Ze
`(1)1soe
`“‘seerd01g
`(asGo9)
`yeadoy
`qeoday
`sdjog
`wor
`WOT
`wor
`(€)Isdygz
`“‘ssedd0lg
`908[419}U]
`aoepIequy__a|@)dsdj¢z
`aoe]Ia]
`(dqsdygq)
`sdjog
`TmaolA
`wor
`aoRLeqUy
`
`adeleyuy
`uoranpey
`worjonpey
`
`sdgg
`
`
`
`ag
`
`
`sdoe
`
`woly
`
`WHOr.J
`
`Wotms
`
`mor.
`
`saps
`
`
`
`emmelyTS
`worpnpay
`uorpnpey
`“‘s80Td01g
`(as4og)
`PIESIP3aoeyraquy
`PIBOSIp2aouplequyadog)
`SORTS]
`sdog
`morg
`moly
`owedT:Z
`auMely1:3
`Wornpay
`euely1:3
`uorpnpey
`OTNpHy
`‘ssord01g
`(@sdyez)
`aovpieyuy
`
`V2AYuNOIS
`
`says
`
`sdygg
`
`WoLyz
`
`moi
`
`aurelyT:g
`
`uorjanpey
`
`woig
`
`goBTrequy
`aoepieyuy
`(8dypZ)
`sdf£Z
`aoepzaqUy
`eoey1eyuy
`
`(sdjpz)
`
`sdez
`
`{Iedjgz)
`
`UMOp-MOTS
`aae[1ejUy
`sdyqz
`
`dsdyeg
`
`moraf
`
`moag
`“spaIs01T
`
`(ds¢z)
`
`eurelgy:S
`
`sdyog
`
`OroTf
`
`2sOB[ASTUTwyVSImor
`uorjonpsy
`29eoR]seqUy
`aRIa]
`
`asdog)
`
`Motorola v. Stellar
`
`Motorola Exhibit 1043
`Page 009
`
`

`

`U.S. Patent
`
`Apr. 9, 2002
`
`Sheet 9 of 14
`
`US 6,370,198 B1
`
`
`
`-3q
`
`aoe]equ]
`
`uoronpey
`
`uorpnpay
`
`Torwf
`
`(L.NOO)v2SYNOIS
`pABOSIp7pUWOLTsoeprieyuy|wmop-aorg
`
`
`
`
`
`
` @)1sGoe|&)1sez|worao8]ASU]~~|moronpey|Usop-TngmortWoLworpnpeyyaoRfAeqwy]“esaTB01gwWOlT(dser)
`
`
`
`
`
`soupreyay|=aouyzoquy|uorpnpeyworjonpey|2pesuyzequy]I2a0RNpIEWU]“seaxSorg
`
`
`
`aoE[1eyUyaoR[teqUy]woruorpnpey|2psoepeyuy|=uoTIONpey|2soBpTeqUy“"s901301g
`
`
` @)Tsdjezg|)Ishre)®ashoe|@asdoo|GH)asder|@)asdjog
`(G)T8dyez|(1)d84396|(€)dsdog()dsUer(g)
`
`soBjreyuy|uoonpey|uoronpey-qd:“:aouyzequywoTNnpRywor
`
`mwoz,ymoigq“WAYOst(dsd7g9)
`
`
`
`Wormo1,]dasdjogWMory|Way‘osiqWoLy|“Way“osiq(asoe)
`WorWHOL,WoLword8Grz
`-3q8d309
`
`
`aqaUeAy1:3zig|dn-paadgaoetreyuy|~ourergT:ZaoupzeqUysdjgr
`
`
`aq2gaoeyaquy©eMBATT:Z|palyzywg)surely[:¢|“pelTimsBO
`
`
`
`
`
`
`sdjog
`
`“‘ssa1d01g
`
`(adz)
`
`sdjog
`
`“‘ssa1S0lg
`
`(d8d3g6)
`
`Motorola v. Stellar
`
`Motorola Exhibit 1043
`Page 010
`
`
`
`
`
`
`
`
`
`
`
`
`

`

`U.S. Patent
`
`Apr. 9, 2002
`
`Sheet 10 of 14
`
`US 6,370,198 B1
`
`Figure 7b
`1nput
`
`Speed-up
`
`
`
`
`
`input
`
`Gs)
`
`output
`(except 96P)
`
`
`
`Interlace
`& Discard
`
`
`
`6th Frame red,
`Int. & Disc.
`
`2:1 Frame Reduction
`
`2:l Frame red.
`
`
`
`Interlace
`& Discard
`
`input/output
`
`(se)
`
`output
`
`2:1 Frame
`
`4th Fr.
`
`3:2 Pull-down
`
`
`
`p
`
`Repeat
`
`& Discard
`
`Int. & Disc.
`
`output
`
`G)—
`
`2:1 Frame Reduction
`
`

`

`U.S. Patent
`
`Apr. 9, 2002
`
`Sheet 11 of 14
`
`US 6,370,198 B1
`
`Figure 7c
`
`3 : 2 Pull-Down
`(24 fps P to 30 fps I)
`
`--
`(Interlace)
`24 fps I ------ 1A
`1B
`
`(3.2 Pull-Down)
`
`2
`--
`2A
`2B
`
`?
`
`3
`--
`3A
`3B
`
`X
`
`4
`--
`4A
`4B
`
`30 fps I ------ 1A
`
`1B
`
`2A
`
`2B 2A
`
`3B 3A
`
`4B
`
`4A
`
`4B
`
`2
`
`3'
`
`4'
`
`5
`
`Figure 7e
`
`4th & 7th Field Repeat plus Re-Interlace
`(24 fps I to 30 fps I)
`
`--
`24 fps I ------ A
`B
`(4th & 7th field
`repeat plus
`
`re-interlace)
`
`2
`--
`2A,
`2B
`
`3
`--
`3A
`3B
`
`4.
`--
`4A
`4B
`
`. \ l l /
`
`4B
`4A
`4A
`3B
`3A
`2B
`2B
`2A
`1R
`30 fps I ------ 1A
`-- -- -- -- -T-
`1.
`2
`3.
`4'
`5
`
`* Revresed- Field
`
`Figure 7g
`
`3:2 Frame Repeat (24 fps P to 60fps P)
`
`24 fps P ------
`
`1
`
`2
`
`3
`
`4
`
`(3:2 Frame Repeat) \\ , / \ ?
`
`60 fps P ------
`
`1.
`
`2
`
`3.
`
`4'
`
`5
`
`6
`
`7|
`
`8
`
`9.
`
`10'
`
`

`

`U.S. Patent
`
`Apr. 9, 2002
`
`Sheet 12 of 14
`
`US 6,370,198 B1
`
`Figure 7d
`
`25 fps Interlace to 30 fps Interlace Conversion
`(from PAL CCD camera source to NTSC out)
`2
`3
`4.
`5
`1.
`--
`--
`-
`--
`--
`2A,
`2B
`3A
`3B
`4B
`5A
`5B
`25 fps I ------ 1A,
`1B
`t
`(de-interlace)
`9'
`10"
`8
`7'
`6
`5'
`3'
`4'
`2
`50 fps P ------ 1
`/ /
`/ /
`l
`l
`/ / \
`\
`repeat)
`\
`(4th frame
`1
`12
`8
`9'
`10
`7
`6”
`3
`4
`5’
`2
`62.5 fps P-----. 1
`- -
`-
`- -
`- -
`-
`- -
`(interlace)
`-
`- -
`-
`- -
`- -
`-
`62.5 fps I --- A B A B A B A B A B A B A B A B A B A B A B A B
`(discard)
`\
`1
`\
`|
`\
`1
`\
`1
`\
`31.25 fps I-1A, 1B' 2A, 2B' 3A 3B' 4A 4B' 5A 5B' 6A, 6B'
`(C ER, .
`.
`.
`.
`.
`.
`.
`\
`\
`\
`\
`\
`30 fps I ------ 1A, 1B 2A, 2B' 3A' 3B 4A 4B' 5A 5B' 6A, 6B'
`
`4A
`
`13
`
`2
`
`3.
`
`4
`
`5
`
`6
`
`Figure 7f
`
`4th Frame Repeat (24fps I to 30fps I)
`
`2
`--
`2A
`2B
`
`2
`
`3
`--
`--
`3A
`3B
`24 fps I ------ lA
`B
`,
`(de-interlace)
`6
`3
`4
`5'
`7
`48 fps P ------
`/ \\ l l /
`(4th Frane Repeat)
`3
`4
`5’
`6”
`7:
`8
`9°
`2
`12
`60 fps P
`-
`- -n
`r -n
`-
`- -
`-
`A B A B A B A B A B A B A B A
`(Iriterlace &
`l
`Discard)
`30 fps I ------ 1A 1
`B
`-- l-T-
`
`4
`--
`4A
`4B
`
`8
`
`10'
`-
`
`-T-
`
`--
`
`

`

`U.S. Patent
`
`Apr. 9, 2002
`
`Sheet 13 of 14
`
`US 6,370,198 B1
`
`Figure 7h
`
`6th Frame Reduction (30fps I to 25fps I)
`
`2
`
`3
`
`4
`
`5
`
`3A
`
`5
`
`3B
`
`4A
`
`6'
`
`4B
`t
`8
`
`5A
`t
`1
`9
`
`lB
`
`2A
`
`2B
`
`30 fps I ------ 1A
`(de-interlace)
`t
`1
`4
`3
`2
`60 fps P ------ 1'
`-l
`--
`(6th Frame red. A B A B A B A
`Inter. & disc.) \ l Y.
`25 fps I ------ 1A, 1B
`2A, 2B,
`-T-
`-T
`2.
`
`6
`6B
`t
`12
`
`SB
`
`6A,
`
`11
`10'
`r
`-
`A B A
`
`A B A B
`
`i
`
`3A
`--
`3.
`
`4A 4B'
`--
`4.
`
`5
`--
`5
`
`Figure 7i
`
`5th Frame Reduction (30fps I to 24fps I)
`
`4
`3
`2
`-- -- -- --
`-
`30 fps I ------ A
`1B
`2A
`2B
`3A
`3B
`4A
`4B
`5A
`(de-interlace)
`t
`!
`t
`- - - -
`2
`6"
`-
`- 1
`3
`4
`7
`8
`9'
`60 fps P-1 4
`-
`(5th Frame red. A B A B A B A B
`A B A B A B A B
`Inter. & disc.)
`l N.
`24 fps I ------ 1A, 1B
`3A 3B'
`4A 4B'
`--
`
`5'
`
`2A, 2B'
`--
`
`5
`-
`5B
`
`10"
`
`

`

`U.S. Patent
`
`Apr. 9, 2002
`
`Sheet 14 of 14
`
`US 6,370,198 B1
`
`
`
`
`
`
`
`Figure - 8
`
`- - -
`822
`\- Audio
`Processor
`
`Audio
`Outputs
`
`Wideo
`
`Signal
`Processor
`
`Graphics
`Processor
`
`
`
`Standard
`Widescreen
`Video AF
`
`Digital
`Outputs
`
`Analog/Component
`Video Outputs
`Analog YAC
`Outputs
`
`Digital
`--> Outputs
`
`Analog/Cornponent
`F: Video Outputs
`Digital
`Outputs
`
`HDTW
`Wideo AF
`
`Film Out
`Wideo AF
`
`Controller
`
`User
`interface
`
`

`

`US 6,370,198 B1
`
`1
`WIDE-BAND MULTI-FORMAT
`AUDIO/VIDEO PRODUCTION SYSTEM
`WITH FRAME-RATE CONVERSION
`
`REFERENCE TO RELATED APPLICATION
`This application is a continuation-in-part of U.S. patent
`application Ser. No. 08/834,912, filed Apr. 7, 1997, and also
`claims priority of U.S. Provisional Patent Application Serial
`No. 60/084,522, filed May 7, 1998. Both of these prior
`applications are incorporated herein by reference in their
`entirety.
`
`FIELD OF THE INVENTION
`This invention relates generally to Video production,
`photographic image processing, and computer graphics, and,
`more particularly, to a multi-format digital Video production
`System capable of maintaining the full bandwidth resolution
`of the Subject material, while providing professional quality
`editing and manipulation of images intended for digital
`television and other applications, including digital HDTV
`programs.
`
`15
`
`2
`fewer than 1200 Scan lines, with picture expansions to create
`a hierarchy of upward-converted formats for theatrical
`projection, film effects, and film recording. In addition,
`general-purpose hardware enhancements should be capable
`of addressing the economic aspects of production, a Subject
`not considered in detail by any of the available references.
`For the first fifty years of television in the United States,
`the history ShowS continuous development and improve
`ment of a purely analog-based System for Video production
`broadcasting. The nature of the NTSC system is to limit the
`video bandwidth to 4.2 MHZ, which corresponds to approxi
`mately 340 TV-lines of resolution. In countries where PAL
`or SECAM systems are employed, the bandwidth is 5.5
`MHZ, which corresponds to approximately 440 TV-lines of
`resolution.
`During the past ten years, digital processing has become
`the Standard for Video production equipment. However, to
`preserve compatibility with existing equipment and
`standards, the video bandwidth typically has been limited to
`4–6 MHZ (for NTSC and PAL applications, respectively).
`This also has tended to reduce the apparent generation loSS
`during Video production Steps.
`In the past five years or So, digital image compression
`technology has matured greatly. Furthermore, there are
`many incompatible Standards, Such as the different forms of
`JPEG systems, the Quick-Time system, MPEG-1, and the
`numerous forms of the MPEG-2 standard. In addition, the
`latest recording formats for Video production have intro
`duced a new set of variations, including the 4-inch DVC
`formats from Sony and Matsushita. While the signal dete
`rioration characteristics of multi-generation analog-based
`production Systems are well known, those imperfections
`resulting from diverse-format digital video compression and
`the conversions between these formats can be just as
`troublesome and unpredictable. In practice, these repeated
`Steps of analog-to-digital (A/D) conversion and digital-to
`analog (D/A) conversion, as well as data compression and
`decompression, introduce many Signal artifacts and various
`forms of Signal noise. Although digital Video production
`promises multiple-step production processes free of genera
`tion losses, the reality is different, due to the repeated Steps
`of A/D and D/A conversions, as well as data compression
`and decompression, present when utilizing the various
`incompatible image data compression formats.
`Meanwhile, during the last twenty years, camera technol
`ogy has advanced to a point far Surpassing the performance
`of traditional production equipment. The video bandwidth
`capability has increased from 4.2 MHZ (corresponding to
`340 TV-lines of resolution) to approximately 12 MHZ
`(corresponding to nearly 1000 TV-lines of resolution).
`Because of the limitations of conventional broadcast and
`production equipment, most of the detail information pro
`duced by today's high-performance camera Systems is lost.
`For HDTV Systems, one goal is to produce images having
`approximately 1000 TV-lines of resolution per picture
`height, which requires a bandwidth of approximately 30
`MHZ. This, in turn, raises a new problem in terms of
`Signal-to-noise ratio. While conventional broadcast cameras
`can produce Signals having a S/N ratio of 65 dB, utilizing
`10-bit digital processing, HDTV cameras typically produce
`signals having a S/N ratio of 54 dB, and utilize only 8-bit
`digital processing. In addition, the typical HDTV camera
`utilizes a 2 Megapixel CCD, in which the elements are
`approximately one-quarter the size of conventional broad
`cast cameras. This translates into a much lower Sensitivity (a
`loss corresponding to 1-2 lens f-stops), higher levels of
`“Smearing”, and lower highlight compression ratios.
`
`25
`
`35
`
`40
`
`45
`
`BACKGROUND OF THE INVENTION
`As the number of television channels available through
`various program delivery methods digital TV (DTV)
`broadcasting, cable TV, home Video, broadcast, etc. contin
`ues to proliferate, the demand for programming, particularly
`high-quality HDTV-format programming, presents Special
`challenges, both technical and financial, to program produc
`ers. While the price of professional editing and image
`manipulation equipment continues to increase, due to the
`high cost of research and development and other factors,
`general-purpose hardware, including personal computers,
`can produce remarkable effects at a cost well within the
`reach of non-professionals, even novices. As a result, the
`distinction between these two classifications of equipment
`has become less well defined. Although general-purpose
`PC-based equipment may never allow professional-style
`rendering of images at full resolution in real-time, each new
`generation of microprocessors enables progressively faster,
`higher-resolution applications. In addition, as the price of
`memory circuits and other data Storage hardware continues
`to fall, the capacity of Such devices has risen dramatically,
`thereby improving the prospects for enhancing PC-based
`image manipulation Systems for Such applications.
`In terms of dedicated equipment, attention has tradition
`ally focused on the development of two kinds of profes
`Sional image-manipulation Systems: those intended for the
`highest quality levels to Support film effects, and those
`50
`intended for television broadcast to provide “full 35 mm
`theatrical film quality, within the realities and economics of
`present broadcasting Systems. Conventional thinking holds
`that 35 mm theatrical film quality as projected in theaters is
`equivalent to 1200 or more lines of resolution, whereas
`camera negatives provide 2500 or more lines. As a result,
`image formats under consideration have been directed
`towards video systems having 2500 or more scan lines for
`high-level production, with hierarchies of production,
`HDTV broadcast, and NTSC and PAL compatible standards
`which are derived by down-converting these formats. Most
`proposals employ progressive Scanning, although interlace
`is considered an acceptable alternative as part of an evolu
`tionary process. Another important issue is adaptability to
`computer-graphics-compatible formats.
`Current technology directions in computers and image
`processing should allow production equipment based upon
`
`55
`
`60
`
`65
`
`

`

`US 6,370,198 B1
`
`15
`
`25
`
`35
`
`40
`
`45
`
`50
`
`3
`Analog-based HDTV Systems, Such as the Japanese
`MUSE system, do not approach the design goal of 1000
`TV-lines. In reality, only one quarter of the picture infor
`mation is transmitted. Although the nominal reduced lumi
`nance bandwidth of 20 MHZ provides approximately 600
`TV-lines of resolution per picture height in Static program
`material, this resolution is drastically reduced to only 450
`TV-lines where motion is occurring. The chrominance band
`width is even further reduced by the Sub-Sampling Scheme,
`to 280 TV-lines for the I-signal and 190 TV-lines for the
`Q-signal (in static scenes), and to 140 TV-lines for the
`I-Signal and 50 TV-lines for the Q-signal (in moving Scenes).
`Although this System provides a wide-screen aspect ratio of
`16:9, it does not really qualify as a High-Definition Televi
`Sion System.
`Because of the aforementioned compatibility issues, it is
`clear that conventional Video recorders cannot match the
`technical performance of modern camera Systems. Although
`“D-6 format' digital recorders are available, the cost and
`complexity of Such equipment place these units beyond the
`means of the vast majority of broadcast Stations.
`Furthermore, the capability of conventional Switchers and
`other production equipment Still fail to match that of avail
`able camera Systems.
`Other recorders have been produced, Such as the one
`half-inch portable recorder (“Uni-HI”), but this system only
`achieves 42 dB Signal-to-noise ratio, and records in the
`analog domain. These specifications render this unit unsuit
`able for multi-generation editing applications. Furthermore,
`the luminance bandwidth is only 20 MHZ, corresponding to
`approximately 600 TV-lines of resolution.
`W-VHS (“Wideband-VHS”) recorders provide a wide
`aspect-ratio image, but only 300 TV-lines of resolution,
`which also renders this unit unsuitable for any professional
`applications. Other distribution formats (such as D-VHS)
`require the application of high compression ratioS to limit
`the data-rate to be recorded, So these formats only achieve
`W-VHS quality (less than 400 TV-lines of resolution).
`The newly-introduced HD Digital Betacam format
`(HDCAM) video recorder utilizes a 3:1:1 digital processing
`System rather than the 4:2:2 processing. However, it has a 24
`MHZ luminance bandwidth corresponding to 700 TV-lines
`of resolution, and a narrower chrominance bandwidth.
`Although this System is clearly Superior to any existing
`analog HDTV recording system, it still falls short of deliv
`ering the full resolution produced by an HDTV digital
`camera. Because of its proprietary image data compression
`format, the production proceSS results in repeated data
`compression and decompression Steps, as well as A/D and
`D/A conversions, which, in turn, results in many Signal
`artifacts and various forms of Signal noise.
`In Summary, the conventional technology for these mar
`kets utilizes professional cameras having a 30 MHZ
`bandwidth, and capable of 1000 TV-lines of resolution.
`55
`However, they produce quality levels more characteristic of
`consumer-grade equipment (in terms of resolution and
`Signal-to-noise ratio). In addition, the price of these Systems
`is cost-prohibitive both on an absolute and also a cost/benefit
`basis, employing digital Systems which produce only
`analog-type performance.
`
`60
`
`SUMMARY OF THE INVENTION
`The present invention takes advantage of available
`general-purpose technology, where possible, in order to
`provide an economical multi-format digital Video produc
`tion System. In the preferred embodiment, Specialized graph
`
`65
`
`4
`ics processing capabilities are included in a high
`performance personal computer or WorkStation, enabling the
`user to edit and manipulate an input video program and
`produce an output version of the program in a final format
`which may have a different frame rate, pixel dimensions, or
`both. An internal production format is chosen which pro
`vides the greatest compatibility with existing and planned
`formats associated with HDTV standard 4:3 or widescreen
`16:9 high-definition television, and film. For compatibility
`with film, the frame rate of the internal production format
`preferably is 24 fps (for program materials originated in film
`format) and 48 fields-per-Second (for live program materials
`Such as Sporting events). Images are re-sized horizontally
`and Vertically by pixel interpolation, thereby producing
`larger or Smaller image dimensions So as to fill the particular
`needs of individual applications. Frame rates are adapted by
`inter-frame interpolation or by traditional Schemes, includ
`ing “3:2 pull-down” for 24-to-30 fps conversions. Simple
`speed-up (for 24-to-25 conversions) or slow-down (for
`25-to-24 conversions) for playback, or by manipulating the
`frame rate itself using a program Storage facility with
`asynchronous reading and writing capabilities. The Step of
`converting the signal to a HDTV format is performed by a
`modified upconversion process for wideband Signals
`(utilizing a higher sampling clock frequency) and a resizing
`to HDTV format frame dimensions in pixels.
`The invention preferably incorporates one or more inter
`face units, including a Standard/widescreen interface unit
`operative to convert the Video program in the input format
`into an output signal representative of a Standard/wideScreen
`formatted image, and output the Signal to an attached display
`device. A high-definition television interface unit is opera
`tive to convert the Video program in the input format into an
`output signal representative of an HDTV-formatted image,
`and output the Signal to the display device. A centralized
`controller in operative communication with the Video pro
`gram input, the graphics processor, and an operator
`interface, enables commands entered by an operator to cause
`the graphics processor to perform one or more of the
`conversions using the television interfaces. The present
`invention thus encourages production at relatively low pixel
`dimensions to make use of lower-cost general-purpose tech
`nology and to maintain high Signal-to-noise ratio, and then
`Subsequently expands the resultant image into a So-called
`up-converted program. This is in contrast to alternative
`approaches, which recommend operating at HDTV-type
`resolution, then down-converting, as necessary, to Smaller
`image formats. This has led to the use of expensive dedi
`cated hardware, the need for which the present invention
`SeekS to eliminate. In addition, the flexible Storage and
`playback facilities allow extensive control of the playback of
`the program material, enabling frame rate adjustments and
`alterations, and providing for time-shifting of the Start and
`end points of the program reproduction in those cases
`wherein direct control of the Source material frame rate is
`not practical, due to physical Separation of the equipment or
`multiple reception points simultaneously producing outputs
`at different frame rates from the same Source Signal playback
`data Stream. In commercial implementations, the invention
`readily accepts and processes enhanced information, Such as
`pan/scan information or identification information to restrict
`Viewing based on regional or geographical marketing plans.
`The method and associated technology provide for main
`taining the original high bandwidth of conventional cameras
`(up to 15 MHZ, which corresponds to more than 600
`TV-lines of resolution-per picture height for 16:9 aspect
`ratio) and provide optimized compression techniques to
`
`

`

`US 6,370,198 B1
`
`15
`
`25
`
`S
`fully utilize the available capacity of general Storage media,
`such as the commercially available Panasonic DVCPRO,
`DVCPRO50, Sony DVCAM, JVC Digital-S, and Sony
`Betacam SX recorders. The system preferably employs a
`consistent compression Scheme utilizing only intra-frame
`compression (Such as Motion-JPEG-type Systems, Systems
`used in DV-format recorders, MPEG-2 4:2:2P(a)ML)
`throughout the entire production process. This avoids many
`Signal artifacts, ensures high Signal-to-noise ratios, and
`provides for editing the program material in data
`compressed format. This enables the System to preserve the
`original camera capability of 600+ TV-lines of resolution per
`picture height, and with 4:2:2 processing provides a chromi
`nance bandwidth of up to 7.5 MHZ. Utilizing 10-bit pro
`cessing results in 65 dB Signal-to-noise performance and
`improved camera Sensitivity (rating of f-11). In contrast,
`available and proposed systems for HDTV are based on 8-bit
`processing, and offer performance of less than 54 dB signal
`to-noise ratio and camera Sensitivity rating of only f-8.
`The invention provides for optimization of the available
`Storage media as well. Utilizing hard-disks, optical discs
`(such as DVD, DVD-R, and DVD-RAM), magneto-optical
`discs, or digital tapes (such as DAT-format, DVC,
`DVCPRO, DVCPRO50, DVCAM, Digital-S, or 8-mm
`format) the data-rate to be recorded is nearly one-quarter
`that of conventional HDTV systems, and consumes only 20
`GB of Storage Space to record more than 60 minutes in the
`Production Format compression Scheme, which utilizes a
`data-rate of 50 Mb per second or less, which is well within
`the capabilities of certain conventional recording devices.
`Horizontal and vertical pixel-interpolation techniques are
`utilized to quadruple the image size, preferably resulting in
`an image frame size of 1920x1080 pixels. The resulting
`program information may then be distributed in a conven
`tional compression format, such as MPEG-2.
`Three alternative image frame sizes preferably are
`Suggested, depending on the intended application. For gen
`eral usage, an image frame size of 1024x576 is recom
`mended. As an option, a frame size of either 1280x720 or
`1920x1080 may be utilized, at 24 frames-per-second. A
`sampling frequency of up to 74.25 MHZ for luminance is
`40
`utilized for 1920x1080. Sampling frequencies of up to 37
`MHZ are preferably are utilized for 1024x576 and 1280x
`720. Chrominance components preferably are Sampled con
`Sistent with a 4:2:2 System, and 10-bit precision is preferred.
`The technology of display devices and methodology has
`progressed as well, offering alternative features Such as
`conversion of interlaced signals to progressive Scan, line
`doubling, pixel quadrupling, and improved general tech
`niques for horizontal and Vertical pixel interpolation. Avail
`ability of these features as part of display devices will
`Simplify the process of implementing multi-format digital
`production.
`BRIEF DESCRIPTION OF THE DRAWINGS
`FIGS. 1A-1D show the preferred and alternative image
`aspect ratioS in pixels,
`FIG. 2 shows a functional diagram for disk/tape-based
`Video recording;
`FIG. 3 shows the components comprising the multi
`format audio/video production System;
`FIG. 4 is a block diagram of an alternative embodiment of
`Video program Storage means incorporating asynchronous
`reading and writing capabilities to carry out frame-rate
`conversions,
`FIG. 5 shows the inter-relationship of the multi-format
`audio/video production System to many of the various
`existing and planned Video formats,
`
`6
`FIG. 6 shows the implementation of a complete television
`production System, including Signals provided by broadcast
`Sources, Satellite receivers, and data-network interfaces,
`FIGS. 7A-7B show the preferred methods for conversion
`between Several of the most common frame-rate choices,
`FIGS. 7C-7I show details of possible methods for frame
`rate conversion processes, and
`FIG. 8 shows a block diagram of an embodiment of a
`universal playback device for multi-format use.
`DETAILED DESCRIPTION OF THE
`INVENTION
`The present invention resides in the conversion of dis
`parate graphics or television formats, including requisite
`frame-rate conversions, to establish an inter-related family
`of aspect ratios, resolutions, and frame rates, while remain
`ing compatible with available and future graphics/TV
`formats, including images of pixel dimensions capable of
`being displayed on currently available multi-Scan computer
`monitors. Custom hardware is also disclosed whereby
`frames of higher pixel-count beyond the capabilities of these
`monitors may be viewed. Images are re-sized by the System
`to larger or Smaller dimensions So as to fill the particular
`needs of individual applications, and frame rates are adapted
`by inter-frame interpolation or by traditional Schemes Such
`as using “3:2 pull-down” (Such as 24 frame-per-Second (fps)
`Progressive to 30 fps interlace shown in FIG. 7C or 48 fps
`Progressive to 60 fps Progressive, as would be utilized for
`film-to-NTSC conversions) or by speeding up the frame rate
`itself (such as for 24 to 25 fps for PAL television display)
`The re-sizing operations may involve preservation of the
`image aspect ratio, or may change the aspect ratio by
`“cropping certain areas, by performing non-linear
`transformations, Such as "Squeezing” the picture, or by
`changing the vision center for “panning,” “scanning” and So
`forth. Inasmuch as film is often referred to as “the universal
`format.” (primarily because 35-mm film equipment is stan
`dardized and used throughout the world), the preferred
`internal or “production' frame rate is preferably 24 fps. This
`Selection also has an additional benefit, in that the 24 fps rate
`allows the implementation of cameras having greater Sen
`sitivity than at 30 fps, which is even more critical in systems
`using progressive Scanning (for which the rate will be 48
`fields per second interlaced (or 24 fps Progressive) vs. 60
`fields per Second interlaced in Some other proposed
`Systems).
`The image dimensions chosen allow the use of conven
`tional CCD-type cameras, but the use of digital processing
`directly through the entire Signal chain is preferred, and this
`is implemented by replacing the typical analog RGB pro
`cessing circuitry with fully digital circuitry. Production
`effects may be conducted in whatever image Size is
`appropriate, and then re-sized for recording. Images are
`recorded by writing the digital data to Storage devices
`employing internal or removable hard-disk drives, disk
`