United States Patent [191
`Sandrew
`
`[11] Patent Number:
`[45] Date of Patent:
`
`4,984,072
`Jan. 8, 1991
`
`[54] SYSTEM AND METHOD FOR COLOR
`IMAGE ENHANCEMENT
`
`,
`_
`_
`[75] Inventor: Barry B- Sandrew, Encmltas, Cahf-
`.
`_
`.
`.
`.
`[73] Ass‘gnee' amen” F‘lm Techmm‘e" Inc-1
`mm» Pa-
`[21] APPL No; 222,543
`
`[22] Filed:
`
`Jul. 25, 1988
`
`[63]
`
`Data
`Related‘; _ A H -
`‘s W “hon
`Continuation-impart of Ser. No- 81,1l6, Aug- 3, 1987,
`abandoned.
`[51] Int. Cl,
`
`HMN 1/46
`
`[52] 1.1.8. C1. . . . . . . . . . . . . . .
`. . . . . .. 358/81; 358/82
`[53] Field 61 Search ...................... .. 353/31, 32, 75, 30
`56
`R fereuces Ci
`[
`1
`U S PA'IfENT DOCleTdlWENTS
`'
`'
`
`- - - -~
`
`i?gggm-lg-r- - - -
`353/30
`2:799:722 7/1957 Neugebauer ..
`353/32
`2,304,500 3/1957 Giacoletto
`_ _ _ " 353/32
`2,374,212 2/1959 Bcchley ' _ _ ' _
`358/104
`2,883,763 4/1959 Schaper
`358/22
`2,974,190 3/1961 Geiger
`353/32
`2,932,314 5/1961 Fine et 31..
`353/332
`3,005,042 10/1961 Horsley ....... ..
`353/32
`3,253,523 6/1966 Oppenheimer
`.. 358/104
`3,486,242 12/1969 Aronson ...... ..
`3,551,539 12/1970 Moskovitz .................. .. 353/32
`3,553,311 1/1971 Montevecchio et a1.
`353/442
`3,560,544 2/1971 Pctmcdli °t 31-
`353/104
`' ' ' ' ‘ ' ' ' ' ' ' ' ' '
`' ' ' "
`
`3:612:755 10/1971 Tadlotf; I:
`3,617,626 11/1971 Blllth et a1.
`3,647,942 3/1972 Siegel . . . . . . . . . . . .
`3,673,317 6/1972 Newell et al.
`3,706,841 12/1972 Novak ......... ..
`3’7lo’oll 1/1973 Altemus et al-
`gcgel """"" "
`358/81
`3’769’458 “V1973 D2523” '
`358/81
`3:770:884 11,1973 cum-an egg-u:
`353/31
`3,770,335 11/1973 Curran et al.
`3,772,465 11/1973 Vlahos et al. ....................... .. 353/31
`
`I: 353/32
`353/23
`. . . .. 353/32
`358/82
`358/82
`353/32
`
`. ........... .. 353/31
`.
`3,734,736 1/1974 Novak ....... ..
`353/76
`3,343,356 11/1974 Reeber Btal. .... ..
`353/32
`3,971,063 7/1976 Gerhardt et a1. ..
`353/31
`3,972,067 7/1976 Peters ............... ..
`353/22
`4,054,904 10/1977 Saitoh Blal.
`353/31
`4,149,135 4/1979 Weinger .... ..
`353/31
`4,133,046 1/1930 Dalke et a1. .
`358/80
`4,258,385 3/1981 Greenbcrg et al.
`353/31
`4,313,121 2/1932 Taite et a1. ....... ..
`353/31
`4,329,710 5/1932 Taylor
`353/30
`4,334,240 6/1932 Franklin ..
`353/31
`4,549,172 10/1935 Welk ......... ..
`353/32
`4,590,511 5/1936 Bocchi et a1.
`353/31
`4,606,625 3/1936 Geshwind .... ..
`358/81
`4,608,596 8/1986 Williams et a].
`4,725,379 2/1933 Eide et a1, ........................... .. 353/31
`
`FOREIGN PATENT DOCUMENTS
`933938 8/1963 United Kingdom '
`
`OTHER PUBLICATIONS
`G. R. McNeal, “Design of a Pseudocolor Generator”,
`IEEE Eighth Annual Conference of the Engineering in
`Medllggifgfwbw Sway’ N°"' 1986' v°1' 3 °f 3’
`PP-
`-
`_
`,
`E. J. Farrell, “Color Display and Interactive Interpreta
`tion of Three-Dimensional Data”, IBM J. Res. De
`velop. vol. 27, No. 4, Jul. 1984, pp. 356-366.
`_
`,
`P?m?o' hammer-James J- Groody
`AmlvtantExaminer—Mark R- Powell
`Attorney, Agent, or Firm-Ratner & Prestia
`
`Answer
`[57]
`A system and method for color enhancing an image or
`a series of images such as a motion picture by digitally
`
`calm“? the im‘fges’ .inmF‘ivdY de?l‘ing.m?sks 6°“
`respqndms 1° °bJ=°ts “1 the Images havmg 811ml“ hues’
`creams regions from these masks, and for 94911 region»
`defining a color transfer function for converting image
`gray-scale information to unique values of Hue, Lumi
`nance, and Saturation. The gray-scale values within
`each region are then processed through that region’s
`color transfer function, and the resulting colors applied
`to the image and stored for later retrieval and display.
`
`37 Claims, 9 Drawing Sheets
`T 1
`00200 000111 REGION
`
`1 ~ 1
`
`o
`
`°
`
`0
`
`6
`1
`
`° °
`1
`1
`
`0
`1
`
`1
`o
`
`1
`0
`
`1
`o
`
`1
`
`o
`1
`
`1
`1
`1
`1
`o 0
`
`1
`°
`
`Prime Focus Ex 1032-1
`Prime Focus v Legend3D
`IPR2016-01243
`
`

`

`US. Patent Jan. 8, 1991
`
`Sheet 1 019
`
`4,984,072
`
`FRAME #1
`
`FHAME #2
`
`F I G . 1
`
`DIGITAL
`
`FRAME
`
`SOURCE
`
`Prime Focus Ex 1032-2
`Prime Focus v Legend3D
`IPR2016-01243
`
`

`

`US. Patent
`
`m r
`
`4,984,072 '
`
`m 0 1 1 O u
`
`1 0 0 1 % 0011M ‘m
`
`m a u
`“m a’. m
`1 0 2 4.. u
`
`mum
`u
`
`REGION
`
`MASK
`
`Prime Focus Ex 1032-3
`Prime Focus v Legend3D
`IPR2016-01243
`
`

`

`US. Patent Jan. 8, 1991
`
`Sheet 3 of9
`
`4,984,072 '
`
`IMAGE
`
`IMAGE
`
`F I G. 3
`
`HAME 2
`
`FRAME 1
`
`Prime Focus Ex 1032-4
`Prime Focus v Legend3D
`IPR2016-01243
`
`

`

`US. Patent Jan. 8, 1991
`
`Sheet 4 of9
`
`4,984,072 ‘
`
`FHAME 2
`
`Prime Focus Ex 1032-5
`Prime Focus v Legend3D
`IPR2016-01243
`
`

`

`US. Patent vJms, 1991
`
`Sheet 5 of9
`
`4,984,072~ ‘
`
`IMAGE
`
`
`
`FIG. 5 REGION N0.
`
`
`
`COLOR LUTS
`
`1000
`
`Prime Focus Ex 1032-6
`Prime Focus v Legend3D
`IPR2016-01243
`
`

`

`US. Patent Jan. 8, 1991
`
`Sheet 6 of9
`
`4,984,072
`
`82
`
`um .mHm
`
`mm .mHm
`
`[K 55 2%
`
`82
`
`55w 2%
`mmw
`
`55w 2%
`mum
`
`Prime Focus Ex 1032-7
`Prime Focus v Legend3D
`IPR2016-01243
`
`

`

`US. Patent Jan. 8, 1991
`
`Sheet 7 of 9
`
`4,984,072
`
`82
`
`on
`
`2
`
`mm
`
`53m 2%
`mmm
`
`Prime Focus Ex 1032-8
`Prime Focus v Legend3D
`IPR2016-01243
`
`

`

`US. Patent Jan. 8, 1991
`
`Sheet 8 of9
`
`4,984,072
`
`82
`
`um .mHm
`
`55w 2%
`mmm
`
`55m 2%
`mwm
`
`Prime Focus Ex 1032-9
`Prime Focus v Legend3D
`IPR2016-01243
`
`

`

`US. Patent Jan. 8, 1991
`
`Sheet 9 of 9
`
`4,984,072
`
`F I G 9
`
`‘ii/v A
`
`A+B
`SCALE 255-— (/ 0\__
`GRAY
`
`Prime Focus Ex 1032-10
`Prime Focus v Legend3D
`IPR2016-01243
`
`

`

`1
`
`SYSTEM AND METHOD FOR COLOR IMAGE
`ENHANCEMENT
`
`This application is a continuation-in-part of US. pa
`tent application Ser. No. 081,116 ?led Aug. 3, 1987, now
`abandoned.
`
`FIELD OF THE INVENTION
`The present invention pertains to a system and
`method for the color enhancement of monochrome or
`multi-color images by relation of digitized gray-scale
`information from the original image to unique values
`(triplets) of hue, luminance, and saturation derived from
`a color transfer function applied to the achromatic
`gray-scale information.
`BACKGROUND OF THE INVENTION
`In strictly mathematical terms, the image produced
`or black and white ?lm is one dimensional (brightness),
`while an original scene can be viewed as three dimen
`sional (using the HLS model of color, where hue, lumi
`nance, and saturation are the dimensions). The chemical
`reaction of the ?lm emulsion, and the camera mecha
`nisms and ?lters, essentially reduce the three dimen
`sional information to one dimension by complex map
`ping, and forever obliterate the original hue, luminance,
`and saturation information.
`Note: It should be pointed out that luminance and
`brightness are not the same. Since brightness of the
`black and white image results from the complex interac
`tions of the three elements of the HLS system, there is
`only a statistical correlation between brightness and
`luminance, but they are not identical.
`
`5
`
`25
`
`35
`
`‘4,984,072
`2
`determines saturation (proportion of a color which is
`made up of its complementary color), with the axis
`representing zero saturation (equal amounts of the pri
`mary color and its complement and therefore no color)
`and the surface of the cone representing maximum per
`ceived saturation for a given luminance. Finally, angu
`lar displacement around the luminance axis represents
`' hue, as with a conventional color wheel, with comple
`ments separated by 180 degrees.
`The solid space de?ned by the HLS conic model is
`pointed at its top and bottom to represent the fact that
`as colors become whiter or blacker, they are perceived
`to lose hue. Saturation is a characteristic which depends
`chiefly on the narrowness or purity of the band of
`wavelengths comprising a color. At the exact center of
`the space is a medium gray which exhibits a moderate
`emittance or re?ectance for all wavelengths of light.
`The RGB system, is used, for example, in CRT dis~
`plays, where each of the three electron guns (red, green,
`20
`and blue) is controlled directly by a signal indicating its
`' respective intensity. HLS color may be converted to
`the RGB representation (see Foley & VanDam, pp.
`648-9.), however there are many HLS combinations
`which must be “clipped” to be represented in RGB.
`The same is true of NT SC video (also known as YIQ),
`which has an even more restricted range than RGB.
`Thus, (not every “real world” color can be reproduced
`by video systems using either RGB or NTSC standards.
`According to Foley and VanDam, “neither the V in
`HSV [Hue, Saturation, Value] nor the L in HLS corre
`spond to luminance in the YIQ model, so two different
`‘ colors de?ned in either space can easily have the same
`luminance, and be indistinguishable on black and white
`TV or videotape.” (Foley and VanDam p. 618)
`Since approximately 1970, several techniques have
`been developed for the addition of color to mono
`chrome images. Often, these techniques rely on video
`technology to augment the video signal in a de?ned
`manner, typically by adding a chrominance component
`signal to the pre-existing luminance signal. This tech
`nique results in an appearance not unlike that achieved
`by Thomas Edison in his early efforts to add tint to
`black and white ?lm by painting the frames: a colored
`area over a black and white picture. Because of their
`reliance on video technology, all of these sytems suffer
`from lack of vividness of colors, and from an inability to
`accurately reproduce colors.
`'
`BRIEF DESCRIPTION OF THE INVENTION
`The present invention comprises a system and
`method for color enhancing an image or a series of
`images such as a motion picture by digitally capturing
`the images, interactively de?ning areas or regions cor
`responding to objects or regions in the images having
`similar hues, and for each region, de?ning a color trans
`fer function for converting image gray-scale informa
`tion for each picture element to unique values of Hue,
`Luminance, and Saturation. The gray-scale values
`within each region are then processed through that
`region’s color transfer function, and the resulting colors
`created in an image memory and stored for later re
`trieval and display.
`
`40
`
`COLOR REPRESENTATION SYSTEMS
`Color is both an absolute quality of light (its wave
`length or frequency) and a relative quality which de
`pends on the human perceiver. For this reason, various
`color “gamuts” have been proposed which allow per
`ceived colors to be quanti?ed. Among these models is
`the CIE chromaticity scale, which permits quanti?ca
`tion of color by combinations of so-called primaries
`(which are, by themselves, not visible to human observ
`ers). (See Foley & VanDam, Fundamentals of Interactive
`Computer Graphics. Addison WEsley, Reading, Mass.
`1984 pp. 606-608.) In addition to theoretical models of
`color (CIE Chromaticity, Hue-Luminance-Saturation,
`and Red-Green-Blue), there are many representational
`systems for color reproduction in various media such as
`the RGB system for cathode ray tubes, the process
`color system for printing, and the NTSC, PAL, and
`SECAM systems for broadcast video. Each of these
`models is, in fact, a relatively small subset of both the
`theoretical absolute range of colors and of the actual
`range of humanly perceivable colors.
`One of the theoretical models (Ostwald, W.,Col0ur
`Science, Winsor & Winson, London, 1931) which is
`especially useful for analysis of color is the HLS model
`of hue, luminance, and saturation. This model uses a
`three dimensional cylindrical universe (the absolute
`theoretical color space), inside of which is a space
`bounded by a pair of cones having a common base (the
`perception space). The cylinder’s axis (from apex to
`apex of the cones) is the luminance axis, with the lower
`apex having a luminance of zero (all black) and the apex
`of the upper cone having a luminance of 1 (all . white).
`Distance from the luminance axis (normal to the axis)
`
`45
`
`55
`
`65
`
`BRIEF DESCRIPTION OF THE FIGS.
`FIG. 1 depicts in schematic form the creation of two
`adjacent frames of graphic information according to the
`method of the present invention.
`
`Prime Focus Ex 1032-11
`Prime Focus v Legend3D
`IPR2016-01243
`
`

`

`4,984,072
`
`3
`FIG. 2 is a schematic representation of a portion of
`the image memory structure according to one embodi
`ment of the method of the present invention.
`FIG. 3 depicts the replication of a the region memory
`portion of the image memory structure of the present
`invention.
`FIG. 4 depicts the correction of a value stored in the
`replicated region memory portion of FIG. 3. ,
`FIG. 5 depicts the application of the HLS color
`model to the coloring of an array of picture elements
`stored in the image memory structure of FIGS. 24.
`FIG. 6 depicts a schematic representation of the “his
`togram slide” operation according to the method of the
`present invention.
`FIG. 7 depicts a schematic representation of the “his
`togram multiplication down” operation according to
`the method of the present invention.
`FIG. 8 depicts a schematic representation of the “his
`togram multiplication up” operation according to the
`method of the present invention.
`FIG. 9 depicts a schematic represention of the “color
`bending” operation according to the method of the
`present invention.
`
`20
`
`4
`KEY FRAME COLORING
`The ?rst task of the key frame colorist is to identify
`areas of the image which comprise objects which will
`be designated to be of similar hue. Several software
`tools are available to the colorist to aid in this computer
`interactive de?nition process.
`A “free hand” mode allows drawing and erasing as
`well as control over a variably-sized cursor. A pointing
`device controls an interactive cursor displayed on a
`screen overlaying the key frame to be processed. Move
`ment of the pointing device effects an analogous move
`ment of the cursor and the picture elements selected by
`this movement (which are displayed on a graphics plane
`known as the “mask plane”) are used to de?ne the edge
`of an area. Because the cursor size may be reduced to
`only one pixel, the smallest region which may be pro
`duced is also one pixel.
`A second drawing tool available to the colorist is one
`which de?nes polygons, rather than completely free
`hand shapes. Using the pointing device, the colorist
`may indicate the precise placement of vertices which
`are intended to de?ne the perimeter of a polygonal area.
`Each vertex may be assigned or removed and, when
`complete, the vertices may be completed to de?ne a
`closed path.
`After de?nition of a closed perimeter by free-hand or
`polygon methods, the enclosed area may be assigned to
`the region (by setting the bits in the mask plane) by
`using one of two ?lling algorithms. An “area ?ll” as
`signs all picture elements within the closed shape and up
`to the de?ned edge to the area. A “cartoon ?ll” com
`bines the de?ned edges with edge recognition algo
`rithms in order to fill an area which is bounded by a
`border line such as the black pen lines which are typical
`of cartoons. Cartoon ?ll mode ?lls up to, or slightly
`over such heavy border lines.
`After areas have been de?ned for all objects having a
`common hue, the ?nal step is the selection and applica
`tion of a given hue to that area. Colors may be selected
`by interactively pointing to a color wheel displayed on
`a video screen, and then by manipulating control indica
`tors on the video screen which allow re?nement of the
`gray~scale information for the area as described below.
`After selection, the color may be displayed by its appli
`cation to the areas (which are collectively known as a
`color mask.) After a color mask has been de?ned, its
`associated picture elements are identi?ed as a “region”,
`the mask plane is cleared, and additional masks are
`de?ned in a similar manner, one for each hue which will
`be displayed in the frame.
`Once all regions are de?ned, the regions, the underly
`ing frame, and the colors assigned to each region are
`stored for later retrieval by colorists who will reiterate
`the process using the subsequent frames of each scene
`and the key frame which has been de?ned.
`The process of extrapolating color enhancement from
`key frames to entire scenes is performed by identi?ng
`and tracking motion as well as appearances and disap
`pearances of objects. Generally fewer than 5% of the
`picture elements in a frame of a scene change in the
`succeeding frame. Because motion pictures are ?lmed at
`24 frames per second, increments of motion on all but
`the most rapidly moving objects are extremely small
`and easily followed. Once motion has been identi?ed,
`adjustment of the regions and application of the key
`frames’ prede?ned colors to those regions is all that is
`required.
`
`DETAILED DESCRIPTION OF THE
`INVENTION
`GENERAL PROCESS OVERVIEW
`The ?lm to be colored or color enhanced is ?rst
`digitally recorded at a preselected resolution. In in
`stances where the ?nal product will be placed on con
`ventional video tape, appropriate resolutions are em
`ployed.
`(E.g. NTSC=525 lines,
`PAL=640,
`HDTV=11251500) Where the ?nal product will be
`used for theatrical presentation, resolutions of 1024 lines
`or more may be employed. The digitization process
`(depicted in FIG. 1) comprises projecting each frame of
`the original (typically black and white) ?lm through a
`liquid gate device which masks scratches and other
`imperfections. The projected frame is captured using a
`video camera such as a saticon tube and the video signal
`is digitized and stored in a random access memory
`frame buffer. The digitization process allows adjust
`ment of the intensity of light falling on the ?lm in order
`to achieve the best contrast for the particular frame
`being digitized (gamma correction). After acquisition of
`a frame, the digital representation of the frame is stored
`in non- volatile storage for further processing. The
`digitization process is repeated on a frame-by-frame
`basis, marking each frame with a time and sequence
`indication (SMPTE time code).
`For each scene (sequence of frames), a frame is se
`lected as being representative of the majority of the
`elements present in the sequence. This “key frame” is
`used as a template for the enhancement of other frames
`within the scene sequence For this reason, the greatest
`care is applied to the processing of key frames.
`After key frames are color-enhanced and de?ned,
`they are passed with the rest of the frames comprising a
`scene to a work station where the regions de?ning the
`application of color are tracked from frame to frame.
`After each frame has been enhanced, it is re-stored in
`non-volatile storage and sent to a central storage were
`all enhanced scenes are eventually reassembled accord
`ing to their original sequence as a ?nal product motion
`picture.
`
`25
`
`30
`
`35
`
`40
`
`45
`
`55
`
`60
`
`65
`
`Prime Focus Ex 1032-12
`Prime Focus v Legend3D
`IPR2016-01243
`
`

`

`25
`
`35
`
`20
`
`5
`Implicit in all frame-to-frame tracking is the ability to
`copy prede?ned parts of the preceding or key frame’s
`region memory contents and to use those parts in subse
`quent frame processing. After copying, small adjust
`ments may be made to the new region memory contents
`and they may be saved with the new frame. Impor
`tantly, because the digitization process assures consis
`tency of brightness and gray scale, and because colors
`are prede?ned by an art director working with key
`frames, overall consistency from frame-to-frame, and
`scene-to-scene is also assured.
`The simplest form of tracking from frame to frame is
`where motion in the frame is apparent only as a result of
`a camera pan. In such instances, the entire region mem
`ory may be copied from that of the preceding frame and
`“shifted” in the direction of the pan. Such a shift will
`create an unde?ned area at the end opposite the motion.
`This area must then be added to the existing regions in
`order to completely apply color.
`When only some of the objects of a frame move, a
`corresponding portion of the region memory may be
`translated as well. In this operation, that portion of the
`region memory which is to be moved is designated by
`the operator and is moved manually to its new location,
`thus overwriting the former contents of the region
`memory at the new location. The old location must then
`be re-defmed and/or added to one or more pre-existing
`regions in order to apply color.
`In highly complex scenes, where several types of
`motion are present, (especially compound motion of
`subject and camera) a “differencing” procedure is em
`ployed. A frame for which areas have been de?ned and
`to which color has been applied is subtracted from a
`next subsequent frame in order to reveal those picture
`elements which have changed (in gray-scale value).
`These areas of difference are then displayed as a con
`trasting overlay on the subsequent frame to be pro
`cessed. Alternatively, an algorithm for edge detection
`in gray-scale images may be used and may be correlated
`to the previous frame’s region memory.‘ Detected differ
`ences may then cause the subsequent frame’s regions to
`be adjusted or deleted, as required. By examining the
`coincidence of this overlay and the region structure
`which has been duplicated from the preceding frame,
`the production colorist is able to quickly designate areas
`which should be added to the region structure, and
`those which must be removed from a given region and
`reassigned.
`The ?nal operation is to assure that no objects have
`either completely appeared or disappeared from the
`frame. If an object disappears, its region must be re
`moved. Similarly, newly appearing objects must have a
`region de?ned for them “on the ?y” and a color as
`signed as well. After adjusting the region for the next
`subsequent image, the colorist again stores that image
`together with its region memory and repeats the pro
`cess for each subsequent frame.
`_
`To permit quality control and review of a sequence of
`60
`color-enhanced frames, two modes of viewing are used:
`Cine mode and Cycle mode. In cine mode, a series of
`frames is sequentially displayed, shuttling ?rst in a for
`ward direction, and then immediately in reverse. The
`speed may be controlled by the operator In cycle mode,
`65
`display direction and speed are directly tied to the
`pointing device to allow ?ne control over viewing
`frame-to-frame transitions.
`
`4,984,072
`
`6
`HARDWARE OVERVIEW
`The method of the present invention is implemented
`on a special purpose version of an RTI-Station image
`processing system interfaced to a personal computer
`such as the IBM/personal computer AT). The RTI
`image processing subsystem (available from Recogni
`tion Technology, Inc. of Holliston, MA) comprises a
`video digitizer, a pipeline architecture pixel processor,
`and image memory which is all interconnected both by
`a host computer bus and by a sychronous video bus.
`The video digitizer takes its input from a video camera
`such as that described above for digitizing ?lm frames,
`and outputs its information to a high de?nition color
`CRT (though not necessarily one interfaced by conven
`tional NTSC video circuitry). The host computer bus
`permits the image processing subsystem to communi
`cate with the user through the host CPU, its memory,
`associated mass storage devices, and devices including
`pointing devices and keyboards.
`The image processing subsystem of the preferred
`embodiment of the present invention comprises an
`image memory having 512x512 spatial resolution, 8
`bits of brightness resolution, and 4 bits for region identi
`?cation. Alternatively, the same system may be con?g
`ured for 6 bits of brightness resolution and 6 bits of
`region identi?cation or for 7 bits of brightness resolu
`tion and 5 bits of region identi?cation. (one additional
`bit is reserved for use by the system in construction of
`the mask bit plane.) General information regarding the
`structure and operation of the RTI Station and its asso
`ciated software library (RTILIB) may be found in the
`RTILIB/SOO Tutorial Revision 1.00 and the RTI
`LIB/ 500 User Manual Revision A, which are published
`by Recognition Technology, Inc.
`Although the currently used implementation employs
`a 512 line spatial resolution, it will be recognized by
`those skilled in the art that increases to and above 2000
`line spatial resolution are within the scope of the present
`invention. Additionally, the image memory may be a
`part of a larger frame store memory which may be, for
`instance, 4096><4096><l6 bits. The display of frames
`may then be controlled by a memory pointer into this
`frame store.
`For the purpose of color selection, the host CPU is
`provided with a 24 bit RGB video board (such as the
`Targa 24, a product of AT&T Information Systems,
`Inc.) A color wheel which corresponds to the common
`base of the HLS cone is displayed on a video screen.
`Using a pointing device, the colorist selects a particular
`color on the wheel, thus de?ning both hue (angular
`position on the color wheel) and saturation (distance
`from the center). As described more fully below, this
`unique 24 bit RGB value is translated immediately into
`the HLS coordinate space, correlated to the modi?ed
`image gray-scale, and retranslated for output on an
`RGB monitor.
`
`SOFTWARE OVERVIEW
`Referring now to FIGS. 1-4, according to the
`method of the present invention, a series of digitized
`monochrome images may be stored in a random access
`memory of a computer as an array of gray-scale values.
`For example, 8 bits of information may be used to repre
`sent each picture element, thus permitting a total of 256
`unique gray-scale values to be represented. (Of course,
`more bits may be used to achieve better resolution and
`dynamic range for the gray-scale information.) The
`
`Prime Focus Ex 1032-13
`Prime Focus v Legend3D
`IPR2016-01243
`
`

`

`4,984,072
`8
`7
`(Thus, 255 gray-scale corresponds by default to a lumi
`digitization step may optionally include the use of look
`up tables in order to enhance contrast, adjust gamma,
`nance of 627.)
`-In order to affect the color transfer function con
`enhance edge de?nition or “stretch” the gray-scale of a
`structed by the system, the colorist actually manipulates
`faded image. Associated with the memory used to store
`the frame gray-scale information by applying histo
`the gray-scale values is a region memory used to
`gramming techniques such as translation, compression,
`uniquely identify each picture element with one of a
`and stretching.
`number of regions (depicted as a four bit memory struc
`Speci?cally, the entire gray-scale may be made to
`ture in FIGS. 2-5). In addition to the image memory
`appear either more or less luminous by a simple “slide”
`and region memory structures, a mask plane memory is
`of the gray-scale (GS) with respect to the luminance
`also associated with the image for use in identifying
`picture elements. Through conventional techniques, the
`scale (FIG. 6). Instead of corresponding GS =128 to
`L=500, the operator may instead choose to brighten
`contents or parts of the contents of these memory struc
`the frame by choosing L=750. Similarly, the frame may
`tures may be copied from one structure to another (for
`be darkened by choosing L=350. As depicted in FIG.
`example, from the region memory to the mask plane
`6, the portion of the color transfer function falling
`memory) or from one frame’s memory to another (as
`within the HLS conic space which corresponds to the
`depicted in FIG. 3 for the transfer of frame ll region
`gray-scale is controlled by this slide operation. At its
`memory to frame 2 region memory.) The copied con
`extremes, the operation creates a threshhold in the
`tents may, of course, be altered by the operator to con
`transfer function beyond which all gray-scale values are
`form to the underlying frame contents as depicted in
`displayed as either white (above the upper threshhold),
`FIG. 4.
`As depicted in FIGS. 5-9, the colorist has at his or
`or as black (below the lower one).
`her disposal a number of software-implemented tools
`which control the construction of the color transfer
`
`LISTING 1
`
`20
`
`Listing 1
`void slide_map( );
`void stretch_map( );
`void bend._on( );
`void bend_off( );
`void slide._bend( );
`void stretch__bend( );
`/*LOCAL CONSTANTS
`#define TRUE I
`
`/”Local Static Data
`static int brt_l = 0;
`static int brt_u = 1000;
`static int bend = FALSE;
`static int markl;
`static int mark2;
`/‘Global Data (and externals)—
`extem int hue_l;
`extem int sat_ l;
`extem int hue_2;
`extem int sat_2;
`
`void slide_map (dbrt)
`int dbrt;
`(
`
`/“slide brightness mapping “/
`/ ‘stretch or compress brightness mapping ‘/
`/‘enable color bending ’/
`/“disable color bending “/
`/'slide region of color bending ‘/
`/°stretch region of color bending “/
`“/
`
`°/
`/'brightness value for start of rarnp a‘/
`/‘brightness value for end of ramp “/
`/ ‘logical - TRUE if color bend enabled “‘/
`Flower color bending overlap marker “/
`Puppet color bending overlap marker "/
`"/
`/‘primary hue value - [0..360] "/
`/‘primary saturation value - [0..l000] "/
`/'secondary hue value - [0..360] */
`/'secondary saturation value - [0..1000] ’/
`/‘secondaries valid for bend = TRUE only “/
`
`/‘change in brightness - [—999..999] ‘/
`
`/*make sure that the requested change is in range ‘/
`dbrt = max ( dbrt, —999);
`dbrt = min ( dbrt, 999);
`/“check for limiting case ’/
`
`/’Simple case: apply the change ‘/
`
`if ( bend) {
`markl + = dbrt;
`gnarkZ + = dbrt;
`
`/“ End Slide map ’“/
`
`function. For computational efficiency, the luminance
`axis of HLS cylindrical space is represented as an inte
`ger range of [0.1000] (as opposed to the conventional
`representation as a real number in the range [O..l.0]).
`The gray-scale of a frame is linearly related to the inte
`ger range by defaulting a gray-scale value of 128 to a
`luminance value of 500, and corresponding each dis
`crete gray-scale unit to each integer of luminance.
`
`65
`
`Another operation available to the colorist is a rati
`ometric multiplication of the gray-scale by a value (de
`picted in FIGS. 7-8). When the multiplier value is a
`fraction less than one, a ratiometric compression of the
`gray-scale occurs which relates the 256 discrete gray
`scale values to fewer than 256 integer luminance val
`ues. When the multiplier value is greater than unity, a
`“stretch” of the gray-scale occurs which results in a
`‘correspondence of the 256 gray-scale values to a range
`
`Prime Focus Ex 1032-14
`Prime Focus v Legend3D
`IPR2016-01243
`
`

`

`4,984,072
`of luminance greater than 256. These multiplications
`may also occur using either end of the gray-scale as a
`?xed point, yielding 4 variations as depicted in FIGS.
`
`Listing 2
`
`10
`-continued
`
`ramplen '/
`newrlen = (double)( newbrt_..u — newbrt_l);
`/‘new ramplen ‘/
`fmarkl = (double)( markl — brt_l) / rlen;
`l'rnarkZ = (double)( mark2 — brt_l) / rlen;
`markl = (int)( fmarkl ‘ newrlen) + newbrt__l;
`mark2 = (int)( fmark2 ‘ newrlen) + newbrt._l;
`
`}
`
`}
`
`LISTING 2
`
`Listing 2
`void strech_map( db_low, db_up)
`int db_low;
`/‘ change in lower ramp point '/
`int db_up;
`/' change in upper ramp point ‘/
`{
`int newbrt_l, newbrt_u, newmark_l, newmark__u;
`double fmarkl, fmarkZ, rlen, newrlen;
`/" Check that change parameters are within range '/
`db_low = min( db__lowI 999);
`db_low = max( db_low —999);
`db_up = min( db_up, 999);
`db_up = max( db_up, —999);
`/' Apply change to lower ramp point"/
`newbrt_1 = brt_l + db_low;
`newbrt_l = max( newbrt_l, 0);
`newbrt__l = min( newbrt__.l, 999);
`/’ Apply change to upper ramp point ‘/
`newbrt__u = brt_u + db_up;
`newbrt_u = max( newbrt_u, l);
`newbrt_u = min( newbrt.._u, 1000);
`/' Make sure that the new combination is valid '/
`if( (db_up = = 0) && (newbrt > = newbrt__u) )
`newbrt_newbrt_u —- 1;
`else if( newbrt_u <= newbrt_l)
`newbrt_u = newbrt_l + l;
`/’ Reset color-bend markers for new ramp ‘/
`if( bend){
`rlen = (double)( brt_u - brt_l);
`
`/' current
`
`10
`
`15
`
`_20
`
`25
`
`30
`
`/' Move new ramp points into official ramp points “/
`brt_l = newbrt_l;
`brt_u — newbrt_u;
`
`/'End of stretch-map ‘/
`
`‘t
`In depicting re?ected light (either originating from a
`colored source, re?ecting from a colored surface, or
`both), it is necessary to “bend” the color transfer func
`tion. As shown in FIG. 9, the colorist has available the
`option of setting a pair of “bending points” for the func
`tion which determine the nature and extent of the bend
`ing. These points, depicted in FIG. 9, (as points A and
`B) de?ne the endpoints of three line segments (OB, BA,
`and Al). Of these segments, the ?rst and last each have
`associated with them unique, operator-selected values
`of hue, while the AB segment represents a mixture of
`hues and saturations. The locations of points A and B
`are defaulted to 25% and 75% of the current gray-scale
`range, thus providing color mixing over one-half of the
`g