(12) United States Patent
`Sandrew
`
`(10) Patent N0.:
`(45) Date of Patent:
`
`US 7,577,312 B2
`Aug. 18,2009
`
`US007577312B2
`
`IMAGE SEQUENCE ENHANCEMENT
`SYSTEM AND METHOD
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`(54)
`
`(75)
`
`(73)
`
`Inventor: Barry B. SandreW, Encinitas, CA (US)
`
`Assignee: Legend Films Inc., San Diego, CA (US)
`
`(*)
`
`Notice:
`
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 0 days.
`
`5,940,528 A *
`6,337,709 B1 *
`
`8/1999 Tanaka et a1. ............. .. 382/151
`1/2002 Yamaashi et a1. ......... .. 348/143
`
`6,509,926 B1 *
`6,686,591 B2 *
`
`348/143
`1/2003 Mills et a1. ...... ..
`2/2004 Ito et a1. ................... .. 250/311
`
`6,727,938 B1* 4/2004 Randall .................... .. 348/143
`6,744,461 B1 *
`6/2004 Wada et a1. ............... .. 348/143
`
`(21)
`
`Appl. No.: 12/032,969
`
`(22)
`
`Filed:
`
`Feb. 18, 2008
`
`* cited by examiner
`
`Primary ExamineriYosef Kassa
`(74) Attorney, Agent, or FirmiDalina LaW Group, PC.
`
`(65)
`
`(60)
`
`(60)
`
`(51)
`
`(52)
`
`(58)
`
`Prior Publication Data
`
`(57)
`
`ABSTRACT
`
`US 2008/0205783 A1
`
`Aug. 28, 2008
`
`Related US. Application Data
`
`Continuation of application No. 11/324,815, ?led on
`Jan. 4, 2006, noW Pat. No. 7,333,670, Which is a divi
`sion of application No. 10/450,970, ?led as application
`No. PCT/US02/14192 on May 6, 2002, noW Pat. No.
`7,181,081.
`Provisional application No. 60/288,929, ?led on May
`4, 2001.
`
`Int. Cl.
`(2006.01)
`G06K 9/36
`US. Cl. ..................... .. 382/282; 382/212; 382/213;
`358/517; 358/537
`Field of Classi?cation Search ............... .. 382/282,
`382/283, 295, 212, 213, 305; 358/517, 537,
`358/538, 540
`See application ?le for complete search history.
`
`Motion picture scenes to be coloriZed are broken into separate
`elements, backgrounds/ sets or motion/onscreen-action.
`Background and motion elements are combined separately
`into single frame representations of multiple frames Which
`becomes a visual reference database that includes data for all
`frame offsets used later for the computer controlled applica
`tion of masks Within a sequence of frames. Each pixel address
`Within the database corresponds to a mask/lookup table
`address Within the digital frame and X, Y, Z location of sub
`sequent frames. Masks are applied to subsequent frames of
`motion objects based on various differentiating image pro
`cessing methods, including automated mask ?tting of all
`masks or single masks in an entire frame, beZier and polygon
`tracing of selected regions With edge detected shaping and
`operator directed detection of subsequent regions. The gray
`scale actively determines the mask and corresponding color
`lookup that is applied in a keying fashion Within regions of
`interest.
`
`4 Claims, 34 Drawing Sheets
`
`Calculate ?lgrid
`
`image line y=ysbart
`
`image column x=xslarl
`
`Get (i,|) for
`Fit Grid Cell (x, y)
`
`bilinear inter polation
`(FitGrldA, FltGridB,
`FitGridC, FltGridD
`
`cleanup. relurn
`
`PRIME FOCUS EX 1010-1
`PRIME FOCUS v LEGEND3D
`
`

`
`US. Patent
`
`Aug. 18, 2009
`
`Sheet 1 0f 34
`
`US 7,577,312 B2
`
`10\
`
`l
`
`Figure 1
`
`PRIME FOCUS EX 1010-2
`PRIME FOCUS v LEGEND3D
`
`

`
`US. Patent
`
`Aug. 18,2009
`
`Sheet 2 0f 34
`
`US 7,577,312 B2
`
`16
`
`Isolated
`Background
`Background
`
`Mask
`
`'
`
`M-Object
`
`Background
`Mask
`M-Object
`Masks
`
`PRIME FOCUS EX 1010-3
`PRIME FOCUS v LEGEND3D
`
`

`
`US. Patent
`
`Aug. 18, 2009
`
`Sheet 3 0f 34
`
`US 7,577,312 B2
`
`Figure 5A
`
`Key
`frame is -——+ ]
`the only
`frame
`fully
`masked
`with
`color
`lookup
`tables
`
`7
`
`13
`
`Frame
`numbers
`
`19
`
`25
`
`3'
`
`Figure 5B
`
`3
`
`4
`
`2
`
`8
`
`9
`
`14
`
`I5
`
`2O
`
`21
`
`10
`
`16
`
`22
`
`S
`
`ll
`
`17
`
`23
`
`6
`
`12
`
`18
`
`24
`
`3O
`
`26
`
`27
`
`28
`
`29
`
`32
`
`33
`
`34
`
`35
`
`36
`
`lrgw
`
`F
`
`w
`
`I»:
`
`‘11-21
`
`mm W
`
`WW?" my
`wan-v f-qvlzmLq
`
`?gjwfill
`
`____l
`
`PRIME FOCUS EX 1010-4
`PRIME FOCUS v LEGEND3D
`
`

`
`US. Patent
`
`Aug. 18,2009
`
`Sheet 4 0f 34
`
`US 7,577,312 B2
`
`Figure 6A
`
`Controls for previewing underlying images in real
`time motion or single frame stepping to assure
`mask alignment and quality control
`
`Tool
`Bar
`
`Frame
`numbers
`
`51
`
`52
`
`55
`
`54
`
`i5
`
`50
`
`Figure 6B
`
`PRIME FOCUS EX 1010-5
`PRIME FOCUS v LEGEND3D
`
`

`
`US. Patent
`
`Aug. 18, 2009
`
`Sheet 5 0f 34
`
`US 7,577,312 B2
`
`Key frame
`masks or
`individual
`masks are
`automatically
`copied to
`subsequent
`frames (2
`through 36)
`
`Frame
`numbers
`
`1
`
`7
`
`13
`
`19
`
`2
`
`8
`
`14
`
`20
`
`3
`
`9
`
`15
`
`21
`
`4
`
`1 0
`
`16
`
`22
`
`5
`
`1 1
`
`17
`
`23
`
`6
`
`12
`
`18
`
`24
`
`25
`
`26
`
`27
`
`28
`
`29
`
`30
`
`3]
`
`32
`
`33
`
`34
`
`35
`
`36
`
`Figure 7A
`
`Floating
`T001
`B211"
`
`Figure 78
`
`PRIME FOCUS EX 1010-6
`PRIME FOCUS v LEGEND3D
`
`

`
`US. Patent
`
`Aug. 18, 2009
`
`Sheet 6 0f 34
`
`US 7,577,312 B2
`
`PRIME FOCUS EX 1010-7
`PRIME FOCUS v LEGEND3D
`
`

`
`US. Patent
`
`Aug. 18, 2009
`
`Sheet 7 0f 34
`
`US 7,577,312 B2
`
`Figure 9A
`
`Figure 9B
`
`PRIME FOCUS EX 1010-8
`PRIME FOCUS v LEGEND3D
`
`

`
`US. Patent
`
`Aug. 18, 2009
`
`Sheet 8 0f 34
`
`US 7,577,312 B2
`
`Figure 10A
`
`Reference Image
`
`Figure 108
`
`Reference Box (x0, y0)
`
`Search Box (x, y)
`
`Figure 10C
`
`Figure 10D
`
`PRIME FOCUS EX 1010-9
`PRIME FOCUS v LEGEND3D
`
`

`
`US. Patent
`
`Aug. 18,2009
`
`Sheet 9 0f 34
`
`US 7,577,312 B2
`
`Search Image Gradient Descent
`
`Figure 118
`
`Search Box 1
`
`Search Box 2
`
`Figure 11A
`
`Figure 11C
`
`PRIME FOCUS EX 1010-10
`PRIME FOCUS v LEGEND3D
`
`

`
`US. Patent
`
`Aug. 18, 2009
`
`Sheet 10 0f 34
`
`US 7,577,312 B2
`
`Error v alue (x, yl-djr)‘
`
`Error value (x-dx, y)
`
`Error value (x, y-dy)
`
`Figure 12
`
`PRIME FOCUS EX 1010-11
`PRIME FOCUS v LEGEND3D
`
`

`
`US. Patent
`
`Aug. 18, 2009
`
`Sheet 11 0134
`
`US 7,577,312 B2
`
`Figure 13
`
`Figure 14
`
`PRIME FOCUS EX 1010-12
`PRIME FOCUS v LEGEND3D
`
`

`
`US. Patent
`
`Aug. 18, 2009
`
`Sheet 12 0f 34
`
`US 7,577,312 B2
`
`5353115522"?
`
`v
`
`7
`
`.
`
`.
`
`..
`
`_.
`
`V
`
`Figure 16
`
`PRIME FOCUS EX 1010-13
`PRIME FOCUS v LEGEND3D
`
`

`
`US. Patent
`
`Aug. 18, 2009
`
`Sheet 13 0f 34
`
`US 7,577,312 B2
`
`Figure 17
`
`"Jill
`
`Figure 18
`
`PRIME FOCUS EX 1010-14
`PRIME FOCUS v LEGEND3D
`
`

`
`US. Patent
`
`Aug. 18, 2009
`
`Sheet 14 0f 34
`
`US 7,577,312 B2
`
`Figure 19
`
`
`
`rd, l1 1.. v . - l
`
`
`
` n 91.4.. . Tu. . ll-llll ll-IJII...
`
`Figure 2 0
`
`PRIME FOCUS EX 1010-15
`PRIME FOCUS v LEGEND3D
`
`

`
`US. Patent
`
`Aug. 18, 2009
`
`Sheet 15 0f 34
`
`US 7,577,312 B2
`
`Figure 21
`
`Figure 22
`
`Figure 23
`
`PRIME FOCUS EX 1010-16
`PRIME FOCUS v LEGEND3D
`
`

`
`US. Patent
`
`Aug. 18,2009
`
`Sheet 16 0f 34
`
`US 7,577,312 B2
`
`PRIME FOCUS EX 1010-17
`PRIME FOCUS v LEGEND3D
`
`

`
`US. Patent
`
`Aug. 18, 2009
`
`Sheet 17 0f 34
`
`US 7,577,312 B2
`
`Figure 26
`
`PRIME FOCUS EX 1010-18
`PRIME FOCUS v LEGEND3D
`
`

`
`US. Patent
`
`Aug. 18, 2009
`
`Sheet 18 0f 34
`
`US 7,577,312 B2
`
`
`
`WWW WEN WP ,
`
`Figure 28
`
`PRIME FOCUS EX 1010-19
`PRIME FOCUS v LEGEND3D
`
`

`
`US. Patent
`
`Aug. 18, 2009
`
`Sheet 19 0f 34
`
`US 7,577,312 B2
`
`
`
`
`
`. 0.145651‘... iii ‘4.0-1341
`
`Figure 29
`
`Figure 30
`
`PRIME FOCUS EX 1010-20
`PRIME FOCUS v LEGEND3D
`
`

`
`PRIME FOCUS EX 1010-21
`PRIME FOCUS v LEGEND3D
`
`

`
`PRIME FOCUS EX 1010-22
`PRIME FOCUS v LEGEND3D
`
`

`
`PRIME FOCUS EX 1010-23
`PRIME FOCUS v LEGEND3D
`
`

`
`PRIME FOCUS EX 1010-24
`PRIME FOCUS v LEGEND3D
`
`

`
`PRIME FOCUS EX 1010-25
`PRIME FOCUS v LEGEND3D
`
`

`
`PRIME FOCUS EX 1010-26
`PRIME FOCUS v LEGEND3D
`
`

`
`PRIME FOCUS EX 1010-27
`PRIME FOCUS v LEGEND3D
`
`

`
`PRIME FOCUS EX 1010-28
`PRIME FOCUS v LEGEND3D
`
`

`
`PRIME FOCUS EX 1010-29
`PRIME FOCUS v LEGEND3D
`
`

`
`PRIME FOCUS EX 1010-30
`PRIME FOCUS v LEGEND3D
`
`

`
`PRIME FOCUS EX 1010-31
`PRIME FOCUS v LEGEND3D
`
`

`
`PRIME FOCUS EX 1010-32
`PRIME FOCUS v LEGEND3D
`
`

`
`PRIME FOCUS EX 1010-33
`PRIME FOCUS v LEGEND3D
`
`

`
`PRIME FOCUS EX 1010-34
`PRIME FOCUS v LEGEND3D
`
`

`
`PRIME FOCUS EX 1010-35
`PRIME FOCUS v LEGEND3D
`
`

`
`PRIME FOCUS EX 1010-36
`PRIME FOCUS v LEGEND3D
`
`

`
`PRIME FOCUS EX 1010-37
`PRIME FOCUS v LEGEND3D
`
`

`
`PRIME FOCUS EX 1010-38
`PRIME FOCUS v LEGEND3D
`
`

`
`PRIME FOCUS EX 1010-39
`PRIME FOCUS v LEGEND3D
`
`

`
`PRIME FOCUS EX 1010-40
`PRIME FOCUS v LEGEND3D
`
`

`
`PRIME FOCUS EX 1010-41
`PRIME FOCUS v LEGEND3D
`
`

`
`PRIME FOCUS EX 1010-42
`PRIME FOCUS v LEGEND3D
`
`

`
`PRIME FOCUS EX 1010-43
`PRIME FOCUS v LEGEND3D
`
`

`
`PRIME FOCUS EX 1010-44
`PRIME FOCUS v LEGEND3D
`
`

`
`PRIME FOCUS EX 1010-45
`PRIME FOCUS v LEGEND3D
`
`

`
`PRIME FOCUS EX 1010-46
`PRIME FOCUS v LEGEND3D
`
`

`
`PRIME FOCUS EX 1010-47
`PRIME FOCUS v LEGEND3D
`
`

`
`PRIME FOCUS EX 1010-48
`PRIME FOCUS v LEGEND3D