111111
`
`1111111111111111111111111111111111111111111111111111111111111
`US007907793B 1
`
`c12) United States Patent
`San drew
`
`(10) Patent No.:
`(45) Date of Patent:
`
`US 7,907,793 Bl
`Mar.15,2011
`
`(54)
`
`IMAGE SEQUENCE DEPTH ENHANCEMENT
`SYSTEM AND METHOD
`
`(75)
`
`Inventor: Barry Sandrew, Encinitas, CA (US)
`
`(73) Assignee: Legend Films Inc., San Diego, CA (US)
`
`( *) Notice:
`
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 0 days.
`
`EP
`
`4,017,166 A
`4,021,841 A
`4,021,846 A
`4,149,185 A
`
`4/1977 Kent et a!.
`5/1977 Weinger
`511977 Roese
`4/1979 Weinger
`(Continued)
`
`FOREIGN PATENT DOCUMENTS
`0302454
`2/1989
`(Continued)
`
`OTHER PUBLICATIONS
`
`(21) Appl. No.: 12/542,498
`
`(22) Filed:
`
`Aug. 17, 2009
`
`Ohm eta!., An Object-Based System for Stereopscopic Viewpoint
`Synthesis, IEEE transaction on Circuits and Systems for Video Tech(cid:173)
`nology, vol. 7, No.5, Oct. 1997, pp. 801-811.
`
`Related U.S. Application Data
`
`(Continued)
`
`(60) Continuation-in-part of application No. 12/032,969,
`filed on Feb. 18,2008, now Pat. No. 7,577,312, which
`is a continuation of application No. 11/324,815, filed
`on Jan. 4, 2006, now Pat. No. 7,333,670, which is a
`division of application No. 10/450,970, filed as
`application No. PCTIUS02/14192 on May 6, 2002,
`now Pat. No. 7,181,081.
`
`(60) Provisional application No. 60/288,929, filed on May
`4, 2001.
`
`(51)
`
`Int. Cl.
`G06K 9136
`(2006.01)
`(52) U.S. Cl. ........ 382/284; 382/154; 382/307; 348/576;
`348/586
`(58) Field of Classification Search .................. 382/154,
`382/284, 294, 307; 348/576, 586
`See application file for complete search history.
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`3,619,051 A
`1111971 Wright
`1111971 Hope
`3,621,127 A
`3,705,762 A
`12/1972 Ladd eta!.
`3,737,567 A
`6/1973 Kratomi
`1111973 Vlahos eta!.
`3,772,465 A
`3,851,955 A
`12/1974 Kent et al.
`
`Primary Examiner- Yosef Kassa
`(74) Attorney, Agent, or Firm- ARC IP Law, PC; Joseph J.
`Mayo
`
`ABSTRACT
`(57)
`Motion picture scenes to be colorized/depth enhanced
`(2D---;.3D) are broken into separate elements, backgrounds/
`sets or motion/onscreen-action. Background and motion ele(cid:173)
`ments are combined separately into single frame representa(cid:173)
`tions of multiple frames which becomes a visual reference
`database that includes data for all frame offsets used later for
`the computer controlled application of masks within a
`sequence of frames. Each pixel address within the database
`corresponds to a mask/lookup table address within the digital
`frame and X, Y, Z location of subsequent frames. Masks are
`applied to subsequent frames of motion objects based on
`various differentiating image processing methods, including
`automated mask fitting of all masks or single masks in an
`entire frame, bezier and polygon tracing of selected regions
`with edge detected shaping and operator directed detection of
`subsequent regions. Colors and/or depths are automatically
`applied to masks throughout a scene from the composite
`background and to motion objects.
`
`20 Claims, 63 Drawing Sheets
`(6 of 63 Drawing Sheet(s) Filed in Color)
`
`PRIME FOCUS EX 1001-1
`PRIME FOCUS v LEGEND3D
`
`

`
`US 7,907,793 Bl
`Page 2
`
`U.S. PATENT DOCUMENTS
`4,168,885 A
`9/1979 Kent et al.
`111980 Kent et al.
`4,183,633 A
`4,235,503 A
`1111980 Condon
`4,436,369 A
`3/1984 Bukowski
`4,475,104 A
`10/1984 Shen
`10/1985 Ohno
`4,544,247 A
`4,558,359 A
`12/1985 Kuperman et al.
`4,600,919 A
`7/1986 Stern
`4,603,952 A
`8/1986 Sybenga
`4,606,625 A
`8/1986 Geshwind
`4,608,596 A
`8/1986 Williams et a!.
`4,642,676 A
`2/1987 Weinger
`4,645,459 A
`2/1987 Graf et al.
`4,647,965 A
`3/1987 Imsand
`4,697,178 A
`9/1987 Heckel
`4,723,159 A
`2/1988 Imsand
`4,755,870 A
`7/1988 Markle eta!.
`4,809,065 A
`2/1989 Harris eta!.
`4,888,713 A
`12/1989 Falk
`4,903,131 A
`2/1990 Lingemann et a!.
`5/1990 Geshwind et al.
`4,925,294 A
`4,933,670 A
`6/1990 Wislocki
`4,965,844 A
`10/1990 Oka eta!.
`4,984,072 A
`111991 San drew
`5,002,387 A
`3/1991 Baljet eta!.
`5,038,161 A
`8/1991 Ki
`5,050,984 A
`9/1991 Geshwind
`5,093,717 A
`3/1992 San drew
`5,177,474 A
`111993 Kadota
`5,181,181 A
`111993 Glynn
`5,185,852 A
`2/1993 Mayer
`8/1993 Roberts eta!.
`5,237,647 A
`10/1993 Sandrew et a!.
`5,252,953 A
`5,262,856 A
`1111993 Lippman et a!.
`5,328,073 A
`7/1994 Blanding et al.
`8/1994 Obata
`5,341,462 A
`5,347,620 A
`9/1994 Zimmer
`5,402,191 A
`3/1995 Dean eta!.
`6/1995 Sato eta!.
`5,428,721 A
`111996 Lipton
`5,481,321 A
`5,495,576 A
`2/1996 Ritchey
`5,534,915 A
`7/1996 San drew
`5,684,715 A
`1111997 Palmer
`12/1997 Palm
`5,699,444 A
`2/1998 Ado! phi et a!.
`5,717,454 A
`3/1998 Jain eta!.
`5,729,471 A
`5,734,915 A * 3/1998 Roewer ......................... 715/202
`5,739,844 A
`4/1998 Kuwano et al.
`4/1998 Palm
`5,742,291 A
`5,748,199 A
`5/1998 Palm
`5,767,923 A
`6/1998 Coleman
`5,778,108 A
`7/1998 Coleman
`5,784,175 A
`7/1998 Lee
`5,784,176 A
`7/1998 Narita
`5,835,163 A
`1111998 Liou et al.
`5,841,512 A
`1111998 Goodhill
`5,899,861 A
`5/1999 F riemel et a!.
`5,907,364 A
`5/1999 F uruhata et a!.
`5,912,994 A
`6/1999 Norton eta!.
`5,920,360 A
`7/1999 Coleman
`5,929,859 A
`7/1999 Meijers
`8/1999 Tanaka eta!.
`5,940,528 A
`5,959,697 A
`9/1999 Coleman
`5,973,700 A
`10/1999 Taylor eta!.
`10/1999 Kleinberger et a!.
`5,973,831 A
`1111999 Hekmatpour et a!.
`5,982,350 A
`5,990,903 A
`1111999 Donovan
`6,011,581 A
`112000 Swift eta!.
`6,014,473 A
`112000 Hossack eta!.
`6,023,276 A
`212000 Kawai et al.
`6,025,882 A
`212000 Geshwind
`6,031,564 A
`212000 Maetal.
`6,049,628 A
`4/2000 Chen eta!.
`6,056,691 A
`5/2000 Urbano eta!.
`6,067,125 A
`5/2000 May
`7/2000 Urbano eta!.
`6,086,537 A
`6,088,006 A
`7/2000 Tabata
`7/2000 Terrasson
`6,091,421 A
`
`.............. 345/419
`
`8/2000 Hossack eta!.
`6,102,865 A
`8/2000 Starks eta!.
`6,108,005 A
`9/2000 Elenbaas et al.
`6,119,123 A
`10/2000 Hossack eta!.
`6,132,376 A
`10/2000 Moedetal.
`6,141,433 A
`12/2000 Jaszlics eta!.
`6,166,744 A
`3/2001 Kameyama
`6,198,484 B1
`3/2001 Hossack eta!.
`6,201,900 B1
`6,208,348 B1
`3/2001 Kaye
`4/2001 Erland
`6,211,941 B1
`6,215,516 B1
`4/2001 Maet a!.
`4/2001 Hossack eta!.
`6,222,948 B1
`5/2001 Danneels et a!.
`6,226,015 B1
`5/2001 Urbano eta!.
`6,228,030 B1
`6,263,101 B1
`7/2001 Klein
`8/2001 Asente
`6,271,859 B1
`6,337,709 B1
`112002 Y amaashi et a!.
`3/2002 Hossack eta!.
`6,360,027 B1
`4/2002 Hossack eta!.
`6,364,835 B1
`4/2002 Dong eta!.
`6,373,970 B1
`5/2002 Peterson et a!.
`6,390,980 B1
`6,414,678 B1 * 7/2002 Goddard et a!.
`7/2002 Jago
`6,416,477 B1
`6,445,816 B1
`9/2002 Pettigrew
`6,456,340 B1
`9/2002 Margulis
`10/2002 Simpson et a!.
`6,466,205 B2
`6,477,267 B1
`1112002 Richards
`6,492,986 B1
`12/2002 Metaxas et a!.
`6,496,598 B1
`12/2002 Harman
`6,509,926 B1
`112003 Mills eta!.
`6,515,659 B1
`2/2003 Kaye eta!.
`6,535,233 B1
`3/2003 Smith
`6,590,573 B1
`7/2003 Geshwind
`6,611,268 B1 * 8/2003 Szeliski et a!. ................ 345/473
`6,650,339 B1
`1112003 Silva eta!.
`6,677,944 B1
`112004 Yamamoto
`2/2004 Ito et al.
`6,686,591 B2
`6,686,926 B1
`2/2004 Kaye
`6,707,487 B1
`3/2004 Aman eta!.
`6,727,938 B1
`4/2004 Randall
`6,744,461 B1
`6/2004 Wada et al.
`7/2004 Swift et al.
`6,765,568 B2
`6,791,542 B2
`9/2004 Matusik et al.
`6,798,406 B1
`9/2004 Jones eta!.
`6,850,252 B1 * 2/2005 Hofiberg ...................... 7151716
`6,919,892 B1 * 7/2005 Cheiky eta! ................. 345/473
`6,965,379 B2 * 1112005 Lee eta!.
`................... 345/427
`6,985,187 B2 *
`112006 Han eta!. ...................... 348/452
`7,006,881 B1 * 2/2006 Hofiberg eta!. ................ 700/83
`7,035,451 B2
`4/2006 Harman et al.
`7,102,633 B2
`9/2006 Kaye eta!.
`7,116,323 B2
`10/2006 Kaye eta!.
`7,116,324 B2
`10/2006 Kaye eta!.
`2/2007 San drew
`7,181,081 B2
`7,190,496 B2 * 3/2007 Klug eta!. ...................... 359/23
`8/2007 Naske eta!.
`7,254,265 B2
`2/2008 San drew
`7,333,670 B2
`8/2009 Sullivan et a!.
`7,573,475 B2
`7,573,489 B2
`8/2009 Davidson et a!.
`7,577,312 B2
`8/2009 San drew
`2002/0048395 A1
`4/2002 Harman et al.
`2002/0063780 A1
`5/2002 Harman et al.
`2002/0075384 A1
`6/2002 Harman
`2004/0004616 A1
`112004 Konya et al.
`2005/0146521 A1
`7/2005 Kaye eta!.
`2005/0231501 A1
`10/2005 Nitawaki
`2008/0225045 A1
`9/2008 Birtwistle
`2008/0226123 A1
`9/2008 Birtwistle
`2008/0226128 A1
`9/2008 Birtwistle et a!.
`2008/0226160 A1
`9/2008 Birtwistle et a!.
`2008/0226181 A1
`9/2008 Birtwistle et a!.
`2008/0226194 A1
`9/2008 Birtwistle et a!.
`2008/0228449 A1
`9/2008 Birtwistle et a!.
`2009/0116732 A1
`5/2009 Zhou eta!.
`
`EP
`wo
`wo
`
`FOREIGN PATENT DOCUMENTS
`1187494
`3/2002
`wo 97/24000
`7/1997
`wo 99/12127
`3/1999
`
`PRIME FOCUS EX 1001-2
`PRIME FOCUS v LEGEND3D
`
`

`
`US 7,907,793 Bl
`Page 3
`
`wo
`wo
`wo
`wo
`wo
`
`wo 99/30280
`wo 00/79781
`wo 01/01348
`wo 02/13143
`wo 2006/078237
`
`6/1999
`12/2000
`1/2001
`212002
`7/2006
`
`OTHER PUBLICATIONS
`
`Izquierdo et a!., Virtual 3D-View Generation from Stereoscopic
`Video Data, IEEE, Jan. 1998, pp. 1219-1224.
`Kaufman, D., "The Big Picture", Apr. 1998, http://www.xenotech.
`comApr.l998, pp.l-4.
`Hanrahan et a!., "Direct WYSIWYG painting and texturing on 3D
`shapes", Computer Graphics, vol. 24, Issue 4, pp. 215-223. Aug.
`1990.
`Grossman, "Look Ma, No Glasses", Games, Apr. 1992, pp. 12-14.
`Slinker eta!., "The Generation and Animation of Random Dot and
`Random Line Autostereograms", Journal of Imaging Science and
`Technology, vol. 36, No.3, pp. 260-267, May 1992.
`A. Michael Noll, Stereographic Projections by Digital Computer,
`Computers and Automation, vol. 14, No.5 (May 1965), pp. 32-34.
`A. Michael Noll, Computer-Generated Three-Dimensional Movies,
`Computers and Automation, vol. 14, No. 11 (Nov. 1965), pp. 20-23.
`Selsis et a!., Automatic Tracking and 3D Localization of Moving
`Objects by Active Contour Models, Intelligent Vehicles 95 Sympo(cid:173)
`sium, Sep. 1995, pp. 96-100.
`Smeulders et a!., Tracking Nonparameterized Object Contours in
`
`Video, IEEE Transactions on Image Processing, vol. 11, No.9, Sep.
`2002, pp. 1081-1091.
`Murray et a!., Active Tracking, IEEE International Conference on
`Intelligent Robots and Systems, Sep. 1993, pp. 1021-1028.
`Gao et al., Perceptual Motion Tracking from Image Sequences,
`IEEE, Jan. 2001, pp. 389-392.
`Yasushi Mae, eta!., "Object Tracking in Cluttered Background Based
`on Optical Flow and Edges," Proc. 13th Int. Conf. on Pattern Recog(cid:173)
`nition, vol. 1, pp. 196-200, Apr. 1996.
`Di Zhong, Shih-Fu Chang, "AMOS: An Active System for MPEG-4
`Video Object Segmentation," ICIP (2) 8: 647-651, Apr. 1998.
`Hua Zhong, eta!., "Interactive Tracker-A Semi-automatic Video
`Object Tracking and Segmentation System," Microsoft Research
`China, http://research.microsoft.com (Aug. 26, 2003).
`Eric N. Mortensen, William A. Barrett, "Interactive segmentation
`with Intelligent Scissors," Graphical Models and Image Processing,
`v.60 n.5, p. 349-384, Sep. 2002.
`Michael Gleicher, "Image Snapping," SIGGRAPH: 183-190, Jun.
`1995.
`Joseph Weber, eta!., "Rigid Body Segmentation and Shape Descrip(cid:173)
`tion ... ,"IEEE Transactions on Pattern Analysis and Machine Intel(cid:173)
`ligence, vol. 19, No.2, Feb. 1997,pp. 139-143.
`E. N. Mortensen and W. A. Barrett, "Intelligent Scissors for Image
`Composition," Computer Graphics (SIGGRAPH '95), pp. 191-198,
`Los Angeles, CA, Aug. 1995.
`* cited by examiner
`
`PRIME FOCUS EX 1001-3
`PRIME FOCUS v LEGEND3D
`
`

`
`U.S. Patent
`
`Mar.15,2011
`
`Sheet 1 of 63
`
`US 7,907,793 Bl
`
`i4a
`
`Figure 1
`
`PRIME FOCUS EX 1001-4
`PRIME FOCUS v LEGEND3D
`
`

`
`U.S. Patent
`
`Mar.15,2011
`
`Sheet 2 of 63
`
`US 7,907,793 Bl
`
`Figure 2
`
`Isolated
`
`Figure 4
`
`2~~'"'''u< u
`
`u I M~~~l~ct I
`
`~~
`Isolated
`Background
`
`PRIME FOCUS EX 1001-5
`PRIME FOCUS v LEGEND3D
`
`

`
`U.S. Patent
`
`Mar.15,2011
`
`Sheet 3 of 63
`
`US 7,907,793 Bl
`
`Figure 5A , ............................................................................................................................................................................. .
`
`-r:~~ .. v
`fm.!ne [~ _-+
`th<: only
`fi-ame
`fuHv
`1na~;k~d
`·with
`color
`lookun
`tabks'
`
`,. ...... ...
`frarnc ~
`
`numbers I
`
`7
`
`13
`
`19
`
`lO
`
`l6
`
`I 1
`
`l7
`
`... 22··~--23
`
`24
`
`Floaiing
`Tm,J
`Bar
`
`l4
`
`2.0
`
`·--------' .• -;.-;.-;.-;.-;.-;.-;.-;.-;.-;.-;.-;.-;.-;.-;.-;.-;.-;.-;.-;.-;.-;.-;.-;.-;.-;. •• -;.-;.-;.-;.-;.-;.-;.-;.-;.-;.-;.-;.-;.-;.-;.-;.-;.-;.-;.-;.-;.-;. •• -;.-;.-;.-;.-;.-;.-;.-;.-;.-;.-;.-;.-;.-;.-;.-;.-;.-;.-;.-;.-;.-;..;;.-;.-;.-;.-;.-;.-;.-;.-;.-;.-;.-;.-;.-;.-;.-;.-;.-;.-;.-;.-;.-;.-;.-h_-;.-;.-;.-;.-;.-;.-;.-;.-;.-;.-;.-;.-;.-;.-;.-;.-;.-;.-1.-:.-:.-;.-;.-;.-;.-;.-;.-;.-;.-;.-;.-;.-;.-;.-;.-;.-;.~----
`
`Figure 58
`
`PRIME FOCUS EX 1001-6
`PRIME FOCUS v LEGEND3D
`
`

`
`U.S. Patent
`
`Mar.15,2011
`
`Sheet 4 of 63
`
`US 7,907,793 Bl
`
`Figure 6A
`
`Floating: child
`
`windo-vv overlaying: sequential fram:.:s
`
`('ontroJ.s kn· prcvio;:\ving underlying images in real--
`time motion or
`thnnc ;.;tepping to assure
`rmtsk alignment :md
`cuntrol
`
`Frame
`turmbcrs
`
`croo1
`Bar
`
`-------------------------
`
`Figure 6B
`
`PRIME FOCUS EX 1001-7
`PRIME FOCUS v LEGEND3D
`
`

`
`U.S. Patent
`
`Mar.15,2011
`
`Sheet 5 of 63
`
`US 7,907,793 Bl
`
`Figure 7A
`
`Key frame
`masks or
`individual
`111<1sks are
`autOJ:YtaticaJl v
`~
`copied to
`subs:::qucnt
`frarnes
`through 36)
`
`Frame
`numbers
`·················l!t-
`
`3
`
`9
`
`14
`
`15
`
`4
`
`10
`
`16
`
`5
`
`I J
`
`6
`
`12
`
`17
`
`lS
`
`Tnoi
`Bar
`
`23
`21
`24
`22
`20
`19
`1------------------------~--------------------- ------------------------------------------1------------------------------------------
`
`25
`
`:1
`....
`
`26
`
`27
`
`28
`
`29
`
`30
`
`Figure 78
`
`PRIME FOCUS EX 1001-8
`PRIME FOCUS v LEGEND3D
`
`

`
`U.S. Patent
`
`Mar.15,2011
`
`Sheet 6 of 63
`
`US 7,907,793 Bl
`
`Figure 8
`
`PRIME FOCUS EX 1001-9
`PRIME FOCUS v LEGEND3D
`
`

`
`U.S. Patent
`
`Mar.15,2011
`
`Sheet 7 of 63
`
`US 7,907,793 Bl
`
`Figure 9A
`
`Figure 98
`
`PRIME FOCUS EX 1001-10
`PRIME FOCUS v LEGEND3D
`
`

`
`U.S. Patent
`
`Mar.15,2011
`
`Sheet 8 of 63
`
`US 7,907,793 Bl
`
`Figure 10A
`
`Reference Image
`
`Figure 108
`
`Reference Box (xO, yO)
`
`Search Box (;c y)
`
`Figure 10C
`
`Figure 10D
`
`PRIME FOCUS EX 1001-11
`PRIME FOCUS v LEGEND3D
`
`

`
`U.S. Patent
`
`Mar.15,2011
`
`Sheet 9 of 63
`
`US 7,907,793 Bl
`
`Search !mage
`
`Descent
`
`Figure 11 B
`
`Search Box 1
`
`Figure 11A
`
`Figure 11C
`
`PRIME FOCUS EX 1001-12
`PRIME FOCUS v LEGEND3D
`
`

`
`U.S. Patent
`
`Mar.15,2011
`
`Sheet 10 of 63
`
`US 7,907,793 Bl
`
`Figure 12
`
`PRIME FOCUS EX 1001-13
`PRIME FOCUS v LEGEND3D
`
`

`
`U.S. Patent
`
`Mar.15,2011
`
`Sheet 11 of 63
`
`US 7,907,793 Bl
`
`Figure 13
`
`Figure 14
`
`PRIME FOCUS EX 1001-14
`PRIME FOCUS v LEGEND3D
`
`

`
`U.S. Patent
`
`Mar.15,2011
`
`Sheet 12 of 63
`
`US 7,907,793 Bl
`
`Figure 15
`
`Figure 16
`
`PRIME FOCUS EX 1001-15
`PRIME FOCUS v LEGEND3D
`
`

`
`U.S. Patent
`
`Mar.15,2011
`
`Sheet 13 of 63
`
`US 7,907,793 Bl
`
`Figure 17
`
`Figure 18
`
`PRIME FOCUS EX 1001-16
`PRIME FOCUS v LEGEND3D
`
`

`
`U.S. Patent
`
`Mar.15,2011
`
`Sheet 14 of 63
`
`US 7,907,793 Bl
`
`Figure 19
`
`Figure 2 0
`
`PRIME FOCUS EX 1001-17
`PRIME FOCUS v LEGEND3D
`
`

`
`U.S. Patent
`
`Mar.15,2011
`
`Sheet 15 of 63
`
`US 7,907,793 Bl
`
`Figure 21
`
`Figure 22
`
`Figure 23
`
`PRIME FOCUS EX 1001-18
`PRIME FOCUS v LEGEND3D
`
`

`
`U.S. Patent
`
`Mar.15,2011
`
`Sheet 16 of 63
`
`US 7,907,793 Bl
`
`Figure 24
`
`Figure 25
`
`PRIME FOCUS EX 1001-19
`PRIME FOCUS v LEGEND3D
`
`

`
`U.S. Patent
`
`Mar.15,2011
`
`Sheet 17 of 63
`
`US 7,907,793 Bl
`
`Figure 26
`
`PRIME FOCUS EX 1001-20
`PRIME FOCUS v LEGEND3D
`
`

`
`U.S. Patent
`
`Mar.15,2011
`
`Sheet 18 of 63
`
`US 7,907,793 Bl
`
`Figure 27
`
`Figure 28
`
`PRIME FOCUS EX 1001-21
`PRIME FOCUS v LEGEND3D
`
`

`
`U.S. Patent
`
`Mar.15,2011
`
`Sheet 19 of 63
`
`US 7,907,793 Bl
`
`Figure 29
`
`Figure 30
`
`PRIME FOCUS EX 1001-22
`PRIME FOCUS v LEGEND3D
`
`

`
`U.S. Patent
`
`Mar.15,2011
`
`Sheet 20 of 63
`
`US 7,907,793 Bl
`
`Figure 31
`
`Figure 32
`
`PRIME FOCUS EX 1001-23
`PRIME FOCUS v LEGEND3D
`
`

`
`U.S. Patent
`
`Mar.15,2011
`
`Sheet 21 of 63
`
`US 7,907,793 Bl
`
`Figure 33
`
`Figure 34
`
`Figure 35
`
`PRIME FOCUS EX 1001-24
`PRIME FOCUS v LEGEND3D
`
`

`
`U.S. Patent
`
`Mar.15,2011
`
`Sheet 22 of 63
`
`US 7,907,793 Bl
`
`Figure 36
`
`PRIME FOCUS EX 1001-25
`PRIME FOCUS v LEGEND3D
`
`

`
`U.S. Patent
`
`Mar. 15, 2011
`
`Sheet 23 of 63
`
`US 7,907,793 Bl
`
`Figure 37
`
`ask Fit
`
`Initialize region and
`fit-grid parameters
`
`Calculate fit grid
`
`interpolate mask on fit grid
`
`cleanup
`
`PRIME FOCUS EX 1001-26
`PRIME FOCUS v LEGEND3D
`
`

`
`U.S. Patent
`
`Mar.15,2011
`
`Sheet 24 of 63
`
`US 7,907,793 Bl
`
`PRIME FOCUS EX 1001-27
`PRIME FOCUS v LEGEND3D
`
`

`
`U.S. Patent
`
`Mar.15,2011
`
`Sheet 25 of 63
`
`US 7,907,793 Bl
`
`PRIME FOCUS EX 1001-28
`PRIME FOCUS v LEGEND3D
`
`

`
`U.S. Patent
`
`Mar.15,2011
`
`Sheet 26 of 63
`
`US 7,907,793 Bl
`
`PRIME FOCUS EX 1001-29
`PRIME FOCUS v LEGEND3D
`
`

`
`U.S. Patent
`
`Mar.15,2011
`
`Sheet 27 of 63
`
`US 7,907,793 Bl
`
`PRIME FOCUS EX 1001-30
`PRIME FOCUS v LEGEND3D
`
`

`
`U.S. Patent
`
`Mar.15,2011
`
`Sheet 28 of 63
`
`US 7,907,793 Bl
`
`PRIME FOCUS EX 1001-31
`PRIME FOCUS v LEGEND3D
`
`

`
`U.S. Patent
`
`Mar.15,2011
`
`Sheet 29 of 63
`
`US 7,907,793 Bl
`
`PRIME FOCUS EX 1001-32
`PRIME FOCUS v LEGEND3D
`
`

`
`U.S. Patent
`
`Mar.15,2011
`
`Sheet 30 of 63
`
`US 7,907,793 Bl
`
`PRIME FOCUS EX 1001-33
`PRIME FOCUS v LEGEND3D
`
`

`
`U.S. Patent
`
`Mar.15,2011
`
`Sheet 31 of 63
`
`US 7,907,793 Bl
`
`PRIME FOCUS EX 1001-34
`PRIME FOCUS v LEGEND3D
`
`

`
`U.S. Patent
`
`Mar.15,2011
`
`Sheet 32 of 63
`
`US 7,907,793 Bl
`
`PRIME FOCUS EX 1001-35
`PRIME FOCUS v LEGEND3D
`
`

`
`U.S. Patent
`
`Mar.15,2011
`
`Sheet 33 of 63
`
`US 7,907,793 Bl
`
`PRIME FOCUS EX 1001-36
`PRIME FOCUS v LEGEND3D
`
`

`
`U.S. Patent
`
`Mar.15,2011
`
`Sheet 34 of 63
`
`US 7,907,793 Bl
`
`PRIME FOCUS EX 1001-37
`PRIME FOCUS v LEGEND3D
`
`

`
`U.S. Patent
`
`Mar.15,2011
`
`Sheet 35 of 63
`
`US 7,907,793 Bl
`
`FIGURE 42
`
`PRIME FOCUS EX 1001-38
`PRIME FOCUS v LEGEND3D
`
`

`
`FIGURE 43
`
`~
`00
`•
`~
`~
`~
`
`~ = ~
`
`~
`~ :-: ....
`0 ....
`....
`
`~Ul
`N
`
`('D
`('D
`
`rFJ =(cid:173)
`.....
`(.H
`0\
`0 .....
`0\
`(.H
`
`d
`rJl
`-....l
`
`~
`
`\c = -....l
`\C w = """"'
`
`Mask:---·
`All f"
`
`r -::lp;:ar
`r;; Current Mask
`··Transparencies··
`Show ~--
`Paint :--
`
`,._____
`
`Start
`
`Clear
`
`PRIME FOCUS EX 1001-39
`PRIME FOCUS v LEGEND3D
`
`

`
`FIGURE 44
`
`, .. , ,.,.,., , .. ,.,.,.,.'lllli? r ............... l2!~f'laY
`i Color
`See<l Fill
`-------~-~gi~~--J ...... l?.al~~------·
`,,, PaintAIJ
`Undo
`···· FiiiRect
`
`Zoom
`x2
`
`.
`
`x8
`
`r····~;·----
`i Lack Calors Auto Paim
`'
`
`:.- ................ nNn,.••••• '•••••••••
`
`s i
`p
`...... :: ...... ;
`
`~
`00
`•
`~
`~
`~
`
`~ = ~
`
`~
`~ :-: ....
`0 ....
`....
`
`~Ul
`N
`
`('D
`('D
`
`rFJ =(cid:173)
`.....
`(.H
`-....l
`0 .....
`0\
`(.H
`
`d
`rJl
`-....l
`\c = -....l
`\C w = """"'
`
`~
`
`PRIME FOCUS EX 1001-40
`PRIME FOCUS v LEGEND3D
`
`

`
`U.S. Patent
`
`Mar.15,2011
`
`Sheet 38 of 63
`
`US 7,907,793 Bl
`
`PRIME FOCUS EX 1001-41
`PRIME FOCUS v LEGEND3D
`
`

`
`FIGURE 46
`
`__ [}isplay
`Color
`Seed fill
`
`------~!~-~~-----L:::~~~~::::::
`
`Paint All
`FinReGt
`
`Undo
`
`~
`00
`•
`~
`~
`~
`
`~ = ~
`
`~
`~ :-: ....
`0 ....
`....
`
`~Ul
`N
`
`('D
`('D
`
`rFJ =(cid:173)
`......
`(.H
`\0
`0 .....
`0\
`(.H
`
`d
`rJl
`-....l
`\c = -....l
`\C w = """"'
`
`~
`
`PRIME FOCUS EX 1001-42
`PRIME FOCUS v LEGEND3D
`
`

`
`U.S. Patent
`
`Mar.15,2011
`
`Sheet 40 of 63
`
`US 7,907,793 Bl
`
`PRIME FOCUS EX 1001-43
`PRIME FOCUS v LEGEND3D
`
`

`
`FIGURE 48
`
`~
`00
`•
`~
`~
`~
`
`~ = ~
`
`~
`~ :-: ....
`0 ....
`....
`
`~Ul
`N
`
`.j;o.
`
`('D
`('D
`
`rFJ =(cid:173)
`.....
`....
`0 .....
`0\
`(.H
`
`d
`rJl
`-....l
`
`~
`
`\c = -....l
`\C w = """"'
`
`PRIME FOCUS EX 1001-44
`PRIME FOCUS v LEGEND3D
`
`

`
`FIGURE 49
`
`~
`00
`•
`~
`~
`~
`
`~ = ~
`
`~
`~ :-: ....
`0 ....
`....
`
`~Ul
`N
`
`('D
`('D
`
`rFJ =(cid:173)
`.....
`.j;o.
`N
`0 .....
`0\
`(.H
`
`d
`rJl
`-....l
`
`~
`
`\c = -....l
`\C w = """"'
`
`PRIME FOCUS EX 1001-45
`PRIME FOCUS v LEGEND3D
`
`

`
`U.S. Patent
`
`Mar.15,2011
`
`Sheet 43 of 63
`
`US 7,907,793 Bl
`
`FIGURE 50
`
`.'::
`W, A f#
`I
`I
`I
`
`tl'
`
`}~' ')'~' (~
`,;; :;;;
`.. I {
`
`}} )~,~~' ''}
`, .'···~~.·
`.,,,.,.
`f t t
`
`,,,,,
`
`,,,,,,
`) )( '"'''''
`} ',',.,',, ?( ((
`il ( '''~''''
`
`/I ;)( ''''" ,,,,,.
`~'g '-''''''
`'''''''''''''"''''"''''''''''''''
`i } (( ?. •n ::g %'' J 1
`..•.•.•
`~j d < > >< .
`I I /
`I
`/ I
`.;ww '
`i > //
`., •. ',~
`' H •.. ~ }
`I
`I
`I
`,',',',''
`•·',',·,
`i Hi tt
`(
`
`i
`
`''''
`'' "' '"" "'
`',',.,',•
`/ ',',',','
`11.0
`' }~ ~\ ,,,,. 30 OH
`I I '''~'''~'
`'·'·'·.''"
`',',',','
`......
`,,, ,,} 0
`ti t ? ? ?? ,,,,,,,,
`,,,,,,,
`•.. ~~ m
`::: ::: ::: !i ii :t
`
`,;, ,,i '". ,1 ,;1, ;~ ill ir ~ 1,1
`
`/ fu' / ',,',','
`
`;:'. ;,;, ., ;H i~-! ,,,,,.,,
`
`,.,,,.,
`}
`. \ ) ).,.,.,))/ },.,.,.\)\\
`i Hi tt
`ti t i
`i Hi tt
`
`::: :: :::
`
`::: ::: :::
`
`::: :: :::
`
`• •
`
`•
`
`'
`
`: : , : : : : :
`
`PRIME FOCUS EX 1001-46
`PRIME FOCUS v LEGEND3D
`
`

`
`U.S. Patent
`
`Mar.15,2011
`
`Sheet 44 of 63
`
`US 7,907,793 Bl
`
`PRIME FOCUS EX 1001-47
`PRIME FOCUS v LEGEND3D
`
`

`
`U.S. Patent
`
`Mar.15,2011
`
`Sheet 45 of 63
`
`US 7,907,793 Bl
`
`N
`L()
`w
`a::
`:::J
`(.9
`LL
`
`PRIME FOCUS EX 1001-48
`PRIME FOCUS v LEGEND3D
`
`

`
`U.S. Patent
`
`Mar.15,2011
`
`Sheet 46 of 63
`
`US 7,907,793 Bl
`
`PRIME FOCUS EX 1001-49
`PRIME FOCUS v LEGEND3D
`
`

`
`U.S. Patent
`
`Mar.15,2011
`
`Sheet 47 of 63
`
`US 7,907,793 Bl
`
`FIGURE 54
`
`;} n:::,yg'f'
`..
`: iiJ
`"}'.
`:;l;
`
`PRIME FOCUS EX 1001-50
`PRIME FOCUS v LEGEND3D
`
`

`
`U.S. Patent
`
`Mar.15,2011
`
`Sheet 48 of 63
`
`US 7,907,793 Bl
`
`l.()
`l.()
`w
`0::
`::::J
`C)
`LL
`
`PRIME FOCUS EX 1001-51
`PRIME FOCUS v LEGEND3D
`
`

`
`U.S. Patent
`
`Mar.15,2011
`
`Sheet 49 of 63
`
`US 7,907,793 Bl
`
`<.0
`l.()
`w
`r::r:
`:::J
`<.9
`LL
`
`PRIME FOCUS EX 1001-52
`PRIME FOCUS v LEGEND3D
`
`

`
`U.S. Patent
`
`Mar.15,2011
`
`Sheet 50 of 63
`
`US 7,907,793 Bl
`
`PRIME FOCUS EX 1001-53
`PRIME FOCUS v LEGEND3D
`
`

`
`U.S. Patent
`
`Mar.15,2011
`
`Sheet 51 of 63
`
`US 7,907,793 Bl
`
`-
`~···
`. !~::: ::::
`~~~ : ~ y
`i ~ i •••••••
`j ~ ;
`i ·······
`"" Ul
`, w r ) t!
`,.,
`j.
`t t
`~j~···~·~··~·····~··;'''I'•·:I~~~:~~:·;:·:·:·I~~~~J~~~Hi~!~~~·•~·•·•·l ,,, .... mm8;~;,,.} lo
`II ~~':!,;,,
`1°}···········•••·•·. ::: ;! ;:
`' • * I '':I ;;
`, , . ., . .
`. ......... ,. ''I .. ,..
`. 0~'""',; , ·:: .. : ,.
`·::..... , ·', ·',
`
`•••••••
`
`••••••••
`
`··•·•·•
`·•·•·•·•
`·•·•·••
`···~ I , ~~
`
`•''•'•'•'•'•'••
`
`i
`
`•••••••
`··•·•·•
`
`j
`
`;
`,,,,., •. ,
`
`,
`
`oi,
`
`l!
`
`•••••••
`
`FIGURE 58
`
`.. ;::::::
`
`:;:;:;::
`
`:;:;::;
`
`,: ;:: ~i
`';
`"' ·', ·',
`
`1:.
`
`lili
`'~~~~~~~~
`
`PRIME FOCUS EX 1001-54
`PRIME FOCUS v LEGEND3D
`
`

`
`U.S. Patent
`
`Mar.15,2011
`
`Sheet 52 of 63
`
`US 7,907,793 Bl
`
`PRIME FOCUS EX 1001-55
`PRIME FOCUS v LEGEND3D
`
`

`
`U.S. Patent
`
`Mar.15,2011
`
`Sheet 53 of 63
`
`US 7,907,793 Bl
`
`PRIME FOCUS EX 1001-56
`PRIME FOCUS v LEGEND3D
`
`

`
`U.S. Patent
`
`Mar.15,2011
`
`Sheet 54 of 63
`
`US 7,907,793 Bl
`
`PRIME FOCUS EX 1001-57
`PRIME FOCUS v LEGEND3D
`
`

`
`U.S. Patent
`
`Mar.15,2011
`
`Sheet 55 of 63
`
`US 7,907,793 Bl
`
`N
`<0
`w
`0:::
`:::J
`<.9
`LL
`
`PRIME FOCUS EX 1001-58
`PRIME FOCUS v LEGEND3D
`
`

`
`U.S. Patent
`
`Mar.15,2011
`
`Sheet 56 of 63
`
`US 7,907,793 Bl
`
`PRIME FOCUS EX 1001-59
`PRIME FOCUS v LEGEND3D
`
`

`
`U.S. Patent
`
`Mar.15,2011
`
`Sheet 57 of 63
`
`US 7,907,793 Bl
`
`PRIME FOCUS EX 1001-60
`PRIME FOCUS v LEGEND3D
`
`

`
`FIGURE 65
`
`~
`00
`•
`~
`~
`~
`
`~ = ~
`
`~
`~ :-: ....
`0 ....
`....
`
`~Ul
`N
`
`('D
`('D
`
`rFJ =(cid:173)
`.....
`Ul
`QO
`0 .....
`0\
`(.H
`
`d
`rJl
`-....l
`
`~
`
`\c = -....l
`\C w = """"'
`
`PRIME FOCUS EX 1001-61
`PRIME FOCUS v LEGEND3D
`
`

`
`U.S. Patent
`
`Mar.15,2011
`
`Sheet 59 of 63
`
`US 7,907,793 Bl
`
`FIGURE 66
`
`PRIME FOCUS EX 1001-62
`PRIME FOCUS v LEGEND3D
`
`

`
`U.S. Patent
`
`Mar.15,2011
`
`Sheet 60 of 63
`
`US 7,907,793 Bl
`
`F!GURE 67
`
`PRIME FOCUS EX 1001-63
`PRIME FOCUS v LEGEND3D
`
`

`
`U.S. Patent
`
`Mar.15,2011
`
`Sheet 61 of 63
`
`US 7,907,793 Bl
`
`PRIME FOCUS EX 1001-64
`PRIME FOCUS v LEGEND3D
`
`

`
`U.S. Patent
`
`Mar.15,2011
`
`Sheet 62 of 63
`
`US 7,907,793 Bl
`
`PRIME FOCUS EX 1001-65
`PRIME FOCUS v LEGEND3D
`
`

`
`U.S. Patent
`
`Mar.15,2011
`
`Sheet 63 of 63
`
`US 7,907,793 Bl
`
`PRIME FOCUS EX 1001-66
`PRIME FOCUS v LEGEND3D
`
`

`
`US 7,907,793 Bl
`
`1
`IMAGE SEQUENCE DEPTH ENHANCEMENT
`SYSTEM AND METHOD
`
`This application is a continuation in part of U.S. Utility
`patent application Ser. No. 12/032,969, filed 18 Feb. 2008 and
`issued as U.S. Pat. No. 7,577,312, which is a continuation of
`Ser. No. 11/324,815 U.S. Pat. No. 7,333,670 filed4 Jan. 2006,
`which is a divisional of Ser. No. 10/450,970 U.S. Pat. No.
`7,181,081, filed Jun. 18, 2003 which is a national stage entry
`of Patent Cooperation Treaty Application Serial No. PCT/
`US02/14192, filed May 6, 2002, which claims the benefit of
`U.S. Provisional Patent Application 60/288,929 filed May 4,
`2001, the specifications of which are all hereby incorporated
`herein by reference.
`
`BACKGROUND OF THE INVENTION
`
`2
`the creation of wire frame models for objects in images. The
`creation of wire frame models is a large undertaking in terms
`of labor. These systems also do not utilize the underlying
`luminance gray scale of objects in the images to automati(cid:173)
`cally position and correct the shape of the masks of the objects
`to correspond with image feature displacement and distortion
`from one frame to another. Hence, great amounts oflabor are
`required to manually shape masks for applying depth or Z-di(cid:173)
`mension data to the objects. Motion objects that move from
`10 frame to frame thus require a great deal of human interven(cid:173)
`tion. In addition, there are no known solutions for enhancing
`two-dimensional images into three-dimensional images that
`utilize composite backgrounds of multiple images in a frame
`for spreading depth information to background and masked
`15 objects. Hence there is a need for an image sequence depth
`enhancement system and method.
`
`BRIEF SUMMARY OF THE INVENTION
`
`1. Field of the Invention
`One or more embodiments of the invention are related to
`the field of image analysis and image enhancement, (sug- 20
`gested class 382, subclass 254). More particularly, but not by
`way oflimitation, one or more embodiments of the invention
`enable an image sequence depth enhancement system and
`method that allows for the rapid conversion of a sequence of
`two-dimensional images into three-dimensional images.
`2. Description of the Related Art
`Known methods for the colorizing of black and white
`feature films involved the identification of gray scale regions
`within a picture followed by the application of a pre-selected
`color transform or lookup tables for the gray scale within each 30
`region defined by a masking operation covering the extent of
`each selected region and the subsequent application of said
`masked regions from one frame to many subsequent frames.
`The primary difference between U.S. Pat. No. 4,984,072,
`System And Method For Color Image Enhancement, and U.S. 35
`Pat. No. 3,705,762, Method For Converting Black-And(cid:173)
`White Films To Color Films, is the manner by which the
`regions of interest (ROis) are isolated and masked, how that
`information is transferred to subsequent frames and how that
`mask information is modified to conform with changes in the 40
`underlying image data. In the U.S. Pat. No. 4,984,072 system,
`the region is masked by an operator via a one-bit painted
`overlay and operator manipulated using a digital paintbrush
`method frame by frame to match the movement. In the U.S.
`Pat. No. 3,705,762 process, each region is outlined or rota- 45
`scoped by an operator using vector polygons, which are then
`adjusted frame by frame by the operator, to create animated
`masked ROis.
`In both systems the color transform lookup tables and
`regions selected are applied and modified manually to each
`frame in succession to compensate for changes in the image
`data which the operator detects visually. All changes and
`movement of the underlying luminance gray scale is subjec(cid:173)
`tively detected by the operator and the masks are sequentially
`corrected manually by the use of an interface device such as
`a mouse for moving or adjusting mask shapes to compensate
`for the detected movement. In all cases the underlying gray
`scale is a passive recipient of the mask containing pre-se(cid:173)
`lected color transforms with all modifications of the mask
`under operator detection and modification. In these prior
`inventions the mask information does not contain any infor(cid:173)
`mation specific to the underlying luminance gray scale and
`therefore no automatic position and shape correction of the
`mask to correspond with image feature displacement and
`distortion from one frame to another is possible.
`Existing systems that are utilized to convert two-dimen(cid:173)
`sional images to three-dimensional images generally require
`
`Embodiments of the invention classifY scenes to be col-
`orized and/or converted from two-dimensional to three-di(cid:173)
`mensional into movies into two separate categories; either
`background elements (i.e. sets and foreground elements that
`are stationary) or motion elements (e.g., actors, automobiles,
`25 etc) that move throughout the scene. These background ele(cid:173)
`ments and motion elements are treated separately in this
`invention similar to the mauner in which traditional anima(cid:173)
`tion is produced.
`Motion Elements: The motion elements are displayed as a
`series of sequential tiled frame sets or thumbnail images
`complete with background elements. The motion elements
`are masked in a key frame using a multitude of operator
`interface tools common to paint systems as well as unique
`tools such as relative bimodal thresholding in which masks
`are applied selectively to contiguous light or dark areas bifur(cid:173)
`cated by a cursor brush. After the key frame is fully designed
`and masked, all mask information from the key frame is then
`applied to all frames in the display-using mask fitting tech(cid:173)
`niques that include:
`1. Automatic mask fitting using Fast Fourier Transform and
`Gradient Decent Calculations based on luminance and pat(cid:173)
`tern matching which references the same masked area of the
`key frame followed by all prior subsequent frames in succes(cid:173)
`Sion.
`2. Bezier curve animation with edge detection as an auto(cid:173)
`matic animation guide
`3. Polygon animation with edge detection as an automatic
`animation guide
`In another embodiment of this invention, these background
`50 elements and motion elements are combined separately into
`single frame representations of multiple frames, as tiled
`frame sets or as a single frame composite of all elements (i.e.,
`including both motion and backgrounds/foregrounds) that
`then becomes a visual reference database for the computer
`55 controlled application of masks within a sequence composed
`of a multiplicity of frames. Each pixel address within the
`reference visual database corresponds to mask/lookup table
`address within the digital frame and X, Y, Z location of sub(cid:173)
`sequent "raw" frames that were used to create the reference
`60 visual database. Masks are applied to subsequent frames
`based on various differentiating image processing methods
`such as edge detection combined with pattern recognition and
`other sub-mask analysis, aided by operator segmented
`regions of interest from reference objects or frames, and
`65 operator directed detection of subsequent regions corre(cid:173)
`sponding to the original region of interest. In this mauner, the
`gray scale actively determines the location and shape of each
`
`PRIME FOCUS EX 1001-67
`PRIME FOCUS v LEGEND3D
`
`

`
`US 7,907,793 Bl
`
`4
`applied to all frames used to create the single representation
`of the background only where there is no pre-existing mask.
`This allows the designer to include as much detail as desired
`including air brushing of mask information and other mask
`application techniques that provide maximum creative
`expression. Once the background color design is completed
`the mask information is transferred automatically to all the
`frames that were used to create the single composited image.
`For depth projects, the distance from the camera to each item
`10 in the composite frame is automatically transferred to all the
`frames that were used to create the single composited image.
`By shifting masked background objects horizontally more or
`less, there precise depth is thus set in a secondary viewpoint
`frame that corresponds to each frame in the scene. Areas
`15 where no image data exists for a second viewpoint may be
`marked in one or more embodiments of the invention using a
`user defined color that allows for the creation missing data to
`ensure that no artifacts occur during the two-dimension to
`three-dimension conversion process. Any technique known
`20 may be utilized in embodiments of the invention to cover
`areas in the background where unknown data exists, i.e., that
`may not be borrowed from another scene/frame for example.
`After assigning depths to objects in