US008897596B1
`
`(12) United States Patent
`Passmore et al.
`
`(10) Patent N0.:
`(45) Date of Patent:
`
`US 8,897,596 B1
`Nov. 25, 2014
`
`(54) SYSTEM AND METHOD FOR RAPID IMAGE
`SEQUENCE DEPTH ENHANCEMENT WITH
`TRANSLUCENT ELEMENTS
`
`(75) Inventors: Charles Passmore, San Diego, CA
`(US); Tony Baldridge, San Diego, CA
`(US); Barry SandreW, San Diego, CA
`(Us)
`
`(73) Assignee: Legend3D, Inc., Carlsbad, CA (U S)
`
`( * ) Notice:
`
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 465 days.
`
`(21) App1.No.: 13/367,316
`
`(22) Filed:
`
`Feb. 6, 2012
`
`Related US. Application Data
`(60) Continuation-in-part of application No. 13/029,862,
`?led on Feb. 17, 2011, now Pat. No. 8,385,684, which
`(Continued)
`
`(51) Int. Cl.
`G06K 9/00
`(52) US. Cl.
`USPC ......... .. 382/284; 382/154; 382/275; 382/291;
`358/326; 358/327
`
`(2006.01)
`
`(58) Field of Classi?cation Search
`USPC ....... .. 382/275, 278, 284, 291; 358/326, 3.27
`See application ?le for complete search history.
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`2,593,925 A
`2,799,722 A
`
`4/ 1952 Sheldon
`7/ 1957 Neugebauer
`(Continued)
`
`FOREIGN PATENT DOCUMENTS
`
`DE
`EP
`
`003444353
`6/1986
`0302454
`2/1989
`(Continued)
`OTHER PUBLICATIONS
`
`Daniel L. Symmes, Three-Dimensional Image, Microsoft Encarta
`Online Encyclopedia (printed May 28, 2008 and of record, now
`indicated by the website indicated on the document to be discontin
`ued:
`http://encarta.msn.com/texti7615 84746i0/Three-Dimen
`sionaliImagehtm).
`
`(Continued)
`
`Primary Examiner * Yosef Kassa
`(74) Attorney, Agent, or Firm * ARC IP Law, PC; Joseph J.
`Mayo
`
`ABSTRACT
`(57)
`Motion picture scenes to be colorized/depth enhanced
`(2D—>3D) are broken into separate elements, backgrounds/
`sets or motion/onscreen-action. Background and motion ele
`ments are combined into composite frame which becomes a
`visual reference database that includes data for all frame
`offsets used later for the computer controlled application of
`masks within a sequence of frames. Masks are applied to
`subsequent frames of motion objects based on various differ
`entiating image processing methods, including automated
`mask ?tting/reshaping. Colors and/or depths are automati
`cally applied to masks throughout a scene from the composite
`background, translucent, motion objects. Areas never
`exposed by motion or foreground objects in a series of images
`may be partially or fully realistically drawn or rendered and
`applied to the occluded areas of the background and then
`automatically applied throughout the images to generate of
`minimal artifact or artifact-free secondary viewpoints when
`translating foreground objects horizontally during 2DQ3D
`conversion.
`
`20 Claims, 82 Drawing Sheets
`(69 of 82 Drawing Sheet(s) Filed in Color)
`
`Subsequent
`1:5:
`
`5
`
`Prime Focus Ex 1020-1
`Prime Focus v Legend3D
`IPR2016-01243
`
`

`

`US 8,897,596 B1
`Page 2
`
`(60)
`
`(56)
`
`Related US. Application Data
`
`a continuation-in-part of application No.
`is
`12/976,970, ?led on Dec. 22, 2010, now Pat. No.
`8,401,336, Which is a continuation-in-part of
`application No. 12/913,614, ?led on Oct. 27, 2010,
`now Pat.
`No.
`8,396,328,
`Which is
`a
`continuation-in-part of application No. 12/542,498,
`?led onAug. 17, 2009, noW Pat. No. 7,907,793, Which
`is
`a continuation-in-part of application No.
`12/032,969, ?led on Feb. 18, 2008, noW Pat. No.
`7,577,312, Which is a continuation of application No.
`11/324,815, ?led on Jan. 4, 2006, noW Pat. No.
`7,333,670, Which is a division of application No.
`10/450,970, ?led as application No. PCT/USO2/14192
`on May 6, 2002, noW Pat. No. 7,181,081.
`Provisional application No. 60/288,929, ?led on May
`4, 2001.
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`8/1957 Giacoletto
`2,804,500 A
`2/1959 Bechley
`2,874,212 A
`4/1959 Schaper
`2,883,763 A
`3/1961 Fine et a1.
`2,974,190 A
`3,005,042 A 10/1961 Horsley
`3,258,528 A
`6/1966 Oppenheimer
`3,486,242 A 12/1969 Aronson
`3,551,589 A 12/1970 Moskoviz
`3,558,811 A
`1/1971 Montevecchio et a1.
`3,560,644 A
`2/1971 Petrocelliet al.
`3,595,987 A
`7/1971 Vlahos
`3,603,962 A
`9/1971 Lechner
`3,612,755 A 10/1971 Tadlock
`3,617,626 A 11/1971 Bluth et a1.
`3,619,051 A 11/1971 Wright
`3,647,942 A
`3/1972 Siegel
`3,673,317 A
`6/1972 Newellet a1.
`3,705,762 A 12/1972 Ladd et a1.
`3,706,841 A 12/1972 Novak
`3,710,011 A
`1/1973 Altemus etal.
`3,731,995 A
`5/1973 Reiffel
`3,742,125 A
`6/1973 Siegel
`3,761,607 A
`9/1973 Hanseman
`3,769,458 A 10/1973 Driskell
`3,770,884 A 11/1973 Curran et al.
`3,770,885 A 11/1973 Curran etal.
`3,772,465 A 11/1973 Vlahos et a1.
`3,784,736 A
`1/1974 Novak
`3,848,856 A 11/1974 Reeber et al.
`3,971,068 A
`7/1976 Gerhardt et a1.
`3,972,067 A
`7/1976 Peters
`4,017,166 A
`4/1977 Kentet a1.
`4,021,841 A
`5/1977 Weinger
`4,054,904 A 10/1977 Saitoh et a1.
`4,149,185 A
`4/1979 Weinger
`4,168,885 A
`9/1979 Kentet a1.
`4,183,046 A
`1/1980 Dalke et a1.
`4,189,743 A
`2/1980 Schure etal.
`4,189,744 A
`2/1980 Stern
`4,258,385 A
`3/1981 Greenberg et a1.
`4,318,121 A
`3/1982 Taite etal.
`4,329,710 A
`5/1982 Taylor
`4,334,240 A
`6/1982 Franklin
`4,475,104 A 10/1984 Shen etal.
`4,549,172 A 10/1985 Welk
`4,563,703 A
`1/1986 Taylor et al.
`4,590,511 A
`5/1986 Bocchiet a1.
`4,600,919 A
`7/1986 Stern
`4,606,625 A
`8/1986 Geshwind
`4,608,596 A
`8/1986 Williams
`4,617,592 A 10/1986 MacDonald
`4,642,676 A
`2/1987 Weinger
`
`6,314,211 >>>>>>>>>>>>>>>>>>>>D>D>D>D>>>>>>>>D>>>>>>>>D>D>D>D>D>>>>>>>>>>>>>>>>>>>>>>>>>
`
`4,645,459
`4,647,965
`4,694,329
`4,721,951
`4,723,159
`4,725,879
`4,755,870
`4,758,908
`4,760,390
`4,774,583
`4,794,382
`4,827,255
`4,847,689
`4,862,256
`4,888,713
`4,903,131
`4,918,624
`4,952,051
`4,965,844
`4,984,072
`5,038,161
`5,050,984
`5,093,717
`5,177,474
`5,252,953
`5,262,856
`5,328,073
`5,347,620
`5,481,321
`5,528,655
`5,534,915
`5,684,715
`5,717,454
`5,729,471
`5,742,291
`5,748,199
`5,767,923
`5,778,108
`5,784,175
`5,784,176
`5,825,997
`5,835,163
`5,841,512
`5,867,169
`5,880,788
`5,899,861
`5,907,364
`5,912,994
`5,920,360
`5,940,528
`5,959,697
`5,973,831
`5,982,350
`5,990,903
`6,005,582
`6,014,473
`6,023,276
`6,025,882
`6,049,628
`6,056,691
`6,067,125
`6,086,537
`6,102,865
`6,118,584
`6,119,123
`6,132,376
`6,141,433
`6,166,744
`6,173,328
`6,184,937
`6,201,900
`6,211,941
`6,215,516
`6,222,948
`6,226,015
`6,228,030
`6,263,101
`6,271,859
`
`2/1987
`3/1987
`9/1987
`1/1988
`2/1988
`2/1988
`7/1988
`7/1988
`7/1988
`9/1988
`12/1988
`5/1989
`7/1989
`8/1989
`12/1989
`2/1990
`4/1990
`8/1990
`10/1990
`1/1991
`8/1991
`9/1991
`3/1992
`1/1993
`10/1993
`11/1993
`7/1994
`9/1994
`1/1996
`6/1996
`7/1996
`11/1997
`2/1998
`3/1998
`4/1998
`5/1998
`6/1998
`7/1998
`7/1998
`7/1998
`10/1998
`11/1998
`11/1998
`2/1999
`3/1999
`5/1999
`5/1999
`6/1999
`7/1999
`8/1999
`9/1999
`10/1999
`11/1999
`11/1999
`12/1999
`1/2000
`2/2000
`2/2000
`4/2000
`5/2000
`5/2000
`7/2000
`8/2000
`9/2000
`9/2000
`10/2000
`10/2000
`12/2000
`1/2001
`2/2001
`3/2001
`4/2001
`4/2001
`4/2001
`5/2001
`5/2001
`7/2001
`8/2001
`11/2001
`
`Graf et a1.
`Imsand
`Belmares-Sarabia et al.
`Holler
`Imsand
`Eide et a1.
`Markle et a1.
`James
`Maine et a1.
`Kellar et al.
`Lai et al.
`Ishii
`Yamamoto et al.
`Markle et a1.
`Falk
`Lingemann et al.
`Moore et al.
`Lovell et al.
`Oka
`Sandrew
`Ki
`Geshwind
`Sandrew
`Kadota
`Sandrew et al.
`Lippman et a1.
`Blanding et a1.
`Zimmer
`Lipton
`Umetani et al.
`Sandrew
`Palmer
`Adolphi et a1.
`Jain et al.
`Palm
`Palm
`Coleman
`Coleman
`Lee
`Narita
`Yamada et al.
`Liou et al.
`Goodhill
`Prater
`Bregler
`Friemel et al.
`Furuhata et al.
`Norton et al.
`Coleman
`Tanaka et al.
`Coleman
`Kleinberger et al.
`Hekmatpour et al.
`Donovan
`Gabriel et al.
`Hossack et al.
`Kawai et a1.
`Geshwind
`Chen et al.
`Urbano et al.
`May
`Urbano et al.
`Hossack et al.
`Van Berkel et al. ........ .. 359/463
`Elenbaas et a1.
`Hossack et al.
`Moed et al.
`Jaszlics et al.
`Sato
`Williams et al.
`Hossack et al.
`Erland
`Ma et al.
`Hossack et al.
`Danneels et al.
`Urbano et al.
`Klein
`Asente
`Kim et a1.
`
`Prime Focus Ex 1020-2
`Prime Focus v Legend3D
`IPR2016-01243
`
`

`

`US 8,897,596 B1
`Page 3
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`6,337,709
`6,360,027
`6,364,835
`6,373,970
`6,390,980
`6,416,477
`6,445,816
`6,509,926
`6,606,166
`6,611,268
`6,650,339
`6,662,357
`6,665,798
`6,686,591
`6,707,487
`6,727,938
`6,737,957
`6,744,461
`6,765,568
`6,813,602
`6,847,737
`6,859,523
`6,964,009
`6,973,434
`7,000,223
`7,006,881
`7,027,054
`7,032,177
`7,079,075
`7,116,324
`7,117,231
`7,136,075
`7,181,081
`7,260,274
`7,272,265
`7,298,094
`7,308,139
`7,333,519
`7,333,670
`7,343,082
`7,461,002
`7,512,262
`7,519,990
`7,532,225
`7,538,768
`7,542,034
`7,576,332
`7,577,312
`7,610,155
`7,624,337
`7,630,533
`7,663,689
`7,680,653
`7,772,532
`7,894,633
`8,085,339
`8,217,931
`2001/0025267
`2001/0051913
`2002/0049778
`2003/0018608
`2003/0046656
`2003/0089777
`2003/0093790
`2003/0097423
`2003/0154299
`2003/0177024
`2004/0004616
`2004/0062439
`2004/0189796
`2004/0258089
`2005/0088515
`2005/0188297
`2005/0207623
`2006/0061583
`
`B1
`B1
`B1
`B1
`B1
`B1
`B1
`B1
`B1
`B1
`B1
`B1
`B1
`B2
`B1
`B1
`B1
`B1
`B2
`B2
`B1
`B1
`B2
`B2
`B1
`B1
`B1
`B2
`B1
`B2
`B2
`B1
`B2
`B2
`B2
`B2
`B2
`B2
`B2
`B2
`B2
`B2
`B1
`B2
`B2
`B2
`B2
`B2
`B2
`B2
`B2
`B2
`B2
`B2
`B1
`B2
`B2
`A1
`A1
`A1
`A1
`A1
`A1
`A1
`A1
`A1
`A1
`A1
`A1
`A1
`A1
`A1
`A1
`A1
`A1
`
`1/2002
`3/2002
`4/2002
`4/2002
`5/2002
`7/2002
`9/2002
`1/2003
`8/2003
`8/2003
`11/2003
`12/2003
`12/2003
`2/2004
`3/2004
`4/2004
`5/2004
`6/2004
`7/2004
`11/2004
`1/2005
`2/2005
`11/2005
`12/2005
`2/2006
`2/2006
`4/2006
`4/2006
`7/2006
`10/2006
`10/2006
`11/2006
`2/2007
`8/2007
`9/2007
`11/2007
`12/2007
`2/2008
`2/2008
`3/2008
`12/2008
`3/2009
`4/2009
`5/2009
`5/2009
`6/2009
`8/2009
`8/2009
`10/2009
`11/2009
`12/2009
`2/2010
`3/2010
`8/2010
`2/2011
`12/2011
`7/2012
`9/2001
`12/2001
`4/2002
`1/2003
`3/2003
`4/2003
`5/2003
`5/2003
`8/2003
`9/2003
`1/2004
`4/2004
`9/2004
`12/2004
`4/2005
`8/2005
`9/2005
`3/2006
`
`............ .. 382/284
`
`Yamaashi et al.
`Hossack et al.
`Hossack et al.
`Dong et al.
`Peterson et al.
`Jago
`Pettigrew
`Mills et al.
`Knoll
`Szeliski et al.
`Silva et al.
`Bowman-Amueh
`McNally et al.
`Ito et al.
`Aman et al.
`Randall
`Petrovic et al.
`Wada et al.
`Swift et al.
`Thyssen
`Kouri et al.
`Jilk et al.
`Samaniego et al.
`Miller
`Knutson et al.
`Hoffberg et al.
`Cheiky et al.
`Novak et al.
`Hoffberg et al.
`Kaye et al.
`Fischer et al.
`Hamburg
`Sandrew
`Sawhney et al.
`Kouri et al.
`Yui
`Wentland et al.
`Sullivan et al.
`Sandrew
`Cote et al.
`Crockett et al.
`Criminisi et al.
`Xie
`Fukushima et al.
`Kiyokawa et al. .......... .. 345/427
`Spooner et al.
`Britten
`Sandrew
`Timmis et al.
`Sull et al.
`Ruth et al.
`Marks ......................... .. 348/370
`Yeldener
`Olsen et al.
`Harman
`Marks ......................... .. 348/362
`Lowe et al.
`Janiszewski
`Vashistha et al.
`Bell
`Rice
`Saxena
`Or-Bach
`Logan et al.
`Ozawa et al.
`Hamilton
`Tsuchida
`Konya et al.
`Cahill et al.
`Ho et al.
`Derechin et al.
`Geng
`Knight et al.
`Liu et al.
`Spooner et al.
`
`2006/0274905 A1 12/2006 Lindahl et al.
`2007/0052807 A1
`3/2007 Zhou et al.
`2007/0260634 A1 11/2007 Makela et al.
`2007/0296721 A1 12/2007 Chang et al.
`2008/0044155 A1
`2/2008 Kuspa
`2008/0079851 A1
`4/2008 Stanger et al.
`2008/0147917 A1
`6/2008 Lees et al.
`2008/0162577 A1
`7/2008 Fukuda et al.
`2008/0181486 A1
`7/2008 Spooner et al.
`2008/0225040 A1
`9/2008 Simmons et al.
`2008/0225042 A1
`9/2008 Birtwistle et al.
`2008/0225045 A1
`9/2008 Birtwistle et al.
`2008/0225059 A1
`9/2008 Lowe et al.
`2008/0226123 A1
`9/2008 Birtwistle et al.
`2008/0226128 A1
`9/2008 Birtwistle et al.
`2008/0226160 A1
`9/2008 Birtwistle et al.
`2008/0226181 A1
`9/2008 Birtwistle et al.
`2008/0226194 A1
`9/2008 Birtwistle et al.
`2008/0227075 A1
`9/2008 Poor et al.
`2008/ 0228449 A1
`9/2008 Birtwistle et al.
`2008/0246759 A1 10/2008 Summers
`2008/0246836 A1 10/2008 Lowe et al.
`2008/0259073 A1 10/2008 Lowe et al.
`2009/0002368 A1
`1/2009 Vitikainen et al.
`2009/0033741 A1
`2/2009 Oh et al.
`2009/0219383 A1
`9/2009 Passmore
`2009/0256903 A1 10/2009 Spooner et al.
`2011/0050864 A1
`3/2011 Bond
`2011/0074784 A1
`3/2011 Turner
`2011/0169827 A1
`7/2011 Spooner et al.
`2011/0169914 A1
`7/2011 Lowe et al.
`2011/0188773 A1
`8/2011 Wei et al.
`2011/0227917 A1
`9/2011 Lowe et al.
`2011/0273531 A1 11/2011 Ito et al.
`2012/0032948 A1
`2/2012 Lowe et al.
`2012/0087570 A1
`4/2012 Seo et al.
`2012/0102435 A1
`4/2012 Han et al.
`2012/0188334 A1
`7/2012 Fortin et al.
`2012/0274626 A1 11/2012 Hsieh
`2012/0281906 A1 11/2012 Appia
`2013/0051659 A1
`2/2013 Yamamoto
`
`FOREIGN PATENT DOCUMENTS
`
`JP
`JP
`JP
`KR
`KR
`SU
`WO
`WO
`WO
`WO
`
`3/1985
`60-52190
`2/2003
`2003046982
`7/2004
`2004-207985
`2/2012
`20120095059
`11/2013
`20130061289
`1192168 A 11/1982
`2008/075276
`6/2008
`2011/029209
`3/2011
`2012016600
`9/2012
`2013084234
`6/2013
`
`OTHER PUBLICATIONS
`
`Lenny Lipton, Foundations of the Stereo-Scopic Cinema a Study in
`Depth, 1982, Van Nostrand Reinhold Company.
`US. District Court, C.D. California, IMAXv. In-Three, No. 05 CV
`1795, 2005, Partial Testimony, Expert: David Geshwind, WestLaw
`2005, WL 3940224 (C.D.Cal.), 8 pages.
`US. District Court, C.D. California, IMAX Corporation and Three
`Dimensional Media Group, Ltd., v.1n-Three, Ina, Partial Testimony,
`Expert: Samuel Zhou, Ph.D., No. CV 05-1795 FMC(Mcx), Jul. 19,
`2005, 2005 WL 3940223 (C.D.Cal.), 6 pages.
`US. District Court, C.D. California, IMAXv. In-Three. No. 06 CV
`1795. Jul. 21, 2005, Partial Testimony, Expert: Samuel Zhou, Ph.D.,
`2005 WL 3940225 (C.D.Cal.), 21 pages.
`US. District Court, C.D. California, Western Division, IMAX Cor
`poration, and Three-Dimensional Media Group, Ltd. v. In-Three,
`Inc., No. CV05 1795 FMC (Mcx). Jul. 18, 2005. Declaration of
`Barbara Frederiksen in Support of In-Three, Inc.’s Opposition to
`Plaintiffs’ Motion for Preliminary Injunction, 2005 WL 5434580
`(C.D.Cal.), 13 pages.
`Lenny Lipton, “Foundations of the Stereo-Scopic Cinema, a Study in
`Depth” With and Appendix on 3D Television, 325 pages.
`
`Prime Focus Ex 1020-3
`Prime Focus v Legend3D
`IPR2016-01243
`
`

`

`US 8,897,596 B1
`Page 4
`
`(56)
`
`References Cited
`
`OTHER PUBLICATIONS
`
`Interpolation (from Wikipedia encyclopedia, article pp. 1-6).
`Optical Reader (from Wikipedia encyclopedia, article p. 1).
`Declaration of Steven K. Feiner, Exhibit A, 10 pages.
`Declaration of Michael F. Chou, Exhibit B, 12 pages.
`Declaration of John Marchioro, Exhibit C, 3 pages.
`Exhibit 1 to Declaration of John Marchioro, Revised translation of
`portions of Japanese Patent Document No. 60-52190 to Hiromae, 3
`pages.
`Nell et al., “Stereographic Projections by Digital Computer”, Com
`puters and Automation for May 1965, pp. 32-34.
`Nell, “Computer-Generated Three-Dimensional Movies” Comput
`ers and Automation for Nov. 1965, pp. 20-23.
`US. Patent and Trademark Of?ce, Before the Board of Patent
`Appeals and Interferences, Ex Parte T hree-Dimensional Media
`Group, Ltd, Appeal 2009-004087, Reexamination Control No.
`90/007,578, US Patent No. 4,925,294, Decision on Appeal, 88 pages.
`Of?ce Action for EPO Patent Application No. 02 734 203.9 dated
`Sep. 12, 2006. (4 pages).
`Of?ce Action for AUS Patent Application No. 2002305387 dated
`Mar. 9, 2007. (2 pages).
`Of?ce Action for EPO Patent Application No. 02 734 203.9 dated
`Oct. 7, 2010. (5 pages).
`First Examination Report for Indian Patent Application No. 01779/
`DELNP/2003 dated Mar. 2004. (4 pages).
`International Search Report Dated Jun. 13, 2003 (3 pages).
`Declaration of Barbara Frederiksen in Support of In-Three, Inc’s
`Opposition to Plaintiffs Motion for Preliminary Injunction, Aug. 1,
`2005,1MAXCorporation et alv. In-Three, Inc., Case No. CV05 1795
`FMC (Mcx). (25 pages).
`USPTO, Board of Patent Appeals and Interferences, Decision on
`Appeal dated Jul. 30, 2010, Ex parte Three-Dimensional Media
`Group, Ltd, Appeal 2009-004087, Reexamination Control No.
`90/007,578, US Patent 4,925,294. (88 pages).
`Of?ce Action for Canadian Patent Application No. 2,446,150 dated
`Oct. 8, 2010 (6 pages).
`Of?ce Action for Canadian Patent Application No. 2,446,150 dated
`Jun. 13, 2011 (4 pages).
`Ohm et al., An Object-Based System for Stereopscopic Viewpoint
`Synthesis, IEEE transaction on Circuits and Systems for Video Tech
`nology, vol. 7, No. 5, Oct. 1997, pp. 801-811.
`Izquierdo et al., Virtual 3D-VieW Generation from Stereoscopic
`Video Data, IEEE, Jan. 1998, pp. 1219-1224.
`Kaufman, D., “The Big Picture”, Apr. 1998, http://WWW.xenotech.
`com Apr. 1998, pp. 1-4.
`Hanrahan et al., “Direct WYSIWYG painting and texturing on 3D
`shapes”, Computer Graphics, vol. 24, Issue 4, pp. 215-223. Aug.
`1990.
`Grossman, “Look Ma, No Glasses”, Games, Apr. 1992, pp. 12-14.
`Slinker et al., “The Generation and Animation of Random Dot and
`Random Line Autostereograms”, Journal of Imaging Science and
`Technology, vol. 36, No. 3, pp. 260-267, May 1992.
`A. Michael Noll, Stereographic Projections by Digital Computer,
`Computers and Automation, vol. 14, No. 5 (May 1965), pp. 32-34.
`A. Michael Noll, Computer-Generated Three-Dimensional Movies,
`Computers and Automation, vol. 14, No. 11 (Nov. 1965), pp. 20-23.
`Selsis et al., Automatic Tracking and 3D Localization of Moving
`Objects by Active Contour Models, Intelligent Vehicles 95 Sympo
`sium, Sep. 1995, pp. 96-100.
`Smeulders et al., Tracking Nonparameterized Object Contours in
`Video, IEEE Transactions on Image Processing, vol. 11, No. 9, Sep.
`2002, pp. 1081-1091.
`Murray et al., Active Tracking, IEEE International Conference on
`Intelligent Robots and Systems, Sep. 1993, pp. 1021-1028.
`Gao et al., Perceptual Motion Tracking from Image Sequences,
`IEEE, Jan. 2001, pp. 389-392.
`
`Yasushi Mae, et al., “Object Tracking in Cluttered Background Based
`on Optical Flow and Edges,” Proc. 13th Int. Conf. on Pattern Recog
`nition, vol. 1, pp. 196-200,Apr. 1996.
`Di Zhong, Shih-Fu Chang, “AMOS: An Active System for MPEG-4
`Video Object Segmentation,” ICIP (2) 8: 647-651, Apr. 1998.
`Hua Zhong, et al., “Interactive TrackeriA Semi-automatic Video
`Object Tracking and Segmentation System,” Microsoft Research
`China, http://research.microsoft.com (Aug. 26, 2003).
`Eric N. Mortensen, William A. Barrett, “Interactive segmentation
`With Intelligent Scissors,” Graphical Models and Image Processing,
`v.60 n.5, p. 349-384, Sep. 2002.
`Michael Gleicher, “Image Snapping,” SIGGRAPH: 183-190, Jun.
`1995.
`Joseph Weber, et al., “Rigid Body Segmentation and Shape Descrip
`tion .
`.
`. ,” IEEE Transactions on Pattern Analysis and Machine
`Intelligence, vol. 19, No. 2, Feb. 1997,pp. 139-143.
`E. N. Mortensen and W. A. Barrett, “Intelligent Scissors for Image
`Composition,” Computer Graphics (SIGGRAPH ’95), pp. 191-198,
`Los Angeles, CA, Aug. 1995.
`International Search Report Issued for PCT/US2013/072208, dated
`Feb. 27, 2014, 6 pages.
`International Search Report and Written Opinion issued for PCT/
`US2013/072447, dated Mar. 13, 2014, 6 pages.
`International Search Report dated May 10, 2012, 8 pages.
`Machine translation of JP Patent No. 2004-207985, dated Jul. 22,
`2008, 34 pages.
`“Nintendo DSi Uses Camera Face Tracking to Create 3D Mirages”,
`retrieved from WWW.Gizmodo.com on Mar. 18, 2013, 3 pages.
`IPER, Mar. 29, 2007, PCT/US2005/014348, 5 pages.
`IPER, Oct. 5, 2013, PCT/US2011/058182, 6 pages.
`International Search Report, Jun. 13, 2003, PCT/U S02/ 14192, 4
`pages.
`Partial Testimony, Expert: Samuel Zhou, Ph.D., 2005 WL 3940225
`(C.D.Cal.), Jul. 21,2005, 21 pages.
`PCT ISR, Feb. 27, 2007, PCT/US2005/014348, 8 pages.
`PCT ISR, Sep. 11, 2007, PCT/US07/62515, 9 pages.
`CA Of?ce Action, Dec. 28, 2011, Appl No. 2,446,150, 4 pages.
`PCT ISR, Nov. 14, 2007, PCT/US07/62515, 24 pages.
`PCT IPRP, Jul. 4, 2013, PCT/US2011/067024, 5 pages.
`European Of?ce Action dated Jun. 26, 2013, for EP Appl. No.
`027342039 on Jul. 22, 2013, 5 pages.
`International Search Report fro PCT Application No. PCT/US2011/
`067024, dated Aug. 22, 2012, 10 pages.
`Lenny Lipton, “Foundations of the Stereo-Scopic Cinema, a Study in
`Depth” With and Appendix on 3D Television, 325 pages, May 1978.
`Interpolation (from Wikipedia encyclopedia, article pp. 1-6),
`retrieved from Internet URL:http://en.Wikipedia.org/Wiki/Interpola
`tion on Jun. 5, 2008.
`Optical Reader (from Wikipedia encyclopedia, article p. 1), retrieved
`from Internet URL:http://en.Wikipedia.org/Wiki/Opticalireader on
`Jun. 5, 2008.
`Declaration of Steven K. Feiner, Exhibit A, 10 pages, Nov. 2, 2007.
`Declaration of Michael F. Chou, Exhibit B, 12 pages, Nov. 2, 2007.
`Declaration of John Marchioro, Exhibit C, 3 pages, Nov. 2, 2007.
`Exhibit 1 to Declaration of John Marchioro, Revised translation of
`portions of Japanese Patent Document No. 60-52190 to Hiromae, 3
`pages, Nov. 2, 2007.
`US. Patent and Trademark Of?ce, Before the Board of Patent
`Appeals and Interferences, Ex Parte T hree-Dimensional Media
`Group, Ltd, Appeal 2009-004087, Reexamination Control No.
`90/007,578, US Patent No. 4,925,294, Decision on Appeal, 88 pages,
`Jul. 30, 2010.
`Tam et al., “3D-TV Content Generation: 2D-To-3D Conversion”,
`ICME 2006, p. 1868-1872.
`Harman et al. “Rapid 2D to 3D Conversion”, The Reporter, vol. 17,
`No. 1, Feb. 2002, 12 pages.
`Legend Films, “System and Method for Conversion of Sequences of
`Two-Dimensional Medical Images to Three-Dimensional Images”
`Sep. 12,2013, 7 pages.
`
`* cited by examiner
`
`Prime Focus Ex 1020-4
`Prime Focus v Legend3D
`IPR2016-01243
`
`

`

`U S. Patent
`
`NOV. 25, 2014
`
`Sheet 1 0f 82
`
`US 8,897,596 B1
`
`égum
`
`23..
`
`Prime Focus Ex 1020-5
`Prime Focus v Legend3D
`IPR2016-01243
`
`

`

`US. Patent
`
`Nov. 25, 2014
`
`Sheet 2 0f 82
`
`US 8,897,596 B1
`
`12%
`
`isolated
`Background
`
`Mask
`
`Key M—Object
`
`(“MW—12
`
`Fig i3 re &
`
`Subsequent
`MmObject
`
`n
`[119
`
`Background
`
`:
`
`<3 ,,,,,,,, ,.
`
`Mask
`
`M*Qb_§80t
`Masks
`
`isoiated
`' Background
`
`Prime Focus Ex 1020-6
`Prime Focus v Legend3D
`IPR2016-01243
`
`

`

`US. Patent
`
`Nov. 25, 2014
`
`Sheet 3 0f 82
`
`US 8 897 596 B1
`
`9
`
`9
`
`Figure 5A
`
`Key
`frame it? __@
`the uniy
`{Fame
`fully '
`n1_a.skeci
`
`'''' u
`
`lookup
`tables
`
`Fram? “4"
`numbers
`
`7
`
`23
`‘
`
`u}
`
`‘
`
`25
`
`31
`
`2
`
`g
`
`14
`
`20
`
`26
`
`3
`
`4
`
`9
`
`E5
`
`21
`
`27
`
`10
`
`16
`
`22
`
`23
`
`5
`
`I E
`
`17
`
`23
`
`29
`
`6
`
`E2
`
`{q
`
`24
`
`30
`
`32
`
`33
`
`34
`
`35
`
`36
`
`j
`g
`3
`§
`g
`g Floating
`3
`TOO}
`g
`Bar
`g
`3
`g
`3
`3 3
`
`s
`
`Prime Focus Ex 1020-7
`Prime Focus v Legend3D
`IPR2016-01243
`
`

`

`US. Patent
`
`Nov. 25, 2014
`
`Sheet 4 0f 82
`
`US 8,897,596 B1
`
`8
`
`Figure $ A
`
`Floating child preview Win?ow overlaying sequential ?ames
`
`Commis for previewing undei'lying images in reai-
`time motixm or single frame: stcpping to assure
`mask alignment and quality control
`
`T001
`Bar
`
`Frame
`numbers
`
`Figura 8 B
`
`Prime Focus Ex 1020-8
`Prime Focus v Legend3D
`IPR2016-01243
`
`

`

`US. Patent
`
`Nov. 25, 2014
`
`Sheet 5 0f 82
`
`US 8,897,596 B1
`
`Key ?ame
`masks or
`individual
`are
`automaticaiiy
`copicd to
`subsequent
`frames (2
`through 36)
`
`_
`,
`Frame _____ __
`
`numbars
`
`I
`
`7
`
`q
`‘”
`
`8
`
`3
`
`9
`
`A
`I:
`
`M
`
`-
`1:
`
`4
`
`5
`
`6
`
`10
`
`16
`
`1 i
`
`.,
`1 I
`
`32
`
`28
`
`19
`
`.20
`
`2,
`
`22
`
`25
`
`24
`
`'
`23
`
`26
`
`27
`
`28
`
`29
`
`30
`
`3 l
`
`1 Z
`
`3 ®
`
`34
`
`35
`
`3b
`
`?gure "Y A
`
`Figure 7 B
`
`Prime Focus Ex 1020-9
`Prime Focus v Legend3D
`IPR2016-01243
`
`

`

`US. Patent
`
`Nov. 25, 2014
`
`Sheet 6 0f 82
`
`US 8,897,596 B1
`
`Figure 8
`
`Prime Focus Ex 1020-10
`Prime Focus v Legend3D
`IPR2016-01243
`
`

`

`US. Patent
`
`Nov. 25, 2014
`
`Sheet 7 0f 82
`
`US 8,897,596 B1
`
`?gure QA
`
`Figure QB
`
`Prime Focus Ex 1020-11
`Prime Focus v Legend3D
`IPR2016-01243
`
`

`

`U S. Patent
`
`NOV. 25, 2014
`
`Sheet 8 0f 82
`
`US 8,897,596 B1
`
`Figure WA
`
`Reference image
`
`?gure 168
`F
`
`é.
`
`Rafe
`re
`
`9 Box(
`
`)
`
`Figure 166
`
`Figure 198
`
`Prime Focus Ex 1020-12
`Prime Focus v Legend3D
`IPR2016-01243
`
`

`

`U S. Patent
`
`Nov. 25, 2014
`
`Sheet 9 0f 82
`
`US 8,897,596 B1
`
`age Gra
`dieni Descent
`Search im
`
`Figure i’iB
`
`Search Box 1
`
`Figure 11A
`
`Figure 113
`
`Prime Focus Ex 1020-13
`Prime Focus v Legend3D
`IPR2016-01243
`
`

`

`U S. Patent
`
`Nov. 25, 2014
`
`Sheet 10 0f 82
`
`US 8,897,596 B1
`
`Figure 12
`
`Prime Focus Ex 1020-14
`Prime Focus v Legend3D
`IPR2016-01243
`
`

`

`U S. Patent
`
`Nov. 25, 2014
`
`Sheet 11 0f 82
`
`US 8,897,596 B1
`
`Figure ’54
`
`Prime Focus Ex 1020-15
`Prime Focus v Legend3D
`IPR2016-01243
`
`

`

`U S. Patent
`
`Nov. 25, 2014
`
`Sheet 12 0f 82
`
`US 8,897,596 B1
`
`?gure 16
`
`Prime Focus Ex 1020-16
`Prime Focus v Legend3D
`IPR2016-01243
`
`

`

`US. Patent
`
`N v. 25, 2014
`
`Sheet 13 0f 82
`
`US 8,897,596 B1
`
`Fiwmw
`
`Figure ‘18
`
`Prime Focus Ex 1020-17
`Prime Focus v Legend3D
`IPR2016-01243
`
`

`

`U S. Patent
`
`NOV. 25, 2014
`
`Sheet 14 0f 82
`
`US 8,897,596 B1
`
`w. H
`
`Figure W
`
`Figum 2 G
`
`Prime Focus Ex 1020-18
`Prime Focus v Legend3D
`IPR2016-01243
`
`

`

`US. Patent
`
`Nov. 25, 2014
`
`Sheet 15 0f 82
`
`US 8,897,596 B1
`
`Figure 22
`
`Figura 23
`
`Prime Focus Ex 1020-19
`Prime Focus v Legend3D
`IPR2016-01243
`
`

`

`U S. Patent
`
`NOV. 25, 2014
`
`Sheet 16 0f 82
`
`US 8,897,596 B1
`
`Figure 24
`
`Figure 25
`
`Prime Focus Ex 1020-20
`Prime Focus v Legend3D
`IPR2016-01243
`
`

`

`Prime Focus Ex 1020-21
`Prime Focus v Legend3D
`IPR2016-01243
`
`

`

`Prime Focus Ex 1020-22
`Prime Focus v Legend3D
`IPR2016-01243
`
`

`

`Prime Focus Ex 1020-23
`Prime Focus v Legend3D
`IPR2016-01243
`
`

`

`Prime Focus Ex 1020-24
`Prime Focus v Legend3D
`IPR2016-01243
`
`

`

`Prime Focus Ex 1020-25
`Prime Focus v Legend3D
`IPR2016-01243
`
`

`

`Prime Focus Ex 1020-26
`Prime Focus v Legend3D
`IPR2016-01243
`
`

`

`Prime Focus Ex 1020-27
`Prime Focus v Legend3D
`IPR2016-01243
`
`

`

`Prime Focus Ex 1020-28
`Prime Focus v Legend3D
`IPR2016-01243
`
`

`

`Prime Focus Ex 1020-29
`Prime Focus v Legend3D
`IPR2016-01243
`
`

`

`Prime Focus Ex 1020-30
`Prime Focus v Legend3D
`IPR2016-01243
`
`

`

`Prime Focus Ex 1020-31
`Prime Focus v Legend3D
`IPR2016-01243
`
`

`

`Prime Focus Ex 1020-32
`Prime Focus v Legend3D
`IPR2016-01243
`
`

`

`Prime Focus Ex 1020-33
`Prime Focus v Legend3D
`IPR2016-01243
`
`

`

`Prime Focus Ex 1020-34
`Prime Focus v Legend3D
`IPR2016-01243
`
`

`

`Prime Focus Ex 1020-35
`Prime Focus v Legend3D
`IPR2016-01243
`
`

`

`Prime Focus Ex 1020-36
`Prime Focus v Legend3D
`IPR2016-01243
`
`

`

`Prime Focus Ex 1020-37
`Prime Focus v Legend3D
`IPR2016-01243
`
`

`

`Prime Focus Ex 1020-38
`Prime Focus v Legend3D
`IPR2016-01243
`
`

`

`Prime Focus Ex 1020-39
`Prime Focus v Legend3D
`IPR2016-01243
`
`

`

`Prime Focus Ex 1020-40
`Prime Focus v Legend3D
`IPR2016-01243
`
`

`

`Prime Focus Ex 1020-41
`Prime Focus v Legend3D
`IPR2016-01243
`
`

`

`Prime Focus Ex 1020-42
`Prime Focus v Legend3D
`IPR2016-01243
`
`

`

`Prime Focus Ex 1020-43
`Prime Focus v Legend3D
`IPR2016-01243
`
`

`

`Prime Focus Ex 1020-44
`Prime Focus v Legend3D
`IPR2016-01243
`
`

`

`Prime Focus Ex 1020-45
`Prime Focus v Legend3D
`IPR2016-01243
`
`

`

`Prime Focus Ex 1020-46
`Prime Focus v Legend3D
`IPR2016-01243
`
`

`

`Prime Focus Ex 1020-47
`Prime Focus v Legend3D
`IPR2016-01243
`
`

`

`Prime Focus Ex 1020-48
`Prime Focus v Legend3D
`IPR2016-01243
`
`

`

`Prime Focus Ex 1020-49
`Prime Focus v Legend3D
`IPR2016-01243
`
`

`

`Prime Focus Ex 1020-50
`Prime Focus v Legend3D
`IPR2016-01243
`
`

`

`Prime Focus Ex 1020-51
`Prime Focus v Legend3D
`IPR2016-01243
`
`

`

`Prime Focus Ex 1020-52
`Prime Focus v Legend3D
`IPR2016-01243
`
`

`

`Prime Focus Ex 1020-53
`Prime Focus v Legend3D
`IPR2016-01243
`
`

`

`Prime Focus Ex 1020-54
`Prime Focus v Legend3D
`IPR2016-01243
`
`

`

`Prime Focus Ex 1020-55
`Prime Focus v Legend3D
`IPR2016-01243
`
`

`

`Prime Focus Ex 1020-56
`Prime Focus v Legend3D
`IPR2016-01243
`
`

`

`Prime Focus Ex 1020-57
`Prime Focus v Legend3D
`IPR2016-01243
`
`

`

`Prime Focus Ex 1020-58
`Prime Focus v Legend3D
`IPR2016-01243
`
`

`

`Prime Focus Ex 1020-59
`Prime Focus v Legend3D
`IPR2016-01243
`
`

`

`Prime Focus Ex 1020-60
`Prime Focus v Legend3D
`IPR2016-01243
`
`

`

`Prime Focus Ex 1020-61
`Prime Focus v Legend3D
`IPR2016-01243
`
`

`

`Prime Focus Ex 1020-62
`Prime Focus v Legend3D
`IPR2016-01243
`
`

`

`Prime Focus Ex 1020-63
`Prime Focus v Legend3D
`IPR2016-01243
`
`

`

`Prime Focus Ex 1020-64
`Prime Focus v Legend3D
`IPR2016-01243
`
`

`

`Prime Focus Ex 1020-65
`Prime Focus v Legend3D
`IPR2016-01243
`
`

`

`Prime Focus Ex 1020-66
`Prime Focus v Legend3D
`IPR2016-01243
`
`

`

`Prime Focus Ex 1020-67
`Prime Focus v Legend3D
`IPR2016-01243
`
`

`

`Prime Focus Ex 1020-68
`Prime Focus v Legend3D
`IPR2016-01243
`
`

`

`Prime Focus Ex 1020-69
`Prime Focus v Legend3D
`IPR2016-01243
`
`

`

`Prime Focus Ex 1020-70
`Prime Focus v Legend3D
`IPR2016-01243
`
`

`

`Prime Focus Ex 1020-71
`Prime Focus v Legend3D
`IPR2016-01243
`
`

`

`Prime Focus Ex 1020-72
`Prime Focus v Legend3D
`IPR2016-01243
`
`

`

`Prime Focus Ex 1020-73
`Prime Focus v Legend3D
`IPR2016-01243
`
`

`

`Prime Focus Ex 1020-74
`Prime Focus v Legend3D
`IPR2016-01243
`
`

`

`Prime Focus Ex 1020-75
`Prime Focus v Legend3D
`IPR2016-01243
`
`

`

`Prime Focus Ex 1020-76
`Prime Focus v Legend3D
`IPR2016-01243
`
`

`

`Prime Focus Ex 1020-77
`Prime Focus v Legend3D
`IPR2016-01243
`
`

`

`Prime Focus Ex 1020-78
`Prime Focus v Legend3D
`IPR2016-01243
`
`

`

`Prime Focus Ex 1020-79
`Prime Focus v Legend3D
`IPR2016-01243
`
`

`

`Prime Focus Ex 1020-80
`Prime Focus v Legend3D
`IPR2016-01243
`
`

`

`Prime Focus Ex 1020-81
`Prime Focus v Legend3D
`IPR2016-01243
`
`

`

`Prime Focus Ex 1020-82
`Prime Focus v Legend3D
`IPR2016-01243
`
`

`

`Prime Focus Ex 1020-83
`Prime Focus v Legend3D
`IPR2016-01243
`
`

`

`Prime Focus Ex 1020-84
`Prime Focus v Legend3D
`IPR2016-01243
`
`

`

`Prime Focus Ex 1020-85
`Prime Focus v Legend3D
`IPR2016-01243
`
`

`

`Prime Focus Ex 1020-86
`Prime Focus v Legend3D
`IPR2016-01243
`
`

`

`Prime Focus Ex 1020-87
`Prime Focus v Legend3D
`IPR2016-01243
`
`

`

`Prime Focus Ex 1020-88
`Prime Focus v Legend3D
`IPR2016-01243
`
`

`

`Prime Focus Ex 1020-89
`Prime Focus v Legend3D
`IPR2016-01243
`
`

`

`Prime Focus Ex 1020-90
`Prime Focus v Legend3D
`IPR2016-01243
`
`

`

`Prime Focus Ex 1020-91
`Prime Focus v Legend3D
`IPR2016-01243
`
`

`

`Prime Focus Ex 1020-92
`Prime Focus v Legend3D
`IPR2016-01243
`
`

`

`Prime Focus Ex 1020-93
`Prime Focus v Legend3D
`IPR2016-01243
`
`

`

`Prime Focus Ex 1020-94
`Prime Focus v Legend3D
`IPR2016-01243
`
`

`

`Prime Focus Ex 1020-95
`Prime Focus v Legend3D
`IPR2016-01243
`
`

`

`Prime Focus Ex 1020-96
`Prime Focus v Legend3D
`IPR2016-01243
`
`

`

`Prime Focus Ex 1020-97
`Prime Focus v Legend3D
`IPR2016-01243
`
`

`

`Prime Focus Ex 1020-98
`Prime Focus v Legend3D
`IPR2016-01243
`
`

`

`Prime Focus Ex 1020-99
`Prime Focus v Legend3D
`IPR2016-01243
`
`

`

`Prime Focus Ex 1020-100
`Prime Focus v Legend3D
`IPR2016-01243
`
`

`

`Prime Focus Ex 1020-101
`Prime Focus v Legend3D
`IPR2016-01243
`
`

`

`Prime Focus Ex 1020-102
`Prime Focus v Legend3D
`IPR2016-01243
`
`

`

`Prime Focus Ex 1020-103
`Prime Focus v Legend3D
`IPR2016-01243
`
`

`

`Prime Focus Ex 1020-104
`Prime Focus v Legend3D
`IPR2016-01243
`
`

`

`Prime Focus Ex 1020-105
`Prime Focus v Legend3D
`IPR2016-01243
`
`

`

`Prime Focus Ex 1020-106
`Prime Focus v Legend3D
`IPR2016-01243
`
`

`

`Prime Focus Ex 1020-107
`Prime Focus v Legend3D
`IPR2016-01243
`
`

`

`Prime Focus Ex 1020-108
`Prime Focus v Legend3D
`IPR2016-01243
`
`

`

`Prime Focus Ex 1020-109
`Prime Focus v Legend3D
`IPR2016-01243
`
`

`

`Prime Focus Ex 1020-110
`Prime Focus v Legend3D
`IPR2016-01243
`
`

`

`Prime Focus Ex 1020-111
`Prime Focus v Legend3D
`IPR2016-01243
`
`