USOO8922628B2
`
`(12) Unlted States Patent
`Bond
`
`(10) Patent No.:
`(45) Date of Patent:
`
`US 8,922,628 B2
`Dec. 30, 2014
`
`(54) SYSTEM AND PROCESS FOR
`TRANSFORMING TWO-DIMENSIONAL
`IMAGES INTO THREE-DIMENSIONAL
`
`IMAGES
`
`_
`(75) Inventor: Chrls Bond, Los Angeles, CA (US)
`
`(73) Assignee: Prime Focus VFX Services II Inc.,
`Hollywood CA (Us)
`’
`Subject to any disclaimer, the term of this
`patent 1s extended or adjusted under 35
`U.S.C. 154(b) by 836 days.
`
`( * ) Notice:
`
`(56)
`
`References Cited
`
`U~S~ PATENT DOCUMENTS
`
`5,691,843 A * 11/1997 O’Neill ....................... .. 359/464
`5,850,352 A 12/1998 MerZi et al.
`5,929,859 A
`7/1999 Meijers
`6,208,348 B1* 3/2001 Kaye ........................... .. 345/419
`6,400,831 B2
`6/2002 Lee et al.
`6,515,659 B1
`2/2003 Kaye et al.
`6,686,926 B1* 2/2004 Kaye ........................... .. 345/680
`*
`Egg 2: 3i: """"""""" " 345/419
`7,116,324 B2 10/2006 Kaye et al‘
`(Continued)
`
`(21) Appl' NO‘ 12/874490
`
`(22) Filed:
`
`seP- 11 2010
`
`FOREIGN PATENT DOCUMENTS
`
`EP
`
`1353518 A1 10/2003
`
`(65)
`
`Prior Publication Data
`
`OTHER PUBLICATIONS
`
`US 2011/0050864 A1
`
`Mar. 3, 2011
`
`Related US. Application Data
`
`_
`
`_
`
`(60) PrOV1slonal appl1catlon No. 61/239,049, ?led on Sep.
`1, 2009-
`
`_
`
`_
`
`(2006.01)
`(2006.01)
`(200601)
`
`(51) Int- Cl-
`H04N13/04
`H04N13/02
`G06T 7/00
`_
`(52) U‘s‘ Cl“
`CPC """" " H04N13/026 (201301), H04N13/0404
`(2013-01); H04N13/0495 (201301); G06T
`7/0071 (201301); G06T 7/0075 (2013-01)
`USPC ................. .. 348/51; 348/46; 345/6; 345/419;
`345/506; 345/680; 382/154; 382/164
`(58) Field of ClaSSi?catiOIl SeaI‘Ch
`CPC .................................................. .. H04N 13/026
`
`USPC .......................................................... .. 348/ 51
`See application ?le for complete search history.
`
`NPL4Creating full view panoramic image mosaics and environ
`ment maps (Microsoft Research, 1997).*
`NPLiHow t0 make-3D-Ph0t0s (wikihowiusing Photoshop).*
`_
`
`(Cont1nued)
`
`Primary Examiner * William C. Vaughn, Jr.
`Assistant Examiner * Luis M PereZ
`74 Allorn , A enl, or Firm 4 Fulwider Patton LLP
`W 8
`
`ABSTRACT
`(57)
`A system and process is provided for the conversion of a
`stream of two-dimensional images into a pair of streams of
`images for providing the perception of a stream of three
`dimensional images. Each complimentary image inthe image
`stream undergoes the application of a selection and remap
`ping process to independently alter portions Ofthe image, so
`that the remappings shift the image elements in a manner
`which produces a stereo depth effect when the images are
`viewed through the appropriate viewing device.
`
`18 Claims, 14 Drawing Sheets
`
`Step 1
`Great/an 11/ Base
`Slewu PJ/f
`
`Add
`D/sroman
`,2
`
`YES
`
`Step 2
`Radia/ DIEM/11M
`
`Step 4
`SE/CCI/U/l Mask
`Adjustmenr
`
`519, 5
`Weighted
`Msp/acement
`
`Se/ert/D/sp/are
`Mare
`7
`
`Legend3D, Inc. Ex. 2005-0001
`PRIME FOCUS V. LEGEND3D
`IPR2016-01243
`
`

`
`US 8,922,628 B2
`Page 2
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`7,932,874
`7,944,464
`8,427,488
`8,705,847
`8,761,501
`2003/0091225
`2007/0159476
`2011/0050864
`
`4/2011
`5/2011
`4/2013
`4/2014
`6/2014
`5/2003
`7/2007
`3/2011
`
`Kurahashi et al. .............. .. 345/6
`Fukushima et a1.
`348/51
`Hirai et al. .................. .. 345/506
`Kuo et a1. ................... .. 382/154
`Lee et a1. .
`. 382/164
`
`Chen ...... ..
`
`Grasnick ..
`
`. 382/145
`
`. 345/419
`
`Bond ............................ .. 348/51
`
`2012/0086775 A1 *
`
`4/2012 Bae et al. ...................... .. 348/46
`
`OTHER PUBLICATIONS
`
`Lewis Wallace, “Video: How IMAX Wizards Convert Harry Potter to
`3-D, Wired.com,” Aug. 6, 2009 (last visited Aug. 26, 2010), https://
`WWW.Wired.com/underwire/2009/08/video-how-imaX-Wizards-con
`vert-harry-potter-to-3 -d.
`International Search Report, ISA/US, dated Oct. 20, 2010, p. 1-2.
`
`* cited by examiner
`
`Legend3D, Inc. Ex. 2005-0002
`PRIME FOCUS V. LEGEND3D
`IPR2016-01243
`
`

`
`US. Patent
`
`Dec. 30, 2014
`
`Sheet 1 0f 14
`
`US 8,922,628 B2
`
`Step 1
`Creation of Base
`Stereo Pair
`
`Add
`Distortion
`?
`
`YES
`
`Step 2
`Radial Distortion
`
`V
`
`Step 3
`Weighted Image
`Selection
`
`A
`
`Step 4
`Selection Mask
`Adjustment
`
`Step 5
`Weighted
`Displacement
`
`Step 6
`Final Adjustment
`
`FIG. 1
`
`Legend3D, Inc. Ex. 2005-0003
`PRIME FOCUS V. LEGEND3D
`IPR2016-01243
`
`

`
`US. Patent
`
`Dec. 30, 2014
`
`Sheet 2 0f 14
`
`US 8,922,628 B2
`
`Input to Process
`
`203'
`
`\ 0
`Or/g/nal Image
`203
`204 L <4 / 205
`\§"
`I
`Output from Step .7
`206 f 0 0 Creation of Base
`V 2
`Stereo pair
`202
`207
`Right-Eye Image
`
`-
`
`-
`
`Left-E ye Image
`
`208\/®
`Warped
`Left-E ye Image
`
`209
`\
`
`/ 208
`
`6* Output from Step 2
`Radial Distortion
`Warped
`R/ght-Eye Image
`
`LE?'El/e MaSk
`
`2 10 \\
`/
`
`0 O Weighted Image
`Left-Eye Image
`Right-Eye Image
`
`Selection
`
`Output from Step 3
`
`211 \
`
`Output from Step 4
`6 Selection Mask
`Adjustment
`
`Left-Eye Image
`MOW/Ed
`209 Left-Eye Mask 204 \
`
`Right-Eye Image
`
`k/ 3
`
`_L9fl‘-6‘ye
`Displacement
`
`‘
`Output from Step 5
`/O O Weighted
`Displacement
`( Left-Eye Image
`Right-Eye Image
`213
`
`Output from Step 6
`Final Adjustment
`
`Cropped
`Cropped
`Left-Eye Image
`Right-E ye Image
`FIG. 2
`
`Legend3D, Inc. Ex. 2005-0004
`PRIME FOCUS V. LEGEND3D
`IPR2016-01243
`
`

`
`US. Patent
`
`Dec. 30, 2014
`
`Sheet 3 0f 14
`
`US 8,922,628 B2
`
`0 yer/apping Left E ye/Rignt Eye
`Projections
`
`307
`
`R/ght Eye Projector
`with Polarizing Lens
`
`309
`
`Right Eye Image
`
`308
`
`Left Eye Projector
`with Polar/Zing Lens
`
`309
`
`Left Eye image
`
`Interface
`Display
`
`306
`
`30 Glasses
`for Human Operator
`
`FIG. 3
`
`Legend3D, Inc. Ex. 2005-0005
`PRIME FOCUS V. LEGEND3D
`IPR2016-01243
`
`

`
`US. Patent
`
`Dec. 30, 2014
`
`Sheet 4 0f 14
`
`US 8,922,628 B2
`
`w 6E
`
`mum
`
`
`
`SEHNE Q mHRod— Ide m . £21 @2253 $4 89253 SEBWE 35 g“ 293
`
`
`ESQ |@‘ 52 giégz
`
`
`mmcmmsaummé . 111111117111
`ma?a ! [I a!
`
`
`
`_ _> BE; = @925 :5 SEE as: >__8E@>Q_E 255%: DE
`
`
`
`
`
`
`
`
`
`
`
`9%:58 in o 252% Eng E?wz??
`
`
`
`
`
`EEEES i £5523 i 5522 ¥
`
`_ £526 = 268% = 2%; _ mwm?
`
`
`
`
`
`_ 622 =28=%i an; __ ?ma _ w: 583; ES
`
`majomsmesg 3 > 2 X “25
`
`a? _ 28:25 2! ES __ Em _
`
`_ @E a $5 _ :32 95 $5 mm: »
`
`2; N336; $28 5 ll
`
`
`SEBWE $5233 0 632
`25% o 3 liltilww X 55% a .m
`
`5% >3 @3323; 53252 _ Ismmgiigssi Egg __ ém _
`
`
`EELS 52; EE
`
`Tittiiiii . TE 5 5an H _@>3
`
`
`
`Eéew?ws?a I 2025 I
`
`65% g m? E
`
`_ £2 __ 5m __ 5% __ E _ $3.0 :légvm
`
`
`$226 yam: Ezmsaz o ! ¢|
`
`
`
`
`$526 @255 o !> !X 5:8
`6526 V62 55 0 2% E E2: 0
`
`_ 522.8 2 ?g 2 Mom _ H jm
`
`@ 22m .
`
`! l m E c0332»,
`
`AUUUUHHUUUUUUUUUUHV
`
`Legend3D, Inc. Ex. 2005-0006
`PRIME FOCUS V. LEGEND3D
`IPR2016-01243
`
`

`
`US. Patent
`
`Dec. 30, 2014
`
`Sheet 5 0f 14
`
`US 8,922,628 B2
`
`FIG. 5A
`
`501
`
`502
`
`FIG. 5B
`
`FIG. 5C
`
`Legend3D, Inc. Ex. 2005-0007
`PRIME FOCUS V. LEGEND3D
`IPR2016-01243
`
`

`
`US. Patent
`
`Dec. 30, 2014
`
`Sheet 6 0f 14
`
`US 8,922,628 B2
`
`Legend3D, Inc. Ex. 2005-0008
`PRIME FOCUS V. LEGEND3D
`IPR2016-01243
`
`

`
`US. Patent
`
`Dec. 30, 2014
`
`Sheet 7 0f 14
`
`US 8,922,628 B2
`
`FIG. 75 FIG. 7C FIG. 70
`
`FIG. 7A
`
`Legend3D, Inc. Ex. 2005-0009
`PRIME FOCUS V. LEGEND3D
`IPR2016-01243
`
`

`
`US. Patent
`
`Dec. 30, 2014
`
`Sheet 8 0f 14
`
`US 8,922,628 B2
`
`FIG. 88 FIG. 8C FIG. 80
`
`FIG. 8A
`
`Legend3D, Inc. Ex. 2005-0010
`PRIME FOCUS V. LEGEND3D
`IPR2016-01243
`
`

`
`US. Patent
`
`Dec. 30, 2014
`
`Sheet 9 0f 14
`
`US 8,922,628 B2
`
`903
`
`Legend3D, Inc. Ex. 2005-0011
`PRIME FOCUS V. LEGEND3D
`IPR2016-01243
`
`

`
`US. Patent
`
`Dec. 30, 2014
`
`Sheet 10 0f 14
`
`US 8,922,628 B2
`
`k)
`Q
`N
`L'i
`m
`
`'33
`Q
`N
`Q3
`E
`
`3
`N
`Q5
`m
`
`/
`/
`/
`/
`/
`
`Lu
`Q
`N
`£5
`m
`
`S
`N
`Q5
`m
`
`Legend3D, Inc. Ex. 2005-0012
`PRIME FOCUS V. LEGEND3D
`IPR2016-01243
`
`

`
`US. Patent
`
`Dec. 30, 2014
`
`Sheet 11 or 14
`
`US 8,922,628 B2
`
`%
`N
`£5
`E
`
`3
`Q5
`t
`
`é
`N
`£5
`E
`
`.
`
`/
`% N
`Q5
`E
`
`Q
`
`a
`"\|
`95
`E
`
`Legend3D, Inc. Ex. 2005-0013
`PRIME FOCUS V. LEGEND3D
`IPR2016-01243
`
`

`
`US. Patent
`
`Dec. 30, 2014
`
`Sheet 12 or 14
`
`US 8,922,628 B2
`
`FIG. 11/]
`
`FIG. 115
`
`I.
`
`/
`/
`// //
`/
`// l/ / //
`l
`1/
`I
`r/
`//
`//
`,/
`/ /
`/ /
`/
`/ /
`/
`
`/
`
`/
`
`//
`/
`/
`
`7/
`/
`//
`
`,/
`f
`/
`/
`
`FIG. 11C
`
`FIG. IZD
`
`FIG. 115
`
`FIG. 11F
`
`Legend3D, Inc. Ex. 2005-0014
`PRIME FOCUS V. LEGEND3D
`IPR2016-01243
`
`

`
`US. Patent
`
`Dec. 30, 2014
`
`Sheet 13 0f 14
`
`US 8,922,628 B2
`
`T 1 SEEK
`r MNNN
`
`
`
`.1 I I , Emémmi
`
`
`
`Emmi gag A
`
`\IRNNN
`
`$5
`
`EQEBQ $3
`
`mmEEm
`ESQ
`
`83 $52:
`
`\‘BQ
`
`N N1 @E
`
`??mmmwéogm thEQmQS A
`
`WEE WEE f
`NEw?qmm 1 VEEEEE n EN“. (4
`
`
`
`ES.
`
`EEG»:
`
`
`
`VRQEEEFG A ?gwEEES EQEQQS M EEQEBEQEEEBM,
`
`= > 1 k
`
`f @S f
`
`Legend3D, Inc. Ex. 2005-0015
`PRIME FOCUS V. LEGEND3D
`IPR2016-01243
`
`

`
`U S. Patent
`
`Dec. 30, 2014
`
`Sheet 14 or 14
`
`US 8,922,628 B2
`
`-
`
`
`
`FIG. 135 FIG. 13C FIG. 130 FIG. 13E
`
`
`
`
`
`FIG. 13A
`
`In
`
`-. -.
`
`Legend3D, Inc. Ex. 2005-0016
`PRIME FOCUS V. LEGEND3D
`IPR2016-01243
`
`

`
`US 8,922,628 B2
`
`1
`SYSTEM AND PROCESS FOR
`TRANSFORMING TWO-DIMENSIONAL
`IMAGES INTO THREE-DIMENSIONAL
`IMAGES
`
`CROSS-REFERENCES TO RELATED
`APPLICATIONS
`
`This application also claims priority from US. Provisional
`Application No. 61/239,049, ?led Sep. 1, 2009, which is
`incorporated herein by reference in its entirety.
`
`FIELD OF THE INVENTION
`
`The ?eld of the invention is generally related to three
`dimensional ?lm po st-production processes. In particular, the
`invention relates to a system and process for converting two
`dimensional images into three-dimensional images.
`
`BACKGROUND
`
`In human stereo vision, each eye captures a slightly differ
`ent view of the scene being observed. This difference, or
`disparity, is due to the baseline distance between the left and
`right eye of the viewing subject, which results in a different
`viewing angle and a slightly different image of the scene
`captured by each eye. When these images are combined by
`the human visual system, these disparities (along with several
`other visual cues) allow the observer to gain a strong sense of
`depth in the observed scene.
`Stereo image pairs (created either digitally, through anima
`tion or computer generated imagery (CGI), or by traditional
`photography) exploit the ability of the human brain to com
`bine slightly different images resulting inperception of depth.
`In order to mimic this effect, each stereo image pair consists
`of a left eye image and a right eye image. Each complimentary
`image differs in the same manner as the image captured by a
`human left and right eye would when viewing the same scene.
`By presenting the left eye image only to the left eye of a
`viewer, and the right eye image only to the right eye, the
`viewer’s visual system will combine the images in a similar
`manner as though the viewer were presented with the scene
`itself. The result is a similar perception of depth.
`Presenting the appropriate images to the left and right eye
`requires the use of a stereo apparatus, of which there are a
`number of variations on the setup. For viewing a ?lm
`sequence of stereo images, however, a common setup
`includes a pair of left and right digital projectors each pro
`jecting the left and right eye image respectively of the stereo
`pair on to the same ?lm screen space. Each projector has a
`lens which polarizes the light leaving the projector in a dif
`ferent manner. The viewer wears a pair of 3D eyeglasses, the
`viewing lenses of which have a special property. The left-eye
`viewing lens screens out light of the type of polarizationbeing
`projected by the right camera, and vice versa. As a result, the
`left eye sees only the image being projected by the left eye
`projector, and the right eye sees only the image being pro
`jected by the right eye projector. The viewer’ s brain combines
`the images as mentioned above, and the stereo perception of
`depth is achieved. The projectors can be placed side by side,
`but are often stacked on top of one another in a fashion that
`minimizes the distance between the projection sources.
`An alternative setup substitutes the pair of digital proj ec
`tors with a single projector which alternately displays left
`eye/right eye images above some minimum display rate. The
`projector has a synchronized lens which switches polariza
`tion in time with the alternate display of the images to keep
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`2
`the left eye and right eye images correctly polarized. Again, a
`pair ofappropriately polarized 3D eyeglasses are worn by the
`viewer to ensure that each eye only sees the image intended
`for that eye. A similar approach is employed by the high speed
`synchronized left- and right-eye imaging found in modern
`3D-capable digital televisions.
`Although these setups may be suitable for the viewing of
`stereo image pairs, there are a number of variations on the
`apparatus that can achieve a stereo depth effect. Essentially,
`any apparatus that allows for the presentation of two corre
`sponding different images, one to each eye, can potentially be
`used to achieve the stereo depth effect.
`Capturing a stereo pair of images with the aim of repro
`ducing the depth effect as described above is relatively
`simple. For example, a stereo camera rig can be set up with a
`pair of synchronized cameras that capture a scene simulta
`neously. The cameras are separated by a suf?cient baseline to
`account for the distance between the eyes of an average
`human viewer. In this manner, the captured images will effec
`tively mimic what each individual eye of the viewer would
`have seen if they were viewing the scene themselves.
`There exists, however, a substantial library of ?lm (or
`“image streams”) in the industry that were captured by only a
`single camera. Thus, these image streams only contain two
`dimensional information. Various methods have been
`attempted to convert these 2D image streams into three-di
`mensional image streams, most providing reasonable results
`only after expending signi?cant effort and cost.
`Creating a sequence of complimentary stereo image pairs
`from a given sequence of one or more images, each captured
`with only a single camera, to induce the perception of three
`dimensional depth, has been a dif?cult problem. The pair
`must be constructed carefully to mimic the differences a
`human visual system would expect to observe in a stereo pair
`as described above, or the perception of depth will fail and the
`viewer will see an unpleasant jumble of scene elements. Not
`only must each image pair be correctly organized and/or
`reconstructed, but the sequence of image pairs must be orga
`nized and/or reconstructed consistently so that elements in
`the scene do not shift unnaturally in depth over the course of
`the sequence.
`The present industry accepted approach to creating a
`sequence of stereo pairs from a sequence of single 2D images
`involves three very costly steps.
`First, the image sequence of one of the images in the stereo
`pair is rotoscoped. Rotoscoping is a substantially manual and
`complicated process performed on image sequences involv
`ing outlining every element in a frame and extending that over
`a ?lmed sequence, one frame at a time. This requires a human
`operator to manually process almost every frame of a
`sequence, tracing out the scene elements so that they can be
`selected and separately shifted in the image. Common ele
`ments in ?lm can take hours and days to manually rotoscope
`just a few seconds of a completed shot. Despite being a
`complex task, rotoscoping results in a rather limited, low
`quality selection. For example, in order to separately select a
`subset of an actor’s face so that each element can be modi?ed
`separately, in addition to outlining the actor, each element
`would have to be outlined or traced frame by frame for the
`duration of the scene. Selecting elements at this detail is
`known as a form of segmentation. Segmentation refers to the
`selection or sub-selections, or parts, of an image (for
`example, the individual pieces of an actor’ s face) and keeping
`those parts separate for creative and technical control. In a
`more complex scene, with high-speed action and various
`levels of detail and crossing objects, rotoscoping as a segmen
`tation tool, becomes extremely inef?cient due to the increase
`
`Legend3D, Inc. Ex. 2005-0017
`PRIME FOCUS V. LEGEND3D
`IPR2016-01243
`
`

`
`US 8,922,628 B2
`
`3
`in complexity of the scene itself. Rotoscoping thus becomes
`a very cost-intensive process, and one of the reasons convert
`ing 2D to 3D has been so expensive and time consuming.
`Close-up shots of actors are very common and present
`numerous problems for artists using rotoscoping and/or other
`outlining techniques to create a proper separation of the actor
`from even a simple background. For example, creating a
`conversion that successfully includes the ?ne hairs and other
`details on an actor’s head in a close-up camera shot which
`appear frequently in feature ?lms could take between 1-3
`days by a competent artist depending on the segmentation
`detail required. The technique becomes substantially more
`dif?cult in a crowd scene.
`Patents have issued for computer enhanced rotoscoping
`processes for use in converting 2D images into 3D images,
`such as that described by US. Pat. No. 6,208,348 to Kaye,
`incorporated herein by reference; however, these technolo
`gies have done little more than speed up the process of select
`ing and duplicating objects within the original image into a
`left-right stereo pair. Each object must still be manually cho
`sen by an outlining mechanism, usually by an operator draw
`ing around the object using a mouse or other computer selec
`tion device, and the objects then must be repositioned with
`object rendering tools in a complementary image and pre
`cisely aligned over the entire image sequence in order to
`create a coherent stereoscopic effect.
`Second, for life-like 3D rendering of 2D ?lm that
`approaches the quality of CGI or ?lm shot by a true stereo 3D
`camera, the 3D geometry of the scene represented by the
`image must be virtually reconstructed. The geometry creation
`required for such a reconstruction is dif?cult to automate
`effectively, and each rotoscoped element must be assigned to
`its respective geometry in the scene. The geometry must then
`also be animated over the sequence to follow scene elements
`and produce the desired depth effect. The 2D to 3D conver
`sion of Harry Potter and the Half-Blood Prince (2009)
`involved a similar technique. Each object in the original 2D
`scene was analyzed and selected by a graphic artist, 3D object
`models or renditions created from their 2D counterparts, and
`the scene completely or partially recreated in 3D to generate
`depth information appropriate to create a stereoscopic image.
`IMAX Corporation’s computer system processed the infor
`mation to generate the correct offset images in the compli
`mentary images of the stereo pair. See Lewis Wallace, Vzdeo:
`How IMAX Wizards Convert Harry Potter 20 3-D, WIRED.
`COM, Aug. 6, 2009 (last visited Aug. 26, 2010), http://ww
`w.wired.com/underwire/2009/08/video-how-imax-wizards
`convert-harry-potter-to-3-d. Signi?cant drawbacks of recre
`ating entire scenes in 3D include requiring a perfect camera
`track and solution for every shot, countless manual labor
`hours and/or artist oversight to create complex geometry to
`perfectly match and animate within the environment, and
`enormous processing power and/or time to render those ele
`ments together. Similarly, the approach of US. Pat. No.
`6,208,348 to Kaye applies the curvature of simple shapes
`such as cylinders and spheres (as shown by FIGS. 12E, F, G,
`H of that patent) to the image to create a semblance of dimen
`sionality, which is extremely limiting, and results in images
`that are not truly life-like.
`Third, the elements of the scene are then shifted or moved
`horizontally and placed in the complimentary image of the
`stereo pair. Shifting of scene elements is necessary in order to
`produce the disparities between the ?rst and second eye that
`the human visual system would expect to observe in a stereo
`image pair. However, in captured images, the process of shift
`ing 2D elements reveals ‘holes’ that were previously occluded
`by the shifted elements. Essentially, no visual information
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`4
`exists due to the movement of the occlusions. For example, in
`a digital image of a person standing in front of a store, the
`image of the person hides, or occludes, a portion of the store
`in the background. If this person is digitally shifted, no image
`information will remain where the person was originally
`positioned in the image. These image areas left blank by the
`process of shifting elements must be re?lled. Whether the
`scene was reconstructed and re-imaged, or whether the roto
`scoped elements were shifted manually in the image to pro
`duce the disparities required for depth perception, one or both
`images in the pair will have missing information. That is,
`occluding objects in the scene, once shifted in the reconstruc
`tion or otherwise, will reveal portions of the scene for which
`there is no information contained in the image. This missing
`information is very dif?cult to automatically create in gen
`eral, and requires a human operator to manually ?ll in this
`information on virtually every frame. US. Pat. No. 6,208,348
`to Kaye describes a method of pixel duplication to ?ll the
`holes by repeating an equivalent number of pixels horizon
`tally in the opposite direction of the required directional
`placement. However, this “pixel repeat” results in a very
`unrealistic image, and thus manual painting of those holes
`frame by frame is usually required for an optimal result.
`Over the years, as described above, there has been a col
`lective effort by those in the visual effects industry engaged in
`2D to 3D conversion to create new visual material for the
`occlusions or blanks. How to create new occluded visual
`information was a primary topic of discussion at industry
`trade shows. It was thought that creating the occluded new
`visual information was the logical thing to do because it best
`simulates the experience with binocular vision.
`What has been needed, and heretofore unavailable, is a
`system and process that avoids the need for the time and cost
`intensive practice of rotoscoping or manually processing each
`frame of a sequence by tracing out the scene elements, build
`ing or reconstructing 3D geometry, 3D scene tracking, as well
`as image reconstruction and mapping and high-quality ren
`dering of image information, all while, at the same time,
`providing a reliable system and process for rapidly transform
`ing a 2D monocular sequence into a sequence of stereo image
`pairs, reducing human interaction, and improving ?delity and
`detail.
`
`SUMMARY OF THE INVENTION
`
`Contrary to industry practice, the inventor of the system
`and process described herein has discovered that a far easier
`approach to transforming 2D images into 3D images is to hide
`image information rather than create it, and then combine that
`with a superior method of selecting and/ or segmenting
`images. The inventor discovered that hiding certain image
`information, for instance, by warping or distorting portions of
`the image to shrink or increase portions of the image, actually
`triggered the same perceived stereoscopic effect as generat
`ing new information. This process of hiding image informa
`tion was also found to be vastly faster than the process of
`duplication and/or re-imaging. When combined with a pro
`cess of selecting and segmenting an image using the image’s
`own intrinsic qualities, rather than by rotoscoping, a superior
`and much more realistic ?nal 3D image is created with vastly
`improved ?delity and detail, and the speed of the process
`allows for more iterations and re?nement.
`In a general aspect, the proposed approach avoids the prob
`lems of the prior art mentioned above at a signi?cant reduc
`tion to post-production time and cost. First, it avoids the
`rotoscoping step by using a weighted image selection based
`upon a variety of visual image properties. This requires little
`
`Legend3D, Inc. Ex. 2005-0018
`PRIME FOCUS V. LEGEND3D
`IPR2016-01243
`
`

`
`US 8,922,628 B2
`
`5
`interaction from a human operator as compared to rotoscop
`ing. Second, it avoids any costly virtual reconstruction step by
`vector-based remapping of image elements by compressing
`or expanding portions of the image to provide the perception
`of shifting image elements to achieve the disparities required
`for stereo depth perception. There is no need to recreate the
`image by creating 3D geometry or recreating objects. This is
`substantially different than other methods of 2D to 3D con
`version, and removes the need for rotoscoping, 3D scene
`tracking, geometry building, image reconstruction and map
`ping and high-quality rendering of those systems and meth
`ods. Third, it avoids any costly manual re?lling such as pixel
`repeat or frame by frame painting by intelligently applying a
`series of image re-mappings using weighted image masks in
`a soft fashion (for example, blending the remapping with the
`original material so the distortion does not have any hard
`edges). The result of these operations does not cause occlud
`ing objects to reveal any absent information which would
`need to be ?lled. In situations where an occlusion would
`otherwise reveal such absent information, the weighted
`image re-mappings automatically interpolate appropriate
`substitute information and produce a stereo image pair which
`effectively induces a strong perception of stereo depth when
`viewed with the appropriate stereo apparatus.
`In one aspect, the invention includes a process for creating
`a perception of a three-dimensional image from a two -dimen
`sional image, including displacing a portion of a working
`image by distortion of the working image to create a modi?ed
`working image, a magnitude of the displacement being rela
`tive to a perceived depth of the portion, the distortion operat
`ing to interpolate image information over a range of the
`displacement, and replacing a selected image in a stereo pair
`of images with the modi?ed working image. In some aspects,
`the process further includes displaying on a video display the
`modi?ed working image as part of the pair of stereo images.
`The perception of a three-dimensional image is created when
`the modi?ed working image is viewed as part of the pair of
`stereo images through a stereoscopic viewing device.
`In further aspects, the process includes creating an image
`mask associated with characteristics of the working image
`and associated with at least a portion of the working image,
`and assigning a depth value to each pixel location of the
`working image that is associated with the image mask, the
`depth value being chosen from a speci?ed range of values,
`wherein the magnitude of the displacement being relative to a
`perceived depth of the portion includes a maximum magni
`tude of displacement being assigned to each pixel location
`having a ?rst selected depth value and a minimum magnitude
`of displacement being assigned to each pixel location having
`a second selected depth value and an intermediate magnitude
`of displacement being assigned to a pixel location having a
`third selected depth value, the third selected depth value
`selected from a range between the ?rst and second depth
`values. In accordance with this aspect, the process may also
`include creating a boundary on at least one part of the image
`mask so as to preclude assigning the depth values to non
`selected pixel locations outside the boundary. The process
`may also include assigning a depth value to each pixel loca
`tion of the working image includes assigning the depth value
`to each pixel location of the image mask and then applying the
`image mask to the working image to assign each depth value
`to a corresponding pixel location in the working image to
`displace the portion of the working image.
`In yet further aspects, creating an image mask associated
`with characteristics of the working image includes selecting
`an image characteristic from a group consisting of hue, lumi
`nance, saturation, and color, and generating an image repre
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`6
`sentation of a weighted distribution of the image characteris
`tic in the working image, wherein the depth values are
`assigned to each pixel location in accordance with the
`weighted distribution. The process may also include compar
`ing a portion of an image feature present in the working image
`with a portion of the image mask and a selected range of depth
`values, and changing the depth value at a selected pixel of the
`image mask based on a location of the selected pixel relative
`to the image feature. In accordance with these further aspects,
`the process may include applying the image mask to a second
`image to assign each depth value to a corresponding pixel
`location in the second image, displacing by distortion of the
`second image a portion of the second image in a direction
`opposite the displacement of the working image, and replac
`ing a selected image in a stereo pair of images with the second
`image.
`In some aspects, the process includes providing a second
`image associated with the working image, displacing a por
`tion of the second image by a distortion of the second image
`to create a modi?ed second image, a magnitude of the dis
`placement of the second image being relative to a perceived
`depth of the portion of the second image, the distortion of the
`second image operating to interpolate image information
`over a range of the displacement of the second image, and
`replacing the second image in a stereo pair of images with the
`modi?ed second image. In accordance with some of these
`aspects, the perception of the three-dimensional image is
`created when the modi?ed working image and the modi?ed
`second image are viewed as a pair of stereo images through a
`stereoscopic viewing device. The displacement of the second
`image may be in a direction opposite the displacement of the
`working image. Further, the magnitude of the displacement of
`the second image may be equal, or substantially equal, to the
`magnitude of the displacement of the working image.
`In a further aspect, the invention includes a system for
`creating a perception of a three-dimensional image from a
`two-dimensional digitized image, including a video monitor
`con?gured to display a stereo pair of images in response to a
`video signal, an input device con?gured to generate input
`signals, and con?gured to select a working image from a
`group of images, and con?gured to select characteristics of
`the working image, a control con?gured to generate control
`signals, and con?gured to assign a value to a characteristic
`selected by the input device. The processor is in operable
`communication with the monitor, the input device and the
`control, the processor con?gured to displace a portion of the
`working image by distortion of the working image to create a
`modi?ed working image, a magnitude of the displacement
`being relative to a value received by the control, the distortion
`operating to interpolate image information over a range of the
`displacement, the processor, in response to a ?rst input signal,
`further con?gured to replace a selected image in a stereo pair
`of images with the modi?ed working image. The video moni
`tor, in response to a second input signal, displays the modi?ed
`working image as part of the stereo pair of images.
`In accordance with the above aspect, the processor may be
`further con?gured to create an image mask associated with
`characteristics of the working image and associated with at
`least a portion of the working image in response to a second
`input signal, and to assign a depth value to each pixel location
`of the working image that is associated with the image mask,
`the depth value being chosen from a speci?ed range of values,
`wherein the magnitude of the displacement being relative to a
`value received by the control includes a control value received
`by the control multiplied by a scaling factor determinable at
`each pixel location according to the depth value assigned to
`each pixel location. In some aspects, the processor is further
`
`Legend3D, Inc. Ex. 2005-0019
`PRIME FOCUS V. LEGEND3D
`IPR2016-01243
`
`

`
`US 8,922,628 B2
`
`7
`con?gured to, in response to a second input signal, apply the
`image mask to a second image to assign each depth value to a
`corresponding pixel location in the second image, the proces
`sor further con?gured to displace in a direction opposite the
`displacement of the working image a portion of the second
`image by distortion of the second image, and the processor
`further con