l lllll llllllll III lllll lllll lllll lllll lllll 111111111111111111111111111111111
`US007460731B2
`
`c12) United States Patent
`Senftner et al.
`
`(10) Patent No.:
`(45) Date of Patent:
`
`US 7 ,460, 731 B2
`Dec. 2, 2008
`
`(54) PERSONALIZING A VIDEO
`
`(75)
`
`Inventars: Blake Senftner, Burbank, CA (US); Liz
`Ralston, Burbank, CA (US); Miles
`Lightwood, Glendale, CA (US); Todd
`Shifflett, Venice, CA (US)
`
`(73) Assignee: Flixor, Inc., Burbank, CA (US)
`
`( *) Notice:
`
`Subject to any disclaimer, the term ofthis
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 0 days.
`
`(21) Appl. No.: 11/840,834
`
`(22) Filed:
`
`Aug. 17, 2007
`
`(65)
`
`Prior Publication Data
`
`US 2008/0019576 Al
`
`Jan.24,2008
`
`(52) U.S. Cl. ....................................... 382/284; 382/103
`(58) Field of Classification Search ................. 382/100,
`382/103, 305, 284, 282-283; 358/537-538,
`358/540, 452-453; 707/1-7, 104.1; 348/584,
`348/586,589,590, 591, 598, 600, 722,578,
`348/36, 39, 135, 140--143, 169-172; 345/723;
`386/52, 55,95,54, 53
`See application file for complete search history.
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`6,795,818 Bl*
`912004 Lee ............................... 707/3
`6,826,292 Bl* 1112004 Tao et al. .................... 382/103
`6,970,639 Bl* 1112005 McGrath et al.
`.............. 386/52
`7,230,653 Bl*
`6/2007 Overton et al. . . . . . . . . . . . . . . 348/ 5 84
`* cited by examiner
`Primary Examiner-Sherali Ishrat
`(74) Attorney, Agent, or Firm-Fish & Richardson P.C.
`
`Related U.S. Application Data
`
`(57)
`
`ABSTRACT
`
`(63)
`
`(60)
`
`Continuation of application No. PCT/US2006/
`036150, filed on Sep. 14, 2006.
`
`Provisional application No. 60/717,852, filed on Sep.
`16, 2005, provisional application No. 60/717,937,
`filed on Sep. 16, 2005, provisional application No.
`60/717,938, filed on Sep. 16, 2005.
`
`(51)
`
`Int. Cl.
`G06K 9136
`G06K 9100
`
`(2006.01)
`(2006.01)
`
`Processes and apparatus for personalizing video through par(cid:173)
`tial image replacement are disclosed. Personalization may
`include partial or füll replacement ofthe image of an actor, an
`object or both. Personalization may also include insertion or
`replacement of an object, and füll or partial replacement of
`the background and/or sound track. A video preparation pro(cid:173)
`cess may be used to create a library of personalization-ready
`videos.
`
`101Claims,11 Drawing Sheets
`
`Digital video
`
`2D Image of
`new actor
`
`r------- -------1
`
`1
`
`r------ ------
`' 1
`
`1
`
`Track original
`actor's position,
`orientation, and
`expression
`
`210
`
`220
`
`230
`
`240
`
`1
`
`1
`
`Render image of
`new actor over
`Replace
`frames prepared
`background behind
`by 220. in
`key portions of the 1--1--t--+1
`position,
`image ofthe
`orientation and
`original actor
`expression from
`210
`
`1
`:
`1
`1
`
`ldentify and isolate
`non-replaced skin 1--1--+--1>1
`areas
`
`Alter non~replaced
`skin areas to
`match skin
`appearance
`of new actor
`
`:
`1
`
`ldentify and track
`mumination,
`
`200 \
`
`shading, shadows,
`1::, ~ reflections
`
`!:',
`
`Recreate
`illumination,
`shading, shadows,
`
`reflections J
`
`:
`
`1
`
`1
`1
`1
`1
`---------·------
`
`1
`_______ j
`
`Personalized
`video
`
`Create model of
`:
`new actor
`1
`(3D geometry,
`:
`demographic
`1
`profile,and
`:
`1
`personal
`1
`1nformation)
`1 L------- - - - - - - -1
`i------- ------,
`
`1LJ
`
`1
`
`110
`
`100
`
`320
`
`330
`
`340
`
`300
`
`1
`
`Petitioner Samsung 1006
`
`

`

`U.S. Patent
`
`Dec. 2, 2008
`
`Sheet 1of11
`
`US 7 ,460, 731 B2
`
`Digital video
`
`20 lmageof
`new actor
`
`r - - - - - -
`1
`1
`1
`
`- - - - - - -,
`1
`1
`1
`t
`
`Track original
`actor's position,
`orientation, and
`expression
`
`r------- -------,
`1
`1
`Create model of
`1
`1
`new actor
`1
`1
`:
`(30 geometry,
`:
`demographic
`1
`:
`profile, and LJ : ' 100
`personal
`1
`information)
`1
`1
`L------- -------1
`r-------
`
`110
`
`Replace
`background behind
`key portions of the i---r----r-111'1
`image ofthe
`original actor
`
`1
`1
`Render image of
`1
`1
`new actor over
`1
`~_._-Pi frames prepared
`by 220, in
`position,
`orientation and
`expression from
`210
`
`ldentify and isolate
`non-replaced skin 1---+----i-!PI
`areas
`
`Alter non-replaced
`skin areas to
`match skin
`appearance
`of new actor
`
`320
`
`330
`
`340
`
`210
`
`220
`
`230
`
`240
`
`1
`1
`1
`1
`
`ldentify and track
`illumination,
`shading, shadows,
`reflections
`
`200~
`
`1
`1
`
`FIG. 1
`
`Recreate
`illumination,
`shading, shadows,
`
`reflections L-/300
`
`1
`_______ J
`
`Personalized
`video
`
`2
`
`

`

`U.S. Patent
`
`Dec. 2, 2008
`
`Sheet 2of11
`
`US 7,460,731 B2
`
`Pose and
`expression
`
`320 ~ --_ :ara---------_ :;~e d:~ -------~·:~n dara
`
`30 Model of
`new actor
`
`Transform 30
`model
`
`Convert to 20
`image and scale
`
`FIG.2
`
`------1
`1
`1
`1
`1
`
`Insert 20 image
`into frame
`of video clip
`
`To
`330
`
`3
`
`

`

`U.S. Patent
`
`Dec. 2, 2008
`
`Sheet 3of11
`
`US 7 ,460, 731 B2
`
`Model of new
`actor as
`created by a
`prior use of
`process 100
`
`Model of2"d
`new actor as
`created by a
`prior use of
`process 100
`
`Model of N1
`h
`new actor as
`created by a
`prior use of
`process 100
`
`,---- ---------- ------------- --,
`
`FIG. 3
`
`User controlled N-dimensional
`morph of the 3D geometry and
`parameters that define a new actor
`
`1
`1
`1
`1
`1
`1
`1
`
`110A
`
`1
`1
`1
`1
`
`L_J-100A
`
`1
`---------------~
`
`Hybrid model of new actor in identical
`data format as new actor models
`created by use of process 100
`
`4
`
`

`

`U.S. Patent
`
`Dec. 2, 2008
`
`Sheet 4of11
`
`US 7 ,460, 731 B2
`
`Digital video
`
`r------
`1
`1
`1
`1
`
`Track location for
`object placement
`
`30 model of
`new object
`,-------
`
`1
`1
`1
`1
`t - - - . - - - -o - - iP I
`
`Insert image of
`new object
`
`350
`
`200A r~ -------------_:
`
`ldentify and track
`illumination,
`shading, shadows,
`reflections
`
`1
`1
`1
`
`Recreate
`illumination,
`shading, shadows,
`reflections
`
`1
`
`1
`1
`
`'
`1
`
`340A
`
`~ - - - - - - - - - - - - - - ~\_ 300A
`
`Personalized
`video
`
`1
`1
`1
`1
`1
`1
`1
`1
`:
`1
`1
`1
`
`250
`
`240A
`
`FIG.4
`
`5
`
`

`

`U.S. Patent
`
`Dec. 2, 2008
`
`Sheet 5of11
`
`US 7 ,460, 731 B2
`
`Digital video
`
`r------
`1
`1
`1
`1
`1
`
`Track position and
`orientation of
`original object
`
`Remove image of
`original object and
`replace
`background
`
`ldentify and track
`illumination,
`shading, shadows,
`reflections
`
`1
`1
`
`255
`
`260
`
`2408
`
`2008
`
`FIG. 5
`
`1
`
`1
`1
`
`~--------------!
`
`1
`1
`1 - - - -1 - - -1 - - -
`
`30 model of
`replacement
`object
`,-------
`' 1
`
`Insert image of
`replacement object
`
`Recreate
`illumination,
`shading, shadows,
`reflections
`
`1
`1
`1
`1
`
`1
`1
`
`"----------------'
`Personalized
`video
`
`1
`\
`
`360
`
`3408
`
`3006
`
`6
`
`

`

`U.S. Patent
`
`Dec. 2, 2008
`
`Sheet 6of11
`
`US 7 ,460, 731 B2
`
`Digital video
`
`,------
`
`1
`1
`1
`
`Track camera
`location
`
`New
`background
`image
`
`Substitute at least
`a portion of the
`1---.----+--19'! background image
`area
`1
`1
`1
`1
`
`r------- ------,
`1
`1
`1
`1
`1
`1
`
`370
`
`1
`1
`1
`1
`
`: _______ -------~
`
`Personalized
`video
`
`3ooc
`
`265
`
`270
`
`200C
`
`FIG.6
`
`Separate
`foreground and
`background image
`areas
`
`1
`
`_r~ -------------_i
`
`7
`
`

`

`U.S. Patent
`
`Dec. 2, 2008
`
`Sheet 7of11
`
`US 7 ,460, 731 B2
`
`! 1008
`
`New actor
`voice sample
`r------ -------,
`1
`1
`1
`1
`1
`'
`1
`1
`1
`1
`
`ldentify key
`attributes of new
`actor's voice
`
`1
`1
`
`'------- -------
`,-------
`
`Digital video
`,-------
`
`1
`1
`1
`1
`1
`
`1
`1
`1
`1
`1--....,._---1--111M
`
`lsolate original
`actor's dialog
`
`Transform original
`actor's dialog
`
`180
`
`380
`
`390
`
`:
`1..---------------'
`
`\_ '
`
`Personalized
`video
`
`3000
`
`280
`
`290
`
`2000
`
`FIG. 7
`
`Substitute or add
`lsolate
`background audio 1---.-------111M background audio
`elements
`elements
`1
`1
`
`1
`1
`1
`1
`
`1
`1
`
`rl
`
`_)
`
`~-- - - -
`
`:
`
`- - - ______ I
`
`8
`
`

`

`U.S. Patent
`
`Dec. 2, 2008
`
`Sheet 8of11
`
`US 7 ,460, 731 B2
`
`FIG.8
`
`410~
`
`Requestor
`
`450
`
`455
`
`Digital
`Video
`
`415
`
`200
`
`Video
`Preparation
`Process
`
`Request
`
`Personalized
`Video
`
`2DDigital
`lmage(s) +
`other info
`
`425
`
`Actor
`Modeling
`Process
`
`Actor
`Model
`
`100
`
`435
`
`465
`
`470
`
`Selection
`
`Selection
`
`Video
`Library
`
`Prepared
`Digital
`Video
`
`Personalization
`Process
`
`Actor
`Model
`
`300
`
`Actor
`Model
`Library
`
`440
`
`9
`
`

`

`U.S. Patent
`
`Dec. 2, 2008
`
`Sheet 9of11
`
`US 7,460,731 B2
`
`410~
`
`'
`
`FIG. 9
`
`Requestor
`
`550
`
`~420
`
`'~~~
`
`Personalized
`Advertising
`Video
`
`2D Digital
`lmage(s) +
`other info
`
`425
`
`Actor
`Modeling
`Process
`
`100
`
`450
`
`455
`
`Suppliers of
`Digital Video
`
`Digital
`Video
`
`200
`
`465
`
`Video
`Preparation
`Process
`
`Prepared
`Digital
`Video
`
`Selection
`
`Selection
`
`Video
`Library
`
`Prepared
`Digital
`Video
`
`300
`
`470
`
`Personalization
`Process
`
`3D
`Product
`Model
`
`3D Product
`Modeling
`process
`
`Selection
`
`Product
`Model
`Library
`
`510
`
`520
`
`Actor
`Model
`
`Actar
`Model
`Library
`
`440
`
`540
`
`Demograph ic
`1 nformation
`
`Advertising
`Strategy
`
`530
`
`10
`
`

`

`U.S. Patent
`
`Dec. 2, 2008
`
`Sheet 10 of 11
`
`US 7,460,731 B2
`
`FIG. 10
`
`20 Digital
`lmage(s) +
`other info
`
`640
`
`Personalized
`Video
`Clip
`
`600
`
`680
`
`\/ideo
`Library
`
`610
`
`620
`
`Processor
`
`Storage
`Medium
`
`630
`
`Memory
`
`11
`
`

`

`U.S. Patent
`
`Dec. 2, 2008
`
`Sheet 11 of 11
`
`US 7 ,460, 731 B2
`
`FIG.11
`
`Personalized
`Video
`
`Requestor
`...
`L::::.
`1 1
`1 1
`'1
`1 1
`
`'1
`
`720
`
`Memory
`
`Processor
`
`Storage
`Medium
`
`700
`
`730
`
`740
`
`710
`
`II Personalizable
`~Video
`
`760
`
`Video
`Library
`
`750
`
`12
`
`

`

`US 7,460,731 B2
`
`1
`PERSONALIZING A VIDEO
`
`PRIORITY CLAIMS AND RELATED
`APPLICATION INFORMATION
`
`This non-provisional patent applicationis, under 35 U.S.C.
`§ 120, a continuing patent application of and claims priority to
`the co-pending PCT Application No. PCT/US2006/036150
`entitled "PERSONALIZING A VIDEO" and filed on Sep. 14,
`2006 (PCT Publication No. W02007/035558) which, under
`35 U.S.C. §119(e)(l), claims priorities from the following
`prior U.S. patent applications: Application No. 60/717,852
`filed on Sep. 16, 2005 and entitled "Facial image replace(cid:173)
`ment;" Application No. 60/717,937 filed Sep. 16, 2005 and
`entitled "Customized product marketing images;" Applica(cid:173)
`tion No. 60/717,938 filed on Sep. 16, 2005 and entitled "Call
`and message notification."
`Tue above referenced PCT application and U.S. patent
`applications are incorporated by reference in their entirety as
`part ofthe specification ofthe present application.
`
`NOTICE OF COPYRIGHTS AND TRADE DRESS
`
`A portion of the disclosure of this patent document con(cid:173)
`tains material which is subject to copyright protection. This
`patent document may show and/ or describe matter which is or
`may become trade dress ofthe owner. The copyright and trade
`dress owner has no objection to the facsimile reproduction by
`anyone of the patent disclosure as it appears in the Patent and
`Trademark Office patent files or records, but otherwise 30
`reserves all copyright and trade dress rights.
`
`2
`depict the representation of the passage of time within one or
`more settings. The number of images, the rate of display of
`the images, and the dimensions ofthe images is irrelevant. Yet
`the images are still conventionally identified as comprised of
`lines and pixels, even though throughout the various steps of
`the processes disclosed herein, the number oflines and pixels
`per frame may be statistically re-sampled, as necessary, for
`the precision of accuracy required by various steps.
`Each frame of a digital video is therefore comprised of
`10 some total number of pixels, and each pixel is represented by
`some number ofbits of information indicating the brightness
`and color ofthe respective portion ofthe image. Throughout
`the plurality of methods that a digital video may be created, all
`are essentially a series of images, represented as a series of
`15 frames composed of lines and pixels. Various means of rep(cid:173)
`resenting a digital video in bits and bytes exist, but at some
`level all can be referenced as frames, lines and pixels.
`A digital video requires a display medium to view the
`frames in sequence. A display medium is typically electronic,
`20 such as a TV, computer and monitor, a cellular phone or a
`personal digital assistant (PDA). These devices receive or
`possess the digital video in the form of a file, and display the
`frames in sequence to the user. Other potential display medi(cid:173)
`ums that are not electronic exist, which are also a method for
`25 a userto experience a digital video. Examples ofthese medi(cid:173)
`ums are 1) printed holograms of the nature found on credit/
`debit cards and collectable sports cards, 2) digital paper that
`employs chemical and other non-electronic image encoding
`methods, 3) simple printed flipbooks.
`
`SUMMARY
`
`TECHNICAL FIELD
`
`This disclosure relates to systems, processes and apparatus
`for manipulating, processing and producing video creating
`personalized videos through partial image replacement.
`
`BACKGROUND
`
`Digital image recording, storage, and synthesis are now
`widely used in television, motion pictures, and video games.
`A digital video is essentially a series of digital photographs,
`commonly termed "frames," of a scene taken at periodic
`intervals. A digital video may be recorded using a digital
`video camera, may be created by digitizing an analog video
`recording or by digitizing a motion picture film recording,
`may be created by rendering 2D and 3D computer graphics,
`or even a hybrid combination of all the above with analogue
`or digital compositing of the various elements to realize a
`final digital video. To give the viewer an impression of
`smooth, continuous motion, digital or analog video images
`are typically comprised of 25, 30, 60, or more frames every
`second. However, the number of frames per second should
`not be viewed as a limiting factor to identify a digital video; 55
`some digital video formats support as few as 1 frame every N
`seconds, or even a variable frame rate as necessary to achieve
`the effect of perceived motion while attempting to reduce the
`final storage size ofthe resulting digital video. Whatever the
`frame rate, each frame may be divided into a plurality of
`horizontal lines, and each line is typically divided into a
`plurality of picture elements, commonly termed "pixels", per
`line. Standard broadcastvideo recordings in the United States
`have 525 lines per frame, and HDTV recordings have 1080
`lines per frame. However, for the purposes of this description,
`the term "digital video" is meant in the broader sense of
`simply a series of images that when viewed in sequence
`
`Apparatus, systems and techniques for providing person(cid:173)
`alized digital video in various applications are described. One
`35 or more target images, such as an actor and an object, in an
`original digital video can be replaced based on user prefer(cid:173)
`ences to produce a personalized digital video. Such a person(cid:173)
`alized video can be used for advertising a product or service
`by inserting one or more images associated with the product
`40 or service in the personalized video.
`In one implementation, a computer-implemented process
`for providing personalized digital video can include selecting
`a target in original digital video to be replaced by a target
`replacement, wherein the target is a portion or an entirety of
`45 an actor or an object other than an actor in the original digital
`video; analyzing each frame of the original digital video to
`track a change in the selected target in the original digital
`video to capture data on the selected target, wherein the
`captured data includes at least information on a position,
`50 orientation and size of the selected target in the original
`digital video; and replacing the selected target with an image
`that resembles a continuation of a scene adjacent to the target
`in the original digital video to produce altered digital video in
`which the selected target is removed.
`In another implementation, a computer-implemented pro-
`cess for providing personalized digital video can include
`storing altered digital video files based on original digital
`video files in a video library. At least one target in an original
`video file is removed in a corresponding altered digital video
`60 file and is substituted by an image that resembles a continu(cid:173)
`ation of a scene adjacent to the target in a frame ofthe original
`digital video file, and the target is a portion or an entirety of an
`actor or an object other than an actor in the original digital
`video file. This method includes receiving, via a computer in
`65 communication with a network, a request from a user for
`personalizing a user selected altered digital video file by
`replacing a target in a corresponding original digital video file
`
`13
`
`

`

`US 7,460,731 B2
`
`15
`
`3
`with a user target replacement identified by the user; and
`retrieving, from the video library, data on the target that is
`removed from the user selected altered digital video file. The
`data includes at least information on a position, orientation
`and size ofthe target in the original digital video file for the
`user selected altered digital video file. This method further
`includes applying the retrieved data on the target, frame by
`frame, to transform the user target replacement received from
`the user into a modified user target replacement that acquires
`characteristics of the target in the corresponding original
`digital video file; and inserting the modified user target
`replacement at a position of the target in each frame of the
`user selected altered digital video file in which the target
`appears in the original digital video file to substantially match
`at least the position, orientation and size ofthe selected target
`in the original digital video file to produce a personalized
`digital video file.
`In another implementation, a system for providing person(cid:173)
`alized digital video can include a network for communica- 20
`tions; and a video library to store altered digital video files
`based on original digital video files. At least one target in an
`original video file is removed in a corresponding altered
`digital video file and is substituted by an image that resembles
`a continuation of a scene adjacent to the target in a frame of 25
`the original digital video file, and wherein the target is a
`portion or an entirety of an actor or an object other than an
`actor in the original digital video file. This system includes at
`least one computer in communication with the network and
`the video library. The computer is operable to interface with 30
`a user via the network and to receive a request from the user
`for personalizing a user selected altered digital video file by
`replacing a target in a corresponding original digital video file
`with a user target replacement identified by the user. The
`computer is operable to retrieve from the video library data on 35
`the target that is removed from the user selected altered digital
`video file, where the data includes at least information on a
`position, orientation and size of the target in the original
`digital video file for the user selected altered digital video file.
`This computer is also operable to apply the retrieved data on 40
`the target, frame by frame, to transform the user target
`replacement received from the user into a modified user target
`replacement that acquires characteristics of the target in the
`corresponding original digital video file, and to insert the
`modified user target replacement at a position of the target in 45
`each frame of the user selected altered digital video file in
`which the target appears in the original digital video file to
`substantially match at least the position, orientation and size
`ofthe selected target in the original digital video file to pro(cid:173)
`duce a personalized digital video file for the user.
`In another implementation, a computer-implemented pro(cid:173)
`cess for providing personalized video can include providing a
`user selected altered digital video file by replacing a target in
`a corresponding original digital video file with a user target
`replacement identified by the user; and obtaining data on the 55
`target that is removed from the user selected altered digital
`video file. The data includes at least information on a position,
`orientation and size ofthe target in the original digital video
`file for the user selected altered digital video file. This process
`includes applying the obtained data on the target, frame by 60
`frame, to transform the user target replacement into a modi(cid:173)
`fied user target replacement that acquires characteristics of
`the target in the corresponding original digital video file; and
`inserting the modified user target replacement at a position of
`the target in each frame of the user selected altered digital 65
`video file in which the target appears in the original digital
`video file to substantially match at least the position, orien-
`
`4
`tation and size of the selected target in the original digital
`video file to produce a personalized digital video file.
`In another implementation, a process is provided for per(cid:173)
`sonalizing an original digital video containing an image com(cid:173)
`prising an original background scene and a foreground
`including an original actor. This process includes tracking the
`position, orientation, and expression of the original actor;
`replacing at least a key portion of the original actor with an
`image that continues the background scene; inserting a new
`10 actor into the video, the new actor substantially matching the
`position, orientation and expression of the replaced portion of
`the original actor; recreating illumination and shading effects
`on the new actor; and recreating shadows and reflections of
`the new actor.
`In another implementation, a process for personalizing a
`video can include providing a video library of a plurality of
`prepared videos, each of the prepared videos resulting from a
`video preparation process; providing an actor model library
`of one or more new actor models where each of the models
`resulting from an actor modeling process; selecting a video
`from the video library; selecting a new actor model from the
`actor model library; and applying a personalization process to
`create a personalized version ofthe selected video using the
`selected new actor model.
`In another implementation, a process is provided for cre(cid:173)
`ating a personalized version of an original digital video and
`the original video contains an image of an original actor. This
`process includes replacing at least a portion ofthe image of
`the original actor with an image of a new actor; and inserting
`an image of a new object into the video.
`In yet another implementation, a computing device is pro(cid:173)
`vided to create a personalized version of an original digital
`video containing an image comprising a background scene
`and a foreground including an image of an original actor. This
`computing device includes a processor, a memory coupled
`with the processor, a storage medium having instructions
`stored thereon which when executed cause the computing
`device to perform actions. These actions include comprising
`personalizing a video where the video contains an image
`including an image of an original actor, tracking the position,
`orientation, and expression ofthe original actor; replacing at
`least a key portion of the original actor with an image that
`continues the background scene; inserting a new actor into the
`video where the new actor substantially matches the position,
`orientation and expression of the replaced portion of the
`original actor; recreating illumination and shading effects on
`the new actor; and recreating shadows and reflections ofthe
`new actor.
`These and other implementations and associated features
`50 are described in greater detail in the attached drawings, the
`description and the claims.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`FIG. 1 is a flow chart of a process to create a personalized
`digital video.
`FIG. 2 is an expansion ofthe process ofFIG. 1.
`FIG. 3 is a flow chart of optional steps in the process to
`create a personalized video.
`FIG. 4 is a flow chart of optional steps in the process to
`create a personalized video.
`FIG. 5 is a flow chart of optional steps in the process to
`create a personalized video.
`FIG. 6 is a flow chart of optional steps in the process to
`create a personalized video.
`FIG. 7 is a flow chart of optional steps in the process to
`create a personalized video.
`
`14
`
`

`

`US 7,460,731 B2
`
`5
`FIG. 8 is a flow chart of another process for providing
`personalized videos.
`FIG. 9 is a flow chart of a process for providing personal(cid:173)
`ized advertising.
`FIG. 10 is a block diagram of a computer apparatus.
`FIG. 11 is a block diagram of another computer apparatus.
`
`DETAILED DESCRIPTION
`
`10
`
`6
`Depending upon the quality of the desired replacement,
`and the impact ofthis desired quality upon the storyline ofthe
`original video, essential elements ofthe replaced actors inter(cid:173)
`pretation andmeaning within the context ofthe storyline may
`be altered. For example, by replacing a muscular, heroic male
`in a video with that of a petite female, the storyline remains
`essentially unaltered, but the interpretation ofthe storyline is
`drastically altered for effect. In order for such alteration to
`occur, the replacement ofthe face and portions ofthe head is
`not enough to achieve this result; in this situation a complete
`removal of the original actor is executed, their key motions
`are preserved in a secondary storage medium, and then ref(cid:173)
`erenced for the animation and insertion of the petite female' s
`digital double.
`Between the two extremes of face/head with connecting
`skin and a complete digital double lies an infinite range of
`degrees which actor replacement could be executed. Note that
`in all instances ofthis infinite range, the storyline and essen(cid:173)
`tial actions of the actors in the original video may be unal-
`20 tered.
`A personalized video can be personalized in various ways.
`In one example, a personalized video is a video sequence that
`is altered via one or more actor replacements. In another
`example, a personalized video is a video sequence that does
`25 not include any actor replacements-and includes non-actor
`objects and/or set pieces within the video which have been
`alte red and/or replaced with othero bjects and/ or set pieces. In
`yet another example, a personalized video is a video sequence
`that is altered via a combination of (1) at least one actor
`30 replacement or alternation and (2) at least one object replace(cid:173)
`ment or altemation. The objects and/or set pieces that have
`been altered and/or replaced may be selected based a user
`criterion, such as some demographic or other information
`about the receiver and/or viewer of the digital video. For
`35 example, a video advertisement for a pizza restaurant could
`receive personalization such that the video advertisement
`appears to take place at the specific local neighborhood fran(cid:173)
`chise of the video' s viewer. In this instance, no actor replace(cid:173)
`ment has occurred, but the video is personalized due to the
`40 local world-space context ofthe new advertisement having a
`personal connection with the viewer-it now takes place in
`their neighborhood. A further example of a non-actor replac(cid:173)
`ing personalized video would be a video depicting a character
`drinking some brand of soda, where the brand of soda has
`45 been altered to be a brand of soda that appeals to the demo(cid:173)
`graphic of the viewer. Different viewers, being members of
`different demographic populations, would receive different
`alterations for the brand of soda being consumed by the
`video's character-making each video personalized to fit to
`50 the demographic tastes of each viewer.
`Certain theater systems use stereopsis to provide an illu(cid:173)
`sion of a three-dimensional (3D) image. These systems
`present separate images or film 2D channels for each ofthe
`viewer's eyes. The two image channels may be presented on
`55 a common surface and separated at the viewer's eyes by
`specialized glasses with polarized or colored lenses. Other
`optical techniques may be employed such that each image
`channel is only presented to the appropriate eye. Although the
`discussion in this disclosure is primarily directed to person-
`60 alizing conventional (2D) videos, the personalization tech(cid:173)
`niques disclosedherein could be also applied to the two image
`channels of a stereographic 3D display system.
`A personalized video may be provided in a range of pos(cid:173)
`sible formats including, but not limited to, the following:
`(a) A Digital Rights Management free format that allows
`for downloading and free exchange and trading ofthe video.
`In advertising applications, due to the potential presence of
`
`the embodiments and
`this description,
`Throughout
`examples shown should be considered as exemplars, rather
`than limitations on the apparatus and methods disclosed or
`claimed.
`Throughout this description, the terms "digital video clip", 15
`"video clip", "clip" and "digital video" all connote a digital
`encoding of a series of images with the intent to view the
`images in sequence. There is no implied limitation on a digital
`video's duration, or the final medium which a digital video
`may be expressed. Examples of a digital video include, but
`are not limited to, the following: a portion of a current or
`classic movie or television show, an entire film or TV show, an
`advertisement, a music video, or a specialty clip made spe(cid:173)
`cifically for personalization (for example, a clip that can be
`personalized to show the new actor with a "celebrity friend").
`A single frame image, e.g., one from any of the previous
`examples, can be considered a digital video for some imple(cid:173)
`mentations within the context ofthis specification. Likewise,
`a single image composed via a collage technique between still
`frame images and other digital image manipulation and cre(cid:173)
`ation techniques can also be considered a digital video in
`some implementations within the context of this specifica(cid:173)
`tion. A digital video may be generated via various methods.
`For example, a digital video may have been recorded using a
`digital video camera, may have been digitized from an analog
`video camera or film recording, may have been retrieved from
`a digital medium such as a DVD, may have been created by
`combining multiple still images placed on a timeline for
`animation purposes, may have been created by a composite
`process employing any ofthe above and other processes not
`described here, or otherwise.
`Tue creation of personalized video is a combination of
`multiple fields that in totality allow for the alteration of video
`sequences such that individuals are able to replace the par(cid:173)
`ticipants of an original video with themselves, their friends,
`their family members or any individuals, real or imagined,
`which they have images depicting. This replacement of par(cid:173)
`ticipants in an original video may only require, but is not
`limited to, the replacement of the face, portions of the head
`and/or connecting skin as visible in the original video due to
`framing ofthe view and/or occluding individuals and objects
`in the video sequence blocking the view ofthe entire replaced
`individuals' bodies, costumes and/or wardrobe wom by the
`role depicted by the replaced character within the storyline of
`the video sequence and so forth. Depending upon the content
`of the storyline depicted within the original video, the
`replacement of participants within the video may include
`portions of their other visible skin, such as hands, arms, legs
`and so on.
`Furthermore, the replacement desired may be carried to an
`extreme with the essential removal ofthe original actor in the
`video sequence, the removal of their shadow, reflection and
`visual influence upon other objects in the scene with a com(cid:173)
`pletely synthesized version (a "digital double") ofthe replac(cid:173)
`ing individual and the addition of their characteristic shadow, 65
`reflectionandothervisual influences uponotherobjects in the
`scene.
`
`15
`
`

`

`US 7,460,731 B2
`
`7
`product placements, advertisers may benefit from having per(cid:173)
`sonalized videos exchanged, traded and displayed in as many
`locations as possible.
`(b) A Digital Rights Management enabled format that only
`allows the download and playback on specific devices by
`specific individuals, such as the purchase of a personalized
`video for playback on specific devices.
`( c) 3gp, 3gpp, mv4, giforotherpublic orproprietary digital
`video or di