Case 5:16-cv-00349-NC Document 19-10 Filed 04/11/16 Page 1 of 26
`
`
`Exhibit 7
`
`

`
`Case 5:16-cv-00349-NC Document 19-10 Filed 04/11/16 Page 2 of 26
`”°°“”‘eTl||lfi||||Il|lllllllflllllllllfllllfllmIllilllllfllllllillllflllllllllll
`
`US007444656B2
`
`(12) Ulllted States Patent
`Lemmons
`
`(10) Patent No.:
`(45) Date of Patent:
`
`US 7,444,656 B2
`Oct. 28, 2008
`
`(54) POST PRODUCTION VISUAL
`ENHANCEMENT RENDERING
`
`5,515,485 A
`5,543,856 A
`5,794,210 A
`
`5/1996 Liquet et al.
`8/1996 Rosser etal.
`8/1998 Goldhaber et al.
`
`.............. .. 395/135
`
`.......... .. 705/14
`
`(75)
`
`Inventor: Thomas Lemmons, Evergreen, CO (US)
`
`5,929,850 A
`6,177,931 B1
`
`7/1999 Broadwin et al.
`1/2001 Alexander et al.
`
`......... .. 345/327
`
`(73) Assignee:
`
`Intellocity USA, Inc.
`
`( * ) Notice:
`
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`U"S'C' 1540’) by 543 days"
`(21) Appl. No.: 10/403,317
`
`FR
`wo
`W0
`
`(Continued)
`
`FOREIGN PATENT DOCUMENTS
`2 730 837 A
`8/1996
`wo 9115921 A
`10/1991
`W0 9510919
`4/1995
`OTHER PUBLICATIONS
`
`(22)
`
`Filed:
`
`Mar. 27, 2003
`
`(65)
`
`Prior Publication Data
`US 2004/0031062 A1
`Feb’ 12’ 2004
`Related US_ Application Data
`(63) Continuation-in-part of application No. 10/212,289,
`filed on Aug. 2, 2002.
`
`)
`
`51
`
`(
`
`(60) Provisional application No‘ 60/3093714’ filed on Aug‘
`2’ 2001'
`I t Ci
`'
`n '
`H04N 7/025
`H04N 7/10
`H04N 5/445
`G06F 13/00
`G06F 3/00
`
`U.S. Appl. No. 10/041,881, filed Oct. 24, 2001, Rasmussen et al.
`
`(Continued)
`Primary Examiner—Hunter B Lonsberry
`(Z1401):Szgizlleorriiiei, Agent, or Firm—Schwegman, Lundberg &
`
`(57)
`
`ABSTRACT
`
`Disclosed is a method and system that renders enhancement
`images onto receptor sites of a scene of a pre-existing pro-
`gram. A receptor site defines an area of a scene onto which an
`enhancement image may be rendered and provides informa-
`.
`h
`.
`i d
`.
`i
`.
`.
`tion t at may inc u e size, sca e, rotation, perspective, con-
`tour and coloration. Rendering processes may be direct, pla-
`nar, ‘conformal, blended, or migrated and may produce photo-
`realistic images that appear to be a natural part of a scene.
`Selection of enhancements may be targeted or opportunistic
`and may employ Secondary Selection Criteria including deInO_
`graphics, Viewer preferences, pricing and other information.
`Receptor Site identification may employ human or Software
`resources, or a combination thereof. Receptor site informa-
`tion may be determined through human or software resources
`or a combination thereof and may include the acquisition of
`characteristics of a displayed object image that then may be
`applied to an enhancement image.
`
`(2006.01)
`(2006 01)
`(200601)
`(200601)
`200601
`(
`'
`)
`........................... ..
`U.S. Cl.
`725/40
`_
`_
`_
`(58) Field of Classification Search ................. .. 725/32,
`.
`.
`72§/34’35> 40
`See apphcanon file for Complete Search h1St0ry-
`References Cited
`U.S. PATENT DOCUMENTS
`
`(56)
`
`5,264,933 A * 11/1993 Rosseretal.
`5,353,392 A
`10/1994 Liquet et al.
`
`............. .. 348/578
`.............. .. 395/135
`
`47 Claims, 14 Drawing Sheets
`
`200
`
`K
`
`Iwish I had 3
`Yummy Donut
`ngm now
`
`
`
`

`
`Case 5:16-cv-00349-NC Document 19-10 Filed 04/11/16 Page 3 of 26
`Case 5:16-CV-00349-NC Document 19-10 Filed 04/11/16 Page 3 of 26
`
`US 7,444,656 B2
`Page 2
`
`U.S. PATENT DOCUMENTS
`
`2002/0112249 A1*
`
`8/2002 Hendricks et al.
`
`......... .. 725/136
`
`OTHER PUBLICATIONS
`
`U.S. Appl. No. 09/941,148, filed Aug. 27, 2001, Huber et al.
`U.S. Appl. No. 10/046,618, filed Oct. 26, 2001, Markel et al.
`U.S. Appl. No. 10/076,950, filed Feb. 12, 2002, Zenoni.
`U.S. Appl. No. 10/103,545, filed Mar. 20, 2002, Reynolds et al.
`U.S. Appl. No. 10/099,054, filed Mar. 13, 2002, Markel.
`U.S. Appl. No. 10/357,155, filed Jan. 31, 2003, Markel.
`U.S. Appl. No. 60/688,075, filed Oct. 17, 2003, Huber et al.
`Aaron Giles Transparency Notes Aug 19 1994 hlIl.‘p'WWWII1lt
`.
`’
`’
`'
`’
`’
`'
`'
`'
`edu.8001/people/nocturne/etc/Transparency_notes.htrnl.
`International Preliminary Examination Report mailed Oct. 12, 2004.
`* cited by examiner
`
`Shoff et al.
`Reagor ...................... .. 455/40
`Sham et al.
`. 348/589
`Liu et 31.
`725/37
`Eldering
`N 705/14
`Srinivasan et 31.
`.......... N 725/32
`Sum
`.............. N 725/32
`Rangan et 31.
`.
`.
`
`Snmvasan et 31'
`' 709/219
`.................. .. 725/32
`Pack et al.
`xgacinzi :1t 31' ““““““““ 2;
`Kitsukawa et al.
`.......... .. 725/36
`
`
`
`6,240,555
`6,263,189
`6,297,853
`6,308,327
`6,324,519
`6,357,042
`6,446,246
`6,493,872
`2001/0023436
`2002/0010923
`2002/0056090
`2002/0059117
`2002/0059590
`
`B1
`B1
`B1*
`B1
`B1
`B2
`B1
`B1 *
`A1
`A1
`A1
`A1
`A1
`
`5/2001
`7/2001
`10/2001
`10/2001
`11/2001
`3/2002
`9/2002
`12/2002
`9/2001
`1/2002
`5/2002
`5/2002
`5/2002
`
`

`
`Case 5:16-cv-00349-NC Document 19-10 Filed 04/11/16 Page 4 of 26
`Case 5:16—cv—OO349—NC Document 19-10 Filed 04/11/16 Page 4 of 26
`
`U.S. Patent
`
`Oct. 28,2008
`
`Sheet 1 of 14
`
`US 7,444,656 B2
`
`Figure 1
`
`102 Window
`
`100 Program
`Scene
`
`‘(
`
`104 Actor
`
`
`
`1 10
`
`Telephone
`
`1 08
`
`Magazine
`
`106 Desk
`
`

`
`Case 5:16-cv-00349-NC Document 19-10 Filed 04/11/16 Page 5 of 26
`Case 5:16—cv—OO349—NC Document 19-10 Filed 04/11/16 Page 5 of 26
`
`U.S. Patent
`
`Oct. 28,2008
`
`Sheet 2 of 14
`
`US 7,444,656 B2
`
`Figure 2
`
`K 200
`
`
`
`I wish I had a
`
`Yummy Donut
`
`
`
`Enhancement
`
`
`
`Image
`
`

`
`Case 5:16-cv-00349-NC Document 19-10 Filed 04/11/16 Page 6 of 26
`Case 5:16—cv—OO349—NC Document 19-10 Filed 04/11/16 Page 6 of 26
`
`U.S. Patent
`
`Oct. 28,2008
`
`Sheet 3 of 14
`
`US 7,444,656 B2
`
`Figure 3
`
`1’ Direct
`Rendering
`Process
`
`Steps
`
`302
`
`/
`
`Access Program
`Scene
`
`
`
`Access Receptor /
`Site Information
`
`304
`
`306
`
`Access Enhancement /
`Image
`
`308
`
`Select Image Version /
`Or Scale Image
`
`310
`
`
`Render
` Enhancement
`
`312
`
`/
`
`

`
`Case 5:16-cv-00349-NC Document 19-10 Filed 04/11/16 Page 7 of 26
`Case 5:16—cv—OO349—NC Document 19-10 Filed 04/11/16 Page 7 of 26
`
`U.S. Patent
`
`Oct. 28,2008
`
`Sheet 4 of 14
`
`US 7,444,656 B2
`
`Fig u re 4
`
`Planar Enhancement
`Image 403
`
`402 Window
`
`400
`
`Scene with 3
`Planar Enhancement
`
`

`
`Case 5:16-cv-00349-NC Document 19-10 Filed 04/11/16 Page 8 of 26
`Case 5:16—cv—OO349—NC Document 19-10 Filed 04/11/16 Page 8 of 26
`
`U.S. Patent
`
`Oct. 28,2008
`
`Sheet 5 of 14
`
`US 7,444,656 B2
`
`Fig u re 5
`
`500
`5/ Steps for Rendering a
`Planar Enhancement
`
`Access Program
`Scene
`
`Access Receptor
`Site Information
`
`502
`
`/
`
`504
`
`/
`
`506
`
`508
`
`5 10
`
`S12
`
`
`
`Access Enhancement /
`Image Information
`
` Scale / Clip
`
`Enhancement Image
`
`Modify Enhancement /
`Coloration
`
` Render
`Enhancement
`
`

`
`Case 5:16-cv-00349-NC Document 19-10 Filed 04/11/16 Page 9 of 26
`Case 5:16—cv—OO349—NC Document 19-10 Filed 04/11/16 Page 9 of 26
`
`U.S. Patent
`
`Oct. 28,2008
`
`Sheet 6 of 14
`
`US 7,444,656 B2
`
`Figure 6
`
`600 Scene Showing
`Conformal Enhancement
`
`Images
`
`608
`
`609
`
`Conformal
`Enhancement
`
`Image
`
` Magazine
`
`Telephone
`6 1 0
`
`61 1
`Conformal Enhancement
`
`Image
`
`

`
`Case 5:16-cv-00349-NC Document 19-10 Filed 04/11/16 Page 10 of 26
`Case 5:16—cv—OO349—NC Document 19-10 Filed 04/11/16 Page 10 of 26
`
`U.S. Patent
`
`Oct. 28,2008
`
`Sheet 7 of 14
`
`US 7,444,656 B2
`
`Figure 7
`
`700 Steps for
`K Conformaily Mapping
`Enhancement Images
`
`Access Program
`Scene
`
`Access Receptor
`Site Information
`
`702
`
`/
`
`704
`
`/
`
`Access Enhancement /
`Image Information
`
`706
`
`Scale / Warp
`Enhancement Image
`
`708
`
`/
`
`-
`Modify Conformal
`Enhancement Image
`Coloration
`
`/
`
` 07 1
`
`712
`
`Render Processed
`
`Image
`
`

`
`Case 5:16-cv-00349-NC Document 19-10 Filed 04/11/16 Page 11 of 26
`Case 5:16—cv—OO349—NC Document 19-10 Filed 04/11/16 Page 11 of 26
`
`U.S. Patent
`
`Oct. 28,2008
`
`Sheet 8 of 14
`
`US 7,444,656 B2
`
`Figure 8
`
`800 Program Scene
`5/ With Blended Image
`
`807 Blended
`Enhancement
`
`
`
`Image
`
`806 Desk
`
`

`
`Case 5:16-cv-00349-NC Document 19-10 Filed 04/11/16 Page 12 of 26
`Case 5:16—cv—OO349—NC Document 19-10 Filed 04/11/16 Page 12 of 26
`
`U.S. Patent
`
`Oct. 28,2008
`
`Sheet 9 of 14
`
`US 7,444,656 B2
`
`K 900 Steps for
`Rendering a
`Blended Image
`
`Figure 9
`
`Access Program
`
`Scene
`
`Access Receptor
`
`Site Information
`
`Access Enhancement
`
`Image Information
`
`Scale/Warp/Rotate
`Enhancement Image
`
`Modify Enhancement
`Image Coloration
`
`Render Processed
`
`Image
`
`902
`
`904
`
`906
`
`908
`
`910
`
`912
`
`

`
`Case 5:16-cv-00349-NC Document 19-10 Filed 04/11/16 Page 13 of 26
`Case 5:16—cv—OO349—NC Document 19-10 Filed 04/11/16 Page 13 of 26
`
`U.S. Patent
`
`Oct. 28,2008
`
`Sheet 10 of 14
`
`US 7,444,656 B2
`
`Figure 10A
`
`1000 Original
`/ Scene
`
`
`
`
`1006 Scene with
`)/ Migrated Image
`
`

`
`Case 5:16-cv-00349-NC Document 19-10 Filed 04/11/16 Page 14 of 26
`Case 5:16—cv—OO349—NC Document 19-10 Filed 04/11/16 Page 14 of 26
`
`U.S. Patent
`
`Oct. 28, 2008
`
`Sheet 11 of 14
`
`US 7,444,656 B2
`
`Figure 11
`
`Access Program
`
`Scene
`
`1102
`
`X
`
`1100 Steps for
`Generating a
`Migrated Image
`
`Identify Migration
`Receptor Object
`
`Determine Receptor
`Object Perspective
`And Illumination
`
`Scale/WarplRotate
`Enhancement Image
`
`Migrate Illumination to
`Enhancement Image
`
`Render Processed
`
`Image
`
`

`
`Case 5:16-cv-00349-NC Document 19-10 Filed 04/11/16 Page 15 of 26
`Case 5:16—cv—OO349—NC Document 19-10 Filed 04/11/16 Page 15 of 26
`
`U.S. Patent
`
`Oct. 28,2008
`
`Sheet 12 of 14
`
`US 7,444,656 B2
`
`Figure 12
`
`1200 Steps to Schedule
`Opportunistic Receptor
`Site Utilization
`
`1218
`
`

`
`Case 5:16-cv-00349-NC Document 19-10 Filed 04/11/16 Page 16 of 26
`Case 5:16—cv—OO349—NC Document 19-10 Filed 04/11/16 Page 16 of 26
`
`U.S. Patent
`
`Oct. 28,2008
`
`Sheet 13 of 14
`
`US 7,444,656 B2
`
`Fig ure 1 3
`
`1300 Steps to Identify
`5/ Receptor Sites and
`Schedule Targeted
`Enhancements
`
`Access Enhancement
`Database
`
`
`
`Receptor
`Site ?
`
`1314
`\
`
`Add to Receptor
`Site List
`
`
`
`1316
`
`/
`
`1318
`
`P
`
`Enh ncem nt
`rocess
`a
`e /
`Images
`
`Create Enhancement / 1320
`Schedule
`
`
`
`

`
`Case 5:16-cv-00349-NC Document 19-10 Filed 04/11/16 Page 17 of 26
`Case 5:16—cv—OO349—NC Document 19-10 Filed 04/11/16 Page 17 of 26
`
`U.S. Patent
`
`Oct. 28,2008
`
`Sheet 14 of 14
`
`US 7,444,656 B2
`
`Figure 14
`
`1400 Steps to Identify
`K a Receptor Site in a
`Program Scene
`
`Access Program
`Scene
`
`
`
`Detect Object
`Image Area(s)
`
`1402
`
`1404
`
`/
`
`/
`
`
`
`
`
`
`
`IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII
`
`Path
`1420
`
`1418 \
`
`Determine Receptor
`Site Dimensions
`
`And Perspective
`
`Determine Object
`Contour
`
`
`
`
`

`
`Case 5:16-cv-00349-NC Document 19-10 Filed 04/11/16 Page 18 of 26
`Case 5:16—cv—OO349—NC Document 19-10 Filed 04/11/16 Page 18 of 26
`
`US 7,444,656 B2
`
`2
`
`1
`POST PRODUCTION VISUAL
`ENHANCEMENT RENDERING
`
`CROSS REFERENCE TO RELATED
`APPLICATIONS
`
`This application is a continuation-in-part of U.S. patent
`application Ser. No. 10/212,289, entitled “POST PRODUC-
`TION VISUAL ALTERNATIONS,” filed Aug. 2, 2002 by
`Thomas Lemmons, which was based upon U.S. provisional
`patent application No. 60/309,714, entitled “POST PRO-
`DUCTION VISUAL ALTERATIONS,” filedAug. 2, 2001 by
`Thomas Lemmons, the entire disclosure of which is specifi-
`cally incorporated herein by reference for all that it discloses
`and teaches.
`
`BACKGROUND OF THE INVENTION
`
`a. Field of the Invention
`
`The present invention pertains generally to television and
`streaming media video program presentation and more par-
`ticularly to post-production visual enhancements and modi-
`fications applied to existing program content.
`b. Description of the Background
`A wide range of programs is provided through broadcast
`television, cable and satellite systems. Internet or other net-
`work presentation of such programs may likely increase in the
`future. Typically such broadcasts are commercial broadcasts
`where programs are accompanied by advertisements, com-
`mercial messages, previews and the like. Trends in the tele-
`vision industry include the advent of digital video recorders
`that allow viewers to easily skip advertising content. Addi-
`tionally, viewers of non-recorded programs may change
`channels, do chores, get a snack, mute audio output, or oth-
`erwise stop viewing when commercial messages are pre-
`sented. These trends reduce the viewing audience for adver-
`tising and the effectiveness of advertising messages.
`Therefore a new method of presenting advertising is needed.
`
`SUMMARY OF THE INVENTION
`
`The present invention overcomes the disadvantages and
`limitations of the prior art by providing a system and method
`of rendering visual enhancements onto scenes of an existing
`program. A scene that may comprise a sequence of video
`frames is analyzed to determine potential receptor sites for
`the placement ofvi sual enhancements. Enhancement images,
`that may comprise advertising or other information, may be
`processed to produce a visually modified enhancement image
`that matches characteristics of the receptor site, such as the
`size, shape, color, perspective, and illumination of the recep-
`tor site. Image processing may be employed to alter an
`enhancement
`image such that a photo-realistic image is
`attained that appears to be a natural part of a scene. Such
`images may be employed to unobtrusively display products
`or product messages within program scenes.
`The present invention may therefore comprise a method of
`rendering an enhancement image onto a scene ofpre-existing
`program content comprising: detecting a receptor site for the
`enhancement; determining receptor
`site characteristics
`including type, size, and position of the receptor site; select-
`ing the enhancement image from a plurality of enhancement
`images using the receptor site characteristics; processing the
`enhancement image employing a process that corresponds to
`the receptor site type and that produces a processed enhance-
`ment image; and rendering the processed enhancement image
`at the position of the receptor site.
`
`The present invention may further comprise a method of
`rendering an enhancement image onto pre-existing content
`comprising: accessing a first database containing a plurality
`of enhancement images and information associated with the
`enhancement images including types of the enhancement
`images, selecting the enhancement image; accessing a second
`database containing a plurality of receptor sites and informa-
`tion associated with the receptor sites and selecting one
`receptor site of the plurality of receptor sites using the type of
`the enhancement image; processing the enhancement image
`responsive to the receptor site information to produce a pro-
`cessed enhancement image; and rendering the processed
`enhancement image at the receptor site.
`The present invention may further comprise a method of
`rendering an enhancement image onto pre-existing content
`comprising: accessing a first database containing a plurality
`ofreceptor sites and information associated with the receptor
`sites including receptor site type, selecting one receptor site
`of said plurality ofreceptor sites; accessing a second database
`containing a plurality of enhancement images and informa-
`tion associated with the enhancement images including types
`ofthe enhancement images, selecting the enhancement image
`using the receptor site types of the one receptor site; process-
`ing the enhancement image responsive to the receptor site
`information of the one receptor site to produce a processed
`enhancement image; and rendering the processed enhance-
`ment image at the one receptor site.
`The present invention may further comprise a method of
`identifying a receptor site in a scene of a program at which an
`enhancement image may be rendered comprising: accessing
`the scene of the program; recognizing an object in the scene;
`defining a receptor site in the scene based upon recognition of
`the object; and assigning a type to the receptor site.
`The present invention may further comprise a system for
`rendering an enhancement image onto a scene ofpre-existing
`program content comprising: first program code that detects a
`receptor site for the enhancement; second program code that
`determines receptor site characteristics including type, size,
`and position of the receptor site; third program code that
`selects the enhancement image from a plurality of enhance-
`ment images using the receptor site characteristics; fourth
`program code that processes the enhancement image using
`the receptor site characteristics to produce a processed
`enhancement image; and rendering program code that ren-
`ders the processed enhancement image at the position of the
`receptor site.
`Advantageously, embodiments of the present invention
`allow broadcasters to insert advertising into existing content
`to attain higher viewership of advertising messages and to
`tailor messages to a particular customer or demographic.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`In the drawings,
`FIG. 1 is an illustration of a program scene to which a
`supplemental image may be applied.
`FIG. 2 is an illustration of an enhancement image applied
`to the scene of FIG. 1 employing direct rendering.
`FIG. 3 is a flowchart of steps employed to direct render and
`enhancement image.
`FIG. 4 illustrates a planar enhancement image applied to
`the scene shown in FIG. 1.
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`FIG. 5 is a flowchart of steps employed to render a planar
`enhancement.
`
`65
`
`FIG. 6 is an illustration of conformal enhancement images
`applied to the scene of FIG. 1.
`
`

`
`Case 5:16-cv-00349-NC Document 19-10 Filed 04/11/16 Page 19 of 26
`Case 5:16—cv—OO349—NC Document 19-10 Filed 04/11/16 Page 19 of 26
`
`US 7,444,656 B2
`
`3
`FIG. 7 is a flowchart of steps that may be employed to
`conformally map an enhancement image to an apparent
`object.
`FIG. 8 illustrates the application of a blended image to the
`program scene of FIG. 1.
`FIG. 9 is a flowchart of steps that may be employed to
`render a blended image.
`FIGS. 10A and 10B illustrate a program scene to which a
`migrated enhancement image may be applied.
`FIG. 11 is a flowchart of steps performed in migrating the
`characteristics of a displayed object
`to an enhancement
`image.
`FIG. 12 is a flowchart of steps performed to schedule
`opportunistic receptor site utilization.
`FIG. 13 is a flowchart of steps that may be performed to
`identify receptor sites and to schedule targeted enhance-
`ments.
`
`FIG. 14 is a flowchart of steps that may be performed to
`identify a receptor site in a scene of a program.
`
`DETAILED DESCRIPTION OF THE INVENTION
`
`FIG. 1 is an illustration of a program scene to which a
`supplemental image may be applied. Program scene 100
`comprises window 102, actor 104, desk 106, magazine 108,
`and telephone 110. Embodiments of the present invention are
`directed to adding an enhancement image to existing program
`content such as program scene 100 of FIG. 1. Supplemental
`audio may also accompany the enhancement
`image. An
`enhancement image may be employed to provide additional
`information concerning displayed content or may be
`employed to add new information, such as advertising, for
`example. An enhancement image may be processed in a num-
`ber ofways prior to being rendered. Such rendering processes
`may be termed as direct, planar, conformal, blended, or
`migrated. Advantageously, processed enhancement images
`may be employed to add commentary, advertising, humor, or
`other information that may increase viewer enjoyment and
`eflicacy of advertising.
`The selection, processing, and rendering of an enhance-
`ment image uses receptor site information. A receptor site is
`an area within a scene of a program where an enhancement
`image may be rendered and includes label spaces as disclosed
`in the patent application Ser. No. 10/212,289, entitled “POST
`PRODUCTION VISUAL ALTERATIONS,” filed Aug. 2,
`2002 by Thomas Lemmons. A “label space,” is a space within
`the content (which may be a video stream) that is available for
`application ofa label . A label may comprise any form ofvideo
`enhancement,
`including video sequences,
`text, graphical
`images that may be in bitmap format or any other format, or
`other graphical representation. Ifthe label is a “hot label,” the
`viewer may then click on the hot label or otherwise activate a
`link associated with the label to access additional informa-
`tion.
`
`Receptor site information may include size, perspective,
`type, illumination, and clipping information. Human opera-
`tors, pattern recognition and other software, or a combination
`thereof, may be employed to identify a receptor site and to
`assign a type to the receptor site. A plurality of receptor sites
`may exist within one scene. Receptor site identification may
`be targeted or opportunistic. For targeted identification, a
`particular enhancement or set of enhancements is identified
`and program scenes are then inspected for a suitable receptor
`site. For opportunistic identification, scenes of a program are
`inspected for possible receptor sites and once identified, a
`database may be parsed to select an enhancement or group of
`enhancements that correspond to an identified receptor site
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`4
`
`type. Parsing of the database may employ receptor site type
`and size, or other information, to determine a set of “candi-
`date” enhancements. Selection of a specific enhancement
`image may employ a range of processes and may use a com-
`bination of price, frequency, demographics, user preference,
`subscription type, type of service, or other information for
`selection. Such processes are termed secondary selection pro-
`cesses. Price information may be employed to select among
`possible enhancement images to maximize earnings. Price
`based selection of enhancements may be real-time or near
`real-time such that potential advertisers may “bid” on adver-
`tising opportunities. Methods may include those disclosed in
`U.S. Pat. No. 5,794,210 entitled “Attention Brokerage”
`issued Aug. 11, 1998 to Golhaber et al; and in U.S. Pat. No.
`6,253,189 entitled “System and Method for Completing
`Advertising Time Slot Transactions” issued Jun. 26, 2001 to
`Freezell et al; and in U.S. Pat. No. 6,324,519 entitled “Adver-
`tisement Auction System” issued Nov. 27, 2001 to Eldering,
`all of which are specifically incorporated herein by reference
`for all that they teach and disclose.
`Advertising enhancement selection may reflect user pref-
`erences or other information and may employ methods dis-
`closed in U.S. provision patent application Ser. No. 60/420,
`1 10 entitled “iChoose Video Advertising” filed Oct. 18, 2002
`by Tom Huber and Thomas Lemmons, which is specifically
`incorporated herein by reference for all that it teaches and
`discloses. Frequency criteria may be employed to repeat
`advertising messages a predetermined number of times, such
`as may be used to promote product or brand name familiarity;
`or messages may be scheduled to occur in some order such
`that the viewer’s interest or curiosity may be stimulated by
`having a changing array of messages presented. Demo-
`graphic criteria may be employed to select messages tailored
`to a specific segment of the population, time of showing,
`program type, and the like.
`Enhancement images may be “static” or animated and may
`be displayed for a predetermined duration, as may be
`expressed as a period of time or a number of frames of a
`presentation. Animation may include apparent motion and
`may include fade-in, fade-out, dissolve, sparkle or other
`effects. The position of an enhancement image may be varied
`during display. Enhancement images are not limited in format
`and may include GIF, TIFF, TIF, JPEG, bitmaps, motion
`JPEG, MPEG, three dimensional models, and other formats.
`Enhancement images may be rendered at a broadcast site or at
`the receiving site. Processing of enhancement images may
`occur at a broadcast site, at a receiving site, or both where
`tasks may be divided between the broadcast site and the
`receiving site.
`FIG. 2 is an illustration of a modified scene 200 with
`
`enhancement image 205 applied to the modified scene 200 of
`FIG. 1 employing direct rendering. In FIG. 2, modified pro-
`gram scene 200 comprises actor 204 and direct rendered
`enhancement
`image 205. Direct rendering refers to the
`enhancement image being rendered with little or no modifi-
`cation, such that little or no processing ofenhancement image
`205 is required. Rendering may comprise selection of an
`enhancement image that matches, nearly matches, or matches
`without exceeding the size of the receptor site. The enhance-
`ment image may be scaled to attain a desired size. In FIG. 2,
`direct rendered enhancement image 205 is employed to indi-
`cate the actor’ s thoughts. Direct rendered enhancements may
`be employed to add messages, humor, icons, logos, and other
`information.
`
`FIG. 3 is a flowchart of steps employed to directly render an
`enhancement image. Process 300 begins at step 302 where a
`scene of a program containing a suitable receptor site is
`
`

`
`Case 5:16-cv-00349-NC Document 19-10 Filed 04/11/16 Page 20 of 26
`Case 5:16—cv—OO349—NC Document 19-10 Filed 04/11/16 Page 20 of 26
`
`US 7,444,656 B2
`
`5
`accessed. As noted previously, receptor site identification
`may be deterministic or opportunistic. At step 304, receptor
`site information is accessed to determine the position and size
`of the receptor site. At step 306, enhancement image infor-
`mation is accessed. At step 308, a check is performed to
`determine if the size of the enhancement image is to be
`altered. This check may comprise comparing the enhance-
`ment image size with the receptor site size. Ifthe check of step
`308 determines that the size of the enhancement image does
`not need to be altered, processing continues at step 312 where
`the enhancement image is rendered. Ifthe check performed at
`step 308 determines that the enhancement image size is to be
`altered, processing continues at step 310 where the image is
`scaled or clipped, or alternatively, if multiple versions of an
`enhancement image are provided, a version that corresponds
`to the receptor site size may be selected. Processing then
`continues at step 312 where the enhancement image is ren-
`dered. In another embodiment of the present invention, step
`306 may comprise accessing an enhancement image from a
`plurality of images that corresponds to the size ofthe receptor
`site, or there may be only one enhancement image wherein
`the enhancement image is of the desired size, such that steps
`308 and 310 are not employed. Enhancement image informa-
`tion may include how the enhancement image is to be ren-
`dered. Rendering is a process in which the enhancement
`image replaces or is combined with a portion of the program
`scene and is described below in greater detail.
`FIG. 4 illustrates a planar enhancement image applied to a
`scene 400. Planar enhancements are processed to conform to
`an apparent surface shown in a program scene. Receptor sites
`for planar images may correspond to windows, pictures, pic-
`ture frames, signs, billboards, computer or television screens,
`panels in a stadium, or similar surfaces and areas. In FIG. 4,
`the modified program scene 400 comprises window 402,
`planar enhancement
`image 403, and actor 404. Planar
`enhancement image 403 is an “out of the window” scene
`containing an automobile. As shown in FIG. 4, a planar
`enhancement image 403 may be employed to promote prod-
`uct or brand familiarity. Planar enhancement images may be
`employed to generate humor, such as having a different scene
`out a window each time the window is shown, a different
`picture in a picture frame each time the picture frame is
`shown, or by showing humorous or nonsensical images. For
`example, scenes from an old sci-fi movie depicting aliens
`might be shown as a window scene to add humor to existing
`content. Planar enhancement receptor sites may correspond
`to two-dimensional surfaces such that image processing may
`employ two-dimensional
`scaling. Processing of planar
`enhancement images may also include color modification.
`For example, the intensity of an enhancement image may be
`modified to match or closely match the lighting conditions of
`a scene such that the rendered image appears as a natural part
`of the scene.
`
`FIG. 5 is a flowchart of the steps 500 employed to render a
`planar enhancement. At step 502, a scene of a program con-
`taining a suitable receptor site is accessed. At step 504, recep-
`tor site information is accessed to determine position, orien-
`tation, and dimensions of the receptor site. At step 506,
`enhancement image information is accessed. Enhancement
`image information may include how the enhancement image
`is to be combined with program content and if the enhance-
`ment image should be scaled or clipped. At step 508 the
`enhancement image is scaled or clipped employing informa-
`tion from step 504. At step 510 the scaled or clipped enhance-
`ment image may be color modified. At step 512 the processed
`enhancement image is rendered.
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`6
`FIG. 6 is an illustration of conformal enhancement images
`applied to an object already existing in a scene. In FIG. 6,
`modified program scene 600 comprises first conformal
`enhancement image 609 conformally rendered on magazine
`image 608 and second conformal enhancement image 611
`conformally rendered on telephone image 610. The confor-
`mal enhancement images depicted in FIG. 6 illustrate how
`images in a program scene may be associated with a commer-
`cial product.
`FIG. 7 is a flowchart of steps that may be employed to
`conformally map an enhancement image to an object already
`existing in a scene. Process 700 begins at step 702 where a
`scene of a program containing a suitable receptor site on an
`existing object is accessed. At 704, receptor site information
`is accessed to determine information including size, position,
`contour, and illumination. At step 706, enhancement image
`information is accessed. Enhancement image information
`may include how the enhancement image is to be combined
`with program content and what properties of the receptor site
`should be applied to the enhancement image, such as reflec-
`tivity, for example. At step 708, size and contour information
`of the receptor site are employed to scale and/or warp the
`enhancement image. Warping comprises non-linear scaling
`as may be employed to map a bit mapped image to a surface,
`such as a sphere or other curved surface for example. At step
`710, illumination information is employed to modify the
`color of the enhancement image. Illumination information
`may comprise color, gradient, and reflectivity. Color in any of
`the embodiments of the present invention may be expressed
`as hue and saturation or may employ any other color spaces,
`including RGB, or Y-Cr-Cb for example. Gradient informa-
`tion may specify variation in color or intensity across the
`image. The enhancement image information accessed at step
`706 may include information employed to control color
`modification. In one embodiment it may be desired that an
`enhancement image blend into the program scene and appear
`as if it were part ofthe original scene. In another embodiment
`it may be desired that the enhancement object stand out from
`the scene. Color information included as part of enhancement
`image information may override or take priority over color
`information associated with a receptor site. At step 712, the
`color-processed image is rendered at the receptor site. As
`previously noted, enhancement images may be animated. For
`example, the word “YUM” may be mapped to the surface of
`a pizza box that already contains pizza company information.
`The word “YUM” may be animated to appear for a predeter-
`mined duration and then be removed. The word “YUM” may
`dissolve, pop, shimmer, morph into a pizza delivery carica-
`ture and run away, or produce any other animated effect.
`FIG. 8 illustrates the application of a blended image to a
`program scene. A blended image is an image that is rotated
`and scaled, or warped, to match receptor site size and per-
`spective, and that is color processed to correspond to illumi-
`nation of the receptor site. Through this process, a blended
`image is a photo-realistic image that appears to be a natural
`part of a scene. A blended image may be employed to display
`advertising related objects or to introduce humor, game
`objects (find the golden key in scenes, for example) and the
`like. In FIG. 8, modified program scene 800 comprises
`blended enhancement
`image 807 rendered on desk 806.
`Blended enhancement image 807 is an image of a soft drink
`can that has been scaled, rotated, colored and inserted in the
`scene to appear as a natural part of the scene. The color
`processing of blended enhancement image 807 may include
`reflectivity where color from the receptor site or surrounding
`area may be used to define the color of the blended image.
`Advantageously, blended images allow product messages to
`
`

`
`Case 5:16-cv-00349-NC Document 19-10 Filed 04/11/16 Page 21 of 26
`Case 5:16—cv—OO349—NC Document 19-10 Filed 04/11/16 Page 21 of 26
`
`US 7,444,656 B2
`
`7
`be unobtrusively introduced into a scene without interrupting
`the flow of a program presentation. Blended images may be
`employed to place objects on desks, tables, walls, streets,
`buildings, and virtually anywhere in a scene that would
`appear to be a part of the scene. Billboards may be placed in
`outdoor scen