Case 5:16-cv-00349-NC Document 19-11 Filed 04/11/16 Page 1 of 32
`
`
`Exhibit 8
`
`

`
`Case 5:16-cv-00349-NC Document 19-11 Filed 04/11/16 Page 2 of 32
`°°°“’"eTl||lfil||IlIll|ll|F||1||1|I|H|||1||Mlllilllhlillllllllllflllllllllll
`
`US007752642B2
`
`(12) Ulllted States Patent
`Lemmons
`
`(10) Patent No.:
`(45) Date of Patent:
`
`US 7,752,642 B2
`*Jul. 6, 2010
`
`(54) POST PRODUCTION VISUAL ALTERATIONS
`
`5,794,210 A
`
`8/1998 Goldhaber et al.
`
`(75)
`
`Inventor: Thomas Lemmons, Evergreen, CO (US)
`
`(73) Assignee:
`
`Intellocity USA Inc., San Francisco, CA
`(US)
`
`(Continued)
`FOREIGN PATENT DOCUMENTS
`
`( * ) Notice:
`
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 0 days.
`
`FR
`
`2730837 A1
`
`2/1995
`
`This patent is subject to a terminal dis-
`Claim“
`
`(Continued)
`OTHER PUBLICATIONS
`
`(21) APP1. N04 12/255,801
`
`(22)
`
`Filed:
`
`Oct. 22, 2008
`
`(65)
`
`Prior Publication Data
`US 2009/0094637 A1
`Apr. 9, 2009
`
`Related U.S. Application Data
`
`“U.S.Appl. No. 10/212,289,AdvisoryAction mailed Mar. 12, 2007”,
`3 pgs.
`
`(Continued)
`Primary Examiner—Hunter B. Lonsberry
`(74) Attorney, Agent, or Firm—Schwegman, Lundberg &
`Woessner, P.A.
`
`(63) Continuation of application No. 10/212,289, filed on
`Aug. 2, 2002, now Pat. No. 7,444,659.
`
`(57)
`
`ABSTRACT
`
`(60)
`
`Pf0ViSi0I1a1 aPP1iCati0I1 N0- 60/309,714, filed 011 Aug.
`2, 2001-
`
`(51)
`
`Int CL
`(200601)
`H04N 7/025
`(200601)
`H04N 7/10
`(52) U.S. Cl.
`........................... .. 725/32; 725/34; 725/40;
`725/44
`(58) Field of Classification Search ................. .. 725/32,
`725/34, 40, 44
`See application file for complete search history.
`_
`References Cited
`U.S. PATENT DOCUMENTS
`
`(56)
`
`5,264,933 A
`5,353,392 A
`5,515,485 A
`5,543,856 A
`5,774,666 A
`
`11/1993 Rosser et al.
`10/1994 Luquet et al.
`5/1996 Luquet et al.
`8/1996 Rosser et al.
`6/1998 Portuesi
`
`SCENE ONE 400 ;(
`
`402 REFRIGERATOR
`
`Disclosed is a method and system in which “labels,” compris-
`ing supplemental information such as advertising, promo-
`tional, or informational elements including interactive ele-
`ments, may be superimposed post-production into a video
`stream. This can be done by using overlay screens including
`interactive overlay screens or by combining video segments.
`As such, the labels do not have to physically be part of the
`actual scene during filming. Once a video stream is created, a
`space for available advertising is designated either manually
`or automatically by pattern recognition techniques. A list of
`available advertising space is generated. During viewing of
`television broadcasts, advertisements are placed in the video
`signal. Labels are displayed by superimposing an HTML
`page that is blank except for the advertisement onto the video
`signal. Advertisements can be superimposed post-produc-
`tion, and can be personalized and localized. The advertise-
`ments can be customized on a per-household basis.
`
`20 Claims, 19 Drawing Sheets
`
`SCENE TWO 406?
`
`408 REFRIGERATOR
`
`
`
`

`
`Case 5:16-cv-00349-NC Document 19-11 Filed 04/11/16 Page 3 of 32
`Case 5:16—cv—OO349—NC Document 19-11 Filed 04/11/16 Page 3 of 32
`
`US 7,752,642 B2
`Page 2
`
`U.S. PATENT DOCUMENTS
`
`5,929,850 A
`6,177,931 B1
`6,240,555 B1
`6,263,189 B1
`6,297,853 B1
`6,308,327 B1
`6,324,519 B1
`6,357,042 B2
`6,446,246 B1
`6,446,261 B1
`6,493,872 B1
`7,444,656 B2
`7,444,659 B2
`2001/0023436 A1
`2002/0010923 A1
`2002/0056090 A1
`2002/0059117 A1
`2002/0059588 A1
`2002/0059590 A1
`2002/0112249 A1
`2002/0126990 A1
`2002/0131511 A1
`2002/0133817 A1
`2002/0147987 A1
`2002/0174425 A1
`2003/0028873 A1
`2003/0149983 A1
`2004/0031062 A1
`
`7/1999 Broadwin et al.
`1/2001 Alexander et al.
`5/2001 Shoff et al.
`7/2001 Reagor
`10/2001 Sharir et al.
`10/2001 Liu et al.
`11/2001 Eldering
`3/2002 Srinivasan et al.
`9/2002 Suto
`9/2002 Rosser
`12/2002 Rangan et al.
`10/2008 Lemmons
`10/2008 Lemmons
`9/2001 Srinivasan et al.
`1/2002 Pack et al.
`5/2002 Wagner et al.
`5/2002 Yoch et al.
`5/2002 Huber et al.
`5/2002 Kitsukawa et al.
`8/2002 Hendricks et al.
`9/2002 Rasmussen et al.
`9/2002 Zenoni
`9/2002 Markel
`10/2002 Reynolds et al.
`11/2002 Markel et al.
`2/2003 Lemmons
`8/2003 Markel
`2/2004 Lemmons
`
`FOREIGN PATENT DOCUMENTS
`
`WO
`WO
`WO
`WO
`WO
`
`WO-9115921 A1
`WO-9510919 A1
`WO-9828906 A2
`WO-0043899 A1
`WO-03012744 A1
`
`10/1991
`4/1995
`7/1998
`7/2000
`2/2003
`
`OTHER PUBLICATIONS
`
`“U.S.App1.No. 10/212,289,AdvisoryActionrnai1edNov. 17, 2005”,
`3 pgs.
`“U.S.Appl. No. 10/212,289, Appeal Briefmailed Feb. 17, 2006”, 17
`pgs.
`“U.S. Appl. No. 10/212,289, Interview Summary mailed Jan. 28,
`2005”, 3 pgs.
`“U.S. Appl. No. 10/212,289, Final Office Action mailed May 3,
`2006”, 23 pgs.
`
`“L.S. Appl. No. 10/212,289, Final Office Action mailed Aug. 24,
`2005”, 26 pgs.
`“L.S. Appl. No. 10/212,289, Final Office Action mailed Sep. 29,
`2004”, 21 pgs.
`“L .S.Appl. No. 10/212,289, Non-Final OfficeAction mailedFeb. 17,
`2004”, 11 pgs.
`“L .S. Appl. \Io. 10/212,289, Non-Final Office Action mailed Mar.
`11, 2005”, 23 pgs.
`“L .S. Appl. \Io. 10/212,289, Non-Final Office Action mailed Aug.
`23, 2007”, 26 pgs.
`“L.S. Appl. No. 10/212,289, Notice of Allowance mailed Jul. 1,
`2008”, 6 pgs.
`“L .S. Appl. \Io. 10/212,289, Preliminary Amendment filed Jun. 25,
`2003”, 14 pgs.
`“L.S. Appl. \Io. 10/212,289, Response filed Jan. 31, 2005 to Final
`Office Action mailed Sep. 29, 2004”, 14 pgs.
`“L.S. Appl. \Io. 10/212,289, Response filed Feb. 14, 2008 to Non
`Final Office Action mailed Aug. 23, 2007”, 10 pgs.
`“L.S. Appl. \Io. 10/212,289, Response filed May 17, 2004 to Non-
`Final Office Action mailed Feb. 17, 2004”, 14 pgs.
`“L.S. Appl. \Io. 10/212,289, Response filed Jun. 13, 2005 to Non-
`Final Office Action mailed Mar. 11, 2005”, 15 pgs.
`“L.S. Appl. \Io. 10/212,289, Response filed Oct. 24, 2005 to Final
`Office Action mailed Aug. 24, 2005”, 4 pgs.
`“L.S. Appl. \Io. 10/212,289, Response filed Nov. 3, 2006 to Final
`Office Action mailed May 3, 2006”, 27 pgs.
`“L.S. Appl. No. 10/403,317, Final Office Action mailed Oct. 5,
`2005”, 20 pgs.
`“L .S. Appl. No. 10/403,317, Non-Final Office Action mailed Apr. 7,
`2005”, 17 pgs.
`“L.S. Appl. No. 10/403,317, Notice of Allowance mailed Jun. 27,
`2008”, 6 pgs.
`“L.S. Appl. No. 10/403,317, Response filed Feb. 7, 2008 to Non-
`Final Office Action filed Apr. 7, 2005”, 19 pgs.
`“L .S. Appl. No. 10/403,317, Response filed Jul. 7, 2005 Non-Final
`Office Action mailed Apr. 7, 2005”, 14 pgs.
`[Online] Retrieved
`“HTML Support—Multimedia and Images”,
`from the
`Internet:
`<URL:
`http://www.citycat.ru/doc/HTML/
`IExplorer.30/mmedia.htm#Marquee>, (1996).
`“International Application Serial No. PCT/US02/24463, Interna-
`tional Search Report mailed Dec. 10, 2002”, 7 pgs.
`“International Application Serial No. PCT/US02/24463, Written
`Opinion mailed Feb. 25, 2004”, 6 pgs.
`“International Application Serial No. PCT/US02/24463, Interna-
`tional Preliminary Examination Report mailed Oct. 12, 2004”, 15
`pgs.
`Aaron, Giles, “Transparency Notes”, [Online]. Retrieved from the
`Internet:
`<URL:http://www.mit.edu:8001/peop1e/nocturne/etc/
`Transparency_notes.html>, (Aug. 9, 1994), 2 pgs.
`
`

`
`Case 5:16-cv-00349-NC Document 19-11 Filed 04/11/16 Page 4 of 32
`Case 5:16—cv—OO349—NC Document 19-11 Filed 04/11/16 Page 4 of 32
`
`U.S. Patent
`
`Jul. 6, 2010
`
`Sheet 1 of 19
`
`US 7,752,642 B2
`
`100 DISPLAY DEVICE
`
`
`
`@Pnzz2A
`EJAU1.”S
`
`um
`
`

`
`Case 5:16-cv-00349-NC Document 19-11 Filed 04/11/16 Page 5 of 32
`Case 5:16—cv—OO349—NC Document 19-11 Filed 04/11/16 Page 5 of 32
`
`U.S. Patent
`
`6.
`
`US 7,752,642 B2
`
`2,gm
`zo_2s_§z_m:zo:_8<
`
`m$5120:-
`
`
`
`”<EmNN_n_335.
`
`weJz_80230.1
`
`mmzozétfis.
`
`E01,xm_<NN_.::mEo-mEu
`55Hmo2>%w.am/xfizemE250-
`
`
`is515.3%;2mm8<mzzzéu92:SN
`
`Nmaze:
`
`93559.33~59.Ezo_5E:_=§_$§-R..a\/\ §\/\
`
`15>N52,is1%,;:59
`
`
`
`
`
`52582:Nasoca382%5<8<o%ma_ag><.Em_o8“
`
`
`
`

`
`Case 5:16-cv-00349-NC Document 19-11 Filed 04/11/16 Page 6 of 32
`Case 5:16—cv—OO349—NC Document 19-11 Filed 04/11/16 Page 6 of 32
`
`U.S. Patent
`
`m
`
`US 7,752,642 B2
`
`
`
`
`
`<NN_n__<_mmNN_n_m_S<n_o...z_nm_oz:o1-...z_omozoom-M.5:
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`m_o<n_ms_o_+mzonsoo-mmoéms_o:- 9m_m_:oz<E#69-Mmzonsoo-
`
`<mmanor.
`
`
`
`<NN_n_<mm_omo-
`
`$w_Em>2am
`
`m_w<mE<aO8
`
`

`
`0wW_..o.
`emU
`Case 5:16-cv-00349-NC Document 19-11 Filed 04/11/16 Page 7 of 32
`emC
`
`Fn
`
`.m0O0owmfimcmImEm9;._..mE__o§z:Emu.
`
`
`
`vEm_m-uEmmet3omo_N>x.Mmm._.3m_W_.—|_.<cmmbm,_mSaEo0-M9HL.mmomaw._m..c_..n_..Morm5m_mmmwV_\_,mm%.w_
`
`U.S. Patent
`
`mmm_Em>o<05
`
`mmoz<:n_n_<mmon
`
`mw<m<._.<ogm
`
`
`
`._.3_._<NN_n_
`
`
`
`_o§%Emm-xommNN_n_-
`
`
`
`m>9.m-xosmRem-
`
`
`
`._®u_.wNOn_....CEO:
`
`
`
`
`
`MmE:b%k<m._,wzmmwmfie:wz_z_<»zoo
`
`€m._m-cmE:z_4mi:5NnSm_mE<Emmo_:n_<moME.m_.%5.mo\(_.o.§z:NmmvWJ88.0.aEn.§%N~_g-
`
`
`
`
`
`
`
`
`E_o.._c:cmE-__mamm-mfiom-
`
`
`9-mm_o_>mm_mEzmm_.E_m>xmwmm_m_Em>o<Em
`n40.o_Eom:m._<mmzmwmmn_>oE
`
`
`
`
`
`
`
`
`
`36
`
`2B2Hmmmmaom
`
`,m3,dmfi
`
`
`_._/.7,m_.__n_bmzulowz_z_<pzoo
`
`
`

`
`Case 5:16-cv-00349-NC Document 19-11 Filed 04/11/16 Page 8 of 32
`Case 5:16—cv—OO349—NC Document 19-11 Filed 04/11/16 Page 8 of 32
`
`U.S. Patent
`
`Jul. 6, 2010
`
`Sheet 5 of 19
`
`US 7,752,642 B2
`
`402REFRIGERATOR
`
` SCENEONE400
`
`FIGURE4A
`
`

`
`Case 5:16-cv-00349-NC Document 19-11 Filed 04/11/16 Page 9 of 32
`Case 5:16—cv—OO349—NC Document 19-11 Filed 04/11/16 Page 9 of 32
`
`U.S. Patent
`
`Jul. 6, 2010
`
`Sheet 6 of 19
`
`US 7,752,642 B2
`
`
`
`mowO>>._.mzmom
`
`moEmm_o_Emmmay
`
`uUZ_WFWfl
`
`flm<._Nev“
`
`memmaom
`
`
`

`
`Case 5:16-cv-00349-NC Document 19-11 Filed 04/11/16 Page 10 of 32
`Case 5:16—cv—OO349—NC Document 19-11 Filed 04/11/16 Page 10 of 32
`
`U.S. Patent
`
`Jul. 6, 2010
`
`Sheet 7 of 19
`
`US 7,752,642 B2
`
`<.__j__.-_,
`
`».-,.
`
`-.—‘§‘~
`
`DC
`LU
`
`Z 3U
`
`3
`_:
`<E
`UJ
`
`DC
`PHUL'S PIZZERIH
`
`SCENEONE500
`
`HUTUMDTIUESHLESHNHSERIJIEE
`S@Wfl7HWE
`
`-.=-
`
`._.
`
`_»=~_:.~4-—
`
`
`
`514SUPERIMPOSED
`
`LABELAD TVSCREEN
`
`

`
`Case 5:16-cv-00349-NC Document 19-11 Filed 04/11/16 Page 11 of 32
`Case 5:16—cv—OO349—NC Document 19-11 Filed 04/11/16 Page 11 of 32
`
`U.S. Patent
`
`Jul. 6, 2010
`
`Sheet 8 of 19
`
`US 7,752,642 B2
`
`
`SCENETWO530
`
`INFORMATION
`
`LABEL540
`
`
`
`zzmvzzsesmmMBUNTH'NBUUKSHNUMHGHZINES
` TVSCREEN
`
`SUPERIMPOSED
`
`§§
`33§§
`
`%
`
`S
`
`FIGURE5B
`
`

`
`Case 5:16-cv-00349-NC Document 19-11 Filed 04/11/16 Page 12 of 32
`Case 5:16—cv—OO349—NC Document 19-11 Filed 04/11/16 Page 12 of 32
`
`U.S. Patent
`
`Jul. 6, 2010
`
`Sheet 9 of 19
`
`US 7,752,642 B2
`
`
`
`
`
`9%mmmxpmzmomzmmmom>._.
`
`m>:o<mm:z_
`
`
`
`
`
`
`
`
`
`m_>_5<mm:.z_m>:o<mm_._.z_m>_5<m.n._._.z_
`
`om.uE:w_n_
`
`mw_><E
`
`mszz
`
`._mm<._
`
`

`
`Case 5:16-cv-00349-NC Document 19-11 Filed 04/11/16 Page 13 of 32
`Case 5:16—cv—OO349—NC Document 19-11 Filed 04/11/16 Page 13 of 32
`
`U.S. Patent
`
`Jul. 6, 2010
`
`Sheet 10 of 19
`
`US 7,752,642 B2
`
`TATTOO
`
`LABEL
`
`
`INTERACTIVE
`
` SCENEONE502
`
`FIGUREBA
`
`

`
`Case 5:16-cv-00349-NC Document 19-11 Filed 04/11/16 Page 14 of 32
`Case 5:16—cv—OO349—NC Document 19-11 Filed 04/11/16 Page 14 of 32
`
`U.S. Patent
`
`Jul. 6, 2010
`
`Sheet 11 of 19
`
`US 7,752,642 B2
`
`
`
` SCENETWO612
`
`FIGUREGB
`
`

`
`Case 5:16-cv-00349-NC Document 19-11 Filed 04/11/16 Page 15 of 32
`Case 5:16—cv—OO349—NC Document 19-11 Filed 04/11/16 Page 15 of 32
`
`U.S. Patent
`
`0
`
`US 7,752,642 B2
`
`
`
` m0mEmmA........-.mm_>Emzo:<z_o_mom,VSm._<zo_mxom_o_>
`
`
`89>amoz<:zm-z:
`
`am_oz<:zm_woemo»
`
`
`txommo?%mm»mmom_o_>1<Ezmo
`weom_oz<:zm_-z:
`nmm:.Es_oo
`xz:NEOE
`
`
`
`
`
`mzoEqs_mo“_z_4<zoEoa<mm<m<_.<oo:
`
`m_o_>m_a
`
`03
`
`><.Em_oV2
`
`mn__mSo
`
`momaow
`
`
`
`ms:tabdmfi._<Eo<moxz:dmfiaz<m5m_E.Emm<m<e<a
`
`
`
`
`
`EL
`
`Nm_m:o_u_
`
`
`
`
`
`<._.<Qmsoezoo\._.Zm_>_m0<._n_-
`
`
`
`
`
`NC.1/Lwo__.E<movdm<._-
`
`
`
`
`

`
`Case 5:16-cv-00349-NC Document 19-11 Filed 04/11/16 Page 16 of 32
`Case 5:16—cv—OO349—NC Document 19-11 Filed 04/11/16 Page 16 of 32
`
`U.S. Patent
`
`Sheet 13 of 19
`
`US 7,752,642 B2
`
`6,89>Momumwtzmomoz<:zm;:.
`
`mom
`
`..Al.InIIIII-m4<zo_mmnumo:/
`
`mobpmmWmum
`
`yaummmm
`
`Emezmo
`
`mmSn.s_oo
`
`mmanor.
`
`v_z_._
`
`omoz<:zm-z:
`
`omo_>
`
`Newcom
`
`Lu
`
`2 >L
`
`IJ
`Q
`
`Sm
`
`m_n__mSo
`
`momnom
`
`
`
`
`

`
`Case 5:16-cv-00349-NC Document 19-11 Filed 04/11/16 Page 17 of 32
`Case 5:16—cv—OO349—NC Document 19-11 Filed 04/11/16 Page 17 of 32
`
`U.S. Patent
`
`Jul. 6, 2010
`
`Sheet 14 of 19
`
`US 7,752,642 B2
`
`Emezoou_Ozo_Emmu
`
`4<:z<s_
`
`mo<n_mn_Ozo:<zo_mm_o
`
`Dm:.<_>_O._._...<
`
`moéwu_ozo:<zo_mmo
`
`
`
`ombémzmom_m_o<n_mm._m<.__<><._.55:
`
`ormmmm_Em>o<0...Bow2moimm:mS_<><
`
`N5oz_5&<mom
`
`
`
`89>mi»7:omo<._.u_mm?92_oz_2m__>ozaao<9m._mm<..
`
`wmmem
`
`oms_mou.mwn_
`
`mmm:GE
`
`com
`
`
`
`
`
`Rzozmmaox459£20.25s_<mEmomo_>
`
`
`
`

`
`Case 5:16-cv-00349-NC Document 19-11 Filed 04/11/16 Page 18 of 32
`Case 5:16—cv—OO349—NC Document 19-11 Filed 04/11/16 Page 18 of 32
`
`U.S. Patent
`
`Jul. 6, 2010
`
`Sheet 15 of 19
`
`US 7,752,642 B2
`
`
`
`
`
`N8.1<zo_mom_o_>oezomm<n_3:1m>Smm>omm_Sn_s_oo
`
`
`
`
`
`
`
`voowmOI._.3<>m>.:<:z<2n_mz_s_<xmm_4<zo_momo_>
`
`
`
`
`
`om><._n_m_om_4<zo_m89>%
`
`_.oo—
`
`Noo_.
`
`
`
`
`
`mo:S<>m>._._<Dz<_2.om:<z.w_wm5m_..<zw_mOmQ_>z_I._._>>mo<n_mD<m._m<.__<><
`
`
`
`
`
`
`
`
`
`_._<zo_mom_o_>mm»:25omzazoom_<25maopzoooz<»zm_s_moSn_89
`
`
`
`M$5.3009EmamE<4<zo_m89>az<<»<omsopzoooz<_.zmsm_oSn_o2:
`
`
`
`
`
`
`
`mo".mm5o_zIoEoz:m_xm>momeémzmom_<._.<Qysoezoon_z<._.zm_>_mo<._n_
`modem3m:mS_<><82
`
`
`
`
`o_,mm_:o_u_
`
`
`
`
`
`mo<n_m04..._Ozo:<zo_mm_o1<:z<s_
`
`82
`
`
`
`._.3Ooz_>mm<oMO".mnmcum
`
`
`
`
`
`>mm._mmSu_O.zo:<.:<._wz_mI._.
`
`
`
`
`

`
`Case 5:16-cv-00349-NC Document 19-11 Filed 04/11/16 Page 19 of 32
`Case 5:16—cv—OO349—NC Document 19-11 Filed 04/11/16 Page 19 of 32
`
`U.S. Patent
`
`Jul. 6, 2010
`
`Sheet 16 of 19
`
`US 7,752,642 B2
`
`START
`
`1101
`
`VIDEO IS PLAYED
`
`1102
`
`T
`
`APPLY PATTERN RECOGNITION TECHNIQUES
`
`1104
`
`
` HAS ANYTHING
`BEEN RECOGNIZED
`?
`
`
`
`1106
`
`IS IT AN ACTUAL
`
`OBJECT
`
`1108
`
`7
`
`YES
`
`GENERATE PLACEMENT AND CONTOUR DATA BY EXISTING TECHNOLOGY
`
`COMBINE PLACEMENTAND CONTOUR DATA VVITH VIDEO SIGNAL
`
`1110
`
`SEND PLACEMENTAND CONTOUR DATAAND VIDEO T COMPUTER
`
`12
`
`COMPUTER OVERLAYS HTML PAGE ONTO VIDEO SIGNAL
`
`1114
`
`110
`
`STEPS ILLUSTRATING THE CARRYING
`
`OUT OF THE INSTALLATION OF LABELS
`
`BY AUTOMATIC DESIGNATION OF AD
`
`SPACE AND SEARCHING FOR ACTUAL
`
`OBJECTS
`
`FIGURE 11
`
`

`
`Case 5:16-cv-00349-NC Document 19-11 Filed 04/11/16 Page 20 of 32
`Case 5:16—cv—OO349—NC Document 19-11 Filed 04/11/16 Page 20 of 32
`
`U.S. Patent
`
`Jul. 6, 2010
`
`Sheet 17 of 19
`
`US 7,752,642 B2
`
`1201
`
`1202
`
`START
`
`VIDEO IS PLAYED
`%
`
`—
`
`APPLY PATTERN RECOGNITION TECHNIQUES
`
`1204
`
`
`
` HAS ANYTHING
`BEEN RECOGNIZED
`?
`
`
`1206
`
`1208
`
`IS ITA
`BLANK SPACE
`
`?
`
`YES
`
`GENERATE PLACEMENTAND CONTOUR DATA BY EXISTING TECHNOLOGY
`
`COMBINE PLACEMENTAND CONTOUR DATA WITH VIDEO SIGNAL
`
`1210
`
`SEND PLACEMENT AND CONTOUR DATAAND VIDEO TO COMPUTER 212
`
`COMPUTER OVERLAYS HTML PAGE ONTO VIDEO SIGNAL
`
`121
`
`120
`
`STEPS ILLUSTRATING THE CARRYING
`
`OUT OF THE INSTALLATION OF LABELS
`
`BY AUTOMATIC DESIGNATION OF AD
`
`SPACE AND SEARCHING FOR BLANK SPACES
`
`FIGURE 12
`
`

`
`Case 5:16-cv-00349-NC Document 19-11 Filed 04/11/16 Page 21 of 32
`Case 5:16—cv—OO349—NC Document 19-11 Filed 04/11/16 Page 21 of 32
`
`U.S. Patent
`
`Jul. 6, 2010
`
`Sheet 18 of 19
`
`US 7,752,642 B2
`
`START
`
`1301
`AUTHOR CREATES ADVERTISER LIST
`
`AUTHOR ASSIGNS ATTRIBUTES
`
`1302
`
`STEPS FOR
`AUTOMATIC
`INSTALLATION OF
`LABELS AND
`SEARCHING FOR
`ACTUAL OBJECTS
`
`CREATE DB RECORD
`
`1304
`
`AND BLANK
`
`SAVE TO DB 0
`
`SIMULTANEOUSLY
`
`PLAY THE VIDEO
`
`1303
`
`“__2\13OO
`
`‘
`
`1310
`
`EMPLOY PATTERN
`RECOGNITION COMPONENT
`
`1312 fl
`
`YES
`
`
`ACTUAL
`
`
`
`SPACE
`
`GENERATE PLACEMENT 8:
`CONTOUR DATA
`
`MATCHES
`
`ATTR|BUTES
`
`Z’
`
`1314
`
`1315
`
`1318
`
`
`
`QUERY DB FOR COMPANIES WITH
`
`"BLANK SPACE" AS AN ATTRIBUTE
`
`1328
`
`MATCHES
`1330
`CHOOSE INSERTED LABEL
`
`N0
`
`CHOOSE INSERTED LABEL
`
`1320
`
`CREATE XML AND OVERLAY
`ONTO DESIGNATED SPACE
`
`ONTO DESIGNATED SPACE
`1332
`
`1322
`
`CREATE XML AND OVERLAY
`
`PARSER PARSER PARSER
`
`PARSER PARSER PARSER
`
`CODE
`HTML.JS
`CODE
`FOR
`+
`FOR
`WEBTV CODE FOR OTHER
`STB
`LIBERATE
`STBIS
`STB
`
`1324
`
`1334
`
`CODE
`HTML.JS
`CODE
`FOR
`I}
`FOR
`WEBTV CODE FOR OTHER
`STB
`LIBERATE
`sTB's
`STB
`
`A
`
`FIGURE 13
`
`

`
`Case 5:16-cv-00349-NC Document 19-11 Filed 04/11/16 Page 22 of 32
`Case 5:16—cv—OO349—NC Document 19-11 Filed 04/11/16 Page 22 of 32
`
`U.S. Patent
`
`Jul. 6, 2010
`
`Sheet 19 of 19
`
`US 7,752,642 B2
`
`
`
`CREATION OF CONTENT
`
`
`
`1401
`
`
`
`DESIGNATION OF AVAILABLE
`
`ADVERTISING SPACE AT CONTENT PROVIDER
`
`1402
`
`1404
`
`1406
`
`TAGS ARE INSERTED INTO VIDEO STREAM AT CONTENT PROVIDE '
`
`1408
`g___ MARKERS ARE GEN ERATED AT CONTENT PROVIDER
`
`1410
`
`1412
`
`MARKERS ARE INSERTED INTO VIDEO STREAM
`
`AT CONTENT PROVIDER
`
`"MOVING LOCATION" TAGS
`
`ARE INSERTED INTO VIDEO STREAM
`
`TAGS ARE READ AT HEAD END
`
`(OR BROADCASTING STATION)
`
`1414
`
`LIST OF ADVERTISERS IS SEARCHED
`
`1416
`
`SELECTION OF ADVERTISER AND LABEL
`
`1413
`
`INSERTION OF LABEL
`
`1420
`
`-h C)O
`
`STEPS ILLUSTRATING THE INSTALLATION
`
`OF LABELS BY USING TAGS AND MARKERS
`
`FIGURE 14
`
`

`
`Case 5:16-cv-00349-NC Document 19-11 Filed 04/11/16 Page 23 of 32
`Case 5:16—cv—OO349—NC Document 19-11 Filed 04/11/16 Page 23 of 32
`
`US 7,752,642 B2
`
`1
`POST PRODUCTION VISUAL ALTERATIONS
`
`CROSS REFERENCE TO RELATED
`APPLICATIONS
`
`This application is a Continuation of U.S. application Ser.
`No. 10/212,289, filed on Aug. 2, 2002, now U.S. Pat. No.
`7,444,659, which is based upon and claims the benefit ofU.S.
`Provisional Application No. 60/309,714, entitled “POST
`PRODUCTION VISUAL ALTERATIONS”, filed Aug. 2,
`2001 by Thomas Lemmons, which applications are herein
`incorporated by reference in their entirety.
`
`BACKGROUND OF THE INVENTION
`
`a. Field of the Invention
`
`The present invention pertains generally to video inforrna-
`tion and more particularly to post-production visual enhance-
`ments and modifications.
`
`b. Description of the Background
`Currently, both local and national television commercials
`are placed in 30-second blocks during television shows.
`These advertisements provide the financial backing of mod-
`errn television. However, with growing viewer intolerance of
`these advertisements along with technological advances,
`such as the use of personal video recorders, viewing of adver-
`tisements is decreasing. Thus, companies have expanded into
`embedded advertising, such as selectively placing a product
`in the hands of a television personality during the normal
`programming interval.
`In this manner, companies with
`national advertising budgets can promote products/services
`in a less intrusive manner.
`
`If such embedded advertising was more versatile in nature,
`such advertising could have a widespread impact on the way
`products are promoted. Current methods of embedded adver-
`tising are part ofthe original video production and do not have
`such versatility. In addition, embedded advertising is very
`expensive and inflexible because it must be included in the
`original recording. Currently,
`local affiliates and smaller
`cable systems do not provide any way to embed advertising,
`and therefore, cannot offer embedded advertising. Further,
`once embedded advertising is included in a video production,
`there has been no effective way to remove or alter such adver-
`tising.
`A need therefore exists for providing more flexible embed-
`ded advertising in a simple and inexpensive manner. In that
`regard, the term “advertisement” can refer to various forms of
`visual alteration to the video display, including, but not by
`way of limitation, any type of informational visual alterations
`of the video content. Content may include any type of visual
`information on any desired medium capable. Content may
`comprise various types of data.
`Other procedures have included techniques for “on-con-
`tent” enhancements, such as the use of “hot spots” as dis-
`closed in U.S. Provisional Application Ser. No. 60/243,107,
`filed Oct. 24, 2000 entitled “CREATING ON-CONTENT
`ENHANCEMENTS”, now U.S. Non-Provisional application
`Ser. No. 10/041,881, filed Oct. 24,2001 having the same title,
`both of which are specifically incorporated herein by refer-
`ence for all that they discloses and teach. In accordance with
`the above referenced application, various elements of a par-
`ticular scene may have a “hot spot” (enhancement) associated
`with the element. These hot spots may be presented in a
`number ofways such as with a colored border, a colored spot,
`a dim pulsating border, graying ofthe actual object, or by any
`number of other presentation schemes that distinguish the hot
`spot from a traditional video display element.
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`2
`
`The hot spot, which acts as an interactive link, appears
`directly over the object, and may appear as part of the object,
`in a particular scene. For example, if during a television
`episode the characters are eating pizza, the hot spot, in the
`form of a colored border, spot or other scheme may appear
`over or on the pizza box. Activation ofthe hot spot by a cursor
`or pointer links the viewer to the local pizza company so that
`the viewer can interactively order a pizza while watching the
`show.
`
`SUMMARY OF THE INVENTION
`
`The present invention overcomes the disadvantages and
`limitations of the prior art by providing a method and system
`in which “labels,” comprising supplemental information such
`as advertising, promotional, or
`informational elements
`including interactive elements (i.e., “Hot LabelsTM”), may be
`superimposed into a video stream after production of the
`video. In other words, the labels are not embedded in the
`video and may be altered or replaced. This can be done by
`using overlay screens including interactive overlay screens or
`by combining video segments.As such, the labels do not have
`to physically be part of the actual scene during filming. For
`example, a label may be a person’s name stenciled on their
`forehead or shirt/j ersey, changes in color, pattern, or texture
`within the video presentation, changes in the contour and/or
`placement of objects within the video presentation, or other
`changes, or forms of advertising that are applied to blank
`portions of the video scene, or which replace and obscure
`portions of the video scene.
`The present invention may therefore comprise a method of
`superimposing a label that contains informational material
`onto pre-existing video content to supplement the pre-exist-
`ing video content comprising, detecting a suitable location on
`the pre-existing video content for placement of the labels,
`selecting the label for placement on the pre-existing video
`content, superimposing the label onto the pre-existing video
`content so that the label appears as an integral part ofa unified
`video signal.
`
`The present invention may also comprise a system for
`superimposing labels onto pre-existing video content to pro-
`vide a video signal that includes information to supplement
`the pre-existing video content comprising, a video signal that
`contains the pre-existing video content, a label signal that
`contains the information to supplement the pre-existing video
`content, a logical system that superimposes the label signal on
`the pre-existing video content.
`An advantage of the present invention is that broadcast and
`cable companies are able to provide embedded advertising in
`a post-production environment in a simple and easy manner.
`Labels can be unobtrusive, noninvasive and may not compro-
`mise size, color, or continuity ofthe video content. Embedded
`advertising can be changed from scene to scene and from
`viewing to viewing. As products become outdated, new prod-
`ucts and objects can be superimposed (overlaid) in the video
`signal to update broadcast impact. Specific viewers can view
`individualized and targeted labels based on demographics
`that can be as individualized as finely as a per-household or
`per viewer basis. With this individualization, local broadcasts
`can also exploit the advantages of embedded advertising.
`
`

`
`Case 5:16-cv-00349-NC Document 19-11 Filed 04/11/16 Page 24 of 32
`Case 5:16—cv—OO349—NC Document 19-11 Filed 04/11/16 Page 24 of 32
`
`US 7,752,642 B2
`
`3
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`In the drawings,
`FIG. 1 is an illustration of one implementation of the
`present invention.
`FIG. 2 is an illustration of the manner in which interactive
`
`content may be displayed.
`FIG. 3A is a graphical representation of a database that
`provides a list of additional
`information associated with
`advertisers.
`
`FIG. 3B is a graphical representation of a database that
`provides a list of attributes associated with advertisers.
`FIGS. 4A and 4B are illustrations of scenes showing other
`implementations of the present invention.
`FIGS. 5A, 5B and 5C are illustrations of scenes showing
`other implementations of the present invention.
`FIGS. 6A and 6B are illustrations of scenes showing other
`implementations of the present invention.
`FIG. 7 is a schematic block diagram that generally illus-
`trates one embodiment of the present invention.
`FIG. 8 is a schematic block diagram that generally illus-
`trates another embodiment of the present invention.
`FIG. 9 is a flow diagram that illustrates the steps that may
`be performed in applying labels to a video stream.
`FIG. 10 is a flow diagram illustrating the steps that may be
`performed for the installation of labels by manual designation
`of available advertising space.
`FIG. 11 is a flow diagram illustrating the steps that may be
`performed for the installation of labels by automatic desig-
`nation of available advertising space and searching for actual
`objects.
`FIG. 12 is a flow diagram illustrating the steps that may be
`performed for the installation of labels by automatic desig-
`nation of available advertising space and searching for blank
`spaces.
`FIG. 13 is a flow diagram illustrating the steps that may be
`performed for the installation of labels, using automatic
`methods, in a manner which combines the processes of look-
`ing for an actual object and blank space simultaneously.
`FIG. 14 is a flow diagram illustrating the steps that may be
`performed for the installation of labels by using tags and
`markers to describe advertising space.
`
`DETAILED DESCRIPTION OF THE INVENTION
`
`FIG. 1 is an illustration of one implementation of the
`present invention. Referring to FIG. 1, a scene in the video is
`being displayed. The video may be displayed on a display
`device 100, such as a television set, which may comprise any
`type of display such as a plasma display, flat panel display,
`CRT monitor, or any other display. The video may also be
`displayed on a computer monitor displaying labels within
`video streams over the Internet, or the video may be displayed
`on any other type of display system/device including movies
`displayed at movie theaters. In the scene, a person 102 is
`shown as sitting on a chair 104 at a table 106. The person 102
`is seated in front of an object, such as a pizza box 108, from
`which the person is eating pizza. The object, or pizza box,
`may be automatically identified by some existing technique
`such as pattern recognition software, as will be discussed in
`more detail in FIG. 13. Alternatively, tags and markers, as
`disclosed in the above referenced patent applications, may be
`supplied in the video stream that provides the object identi-
`fication information (pizza) and location of the object in the
`video stream. The system may then determine if a pizza
`advertiser has been listed as a label advertiser. That is, a list
`may be compiled that comprises advertisers who wish to
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`4
`exhibit advertisements, in the form of labels, in the current
`video presentation. The list of advertisers may also include
`the item (attribute), which may be in the form of a product,
`idea, etc., that the advertisers wish to promote. Such a list of
`advertisers and attributes may be stored in a database. This
`database may be stored in a server at the head-end (transmis-
`sion site), retransmission site, set-top box, or other site. The
`list of advertisers and attributes is more fully disclosed with
`regard to the description of FIG. 3.
`Referring again to FIG. 1, once an object, in this instance a
`pizza box 108, is recognized, the advertiser list may be que-
`ried for a pizza advertiser. If a pizza advertiser has been listed
`as a label advertiser, a label, which may be in the form of a
`logo, text, or other form of advertisement for the advertiser, is
`graphically displayed (superimposed or overlaid) on the
`pizza box. The process of applying, or superimposing, the
`label on the pizza box is discussed in further detail with regard
`to the description of FIG. 13. The label may comprise any
`form of video enhancement, including video sequences, text,
`graphical images that may be in bitmap format or any other
`format, or other graphical representation. Ifthe label is a “hot
`label,” the viewer may then click on the hot label to order a
`pizza from that advertised store. Details of this process are
`further discussed with respect to FIG. 13.
`Labels may comprise enhancements to the original video
`signal, such as improvements, augmentations or new addi-
`tions. For example, a company logo may be applied to a
`soda-can. The soda-can may already exist within the original
`video signal, or the soda-can may be an addition to the origi-
`nal signal. Alternatively, changes may comprise downgrades
`from the original video signal, such as degradations, reduc-
`tions, or deletions. For example, a portion ofthe video may be
`“erased” in order to create a blank space, which may be used
`for applying labels or displaying advertisements.
`Labels may be applied to any type of video content and
`video signals such as live or taped a television broadcasts,
`animation, filmed video and the like or may be captured as a
`magnetic or optical media file, such as in a Digital Video/
`Versatile/Variable Disc (DVD) feature-length film. Labels
`may also comprise informational content such as markings.
`Content may be created by taping or filming, such as the
`filming of a live broadcast, a feature-length film, a television
`program, or other presentation. Content may also be created
`using animation. Cartoons on television or on feature-length
`films are one example of animation. Content creation may
`comprise computer animation. Computer animation is a prin-
`cipal component of multimedia presentations including fea-
`ture-length films, and many software applications canbe used
`to create animations that can be displayed on any display
`device, such as a computer monitor, television screen, or
`movie screen.
`
`Of course, labels may be applied to any form of viewing
`material, including educational programs, multimedia pre-
`sentations, or entertainment presentations. Such multimedia
`or
`entertainment presentations may include digitally
`enhanced or computer enhanced presentations, animated pre-
`sentations, which may include feature-length animated films,
`animated programs that are broadcast over a television net-
`work, DVD animated presentations, or other forms of ani-
`mated presentations. Additionally, labels may be applied to
`such multimedia or entertainment presentations as feature-
`length video films that may be shown in movie theatres,
`perhaps in a digital format, over fiber-optic connections,
`through laser technology, or through other technology.
`As mentioned above techniques to facilitate supplemen-
`tary content in a video signal can also be accomplished by
`combining video signals. Such a technique is disclosed in
`
`

`
`Case 5:16-cv-00349-NC Document 19-11 Filed 04/11/16 Page 25 of 32
`Case 5:16—cv—OO349—NC Document 19-11 Filed 04/11/16 Page 25 of 32
`
`US 7,752,642 B2
`
`5
`U.S. Provisional Application Ser. No. 60/278,669, filed Mar.
`20, 2001 entitled “DELIVERY OF INTERACTIVE VIDEO
`CONTENT USING FULL MOTION VIDEO PLANES”,
`now U.S. Non-Provisional patent application Ser. No.
`10/103,545, filed Mar. 20, 2002 entitled “VIDEO COM-
`BINER”, both of which are specifically incorporated herein
`by reference in their entirety. In accordance with the above
`referenced application, the system is capable of combining
`video signals that incorporate labels at the viewer’s location.
`For example, multiple video feeds, some ofwhich can include
`labels, are provided to a viewer’s set-top box together with
`instructions for combining two or more video feeds. The
`video feeds can then be combined in a set-top box or other-
`wise located at or near the viewer’s location to generate the
`combined or correlated video signal for display with labels.
`Additionally, one or more video feeds can comprise
`enhanced video that is provided from an Internet connection.
`HTML-like scripting can be used to indicate the layout of the
`enhanced video signal. Instructions can be provided for
`replacement of particular segments comprising the label
`space, to make an interactive label.
`Hot labels (interactive labels) can appear unobtrusively
`during a television broadcast. As disclosed above, the label
`itself can be an interactive element which is placed on specific
`objects that relate to the advertising element, or in a blank
`portion of the screen. As also disclosed above, labels may not
`be interactive, i.e., labels may exist simply as an advertise-
`ment or informational material that is part ofthe video stream.
`Insertion of video content
`to produce combined video
`streams, as well as insertion of interactive content can be
`accomplished by using tags and markers inserted into the
`video stream as disclosed inU.S. Provisional Application Ser.
`No. 60/268,350, filed Feb. 12, 2001 entitled “VIDEO TAGS
`AND MARKERS”, now U.S.