(12) United States Patent
`Gharaat et al.
`
`(10) Patent No.:
`(45) Date of Patent:
`
`US 9,554,176 B2
`Jan. 24, 2017
`
`USO09554.176B2
`
`(54)
`
`(71)
`(72)
`
`(73)
`(*)
`
`(21)
`(22)
`(65)
`
`(63)
`
`(51)
`
`(52)
`
`(58)
`
`MEDIA CONTENT FINGERPRINTING
`SYSTEM
`
`Applicant: TiVo Inc., Alviso, CA (US)
`Inventors: Amir H. Gharaat, Menlo Park, CA
`(US); James M. Barton, Alviso, CA
`(US); Mukesh K. Patel, Fremont, CA
`(US)
`Assignee: TiVo Inc., San Jose, CA (US)
`Notice:
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 108 days.
`Appl. No.: 13/965,125
`Filed:
`Aug. 12, 2013
`
`Prior Publication Data
`US 2013/0332951 A1
`Dec. 12, 2013
`US 2016/0353181 A9
`Dec. 1, 2016
`Related U.S. Application Data
`Continuation of application No. 12/631,783, filed on
`Dec. 4, 2009, now Pat. No. 8,510,769.
`(Continued)
`
`Int. C.
`H04N 5/765
`H04N 2L/435
`
`(2006.01)
`(2011.01)
`(Continued)
`
`U.S. C.
`CPC .......... H04N 21/.435 (2013.01); G06F 3/1454
`(2013.01); G06F 17/3002 (2013.01);
`(Continued)
`Field of Classification Search
`CPC .............. H04N 21/8358; H04N 21/442; H04N
`21/44213: HO4N 21/44222: HO4N 21/422:
`H04H 2201/90; H04H 60/31
`(Continued)
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`4,230,990 A * 10/1980 Lert, Jr. ............. GO6K9/00711
`348/473
`
`4.420,769 A 12/1983 Novak
`(Continued)
`
`FOREIGN PATENT DOCUMENTS
`
`EP
`EP
`
`O627857 A1 12, 1994
`2083568 A1
`T 2009
`(Continued)
`
`OTHER PUBLICATIONS
`
`European Patent Office, “Search Report” in application No.
`108162884-1902, dated Nov. 29, 2013, 6 pages.
`(Continued)
`
`Primary Examiner — David Harvey
`(74) Attorney, Agent, or Firm — Wong & Rees LLP
`
`ABSTRACT
`(57)
`A method of deriving fingerprints for media content that is
`being watched by a user is described. For example, a user
`may select a particular show on an electronic programming
`guide displayed by a media device. The media device may
`then request the content stream, from the content source, that
`includes the particular show. The Source may indicate
`whether a fingerprint is needed for the particular show
`requested by the media device. The indication may be a flag
`in the data received by the media device. If the particular
`show needs to be fingerprinted as indicated by the flag, the
`media device may decompress the corresponding video
`frames, load the decompressed video frames into memory
`and analyze the video frames to derive a fingerprint from the
`video frames.
`
`21 Claims, 12 Drawing Sheets
`
`
`
`Fingerprint Database
`602
`
`
`
`
`
`Fingerprint Management Engine
`604
`
`Media Dewice. A
`606
`
`Media Dewice B
`808
`
`Mediaevice C
`610
`
`Media Device M
`620
`
`
`
`EX1030
`Roku V. Media Chain
`U.S. Patent No. 10,489,560
`
`

`

`US 9,554,176 B2
`Page 2
`
`(60)
`
`(51)
`
`(52)
`
`(58)
`
`(56)
`
`Related U.S. Application Data
`Provisional application No. 61/242.277, filed on Sep.
`14, 2009.
`
`(2006.01)
`(2012.01)
`(2012.01)
`(2012.01)
`(2012.01)
`(2006.01)
`(2006.01)
`(2006.01)
`(2011.01)
`(2011.01)
`(2011.01)
`
`Int. C.
`G06F 7/30
`G06O 30/02
`G06O 30/04
`G06O 30/06
`G06O 50/00
`GIB 27/32
`G06F 3/4
`HO)4N 9/79
`H04N 2L/422
`H04N 2L/8358
`H04N 2L/442
`U.S. C.
`CPC ..... G06O 30/0244 (2013.01); G06O 30/0255
`(2013.01); G06O 30/04 (2013.01); G06Q
`30/0631 (2013.01); G06O 50/01 (2013.01);
`GIIB 27/322 (2013.01); H04N 5/765
`(2013.01); H04N 9/79 (2013.01); H04N
`21/422 (2013.01); H04N 2 1/42201 (2013.01);
`H04N 21/442 (2013.01); H04N 21/8358
`(2013.01)
`
`Field of Classification Search
`USPC ...................................................... 725/19, 20
`See application file for complete search history.
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`7,774.407
`7,784,078
`7,805,055
`7,818,771
`7,822,815
`7,840,980
`7,908,635
`7.912206
`7,930,714
`7,984.473
`7.991, 129
`8,000,685
`8,006.261
`8,032,383
`8,032,909
`8,094,872
`
`8,156.520
`8,233,601
`8,265,990
`8,306,859
`8,346,798
`8,351,761
`8,364,703
`8,370,765
`8,381,241
`8,417,096
`8,538,324
`8,910,199
`8,984,626
`9,036,979
`9,264,758
`9,369,758
`2001/0023431
`2002fOO37104
`2002.0099.552
`2002fOO99774
`2002/01 19800
`2002/O124252
`2002/O127000
`20O2/O152117
`2002fO188945
`2003/0093580
`2003/OO93790
`2003/0093806
`2003/OO95791
`2003.0114145
`2003/O1266.00
`2003/O156827
`2003. O163823
`2003/0177503
`2003/0202773
`2004, OO15992
`2004.0043724
`2004/0059570
`2004f00786.10
`2004/OO78817
`2004.0143349
`2004/O1898.73
`2004/O220858
`2004/0226035
`2004/O250212
`2004/0260682
`2005/0022251
`2005/0O25249
`2005, 00281.61
`2005/0044189
`2005/0044561
`2005/0066366
`2005/0101.303
`2005/0149968
`2005/0172312
`2005, 0183041
`2005/023.9545
`2006, OO15609
`2006, OO23849
`2006.0036688
`2006, OO64721
`2006, OO74658
`
`B2
`B2
`B2
`B2
`B2
`B2
`B2
`B2
`B2
`B2
`B2
`B2
`B2
`B2
`B2
`B2
`A1
`A1
`A1
`A1
`A1
`A1
`A1
`A1
`A1
`A1
`A1
`A1
`A1
`A1
`A1
`A1
`A1
`A1
`A1
`A1
`A1
`A1
`A1
`A1
`A1
`A1
`A1
`A1
`A1
`A1
`A1
`A1
`A1
`A1
`A1
`A1
`A1
`A1
`A1
`A1
`A1
`A1
`A1
`A1
`A1
`A1
`
`8, 2010
`8, 2010
`9, 2010
`10, 2010
`10, 2010
`11/2010
`3, 2011
`3, 2011
`4, 2011
`T/2011
`8, 2011
`8, 2011
`8, 2011
`10, 2011
`10, 2011
`1, 2012
`
`4, 2012
`T/2012
`9, 2012
`11/2012
`1, 2013
`1, 2013
`1, 2013
`2, 2013
`2, 2013
`4, 2013
`9, 2013
`12/2014
`3, 2015
`5/2015
`2, 2016
`6, 2016
`9, 2001
`3, 2002
`T/2002
`T/2002
`8, 2002
`9, 2002
`9, 2002
`10, 2002
`12, 2002
`5/2003
`5/2003
`5/2003
`5/2003
`6, 2003
`T/2003
`8, 2003
`8, 2003
`9, 2003
`10, 2003
`1, 2004
`3, 2004
`3, 2004
`4, 2004
`4, 2004
`T/2004
`9, 2004
`11/2004
`11/2004
`12, 2004
`12, 2004
`1/2005
`2, 2005
`2, 2005
`2, 2005
`2, 2005
`3, 2005
`5/2005
`7/2005
`8, 2005
`8, 2005
`10, 2005
`1, 2006
`2, 2006
`2, 2006
`3, 2006
`4, 2006
`
`Daly et al.
`Ohnuma et al.
`Kimoto et al.
`Duchenaut et al.
`Gress et al.
`Gutta
`Barton et al.
`Miller et al.
`Konig et al.
`Casile et al.
`Timmins et al.
`Benco et al.
`Haberman et al.
`Bhardwaj et al.
`Davis et al.
`Yagnik ............... GO6K9/00711
`380,201
`
`Casagrande et al.
`Mendilloa
`Penberthy et al.
`Lerman et al.
`Spiegelman et al.
`Harris
`Ramanathan et al.
`Martin
`Ramaswamy et al. ......... 725/14
`Gharaat et al.
`Hardacker et al.
`Slaney et al.
`Barton et al.
`Gharaat et al.
`Gharaat et al.
`Gharaat et al.
`Horiguchi
`Myers et al.
`Rubin et al.
`Yamato et al.
`Jaggers et al.
`Schaaefer
`Yamato et al.
`Cristofalo et al.
`McGee et al.
`Thomas et al.
`Logan et al.
`Dureau et al.
`Barton et al.
`Boda et al.
`Heuvelman
`Janevski
`Logan et al.
`Sull et al.
`Dow et al.
`Hasegawa et al.
`Weast
`Mochinaga et al.
`Naitoh
`Horowitz et al.
`Roberts et al.
`Konig et al.
`Maggio
`Hauser, Jr.
`Fish
`Herley et al.
`Ohnuma et al.
`Zhao et al.
`Numakami et al.
`IkeZoye et al.
`McDonald
`Takamine
`Pelaez et al.
`Konig et al.
`Lienhart et al.
`Chiu et al.
`Rowe
`Hagale et al.
`Timmins et al.
`McMahan et al.
`Del Val et al.
`Chadha
`
`6, 1988 Novak
`4,750,213 A
`8, 1994 Nafeh
`5,343,251 A
`5,469,207 A 11/1995 Chambers
`5,481.294 A *
`1/1996 Thomas et al. ................. 725/2O
`5,504,518 A
`4, 1996 Ellis et al.
`5,617,539 A
`4/1997 Ludwig et al.
`5,668.917 A
`9, 1997 Lewine
`5,748,263. A
`5, 1998 Ball
`5,956.455 A
`9/1999 Hennig
`6,167.428 A 12, 2000 Ellis
`6,243,445 B1
`6/2001 Begeja et al.
`6,317,784 B1
`1 1/2001 Mackintosh et al.
`6,393,412 B1
`5/2002 Deep
`6,404,977 B1
`6/2002 Iggulden
`6,469,749 B1
`10/2002 Dimitrova et al.
`6,633,651 B1 10/2003 Hirzalla et al.
`6,636,688 B1 10/2003 Otana
`6,725,250 B1
`4, 2004 Ellis
`6,771,316 B1
`8/2004 Iggulden
`6,771,885 B1
`8/2004 Agnihotri et al.
`6,795,530 B1
`9, 2004 Gilbert et al.
`6,928,231 B2
`8/2005 Tajima
`6,963,897 B1
`11/2005 Hubbard
`7,080,392 B1
`7/2006 Geshwind
`7,170,566 B2
`1/2007 McGee et al.
`7,171,108 B1
`1/2007 Masters et al.
`7,222,071 B2 *
`5/2007 Neuhauser ............ G1 OL19,018
`TO4,231
`
`7,269,330 B1
`7,302,160 B1
`7,418,497 B2
`7,477,739 B2
`7.483,835 B2 *
`
`9/2007 Iggulden
`11/2007 Wells
`8/2008 Hagale et al.
`1/2009 HaitSma et al.
`1/2009 Neuhauser ............. HO4H2O?33
`704/27O
`
`7,486,875 B2
`7,548,950 B2
`7,571,226 B1
`7,668,867 B2
`7,738,704 B2
`
`2/2009 Yamashita et al.
`6/2009 Numakami et al.
`8, 2009 Partoviet al.
`2/2010 Morris et al.
`6/2010 Lienhart et al.
`
`

`

`US 9,554,176 B2
`Page 3
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`5, 2006 Cancel
`2006.0109854 A1
`5, 2006 Dhanakshirur
`2006/011 1917 A1
`6, 2006 Shostak
`2006/0116885 A1
`6/2006 Gurram et al.
`2006, O136220 A1
`7/2006 Laksono et al.
`2006, O161947 A1
`8, 2006 Sezan et al.
`2006/0174277 A1
`8/2006 Konig et al.
`2006/O195859 A1
`2006/0230422 A1 10, 2006 Kunin et al.
`2006/0239642 A1 10, 2006 Yuasa et al.
`2006/0245625 A1
`11/2006 Tichelaar et al.
`2006/0259942 A1 11/2006 Toyama et al.
`2006/027 1947 A1
`11/2006 Lienhart et al.
`2006/0271959 A1 1 1/2006 Jacoby et al.
`2006/0282856 All
`12/2006 Errico et al.
`2006, O294557 A1 12, 2006 Morris et al.
`2007/0001806 A1
`1/2007 PO11
`2007/0O83887 A1
`4/2007 Gutta et al.
`2007/0092204 A1
`4/2007 Wagner et al.
`2007/O112939 A1
`5, 2007 Wilson et al.
`2007. O136744 A1
`6, 2007 Simons et al.
`2007/013.6782 A1* 6/2007 Ramaswamy .......... G11B 27.11
`725, 138
`
`6/2007 Rokusek et al.
`2007/O140187 A1
`6, 2007 Cowell et al.
`2007/O143778 A1
`2007/0157249 A1* 7/2007 Cordray et al. ............. 7.25/58
`2007/O169155 A1
`7/2007 Pasquale et al.
`2007,0186242 A1
`8, 2007 Price et al.
`2007,0192410 A1
`8/2007 Liversidge et al.
`2007/0247477 A1 10/2007 Lowry et al.
`2007, O25071.6 A1 10, 2007 Brunk et al.
`2007/0283380 A1 12, 2007 Aoki et al.
`2007/0286463 A1 12, 2007 Ritzau et al.
`2007/0294081 Al 12/2007 Wang et al.
`2008/0040807 A1
`2/2008 Lu et al.
`2008, OOS2742 A1
`2/2008 Kopfet al.
`2008. O130958 A1
`6, 2008 Ziomek
`2008/0170836 A1
`7/2008 Kinoshita et al.
`2008/O175566 A1
`7/2008 Khedouri et al.
`2008. O184326 A1
`7/2008 Nakajima
`2008, 0235283 A1
`9, 2008 Turnball et al.
`2008/0281601 A1
`11/2008 Reichardt et al.
`2008/0306826 A1 12/2008 Kramer et al.
`2008/031305.0 A1 12, 2008 Basir
`2008/0317439 A1 12/2008 Wong et al.
`2009, OO17792 A1
`1/2009 Matsumoto et al.
`2009, OO63277 A1
`3/2009 Bernosky et al.
`2009, OO77179 A1
`3/2009 Bi et al.
`2009,0083631 A1
`3/2009 Sidi et al.
`2009,008.3742 A1
`3/2009 Ramanathan
`2009, OO882O7 A1
`4/2009 Sweeney et al.
`2009.0089587 A1
`4/2009 Brunk et al.
`2009,009 1629 A1
`4/2009 Casey
`2009, O100469 A1
`4/2009 Conradt et al.
`2009.0102973 A1
`4/2009 Harris
`2009, O103893 A1
`4/2009 Sugino et al.
`2009/O119714 A1
`5, 2009 Schlarb et al.
`2009, O124241 A1
`5/2009 Krishnaswamy et al.
`2009/O133059 A1
`5, 2009 Gibbs et al.
`2009/O133070 A1
`5, 2009 Hamano et al.
`2009, O144156 A1
`6/2009 Girgis et al.
`2009, O144769 A1
`6, 2009 Liebhold et al.
`2009/0172108 A1
`7/2009 Sing
`2009/02O7979 A1
`8, 2009 Russell
`2009, 0210908 A1
`8, 2009 Sammarco
`2009, 0216847 A1
`8/2009 Krishnaswamy et al.
`2009,025 1526 A1 10, 2009 Book
`2009/0259499 A1 10/2009 Bhojwani et al.
`2009/0259690 A1 10/2009 Bogdanov
`20090317052 A1 12, 2009 Sezan et al.
`2010, 0023553 A1
`1/2010 Gausman et al.
`2010.0040342 A1
`2/2010 Kemp et al.
`2010/0111269 A1
`5/2010 Younger et al.
`2010.0122279 A1
`5/2010 Zhang
`2010, 0131601 A1
`5, 2010 Cowen et al.
`2010, O150321 A1
`6, 2010 Harris et al.
`
`2010/01699 11 A1* 7/2010 Zhang ............................. 725/19
`2010, 0212001 A1
`8/2010 Kashyap et al.
`2010, O257052 A1 10, 2010 Zito et al.
`2010/0272420 A1 10, 2010 Soohoo et al.
`2010.0318515 A1 12/2010 Ramanathan ..... GO6F 17,30799
`707/723
`
`2010/0318596 Al 12/2010 Bergman
`2011/0022638 A1* 1/2011 Jiang ................. GO6F 17,30743
`707/797
`
`2011/0041153 A1
`2011/0063317 A1
`2011 OO64377 A1
`2011 OO64378 A1
`2011 OO64385 A1
`2011 OO64386 A1
`2011/0066489 A1
`2011/0066663 A1
`2011/0066942 A1
`2011/0066944 A1
`2011 OO67066 A1
`2011 OO67099 A1
`2011 OO85781 A1
`2011 0135283 A1
`2011/O137976 A1
`2013,022381.6 A1
`2014/0205267 A1
`2014/0229969 A1
`
`2/2011 Simon et al.
`3/2011 Gharaat
`3/2011 Gharaat
`3/2011 Gharaat
`3/2011 Gharaat
`3/2011 Gharaat
`3/2011 Gharaat
`3/2011 Gharaat
`3/2011 Barton
`3/2011 Barton
`3/2011 Barton
`3/2011 Barton
`4/2011 Olson
`6/2011 Poniatowki et al.
`6, 2011 Poniatowski
`8, 2013 Gharaat
`7/2014 Poniatowski et al.
`8/2014 Gharaat
`
`FOREIGN PATENT DOCUMENTS
`
`3, 1993
`HO5-56386
`JP
`8, 1993
`405205343
`JP
`11, 1999
`11-328769
`JP
`3, 2002
`2002O77458 A
`JP
`2, 2004
`2004-056669
`JP
`8, 2004
`2004-222014 A
`JP
`2004297685 A 10, 2004
`JP
`2004-320608 A 11 2004
`JP
`2004-264186
`12, 2004
`JP
`2005-78627
`3, 2005
`JP
`2005-110047
`4/2005
`JP
`2005-509229
`4/2005
`JP
`2005-197987
`7/2005
`JP
`2005328.167 A 11/2005
`JP
`2006-2294.94
`8, 2006
`JP
`2006-524473
`10, 2006
`JP
`2006-311098 A 11 2006
`JP
`2007-159166 A
`6, 2007
`JP
`2007-173918 A
`7/2007
`JP
`2007-214861
`8, 2007
`JP
`2008-21994.7
`9, 2008
`JP
`2008-277.933
`11/2008
`JP
`2008-277933. A 11/2008
`JP
`2008277.933
`11/2008
`JP
`2009-1181.89
`5, 2009
`JP
`2009-130548
`6, 2009
`JP
`2009-522858
`6, 2009
`JP
`H-5-260082 A 10, 1993
`WO
`WOOOf 13415
`3, 2000
`WO
`WOOOf 13416
`3, 2000
`WO
`WOOOf 45291
`8, 2000
`WO
`WO O1/24524
`4/2001
`WO
`WO WO 2005/122579
`12/2005
`WO WO 2008/078093
`T 2008
`WO WO 2008, 119372
`9, 2008
`WO WO 2009/033182 A1
`3f2009
`WO WO 2009/143667
`* 12/2009
`WO WO 2011/032167 A1
`3, 2011
`WO WO 2011/032168 A1
`3, 2011
`WO WO2O11,069035 A1
`6, 2011
`
`OTHER PUBLICATIONS
`
`Current Claims in application No. 108162884-1902, dated Nov.
`2013, 2 pages.
`The State Intellectual Property Office of the People's Republic of
`China, “Notification of the First Office Action', in application No.
`201080063087.7, dated Mar. 6, 2014, 8 pages.
`
`

`

`US 9,554,176 B2
`Page 4
`
`(56)
`
`References Cited
`
`OTHER PUBLICATIONS
`
`Current Claims in application No. 201080063087.7, dated Mar.
`2014, 5 pages.
`Japan Patent Office, "Decision of Rejection” in application No.
`2012-528995, dated Apr. 22, 2014, 3 pages.
`Current Claims in Japan application No. 2012-258995, dated Apr.
`2014, 2 pages.
`Japan Patent Office, "Decision of Rejection” in application No.
`2012-528996, dated May 7, 2014, 4 pages.
`Hiroshi Tuuki, Introduction to CGI program making by Perl, Basic
`edition, initial version, Jan. 25, 1998, Softbank Corporation, 4
`pageS.
`Current Claims in Japan Application No. 2012-528996, dated May
`2014, 2 pages.
`International Searching Authority, International Search Report with
`Written Opinion, PCT/US 10/48811, Applicant: Tivo Inc., Dated
`Oct. 29, 2010, 12 pages.
`Claims, PCT/US 10/48811, applicant Tivo Inc., 6 pages.
`International Search Authority/US, Search Report and Written
`Opinion for International Application No. PCT/US 10/48812 mailed
`Jan. 6, 2011, 15 pages, Alexandria Virginia USA.
`Current Claims as of Jan. 6, 2011 of International Application No.
`PCT/US 10/48812, 11 pages.
`Nokai et al., Machine Generated Translation of JP 2004-364186,
`dated Dec. 2004.
`The State Intellectual Property Office of the People's Republic of
`China, “Notification of First Office Action” in application No.
`201080051626.5, dated Jun. 26, 2014, 11 pages.
`Claims in China Application No. 201080051626.5, dated Jun. 2014,
`5 pages.
`Australian Government, "Patent Examination Report No. 1", in
`application No. 2010291945, dated Jul. 4, 2013, 4 pages.
`Current Claims in application No. 2010291945, dated Jul. 2013, 5
`pageS.
`Australian Government, “Patent Examination Report No. 1", in
`application No. 2010325897, dated May 17, 2013, 3 pages.
`Current Claims in application No. 2010325897, dated May 2013, 4
`pageS.
`Hungarian Intellectual Property Office, “Written Opinion” in appli
`cation No. 201201703-4, dated Jul. 18, 2013, 14 pages.
`Current Claims in application No. 201201703-4, dated Jul. 2013, 5
`pageS.
`European Patent Office, “European Search Report” in application
`No. 10816287.6 dated Nov. 11, 2013, 13 pages.
`Current Claims in application No. 10816287.6, dated Nov. 2013, 3
`pageS.
`The Japan Patent Office, “Notification of Reasons for Rejection” in
`application No. 2012-542214, dated Nov. 11, 2013, 4 pages.
`Current Clams in application No. 2012-542214, dated Nov. 2013, 6
`pageS.
`Danish Patent and Trademark Office, "Singapore Written Opinion
`and Search Report” in application No. 201204066-3, dated Sep. 13,
`2013, 36 pages.
`Current Claims in Singapore application No. 201204066-3, 4 pages.
`Canadian Intellectual Property Office, “Office Action” in application
`No. 2,773,558, dated Oct. 11, 2013, 2 pages.
`Current Claims in application No. 2,773,558, dated Oct. 2013, 12
`pageS.
`Japan Patent Office, “Notification of Reasons for Rejection', in
`application No. 2012-528995, dated Sep. 3, 2013, 12 pages.
`Current Claims in application No. 2012-528995, dated Sep. 2013,
`12 pages.
`U.S. Appl. No. 12/631,782, filed Dec. 4, 2009, Final Office Action,
`Apr. 7, 2014.
`U.S. Appl. No. 12/631,778, filed Dec. 4, 2009, Final Office Action,
`May 22, 2014.
`U.S. Appl. No. 12/631,759, filed Dec. 4, 2009, Final Office Action,
`Apr. 22, 2014.
`U.S. Appl. No. 12/631,765, filed Dec. 4, 2009, Office Action, Apr.
`23, 2014.
`
`U.S. Appl. No. 12/631,751, filed Dec. 4, 2009, Office Action, May
`16, 2014.
`U.S. Appl. No. 12/631,740, filed Dec. 4, 2009, Final Office Action,
`Apr. 25, 2014.
`.S. Appl. No. 13/859,620. Non-Final Office Action dated Aug. 21.
`014.
`.S. Appl. No. 12/631,782. Non-Final Office Action dated Sep. 15,
`014.
`.S. Appl. No. 12/631,781. Non-Final Office Action dated Sep. 22.
`014.
`.S. Appl. No. 12/631,740, Non-Final Office Action dated Oct. 7,
`014.
`.S. Appl. No. 12/631,759. Notice of Allowance dated Nov. 4.
`014.
`.S. Appl. No. 12/631,765. Non-Final Office Action dated Nov. 12,
`014.
`.S. Appl. No. 12/631,762. Non-Final Office Action dated Dec. 3,
`014.
`.S. Appl. No. 12/631,778, Non-Final Office Action dated Jan. 14.
`O15.
`.S. Appl. No. 13/859,620. Notice of Allowance dated Jan. 20,
`O15.
`.S. Appl. No. 12/631,751, Final Office Action dated Feb. 4, 2015.
`.S. Appl. No. 12/631,786, Non-Final Office Action dated Feb. 10,
`O15.
`.S. Appl. No. 12/631,782, Final Office Action dated Mar. 20, 2015.
`.S. Appl. No. 12/631,781, Final Office Action dated May 14, 2015.
`.S. Appl. No. 12/631,740, Final Office Action dated May 21, 2015.
`.S. Appl. No. 12/631,762, Final Office Action dated Jun. 26, 2015.
`.S. Appl. No. 14/225,393, Non-Final Office Action dated Jun. 29.
`O15.
`.S. Appl. No. 12/631,782, Advisory Action dated Jul. 1, 2015.
`.S. Appl. No. 12/631,751. Non-Final Office Action dated Jul. 13,
`O15.
`.S. Appl. No. 12/631,765, Final Office Action dated Jul. 31, 2015.
`.S. Appl. No. 12/631,778, Final Office Action dated Aug. 19, 2015.
`.S. Appl. No. 12/631,786, Final Office Action dated Sep. 4, 2015.
`.S. Appl. No. 14/259,145. Non-Final Office Action dated Sep. 25.
`O15.
`U.S. Appl. No. 12/631,781. Notice of Allowance dated Sep. 28,
`2015.
`Chinese Patent Office, Application No. 201080051626.5. Foreign
`Office Action dated Jun. 26, 2014.
`Chinese Patent Office, Application No. 201080051626.5. Foreign
`Office Action dated Apr. 3, 2015.
`Chinese Patent Office, Application No. 201080051626.5, Pending
`Claims as of Jun. 26, 2014.
`Chinese Patent Office, Application No. 201080051626.5, Pending
`Claims as of Apr. 3, 2015.
`Chinese Patent Office, Application No. 201080051627.X. Foreign
`Office Action dated Jul. 31, 2014.
`Chinese Patent Office, Application No. 201080051627.X. Foreign
`Office Action dated Jun. 18, 2015.
`Chinese Patent Office, Application No. 201080051627.X, Pending
`Claims as of Jul. 31, 2014.
`Chinese Patent Office, Application No. 201080051627.X, Pending
`Claims as of Jun. 18, 2015.
`Chinese Patent Office, Application No. 201080063087.7. Foreign
`Office Action dated Nov. 3, 2014.
`Chinese Patent Office, Application No. 201080063087.7. Foreign
`Office Action dated Apr. 13, 2015.
`Chinese Patent Office, Application No. 201080063087.7, Pending
`Claims as of Nov. 3, 2014.
`Chinese Patent Office, Application No. 201080063087.7, Pending
`Claims as of Apr. 13, 2015.
`Japanese Patent Office, Application No. 2012-542214. Foreign
`Office Action dated Dec. 2, 2014.
`Japanese Patent Office, Application No. 2012-542214, Pending
`Claims as of Dec. 2, 2014.
`Japanese Patent Office, Application No. 2014-165383, Foreign
`Office Action dated Jun. 16, 2015.
`Japanese Patent Office, Application No. 2014-165383, Pending
`Claims as of Jun. 16, 2015.
`
`

`

`US 9,554,176 B2
`Page 5
`
`(56)
`
`References Cited
`
`OTHER PUBLICATIONS
`
`Japanese Patent Office, Application No. 2014-182543, Foreign
`Office Action dated Jun. 16, 2015.
`Japanese Patent Office, Application No. 2014-182543, Pending
`Claims as of Jun. 16, 2015.
`WIPO, “System and Method for Analyzing Video Content Using
`Detected Text Video, Patent Scope, Frames' http://www.wipo.int/
`patentscope/search/en/detail.jsf?docId=EP13463397&recNum=l
`&max, Apr. 9, 2012.
`World Intellectual Property Organization, Application No. PCT/
`US2010/48811, Claims dated Oct. 29, 2010.
`World Intellectual Property Organization, Application No. PCT/
`US2010/48811, International Search Report and Written Opinion
`dated Oct. 29, 2010, 12 pages.
`World Intellectual Property Organization, Application No. PCT/
`US2010/48812, International Search Report and Written Opinion
`dated Jan. 6, 2011, 15 pages.
`World Intellectual Property Organization, Application No. PCT/
`US2010/48812, Claims dated Jan. 6, 2011.
`U.S. Appl. No. 12/631,782, filed Dec. 4, 2009, Final Office Action,
`Aug. 22, 2013.
`.S. Appl. No. 12/631,778, filed Dec. 4, 2009, Office Action, Sep.
`5, 2013.
`.S. Appl. No. 12/631,759, filed Dec. 4, 2009, Office Action, Sep.
`7, 2013.
`.S. Appl. No. 12/631,782, Final Office Action dated Apr. 15, 2016.
`.S. Appl. No. 12/631,765. Non-Final Office Action dated Jun. 17.
`O16.
`.S. Appl. No. 12/631,762, Final Office Action dated Jul 14, 2016.
`.S. Appl. No. 14/259,145. Notice of Allowance dated Feb. 17.
`O16.
`.S. Appl. No. 12/631,751, Final Office Action dated Feb. 19, 2016.
`
`U.S. Appl. No. 14/225,393, Final Office Action dated Feb. 25, 2016.
`U.S. Appl. No. 12/631,786. Non-Final Office Action dated Apr. 1,
`2016.
`U.S. Appl. No. 12/631,778, Non-Final Office Action dated Apr. 7,
`2016.
`Chinese Patent Office, Application No. 201080051627.X. Foreign
`Office Action dated Jan. 28, 2016.
`Chinese Patent Office, Application No. 201080051627.X, Pending
`Claims as of Jan. 28, 2016.
`Japanese Patent Office, Application No. 2014-165383, Foreign
`Office Action dated Feb. 23, 2016.
`Japanese Patent Office, Application No. 2014-165383, Pending
`Claims as of Feb. 23, 2016.
`Japanese Patent Office, Application No. 2014-182543, Foreign
`Office Action dated Feb. 23, 2016.
`Japanese Patent Office, Application No. 2014-182543, Pending
`Claims as of Feb. 23, 2016.
`U.S. Appl. No. 12/631,782. Non-Final Office Action dated Oct. 26,
`2015.
`U.S. Appl. No. 12/631,778. Notice of Allowance dated Dec. 9,
`2015.
`U.S. Appl. No. 12/631,762. Non-Final Office Action dated Dec. 31.
`2015.
`U.S. Appl. No. 12/631,740, Non-Final Office Action dated Jan. 15,
`2016.
`U.S. Appl. No. 12/631,751. Non-Final Office Action dated Sep. 16,
`2016.
`U.S. Appl. No. 12/631,740, Final Office Action dated Sep. 27, 2016.
`U.S. Appl. No. 12/631,778, Final Office Action dated Oct. 17, 2016.
`Japanese Patent Office, Application No. 2014-165383, Foreign
`Office Action dated Sep. 20, 2016.
`Japanese Patent Office, Application No. 2014-165383, Pending
`Claims as of Sep. 20, 2016.
`
`* cited by examiner
`
`

`

`U.S. Patent
`
`Jan. 24, 2017
`
`Sheet 1 of 12
`
`US 9,554,176 B2
`
`
`
`
`
`Media Device N
`120
`
`Fingerprint Server
`130
`
`Media Source
`110
`
`Media Device A
`100
`
`NetWOrk DeVice
`140
`
`
`
`
`
`
`
`
`
`
`
`Web Server(s)
`150
`
`
`
`FIGURE 1A
`
`

`

`U.S. Patent
`U.S. Patent
`
`Jan. 24, 2017
`Jan. 24, 2017
`
`Sheet 2 of 12
`Sheet 2 of 12
`
`US 9,554,176 B2
`US 9,554,176 B2
`
`O}INDCAplA
`
`
`
`Kejdsiq
`
`LAL
`
`
`
`d01Adq]BIPAY|
`
`OO!
`
`waysds-qnsAeidsiq
`
`
`JoLaAuogolpnyjxe|
`
`OLL
`
`Z9L
`
`
`
`waysksAloWsS/\y
`
`GSI
`
`
`
`
`
`
`
`
`
`
`
`
`
`wunsjeseydieg
`
`
`
`a|npoyyWOMISN
`
`06
`
`
`
`indujoapiyoipny
`
`GL
`
`abFYNSIs
`
`BLLI
`
`IWGH
`
`Q/L
`
`
`
`
`
`

`

`U.S. Patent
`
`Jan. 24, 2017
`
`Sheet 3 of 12
`
`US 9,554,176 B2
`
`
`
`Receive Command to present media
`Content
`202
`
`Present Media Content
`204
`
`Derive fingerprint from Media Content
`206
`
`Receive Command to present additional
`Content asSociated with media Content being
`presented
`208
`
`Identify additional content based on
`fingerprint of media content
`210
`
`Present Additional Content
`212
`
`FIGURE 2
`
`

`

`U.S. Patent
`
`Jan. 24, 2017
`
`Sheet 4 of 12
`
`US 9,554,176 B2
`
`
`
`Receive Command to display media
`Content
`302
`
`
`
`Present Media Content
`304
`
`Derive fingerprint from Media Content
`being played
`306
`
`Determine position in playing of media Content
`on a first device based on the fingerprint
`308
`
`Detect that an advertisement is being
`played based on the position
`314
`
`Synchronize playing of media content on
`the first device with playing of the same
`media Content on a second device
`310
`
`Receive play-function Command On first
`device and perform play-function command
`On first device and Second device
`312
`
`FIGURE 3
`
`

`

`U.S. Patent
`
`Jan. 24, 2017
`
`Sheet S of 12
`
`US 9,554,176 B2
`
`
`
`Receive command to view/record media
`Content on a first device associated with a
`first user
`402
`
`Publish viewing/recording of media content
`On the first device associated with the first
`Se
`404
`
`Responsive to receiving Command to
`view/record on first device, record media
`Content. On the first device and a Second
`device associated with a Second user
`406
`
`Detect that the same media Content is being
`recorded on multiple devices
`4.08
`
`Schedule a time for playing of the media
`Content Concurrently on multiple devices
`410
`
`Automatically enroll viewers/recorders of
`the same media Content in a social
`networking group associated with the media
`COntent
`412
`
`FIGURE 4
`
`

`

`U.S. Patent
`
`Jan. 24, 2017
`
`Sheet 6 of 12
`
`US 9,554,176 B2
`
`
`
`Monitor a media device and detect
`OCCurrence of an idleness threshold
`asSociated with the media device
`502
`
`Receive media Content by the media
`device from a broadcaster
`504
`
`Derive a fingerprint from media content by
`the media device
`506
`
`Upload the fingerprint derived from the
`media content by the media device
`508
`
`FIGURE 5
`
`

`

`U.S. Patent
`
`Jan. 24, 2017
`
`Sheet 7 of 12
`
`US 9,554,176 B2
`
`
`
`Fingerprint Database
`602
`
`Fingerprint Management Engine
`604
`
`
`
`
`
`
`
`Media Device A
`606
`
`Media Device B
`608
`
`Media Device C
`610
`
`Media Device N
`620
`
`FIGURE 6
`
`

`

`U.S. Patent
`
`Jan. 24, 2017
`
`Sheet 8 of 12
`
`US 9,554,176 B2
`
`Receive message preferences associated
`With a user
`702
`
`Receive Command to play media Content
`704
`
`Play Media Content
`706
`
`Receive message While playing media
`COntent
`708
`
`Present message
`based on message preferences?
`710
`
`NO
`
`
`
`Yes
`
`Present message
`712
`
`FIGURE 7
`
`

`

`U.S. Patent
`
`Jan. 24, 2017
`
`Sheet 9 of 12
`
`US 9,554,176 B2
`
`
`
`Identify user(s) present near a multimedia
`device
`802
`
`Receive a voice Command from a user
`804
`
`Interpret the Voice Command based on
`preferences associated with one or more
`identified users
`806
`
`Perform the action
`808
`
`FIGURE 8
`
`

`

`U.S. Patent
`
`Jan. 24, 2017
`
`Sheet 10 of 12
`
`US 9,554,176 B2
`
`
`
`Play media content
`902
`
`Record audio input received during playing
`of the media Content
`904
`
`Play the media Content with the audio input
`during a Subsequent playing of the Content
`906
`
`FIGURE 9
`
`

`

`U.S. Patent
`
`Jan. 24, 2017
`
`Sheet 11 of 12
`
`US 9,554,176 B2
`
`
`
`
`
`
`
`
`
`
`
`
`
`User Preferences
`1020
`
`Environment
`Configuration Engine
`1015
`
`
`
`
`
`Environment
`Configurations
`1025
`
`FIGURE 10
`
`

`

`U.S. Patent
`U.S. Patent
`
`Jan. 24, 2017
`Jan. 24, 2017
`
`Sheet 12 of 12
`Sheet 12 of 12
`
`US 9,554,176 B2
`US 9,554,176 B2
`
`YaAdgS
`
`colt
`
`MYOMLAN
`TWOO7
`MYOMLAN
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`AOVYOLS
`
`901}
`
`NIV
`
`AYOWAN
`
`sng
`
`NOILYOINAWWOO
`
`AOVAYSIN
`
`OLL
`
`| |
`
`YOSS300Nd
`
`
`
`Lp‘bly
`
`AV1dSId
`
`vib
`
`
`
`JOIASdLNdNI
`
`??TI
`QLLL
`
`Yoseno
`
`JOYXLNOD
`
`
`
`
`
`
`
`
`
`

`

`1.
`MEDIA CONTENT FINGERPRINTING
`SYSTEM
`
`PRIORITY INFORMATION
`
`This application is a continuation of U.S. patent applica
`tion Ser. No. 12/631,783, filed Dec. 4, 2009, which claims
`the benefit of U.S. Provisional Application No. 61/242.277,
`filed Sep. 14, 2009, the entire contents of which is hereby
`incorporated by reference as if fully set forth herein, under
`35 U.S.C. S 120. The applicant(s) hereby rescind any dis
`claimer of claim scope in the parent applications or the
`prosecution thereof and advise the USPTO that the claims in
`this application may be broader than any claim in the parent
`applications.
`
`10
`
`15
`
`FIELD OF THE INVENTION
`
`The present invention relates to a multifunction multime
`dia device.
`
`BACKGROUND
`
`The approaches described in this section are approaches
`that could be pursued, but not necessarily approaches that
`have been previously conceived or pursued. Therefore,
`unless otherwise indicated, it should not be assumed that any
`of the approaches described in this section qualify as prior
`art merely by virtue of their inclusion in this section.
`Multimedia content streams may be received by a multi
`media player for display to a user. Furthermore, general
`information about multimedia content may be received by
`the multimedia player for display to the user. The multime
`dia content is generally presented in a fixed non-editable
`format. The user is able to jump to particular points in the
`media content via scene selections created by the producer.
`Accordingly, the watching of the media content is generally
`passive and the user interaction is minimal.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`The present invention is illustrated by way of example,
`and not by way of limitation, in the figures of the accom
`panying drawings and in which like reference numerals refer
`to similar elements and in which:
`FIG. 1A is a block diagram illustrating an example system
`in accordance with an embodiment;
`FIG. 1B is a block diagram illustrating an example media
`device in accordance with an embodiment;
`FIG. 2 illustrates a flow diagram for presenting additional
`content in accordance with an embodiment.
`FIG. 3 illustrates a flow diagram for determining a posi
`tion in the playing of media content in accordance with an
`embodiment.
`FIG. 4 illustrates a flow diagram for detecting the playing
`of an advertisement in accordance with an embodiment.
`FIG. 5 illustrates a flow diagram for deriving a fingerprint
`from media content in accordance with an embodiment.
`FIG. 6 shows an exemplary architecture for the collection
`and storage of fingerprints derived from media devices.
`FIG. 7 illustrates a flow diagram for presenting messages
`in accordance with an embodiment.
`FIG. 8 illustrates a flow diagram for interpreting voice
`commands in accordance with an embodiment;
`FIG. 9 illustrates a flow diagram for correlating annota
`tions with media content in accordance with an embodiment;
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`US 9,554,176 B2
`
`2
`FIG. 10 shows an exemplary system for configuring an
`environment in accordance with one or more embodiments.
`FIG. 11 shows a block diagram that illustrates a system
`upon which an embodiment of the invention may be imple
`mented.
`
`DETAILED DESCRIPTION
`
`In the following description, for the purposes of expla
`nation, numerous specific details are set forth in order to
`provide a thorough understanding of the present invention.
`It will be apparent, however, that the present invention may
`be practiced without these specific details. In other
`instances, well-known structures and devices are shown in
`block diagram form in order to avoid unnecessarily obscur
`ing the present invention.
`Several features are described hereafter that can each be
`used independently of one another or with any combination
`of the other features. However, any individual feature might
`not address any of the problems discussed above or might
`only address one of the problems discussed above. Some of
`the problems discussed above might not be fully addressed
`by any of the features described herein. Although headings
`are provided, information related to a particular heading, but
`not found in the section having that heading, may also be
`found elsewhere in the specification.
`Example features are described according to the following
`outline:
`10 FUNCTIONAL OVERVIEW
`20 SYSTEMARCHITECTURE
`3.O PRESENTING ADDITIONAL CONTENT BASED
`ON MEDIA CONTENT FINGERPRINTS
`4.O DETERMINING A PLAYING POSITION BASED
`ON MEDIA CONTENT FINGERPRINTS
`5.O PUBLISHING RECORDING OR VIEWING
`INFORMATION
`6.ODERIVING A FINGERPRINT FROMMEDIA CON
`TENT
`7.O PRESENTING UDPATES
`8.0 INTERPRETING COMMANDS
`9.0 CORRELATING INPUT WITH MEDIA CONTENT
`1O.O ELICITING ANNOTATIONS BY A PERSONAL
`MEDIA DEVICE
`11.O MARKING MEDIA CONTENT
`12.O PUBLIC