(12) United States Patent
`Redford et al.
`
`(10) Patent No.:
`(45) Date of Patent:
`
`US 8,660,545 B1
`Feb. 25, 2014
`
`USOO8660545B1
`
`(54) RESPONDING TO AVIDEO REQUEST BY
`DISPLAYING INFORMATION ON A TV
`REMOTE AND VIDEO ON THE TV
`
`(75) Inventors: Peter Redford, Saratoga, CA (US); Bala
`Chander, Saratoga, CA (US)
`(73) Assignee: ILook Corporation, San Jose, CA (US)
`
`*) Notice:
`
`Subject to any disclaimer, the term of this
`y
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 541 days.
`
`(21) Appl. No.: 12/683,405
`
`OTHER PUBLICATIONS
`
`Kleban, T. "A Tutorial on Watching YouTube Videos From an
`iPhone', published on Ezine on Oct. 19, 2007, pp. 2.
`Wilson, B. "Possibly the Best iPhone Web app yet: vtap trumps
`built-in iPhone YouTube app', published on CNET on Nov. 2, 2007.
`p. 1.
`Maltais, M. “Appiphilia: i.TV app tunes in new features for iPhone'.
`published of the Los Angeles Times on Dec. 23, 2008, pp. 6.
`Lowensohn, J. “i.TV's iPhone app gets deep Netflix integration'.
`published on CNET on Nov. 10, 2008, pp. 2.
`Sadun, E. “i.TV for iPhone: Cluttered but promising entertainment
`guide', published on Aug. 17, 2009 of Weblogs, Inc., pp. 2.
`vTap—Frequently Asked Questions, Recently Raised questions
`(Mobiles using m.vtap.com), retrieved from http://vtap.com/facq.
`html on Nov. 22, 2009, pp. 7.
`Katzmaier, D. "Could an Apple HDTV plus an iPhone remote equal
`living room bliss?', retrieved from http://news.cnet.com/8301
`17938 105-10314746-1.html, Aug. 21, 2009, pp. 4.
`Chen, B.X. "Flash Lands on iPhone—One Appata Time', retrieved
`from http://www.wired.com/gadgetlab/2009/10/iphone-flash/, Oct.
`5, 2009, pp. 13.
`* cited by examiner
`Primary Examiner — Danh Le
`(74) Attorney, Agent, or Firm — Silicon Valley Patent Group
`LLP
`
`ABSTRACT
`(57)
`A handheld device displays multiple applications (“apps),
`from which a user selects one app. The selected app displays
`multiple descriptions of videos, from which the user selects
`one video. The handheld device transmits a video request
`containing an identification of the user-selected video. The
`computer system transmits the user-selected video to the
`television and transmits a signal to the handheld device, based
`on the video request. The television responds by automati
`cally displaying on a television screen, a series of frames in
`the video, on receipt therein. The handheld device automati
`cally displays on a handheld Screen, information that is
`directly or indirectly based on the video request, as the signal
`is received from the computer system. Two displays based on
`a single video request, provide a richer viewing experience
`than a single display.
`50 Claims, 42 Drawing Sheets
`
`Jan. 6, 2010
`
`(2006.01)
`
`(22) Filed:
`(51) Int. Cl.
`H04M 3/00
`(52) U.S. Cl.
`USPC ..................... 455/418; 455/414. 1; 455/556.1:
`725/39; 348/552
`
`(58) Field of Classification Search
`USPC ........ 455/418, 414.1, 556.1; 725/39; 348/552
`See application file for complete search history.
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`5/2003 Allport
`6,567,984 B1
`5, 2004 Kaars
`6,741,684 B2
`9, 2008 Allen et al.
`7,428,023 B2
`1/2009 JobS et al.
`7,479,949 B2
`1/2010 Yoshii ........................... 348,687
`7,643,095 B2 *
`3f2002 Rakib ........................... 370,386
`2002/0031120 A1
`6/2006 Raiyat
`2006, O123076 A1
`2007/0250863 A1* 10/2007 Ferguson ........................ T25/46
`2008/0092198 A1* 4/2008 Hutten .......................... 725,133
`2008/O122796 A1
`5, 2008 JobS et al.
`2008/0281718 A1* 1 1/2008 Morgan .......................... 705/26
`2009,0007198 A1
`1/2009 Lavender et al. ............... 725/91
`2009/0204493 A1
`8, 2009 Kim
`T25,131
`2010/00953.45 A1 *
`4, 2010 Tran et al. ....
`... 327/525
`2010/0321095 A1* 12, 2010 Mikawa et al. .
`2011/0063317 A1
`3f2011 Gharaat et al. ................ 345,545
`
`
`
`LEPONE
`NETWORK
`
`basidheld interia
`IDED THID
`YEO)
`1836
`
`o
`
`282
`
`834
`DABSE
`Bidhadas
`its associated
`
`1
`
`Comcast, Ex. 1005
`
`

`

`U.S. Patent
`
`Feb. 25, 2014
`
`Sheet 1 of 42
`
`US 8,660,545 B1
`
`
`
`FIG. 1A (prior art)
`Smartphone
`
`FIG. 1B (prior art)
`Smartphone
`
`PHOTOS MESSAGING CALCUATE
`
`ALARM
`
`WEATHER
`
`STOR
`
`BLOG
`
`NOTES
`
`101
`
`PHONE CALENDAR BROWSER
`
`MAIL
`
`2
`
`

`

`U.S. Patent
`U.S. Patent
`
`Feb. 25, 2014
`Feb. 25, 2014
`
`Sheet 2 of 42
`Sheet 2 of 42
`
`US 8,660,545 B1
`US 8,660,545 B1
`
`
`
`
`
`3
`
`

`

`U.S. Patent
`
`Feb. 25, 2014
`
`Sheet 3 of 42
`
`US 8,660,545 B1
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`PHOTOS
`
`MESSAGENG
`
`CALCULATOR ALARM CLOCK
`
`WEATHER
`
`SORE
`
`BLOG
`
`NOTES
`
`
`
`i Cori uter
`of Content Provider A
`p (e.g. Netflix)
`
`282
`
`Computer of Authority
`(Authority Server)
`
`EPGAPPA
`
`ihandheld :
`interface
`
`4
`
`

`

`U.S. Patent
`
`Feb. 25, 2014
`
`Sheet 4 of 42
`
`US 8,660,545 B1
`
`FIG. 20
`
`299
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`205
`
`291
`
`as
`
`Saving Private Ryan
`Directed by Steven Spielberg.
`Starring Tom Hanks, Tom
`Sizemore. Matt Damon
`The African Queen
`Directed by John Huston.
`SS Starring Humphrey Bogart,
`(Sig SS Katharine Hepburn
`Star Trek
`Created by Gene Roddenberry
`Starring Chris Pine, Zachary
`Quinto, Eric Bana
`
`
`
`
`
`5
`
`

`

`U.S. Patent
`
`Feb. 25, 2014
`
`Sheet 5 of 42
`
`US 8,660,545 B1
`
`
`
`202J
`
`Feature
`Presentations
`
`102 Views
`01:55:08
`
`89 Views
`01:32:24
`
`352 Views
`01:45:47
`
`6
`
`

`

`U.S. Patent
`
`Feb. 25, 2014
`
`Sheet 6 of 42
`
`US 8,660,545 B1
`
`FIG. 2E
`
`
`
`FIG. 2F
`
`{e
`
`re)
`information
`MISSION: IMPOSSIBLE
`102 Views ...,
`
`Director: Brian De Palma
`Release Date:22 May 1996 (USA)
`
`An American agent, under false
`suspicion of disloyalty, must discover
`and expose the real spy without the
`help of his organization.
`
`Search Zillion
`
`Y U
`
`(GHG)KIL)
`(2xcve)(CE)
`
`7
`
`

`

`U.S. Patent
`
`Feb. 25, 2014
`
`Sheet 7 of 42
`
`US 8,660,545 B1
`
`
`
`FIG. 2G
`
`(ek
`
`Now Playing
`
`Fie)
`
`102 Views
`01:55:08
`
`There is a version of MISSION
`MPOSSIBLE where Jean
`Reno is the hero and kills Tom
`Cruise
`i.
`SUCaSaC
`
`Casual Jackets
`Affordable Casual Women's Jackets.
`Get S10 off on S25 or more order
`
`I like Mission Impossible,
`but the earliest rarely seen
`Avengers episodes are better
`
`Buy Jackets at REl
`REI the only place to shop.
`
`252
`
`8
`
`

`

`U.S. Patent
`
`Feb. 25, 2014
`
`Sheet 8 of 42
`
`US 8,660,545 B1
`
`
`
`There is a version of
`MISSIONMPOSSIBLE
`where Jean Reno is the hero
`and kills Tom Cruise
`
`SCaSall
`
`Boston, MA
`Http://wwyg.com
`
`Online editor, Artist, occasional
`software hacker
`
`There is a version of MISSION
`IMPOSSIBLE Where Jean Reno
`is the hero and kills Tom Cruise
`USUCaSae
`
`(as of ge)k(l
`(2xcvsne
`
`9
`
`

`

`U.S. Patent
`
`Feb. 25, 2014
`
`Sheet 9 of 42
`
`US 8.660,545 B1
`
`FIG. 2J
`
`
`
`US CaSae
`Nov. 26, 2009, 5:07 PM
`There is a version of
`MISSION IMPOSSIBLE
`Where Jean Reno is the
`hero and kills Tom Cruise
`SUCaSane
`
`Nov. 27, 2009, 4:37 PM
`Dear USumCasane:
`| Want to Watch
`Tom Cruise die.
`255E Where can I get it?
`
`There is a version of
`MISSION IMPOSSIBLE
`Where Jean Reno is the
`hero and kills Tom Cruise
`USC3Sae
`
`Nov. 27, 2009, 4:37 PM
`Dear uSumcasane:
`I Want to Watch
`255E Tom Cruise die.
`Where can I get it?
`Nov. 27, 2009, 4:41 PM
`It's called Mission
`Impossible-Antihero &
`u can get it on Zillion
`USUCaSane
`
`10
`
`

`

`U.S. Patent
`
`Feb. 25, 2014
`
`Sheet 10 of 42
`
`US 8,660,545 B1
`
`FIG. 2L
`
`
`
`Nov. 27, 2009
`Waterfall Matt
`Very well, thank you - how r
`things at your end - Sounds li
`USUCSale
`Nov. 27, 2009
`It's called Mission Impossible
`Antihero & u can get it on Zillic
`Cooper George Nov. 26, 2009
`Happy Thanksgiving Peter
`- are you in town or traveling
`Tran Thomas
`Nov. 25, 2009
`Thank u Peter and Rita. We
`also wish u all d best on Than
`Redford Scottie Nov. 24, 2009
`
`56
`
`11
`
`

`

`U.S. Patent
`
`Feb. 25, 2014
`
`Sheet 11 of 42
`
`US 8,660,545 B1
`
`
`
`FG. 2N
`
`
`
`257
`
`Casual Jackets
`
`E. Women's Jacket
`
`Easy fit, casual, durable.
`Prod ins, Color - Jet Black. Looks
`GUESS good with any Style.
`Size
`Small By
`259A
`
`2s' gra pping N259B
`
`BUY
`
`260
`
`)
`
`O:50; if B BiH - A if
`
`Casual Jackets
`
`Women's Jacket
`
`Easy fit, casual, durable.
`Color-Jet Black. Looks
`GUESS good with any style.
`size:
`Small EY
`
`Thank you for buying GUESS
`Casual Jacket Prodi 1276982
`Your Transaction ID: WXB678
`A receipt has been emailed
`
`12
`
`

`

`U.S. Patent
`
`Feb. 25, 2014
`
`Sheet 12 of 42
`
`US 8,660,545 B1
`
`
`
`
`
`FIG. 3A
`
`308
`
`
`
`to go
`COMPUTER
`SYSTEM
`
`WIRELESS
`NETWORK
`
`Video Reduest
`
`V
`Messade?s
`based on requested Video).S. 5 de
`Op
`C
`VIDEOS
`CP
`MESSAGES
`L REPOSITORY
`REPOSITORY SP
`P
`
`--
`
`308C
`
`250
`
`309
`
`Video
`Request
`
`311
`
`305
`
`302
`
`Note E.
`es
`MeSSade(s
`requested
`SEASIS)
`video)
`
`
`
`BUILDING
`
`251
`
`EPG item Area
`
`Message area Y
`
`DFO
`
`Internet-enabled
`TELEVISION
`
`13
`
`

`

`U.S. Patent
`
`Feb. 25, 2014
`
`Sheet 13 of 42
`
`US 8,660,545 B1
`
`COMPUTER SYSTEM
`
`250
`
`
`
`
`
`-------------------p----- Tricia; Mickey
`:
`has ice gloves
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`Both. CauSedby.a.?eqYest fg.a.Single Video.........
`
`Plane SE;
`Crazy Mousé
`
`Tricia: Mickey
`has nice gloves
`
`Internet-enabled TV
`
`FAMILY
`ROOM
`310
`
`14
`
`

`

`U.S. Patent
`
`Feb. 25, 2014
`
`Sheet 14 of 42
`
`US 8,660,545 B1
`
`
`
`lane E.
`razy Mouse
`Mickey Mouse
`Laptop $32
`Mickey Mouse
`10 stickers 99C
`Mickey Mouse
`Gloves $1.99
`
`Plane S;
`Crazy Mouse
`5.
`Comforter $35
`Tide Detergent
`70oz $9.99
`Duracell
`Recharaeable
`AA 8/pk $21
`
`COMPUTER SYSTEM
`o
`o
`
`sesses
`
`o
`
`to
`
`All caused by a request
`for a single video
`
`Internet-enabled TV
`
`FAMILY
`R3M
`
`15
`
`

`

`U.S. Patent
`U.S. Patent
`
`Feb. 25, 2014
`Feb. 25, 2014
`
`Sheet 15 of 42
`Sheet 15 of 42
`
`US 8,660,545 B1
`US 8,660,545 B1
`
`
`
`|
`
`D
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`déOld
`
`sjeubis
`
`t|—ieel[\
`
`ZO3GIAAdYfa7Trapnadofsonoo50ASVNId;oafinAUWINTEd+;TitAawanooasAABVONODIS]|OCI
` |Zoapiy
`(oaksaa
`
`| ||
`
`LL
`
`AIEWLY44
`
`
`
`8JOVSSIINGITIONDS-olny
`
`
`
`@OVSSINneveaS0e
`
`16
`
`16
`
`
`
`
`
`

`

`U.S. Patent
`
`Feb. 25, 2014
`
`Sheet 16 of 42
`
`US 8,660,545 B1
`
`aulfe
`
`aspeCATIONNS“OLNYHAL]
`FeeOLMANYSHLONY
`Sane26OLNMAN
`SOE~H-.
`gcogdaTIONOS-O.LNY
`WooeGIT1OIS-OLNV
`
`GogdaTOWNS-OLNY
`
`JOVSSAWCATIONS
`
`JOVSSIINGITIOWS
`
`d3M90T-OLAV
`:VSSIW.,JOVSSAN062
`
`
`
`aogedT1080S"joie||{Woe_C7TONdSOLNvanny
`
`wpewaydj“EPTheron
`-JOVSSAW
`
`coeGITIONDS-OLAY“F-4]
`
`d34901-O0.LNVpoeGaNI0TOLAV
`
`7VSSAN06heheWVSSAW
`élLL
`opeCITIOWS-OLnv
`
`
`
`os,
`
`JOVSSIW
`
`JOVSSAN
`
`seen enwent
`
`leeaqee
`beet
`
`
`
`Q3T10N0S“OLN
`
`JOVSSAN
`
`Tacpicccennnscccs
`
`JOVSSAN
`
`JVSSIN
`
`“Jou0s
`
`
`
`
`
`17
`
`el
`
`
`
`Wey!Jd4
`
`
`
`PJOLOS.
`
`
`
`‘“OLNYMIN
`
`
`
`JDVSSSGITIONNS
`
`
`
`jjOu0s..
`
`d3711040S-O.LNV
`
`JOVSSIW
`
`neeWe}!Od
`
`
`
`GorANMowIsOLAV
`
`17
`
`
`
`
`
`
`
`
`
`
`
`

`

`U.S. Patent
`
`Feb. 25, 2014
`
`Sheet 17 of 42
`
`US 8,660,545 B1
`
`
`
`?esale
`Plane
`S Craz
`Spongebob
`Comforter $25
`Mickey Mouse
`First Laptop $32
`Mickey Mouse
`T-Shirt $3.00
`Minnie Mouse
`Gioves S2.00
`ae |
`>> AG (UD
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`ag
`
`
`
`
`
`are at EJ
`
`sh Plane Crazy304
`(Sig Starging Mickey Mouse,
`S3 SE by Walt
`CSS' Disney
`Spongebob. 30SD.
`Comforter $25
`Mickey Mouse 308C
`First Laptop $32 --
`Mickey. Mouse 305B
`T-Shift $3.00 .
`Minnie Mouse 305A
`Gloves $2.00
`
`
`
`
`
`
`
`
`
`
`
`Plane Crazy304
`53 Starring Mickey Mouse,
`R Essed by Walt
`
`Spongebo
`Til
`Comforter S25
`a Mickey Mouse
`C
`First Laptop $32
`“Mickey Mouse 305B
`T-Shift $3.00
`
`
`
`
`
`90A
`90
`
`Time
`
`18
`
`

`

`U.S. Patent
`
`Feb. 25, 2014
`
`Sheet 18 of 42
`
`US 8,660,545 B1
`
`
`
`
`
`TIMELY
`
`O
`
`cES S25
`
`
`
`Mickey M T. 305B
`
`Mickey Mouse
`T-Shift S3.00
`
`
`
`SSESS
`
`TIMELY -
`DISPLAYS
`
`Spongebgb
`Comforter $25
`Mickey Mouse
`First Laptop $32
`
`as
`
`time eT3
`
`19
`
`

`

`U.S. Patent
`
`Feb. 25, 2014
`
`Sheet 19 of 42
`
`US 8,660,545 B1
`
`
`
`Mickey Mouse
`First Laptop $32
`
`3. Plane
`Craz
`
`$75 Cameras
`
`20
`
`

`

`U.S. Patent
`
`Feb. 25, 2014
`
`Sheet 20 of 42
`
`US 8,660,545 B1
`
`
`
`5 Cameras
`tSEARS
`
`S2 day EAEitz
`
`sis
`
`21
`
`

`

`U.S. Patent
`
`Feb. 25, 2014
`
`Sheet 21 of 42
`
`US 8,660,545 B1
`
`
`
`
`
`
`
`G ?WSSEW GETTO HOS-0}^\!
`
`
`
`
`
`35}\/SSEN GETTO HOS
`
`22
`
`

`

`U.S. Patent
`
`Feb. 25, 2014
`
`Sheet 22 of 42
`
`US 8,660,545 B1
`
`
`
`hicago Cubs
`White Cap S33
`
`on
`
`TONY; Migkey is
`ready to hit a
`home run
`Baseball Gloves
`$4.99
`Chicago Cubs
`White Cap $33
`
`23
`
`

`

`U.S. Patent
`
`Feb. 25, 2014
`
`Sheet 23 of 42
`
`US 8,660,545 B1
`
`EZ09
`
`h no
`
`v.
`
`
`
`
`
`
`
`
`
`
`
`24
`
`

`

`U.S. Patent
`
`Feb. 25, 2014
`
`Sheet 24 of 42
`
`US 8,660,545 B1
`
`
`
`to
`
`o
`
`a
`
`a...ET"f
`
`O
`
`Baseball Gloves
`S14.99
`Chicago Cubs
`bat $33
`Baseball Bat 27"
`$19.99
`
`o o
`
`3. Plane
`Craz
`Baseball Cap
`$4.99
`Chicago Cubs
`White Cap S33
`Mickey Mouse
`Cap$4.99
`Black cap $1.99
`<< ||
`>> Y (D
`
`25
`
`

`

`U.S. Patent
`
`Feb. 25, 2014
`
`Sheet 25 of 42
`
`US 8,660,545 B1
`
`
`
`FIG. 6A
`
`Miney Mouse
`Gloves $2.00
`Mickey Mouse
`T-Shirt S3.00
`Mickey Mouse
`First Laptop S3
`Spongebob
`OOter SS
`Starring Mickey
`Plane YESS d
`Crazy ESE,
`
`250
`
`COMPUTER
`SYSTEM
`
`es
`
`OCC
`
`Starring Micke
`Plane E. y
`Crazy Walt SE
`Spongebob
`Comforter $35
`Mickey Mouse
`First Laptop $32
`Mickey Mouse
`T-Shit $3.00
`
`FAMILY
`ROOM
`310
`
`26
`
`

`

`U.S. Patent
`
`Feb. 25, 2014
`
`Sheet 26 of 42
`
`US 8,660,545 B1
`
`FIG. 6B
`
`RECEIVE WIDEO REQUEST
`FOR SESSEES WIDEO
`(i.e. primary video)
`
`COMPUTER WEBR9CESSOR(s) &
`25 250
`13
`
`TRANSMITAT LEAST A PORTION OF SELECTED WIDEO TO TELEVISION
`
`REQUEST
`
`TRANSMITA SIGNAL TO HANDHELD, CONTAINING ATEXT/GRAPHIC
`MESSAGE(s)'ORIDENTIFIER(STEREOF
`
`sittime finYESSEESARSENSY
`secondary video?
`SELECTED WIDEO TO
`TELEVISION
`
`SELECTA NEXT
`PORTION OF
`PRIMARY WIDEO
`
`
`
`VIDEO.
`/i ...YIDEO-REQUEST CAUSED.Y
`REQUEST
`a? sugg57GSSS" 31
`FAMILY ROOM Fr.
`RECEIVE USER
`:
`310
`INESSENGASA.SY
`AYREQANRSENPI
`REGOES
`A WIDEO REQUEST:
`WESS;"| WIRELESSLY
`
`RECEIVE
`VIDEO
`
`RECEIVE SIGNAL and
`RETRIEVE MESSAGE
`- - - - - - - - - - - 2 CONTAINED/IDENTIFIED
`IN RECEIVED SIGNAL
`
`AUTOMATICALLY
`DISPLAY WIDEO
`ON TW SCREEN
`
`TIMELY
`DISPLAYS
`
`AUTOMATICALLY DISPLAY
`MESSAGE(s) ON HANDHELD
`SCREEN (e.g. messages are
`Ositioned in a list defendin
`p
`On message SES O
`
`434
`HANDHELD DEVICE
`WITH PROCESSOR(s) & MEMORY
`
`27
`
`

`

`U.S. Patent
`
`Feb. 25, 2014
`
`Sheet 27 of 42
`
`US 8,660,545 B1
`
`COMPUTER
`SYSTEM
`
`281A,281N
`
`DATA
`NETWORK
`
`DATANETWORK iNTERFACE
`CIRCUIT (external), such as a
`Set-Top Box Or DVD Player
`
`coast
`
`
`
`III
`
`a daries to get gold
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`WINDOW.
`(auto-scrolling
`messages)
`
`28
`
`

`

`U.S. Patent
`
`Feb. 25, 2014
`
`Sheet 28 of 42
`
`US 8,660,545 B1
`
`
`
`RECEy. E. FER
`
`USE HANDHELD D
`foLOOKSTW D 802
`
`COMPUTER
`(authority server)
`282
`
`TRANSMIT TO CONTEN
`PROVIDER COMPUTER
`D6FWDE6ANDTW
`
`804 SEESANFER
`SELECTED WIDEO
`TRANSMIT TO internet-
`OR SELECT
`enabled TV, ID OF WIDEO
`ANOTHER VIDEO)
`AND ID OF CONTENT
`PROVIDER COMPUTER
`
`SELECT
`
`LIST OF MESSAGES (e.g. RSEN YES
`Ordered by size of bid)
`WIDEO
`DEVESSENSERENG
`SELECTEDMESSAGES
`(OR IDs thereof)
`
`E.
`SE
`(default)
`
`COMPUTER
`(message Server
`Or Client
`Receive or generate (automatically)
`KYS laid fymessage to
`e Sed by autory See
`Selecting EE dispalyed
`O1a ance eCe
`
`Receive or generate (aga:
`value of attribute formessage to be
`used by authority server in Sorting
`messages to be displayed
`On the handheld device
`
`Transmit message with keyword(s)
`and attribute value to authority server
`
`Receive report on message
`
`Display report to user
`
`Receive WideO authorization
`818
`S.E.E.S.E
`M token)
`(with optional D
`Transmit Video to
`authorized
`Receive Video Fetch (including
`citially
`video identifier) from authorized
`Authority Server ESSESS
`
`811
`
`81 7
`
`COMPUTER
`(Content provider server)
`281
`
`Retrieve video
`Or portion
`SR VS
`Repository
`
`protected with
`DRM
`
`s
`SG e
`available in
`
`ls TV's
`DRM token
`Walid
`
`29
`
`

`

`U.S. Patent
`
`Feb. 25, 2014
`
`Sheet 29 of 42
`
`US 8,660,545 B1
`
`EPG App
`(authority
`Compatible
`handheld
`interface)
`
`
`
`
`
`E.
`281
`
`DATABASE
`(handheld device
`& its associated TV)
`
`DATA
`NETWORK
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`Descriptions of available EPGApps
`Identifier of EPGA
`EPG App (including handheld interface)
`
`TELEPHONE
`NETWORK
`
`
`
`(ESAER,
`AES
`
`identifier of Handheld
`& Identifier of TW to beaSSOciated
`
`
`
`
`
`dentifier of Selected Video and
`ldentifier of Content Provider
`essaces(non-EPG intormation
`EPG item of Currently displayed video
`"2121
`835NSA 2 AgSY
`Promotional
`SSSy
`
`
`
`
`
`ESE
`REPOSITORY
`
`
`
`832
`
`311
`
`839
`
`
`
`-C
`
`as
`
`SOCIAL
`MEDIA
`MESSAGE
`SREI
`
`
`
`Li is a 3
`
`cas go
`
`as
`
`
`
`s
`S S
`is a
`S2 is
`S
`as
`,
`S
`5 E.
`55 ;
`i is E-283
`5
`e
`Eg is
`Esc
`S.
`Promotional
`s: i- E 15
`6
`Messages
`5
`5
`MESSAGE keywords
`?o
`22
`sp
`CLENT
`Reports On
`EE b SS
`Messaqe Displa
`SES
`9
`g
`5
`play
`r S.
`
`ma
`
`V.
`
`
`
`The Killers
`Day & Age
`Top 10
`Cameras
`
`Bird Feed
`Cameras
`at SEARS
`
`Emo
`
`30
`
`

`

`U.S. Patent
`
`Feb. 25, 2014
`
`Sheet 30 of 42
`
`US 8,660,545 B1
`
`FIG. 9A (PRIOR ART)
`
`
`
`DATANETWORK INTERFACE CIRCUIT
`
`902
`
`904.
`
`POWER
`SUPPLY
`
`WiFi
`Or 4G
`CIRCUIT &
`ANTENNA
`
`MICROCONTROLLER
`(including PROCESSOR
`and Read Only Memory)
`
`HDM
`CONNECTOR
`
`905
`
`USB
`CONNECTOR
`
`ETHERNET
`CONNECTOR
`
`HDM
`CONNECTOR
`
`SCREEN
`307
`
`303
`internet-enabled Television
`
`31
`
`

`

`U.S. Patent
`
`Feb. 25, 2014
`
`Sheet 31 of 42
`
`US 8,660,545 B1
`
`FIG. 9B
`
`Set-top Application
`
`AUTHORTY-COMPATIBLE
`SET-TOP INTERFACE
`(Remote Control Interface
`Authority Server Interface,
`Content Provider Interface)
`
`SDK Shared Library
`
`
`
`
`
`
`
`LINUX KERNEL
`
`DEVICE DRIVERS
`
`912
`
`MEMORY
`
`PROCESSOR
`
`916
`
`915
`
`
`
`914
`
`913
`
`911
`
`921
`
`FIG. 9C
`
`
`
`
`
`Play video
`
`
`
`Decrypt wideO
`
`
`
`92 2
`
`923
`
`NO
`
`928
`
`Transmit Wideo Fetch to Content Provider Serve
`(with Optional DRM token)
`924
`
`ls video protected with DRM 2
`
`YES
`
`926
`ls
`a matching DRM module
`available?
`
`Send error message
`to Authority Server
`
`32
`
`

`

`U.S. Patent
`
`Feb. 25, 2014
`
`Sheet 32 of 42
`
`US 8,660,545 B1
`
`FIG. 10A (PRIOR ART)
`
`
`
`HANDHELD
`DEVICE
`
`1005
`
`BATTERY
`
`WRELESS
`ANTENNA
`
`1000
`
`MICROCONTROLLER
`(including PROCESSOR
`and Read Only Memory)
`
`TOUCH
`SCREEN
`
`FIG. 1 OB
`202B 202.
`
`202N
`
`202A
`
`
`
`EPG
`EPG
`APPB E. APPN
`209A. A.E.-6::::::
`Authority-
`Authority-
`Authority-
`209N
`
`Handheld
`Interface
`SDK
`Shared
`Library
`
`Handheld
`Handheld
`Handheld
`Interface
`Interface
`Interface
`SDK
`SDK
`SDK
`Shared
`Shared
`Shared
`Library
`Library
`Library
`OPERATING SYSTEM 1011
`
`
`
`MEMORY
`
`PROCESSOR
`
`1001
`
`FIG. 10C
`
`202A
`
`202B
`
`202
`
`202N
`
`EPG |
`EPG
`APPA APP B
`
`EPG
`APP
`
`EPG
`APP N
`
`Authority-Compatible Handheld Interface
`SDK Shared Library
`1012
`OPERATING SYSTEM 1011
`
`MEMORY
`
`PROCESSOR
`
`209
`
`
`
`
`
`
`
`
`
`1001
`
`33
`
`

`

`U.S. Patent
`
`Feb. 25, 2014
`
`Sheet 33 of 42
`
`US 8,660,545 B1
`
`Message
`
`1031
`
`
`
`1032
`
`ls User-selected video free?
`
`103
`
`NO
`
`DisplayList of
`Payment Plans
`
`Request List(s) of
`Video Descriptions
`
`Display List(s) of
`Video Descriptions
`
`FIG. 1 OE
`
`
`
`
`
`1035
`
`:
`
`Transmit dentifier of
`Video to Authority Server
`
`Receive Messages (e.g.
`caused by Video Request)
`
`Wait for User Command
`or for Messages from
`Authority Server
`
`1039
`
`1036
`Determine type of each
`received message
`
`Receive user input on payment plan &
`transmit payment information to
`authority server & optionally store user
`Selection of payment plan for future use
`
`
`
`1037
`
`Disayas E after
`displaying additional
`information based On
`message type
`
`1038
`
`34
`
`

`

`U.S. Patent
`
`Feb. 25, 2014
`
`Sheet 34 of 42
`
`US 8,660,545 B1
`
`FIG. 11A (prior art)
`
`to
`
`
`
`1103
`
`1104
`
`
`
`FIG. 11B
`Receive Connection Request from an internet-enabled TV
`
`
`
`
`
`internet-enabled TW
`exist in database2
`
`
`
`Create a Record for internet
`enabled TV in Database
`
`
`
`Retrieve Record
`
`
`
`Mark TV as available;
`instruct TW to aWait COmmand
`
`35
`
`

`

`U.S. Patent
`
`Feb. 25, 2014
`
`Sheet 35 of 42
`
`US 8,660,545 B1
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`FIG. 11C
`
`
`
`Receive an Application Launch signal
`from Handheld Device
`
`1126
`
`Handheld device
`exist in database2
`
`
`
`Retrieve Record
`
`Create a Record for handheld
`device in Database (including
`user name and passWOrd)
`
`Mark handheld device as available;
`Senda Session identifier to
`handheld device & await Command
`
`Receive Command to form an association, from a
`Handheld Device identifying an internet-enabled TW
`
`1131
`
`1132
`ls internet-enabled TW
`marked as available?
`
`Send Error message back to
`Handheld Device indicating
`that the TW is not available
`
`1134
`
`
`
`Mark handheld device and TV as
`being aSSOciated
`
`Return Success to Handheld Device
`
`1136
`
`AWait COmmand from Handheld Device
`
`36
`
`

`

`U.S. Patent
`
`Feb. 25, 2014
`
`Sheet 36 of 42
`
`US 8,660,545 B1
`
`FIG. 11E
`
`FIG. 11F
`
`FIG. 11G
`
`
`
`
`
`
`
`Execute procedure to
`law wideO
`Hs y
`(see FIG. 11
`
`37
`
`

`

`U.S. Patent
`
`Feb. 25, 2014
`
`Sheet 37 of 42
`
`US 8,660,545 B1
`
`FIG 1 1 H
`
`Receive Command to Play Video
`(including video identifier)
`
`1151
`
`Retrieve from Database, association of internet
`enabled TV with Handheld Device
`
`1152
`
`Send Command to Computer of Content Provider
`to Play Video (including video identifier and TV
`identifier) & Optionally DRM token
`
`1153
`
`Send to internet enabled TV, the URL Ofyideo at
`Content Provider Computer & Optionally DRM token
`
`1154
`
`Retrieve list of frames predetermined to trigger messages;
`Send list of frames to internet-enabled TW
`
`155
`
`1156
`
`
`
`Retrieve keywords associated with Current frame number
`
`Match retrieved keywords (of frame number) to keywords
`associated with E (in a repository of messages),
`optionally using attributes of the user of handheld device
`
`Sort messages resulting from match,
`using a ranking Criterion (e.g. attribute of each message)
`
`Send a predetermined number
`(e.g. five) messages from the
`top of a list of sorted messages
`
`
`
`159
`
`AWait COmmand from internet-enabled TV
`
`
`
`
`
`
`
`
`
`YES
`
`Has WideO Ended?
`
`Automatically select a TV
`channel and transmit channel
`number to internet-enabled TV
`
`164
`
`Exit
`
`ls a Frame
`umber received
`
`38
`
`

`

`U.S. Patent
`
`Feb. 25, 2014
`
`Sheet 38 of 42
`
`US 8,660,545 B1
`
`
`
`FIG. 11
`
`Receive edits to keywords
`
`FIG. 11
`
`
`
`
`
`
`
`
`
`
`
`Store Message Provider's Configuration in Database
`
`
`
`Receive metadata of a message, including specific
`keywords to be used in matching videOS being displayed
`and value of a ranking Criterion to be used in Sorting
`
`
`
`176
`
`1177
`
`178
`
`Store the message's metadata, including keywords and attributes
`
`Receive edits to keywords Or attribute(s)
`118O
`
`39
`
`

`

`U.S. Patent
`
`Feb. 25, 2014
`
`Sheet 39 of 42
`
`US 8,660,545 B1
`
`FIG 1 1 K
`
`Receive Command to Play Video
`(including video identifier)
`
`1151
`
`Retrieve from Database, association Of internet
`enabled TW With Handheld Device
`
`152
`
`Send Command to Computer of Content Provider
`to Play Video (including video identifier and TV
`identifier) & Optionally DRM token
`
`1153
`
`Send to internet-enabled TV, the URL Of Video at N 154
`Content Provider Computer & optionally DRM token
`
`YE
`
`Sa Trig
`received?
`
`
`
`Automatically select a TV
`channel and transmit channel
`number to internet-enabled TW
`
`Exit
`
`
`
`
`
`
`
`
`
`
`
`DOne
`messaging all
`handheld devices
`asSOciated with
`TV
`
`
`
`Match trigger keywords to keywords associated with
`messages (in a repository of messages)
`Optionally using attributes of the user of handheld device
`
`Sort messages resulting from match,
`using a ranking criterion (e.g. attribute of each message)
`
`Send a predetermined number
`(e.g. five) messages from the
`top of a list of SOrted messages
`
`40
`
`

`

`U.S. Patent
`
`Feb. 25, 2014
`
`Sheet 40 of 42
`
`US 8,660,545 B1
`
`F.G. 12
`Receive instructions for logic to be used
`in an EPG application from a developer
`
`
`
`
`
`Build EPG application by Compili and linkind instructions
`for logic with authority-Compati le handheld interface
`
`
`
`2O3
`
`41
`
`

`

`U.S. Patent
`
`Feb. 25, 2014
`
`Sheet 41 of 42
`
`US 8,660,545 B1
`
`FIG. 13A
`
`Receive a list of wide0 keVWOrds ranked
`by rel
`to th
`t
`id
`y relevance to the Content proVICer
`
`301
`
`Generate a first list of KRAD messages
`(by matching video keywords to all KRAD
`messages in promotional message repOsitory)
`
`302
`
`
`
`Generate a second list of SOcial media
`messages (by matching vide0 keyWOrds
`to all social media messages in Social
`media message repository)
`
`1303
`
`Sort first list by size of bid associated with each KRAD message
`
`304
`
`Sort second list, based on the relevance of the keyword(s) that match
`each SOcial media message
`
`130
`
`1306
`
`130
`
`Create a new list by repeating apredetermined
`number of times: insert an item from the first list
`followed by an item from the second list
`
`Transmit the new list to handheld device
`
`42
`
`

`

`U.S. Patent
`
`Feb. 25, 2014
`
`Sheet 42 of 42
`
`US 8,660,545 B1
`
`FG 1 3B
`
`Receive a list of video keywords ranked
`by relevance to the Content provider
`
`Generate a first list of CRAD messages
`(by matching video keywords to all CRAD
`messages in promotional message repository)
`
`1311
`
`1312
`
`Sort first list by size of bid associated with each CRAD message
`
`Select the highest ranked CRAD from the SOrted version of the first list
`
`Generate a second list of KRAD messages
`(by matching video keywords to all KRAD
`messages in promotional message repository)
`
`315
`
`1313
`
`1314
`
`Sort second list by size of bid associated with each KRAD message
`
`1316
`
`Select the highest ranked KRAD from the sorted version of second lis
`
`1317
`
`
`
`
`
`
`
`
`
`
`
`ls
`e size of bid of highest ranked CRAD
`greater than the size of bid of highest
`ranked KRAD
`
`Create a new list With the
`highest ranked CRAD at the
`top and appending a
`predetermined number of
`KRADS from the sorted
`version of the second list
`
`
`
`
`
`
`
`Transmit the new list
`Of ES to display
`On handheld device
`
`1321
`
`
`
`
`
`
`
`43
`
`

`

`US 8,660,545 B1
`
`1.
`RESPONDING TO AVIDEO REQUEST BY
`DISPLAYING INFORMATION ON A TV
`REMOTE AND VIDEO ON THE TV
`
`BACKGROUND
`
`A smartphone 100 (FIG. 1A) of the prior art enables a user
`to make a telephone call as well as provides Support for
`multiple applications ("apps'), such as a web browser or
`email client. An example of a Smartphone is the product
`called “iPhone' available from APPLE INC, e.g. as described
`in US Patent Publication 2008.0122796 published on May 29,
`2008 by Jobs etal, and in U.S. Pat. No. 7,479,949 granted to
`Jobs etal on Jan. 20, 2009 both of which are incorporated by
`reference herein in their entirety as background information
`on conventional Smartphones.
`One of the applications (“apps') in such a smartphone 100
`may be, for example, a store application program ('store
`app') 101 (FIG. 1A) that enables users to download addi
`tional applications. Specifically, on touching store app 101
`(FIG. 1A), Smartphone 100 displays the user interface illus
`trated in FIG. 1B, wherein the user may enter a search term in
`box 102. Below search box 102 is displayed a list 103 of
`identifications 103A-103D of application programs, such as a
`music app, a camera app, a travel app and an app to search for
`toys, any or all of which may be downloaded into Smartphone
`100, from a computer at a predetermined address on the
`Internet.
`One such app is created by YouTube, to enable a user to
`view on the Smartphone iPhone, any video that is posted on
`the YouTube website. Specifically, touching a YouTube icon
`shown by the user interface of the iPhone opens the YouTube
`app, which displays on a touch screen of the iPhone, a main
`page of the YouTube website. The user may then enter appro
`priate keywords in a searchfield on the iPhone's touchscreen,
`to find and select a video which is then displayed on the
`iPhone's touch screen. For more information, see an article
`entitled 'A Tutorial on Watching YouTube Videos From an
`iPhone” by TI Kleban published on Ezine on Oct. 19, 2007.
`Another app called “vTap' allows a user to search for videos
`on the web, for example at the following websites: YouTube,
`MySpace, Dailymotion, news sites like the Associated Press
`and Reuters, even ESPN. For more information, see “Possibly
`the best iPhone Web app yet: Vitap trumps built-in iPhone
`YouTube app” by Ben Wilson published on CNET, on Nov. 2,
`2007.
`The two apps described in the previous paragraph require
`the user to view video on a screen built into a Smartphone.
`Another app called “i.TV enables a user who subscribes to
`Netflix to manage their queue of films available via Netflix,
`and also enables the user to control a digital video recorder
`(DVR) of the type available from TiVo, Inc. For more infor
`mation, see the following articles each of which is incorpo
`rated by reference herein in its entirety as background:
`(A)'Appiphilia: i.TV app tunes in new features for iPhone'
`published by Michelle Maltais of the Los Angeles Times on
`Dec. 23, 2008; (B) “i.TV’s iPhone app gets deep Netflix
`integration' published Nov. 10, 2008 by Josh Lowensohn of
`CNET and (C) “i.TV for iPhone: Cluttered but promising
`entertainment guide” published Aug. 17, 2009, by Erica
`Sadun of Weblogs, Inc.
`
`10
`
`15
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`SUMMARY
`
`In accordance with the invention, a handheld screen is used
`as a Supplement to a television screen. Specifically, informa
`tion displayed on a handheld Screen in accordance with the
`
`65
`
`2
`invention Supplements a video displayed on the television
`(TV) screen. In many embodiments of the invention, display
`of supplementary information on the handheld Screen is coor
`dinated with the display of video on the television screen. In
`Some embodiments, the Supplementary information on the
`handheld screen is selected by a computer system to match
`the specific video being displayed on the TV screen. Addi
`tionally, in several such embodiments of the invention, the
`display of Such supplementary information occurs simulta
`neously or contemporaneously relative to the video display
`e.g. the two displays are made to overlap in time by the
`computer system transmitting the selected information (oran
`identifier thereof) to a handheld device (containing the hand
`held Screen) at the same time as (or within a predetermined
`duration of) transmission of the video to a television (con
`taining the television screen). Such display of Supplementary
`information on a handheld Screen in a timely manner relative
`to video display on the TV screen (i.e. while the video is still
`being displayed) in accordance with the invention provides a
`richer viewing experience to a user than any prior art combi
`nation known to the inventors of the current patent applica
`tion.
`In Some embodiments, transmission of the video and trans
`mission of the Supplementary information are both transmit
`ted in response to a single event, for example receipt by the
`computer system, of a request to display the video, on
`demand. In numerous embodiments, the handheld device
`transmits to the computer system, a single video request
`containing an identification of a user-selected video (e.g. a
`URL of the file containing the video. The computer system
`responds by transmitting to the television a first signal carry
`ing a series of frames in the user-selected video, and further
`transmitting to the handheld device, a second signal carrying
`Supplementary information that is automatically selected for
`display on the handheld device, both signals being based on
`the single video request. The television responds by automati
`cally displaying on the television screen, the series of frames
`being received in the first signal. The handheld device
`responds by automatically displaying on the handheld Screen,
`the Supplementary information received in the second signal.
`As described above, the two signals are both transmitted by
`the computer system in response to the same event (for
`example, in the numerous embodiments described above, by
`a request for a user-selected video). In Such embodiments,
`coordinated displays, of video on the television and Supple
`mentary information on the handheld device, occur in a
`timely manner relative to one another.
`
`BRIEF DESCRIPTION OF THE FIGURES
`
`FIGS. 1A and 1B show a smartphone 100 of the prior art.
`FIG. 2A illustrates a handheld device 200 (e.g. a smart
`phone) programmed to display multiple apps available for
`download from an app store, each app containing therein an
`electronic program guide (EPG) of videos available for on
`demand display on an internet-enabled television in accor
`dance with the invention.
`FIG.2B illustrates, in a high level block diagram, handheld
`device 200 of FIG. 2A, wherein multiple EPG apps have been
`downloaded and installed as represented by icons