111111
`
`1111111111111111111111111111111111111111111111111111111111111
`US0084 79238B2
`
`c12) United States Patent
`Chen et al.
`
`(10) Patent No.:
`(45) Date of Patent:
`
`US 8,479,238 B2
`Jul. 2, 2013
`
`(54) METHOD FOR CONTENT-BASED
`NON-LINEAR CONTROL OF MULTIMEDIA
`PLAYBACK
`
`(75)
`
`Inventors: Edward Y. Chen, Holmdel, NJ (US);
`David Crawford Gibbon, Lincroft, NJ
`(US); Laurence W. Ruedisueli,
`Berkeley Heights, NJ (US); Behzad
`Shahraray, Freehold, NJ (US)
`
`(73) Assignee: AT&T Intellectual Property II, L.P.,
`Atlanta, GA (US)
`
`( *) Notice:
`
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 1364 days.
`
`(21) Appl. No.: 10/146,300
`
`(22) Filed:
`
`May 14,2002
`
`(65)
`
`Prior Publication Data
`
`US 2002/0170062 AI
`
`Nov. 14, 2002
`
`Related U.S. Application Data
`
`(60) Provisional application No. 60/290,788, filed on May
`14, 2001.
`
`(51)
`
`Int. Cl.
`H04N51445
`H04N71173
`(52) U.S. Cl.
`USPC ................... 725/86; 725/39; 725/40; 725/44;
`725/88
`
`(2011.01)
`(2011.01)
`
`(58) Field of Classification Search
`USPC ....... 725/39-41, 52-53, 86-88, 91, 102-104;
`715/723-725, 835, 838
`See application file for complete search history.
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`5,132,992 A * 7/1992 Yurt eta!. ..................... 375/240
`5,410,326 A
`4/1995 Goldstein
`5,539,479 A
`7/1996 Bertram
`5,572,260 A * 1111996 Onishi eta!. .................. 348/460
`5,606,359 A * 2/1997 Youden eta!. .................. 725/88
`5,610,653 A * 3/1997 Abecassis ..................... 348/170
`5,619,249 A * 4/1997 Billock et al.
`.................... 725/5
`5,818,439 A * 10/1998 N agasaka et al. ............... 725/87
`5,835,667 A * 1111998 Wactlar eta!. ................ 386/241
`5,844,620 A
`12/1998 Coleman
`5,854,894 A * 12/1998 Lancaster et a!. ............. 709/219
`5,864,682 A *
`111999 Porter eta!. .................. 709/247
`5,877,755 A
`3/1999 Helhake
`5,884,056 A * 3/1999 Steele ........................... 715/738
`5,900,868 A * 5/1999 Duhault et a!.
`................. 725/42
`5,917,488 A * 6/1999 Anderson et a!. ............. 715/838
`5,966,122 A * 10/1999 Itoh ............................... 715/838
`5,995,155 A
`1111999 Schindler
`6,061,056 A * 5/2000 Menard eta!. ................ 715/704
`6,098,082 A
`8/2000 Gibbon
`(Continued)
`Primary Examiner- An Son P Huynh
`ABSTRACT
`(57)
`A system and method are provided for content-based non(cid:173)
`linear control of video data playback. A multimedia database
`having multimedia data including multimedia content data is
`searched based on a user query to determine a first set of
`multimedia data. The multimedia data includes indexes to
`and condensed representations of corresponding video data
`stored in a video database. A portion of the first set of multi(cid:173)
`media data is displayed at a control device in response to the
`user query. A user of the control device selects an element of
`the first set of multimedia data for video playback and video
`data corresponding to the element delivered to a video device
`for playback. A user of the control device selects an element
`of the first set of multimedia data for additional information
`and a second set of multimedia data corresponding to the
`element delivered to the control device.
`21 Claims, 5 Drawing Sheets
`
`222
`
`212
`
`Control Device
`
`218
`
`Video Device
`
`Video
`Display
`316
`
`Page 1 of 14
`
`Unified Patents Exhibit 1003
`
`

`
`US 8,479,238 B2
`Page 2
`
`U.S. PATENT DOCUMENTS
`6,104,334 A *
`8/2000 Allport ......................... 3411175
`6,118,450 A *
`9/2000 Proehl et al ................... 715/810
`6,119,154 A *
`9/2000 Weaver eta!. ................ 709/219
`6,139,197 A *
`10/2000 Banks ........................... 709/217
`6,144,375 A *
`1112000 Jainetal. ................... 715/500.1
`6,154,600 A *
`1112000 Newman eta!. .................. 386/4
`6,154,771 A *
`1112000 Rangan eta!. ................ 709/217
`6,173,317 B1 *
`112001 Chaddha et a!. .............. 709/219
`6,185,573 B1
`212001 Angelucci eta!.
`6,188,398 B1 *
`212001 Collins-Rector et al ........ 725/37
`6,236,395 B1 *
`5/2001 Sezan eta!. ................... 715/723
`6,271,892 B1 *
`8/2001 Gibbon et al ................. 348/700
`6,342,904 B1 *
`112002 Vasudevan eta!. ........... 715/723
`6,349,410 B1 *
`212002 Lortz ............................ 725/110
`6,405,371 B1 *
`6/2002 Oosterhout et al.
`. ........... 725/39
`6,414,725 B1 *
`7/2002 Clarinet a!.
`.................. 348/714
`6,415,326 B1 *
`7/2002 Gupta et a!.
`.................. 709/231
`6,421,706 B1 *
`7/2002 McNeill et al ................ 709/204
`6,449,608 B1 *
`9/2002 Morita eta!. ..................... 707/3
`6,470,378 B1 *
`10/2002 Tracton et a!. ................ 709/203
`6,473,778 B1 *
`10/2002 Gibbon ......................... 715/201
`6,509,908 B1 *
`112003 Croy eta!. .................... 715/716
`6,535,639 B1 *
`3/2003 Uchihachi eta!. ............ 382/225
`6,567,980 B1 *
`5/2003 Jain eta!. ........................ 725/61
`6,580,437 B1 *
`6/2003 Liou et al ...................... 715/719
`6,584,463 B2 *
`6/2003 Morita eta!. ..................... 707/3
`6,601,103 B1 *
`7/2003 Goldschmidtlki eta!. .. 709/231
`6,615,252 B1 *
`9/2003 Oka eta!. ...................... 709/219
`6,631,523 B1 *
`10/2003 Matthews eta!. ............... 725/53
`6,668,377 B1 *
`12/2003 Dunn .............................. 725/92
`6,710,822 B1 *
`3/2004 Walker et al .................. 348/722
`6,711,741 B2 *
`3/2004 Yeo ................................. 725/87
`6,757,907 B1 *
`6/2004 Schumacher eta!. .......... 725/87
`6,760,916 B2 *
`7/2004 Holtz et al ...................... 725/34
`6,788,882 B1
`9/2004 Geer eta!.
`
`6,789,106 B2 *
`6,829,781 B1 *
`6,870,573 B2 *
`6,877,134 B1 *
`6,880,171 B1 *
`6,882,793 B1 *
`6,918,132 B2 *
`6,938,208 B2 *
`6,956,573 B1 *
`6,961,954 B1 *
`6,965,724 B1 *
`7,065,250 B1 *
`7,073,127 B2 *
`7,107,532 B1 *
`7,174,512 B2 *
`7,178,107 B2 *
`7,200,857 B1 *
`7,237,254 B1 *
`7,293,280 B1 *
`7,301,944 B1 *
`7,313,808 B1 *
`7,325,199 B1 *
`7,401,351 B2 *
`8,091,112 B1 *
`200110049826 A1 *
`2002/0056098 A1 *
`2002/0078176 A1 *
`2002/0170068 A1 *
`2002/0194151 A1 *
`2003/0066085 A1 *
`2004/0117831 A1 *
`2005/0028194 A1 *
`2005/0028208 A1 *
`2005/0229213 A1 *
`2005/0240961 A1 *
`* cited by examiner
`
`9/2004
`12/2004
`3/2005
`4/2005
`4/2005
`4/2005
`7/2005
`8/2005
`10/2005
`1112005
`1112005
`6/2006
`7/2006
`9/2006
`2/2007
`2/2007
`4/2007
`6/2007
`1112007
`1112007
`12/2007
`112008
`7/2008
`112012
`12/2001
`5/2002
`6/2002
`1112002
`12/2002
`4/2003
`6/2004
`2/2005
`2/2005
`10/2005
`10/2005
`
`Eyer eta!. ..................... 709/205
`Bhagavath eta!
`............. 725/94
`Yeo et al.
`. ............ 348/569
`Fuller eta!. ................ 715/500.1
`Ahmad et al.
`. ............... 725/134
`Fu et al ........................... 386/95
`Gargi .............................. 725/45
`Reichardt ..................... 715/719
`Bergen eta!. ................. 345/473
`Maybury eta!. ................ 725/53
`Boccon-Gibod eta!. ....... 386/68
`Lennon ......................... 382/224
`Zhao eta!. .................... 715/719
`Billmaier eta!. ............. 715/720
`Martinet a!.
`................. 715/719
`Sezan eta!. ................... 715/719
`Rodriguez eta!. .............. 725/87
`Omoigui ......................... 725/94
`Gupta eta!
`.................. 725/139
`.................... 370/390
`Redmond
`Gupta et a!
`.................... 725/89
`Reid
`.................... 715/723
`............... 725/88
`Boreczky et al
`Elliott eta!.
`............... 725/135
`Wilf .............................. 725/120
`White ............................. 725/39
`Nomura et al ................ 709/219
`Rafey et al .................... 725/112
`Fenton eta!. ..................... 707/1
`Boyer et al ................... 725/104
`Ellis et al ........................ 725/53
`Elenbaas et al ................. 725/32
`Ellis et al ........................ 725/58
`Ellis et al ........................ 725/58
`. ................. 725/37
`Jerding et al.
`
`Page 2 of 14
`
`

`
`U.S. Patent
`
`Jul. 2, 2013
`
`Sheet 1 of 5
`
`US 8,479,238 B2
`
`110
`
`I
`
`I
`Video Source
`Material
`
`c
`
`120
`
`13 0
`
`/
`
`Ancillary
`Source Ma::erial
`
`Media Analysis -
`Metadata
`Extraction
`
`... ...
`
`:::
`
`--
`
`Figure 1
`
`--::;
`
`140
`I
`~ I
`!vlul t imedia
`Database &
`Video Database
`---
`
`'--
`
`-' ' ' ' ' ' ' ' ' ' ' '~
`214 t
`
`Network
`Access Point
`
`216....-------
`
`Control
`Device
`212
`
`Consumer Premises 210
`Video
`Device
`218
`
`Multimedia
`Database
`222
`
`Video
`Database
`220
`
`~
`
`,
`
`Figure 2
`
`Page 3 of 14
`
`

`
`..
`Processing ~ 324
`----
`I
`
`N0twork
`Interface
`332
`
`-..
`
`...
`
`Network
`Interface
`314
`
`Video
`Decoder
`318
`
`Network
`Interface
`320
`
`..
`
`U.S. Patent
`
`Jul. 2, 2013
`
`Sheet 2 of 5
`
`US 8,479,238 B2
`
`222
`
`212 Control Device
`
`--. _.
`rC
`Multi:necia
`Database
`l?R
`
`_.
`
`'-
`
`Query
`Processing
`330
`
`218
`
`Video Device
`
`Video
`Display
`316
`
`Graphic
`Display
`310
`
`Display
`Controller
`312
`
`220
`
`Video
`Database
`322
`
`Request
`
`Network
`Interface
`l?h
`
`~
`
`Streaming Video and Control
`
`410
`
`412
`
`Figure 3
`
`?igure 4
`
`Page 4 of 14
`
`

`
`U.S. Patent
`
`Jul. 2, 2013
`
`Sheet 3 of 5
`
`US 8,479,238 B2
`
`530
`
`20
`
`22
`
`28
`
`610
`
`Figure 5
`
`Figure 6
`
`Page 5 of 14
`
`

`
`U.S. Patent
`
`Jul. 2, 2013
`
`Sheet 4 of 5
`
`US 8,479,238 B2
`
`Figure 7
`
`810
`
`12
`
`Figure 8
`
`Page 6 of 14
`
`

`
`U.S. Patent
`
`Jul. 2, 2013
`
`Sheet 5 of 5
`
`US 8,479,238 B2
`
`Figure 9
`
`Figure 10
`
`1010
`
`Page 7 of 14
`
`

`
`US 8,479,238 B2
`
`1
`METHOD FOR CONTENT-BASED
`NON-LINEAR CONTROL OF MULTIMEDIA
`PLAYBACK
`
`FIELD OF THE INVENTION
`
`The present invention generally relates to the control and
`playback of user selected video material based on the content
`of the video material.
`
`BACKGROUND OF THE INVENTION
`
`Audio and video signal streams, whether carrying analog
`or digitally encoded information, have come to represent a
`significant source of consumer information and entertain(cid:173)
`ment. Traditionally such signal streams were provided to
`consumers over radio frequency bands through use of a trans(cid:173)
`mitter and a receiver. Changes in distribution methods have
`led to the use of various alternative devices such as cable set
`top boxes, video recorder/player devices such as cassette
`recorders and video disc players, or direct live video sources
`such as home cameras, game systems, and computers, and
`other alternative devices. As such devices have proliferated,
`so too have their associated control devices such that consum-
`ers are frequently faced with a proliferation of control 25
`devices. For example, in "home theater systems" having
`numerous interconnected devices, several or even all the
`devices may have its own control device which enables the
`consumer to control the functionality of the respective device
`remotely without directly manipulating controls on the 30
`device.
`In addition, increased availability of networks for the trans(cid:173)
`mission of signal streams has vastly increased the number and
`variety of information and entertaiument services available to
`consumers. As a result, consumers have trouble keeping 35
`informed as to the scheduling of available services. Accord(cid:173)
`ingly, some cable television systems provide a dedicated
`channel that displays a current programming guide for sched(cid:173)
`uled services. However, such a channel typically slowly
`scrolls a limited and fixed amount of information, usually 40
`only one to three hours of program scheduling. Moreover,
`these systems lack any user interface capability and cannot be
`customized to suit individual consumer's needs so that a
`consumer must inconveniently wait to potentially view infor(cid:173)
`mation of interest and then, may not even be provided with 45
`information suited to their particular interests.
`Some systems enable searching of displayed data, such as
`the title of a movie or the name of a song, associated with
`stored video data. However, such systems do not facilitate
`navigation of large aggregates of video material, or naviga- 50
`tion within a particular video material entry, such as a single
`video program, to enable the consumer to find information of
`particular interest. A consumer will thus often times have to
`navigate linearly through a large amount of irrelevant video
`material in order to find desired video programming, wasting 55
`time and effort. Moreover, such excessive navigation likewise
`requires excessive bandwidth when in communication with a
`video server, failing to minimize the time necessary for video
`material retrieval, and thereby slowing the network over
`which such communication occurs.
`
`2
`first set of multimedia data. The multimedia data includes
`indices to information pertaining to the information content
`of corresponding video data stored in a video database. The
`multimedia, data also includes condensed representations of
`corresponding video data, such as image scenes or closed
`captioned information concerning what was spoken in a
`image scene. A portion of the first set of multimedia data is
`displayed at a control device. A user of the control device
`selects an element of the first set of multimedia data for video
`10 playback and video data corresponding to the element deliv(cid:173)
`ered to a video device for playback. A user of the control
`device selects an element of the first set of multimedia data for
`additional information and a second set of multimedia data
`corresponding to the element is delivered to the control
`15 device.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`Further aspects of the instant invention will be more readily
`20 appreciated upon review of the detailed description of the
`embodiments included below when taken in conjunction with
`the accompanying drawings, of which:
`FIG. 1 illustrates an exemplary architecture for populating
`a multimedia system with detailed content specific informa(cid:173)
`tion;
`FIG. 2 is an exemplary network topology for implementing
`the invention;
`FIG. 3 illustrates additional detail of the exemplary net(cid:173)
`work topology shown in FIG. 2;
`FIG. 4 illustrates an exemplary control device screen for
`specifYing the video device that is to be controlled;
`FIG. 5 illustrates an exemplary control device screen for
`the main search term selection screen (i.e., "home" screen);
`FIG. 6 illustrates an exemplary control device screen for an
`topics list of exemplary common search terms;
`FIG. 7 illustrates an exemplary control device screen of
`multimedia data indicating video data relevant to a particular
`search term;
`FIG. 8 illustrates an exemplary control device screen of
`multimedia data indicating portions of individual video data
`relevant to a particular search term;
`FIG. 9 illustrates an exemplary control device screen of an
`interface for browsing video material that has been close(cid:173)
`captioned; and
`FIG.10 illustrates an exemplary control device screen of an
`interface for browsing video material that has not been close(cid:173)
`captioned.
`
`DETAILED DESCRIPTION OF THE INVENTION
`
`Referring now to FIGS. 1-10, wherein similar components
`of the invention are numerically referenced in like manner,
`disclosed are embodiments of an architecture and method for
`accessing, over a network, user selected video data based on
`selected multimedia data.
`FIG. 1 illustrates an exemplary architecture for populating
`a multimedia system with detailed content specific informa(cid:173)
`tion. Video source materialllO is provided to a media analy(cid:173)
`sis/metadata extraction device 120. The media analysis/meta-
`60 data extraction device extracts content specific index data
`from the provided video material using automatic media pro(cid:173)
`cessing techniques such video indexing and/or audio index(cid:173)
`ing. The media analysis/metadata extraction device converts
`the video source material into video data and multimedia data
`65 for storage in a multimedia database and video database 140.
`Content specific index data determined by the media analysis/
`meta data extraction device is included in the multimedia data.
`
`SUMMARY OF THE INVENTION
`
`A system and method are provided for content-based non(cid:173)
`linear control of video data playback. A multimedia database
`having multimedia data including multimedia content data is
`searched based on a user-selected parameter to determine a
`
`Page 8 of 14
`
`

`
`US 8,479,238 B2
`
`3
`Each logical unit of video source material (e.g., a television
`program, a video-tape, a corporate speech) is broken down by
`the media analysis/metadata extraction device into a distinct
`entry for the multimedia database and video database.
`In addition, the media analysis/metadata extraction device
`may also break video source material into smaller entries
`based on topic or story boundaries of the video source mate(cid:173)
`rial. The video source material may also be analyzed auto(cid:173)
`matically to detect video shot boundaries and to record any
`associated closed caption information. Further, automated 10
`speech recognition, such as Large Vocabulary Automated
`Speech Recognition (LVASR) can be used to obtain a tran(cid:173)
`scription for inclusion in the multimedia data. In another
`alternative, the content specific index data is extracted from 15
`the video data manually, a human functioning as that portion
`the media analysis/metadata extraction device that deter(cid:173)
`mines content specific index data.
`Video data includes that data used to display the video
`source material100 on a video device (i.e., the video itself). 20
`Multimedia data includes multimedia content data, which is
`data based on the content of the video data, such as content
`specific index data, for example, an index of select keywords
`referenced in the video data. Examples of multimedia content
`data include still images (e.g., images of a scene from a 25
`movie), preview videos (e.g., a theatrical trailer), detailed
`information about the video data (e.g., names of actors in a
`video program, the cinematography of the director of a video
`program), and the like. Multimedia data may also include
`metadata, which is other data used to index both the video 30
`data and multimedia data, such as the title of a movie, the
`name of a song, or the like. In addition to creating multimedia
`data, the media analysis/metadata extraction device also may
`digitize and compress the video source material for storage on
`the multimedia database and video database 140. For 35
`example, video data may be stored in a format such as MPEG-
`1, MPEG-2, MPEG-4, or other formats and delivered across
`the network at various bitrates. Preferably, multiple versions
`of the video data in various formats are stored so that the
`highest possible quality version may be later delivered to a
`consumer based on available bandwidth and the consumer's
`video device capabilities.
`Ancillary source material120 may also be provided to the
`media analysis/metadata extraction device 130 to improve the
`accuracy of indexing or to utilize other information suitable
`for indexing and retrieving video data. Examples of ancillary
`data include offline transcriptions, manual annotations, topic
`classifications, post-production scripts, metadata, genre clas(cid:173)
`sifications, and the like. With ancillary source material, the
`media analysis/metadata extraction device 130 would utilize
`both the video source material 110 and the ancillary source
`material120 during the conversion into video data and mul(cid:173)
`timedia data, including content specific index data, for stor(cid:173)
`age in the multimedia database and video database 140.
`For each video data (e.g., television program) in the video
`database corresponding multimedia data is stored in the mul(cid:173)
`timedia database. Multimedia data that may be stored
`includes a program stream, image frames and associated
`metadata (e.g., time within broadcast that the frame was
`sample, type of video transition) representing each scene,
`program metadata including title, broadcaster, time and date
`of airing, closed caption text, automatic speech transcrip(cid:173)
`tions, and various data structures. For example, to indicating
`pagination each multimedia data representation of video data
`can be divided into sets of HTML pages and data structures 65
`established that indicate the number ofHTML pages, and the
`captions and images that appear on given page. Optionally,
`
`4
`the multimedia data that is stored can include an offline tran(cid:173)
`scription that has been synchronized with the video data.
`A number of such sets of video data and corresponding
`multimedia data are stored in the multimedia database and
`video database. Index services, (e.g., MS Indexing Service)
`are used to provide full text search capability on either the
`closed caption text and/or the offline transcription. HTML
`metatags are included with the text so that relational queries
`can be supported (e.g., "find all video data containing the term
`"NASA" from the broadcaster "NBC" that are less than one
`year old.") The system combines the basic closed caption text
`and or speech transcription with the metadata to generate files
`that a sever can use as content for searching. Once video data
`matching a consumer's query has been identified, application
`software generates a content rich interface for browsing the
`corresponding multimedia data and initiating video data play-
`back.
`In this marmer, the invention provides a database of infor(cid:173)
`mation relating to video material, facilitates navigation of
`such information (e.g., a plurality of television programs) and
`further, facilitates navigation with a particular item of such
`information, such as, a particular television program. An
`illustrative example of such a database of information is an
`archive of all television programs in a particular geographic
`regions for a seven-day period.
`The multimedia database and video database may be indi(cid:173)
`vidually stored on a multimedia server and video server,
`respectively. Alternatively, the multimedia server and video
`server can be one in the same server. In either event, the video
`server and the multimedia server are in communication with
`the media analysis/metadata extraction device 130, to receive
`video data and multimedia data, respectively. While central(cid:173)
`izing the video database offers economies of scale and eases
`maintenance, the video data can be stored locally (i.e., prox(cid:173)
`imite a consumer's video device). Additionally, hybrid
`embodiments are possible in which some of the video data is
`stored locally and other video data is stored remotely, perhaps
`in several different geographic locations. Such embodiments
`may be organized such that popular or frequently viewed
`40 video data is stored proximately locally to minimize network
`communication and congestion. Further, intelligent content
`distribution networks can be utilized to efficiently distribute
`the video data from the video database to consumers.
`FIG. 2 illustrates an exemplary network topology for
`45 implementing the invention. At a consumer premises 210, a
`control device 212 communicates via a network access point
`214 over anetwork216. The control device acts as a dynamic
`control pad for initiating video playback of content specific
`information and may be a suitably programmed Personal
`50 Digital Assistant (PDA) device. The control device is capable
`of inputting control command, communicating data, and
`playing multimedia data such as, preferably, still images, text,
`preview videos, or the like. Alternatively, the control device
`may have capabilities limited to textual display and not be
`55 able to play video material. The control device utilizes wire(cid:173)
`less data communications and may employ Bluetooth, IEEE
`802.11 b, infrared protocols, or other wireless protocols, to
`establish communication with the network access point.
`Alternatively, the control device can be a palm-top computer,
`60 a tablet computer, a lap-top computer, a desk top computer, or
`any device capable of display of the multimedia data and
`solicitation of consumer input and generating control queries
`and commands.
`The network access point negotiates data communications
`between the control device and the network. The network
`preferably is an Internet Protocol (IP) network and may have
`bandwidth of 10 Mbps or greater. The control device com-
`
`Page 9 of 14
`
`

`
`US 8,479,238 B2
`
`5
`municates with the network and may establish communica(cid:173)
`tions at a bandwidth lower than that of the network proper, for
`example at a bandwidth under 100 Kbps.
`Additionally, a video device 218 is located at the consumer
`premises 210. The video device can decode compressed digi-
`tal video data and may be a television monitor, computer
`monitor, or the similar device for display video data. The
`video device communicates with the network using compat(cid:173)
`ible protocols, as is known in the art. The video device may
`communicate with the network through a wireless connection 10
`that has sufficient bandwidth to support the playback of com(cid:173)
`pressed digital video data. There may be more than one video
`device that may be controlled by the control device.
`Video server 220 stores video database 140 and is inter- 15
`connected with the network 216. The video server is capable
`of indexing and accessing the video database via methods
`known in the art. Multimedia server 222 stores the multime(cid:173)
`dia database 140 and is interconnected with the network 216.
`The multimedia server is capable of capable of indexing and
`accessing the multimedia database via methods known in the
`art. The multimedia server is also capable of generating con(cid:173)
`tent for remote devices. Together, the video data and the
`multimedia data comprise the content information for utili(cid:173)
`zation by the consumer. During operation of the method of the
`invention, the consumer interacts with the control device and
`multimedia data displayed thereon and observes video data
`displayed on the display device.
`From the consumer premises 210, the control device 212
`communicates over the network 216 with the multimedia 30
`server 222. The multimedia server may be located remotely
`with respect to the consumer premises. In such an instance,
`the network may use Wide Area Network (WAN) or other
`network protocols to route packets between the control device
`and the multimedia server, as is known in the art. Using the
`control device, a consumer is able to selectively retrieve mul(cid:173)
`timedia data located on the multimedia server. More than one
`multimedia server may be located on the network and com(cid:173)
`municated with in the same manner.
`As described above in regard to FIG. 1, the multimedia data
`includes multimedia content data that is content specific data
`that is based on and corresponds to video data stored on a
`video server. Each video program having video data stored on
`a video server will have corresponding multimedia data per(cid:173)
`taining to the content of its corresponding video data stored
`on at least one multimedia server. By viewing the multimedia
`content data for a particular multimedia data, the multimedia
`data is not limited to conventional metadata, such as a title,
`but also includes information pertaining to the content of the
`video data. A consumer may choose to retrieve a particular
`video program having particular video data to the video
`device located at the consumer premises. The multimedia
`data may be indexed and searched in a variety of ways, both
`consumer-defined and/or predefined by indexing software on
`the multimedia server. For example, the consumer may search
`for video programs having a particular actor or for video
`programs located on a particular video server.
`After the user has selected multimedia data in order to view
`the corresponding video data for at least one video program,
`the control device 212 communicates with video server 220 60
`over the network 216 to instruct the video server to retrieve
`and provide the desired video data to the video device 218.
`The video server 220 may be located remotely with respect to
`the consumer premises 210. In such an instance, the network
`216 may use WAN or other network protocols to route packets
`between the control device and the video server, as is known
`in the art.
`
`6
`In other embodiments, the video device may be able to
`download video data or multimedia data and store such data
`locally within the video device, or utilize some other storage
`means at the consumer premises for the same purpose. Fur(cid:173)
`ther, there may be more than one video server 220 located on
`the network 216, and video data for a particular video pro(cid:173)
`gram may be located on more than one video server. The
`video data may be indexed in a variety of ways, as known in
`the art. It is also possible that the multimedia server and the
`video server are one in the same.
`FIG. 3 illustrates additional detail of the exemplary net-
`work topology shown in FIG. 2. The control device 212
`includes a graphic display 310 which displays multimedia
`data, a display controller 312 which controls the consumer
`interface (display of multimedia data and input of consumer
`selections in response thereto), and a control device network
`interface 314 which interfaces the control device to the net(cid:173)
`work. The video device includes a video display 316 which
`20 displays video data, a video decoder 318 which decodes
`compressed video data, and a video device network interface
`320 which interfaces the video display to the network. The
`video server 220 includes a video database 322 which stores
`the video data, a request processing controller 324 which
`25 handles queries and instructions to the video server, and a
`video server network interface 326 which interfaces the video
`server to the network. The multimedia server 222 includes a
`multimedia database 328 which stores the multimedia data, a
`query processing controller 330 which handles queries and
`instructions to the multimedia server, and a multimedia server
`network interface 332 which interfaces the multimedia server
`to the network. The video server network interface 326, the
`multimedia server network interface 332, the control device
`network interface 314 and the video device network interface
`35 320 are connected via the network 216. Communication
`between each of the network interfaces is accomplished as
`described above and via other methods known in the art.
`FIGS. 4-10 are screen views for an exemplary control
`device and disclose one possible implementation of the inven-
`40 tion. The control device provides an interface, such as a
`Graphical User Interface (GUI), for display of information
`and solicitation of consumer input/instruction as explained
`below in conjunction with FIGS. 4-10. For initiation and
`control of video data playback, application software on the
`45 control device generates URLs, which pass parameters to a
`CGI application running under the HTTP server on the video
`device. A CGI syntax of name/value pairs is used for passing
`parameters from the control device to the video device.
`Parameters include MediaURL, VideoDevice, StartTime,
`so Volume, Reply, and Command.
`MediaURL is a URL, URN, or URI indicating the video
`data stream. VideoDevice is an IP address name or number
`indicating the video device that is being directed to display
`the video data. StartTime indicates the video data play posi-
`55 tion in units since the start of the video data. Volume indicates
`volume amplitude on a liner scale. Replay indicates whether
`to send a status message response or issue a no response
`message. The Command parameter may take on one of sev-
`eral values. Those values include play (play the video data at
`the given StartTime), stop (stop the video data playback and
`blank the video display), mute (set volume to zero), volup
`(increase the volume by a predetermined number of units),
`voldown (decrease the volume by a predetermined number of
`units), volume (set the volume using the given Volume param-
`65 eter) and pause (stop the video data playback, freezing the
`display on the current video frame), and other values within
`the skill of one in the art. The control device also queries the
`
`Page 10 of 14
`
`

`
`US 8,479,238 B2
`
`7
`multimedia server and requests action of the video server to
`commend the search for and delivery of information of inter(cid:173)
`est to the consumer.
`FIG. 4 illustrates an exemplary control device screen for
`specifYing the video device that is to be controlled. The con(cid:173)
`trol device provides an input field 410 for t