(19)
`
`(12)
`
`Europaisches Patentamt
`
`European Patent Office
`
`Office europeen des brevets
`
`111111111111111111111111111111111111111111111111111111111111111111111111111
`
`(11)
`
`EP 1 100 268 A2
`
`EUROPEAN PATENT APPLICATION
`
`(43) Date of publication:
`16.05.2001 Bulletin 2001/20
`
`(21) Application number: 00124464.9
`
`(22) Date of filing: 08.11.2000
`
`(84) Designated Contracting States:
`AT BE CH CY DE DK ES Fl FR GB GR IE IT LI LU
`MC NL PT SE TR
`Designated Extension States:
`AL LT LV MK RO SI
`
`(30) Priority: 09.11.1999 US 164361 P
`07.04.2000 us 544808
`
`(71) Applicant: Sharp Kabushiki Kaisha
`Osaka (JP)
`
`(72) Inventors:
`• Tomioka, Yoshiaki
`Chiba 274-0801 (JP)
`
`(51) Int Cl.7: H04N 7/16
`
`• Shibao, Tadahide
`Osaka (JP)
`• Yoshikawa, Kohei
`Nara (JP)
`• Van Beek, Petrus
`Vancouver, Washington 98684 (US)
`• Sezan, Muhammed Ibrahim
`Camas, Washington 98607 (US)
`
`(7 4) Representative:
`MULLER & HOFFMANN Patentanwalte
`lnnere Wiener Strasse 17
`81667 Munchen (DE)
`
`(54)
`
`Audivisual information management system
`
`A system, may include, at least one of audio,
`(57)
`image, and a video comprising a plurality of frames. A
`usage preferences description (500), describing prefer(cid:173)
`ences of a user with respect to the use of at least one
`of the audio, image, and video, where the description
`
`normally includes multiple preferences. In one aspect,
`a protection attribute with respect to at least one of the
`preferences indicates whether one of the preferences is
`considered public or private. Other aspects of the user
`preferences description (500) include other attributes,
`alone or in combination.
`
`FIG.2
`
`--16
`38
`..-----------'-~ ,-------------- - ---- - --- ------- - - - - - - - - - --'---,
`:
`42
`50
`52
`:
`AV PROGRAM
`AUDIOVISUAL
`SEARCH AND
`-VIDEO
`PROGRAM
`FILTERING AND
`-AUDIO
`ANALYSIS
`BROWSING
`-STILL IMAGES
`MODULE
`MODULE
`-WEB CONTENTS
`
`DATA
`
`STORAGE
`UNIT
`
`-
`
`DATA AND SERVICES
`
`-PSIP/DVB-SI
`-DTV DATA SERVICES
`-INTERNET SERVICE
`
`N
`<C
`co
`«>
`N
`0
`0
`T"'"
`
`T"'"
`
`a. w
`
`I
`I
`I
`I
`I
`I
`I
`I
`1
`
`DS
`
`PROGRAM
`INFORMATION DS
`FOR PROGRAM
`DS
`
`DESCRIPTION SCHEME(DS)
`GENERATION MODULE
`
`46
`
`Printed by Jouve, 75001 PARIS (FR)
`
`USER/USAGE
`INFORMATION
`FORDS
`
`82
`GRAPHICAL
`USER
`INTERFACE
`DISPLAY
`
`I
`
`80
`DISPLAY
`
`SYSTEM
`:
`INFORMATION
`I
`L--------------------------------------------J
`48
`USER INPUT
`INFORMATION
`
`Apple 1003
`
`

`
`EP 1 100 268 A2
`
`Description
`
`BACKGROUND OF THE INVENTION
`
`5
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`[0001] The present invention relates to a system for managing audiovisual information, and in particular to a system
`for audiovisual information browsing, filtering, searching, archiving, and personalization.
`[0002] Video cassette recorders (VCRs) may record video programs in response to pressing a record button or may
`be programmed to record video programs based on the time of day. However, the viewer must program the VCR based
`on information from a television guide to identify relevant programs to record. After recording, the viewer scans through
`the entire video tape to select relevant portions of the program for viewing using the functionality provided by the VCR,
`such as fast forward and fast reverse. Unfortunately, the searching and viewing is based on a linear search, which may
`require significant time to locate the desired portions of the program(s) and fast forward to the desired portion of the
`tape. In addition, it is time consuming to program the VCR in light of the television guide to record desired programs.
`Also, unless the viewer recognizes the programs from the television guide as desirable it is unlikely that the viewer will
`select such programs to be recorded.
`[0003] RePlayTV and TiVo have developed hard disk based systems that receive, record, and play television broad(cid:173)
`casts in a manner similar to a VCR. The systems may be programmed with the viewer's viewing preferences. The
`systems use a telephone line interface to receive scheduling information similar to that available from a television
`guide. Based upon the system programming and the scheduling information, the system automatically records pro-
`grams that may be of potential interest to the viewer. Unfortunately, viewing the recorded programs occurs in a linear
`manner and may require substantial time. In addition, each system must be programmed for an individual's preference,
`likely in a different manner.
`[0004] Freeman et al., U.S. Patent No. 5,861,881, disclose an interactive computer system where subscribers can
`receive individualized content.
`[0005] With all the aforementioned systems, each individual viewer is required to program the device according to
`his particular viewing preferences. Unfortunately, each different type of device has different capabilities and limitations
`which limit the selections of the viewer. In addition, each device includes a different interface which the viewer may be
`unfamiliar with. Further, if the operator's manual is inadvertently misplaced it may be difficult for the viewer to efficiently
`program the device.
`
`BRIEF SUMMARY OF THE INVENTION
`
`[0006] The present invention overcomes the aforementioned drawbacks of the prior art by providing a method of
`using a system, which may include, at least one of audio, image, and a video comprising a plurality of frames. A usage
`preferences description, describing preferences of a user with respect to the use of at least one of the audio, image,
`and video, where the description normally includes multiple preferences. In one aspect, a protection attribute with
`respect to at least one of the preferences indicates whether one of the preferences is considered public or private.
`Other aspects of the user preferences description include other attributes, alone or in combination.
`[0007] The foregoing and other objectives, features and advantages of the invention will be more readily understood
`upon consideration of the following detailed description of the invention, taken in conjunction with the accompanying
`drawings.
`
`BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS
`
`[0008] FIG. 1 is an exemplary embodiment of a program, a system, and a user, with associated description schemes,
`of an audiovisual system of the present invention.
`[0009]
`FIG. 2 is an exemplary embodiment of the audiovisual system, including an analysis module, of FIG. 1.
`[0010]
`FIG. 3 is an exemplary embodiment of the analysis module of FIG. 2.
`[0011]
`FIG. 4 is an illustration of a thumbnail view (category) for the audiovisual system.
`[0012]
`FIG. 5 is an illustration of a thumbnail view (channel) for the audiovisual system.
`[0013]
`FIG. 6 is an illustration of a text view (channel) for the audiovisual system.
`[0014]
`FIG. 7 is an illustration of a frame view for the audiovisual system.
`[0015]
`FIG. 8 is an illustration of a shot view for the audiovisual system.
`[0016]
`FIG. 9 is an illustration of a key frame view the audiovisual system.
`[0017]
`FIG. 1 O is an illustration of a highlight view for the audiovisual system.
`[0018]
`FIG. 11 is an illustration of an event view for the audiovisual system.
`[0019]
`FIG. 12 is an illustration of a character/object view for the audiovisual system.
`[0020]
`FIG. 13 is an alternative embodiment of a program description scheme including a syntactic structure de-
`
`2
`
`Apple 1003
`
`

`
`EP 1 100 268 A2
`
`scription scheme, a semantic structure description scheme, a visualization description scheme, and a meta information
`description scheme.
`[0021] FIG. 14 is an exemplary embodiment of the visualization description scheme of FIG. 13.
`[0022] FIG. 15 is an exemplary embodiment of the meta information description scheme of FIG. 13.
`[0023] FIG. 16 is an exemplary embodiment of a segment description scheme for the syntactic structure description
`scheme of FIG. 13.
`[0024] FIG. 17 is an exemplary embodiment of a region description scheme for the syntactic structure description
`scheme of FIG. 13.
`[0025] FIG. 18 is an exemplary embodiment of a segment/region relation description scheme for the syntactic struc-
`ture description scheme of FIG. 13.
`[0026] FIG. 19 is an exemplary embodiment of an event description scheme for the semantic structure description
`scheme of FIG. 13.
`[0027] FIG. 20 is an exemplary embodiment of an object description scheme for the semantic structure description
`scheme of FIG. 13.
`[0028] FIG. 21 is an exemplary embodiment of an event/object relation graph description scheme for the syntactic
`structure description scheme of FIG. 13.
`[0029] FIG. 22 is an exemplary embodiment of a user preference description scheme.
`[0030] FIG. 23 is an exemplary embodiment of the interrelationship between a usage history description scheme,
`an agent, and the usage preference description scheme of FIG. 22.
`[0031] FIG. 24 is an exemplary embodiment of the interrelationship between audio and/or video programs together
`with their descriptors, user identification, and the usage preference description scheme of FIG. 22.
`[0032] FIG. 25 is an exemplary embodiment of a usage preference description scheme of FIG. 22.
`[0033] FIG. 26 is an exemplary embodiment of the interrelationship between the usage description schemes and an
`MPEG-7 description schemes.
`[0034] FIG. 27 is an exemplary embodiment of a usage history description scheme of FIG. 22.
`[0035] FIG. 28 is an exemplary system incorporating the user history description scheme.
`[0036] FIG. 29 is an exemplary user preferences description scheme.
`
`DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT
`
`[0037] Many households today have many sources of audio and video information, such as multiple television sets,
`multiple VCR's, a home stereo, a home entertainment center, cable television, satellite television, internet broadcasts,
`world wide web, data services, specialized Internet services, portable radio devices, and a stereo in each of their
`vehicles. For each of these devices, a different interface is normally used to obtain, select, record, and play the video
`and/or audio content. For example, a VCR permits the selection of the recording times but the user has to correlate
`the television guide with the desired recording times. Another example is the user selecting a preferred set of prese(cid:173)
`lected radio stations for his home stereo and also presumably selecting the same set of preselected stations for each
`of the user's vehicles. If another household member desires a different set of preselected stereo selections, the pro(cid:173)
`gramming of each audio device would need to be reprogrammed at substantial inconvenience.
`[0038] The present inventors came to the realization that users of visual information and listeners to audio information,
`such as for example radio, audio tapes, video tapes, movies, and news, desire to be entertained and informed in more
`than merely one uniform manner. In other words, the audiovisual information presented to a particular user should be
`in a format and include content suited to their particular viewing preferences. In addition, the format should be dependent
`on the content of the particular audiovisual information. The amount of information presented to a user or a listener
`should be limited to only the amount of detail desired by the particular user at the particular time. For example with the
`ever increasing demands on the user's time, the user may desire to watch only 1 O minutes of or merely the highlights
`of a basketball game. In addition, the present inventors came to the realization that the necessity of programming
`multiple audio and visual devices with their particular viewing preferences is a burdensome task, especially when
`presented with unfamiliar recording devices when traveling. When traveling, users desire to easily configure unfamiliar
`devices, such as audiovisual devices in a hotel room, with their viewing and listening preferences in a efficient manner.
`[0039] The present inventors came to the further realization that a convenient technique of merely recording the
`desired audio and video information is not sufficient because the presentation of the information should be in a manner
`that is time efficient, especially in light of the limited time frequently available for the presentation of such information.
`In addition, the user should be able to access only that portion of all of the available information that the user is interested
`in, while skipping the remainder of the information.
`[0040] A user is not capable of watching or otherwise listening to the vast potential amount of information available
`through all, or even a small portion of, the sources of audio and video information. In addition, with the increasing
`information potentially available, the user is not likely even aware of the potential content of information that he may
`
`5
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`3
`
`Apple 1003
`
`

`
`EP 1 100 268 A2
`
`be interested in. In light of the vast amount of audio, image, and video information, the present inventors came to the
`realization that a system that records and presents to the user audio and video information based upon the user's prior
`viewing and listening habits, preferences, and personal characteristics, generally referred to as user information, is
`desirable. In addition, the system may present such information based on the capabilities of the system devices. This
`permits the system to record desirable information and to customize itself automatically to the user and/or listener. It
`is to be understood that user, viewer, and/or listener terms may be used interchangeability for any type of content.
`Also, the user information should be portable between and usable by different devices so that other devices may
`likewise be configured automatically to the particular user's preferences upon receiving the viewing information.
`[0041]
`In light of the foregoing realizations and motivations, the present inventors analyzed a typical audio and video
`presentation environment to determine the significant portions of the typical audiovisual environment. First, referring
`to FIG. 1 the video, image, and/or audio information 1 O is provided or otherwise made available to a user and/or a
`(device) system. Second, the video, image, and/or audio information is presented to the user from the system 12
`(device), such as a television set or a radio. Third, the user interacts both with the system (device) 12 to view the
`information 1 O in a desirable manner and has preferences to define which audio, image, and/or video information is
`obtained in accordance with the user information 14. After the proper identification of the different major aspects of an
`audiovisual system the present inventors then realized that information is needed to describe the informational content
`of each portion of the audiovisual system 16.
`[0042] With three portions of the audiovisual presentation system 16 identified, the functionality of each portion is
`identified together with its interrelationship to the other portions. To define the necessary interrelationships, a set of
`description schemes containing data describing each portion is defined. The description schemes include data that is
`auxiliary to the programs 10, the system 12, and the user 14, to store a set of information, ranging from human readable
`text to encoded data, that can be used in enabling browsing, filtering, searching, archiving, and personalization. By
`providing a separate description scheme describing the program(s) 10, the user 14, and the system 12, the three
`portions (program, user, and system) may be combined together to provide an interactivity not previously achievable.
`In addition, different programs 10, different users 14, and different systems 12 may be combined together in any com(cid:173)
`bination, while still maintaining full compatibility and functionality. It is to be understood that the description scheme
`may contain the data itself or include links to the data, as desired.
`[0043] A program description scheme 18 related to the video, still image, and/or audio information 1 O preferably
`includes two sets of information, namely, program views and program profiles. The program views define logical struc-
`tu res of the frames of a video that define how the video frames are potentially to be viewed suitable for efficient browsing.
`For example the program views may contain a set of fields that contain data forthe identification of key frames, segment
`definitions between shots, highlight definitions, video summary definitions, different lengths of highlights, thumbnail
`set of frames, individual shots or scenes, representative frame of the video, grouping of different events, and a close(cid:173)
`up view. The program view descriptions may contain thumbnail, slide, key frame, highlights, and close-up views so
`that users can filter and search not only at the program level but also within a particular program. The description
`scheme also enables users to access information in varying detail amounts by supporting, for example, a key frame
`view as a part of a program view providing multiple levels of summary ranging from coarse to fine. The program profiles
`define distinctive characteristics of the content of the program, such as actors, stars, rating, director, release date, time
`stamps, keyword identification, trigger profile, still profile, event profile, character profile, object profile, color profile,
`texture profile, shape profile, motion profile, and categories. The program profiles are especially suitable to facilitate
`filtering and searching of the audio and video information. The description scheme enables users to have the provision
`of discovering interesting programs that they may be unaware of by providing a user description scheme. The user
`description scheme provides information to a software agent that in turn performs a search and filtering on behalf of
`the user by possibly using the system description scheme and the program description scheme information. It is to be
`understood that in one of the embodiments of the invention merely the program description scheme is included.
`[0044] Program views contained in the program description scheme are a feature that supports a functionality such
`as close-up view. In the close-up view, a certain image object, e.g., a famous basketball player such as Michael Jordan,
`can be viewed up close by playing back a close-up sequence that is separate from the original program. An alternative
`view can be incorporated in a straightforward manner. Character profile on the other hand may contain spatio-temporal
`position and size of a rectangular region around the character of interest. This region can be enlarged by the presen(cid:173)
`tation engine, or the presentation engine may darken outside the region to focus the user's attention to the characters
`spanning a certain number of frames. Information within the program description scheme may contain data about the
`initial size or location of the region, movement of the region from one frame to another, and duration and terms of the
`number of frames featuring the region. The character profile also provides provision for including text annotation and
`audio annotation about the character as well as web page information, and any other suitable information. Such char(cid:173)
`acter profiles may include the audio annotation which is separate from and in addition to the associated audio track of
`the video.
`[0045] The program description scheme may likewise contain similar information regarding audio (such as radio
`
`5
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`4
`
`Apple 1003
`
`

`
`EP 1 100 268 A2
`
`broadcasts) and images (such as analog or digital photographs or a frame of a video).
`[0046] The user description scheme 20 preferably includes the user's personal preferences, and information regard(cid:173)
`ing the user's viewing history such as for example browsing history, filtering history, searching history, and device
`setting history. The user's personal preferences includes information regarding particular programs and categorizations
`of programs that the user prefers to view. The user description scheme may also include personal information about
`the particular user, such as demographic and geographic information, e.g. zip code and age. The explicit definition of
`the particular programs or attributes related thereto permits the system 16 to select those programs from the information
`contained within the available program description schemes 18 that may be of interest to the user. Frequently, the user
`does not desire to learn to program the device nor desire to explicitly program the device. In addition, the user description
`scheme 20 may not be sufficiently robust to include explicit definitions describing all desirable programs for a particular
`user. In such a case, the capability of the user description scheme 20 to adapt to the viewing habits of the user to
`accommodate different viewing characteristics not explicitly provided for or otherwise difficult to describe is useful. In
`such a case, the user description scheme 20 may be augmented or any technique can be used to compare the infor(cid:173)
`mation contained in the user description scheme 20 to the available information contained in the program description
`scheme 18 to make selections. The user description scheme provides a technique for holding user preferences ranging
`from program categories to program views, as well as usage history. User description scheme information is persistent
`but can be updated by the user or by an intelligent software agent on behalf of the user at any arbitrary time. It may
`also be disabled by the user, at any time, if the user decides to do so. In addition, the user description scheme is
`modular and portable so that users can carry or port it from one device to another, such as with a handheld electronic
`device or smart card or transported over a network connecting multiple devices. When user description scheme is
`standardized among different manufacturers or products, user preferences become portable. For example, a user can
`personalize the television receiver in a hotel room permitting users to access information they prefer at any time and
`anywhere. In a sense, the user description scheme is persistent and timeless based. In addition, selected information
`within the program description scheme may be encrypted since at least part of the information may be deemed to be
`private (e.g., demographics). A user description scheme may be associated with an audiovisual program broadcast
`and compared with a particular user's description scheme of the receiver to readily determine whether or not the
`program's intended audience profile matches that of the user. It is to be understood that in one of the embodiments of
`the invention merely the user description scheme is included.
`[0047] The system description scheme 22 preferably manages the individual programs and other data. The man-
`agement may include maintaining lists of programs, categories, channels, users, videos, audio, and images. The man(cid:173)
`agement may include the capabilities of a device for providing the audio, video, and/or images. Such capabilities may
`include, for example, screen size, stereo, AC3, DTS, color, black/white, etc. The management may also include rela(cid:173)
`tionships between any one or more of the user, the audio, and the images in relation to one or more of a program
`description scheme(s) and a user description scheme(s). In a similar manner the management may include relation-
`ships between one or more of the program description scheme(s) and user description scheme(s). It is to be understood
`that in one of the embodiments of the invention merely the system description scheme is included.
`[0048] The descriptors of the program description scheme and the user description scheme should overlap, at least
`partially, so that potential desirability of the program can be determined by comparing descriptors representative of the
`same information. For example, the program and user description scheme may include the same set of categories and
`actors. The program description scheme has no knowledge of the user description scheme, and vice versa, so that
`each description scheme is not dependant on the other for its existence. It is not necessary for the description schemes
`to be fully populated. It is also beneficial not to include the program description scheme with the user description
`scheme because there will likely be thousands of programs with associated description schemes which if combined
`with the user description scheme would result in a unnecessarily large user description scheme. It is desirable to
`maintain the user description scheme small so that it is more readily portable. Accordingly, a system including only the
`program description scheme and the user description scheme would be beneficial.
`[0049] The user description scheme and the system description scheme should include at least partially overlapping
`fields. With overlapping fields the system can capture the desired information, which would otherwise not be recognized
`as desirable. The system description scheme preferably includes a list of users and available programs. Based on the
`master list of available programs, and associated program description scheme, the system can match the desired
`programs. It is also beneficial not to include the system description scheme with the user description scheme because
`there will likely be thousands of programs stored in the system description schemes which if combined with the user
`description scheme would result in a unnecessarily large user description scheme. It is desirable to maintain the user
`description scheme small so that it is more readily portable. For example, the user description scheme may include
`radio station preselected frequencies and/or types of stations, while the system description scheme includes the avail(cid:173)
`able stations for radio stations in particular cities. When traveling to a different city the user description scheme together
`with the system description scheme will permit reprogramming the radio stations. Accordingly, a system including only
`the system description scheme and the user description scheme would be beneficial.
`
`5
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`5
`
`Apple 1003
`
`

`
`EP 1 100 268 A2
`
`[0050] The program description scheme and the system description scheme should include at least partially over(cid:173)
`lapping fields. With the overlapping fields, the system description scheme will be capable of storing the information
`contained within the program description scheme, so that the information is properly indexed. With proper indexing,
`the system is capable of matching such information with the user information, if available, for obtaining and recording
`suitable programs. If the program description scheme and the system description scheme were not overlapping then
`no information would be extracted from the programs and stored. System capabilities specified within the system
`description scheme of a particular viewing system can be correlated with a program description scheme to determine
`the views that can be supported by the viewing system. For instance, if the viewing device is not capable of playing
`back video, its system description scheme may describe its viewing capabilities as limited to keyframe view and slide
`view only. Program description scheme of a particular program and system description scheme of the viewing system
`are utilized to present the appropriate views to the viewing system. Thus, a server of programs serves the appropriate
`views according to a particular viewing system's capabilities, which may be communicated over a network or commu(cid:173)
`nication channel connecting the server with user's viewing device. It is preferred to maintain the program description
`scheme separate from the system description scheme because the content providers repackage the content and de-
`scription schemes in different styles, times, and formats. Preferably, the program description scheme is associated
`with the program, even if displayed at a different time.
`[0051] Accordingly, a system including only the system description scheme and the program description scheme
`would be beneficial.
`[0052] By preferably maintaining the independence of each of the three description schemes while having fields that
`correlate the same information, the programs 10, the users 14, and the system 12 may be interchanged with one another
`while maintaining the functionality of the entire system 16. Referring to FIG. 2, the audio, visual, or audiovisual program
`38, is received by the system 16. The program 38 may originate at any suitable source, such as for example broadcast
`television, cable television, satellite television, digital television, Internet broadcasts, world wide web, digital video discs,
`still images, video cameras, laser discs, magnetic media, computer hard drive, video tape, audio tape, data services,
`radio broadcasts, and microwave communications. The program description stream may originate from any suitable
`source, such as for example PSIP/DVB-SI information in digital television broadcasts, specialized digital television data
`services, specialized Internet services, world wide web, data files, data over the telephone, and memory, such as com(cid:173)
`puter memory. The program, user, and/or system description scheme may be transported over a network (communication
`channel). For example, the system description scheme may be transported to the source to provide the source with
`views or other capabilities that the device is capable of using. In response, the source provides the device with image,
`audio, and/or video content customized or otherwise suitable for the particular device. The system 16 may include any
`device(s) suitable to receive any one or more of such programs 38. An audiovisual program analysis module 42 performs
`an analysis of the received programs 38 to extract and provide program related information (descriptors) to the descrip(cid:173)
`tion scheme (DS) generation module 44. The program related information may be extracted from the data stream in-
`eluding the program 38 or obtained from any other source, such as for example data transferred over a telephone line,
`data already transferred to the system 16 in the past, or data from an associated file. The program related information
`preferably includes data defining both the program views and the program profiles available for the particular program
`38. The analysis module 42 performs an analysis of the programs 38 using information obtained from (i) automatic
`audio-video analysis methods on the basis of low-level features that are extracted from the program(s), (ii) event detec-
`tion techniques, (iii) data that is available (or extractable) from data sources or electronic program guides (EPGs, DVB(cid:173)
`SI, and PSIP), and (iv) user information obtained from the user description scheme 20 to provide data defining the
`program description scheme.
`[0053] The selection of a particular program analysis technique depends on the amount of readily available data and
`the user preferences. For example, if a user prefers to watch a 5 minute video highlight of a particular program, such
`as a basketball game, the analysis module 42 may invoke a knowledge based system 90 (FIG. 3) to determine the
`highlights that form the best 5 minute summary. The knowledge based system 90 may invoke a commercial filter 92
`to remove commercials and a slow motion detector 54 to assist in creating the video summary. The analysis module
`42 may also invoke other modules to bring information together (e.g., textual information) to author particular program
`views. For example, if the program 38 is a home video where there is no further information available then the analysis
`module 42 may create a key-frame summary by identifying keyframes of a multi-level summary and passing the infor(cid:173)
`mation to be used to generate the program views, and in particular a key frame view, to the description scheme.
`Referring also to FIG. 3, the analysis module 42 may also include other sub-modules, such as for example, a de-mux/
`decoder 60, a data and service content analyzer 62, a text processing and text summary generator 64, a close caption
`analyzer 66, a title frame generator 68, an analysis manager 70, an audiovisual analysis and feature extractor 72, an
`event det