`Case 2:11-cv—10692-DOC-JPR Document 1-2
`Filed 12/27/11 Page 1 of 60 Page ID #255
`
`EXHIBIT 2
`
`
`
`Case 2:11-cv-10692-DOC-JPR Document 1-2 Filed 12/27/11 Page 2 of 60 Page ID #:56
`Case 2:11-cv—10692-DOC-JPR Document 1-2
`Filed 12/27/11 Page 2 of 60 Page ID #256
`Hllll||||||||ll|||l|||||||.......:||l||||l||||||||1||||||l||||||||l|||llll
`
`USOO5813014A
`
`United States Patent
`
`[19]
`
`[11] Patent Number:
`
`5,813,014
`
`Gustman
`
`[45] Date of Patent:
`
`Sep. 22., 1998
`
`[54] METHOD AND APPARATUS FOR
`MANAGEMENT OF MULTIMEDIA ASSETS
`
`Primary Examiner———Wayne Amsbury
`Assistant Examiner—-Diane D. Mizvahi
`
`[75]
`
`Inventor: Samuel Gustman, Santa Monica, Calif.
`
`Atmmey’ Agent’ or Fi""—Hc°k°r & Hamman
`
`[73] Assignee: Survivors of the Shoah Visual History
`Foundation, Los Angclcs, Calif.
`
`[21] APPL No_; 573,727
`[22] Filed:
`JHL 10, 1996
`
`Int‘ CL6 ““““““““““““““““““““““““““““
`[52] U.S. Cl. . ............................. 707/103; 707/3; 707/103;
`707,104, 707/102. 707/1. 707/4
`[58] Field of Search .................................. 395/615, 234,
`395/285, 806, 701, 601—10, 610-619, 620—29;
`434/500, 308; 707/1: 3, 103, 4, 102, 104
`
`[56]
`
`References Cited
`U.S. PATENT DOCUMENTS
`
`[57]
`
`ABSTRACT
`g
`_
`_
`_
`.
`The mvcntion is a generalized solution for management of
`
`multimedia assets. Generalized interfaces are used between
`a browser component, indexing server, archive server, ter—
`tiary storage manager, and method player components. The
`ggnefafized interface defines a Communication protocol
`.
`.
`.
`can be used by any browser, indexing server, archive server,
`twialy SW26 manage-rs Of method Play“ Component-
`Thus, system components supplied by multiple vendors can
`be interconnected to form a multimedia system that com-
`
`municates using the generalized interfaces of the invention.
`A browser component can accept input such as search
`requests and display multimedia data. An indexing server
`component includes a data catalogue that catalogues multi-
`media data. The catalogue can be used to identify catalogue
`elements associated with data that can satisfy a search
`request. A search mechanism is used to identify the desired
`multimedia data. A search mechanism can create additional
`catalogues that are a subset of the catalogue provided on the
`indexing server. An archive server identifies a tertiary stor-
`age manager that manages the store containing the multi-
`_
`_
`,
`media data. The tertiary storage manager retrieves the cor-
`responding multimedia data from its store and sends it to a
`method play _Which‘ Plays ‘ll;
`in a disp1aY region of the
`browser. The invention retains the content and results of 21
`search such that it is only necessary to perform a search
`once. Search elements are used to store the content of a
`search, i.e., search criteria. A segment element is used to
`store the results of a search.
`
`34 Claims, 24 Drawing Sheets
`
`- 348/552
`
`__ 379/93.01
`345/356
`
`'
`
`3/1993 Graczyk et al.
`8/1993 Reed ..............
`2/1994 Enberg et al.
`2/1994 Glick et al.
`3/1994 Bernstein et all.
`4/1994 MacKay .........
`3/1995 Robison et al.
`707/4
`4/1995 I-'ujisawa
`- 345/335
`5/1995 Bakfif 6‘ 31-
`345/133
`7/1995 Dinwiddie, Jr. et al.
`. 370/352
`7/1995 Bowen et al.
`.......N 707/9
`9/1995 Bergen er 3]. N
`395/20035
`5/1995 M35116, 51- ell 31» -
`707/102
`gfacgictfir
`707/9
`7/1997 Amognini cl ,,1_ N
`7/1997 Ferguson ................................... 707/10
`OTHER PUBLICATIONS
`
`
`
`5,192,999
`5,241,571
`5,283,638
`5,283,819
`5,297,249
`5,307,456
`5,402,499
`5,404,506
`$423,730
`5,434,592
`5,436,398
`5945/1531
`5,530,352
`
`5349435
`5,649,186
`
`Musei Vaticani, Internet, (http://wWW.virt:ual.co.il/orgs/orgs/
`archival/bookintnhmi), pp. 1-10, Dec. 8, 1994.
`The Judaica Archival Project, Internet, (http://Www.virtual-
`.co.il.orgs/archiva/hookintrhtrn), pp. 1-11, Apr. 1996.
`
`
`
`Case 2:11-cv-10692-DOC-JPR Document 1-2 Filed 12/27/11 Page 3 of 60 Page ID #:57
`Case 2:11—cv—10692-DOC-JPR Document 1-2
`Filed 12/27/11 ’ Page 3 of 60 Page ID #257
`
`U.S. Patent
`
`Sep. 22, 1998
`
`Sheet 1 of 24
`
`5,813,014
`
`106
`
`102 ’
`
`104
`
`
`
`Method
`
`Player
`
`110
`
`
`
`Tertiary
`Storage
`Manager
`
`
`
`
`
`hVu11\h\...s»
`
`.uusushusauiwIsulnawxvivvvsiwvvvviiii
`
`113
`
`Archive
`Server
`
`Index
`Server
`
`108
`
`Hewlett Packard
`Hardware Platform
`
`Client and
`Method
`
`Player
`
`
`
`Intermedia
`Server
`
`with
`Archive Server
`
`
`and
`
`
`Tertiary Storage
`Manager
`
`
`
`3
`
`3
`
`Index
`Server
`
`‘
`
`’
`
`116
`
`. . . ¢ . u » - — — « 4 . — . . . . . . . . . . . » . . . . 4 . . . . . 4 . a - n u - . . . a — o - a ¢4 o — - y a - - a - a o a 4 a o - . . . — . at
`
`
`
`Case 2:11-cv-10692-DOC-JPR Document 1-2 Filed 12/27/11 Page 4 of 60 Page ID #:58
`Case 2:11-cv—10692-DOC-JPR Document 1-2
`Filed 12/27/11 L Page 4 of 60 Page ID #258
`
`U.S. Patent
`
`Sep.22,1998
`
`Sheet 2 of 24
`
`5,813,014
`
`120
`
`E
`E
`5
`
`’
`......~.
`
`\( 136
`
`134
`
`\/
`
`Media
`Server
`
`130
`
`Index
`Server
`
`122
`
`Method
`Player
`
`124
`
`
`
`Archive
`Server
`
`1-‘K
`
`
`
`Tertiary
`
`Storage
`Manager
`
`
`
`\—’ _.>\U1:3
`
`%
`
`'°5:'1OOFD§"1
`
`U1
`
`2*:
`
`146 , . . . . . . . . — . - u . . — . . . r . . . . . . . . . . - . . . . . . . . . . . . . . . . . . . . . . -.
`
`
`
`Case 2:11-cv-10692-DOC-JPR Document 1-2 Filed 12/27/11 Page 5 of 60 Page ID #:59
`Case 2:11-cv—10692-DOC-JPR Document 1-2
`Filed 12/27/11 Page 5 of 60 Page ID #259
`
`U.S. Patent
`
`Sep.22, 1998
`
`Sheet 3 of 24
`
`5,813,014
`
`. . . , . . . . . . . . _ . . . . . . . . _ . . V . . . . . . . . . . . . . . . . .. - . . . . . . — . . . . . . . . . . . . . . .. . . . . . . . _ _ . , . . . . . ... . . . . . _ . --
`
`190
`
`182
`
`Method
`Player
`
`\ss1>\s:»Irrr1rIIvIa
`
`.:
`
`178
`
`‘
`X
`5
`1
`'
`'
`5
`-
`’
`3
`I
`1
`'
`'
`7
`I
`‘
`2
`‘
`‘
`‘
`‘
`I
`‘
`‘
`
`a ; a — . ¢ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . » . . . . . . . . . . . . . . . . . . . . . . ..
`3
`:
`1
`:
`I
`I
`1
`1
`.
`7
`I
`:
`1
`I
`r
`E
`1
`E
`4
`I
`I
`I
`I
`:
`4
`
`
`
`
`
`E
`1
`1
`
`2
`:
`.
`5.
`Z
`1
`3
`# . — ¢ . . . . . . . . . . V . . . . . . . . . . . . 4 ¢ . . - » . . a r o 4 v c e : - o u « ¢ u - - . . . . . - - r . . . . : ¢ : tr
`
`
`
`Case 2:11-cv-10692-DOC-JPR Document 1-2 Filed 12/27/11 Page 6 of 60 Page ID #:60
`Case 2:11-cv—10692-DOC-JPR Document 1-2
`Filed 12/27/11 Page 6 of 60 Page ID #260
`
`U.S. Patent
`
`Sep. 22, 1998
`
`Sheet 4 of 24
`
`5,813,014
`
`219
`
`216
`
`1/0
`
`VIDEO AMP
`
`CRT
`
`213
`
`214
`
`CPU
`
`VIDEO MEMORY
`
`MAIN MEMORY
`
`218
`
`210
`
`KEYBOARD
`
`MOUSE
`
`2“
`
`MASS STORAGE
`
`217
`
`215
`
`212
`
`FIG. 2
`
`
`
`Case 2:11-cv-10692-DOC-JPR Document 1-2 Filed 12/27/11 Page 7 of 60 Page ID #:61
`Case 2:11—cv—10692-DOC-JPR Document 1-2
`Filed 12/27/11 Page 7 of 60 Page ID #261
`
`U.S. Patent
`
`Sep.22, 1998
`
`Sheet 5 of 24
`
`5,813,014
`
`313
`
`310
`Browse: Mefhod
`
`Player
`
`314 V /, 312
`
`302
`
`316
`
`Indexing
`Server
`
`Archive
`Server
`
`306
`
`Tertiary
`
`
`
`FIG. 3
`
`
`
`Case 2:11-cv-10692-DOC-JPR Document 1-2 Filed 12/27/11 Page 8 of 60 Page ID #:62
`Case 2:11—cv—10692-DOC-JPR Document 1-2
`Filed 12/27/11 Page 8 of 60 Page ID #262
`
`U.S. Patent
`
`Sep.22,1998
`
`Sheet 6 of 24
`
`5,813,014
`
`403
`
`418
`
`424
`
`414
`Erogosed Person
`
`
`ID
`Name
`
`
`
`
`
`
`
`
`
`
` Ehmic 406
`input data ID
`{D
`
`descriptive phrzge
`
`
`
`In/Out Tmwco 5
`sctfkeywards)
`ty
`
`
`utfinuzges}
`In/OuF:I'I'nIcc0d£S
`setfivasons)
`mmment
`5“(P"’P°5“1 k’33f”""’d-‘)
`sel(proposed pasons)
`
`Segment
`ID
`dgsaipfiut phrase
`Smphmses)
`
`ID
`setflypes)
`
`
`
`
`Person ID
`Person Am-ibute
`
`
`
`
`PIG.4A
`
`
`
`Case 2:11-cv-10692-DOC-JPR Document 1-2 Filed 12/27/11 Page 9 of 60 Page ID #:63
`Case 2:11-cv—10692-DOC-JPR Document 1-2
`Filed 12/27/11 Page 9 of 60 Page ID #263
`
`U.S. Patent
`
`Sep.22,1998
`
`Sheet 7 of 24
`
`5,813,014
`
`412
`
`Thesaural
`
`Keyword
`
`
`
`Case 2:11-cv-10692-DOC-JPR Document 1-2 Filed 12/27/11 Page 10 of 60 Page ID #:64
`Case 2:11—cv—10692-DOC-JPR Document 1-2
`Filed 12/27/11 Page 10 of 60 Page ID #264
`
`U.S. Patent
`
`Se .22 1998
`P
`2
`
`Sheet 8 of 24
`
`5 813 014
`9
`9
`
`435 \
`
`Thesaural
`Keyword One
`
`Thesaural
`Keyword Two
`
`Thesaural
`Keyword Three
`
`Thesaural
`Keyword Four
`
`4923
`492*‘
`
`4938
`493A
`
`496C
`
`4963
`496/4»
`
`4973
`497A
`
`433
`\
`Keyword
`Instance
`one
`
`490D
`49039“:
`490A
`
`4918
`
`491A
`
`49113
`491”
`
`
`
`4943
`494A
`
`4953
`495A
`
`Keyword
`Instance
`Two
`
`FIG. 4C
`
`
`
`Case 2:11-cv-10692-DOC-JPR Document 1-2 Filed 12/27/11 Page 11 of 60 Page ID #:65
`Case 2:11-cv—10692-DOC-JPR Document 1-2
`Filed 12/27/11 Page 11 of 60 Page ID #265
`
`U.S. Patent
`
`Sep. 22,1998
`
`Sheet 9 of 24
`
`5,813,014
`
`313
`
`‘
`
`Browser
`
`512
`
`514
`
`\ Segment
`
`\ Phrase
`
`516
`
`'
`\ Keyword
`
`314
`
`I
`
`185
`
`522
`
`-\ Keyword Hierarchy
`520
`‘
`\ Person
`
`\ Thesaural Keyword
`
`
`
`FIG. 5
`
`
`
`Case 2:11-cv-10692-DOC-JPR Document 1-2 Filed 12/27/11 Page 12 of 60 Page ID #:66
`Case 2:11-cv—10692-DOC-JPR Document 1-2
`Filed 12/27/11 Page 12 of 60 Page ID #266
`
`U.S. Patent
`
`Sep.22,1998
`
`Sheet 10 of 24
`
`5,813,014
`
`602
`
`No—>©
`
`Yes
`
`606
`
`perform search
`
`Yes
`transmit command
`
`
`play command?
`
`FIG. 6%
`
`
`
`Case 2:11-cv-10692-DOC-JPR Document 1-2 Filed 12/27/11 Page 13 of 60 Page ID #:67
`Case 2:11—cv—10692-DOC-JPR Document 1-2
`Filed 12/27/11 Page 13 of 60 Page|D #267
`
`U.S. Patent
`
`Sep.22, 1998
`
`Sheet 11 of 24
`
`5,813,014
`
`
`
`702
`
`704
`
`Yes
`
`search already
`exists?
`
`No—>®
`
`7 segment
`associated with
`search?
`
`
`
`Yes
`
`
`
`
`NoA>®
`
`706
`
`708
`
`retrieve query
`segment
`
`
`
`retrieve phrase ids
`associated with
`
`
`
`query segment
`
`
`
`
`
`FIG. 7A
`
`
`
`Case 2:11-cv-10692-DOC-JPR Document 1-2 Filed 12/27/11 Page 14 of 60 Page ID #:68
`Case 2:11-cv—10692-DOC-JPR Document 1-2
`Filed 12/27/11 Page 14 of 60 Page ID #268
`
`U.S. Patent
`
`Sep. 22, 1998
`
`Sheet 12 of 24
`
`5,813,014
`
`722
`
`728
`
`
`
`Person
`
`background
`in formation
`
`
`
`Search person
`
`’
`
`and/ or keyword
`
`search background
`information
`
`
`
`
`
`
`
`
`phrase id(s)
`identified?
`
`Yes
`
`732
`
`create query
`e1en“r1‘en'ts
`
`'
`
`‘
`
`FIG. 7B .
`
`
`
`Case 2:11-cv-10692-DOC-JPR Document 1-2 Filed 12/27/11 Page 15 of 60 Page ID #:69
`Case 2:11-cv—10692-DOC-JPR Document 1-2
`Filed 12/27/11 Page 15 of 60 Page ID #269
`
`U.S. Patent
`
`Sep. 22, 1998
`
`Sheet 13 of 24
`
`.
`
`5,813,014
`
`
`
`all search elements
`
`processed?
`
`802
`
`Yes
`
`find all subelements
`of elements in set
`
`
`
`add subelements
`t
`lement set
`
`0 e
`
`N0
`
`4
`
`80
`
`
`
`search element
`attribute for
`search element
`
`806
`
`
`
`No
`
`
`
`Yes
`
`
`
`Yes
`
`element in
`set?
`
`
`
`
`
`
`N°
`
`81
`
`4
`
`add element to
`element set
`
`FIG. 8A
`
`
`
`Case 2:11-cv-10692-DOC-JPR Document 1-2 Filed 12/27/11 Page 16 of 60 Page ID #:70
`Case 2:11—cv—10692-DOC-JPR Document 1-2
`Filed 12/27/11 Page 16 of 60 Page ID #270
`
`US. Patent
`
`Sep. 22, 1998
`
`Sheet 14 of 24
`
`5,813,014
`
`822
` Search element
`all elements in
`set processed?
`Yes
`hierarchy?
`
`
`
`
`
`find all child elements for
`elements in
`found element set
`
`
`
`
`
`add child elements to
`, found element set
`
`
`
`
`
`FIG. 8B
`
`834
`
`
`
` 826
`
`N°
`
`324
`
`set next element as
`current search
`element
`
`
`
`Search element
`
`attribute using
`search element
`
`828
`
`No
`
`
`
`830
` element
`instance found
`in set?
`
`
`
`832
`
` add element
`instance to found
`
`element set
`
`
`
`
`
`Case 2:11-cv-10692-DOC-JPR Document 1-2 Filed 12/27/11 Page 17 of 60 Page ID #:71
`Case 2:11—cv—10692-DOC-JPR Document 1-2
`Filed 12/27/11 Page 17 of 60 Page ID #271
`
`U.S. Patent
`
`Sep. 22, 1998
`
`Sheet 15 of 24
`
`5,813,014‘
`
`
`
`
`850
`
`all keywords in
`set processed?
`
`
`
`N°
`
`852
`
`set next keyword as
`current keyword
`
`-
`
`8
`
`54
`
`search phrase
`elements for
`0rd association
`
`ke
`
`Yes
`
`end
`
`856
`
`phrase instance
`found not in phrase
`set?
`
`
`No—§®
`
`
`
`Yes
`
`85
`
`8
`
`add phrase instance
`to phrase set
`
`
`
`FIG. 8C
`
`
`
`Case 2:11-cv-10692-DOC-JPR Document 1-2 Filed 12/27/11 Page 18 of 60 Page ID #:72
`Case 2:11-cv—10692-DOC-JPR Document1-2
`Fi|ed12/27/11
`Page 18 of 60 Page ID #272
`
`Patent
`
`Sep. 22, 1998
`
`Sheet 16 of 24
`
`5,813,014
`
`902
`
`search background information
`using selection criteria to
`identify person instances
`
`
`
`
`
`
`select segment
`instances for each
`erson instance
`
`
`904
`
`906
`
`
`
`create set of phrases
`associated with
`
`selected segments
`
`
`
`
`
`918
`
`Yes
`
`end
`
`all phrases in
`set processed?
`
`No
`
`
`
`9
`
`12
`
`14
`
`9
`
`
`
`916
`
`search phrase's
`keywords for
`s ecified content
`
`
`
`create set of phrases
`having specified
`
`content
`
`
`
`FIG. 9
`
`
`
`Case 2:11-cv-10692-DOC-JPR Document 1-2 Filed 12/27/11 Page 19 of 60 Page ID #:73
`Case 2:11—cv—10692-DOC-JPR Document 1-2
`Filed 12/27/11 Page 19 of 60 Page ID #273
`
`U.S. Patent
`
`Sep. 22, 1998
`
`Sheet 17 of 24
`
`5,813,014
`
`1002
`
`identify segment
`
`
`instance(s) associated
`with person instance(s)
`
`
`1004
`
`identify phrase
`instance(s) associated
`with segment insance(s)
`
`
`
`1 006
`
`
`
`
`
`
`create set of phrases
`associated with
`
`
`
`selected segments
`
`
`
`
`content search?
`
`No—>®
`
`Yes
`
`1010
`
`all phrases in set
`processed?
`
`
`
`
`No
`
`Yes+®
`
`PIG. 10A
`
`
`
`Case 2:11-cv-10692-DOC-JPR Document 1-2 Filed 12/27/11 Page 20 of 60 Page ID #:74
`Case 2:11-cv—10692-DOC-JPR Document 1-2
`Filed 12/27/11 Page 20 of 60 Page ID #274
`
`U.S. Patent
`
`Sep. 22, 1998
`
`Sheet 18 of 24
`
`5,813,014
`
`
`
`get next phrase
`
`1012
`
`1014
`
`
`
` 1016
`
`
`
`
`search phrases
`keywords for
`s ecified content
`
`create set of phrases
`having specified
`content
`
`PIG. 10B
`
`
`
`Case 2:11-cv-10692-DOC-JPR Document 1-2 Filed 12/27/11 Page 21 of 60 Page ID #:75
`Case 2:11-cv—10692-DOC-JPR Document 1-2
`Filed 12/27/11 Page 21 of 60 Page ID #275
`
`U.S. Patent
`
`Sep. 22, 1998
`
`Sheet 19 of 24
`
`5,813,014
`
`PIG. 11A
`
`
`
`Case 2:11-cv-10692-DOC-JPR Document 1-2 Filed 12/27/11 Page 22 of 60 Page ID #:76
`Case 2:11-cv—10692-DOC-JPR Document 1-2
`Filed 12/27/11 Page 22 of 60 Page ID #276
`
`U.S. Patent
`
`Sep. 22, 1998
`
`Sheet 20 of 24
`
`5,813,014
`
`1102
`
`.&’:1£
`ID
`
`1106
`
`§2ue_r|¢ Elam!‘ Record
`
`
`
`Query ID
`
`
`Query Element ID
`ID
`
`
`AndOr
`
`Element Order
`
`1104
`
` t
`
`Query Element ID
`
`PIG.11B
`
`
`
`Case 2:11—cv—10692-DOC-JPR. Document 1-2
`Filed 12/27/11 Page 23 of 60 Page ID #277
`Case 2:11-cv-10692-DOC-JPR Document 1-2 Filed 12/27/11 Page 23 of 60 Page ID #:77
`
`U.S. Patent
`
`Sep. 22, 1998
`
`Sheet 21 of 24
`
`5,813,014
`
`1208
`
`
`
`Storage
`
`flmzsz
` Eluzasc
`input data ID
`inpuf data path
` ID
`ID
`
`
`
`descriptive phrase
`
`In/Out Timecodes
`
`In/Out Timecodes
`
`
`
`
`
`Cache 1
`
`1212
`
`1214
`
`set(Ia:y-words)
`seflimages)
`seflpersons)
`set(proposed keywords)
`set(prapased persons)
`
`
`Key Frame
`
`
`
`
` Cache
`Manager
`
`
`
`FIG. 12
`
`
`
`Case 2:11-cv-10692-DOC-JPR Document 1-2 Filed 12/27/11 Page 24 of 60 Page ID #:78
`Case 2:11-cv—10692-DOC-JPR Document 1-2
`Filed 12/27/11 Page 24 of 60 Page ID #278
`
`U.S. Patent
`
`Sep.22, 1998
`
`Sheet 22 of 24
`
`5,813,014
`
`
`
`1300
`
`
`
`1302
`
`
`
`1304
`
`retrieve
`
`multimedia data
`
`
`
`start thread
`
`transfer data
`to thread
`
`
`
`
`
`thread prepares and
`transfers data to browser
`
`1
`
`end of data?
`
`
`
`1306
`
`
`No..@
`
`
`
`Yes
`
`1310
`
`terminate thread
`
`
`
`FIG. 13A
`
`
`
`Case 2:11-cv-10692-DOC-JPR Document 1-2 Filed 12/27/11 Page 25 of 60 Page ID #:79
`Case 2:11—cv—10692-DOC-JPR Document 1-2
`Filed 12/27/11 Page 25 of 60 Page ID #279
`
`U.S. Patent
`
`Sep.22,1998
`
`Sheet 23 of 24
`
`5,813,014
`
`
`
`play command?
`
`No
`
`
`
`
`
`
`No
`
`process play
`command
`
`PIG. 13B
`
`
`
`Yes
`
`stop command?
`
`Yes—->®
`
`
`
`Case 2:11-cv-10692-DOC-JPR Document 1-2 Filed 12/27/11 Page 26 of 60 Page ID #:80
`Case 2:11-cv—10692-DOC-JPR Document 1-2
`Filed 12/27/11 Page 26 of 60 Page ID #280
`
`U.S. Patent
`
`Sep.22, 1998
`
`Sheet 24 of 24
`
`5,813,014
`
`'
`Browser
`Obie“
`
`1402
`
`SE5
`
`U
`
`1404
`
`M th d
`Piy;
`Object
`
`Us/ \Uses
`
`\
`se
`Ph
`Matfzger
`objec‘,
`
`1414
`
`1416
`
`I Uss
`
`\
`
`Phrase
`Object
`
`1410
`
`1418
`
`U
`C13?
`Object ? 55
`
`Player
`Ob)?“
`
`/
`\ Uses
`
`1412
`
`Phrase
`' 3233522“
`
`Contains —--
`
`PIG. 14
`
`
`
`Case 2:11-cv-10692-DOC-JPR Document 1-2 Filed 12/27/11 Page 27 of 60 Page ID #:81
`Case 2:11-cv—10692-DOC-JPR Document1-2
`Fi|ed12/27/11
`Page 27 of 60 Page ID #:81
`
`5,813,014
`
`1
`METHOD AND APPARATUS FOR
`MANAGEMENT OF MULTHVIEDIA ASSETS
`
`BACKGROUND OF THE INVENTION
`
`1. Field of the Invention
`
`The invention relates to a system of managing multimedia
`assets.
`
`2. Background
`Increasingly, computer systems are being used to present
`multimedia material. Such material is usually in the form of
`text, graphics, video, animation, and sound. Two or more of
`these data types are usually combined to form the multime-
`dia data presented by the computer system. A computer
`system that is used to present multimedia material is called
`a multimedia system.
`A multimedia system should include components that
`allow input, information retrieval, and display. FIG. 1A
`provides an illustration of components for a multimedia
`system. Client 110 includes an interface capability to accept
`user input. The user input can specify criteria for identifying
`multimedia data. Method player 102 converts the media data
`from its storage version to a playback version for use by
`Client 110.
`
`Index server 108, archive server 106, and tertiary storage
`104 provide storage and retrieval capabilities. Index server
`108 is used to store and retrieve text data. Information
`entered in the user interface can be used to retrieve textual
`information stored by index server 108. The user input might
`identify a video segment using a title that is associated with
`the video segment using index server 108, for example. The
`multimedia data is stored on a physical storage device. A
`retrieval mechanism is provided to retrieve data from a
`physical storage device. Archive server 106 locates the
`multimedia data wherever it is stored and interfaces with
`tertiary storage manager 105 to access the media data.
`Current multimedia systems attempt to provide some or
`all of the components described in FIG. 1A. However, the
`components provided by these systems are merged to fonn
`a single component thereby making it impossible to separate
`the merged components into the discrete components
`described in FIG. 1A. This architecture is disadvantageous
`for at least two reasons: 1) there is no ability to replace a less
`capable component with another, more capable component;
`2) it forces each system to run on a single hardware platform.
`Each system provides vendor-specific implementations in
`which at least one of the vendor—specific components used in
`these systems renders the system less viable as a solution for
`multimedia applications. Since, the services are merged into
`a single server, there is no ability to remove a less viable
`component and substitute a more capable component in its
`place. For example, a vendor may olfer a multimedia system
`that includes its premiere DBMS product as an index server
`along with a less capable archive server or client. All of the
`components are intertwined such that it is impossible to
`substitute another archive server or client for the existing
`archive server and client. The following provides a brief
`summary of some existing multimedia systems.
`FIG. 1B illustrates one multimedia system having an
`intermedia server, client and index server combined as a
`single component that runs on one, monolithic hardware
`platform. The system is not portable to another hardware
`platform. It runs on a single hardware platform (i.e., plat-
`form 112, a Hewlett Packard platform). The system includes
`intermedia server 118, index server 116, and client 114.
`
`10
`
`‘I5
`
`20
`
`30
`
`45
`
`50
`
`55
`
`60
`
`2
`Client 114 is used to make a multimedia request to inter-
`media server 118. Intermedia server 118 queries index server
`116 for information to locate the requested data. Intermedia
`server 118 uses the data returned from index server 116 to
`request the data from its archive server and tertiary storage
`management subsystems.
`The archive server and tertiary storage management sub-
`systems are integrated in Intermedia server 118. Client and
`method player 114 and index server 116 are interconnected
`with intermedia server 118 to form a single component. The
`interconnections are hardwired such that it is impossible to
`replace one of the existing components. Another aspect of
`the design of this system results in the inability to substitute
`another component. Client 114 has a buflt-in method player
`functionality. Thus, the system expects the client component
`to include a method player. Yet, most client components that
`are offered by vendors do not include a method player.
`Therefore, it is impossible to use a different client or method
`player component with intermedia server 118.
`A multimedia system having a hub is illustrated in FIG.
`1C. Such a system is provided by Oracle. The hub is
`provided by media server 130. The system runs on a specific
`hardware platform (hardware platform 138, an N-Cube
`hardware platform) and is not portable to other platforms.
`Media server 130 acts as a hub that uses a proprietary
`interface to communicate with the other services. Compo-
`nents with which media server 130 can communicate are
`method player 122, tertiary storage manager 124, and client
`120 (Via lines 134, 132, and 136, respectively). These
`components are merged into a single component thus mak-
`ing it impossible to replace a component. 'lhe component
`formed by media sewer 130, method player 122, tertiary
`storage manager 124, index server 128, archive server 126
`and client—120 must run on a single hardware platform,
`platform 138. Further, while index server 128 is a powerful
`database management system, client 120, archive server
`126, method player 122 and tertiary storage 124 offer weak
`solutions.
`
`tertiary storage
`A system having an index server,
`manager, and archive server is illustrated in FIG. 1D. Such
`a system is provided by IBM. The system oifers index server
`148 (e.g., lBM’s DB2 or Oracle’s DBMS), archive server
`146, and tertiary storage manager 144 in an integrated
`system that runs on mainframe 140. Mainframe 140 is an
`IBM SP2 hardware platform comprised of two RS6000
`machines. Mainframe 140 is comprised of processing units
`144 in which components (e.g., index server 148) can run.
`Tertiary storage 144 is connected to one of processing units
`144. A request for multimedia content from a client is
`processed by index server 148. Index server 148 provides
`selection information that identifies the content which is
`
`retrieved from tertiary storage 144. The system is built to run
`in a mainframe environment using IBM hardware. Further,
`the system does not include a client or method player. Index
`server 148, tertiary storage manager 144 and archive server
`146 are combined as a single component such that it is
`impossible to replace one or more of them.
`A system having a kernel and data blades that interface
`with the kernel is illustrated in FIG. IE. Such a system is
`provided by Informix. As illustrated below,
`the system
`includes kernel 170 that acts as a hub. Kernel 170 is
`supplemented by a plurality of data blades 160A—160H.
`Data blades l60A—l 60H communicate with the kernel using
`a proprietary interface. A data blade is a complex data type
`that is defined using a programming language such as “C”.
`Each data blade (e.g., 160A—160H) is attached to kernel 170
`and integrated into data dictionary 172. Data blades
`
`
`
`Case 2:11-cv-10692-DOC-JPR Document 1-2 Filed 12/27/11 Page 28 of 60 Page ID #:82
`Case 2:11-cv—10692-DOC-JPR Document1-2
`Fi|ed12/27/11
`Page 28 of 60 Page ID #282
`
`5,813,014
`
`3
`160A—160H can then be used over a file system. Kernel 170
`includes a index. Thus, another component must commitm-
`cate with the index via a proprietary interface provided by a
`data blade (e.g., data blades 160A—160H). Data blades
`160H, 160B, and 160D provide a proprietary interface to
`method player 162,
`tertiary storage manager 164, and
`archive server 166, respectively. The components provided
`by this system are merged to form a single component that
`use a proprietary interface to communicate. The component
`combination runs in a single hardware platform 174. Data
`dictionary 172 can become large and cumbersome.
`In
`addition, a fault that occurs in one data blade that is included
`in data dictionary 172 causes a fault for the entire system.
`This system construction is not fault tolerant and is unac-
`ceptable for a production environment.
`A pre—packaged system having interconnected system
`components with hardwired, proprietary interconnections is
`illustrated in FIG. 1F. Such a system is provided by Cine-
`base. System 178 includes a component formed by method
`player 182, tertiary storage manager 184, archive server 186,
`index server 188, and client 190. There is no clear delinea-
`tion between components. Further,
`there are no clearly
`defined lines of communication between the components.
`Component interconnections are hardwired, and it is there-
`fore impossible to substitute components that can commu-
`nicate using the existing connections for the existing com-
`ponents. There is no ability to split the component into
`discrete components such that replacement component can
`be substituted for one of the existing components. Further,
`it
`is impossible to split
`the combined component
`into
`separate components that can run on multiple hardware
`platforms. The combined component olfers a weak solution.
`The archive server is fast and eificient. Several tertiary
`storage managers are also provided. The method players are
`capable. However, the client and index server components
`provide a weak solution.
`None of these systems illustrated in FIGS. 1B—1F provide
`a general cataloguing capability that can catalogue any type
`of multimedia data. In addition, none of the systems pro-
`vided a viable multimedia system solution. That is, none of
`the systems provide viable options for each of the multime-
`dia components identified in FIG. 1A. All of the systems
`merge the components identified in FIG. 1A into a single,
`component that makes it impossible to replace one of the
`components. Further, by combining the components into a
`single component, each system must nm on a single hard-
`ware platform. Further,
`there is no vendor-independent
`interface available to integrate components from different
`vendors to construct a optimum multimedia system.
`A number of prior art patents that describe software
`and/or hardware systems are provided below. These systems
`do not provide a general cataloguing capability or interfaces
`to interconnect the multimedia system components identi-
`fied in FIG. 1A.
`
`A computer system having a television circuit that is
`capable of receiving television signals and forwarding these
`signals to the computer system’s audio and video sub-
`systems is described in U.S. Pat. No. 5,192,999 (Graczyk et
`al.,
`issued on Mar. 9, 19931 U.S. Pat. No. 5,283,638
`(Engberg et al., issued Feb. 1, 1994), and U.S. Pat. No.
`5,283,819 (Glick et al., issued Feb. 1, 1994). Video signals
`are received for displayon the computer system’s monitor.
`Audio signals are output using the computer system’s
`speal<er(s). A remote control device is used to control the
`television circuit within the computer system. A computer
`system having the television circuit can be used in combi-
`nation with a data/fax,/voice modem circuit for telephonic
`transmission of television signals for teleconferencing pur-
`poses.
`
`4
`A service for managing hypertext/hypermedia links is
`described in U.S. Pat. No. 5,297,249, Bernstein et al., issued
`on Mar. 22, 1994. A Link Manager Services (LMS) com-
`ponent provides a universal End User Interface
`including menus and dialog boxes that can be called by an
`application program when a link, or marker, is selected by an
`application user. The LMS accesses a database that contains
`information about markers and their associated “presenters”
`(e.g., application or program). In addition, the LMS includes
`a viewer capability for viewing and maintaining existing
`links and creating new links. The LMS is used at runtime to
`generate menus and dialog boxes and manage the hypertext/
`hypermedia links.
`A system that interconnects audio-video equipment such
`as video tape record or video production switcher using one
`or more “AV LANs” in U.S. Pat. No. 5,307,456, Mackay,
`issued on Apr. 26, 1994. Generic device commands are
`communicated over an AV LAN to a device translator. The
`device translator translates the generic command into a
`device-specific command. The AV LANs are interconnected
`using bridges. The AV LANs can further be used to inter-
`connect workstations, minicomputers, mainframes and per-
`sonal computers. The workstations can be used to display
`information about the resources attached to an AV LAN.
`A controller apparatus for use with a computer system that
`provides programmed volume control of multiple analog
`input signals such as CD-ROM, telephony, MIDI, PCWI,
`FM, and microphone in U.S. Pat. No. 5,402,499, Robison et
`al., issued on Mar. 28, 1995. Additionally, the controller can
`sum multiple inputs and produce multiple audio outputs. The
`controller provides telephony support such that it receives
`voice, DTMF and caller ID signals and sends telephonic
`audio signals.
`A multimedia system that provides a multimedia device
`control program (MMDCP) that interfaces with a multime-
`dia application program (MMAP) and a multimedia presen-
`tation manager (MMPM) in OS/2 in US. Pat. No. 5,428,
`730, Baker et al., issued on Jun. 27, 1995. The MMDCP
`provides a standardized user interface. Because many of the
`control functions are performed by the MMDCP, a MJVIAP
`can be simplified. The MMDCP creates an instance of a
`control panel for use with a multimedia device. A control
`panel instance includes UI widgets (e.g., buttons) that allow
`a user to control the associated device via the control panel
`instance and the device driver.
`
`An expansion unit to implement multimedia capabilities
`on an existing computer system (e.g., IBM PS/2) in U.S. Pat.
`No. 5,434,592, Dinwiddie, Jr. et al., issued on Jul. 18, 1995.
`The expansion unit is connected to the computer system via
`an expansion slot and to the computer system’s monitor. The
`expansion slot controls the video presentation displayed on
`the monitor.
`
`A multimedia network bus (MMNB), multimedia network
`controller (MMN C) and protocol for transmitting isochro—
`nous and packet data between a backbone or wide area
`network and multiple local area network configurations in
`U.S. Pat. No. 5,436,898, Bowen ct al., issued on Jul. 25,
`1995. The MMNC allocates isochronous channels and
`packet channels to users over the MMNB taking into
`account the time delay requirements of various data types
`(e.g., voice, moving images, graphics, or text). The MMNB
`and MMNC is for use with multimedia applications that
`require a varying mix of isochronous data and packet data
`transfers.
`
`SUMMARY OF THE INVENTION
`
`The invention is a generalized solution for management of
`multimedia assets. Generalized interfaces are used between
`
`10
`
`15
`
`17.5
`
`30
`
`45
`
`50
`
`55
`
`60
`
`
`
`Case 2:11-cv-10692-DOC-JPR Document 1-2 Filed 12/27/11 Page 29 of 60 Page ID #:83
`Case 2:11-cv—10692-DOC-JPR Document1-2
`Fi|ed12/27/11
`Page 29 of 60 Page ID #283
`
`1-
`3
`
`5,813,014
`
`a browser component, indexing server, archive server, ter-
`tiary storage manager, and method player components. The
`generalized interface defines a communication protocol that
`can be used by any browser, indexing server, archive server,
`tertiary storage manager, or method player component.
`Thus, system components supplied by multiple vendors can
`be interconnected to form a multimedia system that com-
`municates using the generalized interfaces of the invention.
`The browser component can accept input such as search
`requests and display multimedia data. Search requests are
`processed by the browser. The indexing server component
`includes a data catalogue that catalogues multimedia data.
`The catalogue can be used to identify multimedia data that
`can satisfy a search request entered using the browser
`component’s user interface. Search mechanisms provided by
`the browser send requests to the indexing server to query the
`catalogue and obtain attribute information associated with
`the catalogue. A search mechanism can create additional
`catalogues that are a subset of the catalogue provided on the
`indexing server.
`The indexing server’s multimedia catalogue includes one
`or more catalogue elements which can be a complex mul-
`timedia asset. A complex multimedia asset can consist of one
`or more attribute elements. An attribute element
`is an
`attribute that can have attributes. A catalogue element is
`associated with a portion of multimedia data (e.g., one or
`more frames of video data). The result of a search operation
`performed by the browser identifies a set of catalogue
`elements that can satisfy a search request. Each catalogue
`element has an associated identifier (ID) (e.g., an integer ID)
`that uniquely identifies the catalogue element. A set of IDs
`that represent the set of catalogue elements identified in a
`search operation are sent to the archive server component for
`retrieval of the associated multimedia data.
`The invention retains the content and results of a search
`such that it is only necessary to perform a search once.
`Thereafter, the results of the search can be retrieved without
`performing the search. Search elements a