`
`FILE HISTORY
`US 6,199,060
`
`6,199,060
`PATENT:
`INVENTORS: Gustman, Samuel
`
`TITLE:
`
`Method and apparatus management of
`multimedia assets
`
`APPLICATION
`NO:
`FILED:
`ISSUED:
`
`US199876504A
`
`12 MAY 1998
`06 MAR 2001
`
`COMPILED:
`
`13 OCT 2014
`
`Page 1 of 323
`
`MINDGEEK EXHIBIT 1005
`
`

`

`w4_ 1 _
`-044
`1
`
`ft
`il
`"Ek I fbi
`
`I
`
`IU
`
`-
`
`PATENT NI
`01991
`
`U.S. UTILITY PATENT APPLICATION"
`PATENT DATE
`.IPE
`et
`I4ZJC WR 0 6 001
`
`C - f GA.
`
`SCANED
`
`FILED WITH:
`
`_J DISK (CAF) Ui FICHE
`al lap)
`(AsdschipoAanI n
`
`PREPARED AND APPROVED FOR ISSUE
`
`ORIGINAL
`
`SUBCLASS
`CLASS
`3
`7e-
`INTERNATIONAL CLASSIFICATION
`
`ISSUING CLASSIFICATION
`CROSS REFERENCE(S)
`SUBCLASS (ONE SUBCLASS PER BLOCK)'
`..L...E~ T
`
`CLASS
`201
`-70ft
`
`i
`
`~
`
`___
`
`ElContinued on Issue Blip Inhid.949le, Jacket
`
`riTERMIN!A
`L9DISCLAIMER
`
`DFiAWINGSj,
`Sheets Drwg.
`Figs. *g.
`2jA
`
`ti ll
`Print Fig.
`
`CLAIMS ALLOWED
`Total Ci m's
`Print Claim ford
`
`:7
`
`(date)I2 A Q J± 4
`(A"Ja E(*D
`
`-NOTICE OF ALLO WAN
`
`MAILED
`
`c
`
`i.T
`
`Ea) The termi of this patent
`_
`subsequentO
`has been discilimed.
`Ob) The term of this patent shall
`rot extend beyond toe expiration date
`of U.S O~atent: No.
`
`THOAA G. BLACK
`SUPERVISORY PATENT EXAMiINERISUFE
`GROUP 2700
`
`Amount Due
`
`Date Paid
`
`29
`ISSUE BATCH NUMBER
`
`-
`
`o
`
`______[Pdm
`
`EWMS)
`
`-'_months of
`0] c) The terminal
`this patent.have been disclaimed.
`
`____
`
`WARNING:
`The Informion disclosed herein my be retricted. Unauthorized disclosure may be prohiited by the United States Code Ties. 35. Section
`Pomsusion oftd th U.S. Patenit & Trademnark Offic Is restuicted to aUthouiZed emPloyees And 0ontrkeft)( wily.
`
`122Z 181 and 3M6.
`
`Fo.t PTO431A
`(Rty. IOrt7)
`
`NIUE
`
`MAr-M
`
`Page 2 of 323
`
`MINDGEEK EXHIBIT 1005
`
`

`

`6,199,060
`
`METHOD AND APPARATUS MANAGEMENT OF MULTIMEDIA
`ASSETS
`
`Transaction History
`
`Transaction Description
`Date
`5/12/1998 Preliminary Amendment
`5/12/1998 Information Disclosure Statement (IDS) Filed
`5/12/1998 Information Disclosure Statement (IDS) Filed
`5/14/1998 Initial Exam Team nn
`5/26/1998 IFW Scan & PACR Auto Security Review
`6/5/1998 Application Dispatched from OIPE
`6/15/1998 Case Docketed to Examiner in GAU
`7/8/1998 Non-Final Rejection
`7/13/1998 Mail Non-Final Rejection
`12/18/1998 Response after Non-Final Action
`12/18/1998 Request for Extension of Time - Granted
`12/30/1998 Date Forwarded to Examiner
`1/7/1999 Final Rejection
`1/8/1999 Mail Final Rejection (PTOL - 326)
`7/6/1999 Request for Extension of Time - Granted
`7/6/1999 Notice of Appeal Filed
`7/23/1999 Mail Advisory Action (PTOL - 303)
`7/23/1999 Advisory Action (PTOL-303)
`2/3/2000 Continuing Prosecution Application - Continuation (ACPA)
`2/3/2000 Mail Express Abandonment (During Examination)
`2/3/2000 Express Abandonment (during Examination)
`2/25/2000 Date Forwarded to Examiner
`4/17/2000 Final Rejection
`4/18/2000 Mail Final Rejection (PTOL - 326)
`9/15/2000 Notice of Appeal Filed
`9/15/2000 Response after Final Action
`9/15/2000 Request for Extension of Time - Granted
`9/21/2000 Date Forwarded to Examiner
`9/25/2000 Mail Notice of Allowance
`9/25/2000 Notice of Allowance Data Verification Completed
`11/13/2000 Workflow - File Sent to Contractor
`12/22/2000 Issue Fee Payment Verified
`12/22/2000 Workflow - Drawings Finished
`12/22/2000 Workflow - Drawings Matched with File at Contractor
`12/22/2000 Workflow - Drawings Received at Contractor
`12/22/2000 Workflow - Drawings Sent to Contractor
`1/11/2001 Workflow - Complete WF Records for Drawings
`1/16/2001 Application Is Considered Ready for Issue
`2/15/2001 Issue Notification Mailed
`3/6/2001 Recordation of Patent Grant Mailed
` 
`
`Page 3 of 323
`
`MINDGEEK EXHIBIT 1005
`
`

`

`PATENT APPILICATION
`
`09076504
`CONTENTS
`Dat received
`QIncL r. of M.)
`DateWie
`
`INITIALS
`
`_
`
`___
`
`Dnts received
`QIn0LCQ atM.)
`or
`Date Melted
`
`1.~ APPlCaWin
`2.ffL~
`
`p prt
`
`-
`
`46.
`
`4__
`
`59.
`so.
`
`-'.' 54.
`
`il-55.
`
`-a&
`
`t
`
`62.
`63.
`
`64.
`
`65.
`
`66.
`
`68.
`69.
`70.
`
`71.
`7t
`
`73.
`
`74.
`
`76.
`
`77.
`
`78.
`
`79.
`
`82.
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`(FRONT)
`
`1L.
`16.
`17.
`
`1IL
`19.
`
`20.
`
`21.
`22.
`23.
`24.
`25d
`26.
`27.
`28.
`29.
`30.
`31.
`32.
`33.
`34.
`35.
`36.
`37.
`38.
`39.
`
`41.
`
`-
`
`__
`
`_
`
`.__
`_
`
`_
`
`Page 4 of 323
`
`MINDGEEK EXHIBIT 1005
`
`

`

`SEARCH NOTES
`(INCLUDING SEARCH STRATEGY)
`
`Date
`
`Exmr.
`
`Qv"
`
`INTERFERENCE SEARCHED o
`Exm/
`Date
`Sub.
`Class
`q-2 -o2i
`
`7tr7
`
`I
`
`of
`
`MRAHT MJTRIDE)
`
`Page 5 of 323
`
`MINDGEEK EXHIBIT 1005
`
`

`

`issim 3t,W s'rApl APXA (for k..411ional cross references)
`
`POSITION
`
`INITIALS
`INITIALS
`
`ID NO.
`ID NO.
`
`DATE
`
`I
`
`FEE DETERMINATION
`O.I.P.!. CLASSIFIER
`FORMALITY REVIEW
`
`_________
`
`INDEX OF CL
`
`Allowced
`............................
`(Throghnume Allnceed
`..Restricted
`. ...............
`
`-
`
`klMS
`N .................. Non-elected
`Interference
`.......................
`A ...................... Appeal.
`0a...................... Objected
`
`4,
`
`Claim
`
`'I
`
`/051?
`
`2
`, I
`
`~58
`as.............
`59
`
`Date
`
`Claim
`
`Date:
`
`151
`
`111
`
`Il
`
`I
`
`I
`
`Id
`
`_
`
`10
`
`I
`
`I
`
`I
`
`I'
`
`I
`
`65
`66
`
`1 1 1 1
`1 1 1 1 1
`
`1 1
`
`1
`
`121
`
`1 1 1 1 1
`
`1
`
`1
`1
`1 1 1
`4-1
`
`11
`12N1
`
`1_
`
`1
`
`1_
`1
`
`1
`
`1_
`
`I
`
`121
`
`1 3
`
`121
`
`I
`
`43 1
`
`1
`
`1_
`
`I
`
`I I
`
`34 1
`43
`38
`1
`4AI3
`
`1
`
`40
`
`1 1 1 1 L L-
`
`-
`
`70
`71
`72
`
`75
`78
`77-------------
`78
`
`82
`8s3---------
`
`so
`
`91
`
`93
`94
`
`98
`
`97
`lo-----------
`
`If more than 150 claims or 10 actions
`staple additional sheet here
`
`is CCr WaunIM
`
`Page 6 of 323
`
`MINDGEEK EXHIBIT 1005
`
`

`

`(12) United States Patent
`Gustman
`
`US0061990601
`US 6,199,060 Ri
`*Mar. 6, 2001
`
`(10) Patent No.:
`(45) Date of Patent:
`
`(54) METHOD AND APPARATUS MANAGEMENT
`OF MULTIMEDIA ASSETS
`
`(75)
`
`Inventor:
`
`Samuel Gustman, Santa Monica, CA
`(US)
`
`(73) Assignee: Survivorsof THWShoahMVsual
`History Foundation, Los Angeles, CA
`(US)
`
`Notice:
`
`This patent issued on a continued pros-
`ecution application filed under 37 CFR
`1.53(d), and is subject to the twenty year
`term provisions of 35 U .S.C.
`patent
`154(aX2).
`
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 0 days.
`
`(21) Appl. No.: 09/076,504
`
`(22) Filed:
`
`May 12, 1998
`
`Related U.S. Application Data
`
`(62) Division of application No. 08/678,727, filed on Jul. 10,
`1996, now Pat. No. 5,813,014.
`. . . . . . . . . . . . . . . . . . . . G06F 17130
`It. C1. 7 .................
`(51)
`707/3; 707/1; 707/2; 707/4;
`..................
`(52) U.S. CIl
`715; 707/104; 709/216
`707/104, 1, 2,
`(58) Field of Search .........................
`707/3, 4, 5; 370/338, 395, 367, 452, 231;1
`395/500, 200.7, 38, 49, 79; 345/352; 709/216;
`710f74; 712/2
`
`References Cited
`
`5,192,999
`5,241,671
`
`U.S. PATENT DOCUMENTS
`348/552
`3/1993 Graczyk el t ................
`.................. 707/104
`8/1993 Reed et al
`(List continued on next page.)
`
`OTHER PUBLICATIONS
`
`IEEE publication, "Database interoperation support in mul-
`timedia applications architecture and methodology" by W.
`A. Gray et at., The Institute of Electrical Engineers, Dept. of
`Computer Science, UK, pp. 2/1-2/6. Jan. 1998.*
`
`(List continued on next page.)
`
`Primary Examiner-Thomas G. Black
`Assistant Examiner-Diane D. Mizrahi
`(74) Attorney Agent, or Firm-The Meeker Law Group
`
`(57)
`
`ABSTRACT
`
`The invention is a generalized solution for management of
`multimedia assets. Generalized interfaces are used between
`a browser component, indexing server, archive server, ter-
`tiary storage manager, and method player components. The
`generalized interface defines a communication protocol that
`can be used by any browser, indexing server, archive server,
`tertiary storage manager, or method player component.
`Thus, system components supplied by multiple vendors can
`be interconnected to form a multimedia system that com-
`municates using the generalized interfaces of the invention.
`A browser component can accept input such as search
`requests and display multimedia data. An indexing server
`component includes a data catalogue that catalogues multi-
`media data. The catalogue can be used to identify catalogue
`elements associated with data that can satisfy a search
`request. A search mechanism is used to identify the desired
`multimedia data. A search mechanism can create additional
`catalogues that are a subset of the catalogue provided on the
`indexing server. An archive server identifies a tertiary stor-
`age manager that manages the store containing the multi-
`media data. The tertiary storage manager retrieves the cor-
`responding multimedia data from its store and sends it to a
`method play which plays it in a display region of the
`browser. The invention retains the content and results of a
`search such that it is only necessary to perform a search
`once. Search elements are used to store the content of a
`search, i.e., search criteria. A segment element is used to
`store the results of a search.
`
`18 Claims, 24 Drawing Sheets
`
`Page 7 of 323
`
`MINDGEEK EXHIBIT 1005
`
`

`

`US 6,199,060 B1
`Page 2
`
`U.S. PATENT DOCUMENTS
`
`..................
`348/14
`2/1994 Enberg et a[l
`5,283,638
`2/1994 Glick ct al...................... 379/93
`5,283,819
`3/1994 Bernstein et al ................
`384/14
`5,297,249
`4/1994 MacKay ...................... 345/328
`5,307,456
`381/119
`...............
`3/1995 Robison et a[l
`5,402,499
`707/4
`..................
`4/1995 Fujisawa et a[l
`5,404,506
`Baker ........................
`395/154
`5,428,730 *6/1995
`........ 345/133
`7/1995 Dinwiddie, Jr. et atl
`5,434,592
`370/352
`................
`7/1995 Bowen et a]l
`5,436,898
`9/1995 Bergen et al..................... 707/9
`5,450,581
`709/206
`.............
`6/1996 Meskre, Jr. et at
`5,530,852
`707/102
`................
`3/1997 Dockter et alt
`5,608,900
`707/102
`5/1997 Braden-Harder et at .....
`5,630,121
`................ 707/9
`7/1997 Antognini et at
`5,649,185
`
`5,649,186
`5,754,784*
`5,774,698
`
`-
`
`7/1997 Ferguson ................. 707/10
`5/1998 Garland .............. 395/200.49
`6/1998 01nowich ............... 395/500
`
`OTHER PUBLICATIONS
`IEEE publication, "Content based navigation in a open
`hypermedia environment" by RH. Lewis et at., The Institute
`of Electical Engineers, Department of Electronics and Com-
`puter Science, UK pp. 5/1-5/6, Jan. 1995.*
`Musei Vaticani, Internet, (http://www.virtual.co.iI/orgs/ar-
`cbival./bookintr1rum), pp 1-10, Dec. 8, 1994.
`The Judaica Archival Project, Internet, (http://ww.virtual-
`.co.il/orgs/archivaThookintr.btin), pp 1-11, Apr. 1996.
`* cited by examiner
`
`Page 8 of 323
`
`MINDGEEK EXHIBIT 1005
`
`

`

`U.S. Patent
`
`Mar. 6, 2001
`
`Sheet 1 of 24
`
`US 6,199,060 Bi
`
`, 102
`
`110-
`
`FIG. 1A
`
`1142..................
`....
`Hewlett Packard
`Hardware Platform
`
`IndJ
`
`118
`
`\ intermedia
`
`Tertiary Storage
`Manager
`
`rveCId
`
`FIG.
`
`1B
`
`Page 9 of 323
`
`MINDGEEK EXHIBIT 1005
`
`

`

`U.S. Patent
`
`Mar. 6,2001
`
`Sheet 2 of 24
`
`US 6,199,060 BI
`
`..
`112012
`
`................ ............ ........
`
`ClientPlayer
`
`*<
`
`Media(C:
`Server
`
`124
`
`Srae
`
`132o
`
`130
`
`~-..
`
`2
`
`*
`
`128
`
`.........................
`
`......
`
`138
`
`FIG. 1iC
`
`Mainframe14
`
`144
`
`~sSor5}4
`
`Tertiary
`Storage
`anager
`
`*
`
`150A
`
`148 -C
`
`* indexArchive
`~
`
`14C sererJServer
`
`146
`
`'
`
`FIG. 1D
`
`Page 10 of 323
`
`MINDGEEK EXHIBIT 1005
`
`

`

`U.S. Patent
`
`Mar. 6,2001
`
`Sheet 3 of 24
`
`US 6,199,060 B1
`
`174
`
`162
`
`160H4
`
`160A6
`
`167G
`
`1618F
`
`...............................
`
`FIGageF
`
`Page 11 of 323
`
`MINDGEEK EXHIBIT 1005
`
`

`

`U.S. Patent
`
`Mar. 6, 2001
`
`Sheet 4 of 24
`
`US 6,19060 Bi
`
`216
`
`-:RT
`
`217
`
`I215
`
`FIG. 2
`
`Page 12 of 323
`
`MINDGEEK EXHIBIT 1005
`
`

`

`U.S. Patent
`
`Mar. 6,2001
`
`Sheet 5 of 24
`
`U.S.Patnt
`ar. , 201 heetS o 24US
`6,199,060 BI
`
`FIG. 3
`
`Page 13 of 323
`
`MINDGEEK EXHIBIT 1005
`
`

`

`U.S. Patent
`
`Mar. 6,2001
`
`Sheet 6 of 24
`
`US 6,199,060 BI
`
`Phra
`input data IV
`ID
`
`WIt(keymsrds)
`
`set(perumn)
`set(projewd keymords)
`
`Kamord
`ID
`Wettypas)
`selt hmsural kEpwrds)
`set(pamtfl keyuwrd)
`set(child keywrds)
`
`Thesaural Person
`II,
`
`Nam
`Preferred Flag
`
`ID
`Priwary Nome
`
`Date of Birth
`set('proposed perxmn)
`
`412
`
`PmpsedKewo
`ID
`
`Labdam
`
`426
`
`-Y---,
`
`1
`
`Thesaral KeurT
`Iir
`ey"?d IV
`LAbel
`
`Chamacurktic
`
`422
`
`420
`
`1 1 1)
`Influt Tvuucodes
`
`Perv;n ID
`
`Per"n Attribute
`
`41442
`
`roroedPeo
`IV
`
`FIG. 4A
`
`Page 14 of 323
`
`MINDGEEK EXHIBIT 1005
`
`

`

`U.S. Patent
`
`Mar. 6,2001
`
`Sheet 7 of 24
`
`U.S.Patnt
`6,199,060 Bl
`ar. , 201 heet7 o 24US
`
`410
`
`420
`
`424
`
`FIG. 4B
`
`Page 15 of 323
`
`MINDGEEK EXHIBIT 1005
`
`

`

`U.S. Patent
`
`Mar. 6, 2001
`
`Sheet 8 of 24
`
`US 6,199,060 BI
`
`Thesaural
`Keyword One
`
`Thesaural
`Keyword Two
`
`Thesaural
`Keyword Three
`
`Thesaural
`Keyword Four
`
`488
`
`490D
`
`49
`490A xI
`
`Keyword
`Instance
`One
`
`4930
`493A
`
`494B\\
`494A-\
`
`Keyword
`Instance
`Two
`
`495A
`
`496A
`
`FIG. 4C
`
`Page 16 of 323
`
`MINDGEEK EXHIBIT 1005
`
`

`

`U.S. Patent
`
`Mar. 6,2001
`
`Sheet 9 of 24
`
`US 6,199,060 RI
`
`Data
`
`510 NFile
`
`512 NSegment
`
`514 NPhrase
`
`N1 Keyword
`
`/314
`
`L8 N Keyword Hierarchy
`
`520 N Person
`
`Z2N Thesaural Keyword
`
`524 N\ Type
`
`526 N Type Hlierarchy
`
`Request
`
`504/
`
`FIG. 5
`
`Page 17 of 323
`
`MINDGEEK EXHIBIT 1005
`
`

`

`U. S. Patent
`
`Mar. 6, 2001
`
`Sheet 10 of 24
`
`U.S.Patnt
`6,199,060 Bl
`
`Mr. , 201 Seet 0 o 24US
`
`No--0
`
`FIG. 6
`
`Page 18 of 323
`
`MINDGEEK EXHIBIT 1005
`
`

`

`U.S. Patent
`
`U.S.
`Patent
`Mar. 6, 2001
`
`US 691999060 Bl
`Sheet 11 of 24US61900B
`
`No-<Q
`
`No--O-
`
`FIG. 7A
`
`Page 19 of 323
`
`MINDGEEK EXHIBIT 1005
`
`

`

`U.S. Patent
`
`Mar. 6, 2001
`
`Sheet 12 of 24
`
`US 6,199,060 Ri
`
`background
`information
`
`728
`
`N o
`
`*
`
`FIG. 7B
`
`Page 20 of 323
`
`MINDGEEK EXHIBIT 1005
`
`

`

`U. S. Patent
`
`Mar. 6,2001
`
`Sheet 13 of 24
`
`U.S.Patnt
`6,199,060 Ri
`
`Mr. , 201 Seet 3 o 24US
`
`FIG. 8A
`
`Page 21 of 323
`
`MINDGEEK EXHIBIT 1005
`
`

`

`U.S. Patent
`
`Mar. 6, 2001
`
`Sheet 14 of 24
`
`US 6,199,060 RI
`
`search element
`hierarchy?
`
`834
`
`No-
`
`all child elements
`elements in
`found element set
`
`v
`add child elements to
`found element set
`
`840
`
`end
`
`7
`
`FIG. 8B
`
`Page 22 of 323
`
`MINDGEEK EXHIBIT 1005
`
`

`

`U.S. Patent
`
`U.S. Patent
`Mar. 6, 2001
`
`US 611999060 Bl
`Sheet 15 of 24US61900B
`
`Yes-*:
`
`!
`
`No-1-
`
`FIG. 8C
`
`Page 23 of 323
`
`MINDGEEK EXHIBIT 1005
`
`

`

`U.S. Patent
`
`Mar. 6, 2001
`
`Sheet 16 of 24
`
`US 6,199,060 RI
`
`start _ _90
`
`search background information
`using selection criteria to
`identify person instances
`
`904]
`
`906
`
`select segment
`instances for each
`person instance
`
`4
`create set of phrse
`associated wIhs
`selected segmentsI
`
`al Yhaes
`
`i
`
`ea rc phrase
`
`'s
`
`916
`
`ceaeseho phrases
`
`having specified
`content
`
`n
`
`8
`
`9091
`
`e
`
`9141
`e
`
`FIG.
`
`9
`
`Page 24 of 323
`
`MINDGEEK EXHIBIT 1005
`
`

`

`U.S. Patent
`
`Mar. 6, 2001
`
`Sheet 17 of 24
`
`US 6,199,060 B1
`
`start
`
`1C
`
`identify segment
`instance(s) associated
`with person instance(s)
`
`identify phrase
`instance(s) associated
`with segment insance(s)
`
`create set of phrases
`associated with
`selected segments
`
`< content search?
`
`No-o
`
`Yes 1.
`
`FIG. I A
`
`Page 25 of 323
`
`MINDGEEK EXHIBIT 1005
`
`

`

`U.S. Patent
`
`Mar. 6,2001
`
`Sheet 18 of 24
`
`U.S.Patnt
`6,199,060 BI
`
`Mr. , 201 Seet 8 o 24US
`
`get next phrase
`
`1014
`
`search phrase's
`keywords for
`specified content
`
`QD
`
`create set of phrases
`having specified
`content
`
`1018
`
`A
`
`end
`
`FIG. IJOB
`
`Page 26 of 323
`
`MINDGEEK EXHIBIT 1005
`
`

`

`U.S. Patent
`
`Mar. 6, 2001
`
`Sheet 19 of 24
`
`US 6,199,060 BI
`
`418
`
`FIG. 11A
`
`Page 27 of 323
`
`MINDGEEK EXHIBIT 1005
`
`

`

`U.S. Patent
`
`Mar. 6, 2001
`
`Sheet 20 of 24
`
`US 6,199,060 Bi
`
`owc
`
`1106
`
`OucX, Elemmnt Record
`
`QuerylID
`Query Element ID
`ID
`AndOr
`Element Order
`
`FIG. IIB
`
`Page 28 of 323
`
`MINDGEEK EXHIBIT 1005
`
`

`

`U.S. Patent
`
`Mar. 6, 2001
`
`Sheet 21 of 24
`
`US 6,199,060 Ri
`
`Storage
`
`1208
`
`1212
`
`~1214
`
`IW/Owl Timewodes
`
`Ke-y Fran
`
`FIG. 12
`
`Page 29 of 323
`
`MINDGEEK EXHIBIT 1005
`
`

`

`U.S. Patent
`
`U.S. Patent
`Mar. 6, 2001
`
`US 691"9060 Bl
`Sheet 22 of 24US61,00B
`
`>-~No-*
`
`1310
`
`FIG. 13A
`
`Page 30 of 323
`
`MINDGEEK EXHIBIT 1005
`
`

`

`U.S. Patent
`
`U.S. Patent
`Mar. 6, 2001
`
`US 691999060 Bl
`Sheet 23 of 24Us61900B
`
`FIG. 13B
`
`Page 31 of 323
`
`MINDGEEK EXHIBIT 1005
`
`

`

`U.S. Patent
`
`Mar. 6, 2001
`
`Sheet 24 of 24
`
`US 6,199,060 BI
`
`B140
`
`-Uses
`
`CmXhVd
`Ob)ae
`
`1404
`
`use"
`
`"" 14 D
`
`-Uses
`
`-
`
`(Player\
`
`/ bec
`
`uses
`
`1416
`
`1414
`
`1412
`
`yhrs-.Phae
`
`1410
`
`(Soquencej
`
`-
`
`Contains
`
`-t
`
`bet
`
`FIG. 14
`
`Page 32 of 323
`
`MINDGEEK EXHIBIT 1005
`
`

`

`US 6,199,060 B1
`
`1
`METHOD AND APPARATUS MANAGEMENT
`OF MULTIMEDIA ASSETS
`
`The invention relates to a system of managing multime-
`dia assets. This is a divisional of application Scr. No.
`08/678,727, filed Jul. 10, 1996 and now U.S Pat. No.
`5/813,014 filed Sep. 22, 1998.
`
`BACKGROUND OF THE INVENTION
`
`1. Field of the Invention
`2. Background
`Increasingly, computer systems are being used to present
`multimedia material. Such material is usually in the form of
`text, graphics, video, animation, and sound. 'No or more of
`these data types are usually combined to form the multime-
`dia data presented by the computer system. A computer
`system that is used to present multimedia material is called
`a multimedia system.
`A multimedia system should include components that
`allow input, information retrieval, and display. FIG. 1A
`provides an illustration of components for a multimedia
`system. Client 119 includes an interface capability to accept
`user input. The user input can specify criteria for identifying
`multimedia data. Method player 102 converts the media data
`from its storage version to a playback version for use by
`client 110.
`Index server 108, archive server 196, and tertiary storage
`194 provide storage and retrieval capabilities. Index server
`198 is used to store and retrieve text data. Information
`entered in the user interface can be used to retrieve textual
`information stored by index server 108. The user input might
`identify a video segment using a title that is associated with
`the video segment using index server 108, for example. The
`multimedia data is stored on a physical storage device. A
`retrieval mechanism is provided to retrieve data from a
`physical storage device. Archive server 106 locates the
`multimedia data wherever it is stored and interfaces with
`tertiary storage manager 195 to access the media data.
`Current multimedia systems attempt to provide some or
`all of the components described in FIG. IA. However, the
`components provided by these systems are merged to form
`a single component thereby making it impossible to separate
`into the discrete components
`the merged components
`described in FIG. IA. This architecture is disadvantageous
`for at least two reasons: 1) there is no ability to replace a less
`capable component with another, more capable component;
`2) it forces each system to run on a single hardware platform.
`Each system provides vendor-specific implementations in
`which at least one of the vendor-specific components used in
`these systems renders the system less viable as a solution for
`multimedia applications. Since, the services are merged into
`a single server, there is no ability to remove a less viable
`component and substitute a more capable component in its
`place. For example, a vendor may offer a multimedia system
`that includes its premiere DBMS product as an index server
`along with a less capable archive server or client. All of the
`components are intertwined such that it is impossible to
`substitute another archive server or client for the existing
`archive server and client. The following provides a brief
`summary of some existing multimedia systems.
`FIG. lB illustrates one multimedia system having an
`intermedia server, client and index server combined as a
`single component that runs on one, monolithic hardware
`platform. The system is not portable to another hardware
`platform. It runs on a single hardware platform (i.e., plat-
`
`15
`
`form 112, a Hewlett Packard platform). 'The system includes
`intermedia server 118, index server 116, and client 114.
`Client 114 is used to make a multimedia request to inter-
`media server 118. Intennedia server 118 queries index server
`5116 for information to locate the requested data. Intermedia
`server 118 uses the data returned from index server 116 to
`request the data from its archive server and tertiary storage
`management subsystems.
`The archive server and tertiary storage management sub-
`10 systems are integrated in Intermedia server 118. Client and
`method player 114 and index sewver 116 are interconnected
`with intermedia server 118 to form a single component. The
`interconnections are hardwired such that it is impossible to
`replace one of the existing components. Another aspect of
`the design of this system results in the inability to substitute
`another component. Client 114 has a built-in method player
`functionality. Thus, the system expects the client component
`to include a method player. Yet, most client components that
`are offered by vendors do not include a method player.
`20 Therefore, it is impossible to use a different client or method
`player component with intermedia server 118.
`A multimedia system having a hub is illustrated in FIG.
`1C. Such a system is provided by Oracle. The hub
`is
`provided by media server 130. The system runs on a specific
`25 hardware platform (hardware platform 138, an N-Cube
`hardware platform) and is not portable to other platforms.
`Media sewver 130 acts as a hub that uses a proprietary
`interface to communicate with the other services. Compo-
`nents with which media server 139 can communicate are
`30 method player 122, tertiary storage manager 124, and client
`120 (via lines 134, 132, and 136, respectively). These
`components are merged into a single component thus mak-
`ing it impossible to replace a component. The component
`formed by media server 130, method player 122, tertiary
`35 storage manager 124, index server 128, archive sewver 126
`and client 120 must run on a single hardware platform,
`platform 138. Further, while index sewver 128 is a powerful
`database management system, client 120, archive server
`126, method player 122 and tertiary storage 1.24 offer weak
`40 solutions.
`A system having an index server, tertiary storage
`manager, and archive server is illustrated in FIG. 1D. Such
`a system is provided by IBM. The system offers index server
`148 (e.g., IBM's DB2 or Oracle's DBMS), archive server
`45 146, and tertiary storage manager 144 in an integrated
`system that runs on mainframe 140. Mainframe 149 is an
`IBM SP2 hardware platform comprised of two RS6000
`machines. Mainframe 140 is comprised of processing units
`144 in which components (e.g., index server 148) can run.
`5o Tertiary storage 144 is connected to one of processing units
`144. A request for multimedia content from a client is
`processed by index server 148. Index sewver 148 provides
`selection information that identifies the content which is
`retrieved from tertiary storage 144. The system is built to run
`in a mainframe environment using IBM hardware. Further,
`the system does not include a client or method player. Index
`server 148, tertiary storage manager 144 and archive sewver
`146 are combined as a single component such that it is
`impossible to replace one or more of them.
`A system having a kernel and data blades that interface
`with the kernel is illustrated in FIG. lE. Such a system is
`provided by Informix. As illustrated below, the system
`includes kernel 179 that acts as a hub. Kernel 170
`is
`supplemented by a plurality of data blades 160A-160H.
`65 Data blades 160A-160H1 communicate with the kernel using
`a proprietary interface. A data blade is a complex data type
`that is defined using a programming language such as "C".
`
`55
`
`6o
`
`Page 33 of 323
`
`MINDGEEK EXHIBIT 1005
`
`

`

`US 6,199,060 B1
`
`3
`Each data blade (e.g., 160A-160H) is attached to kernel 170
`and integrated into data dictionary 172. Data blades
`160A-160H- can then be used over a file system. Kernel 170
`includes a index. Thus, another component must communi-
`cate with the index via a proprietary interface provided by a
`data blade (e.g.. data blades 160A-160H1). Data blades
`160H1, 160B, and 160D provide a proprietary interface to
`method player 162, tertiary storage manager 164, and
`archive server 166, respectively. The components provided
`by this system are merged to form a single component that
`use a proprietary interface to communicate. The component
`combination runs in a single hardware platform 174. Data
`large and cumbersome. In
`dictionary 172 can become
`addition, a fault that occurs in one data blade that is included
`in data dictionary 172 causes a fault for the entire system.
`This system construction is not fault tolerant and is unac-
`ceptable for a production environment.
`A pre-packaged system having interconnected system
`components with hardwired, proprietary interconnections is
`illustrated in FIG. 1F. Such a system is provided by Cine-
`base. System 178 includes a component formed by method
`player 182, tertiary storage manager 184, archive sewver 186,
`index sewver 188, and client 190. There is no clear delinea-
`tion between components. Further, there are no clearly
`defined lines of communication between the components.
`Component interconnections are hardwired, and it is there-
`fore impossible to substitute components that can commu-
`nicate using the existing connections for the existing com-
`ponents. There is no ability to split the component into
`discrete components such that replacement component can
`be substituted for one of the existing components. Further,
`into
`to split the combined component
`it is impossible
`separate components that can run on multiple hardware
`platforms. The combined component offers a weak solution.
`The archive server is fast and efficient. Several tertiary
`storage managers are also provided. The method players are
`capable. However, the client and index server components
`provide a weak solution.
`None of these systems illustrated in FIGS. lB-IF provide
`a general cataloguing capability that can catalogue any type
`of multimedia data. In addition, none of the systems pro-
`vided a viable multimedia system solution. That is, none of
`the systems provide viable options for each of the multime-
`dia components identified in FIG. IA. All of the systems
`merge the components identified in FIG. 1A into a single,
`component that makes it impossible to replace one of the
`components. Further, by combining the components into a
`single component, each system must run on a single hard-
`there
`is no vendor-independent
`ware platform. Further,
`interface available to integrate components from different
`vendors to construct a optimum multimedia system.
`A number of prior art patents that describe software
`and/or hardware systems are provided below. These systems
`do not provide a general cataloguing capability or interfaces
`to interconnect the multimedia system components identi-
`fled in FIG. lA.
`A computer system having a television circuit that is
`capable of receiving television signals and forwarding these
`signals to the computer system's audio and video sub-
`systems is described in U.S. Pat. Nos. 5,192,999 (Graczyk et
`al., issued on Mar. 9, 1993). 5,283,638 (Engberg et al.,
`issued Feb. 1, 1994), and 5,283,819(Glick et al., issued Feb.
`1, 1994). Video signals are received for display on the
`computer system's monitor. Audio signals are output using
`the computer system's speaker(s). A remote control device
`is used to control the television circuit within the computer
`system. A computer system having the television circuit can
`
`30
`
`be used in combination with a data/fax/voice modem circuit
`for telephonic transmission of television signals for telecon-
`ferencing purposes.
`A service for managing hypertext/hypermedia links is
`5described in U.S. Pat. No. 5,297,249, Bernstein et al., issued
`on Mar. 22, 1994. A Link Manager Services (LMS) com-
`ponent provides a universal End User Interface (EUI)
`including menus and dialog boxes that can be called by an
`application program when a link, or marker, is selected by an
`10application user. The LMS accesses a database that contains
`information about markers andt their associated "presenters"
`(e.g., application or program). In addition, the LMS includes
`a viewer capability for viewing and maintaining existing
`links and creating new links. The LMS is used at runtimne to
`Sgenerate menus and dialog boxes and manage the hypertext/
`15hypermedia links.
`A system that interconnects audio-video equipment such
`as video tape record or video production switcher using one
`or more "AV LANs" in U.S. Pat. No. 5,307,456, Mackay,
`20 issued on Apr. 26, 1994. Generic device commands are
`communicated over an AV LAN to a device translator. The
`device translator translates the generic command into a
`device-specific command. The AV LANs are interconnected
`using bridges. The AV LANs can further be used to inter-
`25 connect workstations, minicomputers, mainframes and per-
`sonal computers: The workstations can be used to display
`information about the resources attached to an AV LAN.
`A controller apparatus for use with a computer system that
`provides programmed volume control of multiple analog
`input signals such as CD-ROM, telephony, MIDI, PCM,
`FM, and microphone in U.S. Pat. No. 5,402,499, Robison et
`al., issued on Mar. 28, 1995. Additionally, the controller can
`sum multiple inputs and produce multiple audio outputs. The
`controller provides telephony support such that it receives
`35 voice, DTMF and caller ID signals and sends telephonic
`audio signals.
`A multimedia system that provides a multimedia device
`control program (MMDCP) that interfaces with a multime-
`dia application program (MMAP) and a multimedia presen-
`40 tation manager (MMPM) in OS/2 in U.S. Pat. No. 5,428,
`730, Baker et al., issued on Jun. 27, 1995. The MMDCP
`provides a standardized user interface. Because many of the
`control functions are performed by the MMDCP, a MMAP
`can be simplified. The MMDCP creates an instance of a
`45 control panel for use with a multimedia device. A control
`panel instance includes Ul widgets (e.g., buttons) that allow
`a user to control the associated device via the control panel
`instance and the device driver.
`An expansion unit to implement multimedia capabilities
`so on an existing computer system (e.g.. IBM PS/2) in U.S. Pat.
`No. 5,434,592, Dinwiddie, Jr. et al., issued on Jul. 18, 1995.
`The expansion unit is connected to the computer system via
`an expansion slot and to the computer system's monitor. The
`expansion slot controls the video presentation displayed on
`the monitor.
`A multimedia network bus (MMNB), multimedia network
`controller (MMNC) and protocol for transmitting isochro-
`nous and packet data between a backbone or wide area
`network and multiple local area network configurations in
`60 U.S. Pat. No. 5,436,898, Bowen et al., issued on Jul. 2.5,
`1995. The MMNC allocates isochronous channels and
`taking
`into
`to users over the MMNB
`packet channels
`account the time delay requirements of various data types
`(e.g., voice, moving images, graphics, or text). The MMNB
`65 and MMNC is for use with multimedia applications that
`require a varying mix of isochronous data and packet data
`transfers.
`
`55
`
`Page 34 of 323
`
`MINDGEEK EXHIBIT 1005
`
`

`

`US 6,199,060 B1
`
`5
`SUMMARY OF TI-I INVENTION
`The invention is a generalized solution for management of
`multimedia assets. Generalized interfaces are used between
`a browser component, indexing server, archive server, ter-
`tiary storage manager, and method player components. The
`generalized interface defines a communication protocol that
`can be used by any browser, indexing server, archive server,
`tertiary storage manager, or method player component.
`Thus, system components supplied by multiple vendors can
`be interconnected to form a multimedia system that com-
`municates using the generalized interfaces of the invention.
`The browser component can accept input such as search
`requests and display multimedia data. Search requests are
`processed by the browser. The indexing server component
`includes a data catalogue that catalogues multimedia data.
`The catalogue can be used to identify multimedia data that
`can satisfy a search request entered using the browser
`component's user interface. Search mechanisms provided by
`the browser send requests to the indexing server to query the
`catalogue and obtain attribute information associated with
`the catalogue. A search mechanism can create additional
`catalogues that are a subset of the catalogue provided on the
`indexing server.
`Thbe indexing server's multimedia catalogue includes one
`or more catalogue elements which can be a complex mul-
`timedia asset. A complex multimedia asset can consist of one
`or more attribute elements. An attribute element is an
`attribute that can have attributes. A catalogue element is
`associated with a portion of multimedia data (e.g., one or
`more frames of video data). The result of a search operation
`performed by the browser identifies a set of catalogue
`elements that can satisfy a search request. Each catalogue
`element has an associated ID (e.g.. an integer ID) that
`uniquely identifies the catalogue element. A set of IDs that
`represent the set of catalogue elements identified in a search
`operation are sent to the archive server component for
`retrieval of the associated multimedia data.
`The invention retains the content and results of a search
`to perform a search once.
`such that it is only neces