United States Patent 1191
`Liaguno et a].
`
`
`
`1 llllllllllllllllllllllllllllllllllllllllllllllllllllllll
`US005729 741A
`[11] Patent Number:
`[45] Date of Patent:
`
`5,729,741
`Mar. 17, 1998
`
`. [54] SYSTEM FOR STORAGE AND RETRIEVAL
`OF DIVERSE TYPES OF INFORMATION
`OBTAlNED FROM DIFFERENT MEDIA
`SOURCES WHICH INCLUDES VIDEO’
`AUDIO, AND TEXT TRANSCRIPTIONS
`
`[75]
`
`[73]
`
`[2 1T
`[22]
`[5 1]
`[52]
`
`[5 3]
`
`[56]
`
`Inventors: Anthony Shawn Liaguno, Melbourne
`Beach; Andrew Frank Connor,
`Melbourne’ both of Fla‘
`.
`_
`.
`Assignee. glden Enterprises, Inc., Melbourne,
`'
`
`APPl- N°~= 419,152
`Filed:
`APR 10’ 1995
`G06F 17/30
`rm. C16
`Us. (:1. ....................... .. 395/615; 3950.79; 395/603;
`395/606; 395/612; 395/617; 364/551.01;
`364/2821
`Field Of Search ..................................... 364/900, 419,
`364/518, 283.2, 514 R, 401. 551.01; 395/600.
`147, 200. 615, 603. 154. 106. 508. 605,
`208, 796. 761, 604, 2.79,. 612. 617'; 341/67;
`382/36‘ 61‘ 317’ 175’ 340/995’ 434/118
`
`Refemnces Cited
`U5, PATENT DOCUMENTS
`
`men“ a a]
`233%:
`339955/25g
`2
`‘$541531 11/1985 Re?zvugm
`395/26
`4,566I065
`1/1986 TO?l
`386/64
`4,635,136
`1/1987 Ciampa a a1.
`395/307
`4,695,975
`9/1987 Bedn'j
`382/178
`4,741,045
`4/1988 Denning ............ ..
`395/796
`4,858,170
`8/1989 DeWick, Sr. et a1.
`395/203
`4,924,337
`5/1990 Jeppesen ------------ -
`395/615
`4,941,125
`7/1990 Boyne ................ ..
`33g’,
`i’gg’gzg 1(1)’
`$99M 3; a1‘
`“(£995
`5’0l6’o07
`511991 “3:23:31 '
`. 341/67
`5:016:00; 5,1991 whiu-ng e,
`.. 343/159
`5,019,905
`511991 Pshtissky et a1.
`..... .. 395/201
`5,099,422
`3/1992 Foresman et al.
`5,129,061
`7/1992 Wang et a]. .......................... .. 395/118
`
`5,159,667 10/1992 Boney et a1. .
`5,172,281 12/1992 Ardis et a1.
`5,201,011
`4/1993 Bloomberg et a].
`5,218,673
`6/1993 Fujiwara
`5,235,654 8/1993 Anderson et a1.
`
`395/761
`360/722
`38%175
`.... .. 395/508
`387J180
`
`(List continued on next page.)
`
`("HER PUBLICATIONS
`Miale. Nicole “Computer Talk 101: Software-Based
`Text-To-Speech Gives Developers New Options”, Telecon
`acct‘ v12, n5. p720). May 1994'
`Banet, Bernard “Speech to Text: Dicatation Systems from
`IBM, Dragon Systems”. The Seybold Report on Desktop
`Publishing, v8, n7, 153(9), Mar. 7. 1994.
`Eliot. Lance B. “In Very Good Form", Al Expert, v9, n7,
`p96)’ M 1994'
`(List °°“"'““°d °" “ext P1149)
`prim,"y Examiner_1homas G. Black
`Assistant Examiner-Charles Leon Rones
`Attorney, Agent, or Firm-Charms E_ Wands
`
`ABSTRACT
`[571
`An integrated media image information storage and retrieval
`system processes information supplied by di?erent types of
`media. A processor-based network server operates as a
`system interface between one or more user control terminals,
`a media image capture station through which media image
`input/output devices are coupled to the network server, and
`a memory for storing media image ?les to be retrieved for
`reproduction. A supervisory media image manipulation and
`“11ml P‘ W is access“ thm‘lgh a ‘"Pmism? EmPhical
`“SCI 91m“? at any [186$ °°l1"°1t?fmina1;and has emba
`ded subordinate mcdla Image mmplll??o? Programs for
`different types of media and information formats- When
`using the interface to import information from an arbitrary
`medium, the user is able to generate a ?rst, index storage ?le,
`and a supplcmnml text dcsc?ption_bascd ?le. so as to
`facilitate rapid retrieval of any type of data. regardless of its
`original format (e.g. text, picture, text-picture combination.
`video, audio) and regardless of the capture medium or
`50"“ fwm which it is imPO?ed in") the Symm
`
`12 Claims, 3 Drawing Sheets
`
`{-201
`MPORT TEXT
`IMAGE
`
`r203
`
`i
`FORMAT 15x1
`INTO TEXT FILE
`
`r211
`IMPORT VOlCE SPEECH
`AS DIGTTIZE BASE
`WAGE ¥¥LE
`
`f221
`IMPORT GENERAL
`IMAGE AS BASE
`IMAGE FILE
`[213
`l
`WES/?gs"
`PERFORM
`CONVERSION
`TEXT CONVERSION
`______l
`
`r223
`
`I
`
`TEXT FILE
`
`Facebook, Inc. - Inter Partes Review
`EXHIBIT 1013
`
`001
`
`

`

`us. PATENT DOCUMENTS
`
`8/1993 R?edml
`52411671
`4/1994 Manson eta].
`5,303,148
`5,305,396 411994 13665 61 a1.
`5,350,303
`9/1994 Fox et a1.
`5,369,704 11/1994 Bennett et 21.
`
`
`
`5,388,197 5,404,295
`
`
`
`2/1995 Rayner 4/1995 Katz et a1. .
`
`
`
`
`
`4/1995 Rosenbanm 5,404,435 5,414,644 5/1995 Seaman et 21.
`
`
`
`5,418,948 5,434,933
`
`
`
`5/1995 Turtle 7/1995 Karnin et a1. ..
`
`
`
`5,444,615 5,467,288 11/1995 Fasciano et a]. 8/1995 Bennett et a1.
`5,493,677
`2/1996 B61691 61 a1.
`5,500,920
`3/1996 Kupjec
`5,511,148
`4/1996 Wellner
`
`5,729,741
`Page 2
`
`.
`
`395/615
`128/660.01
`.. 387J175
`434/113
`380/9
`.. 395/328
`.. 395/602
`.. 395/777
`364/551-0l
`395/604
`" 382/317
`T564151 4 R
`395/615
`, 395/239
`395/106
`
`395/615
`5,517,605 511996 Wolf
`395/106
`5,523,732 6/1996 K1612, Jr.
`5,544,352 8/1996 Egger .................................... .. 395/605
`
`OTHER PUBLICATIONS
`Lane, Alex “Store-Bought Recognition Tools", v9, n10,
`911(3), Oct 1994
`Dyson et a1. ‘Macworld, Part II: Image Processing. Scan
`ners, Printers, Utilities, etc", The Seybold Report on Desk
`3615 Publishing, v9, n1. 1527(7), Sep. 12, 1994.
`Chen et a]. “Word Spotn'ng in Scanned Images Using
`Hidden Markov Models", ICASSP '93: Acoustics Speech
`and Signal Processing Conference, p.v1-v4, Apr. 1993.
`L°Pf¢5?~ Daniel P- “Robust Rem'eval of Noisy T¢X”~ ADI
`‘96 Forum on Research and Technology, p.76-85, Apr.
`1996.
`
`Facebook, Inc. - Inter Partes Review
`EXHIBIT 1013
`
`002
`
`

`

`US. Patent
`
`Mar. 17, 1998
`
`Sheet 1 of 3
`
`5,729,741
`
`(37
`MULTI
`
`GIGABYTE <—> RAID
`
`r10
`
`‘Jr-r : ;
`
`r41
`
`NETWORK
`sEvER
`
`
`
`'
`I
`:
`E
`:
`:
`:
`OBJECT
`1"‘
`T
`'
`SERVER
`T‘
`i
`;'
`:
`E
`:
`g
`1
`"i333: :IIITIIIZ'J
`
`T
`
`1
`
`K33
`(31
`PRINTER
`-———>
`<—>TEEEPN0NE uNE
`k
`35
`
`r23
`r21
`v|Ew/ *5; SCANNER
`scAN
`CAPTURE '
`AUD'O
`5W0" - vmEo CAMERA
`|
`“7
`
`OTHER INPUT
`DEVICE
`k29
`114
`8
`VIEW
`sTAT|0N
`
`REMOTE ACCESS
`
`(13
`
`r12
`
`14
`V I
`VIEW
`STATION
`
`114
`l
`v|Ew
`sTATToN
`
`FIG. 1
`
`r201
`
`Mime?“
`{T
`r203
`FORMAT TExT
`INTO TExT FILE
`
`(211
`IMPORT VOICE SPEECH
`As DIGITIZE BASE
`IMAGE ENE
`l
`(213
`vo'gg/Tsg’gw
`CONVERS‘ON
`
`r221
`IMPORT GENERAL
`IMAGE As BASE
`IMAGE FILE
`J,
`[223
`PERToRN
`A TExT CONVERSION
`
`—iTT—-—
`
`TEXT FILE
`FIG. 2
`
`Facebook, Inc. - Inter Partes Review
`EXHIBIT 1013
`
`003
`
`

`

`US. Patent
`
`Mar. 17, 1998
`
`Sheet 2 of 3
`
`5,729,741
`
`r301
`FOLDER 1 r 303
`TOPIC
`
`r 305
`1 DESCRIPTION
`
`(FROM FIG.2)
`k511
`
`OPERATOR
`Hm
`FIG. 3
`
`Ls07
`
`401
`
`162 l—
`ml
`407
`\ WH W12 W13 Wi_4 Wi5 VES Wi7 VQB Wi9 W110 WW
`
`Facebook, Inc. - Inter Partes Review
`EXHIBIT 1013
`
`004
`
`

`

`US. Patent
`
`Mar. 17, 1998
`
`Sheet 3 of 3
`
`5,729,741
`
`r505
`r501
`VOICE-TEXT —| CORRELATE
`VOTCE-TEXT
`WlTH
`TRANSCRIBED
`_> TExT
`
`.
`
`TRANSCRIBED
`TExT
`k503
`
`r507
`INSERT VIDEO
`TIME LINE
`——> INDICES INTO
`TRANSCRIBED
`TExT FILE
`
`FIG. 5
`
`GENERATE N
`lNDEX ETLE
`6°‘
`
`1
`
`HASHING
`OPERATOR "T503
`
`l
`
`HASHED TExT ,\
`FILE
`505
`
`l
`
`STORE HASHED K607
`CODE IN INDEX
`SEARCH FILE
`
`FIG. 6
`
`Facebook, Inc. - Inter Partes Review
`EXHIBIT 1013
`
`005
`
`

`

`5,729,741
`
`1
`SYSTEM FOR STORAGE AND RETRIEVAL
`OF DIVERSE TYPES OF INFORMATION
`OBTAINED FROM DIFFERENT MEDIA
`SOURCES WHICH INCLUDES VIDEO,
`AUDIO, AND TEXT TRANSCRIPTIONS
`
`FIELD OF THE INVENTION
`The present invention relates in general to information
`storage and retrieval systems, and is particularly directed to
`a new and improved system for e?ciently storing multiple
`types of media image information, including but not limited
`to text, still images, animation, graphics, video, and audio,
`derivable from a variety of media image sources, such as
`computer data base ?les, hard copy print media,
`photographs, audio cassettes, video camera, etc., and also
`rapidly accessing any piece of media image information for
`reproduction on an output device, such as a user display
`terminal or printer.
`
`BACKGROUND OF THE INVENTION
`Continuing improvements in information gathering and
`processing technology have made it possible for industries
`and professions to avail themselves of a variety of media and
`associated storage and reproduction equipment. For
`example, in the legal profession, access to information
`stored on multiple and diverse types of media is crucial to
`research, the generation and storage of documents. and the
`gathering and evaluation of evidence. Also, success at trial
`often depends upon the ability of the presenting attorney to
`quickly locate and reproduce a critical piece of evidence,
`from what is typically a diverse collection of very large
`quantities of material, including, but not limited to, hard
`copy (paper documents, such as contracts, receipts. letters,
`manuscripts, etc), photographs, audio and video storage
`media, and computer-accessible storage media.
`In order to facilitate this information accessing task, one
`or more electronic information storage and retrieval devices,
`such as document scanners, opto-electronic image digitizers,
`large screen displays and the like, that allow substantially
`any piece of information, regardless of its original physical
`characteristics, to be stored and retrieved in an e?icient and
`organized manner, have become commonplace pieces of
`courtroom equipment. However, because the format of the
`information stored in one type of database for playback by
`an associated reproduction device is not necessarily com
`patible with the format used by ano?rer data base and its
`associated playback device, accessing different pieces of
`information for presentation to a viewer currently requires
`the use of a number of separate, stand alone equipments,
`each of which has its own control software.
`For example, the format of a text database ?le, such as
`that of a contract or will, derived from an opto-electronic
`scanner, is not customarily compatible with the format of a
`still image or dynamic image database ?le, such as one
`derived from a digitized photograph, computer graphics
`image animation, or digitized imagery data frame from a
`video camera. As a consequence, the process of retrieving
`diversely formatted electronically stored information is a
`cumbersome and time-consuming one, requiring the use one
`or more separate software packages for each media and
`information type in the course of operating the appropriate
`storage and retrieval device, to enable the information to be
`accessed and reproduced.
`This problem is exacerbated when the imagery informa
`tion of interest has been captured on video tape, since
`locating a given scene or image clip on video tape often
`
`2
`entails a repetitive series of fast forward, look, and rewind
`operations of a video cassette recorder (VCR). Even when
`equipped with a mechanism for providing a time line index
`of respective frames scenes on video tape, there is still the
`need to wait while the VCR physically transports the tape
`from one clip location to another.
`One way to solve this problem is to transfer the imagery
`information stored on video tape to a faster access, mass
`storage medium, such as a laser platter or disc (CD-ROM).
`However, because a laser recording medium is a write once
`storage device, then whenever it is desired to modify or
`update any of the stored information, it is necessary to both
`‘burn’ a new laser disc or platter and also enter new
`parameter data employed in the access and playback control
`software of the reproduction device, since the location where
`the information was stored on the previous medium has
`changed on the new medium as the result of the update.
`Thus, even though there exist various storage media and
`access devices for autonomously electronically storing and
`reproducing multiple forms of media image information, to
`date there has been no single or unitary system for integrat
`ing diverse pieces of information sourcing, storage and
`playback equipment and in a manner that allows any piece
`of information, irrespective of its original format and
`medium in which it is supplied to the user, to be stored in a
`manner that allows it to be expeditiously located in a storage
`database, retrieved and played back on an image reproduc
`tion device.
`
`SUMMARY OF THE INVENTION
`In accordance with the present invention, the above
`problem is successfully addressed by a new and improved
`information storage and retrieval system, which employs a
`graphical user interface through which data obtained from
`diverse types of media image generation equipments is
`converted to both a user-de?ned storage index format and a
`supplemental text description-based format, that facilitates
`rapid retrieval of any type of data, regardless of its original
`format (e.g. text, picture. text-picture combination, video,
`audio) and regardless of the capture medium or source from
`which it is imported into the system.
`The overall system architecture of the storage and
`retrieval system of the present invention includes a
`processor-based network server, which operates as a system
`interface between one or more user control terminals, a
`variety of media image inputloutput devices and an atten
`dant mass media image store. The fundamental media image
`input/output control mechanism of the system is performed
`through one or more local view stations, through which
`users of the system may control the operation of the network
`server, for storing and/or retrieving media images to be
`displayed, exported or printed.
`Each view station may comprise a processor-based video
`display terminal having a keyboard/mouse arrangement, and
`an associated display device, and interfaces with the network
`server via a local area network bus. The operational control
`program for each media image view station includes a
`supervisory media image manipulation and control program,
`into which subordinate media image manipulation programs
`for use with various media and media image formats are
`embedded, so that a view station user may operate a variety
`of media image processing programs through a single,
`supervisory graphical user interface. Such subordinate
`embedded programs include a media image ?le annotation
`program that allows the user to mark-up/edit documents, an
`image indexing program for indexing a media image and
`
`45
`
`55
`
`65
`
`Facebook, Inc. - Inter Partes Review
`EXHIBIT 1013
`
`006
`
`

`

`5,729,741
`
`10
`
`20
`
`25
`
`30
`
`35
`
`3
`performing a free text search. and a multiple object ?le
`viewer for importing images having diiferent formats from
`various media image sources.
`In order to import and store media images from different
`media, the network server is coupled to a processor-based
`view/scan/capture station, which is interfaced with a plural
`ity of information sources, such as a digitizing document
`scanner. an audio input unit, a video camera and an auxiliary
`input device. Like the other media image processing
`software, each of the media image capture programs
`employed by the view/scan/capture station is accessible
`through the supervisory program of the user view stations.
`The audio input may be derived from a microphone
`associated with video camera and is processed by a voice
`recognition and text conversion correlation operator to pro
`vide a search and identi?cation mechanism for rapidly
`locating image clips in video data generated by the video
`camera. When voice from a video camera microphone is
`imported, the voice/speech signals are processed through a
`voice recognition-to-text translation routine, so as to gener
`ate a ‘voice-text’ ?le.
`The network server is further coupled to a print/fax
`station, which is interfaced with a printer and an external
`communications link (telephone line) through which
`database-sourced media image ?les may be imported to and
`exported from the network server. Imported video data is
`stored in an attendant mass data store, such as a multi
`gigabyte memory. The network server is further coupled to
`an object server, which controls access to all the other media
`type on the system.
`Irrespective of their type or original data format. all media
`inputs to the system are processed by means of a text
`generator in order to create a ‘text’ ?le for that media image.
`The generation of a text version of each media image
`enables the use of a free text search operator to locate any
`media image ?le. The free text search mechanism program
`may comprise an indexing and text search program. through
`which a user at a view station may perform a free text search
`using Boulean and fuzzy logic parameters. By virtue of the
`fact that each stored media image ?le. regardless of its
`original format. is processed to generate an accompanying
`text ?le. then in the event the view station operator lacks a
`priori knowledge of an index search folder and its attendant
`description ?elds. He may still be able to retrieve the media
`image using the free text operator, since the text ?le and base
`?le are stored together in association with the index search
`folder for the media image of interest.
`Since any text-containing document may include. in
`addition. to standard typed or printed text. other ‘text’-type
`marks. such as a date stamp. signatures. hand notations etc..
`any scanned document scanned is processed through an
`optical character recognition operator program. so that the
`resulting text ?le for the media image of interest will include
`not only standard typed or printed text. but all text-type
`markings on the document being scanned. For general media
`image inputs. such as a photograph, the output of a video
`camera. or voice input. precursor text-detection processing
`of the original information signals is also performed to
`produce a ‘text’ image ?le, and thereby allow use of a free
`text search mechanism.
`For speech/voice signals. such as those provided by a
`microphone associated with the video camera. in addition to
`generating a base media image ?le that corresponds to a
`digitized representation of the original voice signals, the
`voice signals are also processed through a precursor voice
`recognition-to-text translation routine. so as to produce an
`
`50
`
`SS
`
`65
`
`4
`auxiliary base ‘text image’ ?le, similar to that obtained by
`the high speed scanner.
`For general, non-text image media, such as photographs,
`the original media image scene may or may not contain a
`text object that can be detected by processing the media
`image through a character recognition routine. To determine
`whether an original media image contains any text, the base
`media image derived from the image-containing medium is
`subjected to a an optical character recognition text conver
`sion operator. This precursor text conversion operator exam
`ines the contents of the base media image ?le, which is a
`digitized pixel map representation of the original scene and,
`using a character recognition routine, searches the digitized
`media image ?le for the presence of text anywhere in the
`media image. thereby creating a secondary, ‘text’ ?le for that
`media image.
`Once a general media image has been imported. that
`media image will have two associated media image ?les.
`The ?rst is a base media image ?le obtained as a direct result
`of the digitizing process carried out by the media image
`import mechanism. The second is a text media image ?le,
`resulting from the precursor text detection operator carried,
`and contains whatever text, if any, is found in the original
`media image.
`Advantageously, the manner in which imported media
`images are processed for storage in accordance with the
`present invention allows two alternative retrieval routines to
`be used to access a stored media image for playback The
`?rst retrieval routine, termed an index search, relies upon the
`ability of the user to access folder and descriptor ?elds
`within the folder where the media image has been ?led. The
`second, free text search is intended to be used when the user
`does not have su?icient a priori information to access the
`folder and descriptor ?elds within the folder where the
`media image has been ?led. Instead, it relies upon the
`contents of the text image ?le associated with the media
`image to be retrieved.
`In accordance with the index search storage routine. using
`an archival document storage and retrieval graphical user
`interface program. and the view station keyboard and mouse
`devices. the user either opens a new folder or opens an
`already existing folder in which one or more media images
`may be ?led For each media image being stored within the
`folder, the user enters various identi?cation information.
`including media image topic and a media image ?le descrip
`tion. The media image index ?le also includes a description
`?eld. termed a ‘key word’ index ?eld. The purpose of the key
`word index ?eld is to provide a relatively concise descrip
`tion of the media image, that facilitates an index search by
`a user, and expedites retrieval of the media image by the free
`text search mechanism. when a user lacks su?icient infor
`mation to open the folder in which the media image ?le is
`stored.
`Even though the key word index ?eld can be prepared by
`the user directly from keyboard of the view station, it is
`cumbersome, time consuming and relies upon the expertise
`of the terminal operator to select the appropriate key words.
`Pursuant to the present invention. the contents of the key
`word index ?eld are initially generated by a default, non
`essential word extraction subroutine, that derives the key
`words of the key word index ?eld from the contents of the
`text ?le. According to this key word ?eld-generation
`subroutine, as each ‘text’ image ?le is generated, it is
`subjected to a non-essential word extraction operation,
`which reduces the contents of the media image’s text ?le to
`one or more ‘key words’. that are loaded into the key word
`
`Facebook, Inc. - Inter Partes Review
`EXHIBIT 1013
`
`007
`
`

`

`5,729,741
`
`5
`index ?eld, but contain no auxiliary or connecting words,
`such as de?nite or inde?nite articles, prepositions and other
`similar textual connectives, that may be parsed from the text
`without removing its essential content describing the media
`image.
`The non-essential word extraction subroutine is prefer
`ably a default operation for all text-containing media image
`?les, such as text database ?les, digitized document ?les, or
`voice-image text ?les generated by the speech recognition
`to-text conversion operator. However, it may be turned oil"
`for media images, such as photographs or video images, that
`typically contain only a limited, if any, quantity of alpha
`numeric characters. ’I‘urning o?’ the non-essential word
`extraction routine for such signi?cantly reduced, or limited
`text content media images prevents it from excising any of
`the alpha-numeric characters in the text ?les obtained by the
`above-described character recognition preprocessing of such
`media images. In such a case, the entire contents of the
`character recognition-derived text image are inserted
`directly into the key word index ?eld.
`According to a further feature of the present invention,
`where imported video is accompanied by voice/speech
`signals, ‘voice-text’ images, obtained by the voice recogni
`tion and text conversion routine, may be used in combina
`tion with a transcribed text ?le of the deposition, so that the
`transcribed text ?le may be augmented with the time line
`data of the video, thereby enabling the transcribed text ?eld
`to provide a text-based search and identi?cation mechanism,
`that is capable of rapidly locating the exact portion of the
`video where a point in the testimony took place.
`For this purpose, the contents of the transcribed text ?le
`and the contents of the voice-text ?le are correlated with one
`another, and thereby associated the time line indices of the
`video tape time line with the transcribed text ?le. The
`transcribed text ?le is then augmented to include time line
`indices of the video tape time line, so that the transcribed
`text ?le, which is derived from an essentially one hundred
`percent complete record of the videotaped deposition, can be
`used to locate both the audio contents of the video tape, and
`the associated video.
`Pursuant to a further feature of the invention, in the course
`of storing a media image ?le, the contents of the media
`image ?le are ‘hashed’, so as to produce a hashing code
`representative of the digitized information contained in the
`?le. This hashing code is stored in a hidden ?eld as part of
`the index ?le. Then. whenever a ?le is searched. duplicate
`copies of the media image may be rapidly located by
`invoking the hash code of any located ?le as a search
`parameter.
`
`BRIEF DESCRIPTION OF THE DRAWlNGS
`FIG. 1 diagrammatically illustrates the overall architec
`ture of the storage and retrieval system of an embodiment of
`the present invention;
`FIG. 2 is a process ?ow diagram illustrating the genera
`tion of media image text ?les in the course of importing
`media images from a variety of information sources;
`FIG. 3 diagrammatically illustrates the manner in a media
`image that has been imported by an import and ?le genera
`tion mechanisms is stored, so that it may accessed by means
`of an index search;
`FIG. 4 is a timing diagram of the relationship among
`video tape image frames, a time line containing a time index
`track associated with the (recorded video and audio) con
`tents of the video tape. successive words of voice-text of the
`video tape as derived from a speech recognition-to-text
`
`6
`conversion operator, and successive words of text of the
`video tape, as derived from a court reporter transcription;
`FIG. 5 shows a correlation operation through which the
`contents of a transcribed text ?le and the contents of a
`voice-text ?le are correlated with one another to provide an
`association between time line indices of a video tape time
`line with a transcribed text ?le; and
`FIG. 6 is a ?ow diagram illustrating a hashing operation,
`through which the contents of the digitized media image
`contained in the index ?le are processed through a numerical
`compression algorithm (hashed) to produce a hashing code,
`representative of the media image.
`
`30
`
`35
`
`45
`
`DETAILED DESCRIPTION
`Before describing in detail the new and improved (media
`image) information storage and retrieval system in accor
`dance with the present invention, it should be observed that
`the invention resides primarily in what is e?=ectively the
`interfacing of conventional data storage, retrieval, reproduc
`tion and communication components, and the integration of
`their associated signal processing and media image manipu
`lation control mechanisms, that are embedded in the opera
`tional control software resident in the system's integrated
`computer network, which enables a user of the system to
`both e?ciently store multiple types of information, deriv
`able from diverse media, and to rapidly access any such
`stored media image for reproduction on an output device.
`Accordingly, the structure. control and arrangement of
`these conventional components and control mechanisms
`have been illustrated in the drawings by readily understand
`able block diagrams and associated processing ?ow charts,
`which show only those speci?c details that are pertinent to
`the present invention, so as not to obscure the disclosure
`with structural details which will be readily apparent to
`those skilled in the art having the bene?t of the description
`herein. Thus, the block diagram illustrations of the Figures
`do not; necessarily represent the mechanical structural
`arrangement of the exemplary system. but are primarily
`intended to illustrate the major structru'al components of the
`system in a convenient functional grouping, whereby the
`present invention may be more readily understood
`Moreover, since the details of the various commercially
`available programs, referenced below, are not necessary for
`an understanding of the present invention, they will not be
`described. Where speci?cs of any of the identi?ed commer
`cially available programs or hardware components
`employed in the system are desired, reference may be had to
`the supplier/manufacturer of the item of interest.
`Referring now to FIG. 1. the general architecture of the
`storage and retrieval system of the present invention is
`ditically illustrated as comprising a multi-level
`network system, including a network server processor 10,
`that is coupled via a local area network (LAN) 11 to various
`input/output and control elements of the system. to be
`described. Network server processor 10 operates as a system
`interface between one or more (local or remote) user control
`terminals, a media image input/output device and an atten~
`dant media image store. as will be described. As a non
`limiting example, network server processor 10 may com
`prise an Intel processor chip-based computer (e.g. an 80486/
`66 MHz chip, driven by a Novel 3.12 network server
`program), equipped with 64 Mb of local memory. an atten
`dant 1 GB hard drive, a lObaseT hub, and associated
`communication cables for the local area network. DOS 6.22
`and WINDOWS 3.11 control software may be employed as
`the operating system program for each of the processors of
`
`Facebook, Inc. - Inter Partes Review
`EXHIBIT 1013
`
`008
`
`

`

`5,729,741
`
`25
`
`35
`
`45
`
`7
`the system. Database control software for the network server
`processor 10 may comprise a Gupta SQLBase program.
`The principal media inmge input/output control mecha
`nism of the system is performed through one or more local
`view stations 14, which are coupled via a digital commu
`nication bus 12 of the LAN 11 to the network server
`processor 10. so that users of the system may control the
`operation of the network server 10 and store and/or access
`data (media images) to be displayed, exported or printed. (In
`addition to being accessed and controlled by one or more
`local view stations 14, the network server 12 may also be
`remotely accessed via a remote access interface 13.)
`Each view station 14 preferably comprises a processor
`based video display terminal (VDT) having one or more
`input/control devices, such as a keyboard/mouse
`arrangement, and an associated display device (e.g. CR1‘).
`together with a network card that interfaces the view station
`with local area network bus 12. The (storage and retrieval)
`operational control program for each media image view
`station 14 comprises a supervisory media image manipula
`tion and control program. such as FYI, by Identitech, into
`which subordinate media image manipulation programs for
`use with various media and media image formats are layered
`or embedded. so that the user has the ability to operate a
`variety of media image processing programs through a
`single. supervisory controller. Such subordinate embedded
`programs include an image ?le annotation program (e.g.,
`provided by Spicer Imagination), that allows the user to
`mark-upledit documents, an image indexing program (such
`as ZYIndex) that allows the user to index a media image and
`perform a free text search using Boolean and fuzzy logic
`parameters, a multiple object ?le viewer (such as FYIView
`by Identitech, or Picview by Pegasus) for controlling the
`importing of media images having di?ierent formats from
`various media image sources, and which may be supple
`mented by one or more auxiliary media image access
`programs (such as OUTSIDE IN).
`For importing and storing images from diiferent media,
`network server processor 10 is coupled to a view/scan!
`capture station processor 21. Viewlscanlcapture station pro
`cessor 21 is interfaced with a plurality of information
`sources. shown as including a digitizing document scanner
`23, an audio input unit 25, a video camera 27 and an
`auxiliary input device 29 (such as a VCR, laser disc unit,
`digitizing still camera, and the like).
`As a non-limiting example, view/scan/captnre station
`processor 21 may comprise an Intel processor chip-based
`computer (e.g. a Pentium chip) together with associated
`Hauppage, MPEG, Xionics Turbo and Daughter cards. Also
`included in each view station terminal is a motion pictures
`expert group (MPEG) decompression card (such as an
`OPI‘IVIew or Reelmagic card) and a network card for
`interfacing the view/scanlcapture station 21 with the local
`area network 11.
`For controlling the operation of vieW/scan/capture station
`processor 21, a Hauppauge WIN/I’V or Pegasus Image
`Capture program may be employed for media image
`capture, Word Scan Plus may be used for optical character
`recognition, OPI'IVIew or Reelmagic control software for
`the corresponding MPEG decompression card, while text
`format processing for the document scanner may use Win
`Word for Microsoft Windows. Like the other media image
`processing software described previously, each o