(12) United States Patent
`Van der Wall et al.
`
`USOO6188381B1
`US 6,188,381 B1
`(10) Patent No.:
`*Feb. 13, 2001
`(45) Date of Patent:
`
`(54) MODULAR PARALLEL-PIPELINED VISION
`SYSTEM FOR REAL-TIME WIDEO
`PROCESSING
`
`(75) Inventors: Gooitzen Siemen van der Wal,
`Hopewell; Michael Wade Hansen,
`Lawrenceville; Michael Raymond
`Piacentino, Princeton; Frederic
`William Brehm, Lawrenceville, all of
`NJ (US)
`(73) Assignee: Sarnoff Corporation, Princeton, NJ
`(US)
`This patent issued on a continued pros
`ecution application filed under 37 CFR
`1.53(d), and is subject to the twenty year
`patent term provisions of 35 U.S.C.
`154(a)(2).
`Under 35 U.S.C. 154(b), the term of this
`patent shall be extended for 0 days.
`
`(*) Notice:
`
`(21) Appl. No.: 09/002,265
`(22) Filed:
`Dec. 31, 1997
`Related U.S. Application Data
`(60) Provisional application No. 60/058,270, filed on Sep. 8,
`1997.
`(51) Int. Cl." ............................ G06K9/40; G06F 15/173
`(52) U.S. Cl. ............
`345/1.12; 34.5/518; 348/721
`(58) Field of Search ............................ 348/721; 315/676;
`345/118, 112, 18; 711/151; 712/11
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`6/1993 Suzuki et al. ........................ 395/425
`Re. 34.282
`4,400,771 * 8/1983 Suzuki et al. ..
`... 711/151
`4,601,055 * 7/1986 Kent ...............
`... 382/495
`4,703,514 * 10/1987 Van Der Wal ......................... 382/41
`4,709,327
`11/1987 Hillis et al. .......................... 711/150
`(List continued on next page.)
`
`
`
`OTHER PUBLICATIONS
`(Thurber, Kenneth; A Systematic Approach to theDesign of
`Digital Bussing Structures; Fall Joint Computer Conference;
`pp. 1-10), 1972.*
`(Guttage, et al; A Single-Chip Multiprocessor for Multime
`dia: The MVP; IEEE Computer Graphics & Applications,
`vol. 12, ISS.6), Nov. 1992.*
`“Sensar VFE-100” brochure by the Pyramid Vision Com
`pany, 1994.
`“PVT-200” brochure byPyramid Vision Technologies, Jun.
`1997.
`* cited by examiner
`Primary Examiner Andrew I. Faile
`ASSistant Examiner-Reuben M. Brown
`(74) Attorney, Agent, or Firm William J. Burke
`(57)
`ABSTRACT
`A real-time modular video processing system (VPS) which
`can be scaled smoothly from relatively small systems with
`modest amounts of hardware to very large, Very powerful
`Systems with Significantly more hardware. The modular
`Video processing System includes a processing module con
`taining at least one general purpose microprocessor which
`controls hardware and Software operation of the Video
`processing System using control data and which also facili
`tates communications with external devices. One or more
`Video processing modules are also provided, each containing
`parallel pipelined Video hardware which is programmable by
`the control data to provide different Video processing opera
`tions on an input stream of video data. Each video process
`ing module also contains one or more connections for
`accepting one or more daughterboards which each perform
`a particular image processing task. A global Video bus routes
`Video data between the processing module and each Video
`processing module and between respective processing
`modules, while a global control bus provides the control data
`to/from the processing module from/to the Video processing
`modules Separate from the Video data on the global Video
`bus. A hardware control library loaded on the processing
`module provides an application programming interface
`including high level C-callable functions which allow pro
`gramming of the Video hardware as components are added
`and Subtracted from the Video processing System for differ
`ent applications.
`30 Claims, 6 Drawing Sheets
`
`380
`k
`38
`|E-NET
`
`-99
`as EPRO
`al A 320
`is
`is
`E
`l
`"T
`S
`s
`g
`
`322
`2
`E.
`
`'--
`310 326
`302
`SRAM
`
`PROC
`
`;
`,
`
`BRIDGE
`
`-304
`SRAM
`
`!IDEO
`SRAM
`
`342
`31232B
`WIDEO
`SRAM
`
`BRIDGE
`
`PROC
`
`344
`: 34
`
`PROC
`PRO
`
`g
`g
`) s
`sa-- s
`WIDEO
`SRAM3
`C)
`
`REDGE
`
`: 346
`306
`PMDAUGHTERBOARDB.
`384
`I
`SRAM
`
`|
`
`| 32
`30 ; is
`|
`
`PROC
`
`316 332
`y
`xt
`WIDEO
`SRAM
`
`3.
`3.
`.
`
`BRIDGE
`308
`348-
`PMDAUGHTERBOARDB2
`356
`FC
`ER BE
`350
`C)
`5
`C-392
`e
`ARBTER
`5 40
`
`-34
`
`TAG
`
`s
`
`336
`GLOBAL
`MEMORY
`340
`RS-232
`DUART -
`p
`FLASH
`BANK1
`Flash
`BANK2
`FLASH
`BANK3
`FLASH
`BANK4
`
`343
`J
`SEAPHORE
`PEG i
`SEMAPHORE
`REG2
`
`L-334
`
`Unified Patents, LLC v. Elects. & Telecomm. Res. Inst., et al.
`
`Ex. 1025, p. 1
`
`

`

`US 6,188,381 B1
`Page 2
`
`U.S. PATENT DOCUMENTS
`382/41
`4,797.942 * 1/1989 Burt
`2
`2
`lll . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
`... 358/160
`4.942,470 * 7/1990 Nishitani et al.
`4.959,717 * 9/1990 Faroudja ........
`... 358/147
`4,985,848
`1/1991 Pfeiffer et al. .
`... 364/518
`5,046,023
`9/1991 Katsura et al.
`... 364/518
`5,051.835
`9/1991 Bruehl et all
`358/311
`5.120.002
`7/1992 Wilson
`395/800
`5.241,389 - 8/1993 Bilbrey.
`... 348/552
`
`5,291,368
`
`3/1994 Conroy-Wass
`
`361/796
`
`... 361/796
`5,339,221 * 8/1994 Wass ..........
`... 710/38
`5,339,443 * 8/1994 Lockwood ..
`... 38.2/41
`5,359,674 * 10/1994 Van Der Wall.
`... 348/476
`5,402,488 * 3/1995 Karlock ......
`... 439/157
`5,435,737
`7/1995 Haga et al. .
`... 711/148
`5,455,920 * 10/1995 Muramatsu ...
`... 382/304
`5,461,679
`10/1995 Normile et al.
`... 348/706
`5,502,512 * 3/1996 Toyoda et al. .
`5,544,292
`8/1996 Winser ................................. 395/130
`
`5,550,825 * 8/1996 McMullan et al. .................... 370/73
`5,606,347
`2/1997 Simpson ...
`348/187
`5610,653
`3/1997 Abecassis.
`348/110
`2Y----Y-2
`3. : 3. A. al - - - - - - - - - - - - - - - - - - - - - - - - - - s:
`Y/ - 12
`:
`SCT . . . . . . . . . . . . . . . . . . . . . . . . . . . .
`5,664,214
`9/1997 Taylor et al. ......
`395/800.2
`5,701,507
`12/1997 Bonneau, Jr. et al.
`395/800
`5,732,164 * 3/1998 Kawaguchi et al. ...
`348/721
`5,734,808
`3/1998 Takeda ................................. 345/326
`
`5,761,466 : 6/1998 Chau - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 348/721
`
`5,768,609 * 6/1998 Gove et al. ..................... 395/800.11
`5,790,642
`8/1998 Charles et al. ....................... 395/559
`5,809,174 * 9/1998 Purcell et al. ..
`... 382/236
`5.835,147
`11/1998 Florentin et al. .................... 348/416
`5,910,117 * 6/1999 Basoglu et al. ...................... 600/454
`5,915,088 * 6/1999 Basavaiah et al.
`395/200.28
`5,923,339 * 7/1999 Date et al. ......
`... 34.5/505
`5,963,675 * 3/2000 Van Der Wal et al.
`... 382/260
`6,044,166
`3/2000 Bassman et al. .................... 382/103
`
`
`
`Unified Patents, LLC v. Elects. & Telecomm. Res. Inst., et al.
`
`Ex. 1025, p. 2
`
`

`

`U.S. Patent
`
`Feb. 13, 2001
`
`Sheet 1 of 6
`
`US 6,188,381 B1
`
`
`
`
`
`PROGRAMCONNECTION FRO
`"NULL"TOXB OUT
`
`STATEDIAGRAM FOR CROSSBARINPUT PORTSXB IN
`
`Unified Patents, LLC v. Elects. & Telecomm. Res. Inst., et al.
`
`Ex. 1025, p. 3
`
`

`

`U.S. Patent
`
`Feb. 13, 2001
`
`Sheet 2 of 6
`
`US 6,188,381 B1
`
`
`
`FIG. 2
`
`efd
`a
`
`S 3 is O
`t 2S as g
`
`C5
`2
`O
`
`95. O CfO as H.
`
`Cld
`
`202
`
`TO 204
`VPMLOCALCROSSPOINT SWITCH 2.0
`WADATA
`CROSSPOINT
`STATE
`MACHINE
`
`238
`
`INT
`
`INTERRUPT
`OVER
`BACKPLANE
`TO PM 10
`
`242
`
`INTERRUPTFLAGS
`J INTERRUPT MASKS INTERRUPTS
`244
`FROM
`WPMDEVICES
`
`Unified Patents, LLC v. Elects. & Telecomm. Res. Inst., et al.
`
`Ex. 1025, p. 4
`
`

`

`U.S. Patent
`
`Feb. 13, 2001
`
`Sheet 3 of 6
`
`US 6,188,381 B1
`
`
`
`
`
`
`
`
`
`a. SRAM3 O
`BRIDGER)
`PMDAUGHTERBOARDB
`
`
`
`341
`
`336
`
`GLOBAL
`MEMORY
`
`340
`
`RS-232
`DUART
`
`338
`
`FLASH
`BANK
`
`FLASH
`BANK2
`
`FLASH
`BANK 3
`
`FLASH
`BANK 4
`
`343
`
`SEMAPHORE
`REG
`
`334
`
`FIG. 3
`
`f
`3.
`:
`
`
`
`PMDAUGHTERBOARDB2
`356
`2 KO) PC KX
`a
`BRIDGE
`C
`350
`C
`C
`392
`CD 40
`
`PC
`
`Unified Patents, LLC v. Elects. & Telecomm. Res. Inst., et al.
`
`Ex. 1025, p. 5
`
`

`

`U.S. Patent
`
`Feb. 13, 2001
`
`Sheet 4 of 6
`
`US 6,188,381 B1
`
`UV VIDEO
`
`Y WIDEO
`INPUT
`
`
`
`OVERLAY
`VIDEO
`INPUT
`
`22
`
`418
`
`D1
`TERFACE
`
`DIGITAL
`
`Y
`
`GE
`
`420,
`
`COLORKEY
`OVERLAY
`
`VIDEO
`ENCODER
`
`RGB
`ANALOG
`OUTPUT
`S-VIDEO
`ANALOG
`OUTPUT
`CVBS
`ANALOG
`OUTPUT
`
`FROMDSP
`CONTROL
`BUS
`
`FIG. 4
`
`Unified Patents, LLC v. Elects. & Telecomm. Res. Inst., et al.
`
`Ex. 1025, p. 6
`
`

`

`U.S. Patent
`
`Feb. 13, 2001
`
`Sheet S of 6
`
`US 6,188,381 B1
`
`CVBS
`
`SVIDEO
`
`CVBS
`
`SVIDEO
`
`CVBS
`
`SVIDEO
`
`PARALLEL
`D INPUT
`
`
`
`FROM
`WPM
`CROSS-
`POINT
`SWITCH
`202
`
`UV FIELD
`BUFFER
`
`24
`
`TO
`WPM
`CROSS
`POINT
`SWITCH
`202
`
`TO
`WPM
`CROSS
`POINT
`SWITCH
`202
`
`Unified Patents, LLC v. Elects. & Telecomm. Res. Inst., et al.
`
`Ex. 1025, p. 7
`
`

`

`U.S. Patent
`
`Feb. 13, 2001
`
`Sheet 6 of 6
`
`US 6,188,381 B1
`
`
`
`VIDEON
`-
`->
`
`SRAM
`
`Unified Patents, LLC v. Elects. & Telecomm. Res. Inst., et al.
`
`Ex. 1025, p. 8
`
`

`

`1
`MODULAR PARALLEL-PIPELINED VISION
`SYSTEM FOR REAL-TIME WIDEO
`PROCESSING
`
`CROSS-REFERENCE TO RELATED
`APPLICATIONS
`The present application claims priority from U.S. Provi
`sional Patent Application Ser. No. 60/058,270 filed Sep. 8,
`1997.
`
`STATEMENT OF GOVERNMENTAL INTEREST
`
`This invention was made under Government Contract No.
`DAAK70-93-C-0066. The Government has certain rights in
`the invention.
`
`15
`
`BACKGROUND OF THE INVENTION
`
`1. Field of the Invention
`The present invention relates to a parallel-pipelined vision
`System for real-time Video processing and a corresponding
`method for processing Video signals, and more particularly,
`to Such a System which has been designed to be modular and
`readily Scalable for different applications.
`2. Description of the Prior Art
`Real-time Video processing applications are continuing to
`grow as the capabilities of Video processing Systems
`increase. Typical examples of applications of Video proceSS
`ing include autonomous vehicle navigation, vehicle anti
`collision systems, video Surveillance (both manned and
`unmanned), automatic target recognition, industrial
`inspection, and electronic video Stabilization and mosaic
`construction. The common requirement for all of these
`applications is a need to be able to process the Video imagery
`at rates Suitable for processing live Video. Vehicular obstacle
`avoidance, for example, requires very fast feedback of
`obstacle positions to the vehicle navigation System So that
`the vehicle can be guided around the impending obstacle.
`Other applications, Such as Video Surveillance and electronic
`Video Stabilization, require output that is provided at full
`Video rates for viewing by an operator.
`The amount of data to be processed for real-time imaging
`can easily get into the billions of operations per Second.
`Standard video provides approximately 30 Mbytes/sec of
`Video information that needs to be processed. Thus, even just
`50 operations per pixel, which is a modest amount of
`processing on a per-pixel basis, results in over 1.5 billion
`operations per Second to be performed to process a Standard
`Video stream. Standard personal computers and WorkStations
`are incapable of providing video processing at Such real
`time rates. The processing capabilities of these Systems are
`typically an order of magnitude or more lower than required,
`and the internal bussing of the Standard WorkStation is
`incapable of providing the data throughput required for both
`receiving a digital Video signal and providing a full
`resolution video display output. As a result, dedicated pipe
`lined Video processing Systems have been designed by a
`number of different manufacturers to provide real-time
`performance.
`For example, the Max Video 250 system, developed and
`distributed by Datacube, is one example of a System capable
`of processing multiple video channels, each of which can
`process pixels at 20 Mpixels/sec video rates. However, the
`MV250 only contains a single filtering element, while many
`are often required for real-time multi-resolution image pro
`cessing. The MV250 also provides limited capability of
`operating on multi-resolution images, because it has only
`
`25
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`US 6,188,381 B1
`
`2
`five timing channels, while there are 32 input and output
`paths to the Video crossbar Switch. In addition, a significant
`amount of control overhead is required to calculate the
`appropriate delays for each processing path. Moreover, there
`are only a limited amount of Video paths available between
`the Video processing motherboards, and the five timing
`channels available on one board are shared among all
`processing boards. As a result, Scaling up of the multi
`resolution processing becomes very cumberSome, and use of
`the available resources is inefficient.
`The MV250 also relies on using standard general purpose
`processor boards for System control and, therefore, does not
`provide a means to transfer processed Video data efficiently
`to the general purpose processor for higher level Video data
`processing. Datacube has provided Specific Solutions to add
`general purpose processing capability to the System with fast
`Video data Stream transfer by providing Special purpose DSP
`boards with such a video interface (e.g., an 1960 based
`processing board); however, in that case, the control of the
`Video hardware and the high level processing of Video data
`is decoupled, significantly reducing the efficiency of algo
`rithm implementations and adding additional control Soft
`ware and control time to Synchronize the taskS.
`The Imaging Technologies 150/40 is another example of
`a parallel-pipelined System, which is capable of processing
`40 Mpixels/Sec, also through multiple channels Simulta
`neously. This system has similar limitations as the MV250
`except that the distribution of video data paths within the
`Video processing boards and among the Video processing
`boards is even more limited.
`Effective real-time Video processing by Such dedicated
`pipelined Video processing Systems requires the efficient
`execution of a number of operations referred to as "front
`end operations. Real-time Video processing relies heavily
`on the operation of these front-end operations to perform
`higher-level operations at the desired rates. One of the
`principal front-end operations is the generation of multireSo
`lution image representations. These representations, referred
`to commonly as image pyramids, involve decomposing an
`original image at its full resolution into Successive repre
`Sentations at lower spatial resolutions. This is performed
`through iteratively filtering the image and Subsampling the
`filtered results. The most common of the pyramids, called
`the Gaussian pyramid, involves Successively low-pass fil
`tering and decimating the original image, providing a
`Sequence of Smaller and Smaller images that represent image
`features at lower and lower Spatial resolutions. A pyramid
`processor integrated circuit which provides Such pyramid
`filtering has been described, for example, by one of the
`present inventors in U.S. Pat. No. 5,359,674 and U.S. patent
`application Ser. No. 08/838,096.
`An efficient real-time Video processing System must be
`able to perform front-end operations at real-time video rates
`and to provide the results of the front-end processing to
`general-purpose processors, which analyze the results and
`make decisions based on the results. However, the higher
`level operations Subsequent to the front-end processes are
`typically very much application Specific and Significantly
`more complex. This makes the higher-level operations leSS
`suitable for optimization in hardware. The front-end
`processes, on the other hand, are ubiquitous and should be
`efficiently implementable in hardware.
`Based on these considerations, the following list of fea
`tures can be used to define an effective real-time video
`processing System:
`Fast convolution and pyramid generation. This includes
`the generation of Gaussian and Laplacian pyramids, as
`
`Unified Patents, LLC v. Elects. & Telecomm. Res. Inst., et al.
`
`Ex. 1025, p. 9
`
`

`

`US 6,188,381 B1
`
`3
`well as gradient filters and other generally-applicable
`filtering operations.
`Reconfigurable arithmetic logic units. Image pointwise
`addition, Subtraction, multiplication, and other more
`arbitrary operations are very common with front-end
`proceSSeS.
`Look-up table operations. These operations involve
`Single-image transformations that are performed point
`Wise on the image. Adding gain to an image, inverting
`an image, Scaling an image, thresholding an image, and
`other Such functions are typical of look-up table trans
`formations.
`Efficient parallel architecture. This describes the ability to
`use multiple components in parallel in an efficient way.
`Processing resources are not useful if they cannot
`flexibly be used while other processing resources are
`busy.
`Fast transferal of Video data to general-purpose proces
`sors. When image data must be analyzed by a DSP or
`general-purpose microprocessor, the image data must
`be quickly accessible.
`High-level hardware control. A reentrable, multitasking
`environment must be available for hardware control to
`achieve maximum efficiency and programmability.
`A real-time video processing System developed by the
`present assignee and known as the Sensar VFE-100 has been
`developed in view of these considerations. The VFE-100
`provides real-time image Stabilization, motion tracking,
`change detection, Stereo vision, fast Search for objects of
`interest in a Scene, robotic guidance, and the like by focusing
`on the critical elements in each Scene using the pyramid
`filtering technique to perform initial processing at reduced
`resolution and Sample density and then progressively refin
`ing the processing at higher resolutions as needed.
`Video image processing by the VFE-100 occurs in three
`Stages: Signal transformation, Signal Selection, and attribute
`estimation. Two basic Signal transformations are Supported.
`First, an image warp brings pairs of images into a common
`coordinate System (motion, Stereo or observed and reference
`images) So that Subsequent processes can be uniform and
`local. Second, the pyramid transform decomposes the image
`Signals into band-pass components in the Spatial domain.
`Signal Selection is performed in the pyramid transform
`domain. The location and resolution of data used in Subse
`quent analysis is controlled by Selecting those data from an
`appropriate window at an appropriate level of the pyramid.
`Selected data are then processed to obtain estimates of the
`attributes of interest. These include displacement vectors in
`the case of motion or Stereo, or texture or feature energy for
`altering and orienting. The transformed data are formatted to
`Suit analysis through the application of compact filters which
`can describe measures Such as local correlation, variance,
`texture energy, and local moments.
`The VFE-100 is designed as a general purpose computing
`engine for real-time vision applications. Pipeline processing
`is performed as image data flow through a sequence of
`elements, and data flow paths and processing elements can
`be reconfigured to Serve a wide variety of tasks. Once
`configured, a Sequence of Steps can be performed for an
`entire image or a sequence of images without external
`control. It is now desired to improve upon the VFE-100 by
`designing Such a real-time Video processing System Such that
`it is modular and can be Scaled Smoothly from a relatively
`Small System with modest amounts of hardware to a very
`large, very powerful System with Significantly more hard
`ware. In particular, it is desired to design a modular real-time
`
`4
`Video processing System which may be custom tailored to
`Specific applications through the addition of new processing
`components, and the reconfiguration of available devices.
`The present invention has been designed to meet these needs
`in the art.
`
`SUMMARY OF THE INVENTION
`The above-mentioned needs in the art have been met in
`accordance with the invention by providing a modular Video
`processing system (VPS) which allows the VPS to be scaled
`smoothly from a relatively small system with modest
`amounts of hardware to a very large, very powerful System
`with significantly more hardware. The modularity of the
`design of the VPS of the invention enables the VPS to be
`custom tailored to Specific applications through the addition
`of new processing components, and the reconfiguration of
`available devices.
`In particular, the present invention is a real-time Video
`processing System and a method for processing Video Sig
`nals. In accordance with a preferred embodiment of the
`invention, the VPS is broken up into a modular multi
`processor processing module (PM) with up to two additional
`daughterboards containing a microprocessor (preferably a
`digital signal processor (DSP)) and a hardware control
`library (HCL), one or more modular pipelined video pro
`cessing modules (VPM) with application specific
`daughterboards, and a global control bus (GCB) and a global
`video bus (GVB) connecting the PM and VPMs.
`The VPS of the invention combines a number of elements
`in a modular configuration including one or more VPMs
`which provide basic processing functions that operate effi
`ciently on multiple Video data Streams and which can Store
`and retrieve multiple video data streams. For “Front-End”
`Video processing, multiples of the basic video processing
`elements are required, Such as frame Stores, 2D filter
`functions, look-up table (LUT) operations, and one and two
`input arithmetic operations.
`Since the Video processing elements and the Video data
`Streams efficiently handle Video data of variable image
`resolution in parallel, and without difficulty, each Video data
`Stream may be of a different image resolution. In a preferred
`embodiment, this is implemented by providing independent
`timing information along with each Video stream using the
`techniques described in the afore-mentioned U.S. Pat. No.
`5,359,674 and U.S. patent application Ser. No. 08/838,096.
`This has the additional benefit of Significantly reducing the
`control overhead, because the pipeline delay does not have
`to be calculated for each processing operation, and many
`functions can automatically adapt their parameters based on
`the timing information.
`The VPS of the invention preferably contains a variety of
`Smaller, more Specific video processing modules or Video
`Processing Daughterboards (VPD) that are designed to
`perform more application specific functions. Typically the
`VPDs are implemented so that multiple VPDs can be
`mechanically attached to each VPM. The VPDs can range
`from very Specific video processing functions, Such as a
`Video digitizer, or Video display, to a range of dedicated
`Video processing functions Such as image correlations and
`image Warping, or Special image Storage functions, or can
`include general processors Such as one or more DSPs.
`The VPS of the invention also contains at least one
`general purpose processing module (PM) with up to four
`processors, which has the task of controlling all the func
`tions in the VPS, providing control and/or data interface to
`the external World, and providing general purpose proceSS
`
`15
`
`25
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`Unified Patents, LLC v. Elects. & Telecomm. Res. Inst., et al.
`
`Ex. 1025, p. 10
`
`

`

`S
`ing to data that was first processed by the Video processing
`functions of the VPS. The VPS preferably provides a way to
`Scale the number of general purpose processors for efficient
`multi-processor applications as well as a technique for
`efficient multi-processor multi-tasking and multi-threading.
`The global video bus is an efficient and high bandwidth
`Video Stream data bus that can Scale in bandwidth when
`multiple VPMs are added into the system, while the global
`control bus is an efficient control bus between the PM and
`the rest of the VPS.
`The global control bus passes control data from a hard
`ware control library of the PM to the VPMs which allows for
`on-the-fly programming of field programmable gate arrayS
`used to perform many of the processing functions of the
`VPMs. Transmission of the control data over the global
`control bus Separately from the Video data transmitted over
`the global Video bus is important, for if the Setting up the
`processing of the Video operations is not efficient, the
`real-time performance of the System degrades significantly.
`BRIEF DESCRIPTION OF THE DRAWINGS
`The present invention will be better understood after
`reading the following detailed description of the presently
`preferred embodiments thereof with reference to the
`appended drawings, in which:
`FIG. 1 is a block diagram of the modular video processing
`System of the invention.
`FIG. 2 is a top-level block diagram of the Video Processor
`Motherboard (VPM).
`FIGS. 2A and 2B illustrate the state diagrams of the
`croSSpoint State machine of the croSSpoint Switch for the
`output ports (XB OUT) (FIG. 2A) and input ports (XB
`IN) (FIG. 2B).
`FIG. 3 is a block diagram of the Processor Motherboard
`(PM).
`FIG. 4 is a block diagram of a Display Video Processor
`Daughterboard (VPD).
`FIG. 5 is a block diagram of a Digitizer Video Processor
`Daughterboard (VPD).
`FIG. 6 is a block diagram of a Correlator Video Processor
`Daughterboard (VPD).
`FIG. 7 is a block diagram of a Warper Video Processor
`Daughterboard (VPD).
`FIG. 8 is a preferred embodiment of the Warper VPD
`including five Video inputs and outputs.
`DETAILED DESCRIPTION OF THE
`PREFERRED EMBODIMENT
`A System and method which meets the above-mentioned
`objects and provides other beneficial features in accordance
`with the presently preferred exemplary embodiment of the
`invention will be described below with reference to FIGS.
`1-8. Those skilled in the art will readily appreciate that the
`description given herein with respect to those figures is for
`explanatory purposes only and is not intended in any way to
`limit the Scope of the invention. Accordingly, all questions
`regarding the Scope of the invention should be resolved by
`referring to the appended claims.
`System Overview
`AS illustrated in FIG. 1, the real-time Video processing
`system (VPS) of the invention is divided into four major
`System components, with numerous Subcomponents. The
`major components the VPS are:
`A Processor Motherboard (PM) 10 which provides
`general-purpose microprocessors or digital Signal pro
`
`5
`
`15
`
`25
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`US 6,188,381 B1
`
`6
`cessors (DSPs) 12 for controlling the dedicated video
`hardware, performing image analysis operations that
`are not easily mapped into the Video hardware, and
`facilitating communications with other components
`that are not an integral part of the VPS system.
`A Video Processor Motherboard (VPM) 20 which is the
`baseline video processing component within the VPS.
`Each VPM 20 contains dedicated, parallel-pipelined
`Video hardware that is capable of performing opera
`tions on Streams of Video at a consistent rate (based on
`a global pixel clock). As shown in FIG. 1, the VPM 20
`also Supports the addition of one or two
`daughterboards, called Video Processing Daughter
`boards (VPDs) 22-28 for specialized image
`acquisition, display, and processing devices. AS
`illustrated, there can be one or more VPMs 20 within a
`single VPS system, each with its own set of VPDs. Of
`course, the VPDS are not limited to those shown and
`described herein but may be other processing boards
`connected in different configurations.
`A Global Video Bus (GVB)30 which defines a dedicated,
`proprietary video backplane that enables Video infor
`mation to be transferred between the VPMS 20 of the
`VPS at 33 MBytes per second, and also for video to be
`transferred to and from the microprocessors 12 on the
`PM 10.
`A Global Control Bus (GCB) 40 provides control bussing
`between the PM 10, the VPMs 20, and the GVB 30 of
`the VPS. Control register accesses from the PM 10 are
`transacted over GCB 40 to the destination boards
`within the VPS. Typically, no video transactions are
`performed over this GCB 40.
`The basic operational and architectural aspects of the
`three major VPS Subsystems will now be described. The
`next Section will provide more detailed information on each
`component of the VPS system in turn. It should be noted that
`the term “reconfigurable' has been given two meanings
`herein, namely: “reconfigurable' devices having control
`registers which may be programmed to a new structure and
`“reconfigurable” devices such as FPGAs which are recon
`stituted to perform a different hardware function.
`PM 10 functions as the microprocessor core of the VPS.
`Two microprocessors 12 are actually used in PM10 with a
`possibility of adding one or two more microprocessors 12 as
`daughterboard components. The primary function of the PM
`10 is to provide the command and control of video process
`ing operations that are performed by the VPMs 20 and their
`associated VPDs 22-28. Video processing operations within
`the VPS require the initial setup of control registers in the
`video hardware, followed by the assertion of an enable
`Signal that defines the beginning of execution for that
`operation. The PM's microprocessors 12 perform the reg
`ister programming and Video processing operation execution
`through accessing memory-mapped VPM/VPD control reg
`isters. A high-level, C-callable hardware control library
`loaded on one or more of the microprocessors 12 is used to
`facilitate the coordination of the Video hardware. A Second
`function of PM 10 is to provide additional processing for
`imagery that cannot be performed more efficiently using the
`available dedicated hardware. The GVB 30 provides high
`speed data paths enabling each of the VPMs 20 to provide
`Video information to each microprocessor's local memory.
`Image operations can then be performed by the micropro
`ceSSorS 12 on the imagery as required by the final applica
`tion. The third function of PM10 is communications. While
`the VPS is usually configured with video inputs and outputs,
`there is also a need for lower-speed, more general data
`
`Unified Patents, LLC v. Elects. & Telecomm. Res. Inst., et al.
`
`Ex. 1025, p. 11
`
`

`

`US 6,188,381 B1
`
`15
`
`25
`
`35
`
`40
`
`7
`communications. Such communications can include RS-232
`Serial data communications and Ethernet.
`VPM 20 is a dedicated video processing board. All video
`hardware in the VPS operates on video streams in a parallel
`pipelined fashion. This means that Video data is read out of
`frame Stores a pixel at a time, with appropriate timing
`Signals framing the active Video information. AS this video
`flows through the System, it is processed by the various
`processing units on VPM 20, or the video can be stored in
`destination memories. All processing components on VPM
`20 are designed to work within this flow-through architec
`ture for data processing. Each processing unit will add a
`fixed amount of pipeline delay in the processing, but will be
`guaranteed to maintain the data throughput of the System.
`Thus, the amount of time required to perform an operation
`on a Video frame is always deterministic, given a fixed
`amount of pipeline delay which depends on the operations
`being performed. The Video routing through the System is
`performed through the use of a digital croSSpoint Switch of
`each VPM 20. This Switch enables video from a Source
`location to be routed to any destination on VPM 20, with no
`restrictions. Also, the croSSpoint Switch enables Video to be
`“fanned out” from one source to multiple destinations with
`no penalties. All hardware operations, including croSSpoint
`Switch routing, are defined through the programming of
`memory-mapped control registers on VPM 2.0. Each pro
`cessing device, croSSpoint connection, and Storage device
`has a Set of registers that are manipulated to define Specific
`operations. The PM's microprocessors 12 are responsible
`for Setting up these control registers and enabling video
`operations to begin.
`The VPDs 22–28 behave similarly to the devices on VPM
`20. Each VPD has dedicated crosspoint Switch inputs and
`outputs, and is also controlled through memory-mapped
`control registers.
`AS will be described in more detail below, Some of the
`crosspoint inputs and outputs present within VPM 20 are
`dedicated to video routing to and from the VPM 20 to other
`VPS boards. GVB 30 is responsible for routing video
`between the VPS system boards.
`Video can be routed between pairs of VPMs 20, and
`between the VPMs 20 and the PM 10. There are a number
`of different possible GVBs 30 depending on the application.
`The simplest GVB 30 provides dedicated, hard routed data
`channels between the VPS system boards with a fixed
`topology. This is the Simplest and most cost-effective
`method for routing video between boards, but it is also the
`least flexible method. Other, more advanced