United States Patent (19)
`Hedges et al.
`
`54 DIRECT DIGITAL AIRBORNE PANORAMIC
`CAMERA SYSTEMAND METHOD
`75) Inventors: Thomas M. Hedges, Great Falls, Va.;
`David G. Weir, Ormond Beach, Fla.;
`Jerry A. Speasl, Pleasanton, Calif.
`
`(73) Assignee: Omni Solutions International, Ltd.,
`Vienna, Va.
`21 Appl. No.: 449,350
`22 Filed:
`May 24, 1995
`6
`51) int. Cl. ................................ H04N 5/33; H04N 7/18
`(52) U.S. Cl. ............................ 348/144; 348/61; 348/106;
`348/113: 348/116; 348/117; 348/145
`58) Field of Search .............................. 348/61, 106, 113,
`348/116, 117, 135, 144, 145; 364/449;
`395/127
`
`(56)
`
`References Cited
`U.S. PATENT DOCUMENTS
`3,066,589 12/1962 Beatty ...................................... 95/12.5
`3.294,903 12/1966 Goldmarket al.
`... 178/6.8
`3,660,594 5/1972 Marsh ...............
`... 348/144
`: A. G. et al. -
`:::A;
`4496.972 1/1985
`CT -------.......
`I./w
`ippmann et al.
`... 358/109
`4,516,158 5/1985 Grainge et al. ......................... 358 io9
`4,682,160 7/1987 Beckwith, Jr. et al. ................ 395/127
`4,774,572 9/1988
`... 358/109
`4,796,090
`1/1989
`... 358/211
`4,814,711
`3/1989
`... 348/144
`4,829,304 5/1989
`... 364/449
`5,166,789 11/1992
`... 348/64
`5,204.818 4/1993
`... 348/116
`5,247,356 9/1993
`
`US005604534A
`Patent Number:
`11
`45) Date of Patent:
`
`5,604,534
`Feb. 18, 1997
`
`5,248,979 9/1993 Orme et al. ............................. 348/144
`5,267,042 11/1993 Tsuchiya ......
`... 348/144
`5,353,055 10/1994 Hiramatsu ........
`... 348/144
`3.
`s E. Wi et al. .
`s3.
`y
`y
`ea Cl .....................................
`5,481,479
`1/1996 Wight et al. ............................ 348/144
`Primary Examiner-Thai Q. Tran
`Assistant Examiner-Frank Snow
`Attorney, Agent, or Firm-Christie, Parker & Hale, LLP
`57)
`ABSTRACT
`An airborne direct digital panoramic camera system and
`method in which an in-line electro-optical sensor eliminates
`the need for photographic film and film transport apparatus
`normally associated with prior art airborne reconnaissance
`cameras and yet still retains the very high image resolution
`quality which is so important in intelligence operations and
`commercial geographic information systems (GIS), map
`ping and other remote sensing applications. The system
`provides a simpler, more efficient and less costly panoramic
`camera by utilizing a lens in conjunction with the electro
`optical line array sensor wherein the lens can be simpler and
`less expensive than a framing camera because it essentially
`requires quality focus in only one dimension and by elimi
`nating the burden and delay necessitated in film processing
`and development. The massive amounts of digital data
`generated by the camera are compressed and any motion or
`panoramic errors are easily correctable in the digital data,
`while such errors were nearly impossible to correct in a cost
`effective fashion from film images. The compressed digital
`image data may be stored and retrieved later for utilization
`in computer type networks or alternatively can be transmit
`ted from the aircraft to a ground station for prompt utiliza
`tion.
`
`42 Claims, 11 Drawing Sheets
`
`
`
`DATA
`HANDLING
`UNIT
`
`ELECTRO
`OPTIC
`SENSOR
`
`
`
`
`
`DIRECTION
`OF FLIGHT
`
`
`
`FLGHT
`PATH
`
`Google Ex. 1029, p. 1
`
`

`

`U.S. Patent
`
`Feb. 18, 1997
`
`Sheet 1 of 11
`
`5,604,534
`
`
`
`Google Ex. 1029, p. 2
`
`

`

`U.S. Patent
`
`Feb. 18, 1997
`
`Sheet 2 of 11
`
`5,604,534
`
`
`
`Google Ex. 1029, p. 3
`
`

`

`U.S. Patent
`
`Feb. 18, 1997
`
`Sheet 3 of 11
`
`5,604,534
`
`FIC.5
`
`FIG.6A 1-60
`
`5O
`
`36
`
`38
`
`|
`
`|
`
`
`
`FIG. 7
`CLOCK - 70
`1 OO PIXELS WIDE
`PULSEC
`|
`|
`|
`|
`LL PE25B4E567. goROW 1
`:
`E233P
`ROW2
`
`
`
`LENGTH
`OF
`PXEL LINE
`
`Google Ex. 1029, p. 4
`
`

`

`U.S. Patent
`
`Feb. 18, 1997
`
`Sheet 4 of 11
`
`5,604,534
`
`e-2-14
`
`FIG. 8
`e-3-14
`
`e-2-14
`/Y
`
`CONTROLLER
`
`DATA
`HANDLNG
`UNIT
`
`ELECTRO
`OPTIC
`SENSOR
`
`DIRECTION
`OF FLIGHT
`
`
`
`FLIGHT
`PATH
`
`Google Ex. 1029, p. 5
`
`

`

`U.S. Patent
`
`Feb. 18, 1997
`
`Sheet 5 of 11
`
`5,604,534
`
`
`
`FIG. 10
`PICTURE WIDTH
`RQW Rgw- Rows/FRAME O O. O. Rw
`
`
`
`Google Ex. 1029, p. 6
`
`

`

`U.S. Patent
`US. Patent
`
`9
`
`5,604,534
`5,604,534
`
`
`
`wfi.1;N:0:
`
`.
`
`
`
`
`
`
`
`
`
`
`
`mN?m,295383mmaz
`
`
`.E032Him/E2957::
`
`
`
`EEm>zoomomzmmaE8#5309:75Qz<2930Fmzmmjfidxao\<aegis?:ombmd
`
`7w?02EEof
`
`
`fl4ww“mAV62592ofg"A’m:23I4A,szomwwmmgooSE28w?@zopmfimjé
`
`
`
`
`
`\EOEmEmo<mokmxmmn_0
`
`Google EX. 1029, p. 7
`
`Google Ex. 1029, p. 7
`
`
`

`

`U.S. Patent
`
`Feb. 18, 1997
`
`Sheet 7 of 11
`
`5,604,534
`
`FIC. 12
`FRAME START
`
`GENERATE DYNAMC
`PXEL
`CALIBRATION DATA
`
`COMPARE DYNAMC
`AND
`SAMPLE MULTIPORT | STATIC PIXEL, DATA
`CCD ARRAY IN
`BURST MODE
`
`
`
`A/D CONVERTER
`
`
`
`MODIFY PXEL
`CALIBRATION
`LOOK-UP TABLE
`
`
`
`
`
`
`
`
`
`GPS LAT/LON
`COORONATES
`
`RECORD LAT/
`LON FOR
`EACH FRAME
`
`MONITOR
`MOTON OF E-O
`SENSOR
`
`GENERATE MOTION
`CORRECTION
`DATA
`
`
`
`PXEL CALIBRATION
`LOOK-UP TABLE
`FUNCTION
`
`REDUCE DATA RATE
`AS FUNCTION OF
`MN CYCLE RATE
`
`COMPRESS
`MERGY DATA
`
`RECORD DIGITAL
`MERGY DATA
`
`PROCESS SEAM DATA
`
`OUTPUT TO USERS
`
`Google Ex. 1029, p. 8
`
`

`

`U.S. Patent
`
`Feb. 18, 1997
`
`Sheet 8 of 11
`
`5,604,534
`
`
`
`rt.
`
`A k
`
`READ-OUT
`
`FIG. 14A
`
`116
`
`12O
`
`AMPLITUDE
`
`118
`
`LIGHT INTENSTY
`(LUMINS)
`FIG. f 4B
`
`
`
`CALIBRATED
`OUTPUT
`
`OUTPUT
`VOLTAGE -D
`
`Google Ex. 1029, p. 9
`
`

`

`U.S. Patent
`
`Feb. 18, 1997
`
`Sheet 9 of 11
`
`5,604,534
`
`FIC. 16
`
`CAMERA DATA
`142
`1 44
`DYNAMC
`CALIBRATION
`TABLE
`
`MISSION
`SET UP DATA
`CONTROLLER
`
`
`
`COMPARATOR
`CIRCUIT
`
`STATIC
`CALIBRATION
`TABLE
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`-- 1 O
`
`
`
`AMPLITUDEo
`
`- 1 O
`
`TIME-SEC
`
`Google Ex. 1029, p. 10
`
`

`

`U.S. Patent
`
`Feb. 18, 1997
`
`Sheet 10 of 11
`
`5,604,534
`
`
`
`
`
`
`
`
`
`
`
`
`AIRCRAFT
`Rx
`
`AIRCRAFT
`RX2
`
`DFFERENTIAL
`PROCESSOR
`
`
`
`
`
`MOTION
`DIFFERENTIAL
`CALIBRATION
`CIRCUIT AND
`FRAME MEMORY
`
`
`
`AMP ITUDE
`
`5. O
`
`WAVELENGTH IN P
`
`Google Ex. 1029, p. 11
`
`

`

`U.S. Patent
`
`Feb. 18, 1997
`
`Sheet 11 of 11
`
`5,604,534
`
`FIC. 19
`
`
`
`FIG. 19A
`
`LAT/LON
`ADDRESS
`
`MAGE
`DATA
`
`Google Ex. 1029, p. 12
`
`

`

`1.
`DRECT DIGITAL AIRBORNE PANORAMC
`CAMERA SYSTEMAND METHOD
`
`5,604,534
`
`BACKGROUND OF THE INVENTION
`The present invention relates to airborne panoramic cam
`era systems and more particularly to a direct digital pan
`oramic camera system and method in which an electro
`optical digital sensor eliminates the need for the film and
`film transport apparatus normally associated with prior art
`airborne reconnaissance cameras.
`Airborne camera reconnaissance or surveillance systems
`are nearly as old as the use of military and civilian aircraft.
`The prior art camera or photo reconnaissance systems gen
`erally involved camera photos taken from an aircraft flying
`over the area of interest, and the exposed film was returned
`to the ground after the flight where it was developed and
`processed before it could be delivered to the intelligence
`agencies or groups who could then determine whether the
`photographs contain the desired intelligence. A number of
`prior art reconnaissance systems, including those disclosed
`in U.S. Pat. No. 3,066,589, disclose an airborne reconnais
`sance system which includes an airborne film processing,
`scanning and transmission of the data to associated ground
`stations. U.S. Pat. No. 4,143,971 discloses an airborne photo
`reconnaissance system in which photo cells and color filter
`techniques are employed to identify specific targets of
`interest which have an unique optical or IR pattern. U.S. Pat.
`No. 4,442,453 discloses a combined film and electro-optical
`sensor for converting the exposed film to data suitable for
`relay over a radio link to a ground station where it may be
`demultiplexed and displayed on television type monitors.
`The above-described and other similar prior art photo
`reconnaissance systems employ film as the light-sensing
`medium and therefore have the attendant drawbacks of a
`bulky film system and film transport apparatus, delays in
`developing the film and further generally include a more
`complex and substantially more costly lens that utilize a
`square focal plane system which must focus in two dimen
`SOS.
`It is therefore a primary object of the present invention to
`prove an improved airborne panoramic camera system and
`method which is less costly and more efficient in operation.
`It is another object of the present invention to provide a
`direct digital airborne panoramic camera system and method
`in which an electro-optical sensoreliminates the need for the
`film and film transport apparatus of the prior art airborne
`camera systems.
`It is yet another object of the present invention to provide
`an improved airborne panoramic camera system and method
`in which panoramic errors are easily correctable in the
`digital image data.
`It is yet another object of the present invention to provide
`an improved airborne panoramic camera system and method
`which employs a simplified, less costly optical system.
`It is yet another object of the present invention to provide
`a direct digital, massive data rate airborne panoramic camera
`and system capable of efficiently supporting mapping and
`other remote sensing applications by producing massive data
`bases which are efficiently organized and appropriate for use
`with geographic information systems.
`It is yet another object of the present invention to provide
`a direct digital airborne panoramic camera system and
`method which eliminates the need for photographic film in
`airborne data collection process.
`
`O
`
`15
`
`20
`
`25
`
`30
`
`35
`
`45
`
`50
`
`55
`
`65
`
`2
`It is yet another object of the present invention to provide
`an improved airborne panoramic camera system and method
`having high image quality data particularly with respect to
`contrast and dynamic range.
`It is yet another object of the present invention to provide
`an improved direct digital airborne panoramic camera sys
`tem and method in which the images may be conveniently,
`accurately, and easily geo-referenced.
`It is a further object of the present invention to provide an
`improved direct digital airborne panoramic camera system
`and method in which the massive amounts of digitized
`image data are rapidly and efficiently available to the user in
`computer friendly formats.
`These and other advantages of the present invention will
`become more apparent from the following detailed descrip
`tion taken in conjunction with the illustrative embodiments
`shown in the accompanying drawings.
`
`SUMMARY OF THE INVENTION
`The present invention relates to an improved airborne,
`direct digital panoramic camera system and method in which
`an in-line, electro-optical sensor eliminates the need for
`photographic film and film transport apparatus normally
`associated with prior art airborne reconnaissance cameras
`and yet still retains the very high image resolution quality
`which is so important in intelligence operations and in
`commercial geographic information systems (GIS), map
`ping and other remote sensing applications. The present
`invention provides a simpler, more efficient and less costly
`panoramic camera by utilizing a simplified optical system in
`conjunction with the electro-optical line array sensor
`wherein the lens can be simpler and less expensive because
`it essentially requires quality focus in only one dimension
`and in only one place. The massive amounts of digital data
`generated by the camera are compressed, and any motion
`induced or panoramic errors are easily correctable in the
`digital data while such errors were nearly impossible to
`correct in film images. The compressed digital image data
`may be stored and retrieved later for utilization in computer
`type networks or alternatively can be transmitted from the
`aircraft to a ground station for prompt utilization.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`FIG. 1 is a schematic pictorial diagram of one embodi
`ment of applicants' improved airborne direct digital pan
`oramic camera.
`FIG. 2 is a geometric optical diagram illustrating pixel
`design dimensions of applicants' improved airborne pan
`oramic camera system.
`FIG. 3 is a partial block diagram of the preferred embodi
`ment of applicants' improved airborne panoramic camera.
`FIG. 4 is a perspective view of an electro-optical in-line
`imaging device usable in applicants' improved airborne
`panoramic camera system and method.
`FIG. 5 is a top view of an improved in-line electro-optical
`sensor in accordance with another aspect of applicants'
`invention.
`FIG. 6A is a top view of an alternate embodiment of an
`improved in-line electro-optical sensor utilizable in appli
`cants' improved airborne digital panoramic camera.
`FIG. 6B is a perspective view of another alternate
`embodiment of an electro-optical sensor utilizable in appli
`cants' improved digital panoramic camera system and
`method.
`
`Google Ex. 1029, p. 13
`
`

`

`5,604,534
`
`3.
`FIG. 7 is a pictorial representation of a portion of the
`improved in-line pixel array of applicants' electro-optical
`sensor utilizable in accordance with applicants' improved
`digital panoramic camera and method.
`FIG. 8 is a logic block diagram illustrating principal
`system components of applicants' improved airborne digital
`panoramic camera system and method.
`FIG. 9 is a partial pictorial illustrating the picture or frame
`relationship generated in the operation of applicants'
`improved airborne digital panoramic camera system and
`method.
`FIG. 10 is an expanded pictorial diagram of the multiple
`frame interrelationship of the video data generated in accor
`dance with applicants' improved airborne panoramic camera
`and method.
`FIG. 11 is a logic block diagram illustrating the overall
`system operation of applicants' improved airborne direct
`digital panoramic camera and method.
`FIG. 2 is a logic flow diagram illustrating another
`embodiment of applicants' improved airborne digital pan
`oramic camera and process.
`FIG. 13 is a schematic logic diagram illustrating the
`overlap operation of a dual port memory utilizable in
`accordance with principles of applicants' improved airborne
`digital panoramic camera and system.
`FIGS. 14A and 14B are graphs illustrating pixel calibra
`tion data utilizable in accordance with another aspect of
`applicants invention.
`FIG. 15 is a block diagram of a dynamic calibration
`system utilizable in accordance with another aspect of
`applicants' improved airborne digital panoramic camera and
`method.
`FIG. 16 is a graph of platform attitude parameters utiliz
`able in accordance with a dynamic platform stabilization
`apparatus and process in accordance with another aspect of
`applicants' invention.
`FIG. 17 is a block diagram of an error correction system
`and method for the improved sensor arm positioning mecha
`nism utilizable in accordance with another aspect of appli
`cants' invention.
`FIG. 18 is a graph of a portion of the electromagnetic
`spectrum illustrating remote multi-spectral data utilizable in
`accordance with another aspect of applicants' improved
`airborne digital panoramic camera.
`FIGS. 19 and 19A are a schematic logic illustration of
`data storage utilizable in applicants' improved airborne
`direct digital panoramic camera system and method.
`
`10
`
`5
`
`20
`
`25
`
`30
`
`35
`
`4
`high data rates involved present substantial problems in
`terms of system design, cost and implementation.
`Referring now to FIG. 1, there is schematically shown an
`improved direct digital airborne panoramic camera system
`and method in accordance with applicants' invention. As
`shown in FIG. 1, an aircraft 10, flying at an altitude of 5000
`feet, would yield a bowtie photo frame 12 that is approxi
`mately 1774 feet long and 10,000 feet wide. As will be more
`fully described hereafter in connection with FIGS. 16 and
`17, aircraft 10 incorporates as part of its navigation instru
`ments a global position system (GPS) receiver, not shown,
`which receives navigational data from a series of orbiting
`satellites 14 which are part of the global positioning system
`which are well known to those skilled in the art of naviga
`1On.
`The GPS navigation system permits the exact location of
`the aircraft in terms of latitude and longitude when a picture
`is taken and as will be hereinafter described to accurately
`determine and correlate the position of the sensor arm 16 of
`camera 18 shown in FIG.3 in relationship to a fixed position
`on the earth.
`In order to be commercially competitive, applicants'
`improved airborne direct digital panoramic camera system
`and method must be equivalent in performance to the
`currently available film systems. While comparison between
`film and electro-optical cannot be exact, set forth below in
`Table 1 is a set of performance parameters based on design
`and system tradeoffs.
`
`Design Objectives
`
`TABLE
`System Parameters
`
`Image Quality at 5000 AGL
`Velocity at 5000'
`Sun Angle
`Modes of Operation
`Sustainability
`
`Nadir 5
`IRS-7
`300 Knots or less
`Local Noon 2 hours
`10% and 55%. Overlap
`>1000 Frames of Data
`
`As illustrated schematically in FIG. 1, the above-listed
`parameters yield a bowtie photo frame 2 of approximately
`1700 feet long and 10,000 feet wide for an aircraft flying at
`an altitude of 5000 feet at 300 knots or less. The number of
`picture elements (pixels) in a frame 12 can be calculated as:
`
`Pixels/Frame=(Frame Width/GSD)x(Frame Length/GSD)
`where GSD is the ground's sample distance as illustrated in
`FIG. 2. Image Quality (IQ)27 implies a GSD of no more
`than /3 of a foot. Therefore each frame of data will contain
`at least 153,000,000 pixels, 26. Thus the average data rate
`can be calculated as:
`
`Data Rate=Pixels PIFramexFrames PISecxBytes/Pixel
`Bytes/pixel is measured after compression and the frames
`per second is determined by the mission parameters. For a
`minimum set of requirements, 0.25 bytes/pixel will be stored
`and frames or pictures will be taken as often as every 2
`seconds with 0.5 frames per second. This yields a minimum
`data rate of 15 megabytes per second.
`Turning now to FIG. 3 there is shown a partial breakout
`of an the airborne panoramic camera 18 having a rockably
`mounted sensor arm 16 which houses the lens and optical
`train, not shown. Mounted on top of the rockably mounted
`sensor arm 16 is an electro-optical assembly sensor 20.
`There are a number of commercially available airborne
`
`45
`
`50
`
`DETAILED DESCRIPTION
`In today's rapidly changing world of instant communica
`tions, governmental agencies, private businesses and the
`news media require more intelligence in ever more detail
`and from more remote locations thus making the efficient,
`flexible gathering of image information more critical every
`day. Panoramic airborne cameras are a key to these new
`mission requirements due to their ability to collect, in a very
`short time, massive data on geographically dispersed areas
`at very high image quality. For example, at IQ-7, a pan
`oramic camera at an altitude of 5000 feet in one hour can
`collect imagery data covering more than 500 square miles
`which yields over 87 gigabytes of compressed digital image
`data. This is the equivalent of 87,000 3-/2" floppy disks
`which, as will be recognized by those skilled in the digital
`memory arts, is a massive volume of data, and the extremely
`
`55
`
`60
`
`65
`
`Google Ex. 1029, p. 14
`
`

`

`5,604,534
`
`5
`panoramic cameras, however in the preferred embodiment,
`applicants have utilized an optical system of an airborne
`panoramic camera built by Hycon and used by the U.S.
`military for several years. Panoramic cameras generally take
`pictures over a 90-degree angle and each frame represents a
`t45 degree by a +10 degree look at the ground. As the
`aircraft moves forward along a line of flight 22, the sensor
`arm 16 sweeps from starboard to port in about 0.5 seconds
`thereby transferring or generating the image signals to the
`electro-optical sensor. This mode of taking picture data in a
`cross flight line direction is commonly called a whisk broom
`technique.
`As is known to those skilled in the photographic arts,
`airborne panoramic cameras, for example the KA-54A,
`KA-56A and KA-60 manufactured by Fairchild, the
`KA-55A manufactured by Hycon and the KA-80A manu
`factured by Itek, incorporate an image motion compensation
`(IMC) feature, not shown. The IMC virtually ensures that
`the picture is not blurred by aircraft movement because the
`forward motion of the aircraft in flight is removed or
`compensated for by the image compensation feature. In
`general, the image compensation feature moves the lens
`forward before initiating a picture-taking cycle and during
`the picture-taking period the lens moves aft as the aircraft
`moves forward thus compensating for the aircraft motion
`during frame exposure.
`As is known to those skilled in the airborne panoramic
`camera arts, the relationship of the image compensation rate,
`aircraft velocity, aircraft height and aircraft velocity are
`important parameters in determining the pixel data rates.
`The image compensation rate (TMC) may be expressed as:
`
`IMC-Focal Length}x(Velocity/Height of Aircraft)
`Referring again to FIG. 2, one of the most important
`parameters of the electro-optical sensor design is the deter
`mination of the appropriate ground sample distance (GSD)
`24. As shown in FIG. 2, the ground sample distance 24 is
`simply how much ground each picture element (pixel)
`represents in a frame. The conversion of the GSD to pixel
`pitch (assuming for simplicity the pixel size for square
`pixels) may be stated by the following formula:
`
`Pixel Pitch=Focal Length:X(Min GSD/Altitude)
`Therefore in applicants' preferred embodiment, any sensor
`with a pitch less than 20 microns would produce an Image
`Quality (IQ) of 7 at an altitude of 5000 feet. In applicants'
`preferred embodiment, the pixel pitch of the sensor is 13
`microns, which at an altitude of 5000 feet yields a GSD of
`2.56 in. and 3.62 in. at nadir and 45 degree look angle,
`respectively. This is equivalent to Image Quality 8.
`With a pixel pitch of 13 microns the line rate can be
`calculated as:
`
`45
`
`50
`
`55
`
`Line Rate=(Cross Track Width/GSD)/Scan Time
`The preferred embodiment of applicants' improved cam
`era has a scan time as small as 0.5 seconds and therefore at
`an altitude of 5000 feet has a cross-track width of 10,000
`feet. From the above equation it can be seen that this yields
`a line rate of 73,631 lines per second. Knowing the line rate
`permits the calculation of the pixels per port and the clock
`rate for the sensor from the following formula:
`
`60
`
`65
`
`Max Pixels/Port=Clock Ratefine Rate
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`6
`Thus assuming a reasonable clock rate of 12.5 MHz, this
`would yield a maximum pixels per port of 170. This implies
`that the line array sensor is preferably segmented into
`sections of length of 170 pixels or less. In building appli
`cants' improved camera, a number of design tradeoffs
`involving costs/service were made between scan time, clock
`rate and maximum pixels per port. In the preferred embodi
`ment of applicants' improved camera, the scan time is in the
`order of 0.75 seconds, the clock rate is in the order of 12.5
`MHz, and the maximum pixels per port were set at 256.
`These values yield a corresponding line rate of 49,100 lines
`per second. This slight slowing of the camera's scan time
`allows very useful tradeoffs and facilitates using less costly,
`commercially available sensors.
`Referring now to FIG. 4, there is shown an electro-optical
`sensor assembly 20 which is designed to be mounted on the
`optical sensor arm 16 and become an integral part of the
`optical assembly, not shown. In this manner the electro
`optical detector is fixedly mounted at the top of the optical
`path thus facilitating the maintaining of focus during flight
`operation. In prior art panoramic cameras, failing to keep the
`film flat and thus at a fixed distance from the lens was a
`major source of errors. Applicants, by fixedly mounting the
`sensor 20 to the arm 16 and thus with respect to the lens,
`eliminate a major source of so-called panoramic type errors.
`The electro-optical sensor assembly comprises planer
`mounting board 30 which may be made of any insulating
`material, for example a glass epoxy type. Mounted on board
`30 are four time-domain integration lined arrays 32, 34, 36
`and 38. There are a number of commercially available TDI
`line arrays which are suitable for use, including the Dalsa
`2048x96 sensor elements from Dalsa, Inc. in Waterloo,
`Ontario, Canada. The Dalsa 2048x96 sensor line array
`elements are a 13x13 micron pitch and can be operated at a
`15 megahertz clock rate. An alternative TDI line array is
`commercially available from Loral Fairchild which includes
`a 1024x128 element with a 15x15 micron pixel pitch. Both
`the Dalsa and the Loral arrays are front side illuminated
`charge coupled (CCD) devices and imaging is accomplished
`by integrating photons which must pass through one of more
`levels of polysilicone. Thus the resulting image is heavily
`filtered in the blue and green response of those imagers. The
`plain sensor array assembly 20 further includes a plurality of
`power and control connections 40, a digital timing and
`control processor 42, and thirty-two channel video pream
`plifiers 44. Preamplifiers 44 on the sensor array assembly
`perform the first step of video signal conditioning and may
`be any of the commercially available preamplifiers used, for
`example an emitter follower circuit MMBT3904, manufac
`tured by Motorola. As shown in FIG. 4 the electro-optical
`sensor assembly 30 preferably includes the sensors 32, 34,
`36 and 38, drive electronics 42 and preamplifiers 44 to
`precondition the image signals. The minimal inter-array
`spacing is desirable because it facilitates the use of relatively
`short interconnecting wires, thus facilitating impedance
`matching of the CCD clocking signals which helps in
`maintaining the low signal-to-noise ratio required when
`dealing with low-level signals and the short integration
`times, as will be discussed hereinafter.
`Design of the sensor assembly 30 is in a large measure
`dictated by the active picture area of the airborne camera.
`Airborne panoramic cameras generally have an active pic
`ture area of 4 inches, and therefore an ideal sensor array
`would use a 4-inch line array with something in the order of
`2000 pixels per inch. While an in-line array on the order of
`4 inches is certainly commercially feasible, the actual design
`of applicants' improved sensor array is a tradeoff utilizing
`
`Google Ex. 1029, p. 15
`
`

`

`7
`cost, number of ports and clock speed as the primary
`determinants. In the preferred embodiment of applicants'
`improved electro-optical sensor, 4 Dalsa 2048x96 element
`TDI line arrays 32, 34, 36 and 38 are utilized as shown
`which gives an active optical area of four inches. Obviously
`for other applications a single one-inch line array, or alter
`natively up to seven or eight line arrays in straight line or
`staggered parallel lines, could be utilized for different appli
`cations, as will be understood by those skilled in the
`photographic and airborne reconnaissance arts.
`It should be understood that the one-inch line array sensor
`will result in a substantially less resolution or picture quality
`and the seven or eight-inch sensor would be comparable to
`a framing camera operation which could be utilized for
`mapping.
`In another embodiment of applicants' improved airborne
`panoramic digital camera and method, the electro-optical
`sensor 30 or one or more of the line arrays 32, 34, 36 or 38
`of FIG. 4 or line arrays illustrated in other embodiments may
`be replaced by or supplemented with an additional infrared
`sensor for various applications. The structure and function of
`commercially available infrared sensors, for example
`HgCdTe (Mercury Cadmium Telluride) are well known to
`those skilled in the remote sensing arts. By utilizing various
`sensors, detectors or filters alone or in combination facili
`tates remote sensing of radiation from 0.3 microns wave
`length up to about 14 microns wavelength. In general,
`multi-spectral imagery data is collected in several discrete
`bands, as illustrated in FIG. 18, to obtain good quality data.
`Referring now to FIG. 5, sensor line array overlap, as
`illustrated in FIGS. 4 and 5, will be described. Four electro
`optical sensors, for example of the Dalsa type each com
`prising a 2048x96 element TDI line array are mounted on a
`glass member substrate 50 which is, for example 1.5 inches
`long by 0.5 inches wide. The pixel line array length of the
`illustrated Dalsa sensor is one inch with in the order of 2000
`pixels per inch. The overlap 52 as shown in FIG. 5 is
`necessary in order to avoid gaps in picture line data which
`if uncorrected would be a major source of error in the picture
`data. The optical sensor 20 moves as the optical column
`scans the ground scene for each frame. If a single line of
`pixel elements were employed, this movement would not
`present any optical difficulties in the resulting picture data.
`However with the use of Timed Domain Integration (TDI),
`it is critical that the line arrays be oriented perpendicularly
`to the travel of the camera scan. As is known to those skilled
`in the art of airborne photography, this can be accomplished
`through the appropriate positioning of the fixtures and the
`use of optical correction mechanism, to be more fully
`explained hereinafter. In applicants' improved airborne digi
`tal panoramic camera a new picture (frame) is taken every
`1/75 of a second. With 2000 pixels per sensor element and
`four sensor elements there are thus 320 million pixel signals
`per second generated by the operation of the airborne
`camera as may be calculated utilizing the aforesaid equation
`for data rate as a function of the number of pixels perframe
`times the number of frames per second multiplied by the
`bytes per pixel having a minimum of 0.25 for the defined
`system requirements. The analog video signals are read from
`the sensor array 30, as will hereinafter more fully be
`described, via four sets of 8-channel graphic data outputs 44
`from each Dalsa sensor line array at an effective clock speed
`of 28 megahertz. A new picture scan is initiated every 2.25
`seconds. As shown in FIG. 5 there is a pixel line seam
`separation 52 of approximately 34-inch, which for the Dalsa
`line array is equal to approximately 1500 pixel widths. There
`is also a sensor line overlap of approximately one millimeter
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`65
`
`5,604,534
`
`10
`
`15
`
`20
`
`8
`creating duplicate pixel data on both sides of the seam 52.
`The seam separation is a potential error source which
`requires control of the scan arm velocity VO to within 0.1%.
`As will be hereinafter discussed in connection with FIG. 11
`the post processing of the seam data uses the overlap data to
`calibrate the actual velocity of the scan arm. The post
`processing uses a Fourier type analysis in two dimensions to
`calculate the actual velocity errors which can be used to
`compensate or correct the graphic or picture data.
`Referring now to FIG. 6, an alternate embodiment of
`applicants' improved electro-optical line sensor array 60 is
`shown which is particularly useful in multi-spectral imagery
`of agricultural crops. For years the U.S. Agriculture Depart
`ment has undertaken a number of research projects to
`explore the feasibility of developing improved remote multi
`spectral sensing to identify and quantify various ground
`conditions which would be useful in improving land use.
`Remote spectral sensing in agriculture is concerned with a
`determination of various parameters and characteristics of
`crops through an analysis of data taken at a distance. Remote
`multi-spectral sensing in agriculture has the broad objective
`to increase the yield and quality of agricultural cultivation
`and decrease losses in production which are due to disease
`or weed or insect infestation thereby increasing the quantity
`and quality of agricuitural production. As will be hereinafter
`more fully described, certain portions of the electromagnetic
`spectrum are particularly useful in agricultural multi-spec
`tral imaging. A biological mass or other properties of certain
`vegetation can be estimated as a ratio of the infrared to the
`green-yellow spectrum. An improved electro-optical line
`sensor assembly 60 illustrated in FIGS. 6A and 6B would be
`useful in such agricultural remote multi-spectral imagery of
`crops. As shown in FIG. 6A, there are two electro-optical
`line arrays 62 and 64 which are overlaid by color filters 66
`and 68. As hereinabove de