Image acquisition of microscopic slides
`
`Lister Hill National Center for Biomedical Communications
`
`Earl Henderson
`
`National Library of Medicine
`Bethesda, Maryland 20894
`
`James R. Seamans
`
`Management Systems Designers
`Vienna, Virginia 22180
`
`Abstract
`
`This paper addresses the Color Medical Imaging System (CMIS) Program, which entails the
`development of a prototype system to evaluate spatial correlation techniques to convert
`microscopic images into full color digital electronic files. Program objectives were directed
`toward the creation of high resolution 2D images using spatial template matching. Full color
`image segments were captured using NTSC CCD array cameras. These lower resolution
`segments were captured in an overlapping coverage and combined at their borders as complete
`seamless high resolution files. The use of segmented capture provides two technical advantages.
`First, it overcomes the resolution limitation of the capture system and second, it expands the field
`of view of the microscope for a fixed magnification.
`
`Conversion of glass specimens into a computer—based format is a multistep optical-to—electronic
`process consisting of three phases:
`segment capture, record development, and image display.
`In contrast to some types of medical records, such as x-ray and 35mm film, glass slides hold the
`actual specimen, thus containing image information at the tissue, cell and molecular level.
`Therefore, the information yielded by conversion into electronic format is dependent on the
`magnification level and targeted area.
`
`CMIS was used to capture image segments from medical glass slide specimens using a light
`microscope, and to convert these segments into full color electronic image files. This paper
`describes the CMIS system, its image capture and conversion process.
`
`2.
`
`Introduction
`
`Digital image libraries are increasingly becoming integral components of modern information
`systems‘. The development of these emerging information systems is stimulated by the
`availability of cost effective high resolution display devices, mass storage systems, and wideband
`communications networks.
`In addition to the development of improved hardware systems,
`improved software designs have provided retrieval systems with graphical user interfaces and
`databases with complex data structures, both of which improve the access to image files. As
`storage and retrieval techniques improve, there is also a need to improve techniques for the
`acquisition of image libraries. Although many of the required image collections will be newly
`generated, a wealth of visual material already exists in traditional medical record collections such
`
`0-8 794-7468-9/94/$6.00
`
`SPIE Vol. 2173/21
`
`ZEISS, et aI., EX. 1016
`
`

`
`
`
`If converted into an electronic format, these
`as paper, x-rays, film, and microscopic slides.
`collections could provide valuable additions to future digital image libraries.
`
`In pursuit of systems and techniques to enhance future health databases the National Library of
`Medicine sponsors several programs which investigate the conversion of image information into
`electronic format. One such program is the Color Medical Imaging System (CMIS) Program,
`which evaluates techniques to convert microscopic images into full color digital electronic files.
`
`The CMIS Program is a continuation of an earlier effort to investigate segmentation techniques
`for the display of high resolution x-ray imagesz- This earlier program showed tl1at segmented
`window display techniques could be used to upgrade information databases and computer-based
`educational systems with high resolution medical
`images. The present program evaluates
`techniques to capture microscopic image segments and map these segments into high resolution
`electronic files.
`
`The objectives of CMIS were directed toward the creation of high resolution images using
`segmented capture with lower resolution CCD array cameras. These lower resolution segments
`were captured in an overlapping coverage and combined at their borders as complete seamless
`high resolution files. The use of segmented capture provides two technical advantages. First,
`it overcomes the resolution limitation of the capture system and second, it expands the field of
`view of the microscope for a fixed magnification.
`
`3. Information format
`
`A unique aspect of the problem that the CMIS Program addresses is the composition of
`information stored on microscopic slides. In contrast to information stored on film (x-ray, 35mm,
`etc.) which contains a visual record of the original biological specimens, microscopic slides
`contain the original specimen, in which information lies at the tissue, cellular, and molecular
`level. This information may be examined at different magnification levels for specimen detail.
`Therefore, the process to convert image information from a slide format into an electronic format
`is highly content dependent and must resolve several format constraints. The first constraint is
`imposed by a fixed magnification setting of the microscope and the second by a video resolution
`limitation. The CMIS techniques partially offset these limitations by capturing a multilevel
`image file structure which stores visual records of a selected specimen at several levels of
`magnification.
`
`However, CMIS does encounter a magnification constraint because its images are captured using
`a light microscope with magnification ranges between 10x and lO00x. This range can provide
`visual records of only the tissue and cellular level. Each captured record consists of a base image
`and one or more visuals of the specimen at higher magnification levels. A base image is defined
`as the segment record of the specimen which provides a full view of the target area. Depending
`on the size of the display monitor, the base image can be a 512 X 512 or a 1k x lk pixel image.
`Additional visual records of the specimen can be captured for 2k and 4k pixel frame sizes. For
`example, a target area of 10mm square can be identified within a glass slide specimen and then
`captured full view at 100x as a base image file. Additional magnified views at 200x, 400x and
`800x can also be captured using image segments.
`
`22 /SPIE Vol. 2173
`
`

`
`
`
`
`
`images are actually
`Using the NTSC format,
`captured into a frame size of 512 x 480 pixels
`using a CCD array camera interfaced to a 512 x
`512 digital frame grabber board. Although
`CMIS presently uses 512 x 512 frame segments,
`the system elements and techniques can be scaled
`to higher resolution cameras and mapped into
`higher resolution images. The NTSC format was
`employed to retain compatibility with existing
`NTSC facility test equipment used to align and
`measure system performance. Whereas
`the
`camera resolution determines the size of each
`
`HIBQG segments with overlap
`/\\.
`
`
`
`Figure 1
`
`Image Segments
`
`required number of
`the
`captured segment,
`segments to capture a complete image depends on the magnification setting for the image and
`the amount of overlap allowed for adjacent image segments.
`In Figure 1, a partial capture is
`shown for the top area of the image for an atomic structure. This capture uses three segments
`along the horizontal dimensions.
`
`Two of the three segments shown in Figure 1 are prime segments, the center shaded segment
`provides overlap coverage. Prime segments are adjacent segments which determine the effective
`frame resolution of the captured area. Each mapped image is increased in resolution in the x or
`y direction in direct proportion to the number of prime segments used along the selected axis.
`Overlapped segments are required to compensate for camera-to-image coordinate positioning
`errors at the neighboring segment boundaries. The amount of overlap is a function of the
`expected feature density of the image. CMIS uses a 50% overlap pattern. This coverage was
`determined empirically to minimize correlation errors caused by feature gaps in the image under
`high magnifications. Using a 50% coverage pattern the total number of segment images required
`to capture a selected microscopic area in two dimensions is (2m-1)2, where m represents the ratio
`of the magnitude of the image frame resolution over the segment resolution. A single x-y
`p1ane(z—plane) dataset of captured segments for a single color band is shown in Table 1. To
`capture a thick specimen target area, multiple z-planes can be mapped from adjacent focal planes.
`
`Table 1
`
`
`IMAGE SEGMENT RECORD
`
`Magnification
`
`Base Image
`x2
`
`x4
`x8
`
`No. of
`Segments
`1
`9
`
`49
`225
`
`File Size
`(Mbytes)
`0.25
`2.25
`
`12.25
`56.25
`
`SPIE Vol. 2173/23
`
`

`
`
`
`CMIS maps the captured segment images into seamless full color images of 1k, 2k, and 4k file
`sizes. Each image is stored as a visual record consisting of three color(rgb) bands. A visual
`record consists of a base image and one or more mapped images. The uncompressed file size
`of a visual record varies, based on the number and size of the mapped image. Table 2 shows
`a single z-plane visual record for file sizes presently used with the CMIS system.
`
`Table 2
`
`VISUAL RECORD FILE SIZE
`
`FILE MONOCHROME(Mbytes)
`512 Base
`0.25
`
`FULL COLOR§Ml_)@)
`0.75
`
`1k image
`2k image
`4k image
`
`1
`4
`16
`
`3
`12
`48
`
`The CMIS display uses a window magnification technique which emulates the microscope. This
`windowing technique enables the user to scan the specimen area and view it at higher
`microscopic magnification levels. This capability is limited to the magnification settings selected
`during the capture process. When multiple z—plane representations of the image are processed,
`the microscope focus function can also be emulated.
`
`Ft le Storage
`
`I er
`
`Imge Process! ng
`Vlcrkstat Ion
`
`Stage Contro I
`
`Figure 2
`
`System Diagram
`
`24 /SPIE Vol. 2173
`
`

`
`
`
`4. Technical approach
`
`The CMIS system design is based on commercially available system components. The system
`consists of separate operational units linked by a local area network. These units include a DOS
`OS capture workstation, a UNIX OS file server, and a UNIX OS image workstation. As shown
`in Figure 2, the image capture workstation controls the acquisition subsystems, and is linked via
`a local area network to the file storage and image workstation. Any of the hardware system
`components listed may be modified, upgraded or replaced with a functionally equivalent part.
`
`The information flow diagram, as shown in Figure 3, illustrates the conversion process. It shows
`a three-step process to convert a specimen area into an electronic format: acquisition of the
`segment record, development of the visual record, and image display.
`
`Biological
`Saoclnnn
`
`60539 Slide
`
`GNVEFIS I U! PIXSS
`
`
`
`Biological
`,,,"e
`
`W9"
`Resolution
`Irrnm F! Io
`
`Figure 3
`
`Information Flow Diagram
`
`
`
`1
`‘
`
`The specimen area and the focal plane are selected in the first step of the acquisition process.
`This step requires biological domain expertise and a clear understanding of the end user
`objectives. During the acquisition process, a tradeoff is made among three image parameters,
`target area, magnification level, and image frame size. While evaluating the CMIS system, the
`maximum image frame size was limited to 4k x 4k pixels. However, the frame size can be
`expanded compatibly with other system improvements. CMIS system performance is affected
`by hardware factors such as the microscope stage mechanical tolerance, disk storage, and system
`display memory. Based on the initial program objectives, we concluded that a 4k x 4k image
`format provided sufficient resolution and specimen target area to evaluate the conversion process.
`After identifying the specimen target area and setting the system acquisition parameters, a 24 bit
`full color image segment record is captured.
`*
`
`During the acquisition process magnified segments are captured as an order set and identified by
`row-column number. Row—column identification is critical to the correlation process. CMIS
`
`, SPIE Vol. 2173/25
`
`

`
`
`
`calculates boundary coordinates of overlapping segments using spatial domain template matching3
`techniques.
`
`System software implements an autocorrelation process as expressed in the following equation.
`This process compares the relative location of an extracted window (F3) from a subset of a
`segment image with a similar, but smaller window (Fw) extracted from the adjacent overlapping
`segment. Computation processing time can be reduced by restricting the correlation to a subset
`
`R->'(u'V)_
`
`J
`K
`J]=1i:l
`K
`[Z§:F§(1,J)l“2[ZZF5<1,J)]1/2
`_7=11=1
`j=11=l
`
`of the window (F5) domain based on the expected value of the boundary locations. These
`expected values are equal to the offset values of the microscope stage during the capture phase.
`The computation range near the expected boundary coordinates of the image subset is based on
`the microscope stage positioning error.
`
`Rotation and translation errors increase the processing time. Therefore camera—to—microscope
`system alignment procedures are used to minimized intersegment rotation errors. Translation
`errors are reduced by keeping the microscope focus fixed during the capture period.
`
`Correlation is conducted using an intensity transformation model of the image segments. The
`intensity format‘ preserves image detail missing in the individual color bands.
`
`0.299 0.587
`
`0.114
`
`
`
`
`
`9 0.212 ~o.523 0.311 3
`
`R
`Y
`I=O.596 -0.275 -0.321 G
`
`
`
`
`
`(Where Y = 0.299R + O.587G + 0.l14B is the intensity)
`
`Following each correlation process a mapping algorithm is invoked to generate a seamless
`mapped image. This algorithm utilizes the coordinate data extracted from the image segments.
`CMIS correlation and mapping processes are automated. A manual approach is too time
`consuming to be a practical solution. Image segments are transferred to archival files, following
`the mapping process, to conserve online disc storage.
`
`A correlation coordinate record is produced while processing each image. This record shows the
`boundary locations for the mapped image, and it is also used for error correction. An automated
`error correction algorithm uses values from the coordinate record as a boundary estimate to
`replace out-of—range values resulting from low correlation coefficients.
`In addition,
`the
`coordinate record for highly correlated z—plane images is used to map low correlated images of
`the same target volume. A command line image editor also utilizes the coordinate record for
`
`26 /SPIE Vol. 2173
`
`

`
`final manual realignment. Approximately two percent of the 2k image boundaries, and four
`percent of the 4k image boundaries required manual editing.
`
`5. System applications
`
`The CMIS program addresses a technique to add microscopic images to medical image libraries.
`During the CMIS evaluation, single plane visual records were made from seven glass slide
`specimens. Also a multilevel z—plane visual record was made, from sixteen 1k images,
`accumulated in one micron intervals.
`
`6. Acknowledgements
`
`We would like to acknowledge the work of Thomas E. Neuse, Diane Solomon, and James R.
`McArthur for their contributions to this project.
`
`7. References
`
`Ramesh, Jain. Workshop Report: NSF Workshop on Visual Information Management
`1.
`Systems", Computer Science ‘and Engineering Division, National Science Foundation, 1992.
`
`"PC Based X-ray Imaging System", Proceeding of SPIE - The
`Henderson, B.E.
`2.
`International Society for Optical Engineering, Feb. 1988, vol. 914, pp. 1232-1237.
`
`3. Duda, Richard and Hart, Peter. "Pattern Classification and Scene Analysis", John Wiley
`and Sons, 1973.
`
`Gonzalez, Rafael C., Woods, Richard E.
`
`"Digital Image Processing", Addison—Wesley,
`
`4.
`1991.
`
`
`
`
`
`
`
`SPIE Vol. 2173/27