Microscopes were invented almost four hundred years ago. But today, there’s a revolution in microscopy (as in so many other fields) associated with computation. We’ve been working hard to make the Wolfram Language a definitive platform for the emerging field of computational microscopy.

It all starts with getting an image of some kind—whether from a light or x-ray microscope, transmission electron microscope (TEM), confocal laser scanning microscope (CLSM), two-photon excitation or a scanning electron microscope (SEM), as well as many more. You can then proceed to enhance images, reconstruct objects and perform measurements, detection, recognition and classification. At last month’s Microscopy & Microanalysis conference, we showed various examples of this pipeline, starting with a Zeiss microscope and a ToupTek digital camera.

Image Acquisition

Use Import to bring standard image file formats into the Wolfram Language. (More exotic file formats generated by microscopes are accessible via BioFormatsLink.) What’s even cooler is that you can also connect to a microscope to stream images directly into CurrentImage.

Once an image is imported, you’re off to the races with all the power of the Wolfram Language.

Brightness Equalization

Often, images acquired by microscopes exhibit uneven illumination. The uneven illumination can be fixed by either adjusting the image background according to a given flat field or by modeling the illumination of the visible background. BrightnessEqualize achieves exactly this.

Here is a raw image of a sugar crystal under the microscope:

Here is a pure image adjustment:

And here is the result of brightness equalization using an empirical flat field:

If a flat-field image is not available, construct one. You can segment the background and model its illumination with a second-order polynomial:

Color Deconvolution

Color deconvolution is a technique to convert images of stained samples into distributions of dye uptake.

Here is a stained sample using hematoxylin C19 and DAB (3,3′-Diaminobenzidine):

The corresponding RGB color for each dye is:

Obtain the transformation matrix from dye concentration to RGB colors:

Compute the inverse transformation from color to dye concentration:

Perform the actual de-mixing in the log-scale of color intensities, since the color absorption is exponentially proportional to the dye concentration:

The color deconvolution into hematoxylin and DAB dye concentration:

False coloring of the dye concentration:

Image Viewing and Manual Measurements

To view large images, use DynamicImage, which is an efficient image pane for zooming, panning, dragging and scrolling in-core or out-of core images:

The following code is all it takes to implement a customized interactive interface for radius measurements of circular objects. You can move the position and radius of the superimposed circle via Alt+drag or Command+drag. The radius of the circle is displayed in the top-left corner:

Focus Stacking

To overcome the shallow depth of field of microscopes, you can collect a focal stack, which is a stack of images, each with a different focal length. You can compress the focal stack into a single image by selectively taking in-focus regions of each image in the stack. The function ImageFocusCombine does exactly that.

Here is a reimplementation of ImageFocusCombine to extract the depth and to go one step further and reconstruct a 3D model from a focal stack.

Take the norm of the Laplacian filter as an indicator for a pixel being in or out of focus. The Laplacian filter picks up the high Fourier coefficients, which are subdued first if an image is out of focus:

Then for each pixel, pick the layer that exhibits the largest Laplacian filter norm:

Multiply the resulting binary volume with the focal stack and add up all layers. Thus, you collect only those pixel values that are in focus:

The binary volume depthVol contains the depth information of each pixel. Convert it into a two-dimensional depth map:

The depth information is quite noisy and not equally reliable for all pixel locations. Only edges provide a clear indication if an image region is in focus or not. Thus, use the total focusResponse as a confidence measure for the depth map:

Take those depth measures with a confidence larger than 0.05 into account:

You can regularize the depth values with MedianFilter and close gaps via FillingTransform:

Display the depth map in 3D using the in-focus image as its texture:

An Example with Machine Learning: Pollen Classification

The Wolfram Language has powerful machine learning capabilities that allow implementation of various detection, recognition or classification applications in microscopy.

Here is a small dataset of six flower pollen types that we would like to classify:

Typically, one requires a huge dataset to train a neural network from scratch. However, using other pretrained models, we can do classification using such a small dataset.

Take the VGG-16 network trained on ImageNet available through NetModel:

Remove a few layers at the end that perform the specific classification in this network. This leaves you with a network that generates a feature vector:

Next, compute the feature vector for all images in the pollen dataset:

The feature vectors live in a 4k-dimensional space. To quickly verify if the feature vectors are suitable to classify the data, reduce the feature space to three dimensions and see that the pollen images appear to group nicely by type:

To increase the size of the training set and to make it rotation- and reflection-invariant, generate additional data:

With that training data, create a classifier:

Test the classifier on some new data samples:

An Example with Deep Neural Nets: Detecting Mitosis

The previous classifier relied on a pretrained neural network. If you have enough data, you can train a neural network from scratch, a network that automatically learns the relevant features and simultaneously acts as a subsequent classifier.

As an example, let’s talk about detecting cells that undergo mitosis. Here is a simple convolutional neural network that can do the job:

The data for training and testing has been extracted from the Tumor Proliferation Assessment Challenge 2016. We preprocessed the data into 97×97 images, centered around the actual cells in question.

Use roughly three-quarters of the data for training and the rest for testing:

Again, to increase the training set, perform image mirroring and rotation:

Calculate the classifier metrics and verify the effectiveness of the neural network:

Considering the challenging task, an error rate of less than 10% is comparable to what a pathologist would achieve.

Conclusion

Computational microscopy is an emerging field and an example of how all the different capabilities of the Wolfram Language come to bear. We intend to expand the scope of our functions further to provide the definitive platform for microscope image analysis.