## The Incredible Convenience of *Mathematica* Image Processing

December 1, 2008 — Theodore Gray, Co-founder, Wolfram Research, Inc; Founder, Touch Press; Proprietor, periodictable.com

It’s been possible since Version 6 of *Mathematica* to embed images directly into lines of code, allowing such stupid code tricks as expanding a polynomial of plots.

But is this really good for anything?

As with many extremely nifty technologies, this feature of *Mathematica* had to wait a while before the killer app for it was discovered. And that killer app is image processing.

*Mathematica* 7 adds a suite of image processing functions from trivial to highly sophisticated. To apply them to images, you don’t need to use any form of import command or file name references. Just type the command you want to use, then drag and drop the image from your desktop or browser right into the input line.

Here’s an undersaturated mandrill, and a command to increase the contrast. When you surround an image with textual input the image is automatically displayed at an icon size, but the input line still contains the full data of the image. That means if you save the notebook containing the input, the image is saved with it: the input line is completely self-contained.

Is that too much contrast? An obvious thing to want is a slider that lets you adjust the contrast. The general-purpose `Manipulate` command lets you make just about anything interactive, including the contrast parameter in this example.

OK, but you could just do that in Photoshop, right? Oh shush, let’s do something you definitely can’t.

Here’s a clown fish, and a command that breaks it into 40-pixel squares.

By the way, we’re seeing another neat thing about the integration of images with *Mathematica*‘s typeset input/output system. This result isn’t an image, it’s a list of images. Lists are general things in *Mathematica*, and lists of images are no exception. For example, here are the image patches in reverse order.

And here they are sorted by average pixel value (roughly by brightness):

And here are the images sorted into a scatter plot where the *x* axis represents the red component, the *y* axis represents the green component, and the size represents the blue component. (Image processing commands are fully integrated and compatible with charting commands. This is, after all, *Mathematica*, where one comes to expect things to work with each other.)

See how all the green patches are smooth and all the red patches are high-contrast? It might be interesting to look at these patches in a similarity network rather than a scatter plot. Here’s some code to do that.

This function identifies patches that are similar in color, then connects them into a network. The parameter says how many neighbors to look at before building the network.

Remember how we made a `Manipulate` to play with the contrast of an image? How about a `Manipulate` to play with the number of neighbors?

So there you go: an interactive slider you absolutely cannot get in Photoshop, or anywhere else for that matter.

Let’s look at another application, edge detection. Here’s a very pretty picture taken by Peter Overmann, who led image processing development in *Mathematica*:

Here’s what they look like processed through three image processing functions:

`LaplacianFilter` applies a form of edge detection, `ImageAdjust` tweaks the brightness and contrast, and then `Dilation` expands the bright areas to emphasize them.

The end result is neon jellyfish!

One of the main advantages of doing something like this in *Mathematica* is that you can trivially replicate operations for many images. Sure, you can write PhotoShop scripts; I’ve done that many times, and every time I end up swearing at the thing wishing I could use a real language (and now that *Mathematica* 7 is out, I finally can). Beyond a certain level of complexity it’s simply not efficient to mess around with dialog boxes and script editors: you want a real language with a real syntax and a real API.

A simple example: let’s apply this image processing chain not to one image, but to a series of them. Just to show off, I’ll download those images directly from the web using programmatically constructed URLs. Here’s a command that constructs ten simple URLs for images from one of my personal websites and imports them all into *Mathematica*.

And now let’s apply the same processing to all ten of those images.

Well that’s not ideal. Let’s make a custom interactive application with a slider that lets us play with the Laplacian filter radius and the dilation parameters:

Every time you move a slider, it’s reprocessing ten images automatically. Now that’s integration, real power that comes from taking core functionality and combining it with powerful, general-purpose tools like `Manipulate` and symbolic processing. Not to belabor the point, but this is really a fundamental advantage of *Mathematica* over any other system out there.

Of course the ultimate example of wanting to apply image processing to many images is video processing. To the goose footage!

Here’s a frame from a video I want to process (click the image to see the whole video):

What I want to do is draw a blue line around each goose. Why? Don’t ask, it’s a demo, OK?

The function I need for this is `MorphologicalComponents`, which identifies objects in an image.

To turn this into a blue outline I apply a series of image processing steps. First I expand the dark area:

Then I take the perimeter (outline) of the areas:

Then thicken the outline:

Then turn the outline into a semi-transparent blue:

And finally composite this outline onto the original image:

There you go, blue-outlined geese! Now I just need to do it for the other 424 frames in the movie. The movie is easily split into separate frames using QuickTime Pro. I put all the frames into a directory called “GeeseIn” in the same directory as this notebook. First I count the number of frames:

This command loads one frame of the movie:

The frames load pretty fast, so I can make a perfectly usable little movie viewer just using `Manipulate`. Put into animation mode and adjusted to the right speed this actually plays the movie quite smoothly, despite the fact that it’s reimporting each frame from disk at every step.

To process the movie automatically I put all the image processing steps together into one function:

This function is fast enough that I can make another little viewer that lets me flip through the processed movie in pretty close to real time:

To output a full movie I need to convert and save all the frames. This function writes one frame out to a directory:

And this `Do` command processes all the frames:

Finally, it’s a simple matter to assemble the movie using QuickTime Pro, resulting in this final product:

This is of course just the tip of the iceberg. For example, I’m not using the fact that `MorphologicalComponents` actually separately identifies and numbers each object in the image. It returns an image where each pixel is assigned the index number of the object occupying it. In other words, I can color each goose differently!

This constructs a table of different colors for each object index number, and black for the background.

This command identifies the objects, then colors each one according to the given rules.

And this command processes all the frames.

The mod (or is that rad?) result is this movie:

The flickering of colors, well, that’s because the index numbering of the objects changes from one frame to the next. Avoiding that, in effect doing persistent object detection across multiple frames, is a whole other can of worms, and beyond the scope of this blog post.

Though if you were wanting to develop such an algorithm, *Mathematica* might not be a bad place to start.

*Thank you to David Eisenman for the suggestion.*

## 9 Comments

Theodore, I have an animated video of ground penetrating radar shooting down into the earth every 10th of a second or so. There are lots of animal burrows in the underground,so we see lots of “holes”. I want to calculate the fractal dimension of this 3D space. Can I do that with Mathematica? Thanks for responding!

Al Kinlaw,PhD

Hi,

I met you at the conference in Champaign this past October. I want to video my butterfly swim stroke then analyze it in hopes of improving my efficiency.

Eventually, I would like to upload video files and be able to have mathematica analyze the data. Can you walk me through how to do this? I have these camera goggles. ( http://www.cameragoggles.com)

> I want to swim the butterfly and record it while I am swimming, then take the video and any see the ‘sine wave’ that

> represents my stroke. Can this be done?

natwilliams@mac.com

Hi, very nice, well i have a doubt, is there an algorithm in Mathematica to determine if an arbitrary image has chromatic harmony?, is there an algorithm in Mathematica to harmonize an image chromatically?

Thank you.

I know nothing about Mathematica and very little about Image Processing. However, your article was an eye-opener and very well written. I am fascinated by the possibilities…

hai iam studing now DCsE in final year

Hey guys, this good wook!

Even though I couldn’t understand your article, I still found it very amazing.

Bonjour …

Just discovering Math8 … Incredible

Am 59, from Belgium

Have a Hobby : try to save/restore/enhance and finally HD publish very very old B & W movies.

Imagine you have 10 scanned images. Because the camera was fixed the images are more or less the same. Some have dust, scratches, white points … Images can move a bit right, left, up down of some pixels . How to group then 10 images into a single best one ? I have quite a lot of ideas to achieve the goal . Has this topic some interest ? How can you help me ? Behind this is ‘old movie preservation’ which is very important for me.

Best regards

Pascal

These demos are all fine and dandy, but to my way of thinking Mathematica programming is highly counterintuitive and impossible to debug. How is the end-user supposed to guess the names and parameter lists of all the functions in order to solve his specific problem? That code to draw the blue line around the birds looks pretty bizarre and hard to adapt to other uses, especially for the uninitiated who are not Wolfram-gurus. This is a strange way to program: f(g(h(i(…))).

What I mean is, starting with the problem of the birds needing blue lines, how are you supposed to intuit all those function names? And how would you step through this composition of functions if your matryoshka of a program doesn’t do what you expect it to do?

I used to use Mathematica a lot back when it was version 2.x in my student days. It was impossible without constantly flipping through the manual to find the names of the functions and their zillions of parameters. The only reason I’m messing with it now is Mathematica is free on the Raspberry Pi.