When we first added the MathLink libraries for C, there were real benefits in farming out intensive numerical work (though Mathematica performance improvements over the years and development of the compiler have greatly reduced the occasions where that would be worth the effort). Creating an Excel link added an alternative interface paradigm to Mathematica that wasn’t available in the Mathematica front end. But in the case of R, it isn’t immediately obvious that it does many things that you can’t already do in Mathematica or many that it does significantly better.

However, with RLink I now have immediate access to the work of the R community through the add-on libraries that they have created to extend R into their field. A great zoo of these free libraries fill out thousands of niches–sometimes popular, sometimes obscure–but lots of them. There are over 4,000 right here and more elsewhere. At a stroke, all of them are made immediately available to the Mathematica environment, interpreted through the R language runtime.

Let’s look at a simple example. While Mathematica supports FisherRatioTest, it doesn’t know the exact Fisher test. (This is a hypothesis test where the null hypothesis is that rows and columns in a contingency table with fixed marginals are independent.)

Well, now it does.

Finding the right library is more work than phoning Tank, and I skipped over any error checking. But the only complicated bit was extracting the p-value from the result (the “[[1,1,1]]” part) because RFunction returns an RObject that contains additional metadata, which, this time, I didn’t care about.

I can now use this just like any built-in function.

I can plot it:

I can manipulate it:

And I can use it with libraries from other languages in a similar way:

The future is always hard to predict. When I started here (many) years ago, general linking to FORTRAN seemed like the most important thing, but no one ever asks me about that any more–C and Java linking are the most popular. Links to some specific libraries (BLAS/LAPACK, GMP, and others) have ended up being core infrastructure components in Mathematica. Whether RLink finds extensive use in Mathematica‘s future features, or remains a more or less stand-alone added chunk of functionality within Mathematica‘s infrastructure, is not yet clear.

There are also issues that need to be considered. R code isn’t going to handle symbolic arguments or high-precision numbers, so, for robustness, you will want to type-check more carefully than you might with Mathematica code. You don’t always have the elegant design and quality of Mathematica. Some of it is quite raw, while some of it is excellent. But design and quality take time and resources, and so it will be quite a while before we fill out Mathematica to fill every one of these niches, and through RLink they are available right now.

A big chunk of extra functionality just became a part of the Mathematica ecosystem.

Download this post as a Computable Document Format (CDF) file.

Recently we added formulas for a variety of shapes and forms, and the Wolfram|Alpha Blog showed some examples of shapes that were represented through mathematical equations and inequalities. These included fictional character curves:

Shape curves:

And, most popular among our users, person curves:

While these are curves in a mathematical sense, similar to say a lemniscate or a folium of Descartes, they are interesting less for their mathematical properties than for their visual meaning to humans.

After Richard’s blog post was published, a coworker of mine asked me, “How can you make an equation for Stephen Wolfram’s face?” After a moment of reflection about this question, I realized that the really surprising issue is not that there is a formula: a digital image (assume a grayscale image, for simplicity) is a rectangular array of gray values. From such an array, you could build an interpolating function, even a polynomial. But such an explicit function would be very large, hundreds of pages in size, and not useful for any practical application. The real question is how you can make a formula that resembles a person’s face that fits on a single page and is simple in structure. The formula for the curve that depicts Stephen Wolfram’s face, about one page in length, is about the size of a complicated physics formula, such as the gravitational potential of a cube.

In this post, I want to show how to generate such equations. As a “how to calculate…”, the post will not surprisingly contain a fair bit of Mathematica code, but I’ll start with some simple introductory explanations.

Assume you make a line drawing with a pencil on a piece of paper, and assume you draw only lines; no shading and no filling is done. Then the drawing is made from a set of curve segments. The mathematical concept of Fourier series allows us to write down a finite mathematical formula for each of these line segments that is as close as wanted to a drawn curve.

As a simple example, consider the series of functions y_n(x),

which is a sum of sine functions of various frequencies and amplitudes. Here are the first few members of this sequence of functions:

Plotting this sequence of functions suggests that as n increases, y_n(x) approaches a triangular function.

The sine function is an odd function, and as a result all of the sums of terms sin(k x) are also odd functions. If we use the cosine function instead, we obtain even functions. A mixture of sine and cosine terms allows us to approximate more general curve shapes.

Generalizing the above (-1)^{(k – 1)/2)} k^-2 prefactor in front of the sine function to the following even or odd functions,

allows us to model a wider variety of shapes:

It turns out that any smooth curve y(x) can be approximated arbitrarily well over any interval [x₁, x₂] by a Fourier series. And for smooth curves, the coefficients of the sin(k x) and cos(k x) terms approach zero for large k.

Now given a parametrized curve γ(t) = {γ_x(t), γ_y(t)}, we can use such superpositions of sine and cosine functions independently for the horizontal component γ_x(t) and for the vertical component γ_y(t). Using a sum of three sine functions and three cosine functions for each component,

covers a large variety of shapes already, including circles and ellipses. The next demonstration lets us explore the space of possible shapes. The 2D sliders change the corresponding coefficient in front of the cosine function and the coefficient in front of the sine function. (Download this post as a CDF to interact)

If we truncate the Fourier expansion of a curve at, say, n terms, we have 4n free parameters. In the space of all possible curves, most curves will look uninteresting, but some expansion coefficient values will give shapes that are recognizable. However, small changes in the expansion coefficients already quickly change the shapes. The following example allows a modification of the first 4 × 16 Fourier series coefficients of a curve (meaning 16 for the x direction and another 16 for the y direction). Using appropriate values for the Fourier coefficients, we obtain a variety of recognizable shapes.

And if we now take more than one curve, we already have all the ingredients needed to construct a face-like image. The following demonstration uses two eyes, two eye pupils, a nose, and a mouth.

And here is a quick demonstration of the reverse: we allow the position of a set of points (the blue crosses) that form a line to be changed and plot the Fourier approximations of this line.

Side note: Fourier series are not the only way to encode curves. We could use wavelet bases or splines, or encode the curves piecewise through circle segments. Or, with enough patience, using the universality of the Riemann zeta function, we could search for any shape in the critical strip. (Yes, any possible [sufficiently smooth] image, such as Jesus on a toast, appears somewhere in the image of the Riemann zeta function ζ(s) in the strip 0 ≤ Re(s) ≤ 1, but we don’t have a constructive way to search for it.)

To demonstrate how to find simple, Fourier series-based formulas that approximate given shapes, we will start with an example: a shape with sharp, well-defined boundaries—a short formula. More concretely, we will use a well-known formula: the Pythagorean theorem.

Rasterizing the equation gives the starting image that we will use.

It’s easy to get a list of all points on the edges of the characters using the function EdgeDetect.

Now that we have the points that form the edges, we want to join them into straight-line (or curved) segments. The following function pointListToLines carries out this operation. We start with a randomly chosen point and find all nearby points (using the function Nearest to be fast). We continue this process as long as we find points that are not too far away. We also try to continue in a “straight” manner by slightly penalizing sharp turns. To see how the curve construction progresses, we use Monitor.

For the Pythagorean theorem, we obtain 13 individual curves from the edge points.

Joining the points and coloring each segment differently shows that we obtained the expected curves: the outer boundaries of the letters, the inner boundaries of the letters a and b, the three squares, and the plus and equal signs.

Now for each curve segment we want to find a Fourier series (of the x and y component) that approximates the segment. The typical textbook definition of the Fourier coefficients of a function f(x) are integrals of the function multiplied by cos(k x) and sin(k x). But at this point we have sets of points, not functions. To turn them into functions that we can integrate, we make a B-spline curve of each curve segment. The parametrization variable of the B-spline curve will be the integration variable. (Using B-splines instead of piecewise linear interpolations between the points will have the additional advantage of making jagged curves smoother.)

We could find the integrals needed to obtain the Fourier coefficients by numerical integration. A faster way is to use the fast Fourier transform (FFT) to get the Fourier coefficients.

To get more uniform curves, we perform one more step: re-parametrize the spline interpolated curve of the given curve segments by arclength. The function fourierComponents implements the B-spline curve making, the re-parametrization by arclength, and the FTT calculation to obtain the Fourier coefficient. We also take into account if a curve segment is open or closed to avoid Gibbs phenomena-related oscillations. (The above demonstration of approximating the pentagram nicely shows the Gibbs phenomenon in case the “Closed” checkbox is unchecked.)

For a continuous function, we expect an average decay rate of 1/k² for the kth Fourier series coefficient. This is the case for the just-calculated Fourier series coefficient. This means that on average the 10th Fourier coefficient is only 1% in magnitude compared with the first one. This decay allows us to truncate the Fourier series at a not too high order, as we do not want to obtain formulas that are too large. This expression gives the exponent in the decay rate of the Fourier components for the a² + b² = c² curve above. (The slightly lower than 2 exponent arises from the discretization points in the B-spline curves.)

Here is a log-log-plot of the absolute values of the Fourier series coefficient for the first three curves. In addition to the general trend of an approximately quadratic decay of the Fourier coefficients, we see that the magnitude of nearby coefficients often fluctuates by more than an order of magnitude.

Multiplying the Fourier coefficients by cos(k t) and sin(k t) and summing the terms gives us the desired parametrizations of the curves.

The function makeFourierSeriesApproximationManipulate visualizes the resulting curve approximations as a function of the series truncation order.

For the Pythagorean theorem, starting with a dozen ellipses, we quickly form the characters of the inequality with increasing Fourier series order.

We want a single formula for the whole equation, even if the formula is made from disjoint curve segments. To achieve this, we use the 2π periodicity of the Fourier series of each segment to plot the segments for the parameter ranges [0, 2π], [4π, 6π], [8π, 10π], …, and in the interleaving intervals (2π, 4π), (6π, 8π), …, we make the curve coordinates purely imaginary. As a result, the curve cannot be drawn there, and we obtain disjoint curve segments. Here this construction is demonstrated for the case of two circles:

The next plot shows the real and imaginary parts of the complex-valued parametrization independently. The red line shows the purely imaginary values from the parameter interval [2π, 4π].

As we want the final formula for the curves to look as short and as simple as possible, we change sums of the form a cos(k t) + b sin(k t) to A sin(k t + φ) using the function sinAmplitudeForm and round the floating-point Fourier series coefficients to nearby rationals. Instead of Piecewise, we use UnitStep in the final formula to separate the individual curve segments. The real segments we list in explicit form, and all segments that should not be drawn are encoded through the θ(sgn(sin(t/2)^(1/2))) term.

Now we have everything together to write down the final parametrization {x(t), y(t)} of the typeset form of the Pythagorean theorem as a mathematical formula.

After having discussed the principal construction idea for the parametrizations, let’s look at a more fun example, say the Pink Panther. Looking at the image search results of the Bing search engine, we quickly find an image that seems desirable for a “closed form” parametrization.

Let’s use the following image:

We apply the function EdgeDetect to find all edges on the panther’s face.

Connecting the edges to curve segments yields about 20 segments. (Changing the optional second and third argument of pointListToLines, we obtain fewer or more segments.)

Here is each segment shown with a different color. We see that some closed curves arise from two joined curve segments; we could separate them by changing the second argument of pointListToLines. But for the goal of sketching a line drawing, the joined curve will work just fine.

Proceeding now as above, it is straightforward to approximate the curve segments by trigonometric series.

Plotting the series shows that with 20 terms per segment, we obtain a good representation of the Pink Panther.

As some of the segments of the panther’s face are more intricate than others, we define a function makeSegmentOrderManipulate that allows the number of terms of the Fourier series for each segment to be varied. This lets us further reduce the size of the resulting parametrizations.

We use initial settings for the number of Fourier coefficients that yield a clearly recognizable drawing of the Pink Panther.

For simple cases, we can now roll up all of the above function into a single function. The next function makeSilhouetteFourierResult takes a string as an argument. The function then 1) performs a search on Bing’s image site for this string; 2) selects an image that seems appropriate from the algorithmic point of view; 3) calculates the Fourier series; and 4) returns as the result plots of the Fourier series and an interactive version that lets us change the order of the Fourier series. For simplicity, we restrict the program to only single curves. (In the web search, we use the word “silhouette” to mostly retrieve images that are formed by just a single curve.) As the function relies on the result of an external search engine, there is no guarantee that the function will always return the wanted result.

Here are three examples showing the function at work. We build the Fourier series for a generic spider, Spiderman’s spider, a couple dancing the tango, and a mermaid. (Evaluating these functions might give different results, as the search engine results might change over time.)

So far, the initial line segments were computed from images. But we can also start with hand-drawn curves. Assume we want a formula for Isaac Newton. As I am not good at drawing faces, I cheated a bit and used the curve drawing tool to draw characteristic facial and hair curves on top of an image of Sir Isaac. (For algorithmic approaches on how to extract feature lines from faces, see the recent paper by Zhao and Zhu.) Here is the image that we will use:

Fortunately, small random wiggles in the hand-drawn curve will not matter, as they will be removed by omitting higher-order terms in the Fourier series.

To better see the hand-drawn curves, we separate the image and the lines.

This time, we have 16 segments. We build their Fourier series.

And here are again various approximation orders of the resulting curves.

We use different series orders for the various segments. For the hair, we use relatively high orders, and for the eyes relatively low orders. This will make sure that the resulting equations for the face will not be larger than needed.

Here are the first 50 approximations shown in one graphic with decreasing opacity of the curves, and with each set of corresponding curve segments shown in a different color.

This gives the following final curve for Sir Isaac’s portrait.

And this is the plotted form of the last formula.

This ends today’s discussion about how to make curves that resemble people’s faces, fictional characters, animals, or other shapes. Next time, we will discuss the endless graphics capabilities that arise from these formulas and how you can use these types of equations in a large variety of images.

To interact with the examples above, first, download the Wolfram CDF Player. Then, download this post as a Computable Document Format (CDF) file.

Attendees of all experience levels joined the event to gain new insights on how to get the most out of our technologies, including Mathematica‘s Predictive Interface, CDF and EnterpriseCDF, Wolfram SystemModeler, and more.

Miss a talk or couldn’t attend? Not a problem. Visit the conference’s Resource Center for videos, downloadable presentation notebooks, and additional materials.

Stay tuned to the Wolfram Blog for announcements about future Wolfram events.

Skin tones and appearances vary, and so do flavors of skin detectors. The detector I wanted to develop is based on probabilistic models of pixel colors. For each pixel of an image given as input, the skin detector provides a probability that the pixel color belongs to a skin region.

Let’s first look at our detection task wearing probability goggles. We would like to estimate the probability that a pixel belongs to skin given its color. Using Bayes’s rule, this can be expressed as (we’ll call this equation 1):

Please note that in this blog post, probabilities are denoted with an uppercase P[.].

The three terms on the right-hand side of this equation can be expressed in computable forms. That’s where the image processing goggles and these two training datasets will come in handy: one that consists of pixels that belong to skin regions, and one that consists of pixels that do not. We will train a statistical model and derive a computable formula for P[color|skin]. Estimating a priori P[skin] can be arbitrary in the sense that it depends on the final application that the skin detector is built for; we will make a simple choice and estimate the a priori probability of skin as the proportion of skin pixels from the two training datasets. The third term, P[color], is at first more problematic because robustly modeling the probabilities of every possible color requires an immense training dataset. Thankfully, the law of total probability allows us to work around this problem by decomposing this term as:

Finally, our probabilistic skin detector will be an implementation of this formula (we’ll call this equation 2):

Let’s now put on image processing goggles, create the two required datasets, and train our probabilistic models from it.

I know firsthand that creating datasets of images is a fine art that requires a lot of dedicated effort. Frankly, this time we won’t try so hard, gathering a mere dozen images containing skin regions instead. It’s likely about a few hundred images short of being able to derive statistically meaningful results with complex skin models!

Using Mathematica‘s Graphics menu, I can manually select regions with skin in an image and repeat the process for the images of the datasets—admittedly not the best part of my day:

There have been reports that the standard RGB color space is not the best color space to implement models of skin and non-skin colors. Most of the difficulties arise from how different skin can look like depending, for example, on complexion or changes in illuminating conditions.

As you can see, the skin and non-skin colors (displayed in red and green, respectively) in the datasets do not overlap much, but they do span a large portion of the RGB 3D space:

Let’s experiment with another color space where such changes in skin appearance are supposedly modeled more robustly.

We can transfer to the CIE XYZ color space and then decompose a skin color into two chromaticity coordinates and one luminance coordinate. To build models more robust to changes in illuminating conditions, we only keep the two chromaticity coordinates, x and y, defined as x = X / (X + Y + Z) and y = Y / (X + Y + Z):

This function colorConvertxy gives a two-channel xy image:

The extraction of the list of skin xy pairs is performed seamlessly using the image, the mask of skin regions, and the functions PixelValuePositions and PixelValue. Similarly, one can extract the list of non-skin xy pairs.

In the 2D xy chromaticity space, the skin coordinates for all the training images are not as widespread as in the RGB color space, and the distinction between skin and non-skin is more apparent. In the following representation of the skin and non-skin color distributions, skin colors are displayed in shades of red and non-skin regions in shades of green:

Working in the 2D xy chromaticity plane looks promising. Let’s move ahead and implement data-based statistical models for equation 2. For reading convenience, here is the formula again:

The proportion of skin pixels is about 13% in the million-pixel training datasets. We’ll keep this empirical value in our model for the a priori probability P[skin]:

To model the probability density functions of P[color|skin] and P[color|nonskin], a mixture of Gaussian distributions come to mind. These distributions could be suited to the xy data represented above. However, on my laptop computations are a bit snappier when selecting a model based on smooth kernel distributions instead:

Finally, the probability that a given xy color corresponds to skin is implemented as the following skin-ness function:

How does this implemented model perform? Let’s apply the function to a few test images:

Not bad, is it? With this other test image below, it is interesting that the blurred region at the boundary between the red T-shirt and the green foliage has been incorrectly given a very high skin-ness.

We’re nearly done. Actually detecting skin requires deciding whether a pixel belongs to skin. To create such a binary image, we could threshold the probabilistic images we just obtained. How should the threshold be chosen? If the threshold is too high, actual skin pixels may not be detected. If the threshold is too low, non-skin pixels may be incorrectly detected. This is moving on to receiver operating characteristics (ROC) graphs and analysis, and while it would be a pleasant recreation with Mathematica, I feel it deserves more attention. Another blog post, another time.

An alternative strategy is to compare the probabilities of a pixel being skin or non-skin. One last time, let’s wear the probability goggles and revisit equation 1: P[skin|color] == P[color|skin]*P[skin]/P[color]. Similarly, we can express the a posteriori probability of not belonging to a skin region:

To classify a color as skin or as non-skin, we just need to find the greater of the two a posteriori probabilities. The term P[color] can be eliminated, and thereby a pixel will be detected as skin if: P[color|skin]*P[skin] > P[color|nonskin]*P[nonskin].

Here comes today’s skin detector:

A quick assessment on test images suggests it can perform quite well:

Let’s leave this topic for now, and wrap it up with a skin detector app:

My initial intention was to build a skin detector for the mere pleasure of exploring the interplay between probability, statistics, and image processing. Pretty much every step that leads to our final skin detector (e.g. training dataset, choice of statistical distributions, classifier, quantitative assessment) can be studied further and improved rather easily using the large breadth of Mathematica‘s features.

Download this ZIP file which includes the blog post as a Computable Document Format (CDF) file and a corresponding notebook with all the needed code.

One set of new capabilities that Finance Platform 2 introduces is a major enhancement to the way financial analysis is deployed: automated report generation.

Report Generation allows you to create documents quickly and easily using Wolfram Finance Platform documents. Since Report Generation is built on Finance Platform‘s Computable Document Format interface, it’s easy to add it into your normal workflow.

Data for the report can come from a variety of sources, such as the result of a computation, a database query, or Finance Platform‘s integrated computation data source or integrated market data streams. Portfolio performance, risk analyses, and market/economic outlook are just a few of the applications that can take advantage of Report Generation.

How it all works:

Report Generation uses templates to define the various elements that will appear in a report. Think of the template as a blueprint, containing the style and structure of your report, as well as instructions for how data is inserted into the report.

Within the template there are two types of special markers for information in the report: template variables and evaluation expressions.

Template variables are temporary placeholders that are replaced by expressions such as text, graphics, and function names when the report is generated. The specific expression that’s inserted for each template variable is specified when the report is generated, so that the same template can be used in any number of applications.

Evaluation expressions, on the other hand, are code snippets that are evaluated automatically when the report is generated. These are useful for including additional information with the report, such as the date on which it was run, without having to specify the value manually.

The data in your report can be organized in a hierarchical manner using template groups, which can generate nested subgroups in your report without having to copy and paste sections in the report template. This also makes report templates highly scalable, requiring no modification to expand the scope of an application.

Report Generation also includes options for input cells that allow you to evaluate code when generating the report, hide the input code after evaluation, or even delete code after results have been generated.

When a report template is complete, the ReportGenerate function allows you to generate reports using the template on demand, or it can be scheduled to run automatically.

To run a report, simply provide ReportGenerate the name of the template you’d like to use and the replacement rules for any template variables included in the report.

ReportGenerate also takes an optional argument for an output file, and generated reports can be exported in a variety of different formats, including CDF, PDF, and HTML.

You can find more information on all the features of Wolfram Finance Platform 2 at the product page.

Join our Virtual Seminar showcasing all of the new functionalities and features »

Within months, he passed away, probably from hepatic amoebiasis. His final notebook was sent by the University of Madras to G. H. Hardy, who in turn gave it to mathematician G. N. Watson. When Watson died in 1965, the college chancellor found the notebook in his office while looking through papers scheduled to be incinerated. George Andrews rediscovered the notebook in 1976, and it was finally published in 1987. Bruce Berndt and Andrews wrote about Ramanujan’s Lost Notebook in a series of books (Part 1, Part 2, and Part 3). Berndt said, “The discovery of this ‘Lost Notebook’ caused roughly as much stir in the mathematical world as the discovery of Beethoven’s tenth symphony would cause in the musical world.”

In his book analyzing Ramanujan’s results, Berndt notes the existence of a solution for , but follows with, “We do not record the value here, because it is not particularly elegant.” As we will show below, a solution exists as elegant as other values found by Ramanujan himself.

What does the equation mean? We start by comparing arithmetic sequences to geometric sequences.

Arithmetic: 1 + 2 + 3 + … + n.

Geometric: a¹ + a² + a³ + … + aⁿ.

For each type, we can predict behaviors with such things as partial sum formulas. Another form of arithmetic progression, in the realm of continued fractions, is the following:

where symbol corresponds to the Mathematica function ContinuedFractionK.

The geometric version of continued fractions is known as the Rogers–Ramanujan function R. There is a related Rogers–Ramanujan function S (after Leonard James Rogers, who published papers with Ramanujan in 1919). In the lost notebook, F(q) represents S(q).

R(q) is a continued fraction of the form:

and similarly for S(q). (The presence of the prefactor makes various formulas nicer.) More formal definitions are as follows:

These functions are related by . Many published works mention S(q) = -R(-q), but that’s incorrect due to branch cuts. We can also define R and S in a way that can be evaluated more quickly through q-Pochhammer symbols.

Here are pictures of the behavior of the R function on the unit disk in the complex plane. Values returned can be complex, so these pictures show the imaginary, real, argument, and absolute values (Im, Re, Arg, and Abs) of the function R(q). The unit circle itself is the natural boundary of analyticity and has a dense set of singularities of the function R(q). As one can see, the Roger–Ramanujan functions are beautiful, not just due to their mathematical properties, but also visually.

The functions R and S are two of the few named functions devoted to continued fractions. Recently, we’ve been collecting theorems and formulas for R and S, including the uncompleted ones in this piece of Ramanujan’s original “lost” notebook. That line at the end is equivalent to .

Many of these have been found since Ramanujan wrote them down. All of these are readily solved with Mathematica. We list the values together with the first known solvers, with solutions by Oleg Marichev being first realized by Mathematica.

Bruce Berndt noted, “The value of can be determined by using the value of along with a famous modular equation connecting R(q⁵) with R(q). We do not record the value here, because it is not particularly elegant.”

With Simplify, RootReduce, and many other Mathematica functions, large equations can be boiled down to their most elegant form. Ramanujan used chalk and his mind to simplify most of his results—the long results he erased from his slate, but the elegant results he wrote down. It seems likely to us that Ramanujan actually did know the elegant solution, or at least a method to find it, he just didn’t have the time to write it down. Here’s a method we used. First, calculate a numerical value for the point of interest. Second, conjecture a closed algebraic form for this number. Third, express the algebraic number as nested radicals. Finally, check the conjectured form with many digits of accuracy.

Then we check that the numerical value of the conjectured form is the same as the value of the function. The values agree to at least 10000 places.

Since both of these are algebraic numbers with elegant representations, this is a rather convincing check. And the method can easily be generalized to find many more, so far unknown, values for S(q), and similarly for R(q).

An actual proof can be accomplished using modular equations. This is the modular equation of order 5 for S:

We use the previously known value for for S(q⁵) and solve for S(q) to obtain a value for .

Clearing denominators, we obtain the above form of the result.

Ramanujan’s equations are related to work we’ve done recently to add a lot of continued fraction knowledge to Wolfram|Alpha. In a future blog we will expand on the new capabilities, such as the input continued fraction K (1, n, {n, 1, inf}).

We also put together a list of hundreds of exact values in the “Ramanujan R and S” interactive Demonstration.

“Not particularly elegant”—never a good thing to say about Ramanujan. We’re glad we were able to show that Ramanujan had something elegant in mind.

Download this post as a Computable Document Format (CDF) file.

One of Mathematica 9′s new and exciting features is out-of-core image processing. What does the out-of-core term really mean? It is a way to process very large images that are too big to fit into main memory. Let’s say we have a machine with 8 GB of RAM, and let’s assume that Mathematica can use up to 7.2 GB of that memory (the remaining 0.8 GB will be used by the operating system). Freshly started, Mathematica 9 on Windows 8 takes up about 200 MB of memory, so the kernel can use about 7 GB of RAM. What is the maximal size of the image that we can load into the kernel (we don’t want to visualize it at this point)? If we assume that the image is in the RGB color space and a single byte encoding, then the following formula gives a maximal width (and height) of an image that can be loaded at once into the memory:

50 k by 50 k is a very large image, but remember that we haven’t left any memory for processing and visualization. If we want to perform some operations on that image and visualize the results, then the realistic memory limit would be something around 0.5 GB. Why only 0.5 GB? We should leave at least 2 GB for the front end, and we have to keep in mind that any nontrivial image-processing algorithm requires additional storage for temporary objects. So, when we restrict a single image storage to 0.5 GB, then the maximal dimensions would be:

The above value is a good approximation provided that we only use image processing functions that preserve byte encoding and do not require internal conversion to Real32 or Real type. One example of such a function is ColorNegate. What if we want to call ImageConvolve? Then, not only the internal computations have to be performed using floating points, but also the resulting image is stored in the Real type (8 bytes per value). Our memory limit decreases to something like 0.1 GB, and the maximal width and resolution (in megapixels) become:

Thus, to do image processing on images larger than about 36 megapixels, we need out-of-core processing. Before we show how to do out-of-core processing on an image, let’s write some code that will allow us to visualize large images. In this post we will use an image from the NASA Visible Earth collection. It is a 222 megapixel JPEG file:

Let’s start with inspecting the data and visualizing a thumbnailed version of our large image. The function LargeImageInspect allows us to specify the range of rows and columns of the image that we want to see. Notice that we use the “FirstRow” and “LastRow” options of the Import function that were introduced in Mathematica 9:

Now, let’s create a function that shows a thumbnail of the large image. The function LargeImageThumbnail takes two arguments: a file path and optionally the width of a thumbnail image. It uses Import with the “FirstRow” and “LastRow” options to read a strip of a large image, re-sizes it according to the value of the second argument, and places the result in a list. At the end, ImageAssemble is called on that list to create a final thumbnail.

To explore large images, we need one more utility—an out-of-core image viewer that will allow us to visualize actual pixel values stored in a file. Here I implement the function LargeImageViewer that shows the thumbnail of a large image in the top pane, and a part of the actual image data in the bottom pane. To see the implementation and all the code, please download this post as a Computable Document Format (CDF) file.

Now that we know how to explore large images, let’s take a closer look at the out-of-core image processing functions. Mathematica 9 offers three functions (ImageFileApply, ImageFileFilter, and ImageFileScan) that allow processing of large images in chunks and provide file-to-file workflow. All three functions work with TIFF, JPEG, and PNG files.

ImageFileApply is an out-of-core equivalent of the ImageApply function introduced in Mathematica 7. It applies a function specified as a first argument to the list of channel values for each pixel of the image.

The above function call is equivalent to Export[outputFile, ColorNegate[Import[inputFile]]], however, ImageFileApply reads, processes, and writes successive blocks of image data so it requires significantly less memory than the in-core workflow. Here is the proof. The following code shows that ImageFileApply can be successfully executed when memory is constrained to 100 MB. Notice that in the call to ImageFileApply we used a non-default value of the “MaxBlockSize” option that defines the maximal size of image blocks in pixels. The default value of 8000000 is too large to run computations with the 100 MB memory constraint.

Trying to do the same thing with the in-core image processing functionality fails because it runs out of memory.

ImageFileApply‘s function parameter can be arbitrarily complex as long as it yields a number or a list of numbers. Here, we create a “blue snow” effect:

ImageFileApply is a pixel operation and cannot be used for neighborhood operations such as filters. For this purpose, Mathematica 9 has a function called ImageFileFilter. We will use that function on an image from the ExampleData paclet:

A simple example of ImageFileFilter is to replace every value by the minimum in its range-r neighborhood (MinFilter). In the example below we have not specified the output file. In that case, both ImageFileApply and ImageFileFilter create an output file in the user directory with a name that is formed from the input file name by appending a time stamp.

A popular Gaussian filtering is not much harder than the MinFilter. We create a kernel using the GaussianMatrix function, and then we sum up a product of that kernel with a neighborhood of each pixel, that is, Total[ker #,2]&:

The last function that I want to discuss here is ImageFileScan. It is an out-of-core equivalent of ImageScan (also new in Mathematica 9) that evaluates a given function applied to each pixel of an image. Since ImageFileScan discards the results and does not create any file, it is useful in carrying out operations that have a “side effect.” For example, we can use this function to obtain different image statistics such as minimum and maximum values or image histograms:

All three out-of-core functions have one more interesting option called “ImageList”. It allows us to specify which frames in a multiframe TIFF file should be processed. In this example we color negate only the second and third frame of a multiframe image:

The three functions that I have described here—ImageFileApply, ImageFileFilter, and ImageFileScan—are just the beginning of our work on out-of-core image processing. Ultimately, in future versions of Mathematica, the file-to-file workflow will be possible for all built-in image processing functions.

Download this post as a Computable Document Format (CDF) file.

We’re very interested in academic data. Over the past 20 years or so, publishers have changed in some big ways, such as shifting from print to online or adopting new open access business models. But one thing they haven’t fully tackled yet is how to handle the increasingly large amounts of data coming out of academic research.

The Wolfram technology stack already has many of the necessary components for putting research data online in a useful form. Mathematica can import and export a large number of file formats. We have data file upload and powerful data analytics in Wolfram|Alpha Pro. Plus we have the technology for quickly making easy-to-use online interactive widgets.

We’re working on a few more things that will let us make a more comprehensive contribution to publishing and sharing academic data. Come to the Oxford meeting to hear more.

A few weeks ago we decided to start analyzing all this data. And I have to say that if nothing else it’s been a terrific example of the power of Mathematica and the Wolfram Language for doing data science. (It’ll also be good fodder for the Data Science course I’m starting to create.)

We’d always planned to use the data we collect to enhance our Personal Analytics system. But I couldn’t resist also trying to do some basic science with it.

I’ve always been interested in people and the trajectories of their lives. But I’ve never been able to combine that with my interest in science. Until now. And it’s been quite a thrill over the past few weeks to see the results we’ve been able to get. Sometimes confirming impressions I’ve had; sometimes showing things I never would have guessed. And all along reminding me of phenomena I’ve studied scientifically in A New Kind of Science.

So what does the data look like? Here are the social networks of a few Data Donors—with clusters of friends given different colors. (Anyone can find their own network using Wolfram|Alpha—or the SocialMediaData function in Mathematica.)

So a first quantitative question to ask is: How big are these networks usually? In other words, how many friends do people typically have on Facebook? Well, at least for our users, that’s easy to answer. The median is 342—and here’s a histogram showing the distribution (there’s a cutoff at 5000 because that’s the maximum number of friends for a personal Facebook page):

But how typical are our users? In most respects—so far as we can tell—they seem pretty typical. But there are definitely some differences. Like here’s the distribution of the number of friends not just for our users, but also for their friends (there’s a mathematical subtlety in deriving this that I’ll discuss later):

And what we see is that in this broader Facebook population, there are significantly more people who have almost no Facebook friends. Whether such people should be included in samples one takes is a matter of debate. But so long as one looks at appropriate comparisons, aggregates, and so on, they don’t seem to have a huge effect. (The spike at 200 friends probably has to do with Facebook’s friend recommendation system.)

So, OK. Let’s ask for example how the typical number of Facebook friends varies with a person’s age. Of course all we know are self-reported “Facebook ages”. But let’s plot how the number of friends varies with that age. The solid line is the median number of friends; successive bands show successive octiles of the distribution.

After a rapid rise, the number of friends peaks for people in their late teenage years, and then declines thereafter. Why is this? I suspect it’s partly a reflection of people’s intrinsic behavior, and partly a reflection of the fact that Facebook hasn’t yet been around very long. Assuming people don’t drop friends much once they’ve added them one might expect that the number of friends would simply grow with age. And for sufficiently young people that’s basically what we see. But there’s a limit to the growth, because there’s a limit to the number of years people have been on Facebook. And assuming that’s roughly constant across ages, what the plot suggests is that people add friends progressively more slowly with age.

But what friends do they add? Given a person of a particular age, we can for example ask what the distribution of ages of the person’s friends is. Here are some results (the jaggedness, particularly at age 70, comes from the limited data we have):

And here’s an interactive version, generated from CDF:

The first thing we see is that the ages of friends always peak at or near the age of the person themselves—which is presumably a reflection of the fact that in today’s society many friends are made in age-based classes in school or college. For younger people, the peak around the person’s age tends to be pretty sharp. For older people, the distribution gets progressively broader.

We can summarize what happens by plotting the distribution of friend ages against the age of a person (the solid line is the median age of friends):

There’s an anomaly for the youngest ages, presumably because of kids under 13 misreporting their ages. But apart from that, we see that young people tend to have friends who are remarkably close in age to themselves. The broadening as people get older is probably associated with people making non-age-related friends in their workplaces and communities. And as the array of plots above suggests, by people’s mid-40s, there start to be secondary peaks at younger ages, presumably as people’s children become teenagers, and start using Facebook.

So what else can one see about the trajectory of people’s lives? Here’s the breakdown according to reported relationship status as a function of age:

And here’s more detail, separating out fractions for males and females (“married+” means “civil union”, “separated”, “widowed”, etc. as well as “married”):

There’s some obvious goofiness at low ages with kids (slightly more often girls than boys) misreporting themselves as married. But in general the trend is clear. The rate of getting married starts going up in the early 20s—a couple of years earlier for women than for men—and decreases again in the late 30s, with about 70% of people by then being married. The fraction of people “in a relationship” peaks around age 24, and there’s a small “engaged” peak around 27. The fraction of people who report themselves as married continues to increase roughly linearly with age, gaining about 5% between age 40 and age 60—while the fraction of people who report themselves as single continues to increase for women, while decreasing for men.

I have to say that as I look at the plots above, I’m struck by their similarity to plots for physical processes like chemical reactions. It’s as if all those humans, with all the complexities of their lives, still behave in aggregate a bit like molecules—with certain “reaction rates” to enter into relationships, marry, etc.

Of course, what we’re seeing here is just for the “Facebook world”. So how does it compare to the world at large? Well, at least some of what we can measure in the Facebook world is also measured in official censuses. And so for example we can see how our results for the fraction of people married at a given age compare with results from the official US Census:

I’m amazed at how close the correspondence is. Though there are clearly some differences. Like below age 20 kids on Facebook are misreporting themselves as married. And on the older end, widows are still considering themselves married for purposes of Facebook. For people in their 20s, there’s also a small systematic difference—with people on Facebook on average getting married a couple of years later than the Census would suggest. (As one might expect, if one excludes the rural US population, the difference gets significantly smaller.)

Talking of the Census, we can ask in general how our Facebook population compares to the US population. And for example, we find, not surprisingly, that our Facebook population is heavily weighted toward younger people:

OK. So we saw above how the typical number of friends a person has depends on age. What about gender? Perhaps surprisingly, if we look at all males and all females, there isn’t a perceptible difference in the distributions of number of friends. But if we instead look at males and females as a function of age, there is a definite difference:

Teenage boys tend to have more friends than teenage girls, perhaps because they are less selective in who they accept as friends. But after the early 20s, the difference between genders rapidly dwindles.

What effect does relationship status have? Here’s the male and female data as a function of age:

In the older set, relationship status doesn’t seem to make much difference. But for young people it does. With teenagers who (mis)report themselves as “married” on average having more friends than those who don’t. And with early teenage girls who say they’re “engaged” (perhaps to be able to tag a BFF) typically having more friends than those who say they’re single, or just “in a relationship”.

Another thing that’s fairly reliably reported by Facebook users is location. And it’s common to see quite a lot of variation by location. Like here are comparisons of the median number of friends for countries around the world (ones without enough data are left gray), and for states in the US:

There are some curious effects. Countries like Russia and China have low median friend counts because Facebook isn’t widely used for connections between people inside those countries. And perhaps there are lower friend counts in the western US because of lower population densities. But quite why there are higher friend counts for our Facebook population in places like Iceland, Brazil and the Philippines—or Mississippi—I don’t know. (There is of course some “noise” from people misreporting their locations. But with the size of the sample we have, I don’t think this is a big effect.)

In Facebook, people can list both a “hometown” and a “current city”. Here’s how the probability that these are in the same US state varies with age:

What we see is pretty much what one would expect. For some fraction of the population, there’s a certain rate of random moving, visible here for young ages. Around age 18, there’s a jump as people move away from their “hometowns” to go to college and so on. Later, some fraction move back, and progressively consider wherever they live to be their “hometown”.

One can ask where people move to and from. Here’s a plot showing the number of people in our Facebook population moving between different US states, and different countries:

There’s a huge range of demographic questions we could ask. But let’s come back to social networks. It’s a common observation that people tend to be friends with people who are like them. So to test this we might for example ask whether people with more friends tend to have friends who have more friends. Here’s a plot of the median number of friends that our users have, as a function of the number of friends that they themselves have:

And the result is that, yes, on average people with more friends tend to have friends with more friends. Though we also notice that people with lots of friends tend to have friends with fewer friends than themselves.

And seeing this gives me an opportunity to discuss a subtlety I alluded to earlier. The very first plot in this post shows the distribution of the number of friends that our users have. But what about the number of friends that their friends have? If we just average over all the friends of all our users, this is how what we get compares to the original distribution for our users themselves:

It seems like our users’ friends always tend to have more friends than our users themselves. But actually from the previous plot we know this isn’t true. So what’s going on? It’s a slightly subtle but general social-network phenomenon known as the “friendship paradox”. The issue is that when we sample the friends of our users, we’re inevitably sampling the space of all Facebook users in a very non-uniform way. In particular, if our users represent a uniform sample, any given friend will be sampled at a rate proportional to how many friends they have—with the result that people with more friends are sampled more often, so the average friend count goes up.

It’s perfectly possible to correct for this effect by weighting friends in inverse proportion to the number of friends they have—and that’s what we did earlier in this post. And by doing this we determine that in fact the friends of our users do not typically have more friends than our users themselves; instead their median number of friends is actually 229 instead of 342.

It’s worth mentioning that if we look at the distribution of number of friends that we deduce for the Facebook population, it’s a pretty good fit to a power law, with exponent -2.8. And this is a common form for networks of many kinds—which can be understood as the result of an effect known as “preferential attachment”, in which as the network grows, nodes that already have many connections preferentially get more connections, leading to a limiting “scale-free network” with power-law features.

But, OK. Let’s look in more detail at the social network of an individual user. I’m not sufficiently diligent on Facebook for my own network to be interesting. But my 15-year-old daughter Catherine was kind enough to let me show her network:

There’s a dot for each of Catherine’s Facebook friends, with connections between them showing who’s friends with whom. (There’s no dot for Catherine herself, because she’d just be connected to every other dot.) The network is laid out to show clusters or “communities” of friends (using the Wolfram Language function FindGraphCommunities). And it’s amazing the extent to which the network “tells a story”. With each cluster corresponding to some piece of Catherine’s life or history.

Here’s a whole collection of networks from our Data Donors:

No doubt each of these networks tells a different story. But we can still generate overall statistics. Like, for example, here is a plot of how the number of clusters of friends varies with age (there’d be less noise if we had more data):

Even at age 13, people typically seem to have about 3 clusters (perhaps school, family and neighborhood). As they get older, go to different schools, take jobs, and so on, they accumulate another cluster or so. Right now the number saturates above about age 30, probably in large part just because of the limited time Facebook has been around.

How big are typical clusters? The largest one is usually around 100 friends; the plot below shows the variation of this size with age:

And here’s how the size of the largest cluster as a fraction of the whole network varies with age:

What about more detailed properties of networks? Is there a kind of “periodic table” of network structures? Or a classification scheme like the one I made long ago for cellular automata?

The first step is to find some kind of iconic summary of each network, which we can do for example by looking at the overall connectivity of clusters, ignoring their substructure. And so, for example, for Catherine (who happened to suggest this idea), this reduces her network to the following “cluster diagram”:

Doing the same thing for the Data Donor networks shown above, here’s what we get:

In making these diagrams, we’re keeping every cluster with at least 2 friends. But to get a better overall view, we can just drop any cluster with, say, less than 10% of all friends—in which case for example Catherine’s cluster diagram becomes just:

And now for example we can count the relative numbers of different types of structures that appear in all the Data Donor networks:

And we can look at how the fractions of each of these structures vary with age:

What do we learn? The most common structures consist of either two or three major clusters, all of them connected. But there are also structures in which major clusters are completely disconnected—presumably reflecting facets of a person’s life that for reasons of geography or content are also completely disconnected.

For everyone there’ll be a different detailed story behind the structure of their cluster diagram. And one might think this would mean that there could never be a general theory of such things. At some level it’s a bit like trying to find a general theory of human history, or a general theory of the progression of biological evolution. But what’s interesting now about the Facebook world is that it gives us so much more data from which to form theories.

And we don’t just have to look at things like cluster diagrams, or even friend networks: we can dig almost arbitrarily deep. For example, we can analyze the aggregated text of posts people make on their Facebook walls, say classifying them by topics they talk about (this uses a natural-language classifier written in the Wolfram Language and trained using some large corpora):

Each of these topics is characterized by certain words that appear with high frequency:

And for each topic we can analyze how its popularity varies with (Facebook) age:

It’s almost shocking how much this tells us about the evolution of people’s typical interests. People talk less about video games as they get older, and more about politics and the weather. Men typically talk more about sports and technology than women—and, somewhat surprisingly to me, they also talk more about movies, television and music. Women talk more about pets+animals, family+friends, relationships—and, at least after they reach child-bearing years, health. The peak time for anyone to talk about school+university is (not surprisingly) around age 20. People get less interested in talking about “special occasions” (mostly birthdays) through their teens, but gradually gain interest later. And people get progressively more interested in talking about career+money in their 20s. And so on. And so on.

Some of this is rather depressingly stereotypical. And most of it isn’t terribly surprising to anyone who’s known a reasonable diversity of people of different ages. But what to me is remarkable is how we can see everything laid out in such quantitative detail in the pictures above—kind of a signature of people’s thinking as they go through life.

Of course, the pictures above are all based on aggregate data, carefully anonymized. But if we start looking at individuals, we’ll see all sorts of other interesting things. And for example personally I’m very curious to analyze my own archive of nearly 25 years of email—and then perhaps predict things about myself by comparing to what happens in the general population.

Over the decades I’ve been steadily accumulating countless anecdotal “case studies” about the trajectories of people’s lives—from which I’ve certainly noticed lots of general patterns. But what’s amazed me about what we’ve done over the past few weeks is how much systematic information it’s been possible to get all at once. Quite what it all means, and what kind of general theories we can construct from it, I don’t yet know.

But it feels like we’re starting to be able to train a serious “computational telescope” on the “social universe”. And it’s letting us discover all sorts of phenomena. That have the potential to help us understand much more about society and about ourselves. And that, by the way, provide great examples of what can be achieved with data science, and with the technology I’ve been working on developing for so long.

My obsession with diagrams and simple line drawings began almost unnoticeably in the winter of 2003 in New York City after attending an exhibition at The Metropolitan Museum of Art: “the first comprehensive survey of Leonardo da Vinci’s drawings ever presented in America.” You may think it’d be a drag—crowds marching very slowly in a single long line coiling through the exhibition hallways. But perception of time transforms when you stare at 500-year-old craft. I think it was then that it started to dawn on me what special value a first sketch has. A first act when an idea, something very subjective, evasive, living solely inside one’s mind, materializes as a solid reality, now perceivable by another human being. Imagine it happened ages ago. Wouldn’t you be curious what was going on at that moment in time, what got frozen in this piece of craft in front of you?

Cave Paintings, Paleolithic age

Vitruvian Man, Leonardo da Vinci, circa 1490

People have been expressing things visually since immemorial times. The oldest things we can find, like cave paintings, are dated to 40,000 years ago. And we still keep doodling on napkins, sand, and touch screens—solidifying ideas. There is a tremendous body of work, celebrated or little known, that has accumulated through the ages in libraries, antique book shops, private collections, ruins, burial sites, forgotten attics. These abiding footprints of imagination possess striking visual power that survived their creators. They fuse art and history because the enigma of their stories is no less elegant than their art. I am being intentionally dramatic (you’ll miss it when we’ll get to coding), because art is a tricky thing. How do you share with others a fascination with subjective matters, understanding gained through time and experience? How do you engage others and transfer your perception? Well, maybe you can’t do it directly, but you can still orchestrate how others encounter art. For example, instead of showing an image, you can make it a discovery process. An image is a mystery until we find a meaning and learn its story. I wonder if this can be a metaphor for the image presentation. Can we tease, intrigue beholders with a conundrum and set them on a quest? The way we comprehend art echoes the way the art was created in the first place. Everything is a process, everything has a story, even merely finding an interesting image lost in the pages of an old manuscript. By analogy, we turn presentation into a process—so an encounter with an image becomes more personal. With longer positive experience comes deeper immersion. Jigsaw puzzles are a perfect example.

So now that we’re done with drama and profundities, how can we get to action? Recently I was lucky to come across the brilliant iOS game Blueprint3D by FDG Entertainment, which has been critically acclaimed and praised for its originality. The app shatters blueprint drawings into myriad pieces scattered almost randomly in 3D space—“almost” because there is a single correct perspective under which the fragments suddenly assemble back into the original drawing. The game play is to find this perspective by rotating a 3D cloud of scattered pieces. The images seemed to be created by the company developers à la simple technical blueprints. While I was indulging in several game rounds, it suddenly struck me—this would make a perfect game for “actual” historical drawings that I enjoy so much! And it would be wonderful to have a short story behind the images in the game too. I had no idea about the algorithms that the app used and how to bypass the many challenges of complete app development. But I really wanted to make it work. And what do you do when you just can’t wait for something to happen?

I launch Mathematica. For me it’s the shortest path between A and B, idea and result, a universal beeline for almost every task. And so, without further ado, below is the result of my implementation in Mathematica of Blueprint3D’s idea. It is far from its final polished form, but was achieved in a surprisingly short time (less than an hour) and with a small amount of code (see the attached notebook). Rotate 3D graphics with your mouse until you reveal a hidden image. The 3D graphic object has different rotation modes depending on whether you drag it with your mouse from the middle or from the corners. The “hint” control can help you to navigate in the space of 3D rotations: the puzzle is solved when the green and red dots overlap. Though “hint” is very approximate, it helps to understand if the correct up-down and left-right orientation of the image is found. It is important especially when images have text.

Please let me know in the comments which images you found to be most interesting. And now let’s go over major ideas and challenges of this app.

The game mechanics

So how does it work? We start from texture mapping a square image onto a large square polygon drawn in 3D space. The whiter the pixels are in the original image, the more transparent they are on the polygon. We then subdivide the square polygon into hundreds of small random polygons with the help of Voronoi tessellation that covers everything without gaps. It is very easy to get Voronoi tessellation in Mathematica, and I will use the following function:

The next step is to apply random geometric transformations to each small polygon. We will stretch, rotate, and scatter them around in 3D space in almost perfect chaos, but with one restriction: there is a single direction in space along which things happen in a similar way for all polygons. This is where we will hide the image. Now for a few more details.

The age of interactive blogging

The technology behind this app is CDF, Computable Document Format, which we embedded in the blog web page. Electronic publishing is old news. Please meet interactive blogging. We’ve been practicing this with the Wolfram Blog for a while already, but it is truly a new paradigm. For a blogger, it allows the new freedom to be able to sculpt ideas not only in writing, but with an app, a puzzle, a discovery tool: to be able to capture your readership within a new, more engaging dimension. CDF is powerfully flexible. For instance, I wanted to keep an expandable library of images that I could add to and remove from without having to re-deploy the CDF app every time on the site. So I decided to store images and their descriptions on a server and make the app access them via URLs. Such advanced CDF deployment features are available in Mathematica Enterprise Edition.

The magic of HistogramTransform

As I mentioned already, I needed a pure white image background for it to be transparent in the chaotic cloud of polygons. But the images I found online had various non-white backgrounds, especially antique ones that had heavy yellowish-gray backgrounds like the top images in this post. And though I could convert to gray levels, opacity of the background was a problem. It needed to be completely transparent for the game to be challenging, so one can see through and get visually overwhelmed by a random mixture of line fragments. If the background is opaque, then the polygon boundaries of the fragments will be visible, and it will be much easier and less interesting to solve the puzzle. After trying to code a few image processing techniques on background removal, I realized that, as it often happens in Mathematica, there is a suitable function for that. In the latest version we introduced the almost magical HistogramTransform, which can make one image look like another by matching their histograms. The following code compares an original image, the same image converted to “Grayscale,” and the image converted via HistogramTransform to look like a sample image:

You can clearly see that simple color conversion to “Grayscale” is not enough to remove heavy background residue. But if we find another image (here the celebrated Benson Lossing Reverse) with a mostly clean background, then the result after HistogramTransform is incredibly clear. It also nicely preserves various intensities of gray levels corresponding to different pencil pressure or ink concentration in the paper. This simple “statistical” method based on probability distributions for image pixels is much better than my custom coding based on blunt morphological and threshold operations. White backgrounds will become transparent in a 3D fragment cloud.

Hiding power of ShearingTransform

The main trick in this application is to figure out how to hide the original 2D image in a single 3D direction or line of sight among infinitely many others. I had no knowledge of the algorithm behind Blueprint3D and had to come up with my own. I suspected that something like ShearingTransform was the key. This particular GeometricTransformation function in Mathematica has quite a few variables. In the code below I illustrate their action. We can start from any 3D graphics object, in this case a hexagon, and transform it in such a way that it stretches only along a particular direction (red arrow), perpendicular to some “normal” (yellow arrow), and under a certain “shear” angle. There is also a point we can choose—marked green—that always stays fixed. Try playing with various controls and rotating 3D graphics. This transformation has an interesting hiding property: if you look directly from above or below (parallel to red arrow) from far away, all sheared polygons will look the same, hiding behind each other.

Enigma of a shattered image

The red arrow designates that unique single perspective where we will hide the 2D image. If we break an image into fragments and shear them under different angles and normals, but keep the sheering direction the same, the view from below or above will give us the whole unbroken image. We can illustrate this with an app below. The polygon subdivision was done with the help of Voronoi diagrams. Try controls and rotating 3D graphics below. As you can see, we also displaced polygons at random heights. Note how looking “very close above” leaves the unhealed gaps between the polygons and how these gaps are healed when the height is increased. This explains intuitively that we should look “from far” to not see the gaps. This relates to the fact that during shearing, all points of an object move along parallel lines. Don’t forget you can rotate the object below.

You may have noticed that in the game (the very first CDF app above), fragments are scattered inside a cubic volume. This is the simplest solution, which unfortunately reveals the orientation of the cube by its sharp edges. To make the puzzle more challenging, we made the red cloud above to be of a spherical shape, so there are no edges.

When in trouble, dot multiply

I don’t think this game is difficult, especially after some practice. Yet I think some people, especially kids, would appreciate a bit of help, at least sometimes. This is why I added the “hint” option. When red and green spots of the hint overlap, you are close to the solution. It is not a perfect hint, but it does help you to see if the image is flipped from left to right or upside down, or how far you are from a solution. How does this work? If no panning or zooming is involved, just rotation, a unique object orientation in 3D space is controlled by two properties: ViewPoint and ViewVertical. Both are three-element vectors or lists. While we rotate a 3D object, the elements of these lists change. When a vector is aligned with a specific direction in 3D space, its normalized dot product with that direction equals 1. All other alignments will produce different numbers. So these two dot products can be interpreted as coordinates of a point in a 2D plane. A simplest example is shown below—rotate the left 3D cube to understand the relation to the 2D plane.

I keep getting surprised at how easily we can solve various challenges just because Mathematica always has something up its sleeve—it is an amazing collection of algorithms. You can find the complete code of the game in the attached CDF file. No doubt there are many possible improvements. Let me know in the comments about any suggestions, ideas, or your favorite images. I suspect this game can be quite beneficial for kids, giving basic notions about the relation between 2D and 3D space and some history lessons.

Download this post as a Computable Document Format (CDF) file.