A few days ago we built the millionth version of our software products. For the outside world, we recently shipped Mathematica 6.0.2. But internally we’ve now built a million versions of Mathematica and our other products.

I’ve been at Wolfram Research for 17 years, and for the past 13 years I’ve been responsible for our automated product build systems. Every night (and sometimes during the day) a large cluster of computers builds new versions of every product we make.

Building a single Mathematica is a complex process, involving a host of different computer languages and systems, and a final Mathematica contains more than 10,000 separate files.

Each day, at 6pm Central Time, the builds begin (sometimes with great new features that were added in the minutes beforehand). We develop our products in multiple source-control branches. The first step in the builds is to tag the versions of files on each branch that will be used in each build.

Our Build Scheduler reads a symbolic representation of the current dependency graph for all the components that will be used. Then it monitors when each component is moved into place, and assigns different machines in the cluster to build it.

How each component is built depends on what the component is. The source code of Mathematica is written in several languages (not least, Mathematica itself), and the build process must pre-process, compile and post-process each component, then arrange all of them in the final file layout.

All of this has to be done across the many different machine and operating system architectures that Mathematica supports.

If nothing goes wrong, it takes about eight hours for the builds to complete, so around 1am Central Time there are fresh new Mathematicas ready to test.

We’ve always been responsible for building all our products, not just Mathematica. These days, one of the most complex builds is for our in-product and web documentation. Whenever you look at the Wolfram Mathematica Documentation Center, all the nearly 10,000 pages of documentation you see were automatically built using a large build system written in Mathematica that creates HTML and JavaScript files.

Although we had some automation in the build process even for Mathematica 1.0, the first officially recorded build code in the database we use today is build 29, Mathematica 2.2.5 for Windows, built on May 19, 1996 (for Mathematica aficionados: yes, Mathematica 2.2.5 was never shipped).

On January 28, 2000, our build codes reached six digits; build 100001 was of Mathematica 4.1.0.

A few days ago, we reached build 1000000. Needless to say, it’s for a product that hasn’t shipped yet, so I can’t talk about it here.

Every night, after each build completes, it’s put into testing. Each version is checked out into our Quality Assurance group’s cluster, and their system begins to run millions of tests on it.

When I wake up each morning, one of the first things I do is to check our build monitor and see what happened the night before.

It’s all color coded. Green or gray is good; it means builds completed successfully. Pink is not so good. It means something went wrong with a build.

Since we’re probably building one of the most complex software systems on the planet, and we’re always adding new and different elements, it’s not surprising that things sometimes go wrong.

But that’s exactly why we’ve worked so hard to automate our build systems: so we can run builds every day, automatically, and find out what’s wrong immediately.

We have complete configuration management: for a decade, our database has tracked every single component that goes into every single build. With typically 10,000 components per build, and now a million builds, that’s a fairly serious number of different elements.

When we ship a product, like Mathematica, we’re shipping a particular build that is the culmination of our development process, and has a name like 6.0.2. But if you want to get some idea of what’s behind the scenes, the inner ring of a Mathematica CD, or the output of SystemInformation[], shows the raw build code.

Every one of those builds goes through my automated build system… putting what is by now centuries of CPU time into the creation of Mathematica and our other products.