Former Astronaut Creates Virtual Copilot with Wolfram Neural Nets and a Raspberry Pi
August 16, 2018 — Erez Kaminski , Wolfram Technology Specialist, Wolfram Technology Group
For the past two years, FOALE AEROSPACE has been on an exhilarating journey to create an innovative machine learning–based system designed to help prevent airplane crashes, using what might be the most understated machine for the task—the Raspberry Pi. The system is marketed as a DIY kit for aircraft hobbyists, but the ideas it’s based upon can be applied to larger aircraft (and even spacecraft!).
FOALE AEROSPACE is the brainchild of astronaut Dr. Mike Foale and his daughter Jenna Foale. Mike is a man of many talents (pilot, astrophysicist, entrepreneur) and has spent an amazing 374 days in space! Together with Jenna (who is currently finishing her PhD in computational fluid dynamics), he was able to build a complex machine learning system at minimal cost. All their development work was done in-house, mainly using the Wolfram Language running on the desktop and a Raspberry Pi. FOALE AEROSPACE’s system, which it calls the Solar Pilot Guard (SPG), is a solar-charged probe that identifies and helps prevent loss-of-control (LOC) events during airplane flight. Using sensors to detect changes in the acceleration and air pressure, the system calculates the probability of each data point (an instance in time) to be in-family (normal flight) or out-of-family (non-normal flight/possible LOC event), and issues the pilot voice commands over a Bluetooth speaker. The system uses classical functions to interpolate the dynamic pressure changes around the airplane axes; then, through several layers of Wolfram’s automatic machine learning framework, it assesses when LOC is imminent and instructs the user on the proper countermeasures they should take.
You can see the system work its magic in this short video on the FOALE AEROSPACE YouTube channel. As of the writing of this blog, a few versions of the SPG system have been designed and built: the 2017 version (talked about extensively in a Wolfram Community post by Brett Haines) won the bronze medal at the Experimental Aircraft Association’s Founder’s Innovation Prize. In the year since, Mike has been working intensely to upgrade the system from both a hardware and software perspective. As you can see in the following image, the 2018 SPG has a new streamlined look, and is powered by solar cells (which puts the “S” in “SPG”). It also connects to an off-the-shelf Bluetooth speaker that sits in the cockpit and gives instructions to the pilot.
Building the System: Hardware and Data
While the probe required some custom hardware and intense design to be so easily packaged, the FOALE AEROSPACE team used off-the-shelf Wolfram Language functions to create a powerful machine learning–based tool for the system’s software. The core of the 2017 system was a neural network–based classifier (built using Wolfram’s Classify function), which enabled the classification of flight parameters into in-family and out-of-family flight (possible LOC) events. In the 2018 system, the team used a more complex algorithm involving layering different machine learning functions together to create a semi-automatic pipeline. The combined several layers of supervised and unsupervised learning result in a semi-automated pipeline for dataset creation and classification. The final deployment is again a classifier that classifies in-family and out-of-family (LOC) flights, but this time in a more automatic and robust way.
To build any type of machine learning application, the first thing we need is the right kind of data. In the case at hand, what was needed was actual flight data—both from normal flight patterns and from non-normal flight patterns (the latter leading to LOC events). To do this, one would need to set up the SPG system, start recording with it and take it on a flight. During this flight, it would need to sample both normal flight data and non-normal/LOC events, which means Mike needed to intentionally make his aircraft lose control, over and over again. If this sounds dangerous, it’s because it is, so don’t try this at home. During such a flight, the SPG records acceleration and air pressure data across the longitudinal, vertical and lateral axes (x, y, z). From these inputs, the SPG can calculate the acceleration along the axes, the sideslip angle (β—how much it is moving sideways), the angle of attack (α—the angle between the direction of the noise and the horizontal reference plane) and the relative velocity (of the airplane to the air around it)—respectively, Ax, Ay, Az, β, α and Vrel in the following plot:
A plot of the flight used as the training set. Note that the vertical axis is inverted so a lower value corresponds to an increase in quantity.
Connecting the entire system straight to a Raspberry Pi running the Wolfram Language made gathering all this data and computing with it ridiculously easy. Looking again at the plot, we already notice that there is a phase of almost-steady values (up to 2,000 on the horizontal axis) and a phase of fluctuating values (2,000 onward). Our subject matter expert, Mike Foale, says that these correspond to runway and flight time, respectively. Now that we have some raw data, we need to process and clean it up in order to learn from it.
Taking the same dataset, we first remove any data that isn’t interesting (for example, anything before the 2,000th data point). Now we can re-plot the data:
In the 2017 system, the FOALE AEROSPACE team had to manually curate the right flight segments that correspond to LOC events to create a dataset. This was a labor-intensive process that became semi-automated in the 2018 system.
We now take the (lightly) processed data and start applying the needed machine learning algorithms to it. First, we will cluster the training data to create in-family and out-of-family clusters. To assess which clusters are in-family and which are out-of-family, we will need a human subject matter expert. We will then train the first classifier using those clusters as classes. Now we take a new dataset and, using the first classifier we made, filter out any in-family events (normal flight). Finally, we will cluster the filtered data (with some subject matter expert help) and use the resulting clusters as classes in our final classifier. This final classifier will be used to indicate LOC events while in flight. A simplified plot of the process is given here:
We start by taking the processed data and clustering it (an unsupervised learning approach). Following is a 3D plot of the clusters resulting from the use of FindClusters (specifying we want to find seven clusters). As you can see, the automatic color scheme is very helpful in visualizing the results. Mike, using his subject matter expertise, assesses groups 1, 2, 3, 6 and 7, which represent normal flight data. Group 5 (pink) is the LOC group, and group 4 (red) is the high-velocity normal flight:
To distinguish the LOC cluster from the others, Mike needed to choose more than two cluster groups. After progressively increasing the number of clusters with FindClusters, seven clusters were chosen to reduce the overlap of LOC group 5 from the neighboring groups 1 and 7, which are normal. A classifier trained with clearly distinguishable data will perform better and produce fewer false positives.
Using this clustered data, we can now train a classifier that will classify in-family flight data and out-of-family flight data (Low/High α—groups 4, 5). This in-family/out-of-family flight classifier will become a powerful machine learning tool in processing the next flight’s data. Using the Classify function and some clever preprocessing, we arrive at the following three class classifiers. The three classes are normal flight (Normal), high α flight (High) and low α flight (Low).
We now take data from a later flight and process it as we did earlier. Here is the resulting plot of that data:
Using our first classifier, we now classify the data as representing an in-family flight or an out-of-family flight. If it is in-family (normal flight), we exclude it from the dataset, as we are only looking for out-of-family instances (representing LOC events). With only non-normal data remaining, let’s plot the probability of that data being normal:
It is interesting to note that more than half of the remaining data points have less than a 0.05 probability of being normal. Taking this new, refined dataset we apply another layer of clustering, which results in the following plot:
We now see two main groups: group 3, which Mike explains as corresponding with thermaling; and group 1, which is the high-speed flight group. Thermaling is the act of using rising air columns to gain altitude. This involves flying in circles inside the air column (at speeds so slow it’s close to a stall), so it’s not surprising that β has a wide distribution during this phase. Groups 1 and 6 are also considered to be normal flight. Group 7 corresponds to LOC (a straight stall without sideslip). Groups 4 and 5 are imminent stalls with sideslip, leading to a left or right incipient spin and are considered to be LOC. Group 2 is hidden under group 1 and is a very high-speed flight close to the structural limits of the aircraft, so it’s also LOC.
Using this data, we can construct a new, second-generation classifier with three classes, low α (U), high α (D) and normal flight (N). These letters refer to the action required by the pilot—U means “pull up,” D means “push down” and N means “do nothing.” It is interesting to note that while the older classifier required days of training, this new filtered classifier only needed hours (and also greatly improved the speed and accuracy of the predictions, and reduced the occurrences of false positives).
As a final trial, Mike went on another flight and maintained a normal flight pattern throughout the entire flight. He later took the recorded data and plotted the probability of it being entirely normal using the second-generation classifier. As we can see here, there were no false positives during this flight:
Mike now wanted to test if the classifier would correctly predict possible LOC events. He went on another flight and, again, went into LOC events. Taking the processed data from that flight and plotting the probability of it being normal using the second-generation classifier results in the following final plot:
It is easy to see that some events were not classified as normal, although most of them were. Mike has confirmed these events correspond to actual LOC events.
Mike’s development work is a great demonstration as to how machine learning–based applications are going to affect everything that we do, increasing safety and survivability. This is also a great case study to showcase where and why it is so important to keep human subject matter experts in the loop.
Perhaps one of the most striking components of the SPG system is the use of the Wolfram Language on a Raspberry Pi Zero to connect to sensors, record in-flight data and run a machine learning application to compute when LOC is imminent—all on a computer that costs $5. Additional details on Mike’s journey can be found on his customer story page.
Just a few years ago, it would have been unimaginable for any one person to create such complex algorithms and deploy them rapidly in a real-world environment. The recent boom of the Internet of Things and machine learning has been driving great developmental work in these fields, and even after its 30th anniversary, the Wolfram Language has continued to be at the cutting edge of programming. Through its high-level abstractions and deep automation, the Wolfram Language has enabled a wide range of people to use the power of computation everywhere. There are many great products and projects left to be built using the Wolfram Language. Perhaps today is the day to start yours with a free trial of Wolfram|One!