Introduction

Recognizing words is one of the simplest tasks a human can do, yet it has proven extremely difficult for machines to achieve similar levels of performance. Things have changed dramatically with the ubiquity of machine learning and neural networks, though: the performance achieved by modern techniques is dramatically higher compared with the results from just a few years ago. In this post, I’m excited to show a reduced but practical and educational version of the speech recognition problem—the assumption is that we’ll consider only a limited set of words. This has two main advantages: first of all, we have easy access to a dataset through the Wolfram Data Repository (the Spoken Digit Commands dataset), and, maybe most importantly, all of the classifiers/networks I’ll present can be trained in a reasonable time on a laptop.

It’s been about two years since the initial introduction of the Audio object into the Wolfram Language, and we are thrilled to see so many interesting applications of it. One of the main additions to Version 11.3 of the Wolfram Language was tight integration of Audio objects into our machine learning and neural net framework, and this will be a cornerstone in all of the examples I’ll be showing today.

Without further ado, let’s squeeze out as much information as possible from the Spoken Digit Commands dataset!

The Data

Let’s get started by accessing and inspecting the dataset a bit:

&#10005 ro=ResourceObject["Spoken Digit Commands"]

The dataset is a subset of the Speech Commands dataset released by Google. We wanted to have a “spoken MNIST,” which would let us produce small, self-enclosed examples of machine learning on audio signals. Since the Spoken Digit Commands dataset is a ResourceObject, it’s easy to get all the training and testing data within the Wolfram Language:

&#10005 trainingData=ResourceData[ro,"TrainingData"]; testingData=ResourceData[ro,"TestData"]; RandomSample[trainingData,3]//Dataset

One important thing we made sure of is that the speakers in the training and testing sets are different. This means that in the testing phase, the trained classifier/network will encounter speakers that it has never heard before.

&#10005 Intersection[trainingData[[All,"SpeakerID"]],testingData[[All,"SpeakerID"]]]

The possible output values are the digits from 0 to 9:

&#10005 classes=Union[trainingData[[All,"Output"]]]

Conveniently, the length of all the input data is between .5 and 1 seconds, with the majority for the signals being one second long:

&#10005 Dataset[trainingData][Histogram[#,ScalingFunctions->"Log"]&@*Duration,"Input"]

Encoders

In Version 11.3, we built a collection of audio encoders in NetEncoder and properly integrated it into the rest of the machine learning and neural net framework. Now we can seamlessly extract features from a large collection of audio recordings; inject them into a net; and train, test and evaluate networks for a variety of applications.

Since there are multiple features that one might want to extract from an audio signal, we decided that it was a good idea to have one encoder per feature rather than a single generic "Audio" one. Here is the full list:

• "Audio"
• "AudioSTFT"
• "AudioSpectrogram"
• "AudioMelSpectrogram"
• "AudioMFCC"

The first step (which is common in all encoders) is the preprocessing: the signal is reduced to a single channel, resampled to a fixed sample rate and can be padded or trimmed to a specified duration.

The simplest one is NetEncoder["Audio"], which just returns the raw waveform:

&#10005 encoder=NetEncoder["Audio"]

&#10005 encoder[RandomChoice[trainingData]["Input"]]//Flatten//ListLinePlot

The starting point for all of the other audio encoders is the short-time Fourier transform, where the signal is partitioned in (potentially overlapping) chunks, and the Fourier transform is computed on each of them. This way we can get both time (since each chunk is at a very specific time) and frequency (thanks to the Fourier transform) information. We can visualize this process by using the Spectrogram function:

&#10005 a=AudioGenerator[{"Sin",TimeSeries[{{0,1000},{1,4000}}]},2]; Spectrogram[a]

The main parameters for this operation that are common to all of the frequency domain features are WindowSize and Offset, which control the sizes of the chunks and their offsets.

Each NetEncoder supports the "TargetLength" option. If this is set to a specific number, the input audio will be trimmed or padded to the correct duration; otherwise, the length of the output of the NetEncoder will depend on the length of the original signal.

For the scope of this blog post, I’ll be using the "AudioMFCC" NetEncoder, since it is a feature that packs a lot of information about the signal while keeping the dimensionality low:

&#10005 encoder=NetEncoder[{"AudioMFCC","TargetLength"->All,"SampleRate"->16000,"WindowSize" -> 1024,"Offset"-> 570,"NumberOfCoefficients"->28,"Normalization"->True}] encoder[RandomChoice[trainingData]["Input"]]//Transpose//MatrixPlot

As I mentioned at the beginning, these encoders are quite fast: this specific one on my not-very-new machine runs through all 10,000 examples in slightly more than two seconds:

&#10005 encoder[trainingData[[All,"Input"]]];//AbsoluteTiming

Machine Learning, the Automated Way

Now we have the data and an efficient way of extracting features. Let’s find out what Classify can do for us.

To start, let’s massage our data into a format that Classify would be happier with:

✕ classifyTrainingData = #Input -> #Output & /@ trainingData; classifyTestingData = #Input -> #Output & /@ testingData;

Classify does have some trouble dealing with variable-length sequences (which hopefully will be improved on soon), so we’ll have to find ways to work around that.

Mean of MFCC

To make the problem simpler, we can get rid of the variable length of the features. One naive way is to compute the mean of the sequence:

&#10005 cl=Classify[classifyTrainingData,FeatureExtractor->(Mean@*encoder),PerformanceGoal->"Quality"];

The result is a bit disheartening, but not unexpected, since we are trying to summarize each signal with only 28 parameters. Not stunning.

&#10005 cm=ClassifierMeasurements[cl,classifyTestingData]; cm["Accuracy"] cm["ConfusionMatrixPlot"]

Adding Some Statistics

To improve the results of Classify, we can feed it more information about the signal by adding the standard deviation of each sequence as well:

&#10005 cl=Classify[classifyTrainingData,FeatureExtractor->(Flatten[{Mean[#],StandardDeviation[#]}]&@*encoder),PerformanceGoal->"Quality"];

Some effort does pay off:

&#10005 cm=ClassifierMeasurements[cl,classifyTestingData]; cm["Accuracy"] cm["ConfusionMatrixPlot"]

Even More Statistics

We can follow this strategy a bit more, and also add the Kurtosis of the sequence:

&#10005 cl=Classify[classifyTrainingData,FeatureExtractor->(Flatten[{Mean[#],StandardDeviation[#],Kurtosis[#]}]&@*encoder),PerformanceGoal->"Quality"];

The improvement is not as huge, but it is there:

&#10005 cm=ClassifierMeasurements[cl,classifyTestingData]; cm["Accuracy"] cm["ConfusionMatrixPlot"]

Fixed-Length Sequences

We could continue dripping information about statistics of the sequences, with smaller and smaller returns. But with this specific dataset, we can follow a simpler strategy: remember how we noticed that most recordings were about 1 second long? That means that if we fix the length of the extracted feature to the equivalent of 1 second (about 28 frames) using the "TargetLength" option, the encoder will take care of doing the padding or trimming as appropriate. This way, all the inputs to Classify will have the same dimensions of {28,28}:

&#10005 encoderFixed=NetEncoder[{"AudioMFCC","TargetLength"->28,"SampleRate"->16000,"WindowSize" -> 1024,"Offset"-> 570,"NumberOfCoefficients"->28,"Normalization"->True}]

&#10005 cl=Classify[classifyTrainingData,FeatureExtractor->encoderFixed,PerformanceGoal->"DirectTraining"];

The training time is longer, but we do still get an accuracy bump:

&#10005 cm=ClassifierMeasurements[cl,classifyTestingData]; cm["Accuracy"] cm["ConfusionMatrixPlot"]

This is about as far as we can get with Classify and low-level features. Time to ditch the automation and to bring out the neural networks machinery!

Convolutional Neural Network

Let’s remember that we’re playing with a spoken versions of MNIST, so what could be a better starting place than LeNet? This is a network that is often used as a benchmark on the standard image MNIST, and is very fast to train (even without GPU).

We’ll use the same strategy as in the last Classify example: we’ll fix the length of the signals to about one second, and we’ll tune the parameters of the NetEncoder so that the input will have the same dimensions of the MNIST images. This is one of the reasons we can confidently use a CNN architecture for this job: we are dealing with 2D matrices (images, in essence—actually, that’s how we usually look at MFCC), and we want the network to infer information from their structures.

Let’s grab LeNet from NetModel:

&#10005 lenet=NetModel["LeNet Trained on MNIST Data","UninitializedEvaluationNet"]

Since the "AudioMFCC" NetEncoder produces two-dimensional data (time x frequency), and the net requires three-dimensional inputs (where the first dimensions are the channel dimensions), we can use ReplicateLayer to make them compatible:

&#10005 lenet=NetPrepend[lenet,ReplicateLayer[1]]

Using NetReplacePart, we can attach the "AudioMFCC" NetEncoder to the input and the appropriate NetDecoder to the output:

&#10005 audioLeNet=NetReplacePart[lenet, { "Input"->NetEncoder[{"AudioMFCC","TargetLength"->28,"SampleRate"->16000,"WindowSize" -> 1024,"Offset"-> 570,"NumberOfCoefficients"->28,"Normalization"->True}], "Output"->NetDecoder[{"Class",classes}] } ]

To speed up convergence and prevent overfitting, we can use NetReplace to add a BatchNormalizationLayer after every convolution:

&#10005 audioLeNet=NetReplace[audioLeNet,{x_ConvolutionLayer:>NetChain[{x,BatchNormalizationLayer[]}]}]

NetInformation allows us to visualize at a glance the net’s structure:

✕ NetInformation[audioLeNet,"SummaryGraphic"]

Now our net is ready for training! After defining a validation set on 5% of the training data, we can let NetTrain worry about all hyperparameters:

&#10005 resultObject=NetTrain[ audioLeNet, trainingData, All, ValidationSet->Scaled[.05] ]

Seems good! Now we can use ClassifierMeasurements on the net to measure the performance:

&#10005 cm=ClassifierMeasurements[resultObject["TrainedNet"],classifyTestingData]; cm["Accuracy"] cm["ConfusionMatrixPlot"]

It looks like the added effort paid off!

Recurrent Neural Network

We can also embrace the variable-length nature of the problem by specifying "TargetLength"→All in the encoder:

&#10005 encoder=NetEncoder[{"AudioMFCC","TargetLength"->All,"NumberOfCoefficients"->28,"SampleRate"->16000,"WindowSize" -> 1024,"Offset"-> 571,"Normalization"->True}]

This time we’ll use an architecture based on the GatedRecurrentLayer. Used on its own, it returns its state per each time step, but we are only interested in the classification of the entire sequence, i.e. we want a single output for all time steps. We can use SequenceLastLayer to extract the last state for the sequence. After that, we can add a couple of fully connected layers to do the classification:

&#10005 rnn= NetChain[{ GatedRecurrentLayer[32,"Dropout"->{"VariationalInput"->0.3}], GatedRecurrentLayer[64,"Dropout"->{"VariationalInput"->0.3}], SequenceLastLayer[], LinearLayer[64], Ramp, LinearLayer[Length@classes], SoftmaxLayer[]}, "Input"->encoder, "Output"->NetDecoder[{"Class",classes}] ]

Again, we’ll let NetTrain worry about all hyperparameters:

&#10005 resultObjectRNN=NetTrain[ rnn, trainingData, All, ValidationSet->Scaled[.05] ]

… and measure the performance:

&#10005 cm=ClassifierMeasurements[resultObjectRNN["TrainedNet"],classifyTestingData]; cm["Accuracy"] cm["ConfusionMatrixPlot"]

It seems that treating the input as a pure sequence and letting the network figure out how to extract meaning from it works quite well!

An Interlude

Now that we have some trained networks, we can play with them a bit. First of all, let’s take the recurrent network and chop off the last two layers:

&#10005 choppedNet=NetTake[resultObjectRNN["TrainedNet"],{1,5}]

This leaves us with something that produces a vector of 64 numbers per each input signal. We can try to use this chopped network as a feature extractor and plot the results:

&#10005 FeatureSpacePlot[Style[#["Input"],ColorData[97][#["Output"]+1]]->#["Output"]&/@testingData,FeatureExtractor->choppedNet]

It looks like the various classes get properly separated!

We can also record a signal, and test the trained network on it:

&#10005 a=AudioTrim@AudioCapture[]

&#10005 resultObjectRNN["TrainedNet"][a]

RNN Using CTC Loss

We can attempt something more adventurous on this dataset: up until now, we have simply done classification (a sequence goes in, a single class comes out). What if we tried transduction: a sequence (the MFCC features) goes in, and another sequence (the characters) comes out?

First of all, let’s add string labels to our data:

✕ labels = <|0 -> "zero", 1 -> "one", 2 -> "two", 3 -> "three", 4 -> "four", 5 -> "five", 6 -> "six", 7 -> "seven", 8 -> "eight", 9 -> "nine"|>; trainingDataString = Append[#, "Target" -> labels[#Output]] & /@ trainingData; testingDataString = Append[#, "Target" -> labels[#Output]] & /@ testingData;

We need to remember that once trained, this will not be a general speech-recognition network: it will only have been exposed to one word at a time, only to a limited set of characters and only 10 words!

&#10005 Union[Flatten@Characters@Values@labels]//Sort

A recurrent architecture would output a sequence of the same length as the input, which is not what we want. Luckily, we can use the CTCBeamSearch NetDecoder to take care of this. Say that the input sequence is n steps long, and the decoding has m different classes: the NetDecoder will expect an input of dimensions (there are m possible states, plus a special blank character). Given this information, the decoder will find the most likely sequence of states by collapsing all of the ones that are not separated by the blank symbol.

Another difference with the previous architecture will be the use of NetBidirectionalOperator. This operator applies a net to a sequence and its reverse, catenating both results into one single output sequence:

&#10005 net=NetGraph[{NetBidirectionalOperator@GatedRecurrentLayer[64,"Dropout"->{"VariationalInput"->0.4}], NetBidirectionalOperator@GatedRecurrentLayer[64,"Dropout"->{"VariationalInput"->0.4}], NetMapOperator[{LinearLayer[128],Ramp,LinearLayer[],SoftmaxLayer[]}]}, {NetPort["Input"]->1->2->3->NetPort["Target"]}, "Input"->NetEncoder[{"AudioMFCC","TargetLength"->All,"NumberOfCoefficients"->28,"SampleRate"->16000,"WindowSize" -> 1024,"Offset"-> 571,"Normalization"->True}], "Target"->NetDecoder[{"CTCBeamSearch",Alphabet[]}]]

To train the network, we need a way to compute the loss that takes the decoding into account. This is what the CTCLossLayer is for:

&#10005 trainedCTC=NetTrain[net,trainingDataString,LossFunction->CTCLossLayer["Target"->NetEncoder[{"Characters",Alphabet[]}]],ValidationSet->Scaled[.05],MaxTrainingRounds->20];

Let’s pick a random example from the test set:

&#10005 a=RandomChoice@testingDataString

Look at how the trained network behaves:

&#10005 trainedCTC[a["Input"]]

We can also look at the output of the net just before the CTC decoding takes place. This represents the probability of each character per time step:

&#10005 probabilities=NetReplacePart[trainedCTC,"Target"->None][a["Input"]]; ArrayPlot[Transpose@probabilities,DataReversed->True,FrameTicks->{Thread[{Range[26],Alphabet[]}],None}]

We can also show these probabilities superimposed on the spectrogram of the signal:

&#10005 Show[{ArrayPlot[Transpose@probabilities,DataReversed->True,FrameTicks->{Thread[{Range[26],Alphabet[]}],None}],Graphics@{Opacity[.5],Spectrogram[a["Input"],DataRange->{{0,Length[probabilities]},{0,27}},PlotRange->All][[1]]}}]

There is definitely the possibility that the network would make small spelling mistakes (e.g. “sixo” instead of “six”). We can visually inspect these spelling mistakes by applying the net to all classes and get a WordCloud of them:

&#10005 WordCloud[StringJoin/@trainedCTC[#[[All,"Input"]]]]&/@GroupBy[testingDataString,Last]

Most of these spelling mistakes are quite small, and a simple Nearest function might be enough to correct them:

&#10005 nearest=First@*Nearest[Values@labels]; nearest["sixo"]

To measure the performance of the net and the Nearest function, first we need to define a function that, given an output for the net (a list of characters), computes the probability per each class:

&#10005 probs=AssociationThread[Values[labels]->0]; getProbabilities[chars:{___String}]:=Append[probs,nearest[StringJoin[chars]]->1]

Let’s check that it works:

&#10005 getProbabilities[{"s","i","x","o"}] getProbabilities[{"f","o","u","r"}]

Now we can use ClassifierMeasurements by giving an association of probabilities and the correct labels per each example as input:

&#10005 cm=ClassifierMeasurements[getProbabilities/@trainedCTC[testingDataString[[All,"Input"]]],testingDataString[[All,"Target"]]]

The accuracy is quite high!

&#10005 cm["Accuracy"] cm["ConfusionMatrixPlot"]

Encoder/Decoder

Up till now, the architectures we have been experimenting with are fairly straightforward. We can now attempt to do something more ambitious: an encoder/decoder architecture. The basic idea is that we’ll have two main components in the net: the encoder, whose job is to encode all the information about the input features into a single vector (of 128 elements, in our case); and the decoder, which will take this vector (the “encoded” version of the input) and be able to produce a “translation” of it as a sequence of characters.

Let’s define the NetEncoder that will deal with the strings:

&#10005 targetEnc=NetEncoder[{"Characters",{Alphabet[],{StartOfString,EndOfString}->Automatic},"UnitVector"}]

… and the one that will deal with the Audio objects:

&#10005 inputEnc=NetEncoder[{"AudioMFCC","TargetLength"->All,"NumberOfCoefficients"->28,"SampleRate"->16000,"WindowSize" -> 1024,"Offset"-> 571,"Normalization"->True}]

Our encoder network will consist of a single GatedRecurrentLayer and a SequenceLastLayer to extract the last state, which will become our encoded representation of the input signal:

&#10005 encoderNet=NetChain[{GatedRecurrentLayer[128,"Dropout"->{"VariationalInput"->0.3}],SequenceLastLayer[]}]

The decoder network will take a vector of 128 elements and a sequence of vectors as input, and will return a sequence of vectors:

&#10005 decoderNet=NetGraph[{ SequenceMostLayer[], GatedRecurrentLayer[128,"Dropout"->{"VariationalInput"->0.3}], NetMapOperator[LinearLayer[]], SoftmaxLayer[]}, {NetPort["Input"]->1->2->3->4, NetPort["State"]->NetPort[2,"State"]} ]

We then need to define a network to train the encoder and decoder. This configuration is usually called a “teacher forcing” network:

&#10005 teacherForcingNet=NetGraph[<|"encoder"->encoderNet,"decoder"->decoderNet,"loss"->CrossEntropyLossLayer["Probabilities"],"rest"->SequenceRestLayer[]|>, {NetPort["Input"]->"encoder"->NetPort["decoder","State"], NetPort["Target"]->NetPort["decoder","Input"], "decoder"->NetPort["loss","Input"], NetPort["Target"]->"rest"->NetPort["loss","Target"]}, "Input"->inputEnc,"Target"->targetEnc]

Using NetInformation, we can look at the whole structure with one glance:

&#10005 NetInformation[teacherForcingNet,"FullSummaryGraphic"]

The idea is that the decoder is presented with the encoded input and most of the target, and its job is to predict the next character. We can now go ahead and train the net:

&#10005 trainedEncDec=NetTrain[teacherForcingNet,trainingDataString,ValidationSet->Scaled[.05]]

Now let’s inspect what happened. First of all, we have a trained encoder:

&#10005 trainedEncoder=NetReplacePart[NetExtract[trainedEncDec,"encoder"],"Input"->inputEnc]

This takes an Audio object and outputs a single vector of 150 elements. Hopefully, all of the interesting information of the original signal is included here:

&#10005 example=RandomChoice[testingDataString]

Let’s use the trained encoder to encode the example input:

&#10005 encodedVector=trainedEncoder[example["Input"]]; ListLinePlot[encodedVector]

Of course, this doesn’t tell us much on its own, but we could use the trained encoder as feature extractor to visualize all of the testing set:

&#10005 FeatureSpacePlot[Style[#["Input"],ColorData[97][#["Output"]+1]]->#["Output"]&/@testingData,FeatureExtractor->trainedEncoder]

To extract information from the encoded vector, we need help from our trusty decoder (which has been trained as well):

&#10005 trainedDecoder=NetExtract[trainedEncDec,"decoder"]

Let’s add some processing of the input and output:

&#10005 decoder=NetReplacePart[trainedDecoder,{"Input"->targetEnc,"Output"->NetDecoder[targetEnc]}]

If we feed the decoder the encoded state and a seed string to start the reconstruction and iterate the process, the decoder will do its job nicely:

&#10005 res=decoder[<|"State"->encodedVector,"Input"->"c"|>] res=decoder[<|"State"->encodedVector,"Input"->res|>] res=decoder[<|"State"->encodedVector,"Input"->res|>]

We can make this decoding process more compact, though; we want to construct a net that will compute the output automatically until the end-of-string character is reached. As a first step, let’s extract the two main components of the decoder net:

&#10005 gru=NetExtract[trainedEncDec,{"decoder",2}] linear=NetExtract[trainedEncDec,{"decoder",3,"Net"}]

Define some additional processing of the input and output of the net that includes special classes to indicate the start and end of the string:

&#10005 classEnc=NetEncoder[{"Class",Append[Alphabet[],StartOfString],"UnitVector"}]; classDec=NetDecoder[{"Class",Append[Alphabet[],EndOfString]}];

Define a character-level predictor that takes a single character, runs one step of the GatedRecurrentLayer and produces a single softmax prediction:

&#10005 charPredictor=NetChain[{ReshapeLayer[{1,27}],gru,ReshapeLayer[{128}],linear,SoftmaxLayer[]},"Input"->classEnc,"Output"->classDec]

Now we can use NetStateObject to inject the encoded vector into the state of the recurrent layer:

&#10005 sobj=NetStateObject[charPredictor,<|{2,"State"}->encodedVector|>]

If we now feed this predictor the StartOfString character, this will predict the next character:

&#10005 sobj[StartOfString]

Then we can iterate the process:

&#10005 sobj[%] sobj[%] sobj[%]

We can now encapsulate this process in a single function:

&#10005 predict[input_]:=Module[{encoded,sobj,res}, encoded=trainedEncoder[input]; sobj=NetStateObject[charPredictor,<|{2,"State"}->encoded|>]; res=NestWhileList[sobj,StartOfString,#=!=EndOfString&]; StringJoin@res[[2;;-2]] ]

This way, we can directly compute the full output:

&#10005 predict[example["Input"]]

Again, we need to define a function that, given an output for the net, computes the probability per each class:

&#10005 probs=AssociationThread[Values[labels]->0]; getProbabilities[in_]:=Append[probs,nearest@predict[in]->1];

Now we can use ClassifierMeasurements by giving as input an association of probabilities and the correct labels per each example:

&#10005 cm=ClassifierMeasurements[getProbabilities/@testingDataString[[All,"Input"]],testingDataString[[All,"Target"]]]

&#10005 cm["Accuracy"] cm["ConfusionMatrixPlot"]

Audio signals are less ubiquitous than images in the machine learning world, but that doesn’t mean they are less interesting to analyze. As we continue to complete and optimize audio analysis using modern machine learning and neural net approaches in the Wolfram Language, we are also excited to use it ourselves to build high-level applications in the domains of speech analysis, music understanding and many other areas.

---

Download this post as a Wolfram Notebook.