Fishackathon

Every year, the U.S. Department of State sponsors a worldwide competition called Fishackathon. Its goal is to protect life in our waters by creating technological solutions to help solve problems related to fishing.

The first global competition was held in 2014 and has been growing massively every year. In 2018 the winning entry came from a five-person team from Boston, after competing against 45,000 people in 65 other cities spread across 5 continents. The participants comprised programmers, web and graphic designers, oceanographers and biologists, mathematicians, engineers and students who all worked tirelessly over the course of two days.

To find out more about the winning entry for Fishackathon in 2018 and how the Wolfram Language has helped make the seas safer, we sat down with Michael Sollami to learn more about him and his team’s solution to that year’s challenge.

Tell us a little bit about yourself and how you became familiar with Wolfram tech…

I became involved with Wolfram technologies during college and graduate school. I used Mathematica extensively for research in mathematics and computer science, and I ended up working with Stephen Wolfram in his Cambridge-based advanced research group. Today, I’m Lead Data Scientist at Salesforce Einstein, where I work with an amazing team of engineers and researchers to build new machine learning systems to enhance our prediction and search platforms. From experimental deep neural architectures to prototyping ever-more-efficient information retrieval methods, we are designing next-generation search, recommendation and predictive analytics technologies.

What drew you to Fishackathon?

Growing up near the Long Island Sound, where my family had a small sailboat, we spent a lot of time on or near the water. I love scuba diving (random trivia: Saba is my favorite dive site), and over the years I have seen firsthand the unnerving levels of aquatic devastation. The data coming from oceanographic research paints quite a bleak future. In fact, according to the best oceanographic computer models, there won’t be any fish larger than minnows by the 2040s.

In just 50 years, we’ve reduced the populations of large fish, such as bluefin tuna and cod, by over 90 percent. Industrial fishing uses nets that are 20 miles long, and trawlers drag something the size of a tractor trailer along the ocean floor. In just a few short decades, we clear-cut our seabed floors, once vital nurseries of sponges and corals, into millions of square miles of lifeless mud. According to coral reef ecologist Jeremy Jackson, “The total area of underwater habitat destruction is larger than the sum of all forests that have ever been cut down in the history of humanity.”

Many factors contribute to the threat of sea-life extinction, and they are not independent: biological and chemical pollution, acidification, deoxygenation, plastics, the climate crisis—it can be overwhelming. So we thought illegal, unreported and unregulated (IUU) overfishing was a good place to brainstorm possible solutions. It was incredibly exciting to compete simultaneously with over 3,500 other people around the world, and in the space of a weekend produce code with the potential for positive environmental impact.

What was last year’s challenge?

Each year Fishackathon offers multiple challenge statements that competing teams can chose from. The topics are developed by industry professionals with real-world needs and made available to participants nearer to the time of the event. In 2018, we chose challenge #10: passive illegal fishing detection.

Challenge Statement: Protecting restricted fishing zones (e.g. marine reserves, remote areas) from illegal fishing is a huge challenge. A passive tool (maybe using sonar?) that helps identify fishing activity in restricted areas would help agencies monitor, track and enforce laws more effectively.

You might recall from watching The Hunt for Red October that the United States Navy operates a chain of underwater listening posts located around the world, a sound surveillance system called SOSUS. This collection of bottom-mounted hydrophone arrays was originally constructed in the 1950s to track and fingerprint Soviet submarines by their acoustic signals. Almost 70 years later, there is no reason why we can’t apply (newer and cheaper versions of) this technology to track and identify fishing vessels.

What was the process of developing your solution?

We knew that the heart of our solution was an accurate fishing acoustics model, so we started with the software side of the problem. Once we had proof of life in the core detection algorithm, we iterated on designing the web app/interface and hardware components.

We need to design the submersible listening device to be outfitted with a hydrophone while making it as inexpensive as possible. The deployed devices would also need to be able to transmit network data back to our servers, for triangulating and tracking the positions of any detected poaching vessels operating in protected waters.

Our eventual submission, which we called PoachStopper, would not only recognize sounds associated with fishing, but also compute a unique signature for each boat that passes within a detection radius of 50 kilometers.

How did you incorporate deep learning into the solution?

This is where Mathematica shined. Mathematica’s ability to import and manipulate audio files made the data preprocessing pipeline dead simple. NetEncoders for "Audio" that handle varying dimensionalities made preprocessing inputs much easier than in many other frameworks at the time.

✕ net=ResourceData["HiddenAudioIdentifyMobileNetDepth1-0"]; ne=NetExtract[net,"Input"]; encoder=NetEncoder[{"Function",Function[feat,First@Partition[If[Length[feat]<96,ArrayPad[feat,{{0,96-Length[feat]}},Padding->"Fixed"],feat],96,64]]/@Normal[NetEncoder[{"AudioMelSpectrogram","NumberOfFilters"->64,"WindowSize"->600}][#1]]&,{96,64},"Pattern"->None,"Batched"->True}]; Image3D[ne@Normal@ExampleData[#],ImageSize->Tiny,ColorFunction->"RainbowOpacity"]&/@ExampleData["Audio"][[;;20]]//Multicolumn

At the time Fishackathon was held, the Neural Net Repository had yet to exist, so I couldn’t just pull some state-of-the-art model and perform some transfer learning. So we needed to train the network from scratch with the additional requirement of maintaining a very-low-energy consumption profile. For these reasons, I decided to to start working with a variant of Google’s MobileNet architecture, which features depth-wise separable convolutions as efficient building blocks. To the basic skeleton I added some linear bottlenecks between the layers with skip connections, and which led to higher accuracies and accelerated convergence. After processing the raw audio of our ocean sounds dataset into normalized spectrograms, I trained on AWS overnight on a GPU instance, and by Sunday morning had a working detector.

This was before Version 11.3 came out last year, but it would have been nice to use tools like NetEncoder["AudioMFCC"] and WebAudioSearch. The network we designed could delineate with 98% accuracy the difference between the normal sounds of the ocean, non-fishing boats and actively fishing trawlers operating in their different modes. We also built a separate recognition module that learned a nicely invariant mapping between the sounds of individual vessels’ engines and propellers—essentially a perceptual hash—for tracking specific vessels.

Using the Raspberry Pi seems like a good choice for inexpensive deployment…

Indeed! Thanks to Mathematica being packaged with the Raspbian OS, that could be a good route for people to take. Raspberry Pis can come with GPUs—however, they aren’t from NVIDIA, and so are not supported for TargetDevice→ "GPU". For production, we ended up porting the network into PyTorch, but I eventually ported the model to TensorFlow Mobile for testing with IoT and mobile devices, which was better from a real-time and power-usage profile.

What’s next for PoachStopper?

After we won the finals, I handed the project off to my very capable teammates, who are pursuing various partnerships and funding opportunities. PoachStopper and other eco-startups like it have the potential to make very measurable impacts in the fight for the future of oceanic life. However, the hard truth is that governments are the only entities that can prevent the end of fish. And it is easy to do, with just a few simple legislative steps:

• Create sufficiently large un-fishable areas for populations to begin regenerating
• Impose quotas on the amount of fish caught in any given year
• End government subsidies for the fishing-industrial complex

Companies need to pay for the privilege of fishing, and governments need to ensure that our oceans will not become a vast toxic desert, as we predict they will by 2050.

What’s in store for this year’s Fishackathon?

This year’s Fishackathon has a large focus on Fishcoin, and will focus on developing solutions for data capture and sharing using open platforms. This essentially means building apps that can capture and process sensor data in the field and then publishing it with decentralized ledger technology. With all its new blockchain-related features, Mathematica could be the weapon of choice in 2019!

Anything you’d like to add?

I consider Mathematica to be a “killer app” for any hackathon. Python and machine learning frameworks are my primary tools for machine learning engineering, but Mathematica remains the fastest language to prototype small-scale things. With Version 12 on the horizon, I’m very excited for the ability to compile everything down to machine code for running at C++ speed—not to mention the ability to export networks to ONNX for use in production-grade servers.