News, Views & Insights
How to Make a Webcam Intruder Alarm with Mathematica
Ever since I wrote the “Doing Spy Stuff with Mathematica” blog post, I have had a feeling that I am being watched. Time to build some office security using Mathematica Home Edition!
First, I am going to make use of an imminent new Mathematica command CurrentImage, which will import a real-time image from a video device. Let’s get some test images using the webcam on my laptop.
And again, but with a couple of seconds to hide from the camera:
ImageDifference gives us a pixel-by-pixel absolute difference between the two images. My camera isn’t very good, so there is a bit of noise. As a quick way to deal with that, I first blur both images slightly.
This image is black where there was no difference between the images. I don’t care how different they are, so I will apply a threshold. I have applied this to the maximum difference out of all the color channels, since we want a change in color to be as important as an overall change in brightness.
Now we clump together these differences by filling in gaps of 10 pixels or less.
This is all we need to detect an intruder. So let’s put those steps together into a function.
We can use this simplified scene-change image in two important ways.
Firstly, we can use it to highlight what has changed in an image. I will use it to darken the rest of the image by 75% and add a white line around the edge of the changed area.
Secondly, we can use it as a measure of the amount that has changed.
In this case there has been one scene change of over 37,000 pixels. Had the coffee cup been removed or another added, we would see a second smaller component listed as well.
Now I can create a function that tests and acts on the result of the scene changes. If the size of the biggest change is more than 1,000 pixels, then we will call that an intruder event, and we will use the SendMail command to send me email to tell me about it together with the highlighted changed image. I ignore the case of over 70,000 pixels, as I don’t want an alarm when someone turns off the office lights and the entire image changes.
Next I want to check the current scene against a time lagged store of what the image should look like. We want to update the store regularly, so that lasting changes such as an item being taken from the scene or placed there, or the sun setting, do not produce repeated alerts.
Finally, we need to call the function repeatedly. I could use an infinite loop, but that would lock my Mathematica completely. I might want to run overnight computations at the same time, or have other events happen periodically, so instead I program a scheduled interrupt (another new capability) that will suspend any running Mathematica program, run this command, and then return to the program.
Now let’s use it.
This gives me 60 seconds to leave the room and then runs the task every 15 seconds, keeping a 4 minute buffer of background images.
Later that night I get the following emails. (OK, not really!) Not bad for a couple dozen lines of code!
You are on a roll, Jon. One interesting blog after another. This is teasing! I could right now use the functionality you are show casing here. Many thanks.
Nice application, quite educational. However, if this is supposed to be used as a practical application, I’d caution a few things: a) 15 seconds is a lot of time. Plenty of time to get in, steal or break something, and disappear (or momentarily hide beneath the desk or somewhere else off camera). Most professional motion-detection software uses at least 5 frames per second. The one pointing towards the Pentagon on 911 couldn’t even capture the plane or rocket (whatever theory you believe in) using even more frames per second. b) My motion-detection specifically DOES trigger an alarm if the lights are turned off, because now it WOULD be possible for someone to get in unseen and do whatever they want, as long as they don’t need light for that. c) Also make sure you cover the entry area, not just the desk area, because otherwise the intruder just needs to sneak towards the computer (rob on the ground), pull the power cable, and now your whole system is rendered useless. There’s no point in an intrusion detection system if the intruder can easily turn it off. d) If you run this in complete darkness, make sure you turn off the screen, lest some screen reflections from sending you an email (in the M f/e or in the email program) may be picked up by the camera and trigger a new alert due to pixel changes. You’d get an email every 15 s! Then I’d call that a “low-frequency oscillator”, not an intrusion-detection system.
One could write an interesting “fixed-point study” that uses M to display images based on the images it gets from the camera, when that camera is pointing at the screen (how fast can it be done, color divergences, flip-flops, etc.).
I have developed a package which adds Python style object-orientation to Mathematica. But I would like to develop automatic garbage collector as well.
About MathOO: http://www.voofie.com/content/169/mathoo-adding-python-style-object-orientation-to-mathematica-with-mathoo-10-beta-launch-alternative-/
And here is the problem I am facing about Temporary variables in Mathematica:
http://www.voofie.com/content/178/adding-garbage-collector-to-mathoo/
Is there any way to workaround the problem? And can Mathematica deal with reference cycles? Thank you very much.
This one might be your best yet. Totally cool.
Wow! If this post isn’t marketing then what is? After reading post after post of image processing , who wouldn’t upgrade?
Couldn’t help but notice that dropping a cloth over the webcam beforehand would not only defeat the webcam from capturing an image of you as an intruder, but given the drastic image change, would not trigger an alert either.
An interesting problem, hm?
@Daniel Crowley
As a workaround you can use an imminent new Mathematica command NeglectWebcamCloth[]. A systematic solution would be to use a pair of webcams that would monitor each other, besides the door.
” I must equiped with Mathematica Home Edition, instantly.” — Bond
I had a good laugh at the NeglectWebCamCloth[] imminent Mathematica command! :-)
But we could also solve the problem by placing a mirror so that the camera can see itself and therefore able to check itself.
Despite my positive comment further up, I’d like to mention that there really seems to be a break in the previous “must run on all platforms” philosophy. For CUDA support this is, of course, totally understandable. But not in general. Already the first command, a=CurrentImage[] throws an error
CurrentImage::notsupported: “Image acquisition is not supported on Unix.
I could trace this down to
$DefaultImagingDevice::notsupported: Image acquisition is not supported on Unix.
and
$ImagingDevices::notsupported: Image acquisition is not supported on Unix
Is it really so that there is no way to control the settings of the webcam? White balance, contrast brightness and such?
@ Vincent
As long as you are within the dynamic range of your webcam, then you can achieve that by post-processing the image with ImageAdjust. On-camera adjustments will vary between cameras, but it is common that such adjustments are actually post-processing of the original CCD data.
Jon, I really enjoy your posts. i am trying to do a motion capture (attach ping-pong hemispheres — makeshift markers — to a student wearing a black leotard, capture images from multiple webcams, locate the markers on 2d images, do some math, and infer the 3d position of the markers). Do you know if we can make Mathematica 8.0 capture images from more than one webcam simultaneously? What would you suggest?
@Dan
Thanks.
The methods for switching cameras, is not exposed at the user level except throught the intereractive interface of ImageCapture[].
I suspect that because the image capturing is set up for streaming, that the time taken to swtich between them, even if you could programatically switch, would be too slow to keep up any kind of frame rate. (At the moment). I would probably capture both images to files and then post process them using Import to pick out specific frames at leasure, after the event.
How can I capture sound (without picture) in Mathematica?
Best regards from Prague
Pavel
I am trying to prototype a mathematica application tha will enable Autonomous Intruder & Anomalous Event Detection, Tracking and Alarm to web users those who want to autonomously and asynchronously monitor their private spaces such as homes, offices as well as outdoor spaces such as backyards, roads and recreation facilities using inexpensive commerical Internet-connected IP-cams and webcams.
I want to employ a proven and tested unsupervised algrothm that will help me accomplish my goal. Any idea which one would be the best? What are the mathematica functions that will help me customize such an algorithm? I am using network license of Mathematica-8.
Great post. I found your steganography one some weeks ago by googling something, and now that I looked for something like this in order not to program it, I came here.
It would be lovely to also have here a code download link like in the other post, in order not to use some OCR code…
Cheers!
ImagesToOCR =
Table[Import[
“https://blog.wolfram.com/data/uploads/2010/11/IntruderAlarm”
ToString[i] “.png”], {i, 2, 3}]~Join~
Table[Import[
“https://blog.wolfram.com/data/uploads/2010/11/IntruderAlarm”
ToString[i] “.png”], {i, 5, 14}]
OCRedCode = Table[TextRecognize[ImagesToOCR[[i]]], {i, 1, 12}];
…
I had actually made a very similar project last year at Wildhacks (Northwestern University Hackathon) with 3 others. Ours didn’t use Mathematica, but it had used multiple images of one person and classified that as a specific person called “owner”. When another person comes onto a video screen, it takes screenshots of the video and if a person is unrecognized then it sends an automatic text message to the owner.
Come to think, however, mine was flawed whereas yours seemed to be on point. I’m glad that after reading this post I can get a better understanding on how I could have done it. This would’ve been very useful for me, and I really wished I could’ve read this beforehand.
Cheers.
I like the idea of checking if the intruder is a known person. If I were to update this now, the approach would be to use FindFaces to get the bounding box and then the FeatureExtractor described at https://resources.wolframcloud.com/NeuralNetRepository/resources/ResNet-101-Trained-on-Augmented-CASIA-WebFace-Data to test the face against known people, with an APIFunction or FormFunction link in the email to follow to signify “I know this person”. I have added it to my list of “one day, when I have time” blog items.