Augmented Reality meets Drones

You probably know what these two words mean (if you’re interested in technology or follow our blog). Now, we thought it might be interesting to combine them and create something like an “augmented reality flight mode”. Thus, the concept for dronAR was born. But let’s be honest: we also had a lot of fun without the AR part.

DJI-Phantom4-drone-augmented-reality

Now, I think I can safely say that we do a lot with Augmented Reality here at CodeFlügel. You could even say it’s what we do best. But our competitors aren’t exactly twiddling their thumbs either and new AR frameworks keep appearing left, right and center. So it’s definitely useful to have something special up your sleeve when trying to market AR applications – and if that something can fly, potential customers will be dumbstruck by the sheer awesomeness of the concept. That’s the theory, anyway – let’s see whether you agree. Although, if “Augmented Reality” is still Greek to you at this point, you might want to read this short explanation first.

The Concept

So basically, AR is great at visualizing two-dimensional content (i.e. photos and videos), as well as three-dimensional models – I think we can agree on that. And with mobile AR applications, meaning apps for your smartphone or tablet, anyone owning such a device can use them (and yes, that includes Windows Phones and even – *drumroll* –  Symbian Phones). Be that as it may, but how and why are we using drones, when it works perfectly well with handheld devices? The answer to that is simple: you’ll never be able to capture the same views and perspectives with a smartphone/tablet as by using a drone.

Sure, handhelds are really amazing, if what I’m trying to do is check out some furniture I’d like to buy by digitally placing it in my living room and looking at different colors and variations (watch a short demo here, about 1 minute in), or if I want to display a 3D model of myself on my business card, when I hold my phone over it. We’re big fans of that. But in this case, we’re thinking on a grander scale. Imagine: instead of your living room, the piece of land you just bought to build your house on. Instead of a 3D model of some furniture item, a 3D model of your future home. Now it’s obvious to you, I’m sure, that you might have a hard time with your phone (yes, even with your iPad!) when you place the marker (a sheet of paper like you saw in the video) somewhere in the field instead of your living room. If it’s not very windy that day, the marker will stay where it is, no doubt about it. But you’re going to have to move back about 30 meters to be able to view the entire life-sized thing (unless you’re building a shed, then you might be fine). And here’s where you run into a problem, because your device will have a hard time seeing the marker from that distance.

“Just print it bigger, then!”, you say? Good input, thanks, let’s put a pin in it for now. Because even if I print it one by one meters square, position it and then move back 30 meters, the angle between the ground and my device is less than 4 degrees (and that was calculated for huge basketball players who can hold their phone 2 meters above the ground – which I am not).

degrees-augmented-reality

And even if I were to climb onto a ladder to reach a theoretical height of 5 meters, the angle is still less than 10 degrees, which just isn’t enough to achieve stable and reliable marker detection.

Therefore, this is where we unleash our not-so-secret weapon.

The Drone

By now, there’s a huge variety of drones to choose from, and if it’s your dearest ambition to try your hand at being a pilot, you can get a tiny one for little money and have a lot of fun with it (as our developers will be more than happy to tell you). However, those tiny drones aren’t quite enough to suit our needs, as you probably already figured out. We had to look for a more high-powered alternative that also has a camera. Now, even in this league, there are huge differences in terms of the kind of equipment you can get and thus also in terms of price.

We eventually chose the DJI Phantom 4, a model with a stabilized, swivel-mounted camera and object recognition sensors. Thus, a short lack of attention an unforeseen obstacle on the part of the pilot during the flight can be spotted (and , if desired, avoided) autonomously by the drone. In addition to simply capturing and saving photos and videos on an integrated memory card, the drone also streams the live camera view to the remote (on which you can mount a smartphone or, even better, a tablet to use as a livestream display) in 720p with 30 frames per second. If you’d like to check out the video quality yourself, you can have a look at the following video, which was captured in the highest possible resolution by the same drone model as ours. If you set the YouTube video quality to 720p, you’ll get a good idea of what the drone’s livestream will look like.

So…what do I do with all that?

Well, this is where the technological mumbo jumbo starts – and we really can’t tell you all the details here, because, as I said, we’re keeping them up our sleeve for now. Short version: we’ve basically created a flying eye that can see our marker from the sky. As long as it’s big enough, which is why we put a pin in that thought, because yeah – even with the drone’s really high resolution, there’s limits to what it can do.

So, just like on a smartphone or tablet, the live camera stream is put through a software which can detect and interpret the marker and, most importantly, knows where and how to place the desired 3D model. Finally, the resulting augmented reality view as we know it – meaning a live camera stream with a digitally added 3D model – just has to be displayed. For our little thought experiment, this means that you can now marvel at your future home in the real surroundings you’re going to build it in.

Any further details and fantasies, I’ll leave to your creative mind. After all, you can build anything you like – in your mind, at least. Or you buy a drone. If you do that, definitely get in touch so we can race each other.