Vuforia4Unreal 2.0 – Open Source Edition

It’s been a while since our last blog about the Vuforia4Unreal plugin. One year, two months and two days to be exact… And it took exactly one year for us to get it out of its drawer, dust it off, get it up to scratch and expand it a little bit in order to be able to publish it as an open source project. Today, we would like to tell you what’s new with our Vuforia4Unreal plugin and present a few of our many ideas for its future. I hope you’re as hyped as we are right now.


Vuforia4Unreal as a „real plugin” and other news

Ok, now look – this is the moment where we have to admit that we might have told a little white lie last time. Back then, almost exactly a year ago, Vuforia4Unreal wasn’t technically a full plugin, but an Unreal project. While that distinction doesn’t really matter much, as long our AR code does what it’s supposed to (be it as a project or a plugin), it needs to be a full-fledged plugin to open source it. And not least from a usability point of view, a plugin is a much nicer and smoother solution. If you’d like to know a little more about all this, you can find some info on plugins in the depths of the Unreal documentation.

Also this might be a good time to state that we are not delivering any PTC code here. The Vuforia SDK is owned and distributed by PTC and is used solely as an external dependency in our plugin. Therefore, you only have to copy the Vuforia SDK into our plugin folder and configure the license key in order to use the plugin. Everything else can be done comfortably within the Unreal user interface.

In order to further increase the usability of our plugin, we started forwarding AR events all the way to the Unreal UI as well. I mean, how great is it to just put together your entire AR application in a graphics engine? (And not just any graphics engine either, but in what is currently probably the best one.)

Above is a small level blueprint with the OnTrackingLost/Found events of the Vuforia SDK. In this example, the AR content is simply shown or hidden, depending on whether the tracking target is visible or not. In theory, however, this event can trigger anything you like. How about having a dragon fly across the room?

Under the hood

Next to all the visible improvements, we also made some small (or actually large) enhancements ‘under the hood’ over the past month or so. Instead of copying the texture of the camera stream in order to render it in Unreal, we now use texture pointers. Thus, we not only save on memory, and copying time, but also the detour from the graphics memory to the main memory and back. Okay, on most mobile devices, that’s probably the same memory anyways, but the figures speak for themselves and the copying time is saved in any case.

Initially, we were working with a laggy 20 FPS on a Nexus 5 and now we can get a constant frame rate of up to 60 FPS on modern devices such as the Galaxy S5, S6 and S7 or the Pixel C. For a direct comparison, we also tested the plugin on a Nexus 5, of course, and were able to measure a performance boost from 30 FPS to 60 FPS.

As a last technical detail, before we talk more about the future of Vuforia4Unreal, we also upgraded to the latest version of Unreal Engine (version 4.16.2), which mainly brought an improvement of the build process.

What now? Help us to design the future!

The idea of making our Vuforia4Unreal (now actual) plugin open source has been knocking around since we started on it. But on the one hand, there were some processes we went through, which impacted the further development a little bit (and that’s all we’re going to say about this ;)). And on the other hand, there just wasn’t time to complete the plugin to such a degree that we could in good conscience publish the first commit on Github.

But yay, the time has finally come now! So where will we go next? First, what’s most important is the stability of the existing features. So far, we’ve only had the opportunity to test or plugin on a few select devices. But, of course, we want Vuforia4Unreal to deliver a great performance and high-quality AR on as many devices as possible. Therefore, we need people who will try the plugin on their own devices and then also people who know their stuff when it comes to Unreal and who want to work with us in order to increase the plugin’s efficiency.
Additionally, there are lots of features which we would like to develop (or have developed by you). The list seems endless, since the Vuforia SDK allows for so many possibilities:

• Multi Target Tracking
• Cloud Recognition
• Object Targets
• Cylindrical Targets
• Smart Terrain
• Instant Tracking
• Extended Tracking

And for all the technically-minded among us, there’s obviously also the device support. I’m talking about supporting everything from OpenGL ES 2.0 and Metal 2 in iOS, Vulkan in Android and even DirectX 11 for a possible Windows Plugin. And also, how about a live preview in the Unreal editor for debugging or including target authoring functionalities? That sounds pretty utopic right now and without well-thought-through structures, Unreal UI integration and an absolute ton of testing, it’s going to be hard – but definitely not impossible!

We are highly motivated and want to further develop our Vuforia4Unreal plugin. Maybe there will also be some time to integrate AR into a second (maybe even better?) game engine with Vuforia, who knows? But now, there’s nothing more to say except: here’s the link for the Vuforia4Unreal GitHub repository. Have fun!