splatlog_2: OpenCV image recognition in iOS 11

Note: While I applied these techniques in OpenCV to augmented reality their applications stretch well beyond this context.

OpenCV Take Aways

  • Template Matching is scale and orientation dependent, don’t use it.
  • True image recognition requires SIFT (Scale-Invariant Feature Transform) or SURF (Speeded-Up Robust Features)
  • True image recognition also requires performing thresholding on the results of any function you use in OpenCV

As part of my journey into creating tools to share augmented reality art, I quickly ran into a well documented problem in the world of AR: mapping. In VR we have the luxury of a perfectly understood universe, while in AR the whole problem is understanding the universe that already exists in front of us. We’re getting to the point where we can understand the universe in our field of view, but not nearly at the point where that understanding is shared between instances.

In a rudimentary attempt to solve this problem, I decided to see if I could use OpenCV to find anchor images and place their associated object in the right place relative to the anchor. The idea being:

  • Download a list of nearby drawings with an anchor image and offset vector
  • Once a second, pull the latest camera image from ARKit
  • Use OpenCV to search the camera image for instances of all nearby anchor images
  • When an image is found, ask ARKit where the nearest recognized point in the room is to the anchor image
  • Then add the offset vector to the anchor image position to get the location of where the drawing should be placed
  • Place drawing at computed position!

After a lot of fiddling I got this to kinda work. It’s not great, and I wouldn’t recommend it be used in practice for AR object placement with an offset because it’s not very performant and the translation into the AR world is not very accurate. This might still be useful for others to anchor objects directly to a fewer number of images, and definitely has many use cases outside of AR.

GitHub Example:

Here is a link to a GitHub example project for iOS 11 and OpenCV 3 that uses SURF to track reference images in the camera.

splatlog_1: conception and direction

Art challenges technology, and technology inspires art.

– John Lasseter

Every technological advancement brings with it innovative artists who use it in previously unimaginable ways to convey the wonderfully broken ways in which they perceive the world. Now we stand at the dawn of a new technological era that is poised to forever change the way humans interact with technology, the world, and each other. Augmented reality.

We are on the cusp of a reality that is no longer comprised of the strictly tactile, and the opportunities are tremendous. While the various technologies that comprise AR have been around for a long time, in 2017 both Apple and Google have announced developer tools to significantly lower the barrier to entry to AR software development. I believe we have now reached an inflection point in the adoption and proliferation of AR in the popular culture.

As they do with any new medium, artists have been pushing the boundaries of AR and VR for decades to create and share experiences that eschew the normal boundaries of perception. The elephant in the room for any work like this is, of course, Zach Lieberman. Zach’s pieces obliterate our traditional understanding of reality and construct a new and interactive universe before our us.

Here’s @jonburgerman drawing with an arkit app I am working on #openframeworks

A post shared by zach lieberman (@zach.lieberman) on

Now that the technologies behind this kind of work have begun to reach a point of maturation and large scale accessibility, it’s imperative that the creative efforts at the heart of them be shared with the world as well. It is for that reason that I’m working on splatwire: an app to create and share augmented reality art.

As it currently stands, splatwire consists of two parts: a backend with a public API for uploading 3D art and marking its geo-location, and a client app that allows you to view the art people have uploaded and create simple pieces of your own. With this setup, every phone becomes a portal that turns the world into an art museum for everyone: virtual street art accessible to novices and professional artists alike who look to push the boundaries of digital art.

splatwire is in the very early stages of its development, but it’s current progress can be seen in this video recorded as a submission to NYC Media Lab ’17. (splatwire has been accepted and will be featured in the Demo Expo)

In the short term, my to-do list is as follows:

  • Finish adding anchor points to drawings
  • Implement OpenCV template searching to use anchor images to place drawings
  • Determine standard for sharing more intricate drawings, especially 3D ones
  • Upload drawings to database
  • Standardize drawing view with ability to switch between multiple kinds of drawing techniques including 3D drawing
  • Show drawings on map
  • Add ability to view user profiles and see all drawings