Code AR apps with one line of code

There’s a new spatial AR platform in town, and it's empowering app developers with the tools to quickly create and integrate location-based AR into their apps with a single line of code.

To create excitement for their new ReSight SDK before launch, the company recently held a developer contest, where more than 75 developers submitted their ideas for integrating AR into their apps.

We caught up with Omri Stein, the Co-Founder, and CEO of ReSight to talk about their contest, along with all things spatial AR. Plus how developers can get started using the platform.

ADM: What exactly is spatial augmented reality?

Stein: Spatial augmented reality is a subset of AR, where the digital information that is added to the world takes into account the physical space and its location in centimeters accuracy and changes its behavior accordingly. For example, in an AR pet application that is spatial, your AR pet will walk on the floor, won't sit on the table and would sleep only in its bed in your specific home.
 

ADM: How is ReSight different from other spatial/location-based augmented reality solutions?

Stein: ReSight was designed from the bottom up to be a privacy-first, distributed solution, that not only confers all kinds of neat features (near-zero latency, E2E encryption, mapless relocalization, etc.), but is also fundamentally different from all the other solutions. Most of our competitors talk about creating the "digital twin" of the world, a detailed 3D model of the entire world, that will provide the spatial AR capabilities; we, on the other hand, provide you with the better features without ever having to create or own this data.
 

ADM: How did ReSight get started? 

Stein: ReSight was started when our CTO, Eran, learned that technology has matured to the point that you can use common devices to add a digital memory layer to physical spaces. We knew that this technology was possible, but when we were able to make it work, we immediately recognized the possibility that it could explode in popularity, and thus went to work making the technology easy to implement.
 

ADM: Spatial AR has been historically costly and complex to implement, how does ReSight drive costs down for developers? 

Stein: ReSight is built by developers and for developers. This means that we do all the heavy lifting (such as CV, DL, mapping and localization, privacy, multiuser AR, data synchronization, etc.) in order to let you focus on what is really important: building a great AR application. One of our most loved features is our mapless relocalization, which allows you to add information to the physical space without having to handle the mapping of the area or updating and merging of maps, which saves a lot of time and development.

Epic Marble game made with Resight SDK

ADM: Code AR apps with one line of code: You’re providing an SDK that promises simple and seamless spatial AR integration into apps, how does this work?

Stein: To use the SDK all you have to do is to add a few lines of code and the binary into your application. We have plugins both for Unity3D and for RealityKit. Once you do that the API is very simple, a function to add an anchor to the physical world on which you can hang content and an event list you subscribe to that you get notified about the content found around you and its relative position to you. Everything else is done automatically.

ADM: How does ReSight map locations in real-time?

Stein: The ReSight engine harnesses AI and deep learning with our own proprietary computer vision pipeline to get from the device video stream to a compressed 3D map in real-time. The ability to do this mapping collaboratively with other users in real-time with the map merging done locally on the device (no calculation on any server) is unique to only our platform. So every time a user opens an application with the SDK installed, he is creating this 3D map that is correlated to the other maps that exist in this area immediately.
 

ADM: How will ReSight change the way spatial AR applications are created?

Stein: We enable a different type of spatial AR experience in apps that can be operated anywhere and on any scale, with other solutions such as Azure Spatial Anchors or Google Cloud Anchors. When you create a spatial app you have to know beforehand the location that it's going to operate in, because you have to map and then update this map constantly. This limits the applications to being POC or demos. You can't, for example, build even a simple app like digital graffiti. With ReSight, you can start creating these types of applications that users can use anywhere in the world and generate their content to persist. Another powerful ability is multi-user augmented reality, which utilizes our own network protocol, enabling spatial user discovery and super low latency (which is a must for multiuser AR).
 

ADM: What are a few new things developers can do with spatial augmented reality?

Stein: The possibilities are endless; with spatial AR you can enhance physical space with persistent and shared digital information. It can be for entertainment value (i.e. a game) or it can be a useful feature, like controlling the aircon in the room or getting from one place to another. We have seen a very wide variety of ideas being implemented and worked on. The most powerful application are those in which the actual physical location is as important as experience. For example, a game where the actual physical environment is important for the gameplay. Another example would be an education app that utilizes the child's spatial memory of his house to teach him different languages.

ADM: You recently hosted a developer contest for the ReSight SDK, what kinds of creative applications were submitted?

Stein: We had almost a hundred submissions for the competition from wildly different types of applications. Some examples include an app for teaching languages to children by adding notes to known object in their known surrounding, a social network app where you can take a selfie and leave them in space like a ghost for others to discover and interact with, an application for business to train their employees for a fire emergency, learning where are the fire extinguishers and correct behavior during a fire breakout, an app for creating and sharing an art exhibition in the streets of your city using famous works of art, an app for producing music in spatial space, which allows you to add musical elements to space and to generate music by interacting with them and of course sharing them with others. And the list goes on and on...

ADM: Which app won the developer contest and why?

Stein: Although many of the applications submitted were fantastic, the one that stood out for its effectiveness and beautiful execution is Epic Marble by Patched Reality. The app is a social AR game that utilizes the new LiDar sensor in the new phone to use the geometry of the environment for its gameplay. In the app, you place a digital marble anywhere you want and give it a slight nudge to start to roll out in the real world just like a physical marble (down the stairs for example), with the goal of making the longest and most elaborated run. By using ReSight's SDK, they were able to save their paths to the physical space so that the next time you or anyone else open the application in the same space, they will see and can compete with the best run that was there previously. This really shows how even simple use cases that use this memory layer can become powerful and social. By the way, you can actually check it out and download it from the App Store. It's really fun!
 

ADM: How will app developers benefit most from the ReSight SDK?

Stein: To really benefit from the SDK, we believe that developers should explore and better understand what type of experiences they can unlock with spatial applications. Our advice is not to look for the hype value of it, but rather to understand that having a persistent or multiuser AR experience offers real value for the end-users. We saw tremendous improvement from developers that started with a rough idea, but after actually experimenting with the SDK and seeing what it was capable of it helped bring their ideas to life.

ADM: How do developers gain access to your SDK?

Stein: Developers can register for early access to the SDK on our website: https://ReSight.io and usually gain access within a few hours. We do try to engage with the community and developers to offer them our advice and support, as it is really a fast-growing community of builders and creators.

About Omri Stein

Omri Stein is the Co-Founder and CEO of ReSight. He has an entrepreneurship background where he was focused on data science and machine learning. At his last company, G-Stat (Israel's biggest data science company), he created a new department that provided data science in the cloud to enterprise clients, which has grown into a successful business. Omri also taught mathematics at Tel-Aviv University. He is also a globetrotter, traveling around the world whenever possible.