Open Sesame technology changes the world for paralyzed people

Sesame Enable seeks to open a new door to independence for millions of quadriplegics and people with disabilities around the world through the development of the first touch-free smartphone and tablet app called Open Sesame.

Open Sesame technology works by tracking user’s head movements using the front-facing camera of a smartphone or tablet. Users activate the app through voice command, then cutting-edge computer vision algorithms create a cursor that appears on the users’ screen that is moved through head movements. It is available on Android 7 and higher operating systems and through the Google Play store. We recently spoke to co-founder, Oded Ben Dov about how this technology was born and the implications it has on the future.

ADM: Let’s talk about how the Sesame Enable technology behind the app works. Walk us through it.

Dov: Users will first need to download Open Sesame from the Google Play store. Open Sesame is compatible with smartphones and tables running an Android 7 or higher operating system. Once installed, using voice control to activate the front-facing camera of the user’s device, Open Sesame tracks head movement to unlock touch-free texting, social activity, surfing the web and so much more.

Say “Open Sesame” to your mobile device to unlock it. From there, the device’s camera scans for the user’s face and then tracks head movements as commands to tap or swipe. The cursor will follow your head movements. So, if you turn right, the cursor will go right, or left and up and down. We use proprietary computer vision algorithms to track the user’s head with extreme precision, constantly learning the face and bettering the detection in real time. The great library OpenCV was of help in the basic building blocks of our home-grown technology.

Last - often disregarded but ultra-important - our UX/UI has been refined over the past 5 years, always accounting for user feedback and seeing how we can better the experience. For our users our UX is their gateway to the device, so we have to get it right. We are extremely proud of what Open Sesame has become today and we’re excited to continue its evolution alongside the evolution of mobile devices.

ADM: Tell me about the latest updates to Open Sesame. How has the technology evolved?

Dov: In our 5-year journey, the biggest earthquake took place last year. In September 2016, Android announced a new accessibility feature called Touch Dispatch API. For apps it designated “Accessibility Services,” Android provided the option to mimic a touch event anywhere on the screen, including other apps.

Up to that point, an app couldn’t control other applications, which is an obvious and important security measure. Prior to Touch Dispatch API we had a custom-ROM built, in which there was much more power over the system, but the end product would not be supported anywhere. We knew it would be a challenge for carriers to deal with custom OS installations, so we actually installed our technology on phones ourselves and sold them to our users.

From the September date Touch Dispatch API was available, we transitioned as fast as we could to a pure PlayStore app model, which is night and day compared to the custom ROM and hardwired phones. Our users can now download the app to any Android device they like, in real time, from anywhere in the world. We’re seeing great traction.

In fact, after we first read the description of the API, and receiving previous recognition from Google, we like to think we had some part in making this important evolution happen. We were strong in our belief of proving life-changing value to each of those first-generation pre-installed phone users, and we believed a solution would eventually come.

ADM: I hear you have background in gaming, how did your journey take you from the gaming industry to assistive technology for people with physical disabilities?

Dov: I was working in the gaming industry you’re right, and was in fact demo’ing a game using head gesture technology on TV that caught the attention of my now co-founder. After that appearance, he called me and said: "Hello, my name is Giora, I can't move my hands or legs. Can you make me a smartphone I could use?"

Giora Livne, my co-founder, became quadriplegic after suffering an accident helping a friend, and contacted me in search for a solution to help him regain independence in his life, along with the independent use of a smartphone. The rest is history.

ADM: Tell me about the team behind Sesame Enable. How did you come together?

Dov: While speaking with Giora after my television appearance, we realized he lives a block away from my parent’s home. We set to meet, I got to know him and understand his needs, and realized we would need an Android developer to tackle our proposed solution (I was an *ahem* awesome iOS developer before, and didn’t have the resources to take on Android at the time).

So we brought in our third partner - Vlad Ostrovski - and that really marked the kick-off for Sesame Enable. At this point, we were all of the same alma mater -The Technion, Israel’s Institute of Technology - representing three generations.

As we grew and started dealing internationally Rowee Benbenishty joined as VP Business Development, and just recently Shai Eilat who is based in New York and heads our U.S. operations and user awareness campaigns.

ADM: What is the need for hands-free technology like this? Where are other’s lacking?

Dov: The need is great - there are more than 35 million people around the world and 1.4 million Americans with a physical condition that prevents them from using a smartphone or tablet. These people are essentially shut out of the mobile device revolution - no matter how advanced smartphones or apps become. Without a way to participate, let alone communicate, it makes this a larger societal issue with an easy solution based on technology advancements. Sesame Enable was the company to take on the challenge and develop the solution. It’s not that technology is lacking, we’re just the company to have dedicated resources to furthering assistive devices with gesture technology. We hope to be a positive example to others to continue to adapt technologies and solutions for this demographic.

Other gesture technology solutions today are a bit outdated and rely on external dedicated cameras that ask that you place a marker on your head. Sesame Enable technology can run off the simplest of cameras, no marker required. We also see other hands-free technology for PC’s, not smartphones or mobile devices. The entire mobile eco-system, either Android or iOS, has overlooked the ability to provide a truly mobile solution. I say “outdated” because we believe Sesame Enable brings the 21st century to our users, and we hope to revolutionize their world like able-bodied people have witnessed with smartphones.

ADM: How will apps like Open Sesame change the app development landscape moving forward?

Dov: Our mission is to connect people with physical challenges to the world around them. We hope our mission and success will encourage other individuals and companies to develop or adapt their products and applications to a population that can greatly benefit from assistive technology.

ADM: How is does a hands-free app work with other apps on a phone or tablet?

Dov: Open Sesame is special type of app on the Google Play store called an Accessibility Service. To be considered for the designation the app has to abide to strict regulation, and the user needs to very consciously approve the app to run on their device.

Once the user approves Open Sesame, we use the Accessibility Service dispatchGesture API to interact with any application on the screen. We use the camera with computer vision algorithms to track the user’s face, and translate that to a location on the screen. When the user dwells with the cursor in one place, he can then interact by tapping or swiping. We use dispatchGesture to send the user’s intention to the app. It’s important to say we are oblivious to the content for the app we’re interacting with - we are just creating a virtual finger for our user who can’t use his own.

ADM: How will expanding into the US help drive development?

Dov: Having a headquarters in the U.S. was a strategic business decision to support continued growth in users. From Veterans to those with chronic diseases or sudden injuries, there are millions of Americans who could benefit from Sesame Enable, regain control of their lives and feel as though they are connected to society without barriers. To best support these potential users we felt we needed to have a team on the ground in the U.S. to generate awareness and advocate for assistive technologies. We have a team of five based in our new Bethesda, Maryland office and they will work closely with health and disease-focused organizations to generate awareness of the app among those who can benefit from it, as well as caregivers. Our location in Bethesda allows for close proximity to Washington, D.C., where we know we’ll have opportunity to influence healthcare and subsidy policy changes needed to make our technology available to all Americans who can benefit from it. 

ADM: What else is coming down the pipe? Anything you can preview for us?

Dov: We’re really proud of our new lock-mechanism that came out just last week - you can see an example here but it’s really something else to try it for yourself. It’s just one of many examples of UI challenges we face along the way - a lock that won’t annoy you, always watch you, won’t unlock by mistake as you move/talk to people in the room, and still unlock with a simple head gesture.

There are several exciting things I can’t divulge right now, but looking forward we expect to expand both vertically serving the same audience with more products and innovation, and expanding to perhaps other disabilities where technology can bridge gaps.

About Oded Ben Dov

Oded Ben-Dov is fluent in 11 languages, two spoken and nine programming. It was this fluency that enabled him to solve the most important question of his professional career - "Hello, my name is Giora, I can't move my hands or legs. Can you make me a smartphone I could use?"

A globally recognized technology expert turned social entrepreneur, today Ben-Dov is CEO of Sesame Enable, creator of the first touch-free smartphone app for quadriplegics and people with disabilities called Open SesameTM, and an advocate for life-changing, technology solutions for people with physical challenges.