Embracing operational analytics for better development

Operational analytics defines how we use business analytics to improve business operations. Amazon DynamoDB is a key-value and document database that delivers single-digit millisecond performance at any scale. Hundreds of thousands of AWS customers have chosen DynamoDB as their database for mobile, web, gaming, ad tech, IoT and other applications that need low-latency data access at any scale.

With DynamoDB usage maturing in organizations, there is an increasing need for operational analytics and real-time business reporting on it, which requires the ability to search transactional data, run aggregations and join the data with other datasets.

However, as stated in the Amazon Web Services blog, "DynamoDB is not suitable for running scan operations or fetching a large volume of data because it’s designed for fast lookup using partition keys. Additionally, there are a number of constraints (like lack of support for powerful SQL functions such as group by, having, intersect and joins) in running complex queries against DynamoDB."

All businesses today are a series of real-time events. But what separates the good from the great is how they capture and operationalize that data. In this Q&A, Rockset co-founder and CEO Venkat Venkataramani explains how organizations can embrace operational analytics, and the role of DynamoDB in that journey.

ADM: Can you explain the role of operational analytics within digital organizations?

Venkataramani: Operational analytics can be best explained through an example of how it is being used within a modern organization.

For the last decade, the big data movement has been about capturing a lot of data and crunching it to identify problems and make better decisions. Companies like Uber have talked in-depth about how they use real-time analytics to create seamless trip experiences. What's refreshing about Uber's approach is that it doesn't collect and store data hoping to find insights -- instead, it has operationalized event data to take automated actions in the Uber app in real-time.

This need has spurred the growth of operational analytics more broadly, even beyond Uber. Operational analytics is a very specific term for a type of analytics that focuses on improving existing operations. This type of analytics, like others, involves the use of various data mining and data aggregation tools to get more transparent information for business planning.

The main characteristic that distinguishes operational analytics from other types of analytics is that it is “analytics on the fly," which means that signals emanating from the various parts of a business are processed in real-time to feedback into instant decision making for the business.

ADM: In which industries are you seeing the biggest surge in operational analytics?

Venkataramani: It's not just modern companies that have real-time data that can drive intelligent actions at the drop of a hat. In fact, IDC predicts that by 2025, nearly 30% of all data created will be real-time (compared to 15% in 2017).

A cosmetics company can use point-of-sale data to manage inventory and ship more units to local stores that are running promotions. A medical device company can deliver more insulin through a smart pump based on a patient's fluctuating glucose levels. The possibilities are endless, and it is a movement that organizations spanning industries are experiencing. “Big data” is no longer relevant; any digital organization has massive amounts of data, now it is all about what you can do with it.

ADM: In which industries do you see the most potential?

Venkataramani: While companies spanning a variety of industries are finding ways to operationalize their data, there are a few in particular that can benefit from Rockset on DynamoDB:

• Gaming companies can monetize their products by providing opportunities for in-app purchases
• Organizations in the virtual reality space can tailor content for customers, as they face an imperative to provide viewers with the right content at the right time, depending on when and where they are watching
• SaaS companies can provide an inside, real-time look into their users, such as when they sign up, what the journey is with their product and how many hours users are spending with the product

ADM: What is causing this shift, and how is it changing the way developers work?

Venkataramani: The combination of a variety of technological advancements – from public cloud to open source to microservices – has helped democratize data and give developers more freedom than ever before. No longer are they stuck using the language and environment their IT department provided; developers are now free to use anything they want as long as it shows business value. And now, we have machine learning and AI, which are in the early days of perhaps reshaping the world as we know it.

As a result of these innovations, the world and our expectations have changed dramatically. We expect to have access to everything immediately – whether that’s the delivery of our groceries, the ride to our destination or any information about anything we want to know about. It is now up to developers and data engineers to keep pace with consumer demands and identify the technologies that will most benefit their organization on their journey to operational analytics.

ADM: How does Rockset's integration with DynamoDB help developers?

Venkataramani: Rockset is an operational analytics engine that is entirely serverless, which means it does not require provisioning, capacity planning or server administration in the cloud. Developers and data engineers can run complex queries with the full power of SQL and choose to visualize the data in live dashboards such as Tableau or build custom applications using Rockset’s real-time SQL.

ADM: What makes this announcement different than anything else on the market?

Venkataramani: Millions of developers have embraced NoSQL databases. And the biggest trend in data today is around real-time operational analytics. The problem is that NoSQL databases have just one big downside: the lack of SQL for powerful analytics on that data which translates into the inability to use favorite BI tools (like Tableau) in real-time because those tools expect SQL.

Rockset is a serverless search and analytics engine that provides real-time SQL on NoSQL data from DynamoDB — an industry first.

Once provided with reading access to a DynamoDB table, Rockset reflects changes as they occur in DynamoDB by making use of changelogs in DynamoDB streams. This gives users an up-to-date – within a few seconds – indexed version of their DynamoDB table in Rockset. On top of that, each SQL query against this table is distributed and executed in parallel to ensure that query results return in milliseconds.

With this release, Rockset supports the ability to:

• Visualize DynamoDB data in leading SQL-based visualization tools, including BI market leader Tableau, Apache Superset, Redash, and Grafana, in real-time.
• Build custom interactive dashboards and real-time applications using SQL on DynamoDB data.
• Join DynamoDB data with data in Kafka event streams, Amazon Kinesis or Amazon S3.

ADM: Which other tools can developers use to harness operational analytics?

Venkataramani: A scalable real-time data pipeline is vital to building an operational analytics stack. Open source projects such as Apache Kafka, managed services such as Confluent Cloud, Amazon Managed Streaming for Apache Kafka, and customer data platforms such as Segment are all great options to provide that scalable real-time data pipeline.

ADM: Are there any outdated tools or systems that are stifling innovation?

Venkataramani: Unfortunately, many businesses are still stuck in the old world of data where they had to choose between transactional and analytical data systems. Typically, transactional systems are online databases that are best suited for order entry, financial transactions, customer relationship management, and retail sales, but they're not ideal for complex queries like determining how much of a particular product the business sold in a certain region this week and how that compares to last week.

For such complex queries, analytical systems like data warehouses have been the go-to solution, but they tend to be too slow because they need new data to be prepared, loaded and analyzed in batches. Operational analytics demands new capabilities that are not possible with transactional databases or data warehouses.

ADM: What is the biggest challenge developers face as they get further along in this journey?

Venkataramani: One of the biggest challenges in the move toward operational analytics is that the existing data stack is simply not able to handle the pace at which new data comes and is not set up to process the new types of data being generated. It is incredibly tempting to make small incremental changes to modernize the current data infrastructure, but the reality is that the most successful new projects embrace a whole new cloud-native stack that allows them to move fast and show real value quickly.

About Venkat Venkataramani

Venkat Venkataramani is CEO and co-founder of Rockset. He was previously an Engineering Director in the Facebook infrastructure team responsible for all online data services that stored and served Facebook user data. Collectively, these systems worked across five geographies and served more than five billion queries a second. Prior to Facebook, Venkat worked on the Oracle Database.