Companies plan to invest in DataOps new survey finds

Inspired by the DevOps movement, Nexla, the inter-company Data Operations platform, is proving the DataOps movement is real by just announcing the results of the industry’s only annual data operations survey. The survey tracks the adoption and best practices of Data Operations (DataOps). It found that a staggering 73% of companies are investing in DataOps in 2018.
 
The survey, commissioned by Nexla and conducted by research firm Pulse Q&A, asked 266 data professionals about how they use data, their team structure, and data challenges.

Machine Learning and AI, real-time streaming fueling need for DataOps

Eighty-five percent of companies report that their company is working on machine learning and AI today, up from 70% in 2017. This work will continue, with 83% of companies saying they will do more in machine learning and AI next year.
To feed their models more data, 85% of companies are ingesting data from third parties. The survey found 54% of companies are ingesting data from more than 10 partners.
Real-time streaming data is becoming critical to these efforts, with 58% of companies reporting they ingest data this way.

Data teams understaffed to capitalize on opportunity

Fifty percent of respondents reported they do not have enough backend data engineers to support their company’s data needs 
Seventy-three percent of companies report that they have plans to hire DataOps professionals in 2018
The average company only has one data engineer for every 5 business users, processes 2.7 GB of new data a day, and manages 4,300 data sets.

Overworked data teams are craving automation

• Data engineers reported they spend an average of 18% of their time on troubleshooting. The average company loses 180 hours a week to troubleshooting.
   
• Data pros see an opportunity for automation in all aspects of their work. 56% believe data clean-up could benefit from automation in the next two years, with 47% saying analytics could benefit and 46% saying integration could.
   
• Data format consistency was the number-one challenge cited by data pros, with 39% saying this was a key challenge. 60% of companies ingest data in three or more formats, adding complexity.
   
• Data integration and reliability of data pipelines were the #2 and #3 most-cited challenges with 36% and 35% respectively. Data pros are spending 18% of their time on these activities.

 
“It’s clear that backend data teams are strapped for resources, which is why 73% have plans to invest in DataOps,” said Saket Saurabh, Nexla Co-founder and CEO. “DataOps is as much about people as it is about tools and processes. To really drive value from machine learning, AI, and advanced analytics, data teams need to stop troubleshooting and start automating. We built Nexla to help data teams create automated, repeatable, and scalable data flows so they can focus on deriving value from data.”
 
Data Operations is an emerging organization-wide data management practice that controls the flow of data from source to value, with the goal of speeding up the process of deriving value from data. The outcome is scalable, repeatable, and predictable data flows for data engineers, data scientists, and business users. Nexla is a leader in this new movement, building tools to help teams execute against their DataOps strategy.