A progress report on big data initiatives

Qubole, the big data-as-a-service company, has released the results of its State of DataOps report, a survey of IT and data professionals on the progress of their big data initiatives. The survey revealed a clear reality gap: while data teams have high confidence they can enable self-service insights to meet growing demands across the enterprise, few have delivered on that promise.
 
According to the survey, 76 percent of respondents said their company currently has a big data initiative, and another 20 percent said they plan to soon. In addition, 93 percent of respondents said business demand for big data analysis is growing.

Sixty-five percent of IT teams recognize that to get to ubiquitous access to data and analytics, they need to enable a self-service DataOps approach. And most respondents - 87 percent - felt confident to extremely confident that they could deliver self-service analytics.
 
Yet, respondents characterized their big data processes as still in the earliest stages of maturity: only 8 percent of respondents consider their big data initiatives to be fully mature.
 

A deeper dive reveals that IT is besieged by operational and technological challenges that interfere with improving big data maturity:

- Only 12 percent of respondents said they have multiple big data projects running
- 98 percent said they face numerous challenges with their big data initiatives
- 78 percent still support data requests on a project-by-project basis
- 45 percent can't satisfy business needs and expectations
- 61 percent rely on third-parties for big data expertise
 
"Having experienced this firsthand at Facebook, delivering on the promise of self-service access to data and analytics across the enterprise is extremely difficult and goes way beyond technology, involving rethinking processes, company culture and the operational model of the data team," said Qubole founder and CEO, Ashish Thusoo. "Until IT teams adopt a DataOps approach versus a more traditional command-and-control model, they'll remain a primary bottleneck to insights and their big data initiatives will continue to struggle. But there is a path - some companies have successfully made the transformation, and others can learn from their experiences."
 

Additional findings:

Data analytics is moving rapidly to the cloud:

Nearly six in 10 companies are currently using at least some cloud resources for big data processing: 14 percent are running all big data processing in the cloud and 41 percent are running at least some data processing in the cloud
Another 30 percent of respondents say that while they are currently running big data processes on-premises, they are considering cloud as a future option.

Amazon Web Services (AWS) leads the pack, with 32 percent of respondents saying they use Amazon's cloud platform for big data processing. Microsoft Azure, however, is not far behind, with 26 percent of respondents using it for big data projects. Google Cloud Platform is used by 12 percent of respondents and Oracle Cloud is used by 11 percent.
 
Businesses are in need of big data expertise:

83 percent of respondents said their data teams are growing
36 percent of respondents said they are having difficulty finding people with expertise in big data projects
31 percent said there aren't enough technical resources to run big data operations effectively
61 percent of respondents reported that their organization uses third-party consultants with big data expertise