What AtScale wants you to know about Big Data's future

2017 is looking like it will be the year of BI on Big Data. Seventy-five percent of respondents in the 2016 Big Data Maturity Survey indicated that Business Intelligence is taking over as the #1 workload for Big Data.

AtScale, a company that provides enterprises with a fast and secure self-service analytics platform for Big Data, released the results of the survey, based on 2,550+ responses from big data professionals at 1,400 companies across 77 countries. The report reveals new and unexpected insights about Cloud, Big Data, Business Intelligence and Spark.

So what’s the best way to gain value from Big Data?

According to the survey, it is to solve scale-out needs with 56% of respondents claiming that they are more likely to yield tangible value from their Big Data than companies who use Big Data out of curiosity. The survey also confirmed that Tableau is the number 1 tool used for BI on Big Data, while Microsoft maintains its top spot as the BI tool for small data with Excel, with new entrant PowerBI helping maintain its leading position.

Self-service is another key to driving value from Big Data, with companies that have self-service 32% more likely to succeed. Unfortunately, 53% of companies surveyed still lack self-service access to Big Data. “Some of the findings of this year’s survey were surprising and indicative of upcoming market shifts impacting the industry,” said Bruno Aziza, CMO of AtScale. “The Cloud is taking on a surprising role when it comes to big data, while the viability of new technologies like Spark are still in question.”

Big Data’s future is Cloudy

Among many other insights, the survey reveals a surge in Cloud deployments for Big Data. More than half of respondents deploy Big Data in the Cloud today and 72% plan on doing so in the future.

“There’s been a clear surge in use of Big Data in the Cloud over the last year and what’s perhaps as interesting, is the fact that respondents are far more likely to achieve tangible value when their data is in the Cloud,” explains AtScale CTO and co-founder Matt Baird, whose company recently announced the newest version of it platform will provide a universal semantic layer that provides BI on Big Data across traditional data warehouses (like Teradata), today’s data platforms (like Hadoop) and future cloud based services (Google BigQuery, Microsoft Azure…etc).

Mature and savvy respondents

“The maturity of respondents in this survey is a key consideration,” says Thomas Dinsmore, big data analytics industry analyst and author of the book “Disruptive Analytics.” “One in 5 has more than 100 nodes and 74% of them are in production, indicating double-digit growth year over year.” The survey is a good representation of what AtScale sees across their customer base, which includes leaders across the financial services, insurance, healthcare, telecommunication, online and retail industry. Just this past week, Ventana rewarded Macy’s AtScale deployment with their Business Technology Leadership Award for Big Data, showcasing the maturity and savvy such enterprises have developed in the BI on Big Data field.

Business Intelligence soaring on Big Data

Another key trend observed in this report is the continued momentum of Business Intelligence for Big Data. Last year’s survey indicated that Business Intelligence had overtaken ETL and Data Science as predominant workload planned for Big Data. This trend continues moving forward.

Business Intelligence is #1 workload for Big Data and 75% of respondents are planning on using BI on Big Data. In addition, 97% of respondents indicated that they would do as much or more with Big Data over the next 3 months… 2017 might very well be the year of Business Intelligence on Big Data! The survey was conducted in collaboration with Cloudera, Hortonworks, MapR, Trifacta and Tableau.

Key findings

Mature and Experienced Respondents

- Close to 70% have been using Big Data for more than a year (vs. 59% last year)

- 74% of respondents have more than 10 nodes, close to 20% have over 100 nodes

- 73% of respondents now in production (vs. 65% last year)

- 95% of respondents have achieved positive value or are anticipating they would

Big Data Adoption Soaring

- 97% will do as much or more with Big Data over the next 3 months

- More than 53% of respondents are using Cloud for their Big Data deployment today 14%
of respondents have ALL their Big Data in the Cloud

- 72% of respondents plan on doing Big Data in the Cloud

- Two-thirds (66%) view Big Data as “Strategic” or “Game Changing”, while only 19% of
respondents consider it “Experimental”

#1 Opportunity: BI on Big Data

- Business Intelligence is #1 workload for Big Data with 75% of respondents planning on
using BI on Big Data.

- Self-service access on Big Data grew by 15% year over year, however, most companies
(53%) still suffer from a lack of self-service access to Big Data.

- Accessibility, Security and Governance have become the fastest growing areas of concerns
year-over- year, with Governance growing most at 21%.

- Organizations who have deployed Spark in production are 85% more likely to achieve
value.

The AtScale Big Data Maturity assessment methodology was developed in association with survey experts. The assessment tool is composed of over 20 questions, designed to assess a company’s stage in its Big Data journey. By taking the assessment, participants get an immediate score, access to a primer to help them mature and as well as aggregated results that they can use to benchmark their efforts against industry peers.