The Rise of DevOps: From Backroom to Boardroom

Tuesday, April 1, 2014

The world of online games isn't just pixel placement, optimization tests, and timely calls to "grow revenue." The foundation supporting and delivering those activities is an underworld built on data, and DevOps might just carry more weight in revenue than Marketing.

In the competitive world of online games, you can’t succeed simply having the freshest title or most creative storyline. Game developers live and die by the quality of the player experience they deliver every day. But how do you balance the needs of millions of concurrent users and hundreds of developers, optimize a distributed cloud environment, push code daily, and stay sane?

Enter the realm of DevOps, a software development method that stresses communication, collaboration and integration of developers and IT, the new kings of generating and retaining customers and revenue.

While Marketing focuses on customer acquisition and channel management and Product Management focuses on pixel placement and in-game optimization tests, DevOps plays the role of unsung hero that keeps the release cycle on track and ensures the gameplay meets or exceeds the expectations set for player experience.

Global gaming means it’s always “peak game hour” somewhere around the globe, and DevOps is responsible for finding problems and implementing solutions when seconds count. Given its role in pushing out a constant stream of releases and staying so close to the player experience, DevOps might just have a bigger impact on revenue and LTV than Marketing.

It’s 10:00 p.m. in Russia - Do You Know Where Your Game is Failing?

“Our games consume our players’ valuable and limited entertainment time,” says Albert Ho, Executive Producer/Product Manager for Platform at Rumble. “If anything isn’t behaving correctly in our game or if key features are running too slowly, we could lose that customer forever.” An always-on game isn’t just an aspirational goal. It’s what your players expect - period.

Players vote with their wallets, immediately. According to Rumble, a bad player experience can reduce revenue from new players by as much as 70%. Even a small decrease of a very big number (for example, the number of active players who abandon your game due to a failure in one feature) is still a big number. Furthermore, features tend to fail under load, compounding the negative effects. If you stumble, you’ll feel the impact on your bottom line pretty quickly.

No One is Closer to the Customer than DevOps

When it comes to keeping tabs on what’s happening to the player experience, all roads lead through the massive streams of data generated by online games, including their log files. That’s why more game developers are beefing up their DevOps function and putting tools in place to help them understand the player experience in real-time. Here’s how you can get started.

Step One: Monitor KPIs that Signal Application Stability and Health

Too many game developers only find problems via “black box” testers or player complaints. With as much as 70% of new revenue at risk every time a release gets pushed, this approach does bring predictability to your business. Even the best code can fail when it meets the real world. You need to be ready.

The best game developers have established KPIs that signal the stability and health of their games; and they monitor these KPIs very closely after every code push. They might include:

Number of stack traces and error exceptions.
Incidence of client-side problems such as failure to connect, game lag, characters disappearing, or models not rendering.
Player activity in key features.
Completed and uncompleted transactions.

With KPIs in place, DevOps has an alarm system that tells it when action is required and that provides objective data to galvanize the development team.

Step Two: Isolate Problems with Insights from your Logs

Game developers know the value of data in understanding the player experience; and with the right instrumentation, you can understand what your system is doing at that same level of granularity. In order to get the most out of your log data, you should:

Structure your logs so they can be easily parsed and analyzed. JSON offers a huge amount flexibility in defining and updating your fields.
Log unique session or user identifiers so you can trace events from client to server side, and across multiple servers and APIs. One of the unique challenges for game developers face is the huge diversity in players’ client hardware, software, and networking environments.
Put a solution in place that allows you to efficiently search and analyze your log data. Writing shell scripts or grepping through tons of log data is no fun.

During the problem isolation phase, the DevOps team works closely with development to get the right people involved and bring the game back to a healthy state.

At Loggly we have put together some best practices for appropriate instrumentation that apply to any cloud-centric application. In addition to the areas we outline, game developers need to look carefully at the client side for the reasons we describe earlier.

Step Three: Help Marketing and Customer Support Close the Loop

Depending on the magnitude of the problem, DevOps might give Marketing, Customer Support, or Community Managers the information they need to proactively reach out to affected players. It’s always less expensive to save a customer than to replace one, so the most important customer isn’t your next one but your last one.

DevOps is in a unique position to quantify the impact of a particular issue, determine its cause and effect, and provide data on exactly which players were affected. Then, the business can execute on retention programs that make bottom-line sense.

DevOps Provides Valuable Insurance

The cost of player acquisition is growing in many markets. Players are fickle, and they often have a “hive-mind” mentality and very little patience for tolerating a bad experience. DevOps is the insurance that your marketing dollars are not being diluted by an in-game experience that is not meeting player expectations. It’s data-driven optimization of the most important variable - the core of your application. And the math tells you that pound-for-pound, a dollar spent on DevOps to optimize player experience drives more revenue than that same dollar invested in customer acquisition.

This content is made possible by a guest author, or sponsor; it is not written by and does not necessarily reflect the views of App Developer Magazine's editorial staff.