3 steps to Succes
Getting a game analytics project off the ground is not tricky—succeeding with such a project....that's a different story. Preparation is key.
To quote Benjamin Franklin, "By failing to prepare, you are preparing to fail."
Let's explore some essential steps we take to ensure success for our development teams and publishing partners.
Start Early!
Developers and publishers widely agree that early knowledge of how players perceive the game is crucial in identifying gameplay changes and final balancing before launch. To achieve this, a common practice is to capture data from your golden cohort—early super-engaged players—to establish a baseline for retention, progression, and general engagement. Unfortunately, this opportunity is all too easily squandered if telemetry is added too late or is not correctly validated before launch.
So start early. Implement and validate data before launch to benefit from the following:
Time to design and iterate your telemetry
Conduct continuous quality checks
Early data analysis and identify telemetry shortcomings
Almost without fail, projects that delay telemetry integration are inevitably hectic at launch. These projects usually prioritize user-facing features and bugs, defer telemetry integration, and typically enter launch with telemetry technical debt, drastically slowing the team's time to insights.
A couple of years ago, we were called in to consult on a project just weeks before it was scheduled to go live. As a competitive co-op experience, the game's ability to provide a fantastic player experience relied heavily on matchmaking. After reviewing the incoming data, we noticed one grave omission from the telemetry design and events - there were no unifying IDs.
While we could evaluate game difficulty and match outcomes for individual players, certain critical data points associated with matchmaking and competitive balancing were unavailable. There was no way to link players by their groups. This denied us the tools to determine if players were matched with players who outskilled or outranked them.
As a result, the developer was flying blind. They had to rely on gut feeling and common sense when updating the game to decrease player drop-off due to matchmaking issues. Our limited findings were better than nothing, but as you can imagine, without any chance to validate if the proper fixes were being deployed, a huge opportunity was wasted.
Get buy-in!
Building on the motto of starting early, we always recommend promoting analytics early within the team. The goal is to ensure that telemetry and analytics are considered integral to the game and to facilitate data literacy within the organization, ground up, and one project at a time. One of the more critical aspects of promotion of analytics is to acquire champions from outside the data team. By starting the analytics project early, you gain additional time to promote the value of analytics, build rapport, and acquire buy-in.
Start with production. Producers can be some of the most influential people for whom to get buy-in. They are often accountable for the product's success and can help secure telemetry implementation and support resources. As masters of the schedule, they will ensure that telemetry work is adequately prioritized and can play a critical role in deciding whether or not to "shoehorn" telemetry fixes into an update or delay submission until a critical telemetry issue is resolved.
Include your tech leads, but don't underestimate the effort. Tech is critical to your analytics project's success and product insights' efficiency. Organizations new to telemetry and data often face two recurring challenges:
Telemetry code is critical to business success but often falls in the hands of the least experienced technical resources.
Telemetry code is considered unnecessary technical overhead, compromises game performance, and adds to the bugs overhead.
Implementing events too early can be problematic if gameplay systems and mechanics are immature or planned to undergo significant changes. However, getting buy-in and starting telemetry design early can mitigate many underlying technical concerns. This is where communication across multiple teams is essential - it provides an opportunity to identify gameplay systems that still need to mature. This is also where early cross-team bridges are built that analytics projects will benefit from later, which leads to our last point.
Communicate and collaborate!
Within a development studio, telemetry design is a joint, multidisciplinary effort requiring discussion and alignment. Things get more complicated when multiple remote teams are involved. Few things can replace a good face-to-face conversation, but calling a meeting to discuss every minute change isn't possible. Installing a best practice is to have a forum to discuss telemetry items between meetings, which remains as close to telemetry documentation as possible — cue the collaborative telemetry document.
Collaborative telemetry documents come in many shapes and forms. Still, they are invaluable in serving two equally important purposes — documentation and communication — for multiple stakeholders. As a cross-functional tool, they support the analytics project from start to finish, including telemetry design, event integration, and insights creation.
Regardless of the solution or format, the tool should fulfil the following:
It supports asynchronous, collaborative telemetry designing
It supports event implementation status tracking
It serves as a de facto source for telemetry change tracking
It serves as a critical tool for telemetry bug tracking
It is accessible to multiple teams and supports varying needs
It serves as a historical record of all telemetry-related activities
All game analytics projects should begin with telemetry design. As soon as telemetry ideas are generated, they need to be documented. Those ideas will then travel to a developer for implementation, who will likely have questions about the design to identify the best way to retrieve the necessary information from the game.
This conversation could happen in Slack or via a Jira ticket, but imagine the potential for these discussions to happen in the same place as your telemetry design.
Once implemented, clarifying comments from the developer or notes related to challenges or quirks related to the event implementation can be added. Such notes can be about difficulties expected or use cases not yet covered.
Notes like "This information comes from the client and may not be reliable. We should consider making this a server-side event" or "This event may not fire frequently enough to capture changes in player wallets" can be tremendously helpful for everyone involved! Such notes can prove invaluable later as data quality checks or analyses are performed.
Let's not forget that teams change, and people transition roles and projects. Historical information about quirks and challenges will allow easy onboarding of a new analyst or analytics stakeholder, making things flow.
Consider the example: the event "Tutorial Completed" was initially implemented to fire after the first mission concluded and player onboarding was completed. As the game grew and the feature set expanded, additional tutorial steps were added and extended into the second mission; the event "Tutorial Completed" was moved to reflect this design change.
This critical information would be captured in our collaborative telemetry documentation, which would define how event firing should be revised and inform analysts on how their analysis must evolve.
We often expand the document to include descriptions of the purpose of each telemetry event. This provides context, explanation, and, most importantly, buy-in, as it outlines a clear benefit of telemetry events for the product as a whole.
Below is an exampleof a simple telemetry implementation tracker and an extended collaborative telemetry document.
Like a feature or gameplay mechanic, telemetry will undergo many iterations in the prelaunch phases of product development. Launch your analytics project with early telemetry design. Build buy-in, establish champions throughout your team, and leverage collaborative telemetry documentation to ease the process. By successfully preparing, you are preparing for success.
