Under the Hood of Effective Game Testing

A long time ago, in a galaxy not far away, there was a company, produced the very first maze game named “CocoJambo”. Players loved the gameplay, however, it was riddled with bugs, and that fact ruined the fun and sales. As a result, it was decided to create a team that will test the game before launch to ensure there were no defects that could hinder the gaming experience. Thus, the Dark Team arose.

Right Time, Right Place

Nowadays there are tons of amazing games on the market that win the hearts of millions of people. The gaming craze is immense. The global gaming industry continues to grow with the increasing per capita income, rising number of players, considerable activities from startups. Its revenue is expected to hit $300 billion by 2025, according to GlobalData, one of the well-known analytics companies. Further, NewZoo statistics tell us that one-third of the world’s population are gamers. But it was not always like that, wasn’t it? In 1983, the Atari shock brought the entire industry down, leading to the bankruptcy of the companies producing video games consoles and home computers. Curiously, abysmal quality games have been one of the reasons why the game market might have died forever.

Always About the Quality

The game testing is a crucial part of the game development process that allows to reveal and nip as many crappy problems as possible in the bud. Effective game testing derives from a well-organized approach and seamless implementation of the test automation framework. Add to that the picture-perfect integration with your agile process at your back, and your players won’t ever face invisible walls, missing textures, or unpleasant interruptions. From now on, there is no need to bury your game in the desert. Just start with building your testing process.

“Gorilla Jam” Testing Rush

“Gorilla Jam” Testing

Imagine we are working on “Gorilla Jam”, a remake of the good-old “CocoJambo” maze. The gameplay is simple: the player controls a tourist through an enclosed maze. The goal of the game is to eat all bananas placed in the maze while avoiding the massive gorilla that pursues your clumsy character. The core difference between “CocoJambo” and its remake is the boosters that allow your cumbersome tourist to defeat an angry animal.

Our development team includes one manual QA, one automated QA, two full-stack developers, one junior developer, and two artists. The team created two POCs to verify if our boosters — orchids to move faster, vines to fly over the fury gorilla — would be entertaining enough. The release is planned at the end of the two-week sprint. Thus, applying QA efforts for maximum impact can be done in a few basic steps.

1. Requirements’ Validation

In the first place, we need to break down our project into reasonable chunks and analyze them from the perspective of testing. Let’s assume that our game design document (hereinafter GDD) is unambiguous, completed, and specific, and includes all the essential updates, new configurations, extra features, and future expectations. It’s not a secret that top-notch GDD will let the team build a cohesive, consistent “Gorilla Jam” with “easy to read and maintain” code — the crown jewel of any project.

2. Creation of Test Strategy & Test Cases

From my perspective, the most monotonous, but indispensable activity. Test strategy is a GDD for the game testing process. Carefully crafted and well-described test strategy will include, in the first place, the key points in manual and automated testing along with the definition of the test approach, test environments, and bug tracking process. Thus, we will know precisely what we are going to test and why it should be tested, as well as whether these tests can be automated with our Best in the World test automation framework. On top of that, we will be aware of the most relevant types of testing, namely functionality, compatibility, regression, progression, ad-hoc, etc.

As a result, we are ready to design test cases for both manual and automated testing. For instance, in “Gorilla Jam” we will test manually the new features like boosts. Therefore, while the developers are working on orchids’ and vines’ implementation, QAs are designing their test cases based on the requirements, described in the GDD. All the test cases essential for release will be added to the test plan.

3. Running the Tests

Declaring the build with boosts ready for testing, developers are rubbing their hands in glee. Make sure the build version we are going to test is correct and contains the necessary changes. Both manual and automated test suites are running against the new build. Manual — for the new feature, automated — for regression to confirm nothing was broken during implementation. All the steps in test cases are rigorously marked as pass/fail.

Another helpful hint here is to record the whole test session. This allows the developers to analyze the game’s behaviour smoothly, helps to avoid the misleading description of the defects discovered during the testing, or ensures the new build delivered is error-free. All the bugs found should be documented, even if they were fixed on the fly.

4. Reporting Test Results

Sometimes there is no need to provide the full test report. A short test summary could be enough to notify your stakeholders/team about the test result. These summaries could be generated daily, weekly or monthly basis, depends on the testing and business agreements, and should contain information about:

• the current test plan status (% of test cases and tasks that have been tested)
• link to the test plan in Jira
• the list of critical defects/blockers + links on Jira
• the recommendations for the release (from the QA perspective should be released or required additional work)

5. Bug fixing

The crappy errors discovered during testing are discussed with the stakeholders/team. It occurs that when a player uses orchids as the boosts, the tourist character runs backwards. This defect impacts the whole gameplay since it’s challenging to avoid a gorilla moving back to front. Add to that the crash when a gorilla reaches vines, and you will have the new date for your release.

The Dark Team members wore black cloth and laughed ominously each time the new defect was found. Their testing strategy included emotional testing, implying that the Dark Ones will interact with the game as close as a final user would do. To bring up the verdict whether the game feels nice to play.

Manual Testing vs. Game Test Automation

Good to have both manual testing and game test automation. We need to ensure our “Gorilla Jam” works as intended on all the platforms, on various OS versions, and after all updates in a quite short time — two weeks. The team smudges testing over time avoiding “last-minute” accumulation of the efforts and prevents errors in production using manual and automated testing.

Having Well-Balanced Team

We are lucky to have one strong manual QA (let’s call him Luke) free of deadly sins and one automated QA (lets his name be Han) in the project. Thus, while developers are working on boosts’ implementation, based on the art created in the last sprint, Luke is crafting his test cases for manual testing, and Han is engaged in polishing and updating testing scripts created before according to the changes. Our automated QA will reuse the improved version of the scripts during testing after code freeze.

At the moment, Han has already automated login flow, tutorial, gameplay flow including starting and completing the level, ads and purchases, press actions, and setting menu selections. Han is going to create automated tests for boosts in the next sprint, adding this feature to his collection of automated test cases for the production environment. On top of that, he has some ideas related to the stress and load testing for “Gorilla Jam” which he will discuss with the Product Owner just after the sprint will be completed and an increment released.

On top of that, Han is thinking about creating a powerful “Gorilla Jam” device farm to run more tests in less time on a variety of devices.

False Expectations from Automated Tests

Automated tests improve time to market and overall quality revenue. In the case of our simple 50-levels maze game called “Gorilla Jam” automated test scripts improve test coverage and make the developers happier. However, nothing happens in a flash and effortlessly. Please don’t expect that:

• Manual QA can easily write test scripts
• All the game features will be automated
• Test scripts don’t require maintenance
• Automation tools can be purchased off the shelf
• No QA is needed anymore
• No ramp-up time is required
• Profits will be instant

Options for Games’ Automated Testing

Naysayers claim creating automated tests for games doesn’t make much sense. Players never do what the sane person is expected to do. There are always tons of new bugs that can be discovered only after launching since thousands of players are doing stuff that the developers and the testers never thought to try. Well, it’s a bit short-sighted. For some point, players are the most unpredictable creatures in the world, however, it’s the task of the developers to deliver the top-quality game to the audience.

Let’s take a look from the player’s perspective. Once I got “an exciting groundbreaking open-world horror RPG”. The game has had an awesome graphic and mind-blowing gameplay (according to the trailer). After the intro, my character should have started his first mission by opening the door. Not rocket science, not an insane clicking on various buttons simultaneously, no sudden switching between keyboard and pad. Just open the door. I was trying to do it in every possible way. I wanted to play this game. Unfortunately, I couldn’t play this RPG, since there was no trigger for me to open this bewitched door. Needless to say, I was extremely disappointed. In a couple of weeks, the company launched a patch with a fix, but for me, it was done. I reinstalled this game. I believe this issue could be easily covered with a quick and simple smoke test, automatically or manually.

Terrible frame rates, crashing on loading or startup, players stuck in different places of the game, the inability of restarting, stuttering, NPCs missing their faces, servers that can’t handle the overdose of players, all of this and much more can be effortlessly discovered in any bug-laden launch. However, it’s not the players who should discover these issues.

In “Gorilla Jam”, we can significantly reduce the amount of unpleasant experience by implementing one of the following:

• Basic AIs (e.g. our tourist selects randomly the direction in the maze)
• Quick smoke tests during the automated build process (this may include loading the game, login, starting the tutorial, selecting the map, moving forward, running simple UI tests)
• “Fake player” that would randomly choose the location, run, jump, avoid banana eating
• Functional test server that will run the game for a fixed number of frames to check whether each level is loading correctly and there are no crashes
• Stress tests with selected stress test software
• Shortcuts or “cheat codes” built-in for devs and testers to buy something or get to a specific aspect of the game that would otherwise require extensive amounts of time (e.g. we can jump between levels, win them all, purchase the most expensive booster)
• Device farm, created by Han, allows running regression tests in parallel on 5 various devices.

Automated tests are perfect for regression testing. They will allow Luke not to get crazy during the pre-release activities. Thus, we can expect 50% of blocker and critical level bugs are discovered by our the Best in the World Automation framework, which obviously can not fully replace our manual QA.

The Dark Team preferred to apply ad-hoc testing at the end of the sprint. The goal was to destroy the game. This allowed them to find glitches, vulnerabilities and collapses a typical player would never discover. The Dark Ones started to utilize automation testing after the test automation framework was created.

“Chip ’n’ Dale” for Your Project

Let’s imagine, Luke decided to leave the company before the launch. Theoretically, Han can do the job of his colleague, but it won’t be the best solution, since we risk Han leaving after Luke, unable to cope with the new status of the only QA in the project two days before launch. There is no chance to hire another QA in such a short term. How to test your game while you are critically short on time and people? There are several ways.

• Unit testing as the productive part of your QA program

Typically, developers don’t like to deal with testing at all, but if we have 60–80% of coverage with the unit and integration testing in “Gorilla Jam”, we can reduce release issues and downtime on production in a half.

• Better code reviews

We can prevent a lot of problems at the code review stage, thus building an effective and positive code review culture will help our team to avoid such issues as memory leaks or server crashes while reaching 1000 players. Every low-quality code review will cost a serious amount of time. The team can create a checklist for that.

• Well-balanced traceability matrix.

Make sure all the core features and functionality are covered with test cases. Prioritize your test cases in the test plan and focus on the “must-have” for launch. If our tourist is moving in the directions the player selected, flying over gorilla with vines, and reaching the banana in the maze doesn’t crash the game, it’s a sign of success.

• Dev Test session

Managing dev-test sessions before release can reveal a lot of unexpected behaviour in the game. You can ask your developers to play the pre-release build for 1 hour. Don’t be surprised to hear: “Who made this crap?”

• Game testers from other projects

“Gorilla Jam” is the third, the youngest project in your company. You can ask other product owners from “Gorilla Rush” and “Pinky Cat” in your company for help. Would they be so kind as to provide at least one QA for pre-release testing?

• QA outsource

“Vetted & Handpicked QAs For Your Needs!” All the slogans and web resources for such services are looking appealing, begging you just to name your requirements, and forget about all QA troubles. However, it is always better to find the proper outsource QA team according to the recommendations from someone who has been using their services and get an advantage.

The Dark Team felt good about their day-to-day work. They achieved incredible synergy, acted and thought like a true team. In the regular retro meetings, they were creating harsh testing traps for discovering a greater number of issues.

Five Deadly Sins of a Game Tester

We are all humans after all. Product owners, game developers, game testers. We are all aware that a bug-free game is not possible without a human-eye approach. However, it doesn’t mean that the game tester should simply play the game and have a gala time without extracting a part of the code that is creating trouble for it. The deadly sins of a game tester, that simply prevent Luke, Han, or Chewie from succeeding in the game testing field are the following.

• Excessive trust in the game developer

As a junior game tester, Chewie could often face the “Trust me, everything is working as expected…” friendly approach. A kind-hearted and caring developer would like to save time for young Chewie on testing. It occurred, everything is working, but with one remark: “on my local build”. If Chewie trusts the developer’s experience, he can close the task without checking it.

• No ad-hoc testing

Typically, game testers are burdened with an extensive number of features to test within a limited time. Thus, most of the pre-release testing activities are derived from the formalized approach, avoiding ad-hoc testing that requires in-depth knowledge about the underlying architecture. However, one round of ad-hoc testing can add significant value to the game quality.

• Unreadable bug reports

“The game doesn’t work as expected” vs. “Crash on the loading screen after quick tapping on the progress bar”. This deadly sin can complicate the life of all the team. The best bug summary report is the most precise one. It’s good practice to describe a defect in just 10 words. without overwording.

• No bug reporting

Ignorance in documenting the defects that were fixed on the fly is one of the most common sins of the game tester. “ — Why waste time on things that no longer exist?” — would ask Chewie. But it is worth remembering that bugs are always finding their way back to the game.

• Lost focus

Testing games is challenging, it requires the game tester to stay aware of both the peculiar functionality and the entire context of the game for a long time. Testing games could be exhaustive. Generally, 1 hour of non-stop game testing weakens the focus, and Luke starts to play the game instead of testing it. Short breaks during game testing are a “must-have”.

Wrapping Up

What I have learned from testing games is that the work is easier if you love what you do. The first question I usually asked in job interviews was “Do you like games?”. Sometimes the answer was negative. Testing games will temper your character, you become patient and stress-tolerant, your brain is working as never before, your attention is sharp as a laser. You need to feel the game and reach a verdict, whether it is good to play or not and quite often you don’t have much time for it. Testing games is an art. However, when we are talking about efficiency, there are the following points that need to be marked:

• The recipe for effective game testing is = a well-organized approach + test automation framework + agile process
• A well-organized approach is = well-described GDD + carefully crafted test strategy + testable and clear test cases for every feature in GDD + test execution according to the test plan + bug reporting + bug fixing
• Test automation framework is = test cases for automated testing + testing scripts
• Creating automated tests can reduce pre-release testing time by half, however, it doesn’t mean we don’t need a manual game tester on the project.
• A well-established agile process will help the team to be the team and to be in time for the launch.
• High-quality starts from the effective and positive code review culture
• Long-time keeping focus during testing is not optional. Without this skill, it’s impossible to test games.

Effective game testing is not only about checking the code but the feeling about the game. Typically, if the players do not like something in the game, it becomes a bug. Thus, if you know your game and have an extensive intuition about how it should work to bring a positive experience to your players, then your testing is truly fruitful.