During prototyping, it’s likely that you’ll test tens - if not hundreds - of concepts to see what’s a market fit. Testing quickly without wasting too much time or resources - and without missing an opportunity to help your game succeed - is a tricky balance.
Here, Danielle Reich Cohen, Director of Gaming at Supersonic, Hannah Han, Director of Partnerships at Funloop in China, and Nebih Başaran, CEO of Garawell Games and the developer behind the hit game Bridge Race, all sat down together to discuss their prototype testing tips for hitting the sweet spot of speed and variety.
How many hours should you put into a single prototype for testing?
Nebih: I’d suggest not spending more than a week on one MVP. Very few prototypes pass the marketability test - taking more time than a week just to see it fail takes away time from creating new concepts and can be demoralizing. However, I do believe it’s worthwhile to invest in developing a prototype with a brand-new mechanic.
Dani: We also recommend that developers spend a week or less to build a prototype. You need to balance the amount of time you put into building a prototype with how quickly you test it and move on.
What are the most important features of your prototype to iterate on first?
Nebih: Our first focus is on the background - for example, if it’s a sea landscape, we’ll try out different colors for the waves. Then we start iterating on how the characters look and different skins. After trying out variations of the game’s visual elements, we focus on the build itself, like the difficulty level. By analyzing churn rate, we can see if user dropoff is very high - if it is, we make the game easier, such as reducing obstacles.
Dani: Small tweaks like changing the camera angle are quick and easy elements to iterate one. Even a minor change like going from an open environment to a closed one can help, like we saw with our game Hide ‘N Seek. Altering the environment along with further game tweaks helped the game achieve a D1 retention rate of 48% and increased D0 playtime by 33%.
Hannah: For the China market, we usually start by iterating on the characters’ skins. For example, we designed a skin for Hide ‘N Seek based on a Chinese fairy tale and used it in a creative that became the top-performer.
How many creatives should you initially test?
Nebih: We’ve designed many different prototypes - hundreds, in fact - which means we’ve also tested a lot of creatives. Since hyper-casual prototypes are basic and often have no more than 10 levels, I suggest starting with 6 versions of your creatives. Once you achieve a promising result from a marketability test, then try changing how the game looks and start iterating on your creatives, like changing the colors and showcasing different characters. The goal is to discover what iterations will lower CPI even further after the prototype showed potential from the initial test.
Dani: During testing, you need to understand very quickly if your prototype is marketable - it’s a balance of efficiency and variety. To achieve this, we suggest starting with 4-8 creatives that highlight different scenarios you can easily change in Unity, like camera angle, theme, and characters. The aim is to reduce FOMO that developers often have about a missed opportunity in their creatives that could crack CPI before moving onto the next phase (if the game shows promise) or scrapping it and testing the next prototype.
Do you have any tips for designing a winning creative?
Nebih: Be aware that you’re testing creatives in a very controlled environment - many people are watching just the first few seconds of your ad, usually on Facebook. It’s essential that users understand your concept and how to play within those first seconds of your creative. With this in mind, highlight the main mechanic with elements like the camera angle and colors.
Hannah: A winning creative in China tells the story well within the first few seconds - storytelling is key for that market. Also, the game and story should be localized, which means if you have text within your creative, make sure it’s translated. However, it shouldn’t be a literal translation - it should make sense in the local language
Dani: Something exciting should happen every 3 seconds in your creative and in your gameplay - rhythm is important for engaging users and creating a dynamic experience.
What are the most important metrics for testing?
Dani: CPI is our top priority - it indicates if the game is interesting to users and can scale, particularly on social networks. When analyzing your game’s performance on social platforms, take a look at the demographic split with the information the platform provides and see how well it’s balanced. If it’s not balanced, analyze the game elements that may be skewing it one way or another. IPM is also important - we suggest aiming for an IPM over 35. A high IPM shouldn’t come only from one source, though. Supersonic runs prototype tests on multiple networks and analyzes the source apps delivering a high IPM - if we see it’s a pretty even spread of apps, it’s another way to prove the concept has legs.
Nebih: I agree that CPI is an essential metric because it shows how attractive your product is. Once your game shows a promising CPI, then start looking at in-game metrics like playtime, churn rate, and retention. If playtime is low and/or churn rate is high, it means your creative is misleading or an element of the game isn’t holding the attention of users even though the creative did. You can then start building out your game and making tweaks to address the problem and improve in-game metrics. I also agree that IPM has recently become more important because it includes all impressions, like comments, likes, and shares - it’s a different gauge of user engagement and appeal of your game. We saw with one of our games in a CTR test that it achieved high likes and shares, and the IPM was very high. So we decided to run a marketability test and found that CPI was also low, which indicated that IPM and CPI tend to be correlated.
"IPM has recently become more important because it includes all impressions, like comments, likes, and shares - it’s a different gauge of user engagement and appeal of your game."
- Nebih Başaran, CEO of Garawell Games
Hannah: We’re focusing on the later stages of publishing, like monetizing - an important feature for us to look at is the timing of interstitials and if they’re helping to maximize ARPU. Bridge Race, for example, shortened the levels and made them easier so interstitials could show up more seamlessly (e.g. between levels) and monetize more effectively.
This Q&A originally took place at Mobidictum and you can watch the full video below.
Have a prototype ready for testing? Submit it through our platform and if it meets our benchmarks, we’ll grow it into a hit together.
Let's put these tips to good use
Publish your game with Supersonic
