Do Metacritic scores affect game sales?

This is a surprisingly contentious question with significant consequences for people working in the games industry. I first became aware of this issue when news broke out in 2012 that the developers of the game Fallout: New Vegas were denied their royalty bonus because they failed to achieve a certain metacritic score. The story was punctuated with layoffs reported at the developer studio and brought to light the fairly common publisher practice of tying contract agreeements to metacritic scores as a measure of product performance. Since then there’s been a number of articles highlighting the increasing importance of metacritic scores to the industry, from determining publisher-developer contracts to serving as hiring requirements.

It’s gotten to the point where people are attempting to game the system in order to skew metacritic scores upward. The games industry has a practice where developers and publishers hire “mock reviewers” to come to their offices to play and review an early build of a game prior to release. These “mock reviewers” serve as outside consultants that help the developer identify changes they can make to their game in order to achieve higher reviewer approval (and thus a higher metacritic score). These reviewers are then ethically barred from reviewing the game once it is released, since they have already taken money from the publisher. The story involves a high-ranking studio employee who hired a notoriously fickle mock reviewer purely to remove the reviewer from the metacritic review pool. The studio actually did not care about getting any feedback from this mock reviewer, as they were only interested in eliminating a potentially negative review to skew their game’s final metacritic score higher.

Obviously game publishers are using metacritic scores as a proxy for what they are really interested in: game sales. Since so much is riding on the metacritic score of a game, I wondered if there actually was any relationship between sales and metacritic scores. To do this, I scraped metacritic for their critic and user scores, and vgchartz for unit sales data. I limited my data set to games released after Playstation 3, Xbox 360, and Nintendo Wii generation of consoles arrived (around 2006). This provided a picture of game sales approxiately within the past decade.

If you simply plot metacritic scores vs sales, you end up with a chart looking like this:

It’s really hard to see anything other than the fact that unit sales of Wii Sports (and by extension, the Nintendo Wii since the game was bundled with the console system) eclipses everything else by a humongous margin. Changing the scale of the y-axis to a log scale gives a better picture of the data:

While there is still a lot of noise in the data, it generally suggests an upward trend as metascore increases. This is made more obvious by the median sales for each metacritic score (shown in red diamonds). Median sales generally hover around 100-300k units sold until you reach a metascore of 76, after which the median sales begin to noticeably climb upward as metascore increases. The largest jump comes at a metascore of 84, where the median sales jumps up to 1.61 million from a previous median sale of 0.54 million at a metascore of 83. This pattern is likely what games industry members refer to when associating higher metascore with sales.

However it would be misleading to focus only on the upward trajectory of the median sales. Looking at the raw data you can still see that range of sales at each metascore is huge. If we overlay a histogram, we can see the metascore for most of the games released in the past decade is in the 75-85 range:

Despite the increasing median sales, games in this metascore range can expect to sell anywhere between 10,000 to 20+ million units. At the extreme upper end of the metascore scale, even achieving a metascore as high as 90 does not guarantee success. Games such as Portal, Braid, and Company of Heroes all achieved 90+ metascore rating but did not exceed 50,000 unit sales, a value well below the median sales amount at any metascore rating. Examples of individual games breaking the trend in the opposite direction can be found as well: Kinect Adventures! received a mediocre metascore of 61 yet went on to achieve phenomenal success with 21.24 million units sold.

The results here are very similar to a recent article from Ars Technica where they looked at metacritic score vs sales on the Steam PC game digital distribution platform (measured as the number of user accounts owning a certain game). They binned metacritic scores into 10-point ranges found games with a metascore exceeding 90 sold for at least ~800,000. No game with a metascore below 60 ever exceeded 1 million units sold.

The differences in observations between the Ars Technica article and mine are probably due to examining different data sets. The sales data I obtained from vgchartz primarily focuses on the retail channel, while the Ars Technica article focuses on game sales on a PC digital distribution platform so they aren’t exactly the same market. A game like Kinect Adventures!, which was sold specifically for the Xbox 360 Kinect peripheral is not even availble for sale on Steam. Conversely, the high-rated underselling game Portal has been given away for free before as a limited time promotion. Anecdotally, I have also noticed a repeated sales at steep discount for that particular game, so it is unsurprising that it performed poorly in the retail channels.

Specific observations aside however, their general conclusion is the same I have here: there is just too much variability for metascore alone to be a good predictor of game sales.

This made me wonder if there was another metric we could include to aid in making predictions. Twitch.tv has been something of a phenomenon in recent times, especially with their sale to Amazon for $970 million. For those unaware, Twitch.tv is a streaming website where gamers can stream gameplay from their computer or compatible console and share their play session with the world. Twitch gets an incredible amount of internet traffic, and I think game spectating is fast becoming a new industry trend.

I’m thinking that twitch.tv activity could also be use as another metric to measure “user interest” in a certain game and could potentially serve as a feature for any game sale predictive model. Because twitch.tv’s api does not provide historical information, I wrote a script to track total viewers for video games releasing in November and December 2014 and built a D3 visualization to show the total number of viewers per hour:

Interestingly enough you can see how Assassin’s Creed Unity only has a mediocre metascore of 74, yet it’s already sold close to 2.66 million units and it’s peak twitch viewership is around 55,000 total viewers. In it’s first week of release, it consistently had over 10,000 viewers on twitch paying attention to this game. Contrast this to Pro Evolution Soccer 2015 which had a much better metascore of 82, yet only sold 0.53 million units and it’s twitch activity is much smaller.