May 302011
 

Why do so many reviewers use only the top half of their rating scale? Has the Metacritic age led to score inflation? And why do some reviewers refuse to give out perfect scores? Answers to all of these questions, plus why Metacritic no longer lists scores from X-Play.

“Score Inflation”


It’s no secret that most review magazines and websites today only use the top half of their rating scale when issuing a review score. Are reviewers being paid off by videogame publishers? Do magazines and websites give 7/10s to mediocre games out of a fear of losing advertising dollars? Other than few isolated instances, the reality of the situation isn’t nearly so scandalous.

For those who are unaware of Metacritic, it’s a review aggregator that collects review scores from various sources, and displays an overall average score for the game, movie, or album. However, in order to make things easier to average, Metacritic converts every review score into a scale out of 100. So 9/10 would become 90/100, 4/5 would become 80/100, etc.

On the surface this seems simple enough, but things become trickier when you figure in letter grades. Traditionally, 90-100 = A, 80-89 = B, etc. The problem is, one person might give a mediocre game a 50/100, while another will give it a C, which would be a 75/100. Metacritic tries to compensate for this by changing the values that letter grades are given so that a C = 50/100, but in doing so they’re overlooking a larger problem: any time a scoring system is based on a scale of 1-100, there is a natural tendency for many people to view the score as an equivalent of a letter grade. Hence why so many mediocre games are given a 75/100.

What makes videogames different than movies or music is that the videogame industry actually pays attention to Metacritic scores. Indeed, there are even reports of publishers denying developers a bonus if the game receives too low of a Metacritic score. The website has become so embedded in the culture of the industry that it has resulted in reviewers adjusting their own scores to reflect how they want it to show up out of 100.

A “Perfect” Score


I’ve got a funny story for you. In college, I had this teacher who was like a female version of John Malkovich’s character from Art School Confidential. One day she had us participate in an exercise where we were to mix white and black paint to create a sequence of ten rectangles illustrating a perfect scale of shades of gray between white and black. The middle rectangle needed to be a perfect mix of the two, a “true” gray, and each of the other rectangles needed to be as close to a 10% difference in shade as humans are capable of.

We’d each show her what we had, and she’d say “ah, but see, this gray is closer to the one before it than to the one after it,” and things of that sort. We worked on this for two hours, constantly revising to try and achieve perfection, until she revealed to us a little factoid: never in all her years of teaching, has she ever seen a gradient scale she thought was perfect. In fact, even the ones printed in textbooks she never entirely “agreed with.”

Thing is, she wasn’t saying this as some sort of lesson. In fact, she still expected us to spend the next few hours trying, and was still going to grade us on the results! She was merely telling us this anecdotally, as if we would find it fascinating, when in reality we were all looking at each other like “we are so screwed…”

There’s a belief among some gamers and reviewers that there’s no such thing as a perfect game, and thus a perfect score should be given only on rare occasion, if at all. But if a teacher revealed to you that they’d only given 100/100 on six papers over the course of a decade, because there’s no such thing as a “perfect” paper, you’d probably think they were senile or something. You never see a music critic refusing to give any album five stars because there’s no such thing as the perfect album, and Siskel & Ebert never refused to give a movie two thumbs up just because there’s no such thing as a perfect movie. So why should games be any different?

Metacritic Vs. X-Play


Imagine you’ve just been given 50/100—a failing grade—on a paper you’ve written. When you go to the teacher to ask for more clarification on what you did wrong, they tell you that the paper wasn’t actually terrible; it was merely average, and the teacher doesn’t believe in using only the upper half of the grading scale. If that has a negative impact on your grades, well, there’s always going to be some casualties when trying to enact change.

That teacher wouldn’t be X-Play. G4′s X-Play rates games on a five-star scale, which would look kind of silly to only use the upper half of. When a developer complained to Adam Sessler about how a grade of three stars or less from X-Play significantly drags down a game’s average (due in part to scores from reviewers of higher stature being given more weight), Sessler actually contacted Metacritic to see if they could work something out.

Unfortunately, Metacritic has no interest in adapting game scores (rather than converting literally) just because their own 100 point scale can easily be interpreted like an academic grading scale. And X-Play doesn’t want to switch from their long established five star scale just because of the way some website recently started affecting videogame culture. Ultimately, they came to an agreement to simply have X-Play scores no longer listed on the site.

In Conclusion


It might seem like I’m defending review scores that only use the upper half of the scale. In a perfect world, everyone would go back to using a 1-10 or five star scale, rather than a 5-10 scale. Unfortunately, I don’t see that ever happening without Metacritic either falling out of popularity, or just changing their 100 point scale to a 10 point scale (to distance itself from the similarity to a academic grading system).

In the meantime, I don’t see any point in punishing developers out of pure stubbornness, the way a few major review sites continue to do. Taking that approach isn’t going to change anything. On the other hand, perhaps if enough major sites were to ask to be delisted from Metacritic the way X-Play has, they’d start to get the message…


Did you enjoy this article? Would you like to see more like this in a similar vein? Please let us know by commenting below.

Critical Kate

Critical Kate was born on the same day as the Famicom. Primarily a console gamer, Kate has been playing videogames regularly since the NES days (excluding a hiatus during most of the "128-bit" era in college). She prefers her games arcade-y, enjoys a good story, and doesn't understand the appeal of recreational frustration. You don't want to encounter her in the Facility with Remote Mines.

  43 Responses to “Understanding Review Scores In The Metacritic Age”

  1. With at least 5 separate, public cases of developers/publishers calling editors and ‘expressing their belief that game X deserves a higher rating’, after which the editor fires the reviewer and has some unknown ‘reviewer’ give it a +2 (/10) bump, I have to think that you must be new to all this. Of course the larger publishers and game companies are putting pressure on reviewers, since they actually do tend to connect with quite a few people and influence their decisions. They cannot directly blackmail, but the call itself is the threat. No serious reviewing group ever listens to a company like this. The best they can do is to promise that they will fix the product and then ask for a re-review. That is not how the game industry works though, so the calls are threats to pull advertising. I would love to hear what the wrongful dismissal settlements go for, but of course they are all confidential.

    If the review is coherent, lists good and bad aspects, and compares to other games in the genre, then many readers will take it seriously. XBOX and PS3 magazines are widely derided for giving great scores to crappy games to drive up sales (which they get a part of). Some like PC Gamer
    can still be useful, since a static -2 adjustment to their score (on a big-ticket game) will bring them in-line with reality. PC Gamer has been through the above pressure at least twice which probably resulted in their adjustment for major release from major companies. It is sad, since they used to be worth reading…

    With Metacritic, they have wandered towards a more statistically relevant system from their early days. I remember discovering about 4 years ago that a 14 year old in Australia had posted his self-proclaimed, unprofessional review. The old Metacritic decided to use his 10/10 in their aggregate (probably since he was the only one on the planet that thought the particular game was perfect). Even though they no longer grab random forum posts as ‘professional reviews’, if you look at their aggregate scores and compare them to the average player’s scores (from various sites), you will find that they are still quite inaccurate. The best way to judge is to grab 10 facepalm games like Homefront, and see what metacritic gave them. If there are not less than three that are scored 7 or higher, I would be surprised. Metacritic is still working on becoming relevant, but is not quite there yet. Any reviewer that does not give out less than 5/10 is actually only scoring on a X-Play-scale anyway, and should be considered as such (round down so 7/10 = 3/5). The greatest joke about that is, of course, that they are suggesting that all their other sources that they do use are scoring out of 100 which they are not. If they cannot multiple X-Play’s score by 20 then they should not be allowed to multiply a 10 point scale by 10 either.

    As to the industry relying on Metacritic, the TV industry bases all their decisions on the Neilson rating system. The Neilson system has been proven to be inaccurate by more than +/- 25% (and in some cases much more – especially with the advent of online access). Lazy people want simple numbers with no variables. Bean counters are generally lazy, more concerned with how much they can grow their bonuses by saving 5 cents on paperclips, than by doing a proper analysis for themselves (data mining the forums). It is not a surprise that the gaming industry is becoming more business than entertainment. Just ask where the Expansion packs went, and why a mini-mod can be sold if it is called a DLC. Heck, I am surprised someone has not tried selling patches yet.

    Ignore Metacritic until they develop a properly balanced statistical system weighted for the correlative history of the reviewer (not the review group, but the individual to the average player’s scoring of the game, and then they will be worth something. Newbie reviewers are used often on ‘blockbusters’ where they give out 9′s instead of the player’s average of 6, so they have no correlative power with reality, and that is what we all want. A reviewer that will give us an honest review and let us know if the next EAware sequel is going be even worse than the last, or if they actually used the ‘good’ team this time.

    The best are the websites with no game company advertising and X-Play. While X-Play generally calls it like it is, and definitely play the whole game (while some internet reviewers apparently only play the first few hours – not directed at this site but many, many others). If X-Play gives it a 3/5 or lower I have never had reason to disagree (though I do find they give a 4 to some I would only give a 3). Find the reviews you trust and stick with them.

    Cheers!

  2. [...] is often the quickest way toward gaining understanding. I addressed the review scale question in a previous article, but now let’s take a look at the other most frequently asked [...]

  3. 477787 680508I conceive this internet internet site contains some rattling amazing information for every person : D. 189580

  4. 358598 477131An fascinating discussion is worth comment. I believe which you ought to write regarding this subject, it may well not be a taboo topic but typically persons are too couple of to chat on such topics. To yet another location. Cheers 971845

  5. 846175 530473Das beste Webdesign Berlin erhalten Sie bei uns, genauso wie professionelles Webdesign. Denn wir sind die Webdesign Agentur mit pfiff. 583521

  6. 889183 714505Hi there, i just thought i would publish and now let you know your websites style is truly smudged within the K-Melon browser. Anyhow maintain in the extremely great work. 337651

  7. 320338 224706I enjoy your writing style actually enjoying this internet web site . 678369

  8. 773314 812548for however one more wonderful informative post, Im a loyal reader to this weblog and I cant stress enough how considerably valuable data Ive learned from reading your content material. I genuinely appreciate all of the hard function you put into this wonderful weblog. 495974

  9. 324969 36869Dead written articles , Truly enjoyed reading . 39899

  10. 665478 789707I truly appreciate this post. Ive been looking all over for this! Thank goodness I found it on Bing. Youve made my day! Thank you again.. 389442

  11. “Understanding Review Scores In The Metacritic Age » A
    Critical Hit! – Video Game Reviews” ended up being a
    marvelous article, can’t wait to look at alot more of
    ur posts. Time to waste numerous time on the
    internet lolz. Thanks ,Brandi

  12. 390663 248181If youre needing to produce alteration in an individuals llife, during i would say the Are usually Bodyweight peeling off pounds training course are a wide path within the direction of gaining any search. la weight loss 622474

  13. 604897 625612There is noticeably a whole lot to know about this. I believe you created some good points in capabilities also. 568444

  14. 967689 794698Spot on with this write-up, I truly believe this web site needs a lot much more consideration. Ill probably be once more to read a lot much more, thanks for that info. 426715

  15. You actually generated several outstanding stuff with ur blog,
    “Understanding Review Scores In The Metacritic Age
    » A Critical Hit! – Video Game Reviews”.
    I’ll possibly be coming back again to ur web-site in the near future. Thx -Lanny

  16. 569156 179232Id require to consult you here. Which isnt some thing Which i do! I enjoy reading a post that can make individuals feel. Also, appreciate your allowing me to comment! 796332

  17. […] buenos no bajan del 9, lo que hay debajo no merece la pena”). Se da así un fenómeno de inflación de notas que puede crear distorsiones cuando alguien, en lugar de utilizar una escala del uno al diez, […]

  18. 704893 318702As I web-site possessor I believe the content matter here is rattling wonderful , appreciate it for your efforts. You should keep it up forever! Best of luck. 818746

  19. 735273 867278We stumbled more than here coming from a different internet page and thought I may possibly check items out. I like what I see so now im following you. Appear forward to exploring your internet page however once again. 24025

  20. 812122 683800noutati interesante si utile postate pe blogul dumneavoastra. dar ca si o paranteza , ce parere aveti de inchiriere vile vacanta ?. 986984

  21. 110460 121552Nice site, nice and easy on the eyes and great content too. Do you need many drafts to make a post? 45439

  22. 292385 161545There will probably be several totally different portions about the LA Weight reduction eating plan and one is truly essential. Begin stage is your really truly of these extra load. weight loss 198839

  23. 824742 454488Thanks for the write up! Also, just a heads up, your RSS feeds arent working. Could you take a appear at that? 792502

  24. 835446 792673hey great website i will definaely come back and see once more. 351635

  25. 570484 960231It is difficult to get knowledgeable folks on this subject, but the truth is be understood as what happens you are preaching about! Thanks 711308

  26. 489906 322549I visited a great deal of internet site but I conceive this 1 holds something extra in it in it 718475

  27. 221999 151149Glad to be one of numerous visitants on this incredible internet website : D. 640576

  28. 44852 218196You could certainly see your expertise within the work you write. 250589

  29. 90725 530242Ive writers block that comes and goes and I need to find a way to get rid of my writers block. It can occasionally be so bad I can barley make sentences. Any tips? 93438

  30. 998950 596097Your talent is truly appreciated!! Thank you. You saved me lots of frustration. I switched from Joomla to Drupal to the WordPress platform and Ive fully embraced WordPress. Its so a lot easier and easier to tweak. Anyway, thanks again. Awesome domain! 684436

  31. 391864 8006It can be difficult to write about this subject. I believe you did an outstanding job though! Thanks for this! 813412

  32. 222857 683725You produced some decent points there. I looked online for your problem and located most people will go together with with the website. 98991

  33. 766966 636566You may locate effective specific development of any L . a . Weight loss program and each and every you are incredibly crucial. To begin with level is an natural misplacing during the too significantly weight. shed belly fat 196870

  34. 329507 214703I enjoy the useful info you offer inside your articles. 340046

  35. 876533 777040This is the suitable blog for anybody who needs to seek out out about this topic. You notice so a lot its virtually laborious to argue with you (not that I truly would wantHaHa). You undoubtedly put a brand new spin on a subject thats been written about for years. Fantastic stuff, just fantastic! 811626

  36. 386079 675476Hey! Good stuff, please keep us posted when you post something like that! 492955

  37. 309711 17185Hey this really is kinda of off subject but I was wanting to know if blogs use WYSIWYG editors or in case you need to manually code with HTML. Im starting a weblog soon but have no coding experience so I wanted to get advice from someone with experience. Any support would be greatly appreciated! 621573

  38. 379713 552755Yay google is my king helped me to find this great web site ! . 46374

  39. 954848 625593Oh my goodness! an incredible write-up dude. Thank you Nonetheless Im experiencing challenge with ur rss . Don know why Unable to subscribe to it. Is there anyone acquiring comparable rss drawback? Anybody who knows kindly respond. Thnkx 502888

  40. 11709 456062youve gotten an critical weblog appropriate here! would you wish to make some invite posts on my weblog? 307120

  41. 722404 507127Your home is valueble for me. Thanks!? This internet page is actually a walk-via for all with the details you required about this and didn know who to ask. Glimpse proper here, and you l undoubtedly uncover it. 85414

  42. 362432 687015As being a Newbie, Were permanently exploring online for articles which can be of help to me. Many thanks 100757

  43. 576163 478275This really is the proper weblog for anybody who hopes to learn about this subject. You know a complete lot its almost tough to argue along (not that I truly would wantHaHa). You undoubtedly put a complete new spin for a subject thats been written about for years. Excellent stuff, just wonderful! 403656