Frank Branham: Rating Systems get a 2 out of 5

There is a thread on BGG about declaring January 10 as rating day to update and clean up your ratings. The idea is to make sure you've really rated all of the games you know, and alter your ratings for games about which your opinion has been changed by time.

It seems like so very much work for something so useless. I almost never look at ratings. Instead I wait until I can read a review or comments either by someone whose taste I trust, or someone who can explain their reasons for liking a game well enough to let me work out if I should like the game.

(As an aside, I've found that Mark Jackson, Alan How, Mike Siggins are my most reliable sources. Two of those I've met, and what little I've read about Alan How ALMOST makes me wonder if he's some sort of lost good twin. I'm pretty sure I'm the evil one. )

Ratings have so many odd things about the way they are collected and ranked that make them suspicious. In practice, I don't believe that people can distinguish more shades of gray than Superb, Decent, Meh, and Terrible. So the idea of a rating system with more than 5 categories seems baffling, as I might be willing to allow that we can distinguish the merely bad from the truly and awe-inspiringly appalling.

Those further categories seem to assault the idea of having a permanent rock steady opinion of a game. The BGG rating guidelines are fairly typical, and include concepts like:

"Always want to play it." This has so far failed to be my opinion on any game ever made. Even my beloved Crokinole I'm not so fond of unless there are decent players.

The two point difference between "will play sporadically if in the right mood" and "could be talked into it on occasion."

When you look at a name with a number next to it or even a review, the review and especially the number are probably even fairly useless without knowing the background of the rater. This makes the idea of average ratings seem appealing.

Averaging the ratings creates tiny differences between games. The mediocre Candamir is a single point off the superb basic Settlers. The ratings only average to interesting values in cases where everyone seems to agree on the ratings.

The best example of the pointlessness of averages is IMDB. Practically every movie sits between a 6 and 7 ranking, the same as BGG. I know far less about movies than games, and so I've tried to use IMDB to look up an obscure movie that Tivo has recorded. I look at the rating, and it means absolutely nothing to me.

Ratings aren't for people who want to find out if a game is good. Ratings averages are for people that get all excited about a tenth of a point jump in their favorite game.

Curiously, I do find the Rotten Tomatoes movie review ratings system SOMEWHAT useful. They obtain a score from online reviews. Each review is somehow assigned a rating of "Fresh" or "Rotten". No grey at all. The rating is simply the percentage of good ratings. This produces a fairly wide range of ratings, and really gives you a sense of how very awful Uwe Boll movies actually are.

Even the Rotten Tomatoes style averaging does not always work. Its ratings pool does not seem to like horror movies, and those are always given short shrift. Complicated plots or non-linear films totally throw the review averages as well. (The rather clever and deliberately-paced "Darkness" rates a 4%.)

The BGG ratings are also quite tainted. If you look at the comments for Black Vienna, you'll find comments like "I just don't care for deduction games." Dexterity games, trick-taking games, and abstract games are also peppered with odd ratings and comments. You'll even get ratings from people who've never played the game.

On top of this is the hobby that some people have of shilling, hopping on en masse to hurl 10's at friend's games. Or hyping a new game from a favored company. Or posting a bad rating because of some slight. Or...we really can't trust the ratings numbers.

Even magazines and sites that hire "proper" reviewers, and have a strict set of guidelines to produce a finalized score produce pointless data. Lots of people have made fun of videogame site IGN's review scores. Finding a game outside of the 6-9 range is very hard. They also use decimal scores, just so you know that a 7.2 is better than a mere 7.1.

Therefore, we really cannot respect ratings even from reliable sources, even the best methods of combining them are fraught with peril, and the collections of ratings are certainly tainted with weird anomalies. All of this means that ratings are remarkably useless, both for people with thorough knowledge of the subject, as well as neophytes.

Sooo...if ratings are not useful, what is a newbie who is trying to learn about the hobby supposed to do? My technique of finding people with similar tastes doesn't really work, as the newbie doesn't have much of an idea what they like. They really do not know any of the folks online.

A couple of years back I did have to do a similar thing. I really wanted a High Definition television, and venturing into that lair is like walking into the largest pit of snake-oil salesmen ever created. The commercial sites were not very useful, as their reviews lagged months and years behind the TV ship dates. The fan sites like AVSforums and Hometheaterspot had much more current information. The big issue was that there was so much noise, wailing and bemoaning, and insults that it was hard to get a solid idea of anything. However, an occasional post on the forums was from a guy who provided reasonable technical details, or who worked as a calibrator, or had been doing broadcast video.

It took a couple of hours to find these guys in the forums, and start tearing through their comments. I ended up with a nice deal on a DLP tv that does a passable job on DVD's, and plays videogames flawlessly. It was also $1500 less than the other TV's I was considering. Like games, there is really no perfect piece of kit. You have to find something that really suits you. I actually would not really recommend the TV I ended up with to many people. However, it does suit my needs quite well.

© 2007 Frank Branham

---

Comments:

You must register with BGN in order to comment. Registration is free, but if you appreciate the news, previews, reviews and other material posted on Boardgame News, please consider becoming a member to keep the info flowing to your screen!

Frank, I rate your column a “9 out of 13”.  On this scale, that firmly puts it in the “Love reading it every week, but not quite enough to print it out to read on the ‘throne’.” If it had been up this morning, when I was more alert, it may have received a “10 out of 13” meaning “Love reading it every week, and would turn to it even before my morning gander thru ESPN.com”. Oh, and other than the fact that you and Alan are omnivorous in your gaming, the thought of you two being evil twins is quite humorous.... Rumour has it that Alan may be over here in April, so we’ll have to get a picture of you two playing some appropriately obscure game. D Posted by Dale Yu on Jan 9, 2007 at 08:13 PM | #

I’m still chuckling about the Player Rating system someone posted on the Geek a few days back: http://www.boardgamegeek.com/article/1260530#1260530 I give this column a 1 out of 13 because I think people might start shilling it really soon now. Also, I didn’t read it, but I heard a rumour that someone else thought it wasn’t worth the 9 that Dale gave it. Posted by Melissa Rogerson on Jan 9, 2007 at 09:14 PM | #

It doesn’t seem odd to me that games (or whatever) tend to settle at “middling”, because many things simply ARE middling.  A 6-7 seems about right for many games. To me, those ratings are “trustworthy” as far as I understand the will of the majority of people who rate that way.  Sometimes (many times?) ratings are more about the people who create them than about the games themselves.  It is just the general miasma of BGG that many Eurogames are 6-7.something, and only a few rise above the “pack”, and that sometimes other types of games are rated much lower.  Even tainted ratings carry information. If your tastes run counter to the culture, ratings will be of less use on their own. I use ratings to get a feel for how a particular community feels about something, knowing how much or how little I agree with them, and then using more refined methods for the interesting bits of the resulting Venn diagram. My personal rating system falls somewhere around three: keep, here for now, trade pile.  There are no halves.  I have sometimes felt a bit ashamed about such a simplistic view of the world that if I ever do make external ratings, I try to beef them up to a 5-star or even 10-star system, because otherwise it will sort of look bad.... Posted by Ava Jarvis on Jan 9, 2007 at 10:10 PM | #

I don’t think the problem lies with the process of rating a game, but more with the process of averaging those ratings. Ratings are middling not because all games are equal, but rather because there is about as many people who like a game than people who don’t like it. If you were to select a reduced number of like-minded people, then the rating spread over the BGG database would be a lot more important. I think ratings can be useful since they quickly capture information about a person’s favorite and least favorite games. Some way to turn that bunch of numbers into a source of useful data is to use Mikko Saari excellent rating correlation utility: http://www.lautapelaaja.net/bgg/correlation.php This will let you identify a number of BGG users who enjoys and dislike the same games as you. Flag a couple of these users as your geekbuddies and you get a good pool of reviewers who tend to think like you. In conclusion, don’t kill the messenger: just because you think the averages are worthless doesn’t mean the data itself is worthless. Posted by Philippe Beaudoin on Jan 9, 2007 at 10:54 PM | #

I agree with most of your rant, Frank, but I have a hard time believing you can’t find any shade of gray between Decent and Superb.  Decent, Good, Very Good, and Superb is hardly what I’d call fine graduations of judgement (and whaddya know, there’s our 6, 7, 8, and 9 ratings). Posted by Larry Levy on Jan 9, 2007 at 10:59 PM | #

Actually, Philippe, I’d say just the opposite is true.  Individual ratings are close to meaningless, because I have no idea what your “7.5” means (not to mention all of your biases that I’m unaware of).  But put enough ratings together and average them and now you can draw some solid conclusions.  I can say with great certainty that the BGG users find Puerto Rico to be a far superior game to Monopoly and if I consider myself to be a typical BGG user, that’s a useful fact to me. Posted by Larry Levy on Jan 9, 2007 at 11:06 PM | #

I haven’t read this article yet, but I’ve skimmed the rules and I enjoy designer Frank Branham and publisher BGN.  8/10 Posted by Phil Schwarzmann on Jan 10, 2007 at 12:17 AM | #

When you meet someone and try to assess how similar your tastes are, you might ask him questions such as: “Do you prefer game A or game B?” If you wanted to do this more quickly you could ask him to write down a list ranking his games from his favorite to his least-favorite. Now you could go over this list looking for the games you know, and if they fall roughly in the order you would have placed them, then you know that you have somewhat similar tastes. BGG ratings can be used in exactly the same way. That is, not to represent an absolute and user-independent evaluation of the quality of a game, but rather as an efficient way for each user to rank games. The global average does not consider ratings in this way, but the correlation tool does. Posted by Philippe Beaudoin on Jan 10, 2007 at 01:37 AM | #

Philippe: Hmmm. You are totally correct. While the individual ratings are still suspect, a correlation *IS* quite valuable. I think I DID use the old BGG correlation tool to find my tastes close to Alan. Does the tool include a list of worst matches? That would be much funnier. Posted by Frank Branham on Jan 10, 2007 at 08:15 AM | #

Once in a while the tool will list somebody with a negative correlation. This is rare though, since you would have to find somebody who consistently dislikes games you enjoy and vice-versa. However, there is bound to be a large portion of users with whom your correlation is close to 0, meaning their preferences seem totally random to you. Go ahead, try the tool, it’s totally intuitive and outputs nice point clouds. :) Posted by Philippe Beaudoin on Jan 10, 2007 at 11:24 AM | #

...what little I’ve read about Alan How ALMOST makes me wonder if he’s some sort of lost good twin. I’m pretty sure I’m the evil one. Happy to check that one out with you Frank, but I have also found your own comments spookily close to my own on many occasions. I hope to meet you at the Gathering this year. I have a theory that each gamer has a close alter-ego game buddy that thinks and plays and orders games along the same lines. My own pet rant on BGG ratings is where someone puts down Rating “9 - can’t wait to see what the game looks like”. Please don’t bother.  It is puerile. Posted by Alan How on Jan 10, 2007 at 04:28 PM | #

< Back Home