Larry Levy:  Rating the Designers

Gamers sure love to keep track of gaming minutiae.  We keep track of how many times we play games, how long they take, and whom we play them with.  We compile Nickel and Dime lists, Desert Island lists, and Geek lists.  All of it trackable, all of it stackable, every detail of our gaming lives carefully stored in its proper place.  Just like most hobbies, one of the pleasures of gaming is keeping track of what you have, what you’ve done, and what you’d like to do.

But the data we most enjoy tracking is ratings.  There is a large database on the Geek consisting of nothing more than the ratings gamers have given to a huge number of games, complete with comments.  And given such a storehouse of data, who can resist trying to draw some conclusions from it?  Thus, there have been several attempts to determine which is the “bestâ€? or “most popularâ€? game designer, using the Geek ratings as reference.  All of these rankings have been controversial (including that of our own Tom Vasel, who caught some serious crap when his system rated Knizia as the 12th best designer of all time).

I think the reason for the contention is that there are at least two broadly different schools of thought of how to rate designers.  I refer to these as the “Knizia vs. Seyfarthâ€? methods.  It’s kind of a “quantity vs. qualityâ€? distinction, although that’s an oversimplification.

The reasoning behind the first type of measurement is that the objective of a gamer is to play good games and the more good games a designer creates, the better he or she is.  Flops aren’t really a problem, since you play them once, decide they suck, and then avoid them.  The great games, though, you keep playing.

I call this way of rating designers the “Knizia methodâ€? because it favors those who have a prodigious output (it also helps if, like Reiner, you design a lot of great games).  A number of systems fall into this camp, but most of them are additive.  Assign each game a value—the higher rated it is, the higher the value, and no values are negative.  Add up the values for all of a designer’s games.  The designers with the highest total values are considered the best.

The “Seyfarth methodâ€? works for those gamers who are interested in the consistency of a designer’s output.  Basically, they want to know if that designer creates a new game, how good is it likely to be.  This method favors designers who have created a small number of games, most of which are excellent (Andreas Seyfarth certainly qualifies, as does Francis Tresham).  Again, you can get wiggy with the methodology you use, but the simplest thing is to find the average rating of all the games the designer has produced.  Those with the highest average ratings are on top.

I think it’s impossible to declare one of these methods superior to the others, simply because they are trying to measure different things.  You keep telling me these designers taste great, when I’m trying to find the ones who are less filling.  Not, of course, that that stops anyone from arguing about it.

Which method do I subscribe to?  I believe in a pragmatic approach:  choose the one whose results seem to best fit reality.  This doesn’t help too much with Geek ratings, since I don’t know the raters’ overall preferences (and if I did know it, I wouldn’t have to come up with the methodology).  But it is useful if I apply the methods to my own ratings, since I have a good notion of which designers I like even without any analysis.

My first impulse is to go with the Knizia method.  I play a lot of different games, I usually have a chance to try before I buy, and I love trying out new designs (so even a dog isn’t a total loss, as it represents a new gaming experience—it’s just not one I want to repeat).  All this favors designers who can feed me many good titles.  I also do a lot of reading about the new releases, and as a result usually filter out the games I think I’m less likely to appreciate.  This is by no means a foolproof approach (so I’ll usually give any game a try once), but my hunches tend to be pretty accurate.  So it’s less important to me to be able to predict about a designer’s reliability, since I have so much other data to go on.

Still, the proof is in the pudding, so let’s see how the two methods work for me.  For Method 1, I assign each game I’ve played points based upon its rating.  The assignment is as follows:  6 or less – 0 points; 6.5 – 1 pt.; 7 – 2 pts.; 7.5 – 3 pts. 8 – 4 pts.; 9 – 6 pts.; 10 – 8 pts.  (6.5 and 7.5 are the only half point ratings I make, since so many games fall in the 6 - 7 range.) The point value can be determined by using the following formula:  Points = 2 x (Rating – 6), (with a minimum of 0 points).  I’ve experimented with other schemes, including some that give greater weight to games at the top of the scale, but this seems to work best for me.  I then calculate a point value for every game of a designer I’ve played and sum up the points.  The sum is my rating for that designer.

Here are my top 15 designers using this method:

1. Reiner Knizia – 120
2. Wolfgang Kramer – 63
3. Alan Moon – 49
4. Martin Wallace – 28
5. Michael Kiesling – 27
6. Sid Sackson – 26
6. Michael Schacht – 26
6. Aaron Weissblum – 26
9. Friedemann Friese – 24
10. Stefan Dorra – 23
11. Uwe Rosenberg – 22
12. Rüdiger Dorn – 21
13. Klaus Teuber – 19
14. Günter Cornett – 17
15. Alex Randolph – 14

How well does this match my actual preferences?  Very well indeed.  I’ve long considered my three favorite designers to be Knizia, Kramer, and Moon, in that order.  The point gap between Knizia and Kramer is larger than my preference of Reiner over Wolfgang, but that’s mostly due to the very large number of Knizia designs I’ve tried (almost 70 games!—more than twice as many as Kramer’s).  Most of the other designers are reasonably close to where I would rate them.  Schacht is surprisingly high, as I don’t rate any of his games higher than a 7.5, but again, I’ve played a lot of his games.  Dorn is low, but he’s only designed a handful of games.  And Teuber is just about where I would rate him; outside of his sublime Löwenherz, I tend to admire his designs more than I like them.  (I’m hoping his rating will soon take a boost, as I intend to try out Elasund later today.)

How about Method #2?  Here I go as simple as possible, just finding the average rating I assign to the games I’ve tried from each designer.  I usually don’t bother with a minimum number of games in order to qualify (since otherwise, that eliminates the Seyfarths and the Treshams), but just to put the two lists on something of an even footing, I’ll limit it to the designers for whom I’ve played at least five games (much more and there’ll be very few designers who qualify).  Here are my Top 15 designers by average rating:

1. Rüdiger Dorn – 7.75
2. Günter Cornett – 7.70
3. Michael Kiesling – 6.96
4. Robert Abbott – 6.90
4. Marcel-André Casasola-Merkle – 6.90
6. Uwe Rosenberg – 6.89
7. Friedemann Friese – 6.75
8. Wolfgang Kramer – 6.73
9. Sid Sackson – 6.71
10. Eberle/Kittredge/Olotka* – 6.71
11. Reiner Knizia – 6.65
12. Martin Wallace – 6.64
13. Klaus Teuber – 6.63
14. Stefan Dorra – 6.57
15. Mike Fitzgerald – 6.57

*In case you don’t recognize the names, these are the Eon designers

That’s very different, isn’t it?  Needless to say, this list doesn’t correspond to my preferences nearly as well as the first one.  Yes, it does give props to two of my favorite low volume designers, Dorn and Cornett, and also highlights some others I’m fond of, like the Eon designers and Fitzgerald.  But Knizia eleventh?  Abbott and Casasola-Merkle in the Top 5?  That’s just not the way I rank the designers.

And the other designers from List 1’s Top 15 fared even worse:

16. Alex Randolph – 6.45
17. Alan Moon – 6.40
20. Michael Schacht – 6.39
23. Aaron Weissblum – 6.21

Moon’s low average rating may be surprising for a designer I hold in such high regard, but I really do try out an awful lot of his games (as many as for Kramer, even though Wolfgang has produced many more titles), including quite a few lighter designs that haven’t always worked out.  So this ranking doesn’t cause me to re-evaluate his status in my mind at all.  Overall, though, it seems pretty clear:  for me, the Knizia method works best and I can leave the Seyfarth method to others.

But still, it isn’t perfect.  The gap between first and second place is not only too large, but it means that it will take a very long time for any designer to overtake Knizia in my heart, and I honestly don’t think I like him that much.  Besides, my love affair with Reiner Dearest seems to be waning a bit.  Although I liked a couple of his ’05 designs, the only really indispensable game from my point of view that he’s released since 2000 is Amun-Re.  If that trend continues, I’ll have to question whether Knizia truly is my favorite designer.  But the large number of good games he’s already released will make it very difficult for him to be unseated by using Method 1.

There are other flaws in the results as well.  Without question, my favorite designer of the moment is Rüdiger Dorn, who tops the Seyfarth Method list.  Now I’m not saying I’m ready to anoint him as my all-time favorite yet, but he still probably deserves better than twelfth place, which is where Method #1 puts him (mostly because he hasn’t designed many games yet).  Then there are the niche favorites that I mentioned earlier—Mike Fitzgerald and the Eon designers.  Both are ranked 21st in the first method, which is probably too low.  Note that all of these designers do much better in Method #2.

Besides, there’s something peculiar about having a designer release a dog (like Knizia’s Spy in 2004—Reiner, what were you thinking of?) and not having it affect your rating of him one iota.  Of course, a game like that will kill the Method 2 rating (maybe more than it should), but it leaves Method 1 unscathed.  Neither method seems to handle this satisfactorily.

Well, is there a way of combining the two methods, possibly producing something superior to either?  I don’t know about the second condition, but it’s easy enough to find a middle ground between the two extremes.  One way is to compute what in gaming circles these days is called a Bayesian average.  (This term, which is something of a misnomer, seems to have been coined on the Geek, even though Aaron Fuegi was probably the first to use the method for his Internet 100.  Bayesian probability is an actual mathematical discipline, but it means something very different in that field.) As most of you know, an average rating is converted to a Bayesian average by adding a number of dummy ratings to the population, with the dummy ratings being near the midpoint of the rating scale.  The idea is to give an advantage to games that have received more ratings, since they will be hurt less by the inclusion of these poorer ratings.  The justification for this is that you have more confidence in a game with more ratings (since it is less likely to be subject to wild swings as more ratings come in).

I’m not necessarily a huge fan of Bayesian averages (I usually prefer to use a straight average, with a minimum number of ratings), but it does provide a simple way of using averages while compensating those designers with more ratings (and who therefore usually do less well with a straight average).  One nice thing about a Bayesian system is that you can tweak it until it gives you a result that best mirrors reality.  The method I chose is to add 12 games to each designer’s list, each with a rating of 5.  When I do that (and take away the minimum games requirement, since it’s no longer needed), I come up with a list that looks like this:

1. Reiner Knizia – 6.41
2. Wolfgang Kramer – 6.25
3. Alan Moon – 6.01
4. Sid Sackson – 6.00
4. Michael Kiesling – 5.98
6. Rüdiger Dorn – 5.92
7. Martin Wallace – 5.88
8. Friedemann Friese – 5.88
9. Stefan Dorra – 5.87
10. Michael Schacht – 5.83
11. Klaus Teuber – 5.82
12. Uwe Rosenberg – 5.81
13. Günter Cornett – 5.79
14. Aaron Weissblum – 5.74
15. Alex Randolph – 5.69
16. Eberle/Kittredge/Olotka – 5.63
17. Mike Fitzgerald – 5.58
18. Robert Abbott – 5.56
19. Marcel-André Casasola-Merkle – 5.56
20. Andreas Seyfarth – 5.53

This represents a pretty interesting mix between the two methods and one that matches my preferences quite well.  Dorn is rated sixth, Schacht is down in more reasonable territory, Eon and Mike Fitz are represented, and there’s even room for Seyfarth in the Top 20 (despite the fact I’ve only played three of his designs).  In fact, the method ranks Tresham 23rd, even though I’ve only played two of his games (obviously, both are rated highly)!  Sackson seems to be ranked a bit high for me, but it still looks like the Bayesian method does a good job of mixing the high-volume designers with the less productive, but consistently good ones.

Although I’m sure you’re all fascinated to know which designers I like, the real reason I’m posting this is to report on a ranking system that bridges the gap between the point counters and the ratings averagers.  Obviously, you can come up with lots of variations (such as giving greater weight to highly rated games and such), but even this simple method gives me pretty good results.  I don’t know if some intrepid soul (at least, someone more intrepid than me) wants to try something like this with the Geek ratings, but I’d sure be interested to see how it worked out if they did.

© 2006 Larry Levy

---

Comments:

You must register with BGN in order to comment. Registration is free, but if you appreciate the news, previews, reviews and other material posted on Boardgame News, please consider becoming a member to keep the info flowing to your screen!

Too much maths for me in there, but the real problem is that I’m in none of these listings ;-) Posted by bruno faidutti on Jan 28, 2006 at 03:09 AM | #

That’s quite an exhaustive analysis.  Interesting as well.  Personally, I’ve never really thought about who my favorite designers are.  I’m not really sure why!  There are certainly some designers whose games I will buy sight unseen - Richard Breese springs to mind - and that is certainly a bit vote of confidence.  Someday I’ll have to crunch my numbers and see what the results are. Posted by Rick Thornquist on Jan 28, 2006 at 04:37 PM | #

Sorry, Bruno, but the overlap between the games you love (including the ones you design) and the ones I love isn’t great.  You are Mr. Chaos, and that’s something I’ve made a career of avoiding.  Fortunately for you and the rest of the gaming world, my opinion is not a majority one.  And I have to say, I’m really looking forward to seeing what you, Derek, Corey, and Pierre did with Warrior Knights. Posted by Larry Levy on Jan 29, 2006 at 12:59 AM | #

< Back Home