And just placing well (say, 5/6 in a 70-person Regional) seems likely to be worth more than placing 1st at a 12-person SC… but that doesn’t make that placement nothing, and many people are unable to make it to as many larger events. Seems ideal, then, to include SCs, even if their influence is marginal.
Honestly, including Leagues – particularly online leagues such as future Stimhack Leagues – seems not unreasonable, but I can see why people might be hesitant.
I might also be a pinball nerd - they have an unofficial world ranking system that deals with the same types of challenges (varying tournament formats, varying tournament sizes).
Full details:
Executive Summary
-Your ranking is weighted sum of the top 20 events that you’ve participated in the last 4 years.
-An event is worth some number of points that scales based on the size, the prestige of the event, and the quality of the players therein
There are weaknesses to be sure, but food for thought.
Well… When you design a ranking project like this you go for one of two design goals:
You apparently go for number one while I’m in favor of number two.
I really like the idea of capping the number of tournaments somewhere, (for netrunner, your best 10 finishes, point-wise, in the last 2 years or something would be great), mostly because it would create a cap so that the person who is at the top isn’t just the person who makes it out to the most tournaments. I think this is a good compromise between ELO (too hard to track without automation and universal tournament software), and handing out points for every tournament finish, (will tell us next to nothing about how good people are).
The concept of achieving more accurate results by processing less data is fascinating to me. Especially if what you’re trying to do is rank people that haven’t ever played together in the same event (because that’s what a global ranking system is).
I just don’t think 10 man minor tournaments give compareable data to 40 man tournaments, but to each his own.
It seems to me like the way to go is to include all data possible but construct a confidence / error bars metric. Put another way, I think you can have both #1 and #2 if you directly track and display the accuracy for each player’s score/rank.
That’s what weighing the data is for, man. By ditching the 10-man tournaments you’re saying they don’t matter at all, which is just horseshit.
I think that they should be weighted to a certain point, but that we should stop counting them after a person has a certain number of finishes. Basically, we use them until we get enough data for each person that we can abandon them. This would fall in line with what I actually believe, which is that there is no amount of 10-man tournaments in the same store that you win that can determine whether you have the skill to win a regionals or nationals.
How about top 10 events you’ve finished where each event awards points as:
Placement.
So, if I finished first at worlds, and then had a total of 3 other store championships where I had 3,4 and 7 place finish, my points would be:
256 + 12/3 +12/4 +12/7. Or roughtly 264
What if we just capped the number of type of each type of tournament you could use as part of your rating calculation? Like, cap the number of seasonal tournaments at 3 or 4, capped the number of store champs at 2, and cap the total number of tournaments at 10? This way, your store championship wins would be worth something potentially but as you play in more tournaments, and more large tournaments, those results become irrelevant, and the highest ranking players invariably would be the ones that placed highly in large tournaments rather than the ones who just continuously crush their LGS.
Edit: alternatively, if you just capped the total number of tournaments in general, you would achieve similar results, but I think capping each type of tournament, including nationals and worlds, could be even better so that we don’t end up forcing players to travel a lot to attend multiple nationals or anything if they want to achieve the highest possible rank.
(1) Of course we will have weighted results. We can probably stop discussing this. A 10 person SC will be worth less than a 60 person SC, or a 90 person Regional, etc.
(2) I love the idea of capping the total number of tournaments as a way to discourage the highest rated players just the ones that played more often. We’ll definitely integrate this, @mediohxcore. Thanks!
Actually, there’s a really simple way to discern the relative quality of tournament results, and that’s strength of schedule. SoS goes up with the length of the tournament, so it naturally scales with the number of players. The biggest problem with that is getting the data - this is probably only possible for tournaments run with some kind of software one would have an agreement with.
There’s quite a few ways you can control the influence that particular results have on the final ranking. Two of the more obvious ones are weighing, and best-X-of.
Sure, as long as you don’t fall into the trap of equating “this doesn’t say you have what it takes” with “this says you don’t have what it takes”. In the context of a ranking system such as this, you’d fall in said trap by making regionals/nationals wins so valuable that a lack of such win could not be feasibly compensated by any number of excellent results in lower-level events.
(remember, not everyone can actually attend a regional, let alone a national)
Of course a guy who won a Reg and 1 SC is better than just the guy who won 1 SC, but is he better than a guy who won 3 SCs and 15 field tournaments (read: every fucking event he attended, ever)? These are the kinds of questions that we’ll have to answer as we tinker with the math behind the ratings.
edit for the edit:
Precisely. I think the right way of looking at this is to cap the number of events that count for any particular category by the number you’re reasonably attending if you’re not going out of your way to attend as many as possible. So, let’s say that you count the best 3 SCs, the best Reg, the best Nat, and overall you take the best 3 finishes. Depending on what kind of ladder you want, you can then either have Worlds as a separate tier, or count it as a Nat (possibly with a 1.2 coef or something) - this is probably the sane thing to do, at least until Worlds actually take place in a… more reachable location.
Great points. The first iteration won’t be perfect, but we’ll really just have to put something out there and then see what worked and what (most importantly) didn’t work.
At this point, we should have something in place before SC season begins next year and I hope people will be patient with the failures and enjoy the successes!
I think there is certainly a system like this that could work. Now it’s mostly a matter of figuring out fair numbers for the weights and caps I think, and then implementing the system (assuming that we agree that something like this is the best idea).
This was pretty much what I mean to suggest earlier.
The best part is after you win your limit of SC/Regional/National/etc. you can switch your motivation for playing from selfish personal gain to concern for others.
“Sure, I can’t earn any more points, but I can stop insert-name-here from getting his!” 
Or, if we use best placement/most recent, you can go to try and bump your one second-place back up to a win.
If I was to try and describe tournament difficulty (and therefor ranking gain from placement) as a function of number of players, it would be an exponential formulae with increasing gains as additional participants are added. This is not close to say strength of schedule that has declining gains for additional participants (since it scales with the number of rounds).
I really don’t believe that 5 wins in 10-player tournaments equal the difficulty of a win in a 50-player tournament.
Actually, strength of schedule doesn’t scale with the number of rounds in a linear fashion - for each round you add, you have one more opponent, but also all of your opponents play one more round (and thus have a higher score, especially in the case of a tournament winner who presumably played against other people who won a lot).
And there is something to be said for consistent good placement (even in smaller events), as opposed to one win in a large event. One win could be coincidence, five seldom are. If you’re worried about extremely uneven fields skewing the results, you can do all sorts of checks (weighing based on average attendee rating being the most rudimentary).
Long time lurker, first time writer. 
Someone said rankade.
Our group started using it some months ago and we were immediatly addicted.
Free of charges, easy to use (players approvation needed only for first matches), multiplatform (webapp, iOS, Android).
Algorithm is very good and ranking is very significant (as for positions, ree points and time evolution), admin powers are good even for an ‘uncontrolled’ group needs (errata corrections, match deletion if someone tries fake insertion, etc), different weights for matches allows to record all matches in a single group (for us friendly matches are ‘light’ and round-robin tournament matches are ‘normal’).
And checking F.A.Q. it seems not to be a limit for number of players in a group…