This is all fabulous. Thanks for doing this. Some scattered thoughts:

1. Dropping raters with exceptionally high deviation scores will mechanically increase R2 and decrease RMSE, so I would be careful to only use those metrics when comparing weighting schemes that treat high-deviators the same.

2. Using the F stat is a very clever metric. I'm sure you're aware it's imperfect for these purposes, but still it's a really nice way to frame the issue, IMHO. Bravo!

3. I would be curious to see a graph of avg rating by year, comparing the different weighting schemes. Which years win? Which lose? (Ideally extend back to 1.0)

4. You've probably thought of this, but a middle ground option between doing nothing and doing full-on real-time adjusted weights would be to generate the weights at regular intervals, e.g. monthly. This may be easier coding-wise, although of course it would require some human labor with each update. (The idea being that any new accounts created in between weighting updates would get an initial weight of zero or close to zero.)