That said, I wouldn’t worry about this particular experiment being too biased or inaccurate. For every booster root-root-rooting for the home team with inflated estimaes, you’ll have someone like me who thinks nobody but Ichiro will break the Mendoza line and projects the team to win 50 games, tops. I’ve reached the tipping point where I’m going to wish for, and enjoy, the mounting losses more than the victories. I feel exactly like I did with the Mariners in the early 80s, just before I quit following them for almost a decade.

Whenever I’ve seen anyone throw out projections for Mariners players, they have typically been on the high end.

I agree, although I wouldn’t say “anyone”, I’d say “Mariners’ fans and denizens of Mariner blogs”. The non-Mariner writers and sites seem to be more pessimistic in general, maybe even too pessimistic regarding the Ms. (I’m thinking mainly of the seasons from around 2000 to 2003, when pundits thought that Ms were getting too old to continue to succeed. Eventually they did get too old, but it took longer than some people thought.)

I’d like to compare the aggregate results before and after the projections come out from Baseball HQ, BP, BTF, etc.

Or even see the results of the first round of community projections, and let people adjust their projections and submit new ones, after having seen other people’s projections. This is called the “Delphi technique” in forecasting. College football and basketball pools are somewhat similar, in that voters can adjust their votes week after week — in that case however, there’s also a flow of new hard information, namely the results of the latest games.

My knee-jerk reaction is that the community projections are likely to be of higher quality if people have stronger incentives to submit good forecasts, either in the form of a prize for best projections, or by having to … well, bet in a way; to put up some of their own money. The Iowa Electronic Markets work on this principle, people buy “futures” on e.g. what they think the outcome of the next presidential race will be. These markets have shown good quality results, a well-run stock market or futures market is probably the ultimate example of the Wisdom of Crowds.

I think that viewing it as an experiment is a good attitude to take here. Even if it turns out that PECOTA does better than the wisdom of crowds, that would be an interesting outcome.

So, we should be pretty good.

What would be an even more interesting experiment would be to create a “market” where fans could place buy or sell orders based on certain projections of players. That would probably give you even more accurate projections…and would probably be pretty fun.

Even with these flaws, I’d bet our crowd is smarter than PECOTA by itself. That doesn’t make PECOTA useless – it is one of our inputs, after all – but it also makes the effort worthwhile.

It’s not a perfect experiment, but it’s a useful one nonetheless.

Also, I think that most people (myself included) who make their own projections look at the advanced systems, decide whether those are high are low, and make adjustments from there. If it were possible, I’d like to compare the aggregate results before and after the projections come out from Baseball HQ, BP, BTF, etc. Not sure how we’d ensure that we had a representative sample in both of those sets, though, or that other news and noise hadn’t caused any differences that might be manifest. In theory, though, I’d hope that the earlier set might give us some insight as to what fans expected based upon their own methods (or lack thereof), completely separate from any of the advanced systems.