Confirmation Bias
One of the goals of this blog is to be a rational voice in a sea of emotional response. So, take this post in that vein.
A frequent comment that has been brought up as the M’s struggle to score runs and win games to begin the season has been that it’s okay to freak out about the offense because it was a weakness going into the season, and the first eight games are just evidence of the suspected flaws of the team. I understand why this makes sense on some level, but in reality, it’s just bad analysis in a different package.
That belief is really just a manifestation of confirmation bias, or the natural tendency to weigh evidence that agrees with your preconceived opinion too heavily. Everyone gives in to confirmation bias at some point, as their expectations are met and they inaccurately conclude that they “knew” something was going to happen. There are very few things in life that we know – the rest is different levels of speculation.
A lot of people speculated that the Mariners offense was going to struggle to score runs before the season began. That speculation may be founded on logical principles, with evidence to support it, and it may even be correct. However, looking at any small sample of data and deciding that it is “real” because it conforms to what you believed before it happened is simply faulty, and underestimates the uncertainty that was involved (or should have been involved) in your original thinking.
People do this all the time, and it causes them to make poor decisions. We simply do not have the ability to forecast the future with certainty, so we have to leave room for the fact that our forecasts could be wrong. When you decide that eight games is early enough to say that this team can’t score enough runs to win because you believed that was true a week ago and this simply validates your opinion, you aren’t building enough uncertainty into your forecast.
This isn’t to single out anyone – I’m just using the lack of offense as an example. I could use Rob Johnson’s constant dropping of the ball as a similar example. Because he was so frustrating to watch at times last year, there’s a preconceived notion that he can’t catch the baseball, so things like his dropped third strike on Gabe Gross last night get magnified, because it lined up with what we already believed. There’s confirmation bias everywhere in life. It’s just something we have to recognize and attempt to adjust for, because if we don’t, we’ll make bad predictions and end up overreacting to recent events.
So if you expect to find confirmation bias, and when you look you discover a confirmation bias, is the confirmation bias real or is it a confirmation bias?
Preconceived or not, the fact is they arent winning the close low scoring games (yet). That is going to be the need all season. Also turning someone like Duchesner (sic) into a HOF pitcher after having faced him less than a week before makes one nervous. Cant waste pitching performances by our four backup starters such as Snell in game 2 and RRS did yesterday. Everyone including the team need to relax, take a deep breath and go from here!!!!!
Thanks Mike, I now have a headache in addition to being sick.
j/k… about the headache part anyways…
I understand confirmation bias, but is my tempering of my expectations considered a confirmation bias? It’s not a full blown “we’re gonna die” sort of deal, but more of the idea that variance could kick us in the you know where. I could still see us getting to my original expectation of last year’s record, but given the “unluckyness” (is that even a word?) so far we could easily end up a couple of games under .500.
This is exactly why I read this blog. Thank you for confirming it.
You could even see this in the first game as Chone Figgins advanced on errors from the catcher twice while trying to steal. (“I thought he would wreak havoc on the bases.”)
Sure. It’s something Dave touched on last year when the team started 7-2.
Well, for example, you could see Milton Bradley hitting .045 as a deviation. I don’t think it’s random, since there is a reason for it. He’s cracking like an egg. Shattering bats, flipping off fans. But over the course of a season he could find his swagger and gravitate towards the mean with an offensive firestorm. So could the team. They could rip it up for eight games somewhere down the road.
That said, it’s a little problematic to claim we have no idea how these guys are going to perform over the long haul when you’ve always argued that past performance dictates future performance. I know it’s only eight games and agree that over the long haul the offense should be low-middle instead of sub-basement. I’m just saying that it’s a simplification your very own arguments that is feeding this negativity.
Your general point is well taken, and I think it is appropriate for the offense. Given our knowledge of things like BABIP and the probabilistic nature of most batting outcomes, that makes sense. However, I’m not sure if Johnson’s dropsies are analogous. While every catcher is bound to drop a couple balls over the course of the season, do we have the same sort of expectations with regards to regression to the mean (e.g., the percentage of third strikes he does/doesn’t drop)?
If you really want something to worry about, worry about the fact that Nelson Cruz is going to hit 116 HR this year.
Great post, Dave. You have made a very succinct and cogent summary of confirmation bias. I teach decision-making for a living, and you have summarized the problem quite well. We all seek information that confirms our opinions because it gives us the secure feeling of being right.
Our politics is an example of the mess this can get us into. People wish those who have different opinions would be more open-minded, but they are unwilling to put themselves in the uncomfortable position of considering facts that go against what they believe to be true. Conservatives seldom watch MSNBC. Liberals seldom watch Fox News. It is more comfortable to hear information that confirms what you believe.
Dave has never argued this.
Milton Bradley is “cracking like an egg”? Confirmation bias?
charliemountain,
“Past performance” is several years worth of data, not several games. There are plenty of examples from last year’s team why even a month or two of data shouldn’t be taken at face value.
What To Make Of This Team
Dennisss – According to ESPN, “The Mariners sat Milton Bradley down following an incident with fans last week and told him not to put so much pressure on himself… The man who told The Associated Press last month he was baseball’s Kanye West… flipped off heckling fans from the outfield during the fourth inning at Texas on Friday night… shattered his bat into splinters after pounding it to the ground in frustration following a strikeout… He is 1-for-22 (.045) this season.”
Yeah. I’d say he’s having some emotional difficulties. I don’t think that’s a stretch.
About this Rob Johnson dropping balls bit, are there any stats we can use to prove the point one way or another? Because he sure seems to miss more balls than other catchers I’ve watched, both pitches in the dirt and throws to the plate.
We can’t use Passed Balls, because that’s flawed the same way Errors are for the other fielders, since it’s only the most obvious mistakes that get blamed on the catcher instead of the pitcher.
In general, catcher’s defense seems to lag behind other positions. Is anybody working on this?
Great points, Dave, and I agree with you. It cannot be over-emphasized, however, that claims of confirmation bias can just as easily be used as a means to avoid reality and attack those one disagrees with, even when they are making reasonable arguments (it happens against USSM authors all the time). Some decisions – Jose Vidro is not cut out to be a DH – require a shorter period of analysis (especially when coupled with the objective evidence and reasoning that came along with the attempt to make him a DH).
It is similar to a concept in criminal law concerning “unduly prejudicial evidence.” All evidence is prejudicial, or the advocate would not seek to introduce it. Each side is attempting to hurt the others’ theory of the case. There are protections against the introduction of “unduly” prejudicial evidence in court, however, because, even though such evidence is likely relevant, the relevancy is overshadowed by the probability that the fact-finder will draw unfair conclusions based on it.
The same is true here. We are really talking about giving due weight to all evidence, based on the circumstances. Depending on the belief one claims is confirmed by the evidence, a person is either jumping the gun, or proving a hypothesis to be true. Different conclusions require varying types and amounts of evidence. I think there are some conclusions that are supported in a couple of weeks worth of games. The claim that “this year’s offense sucks” is simply not one of them.
DMZ and Liam – You misread what I said, or maybe I didn’t say it clearly. Dave has argued that several seasons of past performance dictates future performance most of the time. Based on a lot of these players’ past performance, we can expect them to hit below the league average. I’m saying some people are seeing these guys hit below the league average and skipping a step. But it grows out of this idea that we already know how guys are going to play before the season starts because we’ve seen their projections. I think Sabermetrics – when hastily applied – lends itself to people forming confirmation bias.
As long as you say “dictates”, you aren’t saying it clearly.
Regarding confirmation bias:
Excellent book for pointing out the morass between knowledge and certainty, if you don’t mind a nonlinear approach. Not nearly as intimidating as the Tractatus.
You’re using the term “dictate” incorrectly. Past performance, even several seasons of it, doesn’t dictate anything. Dave has never argued that.
Past performance is a good indicator of future performance. That’s a far sight from dictates, or even the idea that “we already know how guys are going to play because we’ve seen their projections” which is another thing no one’s ever said.
Jack and his front office are being given a lot more patience and time because they built this team the way many of the Mariner fan blogs wanted to see the team built. That’s not confirmation bias specifically, but it’s some close relative.
If Bavasi and company were still running this organization, there wouldn’t be calls for calm and posts about how confirmation bias works. A 2-6 start to the season in that instance would incur a lot less cries for patience and a lot more jokes about Mascot.
So you want us to treat a 2-6 start from a front office that exhibited years of rank incompetence the same as we do a 2-6 start from a front office that salvaged the team from years of rank incompetence?
Why? Does context ever mean anything?
Go read last year’s posts after the team started 7-2. We loved Jack and crew back then too, but we weren’t getting carried away by the team’s hot start. We were saying the exact same things we’re saying now.
In short, I don’t agree with your premise, nor do I think you have any evidence to support it.
All that aside, this team doesn’t pass the smell test.
Rob Johnson can’t catch. He couldn’t catch last season, and he can’t catch now. Nothing’s changed. Watch a game. He will drop at least two balls. He does it every game. He can’t catch.
Then maybe you should get a new nose.
Wow, it didn’t take long to prove my point that claims of confirmation bias can be used as an improper means to avoid reality and attack reasonable arguments.
4/11, 2009:
4/14, 2009:
and then later that night…
4/16, 2009:
So, yeah. There was a tiny bit of getting carried away by the team’s hot start. And not exactly the same things you’re saying now.
If cautious optimism was the mood of things after 7-2 last year, wary pessimism is perfectly justified after 2-6 this year.
Care to point out which of those quoted comments was, you know, wrong?
2009: 7-2 in the first 9 games of the season = “get ready for some kind of pennant race”.
2010: 2-6 in the first 8 games of the season = ” …looking at any small sample of data and deciding that it is “real” because it conforms to what you believed before it happened is simply faulty…”
Right, let’s just ignore the paragraphs and paragraphs of math that explain the “pennant race” comment, and the fact that it was entirely correct. And the fact that there was zero insinuation of the 7-2 start being “real”, or anything close to it. And ignoring all the comments around those posts that suggested that we still saw the M’s as a true talent 78 win team.
You’re just cherry picking at this point.
Here’s another way to put it, perhaps:
After 8 games, an average team would have scored about 37 runs. (Given the defense-oriented construction of the team, a league-average offense can be counted as success here.)
Given the pre-season PECOTAs (which weren’t terribly optimistic about the Mariners), we could have expected about 35 runs scored by now.
The team has actually scored 21 runs.
So of the shortfall in the offense, only about 10-15% of it is “okay, they probably weren’t going to be that good anyway,” and 85-90% is “doing worse than we could reasonably have expected.”
The more important question going forward, of course, is: How much of that 85-90% is “they’re just slumping, but they’ll pull through it eventually,” and how much is “maybe they’ve taken a further step backwards this year”?
Following on from that: The question which I’d be interested to see the site writers address, if the team’s troubles continue, is: Are there any players yet of whom you would say, “Coming into the season I expected .xxx/.yyy/.zzz from Player A; but now it looks like it’s just not happening (not only because of the initial results, but also because the bat speed just isn’t there anymore, or the pitch recognition, or whatever)”?
(I phrase the question that way because when enough evidence comes in to change your mind on something, then whatever fallacies might be in play, “confirmation bias” isn’t one of them. Then it’s just a question of whether you’re changing your mind too quickly.)
More perspective:
Maybe these teams should be looking to make some moves too?
Michael Young (TX) is hitting .185
Mark Texeira (NYY) is batting .125
Nick Johnson (NYY) is hitting .135
Kurt Suzuki (OAK) is batting .185
(Didn’t folks say we could use him?)
So, who thinks those numbers are indicative of where these guys will be at the end of the year?
(Show of hands) And if not, then why can’t we expect the M’s guys batting below projections to trend back up, too?
(PS- Dave, it seems you didn’t specifically spell out the part about “confirmation bias” that involves people taking quotes and soundbites out of context in an attempt to prove their own point of view and to contradict a prior premise…)
“Entirely correct”? Pennant races, to me, don’t stop in late July…
You asked to go back to last year’s posts after 7-2, then when I did, anything contrary to your position is ‘cherry picking’. Ok.
We can both agree that we’ll have a better idea about what the 2010 team will really look like in about a month from now. Fair enough. I’ll leave it there.
MrZDevotee – Kind of hard to take something out of context when I reposted the whole thing here.
Chris–
I think I remember Dave posting quite a bit about people shouldn’t get ahead of themselves and think this team can continue winning this way early last year. I remember posts talking about the crazy numbers a few folks were putting up that couldn’t be continued (including Branyan and Chavez– neither of whom even finished the season).
And I don’t understand what the point you were making was when Dave explained after the quick start that even if they regressed to original predictions, we could expect them to win 3 more games than originally suggested.
He was telling us we weren’t gonna win 100 games last year.
And now he’s telling us we’re not gonna lose 120 games this year.
Yes, there was reason for optimism last year, and he wrote about it, but that wasn’t the only observation he had. That was the “context” i was referring to.
Our pessimism this April should be just as carefully throttled as our optimism should have been last year.
(I believe using just the posts when we were 7-2 last year would be considered a “small sample size”.)
Psychology 101 right there… lets hope the M’s get this thing turned around, before its too late.
Well, you know… pitching might help. Bleeding doesn’t stop for a while.
Dave,
I’m a big fan of this blog and a big fan of statistical analysis. However, you and others on this site have, at times, come across a bit overconfident and condescending towards opposing views.
Part of being knowledgeable in a subject is understanding that most topical issues are neither black nor white, but instead appear in various shades of gray.
Whether you want to admit it or not, not all in your field agree with the level of importance you place upon defensive metrics. Is it not possible that you have been wrong about the weight given? We are not all morons just because we disagree with you.
A degree of humility would not be the worst thing that could happen to this blog.
Where in this post was there anything about defensive metrics? The example used was actually the offense, not that the nature of the example is essential to the point being made.
Well, you know… power at the plate might help. Bleeding doesn’t stop for a while.
These are two slightly different statements, and I think one of them is good common sense/good analysis, and the other is indeed an example of confirmation bias.
The first statement, “struggled to score runs” — was there anyone who did NOT think that the Ms, given their roster and ballpark, wouldn’t be near the bottom of the AL in scoring runs? That’s what we predicted, and that’s what’s been happening. And that’s what will continue to happen this season.
But the second statement, “can’t score enough runs to win”, that was a much more speculative and controversial statement, and one which is still speculative a mere 8 games into the season. People who are riding on that statement’s bandwagon are indeed guilty of confirmation bias. We don’t yet know — I think our typical prediction around here was maybe 85-88 wins? The losses are real so we’d want to reduce those numbers a bit, maybe it turns into 82-85 wins or whatever. But to conclude that the Ms can’t win — Dave’s right, it’s too early to conclude that, and if you’re letting the early results cloud your judgement, that’s confirmation bias.
Chris – I would have to agree with Dave on this one. And it’s Liam’s response to my inquiry at the beginning of this post that makes sense.
After the 7-2 start last year, Dave said that we have those wins in the bank and didn’t have to give them back. It’s like a sunk cost. (Well, this year it is so far, last year was a sunk benefit).
He then gave 3 scenarios based upon people’s expectations. So for example if one believed that they were expecting a .500 ballclub they would go 77-78 or 78-77 the rest of the way, putting them at 84-85 wins.
With that reasoning, and his initial expectations coming into the season, his expected win total put the M’s in a possible pennant race come Aug/Sep – hence his comment.
So if we were to apply the same reasoning here after the 2-6 start, and we were expecting another .500 ballclub this year, we’re going to see a team at 79-83 – which was part of my “Well, I could easily see us ending up a couple of games under .500″ comment.
The key is that Dave’s perception of the team going forward didn’t change. The 7-2 start didn’t change his expectation of the teams performance going forward. An exaggerated scenario would be if last year the M’s went say 22-0 to start the season. An expectation of a .500 ballclub would have the team at 92-70.
Dave hasn’t mentioned that he still expects the team to go to the playoffs or any win total in the posts so far. All he says is that he expects the team to get back to the expected performance he thought of. With the 2-6 start that will lower the overall win total, sure.
The problem that most other people are having is that people are changing their expectations of the club going forward based upon the results of the 8 games played. I myself am somewhat guilty of said problem, although my end result would have been the same had I used Dave’s reasoning.
So everyone, give the math a whirl using your beginning of season (not today’s) expectation.
I have to admit this is the first time I have ever read every comment in a thread. Wittgenstein? Pass the salt! Perhaps I can add that it has not been easy historically to separate one´s interest in the outcome either from one´s analysis or one´s conclusions. Your post paints with a broad brush. Did Bavasi’s regime merit pessimism? OK. Did that leave the barn door open for confirmation bias? Can you objectively reject that premise?
Food for thought:
2002 Angels started 6-14, won the World Series
1987 Minnesota Twins had a 29-52 road record, won the World Series
2001 Arizona Diamondbacks fielded a team featuring Craig Counsell, Tony Womack, Junior Spivey, Rod Barajas, Damian Miller, MIGUEL BATISTA, Greg Colbrunn… Yeah, okay, Alex Gonzalez… And (ironically enough) two stud pitchers: Randy Johnson & Curt Schilling. And, they won the World Series.
I’m not implying that the 2010 Mariners are gonna win the World Series of course, just saying there’s a lot of season left. Let’s give it time. (And hey, what are the odds Jackie Z can go a whole season without pulling some strings and doing a couple trades that surprise us? Who knows what 2010 still holds…)
Tonight is win #3.
A rational way to look at this 2-6 beginning would be to look at similar 2-6 stretches by good teams, and to see how common they were compared to sucky teams.
so let’s look at the .562 09′STL:
2-7, may 8th to 18th
2-8, may 31st to june 9th
2-7, june 22nd to june 30th
and 2-8 to end the season when they had the division in the bag. (you could call this a 2-11 finish to the season if you count postseason)
so during three distinct stretches the NL Central division champs went a combined 6-22 in 18% of their season.
(the Yankees only had a 2-6 period once the whole year but they were a .636 wpct team)
for good teams a 2-6 stretch isn’t unheard of, but it isn’t common. For a team like last year’s Nationals……2-6 was quite normal.
While there are examples of teams that had really poor stretches and still did well, they are historically rare.
Over at Fangraphs, Dave has Texas winning the AL West in Predictions That Will Be Wrong. (His picks are the same as the aggregate results of 5 different projection systems, as mentioned here.)
Again: IF their 2-6 record was indicative of the quality of the Mariners, in terms of true talent, that would mean you should reasonably project them to go 41-121 or so.
Does anyone want to seriously argue that the 2010 Mariners are comparable to the 1962 Mets? Or the 1935 Braves (picking two all-time horrible teams out of the hat)?
No? Allrighty then.
Yes, it’s unfortunate that we’re in a cold stretch at the beginning of the season, and yes, it’s possible that a couple of precepts the M’s started the season on offense with are pretty badly wrong:
- Casey Kotchman will be a ~2+ WAR player with an improved bat, closer to his 2007 performance,
- Milton Bradley will be a ~2 WAR player and bounce back from HIS blah season,
- Ken Griffey and Mike Sweeney will be OK at DH.
Those had to be legit concerns coming into the season… and, well, nothing we’ve seen so far has convinced me they still aren’t (Sweeney’s spring training is useless for dispelling that.) That being said, it’s EIGHT GAMES. But a stretch of terrible offense shouldn’t surprise folks: Ichiro has months where he hits .270 with no power, Jose Lopez has weeks where he’s a GIDP machine, and combined with the above three things, and minus about 4-7 WAR of talent out of the rotation… yeah, we’re 2-6.
But seriously? We need more time to come to conclusions. The M’s can’t be redoing the roster any time we have a bad week, and very few teams WON’T have a stretch of 10-20 bad games. We’re just getting ours early.
I’ll say this again, because it’s getting lost… Where’s the discussion on the pitching staff? Even with Lee back in full form, the bottom of the rotation is terribly weak. Without considerable run support, losing will out weigh winning for this roster…. not on this level, but I can’t see them winning the division or WC at this stage.
I am sorry I came in late. I just skimmed most of the comments, but I want to point out that those 8 games were not draws from some totally unknown sample. Most of our expectations have been formed by experience if not a lot of data analysis. In that case, a more Bayesian approach is at least permissible if not warranted. 8 games of awful offense allows us to slightly adjust our expectations of the offense downward. They probably won’t hit .220 for the year, but if we were looking for .260, maybe we have to start thinking about .250. You all can take the time to translate that into runs and more meaningful batting stats. I just wanted to point out that confirmation bias isn’t a sin as long as there are also adjustments to expectations when the evidence conflicts with our prior expectations.
I just skimmed the comments as well. There is most definitely an interesting dillemma here. Before the season starts, we can say we have a pretty good idea of a team’s true talent level and thus how many games that team is likely to win. When the team starts off on a streak, good or bad, how does that affect our predictions? Is the hot (or cold) start part of the randomness of an entire season, in which case we shouldn’t alter our predictions for the rest of the season, or is the start truly “in the bank”, in which case we should project the team to play to its talent level from here on out (meaning we must change our season predictions based on nine games)? Dave seems to have taken both sides of this argument at different times. Both arguments are reasoned and rational, and I have no idea which is right. Seems like a great idea for a statistical study.
Jon S.
Here’s a recent post at Fangraphs about this after Vernon Wells hit 3 HRs in his first 8 PAs.
Saving this here for the next time I need an example of confirmation bias.