What Do We Know About Catcher Defense?
From Dave Cameron to Larry Stone, people have seen Kelly Shoppach’s signing with the M’s as something of a formality. He’s a veteran, he’s played for Eric Wedge before, he’s got a solid defensive reputation. Meanwhile, the normally tight-lipped M’s front office has made it clear that John Jaso’s defensive problems limited his value both to the M’s and potential trading partners. So, how do we know that catcher A is good and that catcher B is terrible? Perhaps no other area of analysis has changed so much in the past five years, so it may be time to look at what we think we know about measuring a catcher’s impact. For those looking for an air-tight conclusion, or a single number we can confidently slap onto Ronny Paulino, Shoppach or Jaso, this piece may disappoint. Those of you who are used to my epistemological musings will understand that we can’t yet do that, but yet it’s still interesting to think through the process.
So: what does a catcher do? What’s in the job description? They’re there to call pitches (sometimes), catch/block wayward pitches, to present somewhat less wayward pitches in the best possible light so that they may be called strikes, throw out runners attempting to steal bases, and function as psychologist, coach and motivational speaker to pitchers. Measuring passed balls/wild pitches is easy, though comparing it to an appropriate baseline is. Same with stolen bases – adding up the runs saved/lost in the running game is easy, but apportioning those runs to catchers and pitchers can be a bit trickier. Still, sabermetrics is on pretty firm ground in assigning values to these two components of catching; Fangraphs’ catcher defense metric is the sum of these two measures. However, the range of this metric simply isn’t all that big. The best/worst catchers range from about -5 to +5 runs in each one, so the gap between the best and worst catchers would realistically be in the neighborhood of +1 to -1 wins per year. That’s…not a lot, and several years ago, the idea that teams were too willing to move “questionable” defenders off the position were popular in some sabermetric circles. The squishy stuff like pitch framing and “managing a pitching staff” seemed near impossible to measure, and no one was certain if it was really true, or if it was just a baseball equivalent of an old wive’s tale.
The first real attempt at trying to measure the impact of the psychologist-cum-strategist role of the catcher is Catcher ERA, or cERA. In its simplest form, it just the earned runs scored while a catcher was playing – crediting the pitcher’s ERAs to the catcher. The problems with this are legion, and we’ve dealt with this issue several times at USSM. As Dave said back when people were clamoring for more Rob Johnson (this really happened! There was clamoring!), the fatal flaw with the measure is selection bias. John Jaso looks great by cERA in 2012 because he caught a lot of Felix’s starts.* Craig Wright updated it years ago in his book “The Diamond Appraised” by focusing solely on pitcher/catcher pairs. That is, he took selection bias out by controlling for the pitcher. Still, methodological problems remain – two catchers could have identical ERAs with a set of hurlers and come out with vastly different cERAs depending on how frequently they caught each of them.
The combination of small samples, a lack of evidence for a real effect, potential biases and the lack of an identifiable *reason* for catchers to have vastly different results led many in sabermetrics to dismiss cERA, leaving just the running game and pitch blocking to measure. Two things fundamentally changed that. First, Dan Turkenkopf and others tried to assign a value, in runs, of a pitch being called either a ball or a strike. An fastball on the black is ruled a ball by a vindictive umpire – what did that cost the pitcher and his team? The answer he got (0.161 runs) was stunningly high – if catchers had a repeatable skill in getting on-the-edge pitches called strikes, the impact of it could dwarf the value of CS% or wild pitches. Initial studies of the impact of pitch framing saw massive, almost impossible gaps between the best and the worst catchers, until Mike Fast’s definitive study on the matter put the range at about +/- two wins per year. For some catchers, this skill does in fact dwarf the impact of CS% and passed pitches. The measure is so new, and researchers are still tweaking it to ensure that they control for everything from the umpire, the park’s pitch fx calibration, the hitter, and the pitch type. This gap remains, and, more excitingly, this appears to be an actual skill – good pitch-framers in year X are generally good pitch-framers in year Y. Maybe the old baseball saw about catcher defense wasn’t just precedent mixed with hokum.
Now that pitch fx has been in use for 5 full seasons, and now that researchers can correct for the park-by-park idiosyncrasies in the data, sabermetrics is again looking at something like cERA to see if there’s any other effect a catcher might have on a pitcher’s performance. In reality, something like a new-fangled cERA (or what Tom Tango always called a WOWY -with or without you- study) is more about capturing everything than it is about isolating a particular skill. Still, might framing and, say, pitch blocking taken together exert a bigger impact on a pitcher’s confidence? Might a catcher’s pitch calling or the way he sets a target have an impact? Controlling for pitch fx, pitcher, batter and just about everything else, Max Marchi tried to measure the runs saved or lost by each catcher.
All of this suggests that teams should be focused on catcher defense. Indeed, the Astros snapped up Mike Fast not long after his article on pitch framing appeared (and another just hired Dan Turkenkopf), so we know that teams *are* paying attention to developments in this field. Marchi’s work on handling a staff and Fast’s work on pitch framing are so new, they’re not incorporated into WAR measures.** This may lead to WAR understating the importance of catcher defense, but I think it’s reasonable for sites to wait a few years to see if anyone’s able to shoot holes in these studies. There’s plenty of work to be done teasing out what it is a catcher does to get a pitcher more comfortable, and, perhaps more importantly, why a catcher will work quite well with one pitcher (or kind of pitcher) and not at all well with another.
There’s also the problem that catchers who are great at one aspect of the job are often poor at others. To bring this back to the Mariners’ catching depth (finally!), we see that Ronny Paulino isn’t great at blocking pitches or controlling the running game, but he’s above average in framing. Miguel Olivo was (surprise!) one of the worst in baseball in blocking pitches, but one of the best (at least from 2008-2011) at handling a staff. John Jaso rates poor in blocking and not-so-hot in pitch framing, but is a touch above average in pitcher results/handling a staff. Kelly Shoppach was a bit above average in framing and in the running game, but a bit below in pitch blocking and hide-your-eyes-bad in pitcher results. Jesus Montero’s not been catching long enough to show up on a lot of these lists, but his results in 2012 weren’t great. He was noticeably worse than Jaso/Olivo with many pitchers last year, and posted the worst K:BB ratio and OPS-against of the three backstops. We’ve got a heck of a lot more data, and even some new theories about how to use it, but I’m not sure we can definitively say that John Jaso was terrible*** or that Kelly Shoppach/Ronny Paulino/anyone else is really good. So help me out: what did YOU think of the M’s catching defense in 2012, and how do you think it’ll change in 2013? What aspects of the catcher’s role is sabermetrics ignoring, even now? What do YOU think the M’s care about in assessing a catcher’s performance, and if you had your way, what WOULD they care about?
* – It’s sort of interesting how catching a bunch of Felix’s starts doesn’t count as a credit, though. I mean, why would Felix keep throwing to him – and so well – if he Jaso was god-awful?
** – Baseball-Reference’s WAR measure for catchers – Defensive Runs Saved – apparently incorporates a cERA-like measure, though it’s tough to tell.
*** – John Jaso’s cERA highlights something that may be the next frontier in researching catcher defense. Over his career, he’s been pretty good compared to other catchers he’s played with (like Kelly Shoppach) in things like K:BB ratio and plain old BB%. Where he’s had a problem is giving up long balls. With the M’s in 2012, Jaso yielded runs and walks at the lowest rate of the three M’s catchers, but saw a lot more HRs per PA against him – particularly with Blake Beavan and Jason Vargas. In Tampa, Jaso had solid RA numbers and decent BB% rates, but was beset by HR problems, particularly when paired with fly-ball pitchers like James Shields and Matt Garza. But just as he had good results – better than Olivo/Montero’s results- catching Felix, Jaso seemed to work well with David Price. I initially thought the issue may be the result of issues with a specific pitch: the change-up. But most of the HRs came on fastballs, and Blake Beavan barely even throws a change-up. Still, it’d be interesting to see if there are specific characteristics of catchers that can lead to problems or successes in things like homers, double plays, walks, or strikeouts. Is Jaso’s low BB% and high HR% the product of a catcher who doesn’t call enough breaking pitches out of the zone? This would be an interesting topic for further research.