First, we will reprint the final analysis from last year explaining what CIA is, why it worked and how well it worked. Then, we have the current version of the Algorithm itself, and, lastly, the initial set of predictions. Here is the final explanation:
Why a Closer Identifier Algorithm?
The idea behind the Closer Identifier Algorithm is that a simple algorithm, a series of simple yes/no questions, can make better predictions about the future than a clinical analysis by the best experts. My view was that the industry became too entranced by the ever-widening quest for more accurate projections, which is already at the point of diminishing returns.
Additionally, fantasy players are faced with a veritable alphabet zoo of metrics, each designed to further quantify what happens on a baseball field. While the zoo of metrics may be good for explaining what has happened, many of them do not help predict the future very well.
Fantasy players already have all the metrics they need; the problem is that players do not know how to use them. The traditional clinical analysis by experts is besotted with psychological flaws, innate biases and outright logical inconsistencies. For example, just read any site’s player profiles and you will be aghast if you keep track of how many times the basics are applied inconsistently. And when you apply even more subjective ideas like scouting the battle is virtually impossible.
So, the idea behind CIA is that we need a simple way of using the information that already exists. And what better place to give it a whirl than the most difficult group of players for fantasy owners to handle correctly, namely closers?
Why did it work?
The main benefit of an algorithm is its ability to avoid psychological biases and inconsistencies. I have written many times over the years on these biases, and won’t rehash them here. CIA weighs variables the exact same way in every case; this is impossible to do in a clinical analysis and more than anything else this is the main reason for its success.
Another is that it does not get bogged down in subjective information. CIA never says “his fastball is doing better” or “he has a great changeup” or “his manager thinks he is ready.” Some will undoubtedly counter that this guarantees it will get some things wrong and will never be perfect, with the implication being that a clinical analysis might do so. If someone says that, they are telling a half-truth if not outright lying. It won’t be perfect, but it will be “better,” or so the thesis goes.
A full explanation of why it works would take a lot more than one article, so I am happy to discuss it with anyone interested.
The results were even better than my most optimistic projections. This year was maybe the most difficult ever faced among the ranks of closers. And yet, despite all this, CIA hit a home run in its predictions.
The final tally: 54 correct predictions and 12 incorrect predictions. And of pitchers that ended the season with at least a share of the closer role it was correct 29 times and wrong only 6.
Now, to be fair, many of the correct predictions were for closers that everyone could have predicted correctly. For these closers we designated them as “easy” prediction, either to hold or lose the job. After all, it doesn’t take an algorithm to know that Craig Kimbrel had a very low probability of failure. And the value here can only be derived from situations where CIA got it right and others got it wrong; the tough cases.
In order to figure this out, I made my best guess on how difficult it would be to predict a particular case when they got the job. It is perhaps easy in hindsight to say “I knew Steve Cishek would hold the job in his second chance but would blow it in his first chance,” but whoever says this is lying. You can see the breakdown here.
Looking only at the closers who ended the season in the job, we had 11 “moderate difficulty” predictions: Johnson, Bailey, Reed, Perez, Holland, Janssen, Belisario, Lopez, Street, Romo, Javier Lopez. CIA predicted these correctly in all 11 cases.
For the tough cases, there needs to be some explanation. Brandon League is a “tough” call because no one would have predicted he would have repelled both Belisario and Kenley Jansen upon his return. But Jansen is also a “tough” call to get right because most predicted he would get the job back. The difficulty rating is in how hard it was to get it right in the end, not whether it was tough to hold the job.
With that in mind, we had Frieri, Perkins, Wilhelmsen, Rodney, Marmol, Jansen, League, Francisco, Axford, Clippard. A review of this list shows why I rated them as tough. They all had stiff competition for the job, had bad skills, or were unknown. Some of them failed already early in the year. I am happy to discuss these by the way, but I don’t think it matters a lot; the main part is in getting rid of the easy predictions.
In the “tough” category, CIA missed on Kenley Jansen, League and Clippard. Its record was a solid 7-3, not much better than random chance, but better, and in the most difficult situations.
Where CIA had the most trouble, paradoxically was in the easy group. Its misses among the “easy” group were Valverde and Gregerson. When we add in the other misses we see an easy story to divine: Sean Marshall, Steve Cishek in his first shot, Dale Thayer, Bobby Parnell, Jared Burton and Jim Henderson. All of these cases had easy explanations that almost every analyst could have gotten correct and probably did.
The only flaw in the methodology was in not ignoring closers that change because of injury concerns. Gregerson, Parnell, Thayer and Burton all fell into this group. I am not willing to excuse them, however, because the theory here is that a closer who ascends to the job has a chance to repel the incumbent. Any analysis has to consider that, so why should an algorithm get a pass?
The current version of CIA:
1. Does the pitcher have a share of the closer’s job? In the cases we are writing about this will usually be yes, but it should apply to all relievers if it is valid.
If so, is his save percentage greater than 90% with 10 or more opportunities, or in his last ten opportunities? If “yes” then “hold.” If “no” proceed to step 2.
Is the pitcher’s save percentage less than or equal to 60% over his last 10 opportunities? If “yes” then “lose.” If “no” proceed to step 2.
2. Is his strikeout rate (K/9) above 7.0?
3. Is his strikeout-to-walk (K/BB) rate above 2.5?
4. Is his BaseballHQ BPV above 80?
If the answer to conditions 2-4 is yes, then CIA predicts he’ll keep the job. If the answer is no to any one of the conditions, it predicts he’ll lose the job.
The inital results for 2013, with commentary:
|Jim Johnson||Hold||Weak K rate but 94% SV?SVO|
|Joel Hanrahan||Hold||1.9 K/BB but BARELY holds on with 90% SV/SVO|
|Addison Reed||Hold||Not close to top tier|
|Chris Perez||Hold||One of the big surprises last year|
|Joaquin Benoit||Hold||May have nominal competition, but is easily the best bet in the pen over Rondon.|
|Bruce Rondon||Lose||Wouldn’t qualify for the AAA version of CIA|
|Phil Coke||Hold||Most likely won’t start with a share of the job.|
|Greg Holland||Hold||BB issues need to improve. Third Tier closer for now.|
|Kelvin Herrera||Hold||Might be better than Holland right now, and is a solid backup plan.|
|Ryan Madson||Hold||Based on 2011 season stats.|
|Ernesto Frieri||Hold||Despite hold rating, there are negatives and any K erosion will make him a “lose” status.|
|Glen Perkins||Hold||Despite being a lefty, he is top-five worthy, and under the radar.|
|Jared Burton||Hold||Like Kelvin Herrera, he is a great backup plan.|
|Mariano Rivera||Hold||Based on 9 appearances in 2012.|
|Ryan Cook||Hold||BPV says he is below the better guys but above the third-tier guys.|
|Grant Balfour||Lose||BPV of 79 barely misses “hold” status. 2.6 K/BB marginal.|
|Tom Wilhelmsen||Hold||Rebounded nicely and looks solid.|
|Fernando Rodney||Hold||48/50 SV/SVO|
|Joe Nathan||Hold||A top five option, and no negatives.|
|Casey Janssen||Hold||The most underrated closer? He is right there after Kimbrel.|
|JJ Putz||Hold||Near the top of the heap.|
|Craig Kimbrel||Hold||Highest BPV over Chapman|
|Carlos Marmol||Hold||Skills are well below CIA minimums, but he makes it on SV/SVO in last ten (100%)|
|Kyuji Fujikawa||Lose||Based PURELY on projected K/BB and BB rate. We include him for completeness but does not technically qualify yet.|
|Jonathon Broxton||Hold||He has the least rope of any closer based on CIA minimums.He only barely merits “hold” status.|
|Sean Marshall||Hold||As it looks right now, he is a much better bet than Broxton, but for the manager.|
|Rafael Betancourt||Hold||Is on a big K/9 decline that, if it continues, will earn a “lose” status.|
|Brandon League||Lose||Well below all CIA minimums.|
|Kenley Jansen||Hold||His BaseballHQ BPV says he is a top five closer right now.|
|Steve Cishek||Hold||Gets a “hold” based on his last ten, but the skill set is weak and the leash is short.|
|Jose Ceda||Lose||If he gets a share of the job his BB rate says he will lose it. The situation may be in flux.|
|Ryan Webb||Lose||The Marlins have to hope that Cishek keeps his “hold” rating or they are in trouble.|
|John Axford||Hold||Like Cishek, his success in save conversion merits a “hold” status, but the skills set is still a work in progress.|
|Jim Henderson||Hold||He easily bests CIA minimums.|
|Frank Francisco||Hold||He had a fairly decent run of coverted saves that merits a “hold.” For now.|
|Bobby Parnell||Hold||He may be the new Ryan Madson over the next few years, getting no shot for purely subjective reasons.|
|Jonathan Papelbon||Hold||Virtually no worries.|
|Jason Grilli||Hold||2012′s small sample is the only one that suggests he will hold the job. And he is 36. make of that what you will.|
|Mark Melancon||Hold||Was much better than subjective analysis will give him credit for. Has a good shot to lead the Pirates’ pen.|
|Huston Street||Hold||We all know the drill. If he is healthy he is a top closer.|
|Sergio Romo||Hold||Easily a top ten, safe pick.|
|Jason Motte||Hold||A few too many BS for aethetic purposes, but is fairly safe.|
|Rafael Soriano||Hold||Held a miniscule HR/FB until September, which buoyed him a bit.He is a clear “hold” but not safe.|