NFL QB evaluation is one of the single most important talent evaluations in all of sports. NFL teams spend millions of dollars and invest years of work and hope into these young QBs from college. For this, NFL teams employ hundreds of employees and countless man-hours into evaluating which college QB will best lead their franchise.
So much is at stake in this evaluation, and yet, the 1st round hit rate on QBs is just 46% (hit rate is defined as players who received a 2nd contract with the team that drafted them). If you were an NFL owner with tens of millions at stake (a billion dollars based on franchise value), would you be willing to bet on just 46% success? You might as well gamble on the teams instead of owning them.
Hit rate of 1st round draft picks 2000-2019
So, NFL teams need to improve their talent evaluation of NFL QBs. But, how?
The subjective evaluation of QBs leaves a lot to be desired. Human beings are imperfect in how we evaluate other human beings (just look at the divorce rate). Focusing on how “good” a QB looks, for example, footwork, release, arm strength, etc, can lead talent evaluators astray. Many QBs who “looked the part” did not succeed in the NFL.
So, is there a way to measure how the QBs performed in college and translate that into NFL success? What statistics in college best translate to the NFL? I think you would be surprised at what actually matters – we will go through 3 of the main statistics that people use: TWP rate (turnover-worthy play rate), BTT rate (big-time throw rate), and Accuracy (% of passes that were on target).
And, then we will review the one statistic that matters more than those – and how that led to creating a model that can predict NFL QB performance within 15% of actual NFL QB performance.
Note: All statistics used come from PFF Premium Stats (highly recommend PFF Premium Stats). These stats are great because they break down each stat by play type, defensive play, and play time. Very useful in developing a model for NFL QB performance prediction (NFL QB performance is measured by EPA per play – follow this link for details on EPA).
TWP Rate
PFF defines TWP: A turnover-worthy play is a play that has a high probability of being intercepted or resulting in a fumble. TWP Rate is just the percent of plays that had were turnover-worthy.
Turnovers are the biggest factor in winning in the NFL. NFL teams who have a positive turnover ratio (at least one less turnover than the other team) win ~70% of games. It is highly predictive of winning games, so a QB who has a low TWP rate in college must correlate to a high performing NFL QB.
But, we see only slight correlation of TWP rate in college translating to NFL QB performance. NFL EPA per play has an R2 value of just 0.018 (R2 is explained here). And, we see some of the most elite QBs in the NFL (Lamar Jackson, Patrick Mahomes, Josh Allen, Jared Goff, Dak Prescott, Jordan Love, etc) with TWP rates >3.2% (average of NFL QBs in college since 2015).
NFL EPA Per Play vs TWP Rate (in College) (QBs with at least 150 snaps since 2015)
However, we can look at TWP rate on various plays: play action vs non-play action, pressure vs no-pressure, etc. Is there a better determinant for TWP rate in college to NFL QB performance?
Yes, there is – TWP rate on Plays w/o a Blitz translates much better than overall TWP rate. This metric controls for how defenses played against each offense so we can compare the QBs on a more even field. TWP rate on no-blitz plays had an R2 of 0.046, much better than overall TWP rate. However, we still see some elite QBs above the college average of 3.1%, such as Patrick Mahomes, Lamar Jackson, Josh Allen, Jared Goff, and Jordan Love.
So, TWP rate is important (and more important when there is no blitz), but it is still not incredibly predictive.
NFL EPA Per Play vs TWP Rate on Plays w/o a Blitz (in College) (QBs with at least 150 snaps since 2015)
BTT Rate
PFF defines BTT as a big-time throw (BTT) is a pass that is high in value and difficulty. BTTs are characterized by excellent timing and ball placement, and are often thrown into tight windows or down the field.
So, if TWP rate is not very predictive, then is BTT rate better? We all see the highlights of the downfield passes that are perfectly placed resulting in big plays. QBs who can make these throws downfield in college should be better in the NFL, right? Let’s see:
NFL EPA Per Play vs BTT Rate (in College) (QBs with at least 150 snaps since 2015)
BTT rate is even less predictive than TWP rate with an R2 value of just 0.006. Some of the best QBs from college in this statistic did not perform well in the NFL (Zach Wilson, Nathan Peterman, Brett Rypien, for example).
However, breaking down BTT rate further, we do see a much better correlation to BTT rate on non-play action plays. With an R2 of 0.035, BTT rate on non-play action plays is 5x more predictive than overall BTT rate.
And, this makes sense – offensive coordinators design play action to make it easier for the QB to get the ball downfield. On non-play action plays, the ability to make those big-time throws is more on the QB and less on the offensive design (although still related).
However, this is still not very predictive. Let’s look at Accuracy %.
NFL EPA Per Play vs BTT Rate on non-Play Action (in College) (QBs with at least 150 snaps since 2015)
Accuracy %
PFF defines accuracy % as passes that are on target to the wide receiver. This is equivalent to completion percentage + drop percentage, so the QB is not lessened due to drops.
Obviously, QBs who are more accurate in college should translate better to the NFL. That is the main job of the QB – get the ball into your receivers’ hands. So, does it? Let’s see:
NFL EPA Per Play vs Accuracy % (in College) (QBs with at least 150 snaps since 2015)
So, accuracy does have a better correlation than BTT rate with an R2 of 0.01, but it is still lacking. And, you would have missed on elite QBs like Lamar Jackson, Jordan Love, and Josh Allen (as did I, and many others, did in my early draft scouting days).
Breaking down accuracy further, we do see a better correlation to accuracy on throws between 10-19 yards (or medium distance). This intermediate area is critical in the NFL – NFL defenses are faster at the LB and safety position and defenses are designed to take away the really big play. So, the ability to hit those intermediate throws consistently is critical for NFL QBs.
However, we only see R2 value of 0.038, so although it helps in our evaluation, it still is not a great predictor.
NFL EPA Per Play vs Accuracy % on Throws between 10-19 yards (in College) (QBs with at least 150 snaps since 2015)
The single best overall stat: Pressure-to-sack rate
PFF defines pressure-to-sack rate as the percentage of dropbacks where the QB was pressured that resulted in a sack.
Pressure-to-sack rate (overall) is a better predictor than TWP rate, BTT rate, and Accuracy. Sacks are killers on drive – sacks have an EPA per play of -1.87 (huge). 60% of drives have at least one first down – that plummets to just 20% after a sack (link).
So, it makes sense that QBs who can avoid sacks in college can translate that to the NFL. Nearly all elite QBs (>0.1 NFL EPA per play) have a pressure-to-sack rate <20% (only Jayden Daniels at 20.2% is the lone outlier). Essentially, if you want to find an elite NFL QB, you want a college QB with a low pressure-to-sack rate.
NFL EPA Per Play vs Pressure-to-Sack Rate (in College) (QBs with at least 150 snaps since 2015)
Breaking down pressure-to-sack rate further, there is an even better correlation on plays w/o a blitz. Pressure-to-sack rate on non-blitz plays had an R2 of 0.059 – the best correlation from a single statistic. This makes sense as defenses can design blitzes to get early pressure, but QBs who can avoid sacks when a defense is only rushing 4 is critical to success (a defense getting sacks by only rushing 4 is huge for defensive success).
Nearly all elite QBs are at <17.5%, except Justin Herbert (18.4%) and Dak Prescott (21.7%). So, you want QBs who have a low pressure-to-sack rate overall and especially on plays without a blitz.
NFL EPA Per Play vs Pressure-to-Sack Rate on Plays w/o a Blitz (in College) (QBs with at least 150 snaps since 2015)
In summary, we have seen good correlation to TWP rate on non-blitz plays, BTT rate on non-play action, and Accuracy on intermediate throws that can explain a significant portion of how college performance can translate to NFL QB performance. However, pressure-to-sack rate on non-blitz plays correlate the best.
But, it is still not good enough – even these college metrics put together do not accurately predict NFL QB performance, at least enough to convince an owner to invest millions of dollars.
So, can we create a model that can predict NFL QB performance using these college statistics? The answer is yes – and that’s the next post.