NFL Scouts Have No Clue How to Predict the Next Great Quarterback

It is widely believed that Johnny Manziel will be one of the first quarterbacks selected in the 2014 NFL draft. Mock drafts at ESPN.com, CBS Sports, and Sports Illustrated all have Manziel taken in the first round, with some having him selected with one of the first 10 picks. 

But not everyone agrees that Manziel is destined for pro-level greatness. NFL analyst Merle Hoge had this to say a couple of months ago:

He has absolutely no instinct or feel for pocket awareness. He has an instinct to run. That's a bad instinct if you're going to have that in the National Football League. You have to play in the pocket with traffic around you and throw it. When traffic comes around him, he runs, and that's dangerous in the National Football League. His skill set does not transition to the National Football League, and it is a big, big risk. In fact, I see bust written all over him, especially if he's drafted in the first round.

Many NFL analysts have raised similar questions about how Manziel will play in the NFL. And yet, few people think Manziel will not be selected in the 2014 first round. So who will prove to be right? Is the team that does call his name—presumably early on Thursday night—making the right choice? Or are the skeptics right to claim they would not select Manziel?

To fully understand that question, it’s important to recognize why an NFL team might think Manziel is a top quarterback. In 2009, Rob Simmons and I published a paper on the logic used in drafting NFL quarterbacks. Recently I updated this analysis, looking at the quarterbacks drafted from 1998 to 2013. The results this time—which were quite consistent with our earlier work—reveal some of the factors teams emphasize on the day of the draft.

Manziel's expected competition in the first round of this year’s draft includes Blake Bortles, Teddy Bridgewater, and Derek Carr. This past season, the aforementioned four quarterbacks posted the following ratings:

• Johnny Manziel: 172.9
• Teddy Bridgewater: 171.1
• Blake Bortles: 163.4
• Derek Carr: 156.3

(Their average in 2013 was around 133.0.)

Each of these marks ranked in the top 15 of college football in 2013. So all four were “good.” However, Manziel and Bridgewater were top-five quarterbacks, so one could argue that on the field, Manziel and Bridgewater were somewhat better than Bortles and Carr.

On-field performance, though, is not the only story. After the season is over, the NFL measures both the physical and mental skills of each athlete it might consider in the draft. At the NFL combine, the players are physically measured (height and weight), run the standard 40-yard dash (among other drills), and asked to take the Wonderlic test. The latter is a 50-question test designed to measure intelligence.

When we look at these four quarterbacks, we see the following measurements:

Of these four, Bortles is the biggest but also the slowest. Manziel—if we are to believe the Wonderlic score—is the smartest. But he is also the smallest.

So what does any of this mean? In terms of where a quarterback is drafted, both what we see on the field and at the combine matters (and it seems the latter matters more than the former). Specifically, according to my research, quarterbacks who are bigger, taller, faster, and smarter tend to hear their name first on draft day. And on this list, it appears taller and faster are the most important: If we look at a one standard deviation in each variable, height and speed have the largest impact on draft position. For each, a one standard deviation improvement moves you about one round up in the draft. That’s both bad and good news for Manziel.

Unlike the other three players listed above, though, Manziel won the Heisman Trophy (in 2012) and plays in one of the top college conferences—a list that includes the ACC, Big 10, Big 12, SEC, and Pac-12. Each of these factors are also statistically related to where a quarterback is drafted (though, as NFL Network analyst Daniel Jeremiah pointed out to Paul Glavic this week, team scouts claim they are generally less concerned with awards than fans think).

So we can see why Manziel might be considered a better choice relative to his primary competition. But will he be a top NFL quarterback, as so many NFL insiders believe? 

Consider the following list of names: Brandon Weeden, Blaine Gabbert, Tim Tebow, Mark Sanchez, Josh Freeman, JaMarcus Russell, Brady Quinn, Vince Young, Matt Leinart, Jason Campbell, and J.P. Losman. Each of these 11 quarterbacks were taken in the first round of the NFL draft in the past 10 years. And each quarterback has not only lost their starting job with the team that thought so highly of them on draft day, but these quarterbacks are not likely to be starting anywhere in the NFL in 2014.

In total, there have been 27 quarterbacks selected in the first round in the last 10 years. With each selection, the experts explained why each quarterback was going to be a star. But as we can see, often teams and the experts have, to borrow a term from baseball, swung and missed.

To understand why, let's ask a different question. Do the factors that predict where a quarterback is selected on draft day also predict NFL performance?

Before we answer that question, we must make an important observation about the impact of draft position on a quarterback's career: Quarterbacks who are taken first in the draft will spend more time on the field.For example, it took the Raiders three years—and 25 starts—before JaMarcus Russell was released. Quarterbacks taken later in the draft never get that many chances to succeed or fail.

To correct for the playing-time bias, then, we need to consider performance per play. And when we take that approach, it becomes clear that where a quarterback is selected in the draft doesn't tell us much about a quarterback's per-play performance in his NFL career. And this is what we see whether we look at measures like the NFL's quarterback rating, Wins Produced per play, or Adjusted Yards per Attempt.

Given that there’s no significant link between draft position and per-play performance, it shouldn't be a surprise that none of the factors that predict draft position can generally predict future NFL performance. And the same story emerges when we consider a variety of performance measures (as well as career performance after each of the first five years of a quarterback's career). Simply put, what seems to matter on draft day just doesn't tell us anything about what a quarterback will do when he is in an NFL uniform.

Again, one might think taller, faster, smarter quarterbacks who played well in college would be more likely to play well in the NFL. But that is not what we see in the data.

So does this mean none of this matters? Can teams just draft short, slow, stupid quarterbacks who couldn’t do anything in college?

Not exactly. Our data set only considers players who were drafted and played in the NFL. What we learn is not that this stuff doesn’t matter, but that in this population, being taller, faster, or posting better on-field numbers in college (relative to the other quarterbacks in the draft) doesn’t mean much.

Again, though, the difference we do see in this population affects the evaluation of talent. And that is where we think a mistake is being made. Looking at two college quarterbacks and rating the 6’5” quarterback over the 6’0” quarterback is probably a mistake. Or thinking the quarterback who runs a bit slower is worse. Or, finally, thinking the Wonderlic test—which doesn’t seem to include any football-related questions—is telling us anything about a quarterback’s ability on the field.

What about the on-field performance? Shouldn’t a quarterback who plays better in college play better in the NFL?

To understand why that might not be the case, consider our ability to predict the performance of veteran NFL quarterbacks. A few years ago, Brian Burke (from AdvancedFootballAnalytics.com) and I published a paper looking at the different ways performance in the NFL can be measured. One issue we considered was the correlation between a quarterback’s NFL performance in a particular season and what the same quarterback did in the NFL the previous season. The results indicated that predicting what a veteran quarterback will do from season to season is very difficult. For example, the correlation between a quarterback’s interceptions per attempt from season to season is 0.006 (or virtually zero). For touchdowns per attempt it was only 0.11. And for the NFL’s quarterback rating, it was only 0.15 (this was similar to what we saw for other metrics like Wins Produced per 100 plays, Expected Points per play, and Win Probability Added per play).

Why is performance so erratic? One suspects that the problem lies in how little a quarterback can do by himself. For a quarterback to succeed on any given play, linemen must block and receivers have to get open and catch the ball. That success also depends on the play being called. From season to season, quarterbacks have to play with a variety of teammates and with different coaches. And then there is the issue of injury, which seems to afflict virtually everyone who plays the sport.

All of these mean that predicting what any NFL quarterback will do in the future is very hard. Once we understand that point, we should not be surprised that predicting what a college quarterback will do in the NFL is amazingly difficult.

So although you’ll likely hear different draft analysts insist that they "know" Johnny Manziel is not going to be a "star" or another quarterback will be a difference-maker, our study of past drafts tells us that no one really knows. What we do know (or at least, we think we know) is why Manziel will find out on Thursday night which team he will make happy—or disappoint.