part 2.
http://www.espn.com/nfl/story/_/id/...ating-quarterbacks-means-2018-draft-prospects
History tells us the NFL is terrible at evaluating quarterbacks. Here's what it means for 2018
Jump to Part 1 on the history of the NFL's poor track record of evaluating quarterbacks.
Why is it so difficult?
All those examples aren't to suggest that NFL personnel executives are stupid and don't understand football, or that only a few select experts have any idea what's going on, both before and after quarterbacks have entered the pros. Longtime Ravens GM Ozzie Newsome traded up to grab
Kyle Boller. Bill Belichick, who was on the other end of that swap, famously went back and forth between
Tom Brady and Tim Rattay during the 2000 draft. Howie Roseman and the Eagles traded up to grab Wentz, but they also handed $12 million in guarantees to
Chase Daniel, who threw one pass in an Eagles uniform before being released and replaced by
Nick Foles. Even for the best, this is tough.
The simple response is usually just to peg players who underperform expectations as "busts" and assume that they would have failed regardless of where they ended up. That's a naive way to evaluate draft picks. Think about
Jared Goff, who looked entirely lost during his debut season under Jeff Fisher and offensive coordinator Rob Boras. What if the Rams had passed up the opportunity to hire Sean McVay and held on to Fisher for another interminable season? Would Goff have improved? If he had been truly dismal for a second consecutive campaign, would the Rams have given up on him and gone after yet another quarterback this offseason? Goff very well could have been written off as a bust.
There are more realistic reasons why coaches and executives end up making decisions that seem curious with the benefit of hindsight. Let's understand why:
The guys who drafted the QBs don't get to stick around for long. The coaches and general managers who fall in love with a quarterback don't always get to stick around for long to actually mold their would-be star. In Goff's case, that was clearly a good thing, but other new coaches don't click with their inherited starter.
Matt Leinart's career went south after Dennis Green was replaced by Ken Whisenhunt in the desert.
This is increasingly becoming a problem. Take a look at every top-10 passer over the past 20 years and their first four professional seasons, which form the bulk of their rookie deal. Of the 20 quarterbacks drafted under the old CBA, 10 played under the coach who drafted them for the first four years of their career.
Thirteen quarterbacks have been drafted since the league went to a new rookie scale in 2011, and of those 13, just four --
Cam Newton,
Andrew Luck,
Carson Wentz and
Patrick Mahomes -- have held on to their coach for the first four years of their careers. Even that number is generous, given that Luck lost interim coach Bruce Arians after one season once Chuck Pagano returned, while Wentz is only two years into his career and Mahomes just finished his debut campaign.
Highly drafted quarterbacks don't get to enjoy much stability.
Mitchell Trubisky is already on his second coach after John Fox was fired. The same is true for Goff.
Jameis Winston is on his second after Dirk Koetter was promoted from an offensive coordinator's role.
Marcus Mariota is about to start his fourth season with his third coach; the coach who drafted him, Whisenhunt, was fired after seven games.
The numbers are even more notable when you look at offensive coordinators. Not a single quarterback taken in the top 10 since
Matthew Stafford in 2009 has managed to make it four years with the same offensive coordinator at the helm. Each of the 15 top-10 passers drafted since -- a group that includes Wentz and Mahomes -- has been forced to work with at least two offensive coordinators during their first four pro seasons.
Translating between schemes is difficult. NFL teams have adopted some plays from the more wide-open college game. Heck, the Eagles ran a concept that descends from the Air Raid --
Mesh -- for a number of their big plays in the Super Bowl. For various reasons both practical (the wider college hashmarks) and obstinate (ask Titans fans about the "
exotic smashmouth"), teams are often stuck interpreting college tape of passers into their professional scheme.
NFL offenses do look more like modern college offenses than ever before, given how frequently they throw the ball and how often they line up in the shotgun. Teams that run more traditional attacks have to account for a quarterback's ability to line up under center when they typically took snaps out of the Pistol or shotgun. Successful NFL passers make their throw after one read more frequently than some might suggest, but coaches who will ask their new starter to make deeper progressions also have to project what they think their passer might be able to do versus seeing him do it on a drive-by-drive basis in college.
Players who are drafted for their tools don't necessarily end up in schemes that play to their strengths. Every NFL team covets a quarterback who can "make all the throws." Offenses don't throw downfield as much as they used to -- the average pass traveled 8.0 yards in the air in 2017, down from 8.3 yards per throw a decade ago -- but when you are in third-and-long or driving for a score late in the fourth quarter, you want a quarterback capable of planting his foot and driving an out route 24 yards downfield to the sideline.
The problem is that teams don't necessarily end up using their quarterbacks in that way. The two starters in the league who were drafted more for their arm strength than anything else are Stafford and
Joe Flacco, who have absolute howitzers on their shoulders. They also, curiously, work within two of the most conservative schemes in the league.
Over the past two years with Jim Bob Cooter, Stafford has averaged 7.6 air yards per throw, which is 24th in the league. Flacco is even further down the stat sheet; at 6.8 yards per pass, the only quarterback who has thrown his typical pass shorter in the air is
Sam Bradford. The most famous comeback of the 2017 season came on a deep out thrown by
Case Keenum -- who wasn't drafted in part because of a lack of arm strength -- to
Stefon Diggs in the playoffs.
Analytics haven't helped all that much (yet). In 2008, college student David Lewin looked back at college quarterbacks from the past
and found that completion percentage and games started were the only two meaningful predictors of professional success, and that only worked for quarterbacks selected in the top two rounds of the draft. (Lewin now works for the Boston Celtics as their director of scouting.)
The idea had a couple of fitting issues -- the data set hadn't been backtested, and the endpoint of the top two rounds was arbitrary -- but the concepts were insightful. Quarterbacks who struggled with their accuracy in college rarely developed it at the professional level, and as passers spent more time in college, scouts gained a better sense of their actual skills and were able to provide a more meaningful evaluation of their talents.
Since then, the results have been mixed. Passers with sub-60 percent completion percentages like Stafford and
Matt Ryan have found success at the pro level, while hyper-accurate darlings like
Brian Brohm and
Robert Griffin III haven't been able to parlay college completions into regular NFL work. The disparity in schemes between schools naturally makes it easier to complete passes in some places as opposed to others, and a more robust version of the Lewin model would incorporate air yard data, which is beginning to become available to quantitatively inclined organizations. Teams have done their own internal studies to measure which elements of college success actually do translate to professional staying power, but that data might not be convincing enough to get a traditional owner, general manager or coach to listen.
Furthermore, "analytics" is an overly broad catch-all, because one team's research may be totally different from another's. The Browns were the league's most numbers-friendly team under the stewardship of Sashi Brown, but Cleveland famously didn't believe that Wentz would turn into a franchise quarterback. (It's worth nothing that, contrary to what might be said today,
their coach also felt like they should have passed on Wentz with the second pick.)
The Eagles felt differently and traded up to grab Wentz, and while it's tempting to contrast their subsequent success to that of the Browns, Philadelphia has
been influenced by analytics for more than a decade, going back to the days of Joe Banner and Andy Reid. Both teams were using a mix of analytics and traditional scouting to evaluate Wentz, and in the end, the Eagles appear to have been right.
Unqualified evaluators can use pseudo-psychology to talk themselves into anything. Statistics and tape aren't enough to scout players, as teams rightfully want to know a quarterback might fit in their locker room and how he'll command a huddle. Off-field problems have sunk passers like Ryan Leaf, Manziel and Todd Marinovich. Organizations have to do their due diligence.
The problem, though, is that they often take a leap past those evaluations and adopt rules or administer tests that can't realistically prove anything. The Giants
were famous for administering a 200-page psychological exam to potential draftees, at which Deion Sanders summarily laughed and walked away.
This year, one NFL organization
asked LSU running back Derrick Guice if he was gay, marking at least the third time in recent memory a team has reportedly asked a player this irrelevant (and
possibly illegal) question. Another asked wide receiver prospect Braxton Berrios
to explain both ISIS and Bitcoin to them; given the age of NFL executives, you might expect they were looking for a primer on the latter. You can infer just about whatever you want, positive or negative, from asking Berrios those questions.
Another popular theory is the marriage test, as espoused by Texans owner Bob McNair when Houston drafted
David Carr as opposed to Joey Harrington with the opening pick of the 2002 draft. "David has been married for three years," McNair said at the time. "He's got a wife, Melody, and a 23-month-old son, Austin. He's already settled down. He can handle success."
Carr might very well have been settled down, but it wasn't enough for him to succeed behind a dismal offensive line in Houston. Plenty of quarterbacks weren't married at the time they were drafted. Brady was years away from dating -- let alone marrying -- supermodel Gisele Bündchen.
Aaron Rodgers remains unmarried. Troy Aikman didn't get married until his final year in the NFL.
Teams aren't much smarter about this stuff now. It might not be the single determining factor in deciding who a team selects, but there is plenty of useless information being strewn around this year. Hue Jackson is
impressed that Baker Mayfield got his teammates to chant "Hee hee." Giants owner Steve Tisch
was blown away by Josh Rosen's table manners. Two years ago, the
Jets were impressed that Christian Hackenberg could make easy conversation about the Masters at a local bar.
Others want to believe they can see whether a quarterback will succeed or not by looking into their eyes or figuring out whether a passer has the "it" factor. I wrote about the
"it" factor fallacy back in 2014, noting how quarterbacks like
Mark Sanchez and
Vince Young had been pegged with the moniker early in their careers, only for "it" to disappear when they were exposed.
Everyone from
Jimmy Clausen to
Tim Tebow has been regarded as someone with the "it" factor at one point or another during their careers, but "it" hasn't been remotely predicative of pro success. Two of the quarterbacks who we would say most clearly have "it" are Brady and
Russell Wilson, who fell past the first two rounds of the draft when everyone who swears they can spot "it" should have seen "it" from a mile away. Intangibles clearly exist, but we're not very good at identifying who possesses "it" until well after the fact.
Underclassmen quarterbacks get to decide when we stop seeing them, which is often at a high point. While there are exceptions, many of the quarterbacks who are taken in the first round of the draft are underclassmen who declare for the draft with eligibility still remaining. Over the past four drafts, 10 of the 11 quarterbacks who were chosen in the first round left school before exhausting their eligibility, with Wentz -- who went to FCS North Dakota State -- as the lone exception.
By definition, quarterbacks who declare for the draft are almost always going to be leaving after what they consider to be a career season. Few quarterbacks are going to forgo their remaining eligibility if they feel like they aren't going to be a first-round pick or didn't play well in their final collegiate season. They have the ability to cut off the information spout when they've made a great impression as opposed to quarterbacks who play out their amateur eligibility and don't live up to expectations during their final year. Leinart is a good example of the latter.
Take Trubisky, who declared for the pros after his junior season, which was his only season as a starter. It's hardly out of the question that he could have kept up his excellent play and continued to perform at a high level, which would have kept him in the top five of the draft. There's also a chance that he would have suffered without
Mack Hollins and
Ryan Switzer, thrown a few more picks, and fallen into the second round as a project.
It's difficult to imagine a year we didn't actually see from Trubisky, but young quarterbacks see their stock fluctuate all the time. Think about the differences in how we perceive
Derek Carr and
Dak Prescott between 2016 and 2017. Quarterbacks who play well enough to leave early get to be judged at their best tape as opposed to simply their most recent tape.
We're not really good at drafting in general, let alone quarterbacks. As many folks have documented, including
Neil Paine for 538, no organization has proven to be better than another at actually picking players. The best teams in the league have succeeded by amassing extra picks via compensatory selections and by trading down, which has worked for the
Patriots under Bill Belichick.
The Steelers famously had what was widely regarded as the best draft in league history in 1974, when they
drafted four future Hall of Famers, but their drafts in 1973 and 1975 saw them select 41 players who combined to make two trips to the Pro Bowl. The next Hall of Famer Pittsburgh drafted was Rod Woodson in 1987, 13 years later. Study after study tells us that after you account for the expected return from a given position in the draft, no executive consistently outscores the opposition.
Organizations get impatient. Teams that say one thing during the pre-draft process don't actually follow through on their development plans. The Jaguars wanted to give
Blake Bortles a redshirt year in 2014 to rebuild his mechanics, but after a solid preseason and three mediocre games from backup
Chad Henne, an 0-3 Jags team named Bortles as their starter in Week 4. Bortles' mechanics never took and eventually collapsed altogether in 2016.
The Eagles, meanwhile,
intended to sit Wentz behind Bradford in 2016, but when Philadelphia got a great trade offer from the Vikings for Bradford, they inserted Wentz into the lineup a year earlier than expected. Things obviously went fine for the second overall pick one year later, but teams generally push their new quarterbacks into the lineup earlier than they suggest in April.
In some cases, this might not be the decision of the people who seem to be involved. Ownership might encourage -- or insist -- that a team draft a quarterback or insert their new prospect into the lineup. A coach who is in danger of losing his job might prematurely bench a struggling young quarterback in the hopes of finding a higher floor with an experienced veteran, representing the moral hazard of when a team's incentives to win aren't in line with that of their head coach.
What does that mean for the 2018 draft class?
For one, anyone who says they're sure that any of these prospects is going to turn into a star or a mistake is kidding themselves. It's pretty clear that much of what makes a quarterback is what happens after he arrives in the NFL. How is his team at developing draft picks? What sort of infrastructure does he have to succeed? How much will his coach, offensive coordinator and quarterbacks coach do to ease the transition to the professional level? Educated eyes can certainly pick up elements of a quarterback's game he'll need to improve or rely upon at the next level, and history may suggest some members of the quarterback pool are more likely to succeed than others, but nobody's a surefire hit or a guaranteed disaster.
There are question marks about each of these prospects, such as the pre-draft debate on ESPN comparing Wyoming quarterback
Josh Allen and Louisville passer
Lamar Jackson. Bill Polian, the former GM of the Colts who now works as an NFL analyst for ESPN, suggested earlier this year that Jackson should consider a move to wide receiver owing to his athleticism and size, but Jackson measured in at the NFL combine at 6-foot-2 and 216 pounds, which is nearly identical to the listed height and weight of
Andy Dalton. Dalton, who was a surprisingly influential
running quarterback in terms of the the zone-read during his time at TCU, does not appear to have been asked to switch positions before or during his pro career.
Allen, meanwhile, is a quarterback prospect straight out of the 1970s. The 21-year-old has ideal height (6-5) and a cannon for an arm, but he completed just 56.2 percent of his throws at Wyoming, the lowest completion percentage for a would-be first-rounder since Locker (54.0 percent). ESPN draft guru Mel Kiper Jr. has suggested Allen played in an offense that frequently threw the ball down the field, deflating his completion percentage, with Jackson playing in a scheme in which it was easier to find safe completions.
In talking to NFL coaches and executives at the combine, there are certainly plenty of people within the league who share Kiper's sentiments about the Louisville and Wyoming offenses. It's also impossible to break down every single one of the playcalls for each offense to figure out whether Allen turned down easy completions to take big shots downfield. Remember the air yards statistic I mentioned earlier, though? We have that data for Allen Jackson, and the rest of the quarterbacks in this year's draft, thanks to ESPN Stats & Information research.
It's true that Allen didn't throw many short passes relative to your typical college quarterback, but Jackson actually threw fewer "layups" than Allen when you account for the fact that he has an extra year on tape. Here are the numbers for each of the five passers, split by the distance over which the ball traveled in the air. Note that these numbers don't include the six passes Allen threw in 2015 or
Baker Mayfield's stats from his lone season at Texas Tech in 2013:
Josh Rosen threw more screens and quick hitches than any of the other candidates, as more than 40 percent of his passes were thrown within 4 yards of the line of scrimmage. Jackson didn't throw a ton of screens, but 42.8 percent of his passes were in the midrange between 5 and 14 yards, which was comfortably the most of any of these five signal-callers. And then, it's certainly true that Allen was chucking it downfield more than anybody else. Nearly a full one-third of his passes, 33.2 percent, traveled 15 or more yards downfield. Rosen, who was playing in a version of the West Coast offense under the tutelage of former NFL coach Jim Mora Jr., threw downfield on just 22.9 percent of his passes.
Let's now consider how accurate each quarterback was in each respective zone. Here is each quarterback's completion percentage, split out by those same air yard figures:
Allen and Jackson, ironically, have roughly similar profiles. They were both well below the NCAA average for completion percentages on extremely short throws (73.3 percent), but were better than the typical passer on throws 15-plus yards downfield.
It's difficult to not notice Mayfield, given that the Heisman Trophy winner was significantly ahead of the pack in most categories. It's staggering that Mayfield's completion percentage on throws traveling 20-plus yards in the air (51.8 percent) wasn't all that far behind that of Allen on throws at or behind the line of scrimmage (58.7 percent). Mayfield's passer rating on those throws was 297.6, and he posted a 99.7 QBR on those throws. Allen was second-best in those same categories among the five would-be first-rounders, but his passer rating was a mere 185.8, with a 90.8 QBR mixed in.
One more chart. We can use NCAA-average completion rates and the rate at which each passer threw each particular type of pass to project what his completion percentage "should" have been, given those range of throws.
Jackson actually underperformed his completion percentage by the largest amount, although it's not materially different from Allen and doesn't lend itself to the layups argument. Rosen was essentially about where we would expect given his range of passes.
Sam Darnold was well ahead of expectation, but Mayfield was in another universe.
This isn't the whole story. A deeper analysis would need to account for the types of specific throws each passer was asked to make and the game situation. QBR does exactly that and gives Mayfield an 89.0 mark over the last three years, with Jackson at 82.3, Darnold at 80.2, Allen at 66.6, and Rosen holding up the rear at 65.4. QBR doesn't know the playcall and can only say much, but it's a further reflection of how these quarterbacks actually impacted both drive and win expectancy.
Jackson ranks second despite decent passing numbers because he was simply dominant as a scrambler and runner. Over his three seasons at Louisville, Jackson ran for 4,132 yards and 50 touchdowns. Forget quarterbacks; Jackson would be third in the country in rushing yards and has more rushing touchdowns than any
running back in college football over the last three seasons. He had 171 runs of 10 yards or more, 33 more than any other player over that timeframe.
To put his rushing dominance in context, QBR suggests Jackson generated 180.4 points for Louisville over the last three seasons as a runner. The other four quarterback hopefuls in this year's draft generated 113.8 points
combined with their feet over that same timeframe. Jackson is good, but not great as a passer -- strip out the runs and his QBR on throwing plays is 73.8, which is just ahead of Allen (73.7) and only good for 26th among passers with 600 attempts or more over the last three seasons.
If a team drafts Jackson to play quarterback and discourages him from running the football, he might disappoint for reasons entirely out of his control. Likewise, though, if a team drafts Allen and tells him to never throw the ball any farther than 10 yards downfield, they're not going to be happy with the results. Draft Mayfield, line him up under center, and tell him to never leave the pocket? You'll end up playing to your quarterback's weaknesses as opposed to his strengths.
It sounds downright stupid for a team to do that, but organizations make these sorts of mistakes all the time. Mora inherited
Michael Vick and tried to turn him into a West Coast quarterback. Mariota went from spending all of his time in the shotgun at Oregon to an offense that took 45.1 percent of its snaps under center over the last two years under Mike Mularkey, the 10th-highest rate in football. Imagine if Wilson had been drafted by a team that insisted he develop as a quarterback under center as opposed to the Seahawks, whose obsession with competition led Pete Carroll to award Wilson the job over free-agent signing
Matt Flynn before the season began.
History tells us not to expect each of these five passers to turn into stars. Just seven draft classes since the AFL-NFL merger of 1970 have produced four or more first-round quarterbacks, and their track record is mixed at best. Of the 31 passers who were taken across those seven classes, five -- Elway, Kelly and Marino from 1983 and both
Eli Manning and
Ben Roethlisberger from 2004 -- are likely to make the Hall of Fame. The jury's still out on Luck and Newton, the first overall picks from the 2011 and 2012 drafts.
Thirteen of the 31 passers failed to produce more than one season of above-average football as a starter. That's 42 percent of the quarterbacks chosen, and it's a group that includes passers selected with high picks. Quarterbacks went 1-2-3 in the 1999 draft, and while second overall selection
Donovan McNabb worked out well for the Eagles, the guys chosen first (
Tim Couch) and third (Akili Smith) were disappointments. They combined to make zero Pro Bowls, while the guys chosen with the next five picks -- Edgerrin James,
Ricky Williams,
Torry Holt,
Champ Bailey and David Boston -- combined for 25 trips to Hawaii.
There's significant risk in taking a quarterback in the first round, but the potential reward is enormous. The Eagles had Wentz signed for 2017 at a cap hit of just over $6.0 million, peanuts in a league in which the top young quarterbacks were signing extensions at $25 million per season. Wentz's cap hit won't rise above $8.5 million until 2020, creating a savings that has allowed the Eagles to squeeze in deals for the likes of
Alshon Jeffery and
Tim Jernigan. The Seahawks were able to add veterans such as
Cliff Avril and
Michael Bennett to an already-stacked team because Wilson's four-year rookie deal cost only a combined $3 million.
Under the new CBA, teams are heavily incentivized to draft and develop rookie quarterbacks into useful contributors. To say that drafting and developing passers is an inexact science, though, would be an understatement. Instead of focusing on the players and what we can gauge about their possible success before they get drafted, it seems way more appropriate to evaluate the infrastructure surrounding those quarterback opportunities, since they may be a far better indicator of whether each passer is likely to develop into the franchise icon any first-round pick is expected to become.
talking about Case Keenum, the old trope about how there aren't 32 good starting quarterbacks to go around is naive. The problem is that there aren't 32 good quarterback
situations to play within, a reality that was reinforced by the success of Keenum and Foles after they looked disastrously bad under Jeff Fisher with the Rams. Teams that are spending every waking hour worrying about picking the right guy are likely kidding themselves to think that they can reliably pick the best quarterback from this group. They should worry about making themselves the right fit for whoever they end up choosing.
