NFL: Week 12 APR vs. ESPN

What this is

A comparison of APR's Week 12 power rankings with ESPN's rankings for the same week of games. This acts as a sanity check (both ways) to see what teams are over- or underrated.

Only teams with the widest differential are listed. The number in parentheses is the difference between APR's rank and ESPN's rank.

Teams ESPN likes more than APR:

On the bubble

Another week, another close win for the Falcons (+6).

ESPN somehow gives the Colts (+6) a 5-spot bump on their less-than-dominating win over the 4-7 Bills.

The Ravens (+6) have amassed a 9-2 record, but it's really not a good sign they played behind the 4-7 Chargers most of the game, and needed to convert 4th and 29 just to get to overtime.

Teams APR likes more than ESPN:

Panthers (-7)

The Panthers finally found another team they could beat. And some of their early-season based power is finally getting to the fall-off part of APR's weighting curve.

On the bubble

Nobody closer than the Broncos (-5).

NFL Wrapup: APR Pick Reliability

APR: Home vs. Visitor

Note that APR had the Week 2 Jaguars @ Jets game as a push, and so that game is not included in these totals.

	Visitor Won	Home Won	Total
Picked Visitor	65	48	113
Picked Home	40	86	126
Total	105	134	239

APR's record picking the home team is a respectable 86-40 (68.2%). Picking the visiting team, APR is 65-48 (57.5%).

Matchup Ratio Range

Here's a plot of all the matchup ratios (the ρ value in the game picks), separated by week. You can see that most ρ values are under 1.55 or so, with the few outliers coming in the first few weeks. The two high-ratio picks in the second half of the season are for Colts @ Patriots (week 13), and Colts @ Ravens (week 14).

Here's a plot of just the games where APR picked the visiting team. Each green 'x' correspond to the ρ value of a correctly picked game, and each red '+' corresponds to the ρ value of an incorrectly picked game.

There's no nice correspondence between wrong picks and a low ρ value. Of the incorrectly picked visiting teams, 26 of 48 (54.1%) have a ρ value higher than 1.1.

Wrong picks are a bit more frequent early on—more than half (25 of 48) came in weeks 2 to 8.

Future Changes

During the offseason, I plan to run some simulations to see if I can find a more accurate configuration for APR.

Thanks again for reading. If you're interested, I'll have some posts during the off season using the historical data. If not, I plan to be back as usual next September.

NFL Wrapup: APR Picks by Team

APR's pick results by team, for weeks 2-17 of the regular season. You may remember that the Jets and Jaguars had equal power in the week 1 power rankings, and so APR didn't have a pick for their week 2 matchup. This is recorded as a tie for W-L-T purposes.

Record	Team	Comments
14-1	Packers	APR never picked the Packers to lose
13-2	Colts, Rams	Consistently bad
12-3	Browns, Vikings, Buccaneers
11-4	Falcons, Panthers, Bengals, Texans, Patriots, Saints
10-4-1	Jaguars
10-5	Lions, Dolphins, Steelers	16 teams at or above 10-5
9-6	Ravens, Bears, 49ers, Titans	16 teams at or below 9-6
8-6-1	Jets
8-7	Bills, Cowboys, Broncos, Chiefs, Chargers, Redskins
7-8	Cardinals	Not too surprising, given how many of their games came down to the final play.
6-9	Raiders	A strong start, but too many injuries, along with no depth at quarterback.
5-10	Giants, Eagles, Seahawks	How do you figure a team that got swept by the Redskins, and won the Super Bowl?

NFL Wrapup: APR Pick Reliability

APR: Home vs. Visitor

Here's the breakdown of pick accuracy for picking the visiting or home teams:

	Visitor Won	Home Won	Total
Picked Visitor	67	63	130
Picked Home	42	68	110
Total	109	131	240

So, the good news is, when APR picked the home team, it was right a reasonably good 61.8% of the time (though still short of the Line's 65.0% record). The bad news is, when APR picked the road team, it was little better than a coin-flip.

APR also picked the home team less than half the time

Picks by Matchup Ratio

The matchup ratio (ρ) for each game is the quotient of the corresponding power indexes for each team (larger divided by the smaller). It is intended to be an indication of "pick confidence": ρ values near 1.0 involve teams judged to be very close in power, while large values indicate more of a mismatch.

Confidence Range			Picked Right	Picked Wrong	%
	ρ	<1.02	8	9	47.1%
1.02<=	ρ	<1.04	9	12	42.9%
1.04<=	ρ	<1.06	14	8	63.6%
1.06<=	ρ	<1.08	9	14	39.1%
1.08<=	ρ	<1.11	14	13	51.9%
1.11<=	ρ	<1.15	27	16	61.8%
1.15<=	ρ	<1.20	16	9	64.0%
1.20<=	ρ	<1.30	19	13	59.3%
1.30<=	ρ		19	11	63.3%

There's a pretty clear cut-off right around ρ ≥ 1.11 where the confidence value really seems to work well; pick results are at or above a 60% success rate.

The bad news is, particularly for ρ < 1.08, the confidence value was actually a negative indicator of which team would win, at least this year.

I need to run the pick results for more seasons to find out how much of this is how APR works, and how much it was a quirk of this season. But that's going to have to wait for another time if I want to get this posted while it's still February...

NFL Wrapup: APR Picks by team

Record	Teams	Comments
14-1	Rams	The Rams were not picked to beat the Lions in week 8.
13-2	Chargers
12-3	Lions, Colts, Buccaneers, Bears	The Bears were a pretty bad team this year, especially for making it to 7 wins. APR had them ranked in the bottom 10 from week 9 to the end of the season.
11-4	Vikings	APR never picked against the Vikings.
10-5	Seahawks, Saints, Redskins, Patriots, Packers, Giants, Eagles, Chiefs, Bengals	I'm surprised APR did this well on the Patriots considering the gap between their home record (8-0) and road record (2-6).
9-6	Titans, Ravens, Falcons, Cowboys, Cardinals, Browns, Bills	16 teams picked 9-6 or worse; same as last year.
8-7	Raiders, Panthers, Jets, Broncos, 49ers	The 49ers were another team with a big difference between their home record (6-2) and road record (2-6)
7-8	Texans, Jaguars, Dolphins	How much does a crowd matter to home field advantage? In spite of their often empty stands, the Jaguars were another team did a lot better at home (5-3) than on the road (2-6).
4-11	Steelers	How do you pick a team like this? Four of their wins came against playoff teams (Chargers, Vikings, Packers, and Ravens), while three of their losses came against some of the worst teams in the league (Chiefs, Browns, and Raiders).

Update: I should make clear that this is for regular season games only (weeks 2-17).

NFL Wrapup: The Top 10 Upsets, as Determined by APR

Determining Upset Magnatude

Originally, I was going to measure upset magnatude using the match-up ratio (the "ρ" value) originally published for the game. But after some (unposted) analysis, it seems clear that the match-up ratios are not comparable from week-to-week. They start out very high at the start of the season, and decline as it goes on.

So the first modification I made was to use the APR power index values from the second-to-last week of the season (I avoided the last week to rule out anomolous effects of teams resting their starters). However, the normal APR algorithm gives less weight to early season games.

Rather than try to justify why the week 2 Chargers @ Broncos game was one of the top-10 upsets of the season, I decided to use the unweighted version of APR power indexes, which gives all games played equal weight.

Determining the Top 10 Upsets

Using the unweighted APR power indexes for week 17, a list of all games where the predicted winner lost was compiled. This list was then sorted according to each game's match-up ratio (ρ).

Any loss that was (in my opinion) attributable to teams not playing all-out was eliminated from the list. The top 10 games from this list is presented below.

The Top 10 Upsets of the 2008 Season

As determined by the unweighted APR power indexes from week 16.

10. Week 5: Buccaneers 13, Broncos 16 (ρ=1.1168)
    From week 3 to week 15, APR had the Buccaneers ranked in the top 10. Maybe they just weren't ready to play at Mile High, but somehow the Buccaneers let the Broncos give them their second loss.

9. Week 16: Jets 3, Seahawks 13 (ρ=1.1312)
   Because the Jets couldn't put up two touchdowns on the Seahawks, they went from "win and you're in" to "win (and with help) you're in". The win and the help never came.

8. Week 3: Jaguars 23, Colts 21 (ρ=1.1383)
   The Jaguars won just 5 games this year, and this was one of them. It's hard to argue against Peyton Manning as MVP when a key loss like this is largely attributable to Manning's early season health problems.

7. Week 7: Jets 13, Raiders 16 (ρ=1.1385)
   Because the Jets couldn't put up two touchdowns on the Raiders... yeah, there seems to be a theme here. You might argue whether Jets @ Seahawks or Jets @ Raiders is the bigger upset, but they're both pretty bad.

6. Week 11: Broncos 24, Falcons 20 (ρ=1.1398)
   The Falcons had a chance at the #2 seed going into week 17. If they had won this game (their only home loss of the season), they would have had a much better chance.

5. Week 12: Jets 34, Titans 13 (ρ=1.1398)
   At the time, this didn't seem like that much of an upset. But that was (right) before the Jets went on a 1-4 skid, and finished out of the playoffs.

4. Week 17: Raiders 31, Buccaneers 24 (ρ=1.2029)
   Coming in to week 17, the Buccaneers still had a chance for the playoffs (and would've been in, had they won). All they had to do was beat the 4-11 Raiders. But there was no defense, no win, and no playoffs.

3. Week 6: Rams 19, Redskins 17 (ρ=1.2192)
   It says something about how bad the Rams were this year that this game is on this list in spite of the Redskins late-season troubles.

2. Week 6: Giants 14, Browns 35 (ρ=1.2483)
   Easily the low-point to the Giants season. Given the way the offense was struggling (Eli Manning threw 3 interceptions), the defense would've needed a masterful performance to keep them in the game. Instead, the Browns scored at will for much of the game.

1. Week 7: Cowboys 14, Rams 34 (ρ=1.3023)
   Brad Johnson will get his fair share of blame for this loss (the Cowboys didn't have a second drive for more than 21 yards until garbage time). But the defense was at fault, too: the Rams scored on their first 3 drives, and didn't punt for the second time until the 3rd quarter.

   If the Cowboys had won this game, they wouldn't have needed a win over the Eagles in week 17 to get in the playoffs.

Eliminated Games

The only game eliminated from the above list is the Titan's week 15 loss to the Texans (which would have been the #5 upset). At that point in the season, the Titans had already clinched the AFC South and a bye week, and only needed to beat the Steelers in week 16 to clinch the #1 seed in the playoffs. This was a meaningless game for the Titans.

NFL: APR vs. the Eagles and Broncos

If you've been following along at home, you may recall that I pointed out in the week 11 power rankings that APR seems to really like the Eagles, in spite of a pretty ordinary record, while the Broncos get no respect, in spite of (mostly) maintaining a winning record. The following is a closer look at why APR ranks the Eagles and Broncos the way it does.

As you know if you've read my description of how APR works, each team's power index is computed from power values computed for each game that team has played. Here is a table detailing the game powers for each game played by the Eagles and Broncos, for the week 12 power rankings:

Philadelphia Eagles		Denver Broncos
won vs Rams by 35	1.339	won @ Raiders by 27	1.695
lost @ Cowboys by 4	1.232	won vs Chargers by 1	1.359
won vs Steelers by 9	1.637	won vs Saints by 2	1.399
lost @ Bears by 4	1.231	lost @ Chiefs by 14	0.807
lost vs Redskins by 6	1.045	won vs Buccaneers by 3	1.476
won @ 49ers by 14	1.559	lost vs Jaguars by 7	0.894
won vs Falcons by 13	1.621	lost @ Patriots by 34	0.996
won @ Seahawks by 19	1.585	lost vs Dolphins by 9	0.943
lost vs Giants by 5	1.295	won @ Browns by 4	1.421
tied @ Bengals	1.122	won @ Falcons by 4	1.602
lost @ Ravens by 29	1.077	lost vs Raiders by 21	0.783

Looking at this table, one thing stands out: the Eagles win big and (except for the game at the Ravens) lose close; the Broncos lose big and win close.

The Broncos' losses have really hurt them. They've got 5 games with power values under 1.0, all relatively recent. On the other hand, the Eagles have 4 very strong game powers, 3 of them within the last 6 games (and, of course, the big win over the Cardinals is not included in this table, since week 13 is not complete at this writing).

This is exactly the kind of thing that APR was designed to emphasise. The Broncos close wins and blow-out losses are a pretty good indicator that with just a few more bad breaks, they could have a much worse record (the week 2 game against the Chargers comes immediately to mind).

The Eagles are on the other side of this argument. A successful goal-to-go push against the Bears, a 3rd-and-1 conversion against the Giants, One more made field goal against the Bengals... it's not hard to imagine an alternate series of events where the Eagles are 8-3 going into week 13, and right in the thick of the playoff hunt.

But—part of the problem with the Eagles is that this "almost made it" syndrome seems to be an on-going problem. Last year, in a game decided by 3 points, they muffed a couple punts, which the Packers turned into 10 points. In a game against the Bears, they lost a crucial fumble recovery on a play that (on a rule no one ever heard of before) got called a false start. A game against the Giants was lost because a last-second field goal hit the upright.

A few close games lost can be blamed on bad breaks. When the streak starts to stretch over two seasons, it's not hard to use this as the basis to make the case that the Eagles have some serious coaching issues...

More info on APR

I want to expand and emphasize a statement I made in my earlier post on how APR works. I wrote:

a simple sort of teams from largest to smallest power index is used to generate a power rankings

This is important. Each team's position in the power ranking is based only on the relative values of their respective power index values generated by APR. In particular:

• No consideration is given to the ranking a team had in the previous week. It is possible (and in fact, not unusual) for teams to fall even though they won, or rise even though they lost.

• No consideration is given to win-loss records, and it's very possible for, say, a 4-2 team to be ranked below a 1-5 team.

An example

I know it's possible for a 4-2 team to be ranked below a 1-5 team, because it happened during the 2007 season.

After the week 6 games of the 2007 season, the Baltimore Ravens were 4-2. The ESPN power rankings had them in the #7 spot. But APR ranked them #25. Below the 1-5 Falcons, the 1-4 Bengals, and the 1-4 Saints. In all, APR had 12 teams with .500 or worse records ranked above the Ravens that week.

Why should a team with a winning record be ranked so low? As usual with APR, the Ravens had played a some weak teams, including losses to the Browns and Bengals, and a 2-point victory over a very weak 49ers squad.

APR is meant to be a predictive measure of power (e.g. how well teams perform the following week). In this case, the low ranking was clearly justified, as the Ravens lost their next 9 games (including giving the Dolphins their one win of the season). The only game the Ravens won after week 6 was in week 17, against a Steelers team playing Charlie Batch at QB.

Conclusion:

This is exactly the kind of thing APR is meant to show: the power of each team as it continues to play games in the season. If you read this description of the SRS ranking algorithm, the discussion makes a distinction between predictive systems (which team is more likely to win their next game) and retrodictive systems (which team accomplished more in the past). APR (like SRS) is a predictive system, and should be approached as such.

Algorithms: How APR works, and what it's trying to do

For those of you coming here from footballoutsiders.com, let me say right away that APR is not supposed to be a replacement or alternative to DVOA, DPAR, or any of their other metrics.

The APR algorithm:

1. Initially, every team is assigned a power index of 1.0

2. For each team, a power index detailed below) is assigned to it for each game played.

3. For each team, the game power values for that team are then combined together using a weighted average (old games weighted less than new games) to give a new power index for the team.

4. Steps 2 and 3 are repeated a number of times to create a feedback loop.

The power for a played game is computed according to the following formula:

gamePower = resultPower × marginPower × opponentPower

Where:

• resultPower is a constant based on whether the team won or lost, and whether they were the visiting or home team, with the following constraints:

  road win > home win > tie > road loss > home loss

• marginPower is a constant based on the margin of victory (or loss). Right now, margin power is assigned in the following ranges:

  • won by 14 or more
  • won by 7 or more
  • won (or lost) by 6 or fewer points
  • lost by 7 or more
  • lost by 14 or more

• opponentPower is the power index of the opponent team.

Using the power index values

Once power index values have been generated, a simple sort of teams from largest to smallest power index is used to generate a power rankings. Similarly, picks for the following week's games are made by chosing the team with the largest power index value for games played to that point.

Evaluating design choices

The efficacy of the algorithm (as well as choices for the various constants) is judged strictly on how well it does picking games on the historical data set of NFL games played. I have made attempts to give more power for wider margins of victory (or loss); such attempts have lead to fewer games picked correctly and were discarded. This means teams can generate a only limited amount of power playing very weak teams, even with blow-out wins.

The design of the APR algorithm has also lead to a related phenomenon I call "power by association" (or PBA, if you like TLAs): When a weak team plays a strong team close (especially with the weak team on the road), the weak team will increase in power, even if they lose (and the strong team will decrease in power, even if they win).

The APR ranking system is by no means perfect. It doesn't take into account injuries, break-out players, fluke wins, how different teams match up, the strength of individual units within a team, or the strength of teams in different game situations. At the end of the season, it doesn't take into account that some teams will clinch their post-season fate early, and elect to rest key starters for one or more games.

Still, it does (to my way of thinking, anyway) a remarkably good job at picking games, and often reveals over- and under-rated teams before ESPN or other subjective-based power ranking systems notices.

Update: and, if you've made it this far, be sure to read this post, which expands on the consequences of APR's design.