Some friendly debate

Kevin Bonham has written a very detailed summary of the performance of seat-level betting markets in the last election. It’s well worth reading — go check it out. We’ve really enjoyed reading Kevin’s blog and his analysis. In a spirit of friendly debate, we wanted to respond to a few comments that are directly relevant to this blog. Kevin writes:

electionlab in their final analysis considered that the most likely culprit in the errors made by seat betting markets was their modelling and not the markets themselves.  In my view, there was nothing significantly wrong in their model’s read of what the seat betting markets were thinking – rather, what actually happened was that seat betting markets themselves were in fact wrong.    Different modelling assumptions regarding covariance and so on greatly affect the spread of modelled expectations, but they have little impact on the mean. The seat betting markets were collectively expecting Labor to win fewer than 50 seats at the end.  There is no way to remodel the final odds to find 55 seats for Labor in them because it is just not true that those markets thought Labor would win that many seats.  Or at least, if someone “finds” such a way to read that result into the markets, the next time they test it I can pretty much guarantee the post hoc overfitting in their new model will cause it to blow up.

We disagree here, for two reasons.

First, changes in covariance structure don’t just change the spread of the seat-count distribution; they fundamentally alter the shape of the distribution, too, and this has important implications for estimating seat counts from the betting odds.  Assuming independence between seats results in a unimodal, bell-shaped distribution; on the other hand, assuming maximum covariance between seats (as constrained by the betting markets) results in a bimodal distribution, with very little density in the middle. More generally, it makes intuitive sense that the distribution might become multi-modal if you bring in covariance between seats. We agree that the mean is relatively insensitive to covariance structure. But for bimodal distributions, the mean is a very bad point estimate and lies in a low probability region of the distribution, meaning it is not a very useful tool in this situation (in hindsight, we should not have used the mean as a point estimate for our maximum covariance model because of this). This is really important, because it affects how we make inferences using the distribution derived from the betting odds. There is nothing sacred about the mean, and it appears that a point estimate centered on the mode (or two point estimates, centered around the two modes) may make more sense here. Or perhaps a point estimate is just a bad idea, and we should look at 95% credible intervals instead. We don’t know the true covariance structure, so we don’t know the true underlying seat count distribution. But we have very good reason to believe that the mean will be a poor point estimate in this case.

Second, a final Labor seat count of 55 seats is absolutely consistent with the betting odds, as long as you expect there to be moderate amounts of covariance between seats. According to the maximum covariance model, there was a 95% chance Labor would obtain somewhere between 32 and 64 seats. Of course, the maximum covariance model is an extreme case, but it’s not hard to show that for more moderate covariance structures, the 55 seats result is within a 95% credible interval. Almost the only way to obtain a prediction with 55 seats outside the 95% credible interval is to assume independence between seats. But as we know, that’s unlikely to be a good assumption.

It baffles me that experienced statisticians attempt to determine how many seats betting markets think parties will win by looking at an indirect and problematic measure (aggregation of implied probabilities concerning particular seats) when there are more direct markets available on seat total events and their past track record has been excellent.

This is a reasonable question: why did we derive predicted seat counts from seat-level betting odds, rather than just directly looking at the seat count betting markets? Why use an indirect method when there is a direct one? We did this because we wanted a general approach that allowed us to look at interesting seat-level scenarios (e.g., this and this). The seat count predictions were something we could readily do with our more general model, so we had a go. Unfortunately, the election ended up being a landslide and, for many of the scenarios we thought might be interesting, the betting markets ended up giving pretty obvious and boring predictions! So the seat count predictions ended up being prominent. We agree that, if your sole aim is to predict seat counts, the seat count betting markets are the way to go. But our ultimate goal is not to just get good at predicting seat counts. We want to use the seat-level betting odds to find interesting stories that can’t be revealed without seat-level data.

Our final predictions: how did they go?

The election is finally over! At time of writing, the Coalition stand to hold 89 seats and Labor 57 seats, with the remaining 4 seats going to a Green and independents. It seems likely these numbers may well change in the coming weeks as postal votes come in for a few close electorates, but we’ll go with them for now.

Leng and I have been sorting through the entrails of our predictions, looking at what worked and what didn’t. What did the betting markets get right, and where did they fall down? What could we have done better?

What worked

The betting markets successfully predicted the Coalition would win government with a large majority. This bears repeating, even though by election day pretty much everyone was predicting this. In contrast to the polls, at no point in the campaign were Labor anywhere close to being the favourites in the betting markets. The high-water mark for Labor in the polls was around July 8, when Newspoll recorded a 50-50 2PP. In contrast, our AFR analysis of the electorate-level betting markets on July 11 still put Labor well behind the Coalition, with an expected Coalition-Labor seat count of 84-60. The closest the parties ever got was 81-66 in our July 30 analysis. We aren’t criticizing the polls here — the polls and betting markets measure different things — but this clearly demonstrates that betting markets do more than just parrot polls, contrary to a common misconception.

What didn’t

Assuming the 89-57-4 seat count holds, our predicted expected seat count of 99-48-3 was off by 10 seats. While this was well within the margin of error of our maximum covariance model, this is worse than we’d expected! There are a few possible reasons for this.

Starting with the most general, could this be considered strong evidence against the underlying theory of prediction markets? Not really. Since prediction markets give probabilistic forecasts, they have to be assessed over multiple campaigns. Results from any one campaign don’t tell you much about predictions of underlying probabilities, just as rolling a die once doesn’t tell you enough to know if it’s loaded or not.

Could this result be due to violations of some of the underlying assumptions of prediction markets? That’s certainly possible. We have no information on the amount of money wagered in each seat, although we have been told that around $250,000 was wagered on Sportsbet’s individual seat market between January and the election, with most of it on marginal seats. Nevertheless, we don’t know how that money was distributed, or if it was sufficient, so it’s possible that this had something to do with the seat count error. Longshot bias does not appear to have had any large, obvious impact on the result, with the betting markets ultimately underestimating Labor’s seat count rather than overestimating it (while this may be partly due to our rather crude correction for longshot bias, this doesn’t appear to have altered the results much). Our understanding of these biases is very crude. Predictions from seat-level betting odds have little heritage and it will take time to be able to observe and eventually correct for these biases.

However, we think the most likely culprit is in the modelling, not the betting markets themselves. Using electorate-level betting odds to predict seat counts requires knowledge of the underlying covariance structure between electorates (ad hoc methods, such as counting up the number of seats where Labor has > 50% probability of victory, make no statistical sense). There is no obvious way to estimate this, so we have to make some assumptions about this structure in practice. To try and get an idea of the impact of this uncertainty, we used two models for our estimates: a model that assumed zero covariance between seats (the ‘independent seats’ model), and another that assumed the maximum possible covariance between seats, conditioned on the betting odds (the ‘maximum covariance’ model). These models both generated distributions of seat counts for each party.

Covariance is key

The independent seats model is commonly used, but there seem to be good intuitive reasons to believe it is badly wrong. The final seat counts were outside the margin of error for the independent seats model’s predictions; that is, there was less than a 5% chance of this election outcome occurring by chance, according to the independent seats model. We aren’t big fans of hypothesis testing, but if you were to use the independent seats model as the null hypothesis, you would probably reject this model given the outcome (p < 0.05). The value was well within the margin of error for the maximum covariance model, however. We don’t suggest the maximum covariance model is necessarily a good representation of the real world. But it demonstrates the sensitivity of the results to the assumed covariance structure.

Why did the two models yield the same expected seat counts, even though their seat count distributions were so different? The maximum covariance model represents a scenario where a large, nationwide swing occurs in one direction, resulting in a symmetric distribution that yields the same expected seat count as the independent seats model. However, more subtle covariance structures, such as ones where different states have different magnitude swings (as occurred in this election), may result in asymmetric distributions that shift the expected seat count. So understanding the role of covariance structure in both expected seat counts and the overall distribution is essential. Covariance is key, and it’s something Leng and I want to look at in more detail.

But for now, we’re going to take a break! We’ve had a great time making this blog. We couldn’t have done it without the support of a lot of great people. We want to thank Bob Chen Ren and Jeff Chan for their assistance with the blog, Edmund Tadros and Jason Murphy at the Australian Financial Review for writing up our work, Kevin Bonham and Simon Jackman for their insights, and everyone who showed an interest in this blog!

Update: Labor wipeout in Tasmania?

We posted a few weeks ago about the likelihood of a complete Labor wipeout in Tasmania: our models gave it a probability somewhere between 5% and 25%. Even a probability of 5% seemed pretty high at the time, given Labor currently hold every seat in Tasmania, except for Denison (held by Andrew Wilkie).

With the election one day away, we wanted to revisit this scenario. The odds should have considerable predictive power by now. Using the same methodology as the previous post, we estimate the probability of a Labor wipeout in Tasmania to be somewhere between 22% and 35%! Conservatively, that’s roughly the probability of flipping a coin twice and getting two heads. It’s more likely not to happen, but there is still a very considerable risk it will.

Looking at the individual seat probabilities, Bass and Braddon appear to be comfortable Coalition gains. Wilkie appears likely to retain his seat in Denison. Franklin and Lyons are the two seats that could still go either way, with Franklin leaning towards Labor (61% chance of Labor victory) and Lyons leaning towards the Coalition (58% chance of Coalition victory). If the implied probabilities of Labor victory drop in Franklin, the probability of a Labor wipeout will increase substantially. We’ll take another look at this scenario when we publish our predictions in the next day or so.

Labor’s electoral prospects still deteriorating

The betting odds suggest the ALP is facing a massive defeat on Saturday. But even worse for Labor, their expected seat count is still deteriorating.

Expected seat counts using Sportsbet betting odds from September 5

Expected seat counts using Sportsbet betting odds from September 5

Our latest analysis in the AFR (a link will be up shortly) was conducted earlier in the week, where our analysis showed the betting markets predicting Labor would win 51 seats. Underdogs often regain a little ground leading into the final few days of the election campaign, so we thought that number might increase before election day. In fact, as of September 5, the markets now expect Labor to win more like 49 seats.

Barring an act of God, the overall election outcome looks like a foregone conclusion. Both our independent and maximum covariance models give Labor literally zero probability of victory. This is astonishing, since the maximum covariance model had been giving Labor a small but non-zero chance of winning up until very recently. The reason for this change appears to be many of the probabilities moving towards a clear victor in some of the seats that were relatively uncertain previously. As the seat probabilities move closer to 1 or 0, the choice of covariance structure has a smaller impact on the predicted result.

Leng and I will be posting our final predictions shortly. Putting aside any political views on a big Coalition victory, an easily predictable election is not a good test of our methodology! So we’ll be making predictions about individual seat results, and possibly a few scenarios, too. Stay tuned.

Happy Beattie vs Sad Beattie

Some recent polling in Forde suggests Peter Beattie has a tough fight on his hands, with an estimated 2PP 60-40 against him. As of August 17, the Sportsbet odds give him a 36% chance of victory. Characteristically of betting markets, the market appears to have noted the polling but hasn’t swung as hard against Labor as might be naively expected for such a bad poll. This is consistent with the fact that polls and betting odds are measuring different things.

The electorate of Forde. Source: Wikipedia.

The electorate of Forde. Source: Wikipedia.

But overall, the news doesn’t look good for Beattie. He only has a 36% chance of winning the seat. But wait, for Peter Beattie, it gets worse. There are worse things in politics than not winning the seat. Let’s look at the full range of possible outcomes for Beattie:

(a) Beattie loses, ALP wins

(b) Beattie loses, Coalition wins

(c) Beattie wins, ALP wins

(d) Beattie wins, Coalition wins

Clearly, (a) and (b) are bad for Beattie. But also, even though Beattie wins in (d), he would then need to serve in opposition for at least three years, since he has pledged to serve a full term regardless of the overall election outcome. For a former Premier used to governing, that would really suck.  Let’s combine (a), (b) and (d) into one scenario: Sad Beattie. Call the other scenario, (c), Happy Beattie.  What are the implied probabilities of Sad Beattie and Happy Beattie?

Sad Beattie vs Happy Beattie

Sad Beattie vs Happy Beattie

As with our previous scenario analyses, we ran lots of simulations for two extreme cases using data from August 17. The implied probability of Happy Beattie is somewhere between 0% and 22%, for the independent and maximum covariance cases, respectively. The true value lies somewhere between these two probabilities.

For the independent seats model, the probability of Happy Beattie is zero. This is because the independent seats model gives Labor an effectively zero chance of winning the election. Under this model, even if Beattie is elected, he will automatically be part of the opposition. For the maximum covariance model, Labor victory is possible: around 22%, the same as the maximum covariance probability of Happy Beattie. So the limiting factor on Peter Beattie’s happiness right now isn’t his performance in his own seat; it appears to be the overall performance of the ALP. Note, therefore, that the probability of Happy Beattie is even less than the probability of Beattie winning his seat. With just under three weeks till election day, Sad Beattie is looking more likely than Happy Beattie.

Three myths about betting markets

Since our latest analysis appeared in the AFR and, for the first time, the SMH, there has been some chatter on Twitter about the legitimacy of using betting odds to make predictions.  We’ve written about the strengths and weaknesses of betting markets before, but wanted to address a few specific criticisms.

Betting markets certainly have their weaknesses. They don’t work if there aren’t enough people betting. They suffer from longshot bias. And some situations are just so uncertain that no prediction method, including prediction markets, can offer much additional insight (a good example of this is the election of the Pope, where only a handful of people in the world have any useful knowledge about the outcome).

But betting markets have also been shown, time and again in the literature, to outperform pundits and polls on average. They’re sometimes misunderstood. Here are a few myths we’ve heard recently:

Opening up a can of whoop-ass on myths about betting odds

Leng and I will never be able to be as hirsute as these dudes.

Myth #1: Betting markets were lousy in the 2010 election.
This is not right. The electorate-level betting data (which we use) predicted a hung parliament in 2010, with Labor expected to win 74 seats. This is a remarkable result, given the 2010 election was probably one of the toughest election outcomes to predict in Australian history.

Myth #2: Betting markets just track polls.
Betting markets aggregate information from many sources, including polls. But betting markets and polls measure different things. Polls track instantaneous voter sentiment; betting markets track the probability of victory over time. So while we’d expect a bump in the polls to be noticed by the betting markets, you’d expect the markets to be cautious about weighting them too heavily. That’s exactly what we’ve seen in this campaign. While the Labor victory probability has usually moved up and down with the polls, at no point has Labor been anywhere near the favourite in the betting odds. That’s a very important qualitative difference between the betting odds and the polls (which have recently suggested the race is neck-and-neck).

Myth #3: Betting markets predicted a landslide in election X, but the result ended up being very close.
A probability of victory implied by betting odds is not a prediction of vote proportion. A 90% probability of Labor victory is not a prediction that Labor will win 90% of the vote; the market may just be very confident Labor will win 76 seats. So, for example, while the odds for the overall election result in 2010 were putting Labor on as much as a 70% chance of forming government, this doesn’t say anything about the expected margin of victory. It simply says the markets were moderately confident Labor would form government (which it did).

Betting markets aren’t perfect. But we think they’re a really promising tool for making predictions in very uncertain situations, such as elections. We’re still a long way from election day, but we expect our predictions to become more and more accurate as it draws closer.

Labor wipeout in Tasmania?

It looks like Labor faces a tough fight  to retain its seats in Tasmania. Before the switch to Kevin Rudd, there was some speculation that Labor may even be completely wiped out in Tasmania. The switch back to Rudd seemed to ease those fears in the Labor camp. But with the Coalition appearing to have the momentum in the polls, what do the betting markets think about a Labor wipeout in Tasmania?

Tasmania's five federal electorates. Source: Wikipedia.

Tasmania’s five federal electorates. Source: Wikipedia.

Tasmania has five federal electorates: Bass, Braddon, Denison, Franklin and Lyons. Denison is held by the independent Andrew Wilkie. The other four are held by the ALP.

We ran lots of simulations for two extreme cases (as in this post), and found the implied probability of Labor winning zero seats in Tasmania is between 5% and 25%, according to the Sportsbet odds from August 14. What does this mean? As things stand, the betting markets believe Labor probably won’t get wiped out in Tasmania. Then again, this is an extreme scenario for a state which is currently wall-to-wall Labor (excluding Denison). Yet there is still at least a 1-in-20 chance it will happen.  To give you an idea of what this means, it’s about the same probability as getting four heads when you flip a coin four times. It’s unlikely, but not really a position you’d want to find yourself in.

Will Eden-Monaro remain a bellwether seat?

The bellwether seat of Eden-Monaro has been held by the ruling party for more than 40 years. Will it remain that way this time around?

One of the unique strengths of our model is that we can estimate the probability of a scenario like this from the electorate-level betting odds. You can’t do this with the betting odds for the overall election result. And doing it with national poll results involves a lot of poor assumptions about the size of swings.

We ran our model for both the maximum-covariance and independent cases and simulated lots of elections. We then looked at each election and counted up the number of times Eden-Monaro went with the party who formed government.

The electorate of Eden-Monaro. Source: Wikipedia

The electorate of Eden-Monaro. Source: Wikipedia

Converting these to probabilities, we got probabilities of 48% and 74%  (for the independent model and maximum-covariance model, respectively) that Eden-Monaro will remain a bellwether seat at the 2013 election. In other words, if we assume no covariance, there is a 48% probability that Eden-Monaro will be won by the same party that wins the election. If we assume maximum-covariance, there is a 74% chance that Eden-Monaro will be won by the same party that wins the election.

What does this mean? Recall that the maximum-covariance and independent models are two extreme ends of a spectrum, so the true probability lies somewhere in between (although not necessarily right in the middle). So the true probability that Eden-Monaro keeps its bellwether streak going is somewhere between 48% and 74%, according to the betting markets. That means, at this stage, it’s more likely than not to stay a bellwether seat, but it’s very uncertain.

This is an interesting result since the betting odds in Eden-Monaro are currently favouring the Labor candidate Mike Kelly, despite the fact that overall the betting markets are pointing to a Labor defeat. If you made the mistake of treating probabilities inferred from betting odds as vote proportions, you would incorrectly conclude Eden-Monaro was on track to break its streak. Of course, we are still a long way from the election and things can change.

We’ll be looking at all kinds of other scenarios over the course of the campaign. Let us know if there’s one in particular you’d like us to look at.

Does covariance matter?

Leng and I had a really interesting exchange with Simon Jackman on Twitter yesterday. Me and my whopping eight followers (hi mum) would probably agree that I suck at Twitter. It became difficult to continue the discussion with the 140 character limit, so Leng and I thought we’d try and outline it here.

The discussion was about the importance or otherwise of modelling covariance between seats, in light of the extremely low probability of Labor victory (< 1%) implied by the seat-level betting data, when modelled assuming seats are independent. Leng and I have put a bit of effort into including this covariance in our model, and we think the very low implied probabilities of Labor victory are due to ignoring this covariance. Simon suggested, though, that the Efficient Market Hypothesis (EMH), were it to hold, implied the seats should be treated as independent.
Here’s what we think Simon’s saying:
1. Bettors can bet on any seat.
2. If they think that seats are NOT independent but move in certain directions together, then this will be reflected in how they bet.
3. How they bet affects the prices of each seat.
4. Assuming the EMH holds, the price of each seat then accurately represents the probability of a party winning that particular seat.
5. Built into that probability is any covariance that bettors think exists. Therefore, it is OK to assume independence because each seat’s probability already builds in any covariance.
We agree with 1-4, but disagree with 5. For a correlated multivariate Bernoulli distribution, the probabilities p_{1}, ..., p_{150} don’t uniquely define the covariances (although they do constrain the allowable choices of covariances). So even if the EMH holds, and p_{1}, ..., p_{150} are known exactly, these probabilities still imply a range for each covariance, rather than a single value. If the EMH holds, we would expect this range to include the true covariance, but we don’t have enough information to define it exactly.
With a little algebra, you can show that for any set of seat probabilities, choosing all covariances to be zero (i.e., consistent with assuming independent) is always within the valid range of choices for covariances. But this doesn’t mean it’s the right choice. Indeed, there are good reasons to think that there are significant correlations between many seats. And if we choose valid non-zero covariances, the seat-level betting data imply a much more realistic probability of Labor victory (as much as 34% using odds from August 6), that is in better agreement with national-level betting odds.
We hope we haven’t misrepresented Simon’s views about the role of EMH in making the independence assumption. Hopefully we can clear this up and better understand the proper role of covariance in making predictions using betting markets.