The Öpik History Wars

For many one of the biggest shock results in last year's general election was Conservative Glyn Davies's gain of Montgomeryshire from Lembit Öpik. But for some it wasn't a surprise at all and had been predicted for years.

Now that is standing to be his party's candidate for Mayor of London, there has been a lot of talk about why he lost so dramatically - the biggest anti-Lib Dem swing in a held seat, the most distant Conservative gain and the Lib Dems losing a seat that was supposedly "Liberal forever (so long as they remember to campaign)". And there's some fierce fighting over recent political history.

It's become such an issue that even Öpik's nomination manifesto (warning that's a very BIG file) feels the need to address it, saying the following:

CRITICISMS ANSWERED HEAD ON
Lembit lost his seat in 2010
The failure of the party to retain both the parliamentary and Welsh Assembly seat in 2010 and 2011 in Montgomeryshire is linked to the drunken altercation between Assembly Member Mick Bates and a paramedic which led to his prosecution and to him leaving the party. Many were aware of Lembit's celebrity profile and assumed that this was the cause of the loss. Regrettably the party did worse in 2011 even indicating a common factor for both poor results.

This just does not convince. And it's unsurprising that it's provoked blog post responses from both the Lib Dem Welsh Assembly Member Peter Black (Lembit Öpik and the rewriting of history) and Conservative MP Glyn Davies (Lembit - take it on the chin like a man.).

I've commented myself on the matter - see for instance my comments at Dave Hill's London Blog: London Lib Dems will choose their mayoral candidate from field of four for my trading the statistics - and so for now I'll just confine myself to showing that the defeat was being predicted long before Mick Bates hit the paramedic, even if not everyone picked up on it.

First from a Usenet post I wrote two years ago:

Montgomeryshire isn't quite the "Liberal forever (except when they're too busy planning centenary celebrations to actually campaign to retain it)" bastion of legend but in several ways is very much a personalist constituency. Now sure I have my personal biases but much of what I've heard from there (including from my sister, who until recently lived and worked in Machynlleth at the very west of the seat and is somewhat to the left of me), Lembit Opik's popularity is shrinking faster than an asteroid burning up in the atmosphere because of his clowning around, high profile personal life and lack of any serious political track record.

By contrast the Conservative candidate is Glyn Davies is widely respected across the political spectrum - I've heard many Plaid members openly say a lot of good things about him (and even that they'd vote for him) - and if anything has increased his appeal by losing his list seat in the last Assembly elections only because the Conservatives gained two constituencies. Whilst the Assembly list results aren't a 100% guide to what will happen in a Westminster election, it's notable the Conservatives topped the poll there.

Furthermore with the Liberal Democrats under pressure in the neighbouring seats of Brecon & Radnorshire and Ceredigion they simply can't target all their energy in Mid & West Wales on Montgomeryshire even if they wanted to - and from comments by Lib Dem activists I get the feeling that many would not want to anyway and are certainly not going to haul themselves great distances to allow Lembit to carry on as Mr Celebrity.

(Message from discussion Opinion poll in Wales on uk.politics.electoral at Jun 16 2009, 11:08 am

Next there's UK Polling Report's specific discussion on Montgomeryshire. Many amateur (and not so amateur) psephologists comment there and unrealistic predictions are easily shot down. People were predicting the Lib Dems were in Öpik-specific difficulties as early as February 2007, with the earliest outright prediction of a loss in July 2008.

Finally there's the Vote UK discussion thread on Montgomeryshire. The earliest outright defeat prediction was in September 2007.

One other point has emerged - Öpik neither wrote his own manifesto nor checked it before it was submitted - see his comments in WalesOnline: Peter Black launches furious attack on 'self-obsessive' Lib Dem colleague Lembit Opik. If he's not even in control of his own campaign, how could he ever hope to be in control of London?

Boris didn't win because of the BNP

In the last few weeks there's been some rather high profile discussion within the Labour Party about their London mayoral candidate for 2012, with invariably a focus on why they lost the 2008 election. I don't wish to intrude on a private war, but one point I feel cannot go unaddressed is the claim being thrown around by some Labour members that Boris Johnson won the Mayoralty in part because the British National Party called for its voters to give him a second preference over Ken Livingstone. (Under the Supplementary Vote system used, these were the only two realistic options.) It is unfortunately standard "damn by association" stuff. Whilst there are many in the Labour Party who are honourable and above such behaviour, in my experience there is an element who will all too freely fling around accusations of racism as a means to get votes, not giving a damn about either truth or the polarising effect this can have on the electorate. The argument goes that Johnson's final majority over Livingstone was 139,772 and as the BNP candidate Richard Barnbrook got 69,710 first preferences then more than half that majority came from the BNP.

As is often the case in politics claims are made with only partial reference to the actual results and the truth requires a greater deal of detail to explain and will often have trouble competing with simplistic assertions made frequently and aggressively. To explain things fully it's necessary to go through the details on this.

Now the Mayoral election results are frequently badly announced and reported, because of the way the vote is counted and the second preference reported. Consequently most tables will list three figures for a candidate - their total number of ballot papers with a first preference, the total number with a second preference and (if applicable) the final result. However as that second figure includes people who gave preferences for both of the top two candidates the figures don't always add up.

For instance the figures for Boris Johnson and Ken Livingstone usually appear as follows:

Johnson 1,043,761 - 257,792 - 1,168,738
Livingstone 893,877 - 303,198 - 1,028,966

In actual fact the more relevant figures for the second column are 124,977 and 135,089 - these are total second preferences cast for Johnson and Livingstone respectively by voters who voted for neither for their first preference.

The official results don't go into enough detail to tell us how every individual voter gave their first and second preferences, although it's relatively easy to use the totals given to calculate the number of Boris Johnson 1, Ken Livingstone 2 votes at 168,109, and Ken Livingstone 1, Boris Johnson 2 at 132,815 (see this forum post by David Boothroyd).

But for other candidates it's much harder to work out results. We'll come back to this later, but first lets look at the impact of the voting system on the result. Or rather the non-impact.

Advocates of preferential voting frequently argue that it breaks the "problem" of vote-splitting and can often deliver a different outcome to whoever would have won on first preferences alone. Now the Supplementary Vote is a limited model, and the voting system doesn't get much coverage, but it's notable that here the only difference to the outcome it made was to narrow Johnson's lead over Livingstone by just over 10,000 votes. This is far from unusual - in many preferential vote elections I have seen the transfers frequently make no difference to the order of elimination or who is elected, instead just dividing up in similar proportions to the first preferences. Outside of co-ordinated voting at conference, only when two strong candidates are chasing the same block of votes and both campaigns are savvy enough to appeal for second preferences and their main rival candidate has more homogenous support have there been significant changes to the outcome. (This is not to say that in a closely fought race the distribution of preferences doesn't sometimes tip the balance but it's rare. The only Mayoral election to date where this has happened was North Tyneside in 2005 when a 1400 Conservative lead on first preferences was converted into a 1002 Labour lead on transfers.)

It's true that some parties called for their voters to use their second preference a particular way, but how many voters actually follow such an instruction? Remember the UK is not Australia where use the "How-to-vote card" is perfected to a fine art (although even there vote leakage can and does occur). The Greens' candidate Sian Berry and Livingstone made a reciprocal transfer call, but how many Green voters followed through? Similarly how many BNP voters followed the party's call for a second preference for Johnson?

Well apparently Giles Edwards and Jonathan Isaby's book "Boris v. Ken" has some figures in this area. I don't have the book myself, but this post and this one gives the figures for transfers from the eliminated candidates:

Brian Paddick: Total votes 236,685. Livingstone 73,612, Johnson 70,157, someone else or blank 92,916.
Sian Berry: Total votes 77,374. Livingstone 36,365, Johnson 10,984, someone else or blank 30,025.
Richard Barnbrook: Total votes 69,710. Johnson 22,200, Livingstone 4,353, someone else or blank 43,157.
Alan Craig: Total votes 39,249. Livingstone 10,352, Johnson 10,328, someone else or blank 18,569.
Gerard Batten: Total votes 22,422. Johnson 6,671, Livingstone 1,681, someone else or blank 14,070.
Lindsey German: Total votes 16,796. Livingstone 6,661, Johnson 1,327, someone else or blank 8,808.
Matt O'Connor: Total votes 10,695. Johnson 2,485, Livingstone, 1,120, someone else or blank 7,090.
Winston McKenzie: Total votes 5,389. Livingstone 945, Johnson 825, someone else or blank 3,619.

Redone as percentages:

Brian Paddick: Livingstone 31.1%, Johnson 29.6%, someone else or blank 39.3%.
Sian Berry: Livingstone 47.0%, Johnson 14.2%, someone else or blank 38.8%.
Richard Barnbrook: Johnson 31.8%, Livingstone 6.2%, someone else or blank 61.9%.
Alan Craig: Livingstone 26.4%, Johnson 26.3%, someone else or blank 47.3%.
Gerard Batten: Johnson 29.8%, Livingstone 7.5%, someone else or blank 62.8%.
Lindsey German: Livingstone 39.7%, Johnson 7.9%, someone else or blank 52.4%.
Matt O'Connor: Johnson 23.2%, Livingstone 10.5%, someone else or blank 66.3%.
Winston McKenzie: Livingstone 17.5%, Johnson 15.3%, someone else or blank 67.2%.

Or taking just the active second preferences:

Brian Paddick: Livingstone 51.2%, Johnson 48.8%.
Sian Berry: Livingstone 76.8%, Johnson 23.2%.
Richard Barnbrook: Johnson 83.6%, Livingstone 16.4%.
Alan Craig: Livingstone 50.1%, Johnson 49.9%.
Gerard Batten: Johnson 79.9%, Livingstone 20.1%.
Lindsey German: Livingstone 83.4%, Johnson 16.6%.
Matt O'Connor: Johnson 68.9%, Livingstone 31.1%.
Winston McKenzie: Livingstone 53.4%, Johnson 46.6%.

Unsurprisingly a lot of votes transferred in line with ideology but there were some very significant leakages. Less than half the people who voted Green followed the call for a Livingstone second preference. Less than a third who voted BNP followed the call for a Johnson second preference.

The BNP transfer contributed only about 18,000 of Johnson's final majority of 139,772 - hardly a decisive factor. It was the strong campaigning by Johnson and outreach to the whole of London, as opposed the Livingstone strategy of just being a Mayor for certain inner London boroughs, that produced the massive increase in votes that won him the Mayoralty.