A Numbers Game: The contradictory and confusing opinion polling on the EU referendum

In ordinary circumstances the obvious means by which one might attempt to anticipate the result of the referendum on EU membership to be held on 23 June would be to consult the opinion polls. After all, an essentially binary Yes/No decision should be more straightforward to capture in such surveys than the more complicated multi-party options seen in recent British general elections.
Yet few are putting much faith or trust in the polls on this occasion. That is partly because of the debacle witnessed in May 2015 – when 11 separate polling companies predicted a dead heat between the Conservatives and Labour only for David Cameron to record a six percentage point lead over Ed Miliband and secure a surprise overall majority – but also because the surveys conducted for this referendum have consistently shown a sharp difference between those conducted by telephone and the others (the majority of all polls undertaken) which deploy online panels.
If the former category is to be believed, then Remain holds an average lead of around ten percentage points among those who assert that they are certain to vote. If the internet camp is correct then, by contrast, it is about even between Remain and Leave. The betting markets, which indicate that there is about a 75% probability of a Remain outcome, have clearly decided which methodology they believe to be valid. If this is a miscalculation, however, a shock as profound as that of last May is a realistic possibility.
What are the methodological differences?
Until about 25 years ago, virtually all pollsters employed the same fundamental model. Polls were conducted face-to-face, often in the street, by teams of individuals with clipboards dispersed over different areas of the country. This technique had the advantage that it was relatively easy for those responsible for these polls to ensure accurate gender, race and age sampling. The disadvantages of traditional polling, by contrast, was that they were time-consuming to do and relatively expensive.
From the mid-1990s, led by ICM, much of the polling industry moved to telephone surveys. There was a fierce methodological debate about whether the loss of face-to-face contact was or was not consequential but a new consensus in favour of the change in technique eventually emerged. It did not hurt matters in this regard that telephone surveys were, at this time at least, faster and cheaper to conduct than the previous methodology had been. In relatively short order, almost all UK polls moved towards telephones (with MORI holding out against the shift for the longest period of time).
This unanimity did not last long. Starting with YouGov, certain pollsters started to invest in creating very large panels of online voters (50,000 people or more). Although the process of acquiring such a list was arduous and expensive at first, once a panel had been established, sending out surveys to sections of it was even faster and cheaper than had been the case for telephone polling. And as the shift from land-lines to mobiles and an increased aversion to ‘cold calling’ of any sort was making the collection of data by telephone harder than it had been, the shift towards online polls became even faster. This was further reinforced by the demands of the media – usually the primary client for political polling – which wanted survey results that were ever faster and ever less expensive. By the time of the last election, the majority of polls were online but with a respectable minority still done by telephone. The assumption in the industry, nonetheless, was that online polling was the future.
There is, though, a fundamental difference of principle between the two types of survey. Telephone polling remains random. An online poll involves people who have volunteered to take part in them. The challenge for telephone pollsters is the sheer number of calls that need to be made these days in order to acquire a sample of an acceptable size (20,000 calls for 1,000 full responses is normal).
The difficulty for online pollsters is ensuring that their samples, although invariably larger than those which telephone surveys can reach, are not distorted by involving people who are atypically political and disproportionately passionate about certain issues. In 2015, more than half of the polls taken by telephone suggested that the Conservatives were in the lead. Only 10% of the online polls reported at the same time had that result. As the polling sector conceded once it had held its own inquest in to what had occurred, the online polls had too many younger Labour-inclined voters.
How do the telephone and online polls compare in the EU referendum?
As noted earlier, there is a sharp difference between them. Almost all telephone polls have a clear Remain lead. Almost all online surveys suggest that it is a very close contest indeed. In an intriguing experiment last week, ICM for The Guardian conducted a telephone poll and an online poll at the same time. The telephone poll produced headline numbers (among those certain to vote) of 47% Remain, 39% Leave and 14% Undecided. The equivalent figures for the online poll (once again only including those stating they were certain to vote) were 43% Remain, 47% Leave and 10% Undecided.
This is obviously a vast difference. It is actually even more stark than it might seem. The two polls not only produced very different sorts of results but they did so while offering two very different profiles of the electorate. The telephone poll indicated that there was a huge ‘gender gap’ on the issue with men splitting evenly (45%/46%) while women were decisively in the Remain camp (by 50%/32%). Yet in the online survey there was no gender gap of any kind.
The telephone poll implied that there was a difference according to social class but it was not that vast a distinction. The ABs surveyed divided 56%-30% for Remain while the DEs split 44%-42% for Leave. In a similar vein, the telephone poll suggested that there was also an age gap with the 18-34 category backing Remain by 60%-31% and the 65+ favouring Remain by 45%-39% but this is not a massive split either. The online poll disagreed with this assessment. It revealed much bigger divides by class and age. It offered a UK in which ABs (55%-38% Remain) and DEs (57%-35% Leave) were at loggerheads, while the 18-34s (53%-37% Remain) and the 65+ lobby (58%-34%) Leave) look as if they come from different planets.
Which opinion polls are correct then?
That is a matter of opinion. The polling industry acknowledges there is a large division between the results from the two types of survey (it could hardly deny this) and is a little embarrassed at it. In so far as there is an official collective ‘line’ it is that the failures of the online polls that proved to be fatal in May 2015 have been identified and rectified so there is no rational reason to conclude that one set of polls is ‘right’ and the other collection is ‘wrong’; the true state of national opinion in all probability sits somewhere between the average of the two forms of opinion poll.
This could well be right but those who put money on these matters and much of the pundit class are not convinced yet that the underlying difficulty with online polling (self-selection) can be corrected by a new structure of ‘weighting’ these surveys or, even if progress has been made in this regard, there are features of this referendum, notably that the average Leave supporter appears to be more strongly motivated than the typical Remain adherent, which means that online polls are highly vulnerable to inaccuracy. Those who believe this to be the case point to the fact that at the recent 5 May elections, just as in the General Election, most online polls overestimated the UKIP share of the national vote while the telephone surveys were more or less spot on in their estimate of how Nigel Farage et al would do. If one or other style of polling is unambiguously more accurate than the other, then it seems logical on the basis of the available evidence to assume that telephone polls will end up proving the winner.