Smoke, mirrors and election surveys

SOMETIMES the similarity between election surveys and reality is purely coincidental. The election is around the corner and few things are as heavily discussed in the run-up to May as these surveys.

I am often asked to make a prediction about the outcome of this year's national and provincial elections. Without data, my understanding of what is likely to happen is that the ANC will get between 49% and 52% of support, the DA 18% to 21% and the EFF between 9% and 11%. This “prediction" is based on demographic factors, the quality of leadership, the likelihood of apathy among voters, the respective policies of the parties, their access to capital (the ANC uses the state treasury) and diverse economic trends. However, it's still just an opinion, albeit based on ad hoc or “gut feel" logic. Understanding the actual statistics is more complicated than that.

A good understanding of qualitative trends should give observers and commentators a fairly good insight into what to expect. These are factors that are assumed in opinion surveys to have already been processed by the electorate and that will determine their voting behaviour. But the question is then, how do those variables affect voters on voting day?

A student who still hasn't received his or her National Student Financial Aid Scheme payout, or a pensioner's allowance that is regularly paid out late, or a power outage that unexpectedly hits a neighbourhood, can all affect voting behaviour, sometimes significantly. If South Africa is unexpectedly plagued by nationwide load-shedding in the run-up to the elections, I'm pretty sure already-sceptical voters will have something to say about it on polling day — by simply staying away, for example. If you think people who stay away have nothing to say, then your ignorance is for your own account.

Every pensioner who is saddened by the state's treatment of her also influences 10 family members who are directly affected by it. Every unemployed person has a family and extended family who are also inconvenienced by his joblessness. All of these factors make it difficult to gauge how many and which voters will eventually turn up at the polls. We already know that participation in our elections is statistically dwindling, so turnout is likely to be the most influential statistic of this year's election.

But back to the surveys. A typical mistake journalists sometimes make is to report them as predictions. Sometimes they do state that the data should not be read as a prediction, but then they present the data in such a way that the reader is necessarily under the impression that it is indeed a prediction.

As Victory Research's Gareth van Onselen and independent analyst Dawie Scholtz have had to repeatedly remind readers, a poll is a statistical snapshot of an opinion or a set of opinions at a particular moment and cannot be understood as a prediction of what the result of an election will be months later. And, equally important, local and national elections data need to be clearly distinguished, because statistically they are not necessarily comparable phenomena.

After a Change Starts Now (CSN) poll of 9,000 respondents, Prof David Everatt from the Wits School of Governance wrote in Daily Maverick: “Unless something unforeseen occurs, the shape of post-election South Africa is already reasonably clear." This is misleading and gives the impression that the poll could serve as a prediction. Equally misleading is that, as Van Onselen rightly pointed out, the survey is at least three months old, yet Daily Maverick refers to it as a “new poll".

CSN is the party founded by Roger Jardine. Scholtz believes surveys conducted or funded by political parties should trigger an observer's nonsense detector. According to this survey, Everatt says the ANC is “well below the 50% mark", the DA falls below the psychologically important 20% (19%), and in the Western Cape the DA loses its absolute majority of 56% and gets only 42% of support. The broad brushstrokes used here are confusing rather than enlightening.

Scholtz's argument may also be true of the DA's internal November 2023 poll which showed the party's support at 32% and the ANC's at 39%. We don't know how the data was accounted for but we can say with reasonable certainty — and the wisdom of ad hoc opinion — that the statistics in this case are probably more complicated than the DA would have us believe. However, DA surveys are often quite reliable and this may again be a case of inadequate reporting rather than questionable data management.

I also listened to Everatt on RSG, where he “predicted" that ANC support in KwaZulu-Natal support could drop to 25% — while also stating that an astonishing 30% of respondents in the province indicated they were undecided or not necessarily going to vote. That 30% needs to be accounted for statistically, otherwise the 25% makes little sense. Everatt held the view that respondents who did not want to indicate their preference for a particular party had historically voted for parties other than the ANC. This is certainly not impossible but the argument is a total oversimplification and statistical methods do exist for offsetting undecided votes or potential voters refusing to express an opinion in the poll.

It is also important to note that the decision on how to allocate undecided voters will depend on the specific goals of the poll, its context and even the subjective preferences of the researchers. The method chosen should be communicated transparently and any assumptions made during the allocation process should be clearly stated to protect the integrity of the results. It is entirely possible that the statisticians who worked on the CSN poll did all the necessary calculations but Everatt's writing in Daily Maverick and his discussion of it creates an extremely distorted image.

I haven't seen the survey's original data but it seems that no turnout modelling was done, at least not as far as the reporting on it is concerned, and it also looks like respondents who were undecided were not allocated. Scholtz also points out that non-voters, in other words people who are not registered, should be left out of the calculations. From now until the election, the electoral lists are going to change very little and the first question for a participant in a survey should be, “Are you registered to vote?"

According to Scholtz, the terminology that sceptical analysts and journalists should look out for is, for example, “polling of all registered voters" (poor) or “polling of all South Africans" (very poor). What an analyst or reader would rather see is “polling of likely voters".

Writing about the same survey, journalist Ferial Haffajee referred to the interesting fact that 61% of respondents had never heard of the Multi-Party Charter yet would nevertheless strongly consider supporting the cooperation agreement between 10 minority parties at the polls. She is more specific in her data, stating that the survey counts the ANC's support at 39% should the election be held tomorrow. The EFF with 16% can begin to claim the role of official opposition and Julius Malema's red brigade doubles its support in the Western Cape.

But again, does that 39% take into account the possible turnout and/or uncertain voters? It's not obvious from the reporting. If I can process my own ad hoc method into an opinion: the EFF will not get close to that 16%, and if the turnout is around 55% the party is in danger of dropping below 10% again. But maybe the data isn't on my side of the argument. Still, I will bet my Land Rover that the ANC will get closer to 50% than to 40% in May.

The EFF's support is overestimated in most surveys. For example, a survey conducted at the end of October 2021 by Ipsos and eNCA measured the EFF's likely support for the November 1 elections as follows:

With high voter turnout: 13.4%.
Medium turnout: 12.8%.
Low turnout: 13%.

Categories like medium, high and low are too broad to make sense of, especially since it's entirely possible to be more specific. In the end, the EFF received 10.88% of the support in the 2021 elections. As Scholtz argues: “Surveys that use ‘all registered voters', or ‘all South Africans' often underestimate the impact of minorities and smaller parties and overestimate the impact of black voters."

Ironically, the Ipsos/eNCA poll was accurate on the EFF when the figure for “all registered voters" is used, but it was out by more than 30% when “voters who do want to vote" is applied. In the end, turnout for the 2021 election was indeed low (46%). The survey was conducted  close to the actual election, so there were fewer factors that would drastically influence voters' opinions at that time. As a general rule, Ipsos surveys are reliable when it comes to the ANC but less so for smaller parties.

The reasons for this are simple: the probability error has a major impact on measuring smaller parties' actual support, as does the size of the sample and demographic factors that are difficult to discount because surveys necessarily only process the raw data. Political commentators and journalists then struggle to make sense of sometimes complex data, and sometimes sensationally fixate on the quantified outcomes without mentioning the qualified aspects. With that said, Lerato Mutsila, in Daily Maverick of February 7, manages quite well to explain these complexities in the context of Ipsos' most recent poll. It can be done.

Turnout will be the crucial factor

The turnout level will almost certainly determine the fate of the ANC and the opposition in this year's election. If opposition parties succeed in getting more of their voters to the polling booths, their chances are exponentially better, especially given the expectation that turnout is likely to be the ANC's biggest challenge as well.

Until then, believe me, the ANC is not going to get 39% and the EFF is not going to get 16% in this election. Most surveys do not confirm the qualitative observations of my political instincts.

♦ VWB ♦

BE PART OF THE CONVERSATION: Go to the bottom of this page to share your opinion. We look forward to hearing from you.