After an intense conference week in St. Louis with WAPOR and AAPOR, we at Gallup Nordic return with several insights – about the development of populism, the AI hype, and why real answers from real people are more important than ever.
WAPOR stands for World Association of Public Opinion Research, AAPOR is the American counterpart. This year, I was invited to speak at both conferences held consecutively in St. Louis. About 150 participants from all over the world attended WAPOR, I was the only Swede. Unfortunately, due to visa problems, about 100 additional visitors could not come to the USA. I was the only Swede.
AAPOR was the larger part, with about 1000 participants, mainly from the USA attending the conference. The contrast from the “small” WAPOR and the “large” AAPOR was striking for those of us invited as speakers to both. Wow, so many people!
Researchers from all the major universities in the USA were there, including those everyone recognizes, such as Harvard and Stanford. The number of Swedes could be counted on two fingers, me and one other.
WAPOR: Populism on the agenda
My presentation at WAPOR was held together with researchers connected to ISSP. We presented a broad approach to the global rise of populism – how the interaction between politicians and media has created the foundation we see emerging in so many Western countries in recent years. My presentation was very well received, and we will make it available to everyone who wants to see it afterward by recording it. Unfortunately, there were no opportunities to record any lectures on-site. Since many simultaneous presentations took place, several people heard about my presentation only afterward and wanted to see it. I also know many in Sweden who are very curious about my lecture on populism .
AAPOR: Building trust
At AAPOR, I shared a session with four former AAPOR presidents, very honored to share the stage with them. The topics was around trust and transparency in surveys.
Here is a link to the entire session: Factors in Public Understanding of Polls and Surveys
I explained how we in Sweden – through mine, Gallup Nordic and Novus work – have managed to strengthen trust in opinion polls. For almost 15 years, I have educated the media about what distinguishes scientific measurements from non-scientific ones. The result? Almost all serious editorial offices in Sweden today only rely on surveys based on random samples.
But there are unfortunately setbacks. Some editorial offices have started to compromise on quality and allow self-recruited surveys. The differences between these and the scientific surveys Novus, Ipsos, and Kantar deliver are often striking – sometimes like two parallel realities, serious probability based form one common view, and the non probabilty based form their own, not being the same as the scientificly proven ones.
The harsh truth for journalism is that if you do not use scientifically proven methods as news sources, you are not using a reliable source. Can it really be news if you do not know if it is true?
USA’s election – no longer a benchmark
The latest US election clearly showed that the American survey industry is in crisis. Well-known actors withdrew, while new and untested companies took over. The surveys became suspiciously similar – I have read statements that they are so similar that they could not have been independent from each other. If the purpose is to follow the herd, not present an independent poll from your data collection, we lose the purpose of surveys: to show how reality actually looks.
Saying that you must trust science is appreciated by most, but so many forget to defend the knowledge that the entire industry’s trust rests on. In the USA, for example, it is interesting that most of news media’s published surveys do not rest on a scientific methodology, what I gather because the media cannot afford to pay what they cost, and there are many new actors who gladly appear in the news to build brand awareness. But i also learned that established survey companies increasingly have stopped releasing polls about the political election in the USA. However, the parties only use surveys based on scientific methods.
The media cannot afford to do it right. The political parties cannot afford to get it wrong.
These two worlds rarely meet, political pollsters and the good scientists and insight managers know and use what works, but the media image create a misleading picture that opinion polls no longer work. Despite the established survey companies in the USA not “playing the game” they get the biggest hit by the general distrust that rightly exists against the free nonscientific surveys news media uses. Resulting in both decreased willingness to pay and lost trust in research and serious survey companies. Resulting in more critique and lower response rates. Its a vicious circle that ,ust be broken by protecting the science we know works. And keep the experiments about alternative methods to be just experiments until they can be proven. From my perspective, selling marketresearch and opinion polling, the methodology must be solid, if not what are we selling? Its not facts and knowledge then, but guesses, wishful thinking? Would you pay good money and use the latter as part of your strategy? Can you really stand behind the results if you dont have any scientific backing? Maybe you have singular anectodical proof at best…
Here Gallup Nordic and Novus’ tactic deviates, we only deliver surveys based on scientifically proven methods, even to the media. Unfortunately, media in Sweden can not afford it either, but it is even worse PR if we do not show how to do real surveys in the media. Nopvus is the market leader in Sweden, if we leave the public political polling space, and open acces panels take our place, and get the election wrong. Novus will still be the biggest looser since our brand is synonymous with opinion polling, same with Gallup Nordic as a company name. Every time the polls get it wrong in US I am the one being called to national news having to protect the polling industry provided they use scientifically prooven methods.
As I said, we educate news media on how to evaluate real surveys as news and understand that bad surveys cannot be the basis for news.
If it is not true, it cannot be news. Nonscientific polls is not a reliable source of facts.
Scott Keeter from PEW also at the same session talked about a very interesting course on opinion polls they have developed.
It is available here: Public Opinion Polling Basics | Pew Research Center.
Something I would have gladly made a Swedish counterpart to if we had had the resources.
Free data and synthetic answers – a false sense of security
Research requires data. But when money is lacking, it is easy to use available non representative data from, for example, Twitter, or build analyses on non-scientific open acces/selfrecruited panels. This tend to be be misleading. When althogh the researcher knows of the limitations, and maybe also specify it, but still treat it as representative data. Being used that way in scientific papers, the non representative data source are given an “academic seal of approval”. Misleading both the media and decision-makers. We all know open access panel or twitter data is not representative. We all know that wheighing is not a cure to make the data truly representative. Why pretend it to be?
One growing pitfall is the use of so-called “synthetic respondents” – AI-generated people who “answer” questions. It sounds smart. But it is just an illusion. The answers are fabricated and do not build on real knowledge – just guesses!
Imagine if any other academia treeted the foundation for the research in the same way? If medical science could not afford labs? If engineers did not have a proper workshop with all the tools?
Social studies weirdly have that view on the raw material and tools needed? The replication crisis is the result. Making due with inferiour data and extraplolate on that? We as an industry need to change that. Not being outspoken on this we constantly dig our own grave. Facts cost money, all data is not reliable.
Isn´t free data by definition worhless?
AI – potential with risks
AI was a recurring theme at the conferences. At Gallup Nordic and our daugther company, Novus, we have long used AI to analyze open answers, a relatively safe application. But several researchers showed that AI in more advanced applications does not live up to expectations. It requires extensive control, and in many cases, the cost of validation is higher than the gain. And can you even validate AI generated data? Or are you just being mislead by AI hallucinations?
The common thread in all AI presentations I and others saw was that the gain is very doubtful. The result can be completely wrong, it can hallucinate, and the other Swede who presented at AAPOR was Johan Martinsson from Demoskop, his presentation was an Demoskop experiment where using AI to try to guess answers they asked but omitted to show the AI. Johan concluded that it did not work. In line with all other researchers I listened to at AAPOR and WAPO.
As a presenter from Pew stated, it is unclear if the gain with AI exists if you still have to validate everything.
AI as synthetic respondents trip on its own feet. The role of surveys is to bring out facts that you do not already know. If you know that AI guesses, and maybe even hallucinates, you cannot verify it if you do not already know what the answer should be.
In Automation AI can be helpful. But not to guess/simulate what people think, artificial intelligence gives artificial opinions. It may look like a real presentation of people’s opinions, but it is not, you risk losing money if you make business decisions based on artificial opinions instead of real opinions.
Conclusion: Facts and knowledge cost – guessing is expensive
It is time to remind about what should be obvious: surveys are not about predicting the future but about giving a correct picture of the present.
In a world where changes are fast and unpredictable – from pandemics, war to electricity crises – the right insight at the right time is crucial. Guessing wrong can lead to disaster. Much more expensive than the cost of doing a scientific survey.
“The role of opinion polls is to provide the truth in questions where we do not know the answer – not to guess right afterward.”
Do you want to know more about WAPOR or how to become a member? Feel free to get in touch! Sweden needs more voices in the international dialogue about research and survey quality.
