I woke up this morning to this post shared in my Facebook feed:

On top is a bar chart picked up from a Le Devoir article published last night at 9:12pm¹ and altered by Jean-François Provençal from the hit millennial-humour TV show Les Appendices. Below is what is called a meme. It’s a still from the TV-reality show The Osbournes in which fallen metalhead Ozzy Osbourne doesn’t know what’s going on when a phone rings.²

Read til the end

Adding “who answer the phone” after “Voting intentions of 18-34-year-olds” presupposes that polls are still conducted by calling randomly selected numbers from the phone book. As I’ve explained in the Polling section of the Elections primer, pollsters have developed new methodologies to adapt to new communication habits.

And, lo and behold, if you read the Le Devoir article until the end, you’ll find this box outlining the methodology:

Méthodologie

Le nouveau sondage Web de Léger a été réalisé auprès de 1010 Québécois ayant le droit de vote du 24 au 28 août, alors que la campagne était commencée. Par comparaison, un échantillon probabiliste similaire aurait une marge d’erreur d’environ plus ou moins 3%, 19 fois sur 20. (emphasis added) ³

So the “problem” with this poll is not that it uses a methodology now deprecated in our smartphone world.

Disagreeing polls

Marc-Antoine Berthiaume raised a much more pertinent issue on Twitter:

Comment expliquer que chez @leger360, pour les 18-34 ans, #QS arrive en 5ème place avec 8% et que @MainStResearch place en 2ème position avec 23,4%? C’est un écart de 15,4%!⁴

He thus contrasts the Le Devoir chart based on Léger’s polling data with data from Mainstreet’s Baromètre élections 2018. This tool is funded by Groupe Capitales Médias, a conglomerate of Power-Corporation-subsidiary Gesca’s French-language dailies sold to Martin Cauchon.⁵ There is a paid subscription service for individuals.⁶

Too Close To Call’s Bryan Breguet replied:

Facile: tailles d’échantillon petites. Donc variance est grande⁷

but quickly added:

Cela étant dit la différence est un peu grande ici, je l’avoue⁸

Let’s take a closer look by placing the two datasets side-by-side:

weighted N sizes=	Mainstreet before distribution	Léger9 after distribution	Difference
	602 (out of 2,350)	165 (out of 1,012)
CAQ	25.3%	26%	+1
QS	23.4%	8%	-15
Liberals	22.1%	35%	+13
PQ	12.1%	16%	+4
Greens	4.7%	9%	+4
CPQ	1.2%	3%	+2
NDPQ		3%	+3
Others	1.6%	0%	-2
Undecided	9.7%

What jumps out at first is that Mainstreet data is provided without the undecided having been allocated to parties since it contains the share of undecided young voters. Léger always provides sub-sample data after having allocated the undecided.

It’s therefore normal that Léger percentages are higher than those in Mainstreet: the sum total of voting intentions in Léger is 100% while it’s 90%¹⁰ in Mainstreet. That’s what explains all the blue in the right-hand column (that shows the difference between both datasets).

Incidentally, we can assume that the presence of 1.6% of young voters who intend to cast their ballots for another party in Mainstreet and their absence in Léger is compensated by the presence in the latter of 3% of young voters who intend to back the NPD Quebec. In other words, it’s likely that a fair share of those who would vote for “another” party in Mainstreet would in fact vote for the NPD Quebec.

But that’s not what shocked the interwebz. By bringing together the two datasets and ordering the parties according to their score in Mainstreet, we immediately see where the polls disagree: on voting intentions for Quebec Solidaire and for the Liberals (still with voters aged 18 to 34).

Alexandre Blanchet, a political science Ph.D., offers a convincing demonstration of the uncertainty inherent to polls in his French-language guide to polling for journalists and other geeks. (Just a heads up: I went on the page twice, and it appears to have brought my Internet connection down for a minute both times.)

la bonne question à se poser n’est souvent pas de savoir quel sondage est meilleur qu’un autre, mais plutôt de savoir de quelle réalité il est le plus probable que ces sondages émanent. Les sondages sont une manifestation de la réalité qui nous intéresse. Ils en sont une manifestation plus ou moins précise, et parfois plusieurs réalités différentes pourront être cohérentes avec les sondages que nous observons. Avec le scénario de l’élection de 2003 où la réalité était claire et nette, nous avons obtenu des sondages qui étaient eux aussi très clairs: le PLQ menait, le PQ était deuxième et l’ADQ était troisième. Avec le scénario de l’élection de 2012, où les intentions de vote étaient beaucoup plus serrées, plusieurs réalités étaient concordantes avec les sondages que nous obtenions.¹¹

Indeed, polls can change even if the underlying reality hasn’t itself changed.

Until I get a chance to dive into the latest Léger poll, I encourage you to read the delightful write-up in The Gazette which comes to the conclusion that:

many Quebecers are still indulging in a favourite pastime, which is to vote strategically¹²

Notes

1. Baillargeon, Stéphane. “Sondage: le PLQ et la CAQ champions des jeunes.” Le Devoir. 28 August 2018.
2. Third item in Stryker, Sam. “26 Times “The Osbournes” Were The Funniest Family On TV.” BuzzFeed (blog), 18 November 2015.
3. Léger conducted its new web-panel survey of 1,010 eligible Quebec voters from 24 to 28 August, when the campaign had already started. For comparison purposes, a similarly-sized probability sample would have a margin of error of around more or less 3%, 19 times out of 20.
4. How do you explain the fact that in Léger, for voters aged 18 to 24, QS gets in 5th place with 8% while it places 2nd in Mainstreet with 23.4%? That’s a 15.4% difference!
5. Le Soleil (Quebec City), La Voix de l’Est (Granby), La Tribune (Sherbrooke), Le Nouvelliste (Trois-Rivières), Le Droit (Ottawa and the Outaouais), and Le Quotidien (Saguenay).
6. Noreau, Pierre-Paul. “Suivre les intentions de vote en temps réel!” Le Droit. 14 August 2018.
7. Easy: small sample sizes. So greater variance
8. That being said, I’ll admit that the difference is rather large in this case
9. “Provincial politics in Québec.” Léger, 29 August 2018, p. 4.
10. 100%-9.7% of undecided young voters = 90.3% decided or leaning young voters
11. The right question to ask is often not which poll is better or worse, but from which reality it is most probable that the polls emanate. Polls are manifestations of the reality in which we are interested. They can be more or less precise as a manifestation, and sometimes a multiplicity of realities will be coherent with the polls that we observe. In the 2003 election scenario in which reality was clear cut, we got polls that were also very clear cut: the Liberals led, the PQ came in second and the ADQ was third. With the 2012 election scenario, in which voting intentions were much closer, many realities were in agreement with the polls we got. From Blanchet, Alexandre. “Sous le capot des sondages: Un petit guide pour les journalistes et autres geeks.” Alexandre Blanchet, PhD (blog), 23 August 2018.
12. Authier, Philip. “Quebec Election: CAQ Leading, but Liberals Still in the Game, Poll Shows.” Montreal Gazette. 29 August 2018.