2023-96: Di Millo v Toronto Star

Download Complaint PDF

April 5, 2024 — for immediate release

The National NewsMedia Council (NNC) has considered and dismissed a complaint about accuracy and context in a November 1, 2023 article, “Tories hold lead over Liberals, Canadians report limited trust in institutions: poll,” published by the Toronto Star.

The complainant, Gus Di Millo, filed a complaint stating so-called ‘opt-in’ polls should not be used by media outlets as an accurate reflection of public opinion. The complainant argued these types of polls are not assigned a degree of accuracy and are subject to manipulation by special interest groups who may be promoting the polls.

The complainant stated news organizations should pay to run accurate randomized polls as proper gauges of public opinion.

As the complainant filed the complaint on the same day as the article was published, they were referred to the Star’s public editor for review, as is standard NNC practice.

In their response to the complaint, the news organization stated that while online polls cannot be assigned a margin of error, they are regarded as an accepted industry-standard method of polling. The news organization also stated that the research firm responsible for the poll takes appropriate steps to ensure results are properly weighted against census data and controls are in place to avoid bias.

The complainant was unsatisfied with the news organization’s response. He reiterated that the Star’s editorial policies for reporting on polls, reports, and studies stipulate that the news organization does not publish polls that do not contain a margin of error. Furthermore, he pointed out that the news organization should publish additional information about the individuals or organizations sponsoring the poll.

In a subsequent round of responses, the news organization indicated it would be updating its polling policies in the near future to clarify any methodological complexities.

The complainant reiterated that the news organization should eliminate the use of non-probability polls.

In its review of the materials submitted by both parties, Council considered, specifically, whether the presentation of the poll breached the news organization’s own editorial policies and, more generally, how news organizations use public opinion polls.

To the latter issue, news organizations often cite public opinion data gathered from polls as a means to provide readers with a broad understanding of public opinion on a particular issue, or question, relevant to the public interest.

That said, best practices support the view that news organizations should indicate if a poll was sponsored by a third-party organization, or special interest group, as this may have an impact on the questions being asked of respondents.

At the same time, Council noted that the use of this empirical data is designed to be used as a snapshot of public opinion at a particular time. Citing data from polls in news articles, generally speaking, is not meant to involve quantitative statistical methods often employed by academics in the physical or social sciences.

In considering this specific complaint, Council reviewed the article in light of widely-accepted polling methods and the relevant sections of the Canadian Press Style Guide related to reporting on public opinion polls. This guide was consulted because the original story was published by the Canadian Press.

In noting the increase in online surveys, the CP Guide states that all stories based on online polls should include the following sentence:

“The polling industry’s professional body, the Canadian Research Insights Council, says online surveys cannot be assigned a margin of error because they do not randomly sample the population.”

In its review of the Toronto Star’s own editorial guidelines, Council also considered the specific criteria set out for how to report on polls. For example, the guidelines state:

…we must give readers full context: names of the sponsor and the polling agency, population from which the sample was drawn, sample size, margin of error, type of interview (telephone, mail, online, in-home), dates when the poll was conducted and wording of the question on which the story focuses.” 

With these criteria in mind, Council noted that the news organization clearly identified the polling firm, the total size of the survey sample, the means through which the survey was conducted, and its date.

The news story also describes the limitations of the sample, namely that it was an online poll and, therefore, margins of error could not be properly assigned.

Council also considered the complainant’s concern that not citing a specific sponsor indicated a breach of standards.

In this case, while the NNC agrees that sponsorship may be relevant information for readers in cases where a special interest group has commissioned the poll, Council found no grounds in its review of the complaint to indicate the poll was undertaken by a sponsor with a special interest that may alter readers’ understanding of the results, or that failing to cite a specific sponsor in this case misrepresents the results of the poll.

The NNC appreciated the complainant’s arguments that consumers of news and information should be skeptical of data presented without appropriate context in news articles. This form of critical thinking, and active readership, is important. That said, presenting engaged readers with appropriate context may avoid the pitfalls that the complainant describes. The NNC understands that no polling method is without limitations.

The NNC likewise appreciated the Toronto Star’s acknowledgement that their editorial policies would benefit from updates to reflect modern methodological challenges and communicate important information to readers.

That said, the NNC finds no grounds to support the view that reporting on a poll without a margin of error is in violation of the news organization’s guidelines, so long as the relevant context is provided to readers.

While the NNC supports the news organization’s plans to update its guidelines to account for a range of current polling methods, the article’s inclusion of relevant context, particularly the fact that no margin of error could be assigned and the reason why, aligns with journalistic standards.

For this reason, Council dismissed the complaint.