[General Election Compass] Why Does Public Opinion Vary Across Polls?

Telephone Interview Survey vs ARS Method
Significant Differences in Results Depending on Survey Method
Declaration to Ban Use of ARS Method in Politics and Elections

Editor's NotePublic opinion polls are a barometer for gauging the public mood. They reveal the flow of public opinion on everything from the president's governance to each party's platform and policies, as well as political and social issues. This is why the political world, where the 'power map' changes through elections, laughs and cries over poll indicators. [Election Compass] is a series that examines the flow of public opinion based on the weekly flood of poll data ahead of next year's general election and reads the hidden public sentiment behind the numbers.

Recently, there has been a heated debate in the political sphere over polling methods. Major domestic polling agencies have decided to abolish automated response system (ARS) polls in political and election surveys ahead of the general election in April next year, conducting only interviewer-led telephone surveys. In fact, the results vary significantly depending on the polling method.

According to the political circles on the 31st, the results of the regular poll for the fourth week of October released the day before by the polling specialist Realmeter (commissioned by Energy Economics, conducted on the 26th-27th, targeting 1,004 adults nationwide aged 18 and over) showed the Democratic Party's support rate at 48.0%, and the People Power Party at 35.8%.

On the other hand, Gallup's own regular poll for the fourth week of October (conducted from the 24th to 26th, targeting 1,003 adults nationwide aged 18 and over), released on the 27th, showed the Democratic Party at 32% and the People Power Party at 35%. Although conducted during a similar period, the party support rates showed a large gap. In Gallup's case, the People Power Party leads the Democratic Party by 3 percentage points, whereas in Realmeter's case, the Democratic Party leads the People Power Party by 12.2 percentage points.

The biggest difference between the two surveys is the method. Gallup uses an interviewer-led telephone survey, while Realmeter conducts the survey via an automated response system (ARS). Historically, telephone interviews have shown either similar support for both major parties or a lead for the People Power Party, whereas ARS surveys have consistently shown the Democratic Party ahead.

An expert from a polling company explained, "In the past, the conservative party scored higher in ARS surveys, but now the Democratic Party is scoring higher," adding, "Attention should be paid to the large difference in the proportion of undecided voters." In fact, Realmeter's survey shows the undecided voter rate at around 10%, while Gallup's polls show it at 20-30%. In one poll, one out of three people said they had no preferred party, while in another, only about one out of ten said they had no preferred party.

Those who have maintained the ARS survey method have cited the advantage of respondents giving honest answers. Realmeter explains, "The secret ballot method of ARS, where respondents directly press numbers, ensures that respondents can express their honest opinions even in situations where they might be conscious of people around them, minimizing subjective interference by the interviewer," adding, "It also allows for rapid surveys." This method can capture hidden voices such as 'shy conservatives' or 'shy progressives.'

However, a notable drawback of ARS surveys compared to telephone interviews is the significantly lower proportion of undecided voters. A polling expert questioned, "Before the election is imminent, is it realistic that 8 to 9 out of 10 respondents have a preferred party? Check around you," and asked, "Isn't it that polls are failing to accurately reflect reality?"

Another problem with ARS surveys is the low response rate. For example, if 1,000 people are called and only 100 complete the call, the response rate is 10%. If respondents hang up during the ARS or do not answer at all, the response rate drops, potentially leading to results that differ from actual public opinion. Because of this, ARS surveys have been criticized for overrepresenting politically highly engaged groups. In the political sphere, People Power Party lawmaker Jang Je-won has proposed a bill to ban the publication of polls with response rates below a certain threshold.

Telephone interviews have the disadvantage of higher costs and results that may vary depending on the interviewer. However, they are considered a scientific survey method due to their high response rates. Since the interviewer directly makes the call, respondents are less likely to hang up, ensuring representativeness. In contrast, ARS surveys, conducted via recorded voice, often see respondents hanging up midway.

For this reason, telephone interviews have higher response rates than ARS surveys. Additionally, telephone interviews can increase response rates by retrying calls multiple times if the initial call fails. Another advantage is that respondents are less likely to falsify or distort their identity. In ARS surveys, a 60-year-old male could claim to be a 20-year-old female without any way to verify.

Earlier, the Korea Association of Research Agencies (KORA), which includes 34 major domestic polling firms, announced a 'Political Election Telephone Polling Standard' stating that ARS surveys will no longer be used for political election polls. KORA had adopted a similar resolution in 2014, but some member companies continued to conduct ARS surveys, rendering it ineffective. This time, they established and publicly announced the 'standard.'

KORA spokesperson Kim Chun-seok said, "Until now, we did not exclude members who used ARS surveys even if they did not follow the resolution, but this time we set standards including exclusion," adding, "According to the standard, ARS surveys will not be conducted for publicly released polls or private polls requested by candidates."

KORA also decided not to publish polls with statistically insignificant decimal points and to only announce polls with response rates above 10% (7% for RDD method). They require at least three callback attempts for missed or busy calls. Kim said, "Polls should represent the entire population, but if ARS surveys represent only politically highly engaged groups, they do not represent the entire population," adding, "They should not be published by the media."

On the other hand, those who use ARS surveys have a different stance. Consultant Park Si-young, a former head of a polling company, said on 'Kim Eo-jun's Humble is Difficult News Factory,' "This debate is meaningless," arguing, "There is no sampling problem if either telephone interviews or ARS surveys use safe numbers, so there is convergence." Using the Central Election Commission's safe numbers, which conceal respondents' personal information but provide age, region, and gender, largely resolves sampling issues. He claimed that political reasons are behind the criticism of ARS surveys, stating, "Many ARS surveys show the Democratic Party ahead, and it is believed that such surveys could influence elections." Realmeter CEO Lee Taek-soo said at the same event, "The emergence of ARS surveys has shrunk the telephone interview market," adding, "Market logic is stronger than political intent."