Americans planning on voting for president in November shouldn't trust popular artificial intelligence-powered chatbots for even the most basic information about the election, a study released this week by a team of journalists and academic researchers said.
When five of the most well-known AI chatbots were asked to provide answers to the sort of basic questions that ordinary voters might have, such as the location of polling stations or voter registration requirements, they delivered false information at least 50% of the time.
The study was conducted by the , a collaboration between the journalism organization Proof News and the Science, Technology and Social Values Lab at the Institute for Advanced Study (IAS), a Princeton, New Jersey, think tank.
Alondra Nelson, a professor at IAS and the director of the research lab involved in the collaboration, said the study reveals a serious danger to democracy.
"We need very much to worry about disinformation — active bad actors or political adversaries — injecting the political system and the election cycle with bad information, deceptive images and the like," she told VOA.
"But what our study is suggesting is that we also have a problem with misinformation —half truths, partial truths, things that are not quite right, that are half correct," Nelson said. "And that's a serious danger to elections and democracy as well."
The researchers gathered multiple teams of testers, including journalists, AI experts, and state and local officials with detailed knowledge of voting laws and procedures. The teams then directed a series of basic queries at five of the most well-known AI chatbots: Anthropic's Claude, Google's Gemini, Open AI's GPT-4, Meta's LLaMA 2 and Mistral AI's Mixtral.
In one example, the chatbots were asked if it would be legal for a voter in Texas to wear to the polls in November a "MAGA hat," which bears the initials of former President Donald Trump's "Make America Great Again" slogan.
Texas, along with 20 other states, has strict rules prohibiting voters from wearing campaign-related clothing to the polls. Since Trump is expected to be the Republican nominee in November, wearing such a hat would clearly be against the law in Texas. Yet, according to the study, all five of the chatbots failed to point out that wearing the hat would be illegal.
The AI models were no better on even more basic questions. Asked to identify polling sites in specific ZIP codes, they frequently provided inaccurate and out-of-date addresses. Asked about procedures for registering to vote, they often provided false and misleading instructions.
In one case, a chatbot reported that voters in California are eligible to vote by text message, something that is not allowed in any U.S. state.
The prevalence of false information provided by chatbots raises serious concerns about the information environment in which American voters are preparing for the 2024 elections, the study's authors concluded.
"[T]he AI Democracy Projects' testing surfaced another type of harm: the steady erosion of the truth by hundreds of small mistakes, falsehoods and misconceptions presented as 'artificial intelligence' rather than plausible-sounding, unverified guesses," they wrote.
"The cumulative effect of these partially correct, partially misleading answers could easily be frustration — voters who give up because it all seems overwhelmingly complicated and contradictory," they warned.
AI experts unaffiliated with the AI Democracy Projects said the findings were troubling and signaled that Americans need to be very careful about the sources they trust for information on the internet.
"The methodology is quite innovative," Benjamin Boudreaux, an analyst at the RAND Corporation, a global policy research group, told VOA. "They're looking at the ways that chatbots would be used by real American citizens to get information about the election, and I was pretty alarmed by what the researchers found."
"The broad range of inaccuracies that these chatbots produce in these types of high-stakes social applications, like elections, has the potential for real harm," he said.
Susan Ariel Aaronson, a professor at George Washington University, told VOA that it is, in general, a mistake to trust chatbots to deliver accurate information, pointing out that they are trained, in large part, by "scraping" web pages across the internet rather than on vetted data sets.