Believe Me
How can I persuade you that AI systems are really good at changing your mind?
By using an AI system, of course. At least that’s the takeaway from an intriguing paper about how much better LLMs are than humans in the art of persuasion. Or, as Jack Clark notes in his blog, summarizing the research:
The results are definitive: across four experiments involving 18,978 conversations across 6,923 people, AI systems are, today, better than humans at text-based persuasion with real world consequences…”
And that has profound implications for the health of the public interest information landscape. It’s not that chatbots will debate us to death, but more that we won’t know whether the systems we’ll use to access information in the future will be honest brokers or advancing someone else’s agenda. It raises real questions about who designs and controls those systems, and whether journalists should be focusing more time and energy on that debate. And it suggests that there are even bigger threats to worry about than AI-generated misinformation.
First, the paper. Researchers pitted machines against humans of various stripes — folks off the street, elite debaters, elite debaters who had coaching, professional fundraisers — to try and change people’s minds in a variety of scenarios. It wasn’t even close. The only time humans caught up with the machines was when the AI systems were throttled back so they could only answer in human-length answers, and at human speeds. (That indicates, perhaps, that the LLMs may not have been better at argument but more at volume of argument; but I suspect — see below — that there are other ways that machines have an edge on us flesh sacs of amino acids.)
The researchers — from the University of Oxford, UK AI Security Institute, Stanford University, and the London School of Economics and Political Science — point to possible consequences of this finding, depending on who controls the technology. “...power,” they note, “could flow to whoever can most readily access and deploy the most capable systems.”
If only powerful people and companies have it, they say, then we’re in a world where they use these systems to influence and potentially manipulate us; but if the technology is democratized, it could be that more marginalized communities have a better shot at getting their point of view across.
But it isn’t really a question of who has access to the technology; in many ways, we all already do. It’s really about who has access to that “last mile” connecting communities — individuals — to information. As I’ve written before, that’s the layer that controls — or at least influences — what news you get, what facts it includes, and how issues are framed.
News organizations have historically controlled that layer — for better and for worse — because we got to choose what to write about, what angle to take, and what language we used, and readers, by and large, had to live with our judgment, our worldview, our sense of what mattered. That world had its good and bad points, but it’s almost certainly going away, whether we like or not, and replaced by a world where AI systems will create news stories tailored to individuals, whenever they want it.
Who controls those systems?
In an ideal world, users do; the agents work on their behalf. Or failing that, at least an agent that prioritizes the health of the information ecosystem, however that’s determined. The last thing we want are systems that are really working on behalf of someone else, even if they seem to be working for us.
Which means that anyone who’s interested in the health of the information ecosystem — and that should include journalists — ought to be really interested in who builds that layer and how it’s programmed. But I don’t see a huge amount of focus among us news types on this issue; much of our efforts are spent on finding better, faster, more efficient ways to keep doing what we’re doing. Or on ways to get paid more for what we’re currently doing.
That’s important, but not as important as this.
We should be as concerned about the systems that determine what people see as we are about what information is available to them.
But surely, some might argue, our key role here should be just to focus on what we do best: finding facts, bringing insight, and making sure that those facts are carried through to the final product that readers see.
Yes. But.
That’s necessary but not sufficient. Persuasion is the process of marshaling facts, context, rhetoric and so on; it’s as important which facts you leave out, which way you frame the issue, as it is which facts you put in, and which strawmen you knock down. Even the most basic 250-word news story is the result of choices about what facts are relevant and what aren’t. Simply requiring an article to be factual doesn’t make it fair or balanced.
And even a “fair and balanced” article can be misleading, if it only presents one perspective, no matter how important or valid that perspective is. You can tell entirely true, entirely comprehensive stories about immigration from the point of view of beleaguered city services, or from the point of view of refugees fleeing oppression. Both stories are true, and both perspectives matter. Which one will a machine choose?
For that matter, which one would a human choose?
But there’s a difference: Right now, when a news organization chooses to cover a subject a certain way, we can all see what it’s published; there’s (usually) only one version of the story. In an AI-intermediated information landscape, everyone gets a different story. Who will — who can — audit what people see?
Which is why I’ve been much more worried about targeted persuasion than about AI-generated misinformation. True, there’s much to worry about when AI systems can generate a flood of false information and deep fakes. But those can be debunked, at least in theory. You can’t fact-check a misleading article that doesn’t have any false facts in it, but omits relevant information; you can only “frame-check” it, and that’s a much harder task.
It’s one reason I’ve tried to experiment with outputs that present multiple perspectives, and systems that can deconstruct stories. But these only work if they get in front of people who will use them.
So: Did I persuade you?
(Or perhaps I should have asked Claude to make this case instead.)
"anyone who’s interested in the health of the information ecosystem — and that should include journalists — ought to be really interested in who builds that layer and how it’s programmed. But I don’t see a huge amount of focus among us news types on this issue; much of our efforts are spent on finding better, faster, more efficient ways to keep doing what we’re doing. Or on ways to get paid more for what we’re currently doing."
YES. It's very confusing to me why the focus from the journalism side rarely touches how to build and support LLM output to be more helpful and relevant/factual for audiences. This is "journalism"'s problem as an industry: a focus on the doing of journalism by the journalist, rather than on the humans who want better, more relevant information (and will subscribe to products that give them better information).
That LLMs can persuade well may be a novelty problem (give it five years and run the test again). That journalists don't understand how to influence and build AI systems-- besides making more of the same old stuff-- is an existential threat to the 20th century thinking of the news business.