Process Over Persona

Or, getting beyond cosplaying.

I confess I’m not a fan of prompts that tell an LLM that they are something or someone — “You are a news assistant,” or “you are a book agent focused on non-fiction titles” — in the expectation that that will help improve the system’s output. I’m sure it will make the LLM sound like a literary agent, but does it know what an agent actually does? That’s sort of like turning to George Clooney for medical advice just because he played a doctor in the TV series ER. (Although, for the record, Mr. Clooney can treat me for anything, anytime.)

What LLMs know, versus what they can convincingly mimic, is a critical difference. As I learned when I set two systems up against each other as skeptical editors, only to discover, as Claude told me, “AI is doing something more like “reasoning by analogy to editorial work I’ve seen” than “executing a well-defined editorial process.” In other words, it knows how to look and sound like an editor, not actually be one.

For that, you need much more process and structure. You have to deconstruct what an editor does and how they think, and encode that in a system.

Which is what I did.

I spent a couple of days with Claude talking through the process of reading and deconstructing a story, discussing the steps an editor might take to assess the strengths and weaknesses of its evidence and arguments, and working out repeatable procedures to stress-test its hypotheses until we came up with something we could code and run. Then we took that and turned each part into what Claude calls a “skill” and chained them into an “artifact,” a reusable bot.

This is increasingly what the core skill for working with AI is: it’s less about writing better broad prompts and more about putting the effort in up front to dissect processes into discrete chunks that the system has to follow in sequence; in effect, bringing some order to an inherently probabilistic machine.

The results from the exercise weren’t surprising, per se, but they were instructive. It helped me see what I take for granted as a human — or, mostly human — editor, and how the specific processes and “skills” we (Claude and I) encoded can form the basis of broader news tools and products. And what complementary advantages both humans and machines bring to the table.

It’s a long-ish post; I apologize; you can skip past the details and get to the TL;DR end a couple of paragraphs from the bottom if you’d prefer.

The tool is a more complex version of my original deconstruction bot, which used Claude’s basic language handling skills to extract a story’s main thesis, the facts and assumptions or analysis cited to support that thesis, and the core assumption underpinning the piece. That one worked pretty well — you can read my post — but I wanted to see if I could do better.

This new tool takes Claude through a much more rigorous — and systematic — process:

• First it looks for the stated thesis, or what the story says it’s about

• Then it looks for the “argued thesis,” or what the story really seems to be trying to say.

• And explains why it sees a difference:

  • For example, a story about a mayor who seems to own more $200,000 Rolex watches than he can afford may be ostensibly about his choices of wristwear, but it’s probably really trying to make the case that he’s corrupt

Once it has that:

• It looks for the key facts asserted in the story that support that thesis, and catalogs the level of sourcing they have

  • For example, a named person or report, or an unnamed person, or if it’s just asserted)

• It does the same for assumptions in the story, and whether they are explicitly called out or simply implied

• Then it looks for the key analyses in the piece that support the thesis, and tries to assess if there is a complete chain of reasoning for the analysis laid out in the story

• Finally, it looks to see if the thesis would hold if each of the core assumptions or analyses turned out to be false. (That’s how it identifies the most central — ”load-bearing” — assumptions in the story)

The whole process is pedantic as hell — but also fascinating to watch.

But wait; there’s more!

• Next, it will run a module, based on those previous findings, to see what other framings the same facts and assumptions might support

• Then it can — on request — generate separate notes:

  • To the reporter/writer, suggesting ways to improve the piece

  • To an editor, noting holes that need to be addressed

  • To a journalism professor, offering ideas for using the story as a teaching example

  • And to the public, making explicit what the piece seems to be about, and what it expects readers to take on faith.

The idea is that there’s a common set of information — the thesis, facts, assumptions, analysis etc — that we can now take and build new structures, new analyses, new outputs on top of.

It’s a lot. The system kept timing out, and if I was building this for real, I’m sure I would have built it more efficiently.

(Claude is very good, for example, at finding redundancies and bloat in code — if you know how to prompt it correctly.) But then again, if I was a real developer, I would have a real job.

I tested this on a couple of stories — two that I’d already run systems on, and another, short piece. There was a New York Times piece analyzing how and why the Supreme Court ruled against trans care for minors in the Skrmetti decision, an issue I’m very personally invested in. That was the one I had run my previous deconstruction bot on, and I had no issues with its findings. There was another NYT story about misinformation and AI in the wake of the shootings of US citizens in Minneapolis, which I had sicced dueling bots on, and I had no issues with those findings, too. And finally, a short Guardian piece about California Gov. Gavin Newsom backtracking on criticism of Israel.

There isn’t enough space to show you all their output, but here are some examples of the analyses it did and notes it wrote that struck me:

In the Skrmetti story, for example, it identified as an empirical, structural premise that “The Supreme Court defeat was caused by the ACLU’s strategic choices (politicized evidence, maximalist theories) rather than by the Court’s composition, ideology, or pre-existing disposition on trans rights” — meaning that’s something that could have been reported out (empirical) and that the piece has baked into its framing (structural) and is foundational to the argument of the piece (premise.) But that isn’t explicitly stated.

It goes on, when looking at the core analysis in the story, to conclude that:

… if the medical consensus was genuinely politicized AND the Court would have ruled against trans rights anyway, the thesis survives but loses its ‘catastrophic error’ edge—it becomes ‘the ACLU fought with bad ammunition but would have lost regardless.’ But if either the consensus was sound OR different tactics could have won, the combination collapses the thesis entirely.

Its note to the public, (i.e. normie readers), says:

What this story is actually arguing:

This story argues that transgender rights advocates, especially the ACLU, made a catastrophic strategic mistake by aggressively pursuing lawsuits to protect access to puberty blockers and hormones for minors. According to the story, they ignored both public opinion (which opposed these treatments even among Democrats) and emerging scientific evidence from Europe showing the treatments had weak support. This overreach led to a Supreme Court defeat that now threatens all transgender rights. The story presents this as a case study in how social movements can become so ideologically committed that they make self-defeating choices.

What it’s asking you to accept:

The story asks you to accept several things without proving them. First, that the Supreme Court ruled against trans rights because advocates pushed too hard, rather than because the conservative Court majority was simply hostile to these rights regardless of strategy. Second, that the European medical reviews represent the best science, while the evidence that American medical organizations relied on was politically motivated rather than a different but legitimate interpretation of contested research. Third, that a more cautious, compromise-based approach would have worked better—though the story provides no evidence this alternative was even possible or would have succeeded.

Can’t say I disagree.

For the Minnesota misinformation piece, which the system tagged as arguing that AI was supercharging misinformation, it generated this note for the reporter, under the title, “is this what you meant to argue?”

You’re arguing that AI misinformation plus social media vigilantism caused innocent people to be harassed, revealing a fundamental breakdown in how people verify information. But here’s the disconnect: your evidence shows an AI image appeared and two Steve Groves were harassed, but nothing connects those two things. The story seems to assume the AI image drove the misidentification, when it’s equally possible people just Googled ‘Steve Grove Minneapolis’ the old-fashioned way and landed on the wrong person. If that’s what happened, this is a story about name confusion, not AI breaking our information ecosystem.

It could have been more polite, I guess, but it does make a fair point

And on the Newsom story, it’s fairly pedantic but also offers a useful lesson in what we — journalists, news junkies — take for granted. For example, it notes that an assertion that the governor plans to run for president is entirely unsourced. Well, sure. But not wrong, I suppose to call that out. It also — again, reasonably pedantically — flags the broader question: how do we know the conventional wisdom framing, that he’s walking back politically inconvenient statements in the face of backlash, is more accurate than a more innocent explanation?

Or, as it put it in a note to journalism educators:

When covering political statements and subsequent clarifications, what’s the journalist’s responsibility to distinguish between a politician changing positions under pressure versus refining an imprecise original statement? Both are newsworthy, but they’re different stories — and without additional reporting beyond the public statements themselves, how do we avoid importing our own interpretation as fact?

Perhaps that’s something any seasoned political journalist would dismiss quickly — and probably accurately. But that’s where the different — and complementary skills — of both human and machine come into play. Humans can bring context, experience and judgment to play much more quickly than machines can; but machines can be much more systematic, don’t necessarily have the same blindspots and are tireless — even if they can flag what seem like obvious questions.

It’s the combination that could bring out the best in both; not by trying to make the machine cosplay a human, but by specifying and embedding — and requiring — a dogged set of procedures to make it more systematic and predictable.

More, well, machine-like.

(PS: Of course I ran this post past the deconstruction bot. It told me I shouldn’t depend on Claude to tell me about Claude (fair), that I had cherry-picked three examples to prove my premises (well, yes), and that I hadn’t done any serious QA testing on the system to give confidence to my findings (guilty.))

But then again, if I was a real developer, I would have a real job.