We know AI is muddying traffic metrics, but newsletter metrics are also in its crosshairs. And we can already see it happening.
(I promise I won’t always post about metrics.)
Like Gina, I’ve been exploring AI tools here at the Tow-Knight Center. I’m particularly interested in how they can help and, conversely, disrupt our ability to gather and share information with the public. So I made one that consolidates all the newsletters we receive into a single, weekly digest. It opens every newsletter, but we don’t read them; an LLM does that for us and summarizes what it thinks we care about.
That’s great for us, but not so much for newsletter metrics. And it’s not just us; our own Substack subscribers include several email addresses like TKnewsletters@gmail(.)com, suggesting others are doing the same. And those are just the obvious ones.
More and more people will go this route, or one like it. We’re all drowning in information (and Substacks) these days. We all could use someone — or something — to go through it for us. But importantly, it’s not just that we want the system to sift and sort newsletters into piles we should read and piles we can discard; we want it to find the nuggets we care about in each of them, summarize them, bring them to our attention and, sometimes, find common threads across them. It’s not a sifting mechanism; it’s an understanding mechanism.
And it’s a lot easier to set up than you think.
You don’t need technical skills; AI automation workflows — easily made using platforms like Zapier, Make and n8n — can help you build your own digest quickly. I built ours using Make. It didn’t require technical knowledge, took about 30 minutes to set up, plus a little troubleshooting with a chatbot, and costs about $5 a year (if you want to try it yourself, instructions are at the bottom of this post). But more expensive tools for this kind of thing, like ChatGPT Pulse (currently for Pro users), already abound and are even more user-friendly.
How our digest works: Rather than everyone on staff individually subscribing to all the work-related newsletters they could, I set up a Tow-Knight Center Gmail account just for newsletters and subscribed it to everything we thought might have relevant mentions of our area of focus: the intersection of journalism and AI. Then I created a four-step AI workflow on Make.com. Once a week, it gathers all the newsletters received in the past 7 days, extracts the most relevant information to our interests, and summarizes it into a 5-minute digest that gets emailed to everyone on our team (OK, just me and Gina) at 7am on Tuesday mornings, with an analysis of patterns across them at the top.
This blows up newsletter metrics:
The workflow opens every email labeled “newsletter” in that inbox, but many of them rarely, if ever, show up in our digest. For example, the inbox is subscribed to all emails from Columbia Journalism Review, but only ones related to AI make our digest, and many weeks it excludes 3-4 CJR emails that come in. (Sorry, Betsy!)
We only have a single subscription to each newsletter, but it can email out the digest to an unlimited number of email addresses (well, as unlimited as Gmail allows). Right now our team is just two people, so we’ve only reduced subscriptions by half.
In some ways, it’s not that different from a listserv subscribed to a newsletter. But creating a personalized digest, and being able to continually tweak it over weeks, has changed our behavior as newsletter consumers: it’s increased our willingness to subscribe to things of only moderate relevance, decreased our interest in unsubscribing from newsletters that aren’t making the weekly cut (since we never have to see them), and could exponentially reduce the total number of potential subscribers to our newsletters, if we grow our team.
Newsletters typically include a lot of trackers — Substack allows us to see how many times a subscriber opened an email, which can give hints as to if an email is being forwarded, something I could have easily set up our digest to do for relevant stories. But what about if I instruct the digest to save the full text of the post in a spreadsheet and send that instead? Or strip the metadata from links that shows the traffic source was an email newsletter? All those take are a few small tweaks of the workflow.
Email marketers have been talking for a while about how open rates are becoming increasingly unreliable, along with recent concerns about AI summaries within email providers and the need to focus more on other metrics that reflect audience retention, like engagement. But our secondary inbox and digest affects that too. To use my CJR example (sorry again, Betsy) — their stats will show a single subscriber opening every email, who has never unsubscribed from anything they send. That’d look like a superuser in most newsletter metric systems we’re aware of.
Multiply us across their entire readership, and pretty soon their metrics become meaningless. (For the record: ours, too.)
Once again, the black box of AI-intermediation rears its scary head. How can newsrooms find new, and perhaps better ways of getting audience feedback about what matters to them in an AI age? Can we build systems that let their AI talk to our AI about what they found interesting in our newsletters? Will we even need newsletters, or just newsfeed filters that extract the parts of interest to readers at the source? How will anyone know what their readers care about?
And more broadly, what happens to news providers in a world where all their content is pre-read, pre-digested, pre-summarized and delivered to users nearly devoid of its original sourcing, branding and narrative? Even if it turns out to be a more useful product for readers.
The flip side
Bad news for newsletter writers (like us), but great news for readers (like us). If you’re in the news business, just reading the amount of newsletters and other information sources you receive on a daily basis likely amounts to a part-time job.
But we can’t inform others if we’re not informed. Enter your favorite LLM. Sifting through a series of documents (or emails), sorting them by relevance, summarizing and finding patterns across them, then delivering the digest in the manner and frequency of your preference is something generative AI is extremely good at. You’re not requiring it to be perfectly factual, you’re asking it to surface the information you most need from an overwhelming quantity. As we’ve discussed here, this is language synthesis — where a language model really shines.
Let’s talk common worries:
“How do you know the summary is accurate?”
Like you, we want to check our sources. This is where customizability really shines. I set up our digest to list items as a 1-2 sentence summary with headline, publication name and date, along with a link to the source newsletter, so if we see something we are interested in, we can easily get to the source.
“What if it left out something important?”
I’m concerned about that too, and I want to know how good our digest is at its job, so I can figure out how to improve it. I have it provide stats at the end of each digest, telling us how many newsletters were received that week, how many were included in the digest, and listing the ones that were excluded. And — frankly — I was never going to get around to reading many of those newsletters anyway, so this might actually improve the percentage of important things I see.
“I don’t want to give an AI access to my inbox.”
Then don’t. We’re not giving it access to our work email accounts. We made a separate email address just for this, which we are fine with giving Make and OpenAI (the AI platform we’re using for this workflow) access to. It only contains newsletters, preventing those companies from accessing our work email addresses and decluttering our work inboxes. Our work emails only appear in the “To:” line of the email sent out in the final step.
Is our digest perfect? No, but Gina and I are pretty happy with it, and it’s saving us lots of time. I set this up in December and have been making small tweaks to the summarizing prompt every couple weeks — mostly adjustments to tone, categories and summary length. It’s now delivering something we both reliably find useful and easy to read. Is it missing things? Probably, but our goal isn’t for this to catch everything, it’s to reduce the amount of relevant news we miss. And we think it’s much better at that than an intern (or a few) would be.
There are lots of variations to be done on this – direct RSS feed digests, podcast and YouTube transcript analyzers, even searching the wider internet to deliver the latest news from a variety of sources. You could save all the text to a Google Sheet or add translation from other languages as a part of the process (more on that in a future post).
But enough talk. If you’ve read this far, you’re probably curious to try it out yourself. So, I put a version of this on GitHub for anyone who wants to play around with it.
Don’t be scared by the word GitHub, if you’ve never used the site. Scroll down to the README text (the stuff that looks like a set of instructions with a flow chart, rather than unfamiliar technical words). There’s a little set up required, but all the instructions you need are there, along with troubleshooting tips and advice on how to get a chatbot to help you further if you get stuck. You can also always get in touch — I’d love to hear how it goes for you.
Why you should try our version of this: There are already more sophisticated alternatives of this out there; I’ve even named and linked a few here. But if you’ve never done much more with AI than ask a chatbot some questions, building your own workflow can help you understand the potential of directing large language models for your specific information synthesis needs. We are intentionally using newsletters — a small set of source material that you’re already familiar with — and Make — an automation tool with a free tier and a user-friendly interface that still allows for a lot of customization. Building and fine-tuning a simple version of this will likely give you ideas for other digests you want to make. And I’ll be sharing more tutorials soon.
There’s an irony in telling our readers to make digests of their Substacks. It will certainly disrupt our Substack metrics. But we’re ok with that. Though, if you do make a newsletter digest, we’d love to hear from you more directly, since your open rates may soon be useless to us.
While we wait to see if the future can bring better newsletter metrics for the AI age (or if it will kill the newsletter entirely), don’t be shy about telling us when you’re finding what we write helpful. Leave a comment on the post, on LinkedIn, or reply to this email directly.



