Whenever I used to perform some deep research task, especially to learn about some topic, there was a moment in the process where “it clicked!”. The exact time when I could see it clearly in my head and it made sense from that point on.

However, when reading or listening to the AI-assisted digests I didn’t experience that moment. Now I know it has a name, a reason and more importantly a suggestion on how to improve the experience.

A few days ago I wrote about how I learn now, tens of agents on a topic, a document to read and a podcast for the car. But it left me with a question: if the machines do the finding, the comparing and the explaining, what’s left that has to happen inside my own head?

tl;dr what’s left is retrieval, pulling the thing back out of your own head without looking. Here’s what the papers say, and the small tool I built to try to put it back:

Why the click was missing

Retrieval is the one thing a very good digest takes off my plate, and that’s the whole problem. The sharpest demonstration of it is Roediger and Karpicke, two prose-learning experiments where the people who got tested on the passage beat the people who restudied it on the delayed test, even though restudying is what made them feel more confident. In those experiments how well something reads and how much of it survives came apart, and I’m guessing a great digest maxes out the first one.

Bjork and Bjork put that in a family they call desirable difficulties, things that make practice harder now and can support learning later, together with spacing, variation and interleaving. The part people drop when they quote that chapter is the condition: difficulty only pays when the learner has the prerequisites and gets feedback, so a bad question, or a good one at a terrible moment, is just friction.

Two more I use as guardrails. Cepeda and colleagues synthesized the verbal recall literature and found the useful gap between repetitions depends on how long you want to remember the thing, so “come back later” is supported. And Barenberg and Dutke found retrieval practice was associated with better calibrated confidence a week later, in secondary students reading short texts.

All of that is students and prose passages, not an engineer reading about attestation formats, so I’m not going to pretend it transfers 1:1. “Cognitive friction” is my own phrase for what I did with it, it isn’t a finding.

AI doesn’t have one learning effect

I wanted the papers to say “AI is bad for learning” so my little project would have a villain, but they don’t.

Bastani and colleagues ran a field experiment in high school mathematics where unguarded access to a model improved the assisted practice and made the later unaided exam worse, while a guarded tutor version mitigated that. One population, one short-term exam, and a 2025 correction that only fixes an author’s affiliation. But Kestin and colleagues ran a trial with 194 undergraduates in physics where a tutor deliberately designed for teaching beat active classroom instruction on what they measured. One course, one tutor design, and whatever that assessment caught. Same technology, opposite direction! The difference is the design wrapped around it and what the exam measures.

One of these shaped what I built, the other one just sits in the back of my mind. Buçinca and colleagues tested cognitive forcing with 199 participants on AI-assisted decisions, making people commit before the AI’s answer showed up reduced overreliance compared to just showing them an explanation, and they’re upfront that people liked those interfaces less. That’s decision support and not learning, but it’s where the order of my quiz comes from. And Lee and colleagues surveyed 319 knowledge workers and found more confidence in GenAI was associated with less self-reported critical-thinking effort, an observational survey, so an association and not a cause, but it’s the shape of what I’m trying to catch in myself.

The one that keeps me honest is Risko and Gilbert on cognitive offloading, basically that pushing work onto something external lowers the demand on you and can be a useful thing to do, though they also say the judgement about when to offload can itself be off. When I’m shipping software I want Claude Code holding the details, for example the exact flags of a command I’ll run once, I’m not trying to learn them. Offloading only costs me something when learning is the goal and I skip the part where I’d find out I can’t do it alone.

What I built

So I built a small tool called Flint, it only runs on my machine for now. The name comes from striking a spark through contact, which is a bit grandiose for a quiz window haha.

The loop is narrow on purpose. Nothing counts as read until I say so in writing, about an exact revision of an exact file. Not opened, not imported, not summarized by an agent, read by me. This is the read record for the research seed behind this post, straight out of the repo:

"read": {
  "status": "unread",
  "sha256": null,
  "attested_at": null,
  "actor": null,
  "provenance": null
}

Questions come from a bank a human approved, tied to those same exact revisions, with a deterministic key, so scoring is arithmetic and never a model judging my answer. One item, trimmed:

{
  "id": "synthetic-q2",
  "objective_id": "concept",
  "kind": "application",
  "prompt": "Which proposed action is consistent with the source in item two?",
  "sources": [{"document_id":"synthetic-document","revision_id":"synthetic-revision","sha256":"6911aec2...","quote":"Synthetic source says yes."}],
  "key": {"type":"single","options":["affirmative","negative"],"correct":"affirmative"},
  "validation": "reviewed",
  "provenance": "synthetic manual review"
}

Then the interruption. While a coding agent is busy with a long foreground command, a separate local window can offer 4 questions. The scored order is the answer first, the confidence number second, the explanation only after both, which is the Buçinca idea and the Barenberg one glued together. There’s a reveal button too, but using it marks the attempt assisted, so it can’t score:

The quiz turns up on its own while Haiku 4.5 is mid-command. A demo topic, not my research

4 items at 25 points each, a valid unaided score under 75 reopens the topic, exactly 75 passes, and a bad result never erases the reading history. Same attempt after submitting it:

50/100 reopens the topic, and every item points back at its source revision

Flint works with both Claude Code and Codex CLI, and it fails open, when that local quiz window isn’t running the plugin does nothing and my work carries on.

The 75, the 4 items, the window timing and the revisit intervals I sketched are product choices I made, and no paper I read validates any of them (yes, I know how that sounds).

Sadly none of that tells me whether I actually learn more, it’s all evidence about the tool. I haven’t run the pilot on myself, so I don’t have a score and I haven’t done the delayed recheck, and the research seed for this post, the one in that first fence, is still marked unread, so yeah, I built the thing that quizzes me on my research and then never ran it on my own research haha.

Conclusions

So what I have is an attempt at putting back what the digest removed. My best bet is that a small unaided attempt, plus a confidence number I commit to up front, tells me whether the digest landed or only read well, and it might do more for knowing what I don’t know than for memory. I’ll write about it when/if the pilot tells me something.

The thing is, for me, it’s either learning by using AI-assisted deep research digests or nothing. I don’t have the time to do it “by hand” anymore. So this method is the next best thing, I think. ;)

As always, if you spot something that is incorrect, let me know (@ivMobAppSec).