The Loop That Never Closed
You recorded the decision. You never recorded what happened. That's why your knowledge base can't teach you anything.
In the summer of 2015, Philip Tetlock published the results of the Good Judgment Project — a four-year forecasting tournament involving 25,000 ordinary people making geopolitical predictions. The finding that got the most attention was that the best forecasters outperformed CIA analysts with access to classified intelligence. The finding that got less attention was why: not because the forecasters were smarter or better informed, but because they tracked their predictions and reviewed them against outcomes. The loop was closed. Feedback could happen. They improved.
Most people's decision records don't work this way. They capture the decision, and then stop. The outcome is somewhere in memory, or in a downstream email thread, or in some consequence you never formally connected back to the choice that produced it. The loop never closes. You have a record of what you decided, but no record of whether you were right.
The bias that makes this worse
Jonathan Baron and John Hershey showed in 1988 that people judge past decisions by their outcomes, not by the quality of the reasoning at the time. Show someone a medical decision that led to a good outcome and they rate the thinking as sound. Show them identical reasoning with a bad outcome and they call it reckless. Subjects in the study reported that outcomes shouldn't influence their evaluation — and then they influenced it anyway.
Annie Duke calls this "resulting": retroactively inferring the quality of a decision from the quality of what happened. The problem isn't just that it feels unfair. It's that resulting corrupts your feedback loop. When you reconstruct past decisions in light of outcomes you already know, you can't learn from them. You can only confirm whatever you already believed.
Tetlock's finding is the other side of this. His superforecasters got better because they didn't do it. They wrote down their probability estimates before outcomes were known, with a timestamp and enough specificity to be unambiguous. Six months later, they compared prediction to result, without editorial reconstruction. The process felt clinical. That was the point.
Most personal knowledge tools make both failure modes easy. They're designed for capture, not review. A decision you recorded in 2022 — about a hire, a vendor, a technical direction — sits in your knowledge base with exactly the same weight as one from last week. Confident, undated as a decision record, unreviewed. The note doesn't know it's stale.
What architecture teams got half right
Software teams invented Architecture Decision Records partly to solve this. Michael Nygard's 2011 formulation gave decisions a fixed structure: context, options considered, choice, consequences. A few ADR templates include a "confirmation" field — a way to note, later, whether the decision held up. Most teams skip it. The reason isn't laziness exactly; it's that someone has to set a reminder, return to the record months later, gather the relevant evidence, and fill in the field. That's real work competing with other real work, and it loses that competition almost every time.
The decision record gets written. The loop doesn't close.
What changes when the reader is patient
The bottleneck was always the reviewer. Someone had to come back.
An AI agent with access to your structured knowledge base is not subject to that bottleneck. It can scan your decision records, find ones past a target review date, look for related notes and project documents, and propose an outcome update — a patch you review and confirm or correct. The scanning and drafting is work the agent does. The judgment stays yours.
This only functions if the decision records exist as typed, structured entities, not as prose buried in a document. An agent reading a paragraph can extract a decision. An agent reading a typed decision record — with explicit fields for context, choice, and outcome — can find the field that needs filling and know what evidence to look for.
Harbor stores decisions this way. A decision record knows it's a decision record. It has a status. It has a review date. An agent that reads the workspace doesn't just find text that mentions a choice; it finds entities it can act on.
What the closed loop actually gives you
This isn't about real-time decision-making getting better. You won't suddenly get smarter at individual choices. What changes is calibration over time — the gradual recognition that some categories of judgment you're systematically wrong about, that certain heuristics work and others don't, that a particular kind of vendor relationship or technical bet has a track record you'd never have noticed without a record that told the truth.
Tetlock's superforecasters didn't outperform intelligence analysts because they had better information. They had something the analysts didn't: a system that held them accountable to what actually happened, without letting them quietly revise the story.
Most knowledge bases are full of confident conclusions. The uncertainty that preceded them is gone. The outcome that followed is never recorded. The loop stays open indefinitely — and your notes keep looking like wisdom when some of them were just luck.
Asgeir Albretsen is the founder of Harbor.