Preference Systems That Actually Get Used

The first time you open most apps, there's a preference screen. It asks what language you speak, how you want notifications delivered, whether you'd like a light or dark theme. You click through it in about thirty seconds. You never open it again.

The rest of your "preferences" — what you actually care about, what you want the tool to remember, how you actually want it to behave — never get entered at all. They exist somewhere in your head, briefly activated every time the app does something slightly wrong, and then submerged again.

Studies on default settings find that around 95% of users accept them unchanged. That's not because people don't have preferences. It's because filling out a form about your future behavior is genuinely hard. You're not sure what you want until you encounter a situation. By the time you're in the situation, the moment to enter the preference has passed.

The gap between stated and revealed

There's a term from economics for this: revealed preferences versus stated preferences. Revealed preferences are what you actually choose when given options. Stated preferences are what you tell someone you'll choose in advance. The gap between them is large and consistent.

Netflix took this seriously early. They ran elaborate experiments with five-star rating systems, asking users to tell them what they liked. Then they compared those ratings against what people actually watched. The fit was poor. What users said they enjoyed watching didn't predict what they'd click on Friday night. So Netflix largely moved away from explicit ratings and toward behavioral signals — what you started, skipped, finished, rewatched at 2 AM. That data turned out to be far more predictive than anything a user would type into a preferences screen.

The uncomfortable implication: you probably don't know your own preferences as well as your behavior does.

This creates a real design problem for knowledge tools. If you build a preference system that requires deliberate human input, you'll get sparse, stale data that reflects what users think they prefer rather than how they actually work. Most "settings" screens in productivity apps are graveyards of aspirational choices made during onboarding.

What AI changes about this

There's a version of this problem that AI can actually help with. Not by asking you to fill out forms — but by watching what you do and updating your preference records from observed behavior.

You tell an AI you want meeting notes formatted a certain way. It formats them. The next time, you edit the format slightly. The AI could notice that, propose an update to your preference record: Based on how you edited this, should I update your default meeting note format? You approve. The preference now reflects how you actually work, not how you imagined you'd work when you signed up.

This converts preferences from a one-time input problem into a continuous refinement problem. The preference file isn't something you fill out — it's something that accumulates from your actual behavior, with AI as the observer.

Richard Thaler and Cass Sunstein, in their 2008 book Nudge, argued that the design of default options is itself a form of influence — choice architecture that shapes outcomes without restricting them. Who sets the defaults, and to what end? Applied to AI systems, the same question becomes: who updates your preference record, and how?

Why you still need to see them

Here's where it gets complicated. An AI that silently updates your preferences from observed behavior is building a model of you that you can't inspect. At first this feels convenient. The tool gets better at predicting what you want. But the preferences it's learned from your behavior might not be the ones you'd endorse on reflection.

You format meeting notes a certain way because you're in a hurry, not because you think it's right. You always skip a certain type of notification because you haven't figured out how to route it properly, not because you want to silence it forever. Behavior is a noisy signal of preference. An AI that treats all behavior as signal will learn the wrong things with confidence.

The design that actually works is somewhere in between. Preferences get proposed — "I noticed you do X consistently, should I update your default?" — and then you approve, modify, or reject them. The preference record is stored somewhere readable, not inside the model, not in a cloud system you can't inspect. You can open it, see what it says about you, correct it when it's wrong.

Harbor stores preferences as structured blocks inside Markdown files. They're readable in any text editor. They can be updated by AI, but only through a reviewable proposal. The result is a preference record that's actually current — because it's updated from real behavior — but still under your control.

That's the version of preferences that might actually get used. Not a screen you fill out once during onboarding. A document that gets more accurate over time, because something is watching and asking.

There's something slightly uncomfortable about being known that well by a piece of software. But probably less uncomfortable than a tool that never learns anything about you — and keeps formatting your meeting notes wrong.

---

Asgeir Albretsen is the founder of Harbor.