Most days, I work with capable agents.

The strange part is not that they forget.

That will get better.

The stranger part is where the remembering happens.

One agent helps me reason through a paper. Another edits code. Another has the meeting notes. Another knows the failed attempt.

Over time, each agent learns a slice of how I work.

That is useful.

It is also the problem.

If the record of my work lives inside one agent, then my process becomes part of that agent’s product.

The agent is no longer just helping me use my process.

It is becoming the place where that process lives.

The model is not the only thing that matters. The layer around the model matters too: the context it sees, the memory it keeps, the tools it can use, the skills it loads, the project instructions it follows, and the way it turns yesterday’s work into tomorrow’s default.

Call that the harness.

At first, that feels like the product getting better.

I do not think that anymore.

Now ask the question:

If I leave this agent tomorrow, what leaves with me?

Do the memories leave?

Do the lessons leave?

Do the workflows leave?

Do the review habits leave?

Do the conventions leave?

That is the ownership test.

The Repo Is Not Enough

The obvious objection is simple: put the context in the repo.

Yes.

For code, the repo should carry a lot: ADRs, docs, tests, comments, decision files, and project instructions.

If a decision belongs with the code, put it there.

But the repo is one view of the work, not the whole work.

It does not hold the client meeting, the cross-project habit, the private note, the research trail, or the way five tools were used together.

It cannot find the sentence you said in a meeting.

It cannot find the convention that lives only in your head.

Some of that should be written down.

Some of it should never live inside one provider’s memory.

So you supply it.

You are the missing layer.

That is fine once. It is not fine as a system.

The work you keep carrying is not random context. It is your process. It is how you make decisions. It is the shape of your taste. It is the difference between an agent that can do work and an agent that can work with you.

And right now, most of that know-how is either trapped in your head or scattered across vendor-specific harnesses.

Not Just The Model

The popular story says the model is the product.

That is partly true.

Better models matter. A lot. A good harness does not rescue a bad model.

But the wrapper changes the result in ways that are not cosmetic.

The useful evidence is messy in the right way. In SWE-ContextBench, oracle-selected compact summaries improved coding-task resolution from 26.26% to 34.34%. Raw trajectories barely helped. Mem0 underperformed the no-context baseline.

So the point is not “more memory.”

More memory can make things worse.

I also dislike the word memory here.

It points backward. This layer also holds decisions, workflows, open questions, failed attempts, and ideas still worth trying.

Memory is too small.

Know-how is closer.

But know-how does not mean dumping everything into a bigger memory store.

Own the raw record.

Curate the useful views.

Keep the source attached.

Make derived views disposable.

That is the know-how layer I mean.

Memory Islands

The opposite of that layer is what we have now.

The MemOS paper calls them memory islands: ideas explored in one environment do not carry cleanly to another, so you rebuild context by hand.

That is exactly what daily agent work feels like.

Claude knows one slice.

Codex knows another.

Cursor knows another.

Pi knows another.

Your notes know something. Your commits know something. Your meetings know something. Your calendar knows something.

No one view has the work.

The fix is not to give one vendor every slice.

That only makes the island bigger.

The fix is to own the layer that connects them.

I want a know-how layer outside any one agent: an owned record of what happened across the agents, tools, people, decisions, and artifacts involved in the work.

Not one vendor’s memory.

My layer.

The Black Box

This is where the argument is easy to misread.

The point is not that every vendor is secretly training on every conversation.

Some consumer products can use your chats or coding sessions for training unless you opt out. Business products usually draw a harder line. OpenAI says it does not train on business data by default. Anthropic’s commercial terms say it may not train on customer content.

Policies vary by product, tier, and date.

These are current product promises, not ownership.

The deeper problem is control.

My work sits inside systems I do not control, under policies I do not set, in formats I may not be able to inspect, correct, export, or move.

So I do not really control the record.

I control the buttons the product gives me.

Export.

Delete.

Correct.

Opt out.

Maybe.

And the more useful the agent becomes, the more important that record becomes.

It remembers preferences. It loads skills. It infers patterns. It compresses my work into its own shape.

That can be useful. I use these tools because they are useful.

But useful does not mean harmless.

The black box is not scary because it is magic.

It is scary because it is learning the shape of my work, and the shape stays inside the box.

Not privacy in the abstract.

Ownership of process.

The Company Version

This is not just an individual problem.

A partner at a mid-sized financial-modeling firm told me they were working with a startup to build agentic workflows for parts of their business.

The setup made sense.

The startup was spending time with the firm, learning how the partners think, how the calculations work, where judgment enters, what gets reviewed, and what makes one output acceptable instead of merely correct.

That is how you build a useful system.

It is also the moment to ask the ownership question.

If more of the firm’s work is going to be done through agents, then the firm’s process becomes one of its strongest assets.

Not just the spreadsheets.

The way they use them.

The review habits. The exceptions. The judgment. The workflow.

If that gets turned into a product the startup owns, the firm may end up renting its own process back on subscription.

At first, this can feel fine.

Early customers get attention.

The system feels custom.

The startup listens.

But if the product works, the startup has to scale. The special treatment fades. The workflow hardens into someone else’s platform.

That does not mean the startup is evil.

It means the firm should know what it is trading.

My advice was simple: try to own the know-how layer first.

Own the skills.

Own the workflows.

Own the examples.

Own the evaluation rules.

Own the record of why the system behaves the way it does.

Then connect agents to that layer.

If building it in-house fails, use the vendor. Sometimes that is the right call.

But at least know what failed.

Do not start by handing away the thing that may become your moat.

The Discipline Problem

Companies see the ownership problem when it becomes a contract.

Individuals see it later, when it becomes a habit.

My friend, Hause Lin, pushed back from the other side.

He has spent years building the kind of system I am describing for himself: timestamped markdown notes, meeting transcripts, years of work in one place, searchable and usable by agents.

His objection was practical: if a disciplined person can get most of the benefit with Obsidian, markdown, transcripts, search, and a few habits, why build more infrastructure for the last mile?

Then he made the harder point.

The problem is not only technical.

It is behavioral.

People know they should keep the record clean. They know future work will depend on it. They know the notes, links, names, and sources will matter later.

Then the work gets busy.

The note does not get written.

The file does not get named.

The meeting does not get linked.

The system decays.

He might be right.

It is also the proof.

The people who have solved this for themselves all seem to build some version of the same thing. They centralize their work. They keep it in files they can read. They make it searchable. They let agents use it. They do not rely on one chat window to remember their life.

He said Obsidian helped him get to the point where a fresh agent does not need a long briefing. He can point it at the record, ask for the analysis, and move work that used to take five hours closer to twenty minutes.

The objection is not that the know-how layer is useless.

The objection is that building it well takes too much discipline.

Exactly.

That is the infrastructure gap.

The payoff is real. The discipline required to get it is still too high.

The Record Is Not The View

There is a bad version of this argument.

It says: put everything in files and the problem is solved.

That is not true.

A messy folder is just a messy prompt with better ownership.

The record and the view are different things.

The record should be durable.

What happened. Who changed it. What source it came from. What decision it supported. What should expire. What needs review.

The view should be disposable.

A summary can be wrong.

An index can be rebuilt.

An agent-facing context pack can change with the task.

A human interface can hide the raw files completely.

That is the point.

Own the record.

Let the views change.

The Ledger

Imagine an agent finishes a work session.

Before it leaves, it updates a ledger.

Not however it wants.

In the format I defined.

Under the rules I set.

It records the decision that was made.

It links the source that justified it.

It notes the convention it discovered.

It marks the question that stayed open.

It flags what needs review, what should expire, and what should never leave the machine.

Maybe the agent writes that directly.

Maybe another agent interviews it at the end of the session.

Maybe a human reviews the entry before it becomes durable.

The mechanism can vary.

The point is the governance.

The work leaves behind a record I control.

Then the next agent starts from that ledger instead of from your tired re-explanation.

After enough sessions, the ledger stops being bookkeeping.

It becomes an asset.

The projects I have done, the decisions I made, the failures I diagnosed, the review rules I trusted, and the workflows I reused become examples the next agent can use before I explain myself again.

In model language, they become reusable context: examples, skills, workflows, review rules, and memories that shape the next run.

In human language, they become accumulated know-how.

MIRIX points toward the commercial version: personal memory that can be reused, shared, governed, and monetized. I do not like the word memory for it, but I like the direction.

If an expert’s know-how can guide an agent, it can also become something the expert shares under rules they control.

That is the ledger I mean.

Not a universal brain.

Not a new place to dump every transcript forever.

A governed record of work that future agents can use.

The Question Again

Before you ask which agent harness is best, ask a different question:

If I leave this agent tomorrow, what leaves with me?

Some files might.

That is not enough.

Maybe the answer does not matter for a small task.

Maybe it matters for a career.

Maybe it matters a lot for a company whose expertise is the company.

I am not trying to make agents remember everything.

I am trying not to hand my process to the tool I happen to be using this month.

The Point

Use the best model you can.

Use the best harness you can. Prefer one you can inspect, modify, and own.

Just do not let either one become the place where your process lives.

Keep your know-how in a record an agent can use and a vendor does not control.

Prefer tools that read and write open formats.

Ask every agent vendor the export question: if I leave, what leaves with me?

If you are an expert working with a vertical-agent startup, negotiate process ownership before your process becomes their product.

The future I want is simple:

Every agent session starts with the context it needs.

Every agent session ends having improved the record.

And the record belongs to you.