Categories
AI Learning Meta

Trajectories: how Meta plans to make Muse smarter by watching it work

Buried in the data policy section of Meta’s long post on how it built safety into Muse is a sentence that isn’t about safety at all:

“Inference data, the back and forth conversations between you and your Muse and the tool calls and subagent handoffs that result (‘trajectories’) are useful data for training new checkpoints of the LLM model at the core.”

Two things are worth noticing. First, the technique described isn’t new — training on agent rollouts is standard practice across the field. Second, the company describing it is Meta, in plain language, in a public post, about a product aimed at billions of consumers. The labs usually discuss this stuff in papers about coding agents. Meta just told its future user base: your agent’s work product is our training data. The candor is the story, not the technique.

A trajectory isn’t a chat log. It’s the complete record of an agent doing a job: what you asked, what it tried, which tools it called, where it went wrong, how it recovered, which subagents it spawned, and whether the thing actually got done. Every time you let Muse book the flight, triage the inbox, or research the supplier, you’re generating one.

From text to behavior

The technique matters anyway, because the diet that AI trains on is changing. Pretraining was about text — the whole internet, more or less. Post-training was about preferences — which answer humans liked better. Trajectories are the third course: demonstrations of competent behavior, in full, mistakes included.

There’s a reason for the shift. Text teaches a model what the world looks like. Preferences teach it what people want. But neither teaches it how to do a 40-step task without wandering off, recovering from a dead end, or knowing when to ask for help. That only exists in records of agents actually doing things. And until recently, almost nobody had those records at scale — because almost nobody had agents doing real work at scale.

The demonstrated instance — and what it doesn’t prove

Meta’s concrete example is Muse Spark 1.2, co-trained with Muse Code: model and harness trained together on rejection-sampled harness trajectories — run the agent many times, keep the runs that succeeded, train on those — with recipe-level tuning for goals, context compaction, and subagents. The model isn’t learning to predict text; it’s learning to behave inside a specific set of tools.

This is the end of the “base model plus clever prompting” era. The artifact is the bundle — model and harness, co-designed. A model trained on trajectories from one harness will be genuinely better inside that harness than a smarter general model dropped into it cold. Meta is saying this out loud; OpenAI and Anthropic are doing the same thing more quietly.

But notice the domain: coding. And coding is exactly where trajectories are cheapest to manufacture — verifiable unit tests, sandboxed repos, SWE-bench-style tasks. Nothing about the Muse Code result requires a single consumer or a single inbox. So the one demonstrated instance of Meta’s trajectory training sits squarely in the category where Meta’s distribution advantage matters least. Meta hasn’t shown its hand on the category that actually matters.

The other category is the personal one, and there the evidence is thinner. What exists is a stated intent, not a published result. The data policy says personal Muse trajectories “are useful data for training new checkpoints.” The product is designed to generate them: Meta’s own design example has Muse monitoring school emails, adding dates to a family calendar, filling a supply cart, finding a sale sweatshirt, booking dinner, and catching a sports tryout deadline hours before it closed. That is what an unverifiable-domain trajectory looks like — a morning of small judgments no unit test could grade.

No training run on that data has been published. No benchmark, no “Muse got X% better at inbox triage after training on Y million user trajectories.” So the sharpest version of the argument — that the real moat is the data nobody else can fake — should be labeled for what it is: a prediction, not an observed fact. It’s a prediction with a mechanism, though: these are judgments that can’t be synthesized, in the one distribution channel that reaches the people making them.

The flywheel — and its limits

With that caveat on the table: trajectories get better with scale, and Meta has scale like nobody else: billions of users across its apps, and now an agent — Muse — sitting inside them. Every user interaction is a potential training trajectory. Better trajectories train a better model; a better model makes a better agent; a better agent attracts more users. Meta states the bargain plainly: “every Muse user gets a better personal agent as we all collectively use the product and help the model understand the intricacies of human life.”

But “most users = most trajectories = structural advantage” needs its counter-case, because a lot of the highest-value trajectory data right now doesn’t come from consumers at all. It comes from sandboxes, the same kind that produced Muse Code. Synthetic and simulated trajectories sidestep the need for billions of users entirely — Anthropic and OpenAI are getting rich trajectory data from developers running Claude Code and Codex against real repos, no social-app distribution required.

The honest version of the moat argument is narrower, and more interesting. Synthetic trajectories work brilliantly where success is verifiable — code either passes the tests or it doesn’t. They work poorly where success is a matter of judgment: triaging an inbox, planning a trip around someone’s actual preferences, knowing which email deserves a reply. There is no unit test for a life well managed. And those unverifiable, deeply personal tasks are exactly what Meta means by “personal superintelligence” — and exactly where its distribution gives it trajectories nobody else can synthesize. The moat isn’t “most data.” It’s “the data nobody else can fake.”

The price of the flywheel

There’s a wrinkle, and Meta knows it. The flywheel runs on your data — your emails, your calendar, the messy reality of your life, which is exactly what makes the trajectories valuable. Meta’s answer is sanitization (“trajectories are sanitized to remove key personally identifiable information”), an opt-out switch, no sharing with ad systems, and a forthcoming “Confidential VM” that would cryptographically prevent even Meta from seeing your data.

The tension is fundamental, and it’s the sharpest part of the whole picture: the product gets smarter by watching you, and it earns the right to watch you by being trustworthy. Those two imperatives pull in opposite directions, and no amount of engineering fully resolves it — the Confidential VM, if it ever ships as described, would resolve it by breaking the flywheel, since trajectories Meta can’t see are trajectories Meta can’t train on. The opt-out rate will be the market’s verdict on the deal Meta is offering.

Experience is the missing piece

But the deepest reason trajectories matter has nothing to do with Meta’s strategy. It’s about what intelligence actually is.

A model trained only on text knows the world the way a brilliant student knows it from books. A model trained on trajectories knows it the way a practitioner does — from doing the thing, failing at it, and adjusting. The trajectory is the closest thing AI has to experience. And an agent that records its experience, keeps what worked, and folds it back into itself is doing something that rhymes with learning.

This is why I keep coming back to continual learning as the critical missing piece in AI. The models are frozen at training time; everything they “learn” afterward lives in context windows and memory files, fragile and local. Trajectories are the bridge: today’s version of the loop is slow and centralized (collect trajectories, train a new checkpoint, ship it), but the direction is obvious. The end state is an agent that learns continuously from its own experience — from your experience with it — the way people do.

Meta’s bet is that the path to personal superintelligence runs through watching agents work, at planetary scale, and distilling what works back into the model. No result yet proves the bet pays off — the personal trajectories are still a hypothesis, not a track record. But it’s an unglamorous hypothesis, no new scaling law, just better data about doing things, and unglamorous bets about data have a good track record in this field. The internet made the last generation of models. Trajectories might make the next one — if Meta can show, and not just say, that the data nobody else can fake is data that actually teaches.

Categories
Apple Apple Vision Pro Facebook Meta

Sneakily antisocial with good enough pass through!

Update: If you’d rather listen to a podcast discussing this post, try this!

Sometimes the future arrives when you least expect it. This week I spent an hour watching the Meta Connect keynote presentation by CEO Mark Zuckerberg. It was amazing. The joy of a live technology keynote – flaws and all – was brought to life once again!

Apple’s iPhone Event

Earlier this month, Apple had its annual iPhone reveal event. Beautifully videoed at locations around San Francisco and Apple Park in Cupertino, the presenters told their stories about the various features of the new iPhones. But even though the presentations were truly beautiful, the whole session was boring. Too long and too “slick”.

Meta’s Connect Keynote

Zuckerberg’s presentation, in contrast, was live in front of an audience – and it had the spontaneity we all remember from those earlier Apple keynotes that Steve Jobs pioneered and other Apple execs continued doing before the pandemic forced the elimination of a live audience and the switch to formal video productions instead. The first demo that Zuck featured began nicely but then quickly failed – the demo gods are still alive and lurking in the background. But the demo fail wasn’t a disaster – it brought humor to the event. We’ve all been there – had something go off the rails when we least expect it. Seeing it happen live was in its own way delightful.

Orion vs Vision Pro

Zuckerberg spoke of several things during his keynote but the one that struck me the most was his introduction of the Orion holographic smart glasses. The obvious contrast to Orion is the Apple Vision Pro, a product which certainly failed to interest me and which seems to have had the same effect on many others. Unfortunately, the perception of the Vision Pro being a market failure hasn’t been helped by Apple’s curious failure to show real commitment to the product.

The Vision Pro won high marks from most reviewers for being an elegant and sophisticated piece of technology. And that elegance is reflected in its price – over $3,500. Because of the design decisions that Apple made, a lot of the cost of the Vision Pro seems rooted in the very high end cameras and screens to bring the real world inside the headset where it’s then augmented with other computer generated screens creating a mixed reality experience.

Meta’s Orion, on the other hand, doesn’t try to do that. Instead the metaphor is one of real glasses where the real world is simply directly visible without the need for the Vision Pro’s high end cameras and displays. As one reviewer commented, the real world is displayed in “infinite resolution” unlike in headsets like that Vision Pro that capture the real world with cameras and projects those images instead. Orion has “good enough pass through”!

In one of his discussions of his impressions of Orion, Stratechery’s Ben Thompson commented about how you could be “sneakily anti-social” using Orion because you could interact with other humans just like you would wearing a pair of glasses. You could be looking at your Instagram feed without anyone else realizing you’re doing so as you maintain eye contact with them as well.

Audio and Visual Transparency

This reminded me of one of the hidden benefits I learned about when I was fitted with hearing aids several years ago. Fortunately for me, my need for hearing aids coincided with the technology in hearing aids maturing to the point where they have become very small, rechargeable with all day battery life, and – importantly and surprisingly – perfect for listening to streaming content from my iPhone. When I’m doing so, I still hear the real world around me and can have a conversation with another person even while there’s music or a podcast playing in my ears. Orion seems visually very similar to my audio experience with my hearing aids!

Concept Cars

Ben Thompson pointed out that Orion is really just a “concept car” at the moment whereas Apple’s Vision Pro is a real product that’s been shipping in volume for over a year. Direct comparisons really aren’t appropriate – but the fundamental design differences between the two are fun to think about and cause me to wonder about bigger implications for the future of both products.

Back to the Future

There’s so much happening in high tech right now that it’s an amazing time to be alive. When you overlay AI on top of these audio and visual capabilities, things get even more interesting – and exciting. Zuckerberg’s keynote brought back memories for me of those years of Steve Jobs proudly showing off Apple’s latest and greatest innovations. It’s pretty clear to me that the founder-led aspect does shine through at times like these. As does the joy of watching a live performance!