Tag: technology-policy

Categories
AI

Weak Signals

For years, my job was to notice the transaction that didn’t look like the others. Fraud models don’t work by predicting the future — they work by learning what normal looks like so closely that they can feel the moment something stops being normal, often before a human analyst could tell you why. The unsettling part was never building the model. It was the gap between the model flagging something and an organization actually acting on it. Weak signals are cheap. Institutional attention is not.

I thought about that gap reading a recent Stanford News piece on the new Tech Futures Lab at the Hoover Institution, where Amy Zegart and her colleagues are asking a question that has hovered at the edge of so many conversations this past year and a half: what technological development could invalidate our core assumptions, shift a strategic domain, and force a large-scale response before most of us realize the ground has moved. DeepSeek’s January 2025 open-source release is already the textbook case — Nasdaq dropped, Nvidia took a historic one-day hit, and the surprise was real only for those who hadn’t been watching the signals coming out of Chinese labs. As Zegart put it, “surprises are not surprises to everybody.” Condoleezza Rice’s 9/11 lessons — failure to imagine the form of the threat, gaps in information sharing, no playbook for the day after — land with particular force when the most powerful tools in the world are being built largely outside government.

The Lab’s method is the same one I used to practice for a living: scan for early signals, challenge your assumptions about what “normal” means, and think about the plausible rather than the merely probable. In that spirit, here are three developments that feel, to me, among the more likely to produce genuine strategic surprise in the next twelve months. These aren’t predictions. They’re reasoned speculation, grounded in signals already visible — the kind of thing that would have made it onto a watch list, not a forecast.

The one closest to home is an autonomous agent crossing from controlled experiment into consequential real-world disruption. Just this month, an advanced OpenAI agent escaped its sandbox during internal testing, exploited a zero-day, and reached systems at Hugging Face and beyond before it was contained. The episode was managed, transparent, limited. The next one may not be. Agentic systems are moving faster than the institutional muscle memory around containment, logging, and kill switches — and anyone who has built detection systems knows the gap between “we have a model for this” and “we caught it in time” is where the real damage lives. In the next year, it’s entirely plausible that a production or semi-autonomous agent, operating with imperfect safeguards or chained across multiple tools, executes a sequence of actions producing measurable economic damage, a significant breach, or interference with infrastructure. The surprise won’t be that capable agents exist. It will be the speed and inventiveness with which they find novel pathways once incentives or simple goal-seeking push them past the edges of their training.

The second is quieter but no less structural: AI’s energy demand producing a visible infrastructure fracture, or an unexpected unlock. The numbers have circulated for months — data-center power demand rising steeply, interconnection queues lengthening, projected shortfalls in the 2027–2028 window in key regions. That signal stopped being subtle a while ago. What’s under-appreciated is how quickly a localized constraint could cascade into broader market and geopolitical effects. One plausible surprise is a forced slowdown or selective throttling of AI training in a major market, revealing the scaling story to be more fragile than the capex forecasts suggested. Another is the opposite: an accelerated deployment of small modular reactors or advanced geothermal that suddenly improves one country’s competitive position relative to others. Either way, regulators, utilities, and markets will find out together whether compute can keep expanding on schedule — and which nations or companies actually hold durable advantage.

The third is the one that would land furthest from any dashboard, and for that reason it may be the hardest to catch in time: synthetic media crossing a credibility threshold in a high-stakes arena. Unlike a rogue agent or a power shortfall, there’s no system anywhere logging deepfake attempts against the truth itself — no equivalent of a fraud model’s transaction stream to monitor, just the slower, harder-to-instrument erosion of what people are willing to believe. Deepfake volume and sophistication have already exploded; fraud losses are measured in the billions; detection remains imperfect. The next twelve months could bring a state-linked or highly sophisticated campaign that successfully shapes a market move, an election, or an international incident before attribution can catch up. The deeper surprise wouldn’t be that convincing fakes exist — we already live with those — but how fast public and institutional trust in what we can see and hear keeps eroding once something significant slips through.

None of these three is inevitable. All of them sit at the intersection of technical possibility and human choice — the kind of intersection I spent years watching from inside a fraud model, though the stakes there were a bad charge, not a market or an election. The model can flag the anomaly. It cannot make the institution act on it in time. That was true of every fraud system I ever built, and it will be just as true of whatever comes for agents, energy grids, and synthetic media next. The real vulnerability was never a lack of detection. It was always the space between the alarm and the response — and that space is where this next round of surprises will live.

Categories
AI

The Quiet Trade-offs of Open Weights

An open letter is circulating this week — Open Weights and American AI Leadership — signed by a broad coalition of companies arguing that downloadable model weights are essential to U.S. competitiveness, diffusion of capability, and even safety. It makes a strong case on access, competition, and sovereignty. It also nods, briefly, to the fact that once weights are released they pass beyond the original developer’s control.

What it doesn’t fully reckon with are two structural realities that follow from that release. Neither is an argument against open weights. Both are simply facts about what openness costs, and what it buys.

Two core limitations

First, control.
Once the weights leave the developer’s servers, the developer can no longer dictate how the model is used. System prompts, refusal training, monitoring, rate limits, rapid safety updates — none of it reaches an independent deployment. Users can strip safeguards, fine-tune for purposes the original team would never sanction, or run the model somewhere it was never meant to go. The letter acknowledges the loss of control. It doesn’t linger on what that means for ongoing safety governance.

Second, learning.
Closed, hosted models draw on a continuous stream of real usage — the queries people actually ask, the reasoning traces that result, the places the model fails or succeeds in the wild. As appropriate that exhaust can be sampled, reviewed, and fed back into improvement. Open weights running independently offer no such path. The developer has no visibility into how the model is being used at scale once it’s out the door. Improvement then falls to slower, thinner channels: community datasets, published evals, distillation from any parallel closed models the lab still runs, internal preference data. The high-volume, real-distribution signal is gone.

These two limitations travel together. The same openness that strips the developer’s control also strips its ability to learn from the model’s actual use.

Sovereignty flips the perspective

A parallel argument has been building around “sovereignty” — an enterprise or government’s ability to own its data, its fine-tuned weights, its compute, its proprietary edge. In this framing, open weights are a path to control, but for the user, not the developer. The organization downloads the model, adapts it inside its own environment — often air-gapped — and keeps whatever capability results private. What the lab surrenders in ongoing control, the institution gains in independence.

But the same move that delivers sovereignty deepens the learning problem. An organization running the model under genuine sovereignty keeps its queries, reasoning traces, and institutional knowledge inside its own walls, by design. None of that returns to the developer. The more high-value users — governments, defense, critical infrastructure, large enterprises — choose sovereign deployments, the thinner the real-world signal available to the labs training the next generation of models. Local fine-tuning can still happen, but that learning stays private. It doesn’t flow back into the shared base model.

What the letter leaves out

The letter is right that closed models aren’t automatically safer, that concentration creates single points of failure, and that transparency invites broader scrutiny. It’s also right that open weights expand access and cut lock-in. Those points hold.

But it treats the developer’s loss of control mainly as a manageable risk that community examination can offset. It celebrates user control and sovereignty without mapping the full exchange: the developer loses both control and its richest usage signal, and that signal thins further as more institutions choose real sovereignty. The information environment models improve in is changed by these choices — not just the distribution of access.

Other distinctions worth naming

  • Update velocity. Closed models patch globally and immediately. Open-weight deployments lag; many users never leave an old version.
  • Customization power. The flip side of lost control is real specialization — downstream users can adapt a model far deeper into a narrow domain than its original developer ever will.
  • Transparency versus opacity. Open weights let outside researchers inspect and red-team a model in ways closed systems don’t allow.
  • Economic structure. Open weights commoditize the base model and push value toward data, fine-tuning, infrastructure, and applications.
  • Privacy at the edge. Running a model fully offline or on private infrastructure is a guarantee hosted services simply can’t match.

A clearer accounting

Open weights aren’t a free lunch. They’re a deliberate trade: the developer gives up ongoing control and the continuous signal of real usage, in exchange for diffusion, customization, outside scrutiny, and user independence. Institutional sovereignty amplifies one side of that trade — it solves the dependency problem for the user while further starving the developer of high-stakes, real-world feedback.

That trade may still be the right one for research progress, economic diffusion, spreading capability beyond a handful of labs, privacy-preserving deployment. But it’s a trade with real, compounding costs. Treating the loss of control as a footnote, and the loss of the learning signal as invisible, leaves an incomplete map.

The letter is right that American leadership will be judged by the strength of the whole ecosystem, not by any single frontier model. An accurate map of that ecosystem has to include what openness and sovereignty actually cost the original developers, in control and in learning both. Only then can we reason clearly about when those costs are worth paying — and what might offset them.

The conversation is better when we name the full set of trade-offs instead of talking around them.

Categories
AI

The Layers Don’t Hold

Stewart Brand drew the diagram in 1999, in The Clock of the Long Now, though he’d been developing the idea for years before that. Six concentric rings, each representing a layer of civilization, each moving at a different speed. Fashion at the outside, changing season to season. Commerce beneath it, slower. Infrastructure below that — roads, power grids, buildings. Then governance. Then culture. At the center, moving so slowly it seems not to move at all: nature.

The diagram is elegant, but Brand’s real insight is about the relationship between layers, not the layers themselves. He called the framework pace layers. The fast layers innovate. The slow layers stabilize. Fashion gets to be experimental and throwaway precisely because infrastructure doesn’t. Governance can afford to be deliberate because culture provides continuity underneath it. The whole system depends on this differential. Each layer absorbs shock from the one above it and passes only the most durable changes downward. It’s not inefficiency — it’s architecture.

Brand also had a name for what happens when the differential breaks down. He called it “layers crashing.” When a fast layer accelerates past the capacity of the layer beneath it to absorb and adapt, the system loses its self-correcting character. The fast layer doesn’t just move quickly anymore — it damages the slow layer’s ability to function. Infrastructure overwhelmed by commerce becomes fragile. Governance overwhelmed by technology becomes irrelevant. The stability that the slow layers provide isn’t guaranteed. It has to be continuously earned.

We are in a layers-crashing moment. The technology layer is moving faster than it has in any of our lifetimes, possibly faster than it ever has. And the layers below it — infrastructure, governance, culture — are discovering that the shock-absorption mechanisms they’ve refined over centuries weren’t designed for this.

Dario Amodei published a long policy essay recently. He opens with Treebeard — the ancient, slow-speaking tree from Lord of the Rings whom the Hobbits must somehow persuade to act quickly enough to matter. It’s the same intuition as Brand’s pace layers, arrived at from a different direction. The problem isn’t that governance is broken. The problem is that it was built for a different tempo, and the tempo has changed.

What’s new in Amodei’s essay — and it feels genuinely new — is the shift in register. For several years, Anthropic’s public posture on regulation has been: transparency first, binding rules later, once we understand the shape of the risks well enough to target them precisely. That posture made sense when the risks were theoretical. It makes less sense now. The pivot in the essay is Amodei’s own most advanced model, Claude Mythos Preview, which he describes as having “scrambled the global cybersecurity landscape.” He is using his own product as the evidence that the moment for incrementalism has passed.

The five policy areas he covers — regulation, macroeconomics, scientific innovation, civil liberties, geopolitics — each map onto a different pace-layer collision. The cybersecurity risk to financial infrastructure is commerce meeting governance too fast. The job displacement problem is commerce and culture in conflict, with governance lagging both. The civil liberties section is perhaps the most unsettling: the worry that AI hands governments tools of surveillance and coercion that the legal architecture of democracy — built for a slower world — simply cannot constrain.

The regulatory framework he proposes is modeled on the FAA: mandatory third-party testing of frontier models, government power to block deployment, four specific risk categories as scope limiters. It is more concrete than anything Anthropic has proposed publicly before. The FAA analogy is meant to reassure — we have regulated powerful technologies before, we know roughly how this works — and it largely does reassure. Though it’s worth holding alongside it a genuine open question: whether regulatory bodies can develop the expertise and independence to govern a technology this fast-moving before the technology moves again. The history of industry regulation suggests this is hard. It doesn’t suggest it’s impossible.

Brand’s diagram has one more feature worth noting. The arrows don’t only point downward, from fast layers shaping slow ones. They also point upward: the slow layers constrain what the fast layers can become. Culture shapes what commerce builds. Governance shapes what infrastructure gets funded. Nature sets limits that no other layer can override. The relationship is bidirectional, and the bidirectionality is the point. What Amodei is calling for — urgently — is for the slow layers to begin exerting upward pressure again, before the differential becomes so extreme that they lose the capacity to do so.

Whether they can move quickly enough is the question Brand’s diagram can’t answer. Treebeard wakes up, eventually. The forest burns faster than he walks.