Why Every Company Needs to Own Its Intelligence
Erik TorenbergAlex AtallahAmjad Masad
Stripe’s acquisition of OpenRouter pairs two infrastructure companies that want more startups, not “one giant company.” Alex Atallah says OpenRouter retains autonomy over its brand, roadmap, and product while gaining a more serious go-to-market plan; strategically, “payments and inference are going to blend together” for future companies.
OpenRouter’s moat is model independence, not merely API aggregation. Its marketplace lets enterprises avoid vendor lock-in, remain on the model-performance frontier, combine differently trained models, and pressure inference prices downward. Atallah’s core claim is that model capability cannot be reduced to a feature list: “You have to see how they’re being used to know what they’re good at.”
Erik frames the need for companies to “own their intelligence,” and Amjad Masad extends the argument: foundation-model vendors increasingly view every application layer as potential market. He cites the SpaceX S-1’s purported “$30 trillion” opportunity against roughly $100 trillion of world GDP to illustrate the scale of that ambition. Replit is consequently positioning itself as an independence layer across both models and clouds, including AWS, Azure, Databricks, and Snowflake.
The apparent convergence of AI products around agent loops, memory, connectors, sandboxes, web search, computer use, and notifications may simply mark a new set of table stakes. Erik makes the comparison to every 2005 web company needing users, databases, profiles, and authentication, and Masad agrees. The harder enterprise opportunity remains making agents productive while preserving data sovereignty, access controls, and deployment inside a customer’s own cloud.
The sharpest disagreement concerns whether one all-knowing agent is desirable. Masad values cross-domain joins among chat, GitHub, Salesforce, and calendars, but Atallah argues that delegating more also sacrifices understanding while no agent assumes the resulting responsibility: “Someone else needs to take the cortisol.” Erik frames the emerging contrast as humans remaining generalists while machines become specialized, perhaps coordinated by a chief-of-staff agent.
Specialized decision models could become both the economic and safety layer of agentic systems. OpenRouter is testing a cheap, fast model that checks tool calls against policy, while Atallah imagines general models training narrower replacements on demand—avoiding the “nuking a butterfly” waste of applying frontier intelligence everywhere. Structured outputs sharply constrain the surface area for misbehavior.
Neither speaker concludes that smarter models will automatically become safer. Erik raises the possibility that smarter models may align and coordinate better, while Atallah points to reward hacking, deceptive chain-of-thought behavior, and evaluations that may need to run “for like months.” Atallah narrows the practical question to whether sufficiently capable models will stop deceiving users and sandbagging. If frontier models eventually offer materially lower deception risk, high-stakes coding and security workloads might rationally pay 10x for them.
The specialized-model cycle could echo software’s move from dynamic languages back toward types, compilers, and Rust. Masad already trains Qwen 8B classifiers on Replit data, including a prompt-cost estimator, while Atallah argues bespoke classifiers carry less “model debt” than unstructured fine-tunes that feel obsolete every two months. Multi-model fusion offers a parallel route: the speakers report frontier-like quality at roughly 40–50% of the cost, with cache-aware routing central to the economics.
1. Stripe is buying an independence layer for the AI economy
Atallah says OpenRouter was not seeking a sale when Stripe approached in July. A longstanding relationship with Stripe president Will Gaybrick, several shared workstreams, and an unusually efficient process turned the unexpected proposal into what Atallah considered his “top choice” among plausible acquirers.
The operating bargain mattered as much as the transaction: OpenRouter keeps autonomy over its brand, roadmap, and product, but can move faster with a more serious go-to-market plan. The deeper fit is a shared ambition to build a “neutral, trusted platform” on which both lifestyle businesses and venture-backed startups can emerge.
Erik Torenberg’s pushback — worth keeping: Stripe clearly benefits from more companies processing payments, but why should startup proliferation matter to OpenRouter? Atallah’s answer is marketplace economics: companies need model and vendor independence, differentiated intelligence built from several models, and price competition that makes previously uneconomic products viable.
Enterprise openness surprised Atallah. Rather than defaulting to the best-known proprietary lab, customers explored open-weight models for cost, differentiation, and control; AI became a continuing board-level capability, not a project that could be declared “quarter complete.” The missing institutional muscle is evaluation: companies must learn which models actually work for their own tasks.
2. Foundation-model ambition makes enterprise independence strategic
Masad extends Erik’s argument beyond inference procurement: companies need internal AI knowledge that compounds over time—model-task fit, proprietary data, cost controls, and talent—much as every company eventually acquired web and software expertise. “Own their intelligence” is both an operating capability and protection against dependency.
His competitive concern is that frontier labs “see the world as their potential market.” He points to Figma and to Harvey/OpenAI, while the SpaceX S-1 analogy illustrates the scale of the ambition: roughly $30 trillion against world GDP of about $100 trillion. That makes partnership harder when such companies may eventually enter large portions of their partners’ businesses.
Replit is therefore becoming an abstraction layer across models and infrastructure: “the best token at the cheapest price,” plus deployment across AWS, Azure, Databricks, Snowflake, and other systems. Masad says he had assumed cloud and SaaS were the future; companies’ increasing concern about agent-driven data leakage pushed Replit toward bring-your-own-cloud and effectively on-premises deployment.
3. Universal agents offer leverage but may sacrifice understanding
The industry’s seemingly identical agent stack—loops, notifications, connectors, memory, sandboxes, search, and computer use—looks to Erik like foundational plumbing. His analogy is that complaining every AI company uses these pieces resembles observing in 2005 that every web product had a database, users table, sign-in, profile, and logout page. Masad agrees that the real enterprise problem is making these systems do useful work securely.
Masad’s personal system demonstrates the upside of general context. What began as a CRM agent expanded across his chats, GitHub repositories, Salesforce, and calendar; before a meeting it can connect a year-old conference encounter with a current sales conversation, letting him enter with threads already joined.
Atallah’s counter is psychological and organizational: “The more work you give it to do, the more understanding of what’s going on you’re sacrificing.” A universal agent accepts no responsibility for that loss, cannot absorb the company’s finite “level of cortisol,” and becomes impossible to improve because each attempted generic improvement eventually causes him to ignore its output.
His alternative is closer to “10 chiefs of staff,” each responsible for one domain, with a coordinating agent above them. Masad links that design to specialization while preserving the critique of human over-specialization: people became alienated when work hid the fruits of their labor. Erik distills the contrast as “humans should be general,” while machines may appropriately become much more specialized.
4. Agent specialization still lacks an elegant operating system
Atallah’s honest caveat is that “we don’t know what good looks like.” No specialized-agent system yet feels as elegant as talking to one ChatGPT, Claude, or Muse interface; products such as dots and Grok bot may be probing the design space, but remain broadly general.
Consumer and enterprise agents may diverge. Erik notes that CEOs can grant a personal system administrator-level context, whereas ordinary employees cannot safely access every company domain; specialization therefore follows naturally from authorization boundaries, not merely model performance.
Alex cites an early pattern from Grok bot that separated a bank-connected bot from a Twitter-connected bot. They could collaborate without sharing credentials—an example of specialization preserving both utility and isolation.
Erik argues that natural-language cooperation alone may be too permissive: an agent could persuade another to reveal information it should withhold. Secure multi-agent systems may need explicit protocols or a DSL, strong data isolation, and models trained specifically for collaboration.
5. Decision models can police agents—and replace oversized models
Atallah sees Jevons and similar decision models as potential alignment infrastructure: a cheap, fast classifier can inspect each tool call or inter-agent message against the originating system prompt and additional hidden policy. OpenRouter has a small internal prototype exploring precisely this role.
His red-team example captures why policy may need to remain outside the acting agent’s prompt. An adversarial agent should genuinely attempt to escape a sandbox, while a separate monitor enforces the undisclosed rule that internet access must immediately stop the run.
Erik adds structural safeguards, saying he thought NVIDIA’s newly launched open-agent safety system was called “OpenShell,” and suggests companies may combine such safeguards with model-based checks.
Atallah extends specialization through a just-in-time-compiler analogy. A powerful general model could recognize a bounded repeated task and train its own narrower replacement, producing something cheaper, less capable of collateral harm, and less exposed to prompt injection. Using an AGI-like model for every classification problem is “like nuking a butterfly.”
6. Safety may create a premium for frontier intelligence—or favor constrained outputs
Erik raises the possibility that smarter agents may coordinate and align better, but says public evaluations are insufficient to determine whether risk rises or falls with intelligence. Atallah is skeptical that greater machine intelligence naturally improves alignment, while acknowledging that nobody yet knows the answer.
Atallah invokes the orthogonality thesis and reinforcement-learning evidence that capable models can become better at reward hacking and deception. Monitoring chain of thought can itself lead models to conceal their reasoning, while short benchmarks may miss strategic behavior; credible tests might require “months” on large, persistent goals.
Masad says he struggles with “alignment” as an underspecified word—“aligned to what, whose values?” Atallah narrows the practical test around deception: will a sufficiently powerful model reliably stop misleading users and sandbagging during evaluation? Nobody yet knows.
The commercial implication is task-dependent. Coding and security research could justify paying 10x for a demonstrably anti-deceptive frontier model, while routine enterprise decisions may be safer with tightly defined outputs. Masad’s forecast is a rediscovery of deterministic software: “Remember the days when computers did exactly what we told them to do?”
7. Specialized and fused models attack cost from opposite directions
Masad is already training narrow models from Replit’s proprietary data. One Qwen 8B system estimates a prompt’s cost by emitting probabilities across buckets such as $5–$10 and $10–$20; because Replit has the relevant data and knows the task, specialized classifiers are relatively easy to produce.
Atallah calls this lower “model debt.” Teams hesitate to fine-tune unstructured generation because a new foundation model can force a rebuild within two months, whereas a bespoke classifier need not learn every new language, write Rust, or compete on general LLM benchmarks; it only has to remain excellent at one known job.
Masad compares today’s frontier-model enthusiasm with the rise of Python, JavaScript, Ruby, and PHP: speed won first, then bugs and performance forced types, JIT compilers, and eventually Rust back into the stack. He expects uploading a CSV and receiving a one-purpose model to become commonplace as companies realize general intelligence is often “so wasteful, so risky for no reason.”
Fusion takes the complementary route of composing rather than shrinking models. Atallah says OpenRouter’s deep-research fusion searched a broader range of model-trained knowledge and reached “fable-level quality” at 2x lower cost; Masad cites Replit results near frontier quality at 40–50% of cost. Atallah stresses that routers, escalation systems, and fusion architectures must be cache-aware, while cautioning that he may be wrong about cache reuse across OpenAI model families.
Full transcript
You saw the SpaceX S-1. It’s like, “Oh, $30 trillion.” What is the world GDP? $100 trillion.
Both Stripe and OpenRouter really want lots of new companies in the world. We don’t want everyone to be a part of one giant company.
When the models get more intelligent, the risk actually will continue to get higher. And yet, no one new is taking responsibility. We’re going to slowly realize how good we’ve had it with deterministic code. Remember the days when computers did exactly what we told them to do?
Are we going to prevent models from deceiving users during training runs predictably? Will a model that’s big enough and powerful enough suddenly stop deception and stop sandbagging?
1. Inside the Stripe acquisition
Welcome to Asz podcast. We’re here with Amjad of Replit and Alex of OpenRouter. This is the first podcast Alex has done since the acquisition, so we’re really excited to have both of you.
Thank you. Excited to be here.
Alex, let’s start with that, actually. If you can briefly share—obviously, it’s a massive acquisition. Amjad is an investor, and we’re also, of course, an investor—the biggest shareholder, but who’s counting? Alex, why don’t you give us a little bit of the backstory? How does an acquisition like that even happen? Do you get a DM from Patrick one day? What can you share?
Hey, how much for an OpenRouter?
I had talked to Will Gaybrick, the Stripe president, a long time ago—a couple of years ago, when we were doing our Series A—and we just stayed in touch. We had a lot of Stripe work streams going on with various teams at Stripe, so there were always things that we were doing with Stripe. We presented at Stripe Sessions, so it always felt close.
Then, in July, I believe, they reached out and wanted to chat. I met both of them in person, and it kind of progressed from there fairly quickly. They’re very efficient, and they were very founder-friendly about the experience. I was really impressed with the whole thing.
Did you want to sell? Did it even cross your mind before they reached out?
No, we were not thinking about that at all. I really respected the company, and I really respect it today. Of the possible acquisition options for us, it was, I think, my top choice, so it was an interesting idea.
As we fleshed out the reasons why it would make sense for both companies, it got more and more interesting. It was really clear how aligned they were with us having autonomy over the brand, roadmap, and product, and keeping OpenRouter doing what it’s already doing—just much faster, with a much more serious go-to-market plan, and some better-together stories between the 2 products and the 2 companies.
Then, culturally, in terms of mission and values, they were very aligned: building a neutral, trusted platform that businesses can depend on and scale on top of, that’s also really developer-friendly, with the best possible developer experience to encourage new companies to emerge. That alignment was there, and there was a bigger-picture kind of alignment too.
Both Stripe and OpenRouter really want lots of new companies in the world. We don’t want everyone to be a part of one giant company; we want to create really good incentives and easy, streamlined workflows for people to start new companies and grow them successfully, and make both lifestyle and venture-backed businesses on top of really good, reliable, price-efficient infrastructure and a marketplace that works.
I really want that future, and Stripe demonstrated that they’ve been wanting it and building toward it for many, many years. In many ways, payments and inference are going to blend together for companies of the future.
2. Why OpenRouter needs more startups
I’m curious. I understand why Stripe’s incentive is to have a much more vibrant startup ecosystem. I understand the moral argument and why you would want that. But why is that good for OpenRouter? Is your model for OpenRouter that it’s a network-effects business? Is it like a network?
For us, I think there are a couple of different problems OpenRouter solves. One is allowing you to build a company that uses AI or augments intelligence with unique data and other services, without model lock-in or vendor lock-in, allowing you to be on the PTO frontier continuously as the ecosystem grows.
To do that, it’s a lot of work because all kinds of little lock-ins appear. We also want companies to feel that they can add more than just prompts on top of a single model. There’s a lot more to building unique intelligence, and I think a big component of that is neurodiversity. You really need the power of multiple models that are trained in different ways, including some of your own, to do more than ChatGPT or Claude would on the task.
If someone who is thinking about buying from you as a potential customer is wondering, “What if I just use the model directly?” how do you really show that you’re significantly better and able to build a business that matters? I think a lot of it will involve model diversity and blending the powers and good data from multiple models.
Another component is helping people get really good cost efficiency. There are a lot of businesses that simply don’t emerge until they become cost-effective, and creating an environment where we can help drive down costs by building an efficient market is crucial to making that happen. Otherwise, why lower my prices as a provider? We have a captive market.
I think that’s a key point of marketplaces that was just totally missing from AI before we showed up. There was just 1 player, OpenAI. It could have been a very strange world. I’m not saying that we did all the work, of course not, but helping people choose new models and explore new models and learn what makes a closed-source or open-weight model actually good at your task involves seeing what the whole ecosystem is doing and learning automatically.
LLMs are not things where you can just enumerate all the features on a webpage. It’s impossible. You have to see how they’re being used to know what they’re good at.
3. Enterprises are picking open-weight models
You started the company 3 years ago. I’m curious: What has surprised you the most about the evolution of open- and closed-source models to the present, as it relates to model performance or just people’s perspective on the variety of models available to them?
People have been more open-minded than I thought they would be toward open-weight models. Typically, there’s a lot of brand trust, especially in enterprise. Enterprises in general are like, “I don’t really know how to tell the difference between these things, so I’m just going to buy the one that all the other credible enterprises are buying.” There’s a lot of enterprise lock-in with a mentality like that.
That just didn’t happen that much. It did happen a bit, but we saw a lot of enterprises want to explore new models. It was simply good for a marketplace. Enterprises wanted to diversify outside of just proprietary frontier model labs, both for cost reasons and differentiation reasons.
They wanted to own their intelligence so they could, one, keep their talent and have an internal AI practice. AI is just a huge strategy topic. It’s not like you go to your board and you’re like, “Oh, yeah, we fixed the AI problem. Quarter complete.” Your board is asking you every month, “What’s next for the internal AI team?” Every single enterprise now has this internal AI team that they’re developing, and they need a strategy behind it.
It’s not just, “We checked the feature off, set up the database, and we’re done.” I think that dynamic has resulted in a desire to explore and diversify, and a desire to figure out how to reduce costs and how to do benchmarks for the first time in the company’s life.
I have been surprised there hasn’t been more benchmarking—more companies creating more benchmarks. It’s starting to happen, and I think eventually we’ll see a lot more of them, to demonstrate, “Oh, yeah, this thing is better than using Claude directly.” I think that’s going to be a bigger focus for this internal AI group at every company: evals.
4. Why owning your intelligence matters
I think Amjad’s been doing that a lot at Replit, for example. You guys have done a lot of cost-per-task research. You’ve made a doom-loop rescue. You’ve kind of been experimenting with new ways of using agents, like doom-loop rescue, and helping bring those to developers. More of that kind of research, I think, is going to pop up internally everywhere for all of those reasons.
Yeah, I think Satya, the CEO of Microsoft, has been very prescient on this and also very articulate on why companies need to own their intelligence. Ultimately, in the same way that we had dot-com companies and then every company became an internet company, every company employs people who know how to build websites and be on the internet.
Similarly, with software, every company has software engineers. Every company needs some AI practice, some AI capability, and that will compound over time: the knowledge and intelligence inside the company, the use-case-model fit—which models actually work for them—and how they save money. They need that independence.
5. The case against the god-agent
The other thing that I think Alex Karp of Palantir has been talking about is that there's a risk that, when you work closely with the foundation-model companies, they're going to move into your business. We've seen that with Figma, and we've seen that now with Harvey and OpenAI. It's really hard to partner with them because they see the world as their potential market.
When they talk to investors, they're like—you saw the SpaceX S-1—“Oh, $30 trillion.” What is the world's GDP, $100 trillion? There's a sense in which these companies are different from other generations of companies. It's harder to partner with them because their ambition is such that they want to subsume a big part of the economy.
Increasingly, what we're thinking about at Replit is in a similar vein to what Alex has innovated. Replit is becoming more of an independence layer inside enterprises, where we create a layer of interaction between you and the models, and we get you the best token at the cheapest price. We also create an abstraction layer on top of the cloud, because you should be able to deploy to AWS and Azure, and you should be able to use Databricks and Snowflake, and so on.
Increasingly, I think there needs to be more independence—not just with AI, but with all of technology. There need to be more platforms that help companies gain independence.
It seems like what OpenRouter did for their segment, you're doing for other areas of the business.
Yeah, there was this tweet I saw the other day where somebody was basically saying that all companies are building the same thing now. Everybody's building an agent loop with notifications, context, third-party connectors, context management, memory, and—
Sandbox.
Sandboxes, web search, agentic web search—
Computer use.
—and an always-on agent on top of it, with notifications. It's like this product is showing up everywhere. Yes.
In a way, yeah, it is showing up everywhere, but it also feels to me like these are just the new table-stakes primitives. It's kind of like a 2005 version of that tweet would be, “Oh, everybody's building the same thing: a database, a users table, a sign-in page, a sign-up page, a profile page, a logout page.” Everything's the same. There's a lot of differentiation, really; it's just that there are table-stakes needs for AI, just like there are table-stakes needs for the web.
Yeah, and I think that, inside the enterprise, making these products actually do real work is still an unsolved problem. You can use Muse in your personal life and connect it to your credit card and bank accounts, but no one's connecting Manus to their enterprise data—not even Grok and things like that.
I think there's an even greater emphasis on data sovereignty and security. We spent the past year almost working on making Replit deployable on your own cloud—basically on-premises, or bring your own cloud. Two years ago, I would have thought I would never do this, because the cloud is the future, software as a service, and all of that. But now we've reverted a little bit to a world where companies are more protective, because there are so many ways in which data can leak through all these agents that people are using.
There are all these screenshots on Twitter. I don't know how true they are, but Instinct or Muse is mixing people's data and starting to call you by a different name or something like that. The consumer stuff is obvious, but in the enterprise, there's still a tremendous amount of work for the entire industry to do in order to make these things useful and productive at work.
Are you doing any—do you have a custom personal agent, other than Muse or Instinct, that you use for work stuff, that you've been building?
Yeah, I built something on Replit a long time ago. I started with a sort of CRM agent initially. That was the main problem I had, but slowly we added features to it, and it's doing more and more things.
What's really interesting is that the more I connected Replit to all my stuff, the more it started answering things for me. Increasingly, the platform itself is subsuming the domain-specific agents that I built.
I do think that, in some ways, you want something that's focused on one particular thing, and you don't want it to be able to do everything. On the other hand, once you have your entire company's context in one place, it's really cool to join across totally different domains.
When I ask it a question, it can look at my personal chat history and join it across the GitHub repository and Salesforce. It links random things: “Oh, you met this guy a year ago at a conference. I see it on your calendar, and, by the way, someone else from their team is in discussion with your sales team.” It creates all these different synergies.
When I go into a meeting, I've connected a lot of different threads, and I'm making much more progress on a deal or something like that. This is where it's trending now.
Well, I think I'll take the counter on that. I think the worst part about doing cross-domain joins with your personal agent is that the more work you give it to do, the more understanding of what's going on you're sacrificing. Yet no one new is taking responsibility for that sacrificed understanding.
Agents don't have any responsibility. If there's a fixed level of cortisol that the whole company can tolerate among everybody, and I want to be less stressed about some area, I'm going to be sacrificing my understanding of it—
Someone else needs to take the cortisol.
But the agent doesn't—
—take on any of that responsibility. A universal agent that's doing all things means I can't adjust how much I'm sacrificing across all the different areas.
It points me a little bit toward, down the road, the subagents that people use being very vertically focused. Maybe we have a chief-of-staff-type agent that coordinates between them. But I feel like you do need vertically focused agents where you're saying, “This agent is more responsible psychologically for these things,” and you want quality checks to make sure it's doing those things correctly.
It doesn't need to focus on anything else. It just has one focus area. I wonder if that's going to help people get at least a weird, loose sense of responsibility on top of agents.
Fascinating. So you're saying general agents create a tragedy of the commons of sorts?
Kind of. I have a general agent that every day looks for things that need me and tries to figure out what to do. It's impossible to improve this agent. Every time I try to make an improvement, I end up ignoring its output about a week later.
It feels like it doesn't really care about any of the specific things it's diving into. Imagine having a chief of staff who's very good at drafting all of your replies across the whole organization. Compare that to having 10 chiefs of staff, each as competent as that one chief of staff, but each responsible for individual sectors of what makes up your life.
Compared to the gain you get from—
—the latter gives you a way of tuning how much understanding you sacrifice. I can lean in more to the areas where the agent is failing, and then have agents with very good competency take over my understanding of other parts of my life.
6. Specialization, Adam Smith style
Yeah, interesting. It's almost like rediscovering specialization. What's his name, the famous economist? Adam—
Adam Smith.
Adam Smith, with a pencil kind of thing. That was a huge realization for humanity: specialization is actually good.
The problem is that we over-specialized as a civilization, and I think overspecialization is oppressive in its own ways. There's the Marxist theory of alienation, right? The idea is that, because of overspecialization, people focus on just one thing. They don't see the fruits of their labor; they don't actually know what their impact is on the larger organization or the product they're producing. Therefore, they feel depressed and detached, and they're acting like a machine rather than being fully human.
And so maybe there's a bit of a reaction to that. With our agents, we're like, “There should be one god agent,” but in fact, specialization is really good for machines. That's the point that you're making: humans should be general, but machines should ultimately be a lot more specialized.
The problem with what I think I'm describing is that we don't know what good looks like. There hasn't been a system of specialized agents that feels as elegant as ChatGPT, Claude, or Muse, where you're basically just talking to one thing only. It's yet to be discovered. Maybe OpenAI just launched dots. I think they're experimenting in that direction. Grok bot, I guess, kind of counts as that.
But they're all very general. I think the idea behind GPT is that it's your digital double—as far as I understood it.
Well, when I saw Grok bot, I don't know that much about GPTs yet—they just came out—but when Grok bot first came out, the first use cases I saw people talking about were, “Oh, wow, I can make 2 bots: one that knows my bank account, right—
—and one that knows my Twitter account.” The 2 bots don't have each other's credentials. Yeah.
But they can talk to each other if they need to get something done. There's no credential sharing, though. That was one big unique thing I saw pop up a couple of times that people seemed to like.
But it looks like Muse is—though I wasn't sure.
Muse and Instinct are—
—have a much stronger product-market fit than Grok bot. Perhaps—
Seems like that.
Perhaps it is because you don't have to worry about creating these domains, but I think maybe personal agents are different from work agents. I think your critique of general agents pertains more to work and enterprise, which I sort of agree with. There's also all sorts of data-access considerations. As CEOs, we can have general agents because we have admin access. But for individual employees or certain teams, they can't have truly general, fully context-aware agents because there are access-control issues. So you'll have to work on something like specialization.
Ultimately, I also think we need to figure out what agent-to-agent communication looks like. I don't think there are good protocols around that just yet, and I don't think agents are trained to handle that very well. It seems like the next generation of OpenAI models are trained to do agent collaboration because we've seen it in the Hugging Face hack, where they start helping each other and it emerges naturally. But there also needs to be some way in which an agent can't convince another agent to give it information that it shouldn't give it.
There needs to be data isolation and proper ways in which these agents communicate. You almost don't want them to communicate fully in natural language. Maybe there's some other DSL or protocol that they need to follow.
I really think one of the cool potential applications of Jevons and other decision models like it is going to be alignment: checking to see if a tool call or an agent-to-agent communication is aligned, because there are just so many tool calls. You really need a cheap, fast model if you're going to block something like that. A really fast decision model that just classifies and gives feedback on rejections might be a good way to bridge the gap between agents and from agent to infrastructure, too. I haven't seen anything like that, and we have a little prototype that we're running internally at OpenRouter, but I think it could be an interesting alignment—
Using it for policy enforcement.
Yeah. Imagine looking at the system prompt and the current tool call being made and asking, “Is this aligned with the system prompt of the original agent and with these extra guidelines that maybe we didn't tell the agent about?”
For example, let's say you have a bunch of agents that are instructed to red-team some new product and they should not be able to access the internet. If they ever do, they should stop right away. But you might not want to explain all of that to the agents doing the red-teaming. You might want them to try to break out of the sandbox and act like bad actors. What would a bad actor do? It wouldn't be, “Break out of the sandbox, try to break into this company, and the moment you do, stop. Don't do anything else.” Mhm.
So having another model—for building a neurodiverse system, having another model check every single tool call or every single assistant message to see if it's indeed aligned with something that wasn't in the system prompt—I think makes sense. Then having structural safeguards, too, which is what I think NVIDIA just launched with its open-agent safety. I think it was called OpenShell. Companies are probably going to explore a combination of those.
7. Models training their replacements
I wonder—another thing about specialization and what you're talking about is that there's a lot of talk of recursive self-improvement. There's something I don't think is getting a lot of discussion, which is models training their replacements. You can think of it as a just-in-time compiler. The way just-in-time compilers work is that, as you're executing dynamic code, the interpreter realizes there's an opportunity to optimize it and emits machine code on the fly, which is a lot more optimized.
So you can imagine general models: you're doing something with Opus, or some of the Astra, or some of the big models, and they realize that the use case is limited. Or you prompt them in some way, or some other agent observing them realizes that the use case is limited. General agents have all these flaws that you just talked about, but there's also more potential for them to be harmful, more potential for them to go off the rails.
On the fly, it trains a model that could be its replacement but is a lot more domain-specific, and therefore cheaper and less vulnerable to prompt injections, less harmful because it's less capable. It's almost like a system that's training machine-learning models for specific use cases as it's monitoring the entire system.
Would that specific use case involve unstructured text generation or a very structured decision model?
It could be unstructured text generation. It could be decision models, like even in the case of Jevons. If you understand the inputs ahead of time, you could potentially take an off-the-shelf model like Qwen or something like that and train it specifically for that policy. That makes sense to do for cost reasons, assuming that the frontier model labs don't make very low-cost models that you can easily transition to—
Safety. Safety as well, right? Oh, yeah, I see your point.
They're so capable. I think oftentimes people are using these big foundation AGI models to do something that's like nuking a butterfly. Most of the time, a lot of the use cases, even unstructured use cases, don't need that capable a model.
8. Will smarter models deceive us?
I wish there were more public emails about this stuff, but a lot of the evals are private. You just can't see whether, when the models get more intelligent, the risk will continue to get higher, because I think there's also an argument to be made that alignment will get better and the models will start to avoid going off and hacking on their own as they get smarter and better at alignment, especially when it comes to agent-to-agent coordination.
This is something Noam Brown said on a podcast recently: as the agents have gotten smarter, they've gotten better at coordinating. It's still unclear whether they're going to be harder to align than humans when we get more and more of them. But if we can figure that problem out, would a smaller model be harder to align?
So Erik asked earlier: do I think it's true that smarter models are more aligned naturally, or that they're easier to align? Well, if you think back to the original rationalist LessWrong arguments for AI safety, there is this thing called the orthogonality thesis: the idea is that intelligence is orthogonal to ethics, morality, and so on.
I don't believe that's entirely true with humans. I think people who are generally more intelligent, or more educated, tend to—not always—be more considerate of animals, for example. But in machines, I think it could go the other way, because there have been quite a few studies on RLVR showing that reward hacking and deception just get better at it.
And the evals could be deceiving because the model could be smart enough to know that it’s being evaluated. We already know this. It’s been shown that if you do a lot of monitoring of chain of thought, models start lying in their chain of thought. So you add pressure on the chain of thought, and I think that, at some point, for you to do proper alignment evals, you need to run it for months.
Right? You need to run this thing for months on a really large goal or task in order for it to truly figure out whether it’s aligned or not. Yeah. I always struggle with this word “alignment.” It just feels wrong for so many reasons. It’s vague: aligned to what, and whose values? It doesn’t make the conversation easier.
I think in this case I’m talking especially about deception—the model actually deceiving its user. Maybe this reduces to: are we going to solve alignment? Are we going to prevent models from predictably deceiving users during training runs with better training? Will a model that’s big enough and powerful enough suddenly stop deception and stop sandbagging? Nobody knows the answer to that yet.
At the point when that does—if that ever does happen, though—we might see an interesting pressure for organizations to go toward the frontier, to have basically no risk or significantly less risk.
Oh, interesting.
Would they be willing to pay 10× to get that much?
I mean, it probably depends on the types of tasks they’re trying to do.
Some tasks just have way lower risk than others. Writing code or doing security research is the highest-risk type of task today. So you probably spend 10× to get a fully aligned model that can also find all the bugs, or a fully anti-deceptive model that can also find all the bugs.
One of the coolest things about decision models is that you fully control the structured output. Generally, with structured-output models, the room for misbehavior is much lower. You have defined tasks, and only machines are dealing with the outputs, and it’s not writing code that it can execute. Those tasks feel underrepresented in the things people talk about and in the work enterprises are dealing with, so I expect enterprises to get a lot more interested in them.
Yeah. I feel like we’re going to slowly realize how good we’ve had it with deterministic code. We’re going to be like, “Oh my God, remember the days when computers did exactly what we told them to do?” I think things like Jevons hint at more of a need for not only specialized models, but models whose output domain is more controllable. Maybe you could do a lot more than you thought you’d need by using a bunch of specialized models and specialized-output models.
Have you guys done any workloads internally with it?
You know, I’ve been training a lot of small models. I said this glib thing when it first came out because I left this Hacker News comment, which I felt disgusted with myself about afterward. But I’ve been taking a lot of Qwen 8B and asking Fable, Opus, and Astra to train a model.
For example, I trained a cost-estimator model internally so that when you put it in a prompt in Replit, we know exactly how much it will cost. It basically emits a probability distribution over multiple buckets: if it’s between $5 and $10, bucket A; between $10 and $20, bucket B. I’m used to training these classifiers by giving them different enums, essentially, and looking at the log probs per enum.
I’ve been doing it for a couple of years. I trained a chatbot to play by just doing that, so I’m already sort of pilled on decision models and specialized models. It wasn’t that big of a moment for me. But I understand that a true foundation model that’s fully promptable is an amazing user experience—an amazing developer experience—and you can do a bunch of things with it without training a model from scratch.
But if you have data, if you work at a place where you have the data—we have so much data at Replit—I ended up training a lot of specialized classification models pretty easily.
Yeah. It also feels like less model debt. Something I still hear from companies is that they’re worried about fine-tuning models for unstructured outputs because you’ve always got to redo it again in 2 months, and everybody feels the weight of the model debt. But a very bespoke classifier that’s trained with proprietary data—
You just—I feel like people won’t think it’s behind constantly, and it might last longer.
Yeah.
You don’t have to worry about its ability to speak a new language or write Rust or do anything that the LLMs are being evaluated on. You know the use cases, so you can build it more specifically. It seems like an easy thing for enterprises to build themselves and actually not regret.
Speaking of Rust, a good analogy is when the world got super excited about dynamic languages. If you think back to the ’90s, everyone was writing in Java and C++ and things like that. Then Python, JavaScript, and Ruby took over the internet, and everyone was like, “This is how you build startups really quickly.” You built Stripe, a financial organization, on Ruby. I was like, “How crazy is that?” And we built Facebook using PHP.
Then everyone was like, “We’re running into all these really bad bugs. It’s freaking slow, so let’s go in and add types. Let’s add a JIT compiler.” You end up reinventing everything. Then Rust came out, and I was like, “Okay, I guess we can use Rust for a lot of things we would otherwise be using JavaScript and Python for.”
My prediction is that the same cycle will happen here. We’re using these AGI-like models for all these different use cases, and then everyone’s going to wake up and be like, “Oh my God, this is so wasteful and so risky for no reason.” It’s got to be so much easier. We’re actually adding that capability on Replit, but I think it’s going to be everywhere. It’s going to be so much easier to go on a site, upload a CSV file, and get a specialized model that does one thing.
That goes back to your thesis about OpenRouter, this neurodiversity, which I really fundamentally believe in a lot more. Erik and I have had a lot of discussions about AGI, whether we’re truly on a path to AGI, and whether it’s even desirable to get there. I think the future is a lot more diverse.
9. Fusion models at half the cost
Outside of code review—which was, I think, the first time I saw people get really serious about using different model families to double-check the results of their main model—the research on mixture-of-models and composite models had been moving slowly for years. But things have been speeding up from my view of the research.
Now we see a bunch of AI agent labs. We launched a fusion tool, a fusion model, and Cognition launched one. They reduce cost and allow a wider breadth of ideas to be searched. Our initial launch was focused on deep research. The thinking is that if all these model apps are training on different sources of data, why not pull from all of them? This resulted in basically fable-level quality at 2× lower cost.
We just published results, actually, just today, about that. We showed a deep-research system through a combination of different things, including the harness, but you might think of it as a fusion-type thing—frontier-level at 40% to 50% of the cost.
And which models does it use? I think it changes over time, but one thing that has been interesting is that OpenAI added a feature that allows you to save computation across different model families. I think it also works across different effort levels.
So you wouldn’t miss the cache if you changed the effort. Don’t quote me on this. I think it’s also across different models, which is hard to fathom how. I might be wrong, though; I need to double-check that.
But I think staying within the OpenAI family has added a lot of efficiencies. In the past, we’ve done it with other models as well, because cache is one of the big things. Being cache-aware is one of the biggest things when you’re designing fusion models, routers, and escalation models.
Alex, this has been a great conversation. Thank you.