đŹ The Bitter Lesson is Coming for Proteins - Alex Rives, BioHub
Alex Rivesâs central bet is that protein biology will obey the bitter lesson: scale a generic sequence predictor across evolution, and useful biological structure will emerge without hand-coded priors. Evolution constrains which amino acids can coexist, so predicting masked residues should force ESM to infer hidden variables for structure and function. After increasing models roughly an order of magnitude per generation since 2018, Rives says, âI believe in scaling laws.â
ESM-Câs decisive improvement over ESM-2 came from widening the evolutionary data distribution, especially with billions of noisy metagenomic sequences. ESM-2 showed diminishing returns despite larger models; ESM-C, at approximately the same parameter scale but with more diverse data, showed âno longer diminishing returns to scale.â The thesis-relevant bottleneck is therefore not clever architecture alone, but access to amino acids across as many evolutionary contexts as possible.
The newly MIT-licensed system packages a protein world model, ESMFold 2, mechanistic-interpretability features, and an atlas spanning 6.8 billion nonredundant proteins. The team predicted structures for 1.1 billion representatives clustered at 70% sequence identity, giving structural coverage of the larger set, while sparse autoencoders expose features from basic chemistry through abstract function. Rives calls it âthe most comprehensive picture of protein structure and function thatâs been created.â
The commercially consequential capability is search-based protein design, particularly scFv antibodies reaching affinity levels needed for therapeutic activity in a small number of trials. Rives says therapeutic design âbasically emerges from that searchâ of a general sequence-structure-function model; he also claims significantly better antibody performance, where evolutionary information may be less useful. Full IgGs remain untested, although scFvs can be reformatted and he sees no reason the approach would not work.
The conversation treats static protein models as only the first rung; the larger prize is a virtual cell that predicts genuinely novel interventions in unseen biological contexts. Rives argues todayâs virtual-cell models represent their training data well but have âa very limited abilityâ to answer new experimental questions. Reaching useful cellular oracles requires perturbational and spatial biology, multimodal measurements, experimental feedback, and models spanning molecules, genomes, cells, and ultimately physiology.
Biohub is committing $400 million internally and $100 million externally to build that missing data stack, while acknowledging this is only a fraction of what is needed. The near-term plan is to scale existing assays 10xâ100x, then develop technology for another 10x or more while expanding interventions, measured modalities, and biological contexts. âWe canât wait decadesâ; Rives wants the enabling data created within a couple of years.
Neither protein data nor compute appears exhausted: ESM-C used roughly one billion sequences, while Rives estimates there may be on the order of 100 billion available sequences. Small variations should not be discarded as redundant because they may teach functionââa single mutation is enough to destroy the function of a protein.â A 100x compute increase would improve ESM-C, he says, but only if data scales in tandem; how long returns persist remains âtruly an empirical question.â
1. Evolution supplies the supervision that protein models need
Rives traces the program to summer 2018, when his team at Meta AI trained an early transformer language model for proteins. Across subsequent generations, increasing scale roughly an order of magnitude repeatedly produced new capabilities.
The biological premise is concrete: residues touching in a folded protein cannot evolve independently. A change at one position requires compatible changes elsewhere, leaving statistical patterns in sequence databases that reflect underlying structure and function.
Rivesâs task formulation is deliberately generic: mask amino acids and predict âthe amino acids that evolution will choose.â Solving that across billions of sequences pressures the model to infer the otherwise-hidden constraints shaping proteins.
The hostâs challengeâproteins are not natural languageâdraws Rivesâs empirical answer: AI lacks a complete theory of when scaling transfers, but evolution has already generated an enormous training set through âfour billion years of life running experiments in parallel.â
2. Metagenomics broke ESM-2âs data ceiling
ESM-2 improved from roughly the billion-parameter to 10-billion-parameter scale, yet its structure-representation curve showed diminishing returns. Rives now interprets that result as data limitation rather than evidence against scaling.
UniRef provided curated, clustered coverage of known sequence biology. ESM-C added metagenomic material collected indiscriminately from hydrothermal vents, polar environments, deep oceans, soil, human guts, and other ecosystems.
The metagenomic data is messy by design: researchers sequence environmental DNA, translate likely proteins from fragmented contigs, and often lack complete genomes, organism identities, or certainty that every inferred sequence is a full protein.
That noise bought diversity. With approximately the same parameter scale as ESM-2, somewhat more compute, and billions of additional sequences, ESM-C produced a clean scaling curve whose smaller-model extrapolation predicted the representational fidelity of larger models: âThe data was really the critical thing here.â
3. ESM-C turns a language model into an open protein atlas
Rives describes ESM-C as the fourth generation, trained a little over a year before release and now fully open-sourced under an MIT license. The family contains 300-million-, 600-million-, and 6-billion-parameter models.
Around the language model, the team built ESMFold 2 for structure prediction and sparse-autoencoder tooling for exposing learned features. The result is meant to be a world model spanning protein sequence, structure, and functionânot merely a next-token predictor.
The atlas combines major sequence databases into 6.8 billion nonredundant proteins. The team clustered them at 70% sequence identity and predicted structures for 1.1 billion cluster centers; related members should share the same fold, with smaller variations.
Those structures add hundreds of millions of entries to the accessible picture of protein diversity. Computed features also link distant proteins that share functional or structural patterns despite weak sequence similarity.
4. Interpretability reveals biology the model was never explicitly taught
Sparse autoencoders trained across the ESM-C layers reveal a hierarchy resembling biologyâs experimentally developed reductionist picture: biochemical properties and structural building blocks at the bottom, then large functional themes and abstract concepts.
The nucleophilic elbow is Rivesâs sharpest specimen. Protein families with different topologies may have evolved this structural motif independently, yet ESM-C uses âa single featureâ for it across those evolutionarily distant families.
His explanation remains a hypothesis: prediction requires compression, and compression creates latent variables. Because every amino-acid choice is entangled with the rest of the sequence, reusable concepts such as a nucleophilic elbow help the model predict many otherwise unrelated contexts.
The same feature space clusters distantly related gene-editing systems and other proteins whose functions are unknown. Some might be undiscovered editing systems, though experimental validation is still required; Rives notes that Feng Zhangâs group used the first ESM Atlas to find a new gene-editing system.
5. Distributional structure offers a hypothesis for emergence
Rives invokes Zellig Harrisâs 1954 âDistributional Structureâ: the contexts in which a word appears are constrained by its meaning, so statistical structure can recover semantic structure without receiving explicit definitions.
His biological analogue is direct. The contexts available to an amino acid are determined by a proteinâs structure, function, biological role, and relationships to other proteins; learning those context sets should therefore expose the hidden biological variables producing them.
This framing also explains why small sequence differences are valuable. Broad evolutionary diversity teaches structural abstractions, while dense variation within families may teach function at the resolution where one mutation can destroy function.
6. ESM3 and ESM-C offer two routes to programmable biology
Rives says ESM3 was consistent with the ESM philosophy and that both approaches have a place. ESM3âs goal was explicit programmability, using sequence, structure, and function tracks so biologists could prompt the model with the right biological information.
ESM-C approaches design as world-model search instead: specify desired criteria, then search its predictive landscape for molecules satisfying them. It has generated mini-protein binders and, more notably, scFv antibodies.
The hosts compare this with coding agents that begin with broad pretraining and acquire programmability through post-training or reinforcement learning. Rives calls conversion between these approaches promising, but says the right method is not yet understood: âWe need both.â
7. Antibody results test whether general models beat specialized pipelines
An scFv combines one antibody heavy-chain component and one light-chain component into a single chain, allowing a complex binding interface. Rives estimates antibodies account for roughly a quarter of new drugs, making this a consequential therapeutic modality.
In a small number of trials, ESM-C search found scFvs at affinity levels needed for therapeutic function and activity. Rives emphasizes that this behavior came from a general protein model rather than a system trained solely to engineer antibodies.
The hostsâ pushback is important: mini-binders are increasingly routine, whereas nanobodies, scFvs, and especially antibodies become harder; antibody diversity also makes multiple-sequence alignments less naturally informative than for conserved proteins.
Rives has not tested full IgGs. scFvs can be reformatted as antibodies, and he sees no reason full-IgG design would not work, but that remains a prospective claim; his current, narrower assertion is that ESM-C performs âsignificantly better on antibodies.â
8. Fast structure prediction is an on-ramp, not a virtual cell
ESMFold 2 does not require multiple-sequence alignments, so it can produce atomic-resolution predictions directly from sequence in seconds. Rives says the system is state of the art for open models on multimer prediction.
A host proposes predicting every pairwise interaction in the human proteome as an initial interactome. Rives agrees this computational proxy could be valuable. The host also notes that static structures omit dynamics that are important to much of cellular biology.
At Biohub, the team is building cryo-electron tomography with higher cellular contrast. Rives hopes this kind of work will eventually enable structurally and empirically resolved interactomes, despite major technical hurdles.
First-principles simulation cannot yet bridge the gap: even physical folding simulation works only for a few fast-folding proteins. Rives also presents an information-theoretic view of the cell, linking genome, transcription, cellular programs, and phenotype.
Historically, the field expected protein-structure prediction to come from first-principles simulation; machine-learning pattern recognition instead made major progress. Rives argues that learning the underlying programs of cellular biology may provide the right abstraction available in the current era of information theory at scale.
9. Virtual biology requires a scaled experimental feedback machine
Rivesâs standard for a virtual cell is generalization: it must predict an experiment absent from its training data. Current models are âgood representations of the underlying dataâ but weak at novel interventions in novel contextsâthe capability fundamental science actually needs.
Biohubâs initiative assigns $400 million to internal data generation and enabling technology and $100 million to outside efforts. Priorities include Perturb-seq, combined transcription and imaging, spatial biology, and simultaneous phenotype, transcriptomic, proteomic, genomic, and epigenetic measurements.
Existing programs may encompass roughly a billion cells by Rivesâs estimate, but he wants multiple orders of magnitude more. Current technology might scale 10xâ100x with reasonable investment; another 10x or more requires better assays, automation, flexible robotics, and cheaper multidimensional measurement.
Feedback completes the thesis: models could reason over thousands, millions, or even hundreds of millions of hypotheses, select a small number of experiments, observe outcomes, and update their representationsââsomething like RLVRâ grounded in biology. Compute and data must expand together; whether ESM-C eventually hits diminishing returns remains empirical.
Full transcript
So ESM-C is also approaching programmable biology, but I would say in a very different way. It's approaching it from this kind of world-modeling perspective, where the idea is basically that you have a predictive model and you're going to search the world model to find protein molecules that satisfy whatever design criteria you have. We've been able to use this to design many protein binders. But I think, most excitingly, we've been able to use this to design antibodies, scFvs.
Thank you for having me here. It's great to be here. I'm head of science at EvolutionaryScale. I'm a computer scientist, and I work on AI for biology. A lot of my work has been on language models for biology.
swyx
1. The Bitter Lesson For Proteins
By the time this podcast is released, you will have put out several new, exciting, interesting models. Going over them, I couldn't help but have the thought that you might be the most Bitter Lesson person in protein biology right now. Can you give a little context about what that means for biology and why you're so committed and excited about this route?
Well, I'll take that. I believe in scaling laws. I guess I've been working on this since the summer of 2018. My team, when we were at Meta AI, trained really the first transformer language model for protein biology. I've always thought that there would be an emergence of biological information as you train a model to predict the next token that evolution creates.
So our team has really explored that idea over a number of years, and we've really seen the scaling curve. We've seen that, as we have increased models by an order of magnitude in each generation, there's this emergence of new capabilities.
swyx
Yeah. So you've been talking about the emergence of capabilities and scaling over generations. You've been working at this for, I guess, 8 years now or something like that. It didn't always work that way, right? There were signs that scaling might work.
We'll be getting to some new results where I think you've really clearly demonstrated this hypothesis in a way that hasn't happened before. But you seem to have a strong commitment to this in a way that I'm not necessarily sure I would have been so convinced would work in the same way. Protein language is not the same thing as natural language. There are similarities, but if you start sampling a normal language transformer at temperature, you're going to get gibberish.
If you sample a protein language model at infinite temperature, you're going to get something that is a valid protein, if not an interesting protein, despite the fact that it's a different domain for a different reason. So what is specifically about proteins that you thought was special, or that would make this valid as well?
2. Protein Sequences Encode Biology
Yeah, I mean, it's a really interesting question. I think it's a deep question across AI right now more broadly. What's so interesting is that AI right now is such an empirical science, and we don't have theory that can always guide us in these things. But we have this really strong empirical evidence of scaling.
The thing that motivated me is, if you think about evolution and the data that we have around proteins, we have databases that have billions of protein sequences. Those sequences contain patterns, and it had long been knownâdecades before we started working on this with language modelsâthat there are patterns in the sequences of protein families that arise because of the constraints that evolution is operating under.
So you can think about a protein sequence that folds into a three-dimensional structure in space. You can imagine that there are 2 residues, or amino acids, that are in contact in that folded structure. Evolution isn't free to choose those independently from each other. If it makes a choice at one position, it has to make another choice that's going to be compatible at the next position.
Going back all the way to the beginning of gene sequencing, when people first began to be able to look at the same protein in related organisms, you could start to see these patterns that reflect the fundamental underlying biology.
So the idea behind ESM was, what if you were to apply this principle across all of evolution, across the vast diversity of proteins that have been generated across all of life, and basically have a language model predict the amino acids that evolution will choose to place in proteins across all of those biological contexts?
You can think that there's just this incredible amount of information in that total picture about the underlying biology of proteins. That was really the idea that sparked this. As a model is having to predict the next tokenâand we train these models with masked language modeling, so they're predicting tokens that are masked out of various parts of the sequenceâit would have to learn something about the underlying constraints that are shaping which tokens evolution can choose.
3. Building The Protein World Model
Alessio Fanelli
Yeah. So maybe for a bit of history: you just released ESM Cambrian, right? Is that what it's called?
Well, they were called ESM from the start. We had various branches of the different models. This one, I would say, is a fourth-generation model. It's actually a model that we trained a little over a year ago. Now that we're at EvolutionaryScale, we're open-sourcing this model fully under an MIT license for the first time, so we're really excited to do that.
The big thing that's new here is that we've really built a world model of protein biology. The foundation of that is ESM-C. Using the representations of ESM-C, we've now built a structure-prediction model. This is the next-generation ESMFold model.
We've also used the techniques of mechanistic interpretability and sparse coding to really start to look deeply into the representation space of the language model and pull out the underlying features that the model actually uses to represent protein biology.
Bringing all of this together, we're able to make predictions for protein structure and predictions about the underlying features that proteins are made out of. That allows us to build linkages across evolution. We're able to take this model and invert it to design proteins, and we've used this to create a comprehensive picture of protein biology.
We put together the world's largest protein sequence databases. That amounts to 6.8 billion nonredundant proteins. We've resolved or predicted structures for 1.1 billion of those, and we've also computed features across all of them so that we can make these linkages basically all across evolution and protein biology.
Alessio Fanelli
6.8 billion, of which you've resolved structures for 1.2 billionâis that right?
1.1.
Alessio Fanelli
1.1. So what about the others?
Basically, what we did is take that database and cluster it at 70% sequence identity. It's really resolving structures for everything in the sense that, for each cluster, we have a cluster center. We're predicting the structure there, and we can expect that the other proteins are going to have a similar template structure. There will be small variations, but they have the same fold.
Alessio Fanelli
1.2 billion or so clusters that are covering the 6.8 billion?
That are kind of covering the 6.8 billion, yeah.
Alessio Fanelli
Okay, interesting. Since we're talking about scaling, how do you know that this is the right number? How do you know that focusing on these 1.1 billion is the right resolution for this model?
We've chosen them so that they really cover that entire space. What I can say about this database is that it's really the most comprehensive picture of protein structure and function that's been created. It's adding hundreds of millions of structures to our knowledge of the diversity of protein structure, and it's also creating this feature space that allows us to find these linkages between proteins across evolution.
We can see really interesting themes emerging across evolution, linking, for example, gene-editing systems that are very far apart in sequence but share some kind of underlying functional patterns or structural homology that the model is able to bring together and find those connections.
Alessio Fanelli
Now we're talking about the mechanistic-interpretability part. If I understand correctly, you use sparse autoencoders and other techniques to understand what the patterns of outputs are when you activate the network using a protein, and how they relate to each other.
You have these sequences that are unrelated or only partly related based on the actual sequence, but in terms of behavior they have similar behavior and therefore activate similar networks. Is that right?
swyx
Is that kind of the summary of what you just said?
Yeah. Basically, what we've done is train sparse autoencoders across all the different layers of the ESM-C model family. There are actually 3 models in that family: a 300 million parameter model, a 600 million parameter model, and a 6 billion parameter model. We've done a very deep analysis of the feature space of that 6 billion parameter model, which is really the state-of-the-art protein language model.
What we find that's really interesting is that there's this hierarchy of features that emerges. What's really interesting about it is that it corresponds to the reductive picture of biology that has been developed over many decadesâa centuryâof biological experiments. What's so cool is that this is emerging without any prior knowledge; it's been learned by the language model.
The interesting thing about SAEs, right, is that they're really just revealing the intrinsic structure of the representation space. This model's been trained on protein sequences. It's been trained just to predict the amino acids that evolution will choose. Somehow, this is leading to the emergence of this very ordered feature space that has a hierarchical structure, where you can really see everything from the basic biochemical properties and the basic structural building blocks of proteins to these very large functional themes, these abstract concepts that connect to the human picture of protein function.
swyx
Do you have a hypothesis or feel for why there are relationships between the sequences themselves, even if they're shifted, cut up, and recombined in different ways? I can imagine that might work because proteins are hierarchical in their nature as well. So maybe the hierarchy moves around, but the functional units, I guess, have related structures. What is the hypothesis here?
I mean, it's a really interesting question. I think I can speculate about it. I don't think we completely understand this, but let me give a concrete example.
The nucleophilic elbow is this core functional motif that people have thought maybe has emerged independently in evolution, at different times in different protein families. But it has this very clear structural motif that you can see in a crystal structure. What we found, basically, is that the model has a single feature for this nucleophilic elbow, and it's activating across these very evolutionarily diverse familiesâreally completely different structural topologies, proteins that probably evolved entirely independently from each other. But the model is using this one feature to represent that.
Why does it do that? I think it's a really interesting question. One answer is the idea of compression and the idea that the model needs to have some kind of underlying latent variables that it develops to help solve this sequence prediction task. What's so interesting is that the choice of any amino acid is completely entangled with the choice of all the other amino acids in the sequence. This is a very complex task: trying to predict what amino acids should be where in a protein.
To really do this well, the model would have to have these hidden variables that are representing the biology, which allow it to look at a protein and say, âOkay, what amino acids should be there in all these different contexts?â That's the intuition. I would draw the parallel to language modeling.
I was very influenced by a paper by Zellig Harris called âDistributional Structureâ from 1954. I think that paper influenced a lot of people in the language-modeling field as well. It focuses on language and really articulates this idea that the set of contexts in which a word appears is determined by the meaning of that word. What Zellig Harris imagined is that, as you looked at the statistical patterns of what words appear in what context sets, you would be able to derive the meaning of language. You would have this statistical structure that would mirror the underlying meaning of language.
For me, at least, that's one of the most convincing explanations for why a language model that's trained on the text of the internet is going to learn something about meaning. It's going to learn something deeper and more fundamental. I think you can think about the same thing in biology, where the contexts in which an amino acid can occur are really determined by the structure and function of the protein, its biological rolesâthese very complex phenomena, both the intrinsic biology of the protein and its relation to all of the other proteins and their function and evolution. Those are what determine the context sets, and so you would imagine that those statistical patterns in the use of amino acids directly reflect those underlying hidden variables. The model is going to learn something about those hidden variables.
swyx
4. Metagenomics Unlocks Scaling
I definitely buy that. It seems plausible. I want to be clear: I actually do really believe in this direction, but there are a lot of ways I think about this where maybe I could say, maybe I would imagine it wouldn't work. One of them is data availability. What type of data do we normally have? What type of sequence data do we normally get?
I think ESM-C in particular has some new data sources compared to previous models, which might be helpful, but oftentimes the types of sequences we have available have a very strong bias toward certain specific needs for medicine or human biology or disease biology. It's not necessarily the case that, if you take just a naive data set, you're going to necessarily get an interesting scaling law.
I'm curious about what in particular was the breakthrough in ESM-C. Maybe we can go back a bit and talk about some of the other ESM predecessors, which got here before ESM-C, and how their strengths, but also maybe some of the limitations that ESM-C overcame, led to the developments there.
Alessio Fanelli
Yeah. Well, I'll admit that I am a bit less informed. I am a scaling fan.
I do think that just increasing the data and increasing the parameters and having that compression is going to lead to more powerful models. But it is also trueâand I think you're absolutely rightâthat the underlying structure and distribution of the data is really critical. Some data sets will be far more valuable for learning these general principles than others.
But I think it goes against a lot of biological intuitions about collecting data, I guess, is what I'd say. Normally, when you think about what data you want, you're trying to answer a very specific scientific hypothesis. You want a very well-controlled experiment. You really want multiple replicates. It's something very focused, is the way that I would put it.
I think the change in the way of thinking is to think, okay, what you really want, if you want to learn a general representation of proteins, is to see amino acids in as many evolutionary contexts as possible. That's really what you want. That's really how I think about data.
If you look at what changed between ESM-2, which was kind of the previous-generation model, and ESM-C, which is this new-generation model, they're both at approximately the same scale. Andâ
Alessio Fanelli
The same scale of compute.
Same scale of parameters. Yeah, ESM-2 got a lot of compute, but ESM-C got even more compute. But it's not just the compute. The data was really the critical thing here, actually.
When we trained ESM-2, we observed 2 things. The first was that, as we increased the number of parameters and compute, we saw improvements. We had a model at the billion-parameter scale. We had a model at the 10 billion-parameter scale, and the larger-scale model is better than the smaller-scale model.
But if you look at a plot of parameter scaleâsort of a log plot of parameter scale versus capabilityâand for capability, we're looking at representational fidelity: how well does it capture protein structure? You could see diminishing returns in ESM-2. ESM-2 is trained on UniRef. For ESM-C, we added metagenomics. We added billions more sequences to the training data.
swyx
Could you explain what UniRef and metagenomics mean?
Yeah. UniRef is, I'd say, the gold-standard data set of sequence biology. It's taking sequences from across a wide variety of different sequencing resources and clustering them to remove some of this redundancy that you were mentioning. It creates definitive coverage of protein biology.
In parallel to classical gene sequencing, there's this idea of metagenomic sequencing, where people go out into all kinds of different biomes and environments and collect samples from the world, then just sequence the natural diversity that's present there.
So, proteins from a hydrothermal vent, or proteins from a frigid environment near the South Pole, or the deep ocean, soil, the human gutâall kinds of different environments. This is a very different way of collecting data.
Instead of trying to understand the specific genome of a specific organism or a specific protein, you just collect a bunch of stuff, mix it up in a pot, and get the sequences out. You have no idea what organisms these are from. You don't necessarily even know if a given sequence is a protein, but you can guess based on certain contexts and say, âOkay, we threw these together. These are likely protein sequences we found. We're not assigning them to an organism. We're not assigning them to a larger context. We're just saying this is probably a protein. Let's train on it.â
swyx
That is right. Yeah. You don't even get the full genomes. You just get these kinds of contigs that are often broken and have even partial proteins, so the data is really noisy.
One more little nerdy question that I have here: If I understand correctly, you're not actually using a device that sequences proteins. You're sequencing the DNA that would manufacture those proteins, so you're finding DNA and then looking for markers that indicate the beginning and end of a protein sequence. Is that kind of right?
Yeah, that's exactly right. Basically, we're sequencing genetic sequences, and then we translate the proteins from those sequences.
swyx
So you're digging up, like, sewersânot you, butâ
Me personally, there are sewers, like probably many New York City subways, all kinds of things.
swyx
Yeah. So the natural question to me is: You built this model and you think that you've de-duplicated it, so you have a good representational set without a lot of redundancy in it. How much more is there? If we had an order of magnitude more resources, do you think that there is an order of magnitude more proteins to discover?
I think so. I'm not entirely sure, but there are a lot of proteins, and I think we've barely scratched the surface of measuring Earth's biodiversity. There are core proteins that are conserved across all of life, so I think we know those. But as you go into these different environments, there are constantly new genes and new proteins being created by evolution.
swyx
A lot of my understanding is that this is viruses, bacteria, and otherâ
Microorganisms. Those guys are basically in this long-running conflict with each other that causes them toâ
swyx
Recombine their DNA in ways that help them survive in these extreme, or whatever, environments. And so that's what's causing this incredible diversity of proteins.
That's right. Yeah. And just 4 billion years of life running experiments in parallel all across the Earth, in all kinds of different ecological niches. We just see the outcome of all of that.
Shawn Wang
And the combinatorial effectâthat's why you believe that there's going to be so much more diversity, although maybe from a macroscopic perspective, when we look at it, there's not even nearly as much diversity as there will be at the microscopic scale, because you have this incredible combinatorial effect.
Alexander Rives
Yeah. I mean, there's just tremendous diversity there. So, kind of going back thereâ
Shawn Wang
Yeah, I know. It's great, right? And I think it's reallyâI mean, we could also talk about data and building models of the cell, and really going from the molecular level to higher levels of biological complexity. But, to complete the description of ESMCâ
Alexander Rives
The big change was adding these metagenomic sequences. What we saw, basically, is that there are no longer diminishing returns to scale. That's really saying that ESM2 was data-limited rather than compute-limited for ESMC.
There's a really beautiful scaling law that we can plot. We can train models at the smaller scale, and we can look at the best representational fidelity they can achieve for a given compute budget. Then we can draw a line of extrapolation that beautifully predicts what the larger-scale models will be able to achieve in their representational fidelity.
There's this really beautiful scaling. The only changes to ESMC are some changes to make it a more efficient model for training, but I think the data is really the big thing that's driving that.
Shawn Wang
So it still is basically just a standard vanilla transformer, with a few tricksâeveryone has a few tricks at this pointâa language model, and just a lot of data?
Alexander Rives
So, I mean, this is very much in contrast to something like AlphaFold, where you have a lot of inductive bias built into the model in order to be able toâ
Shawn Wang
Predict protein structure.
Alexander Rives
That's right. And the idea here is, can we just learn the right structure? Don't give any priors; just allow machine learning to figure out what that structure is.
Shawn Wang
You also had your own detour into priors with ESM3âor maybe not priors, but using more intuition or more human design. Do you think ESM3 was a detour? Did you just end up saying, âOkay, let's make ESMC bigger,â and then suddenly it worked, and now you learned that we actually don't need priors anymore? Is that a key insight, or do you still think there's room for priors?
Alexander Rives
I think we need both. I think there's a place for both of them. The goal for ESM3 was to really make biology programmable. We were trying to think, âOkay, what is the programming language? How are you going to be able to allow biologists to prompt a model and design structure and design function and all these things?â We really thought it needed the right tracks.
I would say that ESM3 was very consistent with the philosophy of ESM, because what we did was predict structures for this vast array of evolutionarily diverse proteins, and we used that as the training data. The model is now learning from sequence patterns, structural patterns, and functional patterns.
I think that same kind of synthesisâthe model learning on sequencesâcould be extended by bringing in more multidimensional information. That would build an even better representation space.
Shawn Wang
If you're a coder, or if you're building language models and then building coding agents, you start with pre-training on everything, and then you go to doing the programming part through some sort of post-training, probably RL. Have you thought about post-training ESMC to try to give it the same abilities for programmability? Do you think you could get programmability without all of the inductive biases, which involve an atlas of structures and some sort of interesting distillation? I guess maybe that is some kind of post-training of a different model.
Alexander Rives
Yeah. I think it's a really interesting question: To what degree can you interconvert these models? I don't think that's fully understood yet, but I think it's a very promising direction to think aboutâwhat are the right ways to do that?
5. Searching The Protein World Model
Shawn Wang
So, ESMC is also approaching programmable biology, but I would say in a very different way.
Alexander Rives
It's approaching it from this kind of world-modeling perspective, where the idea is basically that you have a predictive model, and you're going to search the world model to find protein molecules that satisfy whatever design criteria you have.
We've been able to use this to design mini-protein binders, but most excitingly, we've been able to use it to design antibodiesâscFvsâand we're seeing really exciting success rates in a small number of trials now.
Shawn Wang
Can you explain what those scFvs are?
Alexander Rives
Yeah. An scFv is basically a single-chain antibody. It's a kind of therapeutic modality. An antibody has a heavy chain and a light chain, and it basically has a pairâ1 heavy chain and 1 light chainâthat come together to recognize a target.
There are different variations of these kinds of modalities that are used therapeutically. What's interesting about the scFv is that it has 1 heavy chain and 1 light chain. It's able to form these very complex binding interfaces, where you can have 2 different subunits coming together to engage a target.
These are important therapeutic modalities. Something like a quarter of new drugs are antibodies, so it's really 1 of the critical modalities for medicine. What we're able to see is that you can search ESMC and actually find antibodies that are reaching the level of affinity needed for therapeutic function and activity.
Shawn Wang
The protein design space has kind of exploded in the last 5 years. Pretty much everyone is doing protein design, and many people are excited about protein design. My high-level, naive understanding of the field is that things like mini-binders are quite doable.
People have done that quite routinely and successfully in smaller proteins. By the time you get to nanobodies and scFvs, theyâre a little bit harder to design, and antibodies are still actually quite out of reach oftentimes. One of the common reasons for this is that if youâre in the AlphaFold paradigm, you donât have MSAs, right? The evolutionary pressure for antibodies is actually the opposite, in many ways, of what the evolutionary pressure is for everything else: they go for diversity rather than trying to evolve along a very constrained path. So Iâm curious: did you try larger structures, and is that something that youâve seen success on, or is this something that you still think, for some reason, might be hard to do?
Alexander Rives
We can actually take the scFvs and reformat them as antibodies. I think that would be the quickest approach to do that. Weâve not tried full IgGs. I donât see any reason why that wouldnât work. Actually, itâs something we havenât done yet.
Weâve decided weâre basically releasing this now because we feel like itâs reached a point where weâre seeing a really significant step above whatâs been possible in the past. We just wanted to get it out there. I think thereâs a lot more progress thatâs possible. We have a lot of collaborations to look at some of the other applications here.
The thing about it, right, is that itâs a general model. To me, thatâs the most exciting thing about it: a general model for protein sequence, structure, and function. You can search it, and therapeutic design basically emerges from that search.
Shawn Wang
Yeah, the fact that youâre not using MSAs, or multiple-sequence alignmentsâwhich was one of the, or maybe the, critical insights that allowed AlphaFold to work really wellâand the fact that you didnât need that in order to make it work basically as well as AlphaFold 3 is really exciting to me. That means that your thesis is, letâs cover the space of possible proteins as well as we can and see what the emergent behaviors are. If this is an emergent behavior that weâre able to replicate, what happens with multiple-sequence alignments when we use multiple-sequence alignments? What are the other things that maybe we donât have data for but that weâre able to also do in an emergent way?
Alexander Rives
I would say, actually, weâre doing significantly better on antibodies. I think thatâs one of the theses that we had: antibodies are not going to benefit from evolutionary information probably in the same way that predicting the structural topology of a molecule will. I think you see that now, where the representation space contains something thatâs really interesting about antibodies here.
6. Virtual Cells Need Generalization
Shawn Wang
I want to talk about something, because you mentioned something very interesting to me: virtual cell and how this maybe interfaces with this work here. Iâm really interested to know: were you able to find other things in your mechanistic interpretability? What were some interesting things that werenât just validating biology, but where there was a pattern that was unexpected? Did you find anything like that?
Alexander Rives
Itâs complicated. We have to now actually go and validate some of these things, right? I think what we saw were interesting connections. What we can see, for example, is that distantly evolutionarily related gene-editing systems cluster together in this space in ways that are consistent with and reflect our knowledge of the origin of those gene-editing systems. So thatâs really exciting.
But the thing is, there are a number of proteins in that map that are brought together in different ways where we just donât know what they are right now. We donât know what they do. One hypothesis is that these are novel gene-editing systems. I think in this atlas, thereâs going to be some really interesting basis for scientific discovery there.
If you think about how people go out and look for new gene-editing systems, for example, theyâre typically mining the large genetic sequence databases, and theyâre looking for different sequence patterns or structural patterns that are linked to that. Actually, the first version of the ESM Atlas was used by Feng Zhangâs group to find a new gene-editing system. So I think thereâs just a lot of biology out there that we donât understand thatâs waiting to be discovered, and being able to connect the dots between proteins so that we can go from what it is that we know today to make those inferences about the unknownâthatâs what Iâm excited about.
I think there are proteins for so many applications that nature has probably invented. You think about the thermostable polymerase, which enables PCR, that came from a bacterium living in a thermal hot pool. There may be a solution to climate change somewhere in protein biology. There are probably all kinds of building blocks for completely green chemistry infrastructure out there. There are probably new medicines and therapies, but the question is: how do you find those? I think being able to connect the dots is really one way to start opening up that space of protein biology to discovery.
Shawn Wang
Iâm curious: one of the advancements of ESM-C is an improvement in multimer predictionâbasically, proteinâprotein interactions, the ability to predict the way 2 proteins interact. I think you now claim to do better than anyone else, right? Correct me if Iâm wrong.
Alexander Rives
Yeah. I think weâre state of the art for open models.
Shawn Wang
Okay. One thing that I know some people would find very useful for virtual cell is just an entire mapping of every single pair of proteins inside the human proteome. Have you thought about doing this as a beginning to a virtual cellâcreating that map?
Alexander Rives
I think something like that would be really valuable. The other thing about ESMFold 2 is that itâs a really fast model because it doesnât require multiple-sequence alignments. You can do inference directly from the sequence. It takes seconds; you can get an atomic-resolution prediction.
Thatâs one really interesting application at Biohub. The other thing that weâre thinking about is whether we can actually experimentally resolve this. One of the things that weâre building is cryo-electron tomography, and weâre really building systems that can greatly increase the contrast when youâre looking at the cell at the atomic level.
So I think one thing that I hope to see is actually structurally, empirically resolved interactomes at some point in the future. I think there are some pretty big technical hurdles and technologies that have to be developed to overcome that, but I think thatâs something thatâs going to be possible. We can use computational methods to start to get a proxy for that, and I think thatâs going to be really powerful. But I think a lot of the future of structure prediction is going to turn into structure determination, actuallyâreally bringing together these tools that we have for modeling proteins and bringing them together with experimental data so that we can start to develop this picture thatâs informed by empirical biology, by what we can observe.
Shawn Wang
So is that the vision here, if Iâm understanding correctly? You have maybe a lab-in-the-loop kind of thing where you have an agent thatâs talking to your CRO and whatever, and then it predicts a property that youâre interested in. It sequences the genome, or it creates the genome. It creates the protein from the genome, and then it observes it with some version of this microscope. What did you call the microscope again?
Alexander Rives
Cryo-electron tomography.
Shawn Wang
Okay, okay. And then you do whatever experiments, or you observe it, and then you use this as a lab-in-the-loop to say, âOkay, this folds this way. Therefore, I want to check the next one that I want to check,â and use an active-learning system. Is that sort of the vision that youâre articulating here?
Alexander Rives
Well, I think there are going to be a few fundamental principles for the next era of biology. I think itâs such an interesting time right now because weâre really at the beginning of a new scientific paradigm. Itâs really just the beginning of it. So what is defining that paradigm, right?
I think there are a few principles. Data generation is going to be really critical. The second is computational, predictive digital representations of biology. We can talk about that: you can think of ESM as being a first generation, AlphaFold as being a first generation of those kinds of approaches. You can start to think about what that looks like as we can model more and more biological complexity in that way.
Then you have the principle of feedback, and you have the principle that intelligence is now scalable and so can be applied to every unit of a biological problem. What would it mean for all of that to come together? I think weâre going to have increasingly capable and accurate digital representations of molecules, genomes, cells, and ultimately physiology.
That's where you want to get. We're going to have to go up that complexity scale, through the levels of biological complexity, which requires traversing a data barrier. There's, I think, data that does not exist that needs to be generated to achieve that level of predictive fidelity. And then we're going to have reasoning. I think what that will mean is that we can reason over thousands, millions, hundreds of millions of scientific hypotheses in parallel digitally, using predictive oracles which can actually predict the outcome of an experiment.
So the scale at which we can ask questions, and the kinds of questions that we can ask, will just fundamentally change. That feedback is going to be critical. The models are going to need toâthere's going to be sort of a scaling dimension of this, which is building the data to have those accurate representations, and then a feedback dimension where the models can learn from biology, can reason digitally, can reduce that to a small number of experimental hypotheses, examine the outcome of each of those experiments, update their understanding, and build knowledge in that way. So I think that's what it's going to look like, and we kind of have to build each of those components.
What Biohub is really trying to do is bring together the experimental and technology layers that will actually allow us to have these AI models interact with biology and do experiments. We see incredible advances in areas where we can get feedback computationally, so in closed domains, but of course experimental biology is completely open-ended. The feedback principle there is going to be very different. But there's going to be something like RLVR with experiments, where we can have models that are really building knowledge and learning from that knowledge and being able to develop more and more accurate representations.
7. Biohub Builds The New Paradigm
Shawn Wang
You're the head of science at Biohub. Maybe a fun fact for those who don't know: the science section of Latent Space was basically launched after, or in response to, Mark Zuckerberg and Priscilla Chan appearing on this podcast about 6 months ago. It's actually very exciting to have you here and kind of come full circle. Mark laid out quite an ambitious vision for what Biohub wants to accomplish, and I think you just laid out a very natural successor to that. I think you had just joinedâyou were there 2 weeks.
Alexander Rives
I joined at the very end of October and launched at the beginning of November.
Shawn Wang
Yeah. One thing I'm curious about is, in your eyes, where is Biohub now? What do you want to accomplish? What are your big-picture goals for listeners who haven't watched the episode with Mark and Priscilla? Have you learned anything even in just the short time of 6 months you've been here? Has the vision evolved, and where do you see this going? How does ESMC fit into this? How does the virtual biology initiative that you recently announced fit into this? And then I think there's several other things that you're working on that we haven't even touched on.
Alexander Rives
Yeah. I'm learning things every single day. But the way I think about it, we're building a scientific institution for this new paradigm. To do that, it's an institution that's going to be powered by frontier experimental biology, frontier technology for measurement and observation, and frontier artificial intelligence.
Shawn Wang
And this is all open source, right?
Alexander Rives
It's a philanthropy. Our goal is to accelerate science. Our mission is to cure or prevent disease. To do that, our belief is that there's a fundamental gap in our understanding, and we need to accelerate science to traverse that gap.
We're really thinking about every layer of biological understanding, from the most basic level, like the atoms of a protein in a cell, all the way to systems of cells in physiology and disease. How can we create models that can capture that complexity and allow us to understand that complexity? And I think, if you think about what the cure to disease looks like, it's not a pill. It's not a medicine in the conventional sense. It's going to have to be a system that is capable of modeling and understanding the underlying physiology of disease in a way that's differentiated for every single human being, for every single different genome. And it's going to have to be able to link events all the way from the molecular scale to the manifestation of disease in physiology.
So it's an incredibly complex, incredibly hard problem. For us, we're trying to ladder up those layers of complexity, and we're trying to build the foundational tools that scientists can use to answer the fundamental questions there. We're creating atomic-level imaging. We're creating light-sheet microscopy that allows us to observe how all the cells move and develop in a developing organism. We're creating spatially and temporally resolved maps of inflammation.
We're creating cellular programming and immune-cell reprogramming to be able to actually design completely programmable therapies. We're creating these digital representations at each of these layers so that we can accelerate the science, simulate what's happening, and make biological matterâproteins, cells, and genomesâprogrammable. All of that has to come together. If you have the focus and you build the biology and the computational layers together so that they're tightly integrated, that's how we're going to make the fastest progress.
For the last 10 years, I think we've been one of the big champions of open science. We're an organization that does both: we fund and we build. In our funding, we've always supported open science, and in our building, we've always done open science. That's something that's going to continue. It's just really fundamental. We're not a drug development company. We're not trying to generate therapies. We're trying to build the technology that moves science forward.
swyx
I think Mark had this concept that if you provide the right tools, then the entire scientific community can leverage them. So obviously you believe strongly in protein language modeling as a tool. What is the next most important tool for advancing a general improvement in our ability to tackle human disease?
Yeah. I think the next level of complexity that we have to address is the complexity of the cell. This is going to be tremendously hardâbillions of proteins.
swyx
So you say it's tremendously hard. If you came and said it was going to be easy-peasyâ
Well, I think it's a worthy challenge, but it requires technology that doesn't exist today. It requires new modeling approaches and probably architectures and ideas in machine learning that probably don't yet exist. So there's deep and fundamental problems to solve. But again, I think you take it step by step. We start at the molecular layer, and we know that that is really fundamental, and we can begin to link that to observables in cellular biology.
swyx
I'm really curious because this has been the question that's been on my mind for a long time: we have virtual cell models, we have molecular-scale models, and I've seen a few papers about trying to link them. But what are you guys doing? Because it sounds like this is becoming top of mind for you.
So let's maybe make the analogy with protein biology. What I think makes our digital representations of proteins powerful and useful is that they generalize. They're able to make predictions for proteins that are entirely unlike the proteins in their training data. They're able to generalize so that you can design fundamentally new folds, new binding interfaces, and new structures.
So there's this degree of what we call generalization, or generality. In short, they can predict the outcome of an experiment that we haven't already made, that they haven't already been trained on. For digital representations to be valuable, they've got to be able to be used to answer a new question. I think that's the critical thing.
We're not there with cells. I think with the current generation of models that are being called virtual cells, they are good representations of the underlying data, but they have a very limited ability to predict what will happen when you make a novel intervention in a novel, unobserved context. To be able to answer the fundamental scientific questions about cellular biology, we need a model that can do that. Our thinking about this starts with that idea: what's it going to take to get there?
swyx
Going back to protein-protein interaction, the human interactomeâif you had that, just predicting static structuresâstatic structures are, in some sense, not enough for a lot of understanding biology.
swyx
Dynamics are probably, for most people, a much more useful tool to have. You can start with static; it can give you some insight, but itâs very rarely the full answer. So you have a model capable of predicting a lot of different proteins. We probably have many of these resolved in the PDB, and some of them we donât.
Given that dynamics and interactions are more important, how do you bridge that gap? To me, that seems like maybe one of the key steps in going from a really microscopic model of things to something closer to a virtual cell. You actually have to be able to model local interactions of local proteins, RNA, DNA, lipids, or whatever else is floating in the cell. Is that sort of a goal that you would try to bridge, or maybe Iâm misunderstanding? Is there another way you would imagine bridging these two?
I mean, one day itâll probably be possible to have a computer that can simulate the cell from first principles, but weâre very far from that, right? I think thatâs far beyond the reach of current computational technology. I mean, even simulating the physics of the folding of a single protein moleculeâwe could do it for a few fast-folding proteins, but thatâs really about it.
Yeah. So thereâs kind of this dual view of biology, this dual, complementary view of biology. One view is that kind of first-principles reduction, where all of biology is explainable in more basic termsâin basic physical, chemical, and biochemical terms. I think historically thereâs a long line of research thatâs really sought to understand biological phenomena and simulate biological phenomena in that way.
Historically, the field had believed that the solution to the protein-folding problem, or the protein-structure-prediction problem, would come from this kind of first-principles simulation. It really came out of nowhere that this could be solved using essentially pattern recognition, or this type of machine-learning approach. I think historically it has been productive to understand biology through information theory and information.
In informational terms, there are these very basic principles that link the information coded in the genome to the genes that are transcribed, to the phenotypes of the cell that will result. If we could model and understand the cell at the level of its underlying programs, that sort of gives, I think, the right abstraction.
What do I mean by the right abstraction? I mean the abstraction that is possible today, because weâre in the era of information theory at scale. Claude Shannon had this idea of the ideal predictor of the next character, and he had this really beautiful paper where he tried to compute the entropy of the English language and imagine taking an infinite context: What is the entropy of the next character?
At that time, it was unimaginable. Iâd say it took a great leap of imagination to imagine that ideal predictor, but today weâre getting closer and closer to being able to build that, and we can do that for text. What would that predictor be for biology? Thatâs kind of the idea of ESM: It would learn the underlying structure of all biological phenomena.
If you think about that from the standpoint of the cell, if we can collect enough outputs of cellular biology that we can observe to reveal the underlying programs, patterns, and structure, then we could create the information-theoretic description of the cell. I think that would be sufficient for understanding disease.
swyx
This reminds me of a lot of the work that happens in signaling pathways right now, where you have a protein in a cascade of different proteinâprotein interactions that eventually cause a phenotypic change in the cell in some way. How do you translate that into something that can be scaled into a model, or maybe something else? How do you, for example, do that?
Yeah, going back to the Bitter Lesson.
swyx
Going back, letâs just get back to the Bitter Lesson.
We need data. I think the reason these advances in protein biology have been possible is because of decadesâI mean, for protein structure, half a centuryâof work to experimentally determine the structure of proteins, and the effort across the scientific world to sequence genomes and metagenomes. Thatâs created a data set that you can really train at scale and learn these deeper principles.
swyx
But those 2 different data sets are actually, in many ways, quite different. The PDB is a bunch of very painstakingly constructed protein structures, many of which were the subject of an individual PhD thesis. Then maybe similar ones came later, which might have been 10 of them for a PhD thesis.
These people estimate itâs like $13 billion to create the PDB, some very large number. The reason people created the PDB was because each individual protein was independently useful. People didnât create it for the sake of solving protein structure; they saw that, âThis protein, we believe, is involved in this pathway. Letâs understand this protein so we can target it,â and so on.
Of course, there are some caveats here, but at a high level, a lot of this genomic data, especially for humans, viruses, or bacteria, was sequenced for a very specific reason as well, right? Itâs great that these are useful after the fact, but I wonder if now, going forward, especially since the Virtual Biology InitiativeâBiohubâs Virtual Biology Initiativeâis like half a billion dollars, I think, and Iâm sure there will be more large initiatives coming from Biohub in the future, you have the chance to be very specific and deliberate, and now collect data for the sake of solving a problem with ML rather than depending on a data set that was curated and created for some other purpose.
Given that new opportunity, how do you do things differently? How do you think about data collection to enable science broadly when you have the option of doing basically anything from first principles?
8. Building The Biology Data Flywheel
A little bit of context: We announced the Virtual Biology Initiative a few weeks ago. We basically said, âWeâre going to invest $400 million internally in data creation and development of technology to scale data generation, to be able to increase the number of modalities that we can measure simultaneously.â We also announced that weâre going to commit $100 million to catalyzing efforts outside of Biohub to generate data.
We think thatâs a fraction of whatâs actually needed to do this, right? But the hope is that by making this initial commitment, giving starting funds to some of the groups that are really thinking about this and working to build different core areas of the data thatâs going to be needed, thatâs going to be a catalyst. Thatâs going to galvanize other groups to come in and contribute to this. Thatâs what we really hope to see.
The idea is that this is a broad-based effort, so itâs not just us. I can say what my perspective is on what data needs to be generated here or what can be generated, but we also want to approach this really collaboratively with the scientific community. Part of this is also hearing from scientists what they want.
From my view, there are a few key principles here. The first is speed. It took decades to build the data for proteins, and we canât wait decades. We need to figure out how to do this in a couple of years. You look at the rate at which general AI is developing, and the limitation in biology is that weâre going to be fundamentally limited by experimental science and data. We really need to work to address that gap as quickly as possible.
One key thing is looking at what technologies we can scale up today to begin to give this picture of the information architecture of the cell. Thereâs speed, and then thereâs also the idea of generalization. Going back to what I was saying before, we want models that can serve as oracles for biology. They can predict an experiment that you havenât done.
How are we going to be able to do that? Weâre going to need to look at a multitude of different interventions in a multitude of different contexts. Itâs similar to the principle of training a language model on the internet or training a protein language model across all of evolutionary diversity. What does that look like for cellular biology? We have to scale interventional biology.
That looks like things such as perturbation biology and Perturb-seq measurements, where we can look at combined transcription, imaging, and other layers of the cellular information hierarchy. A number of groupsâour teams, as well as groups across the scientific worldâare working on problems like this and are ready to scale.
The second is spatial biology. I think thatâs going to be really important, and itâs going to help us understand the cell in context. Understanding the cell in isolation is really not what we need; itâs not the goal. The cell is part of an incredibly complex system in the body, and to be able to understand disease, we have to understand how cells interact, the systems that they form, and the circuits that they form. We need to see that.
Spatial biology is undergoing rapid progress and is an area thatâs really ready to scale up. Thatâs kind of what can scale now. Biohub has, over the last 10 years, made pioneering funding commitments in those areas.
Weâve funded efforts like the Human Cell Atlas, and weâve built Tabula Sapiens, which is a large cell atlas. Weâve built CELLxGENE, which is kind of a database of single-cell transcriptomics. Weâre really looking to build on that. I donât know how many cells there are in the largest efforts; weâre probably around a billion cells or something like that today.
So we've got to go multiple orders of magnitude from that. That involves scaling the technologies that we have now, but it also involves new, next-generation technology. We're also funding and supporting efforts in that area. There, we really want to look more at cross-modality. Can you simultaneously see the phenotype, observe the transcriptional layer, understand what's happening proteomically, and link that to the genome? We'd like to see the epigenetic state as well. We'd like to be able to see all of that.
We're really pushing technology to be developed faster so that it can reveal more of those connections and more of that biology, and do that in a more scalable way.
Shawn Wang
It's interesting because when I hear most of those ideas, they're often the things that people already think about in terms of scaling biology. What is the next technology that's going to enable data collection? Going back to the theme of the Bitter Lesson for biology, you don't just have scaling laws on compute and parameters, but now the scaling laws are probably in data collection in some meaningful sense. Where are the next big opportunities there? You're talking about developing new technology as part of this initiative.
Alexander Rives
Yeah. I think it's basically the things that I'm saying: scaling what we have now, being able to expand the number of interventions that we can look at, expand the number of parameters that we can measureâreally, more and more multidimensional measurementâand drive down the cost and all of that. Better gene sequencing, better ways of encapsulating cells, and being able to measure what's happening not just in the transcriptome but in other layers simultaneously.
Alessio Fanelli
There's an interesting Pareto frontier there: if you have a fixed budget, how much time do you spend on improving your assay versus how much do you spend on actually scaling it? Where do you weigh in there?
Alexander Rives
We have to do both of those things, right? I think with current technology, we can definitely get data 10Ă to 100Ă where it is today with relatively reasonable investments. But then to get another 10Ă or more beyond that, that's going to require a lot more technology development.
The other really big principle is going to be feedback, and I think that's going to be really critical. You can see that as a layer of technology development that's going to need to occur. There's a lot of great things happening right nowâautomation, flexible roboticsâthat's going to accelerate where that can go, and experimental design as well.
Alessio Fanelli
We typically ask our guests what bottleneck they would remove that would sort of unlock things, but we just spent a long time talking about that.
Shawn Wang
Yeah, I want to ask that question, but I'm going to give it a spin: maybe something a little bit outside of your domain, like language modeling or supply chainâsomething that is a bottleneck that is maybe nonobvious and not directly something that you are working on, but that maybe has an impact on the work of biology or Biohub in particular.
Alexander Rives
I mean, it's a hard question to answer because there are just so many bottlenecks. The one that I always think about is compute, but I think that's a pretty obvious one. It's the bottleneck for all of AI in many ways right now.
Especially because we're training these large-scale models, we're always focused on compute. I think we're limited both by the data and compute. We're in a position where we have incredible compute resources for a team working in biology. But, like all teams working in AI right now, really the limit is just how much compute power youâ
Shawn Wang
So if you could 100Ă your compute, do you think that ESM-C would be way better?
Alexander Rives
It would definitely be way better. We also need to scale data, so both of those things would have to happen in tandem.
Shawn Wang
Have you basically exhausted what's available right now forâ
Alexander Rives
I don't think so. No, I don't think so.
Shawn Wang
Okay. The large datasets out there, orâ
Alexander Rives
Well, I mean, more parameters, you know. We trained ESM-C up to 6 billion parameters.
Shawn Wang
Oh, but I'm saying in terms of data available: have you exhausted most of what's publicly available in terms ofâ
Alexander Rives
No, not yet. And the atlas that we just built actually has more sequences and structures than ESM-C was trained on.
Shawn Wang
So you definitely have a little room to go. Is that an order-of-magnitude jump, or twice as much? How does that work?
Alexander Rives
Yeah, I mean, ESM-C is trained on, say, on the order of 1 billion sequences. There's definitely probably on the order of 100 billion sequences.
Shawn Wang
That's large, but a lot of them are largely redundant. 100 billion.
Alexander Rives
Yeah.
Shawn Wang
Okay. To get that billion, you whittle down from 6 billionâ6.8 billion, right? So, of those 100 billion, if you were to similarly cluster and find unique ones, where do you think it would land?
Alexander Rives
The sequences aren't actually redundant, right? It really depends on what you mean by redundancy, because I think there's a tremendous amount that you can learn from small genetic variations. These are really revealing of the very basic determinants of protein structure and function at a very fine level.
As we think about protein space, having a vast diversity of sequences across a wide range of protein families is really critical for the emergence of this kind of structure-prediction capability, because I think large diversity is what trains the model to understandâto develop a representation of structure. But I actually think that to develop a representation of function, it's these very small variations that are important.
I do think that there's probably a lot more. The models haven't yet been trained at that level of really deep understanding of these very small but critical patterns in sequence. A single mutation is enough to destroy the function of a protein.
Shawn Wang
So you could conceivably actually take all 6.8 billion of those and retrainâeverything's the same, butâ
Alexander Rives
Yeah, you could train on more than that. Even that is kind of clustered down, soâ
Shawn Wang
Yeah, maybe the question is: how far until you hit the law of diminishing returns here? It sounds like you have plans for an ESM-4 or an ESM-3, or whatever you want to develop. I'm just wondering: at some point, is this actually something that you could exhaust? People talk about exhausting the pretraining data inâ
Alexander Rives
Yeah, at some point. Yeah, at some point.
Shawn Wang
But it's not actually something you could conceivably imagine doing in the next few years. Even if you don't exhaust it, you could hit a lot of diminishing returns for the applications that you're trying to predict here, where maybe your resources are better spent somewhere else.
Alexander Rives
I mean, it's basically an empirical question, right? It's truly an empirical question. And so we just don't know. With ESM-2, we weren't sure because there were some diminishing returns. With ESM-C, now there aren't, right? So you can kind of look at that, extrapolate from the scaling law there, and there is enough data to train that next model.
Alessio Fanelli
And the other question that we usually ask is: any call to action? What do you want people to go take action on? If the listeners want to get involved, get hired, or build things, what would you ask people to do?
Alexander Rives
Well, we just announcedâor, I should say, at the time that this podcast comes out, we will have announcedâESMC and this world model for protein biology. It's going to be open source. It's MIT-licensed, and we want people to use it. We want this to be a tool that can unlock science.
We're excited to collaborate. We have a team that works on that, and we want to hear from people and understand what we can build that can help to accelerate their science.
Shawn Wang
Yeah, we might have a demo/paper club of some sort on this channel, so stay tuned.
Yeah, stay tuned for that. We'll invite you and your team, whoever can make it. We'll feature this paper once it's in final preprint form and spend some time on it for an hour on the Latent Space Paper Club.
Alessio Fanelli
Yeah. Thanks for chatting with us.
Awesome. Yeah, great to meet you guys.