Charting the Latent Space of Life: What Protein and DNA Language Models Learn

Nobody showed ESM-2 a protein structure. Meta's protein language model was trained on one game: hide some of the amino acids in a sequence and guess them back. That is all. Yet when the authors scaled the model to 15 billion parameters, they reported that "an atomic-resolution picture of protein structure emerges in the learned representations." Folds, contacts, and the geometry of a working molecule had been compressed into the model's latent space, without anyone asking for them.

That sentence is the reason this site has a biology section. Space Services is about latent space: the high-dimensional interior where a model stores what it has learned. Most of our articles chart that space for language. This one charts it for life. The same recipe that turned the internet into a chatbot, predict the missing token at enormous scale, has been run on the 20-letter alphabet of proteins and the 4-letter alphabet of DNA. The models that came out the other side know things about biology that were never in their training objective, and over the last two years researchers have started reading that knowledge out, one feature at a time.

This article walks through how that works. First, why guessing letters forces a model to learn structure. Then the tool that cracked the latent space open, the sparse autoencoder, which you can train yourself in the sandbox below. Then what the features turned out to be in protein models and in Evo 2, a genome model with a context window of a million DNA letters. We end with the hardest test of all, using the model to write a protein that nature never made, and with what all of this does and does not prove.

Life is written in very small alphabets

A protein is a chain of amino acids, drawn from an alphabet of 20. The chain folds into a three-dimensional shape, and the shape does the work: it binds, cuts, pumps, or signals. DNA is simpler still: four nucleotides, A, C, G, and T, which encode the proteins along with the switches that decide when each one is made.

These alphabets come with a remarkable training set. Every sequence in a modern protein database is a survivor. Evolution has run a filtering experiment for billions of years, and only sequences that fold and function made it into living organisms. A model that learns to predict those sequences is learning the statistics of things that work.

That has a deep consequence. Suppose two amino acids sit far apart in the chain but touch once it folds. If one mutates, the other often has to change to keep the contact stable. Across thousands of related proteins, the two positions vary together. A model that is good at filling in a hidden residue will learn to look at its partner, because the partner is evidence. Contact, a three-dimensional fact, leaves a statistical fingerprint in one-dimensional text, and a predictor that is pushed hard enough will pick it up.

Hiding letters teaches structure: ESM-2

ESM-2 is a family of transformer models trained with exactly that masked-token objective on UniRef50, a clustered database of known protein sequences. Meta released a ladder of sizes, from 8 million parameters (6 layers) up to 15 billion (48 layers), which made it possible to watch what changes as the model grows.

What changes is structure. The 2023 Science paper behind ESM-2 showed that as the models scale, their internal representations hold more and more information about how the protein folds. The authors attached a folding head to the language model and called it ESMFold. It predicts a structure from a single sequence, with no search through evolutionary relatives, which made it fast enough to fold proteins by the hundreds of millions: the team used it to build the ESM Metagenomic Atlas, which launched with predicted structures for more than 617 million metagenomic proteins, over 225 million of them predicted with high confidence.

So the representations contain structure. The harder question is how to see it. A model's hidden state is a vector of a few hundred to a few thousand numbers per residue, and those numbers do not come labeled.

Why the latent space is hard to read: superposition

The natural first move is to look at individual neurons and ask what each one responds to. In language models this mostly fails, and it fails in protein models too. A typical neuron fires for a grab bag of unrelated things. Neurons are polysemantic.

The leading explanation is superposition. A model needs to represent far more concepts than it has dimensions, so it stores each concept as a direction in activation space rather than as a single neuron, and lets those directions overlap. Because most concepts are rare, any one residue only expresses a handful of them at once, and the overlap rarely causes confusion. This is efficient for the model and miserable for us: the concepts exist, but they are smeared across every neuron.

A sparse autoencoder (SAE) is the tool built to undo that smear. It takes a model's activation vector, expands it into a much larger set of candidate features, and is forced to keep almost all of them at zero for any given input. Then it has to rebuild the original activation from the few that stayed on. Sparsity is the trick. If the true concepts are rare and the SAE is only allowed a few active features, the cheapest way to rebuild every activation is to assign one feature to each concept.

You can watch that happen. The sandbox below builds a toy latent space: 12 hidden concepts packed into only 6 dimensions, with each residue expressing one or two at a time. It then trains a small SAE from scratch in plain NumPy and checks whether the learned features line up with the hidden concepts. The concept names are labels for illustration; the directions are random.

Two things to try. Set lam to 0.05 and rerun: with a weak sparsity penalty the SAE spreads each concept across several features and the count drops. Then set the concept probability from 0.12 to 0.4 so that residues express many concepts at once: superposition stops being cheap, and no method recovers the directions cleanly. Real models live in the sparse regime, which is why the technique works on them at all.

What the features turned out to be: proteins

In late 2024, Elana Simon and James Zou at Stanford ran this experiment on a real protein model. Their method, InterPLM (published in Nature Methods in 2025), trained SAEs on every layer of the smallest ESM-2, the 8 million parameter model, expanding its 320-dimensional hidden state into 10,240 features. Then they checked each feature against 433 biological concepts annotated in Swiss-Prot, the curated protein database.

The SAE found up to 2,548 features per layer that strongly matched up to 143 known concepts. Looking at neurons directly found up to 46 neurons covering 15 concepts. The control is the important part: the same SAE trained on a model with random weights found no biological features at all. The concepts are in what the model learned, not in the method.

Some features are startlingly specific. One detects the beta barrel of TonB-dependent receptors, a class of bacterial membrane transporters, with an F1 score of 0.998. Another fires on a single conserved position of the Nudix box, a short motif found in a family of hydrolase enzymes. A third fires on two conserved positions in tyrosine recombinases and was perfectly precise across 45 of them.

The Nudix feature produced the best story in the paper. It fired strongly on a protein, UniProt entry B2GFH1, whose Swiss-Prot record had no Nudix annotation at all. That looked like a false positive. But a second database, InterPro, does annotate a Nudix motif in that gene. The feature was right and the label was missing. Features learned without supervision can point at gaps in human annotation.

Other groups pushed the method to larger models and found more. A team led by Mohammed AlQuraishi at Columbia trained SAEs on ESM-2's 650 million parameter model (InterProt, ICML 2025). Their features include one that fires on bipartite nuclear localization signals, the short address labels that send a protein into the cell nucleus, following the pattern R/K, then 10 to 12 residues of anything, then KRXK. Others mark signal peptide cleavage sites, mitochondrial targeting sequences, and transmembrane segments, so the model's latent space contains a working map of where in the cell a protein is going. A group at MIT showed in PNAS in 2025 that SAE features from ESM-2 line up with Gene Ontology terms at every level of the hierarchy, from broad processes down to specific enzyme activities. And the Reticular team trained SAEs on ESM-2 3B, the backbone of ESMFold, then used a feature to push ESMFold's predicted structure toward higher solvent accessibility without changing the input sequence.

The largest public example arrived in 2026 with ESM C 6B, released by Biohub along with SAEs that split its internal representations into more than 16,000 features. One of them activates on the nucleophilic elbow, a catalytic motif, in 75 of 99 relevant enzymes spanning 25 distinct protein folds. That is a feature recognizing one chemical job across proteins that look nothing alike in shape.

Genomes: a million letters of context

Proteins are the output. DNA is the source code, and it is much harder to model. A gene is scattered across a genome with long noncoding stretches between its pieces, regulatory switches can sit far from what they control, and a bacterial genome alone runs to millions of letters.

Evo, from the Arc Institute and collaborators (Science, 2024), was a first serious attempt: a 7 billion parameter model reading one nucleotide per token with a context of 131 kilobases, trained on 2.7 million bacterial, archaeal, and phage genomes. Evo 2 (Nature, March 2026) went much further. The larger version has 40 billion parameters, was trained on 9.3 trillion tokens from more than 8.8 trillion nucleotides of bacteria, archaea, eukaryotes, and bacteriophage, and reads a context window of one million nucleotides at single-letter resolution. For biosafety, the training data deliberately left out viruses that infect eukaryotic hosts, and the model is measurably worse at predicting them.

Like ESM-2, Evo 2 was only trained to predict the next letter, and like ESM-2 it learned far more. Simple classifiers on top of its internal representations find exons, the protein-coding pieces of genes, with AUROC between 0.91 and 0.99 on eight held-out species, beating dedicated gene finders such as AUGUSTUS. With no fine-tuning at all, it can score mutations in the breast cancer gene BRCA1 by how surprised it is to see them, and it outperformed every other model tested on noncoding variants there. Arc reports over 90 percent accuracy in separating benign BRCA1 variants from potentially pathogenic ones.

The interpretability work on Evo 2 was done with Goodfire and is part of the Nature paper. The team trained a sparse autoencoder on layer 26, using activations from a billion tokens of complete eukaryotic and prokaryotic genomes, and searched for features that track known biology. What they found reads like a genetics textbook rediscovered by statistics:

• Gene anatomy. Separate features for coding sequence, introns, the first bases of an exon after an intron, and the last base of an exon before one. The exon-intron features carry over to a stretch of woolly mammoth genome.
• Protein structure from DNA. Features that fire on the DNA encoding alpha helices and beta sheets in genes such as the one for the bacterial protein EF-Tu. The model only ever saw nucleotides, so it picked up something about the folded protein from letters two steps removed: DNA to protein sequence, protein sequence to shape.
• Broken genes. A feature that activates preferentially on frameshift and premature stop mutations in human coding sequences.
• Viral history. A prophage feature that fires on viral DNA integrated into the E. coli genome. It also fires on CRISPR spacers, the snippets of viral DNA bacteria keep as immune memory, and even on scrambled synthetic spacers, which suggests the model learned the association between spacers and phage rather than memorizing particular sequences.

Transcription factor binding motifs, the short sequences where regulatory proteins grab DNA, are a good benchmark because biologists have spent decades cataloging them. Evo 2's unsupervised features matched 70 percent of the promoter-enriched motifs in the HOCOMOCO v12 database. HOMER, a specialized motif discovery tool, recalled 35 percent of the same motifs.

The most literal version of a latent space map came from a follow-up at Goodfire. Looking at layer 24 of Evo 2 across more than 2,400 bacterial species, they found a 10-dimensional flat subspace in which distances between species correlate with their evolutionary distance at 0.98 on the species used to find it, and about 0.9 on a held-out clade. The tree of life, it turns out, is a shape inside the model.

From reading to writing: a protein nature never made

Interpretation asks what a model knows. Generation tests it. In January 2025, EvolutionaryScale published ESM3 in Science, a model that reasons jointly over a protein's sequence, structure, and function. The largest version has 98 billion parameters and was trained on 2.78 billion proteins.

The team asked the 7 billion parameter version to design a green fluorescent protein, prompting it with six key residues from the natural GFP chromophore site. The result, esmGFP, is a 229-residue protein with 58 percent sequence identity to its closest known fluorescent protein, the engineered tagRFP, 96 mutations away. Its closest natural relative differs at 107 positions, and it shares only 36 percent identity with the classic jellyfish GFP. It glows. The first round of designs produced one that was 50 times dimmer than natural GFPs. A second round built on that design and produced esmGFP, whose brightness falls within the range of natural GFPs once it matures for two days.

The paper's title is "Simulating 500 million years of evolution with a language model," and that number deserves care. The model did not simulate evolution. The authors took a time-calibrated family tree of natural fluorescent proteins, fit a simple estimate of how sequence identity falls with evolutionary time, and asked how much time would separate esmGFP from its nearest relative if it had evolved naturally. Their estimate is about 500 million years. It is a way of saying how far into unexplored sequence space the model reached, measured in nature's units.

What this does and does not prove

It is easy to oversell results like these, so here is what they do not show.

A matching feature is not a mechanism. A feature that tracks a Swiss-Prot annotation shows the model represents that concept. It does not show the model uses it the way a biologist would reason, or that the feature causes anything downstream. Steering experiments, like clamping a feature and watching the prediction change, are stronger evidence, and they are still rare.

The dictionary is a choice. How many features an SAE has, which layer it reads, and how sparse it is forced to be all change what comes out. Concept counts from one paper are not comparable to another, and a concept nobody annotated can only be found by luck, like the Nudix case.

Survivorship cuts both ways. These models learn from sequences that exist. That is why they learn what works, and also why they can be confident about variants nature rarely tried. Zero-shot variant scores are research tools, not clinical tests.

Scale is not the only axis. ESM C's 600 million parameter model rivals ESM-2 at 3 billion on its makers' benchmarks, and the InterPLM authors found that ESM-2 at 650 million parameters holds substantially more readable concepts than at 8 million. Better data and training can matter as much as size.

Takeaways

Protein and genome language models are trained on nothing but letters, yet their latent spaces hold structure, function, cellular address labels, gene anatomy, regulatory grammar, and even the shape of the tree of life. Masked and next-token prediction force it: the statistics of surviving sequences carry the fingerprints of physics and evolution, and a large enough predictor absorbs them.

Sparse autoencoders turned that from a claim into a catalog. By assuming concepts are rare and stored as overlapping directions, they split dense, unreadable activations into features that line up with biology, from a single conserved residue in the Nudix box to the exon boundaries of a woolly mammoth gene. Sometimes they find what the databases missed.

The same latent space that can be read can be written. esmGFP is a glowing protein far from anything in nature, designed by a model that learned what a fluorescent protein needs. Charting latent space was never only about language. The space where a model compresses the world is the same kind of object whether the world is text, images, or the four letters of DNA, and the instruments built to read one are already working on the others.

References

1. Lin, Z., Akin, H., Rao, R., Hie, B., Zhu, Z., Lu, W., et al., and Rives, A. (2023). "Evolutionary-scale prediction of atomic-level protein structure with a language model." Science 379(6637):1123-1130. DOI

2. Meta FAIR. "ESM: Evolutionary Scale Modeling" (ESM-2 checkpoints and code). GitHub

3. Hayes, T., Rao, R., Akin, H., Sofroniew, N. J., Oktay, D., Lin, Z., et al., and Rives, A. (2025). "Simulating 500 million years of evolution with a language model." Science 387(6736):850-858. DOI and preprint

4. EvolutionaryScale (2024). "ESM Cambrian: Revealing the mysteries of proteins with unsupervised learning." Blog

5. Simon, E. and Zou, J. (2025). "InterPLM: discovering interpretable features in protein language models via sparse autoencoders." Nature Methods 22(10):2107-2117. DOI and arXiv

6. Adams, E., Bai, L., Lee, M., Yu, Y., and AlQuraishi, M. (2025). "From Mechanistic Interpretability to Mechanistic Biology: Training, Evaluating, and Interpreting Sparse Autoencoders on Protein Language Models." ICML 2025, PMLR 267:460-476. Proceedings

7. Gujral, O., Bafna, M., Alm, E., and Berger, B. (2025). "Sparse autoencoders uncover biologically interpretable features in protein language model representations." PNAS 122(34):e2506316122. DOI

8. Parsan, N., Yang, D. J., and Yang, J. J. (2025). "Towards Interpretable Protein Structure Prediction with Sparse Autoencoders." arXiv:2503.08764. arXiv

9. Biohub (2026). "ESM C 6B" model card, with sparse autoencoders. Hugging Face and ESM Atlas notes

10. Nguyen, E., Poli, M., Durrant, M. G., Kang, B., Katrekar, D., Li, D. B., et al., and Hie, B. L. (2024). "Sequence modeling and design from molecular to genome scale with Evo." Science 386(6723):eado9336. Full text

11. Brixi, G., Durrant, M. G., Ku, J., Naghipourfar, M., Poli, M., Sun, G., et al., and Hie, B. L. (2026). "Genome modelling and design across all domains of life with Evo 2." Nature 652(8112):1349-1361. Full text and preprint

12. Arc Institute (2025). "AI can now model and design the genetic code for all domains of life with Evo 2." Announcement and interactive feature visualizer

13. Gorton, L., Wang, N., Nguyen, N., Deng, M., Ho, E., Balsam, D., and McGrath, T. (2025). "Interpreting Evo 2: Arc Institute's Next-Generation Genomic Foundation Model." Goodfire Research. Article

14. Pearce, M., Simon, E., Byun, M., and Balsam, D. (2025). "Finding the Tree of Life in Evo 2." Goodfire Research. Article