---
title: "Paired residue prediction dependencies in ESM2"
authors:
  - "Brae M. Bigge"
  - "Evan Kiefl"
doi: "10.57844/arcadia-f52b-1451"
license: "https://creativecommons.org/licenses/by/4.0/"
date: "2025-03-09"
version: 2
canonical_url: "https://thestacks.org/publications/result-paired-token-masking"
---

# Paired residue prediction dependencies in ESM2

_During a quick analysis of the ESM2 model for masked token prediction, we noticed that amino acid probability distributions of residues affect each other in a pattern that mirrors a protein’s 3D contact map. But less so for the larger model sizes. Our question to you is, why?_

## Abstract

This notebook details a quick analysis we performed on the ESM2 (Evolutionary Scale Modeling) models [](https://doi.org/10.1126/science.ade2574) for masked token prediction. We stumbled upon a counterintuitive result related to the effect that masking one residue has on the distribution of another.

Before jumping into our results, let’s first establish what masked token prediction is, how it works, and our consequent motivation for this analysis.

# View the notebook {#h.orpsds5orkyo}
::::::div{.info-box}
The **full notebook pub** is available [here](https://arcadia-science.github.io/2024-paired-token-masking/).

The **source code** to generate it is available in [this GitHub repo](https://github.com/Arcadia-Science/2024-paired-token-masking/tree/v1.1) (DOI\: [10.5281/zenodo.15002836](https://doi.org/10.5281/zenodo.15002836)).
::::::

In the future, we hope to host [notebook pubs](https://dx.doi.org/10.57844/arcadia-ca21-23bb) directly on our publishing platform. Until that’s possible, we’ll create stubs like this with key metadata like the DOI, author roles, citation information, and an external link to the pub itself.