{"publication":{"abstract":"This notebook details a quick analysis we performed on the ESM2 (Evolutionary Scale Modeling) models [](https://doi.org/10.1126/science.ade2574) for masked token prediction. We stumbled upon a counterintuitive result related to the effect that masking one residue has on the distribution of another.\n\nBefore jumping into our results, let’s first establish what masked token prediction is, how it works, and our consequent motivation for this analysis.","body":"# View the notebook {#h.orpsds5orkyo}\n::::::div{.info-box}\nThe **full notebook pub** is available [here](https://arcadia-science.github.io/2024-paired-token-masking/).\n\nThe **source code** to generate it is available in [this GitHub repo](https://github.com/Arcadia-Science/2024-paired-token-masking/tree/v1.1) (DOI\\: [10.5281/zenodo.15002836](https://doi.org/10.5281/zenodo.15002836)).\n::::::\n\nIn the future, we hope to host [notebook pubs](https://dx.doi.org/10.57844/arcadia-ca21-23bb) directly on our publishing platform. Until that’s possible, we’ll create stubs like this with key metadata like the DOI, author roles, citation information, and an external link to the pub itself.","contributors":[{"user_id":4218,"role":"Supervision","first_name":"Brae M.","last_name":"Bigge","avatar":"https://thestacks-01.s3.amazonaws.com/users/avatars/Brae-Arcadia-headshot.png","in_byline":1,"priority":1,"affiliations":[]},{"user_id":4259,"role":"Critical Feedback","first_name":"Daniel","last_name":"Burns","avatar":"https://thestacks-01.s3.amazonaws.com/users/avatars/fly-placeholder.png","in_byline":0,"priority":2,"affiliations":[]},{"user_id":4259,"role":"Formal Analysis","first_name":"Daniel","last_name":"Burns","avatar":"https://thestacks-01.s3.amazonaws.com/users/avatars/fly-placeholder.png","in_byline":0,"priority":2,"affiliations":[]},{"user_id":4214,"role":"Critical Feedback","first_name":"Keith","last_name":"Cheveralls","avatar":"https://thestacks-01.s3.amazonaws.com/users/avatars/Keith-Arcadia-headshot.png","in_byline":0,"priority":3,"affiliations":[]},{"user_id":4214,"role":"Validation","first_name":"Keith","last_name":"Cheveralls","avatar":"https://thestacks-01.s3.amazonaws.com/users/avatars/Keith-Arcadia-headshot.png","in_byline":0,"priority":3,"affiliations":[]},{"user_id":9,"role":"Editing","first_name":"Megan L.","last_name":"Hochstrasser","avatar":"avatars/users/user_9_1765929965.jpg","in_byline":0,"priority":4,"affiliations":[]},{"user_id":4229,"role":"Conceptualization","first_name":"Evan","last_name":"Kiefl","avatar":"https://thestacks-01.s3.amazonaws.com/users/avatars/Evan-Arcadia-headshot.png","in_byline":1,"priority":5,"affiliations":[]},{"user_id":4229,"role":"Formal Analysis","first_name":"Evan","last_name":"Kiefl","avatar":"https://thestacks-01.s3.amazonaws.com/users/avatars/Evan-Arcadia-headshot.png","in_byline":1,"priority":5,"affiliations":[]},{"user_id":4229,"role":"Investigation","first_name":"Evan","last_name":"Kiefl","avatar":"https://thestacks-01.s3.amazonaws.com/users/avatars/Evan-Arcadia-headshot.png","in_byline":1,"priority":5,"affiliations":[]},{"user_id":4229,"role":"Methodology","first_name":"Evan","last_name":"Kiefl","avatar":"https://thestacks-01.s3.amazonaws.com/users/avatars/Evan-Arcadia-headshot.png","in_byline":1,"priority":5,"affiliations":[]},{"user_id":4229,"role":"Software","first_name":"Evan","last_name":"Kiefl","avatar":"https://thestacks-01.s3.amazonaws.com/users/avatars/Evan-Arcadia-headshot.png","in_byline":1,"priority":5,"affiliations":[]},{"user_id":4229,"role":"Visualization","first_name":"Evan","last_name":"Kiefl","avatar":"https://thestacks-01.s3.amazonaws.com/users/avatars/Evan-Arcadia-headshot.png","in_byline":0,"priority":5,"affiliations":[]},{"user_id":4229,"role":"Writing","first_name":"Evan","last_name":"Kiefl","avatar":"https://thestacks-01.s3.amazonaws.com/users/avatars/Evan-Arcadia-headshot.png","in_byline":1,"priority":5,"affiliations":[]}],"created_at":"2025-03-09T18:30:00.000Z","doi":"10.57844/arcadia-f52b-1451","version_doi":null,"feedback_form_embed_src":null,"id":86,"license":"CC BY","linked_assets":[{"type":"Code","url":"https://github.com/Arcadia-Science/2024-paired-token-masking","name":"2024-paired-token-masking"}],"orgs":[{"org_id":2,"priority":null,"slug":"arcadia-science","name":"Arcadia Science","avatar":"avatars/orgs/org_2_1766013565.jpg"}],"pdf_url":"https://thestacks-01.s3.us-west-2.amazonaws.com/publications/result-paired-token-masking/result-paired-token-masking_v2.pdf","references":[{"url":"https://doi.org/10.1126/science.ade2574","text":"Lin Z, Akin H, Rao R, Hie B, Zhu Z, Lu W, Smetanin N, Verkuil R, Kabeli O, Shmueli Y, dos Santos Costa A, Fazel-Zarandi M, Sercu T, Candido S, Rives A. (2023). Evolutionary-scale prediction of atomic-level protein structure with a language model."}],"slug":"result-paired-token-masking","social_posts_count":null,"social_posts_embed_src":"https://publishing-tools.arcadiascience.com/twitter/061385962078399616","state":"PUBLISHED","subtitle":"During a quick analysis of the ESM2 model for masked token prediction, we noticed that amino acid probability distributions of residues affect each other in a pattern that mirrors a protein’s 3D contact map. But less so for the larger model sizes. Our question to you is, why?","tags":["Result","not actively updating"],"title":"Paired residue prediction dependencies in ESM2","version_desc":"Added another H1 heading","version_number":2,"versions":[{"id":330,"version_number":2,"version_desc":"Added another H1 heading","doi":null,"created_at":"2025-12-01T18:30:00.000Z"},{"id":329,"version_number":1,"version_desc":null,"doi":null,"created_at":"2025-03-09T18:30:00.000Z"}]}}