{"publication":{"abstract":"Liu et al. [](https://doi.org/10.48550/arXiv.2602.06923) investigated the world models learned by transformers trained to predict planetary motion. Using linear probes, they found evidence that their transformers learned Keplerian or Newtonian world models depending on inductive biases such as context length. However, while linear probes can reveal what's encoded in a model, they don't necessarily reveal how a model computed its prediction. Here, we open the black box by applying equivalent linear mapping (ELM) to rewrite the trained models as an exact per-input linear map. This lets us directly understand how each transformer arrives at its prediction, revealing the underlying learned world model.","body":"# View the notebook\n\n::::::div{.info-box}\nThe **full pub** is available [here](https://arcadia-science.github.io/world-model-orbits).\n\nThe **source code** to generate it is available in [this GitHub repo](https://github.com/Arcadia-Science/world-model-orbits) (DOI: [10.5281/zenodo.23697208](https://doi.org/10.5281/zenodo.23697208)).\n::::::\n\nIn the future, we hope to host [notebook pubs](https://dx.doi.org/10.57844/arcadia-ca21-23bb) directly on our publishing platform. Until that’s possible, we’ll create stubs like this with key metadata like the DOI, author roles, citation information, and an external link to the pub itself.\n","contributors":[{"user_id":4222,"role":"Supervision","first_name":"Ryan","last_name":"York","avatar":"https://thestacks-01.s3.amazonaws.com/users/avatars/Ryan-Arcadia-headshot.png","in_byline":1,"priority":4,"affiliations":[{"org_id":2,"name":"Arcadia Science","slug":"arcadia-science","ror_id":"052zd4v68","avatar":"avatars/orgs/org_2_1766013565.jpg"}]},{"user_id":4225,"role":"Critical feedback","first_name":"Erin","last_name":"McGeever","avatar":"https://thestacks-01.s3.amazonaws.com/users/avatars/Erin-Arcadia-headshot.png","in_byline":0,"priority":2,"affiliations":[{"org_id":2,"name":"Arcadia Science","slug":"arcadia-science","ror_id":"052zd4v68","avatar":"avatars/orgs/org_2_1766013565.jpg"}]},{"user_id":4225,"role":"Validation","first_name":"Erin","last_name":"McGeever","avatar":"https://thestacks-01.s3.amazonaws.com/users/avatars/Erin-Arcadia-headshot.png","in_byline":0,"priority":2,"affiliations":[{"org_id":2,"name":"Arcadia Science","slug":"arcadia-science","ror_id":"052zd4v68","avatar":"avatars/orgs/org_2_1766013565.jpg"}]},{"user_id":4228,"role":"Validation","first_name":"James R.","last_name":"Golden","avatar":"https://thestacks-01.s3.amazonaws.com/users/avatars/James-Arcadia-headshot.png","in_byline":0,"priority":1,"affiliations":[{"org_id":2,"name":"Arcadia Science","slug":"arcadia-science","ror_id":"052zd4v68","avatar":"avatars/orgs/org_2_1766013565.jpg"}]},{"user_id":4230,"role":"Conceptualization","first_name":"George","last_name":"Sandler","avatar":"https://thestacks-01.s3.amazonaws.com/users/avatars/George-Arcadia-headshot.png","in_byline":1,"priority":3,"affiliations":[{"org_id":2,"name":"Arcadia Science","slug":"arcadia-science","ror_id":"052zd4v68","avatar":"avatars/orgs/org_2_1766013565.jpg"}]},{"user_id":4230,"role":"Experimental design","first_name":"George","last_name":"Sandler","avatar":"https://thestacks-01.s3.amazonaws.com/users/avatars/George-Arcadia-headshot.png","in_byline":1,"priority":3,"affiliations":[{"org_id":2,"name":"Arcadia Science","slug":"arcadia-science","ror_id":"052zd4v68","avatar":"avatars/orgs/org_2_1766013565.jpg"}]},{"user_id":4230,"role":"Formal analysis","first_name":"George","last_name":"Sandler","avatar":"https://thestacks-01.s3.amazonaws.com/users/avatars/George-Arcadia-headshot.png","in_byline":1,"priority":3,"affiliations":[{"org_id":2,"name":"Arcadia Science","slug":"arcadia-science","ror_id":"052zd4v68","avatar":"avatars/orgs/org_2_1766013565.jpg"}]},{"user_id":4230,"role":"Software","first_name":"George","last_name":"Sandler","avatar":"https://thestacks-01.s3.amazonaws.com/users/avatars/George-Arcadia-headshot.png","in_byline":1,"priority":3,"affiliations":[{"org_id":2,"name":"Arcadia Science","slug":"arcadia-science","ror_id":"052zd4v68","avatar":"avatars/orgs/org_2_1766013565.jpg"}]},{"user_id":4230,"role":"Visualization","first_name":"George","last_name":"Sandler","avatar":"https://thestacks-01.s3.amazonaws.com/users/avatars/George-Arcadia-headshot.png","in_byline":1,"priority":3,"affiliations":[{"org_id":2,"name":"Arcadia Science","slug":"arcadia-science","ror_id":"052zd4v68","avatar":"avatars/orgs/org_2_1766013565.jpg"}]},{"user_id":4230,"role":"Writing","first_name":"George","last_name":"Sandler","avatar":"https://thestacks-01.s3.amazonaws.com/users/avatars/George-Arcadia-headshot.png","in_byline":1,"priority":3,"affiliations":[{"org_id":2,"name":"Arcadia Science","slug":"arcadia-science","ror_id":"052zd4v68","avatar":"avatars/orgs/org_2_1766013565.jpg"}]}],"created_at":"2026-10-27T20:45:03.137Z","doi":"10.57844/arcadia-94t4-vs13","version_doi":null,"feedback_form_embed_src":"https://publishing-tools.arcadiascience.com/typeform/AacF8YKM?pub_name=recoMirRiSrisjdgz","id":193,"license":"CC BY","content":{"format":"gdoc"},"linked_assets":[{"type":"Code","url":"https://github.com/Arcadia-Science/world-model-orbits","name":""}],"mediaAssets":[],"orgs":[{"org_id_int":2,"priority":1,"slug":"arcadia-science","name":"Arcadia Science","avatar":"avatars/orgs/org_2_1766013565.jpg"}],"pdf_url":null,"publication_ulid":"01M5JJWY99XBJ84HEMRK4DA9G8","publisher":"The Stacks","references":[{"url":"https://doi.org/10.48550/arxiv.2602.06923","text":"Liu Z, Sanborn S, Ganguli S, Tolias A. (2026). From Kepler to Newton: Inductive Biases Guide Learned World Models in Transformers."}],"slug":"result-world-model-orbits","social_posts_count":null,"social_posts_embed_src":"https://publishing-tools.arcadiascience.com/twitter/135438306233049507","state":"PUBLISHED","subtitle":"Transformers trained on planetary orbits can predict planet movement with high fidelity under multiple possible “world models.” Here, we apply equivalent linear mapping to precisely characterize how a transformer builds its predictions from two past planetary positions.","tags":["Result","Feedback requested"],"title":"From Kepler-ish to Newton-ish: Understanding learned world models in transformers","version_desc":null,"version_number":1,"versions":[{"id":528,"version_number":1,"version_desc":null,"doi":null,"created_at":"2026-10-27T20:45:03.137Z"}]}}