LLMs and World Models Evidence Emergent Realities: An Analysis of World Model Representations in Large Language Models Executive Summary The question of whether Large Language Models (LLMs) possess "world models"—internal, structured representations of the world that support generalization and reasoning—is a central and fi...

Just don't make the mistake of believing that they can't develop world models – they clearly can & do!

With that, here are ChatGPT & Gemini Deep Research reports on AI World Models

Enjoy your trip down the rabbit hole! 🐇 🕳️

chatgpt.com/share/68791...

docs.google.com/document/d/...

(2) In practice, it's hard to know what an LLM has memorized, when it's using random heuristics, what it's truly grokked, or what it was in the process of grokking when training stopped

This makes using & studying LLMs tricky!

Sorry, but nobody said this would be easy!

I could go on indefinitely here, but instead, I'll leave you with a few concluding thoughts:

(1) To steel-man the paper, I'll say this:

That a model can predict the next token in a sequence does NOT imply that it has a robust world model. That much is true.

Most compelling to me: "world models" from non-language problem spaces.

Here's an example where mechanistic interpretability analysis of a Protein Language Model helped biologists identify a new protein motif

x.com/james_y_zou...

Also omitted is this paper showing how Llama-2-70B represents space and time more consistently as you get deeper into the layers.

(If you're noticing that a lot of this work is 2-3 years old, that's because most of the field has moved past this debate!)

x.com/wesg52/stat...

Not mentioned in the paper is this fascinating work from Owain & team which shows that ...

Models fine-tuned on input/output pairs from an unknown function CAN express the function *in closed form*

Seems pretty relevant!

x.com/OwainEvans_...

And besides, it's not like Isaac Newton jumped straight from raw sensory data to a theory of gravity

"Principia" came ~15 years after his work on Calculus and required intensive symbolic reasoning

Relatedly... LLMs are now world-class coders!

x.com/gdb/status/...

One thing you might try, if you really wanted to make this work, is simply ... training longer.

After all, the original Grokking paper showed generalization a full 3 OOMs after memorization.

Neural Networks are weird!

x.com/awnihannun/...

For Othello, the dataset is only 7.7M tokens!

For comparison, the 2022 (!!) work showing that models trained on Othello move sequences DO learn board state "world models" used "a synthetic dataset with 20 million games" – ~50X more data!

x.com/ke_li_2021/...

Beyond that, the models & datasets they use are super small!

For orbital mechanics, they used a 109M param Transformer and 2B tokens – roughly 1 / 10,000th the size of current frontier models & datasets. 🤔

These are not really "foundation models" at all! 🚨

The critical mistake is simple:

You can't generalize from a few failed experiments to the conclusion that something is impossible

Specifically, that "these toy models didn't generalize in the way we hypothesized" does not imply "no world models" in general

The trouble is: what do you do when your first experiments fail?

At a company pushing the AI capabilities frontier, you would try, try again!

In this case, the authors declare victory & invoke Isaac Newton to promote their "no world models" world model

x.com/keyonV/stat...

First, they train small models to predict eg- planetary orbits & Othello move sequences

Then, they fine-tune those models on related tasks – predicting gravitational forces & board states – and look for evidence of fast generalization

They call this the "inductive bias probe"

Let's start with their big idea:

IF foundation models learn "world models" representing the underlying process that generates the training data...

Then they should be able to re-use those world models to make related predictions

Cool idea, imho, but ... how to test it?

Fwiw, I don't enjoy being this guy, but when so many people are taking false comfort in shoddy AI denialism... somebody's got to do it

Plus, because the headline animation is so intuitively appealing, I was almost taken in by it myself!

You keep sharing this paper, but... I do not think it means what you think it means

In this short 🧵:
1) what the authors actually did, and why "no world models" simply does not follow
2) a few highlights from the literature on AI world models

x.com/keyonV/stat...

(10) Arguments for export controls are now explicitly focused on allowing the US to run more "AI workers" than China

OpenAI Deep Research tracks how this evolved from the original national security concerns here: chatgpt.com/share/681a7...

x.com/ohlennart/s...

(9) @Amazon blocks Operator, which I found surprising. This led me to try their shopping assistant Opus, which does work well, though tbh the Amazon shopping experience is so good – at least for routine things – that it's really hard to beat

www.perplexity.ai/search/does...

(8) Relatedly, this Personal MCP concept is super interesting to me

I can see MCP as a standard for all sorts of Personal AIs that support all sorts of access & configurations internally, & help us engage the increasingly noisy world on our own terms

x.com/betterhn20/...

(7) You could try to place blame on the AI that actually called the tool and directly caused some harm.

But … that makes the Operator model challenging. The actual contact-with-the-world agent is likely going to be taking remote direction

See also: Gemini Robotics 🤖 ☁️

(6) Pietro's planning MCP is open source, but with dreams of MCP-as-a-service in the air, many MCPs will be closed, and will be consulted more like oracles for whatever they do.

How does one think about agency in that case? Or liability?

x.com/jeff_weinst...

(5) That said, since you can call an LLM from anywhere, it can be hard to draw a box around an AI agent

For example, Augment's open-source coding Agent uses @skirano's "sequential thinking" MCP

Does that make Pietro's MCP part of the agent?

x.com/augmentcode...

(4) I like this "2 kinds of agents" model. Factored cognition has both reliability & safety benefits.

Task-specific agents with defined action space, curated context, & dedicated monitoring, all supporting human/AI agents working at a higher level 👍
x.com/hwchase17/s...

(3) The Operator model is functionally competent but not that smart, doesn't handle long context, can't use tools, & cannot "write as me" at all

It can, however, consult my @withdelphi AI clone & @Calendly via the browser

When it can call higher intelligences & tools ... 👀

(2) @OpenAI Operator is good now!

Tell it that you don't want it to confirm every step. It still will, in some cases for the better.

It can do ~100-action sequences; it tries new strategies rather than getting stuck.

Here it uses @waymark – sadly no HTML5 canvas support! 😦

10 random thoughts on AI Agents, inspired by LLM product response times, which with inference time scaling are long enough for my mind to wander 🧵

# 1 – LLMs remain weird - new Gemini 2.5 Pro still just 1 for 3 on this task, with catastrophic failures

x.com/labenz/stat...

"It's important to know that Elon's right.

OpenAI is attempting the second-biggest theft in human history.

The amicus brief is completely correct. The terms they've suggested are completely unacceptable."

@TheZvi on the @elonmusk vs @OpenAI lawsuit

"o3 is misaligned. It lies to the user & defends it onto death. It's significantly worse than o1

and GPT-4.1 seems less aligned than GPT-4o

The more RL you apply, the more misaligned they get, & we don't seem to be trying that hard"

@TheZvi on why his p(doom) is up to 70%

There's no truly safe option, but Jeremie & Edouard have published a sobering report outlining what it would really take to secure an "AI Manhattan Project"

Is it a blueprint for victory or a warning against such projects? Listen & decide for yourself!

x.com/harris_edou...

On the other hand, China "hold[s] our infrastructure at risk. One Taiwan invasion scenario, if things are not going as well as they hope ... just turn off all of our grids. Turn out the lights and let the chaos reign. Why wouldn't they do that?"

Doesn't sound good either!