It's generally a good idea to move past the "is there a model of the world" phase into the "what are the strengths and limitations of the world-model implicit in language that LLMs have learned".

Because a LLM's knowledge of physics is highly derived not ingrained, tasks that are easy for human-brain world-models to predict can be very difficult for LLMs. It doesn't mean there isn't a world-model embedded in the weights; it's just not like ours.

For instance, if I ask GPT-4 what will happen if I throw a pool noodle at a wine glass, GPT-4 answers, "If you throw a pool noodle at a wine glass, it is generally unlikely to break the glass, especially if thrown with low to moderate force. Pool noodles are typically made of lightweight foam and are not very dense. Because of this, they usually don't have enough mass or rigidity to exert the necessary force to break glass."

Now, I can use my world-model to guide GPT-4 to get the right answer, but it's just got a really, really bad model of physics. Which isn't really surprising, because we aren't constantly talking about such things.

If you ask it something that is sufficiently adjacent to what people do actually do--like making a rope out of paper towels--it will come up with a sensible-enough answer with appropriate cautions ("Please note that a rope made from paper towels is not meant for heavy-duty use. It can be used for light, decorative purposes, or simple craft projects, but it should not be relied upon for any application where strength and durability are essential.").

I think it's the sheer weirdness of the world-model that trips up people more than anything else.

Of course it can't help but be there, because massive amounts of language is structured the way it is precisely because of how the world is. But understanding what it is like is not a straightforward proposition. We're used to interacting via language with systems that have similar world models (i.e. other humans), but our usual tricks don't work so well on LLMs.