I would call it generating unexpected useless and perhaps dangerous and misleading results. There is no motive like lying. There is no medical condition like hallucinations.

It knows the probabilities that one word or sentence will follow another, based on the data it was trained on. That’s it. It’s like typing exclusively through a smartphone keyboard’s predictive text, just much more complex and trained on a much larger dataset.

It has no way of connecting any word or sentence to what that word or sentence means or represents in the real world, and it’s absolute pants at making up anything new, which is why AI generated stories are cliches from top to bottom.

Like, it knows what words and phrases are associated with the word cat, or with a description prompt like “a cute story about your cat”, so it can imitate that text, or imitate it and mix it with imitations of other text patterns (" give me a cute story about your cat riding a dinosaur in a sarcastic tone"), but it’s just a mirror that reflects back what humans put into it.

If it must be compared to a human brain, it’s more like exclusively broca’s area of the human brain - the part that handled language. If broca’s area is damaged in a head injury, a human can end up with aphasia, unable to speak coherently but maybe still able to understand others perfectly well , or able to speak but unable to understand language from others, or both, but in all other ways the person can be completely fine and still able to live and solve puzzles and interact with other humans normally. (Granted, I am not a neurologist, and I’d imagine head injuries like these can probably get much more complicated.)

Anyway, hypothetically, if you were to take broca’s area out of a human brain and keep it alive in a jar on its own, without the rest of the brain: I don’t think it could be described as sentient or thinking on its own. Although the human brain is so complex and interconnected and poorly understood (beyond what area is generally associated with what function) and malleable/plastic based on experience (e.g. the brains of blind people re-purpose areas normally used for sight to he used for other purposes instead, absent visual input) that I could be wrong / it could be arguable.

The experiences of people who have had the two halves of their brain separated (this used to be a treatment for seizures, apparently), who then seem to have the two halves thinking and operating independently, or of people who manage to live largely normal lives despite missing half or more of their brain entirely, would suggest that maybe even a tiny piece of a brain could be a thinking being, I suppose.

There was an experiment a while back… MIT, maybe? Like a decade or more ago. Where they made little robots that had a kind of scaffold/medium in which they planted and grew neurons sourced from mouse brains. The robot brains, iirc, changed in reaction to the input the robots were given/their environment and sensors, and they showed personality differences in the way they behaved.

LLM’s don’t do anything like that, though, so far as I know. They don’t restructure themselves in response to input like an organic brain can. They’re a really complicated pile of if-thens, and while the inputs and therefore outputs change, the mechanism that turns inputs into outputs never does.

This comment is like 90% tangent at this point. Oh well.

Anyway, there’s a really good classic metaphor for this that explains it much better, about a man in a locked room who receives mail in a language he can’t speak through a slot in the wall, but he is still able to sort the mail in complex ways even though he can’t understand the content of any of it. I can’t remember the whole thing or the name of it off the top of my head though. I’ll try to remember.