Mitch Harris

Constantly astonished and frustrated by the fact that so many cognitive scientists claim that LLMs can teach us about how babies learn language. If anything, we should be talking about how the way babies learn language can tell us about how AI will never be “sentient.”

3 - there is a huge academic interest bubble - the attention being paid to academic progress in AI (LLMs and Diffusion) is sucking -all- the mental energy away from existing AI research efforts. AI science that is not about extensions of prompt engineering will suffer.

2 - there is a huge -economic- AI bubble - the amount of money being invested is way larger than I expect can be recovered by income from consumers/other businesses or savings in labor costs). New investment money will dry up soon once it is noticed that promised profits aren't coming in.

Three AI Bubbles:

At this moment I feel like there are 3 interconnected but distinct AI bubbles.

1 - there is a huge AI hype bubble - the expectation of what AI can do, the new things it can make possible, is well beyond what is actually possible. People will get disillusioned very soon.

This (the idea of using LLMs as stand-ins for humans) is so very, very dumb.

Legitimately surprised there's enough published work using this method that it needed refuting, but good work for doing so!

quick while the government is shut down let’s all switch to metric

Let’s talk about ‘whoa.’

First of all, ‘woah’ is not yet considered an established spelling variant of ‘whoa.’

While ‘woah’ has grown in prevalence since the late 20th century, ‘whoa’ is still about 5 times as common as ‘woah.’

Also, all these LLM solutions (that is in addition to ChatGPT also UpToDate, Open Evidence, and Epic, all except paper textbooks) will have 'hallucinations' (= errors = differences from the source documents) to some degree - they are mitigable but statistically inevitable.

So 'presumably' is complicated but mostly not the case.

(and this most likely also holds for UpToDate's and Epic's models).

Links to source documents (in UpToDate, OpenEvidence, and in ChatGPT et al.) may be created in real time (via good ol fashioned doc search), and updated source documents themselves are still the main business for UpToDate.

In your table summarizing capabilities, you say 'presumably updated all the time', that is not the case with LLMs in general. LLMs take a long time to train (a couple months for ChatGPT) and new versions (with new training data/RLHF) is usually only updated to the public every 6 months or so.

不翼而飞

bù yì’ér fēi

to suddenly disappear

😢💔

The first known use of 'chimpanzee' was in 1738, and it comes from the Kongo dialect word 'chimpenzi.'

Once you realize that some journalists are trying to make their name by being the first to call AGI you also start to realize other journalists are trying to make their name calling the AI bubble burst. Everyone wants to talk their book.

I rarely read mainstream fiction, because of the smallness of the ideas. I've read SF for nearly fifty years, for the exact opposite reason.

If you look back over the last half century of AI, the term is always applied to stuff that doesn't quite work. Once it works reliably, we stop calling it AI.

Also for the longest time I couldn't stop thinking about the possibility of Yoko Ono marrying Victor Buono and then divorcing him and marrying Sonny Bono, rendering her Yoko Ono-Buono-Bono.

This only happens to you once

AI isn't a science fiction term with computer science. It's a decades old field that's produced a lot of useful things.

There is absolutely no good faith reason to use the term “AI” for any technology one is selling.

It serves only for dazzling people into thinking the technology has capabilities that it doesn’t.

If one wants a technology to be trustworthy, just use a transparent, informative term without hype.

For folks considering grad school in ML, my advice is to explore programs that mix ML with a domain interest. ML programs are wildly oversubscribed while a lot of the fun right now is in figuring out what you can do with it

If you’re planning to use LLMs to simulate human responses:
✔️ Treat them as workflows that need validation, thought, and time
✔️ Map analytic choices explicitly
✔️ Pilot against real data when possible
✔️ Be cautious about claims of generalizability

I'm no sociologist but using LLM responses as a proxy for human behavior seems ... nutballs.

Thanks for this kind of paper. People are going to use the tool because it is so easy...good to try to get out in front of it and herd them towards less chaotic and meaningless use.

Third Reaction: "OMG... you're studying this weird machine, not people..."
Response: "I've submitted 6 papers already! Two to be published soon!"

Second Reaction: "Wait, researchers are doing this already? People are only guessing at the training and RLHF data, and it only codes for one homogenized personality which we barely know"
Response: "We're doing the stats now!"

First Reaction: "Why would researchers even consider that?"
Response: "It is so much easier than actual humans, and LLM responses look kinda like human responses.

We are used to according the authority due to unique, once-in-a-generation geniuses to first-look reports of the unthinkably new entering human experience. Our defaults are Heisenbergs and Einstens. But here, the first-look crowd is somewhere between journalists and test-screening audiences.

"The world is everything that is the case"

What does that even mean?