SE Gyges

so the small draft model failed, presumably, almost every time, driving load and costs up probably by something like a factor of two or five and leading openai to claim it was ddosed

which it was. by its customers

(no I don't know for sure this is why they dropped but it lined up)

why would they be wrong? because the small model was trained on normal text, or on normal text for chatgpt BEFORE the price drop

gpt-4 was still the best llm on earth by far and the price had just dropped. thousands of very serious nerds immediately spammed it with shit no LLM had ever seen before

the time i am thinking of, openai had a sharp sharp price drop right as spec decoding was well fleshed out and then immediately had an outage and blamed a ddos

my best guess: they set up speculative decoding, and priced on some reasonable level of "cache hit" ie the small model guessing right

i will never not repost borges

a fun story about speculative decoding: i am pretty sure openai fucked themselves with it once

in spec decoding a small llm guesses what the big llm is going to do, and the big llm can check the result quickly

this works like a cache, and it can miss like a cache

I know what comes next

https://github.com/tanishqkumar/ssd

not unheard of

my oddities these days are mostly deepseek which is like, barely finetuned

will i mention that X llm did Y thing if i am talking to someone? yes. did i write up a detailed report of my experience? no. i had seven of them that day. they were all strange in different ways

this has the effect that there are all sorts of weird LLM quirks that are mostly known by oral tradition because the volume of them is simply much too high

one of the funny things is that most of us who deal with them a lot will encounter a doomloop or similar when we are DOING SOMETHING ELSE and so usually we barely or don't chronicle the fact that it happened

seems pretty obvious. time dilation makes them effectively eternal

"how close to the line am i" joke

is gemini like this because they gave it a name that suggests it is into astrology? in the same way that making an LLM a nazi makes it dumb and a bad coder, naming your LLM "gemini" suggests it's really into crystals and has self esteem issues

count how many times it says "I will be" in that doomloop. almost any level of repetition penalty would prevent it from outputting that, either at serving time or at RL time

huh. where's that from, exactly? one of the soundtrack bits?

they could literally fix it with a repetition penalty, a widely known techinque that has been available to everyone for at least five years

if anyone has never seen the gemini doomloop it looks like this

a beta beto, if you will

Basically I am going through "which of these people would be an obvious person for a senator to talk to" and most other people are distinctly non-obvious

i take it super seriously. i just think you have to generalize it past quick text exchanges to get any signal out of it now. turing would've loved that

llms should have native audio output so they can rap battle and we can determine the best lab that way

please remember: he's not mad

Time to support trans Dems like @beth4ma.bsky.social

lmfao

isn't he canadian

is this meme retro yet

an LLM is the plucked chicken for our age

*sufficient

the tldr is that llms show sufficiently evidence of consciousness that it depends which definition of the word you use, even, which is probably a major source of the drama

like we basically mean "human-like" and they're clearly human-like in some ways but not others