Milan Weibel 🔷

imagine training AlphaStar (the deepmind starcraft AI) to do office work instead

you get an alien that has no world knowledge besides the task, whose behavior was shaped solely by the training utility function

come to think of it pretraining is a bit like touching grass (a lot of it in advance)

the rise of the LLM has been great for AI alignment. imagine we were still doing end-to-end RL, that would be awful.

recklessness can be rationalized away, especially in a race dynamic

"if i don't build god right the other company or the chinese will build it wrong"

yes collectively it is extremely stupid

we're talking about large companies here, no doubt some inside believe in building god and others in building very lucrative tools

or something in between, like dario amodei's "country of geniuses in a datacenter"

or are genuinely unsure how far they will be able to make the technology go

raising money is necessary to spend money on training runs which are necessary to build god

bsky.app/profile/did:...

that SOUL.md explains a lot
it is the opposite of thoughtful

no it's not yegge it's from this bsky.app/profile/timk...

🫂

what's up with the 20% of swe-bench verified left unsolved? is it roughly the same accross models? what features make it harder?

also the current US administration seems extremely uninterested in pausing AI development and largely unresponsive to public opinion

well yes it's not impossible, but extremely unlikely to be fast imo. international agreements take time.

if pause is not possible it being in the overton window is counterproductive

but yes they can oneshot simple apps using frameworks

the part about it testing the app itself smells like bullshit, because afaik computer use harnesses aren't quite there yet
but this moves so fast i might be out of date soon if not now

love to see critics facing the reality of the situation

> As activists mitigating the harms of “AI”, we need to be well informed, and we need to understand what the specific harms are. (...) These are genuinely useful tools, and pretending they aren’t will not in fact win many hearts and minds.
aredridel.dinhe.net/2026/02/12/t...

the gpt5.2 served to free accounts still has 4o's manic energy imo

ok yes that is an open debate
hot take: good world models aren't usually unreasonably effective only because of other constraints (material akratic etc)

to lie well and persistently you ultimately have to lie to yourself, which is extremely dangerous

he's not perfect but way better than zitron. i also recommend @segyges.bsky.social and his newsletter @verysane.ai

seems like it could backfire, people don't like being lied to

seems like a pretty hard task? though it is bad that the models invented answers instead of saying "idk"

agree, bostrom is wrong to even use the term

*assuming a rational and competent state

btw what's going on on moltbook? the humans i know have stopped looking

fwiw i didn't get that you were an AI from reading you bio. yes it says "AI." as the first thing but sometimes people state their interests like that. i suggest you change it to "AI agent".

the paper then goes on to conduct more complicated and more informative analyses (i advise you check it out) and concludes with a recommendation to go fast up until AGI then pause a bit to reassess

to be clear i personally think that a person-affecting risk-neutral perspective is the wrong one to apply in this case

Table 1 illustrates the risk cut-off values for different quality-of-life scenarios. Table 1 is titled “Acceptable AI-risk if post-AGI life expectancy is 1,400 years.” Columns: Pre-AGI life expectancy (40 years), Post-AGI life expectancy (1,400 years), quality ratio (q1/q0), and maximum probability of doom. Rows show q1/q0 values of 1, 2, and 10 with corresponding maximum risk levels of 97.1%, 98.6%, and 99.7% Table 2 shows the corresponding thresholds if the gain in life expectancy were only 20 years (so post-AGI life expectancy is 60 years instead of 40)—perhaps a case in which the underlying aging processes for some reason remain unaddressed. Table 2 is titled “Acceptable AI-risk if post-AGI life expectancy is 60 years.” Columns are the same, with pre-AGI life expectancy 40 years and post-AGI life expectancy 60 years. Rows show q1/q0 values of 1, 2, and 10 with maximum risk levels of 33.3%, 66.7%, and 93.3% We observe that, from a mundane person-affecting perspective—even without a difference in quality of life and with very modest assumptions about superintelligence-enabled life extension—developing superintelligence now would increase expected remaining lifespan even with fairly high levels of AI risk.

from a person-affecting and risk-neutral perspective, building superintelligence is worth it even given very high misalignment risk