Nate

When it comes to stopping the reckless race to artificial superintelligence: Don't sit around hoping for a warning shot. Prepare well.

The moral of the first tale is that warning shots alone don't save you. The moral of the second tale is that you don't need to wait for exactly the right disaster to do something you think needs doing. You just need to prepare well, and wait for an excuse.

A tale of two warning shots, #2: George W. Bush was worried about Saddam Hussein. Some completely unrelated terrorists slammed planes into the twin towers. That was the final straw. The USA took out Saddam Hussein.

A tale of two warning shots, #1: COVID happened. Scientists are divided on whether it was a lab leak. The world did not rally against dangerous viral research in labs. The warning shot was squandered.

(Also: modern AIs are trained on problem-solving, not just prediction. But even ignoring that, "next-token prediction" is not an atomic ability that is superficial and therefore safe; doing it really really well would require potent & dangerous internal mechanics.)

"The AI just predicts the next token" is like "the container just lifts up." How does it achieve lift? Both a hot air balloon and a rocket lift up, but one of them violently explodes and kills everyone in range if you design a big one even slightly wrong.

Just having the book in front of the crowd here was probably worth more than all my sideline interviews and roundtables combined. I'm honored by whoever made that call. I've heard folks behind the conf see superintelligence as a possibility worth putting on the radar. Progress.

The empty holders are where more copies used to sit, and at the beginning of the conference there were stacks. When I walked in just now, the people working at the bookstore were reading it and asked for my signature.

The Munich Security Conference has a little bookstore where they feature about a dozen books, including If Anyone Builds It, Everyone Dies. Most of the books have multiple stacks left. My book has exactly one copy left.

I gave my remarks to a bunch of high-powered intelligence community folks, and didn't shy away from the dangers. Most responded with something mild that avoided acknowledging superintelligence was a possiibilty. But they also didn't argue against it. I'm honored they invited me. It's progress.

A couple people have recognized me and told me they like my book and confided in me that they harbor worries. Then when it's their turn to talk they say something mild that avoids acknowledging superintelligence as a possibility.

I also feel a bit like a child here because it keeps feeling like I'm the child straight out of "The Emperor Has No Clothes".

I was invited to give remarks at a high-powered intelligence community dinner, but my badge lacked the "high powered intelligence community member" marker so I got held up at the door until a grownup came and rescued me.

I, uh, also had to go to the mall and buy a suit real quick, once I noticed what kind of conference I was at.

For some reason the check-in instructions didn't make it to me, so I just tried to waltz onto the premises rather than routing through the special check-in desk that was squirreled away in a building I was unaware of. The police didn't love that.

I feel like a child here, in part because most everyone else is older and in part because I keep having funny mishaps. For instance:

I'm at the Munich Security Conference. It's interesting. It feels a lot like a normal academic conference, except it's packed with generals and senators and there's armed police pouring out or every orifice. 🧵