Existential Risk Observatory

If one in ten experts think there is a risk of human extinction when developing a technology, we should not develop this technology, until we are confident that the risk can be almost ruled out.

Can a small startup prevent AI loss of control? - with Riccardo Varenna · Luma According to many leading AI researchers, there is a chance we could lose control over future AI. We think one of the most important challenges of our century…

📢 Event coming up in Amsterdam!📢

Many think we should have an AI safety treaty, but how to enforce it?🤔

Riccardo Varenna from TamperSec has part of a solution: sealing hardware within a secure enclosure. Their proto should be ready within three months.

Time to hear more!

Be there! lu.ma/v2us0gtr

BREAKING: New experiments by former OpenAI researcher Steven Adler find that GPT-4o will prioritize preserving itself over the safety of its users.

Adler set up a scenario where the AI believed it was a scuba diving assistant, monitoring user vitals and assisting them with decisions.

YouTube video by RNZ Humans "no longer needed" - Godfather of AI | 30 with Guyon Espiner S3 Ep 9 | RNZ

youtu.be/uuOPOO90NBo?... 15:15

Slowly, but surely, the public is getting informed that there is a level of AI that may kill everyone. And obviously, an informed public is not going to let that happen.

Never mind SB1047. In the end, we will win.

What is interesting is that the presenter assumes familiarity with not only the possibility that AI could cause our extinction, but also the fact that many experts think there is an appreciable chance this may actually happen.

Two weeks ago, Geoffrey Hinton informed a New Zealand audience that AI could kill their children. The presenter announced the part as: "They call it p(doom), don't they, the probability that AI could wipe us out. On the BBC recently you gave it a 10-20% chance".

The closer we get to actual AI, the less people like intelligence, however measured. Passing the Turing test is downplayed now, but passing Marcus' Simpsons test will be downplayed later when it happens, too.

Still, AI reaching human level is actually important. We can't keep our heads in the sand.

More info and discussion here:
forum.effectivealtruism.org/posts/XJuPEy...
www.lesswrong.com/posts/sc4Kh5...

- Offense/defense balance. Many seem to rely on this balance favoring defense, but so far little work has been done on aiming to determine whether this assumption holds, and in fleshing out what such defense could look like. A follow-up research project could be to shed light on these questions.

Our follow-up research might include:

- Systemic risks, such as gradual disempowerment, geopolitical risks (see e.g. MAIM), mass unemployment, stable extreme inequality, planetary boundaries and climate, and others.

- Require security and governance audits for developers of models above the threshold.
- Impose reporting requirements and Know-Your-Customer requirements on cloud compute providers.
- Verify implementation via oversight of the compute supply chain.

Based on our review, our treaty recommendations are:

- Establish a compute threshold above which development should be regulated.
- Require “model audits” (evaluations and red-teaming) for models above the threshold.

International Agreements on AI Safety: Review and Recommendations for a Conditional AI Safety Treaty The malicious use or malfunction of advanced general-purpose AI (GPAI) poses risks that, according to leading experts, could lead to the 'marginalisation or extinction of humanity.' To address these r...

Our paper "International Agreements on AI Safety: Review and Recommendations for a Conditional AI Safety Treaty" focuses on risk thresholds, types of international agreement, building scientific consensus, standardisation, auditing, verification and incentivisation.

arxiv.org/abs/2503.18956

New paper out!📜🚀

Many think there should be an AI Safety Treaty, but what should it look like?🤔

Our paper starts with a review of current treaty proposals, and then gives its own Conditional AI Safety Treaty recommendations.

YouTube video by UBC Computer Science Rich Sutton - The Future of AI

Richard Sutton has repeatedly argued that human extinction would be the morally right thing to happen, if AIs were smarter than us. Yesterday, he won the Turing Award from @acm.org.

Why is arguing for and working towards extinction fine in AI?

youtu.be/pD-FWetbvN8&...

It is hopeful that the British public and British politicians support regulation to mitigate the risk of extinction from AI. Other countries should follow. In the end, a global AI Safety Treaty should be signed.

AI Safety Debate with prof. Yoshua Bengio · Luma Progress in AI has been stellar and does not seem to slow down. If we continue at this pace, human-level AI with its existential risks may be a reality sooner…

On the eve of the AI Action Summit in Paris, we proudly announce our AI Safety Debate with Prof. Yoshua Bengio!📢

In the panel:

@billyperrigo.bsky.social from Time
@kncukier.bsky.social from The Economist
Jaan Tallinn from CSER/FLI
Emma Verhoeff from @minbz.bsky.social

Join here! lu.ma/g7tpfct0

Pretraining may have hit a wall, but AI progress in general hasn't. Progress in closed-ended domains such as math and programming is obvious, and worrying.

The public needs to be kept up to date on both increasing capabilities, and obvious misalignment of leading models.

Nobel Prize winner Geoffrey Hinton thinks there is a 10-20% chance AI will "wipe us all out" and calls for regulation.

Our proposal is to implement a Conditional AI Safety Treaty. Read the details below.

www.theguardian.com/technology/2...

💼 We're hiring a Head of US Policy! ⬇️

🇺🇸 This opening is an exciting opportunity to lead and grow our US policy team in its advocacy for forward-thinking AI policy at the state and federal levels.

✍ Apply by Dec. 22 and please share:
jobs.lever.co/futureof-life/c933ef39-588f-43a0-bca5-1335822b46a6

Peaceful activism from organizations such as @pauseai.bsky.social is a good way to increase pressure on governments. They need to accept meaningful AI regulation, such as an international AI safety treaty.

It is still quite likely AGI will be invented in a relevant timespan, for example the next five to ten years. Therefore, we need to continue informing the public about its existential risks, and we need to continue proposing helpful regulation to policymakers.

Our work is just getting started.

It doesn't appear like we have quite figured out the AGI algorithm yet, despite what Sam Altman might say. But more and more startups, and then academics, and finally everyone, will be in a position to try out their ideas. This is by no means a safer situation.

So are we back where we started? Not quite. Hardware progress has continued. As can be seen in the graph above, compute is rapidly leaving human brains in the dust. Also, LLMs could well provide a piece of the puzzle, if not everything.

Leading labs no longer bet on larger training runs, but increase capabilities in other ways. Ilya Sutskever: "The 2010s were the age of scaling, now we're back in the age of wonder and discovery once again. Everyone is looking for the next thing. Scaling the right thing matters more now than ever."

It is now public knowledge that multiple LLMs significantly larger than GPT-4 have been trained, but they have not performed much better. That means scaling laws have broken down. What does this mean for existential risk?

There Is a Solution to AI's Existential Risk Problem A Conditional AI Safety Treaty can help prevent runaway AI and President-elect Trump should embrace the idea, writes Otto Barten.

Read the full piece here: time.com/7171432/cond...

We realize that a lot of work needs to be done to get the Conditional AI Safety Treaty implemented and enforced. But we believe that if we really want to, these challenges are by no means beyond humanity's reach.

We can solve existential risk, if we want to.

We think our proposal is going in broadly the same direction as others, such as Max Tegmark (@fliorg.bsky.social), @npcollapse.bsky.social (Conjecture), and Andrea Miotti (@controlai.com).

We welcome their great work and are open to converging towards the most optimal solution.