ControlAI

Hinton's hope is that we get an international community of AI safety institutes and associations that can share research on developing AIs that are benevolent and don't want to take control away from humans.

Hinton points to the example of how the US and Soviet Union collaborated at the height of the Cold War to prevent nuclear war.

"The situation now is no country wants AI to take over."

Hinton says getting countries to work together on managing dangerous capabilities like cyberattacks could be hard.

But that doesn't mean cooperation is a hopeless endeavour.

At the World AI Conference, AI godfather Geoffrey Hinton warns that our current situation is like someone raising a tiger cub as a pet.

When it grows up, it can easily kill you. To survive, you must either get rid of it, or make sure it never wants to kill you.

AI godfather Geoffrey Hinton says AI isn't just a tool, and that AIs understand things in the way we understand them.

Is there anything preparing humanity for superintelligence within the next 3 to 5 years?

Ex-OpenAI researcher Daniel Kokotajlo: "Basically, no."

And many experts believe that artificial superintelligence — AI vastly smarter than humans — could be developed within the next 5 years.

Importantly, nobody knows how to ensure that superintelligence would be safe or controllable.

OpenAI's recent ChatGPT Agent was the first system classified as "High" capability in biological and chemical domains, able to "meaningfully help a novice to create severe biological harm".

On the one hand, systems like XBOW can be used to find and patch vulnerabilities in computer systems, but in the wrong hands they could potentially be used to launch unprecedented cyberattacks.

What's clear is that AIs are rapidly becoming more capable across domains.

In June, it topped HackerOne's US leaderboard. Now, only weeks later, it's ranked #1 in the world.

AI hackers could be a double-edged sword.

🚨 Breaking: AIs are now beating the best humans at hacking.

An AI just took the top spot on HackerOne's global leaderboard for finding security vulnerabilities in code.

XBOW, a fully automated AI system, has found thousands of these vulnerabilities in recent months.

Ex-OpenAI researcher Daniel Kokotajlo says that even if we don't lose control of AI, the default path could be that a tech company takes over the world, possibly even just its CEO.

Nothing Personal Superintelligence, datacenters, cooperation, and more.

From the latest edition of our newsletter:
controlai.news/p/not...

It doesn’t make sense to abandon efforts to cooperate on the risks of AI before they’ve even been tried.

In A Narrow Path, our policy package we developed to ensure the survival and flourishing of humanity, we include the design of an international framework that we believe could avoid such dangerous dynamics.

development efforts.

Currently, nobody knows how to ensure that smarter-than-human AIs are safe. Leading experts have warned that we could lose control of AI, which could result in extinction.

The principal danger of such a race is that inevitably one must prioritize rapid development over ensuring that AI systems are safe and controllable — Max Tegmark has called it a “suicide race” — though it could also provoke the use of extreme measures by participants to obstruct each other’s

This has notably been supported by AI CEOs such as Sam Altman and Dario Amodei.

In the last year or so, beginning with Achenbrenner’s essay Situational Awareness, a dangerous narrative in AI discourse has emerged of the US and China being in a winner-takes-all race to superintelligence, where the winner would have supremacy over the world, and the future.

He also spoke about the potential benefits of AI, and said China wanted to share its experience and products with other countries.

Li noted that the risks of AI have caused widespread concern, and said there’s an urgent need to build consensus on how to strike a balance between development and security, adding that technology must remain a tool to be harnessed and controlled by humans.

Will countries cooperate to prevent the worst risks from AI?

This week, at the World Artificial Intelligence Conference, Chinese premier Li Qiang proposed the establishment of an international organization to support global cooperation on AI.

"They stop listening, and then what happens?"

"Well, then they kill everyone."

Ex-OpenAI researcher Daniel Kokotajlo explains how AI could go wrong.

Ex-OpenAI researcher Daniel Kokotajlo says AI could drive humanity extinct. He says this is because AI is going to continue to get smarter and smarter, which isn't just something that could happen, it's something that AI companies are trying to make happen.

"I think the urgency of these things has gone up."

Dario Amodei says that AI systems are now as capable as smart PhD students and that this trajectory means that national security and economic issues are becoming "quite near to where we're actually going to face them".

Nothing Personal Superintelligence, datacenters, cooperation, and more.

While AI CEOs confuse and obfuscate, the superintelligence threat creeps up behind us.

Get the latest developments in AI, including Zuckerberg's superintelligence manifesto, the vast AI infrastructure buildout, and an effort to cooperate on AI:
controlai.news/p/not...

Last Tuesday, Sam Altman says things could go wrong by putting AI in charge of OpenAI.

Last Wednesday, he says it would be awesome if you could do that, and it might be possible not that long from now.

As the Future of Life Institute's independent expert panel recently found, none of these AI companies have anything like a coherent plan for how they'll prevent superintelligence from ending in disaster.

But this threat comes from the development of artificial superintelligence, exactly what Zuckerberg and other AI CEOs are racing to develop.

That's because nobody has any idea how to ensure that smarter-than-human AIs are safe or controllable.

Zuckerberg makes some allusion to the danger of AI:
"That said, superintelligence will raise novel safety concerns. We'll need to be rigorous about mitigating these risks and careful about what we choose to open source."