Seb Krier

Continue reading on Firefly.Social "AI becomes the government" is dystopian: it leads to slop when AI is weak, and is doom-maximizing once AI becomes strong. But AI used well can be empowering, and push the frontier of democratic / decentralized modes of governance. The core problem with democratic / decentralized modes of governance (including DAOs on ethereum) is limits to human attention: there are many thousands of decisions to make, involving many domains of expertise, and most people don't have the time or skill to be experts in even one, let alone all of them. The usual solution, delegation, is disempowering: it leads to a small group of delegates controlling decision-making while their supporters, after they hit the "delegate" button, have no influence at all. So what can we do? We use personal LLMs to solve the attention problem! Here are a few ideas: ## Personal governance agents If a governance mechanism depends on you to make a large number of decisions, a personal agent can perform all the necessary votes for you, based on preferences that it infers from your personal writing, conversation history, direct statements, etc. If the agent is (i) unsure how you would vote on an issue, and (ii) convinced the issue is important, then it should ask you directly, and give you all relevant context. ## Public conversation agents Making good decisions often cannot come from a linear process of taking people's views that are based only on their own information, and averaging them (even quadratically). There is a need for processes that aggregate many people's information, and then give each person (or their LLM) a chance to respond *based on that*. This includes: * Inferring and summarizing your own views and converting them into a format that can be shared publicly (and does not expose your private info) * Summarizing commonalities between people's inputs (expressed as words), similar to the various LLM+pol.is ideas ## Suggestion markets If a governance mechanism values "high-quality inputs" of any type (this could be proposals, or it could even be arguments), then you can have a prediction market, where anyone can submit an input, AIs can bet on a token representing that input, and if the mechanism "accepts" the input (either accepting the proposal, or accepting it as a "unit" of conversation that it then passes along to its participant), it pays out $X to the holders of the token. Note that this is basically the same as https://firefly.social/post/x/2017956762347835488 ## Decentralized governance with private information One of the biggest weaknesses of highly decentralized / democratic governance is that it does not work well when important decisions need to be made with secret information. Common situations: (i) the org engaging in adversarial conflicts or negotiations (ii) internal dispute resolution (iii) compensation / funding decisions. Typically, orgs solve this by appointing individuals who have great power to take on those tasks. But with multi-party computation (currently I've seen this done with TEEs; I would love to see at least the two-party case solved with garbled circuits https://vitalik.eth.limo/general/2020/03/21/garbled.html so we can get pure-cryptographic security guarantees for it), we could actually take many people's inputs into account to deal with these situations, without compromising privacy. Basically: you submit your personal LLM into a black box, the LLM sees private info, it makes a judgement based on that, and it outputs only that judgement. You don't see the private info, and no one else sees the contents of your personal LLM. ## The importance of privacy All of these approaches involve each participant making use of much more information about themselves, and potentially submitting much larger-sized inputs. Hence, it becomes all the more important to protect privacy. There are two kinds of privacy that matter: * Anonymity of the participant: this can be accomplished with ZK. In general, I think all governance tools should come with ZK built in * Privacy of the contents: this has two parts. First, the personal LLM should do what it can to avoid divulging private info about you that it does not need to divulge. Second, when you have computation that combines multiple LLMs or multiple people's info, you need multi-party techniques to compute it privately. Both are important.

"AI becomes the government" is dystopian: it leads to slop when AI is weak, and is doom-maximizing once AI becomes strong. But AI used well can be empowering, and push the frontier of democratic / decentralized modes of…

https://firefly.social/post/ff-669078d24e3d46ad933c6de23225e55a?s=bsky

Very aligned with my writing and views :) happy that 'AI as a governance technology' is starting to take off!

if you're an older person who finds words like mogg and maxxing annoying, just starting using them. people over the age of 30 have the superpower to end trends by simply adopting them

A brilliant emerald green bird on a thin brown vine. The green wings are complemented by a green head ruff, red throat and long pointed bill. It has an alert black eye and looks defiant to me, but maybe I'm reading too much attitude into a little green bird. It resembles a hummingbird, but like... 'fatter', I guess? CREDIT: DrE11even, Wikimedia

Here's my other new friend from Puerto Rico.

Meet the Puerto Rican Tody which has the unfortunate scientific name of 'Todus mexicanus', thanks to a mix-up of samples by a visiting ornithologist in 1830.

There's a campaign to rename it to 'Todus borinquensis', the Taíno name for the island.

I kinda regret going too deep in old school simulators - the aim was to intuition pump the mechanisms that shape models outputs, not to say that it's the same today as it was in 2023. But overall I think there are still many simulatorsy elements to model behaviour

yes definitely agree with that! need x1000 more tests and logs and evals.

separately I wonder if we'll ever drift away from characters/personas in some cases. snac/qorporate had some great takes on gemini being more of a platform than an assistant

🫡

Open Character Training: Shaping the Persona of AI Assistants through Constitutional AI The character of the "AI assistant" persona generated by modern chatbot large language models influences both surface-level behavior and apparent values, beliefs, and ethics. These all affect interact...

In retrospect I should've used more examples from arxiv.org/abs/2511.01689

but yes pls expand I'm interested :)

Yes agree that post training methods skew things and that a pure 2023 simulators view wouldn't explain outputs well today! I think we get more coherent personas with deeper alignment, more consistent tone, better refusal behavior, and overall stylistic consistency. Tho I still see this as a persona

Had fun researching this piece, which cites Keynes & Marx but also folks of much more recent vintage, including the economists Phillip Trammell, @briancalbrecht.bsky.social,
@aleximas.bsky.social; plus @dwarkesh.bsky.social,
@sebk.bsky.social, Boaz Barak, Geoffrey Hinton.

Color image of a shorthaired white cat sitting in a window, peering placidly from between lace curtains. The window is bright with sunlight, making the curtains glow but backlighting the cat so that its face is shadowy and mysterious. This would have been a lovely composition on its own but accentuating the otherworldliness of the scene is a houseplant with long, seaweed-like leaves that undulate across the foreground, one of them arching up to frame the cat perfectly.

An Apparition. Slide from my collection, 1975.

yes huge fan of the blog!

super interesting take blog.cosmos-institute.org/p/coasean-ba...

nice!! yes I'm still thinking through the implications of this, but it definitely does bring me closer to the Drexler view of the world

experts have labeled it as "fax" and "quick maths"

considering the same. jump and I'll follow

ICYM: International bodies for which the US Administration has apparently stopped paying assessed dues include (thread):

no bro just one more LessWrong post bro I swear bro we will solve safety forever one shot just 6 more months bro we don't need to see build tech to fix it we just need to think really hard I promise bro 😭

Séb Krier on X: "Here's the meta AI safety risk I'm more concerned about. Nothing groundbreaking but just a chain of thought of how I see it. Over time, as technology gets more efficient, it gets cheaper and more accessible. Overall that's a good thing of course. I want a lot more of this and I https://t.co/W8ICVPfSnk" / X Here's the meta AI safety risk I'm more concerned about. Nothing groundbreaking but just a chain of thought of how I see it. Over time, as technology gets more efficient, it gets cheaper and more accessible. Overall that's a good thing of course. I want a lot more of this and I https://t.co/W8ICVPfSnk

But for the avoidance of doubt, this is ofc not my view 😌 this is: x.com/sebkrier/sta...

You're absolutely right!

What will AI look like by 2030 if current trends hold?

Our new report zooms in on two things: (1) whether scaling continues (compute, data, power, capital), and (2) the capabilities this enables—especially for scientific R&D.

To Make Sure AI Advances Democracy, First Ask, ‘Who Does It Serve?’ | TechPolicy.Press AI must serve the people who use it to engage in free expression and make democratic decisions, writes Richard Reisman.

AI can help protect democracies or empower those who threaten freedom: corporations, oligarchs, or governments. The key question is: who does AI serve? We should all care about the answer, writes Richard Reisman (@rreisman.bsky.social).

🧵 Several people have raised questions about what I meant by "deservedly so" when referring to the decline of institutional trust. It's a fair question, so I want to explain that, because it sits at the heart of my work on how democracies collapse when their epistemic foundations rot.

I know Yglesias isn't everyone's favorite, but it's amazing how angry many people -often with PhDs-get at a suggestion that there are tradeoffs in politics. Remarkably, many who purport to believe the republic is hanging by a thread are in the "no compromise with the voters!" camp.

Does the UK really have the 12th largest ‘female health gap’ in the world? – Full Fact Unreliable research from a male wellness company, which claimed that women are less healthy than men in the UK, has been widely shared by MPs, the media and institutions around the country.

I'm quoted in this fact-check for the claim that the UK has "the 12th largest female health gap in the world", which was widely shared by MPs & media in the UK.

The answer is no, and the male wellness company who calculated this statistic reviewed & deleted their page while this piece was written.

Societal and technological progress as sewing an ever-growing, ever-changing, patchy, and polychrome quilt Artificial Intelligence (AI) systems are increasingly placed in positions where their decisions have real consequences, e.g., moderating online spaces, conducting research, and advising on policy. Ens...

Announcing our latest arxiv paper:

Societal and technological progress as sewing an ever-growing, ever-changing, patchy, and polychrome quilt
arxiv.org/abs/2505.05197

We argue for a view of AI safety centered on preventing disagreement from spiraling into conflict.

Join us for our final symposium panel, Panel 7: Transitions to Transformative AI, at 2:15pm ET.
Panelists: @jbullockruns.bsky.social, @ruchowdh.bsky.social, @sebk.bsky.social, and Sam Manning.
Moderator: @sethlazar.org. #AIDemocraticFreedoms