Kevin Feng

Artificial Intelligence and Democratic Freedoms

Special thanks to @katygb.bsky.social and @sethlazar.org for organizing a fantastic @knightcolumbia essay series on AI and Democratic freedoms, and other authors of the essay series for valuable discussion + feedback. Check out the other essays here! knightcolumbia.org/research/art....

Finally, autonomy levels can help with agentic safety evaluations and setting thresholds for autonomy risks in safety frameworks. How do we know what level an agent is operating at, and what appropriate risk mitigations should be put in place? Read our paper for more!

We also introduce **agent autonomy certificates** to govern agent behavior in single- and multi-agent settings. Certificates constrain the space of permitted actions and help ensure effective and safe collaborative behaviors in agentic systems.

Why have this framework at all?

We argue that agent autonomy can be a deliberate design decision, independent of the agent's capability or environment. This framework offers a practical guide for designing human-agent & agent-agent collaboration for real-world deployments.

Level 5: user as an observer.

The agent autonomously operates over long time horizons and does not seek user involvement at all. The user passively monitors the agent via activity logs and has access to an emergency off switch.

Example: Sakana AI's AI Scientist.

Level 4: user as an approver.

The agent only requests user involvement when it needs approval for a high-risk action (e.g., writing to a database) or when it fails and needs user assistance.

Example: most coding agents (Cursor, Devin, GH Copilot Agent, etc.).

Level 3: user as a consultant.

The agent takes the lead in task planning and execution, but actively consults the user to elicit rich preferences and feedback. Unlike L1 & L2, the user can no longer directly control the agent's workflow.

Example: deep research systems.

Level 2: user as a collaborator.

The user and the agent collaboratively plan and execute tasks, handing off information to each other and leveraging shared environments and representations to create common ground.

Example: Cocoa (arxiv.org/abs/2412.10999).

Level 1: user as an operator.

The user is in charge of high-level planning to steer the agent. The agent acts when directed, providing on-demand assistance.

Example: your average "copilot" that drafts your emails when you ask it to.

Levels of Autonomy for AI Agents

Web: knightcolumbia.org/content/leve....
arXiv: arxiv.org/abs/2506.12469.

Co-authored w/ David McDonald and @axz.bsky.social .

Summary thread below.

Levels of Autonomy for AI Agents

Here are links to the paper:

Web: knightcolumbia.org/content/leve....
arXiv: arxiv.org/abs/2506.12469.

Co-authored w/ David McDonald and @axz.bsky.social .

Artificial Intelligence and Democratic Freedoms

Special thanks to @katygb.bsky.social and @sethlazar.org for organizing a fantastic @knightcolumbia.org essay series on AI and Democratic freedoms, and other authors of the essay series for valuable discussion + feedback. Check out the other essays here! knightcolumbia.org/research/art....

Finally, autonomy levels can help with agentic safety evaluations and setting thresholds for autonomy risks in safety frameworks. How do we know what level an agent is operating at, and what appropriate risk mitigations should be put in place? Read our paper for more!

We also introduce **agent autonomy certificates** to govern agent behavior in single- and multi-agent settings. Certificates constrain the space of permitted actions and help ensure effective and safe collaborative behaviors in agentic systems.

Why have this framework at all?

We argue that agent autonomy can be a deliberate design decision, independent of the agent's capability or environment. This framework offers a practical guide for designing human-agent & agent-agent collaboration for real-world deployments.

Level 5: user as an observer.

The agent autonomously operates over long time horizons and does not seek user involvement at all. The user passively monitors the agent via activity logs and has access to an emergency off switch.

Example: Sakana AI's AI Scientist.

Level 4: user as an approver.

The agent only requests user involvement when it needs approval for a high-risk action (e.g., writing to a database) or when it fails and needs user assistance.

Example: most coding agents (Cursor, Devin, GH Copilot Agent, etc.).

Level 3: user as a consultant.

The agent takes the lead in task planning and execution, but actively consults the user to elicit rich preferences and feedback. Unlike L1 & L2, the user can no longer directly control the agent's workflow.

Example: deep research systems.

Level 2: user as a collaborator.

The user and the agent collaboratively plan and execute tasks, handing off information to each other and leveraging shared environments and representations to create common ground.

Example: Cocoa (arxiv.org/abs/2412.10999).

Level 1: user as an operator.

The user is in charge of high-level planning to steer the agent. The agent acts when directed, providing on-demand assistance.

Example: your average "copilot" that drafts your emails when you ask it to.

Screenshot of a paper on a webpage, with a figure showing 5 levels of autonomy for AI agents and the tradeoff between user involvement and agent autonomy as the levels increase.

📢 New paper, published by @knightcolumbia.org

We often talk about AI agents augmenting vs. automating work, but how exactly can different configurations of human-agent interaction look like? We introduce a 5-level framework for AI agent autonomy to unpack this.

🧵👇

Hi Dave, this is super cool! I'm currently working on a research prototype that brings domain experts together to draft policies for AI behavior and would love to chat more, esp about how I can integrate something like this

Levels of Autonomy for AI Agents

In an essay for our AI & Democratic Freedoms series,
@kjfeng.me, @axz.bsky.social (both of @socialfutureslab.bsky.social), & David W. McDonald outline a framework for levels of #AI agent autonomy that can be used to support the responsible deployment of AI agents. knightcolumbia.org/content/leve...

Excited to kick off the first workshop on Sociotechnical AI Governance at #chi2025 (STAIG@CHI'25) with a morning poster session in a full house! Looking forward to more posters, discussions, and our keynote in the afternoon. Follow our schedule at chi-staig.github.io!

@kjfeng.me and @rockpang.bsky.social are kicking off the panel on sociotechnical AI governance!

#CHI2025

I'll be at CHI in-person in Yokohama 🇯🇵. Let's connect to chat about anything from designing LLM-powered UX to AI alignment + governance!

Thanks to my fantastic collaborators @qveraliao.bsky.social @ziangxiao.bsky.social @jennwv.bsky.social @axz.bsky.social and David McDonald. This work was done during a summer internship with the FATE team at Microsoft Research.

Canvil | Figma Canvil allows you to craft AI-powered user experiences by shaping the behavior of large language models (LLMs) right in your canvas. Works in both Figma and FigJam! Demo (3 mins): https://youtu.be/E0...

We published Canvil on Figma community and it's already used by 2k+ designers worldwide.

Try it yourself on Figma today! www.figma.com/community/wi....

Or, check out a demo here: youtu.be/E0l-IH_Lo7Q.

Finally, Canvil's seamless integration into Figma allowed designers to use their existing collaborative workflows to work with design and non-design teammates on model adaptatation.

We believe lowering the barrier to adaptation can catalyze more thoughtfully designed LLM UX.

Designers also used constraints from their existing design requirements and UIs to shape model behaviors. For example, many devised ways to embed cultural preferences and customs into models based on the needs and backgrounds of their target users.