Anna Tsvetkov

Thinking of all the colleagues and friends I have who are connected to Brown University. What a devastating day for Brown, and for all of us.

The Clock and the Pizza: Two Stories in Mechanistic Explanation of Neural Networks Do neural networks, trained on well-understood algorithmic tasks, reliably rediscover known algorithms for solving those tasks? Several recent studies, on tasks ranging from group arithmetic to in-con...

Some tasks admit dif algorithms that behave the same on the training data, so a model’s learned mechanism can look arbitrary unless we know what the task requires (the goals, constraints, and invariances that define a correct solution)

❓ Other cases like this or other limits of mech interp?
🧵 (2/2)

🔍 What are the limits of interpretability in ML?
Mech interp often stays at Marr’s algorithmic level but without the computational level (what the task is, what counts as the right solution) the mechanisms we find can look arbitrary. Why does a model learn one algorithm rather than another?
🧵 (1/2)

Introspection targets our ongoing or recently past mental states. What could it mean for a system that lacks any obvious analogue of a continuous stream of experience to have current or recently past “internal states” to introspect on?

Robert Long makes a similar point in his substack

Emergent introspective awareness in large language models Research from Anthropic on the ability of large language models to introspect

Anthropic has a great new piece on “Signs of introspection in large language models” 👉 www.anthropic.com/research/int...

🤔 Neat evidence that LLMs can report on manipulated activations, with big caveats!

🧠 But leaves open: what are the “internal states” an LLM can introspect in the first place?

This is a beautiful paper! The first third helpfully labels a stream of recent work in philosophy of AI as "propositional interpretability". The idea is to use propositional attitudes like belief, desire, and intention, to help explain AI in a way that we can understand. 1/n

MIT researchers release a repository of AI risks | TechCrunch A group of researchers at MIT and elsewhere have compiled what they claim is the most thorough databases of possible risks around AI use.

"The AI risk repository, which includes over 700 AI risks grouped by causal factors (e.g. intentionality), and domains (e.g. discrimination), was born out of a desire to understand the overlaps and disconnects in AI safety research"
#AIEthics

techcrunch.com/2024/08/14/m...

Would love to be included!