Benno Krojer

*study in isolation

True, but it's also more loopy! Not just one clean forward pass you can study

Another way to put it:

With ai systems, we'll never have such privileged access to latents than with brains (albeit just our own, so depends how much you think we're all the same roughly)

The flip side:
With ai systems it's still very much unclear which of our human intuitions apply (anthropomorphizing) and when they're a completely different beast that require fully new theories of cognition

Sure, there's lots of fallacies and biases in that process of introspection but I wouldn't discard subjective experience as a very strong source of insight

People often say (myself too):

Interpretability on AI is so much easier than neuroscience! We can inspect everything and even retrain (vs carefully poke a little into the brain)!

One big advantage in neuroscience I often forget: We're quite literally *inside* the thing we're studying

This is a more accurate example since our method explicitly goes beyond single token interpretations --> words in the context of a sentence/paragraph

GitHub - McGill-NLP/latentlens: Code and data for the paper "LatentLens: Revealing Highly Interpretable Visual Tokens in LLMs" Code and data for the paper "LatentLens: Revealing Highly Interpretable Visual Tokens in LLMs" - McGill-NLP/latentlens

For more detailed instructions how to use the library:
github.com/McGill-NLP/l...

You can now "pip install latentlens" 🔨

It comes with:
* pre-computed embeddings for several popular LLMs and VLMs
* a txt file with sentences describing WordNet concepts, which we recommend as a standard corpus to get embeddings from
* ...

Try it out and let us know what we can improve!

Is interpretability at the random fact-gathering stage or beyond?

Finally getting into this classic

Let's see if by the end I'll have a clearer idea what type of science some fields of AI are, like interpretability

What are our paradigms?

Google decided to show this as my first sentence from my website (and not any of the sentences actually at the top of the website)

Keep me posted and feel free to ping me anytime something is confusing!

Re 2) this was a typo and should be "i" for token position consistent with later uses in 3.2 and also how we use "i" in 3.1

Maybe we can formulate it as a description d is text with optional meta-data (token position, layer) that is mapped to a vector r

The general formalism is tricky but i think the intuition is hopefully clear :)

So in our case (LatentLens) i would say:
a description here is something like "a brown *dog*" and not "a brown dog" so the token position makes it a different description (this is also how we highlight it in our demo: bennokrojer.com/vlm_interp_d...)

So I got a chance to look closely and you are right in both cases! Thank you for spotting this. I will upload a new version on arxiv soon with fixes

To clarify things here also:
1) in 3.1 we described things generally but missed that eg LatentLens would match several vectors r with a description d

Thank you! Let me get back to you later today on this when I'm on my laptop

What does it mean for visual tokens to be "interpretable" to LLM? And how to we measure it?

These, and many more pressing questions are addressed!

Introducing LatentLens -- a new, more faithful tool for interpretability! Honoured to have collaborated with
@bennokrojer.bsky.social on this!

Finally on a personal note, this will be the final paper of my PhD... what a journey it has been

Pivoting to interpretability this year was great and i also wrote a blog post on this specifically:
bennokrojer.com/interp.html

This is a major lesson i will keep in mind for any future project:

Test your assumptions, do not assume the field already has settled

This project was definitely accelerated and shaped by Claude Code/Cursor. Building intuitive demos in interp is now much easier

Finally we do test it empirically: finding some models where the embedding matrix of the LLM already provides decently interpretable nearest neighbors

But this was not the full story yet...
@mariusmosbach.bsky.social and @elinorpd.bsky.social nudged me to use contextual embeddings

Then the project went "off-track" for a while, partially because we didn't question our assumptions enough:

We just assumed visual tokens going into an LLM would not be that interpretable (based on the literature and our intuition)

But we never fully tested it for many weeks!

The initial ideation phase:

Pivoting to a new direction, wondering what kind of interp work would be meaningful, getting feedback from my lab, ...

For every one of my papers, I try to include a "Behind the Scenes" section

I think this paper in particular has a lot going on behind the scenes; from lessons learned to personal reflections

let me share some

@delliott.bsky.social joined the project mid-way and somehow still had so much positive influence, ideas and energy. Good research is done with real care for detail and you can sense Des cares about the details

I am very grateful to @sivareddyg.bsky.social's
supervision in all these years, not just in challenging me to do impactful work, but also on the human side

@mariusmosbach.bsky.social
was an amazing mentor, his ideas and writing really shaped not just this work but how i conduct research