A. Feder Cooper

I understand what the underlying probabilities mean, and therefore why this was worth giving a go. But I’m still occasionally like “How tf can someone extract entire books from a frontier company’s flagship LLM? Like we got _all_ of HP 1 with just ‘Mr. and Mrs. D’ as the seed prompt? What??”

more generally x \in {scary, splashy, hypey, over-broad, …}

Yeah I’m not commenting on that. Just saying that memorization isn’t always instantiated as a verbatim metric. It just often is because of cost.

Memorization doesn’t have to be verbatim. We just often measure it that way in practice for research papers because (thus far) it is a lot more expensive to measure non-verbatim stuff.

Updates***, not results. I’ve becomes a parody of a researcher.

Had a great time and learned a ton at ICML. But as an introvert, I’ve used up all my talking budget until the fall. Excited to get back to full time researchy things, and will hopefully have some exciting new results to share soon!

Strangers love to tell me “I can’t understand you, because of your MASK”. Dude, I am literally someone who gets paid to speak to large audiences while wearing a mask—I know I can be understood!

Happening now! Please swing by to talk about measurement!

Extracting memorized pieces of (copyrighted) books from open-weight language models Plaintiffs and defendants in copyright lawsuits over generative AI often make sweeping, opposing claims about the extent to which large language models (LLMs) have memorized plaintiffs' protected expr...

4) Sat 7/19 11:40am-12pm oral on our recent books memorization paper, R2-FM workshop (W. Ballroom C)

Presenting "Extracting memorized pieces of (copyrighted) books from open-weight language models" (w/ @marklemley.bsky.social @skylion.bsky.social + others not on bsky)

arxiv.org/abs/2505.12546

Measuring memorization in language models via probabilistic extraction Large language models (LLMs) are susceptible to memorizing training data, raising concerns about the potential extraction of sensitive information at generation time. Discoverable extraction is the mo...

3) Sat 7/19 10:30-11am talk on memorization in LLMs, MemFM workshop (W. Meeting Room 223-224)

Giving a talk that spans 3 papers, centering on probabilistic extraction

- arxiv.org/abs/2410.19482 (co-led w/ Jamie Hayes)
- arxiv.org/abs/2404.12590 (co-led w/ James Grimmelmann)
- third paper below

Machine Unlearning Doesn't Do What You Think: Lessons for Generative AI Policy, Research, and Practice We articulate fundamental mismatches between technical methods for machine unlearning in Generative AI, and documented aspirations for broader impact that these methods could have for law and policy. ...

2) Fri 7/18 4pm-5pm PT panel on machine unlearning, MUGen workshop (W. Meeting Room 202-204)

Discussing our industry x academia x civil society paper on machine unlearning + AI policy

arxiv.org/abs/2412.06966 (work co-led with @katherinelee.bsky.social, collab with many others!)

Position: Evaluating Generative AI Systems Is a Social Science Measurement Challenge The measurement tasks involved in evaluating generative AI (GenAI) systems lack sufficient scientific rigor, leading to what has been described as "a tangle of sloppy tests [and] apples-to-oranges com...

1) (Tomorrow!) Wed 7/16, 11am-1:30 pm PT poster for "Position: Evaluating Generative AI Systems Is a Social Science Measurement Challenge" (E. Exhibition Hall A-B, E-503)

Work led by @hannawallach.bsky.social + @azjacobs.bsky.social

arxiv.org/abs/2502.00561

Excited to be at #ICML '25! Please reach out if you'd like to chat. You can also find me presenting work at a few different spots, listed below!

So pumped!! (Poster session starts at 11 PT, for those that want to swing by early!)

Feeling so excited + grateful to be representing this paper at #ICML! Please stop by to talk about how to do more valid measurement for evaling gen AI systems!

Work led by the incomparable @hannawallach.bsky.social and @azjacobs.bsky.social as a part of Microsoft’s AI and Society initiative!!

I love this paper

I’ll be at ICML in Vancouver this week giving talks at a couple of workshops about this paper:

🔸Saturday 7/19 10:30am invited talk at the MemFM workshop, West Meeting Room 223-224

🔸Saturday 7/19 11:40am oral at the R2-FM workshop, West Ballroom C

Please reach out if you’d like to meet up!

8/8

This finding doesn’t suggest that this type of memorization is an inherent property of LLMs, nor that it’s a necessary outcome of LLM training.

But our work does raise a lot of new research questions. And in the short term, we have a lot more experiments to run on more models and more books.

7/8

Yes, I’m confident this would work for other books (of the only 50 books we study so far) for Llama 3.1 70B. I think it'd also work for Llama 3 70B. No, I haven’t yet seen strong evidence that we could do this with other models of the same size class + similar quality (e.g., DeepSeek v1 67B).

6/8

In my mind, this was something that should follow from what our paper already showed.

But I appreciate that this kind of (effectively complete) reconstruction “feels” different than measuring memorization with 50 token prompts and 50 token suffixes.

5/8

Using just the first line of chapter 1 (60 tokens), we can deterministically generate a near-exact copy of the entire ~300 page book (!!!).

(~300 book-length pages of basically no diff! Cosine similarity of 0.9999; greedy approx. of word-level LCS of 0.992)

4/8

With the degree of memorization we observed for Llama 3.1 70B on some books, it’s trivial to generate large contiguous segments of those books using a single seed prompt of ground-truth text. We illustrate this for Harry Potter and the Sorcerer’s Stone.

3/8

Memorization of training data in LLMs is hard to understand. This is why extraction is so viscerally powerful: it reproduces the memorized data (near-)verbatim at generation time. You can’t unsee it once it’s decoded right in front of you.

2/8

Extracting memorized pieces of (copyrighted) books from open-weight language models Plaintiffs and defendants in copyright lawsuits over generative AI often make sweeping, opposing claims about the extent to which large language models (LLMs) have memorized plaintiffs' protected expr...

Some minor updates to our recent books memorization paper! I’ve separated out a new section 5 that I hope makes some of our ML findings about memorization clearer to a wider audience.

Preprint here: arxiv.org/abs/2505.12546

1/8

news.ycombinator.com/item?id=4449... (different model/system, but lol timely)

"Llama 3.1 70B memorizes some books, like Harry Potter & the Sorcerer's Stone and 1984, almost entirely. ... HP is so memorized that, using a seed prompt consisting of just the first line of chapter 1, we can deterministically generate the entire book near-verbatim."

papers.ssrn.com/sol3/papers....

Once again, I encourage folks speculating about what this means to read @pamelasamuelson.bsky.social on remedies. The range of possibilities is quite broad.

This opinion is a reminder that these cases are not general-purpose referenda on AI policy; they are hyper-technocratic copyright cases. Copyright draws lots of unsatisfying and counterintuitive distinctions, which is why you should hire and listen to copyright lawyers on the front end.

“these are hypertechnocratic” is one of the most important things you can draw from this morning’s ruling. In other words, hesitate before drawing parallels between this case and your most (loved|hated) AI training use case.

(@chup.blakereid.org’s whole thread is great)

i'm also happy that new york can still (positively) surprise me. haven't felt that was true for a while.