P. Razavi

Information-Guided Identification of Training Data Imprint in (Proprietary) Large Language Models High-quality training data has proven crucial for developing performant large language models (LLMs). However, commercial LLM providers disclose few, if any, details about the data used for training. ...

Want to know what training data has been memorized by models like GPT-4?

We propose information-guided probes, a method to uncover memorization evidence in *completely black-box* models,

without requiring access to
🙅‍♀️ Model weights
🙅‍♀️ Training data
🙅‍♀️ Token probabilities 🧵 (1/5)

I’ve been referring people (esp social science/psych PhD students) to this blog post for years. The headline and opening paragraph are all you really need.

Had a great time presenting our work on LLM-based item difficulty estimation at #NCME .
If you’re in Denver and would like to discuss measurement research or just catchup in the next couple of days, let me know 😊

Fantastic, thoughtful work! 👏👏

Estimating Item Difficulty Using Large Language Models and Tree-Based Machine Learning Algorithms Estimating item difficulty through field-testing is often resource-intensive and time-consuming. As such, there is strong motivation to develop methods that can predict item difficulty at scale using ...

If you're interested in learning more and plan to attend the #NCME conference in Denver next week, we’d love to see you at our coordinated paper session, “Approaches to Optimizing a Personalized Learning System,” on Friday, April 25, from 11:30 AM to 1:00 PM. (🧵9/9)
arxiv.org/abs/2504.08804

We are excited about the potential of these methods to support more efficient item development in education. In the preprint, we provide a seven-step workflow for testing professionals who would want to implement a similar item difficulty estimation approach with their item pool. (🧵8/9)

The feature-based approach presumably benefits from the language model’s extraction of multiple cognitive and linguistic dimensions that an ensemble tree-based algorithm then “learns” to weight in ways that maximize prediction accuracy. (🧵7/9)

The modest performance of direct LLM estimates in some instances, and the more robust performance of feature-based methods, hints that LLMs can add value, but that this value is maximized when the model is “nudged” or structured via psychometric frameworks. (🧵6/9)

The results are promising, especially for the feature-based approach which performed considerably better than the dummy regressor benchmarks and the direct estimation approach. (🧵5/9)

In the second approach, we use the LLM to extract cognitive and linguistic features from each item. We then train tree-based machine learning models (i.e., random forest and gradient boosting machines) to estimate item difficulty based on the features. (🧵4/9)

In the first approach, we use a direct estimation method that prompted the LLM to assign a single difficulty rating to each item based on qualitatively informed criteria. (🧵3/9)

Field-testing assessment items to estimate difficulty can be both costly and time-consuming. In this research, we evaluate two LLM-based approaches to predict item difficulty for K-5 mathematics and reading assessments based on item content. (🧵2/9)

Estimating Item Difficulty Using Large Language Models and Tree-Based Machine Learning Algorithms Estimating item difficulty through field-testing is often resource-intensive and time-consuming. As such, there is strong motivation to develop methods that can predict item difficulty at scale using ...

I'm excited to share our latest work: "Estimating Item Difficulty Using Large Language Models and Tree-Based Machine Learning Algorithms." (🧵 1/9)
arxiv.org/abs/2504.08804

Register your interest for our infrastructure workers' interview study The Developer Success Lab is looking for individuals interested in having a 1 hour, remote, conversation with a member of our research team in the next few weeks (April 16–May 16). Specifically, we're...

The Developer Success Lab is looking for engineers who work in infrastructure for a research study on the working experience of SREs, platform engineers, DevOps engineers, and software developers who maintain or develop infrastructure.

Wooden Shoe Tulip Festival
📍Portland, Oregon 🇺🇲

A yellow street sign in Japanese with three black flying through the sky. Below is text that says ネコ飛出し注意 (neko tobidashi chūi) Means “watch for cats darting out”

ネコ飛出し注意 (neko tobidashi chūi) Means “watch for cats darting out” and I love this sign.

A tricky thing about modern society is that no one has any idea when they don’t die.

Like, the number of lives saved by controlling air pollution in America is probably over 200,000 per year, but the number of people who think their life was saved by controlling air pollution is zero.

HPS in 20 objects This resource was produced by academics from the Centre for History and Philosophy of Science at the University of Leeds, where we have our Museum filled with artefacts that tell a stories about the H

Did you know: our researchers have developed a suite of resources for A-Level students and teachers? "History & Philosophy of Science in 20 Objects" draws on an incredible array of items from our own collection ft. prompts, questions, videos and more! sway.cloud.microsoft/cEekCFBF5CGF... #histsci

It is almost as if we need to interrogate the concepts we're discussing.

@mohammadatari.bsky.social @mdehghani.bsky.social can you help?

Congrats 👏🏽🎉. Very well-deserved! 😊

rough (like uff in buff)
cough (like off in scoff)
drought (like ow in cow)
though (like o in no)
thought (like aw in saw)
through (like oo in woo)

Enough.

Hello to all my friends at SPSP seeing this message in a hallway or lobby as you hope you are staring at your phone with enough noticeable intensity to avoid having to interact with anyone

1/3

Tutorial on exploring ecological momentary assessment data is online at AMPPS, with:
- Accessible ways to visualize data for better understanding
- Models to get some first insights
- Further reading boxes for more advanced topics
- Reproducible pipeline you can run over your own data

Some of us have been meeting up at SPSP for the last few years. This year marks our fifth gathering. Email one of us if you want to join! Location TBD.

@mdehghani.bsky.social @drsanaz.bsky.social @simine.com @dorsaamir.bsky.social

My husband just inadvertently inspired one of the simplest, most relatable XY problem¹ demos I've seen.

He asked if I could buy unscented TP 🧻 next time I grocery shop.

Knowing he had been getting a cold, I probed: when does the scent become a problem?

[1/3]

¹ en.m.wikipedia.org/wiki/XY_prob...

...to examine the differences bet. justified and unjustified anger. No matter how we analyze it, these two variants have differences across cognitive, affective, moral, and relational dimensions. These findings have significant implications for theories of anger and intervention strategies.

I’ll share a more detailed thread on this work later, but for now, I’m excited to share this preprint with the Blsky community! In this research, we used a range of methodologies including thematic analysis, closed- and open-vocabulary analyses (e.g., LIWC, topic modeling), and prototype approach...

Lots of useful info in this thread if you are backing up public data (whether at OSF or elsewhere)

Also, 😉