Aidan Wright

Ah, haven’t mastered the art of a sunny side up yet I see

Has she tried the madame version?

Tempting, isn’t it?

The timing on this post from Twitter…

This is peak lakehouse

Mechanistic Science | Infection and Immunity “Science is the knowledge of consequences, and dependence of one fact upon another.” —Thomas Hobbes (7)

1:1 correspondence seems deeply unlikely. I really love this brief editorial that talks about it in terms of description and mechanism.

To use their eg of a candle and light, there’s no obvious link between the chemical combustion and the concept of illumination.

journals.asm.org/doi/full/10....

If I understand you correctly I totally agree. This absolutely should be hard. Which is why I think it’s so absurd when people act like it isn’t.

Interesting way to say it. I guess that’s probably how they think about it, as assumed. I think the best version of RDoC would be designed at adressing the multilevel/multimodal coherence issue instead of being a neural reductionism model

These are good examples that illustrate the issue.

Though another way to frame it is who was audacious (hubristic?) enough to think they could capture a psychological construct with peripheral psychophys?

I still see this in the logic of some research studies.

For sure. I can’t think of a more important issue.

This is the question of our times. Though I think answers are likely to be at least somewhat concept specific

I was thinking the other day about how much people’s ignorance of survivor bias influences their perspectives (including politics) much to everyone’s detriment

IIRC the average was ~100 words/ minute, which is what you expect when reading out loud, but there was real variability here.

Would be happy to chat more if you're interested in pursuing collaboratively, but if not we include our prompts in the manuscript, etc.

I don't have a great sense for the exact minimum. No doubt it will depend on eliciting prompt, construct, and model. But we do see that absolute error is modestly associated with length and at lower number of diaries/minutes (2 samples) we see things stabilize at some point.

You could also create multi informant latent variables, with either self and LLM or multiple LLMs. Trait scores are highly correlated across models but not perfectly so.

For sure. A limitation of the data we had available for this study is that we lacked non-self-report outcomes. So, due to method overlap self-report almost always would “win”. To show incremental validity ideally we’d try to predict unprivileged outcomes.

It’s already longer than we were shooting for at first. It would be good to add that though on revision or rejection.

I think this is fair criticism and easy to add. True story. We had that as a focus early on when we first did this with just ChatGPT. Then we thought it would be better not as an ad for openAI and went for many models. Then the output started ballooning and it sort of got unwieldy and lost track

Ha

To say nothing of the sand orcs

I draw this distinction because I think there are other ways to set this sort of thing up where you train this or another model up to specifically predict self report. Here we ask the LLM to rate personality based on the text we give it. We discuss in context of self-other agreement literature

Thanks. A nuance, perhaps, but we don’t treat self-reports as “true” in an absolute sense. The personality lit has dispelled the notion of self-as-true a long time ago. We think of this as a different rater based on different modality of data. I think this nuance is critical, others might disagree.

There is also potential for getting info that isn’t easily captured by questionnaire. Here we use the big5 as the target because we had self-reports to compare but also because the info contained is so so broad. I’m pretty convinced we can start extrapolating.

I particularly think this holds potential for taking narrative summaries collected in real- or near-time to events (as we do in Study 2) before they are filtered through days of interpretation and meaning making by folks. Also that is a context where we can’t give long lists of questionnaire items.

If I may, this is sort of a double barreled question. 1. I think we should typically trust our patients. At least that their reports reflect their experiences accurately. 2. The value of this method isn’t in getting more trustworthy info, but in aggregating and distilling info from other data.

It is wild. That’s an apt description. When we started doing this I almost fell out of my chair I was so surprised.

Thanks for engaging, Michael! Great idea. As you noted, one of the big takeaways/demonstrations here was how much this interface can democratize the use of LLMs. But also we’re very interested in understanding this all better and figuring out what works best. Let me chat with team and get back to u

So, ironically perhaps, the spell check AI didn’t catch the error in the figure caption. As much as I wish we had, we did not have “sand elf-reports” in this study. As you all know, sand elves are impossible to work with. It was just human self-report.

H/t @gregdepow.bsky.social for catching this

The robots know us so well.

We show pretrained LLM ratings of personality from brief snippets of natural language achieve agreement with self-reports that are comparable to family and friends.