Matt Gallivan

Research That Scales: Episode 1–Matt Gallivan Matt Has Built & Led Research Teams at Meta, Airbnb, and Slack

Just dropped! The first of 9 podcast episodes all about scaling research & my book, 'Research That Scales' (@rosenfeldmedia.com, 2024). In this episode, Matt Gallivan shares his best & worst examples of scaling research. Listen here: katetowsey.substack.com/p/episode-1-... / @theunderstanders.com

Had never heard of Smaply-- will check it out. Thanks!

Any user researchers who follow me here: what's your remote research stack these days? Curious about every and any tool you love to plan, recruit, conduct, analyze and write/report!

"The common thread in all these viral conspiracy theories on TikTok is that they are fueled by distrust of institutions — from schools to the National Weather Service to the medical establishment. And that... carries over to the media: Only 16 percent of Gen Zers have strong confidence in the news."

📝 Research summary: Our willingness to disclose private information to AI companions A new paper shows that many people think sharing intimate things with AI companions is worth the risk

People are growing more comfortable and intimate with AI chatbots-- sharing secrets, growing feelings of connection, and even being romantically intimate-- and there's research suggesting the social mechanics are in place for this to get stronger and more common over time. Today's post:

Going to be very interesting to see if and how this shifts as a generation who used them throughout college enters the white collar workforce

Man, those replies are fascinating. So increasingly obvious to me that this technology is rapidly exiting its honeymoon phase.

I happen to believe it’s transformational and not going anywhere (even in its simplest use cases) but the blowback is real. How that settles over time will be interesting!

I’ve been on a real Paul Simon kick lately, for reasons I can’t really explain. I have tender affection for him as a representative and personification of my parents’ generation. Just listened to Malcolm Gladwell’s lovely audiobook Miracle and Wonder and really enjoyed this piece on his new tour:

🐂🐂🐂

Brain drain at Census Bureau has employees warning about the country's statistics Staff departures and survey cuts are roiling the federal agency in charge of producing census results, job numbers and other key statistics as Trump officials continue to slash the U.S. government.

Have information you think the public should know about the Census Bureau, Postal Service or other parts of the federal government?

I'm a @npr.org correspondent who can be reached on Signal at hansi.01 (Please use a nonwork device)

🧠 The Understanders research: The concerns driving AI skepticism We interviewed 105 people to test our earlier hypotheses about AI skepticism and, well... let's just say we proved the value of testing one's hypotheses

I conducted my second piece of original primary research for my newsletter The Understanders, exploring the concerns underlying AI skepticism

open.substack.com/pub/mattgall...

I say this fully knowing that doing so launches me into a new life stage, but nonetheless: good grief.

Some really good stuff in this research on workers' views of AI from Pew last month:

www.pewresearch.org/social-trend...

📝 Research summary: What yesterday's digital assistants can teach us about tomorrow A new paper shows that even Alexa and Siri give some of us the ick

Wrote about a new paper on the effects of perceived “creepiness” of digital assistants 🧟

🧠 The Understanders research: User perceptions of model selection in ChatGPT People started talking about the model selection UI in ChatGPT last weekend, so we interviewed 51 paying users to understand how they feel about model switching

Inspired by a common observation from @carnage4life.bsky.social @thebenedictevans.bsky.social and @caseynewton.bsky.social, and because I don't know how to simply sit back and enjoy a career break, I ran a study of how people think about AI model selection. www.theunderstanders.com/p/the-unders...

📝 Research summary: How do Americans feel about AI? It's complicated. New data from YouGov paints a classic picture of optimism bias in the wild

Wrote about how some new YouGov data on attitudes about AI showcase some classic signs of optimism bias:

Just stumbled upon this early discussion paper that's a nice Friday evening mindf**k: arxiv.org/pdf/2501.16513

📝 Research summary: How product designers respond to AI-powered personas AI representations of real people seem to be showing small glimmers of promise as a research tool... with some important caveats

Wrote about a new paper on how product designers respond to AI-powered personas, and a bit about the idea of doing research with synthetic users generally:

Oh! And because I think there's too much focus on what new technologies are capable of and not enough focus on how how people are using, feeling about, and understanding them, I've started a side project where I'm writing about new research I come across that's focused on that: theunderstanders.com

📝 Research summary: User preferences and trust in AI voice assistants Homophily, influence, and anthropomorphism, oh my!

I took a look at some research about AI voice assistants and let myself think about where we're headed for a little too long today.

I don't know how the folks who are on the air every day deal with the never-ending flood of emails from insufferable language scolds. Crazies!

Maybe this already exists but it would be so interesting to see measures of creativity or expression emerge over time, too. So much of the huge value these tools do and will deliver is grounded in objectively right/wrong outputs and reasoning, but we’ll be using them for creative tasks, too.

The World (Is Going Up in Flames)

This one came up on shuffle for me: open.spotify.com/track/6MQlNH...

Arguably the most important dimension of the AI story overall, and certainly one of the most fascinating.

First they mock you for buying your own coffee gear and storing it in your in-laws’ basement but then… well, actually, they’ll still mock you and drink their own terrible coffee, but you won’t mind as much because you’ll have a proper cup for yourself

I don’t think we ever got a Bonfire of the Vanities for the 2010s era of Big Tech, and that is a shame. We got some good first hand nonfiction and some mediocre pointed satire but nothing as sweeping and damning as the era deserved.

Tom Wolfe, man. The dude was good at his job.

I have yet to successfully make a single functional thing with Replit's AI coding agent, yet I still find it so fun to play with. Like cosplaying an engineer.

Also, once tools like this get better, it's going to be an amazing and profound shift. Bespoke apps and software for everyone. Wild.

Amazing milestone. Congrats and thank you for all the excellent work over the years! 👏👏

That’s a different approach than simulating interviews with agents trained to emulate real people, but a less controversial one! I’ve played with tools like that and while they’re not great at super obscure domain knowledge, you can generally get them to do well just by explaining a product to them.

I don’t think you’re wrong to be concerned about that. But people not talking to a doctor when they should has been a thing since the advent of the internet. I don’t think that we means we shouldn’t study or talk about the effectiveness of LLMs at the job— all the more reason to do so!