Martin Saveski

Consider submitting an ICWSM workshop proposal! It’s a great opportunity to create space for discussions around emerging research threads, new methods, or even old but exciting topics. Deadline: Jan 30.

I feel like colleagues are winking at me when they write a descent letter but don't select the highest option 😀

I didn't expect this until I was on the other side, but I do find them a bit informative. The culture is such that ppl feels they need to write a "good" letter and you can emphasize the positive aspects in the letter. But when explicitly asked, I think most people find it hard to be untruthful.

The Center for Information Technology Policy at Princeton invites applications for a Postdoctoral Fellow to work with Andy Guess (Politics/SPIA), Brandon Stewart (Sociology), and me (CS).

puwebp.princeton.edu/AcadHire/app...

Please apply before Sunday, the 13th of December!

Some social media algorithms are feeding us antidemocratic attitudes and intergroup hostility.

But a new field experiment finds that an algorithmic feed can reduce out-group animosity & affective polarization by down-ranking this hostile political content.
www.science.org/doi/10.1126/...

Sure, the "dosage" of the decreased exposure intervention depends on how much AAPA participants had in their feed (per party breakdown in Fig. S3); increased exposure was similar for everyone. But reweighting by party, education, and race doesn’t change the point estimates much (see Sec. S10).

Screenshot of research article in Science titled "Reranking partisan animosity in algorithmic social media feeds alters affective polarization." Full text available at https://www.science.org/doi/10.1126/science.adu5584.

What if you could see fewer hostile political posts on social media? A new paper out in Science by Martin Saveski @msaveski.bsky.social of the iSchool, along with @tiziano.bsky.social, @jiachenyan.bsky.social, Jeff Hancock, Jeanne Tsai and @mbernst.bsky.social, explores this: doi.org/10.1126/scie...

Nice! I'm glad you enjoyed it! Actually, @jugander.bsky.social strongly recommended it to me when I was there a few years ago.

Re SUTVA: My experience doing empirical and methodological work on interference (e.g., doi.org/10.1145/3097...) has kept me humble when trying to predict total treatment effects.

(Also, that's where we got the idea to contextualize with historical change.)

The promise and pitfalls of cross-partisan conversations for reducing affective polarization: Evidence from randomized experiments Cross-partisan conversations can reduce affective polarization, but effects do not persist long-term or spill over.

We actually did a thorough lit review when doing the power analysis and, if you look closely, there aren’t many experiments that used the same outcome to compare with. My best reference point is the excellent paper by Santoro & @dbroockman.bsky.social : doi.org/10.1126/scia...

That’s why we tried to contextualize the results in terms of historical change in the metric.

Well, they ask whether a 2-degree change is a small effect, and I think it’s a reasonable question. I’ve discussed this with quite a few people who have done extensive empirical work in this area and whose opinions I value. For some, 2 degrees is small; for others, it’s huge ...

Thank you! I have been meaning to send all of your a note for while, but I can't overstate how helpful your Green Lab SOP was in analyzing the data!

Thanks for the shoutout! Obviously many possible reasons for the differences but my best guess is (i) content vs. user level intervention (i.e., reranking content likely to polarize) and (ii) much higher prevalence of political content on X (32% on X vs. 13.4% on FB). Curious to hear your thoughts.

Finally, in this work, we focused on affective polarization, but our framework for LLM-based feed ranking is general and can be applied to other outcomes, including well-being, mental health, and civic engagement.

/fin

We hope that other researchers will use our methodology to run experiments that are longer, span multiple platforms, and extend beyond the US.

/13

Important limitations to keep in mind: (i) this was a 10-day experiment, (ii) run on a single platform, and (iii) during a politically charged time.

/12

Increasing exposure to AAPA didn’t lead to any detectable effects on engagement, likely because we reranked far fewer posts.

/11

Reducing exposure to AAPA led to a decrease in engagement in absolute terms: less time spent, less posts viewed, and liked. However, among the posts that the participants viewed, they liked them at a significantly higher rate.

/10

Effects of reducing and increasing exposure to AAPA content in participants’ feeds on their experiences of emotion compared with that of the corresponding control group. (Left) Participants were surveyed within the feed during the intervention [scale ranged from 0 (“none at all”) to 100 (“extremely”)] and (right) off-platform after the experiment [scale ranged from 1 (“never”) to 5 (“all the time”)]. The filled circles represent statistical significance (Padj < 0.05, adjusted for multiple hypothesis testing), and the error bars represent 95% CIs.

Decreasing exposure to AAPA made participants less angry and sad in the moment while increasing exposure had the opposite effect. The reranking didn’t have any effect on calm and excitement.

/9

Average fraction of AAPA posts seen by participants for each day of the experiment.

While the effects are symmetric, it’s worth noting that we upranked a few APAA posts and downranked all AAPA posts in the corresponding conditions.

/8

Effects of reducing and increasing exposure to AAPA content in participants’ feeds on their feeling toward the out-party relative to the corresponding control group. (Left) Participants were surveyed within the feed during the intervention and (right) off-platform after the experiment. The feeling thermometer scale was between 0 (cold) and 100 (warm). The error bars represent 95% CIs.

In a field experiment with 1,256 consenting participants, we found that downranking AAPA posts leads to a decrease and upranking to an increase in affective polarization of 2 degrees on the 0-100 out-party feeling thermometer.

/7

There are many reasonable ways to define “polarizing content.” We focused on antidemocratic attitudes and partisan animosity (AAPA), drawing on the eight factors defined in the excellent study by Voelkel et al.

doi.org/10.1126/scie...

/6

In contrast to previous work that intervened at the level of users (e.g., downranking in-party content) or platform affordances (e.g., switching to a chronological feed), we intervened at the content level, exploiting recent advances in NLP.

/5

We used this method to test how reranking content that is likely to polarize affects participants’ affective polarization and emotions.

/4

The extension intercepts the user’s feed, uses an LLM to score and rerank the posts, and displays the updated feed.

Making this process fast took a lot of @tiziano.bsky.social’s engineering wizardry.

Source Code: github.com/StanfordHCI/...
Technical report: arxiv.org/abs/2406.19571

/3

Feed algorithms impact our lives, but until now only the platforms could run experiments testing the causal effects of different design choices. We propose a possible solution to this challenge by deploying a field experiment on X using a browser extension.

/2

Joint work with @tiziano.bsky.social, @jiachenyan.bsky.social, Jeff Hancock, Jeanne Tsai, @mbernst.bsky.social

Link: doi.org/10.1126/scie...

And a very thoughtful perspective by @jennyallen.bsky.social and @jatucker.bsky.social: doi.org/10.1126/scie...

/1

Today, social media platforms hold the sole power to study the effects of feed-ranking algorithms. We developed a platform-independent method that reranks participants’ feeds in real time and used this method to conduct a preregistered 10-day field experiment with 1256 participants on X during the 2024 US presidential campaign. Our experiment used a large language model to rerank posts that expressed antidemocratic attitudes and partisan animosity (AAPA). Decreasing or increasing AAPA exposure shifted out-party partisan animosity by more than 2 points on a 100-point feeling thermometer, with no detectable differences across party lines, providing causal evidence that exposure to AAPA content alters affective polarization. This work establishes a method to study feed algorithms without requiring platform cooperation, enabling independent evaluation of ranking interventions in naturalistic settings.

New paper in Science:

In a platform-independent field experiment, we show that reranking content expressing antidemocratic attitudes and partisan animosity in social media feeds alters affective polarization.

🧵