James M. Zumel Dumlao

Stack of papers and books

This paper explores academic peer review as a source of knowledge transfer and learning for the reviewers themselves.
spkl.io/63328A1crC
@innovation.bsky.social @jamesmzd.bsky.social

I think that would be an ideal experiment (just need $$ to run it)!! Thinking about how to do this in an ecologically valid way… especially since some folks are chiming in that they only read when reviewing 🤔

Learning vs. familiarity/awareness looks functionally similar under the definition of learning we’re using from thr Org Behavior lit. One suggestion from @mariomalicki.bsky.social is to look at *how* reviewers cite (e.g., substantive vs. rhetorical)

One explanation could be the 3yr citation window we set. Full paper has similar result with infinite citation window, but these papers were *reviewed* in 2018-19 (maybe published later). It’s possible this effect would be stronger with longer windows as time goes on…

We were partially motivated by a 2018 Publons survey where ~1/3 of respondents mentioned keeping up w their field as a reason for reviewing. To our knowledge, we’re the first to observe this behaviorally!!

Had a great time presenting these findings at #metascience2025 and looking forward to getting even more eyes on it at the #NBERSummerInstitute2025 session on Science of Science Funding (July 17-18)!

Catch in online July 17, 3:30pm at www.youtube.com/nbervideos

Implications

1. Highlighting personal learning benefits may help with reviewer recruitment

2. Peer review is an overlooked site of knowledge dissemination. Don't outsource evaluation labor to #LLMs?

3. Editors may take geographic/intellectual proximity into account when assigning reviewers

Additional Findings

1. "Learning effect" from peer reviewing is stronger for reviewers closer in geographic and intellectual distance (same-country and abstract cosine similarity)

2. Reviewing from a greater distance brings citation likelihood close to non-reviewers of a closer distance

‼️Main finding: Reviewers 2x likely to cite w/in 3 yrs‼️

Data: @instituteofphysics.bsky.social on 104k reviews in 55 STEM journals (2018-2019)

Method: Within-submission comparisons of invited reviewers who submitted review (treat) vs. declined due to unavailability (control)

Now available on @ssrn.bsky.social: papers.ssrn.com/sol3/papers....

Learning requires 1) exposure, 2) integration, 3) deployment. Peer review exposes reviewers to cutting-edge work. Critical assessment requires integration with existing knowledge. Citation in future work is evidence of deployment.

Title page of "Learning by Evaluating"

Probability of citing manuscript within 3 years, by review status. Reviewers are more than twice as likely to cite reviewed paper in future work compared to invited reviewers who declined due to unavailability.

🏮NEW WORKING PAPER ALERT🏮

Scientists collectively spend tens of millions of hours peer reviewing each year, mostly for no $$. Why??

In new work w/ Charles Ayoubi and @innovation.bsky.social, we observe a private benefit for participating in evaluation: 💡💡Learning💡💡

papers.ssrn.com/sol3/papers....

@charles @jamesmzd.bsky.social

Why do we peer review?

In a new paper, we establish a self-interested reason: learning.

With a quasi-experimental design applied to admin data from 55 journals, we show reviewing a paper doubles chances of citing it in future work!

1. Another Friday afternoon massacre of sorts:

NIH just rescinded its Scientific Integrity Policy.

h/t @lizborkowski.bsky.social

My Name is Mahmoud Khalil and I Am A Political Prisoner A letter dictated by Khalil over the phone from ICE detention in Louisiana.

In a letter dictated over the phone from ICE detention in Louisiana, Mahmoud Khalil said that his arrest "was a direct consequence of exercising my right to free speech as I advocated for a free Palestine."

"It is our moral imperative to persist in the struggle for their complete freedom."

Screenshot of top half of first page of paper. The paper is titled: "When People are Floods: Analyzing Dehumanizing Metaphors in Immigration Discourse with Large Language Models". The authors are Julia Mendelsohn (University of Chicago) and Ceren Budak (University of Michigan). The top right corner contains a visual showing the sentence "They want immigrants to pour into and infest this country". The caption says: Figure 1: Dehumanizing sentence likening immigrants to the source domain concepts of Water and Vermin via the words "pour" and "infest". The abstract text on the left reads: Metaphor, discussing one concept in terms of another, is abundant in politics and can shape how people understand important issues. We develop a computational approach to measure metaphorical language, focusing on immigration discourse on social media. Grounded in qualitative social science research, we identify seven concepts evoked in immigration discourse (e.g. "water" or "vermin"). We propose and evaluate a novel technique that leverages both word-level and document-level signals to measure metaphor with respect to these concepts. We then study the relationship between metaphor, political ideology, and user engagement in 400K US tweets about immigration. While conservatives tend to use dehumanizing metaphors more than liberals, this effect varies widely across concepts. Moreover, creature-related metaphor is associated with more retweets, especially for liberal authors. Our work highlights the potential for computational methods to complement qualitative approaches in understanding subtle and implicit language in political discourse.

New preprint!
Metaphors shape how people understand politics, but measuring them (& their real-world effects) is hard.

We develop a new method to measure metaphor & use it to study dehumanizing metaphor in 400K immigration tweets Link: bit.ly/4i3PGm3

#NLP #NLProc #polisky #polcom #compsocialsci
🐦🐦

I’m submitting to a student conference with a deadline on xmas eve…

On the last flight on a Sunday night LAX to DTL got me like open.spotify.com/track/1qiJlt...

I’m officially an STSer!! Very excited to continue engaging w the work here at #4SHonolulu and creating/bridging convos w/in #ScienceofScience

They’re storming the beach

Brandon nails it: Karikó's story and the Nobel Prize for her incredible science highlight the flawed foundation of academia's claim to meritocracy. It's an ideal that we can and should aspire to! But we have a lot of hard introspection and institutional work to do to get there...

HELMET CATCH #GOBLUE

Academic productivity is so mysterious to me. Everything I do is angled to get a few high quality working hours each day (or even each week). Hitting that flow state feels unpredictable (could be 9am or 9pm). But once it hits, just 2-4 hours can make the whole day worth it

Wifi was down on campus for 4 days and I was ready to embrace tradition, return to monke, etc. No need! Modernity is back 🤠

Come watch the moonrise w us later?

“Hello darkness my old friend” -me first night back in Munger

Flying back to Ann Arbor today after 3.5 months in the Bay. It was a joy and a pleasure but I’m ready to get back to the grind 😤

It’s been a long road with GEO to come to a tentative agreement, but I’m so proud to vote YES to ratify this historic contract. Our wins will change the lives of thousands of current and future grad workers. RATIFICATION IMMINENT!