Eva Vivalt

That's a great question. We hope that soon there will be a few "sets" of related projects to forecast, via collaborations, which could let us look at this, but to date we have only had studies on similar topics be posted on the platform by chance, so I'd say there's not enough info quite yet.

Several excellent RAs worked on this project, most recently Kevin Didi, Malek Hassouneh, Rohan Jha, and Francis Priestland.

Kevin is a pre-doc of mine currently applying to PhD programs! He also assisted with the recent guaranteed income papers. He's great - watch out for him!

Forecasting Social Science: Evidence from 100 Projects Founded in 1920, the NBER is a private, non-profit, non-partisan organization dedicated to conducting economic research and to disseminating research findings among academics, public policy makers, an...

Together, these results show that forecast data contain real, exploitable information that can help decision-makers prioritize projects and treatment arms, benchmark expected effect sizes, and design better studies.

For more results, check out the paper! www.nber.org/papers/w34493

Again, one of the advantages of this paper is that we can track individuals. There is a lot of work on, e.g., the wisdom of crowds, perhaps in part because you can estimate effects of crowds without a panel. But getting the right people to forecast matters a lot, too. (Both are important!)

Our results also highlight that who is placing the forecasts really matters. We see a clear gap between academics and non-academics, but field and subfield expertise don't improve accuracy in a meaningful way.

Our panelists are great!

We have a paid forecaster panel that takes the majority of the surveys posted on the platform. And they do very well in comparison to other users.

In other words, controlling for forecaster fixed effects, we regain the result that confidence has a (weak) positive correlation with accuracy. But some people are much better forecasters than others, and those who place more accurate forecasts also tend to be more uncertain.

Interestingly, high self-reported confidence is associated with lower accuracy. This is dissimilar to most of the literature.

In our setting, we can track individual forecasters over time. And thus we can observe: this result is driven by overconfident forecasters.

If you are a funding body, policymaker, or researcher, you could benefit from collecting forecasts, but shrink their effect sizes by about 1/2.

First result: forecasters tend to overestimate treatment effects - but there is a lot of signal in the forecasts made.

This means that forecasts can be informative in power calculations or determining which interventions to trial.

Many of the forecasts are thus of *causal* phenomena, like "what effect will X have on Y?"

Super relevant for decision-makers trying to understand the potential impact of their actions.

For context, this paper is based on the most comprehensive set of forecasts of research results, to our knowledge. It uses data from the Social Science Prediction Platform, a platform researchers use to collect forecasts of what their studies will find.

🚨 New working paper!

How well do people predict the results of studies?

@sdellavi.bsky.social and I leverage data from the first 100 studies to have been posted on the SSPP, containing 1,482 key questions, on which over 50,000 forecasts were placed. Some surprising results below.... 🧵👇

It's really fun to be working in a cafe and overhear someone talking about basic income and your work!

An inspired choice! I was thinking there may be a prize for innovation this year, and it is so richly deserved.

Those are my three favorite new plots in the revised paper. Follow for more updates as we continue to put out results about this exciting program.

And here is the full paper: evavivalt.com/wp-content/u... 17/17

But you can check out results from different years in the paper. 16/

Most people pointing to the pandemic seem to think effects would be better afterwards.

If anything, during the pandemic the effects on labor supply were more muted. 15/

Also, some people have wondered about whether results were driven by the COVID-19 pandemic. We can't make conclusive statements here, but it's important to note the majority of the negative labor supply effects only materialized late. 14/

Importantly, if the transfer were of a shorter duration or if we had followed participants for a shorter period of time, we might have come to very different conclusions! 13/

As you can see, the results show a clear time trend, with impacts on employment growing over time until near the end of the program, when the gap starts to close. 12/

3) We previously included quarterly regression results, but we obtained some updated administrative data and made some nicer plots. For example, here is an event study plot looking at employment status. 11/

Yes, there are some negative impacts on labor supply and income excluding the transfers. People also do stuff with that money.

This second new figure helps illustrate the overall effects - and what doesn't move. 10/

2) Zooming out a bit, what can we say about the broader effects of cash transfers?

Some people have focused on the negative effects of cash transfers on labor supply. Others have focused on the consumption the transfers have enabled. 9/

In any case, improving people's subjective well-being over the long term is hard. This isn't necessarily a fault of the intervention - it could be something about human nature or subjective well-being measures. 8/

Many experts expected the transfers - which represent 40% of baseline household income - to make a bigger difference. If participants thought similarly, they might be disappointed.

Treated participants may face unexpected challenges as they make life changes. 7/

But there are other things that could explain it.

For example, it's possible that those receiving the transfers are not doing as well in years 2-3 as they expected. 6/

When I saw these results, my first reaction was that we have good data to be able to pick up this pattern. 5/

This might be surprising, but there is a large literature on "hedonic adaptation" and that could be part of what is happening here. People who win the lottery, for example, often revert to baseline levels of happiness.

Many positive shocks only temporarily improve well-being. 4/

Here, we measure subjective well-being in several different ways: a measure of life satisfaction, an index of satisfaction across several domains, and a measure of "affect balance" (SPANE).

All show the same trend. 3/