Alejandro Schuler

🚨A Researcher's Guide to Empirical Risk Minimization

I put together a guide on regret theory for empirical risk minimization (ERM) as I understand it.

The goal was to compile results and proof techniques I’ve found useful in my own work. I hope people find it useful more broadly

it took an expensive, coordinated effort & a lot of people saying “this doesn’t matter” to get from a bathroom ban costing one state $400 million & a gubernatorial race to “trans people must turn in their documents which become invalid in a few hours or face fines & potential jail time” in 10 years.

Losers

another gem: www.jstor.org/stable/pdf/3...

i feel seen

I, too, have been accused of this.

I'm more upset about how bad we are at training statisticians to engage with domain researchers than I am that domain researchers sometimes still ask bad questions.

most statisticians are bad scientists too!

(academic statisticians are bored by it; for applied statisticians maybe not but they often don't advocate for themselves or are happy turning the crank on power calculations)

However, the blame is just as much with the statisticians imo. The most useful role of the statistician (gut-checking utility of the question and doing a good job formalizing the question) is exactly the part that is not taught and that most statisticians are bored by.

the frustrating thing is the power differential: clinicans are better at finding good questions (though remarkably not as good as one would think) but that's not enough to justify how much control they have over funding, etc

definitely. What's remarkable (to me) is the extent to which clinicians, epidemiologists, psychologists, etc. etc. have the capacity to still ask bad questions despite all that training. It often has to do with bandwagoning on a particular "format" of paper and ease of data collection/access

I'm mostly talking about clinicians when I say "domain scientists", but maybe that's too broad a use of the term

Unfortunately, statisticians spend 95% of their training learning about analysis, maybe 5% on formalization if they're lucky.

Training for domain scientists seems to be 50% grant salesmanship (or who knows), 40% finding good questions, 10% analysis (what's an estimand?). I might be generous here.

The majority of failed research does not pass the *first* bar. I assume we agree that the problems are roughly 65% bad question, 30% bad formalization, 5% bad analysis. And even with bad formalization and bad analysis, you can often still get a qualitatively useful answer to a good question.

3. Was the analysis appropriate? Is the estimator consistent, efficient; multiple testing controlled for, etc.

2. Has the question been formalized sensibly? Is there a nonparametrically defined estimand, identification, sensible theoretical population, etc

1. Is the question good? Is it policy-relevant in any realistic context? I.e. given any natural-language answer, would the world change at all?

In my consulting class I like to say that research has three successive bars to pass:

Can anyone fix science? Science has always been in crisis. This is fine.

Part one of a new blog series: using the discovery of vitamins as a parable for why replication crises in science are actually good.

like, idk, how much goes to sports at public unis? (which I think is mostly fine btw) Or stadiums that cities end up paying for and losing money on? I guess you could argue those give immediate entertainment value whereas for science you gotta wait for museum translation haha

Generally that's the trend for sure (compare eg NIH to NSF funding) but now I'm trying to think of all the multi-million dollar counterexamples where the public supports funding ostensibly useless things...

If forced to rationalize it I'm sure people will tell you space has something to do with the economy and curing moon cancer or whatever but tbh people just think space is cool and that's a totally legit reason to fund NASA that should be considered against tradeoffs.

NASA's budget is 100x the national endowment for the arts, but does NASA deliver 100x the instrumental value of the NEA? Honestly not sure but what I'm more sure of is that the funding disparity is more about the vibe than the return on investment.

The reason the community is driven to do something can be different than the reason others allow or encourage it, in which case there is no one "purpose". It's contextual and fluid.

Yeah I agree with that too :)

Why have art of any kind?

YouTube video by Santi Abad Why do we read and write poetry? (Dead Poets Society)

youtu.be/aS1esgRV4Rc?...

why should there be? Why is a thriving community of learners not inherently more valuable than... ? Social value is social. If we agree it is valuable, it is valuable. For a long time, a lot of people agreed this was valuable.