Its possible big discoveries and important steps lie more in the unpredictable combinations than the predicable ones. Collective behavior is one example (computer graphics techniques + anima behavior questions)
It’s more the difficulty of altering dynamics without degrading functionality. Collective behavior is so fickle and fragile it’s quite strange science works at all, in some ways.
For example, how does noisy and finite but diverse individual reads of the literature differ from all getting summaries
We wrote about the general idea here. From what we know about collective behavior, it’s highly unlikely that altering it for short term profits is going to produce desirable outcomes at scale.
www.pnas.org/doi/10.1073/...
I think the termite analogy gets us to nearly the opposite conclusion. Animal collective behavior often relies on highly selected rules for interaction, that emergently produce functionality. Altering that with robot termites that don’t follow those rules would almost certainly collapse functioning.
I am going to need every single person who said that misinformation was a moral panic to feel enough shame and regret that they leave public intellectual life.
Related point, if you are in misinformation research and have been launching study after study to see what Uncle Fred thinks after seeing misinformation PLEASE listen to what @katestarbird.bsky.social and I have been saying for years: effects on elite and official action is what matters most.
I suspect it is! There’s been some work on ai companies and research, but I think largely focused on ai research rather than its consequences.
I do think the field will turn up messy results unless we start getting cois straightened out.
I'm with @devezer.bsky.social, you *absolutely* cannot test for p-hacking.
These tests universally assume p-hacking, and nothing else under the sun, explain deviations from idealized distributions and then claim to have found it.
Fantastic thread.
For example if someone wants to tell me about space travel, I should treat that information differently if they are a friend I trust who has a PhD in astrophysics, versus some random person on the internet, versus a psychotic billionaire trying to sell a space travel company
9/
If the sign up bonus is good, I’d even give it a depression survey and plot the results over time.
Stupidest job ever, but like… I’d take it.
For $400k a year, I’d ask Claude how it’s doing each day and send Dario Amodei a thumbs up emoji on slack.
AI makes continuous reproducibility and robustness testing trivial. What happens to science under new levels of scrutiny and stress-testing by default?
Some thoughts on how this could play out, informed by watching open science play out over the last decade.
Good piece. Carl and I reached similar conclusions about ai in peer review. It’s good to see the hype tamed with what it is we actually do when we’re doing science.
www.nature.com/articles/d41...
My experience with trying to get LLMs to write statistical models is that they will happily and silently bury an implausible assumption about the data generating process in an unobjectionable, conventional structure.
Of course we shouldn’t condition our model choices on the inferences we draw, but the models form is the questions they ask—based on what we know, assume, value etc..
I truly don’t see how that can be automated, it’s where our knowledge and data actually meet.
It’s as if when we choose the terms in a model or its form we’re guessing and somewhere out there is the right answer, the biggest threat to getting it wrong is that we might choose the model based on our subjectivity.
So we want to hedge across options, or ask a machine to do it.
More seriously, a lot of the LLMs in analysis discourse seems to be stuck in the same sort of thinking that motivated avetsging full specification curves of every possible model, or many analysts studies.
Less seriously, I’m gonna be big mad when the EC2 instance checking my reproducibility OOMs and rejects submission.
Many are appropriately outraged by Altman’s comments here implying that raising a human child is akin to “training” an AI model.
This is part of a broader pattern where AI industry leaders use language that collapses the boundary between human and machine.
🧵/
"There are deeper strata of truth…there is such a thing as poetic, ecstatic truth. It is mysterious and elusive, and can be reached only through fabrication and imagination and stylization." - Werner Herzog
Frankly if you can’t figure out how to use a proper environment for your code it raises deeper questions about the analysis.
Definitely inspired by it. I don’t understand conceptually adding something non reproducible to the problem of reproducibility when it’s a small infrastructure investment to ensure code compiles the way we do with latex.
Most code I’ve struggled to reproduce could be addressed with a little template repository, even just using package management.
Containers make it all easier, but it’s weird how rarely someone makes it clear which version of (say) R they used.
I’ve lost track of downloading someone’s R code and spending three hours figuring out what version of which packages don’t cause things to break, while replacing hardcoded paths everywhere.
I think the benefits sort of pay for themselves in terms of avoiding dependency hell when writing code. There is a learning curve but not particularly steep re: eveything else in science. For example, even using like pip freeze > requirements.txt and relative paths would go a long way.
My old boss and current boss happened to meet halfway across the world today, and im truly hoping they took the piss out of me.
“Have the agentic robot do it”
Vs.
“Use nix, GitHub, docker, or even just decent dependency management”
People excited about computational reproducibility with LLMs are gonna lose their minds when the llm tells them that it’s largely a solved problem with dozens of suitable tools and it just requires knowing basic software development.
There's also plenty of complexity inherent to physics (the beast of social systems) but like lots of complex systems, you get emergent forms of simplicity amenable to models. I'm not sure we've done a good job integrating theory and empirics because of the fallout over homo-economicus.
