Adam Rodman

Screenshot of OpenEvidence investment prospectus, noting revenue of $50,000,000 with valuation of $6,100,000,000.

OpenEvidence has revenue of $50 million.

But investors value it at $6.1 BILLION.

So… a couple of questions for the “AI won’t replace doctors!” crowd:

What do you think OE’s long-term monetization pathway looks like?

And what do investors expect to happen that could justify this valuation?

As I'm writing this, I realize I'm probably not making anyone feel better😀

TL;DR: I largely agree with your premise, but it's going to take longer, and have much more slow and incremental change (and then fast and epistemic change) than the tech bros think

... since most of the proposed regulatory systems rely on tech that doesn't really exist yet.

There's also the regulatory angle. I think that SaMD absolutists are delusional (and probably not talking to anyone in the government), but companies are going to need some sort of regulatory structure in order to make investments in this space. That's a problem (right now) too...

... but I see no signs that OE is doing that, and I don't think that's happening in the short term (which, I mean, is like 5 years)

I think it's much more likely that OE wants to become the indispensable tool for doctors, and leverage the massive amount of data that they collect on us.

I also think that means that eventually, an OE API might become part of an evidence retrieval agent for a semi-autonomous HITL->HOTL system...

But that's the *easy* part. Building out workflows, developing and testing oversight systems, and then deploying and iterating -- that is a MUCH larger challenge. If OE wants to do that, they've made no signs in their hiring, or any acquisitions.

... for many tasks at least.

OE has built an industry standard for an incredibly important part of that (knowledge retrieval), and their moat is now their content agreements, as well as the RL from all of their users.

Maybe I can make you feel better (in the short run at least).

Replacing doctors (or rather creating agentic workflows that allow for moving from HITL to HOTL) is both an immense technological and oversight system challenge (though I believe that it's fully doable).

All of these studies have limitations of course -- but the evidence suggests that LLMs can absolutely collect information in meaningful ways directly from patients at least within certain confines (especially telehealth-style urgent care).

Similarly, the K health studies again show LLMs (though the K-health system uses multiple types of tech) are capable of collecting patient histories (in urgent care)

LLMs work well at eliciting information (from patient actors, compared to a blinded human, via a chat interface). The study the OP is referring to is CRAFT-MD, which uses MedQA questions and two LLMs role-playing a human and a patient.

Clinical trial data coming soon.

I don't think being anti-AI is the solution -- just to be thoughtful about it. EG, me and my resident made a neurological diagnosis (we are not neurologists) by using GPT-5 to help guide us on the bedside neurological exam. Lord knows I'm not usually breaking out Hoffman's sign

Kind words from @medcrisis.bsky.social (and definitely not a techbro -- still use my wood carved stethoscope!). Thread expounds more on my quote in the article: "if we don't figure this out, we're screwed."

So maybe 15 years from now none of this will matter (though I'm skeptical) -- but in the meantime, I think I stand by what I told Dhruv in that story, if we (meaning educators) don't figure out how to train the next gen in this environment, we're all screwed.

(plus, it's not like there's going to be an "AI doctor" -- there will have to be brand new workflows, safety systems, and regulatory environments)

in the near future (reward systems favor classification, though of course we all know that for many important diagnoses we never get anything close to a definitive answer).

But management reasoning at a physician level is probably beyond that these systems can reasonably do

This probably WOULDN'T matter too much if this current generation of tech would uniformly outperform doctors.

But it won't. I'm actually much more optimistic than @ericstrong.bsky.social that diagnostic reasoning will largely be done better than LLMs for a variety of diagnoses (not just esoterica)

We don't have a LOT of evidence yet about what overreliance on the tools does to our diagnostic (and management) reasoning abilities, but there's plenty of reason from other domains (and our psychological understanding of learning) that it degrades them.

My biggest concern about this technology in the short (and, well, medium)-term is what is does to US, especially the current generation of doctors in training. Even the difference between my third year and first-year residents is pretty stark in their use of AI tools (especially Open Evidence)

AI GRAND ROUNDS EPISODE 35 Medicine, Machines, and Magic: Dr. Jonathan Chen on Medical AI

In the latest episode of the NEJM AI Grand Rounds podcast, Dr. Jonathan Chen discusses his path from teenage programmer to Stanford physician-informatician and why machine learning has both thrilled and unnerved him. Listen to the full episode: nejm.ai/ep35

#AI #MedSky

I think another big difference in the ICU is that line volume is also much less than it was in the past. So also fewer opportunities.

I was going to say, as one of our proceduralist hospitalists, this problem stretches back to the 2010s or even 2000s. Back when I was an intern (2013), we were gnashing out teeth about it.

Congrats! They're lucky to have you!

A painted scene where a skeleton holds up a round bottomed glass flask to a man in a long robe and a white cap.

Another one from the Bleibach Totentanz - or Dance of Death. This scene shows death coming for the physician - and you can tell it’s a physician because of the urine flask, which used to be their primary diagnostic tool. It was believed that the colour of the urine would indicate the disease

Four images to illustrate some prominent single-gene myths. Top left shows a photograph of a person deftly rolling their tongue into a U-shape. Top right shows a photograph of a person’s ear, highlighting the shape and features of the earlobe and cartilage. Bottom left shows a close-up photograph of a person’s eye, with a vivid blue colouration. Bottom right shows a photograph of a person poised to write with their left hand on the blank white page of a spiral-bound notebook.

Remember when you first learned about genetics at school? All those fascinating examples of human traits that are each apparently determined by just a single gene? Time to check in on some of your favourites to see how they’re doing. 🧬🧵🧪 1/n

I saw this! There's still quite a gap between hopes of improving access and the realities of the tech, and probably more importantly what sort of regulatory system this would live in. But this is no longer science fiction to speculate.

2 important papers @nature.com today indicate the potential for A.I. to improve diagnostic accuracy and overall clinical performance for patient management and interactions
nature.com/articles/s41...

Page 1 of the editorial "When It Comes to Benchmarks, Humans Are the Only Way"

Editorial by @adamrodmanmd.bsky.social and colleagues: When It Comes to Benchmarks, Humans Are the Only Way nejm.ai/4hMrAM9

#AI #MedSky #MLSky

IMO this is the best argument for robust benchmarks. We have no way of studying new models other than as if they were natural phenomena. Clinical trials are incredibly important -- but with this rate of change so is benchmarking for the foreseeable future.