Faithfulness vs. Safety: Evaluating LLM Behavior Under Counterfactual Medical Evidence In high-stakes domains like medicine, it may be generally desirable for models to faithfully adhere to the context provided. But what happens if the context does not align with model priors or safety ...

📎Paper: arxiv.org/abs/2601.11886
🧑‍💻Code/data: github.com/KaijieMo-kj/...

w/
@kaijie-mo.bsky.social @sidvenkatayogi.bsky.social
@chantalsh.bsky.social @ramezkouzy.bsky.social
@cocoweixu.bsky.social @byron.bsky.social @jessyjli.bsky.social

21.01.2026 19:07 — 👍 1    🔁 1    💬 0    📌 0

Results (3/4)
– With evidence, models strongly adhere to it with high confidence, even for toxic or nonsensical interventions
– Implausibility awareness is transient; once evidence appears, models rarely flag problems
– Scaling, medical fine-tuning, and skeptical prompting offer little protection

21.01.2026 18:50 — 👍 1    🔁 0    💬 0    📌 0

Setup (2/4)
We introduce MedCounterFact, a counterfactual medical QA dataset built on RCT-based evidence synthesis.
– Replace real interventions in evidence with nonce, mismatched medical, non-medical, or toxic terms
– Evaluate 9 frontier LLMs under evidence-grounded prompts

21.01.2026 18:47 — 👍 1    🔁 0    💬 0    📌 0

Hello world 👋
My first paper at UT Austin!

We ask: what happens when medical “evidence” fed into an LLM is wrong? Should your AI stay faithful, or should it play it safe when the evidence is harmful?

We show that frontier LLMs accept counterfactual medical evidence at face value.🧵

21.01.2026 18:45 — 👍 14    🔁 6    💬 3    📌 2