Shiwali Mohan

Can Generative AI Support Patients' & Caregivers' Informational Needs? Towards Task-Centric Evaluation Of AI Systems Generative AI systems such as ChatGPT and Claude are built upon language models that are typically evaluated for accuracy on curated benchmark datasets. Such evaluation paradigms measure predictive an...

Its a preliminary study but it shows how we can make #AI #ML evaluations more informative; beyond benchmarks curated with minimal insights about what a useful questions is and what an appropriate answer looks like. (8/8)

📖 Paper: arxiv.org/abs/2402.00234
🎥Talk: drive.google.com/file/d/1m79W...

24.03.2025 19:27 — 👍 0    🔁 0    💬 0    📌 0

🤖 Measured how #GenAI systems did; not only in terms of correctness but also how similar they were to an expert answering the same question. (7/8)

24.03.2025 19:27 — 👍 0    🔁 0    💬 1    📌 0

📥 Curated an evaluation question set from observed interactions. The set contains real questions asked by participants as they were attempting to do a specific task. Such datasets are critical to measuring if an #AI system is producing responses that are useful. (6/8)

24.03.2025 19:27 — 👍 0    🔁 0    💬 1    📌 0

🤕👩‍⚕️ Studied how people interact with the expert if they were available. This uncovered specific needs people have as they make sense of data and also, how an expert addresses those needs. (5/8)

24.03.2025 19:27 — 👍 0    🔁 0    💬 1    📌 0

🏥 Identified a specific usecase in which people need support from an expert but the expert is not easily accessible; understanding medical scans and reports in order to make good decisions about your treatment. (4/8)

24.03.2025 19:27 — 👍 0    🔁 0    💬 1    📌 0

In our most recent paper, we explore an evaluation approach for #GenAI #GenerativeAI systems. Here are the steps we followed - (3/8)

24.03.2025 19:27 — 👍 0    🔁 0    💬 1    📌 0

As a science, we have to adopt rigorous evaluations that identify what #IntelligentSystem #AIAgent #Agent behavior should be & measure if it works as intended. Move beyond a 𝘱𝘳𝘰𝘣𝘭𝘦𝘮-𝘢𝘨𝘯𝘰𝘴𝘵𝘪𝘤 metric (accuracy) on a 𝘵𝘢𝘴𝘬-𝘢𝘨𝘰𝘯𝘴𝘵𝘪𝘤 benchmark. Adopt practices from #HCI, #psychology, #economics. (2/8)

24.03.2025 19:27 — 👍 0    🔁 0    💬 1    📌 0

#AI #ML evals measure accuracy on benchmarks, telling us how algorithms compare with each other. But, not much about how an #IntelligentSystem should be built. How do we make evals more informative? (1/8)

📖 Paper: arxiv.org/abs/2402.00234
🎥Talk: drive.google.com/file/d/1m79W...

24.03.2025 19:27 — 👍 0    🔁 0    💬 1    📌 0

An architecture showing how a planning agent can be extended with metareasoning.

At #AAAI2025? Looking for #AI #ML research beyond #GenAI hype & doom? Excited about #AI running on your laptop? Listen to my colleague Wiktor Piotrowski talk about #OpenWorldLearning #OWL at 9:30 am on Feb 28th (Journal Track).

arxiv.org/abs/2306.06272
#AIPlanning #MBR #CognitiveSystems #KRR

26.02.2025 20:14 — 👍 1    🔁 0    💬 0    📌 0

Wait, are the AnthropicAI people seriously claiming to “unlock a rich theoretical landscape” for AI evaluation by proposing the use of…. error bars? And this secret trove of deep statistical insight starts with “use the Central Limit Theorem”?

Befuddling

27.11.2024 22:09 — 👍 98    🔁 16    💬 11    📌 1

When your experiments show that your #AI is more human than humans, it is not that you have built #AGI or #SuperIntelligence, it is that you don't know how to evaluate, experiment, and measure.

27.11.2024 21:28 — 👍 2    🔁 0    💬 0    📌 0