Mikyo

This is why going forward all AI features I help build will be natively instrumented with #OTEL. The telemetry data is the "fossil fuel" that feeds understanding and future improvement. AI cannot be treated as a black-box. It has to be inspected and understood.

Telemetry while testing and developing has been critical for me. It let's me hook into and inspect how systems like Vercel's AI SDK and LiteLLM work under the hood and figure out what prompts are being used for judgement.

Take evals. You might pick an eval and trust that it works. But this would be a mistake. It's rare that these evals will work for you across the board. Previously it would have been crazy to enable telemetry during testing. But with evals, you are going to want to inspect how your tests "operate".

Tracing and telemetry traditionally has been an operational requirement, not a development one. But I've found that with AI applications this fundamentally changes.

🐳
@arize.bsky.social OSS Prompt Playground
@arize-phoenix.bsky.social gets Deepseek support! Now you can compare outputs of all the top tier reasoning models.

Which LLM provider would you like to see next? Let us know on GitHub!

github.com/Arize-ai/pho...

👨‍🍳 @arize-phoenix.bsky.social continues to cook

Announcing OpenInference instrumentation for Agno, Mastra, Bedrock Agents, and AutoGen AgentChat!

At @arize.bsky.social we believe observability deserves to be built in the open

s/o @anthonypowell.me and many others

github.com/Arize-ai/ope...

🧪 📊 The @arize-phoenix.bsky.social TS/JS client now supports Experiments and Datasets!

You can now create datasets, run experiments, and attach evaluations to experiments using the Phoenix TS/JS client.

Shoutout to @anthonypowell.me and @mikeldking.bsky.social for the work here!

@arizeai/phoenix-client@1.3.0 -
@arize-phoenix.bsky.social javascript client gets experiments 🧪

s/o @anthonypowell.me !

- native tracing of ai tasks and evaluators,
- async concurrency queues
- support for any evaluator (e.g. bring your own evals) and more!

OpenTelemetry instrumentation for Agno is published! Huge s/o to Dirk Brand.

A true testament that AI observability should be built in the open 👏

@arize-phoenix.bsky.social

pypi.org/project/open...

annotating an llm call

📝Annotation Configs in @arize-phoenix.bsky.social

Part of the "Look at the Data" initiative, create custom rubrics and forms to annotate your spans.

s/o to @anthonypowell.me here who built out all the rich UI features.

9⃣ @arize-phoenix.bsky.social is gonna turn 9 today.

Project Retention Policies

Customize the data retention of your projects by number of days or by trace count. No more cron jobs or manual deleting of traces needed!

A much requested ask from our on-prem users and phoenix-cloud users alike.

Learn to prompt better

A speaker announcement card showing that Ben McHone is going to be presenting at Arize: Observe 2025 on June 25th, 2025.

I'll be speaking at Arize:Observe at SHACK15 on June 25! Looking forward to exploring what’s next for AI agents & assistants. More details on my session to come. @arize.bsky.social

arize.com/observe-2025

I still own plenty of pencils but no erasers. What does that say about me?

Just dropped a tutorial on using the OpenAI Agents SDK + @arize-phoenix.bsky.social to go from building to evaluating agents.

✔️ Trace agent decisions at every step
✔️ Offline and Online Evals using LLM as a Judge

If you're building agents, measuring them is essential.

Full vid and cookbook below

Text reads: Building AI? Demo your app. Arize:Observe community demos. Submit by 4.30.25. Apply.

Demo your app at this year's Observe! Fill out a short application by 4.30 to be considered for our Demo Den. Great opportunity to showcase your work to the AI community in SF.

Apply here: docs.google.com/forms/d/e/1F...

"The purpose of abstraction is not to be vague, but to create a new semantic level in which one can be absolutely precise." - Edsger W. Dijkstra. Just read this and I am going to be using it a LOT.

In case you missed it, Arize AI Phoenix crossed the 5k GitHub star mark last week! ⭐️

Phoenix has changed a TON since its first iteration.

I'm constantly in awe of the execution speed and quality of this team. Here's to the next 5k and beyond!

Love the community we're building!

LLM Evals Office Hours with Arize · Luma Join us for an open coworking session focused on LLM and Agent Evaluations! Whether you're actively working on evaluation strategies or just exploring the…

For all my NYC friends! 🗽🍎

We're hosting an in-person office hours tomorrow all around LLM and Agent Evals.

Join for the free snacks/drinks, stay for the heated discussions about the validity of Pokemon-based model evaluations ⚡️🐀

How much more data does an LLM app really need?

In my latest tutorial, I explore how few-shot prompting boosts accuracy without massive datasets or retraining—using @arize-phoenix.bsky.social prompts and experiments to break it down.

This kicks off my prompting series... more to come!

🤖 OpenAI 's agent framework openai-agents provides a rich set of composable primitives that enable you to build agents.

openinference-instrumentation-openai-agents, an OpenTelememetry instrumentor that is compatible with any OTel backend like @arize-phoenix.bsky.social. Fully OSS and free to use!

How can you programmatically improve your prompts? 🤔 🤖

Forget manual prompt engineering - there are better (read: "more automatic") ways to improve your prompts.

This video and notebook break down these techniques.

Featuring:
- DSPy
- @arize-phoenix.bsky.social

Prompt Management from First Principles In Phoenix 8.0, we built a prompt management system to ensure reproducibility and empower developers with better testing and control.

Learn how we built a holistic prompt management system that preserves developer freedom.

With Phoenix 8.0, we built a prompt management system that prioritizes: LLM reproducibility, prompt versioning & tracking, & developer flexibility—no vendor lock-in

arize.com/blog/prompt-...

AI is all about vibes lately

Trace & Evaluate your Agent with Arize Phoenix We’re on a journey to advance and democratize artificial intelligence through open source and open science.

huggingface.co/blog/smolage...

YouTube video by Arize AI Phoenix 8.0 - Prompts

With Phoenix 8.0 you can version & iterate on prompts seamlessly—both in the UI & in code! What's new:

🔗 TypeScript Client: Sync prompts with your JavaScript runtime
🐍 Python Client: Sync templates & apply them directly to AI SDKs
🔄 Native prompt normalization & much more!

youtu.be/qbeohWaRlsM

🤖 Building agents, but not sure how to measure their performance?

Our newest blog post on @hf.co has you covered!

This post shows you how to use @arize-phoenix.bsky.social to trace and evaluate your smolagents.

Credit to @srichavali.bsky.social and @aymeric-roucher.bsky.social

Your AI Agent isn't an Engineer Table of Contents Why This Conversation Matters How AI Marketing Shaped This Perception The Problem...

AI agents are just tools. They're not engineers. They're not people. While I get the value of describing AI agents as human, I think it's a tad lazy and it creates a self-sabotaging narrative for our industry.

dev.to/blackgirlbyt...

Learning Cursor and Vim at the same time was a bad idea. They philosophically clash. One is for vibe coding driven by AI instinct, the other is vibe coding driven by human instinct. Switching to Zed for a while to focus on building my own intuitions.