MartinDotNet

#O11yDayNYC 2026: Big thinkers. Big ideas. Big conversations.

Founders and engineering leaders are coming together to explore where observability is headed in an AI-driven world and how engineers can evolve with it.

Join us in NYC 👉 buff.ly/fnIwrcZ

lol, I'm less concerned about it messing up my local machine than I am about it using my access to AWS to tank the entire company.

The meme from XKCD that used to refers to "sudo", but instead of "sudo", it uses "bypass permissions on" with a reference to Claude code.

The context is that I want to use KMS keys tied to a Cognito identity to provide an element of security that means that even AWS admins don't have access to the underlying data.

Does anyone know if it's possible to have a lambda execute in a Cognito identity context when the originator is an SQS/EventBridge async execution from a Lambda invoked in API Gateway

APIGW => Lambda (cognito) => SQS => Lambda (cognito)

@theburningmonk.com maybe?

I've spent the last few months building this event, from working with @aparker.io on the keynote about how Knowledge is the new Data to pulling in some amazing industry speakers.

The best thing though, is that I got sign-off to show a sneak peak of the future of honeycomb.

Not to be missed!

#o11y

Hmm, interesting.

If it was a human who wrote the code, and you approved them pushing it (signing it off), would that be the same, even if you gave them the instruction on what to build, but didn't review the code.

When working with an Agent that writes the code for you, would it be better to having the author in git be the agent and the "signed-off-by" be the user?

That seems more accurate to whats going on.

If you yolo it into git and push, there shouldn't be a signed off, showing that noone approved it

100% thats a valid thing to do.

Modelling of telemetry is intensely contextual thing. Breaking a trace can definitely be the right idea.

I would recommend adding a span link to the new span so there is a casual link between for navigation/attribution. That is dependent on your backend though.

I'm happy to be partnering with the @honeycomb.io team to share this, because Martin Thwaites’ recent take on why "AI isn't here to replace your dashboards yet" actually changed my mind about the future of the tools I want to use.

At #o11yDaySFO, Kesha Mykhailov shared how the Intercom team rethought observability for their AI agent, Fin, starting from the customer’s perspective.

In the age of AI, the takeaway is simple: empathy is the most powerful tool we have.

Learn more: buff.ly/MS4gVIU

#HoneycombIntelligence #AI

This quote from Martin Thwaites at #O11yDayLondon is one we still find ourselves coming back to. A great reminder of how powerful curiosity and good questions are in observability.

#honeycomb #observability

Honestly, Claude+MCP has been doing wonders with finding those with little effort.

Thats basically my point to be honest.

If what you want is a different way to use the APIs from your cloud provider, then fine. If you want to use IaC, you need be able to destroy it and bring it back.

PSA: Always try destroying your IaC stack before you rely on it.

That's more a legacy prometheus thing really, you should be running these through a collector pipeline that will do the merging. That's the modern way to control things and is beyond SaaS vs Self-hosted.

I'm not sure thats limited to SaaS. When you host it yourself, you then sacrifice performance (or Infrastructure cost).

The reason its normally seen as a SaaS problem is that the cost is more directly visible, since the cloud provider needs to maintain consistent performance.

I prefer "AI Augmented".

I see it as similar to mob programming, where you're the navigator and its important that everyone understands whats going on.

Vibe software developing is more accurate to what people think will replace developers.

I'm pretty sure a simple prompt for Claude will build the shim for you.

This is down to the fact that we push people into the pit of success, and writing to files isn't the way that gets you there.

Theres been extensive debate on this, with valid arguments on both sides. The decision was that you should sidecar a basic collector to offload the processing.

Have you had a look at replacing your Sentry with our HFO tool? I'd be happy to walk you through it in the new year.

I would tackle this from the backend, as long as the backend should never have propagation. You can set the textMapPropagator to have logic on when it can accept propagation.

That is absolutely a thing FWIW.

What language are you using? There are ways to stop propagation from the backend.

Arc Raiders is going to go down as a seminal moment in gaming.

Its not the best looking, it has bugs (annoying ones), it has a cheating problem, it has a growing toxic element

That said, the moments of pure joy (and terror) that come from proximity comms and terrifying enemies, is just exceptional

I suppose you work with the tools you have, and if all you have is dashboards you’ll revert back to those from an alert

I'm probably little spoilt though since our customers don't do that flow, but I can see how the older platforms lean into that more.

Like I said, o11y isnt metrics and dashboards

I think thats true for the legacy observability backends, but when you look at modern ones, they are replaced with going from granular SLOs into querying the underlying data, rather than precanned dashboards

Dashboards for monitoring are still a thing, but even then, most teams replace them with more reports and SLOs.

Interesting contradiction in that first part, staring at dashboards isn’t o11y, o11y isn't actually any visualisation.

Its precisely because of that, that causes the confusion. O11y is just a measure of how well you know the state of your system from the outputs, thats rarely a dashboard.

Hear from Kesha Mykhailov on how Intercom’s Fin.ai customer support agent leveraged human-centric observability to improve resolution rates and the customer experience, and see how Honeycomb helped spot issues along the way.

Learn more 👉 buff.ly/k4IWxE1

#honeycomb #observability #o11yDaySFO #AI

There was a pretty bad bug in the metrics portion, the latest has apprently fixed it.

That said, don't generate metrics at the application level, its wasteful. Do it in the collector from your trace data, or aggregate the data from traces in the backend.