Greg

MCP Colors: Systematically deal with prompt injection risk

MCP Colors

A riff off of the lethal trifecta for addressing prompt injection, this is a simple heuristic to ensure security at runtime

red = untrusted content
blue = potentially critical actions

An agent can't be allowed to do both

timkellogg.me/blog/2025/11...

Yes please !

my take, after reading replies all day:

we’re still early. people aren’t spending much money on AI so it’s not a lucrative target yet

it’s also inconsistent, which is annoying to design attacks for, especially if the rewards are sparse

claude_code_docs_map.md Something I'm enjoying about Claude Code is that any time you ask it questions about itself it runs tool calls like these: In this case I'd asked it about its …

It's neat how if you ask Claude Code questions about itself it can answer them, because it knows how to fetch a Markdown index of its own online documentation and then navigate to the right place

I wish more LLM tools would implement the same pattern! simonwillison.net/2025/Oct/24/...

An Opinionated Guide to Using AI Right Now What AI to use in late 2025

I wrote an updated guide on which AIs to use right now, & some tips on how to use them (and how to avoid falling into some common traps)

A lot has changed since I last wrote a guide like this in the spring, and AI has gotten much more useful as a result. open.substack.com/pub/oneusefu...

clem @ClementDelangue Am I wrong in sensing a paradigm shift in Al? Feels like we're moving from a world obsessed with generalist LLM APls to one where more and more companies are training, optimizing, and running their own models built on open source (especially smaller, specialized ones) Some validating signs just in the past few weeks: - @karpathy released nanochat to train models in just a few lines of code - @thinkymachines launched a fine-tuning product - rising popularity of @vllm_project, @sgl_project, @PrimeIntellect, Loras, trl,... - 1M new repos on HF in the past 90 days (including the first open-source LLMs from @OpenAI) And now, @nvidia just announced DGX Spark, powerful enough for everyone to fine-tune their own models at home.

correct

i’ve been saying this for a couple months. RL is driving towards specialization

my hunch is it’s temporary and something will shift again back towards generalization, but for now.. buckle up!

Sora 2 Watermark Removers Flood the Web Bypassing Sora 2's rudimentary safety features is easy and experts worry it'll lead to a new era of scams and disinformation.

I think people are still unprepared for a world where you cannot trust any video content, despite years of warning.

Even when Google & OpenAI include watermarks, those can be easily removed, and open weights AI video models without guardrails are coming. www.404media.co/sora-2-water...

I think I have at least one conversation (or argument?) about this every couple of days!

sheesh! AI bluesky has arrived

not just good content, there’s more and more original work, people from labs, and people with genuinely interesting perspectives

when i joined, it was so painful trying to find even traces

Codex in the app is going to open the door to real vibe coding on the go — no computer required. I’m so excited for this to expand, the ceiling is so high and this is the worst the models will ever be.

The key detail people may miss: it looks like an AI company in the USA can train on an author's book by purchasing a used copy, cutting it up and scanning the pages - in which case the author gets no money at all!

I feel seen 👀 Although it’s improved since I use Cursor and have it generate my commit messages!

This is illustrative of common AI issues:
1) When you get an instant AI answer, it is from a small model, which are weak models, especially at math.
2) Non-reasoning models, like the one powering AI overview, only “think” as they write, they make mistakes & then back justify them as they write more

The chart presents the decomposition of Average Treatment Effect (ATE) on cosine similarity into two components: Model Effect (red) and Prompting Effect (blue). • Y-axis: Δ Cosine Similarity (change in similarity). • X-axis: The source of prompts (top labels) and the replay model used (bottom labels). • Points and error bars: Represent mean effects with 95% confidence intervals, bootstrapped and clustered by participant. Breakdown: 1. DALL-E 2 → DALL-E 2 (baseline): Δ Cosine Similarity is ~0, establishing the reference point. 2. DALL-E 2 prompts replayed on DALL-E 3: Shows a Model Effect (increase ~0.007–0.008). This isolates the improvement attributable to the newer model when given the same prompts. 3. DALL-E 3 prompts replayed on DALL-E 3 vs DALL-E 2 prompts on DALL-E 3: The additional boost is attributed to the Prompting Effect (~0.006–0.007). 4. Total ATE (black bracket): When prompts written for DALL-E 3 are used on DALL-E 3, the improvement in cosine similarity reaches ~0.016–0.018. 5. DALL-E 3 prompts replayed on DALL-E 2: Effect is small, close to baseline, showing the limited benefit of improved prompts without the newer model. Summary (from caption): • ATE (black) = Model Effect (red) + Prompting Effect (blue). • Model upgrades (DALL-E 3 vs DALL-E 2) and better prompt designs both contribute to improved performance. • Prompting alone offers some gains, but most improvements come from model advancements.

As LLMs Improve, People Adapt Their Prompts

a study shows that a lot of the real world performance gains that people see are actually because people learn how to use the model better

arxiv.org/abs/2407.14333

When you use coding agents for something that produces the bulk of the code of an application that will be used for years, also factor in the technical debt that you are happily accumulating. When you use LLMs an an aid, you could, on the contrary, improve your coding culture.

Looking at the ARC-AGI benchmark is a useful way of understanding AI progress.

There are two goals in AI, minimize cost (which is also roughly environmental impact of use) & maximize ability. It is clear you can win one goal by losing the other, GPT-5 seems to be a gain on both.

Pelican Riding Bike This is the cat! He's got big wings and a happy tail. He loves to ride his bike! Bike lights are shining bright. He's got a shiny top, too! He's ready for adventure!

The new Gemma 3 270M open weights model from Google is really fun - it's absolutely tiny, just a 241MB download

I asked it for an SVG of a pelican riding a bicycle and it wrote me a delightful little poem instead

simonwillison.net/2025/Aug/14/...

roon y @tszzl model switcher paradigm will be vindicated in the long run. there is a high switching cost into a very new UX on a useful product, but it's the right move

i think i agree. i also think it’ll be a moat against other labs that don’t a strong product

it doesn’t make sense if you don’t have a strong UI, plus it’s obviously hard

Same! Not as wild as I imagined !

As a PM, I really can’t wrap my head around this either 🤔

You are likely going to see a lot of very varied results posted online from GPT-5 because it is actually multiple models, some of which are very good and some of which are meh.

Since the underlying model selection isn’t transparent, expect confusion.

I had access to GPT-5. I think it is a very big deal as it is very smart & just does stuff for you

This is “make a procedural brutalist building creator where i can drag and edit buildings in cool ways" & "make it better" a bunch. I touched no code

Full post: www.oneusefulthing.org/p/gpt-5-it-j...

What could be a two step master plan:
1. Release open model to commoditize much of the model market a tad off the frontier
2. Release GPT 5 as the only model worth paying for

Really curious about all the strategy decisions that went into this open model (& GPT5/Gemini 3)

Really looking forward to see how it performs with tools!

I prompted Gemini 2.5 Deep Think: "create a missile command game that incorporates relativity in realistic ways but is still playable." I then asked it to improve the design a few times.

The full design & all code & calculations came from AI, no errors. Try it: glittery-raindrop-318339.netlify.app

A line chart with vibe on the Y axis and understanding on the X axis with a downwards diagonal line

A helpful graph made by @stevekrouse.com on the inverse relationship between vibes and understanding in AI assisted code.

Put a few thoughts down here: maggieappleton.com/2025-08-vibe...

Original article: blog.val.town/vibe-code

The best available open weight LLMs now come from China Something that has become undeniable this month is that the best available open weight models now come from the Chinese AI labs. I continue to have a lot of love …

July has been a truly incredible month for model releases from China - Moonshot (Kimi K2), Z ai (GLM-4.5) and 5 new releases from Qwen

I think it's undeniable that the best available open weight models now come from the Chinese AI labs simonwillison.net/2025/Jul/30/...

Latest open artifacts (#12): Chinese models continue to dominate throughout the summer 🦦 Artifacts Log 12.

Having covered the open models coming out in the last 2-3 years of the AI boom, it’s safe to say that we’re really starting to reach maturity. There are very serious AI models across many modalities that anyone can download and start to make products with.
buff.ly/lIU1ftr

Don't leave AI to the STEM folks.

They are often far worse at getting AI to do stuff than those with a liberal arts or social science bent. LLMs are built from the vast corpus human expression, and knowing the history & obscure corners of human works lets you do far more with AI & get its limits.