We're publishing our streaming classifier openly — the methodology, weights, and a full tutorial. This is a problem the whole T&S community needs to solve, so we're eager to see others build upon our technique. Full details on our blog:
blog.zentropi.ai/enabling-streaming-classification/
Today, we are releasing a classifier that can score content as it streams, token by token. It can flag that a violation is developing partway through a sequence — early enough to actually do something about it. Interrupt generation. Route to review. Log a warning. Things you can't do with post-hoc.
The idea is simple: if you're already running content through a classifier, the model is already building internal representations at every token. Those representations already encode whether a violation is developing — you just have to ask.
We trained a tiny linear probe to do the asking.
There's a major gap in content safety tooling: classifiers typically only score complete text. When you're working with generative AI, "complete text" means the user already saw it. That's too late.
So we built a streaming classifier that we're releasing today! Here's what we did and why.
🧵...
... image classifiers also!
If you're not using either tool yet, now's a good time to try both! Zentropi's Community Edition is free and gives you unlimited labelers. Coop is fully open source and runs on your infrastructure.
:D
Thank you for your leadership and for being great stewards of Cove/Coop.
@dwillner.bsky.social and I have spent years watching T&S teams rebuild the same infrastructure from scratch. This is what it looks like when open tools actually work together instead. Really proud of this one and appreciative of @roost.tools's leadership!
Details: blog.zentropi.ai/zentropi-now...
Zentropi is now integrated into Coop, @roost.tools's open source moderation platform. You can write a content policy in plain English on Zentropi, plug it into Coop as a signal, and have a moderation pipeline running in minutes.
Dave Willner, who led trust and safety at major tech firms and has cofounded a company that is developing an AI content classification platform, says LLM-driven technology can now accomplish classification at the scale necessary for moderation on large platforms. That has substantial implications.
I can has cats.
Just shipped Zentropi's most requested feature: image classification!
Now analyze images against your own policies, at scale.
To power it we built cope-b-12b, a new multimodal model w/ native vision.
Check out the cat detector we made in < 1 min. 🐱
blog.zentropi.ai/zentropi-now-labels-images/
On the other hand, interesting contribution to do this all with a single transformer and candidate isolation.
If you are looking for a technical description of how X rots your brain, look no further than their github post on the 'X algorithm'. It is pure, unadulterated behavioral engagement maximization that amplifies the very worst human impulses. github.com/xai-org/x-al...
Would love to hear more! What kind of community guidelines were you feeding to CoPE? What worked well and where were there gaps?
Why are we just giving away all our secrets? Well, it is our hope that it helps the ecosystem further advance the state of the art in policy-steerable content classification, which is foundational to a more trustworthy internet.
Dave just published a Zentropi labeler that can precisely identify requests at prompting an AI model to undress a person in a photo. The tools exist to easily deal with this problem -- platforms just need to choose to use them. If you are the developer of an AI system, please use this guardrail!
"We'll make it right for you"
This was such a cool experiment that I created a Zentropi labeler with a simplified version of the authors' Partisan Animosity criteria. Now anyone can experiment directly with using this labeler to try to reduce the temperature of affective polarization in their feeds. zentropi.ai/labelers/b30...
We just wrote an in-depth post about Toxic Content labeling. It presents a new way of defining toxic speech online-- and illustrates the importance of observable features for accurate language model interpretability. Would love to hear how YOU define toxicity, too! blog.zentropi.ai/observations...
Awesome to see how this is already being used! One of the most useful aspects is that the published policies show what it takes to write content rules that can be accurately interpreted by language models. We hope this can be a boost to the broader content policy community.
For clarity, the whole point of this launch is to enable people to easily customize their own policies so that we can support a plurality of content classification perspectives online! It is actually a solution to the problem Evelyn highlights in that piece.
This was a fun launch! It turns Zentropi into a Github for Content Labelers. You can share content policies with others and build off each other's work. It's the easiest way of deploying a fully customizable classifier. Check out the policies @dwillner.bsky.social created at zentropi.ai/u/dave
Content policies are usually private, one-off efforts. You build yours, I build mine, we don't share much about what works or why. This makes sense given products can (and should) set different policies based on their communities, but it leaves us reinventing the wheel. 🧵 1/5
#MakeMasnickWrongAgain
Social media algorithms push people who are near the extremes to further extremes. But it doesn't have to be this way.
We got really positive feedback on the TrustCon workshop we ran on writing good content policies for LLMs...so we're doing it again! If you're interested go sign up here, so we can start to figure out timing: forms.gle/tj7vf7ng8n7R...
The standard damage control post from tech companies in these situations is to implicitly put the responsibility/blame on users. We can (and should) debate whether OpenAI's promised mitigations are sufficient, but the fact that they are tackling the *product problem* head-on is a vital first step.
This response to the Raine tragedy from OpenAI does something remarkable: it has the humility to acknowledge that a *product failure* led to real-world harm. Despite horrific circumstances, it has a rare degree of honesty that I wish tech companies would show more often. openai.com/index/helpin...
Yes, would definitely love to directly integrate with roost/coop. Let's chat!
