Bad actors and harmful content now evolve at the speed of a prompt. 

Nuanced, malicious content can be generated that slips past static rule sets with ease. The sheer volume and variety of AI-generated text and images make manually updating a rules-based system or relying on human moderators a non-starter. The speed, scale, and nuance of AI-generated content have made it clear: the old ways of working are no longer enough. If you’re a T&S leader still wrestling with legacy systems, this is the moment to ask if your tools of the past are equipped for the challenges of today.

With TrustCon behind me and another year ahead, I find myself reflecting on what I heard. The theme this year felt familiar to last year: the slow, grinding frustration of fighting tomorrow’s problems with yesterday's tools. Knowing teams are still tethered to legacy solutions built a decade ago has me thinking about my own experience with these tools.

From the Trenches:
A Story of Good Intentions and Bad Tools

While I was at Microsoft working on Xbox Safety, I remember a time when our team saw a spike in user complaints on social media. It wasn’t due to toxicity. Players were trying to do the very things we wanted them to do: connect, collaborate, and play. But a series of rigid rules in our community moderation tool started inadvertently blocking users from using common in-game terms, eroding user experience. The system saw a text pattern and failed to see the nuance between what the rule was intended to catch and what it shouldn’t catch, resulting in false positives that negatively impacted gameplay. Such is the way with rules-based systems - and it was frustrating for our safety team as much as our users!

Writing rules in that moderation system was simplistic and couldn’t be tested, they just went live and you had to hope you set it up correctly. There was never an understanding of how accurate any change was going to be. Our confidence was a shrug of the shoulders on how many false positives we might see with any change. In times of crisis, like the above, the only solution was to have our team of humans manually “jam things into the system until it worked,” a painfully slow process that pulled our focus from finding genuinely harmful content.

Our confidence was a shrug of the shoulders on how many false positives we might see with any change.

This single incident was a symptom of a deeper problem. We were building our entire safety strategy on a system of categories and risk scores we couldn't fully understand. And for anything visual, the technology of the time was so limited that the full weight of reviewing every reported image fell directly on our human moderators—an expensive, time-consuming, and often draining task. We weren’t just flying blind; we were building inside a box. Our safety features had to conform to the rigid constraints of an external system, rather than being designed around the ideal experience for our players. This limitation didn't just cost us agility—it eroded our ability to innovate and bring delight to players.The Rigidity of Rules in a Fluid World

Many long-standing tools in the moderation space were built on a foundation of rules-based systems. This approach, which relies on predefined lists of keywords, phrases, and patterns, was barely effective as a “first generation” safety solution. However, in the age of generative AI, this rigidity is a significant drawback.

The Rigidity of Rules in a Fluid World

Many long-standing tools in the moderation space were built on a foundation of rules-based systems. This approach, which relies on predefined lists of keywords, phrases, and patterns, was barely effective as a “first generation” safety solution. However, in the age of generative AI, this rigidity is a significant drawback.

The Problem with the “Black Box”

In response to the limitations of rule-based systems, many platforms—including my own at Xbox—evaluated and turned to “black box“ machine learning solutions. These models are trained on vast datasets to recognize harmful content, offering a more dynamic approach than their predecessors. The challenge with many of these systems, however, lies in their opaque nature.

For platforms building vibrant, creative communities that deeply value user trust, understanding the nuances of flagged content is paramount.

When a piece of content is flagged, it can be incredibly difficult to understand why. Was it a specific object in an image? A particular combination of words? Maybe even a bad piece of training data buried somewhere deep. This lack of transparency creates a host of problems. For platforms building vibrant, creative communities that deeply value user trust, understanding the nuances of flagged content is paramount. Without this explainability, it’s impossible to refine moderation policies, educate users, or build a system of trust. You are left at the mercy of a decision you cannot understand or improve upon.

The Clavata.ai Difference:
What I Wish I Had

Looking back on those days in the trenches, I know exactly what my team needed. We needed more than just a better blocklist or a smarter algorithm. We needed a new way to work. We needed:

• Fluidity and Control: We needed the ability to write a rule that understood concepts, not just keywords. The ability to distinguish between “I want to trade my sword” and “I will trade your account credentials.” We needed a tool that gave us, the safety experts, direct control to write, test, and deploy our policies with precision.

• Explainability: When a decision was made, we needed to know why. A transparent, auditable system would have allowed us to build trust with our community and have a fair appeals process, instead of just saying “computer says no.”

• Agility: We needed to be able to react in minutes, not weeks. The ability to identify a new problem, author a policy, test it against live data, and deploy it instantly would have been a game-changer, turning a week-long crisis into a 15-minute fix.

This is precisely what we’ve built at Clavata.ai. It’s the platform I wish I had while being on the frontlines of Xbox community safety. It’s designed to empower Trust & Safety teams, giving them modern, agile, and powerful tools needed to move at the speed of their communities.

The world has changed. The challenges are more complex than ever. It’s time our tools evolved too. It’s time to stop fighting fires with a leaky bucket and start building a truly resilient foundation for trust and safety.