Don't Trust AI. Build Systems That Earn Your Trust.

JB avatar
JB
Cover for Don't Trust AI. Build Systems That Earn Your Trust.

One topic we discuss frequently on our stand-ups is workflow: specifically the bottlenecks and time-sucks in our daily operations.

Code Review was the first and most obvious time-suck we solved last year, with an orchestration system that spawns a finder fleet of up to eight adversarial agents to review AI output before a human ever reads it. Their only job is to attack the work and find the gaps, the shortcuts, and the misguided claims. We shipped that as the deep-code-review skill in our free deep-skills series.

But code review wasn't the only bottleneck. The same problem kept surfacing in non-coding efforts: too much human time spent re-reading and correcting AI output.

For example, you go into planning mode with an agent. The plan gets generated in minutes. Then you spend an hour reviewing it. You submit a few changes, the AI updates the plan, and there you go again, another re-read. Review time is the new bottleneck, and it wasn't isolated to engineers: our QA and BA were living the same pattern.

Here's the formula we're running instead.

The formula

Before any human reads AI output, an adversarial agent reviews it first. Its only job is to attack the work and find the gaps, the shortcuts, and the misguided claims.

The first few runs, you re-read everything anyway. And you notice something. If it's designed correctly, the adversary is catching the same issues you would have caught. In some cases, it's catching things you wouldn't have. An "adversarial" agent wakes up a very different part of inference than one instructed to "create".

That's when trust starts to compound. Not trust in the model. Trust in the process.

Do this enough with a correctly designed adversarial agent and you can reliably point it at most anything in your process: research, planning, evaluations, and beyond. No more manual re-reads.

How to build one
  1. Give your adversarial agent a role, memory, and context, then watch it self-improve.
  2. Trigger it manually until you're comfortable.
  3. Refine it, then automate.
  4. Design and integrate a specific adversary for each type of output.

Soon you'll have trusted systems with predictable outcomes.

Steal this

We shipped our fix for coding plans in deep-plan-review, an adversarial fleet that attacks the plan before you re-read it. It's in the same free deep-skills series as deep-code-review.

Explore the deep-skills series
Trust in AI systems isn't a feeling. It's the byproduct of an adversary that keeps catching what you would have caught.

That's one of the differences between vibe-slop mystery and governed engineering.