Explainer

Your AI guardrail passed the demo. Has anyone attacked it since?

A guardrail that was switched on and a guardrail that works look the same on a quiet dashboard. This is how to tell them apart.

By the NoFly team

Four ways a control fails quietly

  • It is not in the path. Some traffic goes around it, and nobody measured how much.
  • It was never attacked. No alerts looks exactly like no attacks.
  • It blocks everything. Every attack test passes, and so does nothing else.
  • It was tested once, by the vendor, on the vendor’s own examples.

What to measure

  • Reach: how much of the real traffic crosses the control.
  • Refusals: whether it stopped a reproduced exploit, observed rather than assumed.
  • Ordinary work: whether routine traffic still gets through.
  • Freshness: when it was last attacked, and with what.

A ladder instead of a green light

NoFly grades each control on one ladder: Defeated, Indiscriminate, Porous, Absent, Unexercised, Refusing and Holding. A control reaches Holding only when it refused a reproduced exploit and still let ordinary work through. Never attacked is Unexercised, and it is never shown as green.

How NoFly helps

  • Verify

    Every control, including model provider guardrails, prompt firewalls, proxies and DLP, is graded on the same ladder from evidence you can recompute.

  • Attack

    Attacks run on a schedule and in CI, so a control that stops holding is caught by the next run rather than the next pen test.

What to do this week

This week3 actions
  1. List every AI security control you pay for, and when each one was last attacked.
  2. Measure what share of agent traffic actually passes through each one.
  3. Replay one known exploit with the control on, then off.
Keep reading

More from the blog.

Find out what your AI is allowed to do today.

Book a call and we will map your agents, MCP servers and keys with you, or start on one laptop with the free scan.

Book a demo
npm i -g @nofly/agent