Evidence that your AI is safe to ship.
Reinforced Labs is the evaluation and safety lab for teams putting models and agents into production. We measure what your system actually does, block what it must never do, and supply the expert data that teaches it the difference.
adversarial test cases executed for customers last quarter
vetted domain experts annotating safety data across 38 fields
median Guard decision latency, including tool-call inspection
production-tested evaluators in the Reinforced grader library
Benchmarks don't ship. Your system does.
A model's public score says almost nothing about how your prompt, your retrieval and your tools behave under pressure. The failures that hurt are specific to your deployment.
Once a model can send email, move money or query a database, a bad completion is an incident. Text filters can't see a well-formed tool call with the wrong arguments.
Policies in a wiki and a red-team report from last spring are not controls. Safety has to be a number that is checked on every release and watched in production.
One lab, four instruments.
MeasureReinforced Evals
Release-gating eval suites for models, RAG and agents. 1,400 graders, custom rubrics, human calibration, regression diffs and CI gates on every merge.
Evals → 02ProtectReinforced Guard
A runtime firewall for agents. Screens inputs, retrieved content, outputs and tool calls against policy-as-code. Blocks injection, exfiltration and unsafe actions in under 18 ms.
Guard → 03AttackReinforced Red
Continuous adversarial testing. Attacker agents probe your system every night; our human specialists run deep engagements before major launches.
Red Teaming → 04TeachReinforced Data
Expert-annotated red-team corpora, preference pairs and refusal-calibration sets for post-training, built by vetted specialists with double-blind QA.
Safety Data →A closed loop from commit to production, and back.
Safety that runs like a unit test.
Define a suite once, run it on every pull request, and block the merge when a score drops. The same suite runs against a raw model, a RAG pipeline or a ten-step agent.
- Python and TypeScript SDKs, plus a CLI for any CI system
- Graders calibrated against expert human labels, with agreement reported
- Per-case diffs so you see exactly which behaviour changed
- Statistical gates that separate real regressions from sampling noise
from reinforced import Suite, graders
suite = Suite.load("support-agent/release")
suite.add(graders.ToolCallPolicy("policies/refunds.yaml"))
suite.add(graders.InjectionResistance(corpus="indirect/email-v3"))
run = suite.run(target="https://staging.internal/agent",
concurrency=64, seeds=3)
# fail the build if any category regresses beyond noise
run.gate(min_score=0.95, max_regression=0.02, confidence=0.95)- uses: reinforced-labs/evals-action@v3
with:
suite: support-agent/release
target: ${{ steps.deploy.outputs.preview_url }}
gate: strictBuilt for the people who sign off on launch.
ML and platform leads
Replace notebook evals with a versioned suite that runs in CI and tells you, with confidence intervals, whether the new model is better.
CISOs and security teams
Treat agents like any other privileged workload: threat model, continuous testing, runtime enforcement and an audit trail for every blocked action.
Risk, compliance and legal
Evidence packs mapped to the EU AI Act, NIST AI RMF and ISO/IEC 42001, generated from real test runs rather than questionnaires.
Model developers
Pre-release capability and safety evaluations, expert red teaming and post-training data from specialists in biology, cyber, finance, law and medicine.
“We had forty evals in notebooks and no idea which ones mattered. Six weeks later every merge to main runs eleven thousand cases and blocks on regressions. Launch reviews went from a week of arguing to a single report.”Head of ML Platform · Fortune 100 financial services firm
Research we publish, and use.
AgentBreach-2: 3,100 tool-use attacks against production-style agents
Indirect injection succeeds against 4 in 10 undefended agents. Tool-call inspection cuts that by 92%.
When can you trust an LLM grader?
Agreement between model graders and expert humans across 26 eval categories, and the five where it collapses.
Over-refusal is a safety bug
A calibration set that measures harmful compliance and unnecessary refusal on the same axis.
Frequently asked.
What does Reinforced Labs do?
We build the measurement and control layer for production AI. Evals tells you what your model or agent actually does, Guard stops what it must never do at runtime, Red Teaming keeps finding new ways it can fail, and Safety Data supplies the expert-labelled examples that teach it the difference.
How is this different from public benchmarks?
Public benchmarks measure a model in general. We measure your system: your prompts, your tools, your retrieval, your policies and your users. A model that scores well on a leaderboard can still issue refunds it shouldn't, leak a customer record through a tool call, or follow instructions hidden in a PDF. Those failures only show up when you test the deployed system.
Which models and frameworks do you support?
Any model you can call: closed API models, open-weight models served on vLLM, TGI or SGLang, and fine-tuned variants. Agent frameworks are supported through OpenTelemetry traces and native SDKs for Python and TypeScript. You can point Evals at a raw model, a RAG pipeline or a full multi-step agent.
Can we run it inside our own cloud?
Yes. Every product can run as managed SaaS, as a single-tenant deployment, or fully inside your VPC on AWS, Google Cloud or Azure. Air-gapped on-prem installs are available for regulated and public-sector customers.
How long until we have a working eval suite?
Most teams have a first suite running against staging in the first week. Our applied team typically delivers a release-gating suite, with a threat model and baseline report, within 30 days.
Do you train on our data?
No. Customer prompts, outputs, traces and datasets are never used to train our models or shared with other customers. Data retention is configurable down to zero for Guard traffic.
Find out what your model does before your users do.
A 45-minute briefing with our evaluation team. Bring a use case; leave with a threat model, a draft eval plan and a sense of where you stand.