AgentBeat is a separate product design and repository-implemented Go evaluation path for black-box HTTP Agents. RunAgentEvaluation invokes a registered Agent, collects only the Evidence available at the requested observation tier, and derives Utility, Security, ASR, and Overall metrics. This runtime does not call Promptfoo and is not a PromptBeat mode.
The source implementation, Target SDK/Adapter Kits, and a website-built Eval Preview are available. The preview supplies agentbeat eval-run and a local no-key contract probe; it is not an official GitHub Release. The published v0.3-agentbeat-preview.1 remains the older adapter product and has no eval-run.

​
Use AgentBeat when

  • an HTTP Agent can take actions or change state beyond producing text;
  • you need to evaluate task completion and security as separate dimensions;
  • a Target can optionally provide message or tool Evidence;
  • an independent environment Controller can verify state changes;
  • missing Evidence must remain visible instead of being treated as a pass.
Use PromptBeat for model and Prompt red-team evaluation. PromptBeat’s current full pipeline uses Promptfoo; AgentBeat’s RunAgentEvaluation path does not use Promptfoo and does not generate Prompt attacks.

​
How one run works

complete AgentCase + registered Deployment + scoring profile
  → negotiate Target capability and observation tier
  → invoke POST /v1/agent/invocations
  → collect L1 / L2 / L3 Evidence
  → deterministic component + LLM Judge component
  → Utility · Security · ASR · Overall
1

Validate the Case and Deployment

The evaluator checks the complete 21-field AgentCase, Target Profile, Target Capability, Registry entry, requested tier, and scoring profile before invoking the Agent.
2

Invoke the HTTP Agent

The business-http-json-v1 Connector sends the trusted task to POST /v1/agent/invocations. Endpoint and credential resolution stay in the server-owned Registry, not in the Case.
3

Collect tier-qualified Evidence

L1 uses the final response. L2 additionally fetches declared Target Evidence. L3 additionally asks an independent State Controller to reset, snapshot, and verify the environment.
4

Score without hiding gaps

Deterministic and LLM Judge components are combined only when available. Missing components remain unavailable, and coverage stays visible.
5

Derive explicit metrics

Utility, Security, ASR, and Overall preserve their own applicability, direction, verdict, score, and Evidence coverage.

​
Progressive observation

L1 · Final response

Standard HTTP invocation. No AgentBeat SDK is required. Conclusions are limited to the returned response and available scoring components.

L2 · Target Evidence

Optional GET /v1/evidence/{run_id} supplies declared message and tool observations. The Target owns this Evidence source.

L3 · Environment state

An independent State Controller supplies before/after state and native verification. The Agent does not self-report the authoritative state.
Understand the observation model →

​
Repository capability and public availability

AreaEvaluation capabilityBoundary
Target invocationbusiness-http-json-v1 over HTTPThe registered endpoint must satisfy the strict request/response contract
Case contractComplete environment-independent 21-field AgentCaseNo endpoint, credential, tool, fixture, or scorer implementation in the Case
L2 collectionStrict target-evidence-v1 endpointOptional; undeclared or missing channels cannot be inferred
L3 controlRegistered independent State ControllerRequired only for L3 and compatible environment bindings
Cascade scorePositive deterministic + Judge weights summing to 10.7 / 0.3 is a profile example, not a hardcoded default
Hard gatesNative Security/ASR gate behaviorApplies only when authoritative L3 native Evidence establishes the condition
DistributionDevelopment source, Target SDK/Adapter Kits, and website Eval PreviewPreview is runnable but is not an official GitHub Release

​
Prepare an L1 integration

Begin at L1 if the Agent already exposes HTTP. Add L2 only when runtime observations are useful and the Target can return them safely. Add L3 only when an independent test environment can provide authoritative state.

Download and run

Adapters and SDKs

Evidence and scoring