Start with the system under test, not the product name.

Choose PromptBeat

You send an input to a model or prompt endpoint and evaluate its response across red-team and regression scenarios.

Choose AgentBeat

You invoke an HTTP agent and need to evaluate its final response, declared runtime Evidence, or independently verified state changes.

​
Three questions are usually enough

  1. Is the behavior under test a model response? Use PromptBeat.
  2. Can the system take actions or change state behind an HTTP Agent boundary? Use AgentBeat.
  3. Do both layers matter? Run both products independently. Do not interpret one product’s score as evidence for the other layer.

​
Side-by-side

DimensionPromptBeatAgentBeat
Product statusPublic model/Prompt evaluation packageSeparate HTTP Agent evaluator plus runnable website Eval Preview; official GitHub evaluator Release pending
System under testModels and Prompt-driven endpointsBlack-box HTTP Agents
Primary inputPromptBeat project, scenarios, seeds, provider configComplete AgentCase, registered Target deployment, scoring profile
Main runtimeGo CLI with PromptfooGo RunAgentEvaluation; does not use Promptfoo
Execution backendPromptfoo for the current red-team/evaluation pipelineIndependent native Go AgentBeat runtime; no Promptfoo
ObservationProvider request/response and evaluation resultL1 final response; optional L2 Evidence; optional L3 state verification
ScoringBackend assertions/results normalized into PromptBeat artifactsDeterministic component + LLM Judge, with tier-qualified metrics
Command availabilitypromptbeat run --config … --output-dir … is publicly packagedWebsite Eval Preview provides agentbeat eval-run < eval-run.json; older GitHub adapter Release does not
SDK requirementNoneNone for L1; optional JS/Python helpers for Target transport and L2 Evidence
Best fitPrompt injection, jailbreak, policy and refusal regressionTool use, task completion, behavioral boundaries, environment impact

​
When to use both

A RAG assistant, coding assistant, or business automation Agent can expose both boundaries. PromptBeat can test model/Prompt response behavior. AgentBeat can separately test the Agent’s externally observable behavior and state impact. 
Model / Prompt boundary ── PromptBeat ── response evaluation
HTTP Agent boundary     ── AgentBeat  ── response + tier-qualified Evidence
Using both is an architectural choice, not a required sequence. PromptBeat does not generate mandatory inputs for AgentBeat, and AgentBeat is not a deeper mode of a PromptBeat run.

​
Current transition

AgentBeat is a separate product design and Go evaluation implementation that does not use Promptfoo. The website Eval Preview now supplies the selected agentbeat eval-run interface and a no-key local contract probe; it is not represented as an official GitHub Release.

Run a PromptBeat evaluation

Integrate an AgentBeat Target