PromptBeat is an independent open-source product for model and prompt red-team evaluation. It turns scenarios, seeds, and provider configuration into repeatable evaluations and inspectable artifacts. Its current full pipeline uses Promptfoo for red-team generation and execution; PromptBeat owns the project workflow, Go CLI, validation, normalization, datasets, and packaged results around that backend. AVAILABLE NOW

​
Use PromptBeat when

  • you need to test prompt injection, jailbreak, refusal, harmful-content, or policy behavior;
  • you want to compare a model, Prompt, or provider change on the same cases;
  • you need a versioned dataset and repeatable regression workflow;
  • the claim you need to make is scoped to model/provider requests and responses.
For HTTP Agent actions, tool Evidence, or environment state, use AgentBeat instead.

​
One product loop

1

Describe the evaluation

A PromptBeat project defines the model Target, Scenarios, Seeds, Provider roles, and Promptfoo backend configuration.
2

Validate before calling a model

promptbeat validate checks the project structure and referenced files. Validation does not prove that a remote model is reachable.
3

Preview deterministic cases

promptbeat generate deterministically produces local JSON cases without running Promptfoo red-team generation or calling the Target.
4

Run the full pipeline

promptbeat run compiles Promptfoo configuration, runs Promptfoo red-team generation, validates generated cases, evaluates configured targets, and parses the result.
5

Inspect and retain artifacts

The output directory contains the generated backend artifacts, raw result, normalized evaluation_result.json, and report.html.
6

Build regression datasets

Dataset commands can select, version, and carry cases into future runs.

​
Current capabilities

Scenario-driven projects

Keep target definition, risks, expected behavior, and seeds reviewable in source-controlled files.

Promptfoo execution

Compile and execute the current red-team pipeline through the pinned Promptfoo backend.

Provider roles

Configure generator, Judge, and target providers independently through environment-backed values.

Inspectable artifacts

Retain raw backend output, normalized results, and an HTML report under the run directory.

Dataset versioning

Build versioned candidate, evaluation, promotion, lineage, and report artifacts.

Source or release use

Run from a complete release bundle or build the Go CLI from source and supply the required evaluation backend separately.

​
Runtime boundary

PromptBeat project
  → compile Promptfoo configuration
  → Promptfoo red-team generation
  → provider evaluation
  → parse EvaluationResult
  → JSON / HTML artifacts
PromptBeat does not execute the AgentBeat RunAgentEvaluation path and does not evaluate HTTP Agent tool use or environment state.

Quickstart

Design scenarios

Understand artifacts