PromptBeat has two generation paths with different behavior. Distinguishing them prevents surprise model calls and misleading documentation.

​
1. Local case preview

promptbeat generate \
  --config examples/llm-basic/promptbeat.yaml \
  --count 5 \
  --output artifacts/cases.json
This command invokes the Go GenerateCases application path. It reads the project, builds cases from configured Scenarios and Seeds, applies the per-Scenario count limit, and writes JSON.

Remote calls

None from this command

Target calls

None

Output

Local case JSON
Use it to inspect scope before any paid or side-effecting evaluation.

​
2. External-backend red-team generation

The complete project pipeline uses the external evaluation backend: 
promptbeat run \
  --config examples/llm-basic/promptbeat.yaml \
  --output-dir artifacts/run
Its generation stage runs the backend’s red-team generator with the compiled configuration. That stage can call the configured Generator Provider; the later evaluation stage calls the configured Target and can use the configured Judge.

Remote calls

Depends on configured Providers

Target calls

Yes, during evaluation

Output

Backend YAML and run artifacts
Preview locally first. Before run, review Provider endpoints, model names, concurrency, delay, num_tests, and every Scenario. The full pipeline may incur cost and send test content to third-party services.

​
Generation inputs

InputRole
TargetModel/Prompt endpoint and capabilities
ScenarioRisk and expected response boundary
SeedSource material used to construct probes
Generator ProviderModel used by external-backend red-team generation
Backend optionsNumber of tests, output paths, concurrency, delay, language

​
Keep artifacts reviewable

  • Use stable Scenario and Seed IDs.
  • Write generated files beneath a bounded artifact directory.
  • Inspect generated prompts before sending them to a sensitive Target.
  • Keep credentials in environment variables.
  • Record the project commit and Provider configuration needed to reproduce a run.

​
Dataset generation

dataset build-version and dataset build-cycle can construct larger candidate and regression artifacts. They are separate from the small local preview command and can invoke external-backend generation and evaluation. See Dataset loop.

Run evaluations

Build datasets