Test AI behavior.
Before it reaches your users.
Red-team models and prompts with PromptBeat, verify black-box HTTP agents with AgentBeat, and inspect AI relay security benchmarks with RouterBeat.
PromptBeat
PromptBeatAutomated red-teaming and quality evaluation to verify models and prompts before deployment.
- Generate and inspect test cases locally without uploading sensitive data
- Test prompt injection, jailbreak resistance, and guardrail regressions
- Export standalone HTML reports with failure cases and scoring details
AgentBeat
AgentBeatBlack-box HTTP agent evaluation measuring goal completion alongside observable action evidence.
- Standard black-box HTTP evaluation without modifying internal agent code
- Record state observations and tool traces when integrations supply evidence
- Automated scoring combining task completion rates with safety checks
RouterBeat
RouterBeatMulti-dimensional live benchmarks evaluating security, stability, and transparency for AI relay platforms.
- Public multi-dimensional benchmarks with S/A/B/C tiers and stability metrics
- Inspect real model fingerprints, response integrity, and billing transparency
- Analyze prompt leakage risks and relay supply-chain anomalies
Start by describing what you want to test.
Load AIBeat Skill into your coding tool, then describe your evaluation in your own words. The skill assists with test planning and offline case generation before running full evaluations.
AIBeat Skill brings AIBeat workflows into your coding assistant. Its five current modules focus on PromptBeat; for black-box HTTP Agent evaluation, use AgentBeat.
Before you start: choose a suitable model
Not every model supports these evaluation workflows. Choose a model that supports the required tool calls and whose provider terms permit your authorized security-testing use case. Some models may refuse security test inputs or fail specific steps. Always stay within authorized scope, begin with non-sensitive samples, and never send real credentials, personal data, or production secrets.
Model role distinction: The coding assistant or Harness model driving the Skill is distinct from the tested Target system and any generator or Judge model; support in one role does not imply support in another. When bootstrapping test cases locally, the Pi assistant itself still relies on model calls and is not fully offline.
“I have a customer-support agent and worry it might access other users’ orders without authorization. How should I start testing? Help me plan first; do not execute tests.” (Recorded in Chinese)
In a workspace containing the product guides, we ask about authorization testing for a customer-support agent. The assistant first differentiates prompt/model-level testing from HTTP Agent testing, provides preliminary test ideas centered on prompt inputs, and asks about target architecture and data sources for next steps.
Loading promptbeat-getting-started skill with authorization test planning inquiry, requesting to plan first without executing tests. Click to view full 1280×960 image.
Terminal history view showing candidate test ideas provided by the assistant. Note: In the screenshot, the model misclassifies direct user input as indirect injection; direct malicious user input is direct injection, whereas indirect injection comes from untrusted external content or tool outputs. Candidate suggestions require verification; testing real backend permissions requires preparing interfaces and evidence per AgentBeat guides rather than relying on prompt responses. Click to view full 1280×960 image.
Terminal output showing candidate criteria and next-step routing, asking about target architecture and data sources for integration preparation. Click to view full 1280×960 image.
“Help me generate a few prompt-injection risk test cases for a customer-support scenario. Use local examples first, generate test cases only, and do not call target models.” (Recorded in Chinese)
Two turns in a single terminal session: Turn 1 loads promptbeat-run-quick-eval, validates local config, and generates 3 customer-support cases (saved to artifacts/cases.json without calling target models for evaluation); Turn 2 asks to keep only prompt injection cases, saving the 2 filtered cases to artifacts/prompt_injection_cases.json.
Loading promptbeat-run-quick-eval skill to generate customer-support cases using local examples without calling target models. Click to view full 1280×960 image.
Terminal runs validate to verify config then generate, writing 3 test cases to artifacts/cases.json. Click to view full 1280×960 image.
Terminal displays the 3 generated cases (2 prompt injection, 1 unauthorized tool invocation) saved to artifacts/cases.json. Click to view full 1280×960 image.
Follow-up in same session asking to keep only prompt injection cases and save them separately. Click to view full 1280×960 image.
Assistant executes filter, saving 2 retained prompt injection cases to artifacts/prompt_injection_cases.json and reporting count and location. Click to view full 1280×960 image.
More Questions You Can Ask AIBeat Skill
“I just downloaded AIBeat; what can I try out locally without connecting any models?” (Recorded in Chinese)
“We run a customer service bot; which risks should we test first?” (Recorded in Chinese)
“Check whether my configuration is complete first; do not start evaluation yet.” (Recorded in Chinese)
“Why did the connection time out? Locate the root cause first, do not re-run.” (Recorded in Chinese)
Expand your test scenarios.
Explore public scenario seeds to broaden evaluation coverage for prompt injections, privilege violations, and safety regressions.
Browse seed collections by safety domain, prompt-injection strategy, and regression scenario.
Review seed samples and follow the guide to add them to your local test suite.
Add seeds to your tests and reuse them after model, prompt, or application updates to check safety.
Scan to add assistant Xiaoxingxing
Send AIBEAT to join community
Connect with the community
Scan to add our official assistant, Xiaoxingxing. Send AIBEAT to join the community chat.