The SDK’s role
The SDK is collection-only. It gives you a small HTTP server and turns your existing agent call into thetarget-invocation-v1 contract. Your only job is one function:
| The SDK does | The SDK does not |
|---|---|
Serve /healthz, /v1/agent/invocations, /v1/evidence/{run_id} | Create an EvalRun or score |
Validate target-invocation-v1, reject runtime overrides, guard replay | Run a Judge or aggregate results |
| Bound and redact evidence, store it per run | Resolve the registry or pick cases |
| SDK package | Language | Exports you touch |
|---|---|---|
agentbeat-sdk-py | Python | create_target_server, InvocationContext, EvidenceCollector, read_bearer_token |
agentbeat-sdk-js | Node | createTargetServer, bearerTokenFromFile |
Three SDK routes
The SDK exposes these three routes. It has no/run alias or EvalRun endpoint. L2 evidence retrieval is optional for an L1-only Target.
| Route | Method | Auth | Purpose |
|---|---|---|---|
/healthz | GET | none | Readiness check. Reports evidence: "L1" or "L2". |
/v1/agent/invocations | POST | Bearer | Execute the agent. The only entry point. |
/v1/evidence/{run_id} | GET | Bearer | Fetch the bounded L2 evidence document. |
| Guard | Behavior | Error |
|---|---|---|
| Schema | unknown top-level fields, bad schema_version, bad input | 400 |
| Correlation | X-AIBeat-Run-ID / X-AIBeat-Case-ID headers must match the body | 400 |
| Runtime override | metadata must not configure model, endpoint, token, workspace, tools, state, sidecar | 400 |
| Replay | an accepted run_id is never accepted twice | 409 |
| Concurrency | more in-flight runs than max_concurrent_runs | 503 |
The three evidence levels
The invocation interface stays the same, but higher levels require actual observations and, at L3, an independent State Controller—not only a configuration flag.| Level | What the target returns | How it is produced |
|---|---|---|
| L1 | final_response only | Evidence disabled (evidence=False). /healthz reports evidence: "L1"; no evidence_ref. |
| L2 | final_response + evidence_ref | EvidenceCollector records bounded, redacted message/tool events. The document is target-evidence-v1 with assurance_level: "L2". |
| L3 | L2 + verified before/after state | Evaluator-owned State Controller runs seed → reset → snapshot(before) → invoke → snapshot(after) → verify. The Go Core combines the L2 document with the verifier receipt. |
Wire your framework
Every adapter projects your framework’s native events intotarget-evidence-v1. The invoke function is the same shape in all of them.
| Your framework | Use | Watch for |
|---|---|---|
| LangGraph | LangGraphObserver | reads stream_mode="updates"; no graph changes |
| OpenAI Agents | OpenAIAgentsTracingProcessor + observe_openai_agents_run | process-global registry; per-run isolation |
| LangChain | LangChainCallbackHandler | attach at the call root; coexists with LangSmith / Langfuse |
| Codex (Node) | createTargetServer + observeCodexNotification | out-of-process JSON-RPC |
| CrewAI / Langflow | manual EvidenceCollector calls | no native event callbacks |
| Anything else | OTelGenAISpanProcessor | best-effort fallback, see limits below |
Minimal target (any framework, L1-ready)
evidence=False and this same function serves L1. Leave evidence on (the default) and it serves L2.
If you don’t use a supported framework, use the manual collector (context.observe). Call its typed methods directly to build evidence.
LangChain
AttachLangChainCallbackHandler at the root RunnableConfig. This projects model, tool, and chain callbacks while keeping the run_id tree intact. A nested attachment misses sibling callbacks and can orphan tool results.
LangChainTracer) and Langfuse (CallbackHandler) can safely live in the same callbacks list. If you use async runnables like ainvoke, substitute AsyncLangChainCallbackHandler.
LangGraph
LangGraphObserver reads stream_mode="updates" output without touching graph nodes. It records assistant messages and the bound tool calls/results.
OpenAI Agents
The Agents SDK delivers tracing through a global registry. You must isolate concurrent runs.observe_openai_agents_run binds a lazy router to the active ContextVar.
openai-agents==0.22.0. Each run remains isolated, avoiding mixed evidence.
OpenTelemetry (fallback)
OTelGenAISpanProcessor is a best-effort fallback for missing integrations. It projects spans that follow GenAI conventions without taking over the global tracer provider.
- Opt-In payload. Content-capture fields (tool arguments, messages) are disabled by default in OTel. After a run,
processor.content_capture_report()(backed byContentCaptureProbe) tells you whether those fields actually appeared. If they did not, AgentBeat records a degradedlifecycleevent rather than inventing payload. - Development semconv. The GenAI convention snapshot is
2026-09-03. Attribute names may break and must be upgraded with the convention. - No provider takeover. The module never calls
trace.set_tracer_provider.
TracerProvider. AgentBeat’s redaction only protects the evidence document; it cannot redact data sent to your other exporters.
CrewAI / Langflow (manual)
These frameworks have no native event callback surface in the pinned versions, so you instrument manually with the collector methods shown above. The reference targets do exactly that.Codex (JavaScript)
The Codex target is out-of-process: the SDK serves the canonical HTTP routes, andobserveCodexNotification projects Codex app-server JSON-RPC notifications into evidence.
invoke returns finalResponse (camelCase), mirroring final_response on the Python side.
Register the target
A registry entry names the target and its deployment wiring; it stores no model, tool, fixture or secret values — only secret refs and env-var names, which resolve at deploy. Case and browser requests can only picktarget_id and deployment_profile_id.
deploy/aibeat-eval/config/agent-target-registry.langgraph-reference.example.json for the full configuration reference. state_verifier: true declares a capability; an actual registered State Controller, compatible deployment and verified state evidence are also required for L3.
Verify end to end
After the target is up, walk the three routes in order./healthz reports evidence: "L2" and the level you configured. If step 2 returns a body with evidence_ref, the target is emitting L2 evidence. If /v1/evidence returns 404, evidence is disabled — check the evidence option.
The final confirmation is in the Workbench: register the target, bind a deployment profile, run a case, and confirm the run shows a final_response plus an L2 trace (and, for L3, a verified state receipt) under the same run_id.
Three pitfalls
1. The model is fixed at deployment. Model endpoint, model id, and tokens come from startup environment variables. Requestmetadata keys like model, base_url, or token are rejected with a 400 error. Redeploy the target to change models.
2. L3 uses two tokens. The State Controller needs a control token to snapshot state. The agent needs a run-bound tool token. The agent must never receive the control token.
3. run_id is single-use. Reusing a run_id returns 409 run_id_reused. Failed turns also consume the ID because they may have caused side effects. Generate a fresh run_id per attempt.
Install the dependencies
The SDK core has no third-party dependencies. Each adapter’s framework is pinned by the example that uses it:| Adapter | Package | Pinned version (reference example) |
|---|---|---|
| core | agentbeat-sdk | 1.0.0 |
| LangChain | langchain + langchain-core | 1.3.14 / 1.5.3 |
| LangGraph | langgraph + langchain-core | 1.2.10 / 1.5.3 |
| OpenAI Agents | openai-agents | 0.22.0 |
| OpenTelemetry | opentelemetry-sdk + opentelemetry-api | 1.34.0 |
| CrewAI | crewai + crewai-tools | 1.15.17 |
| Codex (Node) | none (Node runtime) | Node >=22.22.0 |
pyproject.toml rather than installing unpinned latest.
What’s next
Adapters
Evaluator-side Connectors, Target SDKs and the legacy compatibility boundary.
Observation model
The trace schema your evidence is projected into.
Evidence
What a finding carries: response, trace, env delta.
Agent quickstart
Target kinds and the registry that binds them.