botz-mcp swarm Documentation

Standards-based Model Context Protocol (MCP) contract, smoke, soak, and load testing — from standalone CLI suites in git repositories to governed multi-server Studio projects.

1. Quickstart & Standalone CLI

Capture any MCP server (Streamable HTTP, SSE, or stdio) to generate a schema-validated contract suite, then execute smoke or load runs locally or in CI without silent greens.

# Install or invoke via botz CLI
botz plugins install mcp-swarm

# Introspect an MCP server and generate a contract suite (.botz/mcp-swarm/suites/demo)
botz mcp-swarm capture --url https://mcp.example.com/mcp --name demo

# Run the suite with parameter overrides or concurrency settings
botz mcp-swarm run --suite demo --concurrency 4 --iterations 10

# Analyse errors from the latest run with AI root-cause & RFC 6902 patch proposals
botz mcp-swarm analyse --suite demo

No silent green: A run that never exercised the server or skipped required capabilities is recorded as a fail verdict with non-zero exit code.
Git-friendly layout: Suites live in .botz/mcp-swarm/suites/<name>/suite.json (committed), while credentials (config.json) and run reports (runs/) are automatically gitignored.

2. MCP Server Authentication: 1-Click API Key & OAuth 2.0 / SSO

When adding or editing an MCP server in Servers or Project Studio, choose an authentication mode in the Authentication panel:

API Key / Bearer Token

Enter a single masked API key or token with Show/Hide toggle. Choose Authorization: Bearer <token>, X-API-Key: <key>, or a custom header name. Credentials are automatically injected on every MCP call and scrubbed from logs and reports.

OAuth 2.0 / SSO (PKCE + DCR)

Click Auto-discover OAuth or Sign in with OAuth SSO. Swarm probes RFC 9728 .well-known/oauth-protected-resource and RFC 8414 .well-known/oauth-authorization-server, performs Dynamic Client Registration (RFC 7591), and completes Authorization Code + PKCE or Client Credentials flows with automatic token refresh.

3. Multi-Server Suites & Variable Extraction

A project can register multiple MCP servers and orchestrate cross-server workflows in a single suite:

Assign any sampler step, group, or loop to a specific server alias (or inherit from its parent frame).
Use Extractor steps (extract.variable + JSONPath) to capture values from one MCP server's tool response and pass ${varName} into subsequent steps on another server.
Reports track per-server latency percentiles, error rates, session counts, and end-to-end Flow p95 across full virtual-user passes.

4. Load Profiles, SLO Thresholds & Graceful Ramp-Down

Load Profile: Configure rampUpMs, hold durationMs, rampDownMs, maxConcurrency (virtual users), dwellMs, and suite safety ceilings (maxCallCount, maxPayloadBytes).
SLO Objectives: Set suite-wide, per-server, or per-step maxP95Ms, maxP99Ms, and maxErrorRate thresholds.
Graceful Ramp-Down: Calls cut off after the drain deadline are tracked under rampDown.cancelledByRampDown and excluded from error rates and pass/fail verdicts.

5. AI Error Analysis, Fix Proposals & Standalone Reports

Analyse Errors: Classifies failures into suite_bug (with validated RFC 6902 JSON patches you can accept into a new suite version in one click), server_bug, or environment.
Standalone HTML & PDF Reports: Every stored run provides a CSP-safe standalone report (/api/admin/projects/:id/runs/:runId/report.html) and downloadable PDF (report.pdf) with response time, TPS, and VU timelines plus scrubbed request/response samples.

6. Remote Workers, Governance & Platform Analytics

Outbound Workers: Run workers behind NAT/firewalls using SWARM_WORKER_TOKEN; workers dial out to the control plane to execute assigned suites.
GitHub Org & Team RBAC: Access is scoped to your GitHub organisation with owner, operator, editor, runner, and destructive role bindings.
Platform Analytics & Costs: Operators can inspect real-time telemetry, structured logs, LLM token spend, and live model routing under Settings → Analytics & Logs and Costs & Models.

7. Remote MCP Server OTLP Instrumentation (Python & Node.js)

Every HTTP/SSE suite run sends X-Swarm-Run-Id: <runId> on each MCP request. Remote Python and Node.js MCP servers can capture X-Swarm-Run-Id into request context, sample in-process memory (process.runtime.rss.bytes) and CPU (process.runtime.cpu.ratio), scrub secret attributes (/authorization|token|secret|password|api[_-]?key|bearer|credential/i), and export OTLP JSON batches to https://swarm.botz.ai/otlp/v1/metrics and /otlp/v1/traces authenticated with X-Swarm-Token: <worker-or-otlp-token>.

Python (`psutil` + ASGI Middleware)

# Extract X-Swarm-Run-Id in ASGI middleware & export redacted OTLP metrics/traces:
run_id = headers.get("x-swarm-run-id", "")
# Tag process.runtime.rss.bytes, process.runtime.cpu.ratio, mcp.rpc.duration, and mcp.tool.call.duration
# with attributes: {"run.id": run_id, "role": "sut", "runtime": "python"}
# POST to https://swarm.botz.ai/otlp/v1/metrics with header X-Swarm-Token: <SWARM_WORKER_TOKEN>

Node.js / TypeScript (`AsyncLocalStorage` + HTTP Middleware)

// Capture req.headers["x-swarm-run-id"] in AsyncLocalStorage:
const runId = String(req.headers["x-swarm-run-id"] ?? "").trim();
// Sample process.memoryUsage().rss and process.cpuUsage() tagged with {"run.id": runId, "role": "sut", "runtime": "node"}
// Scrub secret keys before POSTing OTLP JSON to https://swarm.botz.ai/otlp/v1/metrics and /otlp/v1/traces