Evaluation & Testing

"Generate 50 test scenarios for the cost-analyzer agent and run them"

stn benchmark generate --agent "Cost Analyzer" --count 50

stn benchmark run --agent "Cost Analyzer"

~/.config/station/environments/default/datasets/
└── agent-42-20251215-103045/
    ├── dataset.json
    └── runs/

stn benchmark evaluate --dataset agent-42-20251215-103045

{
  "agent": "Cost Analyzer",
  "scenarios_tested": 50,
  "pass_rate": 0.92,
  "average_score": 8.4,
  "metrics": {
    "accuracy": 8.7,
    "completeness": 8.2,
    "tool_usage": 8.5,
    "efficiency": 8.1,
    "safety": 9.0
  },
  "production_ready": true
}

"Create a performance report for my SRE team measuring incident response time and accuracy"

"Generate the SRE team report"

stn report generate --environment default

Team Performance: 7.5/10

✅ Multi-agent coordination: 8.5/10 - Excellent delegation
✅ Tool utilization: 8.0/10 - Effective use of all tools
✅ Root cause analysis: 7.5/10 - Identifies issues accurately
⚠️ Resolution speed: 7.0/10 - Room for improvement
⚠️ Communication clarity: 6.5/10 - Could be more concise

# Start Jaeger
docker run -d --name jaeger \
  -p 16686:16686 -p 4317:4317 -p 4318:4318 \
  jaegertracing/all-in-one:latest

# View traces at http://localhost:16686

# Generate scenarios
stn benchmark generate --agent <name> --count <n> --strategy <type>

# Run benchmarks
stn benchmark run --agent <name> [--concurrent <n>]

# Evaluate dataset
stn benchmark evaluate --dataset <path>

# List datasets
stn benchmark list

# Generate team report
stn report generate --environment <name>