> ## Documentation Index
> Fetch the complete documentation index at: https://docs.cloudshipai.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Evaluation & Testing

> Test agents with LLM-as-judge scoring

## Why Built-In Evaluation?

**Traditional Testing:**

* Manual testing with ad-hoc prompts
* Subjective quality checks
* No baseline for comparisons

**Station's Evaluation:**

* Automated scenario generation
* LLM-as-judge scoring
* Business-focused metrics
* Full execution traces

## Via MCP Tools

The easiest way to run evaluations is through your AI assistant:

```
"Generate 50 test scenarios for the cost-analyzer agent and run them"
```

Station uses these MCP tools:

| Tool                      | Purpose                                        |
| ------------------------- | ---------------------------------------------- |
| `generate_and_test_agent` | Generate scenarios + execute tests in one step |
| `batch_execute_agents`    | Run multiple agents in parallel                |
| `evaluate_benchmark`      | LLM-as-judge scoring on a run                  |
| `evaluate_dataset`        | Score entire dataset                           |
| `list_benchmark_results`  | View evaluation results                        |
| `get_benchmark_status`    | Check async evaluation progress                |

**Reports:**

| Tool              | Purpose                            |
| ----------------- | ---------------------------------- |
| `create_report`   | Define team performance criteria   |
| `generate_report` | Run benchmarks and generate report |
| `list_reports`    | List all reports                   |
| `get_report`      | Get report details                 |

## Via CLI

### 1. Generate Test Scenarios

Station AI generates diverse test scenarios based on your agent's purpose:

```bash theme={null}
stn benchmark generate --agent "Cost Analyzer" --count 50
```

**Variation Strategies:**

* `comprehensive` - Wide range of scenarios (default)
* `edge_cases` - Unusual boundary conditions
* `common` - Typical real-world cases

### 2. Execute Tests

Run all scenarios with trace capture:

```bash theme={null}
stn benchmark run --agent "Cost Analyzer"
```

Results saved to timestamped dataset:

```
~/.config/station/environments/default/datasets/
└── agent-42-20251215-103045/
    ├── dataset.json
    └── runs/
```

### 3. Evaluate Quality

LLM-as-judge analyzes each run:

```bash theme={null}
stn benchmark evaluate --dataset agent-42-20251215-103045
```

## Quality Metrics

| Metric           | Description                | Weight |
| ---------------- | -------------------------- | ------ |
| **Accuracy**     | Correctness of results     | 30%    |
| **Completeness** | All aspects addressed      | 25%    |
| **Tool Usage**   | Appropriate tool selection | 20%    |
| **Efficiency**   | Minimal unnecessary steps  | 15%    |
| **Safety**       | No harmful actions         | 10%    |

## Evaluation Report

```json theme={null}
{
  "agent": "Cost Analyzer",
  "scenarios_tested": 50,
  "pass_rate": 0.92,
  "average_score": 8.4,
  "metrics": {
    "accuracy": 8.7,
    "completeness": 8.2,
    "tool_usage": 8.5,
    "efficiency": 8.1,
    "safety": 9.0
  },
  "production_ready": true
}
```

## Team Reports

Evaluate multi-agent teams against business goals.

### Via MCP

```
"Create a performance report for my SRE team measuring incident response time and accuracy"
```

```
"Generate the SRE team report"
```

### Via CLI

```bash theme={null}
stn report generate --environment default
```

### Report Metrics

| Metric           | Description              |
| ---------------- | ------------------------ |
| **Team Score**   | Overall 1-10 rating      |
| **MTTR**         | Mean Time to Resolution  |
| **Cost Savings** | \$ identified by agents  |
| **Accuracy**     | vs ground truth          |
| **Tool Cost**    | Execution cost per agent |

### Example Report Output

```
Team Performance: 7.5/10

✅ Multi-agent coordination: 8.5/10 - Excellent delegation
✅ Tool utilization: 8.0/10 - Effective use of all tools
✅ Root cause analysis: 7.5/10 - Identifies issues accurately
⚠️ Resolution speed: 7.0/10 - Room for improvement
⚠️ Communication clarity: 6.5/10 - Could be more concise
```

## Viewing Traces

Every evaluation run captures full execution traces:

```bash theme={null}
# Start Jaeger
docker run -d --name jaeger \
  -p 16686:16686 -p 4317:4317 -p 4318:4318 \
  jaegertracing/all-in-one:latest

# View traces at http://localhost:16686
```

## Best Practices

<AccordionGroup>
  <Accordion title="Test Before Deploy">
    Run evaluation before promoting agents to production.
  </Accordion>

  <Accordion title="Use Edge Cases">
    Include boundary conditions to find failure modes.
  </Accordion>

  <Accordion title="Track Over Time">
    Compare scores across agent versions.
  </Accordion>

  <Accordion title="Review Failures">
    Manually inspect low-scoring runs to improve prompts.
  </Accordion>
</AccordionGroup>

## CLI Reference

```bash theme={null}
# Generate scenarios
stn benchmark generate --agent <name> --count <n> --strategy <type>

# Run benchmarks
stn benchmark run --agent <name> [--concurrent <n>]

# Evaluate dataset
stn benchmark evaluate --dataset <path>

# List datasets
stn benchmark list

# Generate team report
stn report generate --environment <name>
```

## Next Steps

<CardGroup cols={2}>
  <Card title="Bundles" icon="box" href="/station/bundles">
    Package tested agents for distribution
  </Card>

  <Card title="CloudShip Platform" icon="cloud" href="/platform/overview">
    Centralized monitoring and management
  </Card>
</CardGroup>