Evaluation & Simulation

Score agent quality, run multi-turn simulations, and benchmark personality consistency.

Evaluate a Response

Score an agent's response against a template rubric.

const result = await client.agents.evaluate("agent-id", {
  templateId: "template-id",
  messages: [
    { role: "user", content: "I'm feeling really stressed about work" },
    { role: "assistant", content: "I hear you. Work stress can be overwhelming..." },
  ],
});

console.log(result.score);       // 0-100
console.log(result.feedback);    // detailed feedback
console.log(result.categories);  // per-category scores

Evaluation Templates

Create scoring rubrics with weighted categories.

// Create a template
const template = await client.evalTemplates.create({
  name: "Empathy & Support",
  description: "Evaluates emotional intelligence and supportive responses",
  scoringRubric: "Score based on empathy, active listening, and actionable advice",
  categories: ["empathy", "active_listening", "actionable_advice"],
  judgeModel: "claude-sonnet-4-6",
  temperature: 0.3,
});

// List templates
const templates = await client.evalTemplates.list();

Run a Simulation

Run multi-turn simulated conversations to test agent behavior at scale.

for await (const event of client.agents.simulate("agent-id", {
  maxSessions: 3,
  maxTurnsPerSession: 10,
  simulatedDurationHours: 24,
  enableProactive: true,
  enableConsolidation: true,
  userPersonas: [
    {
      name: "Alex",
      background: "College student struggling with math",
      personalityTraits: ["anxious", "eager to learn"],
      communicationStyle: "casual, uses slang",
    },
  ],
})) {
  console.log(`[${event.type}] ${event.message}`);
  if (event.totalCostUsd) {
    console.log(`Cost so far: $${event.totalCostUsd}`);
  }
}

Simulation + Evaluation (runEval)

Combine simulation and evaluation in one step.

for await (const event of client.agents.runEval("agent-id", {
  templateId: "template-id",
  maxSessions: 5,
  maxTurnsPerSession: 8,
})) {
  if (event.type === "evaluation") {
    console.log("Score:", event.score);
  }
}

Eval Runs

Track and manage simulation runs.

// List runs
const runs = await client.evalRuns.list({ agentId: "agent-id" });

// Get a specific run
const run = await client.evalRuns.get("run-id");

// Reconnect to a streaming run
for await (const event of client.evalRuns.streamEvents("run-id")) {
  console.log(event.type, event.message);
}

Async Simulations

Simulations support async mode via simulateAsync() which returns a RunRef immediately, allowing you to poll or reconnect later.

← Knowledge Base Instances→