Documentation

This is a training simulator for health-insurance member-services agents, built as part of the ServisBOT C360 platform. C360 analyses customer conversations; this app adds the practice layer: a realistic AI member an agent can talk to, live guidance while they talk, and a scored evaluation of the call afterwards.

The same rubric scores practice calls and (in production C360) live calls, so improvement can be measured on the same scale it is detected on. Everything you need to try it is behind the sign-in page — ask for a one-time login link if you do not have an account.

1. C360 reviews past calls and finds a skill gap for an agent (e.g. explaining appeal options).
2. A manager drills into the evidence — the finding, the transcript quote, the score.
3. C360 recommends a practice scenario and the manager assigns it.
4. The agent takes the practice call with the AI member — by voice or typed.
5. Live guidance helps during the call in guided mode; the same rubric scores the call afterwards.
6. C360 compares later live calls with earlier ones to show whether the practice worked.

• Agent — sees their own assignments, practice calls and performance page. Cannot see other agents.
• Manager — sees C360-lite for the whole team, assigns scenarios, authors scenarios.
• Admin — everything, plus user management and one-time login links.

1. Open C360-lite — the team overview flags Maria's weakness on appeal options.
2. Open Maria's advisor view and drill into a past call to see the finding and its evidence quote.
3. Assign the recommended denied-claim scenario (guided mode shows the guidance panel; assessment mode hides it).
4. Go to Simulator, pick Maria, and start the call. Speak to the AI member — identify yourself, explain the denial, cover the appeal options without promising an outcome.
5. Watch the live panel: steps tick off, the member record unlocks after verification, the provisional score moves.
6. End the call and review the colour-coded report — score gauge, strengths, focus areas, findings linked to transcript turns.
7. Back on Maria's page, Reveal post-training calls shows the (synthetic) before/after improvement chart.
8. Optional: try an early disclosure first — mention the claim before verifying identity and watch the critical alert.

The Scenario library (managers and admins) lists every scenario. "New scenario" opens a full editor: persona, hidden facts, what the member knows and volunteers, identity-verification rules, sensitive terms that trigger the early-disclosure alert, the member record shown after verification, mood rules and the call flow — exactly one step may unlock the member record.

Generate from transcript turns a real call transcript into a draft scenario: paste or upload the transcript, review how it was parsed, then let the AI draft the scenario. All personal data in the draft is fictionalized; a human reviews and edits the draft before it is saved. Saved scenarios are immediately usable in the simulator.

The editor role is for content designers who build scenarios, training plans and automation rules without access to live agent data. An editor's home page is the scenario library; the header shows Scenarios, Training plans, Rules, Simulator and Docs. Editors cannot reach the C360 team view, agent performance or the admin pages.

To test a scenario or plan, an editor uses the Simulator with the Design sandbox agent — a special agent whose practice calls never pollute real agent statistics. The sandbox agent is the only agent an editor can start a conversation for; this is enforced server-side.

A training plan is a sequenced path of practice steps that moves an agent from guided coaching to an independent assessment. Each step picks a scenario from the library, a mode (guided or assessment), a pass mark and a maximum number of attempts. A plan can only be published once its final step is an assessment.

When an agent is enrolled — manually by a manager or automatically by a rule — the first step's assignment is created. Passing a step advances to the next; failing creates a retry assignment up to the step's max attempts. Failing the last attempt moves the enrolment to needs coaching. Passing the final step completes the plan.

Automation rules watch live-call scores and enrol agents in training plans automatically. A rule has a trigger and a target plan. Two trigger types are supported:

• Skill average below threshold — fires when an agent's average score for a skill over their last N live calls drops below a threshold.
• Critical criterion failure — fires when an agent scores below 50 on a critical criterion (e.g. identity & privacy) on their most recent live call.

Each rule has a cooldown (days before the same agent can be re-enrolled in the same plan). Rules only fire for published plans, and skip agents who already have an active or needs-coaching enrolment in the target plan. A Test against current data button shows which agents would match a rule right now, without enrolling anyone.

Calibration checks that the AI scorer agrees with your experts. A calibration case is a call plus the scores an expert says it should get: a band for each rubric criterion (for example 20–40) and whether the call should critical-fail. Editors create cases from a pasted transcript; managers and admins can also add any scored call from its report. A reviewer approves each case, and changing its expected scores later returns it to draft.

A calibration run scores every case several times with the production scorer and reports, per criterion, how often the score fell inside the expert band (agreement) and how much it varied between repeats (spread), plus critical-fail recall and precision and the average gap to the expected overall score. Green targets: agreement of at least 80%, spread of at most 8 points, every expected critical fail caught, precision of at least 90% and overall error of at most 8 points. The demo ships with 27 draft cases drafted from its sample calls; they need expert approval before the results count as a benchmark. Re-run calibration after any change to the model, prompts or rubric.

Admins can generate passwordless login links at Login links: expiring, single-use URLs (/login?key=…) that sign someone straight in — as an existing user or as a temporary user with a chosen role. Each link is shown once, logs every use, and can be revoked. This site is a demo: links should be treated as credentials and not forwarded publicly.

The team dashboard can generate simulated live calls to demonstrate what C360 would feed in. Each generated call is a realistic transcript between a member and an agent, assessed turn-by-turn by the same live-assist engine that powers practice calls, and scored on completion. Every scored call runs the automation rules, which can enrol an agent in a training plan.

• What is simulated — the generated live calls (transcripts produced by an AI generator following an agent proficiency model) and the proficiency model itself (a per-skill score that rises when an agent passes a plan assessment).
• What is real — the live assessment engine (call-flow steps, criteria states, alerts, provisional score), the final scoring rubric, the automation rules engine, the training plans and the plan progression. These are the same code paths used for practice calls.
• The live monitor page streams the transcript and the C360 live assessment panel in real time, exactly as a real C360 integration would.

• Real — the voice simulation, the live guidance and the scoring of practice calls, all running on OpenAI models.
• Synthetic — the seeded "past live calls", the agents and the before/after improvement chart. These are authored fixtures labelled as illustrative, so the demo loop can be shown end to end.

• One Next.js app: the simulator and C360-lite talk only through an API boundary, so the real C360 can replace C360-lite later.
• Voice calls use an ephemeral OpenAI Realtime credential minted server-side; the API key never reaches the browser.
• The AI member and the evaluator are separate — the member never sees guidance or scores.
• Source, deployment scripts and architecture: github.com/proservisbot/agent-training.