Operate

Drift & retrain

DriftWatch watches quality against golden sets and live slices. FineForge turns an alert into a versioned fix — prompt patch, adapter, or LoRA — then promote or roll back with deploy markers on the same dashboards.

Loop

LensAI + TraceForge  →  live traffic + traces
         ↓
DriftWatch          →  score vs golden / teacher / slices
         ↓  alert
FineForge           →  patch | train | eval gate
         ↓
promote (canary→full)  or  rollback
         ↓
RouteIQ + prompt_version / model_id updated

DriftWatch signals

Signal	Symptom	Operator action
Prompt regression	Score drop + deploy marker	Rollback prompt_version
Slice drift	One intent hurts; others fine	Intent-specific prompt or route
Tool schema drift	Tool errors↑ + score↓	Adapter fix via FineForge
Provider shift	Score drop without your deploy	Pin provider version; escalate
Cost-cutting drift	Quality↓ after budget force-cheap	Restore hard path
Latency-induced	P99↑ with incomplete answers	RouteIQ failover
Golden set stale	Eval–prod gap widens	Refresh cases from failures
Alert fatigue	Ack rate collapses	Raise thresholds; critical slices only

Key metrics: driftwatch.score.prod, driftwatch.slice.score, driftwatch.delta, driftwatch.alerts.open. Details: DriftWatch.

Golden sets and teachers

Golden set — fixed prompts + expected outcomes (business success, not HTTP 200). Stratify by intent.
Teacher model — stronger model scores or critiques owned SLM outputs; freeze the judge prompt to avoid labeler drift.
Shadow — send % traffic to a candidate; compare driftwatch.score.shadow vs prod before promote.

drift:
  enabled: true
  window: 24h
  golden_set: /var/lib/inferix/golden/support.jsonl
  teacher: provider/strong
  slices:
    - intent
    - locale
    - tenant_tier
  alert:
    min_delta: 0.05
    min_samples: 50

FineForge jobs

Job type	When	Artifact
Prompt patch	Surgical regression	prompt_version bump
Adapter fix	Schema / tool 4xx	tool_schema_version + fixtures
Teacher critique	Failure cluster	Repair suggestions → patch
Distill / LoRA	Cheap path specialization	owned model revision
Eval harness expand	New failure mode	Cases from DriftWatch samples
Bundle promote	Prompt+adapter+route	Atomic multi-artifact version

# Start a FineForge job from an alert
curl -X POST http://localhost:4000/v1/fineforge/jobs \
  -H "Authorization: Bearer $INFERIX_MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "alert_id": "dw_alert_8f2a",
    "kind": "prompt_patch",
    "model": "owned/slm-support",
    "shadow_pct": 5
  }'

Promote and rollback

Canary — 1% traffic; auto-promote if driftwatch.delta > ε and error budget holds.
Full promote — pin new model_id / prompt_version; emit deploy marker for LensAI dashboards.
Rollback — instant previous revision; FineForge keeps the bad candidate for postmortem → new eval case.

Change one variable

Cascade interaction drift (D19) confounds RCA when route and prompt change together. Promote prompt, adapter, or route rule in separate steps unless you ship an explicit FineForge bundle.

Chat completions, traces, policies, drift alerts, FineForge jobs.

API reference →