Drift & retrain

DriftWatch watches quality against golden sets and live slices. FineForge turns an alert into a versioned fix — prompt patch, adapter, or LoRA — then promote or roll back with deploy markers on the same dashboards.

Loop

LensAI + TraceForge  →  live traffic + traces
         ↓
DriftWatch          →  score vs golden / teacher / slices
         ↓  alert
FineForge           →  patch | train | eval gate
         ↓
promote (canary→full)  or  rollback
         ↓
RouteIQ + prompt_version / model_id updated

DriftWatch signals

Key metrics: driftwatch.score.prod, driftwatch.slice.score, driftwatch.delta, driftwatch.alerts.open. Details: DriftWatch.

Golden sets and teachers

• Golden set — fixed prompts + expected outcomes (business success, not HTTP 200). Stratify by intent.
• Teacher model — stronger model scores or critiques owned SLM outputs; freeze the judge prompt to avoid labeler drift.
• Shadow — send % traffic to a candidate; compare driftwatch.score.shadow vs prod before promote.

drift:
  enabled: true
  window: 24h
  golden_set: /var/lib/inferix/golden/support.jsonl
  teacher: provider/strong
  slices:
    - intent
    - locale
    - tenant_tier
  alert:
    min_delta: 0.05
    min_samples: 50

FineForge jobs

# Start a FineForge job from an alert
# DESIGN TARGET — :4000 is not shipped; see Status
curl -X POST http://localhost:4000/v1/fineforge/jobs \
  -H "Authorization: Bearer $INFERIX_MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "alert_id": "dw_alert_8f2a",
    "kind": "prompt_patch",
    "model": "owned/slm-support",
    "shadow_pct": 5
  }'

Promote and rollback

• Canary — 1% traffic; auto-promote if driftwatch.delta > ε and error budget holds.
• Full promote — pin new model_id / prompt_version; emit deploy marker for LensAI dashboards.
• Rollback — instant previous revision; FineForge keeps the bad candidate for postmortem → new eval case.