Products

DriftWatch

Alert when quality slips — before users complain. DriftWatch scores traffic and golden sets by slice, correlates with deploy markers, and opens alerts FineForge can act on.

When to use

You ship prompt or model changes and need regression gates
Quality drops on one intent while fleet averages look fine
Schema, retrieval, or provider behavior shifts silently

Inputs

Input	Role
Golden set	Fixed cases; business outcomes
Teacher model	Score / critique owned outputs; freeze judge prompt
Live sample	Stratified by intent — avoid easy-only bias (D12)
Deploy markers	Correlate score drop with prompt/model version
Shadow scores	Candidate vs prod before promote

Metrics

driftwatch.score.prod / score.shadow
driftwatch.delta, cusum
driftwatch.slice.score — heatmap by intent/locale/segment
driftwatch.alerts.open

Operator failure modes

ID	Problem	Next action
D01	Prompt regression	Rollback prompt_version
D02	Partial slice drift	Intent-specific prompt
D03	Tool schema drift	Adapter fix
D04	Provider behavior shift	Pin version; escalate
D08	Cost-cutting quality drop	Restore hard path
D10	Labeler / judge drift	Freeze judge prompt
D16	Format / JSON drift	Structured output gate
D22	Alert fatigue	Raise thresholds; critical slices
D24	Stale golden set	Refresh from FineForge failures

Config

drift:
  enabled: true
  window: 24h
  golden_set: /var/lib/inferix/golden/default.jsonl
  teacher: provider/strong
  slices: [intent, locale, tenant_tier]
  alert:
    min_delta: 0.05
    min_samples: 50
  seasonal_baseline: true

Operate guide: Drift & retrain.

Fits the loop

DriftWatch is off the hot path. Alerts feed FineForge jobs; RouteIQ and LensAI show whether a quality drop came from model mix or latency-induced incompletes.

Source

https://github.com/AkshantVats/driftwatch

Retrain, promote, rollback, prompt+adapter bundles.

FineForge →