Operate

Observability

LensAI tells you what is happening across the fleet. TraceForge tells you why a single request behaved that way. Metrics for SLOs and cost; spans for RCA. Inferix sits next to your gateway and existing traces — not instead of them.

Ports (do not mix them up)

Port	What
:4000	Inferix control plane — chat, policy, /v1/observe, /v1/traces
:3000	Grafana — LensAI / TraceForge dashboards
:8080	Streaming ingest — eBPF/HTTP events into Redpanda

LensAI pipeline

eBPF or HTTP/SDK  →  ingest :8080  →  Redpanda  →  ClickHouse  →  Grafana :3000
                                                      ↑
                                         query also via :4000 /v1/observe/metrics

Labels that must exist: tenant_id, model_id, trace_id, plus agent_id and env. Missing tenant breaks cost; missing trace_id blocks spike→trace jumps.

Latency: lensai.latency.p50/p95/p99
Tokens / cost: tokens.*, cost.usd, cost.per_successful_task
Errors: timeouts, provider 5xx, 429s, empty-200 class
Pipeline health: ingest lag, cardinality, WAL depth

Deep dive: LensAI. Repos: LensAI, streaming, eBPF.

TraceForge spans

OTel gen_ai spans with a versioned tool taxonomy:

inferix.call
├── routeiq.decide
├── agent.tool.<name>
└── gen_ai.chat

Business success predicate on the task — not HTTP 200 alone
Watch N+1 loops, retry storms, skipped required steps
Keep root spans under sampling; hash prompts; scrub PII

Deep dive: TraceForge.

Together

Question	Use	Signal
Is p95 up for slm-support?	LensAI	latency by model_id
Why did this refund stall?	TraceForge	span tree + tools
Tenant X burn today?	LensAI	cost by tenant_id
Which route ran?	TraceForge + LensAI	route_decision
Quality vs yesterday?	DriftWatch overlay	score + deploy marker

Sampling

observe.sample_rate applies to TraceForge spans. LensAI metrics stay complete so cost and SLOs stay honest. Always sample errors and slow traces.

RouteIQ cheap / general / hard path.

Routing policies →