LLMOps & Observability¶

The bar: you can take an LLM feature that "works on your machine" and stand up the tracing, logging, metrics, and cost controls that let you debug it, prove it's healthy, and keep the bill from surprising you once real users are hitting it.

An AI feature that runs in a demo and an AI feature you can operate in production are two different things. The demo tells you nothing about what happens at 2 a.m. when a prompt starts returning garbage, latency triples, or a runaway loop burns through your token budget. Observability is how you find out — and how you fix it before a veteran on the other end notices.

This module is the discipline of running LLM systems the way you'd run any other production service: instrument everything, log the right things, watch the numbers, and control the spend. You can't improve what you can't measure. By the end you can wire a feature so that every request leaves a trail you can follow.

23.1 LangSmith¶

Objective: trace and debug LLM applications end to end instead of guessing at black-box behavior.

LangSmith is a tracing platform built for LLM apps. When a request fans out across a chain — retrieval, a prompt, a model call, a parser — a trace captures every step with its inputs, outputs, timing, and token counts. Setting up tracing is mostly environment configuration; once it's on, runs show up in a UI where you can visualize exactly what the model saw and returned. That run visualization is the single biggest upgrade over print-debugging a prompt.

Beyond debugging, LangSmith holds evaluation datasets — curated input/expected-output pairs you replay against a chain to catch regressions before you ship a prompt change. Pair that with prompt versioning so you always know which prompt produced which behavior, and cost tracking so each run's token spend is visible next to its output.

The common mistake is treating tracing as a one-time bolt-on you enable while debugging and forget. Leave it on in staging and production (with sampling if volume is high). The trace you didn't capture is the incident you can't explain.

• Drill: set the LangSmith environment variables (LANGCHAIN_TRACING_V2, LANGCHAIN_API_KEY) and confirm runs appear in the dashboard.
• Drill: open a multi-step run and read each step's inputs, outputs, latency, and tokens.
• Drill: build a small evaluation dataset and run it against two prompt versions to compare.
• Drill: use prompt versioning and per-run cost tracking to tie a behavior change to a specific prompt.

23.2 Structured Logging¶

Objective: log in a way that makes debugging fast instead of a grep-through-noise slog.

Structured logs are logs as data, not sentences. Emit JSON — {"level":"info","event":"llm_call","request_id":"...","tokens":812} — and your aggregator can filter, sort, and alert on fields instead of forcing you to regex freeform text. Use log levels deliberately: debug for local detail, info for normal events, warn and error for things that need attention, and filter by level so production isn't drowning in noise.

Request correlation is the piece self-taught engineers usually miss. Generate a request ID at the entry point and attach it to every log line for that request, so one user's journey through retrieval, model call, and response is a single filter query — not detective work across unrelated lines.

Handle sensitive data on purpose. Prompts and completions can carry PII; never log raw secrets, tokens, or personal data you wouldn't want in a breach. Redact at the logging boundary. Then ship logs to an aggregator so they're searchable in one place instead of scattered across containers that die on redeploy.

• Drill: emit logs as JSON with consistent field names.
• Drill: wire log levels and filter production down to info and above.
• Drill: thread a request_id through every log line in a request's lifecycle.
• Drill: redact sensitive fields before they hit the logs, and forward logs to an aggregator.

23.3 Metrics and Alerting¶

Objective: monitor system health so you learn about problems from a dashboard, not from an angry user.

Logs tell you what happened in one request; metrics tell you how the whole system is trending. For AI systems the numbers that matter are latency (how long calls take), token usage (how much you're consuming), and error rate (how often calls fail or time out). Watch those three and you'll catch most degradations early.

Prometheus is the standard way to collect them: your app exposes a metrics endpoint, Prometheus scrapes it on an interval, and you build custom dashboards on top to see trends over time. A dashboard nobody looks at is worthless, so put the handful of metrics that predict user pain front and center.

The point of metrics is alerting. Configure alerts on thresholds — p95 latency over budget, error rate spiking, token consumption abnormal — so the system pages you. The gotcha is alert fatigue: alert on symptoms users feel, not on every twitch, or your team learns to ignore the pager and misses the real fire.

• Drill: instrument latency, token, and error metrics for your LLM calls.
• Drill: expose a Prometheus metrics endpoint and confirm it's being scraped.
• Drill: build a dashboard showing latency, tokens, and error rate over time.
• Drill: configure an alert that fires on a realistic threshold breach.

23.4 Cost Management¶

Objective: control AI infrastructure costs before they control you.

Token-based pricing means every request has a dollar cost, and at scale small inefficiencies compound fast. Start by tracking token usage per feature so you know where the money goes. Then compare model costs — a cheaper, smaller model often handles routine work fine, and reserving the expensive model for hard cases can cut spend dramatically with no quality loss users notice.

Caching is the highest-leverage optimization. Identical or near-identical prompts don't need a fresh model call every time; cache responses and you eliminate whole categories of spend and latency at once. Layer in other techniques — trimming prompt bloat, capping output length, batching — where they pay off.

Put a floor under it with budget alerts so a runaway loop or traffic spike triggers a warning instead of a shocking invoice. The classic self-taught mistake is optimizing blind: never guess where the cost is. Measure token usage first, then cut the biggest line item.

• Drill: track token usage broken down per feature or endpoint.
• Drill: compare per-request cost across two models on the same task.
• Drill: add a response cache for repeated prompts and measure the savings.
• Drill: set a budget alert that fires when spend crosses a threshold.

Hands-On Lab: Monitoring Dashboard¶

Objective: build production observability for an LLM feature, front to back.