Tier III — Making It Real¶

The bar: you can ground the agent in real knowledge and prove it works with numbers.

L4 — Knowledge & Confidence¶

COND + CORE — The agent knows things it wasn't trained on — and knows when it doesn't.

Two distinct capabilities share this rung because both are about grounding: one grounds the agent in external knowledge (retrieval — conditional; skip it entirely for pure tool/code agents), the other grounds the loop's decisions in the model's own certainty (confidence — core; the completion of the termination work from L2 using the token-level signal).

Ingest → Chunk → Embed → Store (pgvector) → Retrieve → Rerank → Inject → Model

Retrieval pipeline (conditional). Chunking is the dominant quality lever; reranking is the precision multiplier.

Model output → logprobs / entropy → high → EMIT
                                    low → LOOP BACK

Confidence gate (core). Token certainty becomes a loop-control decision — the stop condition promised at L2.

Retrieval — the conditional half¶

The pipeline: ingest → chunk → embed → store → retrieve → rerank → inject, each stage with its own failure modes. The dominant lever is that chunking dominates quality — size, overlap, and structure-aware splitting matter more than model choice, and most bad retrieval is bad chunking. Embeddings and vector stores give you similarity search and its limits; reranking is the quality multiplier — retrieve broadly, rerank precisely, trading recall for precision.

Context injection connects back to L3's budget — retrieved chunks cost window — and you learn the distinction between retrieval-augmented generation and retrieval as a tool the loop calls. Crucially, when not to use RAG: if the knowledge fits in context, or the task is tool/computation-bound, retrieval is pure overhead.

Confidence — the core half¶

Logprobs as a signal, not a curiosity — token-level log-probabilities expose how certain the model is, per token — and entropy / perplexity over the distribution as an aggregate measure. The payoff is confidence as loop control: low confidence → iterate (gather more, reformulate, ask a human); high confidence → exit and emit. This is the confidence-threshold stop condition promised at L2.

With it comes the calibration caveat — confidence is not correctness; you learn where token confidence misleads and how to combine it with external verification — and constrained decoding revisited: enforcing structure at the token level as a reliability mechanism the harness owns.

Concepts to master¶

• The retrieval pipeline — ingest/chunk/embed/store/retrieve/rerank.
• Chunking strategy — size, overlap, structure-aware; the dominant lever.
• Embeddings + vector search — recall vs. precision.
• Reranking — the precision multiplier.
• RAG-as-tool vs. augmentation — and when to skip RAG entirely.
• Logprobs, entropy, perplexity — confidence as a measurable signal.
• Confidence-gated control — retry / escalate / exit.
• Calibration — confidence vs. correctness.
• Hybrid search — dense vectors + sparse (BM25) for recall.
• Metadata filtering & routing — narrow the space before retrieval.
• Chunk provenance — carry citations through to the answer.
• Self-consistency — sampling multiple traces to estimate certainty.

Clear it when¶

• ☐ Retrieval measurably beats no-retrieval on your task — you have the numbers.
• ☐ Your chunking strategy is a deliberate choice you can defend, not a default.
• ☐ The confidence gate demonstrably catches low-quality turns before they reach the user.
• ☐ You can say where token confidence is trustworthy for your task and where it isn't.

Where people get stuck¶

• ✗ Bolting RAG onto a loop they don't understand — why you did L2/L3 first.
• ✗ Blaming the model for what is actually a chunking problem.
• ✗ Logprobs as a dashboard number instead of a decision input.
• ✗ Assuming high confidence means correct.

L5 — Proof¶

CORE — You can say "it works" and back it with numbers.

Everything up to here can be demoed. None of it can be trusted until it's measured. This rung is the line between someone who made an impressive demo and someone who operates a system — and it's where many engineers stall, because evaluation is less fun than building and more necessary than anything else. Two halves: the eval harness (offline proof) and observability (online proof).

Agent harness	Eval harness
Runs the agent in production — the loop, tools, context, streaming.	Runs benchmarks against the agent — datasets, judges, regression gates.

Same word, different rig. One executes the agent; the other measures it. Offline evals gate the build; observability watches production.

Evaluation — the other harness¶

Eval datasets tied to the spec — your test set encodes what "done" means, drawn from the same source of truth as the build (foreshadowing L7). Offline eval types: task-completion, exact and semantic accuracy, and critically regression testing, so a change that fixes one thing doesn't silently break three. LLM-as-judge for grading at scale, along with its failure modes — length bias, position bias, self-preference — and how to constrain and validate the judge.

Adversarial / red-team evals probe how it breaks under hostile or edge input, and safety evals are dedicated tests for the behaviors your system must never exhibit. Keep the eval-harness-vs-agent-harness distinction sharp: same word, different rig — one runs the agent, the other runs benchmarks against it.

Observability — operating it¶

Tracing makes every run inspectable end-to-end — prompts, tool calls, retrievals, decisions. Cost and latency monitoring per run and per component is the operational reality of shipping. Structured logging and online / production evals grade real traffic, not just the test set, and feedback capture closes the loop with real user signal — the input to L7's revision.

Concepts to master¶

• Spec-derived datasets — the test set encodes "done."
• Eval types — task-completion, accuracy, regression.
• LLM-as-judge — its biases and mitigations.
• Adversarial + safety evals — hostile input and must-not behaviors.
• Eval harness vs. agent harness — same word, different rig.
• Tracing — end-to-end run inspection.
• Cost/latency + structured logs — the operations layer.
• Online evals + feedback capture — grading real traffic.
• Golden datasets — curated, versioned sets that encode "good."
• Component vs. end-to-end — grade sub-agents and the whole system.
• Judge calibration — align the LLM judge against human labels.
• Cost/latency as criteria — budgets are pass/fail, not afterthoughts.

Clear it when¶

• ☐ A change to a prompt or the harness produces a measured delta, not a vibe.
• ☐ You catch a regression before it ships.
• ☐ You can explain any production failure from traces alone.
• ☐ Your evals derive from your spec, not from what was convenient to test.

Where people get stuck¶

• ✗ "It looked good in the demo" — anecdote as evidence.
• ✗ No regression suite — every fix risks a silent break.
• ✗ An unvalidated LLM judge whose biases you inherit.
• ✗ Evals disconnected from the spec — passing them proves little.