Glossary & Field References¶

Glossary¶

The working vocabulary of the stack, in one place — so a cohort reads every rung the same way.

Agent

Model + harness + loop; the smallest unit that pursues a goal on its own.

Agent loop

The observe → decide → act → terminate cycle at runtime.

Augmented LLM

A model equipped with retrieval, tools, and memory.

Chunking

Splitting source text into retrievable units; the dominant RAG quality lever.

Compaction

Summarizing old turns to fit the context budget.

Confidence gating

Using token certainty to decide whether the loop iterates or exits.

Constrained decoding

Forcing output to match a grammar or schema at the token level.

Context window

The fixed token budget a model can attend to per call.

Entropy / perplexity

Aggregate measures of a model's uncertainty over its output.

Eval harness

The rig that runs benchmarks against an agent — distinct from the agent harness.

Guardrails

Safety and policy checks around a model's inputs and outputs.

Harness

The machinery around the loop: tools, context, recovery, state, streaming.

LLM-as-judge

Using a model to grade outputs at scale; prone to position, verbosity, and self bias.

Logprobs

Per-token log-probabilities; the raw signal behind confidence.

Orchestrator

An agent whose action space is other agents.

pgvector

A Postgres extension for vector similarity search.

Plan-execute

A topology that commits a plan up front, then runs the steps.

ReAct

Interleaving reasoning traces with actions each turn.

Reflexion

Self-critique stored in memory to improve the next attempt.

Reranking

Reordering retrieved candidates for precision after broad retrieval.

RAG

Retrieval-augmented generation: grounding output in retrieved passages.

Skill

A loaded-on-demand capability package with no loop of its own.

Spec-driven development

Governing both the build and the evals from one durable specification.

Structured output

Model output shaped as data your code can parse and validate.

Subagent

An agent in the child position; own window and loop, returns a distilled result.

Termination

The stop-condition problem — knowing when the loop is done.

Tool calling

A model emitting a structured request your code executes.

Trace / observability

End-to-end inspection of a run: prompts, calls, cost, latency.

Truncation

Dropping context by recency or relevance to fit the window.

Field references¶

Primary sources behind the rungs — the papers and practitioner guides worth reading in full, grouped by where they land on the ladder.

L1–L2 · Tool use & the loop¶

• ReAct: Synergizing Reasoning and Acting in Language Models. Yao et al., ICLR 2023 · arXiv:2210.03629. Interleaves reasoning traces with actions — the founding pattern for the agent loop.
• Reflexion: Language Agents with Verbal Reinforcement Learning. Shinn et al., NeurIPS 2023 · arXiv:2303.11366. Self-critique held in an episodic memory buffer — the reflexion topology, no weight updates.
• Toolformer: Language Models Can Teach Themselves to Use Tools. Schick et al., NeurIPS 2023 · arXiv:2302.04761. Foundational treatment of models deciding when and how to call a tool.

L2 & L6 · Agent & workflow architecture¶

• Building Effective Agents. Anthropic, 2024 · anthropic.com/research/building-effective-agents. The workflows-vs-agents distinction, the augmented LLM, and five composable patterns including orchestrator-workers. The best single primer on agent architecture.

L3–L4 · Context & knowledge¶

• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. Lewis et al., NeurIPS 2020. The original RAG formulation — parametric generation plus non-parametric retrieval, motivated by provenance and updatability.
• Lost in the Middle: How Language Models Use Long Contexts. Liu et al., 2023 · arXiv:2307.03172. Why position in the window matters — direct input to context-assembly and chunk-injection decisions.

L5 · Evaluation¶

• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. Zheng et al., NeurIPS 2023 · arXiv:2306.05685. Establishes LLM-as-judge and names its biases — position, verbosity, self-enhancement.
• G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment. Liu et al., EMNLP 2023 · arXiv:2303.16634. A widely used pattern for model-graded evaluation with chain-of-thought scoring.
• Evaluation Best Practices. OpenAI, developer docs. Practical eval-driven development — evaluate early and often, wire evals into CI as a merge gate.

L7 · Spec- & eval-driven development¶

• Eval-Driven Development. DeepEval & practitioner guides, 2025–26. The top-down inversion — define the standards (evals) first, then build toward them.
• LLM Evals FAQ. Husain & Shankar, 2026. The counter-view: for open-ended systems, prefer error-analysis-first over writing every evaluator up front. Read alongside the above.