Testing AI Systems¶

The bar: you can prove an AI feature works before it ships — unit tests on the Python, contract tests on the API, load tests under real traffic, and an evaluation harness that catches a model quietly going stupid.

Testing normal code is a solved problem: same input, same output, assert and move on. AI systems break that contract. A model returns different words every call, an upstream API times out, a prompt that worked last week starts hallucinating after a version bump. You cannot eyeball your way out of that at scale, and you cannot ship it to thousands of veterans on a hope.

This module builds the full ladder — from the fast unit test that runs on every commit to the evaluation suite that scores whether the model's answers are actually good. The goal is confidence you can defend, not a green checkmark you crossed your fingers for.

22.1 Python Testing¶

Objective: test Python AI code effectively, including the parts that talk to models and networks.

pytest is the standard runner. You configure it once in pyproject.toml or pytest.ini — test discovery paths, markers, and default flags — so the whole team runs the suite the same way. Fixtures are pytest's best idea: a function decorated with @pytest.fixture builds a piece of setup (a client, a temp database, a fake payload) once and hands it to any test that names it as an argument. Factories go a step further, generating fresh test objects on demand so tests don't fight over shared state.

The part that trips up self-taught engineers is the network. Your tests must never call the real OpenAI or Anthropic API — it's slow, it costs money, and it's non-deterministic, which means your suite goes red for reasons that have nothing to do with your code. Mock the boundary: patch the client with unittest.mock or monkeypatch and return a canned response. You're testing your logic around the model, not the model itself.

AI code is heavily async, so you'll reach for pytest-asyncio and mark coroutine tests with @pytest.mark.asyncio. Finally, run coverage (pytest --cov) to see which branches never execute — but treat the number as a map of untested risk, not a score to game. Ninety percent coverage of the happy path still ships the error handler that crashes in production.

Drill:

• Configure pytest in pyproject.toml with markers and testpaths.
• Write a @pytest.fixture and a small factory that builds test objects.
• Mock an external API call with monkeypatch / unittest.mock so no real request fires.
• Test an async function with pytest-asyncio and @pytest.mark.asyncio.
• Generate a coverage report with pytest --cov and read the missed lines.

22.2 API Testing¶

Objective: test FastAPI endpoints comprehensively, from a plain GET to a streamed token feed.

FastAPI ships TestClient, a wrapper around your app that lets you fire requests in-process — no server, no ports, milliseconds per call. You assert on status codes, JSON bodies, and headers exactly as a real client would see them. This is where you lock down contracts: the shape of a response, the error you return on bad input, the 401 on a missing token.

Streaming endpoints need special handling. An LLM endpoint often returns Server-Sent Events or a chunked stream, so you test it by iterating the response and asserting the sequence of chunks arrives correctly and terminates cleanly. Authentication is its own test surface: prove that a valid token gets through, an expired or missing one gets rejected, and a user can't reach data they don't own.

Integration tests wire several pieces together — endpoint, business logic, a test database — to catch the bugs that unit tests miss because they mocked everything. The common mistake is stopping at unit tests and assuming the seams hold. They don't. The bug is almost always in the wiring.

Drill:

• Hit an endpoint with TestClient and assert status, JSON, and headers.
• Iterate a streaming response and assert the chunk sequence and clean close.
• Test the auth path: valid token passes, missing/expired token returns 401.
• Write one integration test spanning endpoint → logic → test database.

22.3 Load Testing with K6¶

Objective: verify the system holds up under realistic and peak traffic.

Functional tests prove one request works. Load tests prove a thousand concurrent requests work. K6 is a load-testing tool where you write scenarios in JavaScript: a default function is the work each virtual user performs, and the exported options object defines how many virtual users (VUs) run and for how long. Scenarios let you model different traffic shapes — a steady ramp, a sudden spike, a soak test that runs for an hour.

The teeth of a K6 test are checks and thresholds. Checks are per-request assertions (status is 200, body isn't empty). Thresholds are pass/fail gates on aggregate metrics — for example, 95th-percentile latency must stay under 500ms, error rate under one percent. If a threshold is breached, K6 exits non-zero, which is exactly what you want so the run fails a pipeline instead of quietly logging a warning.

Streaming endpoints matter here too: a model that responds fine for one user can fall over when fifty are streaming tokens at once, because each open connection holds resources. Wire the whole thing into CI so a performance regression is caught on a pull request, not by a veteran watching a spinner in production.

Drill:

• Write a K6 script with a default function and an options block.
• Set stages to ramp virtual users and define a load scenario.
• Add check() assertions and a thresholds gate on p95 latency and error rate.
• Load-test a streaming endpoint under concurrent VUs.
• Run K6 in CI so a breached threshold fails the build.

22.4 LLM Output Evaluation¶

Objective: evaluate AI-generated content systematically instead of by vibes.

This is the part that has no equivalent in normal software. The endpoint returned 200 and valid JSON — but was the answer correct? Relevant? Made up? Evaluation frameworks give you structure for asking those questions repeatably. You build a dataset of inputs with known-good expectations and score every model output against it, so a prompt change or model upgrade gets measured, not guessed.

Three axes carry most of the weight. Factuality checking asks whether the claims in the output are true and grounded in the source material — critical for anything retrieval-backed, where hallucination is the failure mode that erodes trust fastest. Relevance scoring asks whether the answer actually addressed what was asked. Both can be automated, often using a second LLM as a judge, but automation has blind spots.

That's why human evaluation stays in the loop: a person rating a sample of outputs catches the subtle wrongness a scoring rubric misses. The mature setup is an automated evaluation pipeline that runs on every meaningful change and flags regressions, with humans spot-checking the edges. The mistake is treating one good demo as proof — a model that answered five questions well can still fail the sixth in a way that matters.

Drill:

• Stand up an evaluation framework with a labeled input/expected dataset.
• Write a factuality check that verifies claims against source material.
• Score output relevance against the original query.
• Run a human evaluation pass on a sampled batch and record ratings.
• Automate the eval as a pipeline that flags regressions on each change.

Hands-On Lab: Test Suite¶

Objective: build a comprehensive, layered test suite for a real AI application.