Ethics, Safety & Governance¶

The bar: you can ship an AI feature that refuses to leak PII, resists a prompt-injection attack, logs every decision it makes, and has a written plan for the day it goes wrong.

AI systems fail differently than the software you already know. A null-pointer bug crashes and you fix it. A biased model quietly denies the right people, a leaky prompt hands an attacker your training data, and neither one throws a stack trace. Nobody files the ticket because nobody sees the harm until it's already shipped to real users.

This module is about building responsibly — not because a regulation forces you to, but because you're putting software in front of thousands of veterans and the people who trust them. The engineer who thinks about failure modes before launch is the one who gets handed the important work. Everything here is a habit you build once and carry into every feature after.

25.1 Bias and Fairness¶

Objective: identify where AI bias comes from and put concrete measures in place to reduce it.

Bias is not something the model invents — it inherits it. If your training data over-represents one group, encodes historical decisions that were themselves unfair, or simply has holes where a population should be, the model learns those patterns and repeats them at scale. That's the first thing to internalize: the model is a mirror of its data, and a dirty mirror ships a dirty answer.

You can't fix what you don't measure, so fairness gets treated as a metric, not a feeling. Techniques like demographic parity and equalized odds let you check whether outcomes differ across groups you care about. Once you have a number, mitigation becomes real work: rebalancing or augmenting the training set, reweighting during training, or adjusting decision thresholds per the tradeoff you can defend.

The gotcha the self-taught engineer hits: they run the fairness check once before launch, see green, and move on. Bias drifts. Your user base shifts, the world changes, and a model that was fair in March is skewed by September. Fairness is a monitor you leave running, not a gate you pass through.

Drill: name the three sources of training bias — sampling, historical, and representation gaps; compute a fairness metric such as demographic_parity or equalized_odds on model output; apply one mitigation (reweighting, resampling, or threshold tuning); and wire a recurring fairness check into monitoring instead of a one-time audit.

25.2 Security Threats¶

Objective: recognize the attacks that target AI systems specifically and defend against them.

Traditional appsec still applies, but AI opens new doors. Prompt injection is the big one: an attacker buries instructions inside the input — a support ticket, a scraped web page, a file the model reads — and your system follows them instead of your own rules. Treat every piece of model input as untrusted, the same way you already treat form data headed for a database.

The others cluster around your data and your model as assets. Data poisoning corrupts training input so the model learns something an attacker chose. Model extraction steals your model's behavior through relentless querying, cloning what you paid to build. PII leakage is the quiet killer — a model trained on or fed personal data can regurgitate it, and a chatbot that repeats one user's email to another is a breach, not a bug.

The common mistake is trusting the model's output as if it were your own code path. If the model can be talked into an action — calling a tool, returning a record, running a query — an attacker will try to talk it into the wrong one. Constrain what the model is allowed to do, validate what comes back, and never let raw model output reach a privileged action unchecked.

Drill: write a prompt that attempts injection and confirm your guardrail catches it; separate trusted system instructions from untrusted user input; rate-limit and monitor query patterns to blunt model extraction; scrub PII from both training data and runtime prompts; and add poisoning defenses like provenance checks on training sources.

25.3 Explainability¶

Objective: make AI decisions transparent to users, auditors, and your future self.

When a model tells a user "no," they deserve to know why — and so do you, six months later, when someone asks how the system reached a decision. Explainability matters most exactly when the stakes are highest: denials, flags, rankings that affect a real person. A black box you can't reason about is a liability you can't defend.

Explanation techniques range from simple to sophisticated: surfacing the input features that drove a result, showing confidence scores, or using methods like SHAP or LIME to attribute a prediction to its inputs. But the technique is only half the job. The other half is user communication — translating "feature 7 exceeded threshold" into a sentence a human understands and can act on.

Behind the user-facing explanation sits the audit trail: a durable record of what the model saw, what it decided, and when. This is your evidence when a decision is questioned, and the raw material for debugging when behavior drifts. The mistake here is treating logs as an afterthought — by the time you need them, it's too late to start collecting.

Drill: state why explainability matters for high-stakes decisions; produce a feature-attribution explanation with SHAP or LIME; rewrite a technical explanation into plain user-facing language; and record every decision to a durable audit trail with input, output, and timestamp.

25.4 Governance¶

Objective: understand the regulations and standards that apply to AI and build to comply with them.

Governance is the paperwork that keeps you out of court and keeps users safe. GDPR reaches AI directly: users have rights over their data, including a say in automated decisions and the right to have their data forgotten — which means your system needs to actually be able to delete someone. Design for that on day one; retrofitting deletion into a model pipeline is brutal.

Standards give you a framework instead of a scramble. ISO 42001 is the AI management-system standard — a structured way to govern how AI gets built and operated. Model cards are the lightweight, practical companion: a short document that states what a model does, what data trained it, how it performs, and where it should not be used. Ship one with anything that makes real decisions.

The last piece is incident response — the plan for when, not if, something goes wrong. Who gets paged, how you shut a model off, how you notify affected users, how you do the post-mortem. The engineer who has this written before launch looks calm during the incident. The one who doesn't is improvising while users are being harmed.

Drill: map GDPR requirements — consent, automated-decision rights, and right-to-erasure — to your data flow; outline what an ISO 42001 management system covers; write a model card documenting purpose, training data, performance, and limits; and draft an incident-response plan naming owners and escalation steps.

Hands-On Lab: Safety Implementation¶

Objective: implement the production safety measures that turn these principles into running code.