L4-M1 / lesson 2 of 2

Test subgroups and justify residual risk

The development team reviews apparently strong test results.

10–15 min · 0/4 steps complete

Start here / A workplace decision

Test beyond the headline metric

The development team reviews apparently strong test results.

Duplicate CVs occur in both training and test sets.

Before you read / Think it through

In “Test beyond the headline metric”, what would you check or do first?

Pause and form your answer, then compare the reasoning. This reflection is not scored.

Compare the reasoning

Remove leakage and repeat a valid separated evaluation.

The ideas below give you a method for this decision. Then work through the rest of this case and try the decisions yourself.

Part 1 of 4 Learn the ideas

Choose a topic0 of 5 read

Training validation and test separation

Use appropriate separation for development and final evaluation.

Detect duplicates and information leakage.

In the workplace

Candidate-level leakage can occur even when rows differ. Split at the relevant entity or time level and document the choice.

Remember this

Keep test use controlled and traceable.

The workplace case

YOUR CASE / RISK MANAGEMENT AND DATA GOVERNANCE

A provider is developing a high-risk recruitment-ranking system. Its dataset reflects past hiring decisions, has few examples from one applicant group and contains duplicate CVs across training and testing. Aggregate accuracy looks strong, but a pilot shows unequal false-rejection rates.

Quick reference / five key ideas

Training validation and test separation: Keep test use controlled and traceable.
Representativeness and missing data: Apply appropriate, context-sensitive quality measures.
Bias detection with sensitive data: Apply access, reuse, security and deletion safeguards.
Metrics and subgroup analysis: Report uncertainty and sample limitations.
Residual risk and release: Connect release conditions to post-market monitoring.