Training data

What training data is

A training dataset is a collection of examples. In supervised learning each example pairs an input (an image, a sentence, a row of measurements) with a label giving the correct output. The model adjusts its internal parameters until its outputs match the labels well. Whatever patterns are in the examples — including accidental ones — are what the model learns.

Splitting the data

• Training set: the examples the model's parameters are fitted to.
• Validation set: held back from fitting and used to make design choices, such as how large the model should be.
• Test set: used once, at the end, to estimate performance on new data. It must not influence any choice about the model.

A classic example is the MNIST database of handwritten digits described by LeCun and colleagues in 1998: they trained on 60,000 examples and tested on 10,000 images. For one of the two collections the digits came from, the first 250 writers went into training and the other 250 into testing, so the test checked handwriting the model had never seen.

How data goes wrong

• Unrepresentative samples: in 2018, Buolamwini and Gebru found two facial-analysis benchmarks were 79.6% (IJB-A) and 86.2% (Adience) lighter-skinned subjects, so performance on darker-skinned faces was measured on comparatively few examples.
• Wrong labels: a 2021 study estimated at least 3.3% label errors on average across the test sets of 10 popular datasets, and at least 6% in the ImageNet validation set.
• Out-of-date data: NIST warns training data can become stale or detached from the context where the system is later used.
• Contamination: test questions can leak into web-scale training data simply because they are published online, which can make scores look better than they are. The GPT-3 authors measured this: the effect was small on most benchmarks, but on a few it could be inflating results, so they flagged or dropped those.

NIST groups AI bias into three categories — systemic, computational and statistical, and human-cognitive — and stresses that each can arise without anyone intending to discriminate. A skewed dataset is an example of how bias can enter quietly, through who and what happened to be collected. The 'Biased Training Cards' activity shows the effect with paper cards: a learner who is only shown red apples tends to write a rule such as 'red and round', which then rejects green apples and accepts a red ball.

Worked example: auditing a dataset

Suppose a face dataset has 1,000 images and 862 of them are of lighter-skinned people. The share is 862 / 1,000 = 86.2% — the same proportion reported for the Adience benchmark. Only 138 images (13.8%) would then represent everyone else. A model can show a high overall accuracy while being far less accurate for the under-represented group, because that group contributes little to the overall score. The fix starts with measuring composition and reporting accuracy separately for each group, not just in aggregate.

Documenting data

Gebru and colleagues proposed 'datasheets for datasets': just as an electronic component comes with a datasheet, every dataset should record its motivation, composition, collection process and recommended uses. NIST's framework similarly says that maintaining the provenance of training data supports transparency and accountability, and notes that training data may be subject to copyright.