How much labelled data do you need to fine-tune a model?

The instinct is to ask for tens of thousands of examples. The evidence points the other way: for adapting an already-capable model to a task or a voice, a small set of excellent examples beats a large set of mediocre ones.

What the research found

The LIMA study, published at NeurIPS 2023, fine-tuned a 65-billion-parameter model on just 1,000 carefully curated prompt-and-response pairs — no reinforcement learning, no preference data. In a controlled human-preference study, its responses were judged equivalent to or better than GPT-4's in 43% of cases. The authors' conclusion was about data quality: almost all of a model's knowledge comes from pretraining, and a small, well-chosen set is enough to teach it the task.

The reverse also holds: bad labels do real damage, and not only in training. A NeurIPS 2021 study by Northcutt, Athalye and Mueller estimated that the test sets of ten widely used benchmark datasets contain an average of at least 3.3% label errors — enough, in their analysis, to change which model appears to perform best. If your evaluation labels are wrong, you may promote the wrong model to production and never know.

What good looks like in practice

Three properties matter more than count. Coverage: the examples span the real variety of inputs, including the awkward ones, not just the easy centre. Consistency: two experts labelling the same example should produce the same label — and that agreement should be measured, not assumed, with disagreements traced back to the labelling guideline and fixed there. Provenance: every dataset should carry its documentation — where the data came from, what consent covers it, what is known to be missing — a practice formalised as datasheets for datasets in Communications of the ACM in 2021.

A working sequence

Start with hundreds, not tens of thousands: enough to cover the input variety once. Have the people who actually do the job label them, measure agreement, and revise the guideline until agreement is high. Hold out a clean slice as the evaluation set before any training happens. Then scale only what the first fine-tune proves is missing. Data budgets are better spent on expert time per example than on example count.