Training data requirements: match your dataset to the right stage of model development
The same corpus can be marginal for pre-training and valuable for evaluation. Deciding which stage your data suits changes how you describe it and which buyers to expect.
One corpus, three stages
- Pre-training Volume, diversity, cleanliness.
- Fine-tuning Instruction pairs and expert labels.
- Evaluation Small, precise, never published.
Stage fit
| Stage | What matters | Example dataset |
|---|---|---|
| Pre-training | Volume, diversity, text cleanliness | A trade publisher's back catalog with clear author agreements |
| Fine-tuning | Instruction-style pairs, expert labels | Resolved support tickets with verified solutions, de-identified |
| Evaluation | Small, precise, never published | Expert-written exam questions in a regulated profession |
Properties to describe
- Record count or token/hour/image counts and time span.
- Human-created vs machine-generated share; whether any content came from AI tools.
- Deduplication and cleaning steps.
- Known gaps, skews and labeling error rates if measured.
'Never published' is a feature only if provable
Evaluation data is valuable because models haven't seen it. If you claim a set was never online, document where it was stored and who had access.
How this affects value
A small evaluation set can command more attention than a large corpus when its unpublished status is documented.
See what shapes a dataset's value