Synthetic Data
Training and evaluation data generated by models
Data produced by a model (or a program or simulator) instead of collected from people: instruction-answer pairs, preference data, test questions. Cheap and fast to scale, but its quality, diversity and legal status need checking separately.
Synthetic data is data that is generated rather than collected from the real world. In the LLM era that usually means text a model wrote: instruction-answer pairs, multi-step reasoning examples, preference labels between two answers, examples of rare classes for a classifier, or test questions to probe a system.
Main uses:
- Instruction and fine-tuning data: generate question-answer pairs that are expensive to label by hand with a strong model, then fine-tune a smaller one.
- Distillation: a large "teacher" model's outputs become a small "student" model's training data. Generating synthetic data is often the first step of distillation.
- Preference and reward data: generate answer pairs for RLHF-style methods and have a model or a person score them.
- Evaluation: when real user questions are scarce, generate test scenarios, edge cases and attack attempts; see evals.
The work that shaped the field is Self-Instruct: Wang et al. (2022), "Self-Instruct: Aligning Language Models with Self-Generated Instructions" (ACL 2023). It starts from a small seed set of hand-written tasks (175 in the released repository); the model generates new instructions, inputs and outputs from them, invalid examples and ones too similar to existing ones are filtered out, and the rest go back into the pool for the next round. The result: about 52K instructions with 82K instances. GPT-3 fine-tuned on this data improved by 33% absolute over the original model on Super-NaturalInstructions, on par with InstructGPT-001.
The main risk is model collapse: Shumailov et al. (2024), "AI models collapse when trained on recursively generated data" (Nature 631, 755–759). The paper shows that indiscriminate use of model-generated content in training causes irreversible defects: as models are trained on their own output generation after generation, the tails of the original distribution (rare but real examples) disappear. The practical lesson: mix synthetic with real data, label its provenance and measure diversity.
On the legal side, using a model's output to train another model depends on the generating model's terms of use. For example, Anthropic's commercial terms prohibit using the services to build a competing product, "including to train competing AI models"; the Gemini API terms prohibit using the services "to develop models that compete". The Llama 3.1 license, by contrast, allows training other models on outputs but requires a distributed model's name to begin with "Llama".
Like practice questions a teacher writes for exam prep. When real exam questions are scarce, writing hundreds of similar ones helps the student a lot.
But if the teacher always writes in the same style, the student only learns that style and is thrown by an unusual question on the real exam. Worse, if the student becomes a teacher and trains new students only on questions they wrote themselves, the questions get a little less varied each generation. That is model collapse.
An e-commerce company wants a small model that classifies customer emails, but it has only a handful of examples of rare combinations like "return + damaged item + international shipping".
- With the category list and 30 real (anonymized) emails, they have a strong model write 200 emails per rare category, varying tone, length and typos.
- They drop repetitive and near-duplicate examples with a similarity measure and spot-check a sample by hand.
- They add the synthetic examples to the real data and train.
- They evaluate only on real emails the model has never seen. On a synthetic-only test set the model could look better than it is, because it learned to recognize the generating model's style.
Evals follow a similar flow: generating question-answer pairs from documents for a RAG system and scoring answers with LLM-as-judge is a fast start, but people need to verify a sample of the generated questions.
- Labeled data is scarce and expensive, especially for rare classes and edge cases
- You want to transfer a large model's behavior to a smaller, cheaper one (distillation) and the license allows it
- Expanding an eval set, generating attack and edge-case scenarios
- As test and development data when you can't share real personal data (together with privacy checks)
- The generating model's terms forbid using output to train a competing model
- Real data is plentiful and clean; synthetic data usually goes next to real data, not instead of it
- As the only source for final success measurement; the final evaluation should run on real data
Underestimating model collapse
Shumailov et al. (2024) showed that indiscriminate, generation-after-generation training on model output erases the tails of the distribution. Don't throw away real data; label synthetic data and track the mix.
The diversity illusion
Generating thousands of examples from one prompt often yields variations of the same pattern. Deliberately vary axes like persona, topic, difficulty and format, and filter near-duplicates.
Error amplification
If the generating model hallucinates, false facts enter the training data and the student repeats them confidently. In verifiable domains (code, math), check outputs with tests or computation.
Not reading the terms of use
Anthropic's and the Gemini API's terms prohibit developing competing models; open-weight model licenses have conditions too, such as naming. Read and keep the terms before generating data.
Assuming synthetic data is anonymous
A model can copy names, addresses or sentences from real records you gave it as examples. Anonymize real data you put in the prompt and scan generated data for personal data.