Open datasets for training language models
The availability of high-quality text collections has become one of the determining factors in the development of artificial intelligence. Where building language models was initially seen primarily as a question of compute and network architecture, the center of gravity is shifting increasingly to the input side. The volume of freely available, human-written text on the public internet is, after all, not inexhaustible.
More and more platforms and publishers are shielding their data from automated indexing. Websites install barriers against bots, adjust their terms of use or place content behind a paywall. At the same time the public web is steadily being polluted with texts generated by language models themselves. This double movement means collecting and preparing high-quality open data is no longer a side issue but a core activity in AI research.
Categories of open data sources
Anyone looking for texts to train a language model encounters widely varying types of source. Every category has specific characteristics in terms of provenance, licensing law and editorial quality. In practice, different types are combined to achieve a balanced mix.
The distinction between these categories determines not only which processing steps are needed but also what kind of language use the eventual model learns. The main sources can be grouped as follows:
- Large-scale web collections: Automated crawls of the public internet. These sources offer an enormous variety of topics and writing styles, but also contain a lot of noise, formatting errors and unwanted content.
- Public domain and heritage institutions: Digitized books, historical newspapers, archives and academic works whose copyright has expired. These sources often offer high grammatical and editorial quality, but may contain dated language.
- Volunteer collections: Knowledge bases, encyclopedias and collaborative projects where a community of users writes and checks articles. Structure and reliability are often high, but coverage varies widely by field.
- Government data and official documentation: Policy papers, parliamentary records, statutory texts and research reports from public bodies. These texts are formal in style, legally and administratively accurate, but often lack everyday colloquial language.
| Category | Typical provenance | Linguistic quality | Cleaning required |
|---|---|---|---|
| Web collections | Public websites and forums | Variable to low | Very intensive |
| Public domain | Libraries and archives | High (formal) | Moderate (OCR correction) |
| Volunteer projects | Open encyclopedias | High (structured) | Limited |
| Government data | Ministries and councils | High (legal) | Limited (style filter) |
Three different meanings of 'open'
The label 'open' is used frequently in the context of training data, but in practice it has different meanings. A dataset developers call open is not automatically open in a legal or practical sense. It is essential to distinguish three levels of openness:
- Free to download: The file sits on a public server or distributed network and can be retrieved without payment or an account. This says nothing about what you may then do with it.
- Free to reuse: The rights holder has given permission to analyze, modify or store the data — for scientific research or education, for instance.
- Free to reuse for commercial training: The license explicitly permits the data to be used for developing commercial products, including training profit-making models.
In practice these three definitions rarely coincide. A file can be downloaded easily from a server while the material is protected by copyright and commercial exploitation is not permitted. Developers pursuing transparency therefore have to analyze precisely under which conditions each subset is offered. For more background on how these usage rights relate to model weights, see the article on the licensing debate around open models.
The difference between inspectability and provenance
There is a fundamental difference between a dataset you may inspect as a researcher and a dataset whose provenance is documented per individual document. Inspectability only means the text files are accessible. You can see which words they contain and how the texts are structured.
Documenting provenance (also called provenance ) goes much further. It means it is recorded exactly which source every text fragment came from, when it was collected, which license applied at the moment of collection, and which edits or filters were applied to that specific document.
Renewed focus on data provenance: Without accurate provenance data per document, a dataset is a black box. If a particular source turns out afterwards not to have been permissible, then without provenance information the entire collection has to be discarded or the model retrained.
Detailed provenance administration is rare because it places heavy technical demands on the collection process. Storing metadata per document increases the size of the administrative databases and requires strict protocols during crawling. Yet this documentation is becoming increasingly valuable. It lets developers remove specific documents in a targeted way when privacy problems or copyright objections arise, without touching the rest of the collection.
The position of smaller language areas such as Dutch
For major world languages, collecting enough training material is mainly a filtering question: there is an abundance of text, and the challenge is separating noise from usable material. For smaller language areas such as Dutch, the situation is different. The absolute volume of available digital text is many times smaller, which brings specific challenges.
When a language area has less material, a model cannot simply rely on the law of large numbers. Simply crawling the public web in Dutch quickly yields a relatively high share of poorly translated product descriptions, automated news items and search-engine-optimized filler. If a model is trained on these texts, it learns subtle grammatical errors and unnatural sentence structure.
For a language such as Dutch, the quality of the source material therefore weighs considerably more heavily than sheer volume. Initiatives in smaller language areas consequently focus more on targeted collection: actively digitizing archives, opening up government registers and working with cultural institutions. Anyone wanting to go deeper into the specific multilingual balance and the processing of Dutch will find extra context in the overview of language diversity in LLMs with an emphasis on Dutch.
The processing pipeline: from raw text to usable data
Collecting raw text is only the first step in a long and labor-intensive process. Raw web text is unsuitable for feeding directly to a language model. It contains navigation menus, cookie notices, advertising blocks, arbitrary punctuation and truncated code. Editing and cleaning the data therefore accounts for the lion's share of the effort in preparing a dataset.
A typical processing pipeline goes through several successive phases:
- Extraction and markup removal: Stripping HTML tags, scripts and formatting elements to leave only the textual content.
- Language identification: Determining the language per paragraph or document, so texts can be classified correctly and foreign languages filtered out of a specific subset.
- Quality filtering: Applying heuristics and statistical filters. Documents with excessive repetition of specific words, an unusual ratio between punctuation and letters, or extremely low information density are removed.
- Removal of personal data: Detecting and anonymizing sensitive information such as phone numbers, email addresses, national identification numbers and physical addresses.
These steps often introduce unintended distortions in the data. If a quality filter is tuned too strictly, specific dialects, professional jargon or informal conversations can be wrongly marked as noise and deleted. A detailed analysis of how data selection choices carry through into a model's behavior can be found in the guide on training data and bias.
Deduplication: the underrated workhorse of data preparation
One of the most critical but often overlooked parts of data processing is deduplication. On the public internet, the same texts occur on a large scale. Think of press releases reproduced in full by hundreds of news sites, standard terms and conditions on web shops, or FAQs on government portals.
If these identical or near-identical texts remain in the training set unprocessed, that has harmful consequences for the trained model:
- Overfitting on specific turns of phrase: The model assigns disproportionate weight to frequently repeated texts, and starts reproducing these fragments literally instead of understanding the underlying structure.
- Wasted compute: Processing duplicate information during training costs valuable capacity without the model learning new concepts.
- Increased memorization of privacy-sensitive data: Unique data that happens to appear in several places gets anchored in the network's parameters faster through the repetition.
Detecting exact duplicates is technically simple through hash functions. The real work, however, lies in finding near-duplicates: texts that are substantively identical but differ in a few words, synonyms or punctuation marks. Advanced algorithms are used for this, converting documents into compact digital fingerprints and computing the degree of overlap.
The pollution of web collections with generated text
A recent phenomenon complicating the assembly of open datasets is the presence of synthetic content on the internet. Since automated text generation became accessible at scale, thousands of websites have been filled with articles, product reviews and blogs written without human intervention.
When an automated web crawler gathers new data, it inevitably picks up these generated texts. This leads to circular reasoning in the training process: a new model is trained on the output of earlier models. Researchers point out that this accumulation can degrade the model's linguistic variation and factual precision.
Synthetic data is not worthless by definition, however; when generated in a controlled way under strict supervision for specific tasks, it can serve as a useful supplement. The problem arises when unmanaged, generic web texts end up in a dataset unfiltered. A deeper look at the pros and cons of controlled data generation can be found in the overview of developments around synthetic data.
Testability, benchmark contamination and verifiability
Openness of training data matters not only to those who want to build a model themselves; it is equally crucial for those who want to evaluate and compare models. To determine how well a language model performs on reasoning, language knowledge or medical questions, standardized tests and questionnaires are used: benchmarks.
A major problem arises, however, when the questions and answers from such a benchmark have accidentally been included in the model's training data. In that case you are not testing the model's ability to solve a problem but simply whether the model has memorized the answers. This phenomenon is known as benchmark contamination or data contamination.
Without full visibility into the exact content and provenance of a dataset, it is impossible to check whether a high score is the result of real understanding or of contamination. Every claim about a closed model's performance therefore rests on assumptions. For a detailed explanation of how contamination is detected, see the article on benchmark contamination explained.
To increase this verifiability, researchers increasingly use open platforms where data sources and model parameters are catalogued precisely. An overview of these collection points can be found in the overview of model and dataset hubs.
The tension between openness, privacy and the right to erasure
Publishing large datasets chafes against privacy law and individual rights on several fronts. Web collections inevitably contain personal data: from names and messages on discussion forums to blog articles sharing privacy-sensitive experiences.
Privacy law gives individuals the right in many cases to request the deletion of their personal data (the right to be forgotten). With a traditional database such a request is easy to carry out: the record in question is looked up and deleted. With open datasets and trained language models, this is technically considerably more complicated.
Once a dataset has been distributed through open networks, it is virtually impossible to guarantee that every copied version worldwide is updated. Moreover, removing a text from a dataset is only half the challenge. If a language model has already been trained on that data, the patterns of that information are anchored in the network's billions of parameters.
Removing specific knowledge from an already trained model after the fact — a process known as machine unlearning — is still at an early stage of research. It requires complex interventions in the network's weights, often at the risk of degrading the model's general abilities. This explains why open data custodians increasingly emphasize preventive filtering before the training process begins.
Conclusion
Open datasets are the foundation of transparent and verifiable AI research. Building high-quality collections, however, takes far more than gathering as much web text as possible. It requires careful filtering, thorough deduplication, clear licensing models and accurate administration of every document's provenance.
As authentic human text becomes scarcer and the web is polluted with machine-generated content, the emphasis shifts from quantity to quality. For smaller language areas such as Dutch in particular, a targeted, quality-driven approach is the only way to develop representative and reliable language models.
Further reading
- The licensing debate around open models and usage rights
- Language diversity in LLMs: the challenges for Dutch
- How training data and bias affect AI output
- The role and risks of synthetic data in AI training
- Benchmark contamination explained: causes and detection
- Overview of open model and dataset hubs
