What LLMs actually need from data infrastructure — and where most pipelines fall short

The executive churn inside major AI labs over the past months has surfaced a pattern worth examining. It is not a story about personalities. It is a structural signal: the hardest unsolved problem in large language model development is not the model itself. It is the data infrastructure that feeds it.

Data center leadership at top-tier AI organizations has become one of the most turbulent roles in the industry. That instability reflects something real. Scaling a model requires scaling a data supply chain. And most organizations — even well-resourced ones — are discovering that the supply chain is where ambitions go to stall.

For teams building on top of LLMs, or training domain-specific models using public web data, the same friction appears at a smaller scale with equally damaging consequences.

The infrastructure gap nobody scoped correctly

When organizations plan an LLM project, they tend to budget for compute and model fine-tuning. They underestimate — systematically — the cost and complexity of the data layer underneath.

What does that layer actually involve?

• Continuous access to a defined slice of the public web at a specified frequency
• Language detection and deduplication before the data enters any model
• Provenance tracking so every text fragment can be traced to its source and timestamp
• Format normalization across heterogeneous sources
• Legal framing that makes the processing defensible under applicable law

Each of these steps has failure modes that only become visible in production. Deduplication logic that works at one million documents breaks differently at one billion. Language detection models trained on balanced corpora misclassify code-mixed content at rates that distort downstream outputs. Provenance metadata, when missing, makes it impossible to audit model behavior or respond to data rights disputes.

None of this is hypothetical. These are the recurring failure patterns in production TDM pipelines.

Why legal framing is not optional infrastructure

Organizations sometimes treat the legal layer as something to sort out after the technical pipeline is running. That sequencing is expensive.

Under Art. 4 of Directive (EU) 2019/790, Text and Data Mining of publicly accessible sources is a permitted activity, provided the access was lawful and the rights holder has not explicitly opted out. This exception applies broadly — including for commercial purposes — and it covers the kind of systematic analysis of public web data that LLM training requires.

But "permitted" does not mean "unstructured." The exception applies to the analytical process, not to redistribution of original content. The processed data — embeddings, derived signals, statistical patterns — is what the regulation shields. Raw reproductions of source texts are not covered by the same logic.

This distinction matters enormously for pipeline design. It determines what gets stored, in what form, for how long, and under what access controls. Organizations that conflate "we can process it" with "we can keep a copy of it" are building on a legal assumption that will not survive scrutiny.

Art. 67 bis of Spain's Intellectual Property Law (LPI) mirrors this framework at the national level, adding a further layer of specificity that applies when processing sources within that jurisdiction.

Building the legal structure into the pipeline architecture from the start — not as a compliance annotation but as a design constraint — is the difference between infrastructure that scales and infrastructure that creates liability as it grows.

What rigorous TDM practice looks like in a data-for-LLM context

There is a gap between what teams think TDM means and what it requires operationally when the output is training data for a language model.

Scope definition before access. The universe of public Internet is not a monolith. Different domains, languages, publication cadences, and content types produce data with very different utility for different model objectives. Defining the scope precisely — by topic, geography, language, source type, and time window — before any processing begins is the step that most projects skip or do under-specify.

Signal density over volume. More data does not automatically produce better models. Data that is topically dense, temporally consistent, and linguistically clean produces better training signal than a high-volume corpus full of near-duplicates, machine-translated filler, and SEO-generated padding. The filtering decisions made during preprocessing determine model behavior more than the raw volume of text ingested.

Temporal consistency as a first-class requirement. LLMs trained on snapshots of the web inherit the temporal biases of those snapshots. If the data pipeline cannot maintain consistent coverage of a defined source set over time, the model's representation of the world will drift in ways that are difficult to detect and harder to correct. Temporal consistency is not a nice-to-have; it is a prerequisite for models that need to reason about current states of the world.

Auditability built in, not bolted on. Regulatory pressure on AI systems is increasing. The ability to answer "what data was used to produce this output, and when did that data enter the pipeline?" is becoming a requirement, not just for legal defense but for scientific reproducibility. Pipelines that do not instrument provenance at ingestion time cannot retrofit it later without rebuilding from scratch.

The bottleneck is not the model

The organizations experiencing the most friction in LLM development right now are not failing at the model architecture level. They are failing — or stalling — at the data supply chain level. Compute is available. Architectures are well understood. The blocking problem is access to sufficient quantities of high-quality, legally defensible, continuously updated public web data processed with the rigor that training and fine-tuning actually require.

This is the problem that TDM infrastructure exists to solve. Not as an abstraction, but as a concrete operational layer: access, processing, filtering, normalization, provenance, and legal framing, running continuously against a defined slice of the public web.

TrawlingWeb operates this infrastructure at scale, applying TDM methodology under the Art. 4 framework to deliver derived analytical signals — not raw content — that organizations can use to build, fine-tune, and monitor AI systems responsibly.

The teams that will have functional, defensible LLM pipelines in twelve months are the ones building the data layer with the same rigor they apply to model evaluation. The bottleneck has always been there. It is just becoming harder to ignore.