Art. 4 Directive 2019/790: What a TDM Compliance Audit Actually Looks Like

Most organisations that do Text and Data Mining know Art. 4 exists. Few have mapped what it demands from their infrastructure in operational terms. The gap between knowing the legal text and running a defensible TDM programme is where real risk lives.

Art. 4 of Directive (EU) 2019/790 — transposed in Spain as Art. 67 bis LPI — grants any person lawful access to publicly available content the right to perform TDM on that content. That sounds permissive. It is, within strict boundaries. Those boundaries are not abstract: they show up in system design, data lifecycle policies, and access logs. When a rights holder challenges your TDM operation, they will not ask whether you read the directive. They will ask for evidence.

This post is about what that evidence looks like — and what a structured compliance audit of a TDM pipeline actually checks.

The Four Operational Questions Art. 4 Forces You to Answer

The directive does not just grant a right. It implicitly demands that any organisation relying on it can answer four operational questions at any point in time:

1. Was access lawful at the moment of processing? Lawful access means accessing the source under the same conditions any other user would — no credential bypassing, no rate-limit circumvention, no exploitation of misconfigured access controls. Your system must be able to show, per domain and per processing session, that access complied with the source's publicly declared terms.

2. Was the content publicly available? Art. 4 only covers publicly available content. Content behind paywalls, authentication walls, or contractual access restrictions falls outside the exception. A compliant infrastructure must log the access tier of every source at processing time — not retrospectively.

3. Was retention limited to what is technically necessary? This is the most frequently overlooked condition. The directive explicitly allows rights holders to apply technical measures to prevent retention beyond what the TDM purpose requires. If your system retains raw source content indefinitely, you are not operating within the exception — you are building an archive, which is a different legal category entirely.

4. Can you demonstrate the analysis is derived, not reproduced? The output of a TDM process must be derived analysis — signals, patterns, aggregations, structured insights — not a reproduction of the original content. The distinction matters in court.

What a Compliance Audit Actually Checks

A TDM compliance audit is not a legal review of your contracts. It is a technical and procedural inspection of your pipeline. Auditors — whether internal, external, or in response to a legal challenge — will typically examine the following:

Access layer logs. Can you produce a timestamped record showing that each source was accessed under lawful conditions? Gaps in logs, or logs that do not capture the access tier, are immediate red flags.

Source classification records. Does your system maintain a live or near-live classification of sources by access type (fully public, login-required, mixed)? This classification must predate the processing run, not be reconstructed after the fact.

Retention and deletion policies. What is the maximum retention period for raw intermediary data? Is that period documented, enforced programmatically, and auditable? Many pipelines retain data in staging environments well beyond operational need — this creates exposure.

Output format analysis. Do your outputs contain extended verbatim extracts from source content, or are they genuinely derived — counts, vectors, entity-tagged summaries, trend signals? The further the output is from the original text, the stronger the TDM claim.

Opt-out compliance records. Art. 4(3) allows rights holders to reserve their content against TDM via machine-readable means (e.g., robots.txt directives, explicit metadata flags). A compliant system must demonstrate that it respects those reservations and that it checks them at processing time, not just at onboarding.

Where Pipelines Fail the Audit

The most common failure points are not in the legal framework — they are in the engineering.

Immutable raw data lakes. Systems designed for reliability often prioritise immutability. Raw content is written once and never deleted. That design choice, made for good engineering reasons, directly conflicts with the "technically necessary retention" requirement of Art. 4.

Static source classifications. A source classified as fully public in 2023 may have introduced a paywall or authentication layer since. Pipelines that do not refresh source classifications regularly cannot prove lawful access at the time of each processing run.

Verbose intermediary outputs. Many pipelines produce intermediary files — enriched raw content, annotated text corpora — that sit somewhere between source content and derived analysis. Those files are the grey zone. If they contain sufficient original expression to be recognisable as reproductions, they undermine the TDM defence.

No opt-out refresh cycle. Robots.txt files change. Metadata flags change. A pipeline that reads opt-out signals once at setup and never refreshes them is operationally non-compliant, even if it was compliant on day one.

Building the Evidence Layer Into the Pipeline

Compliance is not a documentation exercise performed after a legal challenge arrives. It is a design property of the processing infrastructure.

This means building the evidence layer in from the start: timestamped access logs that capture the access tier, automated source reclassification cycles, programmatic enforcement of retention windows, and output validation checks that flag verbatim density above defined thresholds.

Organisations that operate at scale — processing signals from tens of thousands of public sources continuously — cannot manage this manually. The compliance logic must be embedded in the pipeline itself, not layered on top as a periodic review.

At TrawlingWeb, the infrastructure is designed around this principle. Processing operates under the Art. 4 / Art. 67 bis LPI framework, with derived analysis as the output layer — not raw content redistribution. The distinction is not cosmetic; it is structural.

The Audit Is Coming, Whether You Schedule It or Not

Rights holders are increasingly aware of Art. 4's boundaries — and of the conditions that must be met for the exception to hold. Enforcement actions in this space are not hypothetical. They are already happening in multiple EU jurisdictions.

The organisations that will be in the strongest position are those that can produce, on short notice, a documented and auditable record of how their TDM pipeline operates. Not a policy document. Not a legal opinion. An operational record.

If your pipeline cannot produce that record today, the time to build it is not after you receive a legal notice. It is now.