Art. 4 Directive 2019/790: Why Documentation Is Your First Line of TDM Defence

Most organisations that conduct Text and Data Mining know, at least in broad terms, that Art. 4 of EU Directive 2019/790 gives them a legal basis to work with publicly accessible sources. What fewer organisations have is a clear answer to a much simpler question: if someone challenges your TDM activities tomorrow, what exactly would you show them?

That gap — between knowing a right exists and being able to demonstrate that you exercised it correctly — is where the practical risk lives. The directive does not protect the activity in the abstract. It protects the activity as carried out by a specific actor, against specific sources, for a specific purpose. All three dimensions need to be traceable.

What Art. 4 Actually Grants (and What It Does Not)

Art. 4 of Directive (EU) 2019/790 establishes that any natural or legal person with lawful access to content may conduct Text and Data Mining for any purpose, including commercial ones. This is broader than Art. 3, which is limited to research organisations and cultural heritage institutions.

But lawful access is not a blanket pass. It means access that was legitimately obtained — not circumventing technical measures, not violating platform terms that explicitly prohibit automated processing, and not retaining more of the original content than is strictly necessary for the analytical objective.

The directive also explicitly allows rightholders to opt out via machine-readable means (typically robots.txt or equivalent declarations). That opt-out mechanism is legally binding. Ignoring it does not become legally acceptable simply because the content was publicly visible.

The right, in short, is real. But it comes with preconditions that need to be met actively, not assumed passively.

The Three Pillars of a Defensible TDM Position

Organisations that operate TDM pipelines at scale — processing signals from tens of thousands of public sources — cannot rely on a general policy document. Compliance needs to be embedded in the pipeline itself.

1. Access legitimacy logs For each source category in your pipeline, you need to be able to demonstrate that access was obtained through publicly available, non-restricted endpoints. This means keeping records of how sources were identified, what terms applied to them at the time of processing, and whether any opt-out signals were present and honoured.

2. Purpose definition per processing run Art. 4 does not grant a permanent, open-ended licence to process whatever content for whatever purpose. The analytical purpose must be definable. "Competitive intelligence for client X during Q3" is a defensible purpose. "General data enrichment" is not. Purpose documentation does not need to be exhaustive — it needs to be specific enough to map to the access you used.

3. Retention and output boundaries The directive does not permit the retention of source content in its original form beyond what the TDM process requires. What you store should be the derived analysis, not the raw original. This is not just a legal distinction — it is also the correct operational posture. Derived signals, structured outputs, and analytical conclusions are what create value. Retaining original text beyond the processing window adds liability without adding insight.

Where Organisations Typically Fall Short

The most common gap is not malicious — it is architectural. TDM pipelines are built by data engineers optimising for throughput and reliability. Legal defensibility is rarely a design requirement at the point of build.

As a result, three patterns tend to emerge:

• Opt-out signals are checked at crawl time but not logged. The system honours the opt-out, but there is no audit trail showing it did so. If challenged, you cannot prove the check happened.
• Source lists are dynamic but undocumented. The pipeline adds and removes sources automatically based on availability. Nobody has a timestamped record of which sources were active during which processing period.
• Outputs contain more original text than necessary. Summaries that quote extensively, or databases that store full article bodies alongside the derived metadata, blur the line between analysis and reproduction.

None of these are catastrophic on their own. But together they create a profile that is difficult to defend under scrutiny — even when the underlying activity was entirely within the scope of Art. 4.

Making Compliance an Operational Habit, Not a Legal Afterthought

The practical fix is not to hire more lawyers. It is to instrument the pipeline itself so that compliance evidence is generated as a byproduct of normal operation.

Opt-out check results should be written to a log with a timestamp and source identifier. Source lists should be version-controlled, not just maintained. Processing jobs should carry purpose metadata that links the run to a defined analytical objective. And output schemas should be designed from the start to hold derived signals, not to replicate source content.

This is not a burden unique to large organisations. Even a mid-sized team running TDM at moderate scale can implement these patterns without significant overhead — if they treat compliance as an engineering requirement rather than a legal overlay.

At TrawlingWeb, this framing is central to how the infrastructure is designed: Text and Data Mining operations are structured around derived analysis, with source access governed by the boundaries that Art. 4 and Art. 67 bis LPI define. The output is always analytical, never a reproduction of the original public content.

The Question to Ask Before Your Next Processing Run

Before expanding your TDM scope — new source categories, new geographies, new client use cases — the useful question is not "is this legal in principle?" It almost certainly is, under Art. 4.

The useful question is: can I reconstruct, six months from now, exactly what we processed, why, how we accessed it, and what we produced from it?

If the answer is yes, you have a defensible TDM operation. If the answer is "probably, mostly", the gap is worth closing before someone else asks the question on their terms.