One question comes up in many of my meetings right now: can we use a large language model on sensitive data if we filter the sensitive parts out first?
The short answer is yes, and it is worth doing. But where you filter matters more than how. The common answer is a filter in front of the model call. I would put the main control much earlier: in the source, before the data ever reaches the layer the AI is allowed to use. And the European Data Protection Board has just given us a useful way to test whether it works.
The reality check from the EDPB
In July the EDPB adopted draft Guidelines 02/2026 on anonymisation for public consultation. Data is anonymous only if it does not relate to an identified or identifiable person, judged by the means reasonably likely to be used. The guidelines give three criteria, and data is presumed anonymous only if it meets all three:
- No record isolation. You can't single out one person from a unique combination of values.
- No linkage. You can't link the record to another dataset and identify the same person.
- No inference. You can't draw a specific, meaningful conclusion about an individual that isn't available from general knowledge.
Read that third criterion with a language model in mind. Inference is what a model does. Commentators on the guidelines point out that inferences drawn by prompting a model can fail the "no inference" test, and that model output should not be treated as anonymous by default. So removing names and ID numbers from a prompt is a good start, but it does not by itself make the prompt anonymous.
Why a filter in front of the model is the wrong main control
A filter just before the model call is attractive because it is one place and easy to add. In my view it is the wrong place to rely on:
- It works on text, not on meaning. By the time data has become a prompt, the structure that told you what each value was is gone. A job title in a small organisation, a rare event, a date and a place together: no single word is sensitive, but the combination isolates a person.
- Retrieval and agents multiply the paths. Documents fetched to answer a question, and tool responses an agent pulls in, often carry more personal data than the question itself. Every new path needs the same filter, and the one you forget is the one that leaks.
- It decides too late. The question "may this value ever be used by AI?" is a data governance decision. Making it at runtime, on a stream of text, turns a policy into a guess.
Filter at the source, into a semantic layer
This is the pattern I would start from for a protected environment, whether that is the public sector, finance, healthcare or an industrial customer with sensitive designs:
- Classify the data in the source, not just the source. Decide per field, and where needed per record, what may never leave the system it lives in. "The HR system is sensitive" is too coarse. "Salary and personal identity number never leave; role and department may" is a decision you can implement and test.
- Extract into a shared semantic layer. Move the data AI may use into one common layer with shared definitions: the business concepts, their relationships and their meaning. This is where integration architecture earns its keep.
- Filter on the way in. Everything that may not leave the source is removed, aggregated or replaced with tokens during extraction, before it lands in the semantic layer. Free-text fields get special treatment here, because that is where identity hides.
- AI connects only to the semantic layer. Retrieval, agents and tools read from the layer, never directly from the source systems. Data that never reached the layer can't leak through a path you forgot.
- Keep the keys in the source zone. If users need to see real names in the answer, the mapping from token to identity stays inside the protected zone, with separate access rights and logging, and is applied after the AI has done its work.
- Log what the layer contains and what was sent. Which extraction rule, which version, which model. If you can't show it, you can't defend it.
A light check at the model boundary can still catch mistakes. But it is a safety net, not the control.
The trade-offs
This approach costs more up front. A semantic layer is something to design, run and keep current, and it is another copy of data that needs protecting, even if a reduced one. Extraction adds latency, so the layer may lag the source. Field-level classification takes real work with the people who own the data, and it has to be maintained as systems change.
It also takes something away. Every field you keep out can make some answers worse, and a model that only sees "Person A in Department B" sometimes can't do the job. That is the point: the trade-off is then made deliberately, per field, by people who understand the data, instead of by a filter at runtime.
And one point I think is often missed: pseudonymised data is still personal data under the GDPR. Filtering at the source lowers the risk a lot. It doesn't change the legal status by itself, and the guidelines are still a draft. I would design as if what reaches the semantic layer can still be personal data, and choose where the model runs with that in mind.
What would change my mind
If an organisation has a single, well-structured source and one narrow AI use case, a semantic layer may be more machinery than the problem needs, and a careful filter at the boundary can be enough. But as soon as there are several sources, several AI use cases or agents with tools, I would rather decide once, at the source, than hope every path is covered.
How are you handling this today: filtering at the prompt, filtering at the source, or running the model inside your own environment?