Artificial intelligence changes what data protection has to defend. The obligation is unchanged; the surface is not. Training sets, model weights and inference endpoints each hold or leak personal data in ways a classical processing register never described. This guide sets out the risks, the legal duties attached to them, and the controls that answer them.
Quick Overview
What you’ll learn:
- Why AI processing breaks assumptions classical data protection was built on
- How training-data quality became a legal property rather than an engineering one
- Which GDPR duties bind an AI deployment, and where they bite hardest
- What each privacy-enhancing technology actually buys
- Which attacks target the model rather than the database behind it
- What a governance framework for AI projects has to name
- Where to start when the project is already running
Who this article is for:
- Data protection officers reviewing an AI project for the first time
- Data scientists and ML engineers who own a training pipeline
- Security leads accountable for models already in production
Reading time: 8 minutes
Where AI Breaks the Classical Data-Protection Model
Classical data protection assumes you can point at the data: a table, a field, a retention rule attached to both. A trained model breaks that assumption, because the personal data is no longer in one place. It sits in the training corpus, in the parameters derived from it, and in the outputs produced on request. Four shifts follow:
- Re-identification. Records stripped of direct identifiers can be correlated back to individuals by a model that has seen enough adjacent data. Anonymisation is a property of a dataset in a context, not a permanent label.
- Inherited bias. A model trained on historical decisions reproduces those decisions, including the parts nobody intended to keep.
- Opacity. A parameter-heavy model does not explain itself, and an organisation that cannot explain a decision cannot defend it to the person affected.
- Purpose drift. Data collected to run a service and later reused for training has changed purpose, whatever the pipeline diagram says.
The European Data Protection Board’s Opinion 28/2024 on certain data protection aspects related to the processing of personal data in the context of AI models states the consequence plainly: a model trained on personal data is not anonymous by default, and anonymity has to be demonstrated for that specific model rather than inferred from the pipeline that produced it. Teams that want the regulatory frame around this will find it in our guide to what the AI Act changes for IT teams.
Training Data as a Compliance Artefact
Data quality used to be an engineering concern with a business cost attached. Under the Artificial Intelligence Act it is a legal requirement with an audit trail: the Act’s data governance obligation for high-risk systems requires training, validation and testing sets to be relevant, sufficiently representative, and — to the best extent possible — free of errors and complete in view of the intended purpose. That is a documentation duty as much as a modelling one, because the claim has to be evidenced rather than asserted.
The same Act resolves a tension that has stalled more than one bias audit. Detecting discrimination generally requires the very attributes an organisation is least willing to process, and the Artificial Intelligence Act permits processing special categories of personal data specifically for bias detection and correction, subject to safeguards and where strictly necessary. The lawful route exists; refusing to look is a choice, not a compliance position.
The work itself is unglamorous: audit the corpus for skew against protected characteristics, mitigate what is found, document representativeness, and validate quality at ingestion rather than after training. The cheapest moment to fix a dataset is before the first run.
What the GDPR Asks of an AI Deployment
The General Data Protection Regulation applies to AI systems exactly as it applies to anything else, and the parts that bite hardest are the ordinary ones. Purpose limitation and data minimisation constrain what may enter the training set; data protection by design and by default constrains how the system is built rather than how it is described afterwards.
Two provisions deserve separate attention. Article 22 of the General Data Protection Regulation gives a person the right not to be subject to a decision based solely on automated processing, including profiling, that produces legal effects concerning them or similarly significantly affects them. A scoring model wired straight into an outcome sits inside that provision, and a human who rubber-stamps the score does not move it out. Article 35 requires a data protection impact assessment where processing is likely to result in a high risk, and names systematic and extensive automated evaluation of personal aspects among the triggering cases.
Subject rights are where AI projects discover their debt. Erasure, rectification and access all assume you can locate a person’s data, and in a trained model you frequently cannot — an argument for deciding the retention and retraining story before deployment. Teams building that capability from the legal side up will find the ground covered in the Data protection course.
Privacy-Enhancing Technologies and What Each One Buys
Privacy-enhancing technologies are not interchangeable, and treating them as one category is how projects end up with the expensive control and the weak guarantee. Each buys something specific:
- Pseudonymisation removes direct identifiers and keeps the data in scope. It reduces exposure; it does not end the obligation.
- Differential privacy adds calibrated noise so that the presence or absence of any single record barely changes the result. NIST’s Guidelines for Evaluating Differential Privacy Guarantees make the trade-off explicit: the guarantee is governed by a privacy-loss parameter, and a value chosen for utility can leave that guarantee close to meaningless while the label stays intact.
- Federated learning trains across data that never leaves its holder, sending model updates instead of records. It removes the central copy, not the leakage risk carried by the updates.
- Homomorphic encryption allows computation over encrypted data, at a computational cost that decides its own use cases.
- Synthetic data reproduces statistical structure without individual records — provided the generator was not trained in a way that memorises them.
A team that cannot say which specific risk a control removes has bought a label rather than a guarantee.
Attacks Aimed at the Model Itself
Securing the database behind a model is necessary and insufficient, because the model is a separate asset with its own attack surface. The OWASP Top 10 for Large Language Model Applications names sensitive information disclosure and data-and-model poisoning as distinct entries — engineering categories, not theoretical ones. The shapes worth naming:
- Data poisoning — crafted records placed in the training set to bend later predictions.
- Adversarial inputs — perturbations invisible to a person that reliably flip a classification.
- Model extraction — reconstructing a model’s behaviour by querying it systematically.
- Membership inference — establishing whether a particular record was in the training set, which is a personal-data disclosure even when no content is returned.
Defences are correspondingly specific: validating training data provenance, adversarial hardening, rate-limiting and authenticating inference endpoints, and monitoring outputs, not only infrastructure. Our guide to AI in the hands of attackers covers the offensive side of this boundary.
Governance: Naming Who Answers for the Data
Controls without owners decay quietly, and AI projects decay faster because the pipeline changes weekly. A workable governance framework names four things: ownership — who holds each dataset and who stewards its quality; policy — what binds acquisition, storage, sharing and deletion; metadata — what records provenance and processing history; and audit — who checks the result against internal policy and external regulation.
Dates make this concrete rather than aspirational. The Artificial Intelligence Act has applied generally since 2 August 2026, with its prohibitions and AI literacy obligations applicable from 2 February 2025. An organisation that treats governance as documentation to be produced during an audit will produce it during an audit, at the worst possible cost. Our practical guide to AI governance and ethics covers the operating model.
Where to Start
For a project already under way, the order that wastes least: establish what personal data is in the training corpus and on what legal basis it got there; run the impact assessment before the model reaches users; decide the retention and retraining story so that subject rights have a mechanism behind them; select privacy-enhancing technologies against named risks; and monitor the inference endpoint like any other production interface.
None of this is exotic. It is ordinary data protection applied to an asset that keeps changing shape, done early enough to still be cheap. Organisations that find AI compliance painful usually asked these questions after the model was already serving users.
Build Your Skills
Turning these obligations into a working control set is hands-on work: classifying systems by risk, writing the documentation the regulation expects, and building governance that survives contact with a delivery team. The AI governance and the EU AI Act: risk management for AI systems course covers that ground. Check the programme and sign up.
Frequently Asked Questions (FAQ)
Can we train a model on personal data we already hold?
Not automatically. Data collected for one purpose and reused for training has changed purpose, and the General Data Protection Regulation requires a lawful basis for the new processing. The practical question is rarely “is it allowed” but “can we evidence why”, and that answer must exist before the training run.
Does anonymising the training set take the project outside the GDPR?
Only if the anonymisation holds. The European Data Protection Board’s Opinion 28/2024 treats anonymity as something to be demonstrated for the model in question, not assumed from the process that produced it. A model that can be made to reveal training records is not anonymous whatever the pipeline claims.
Do we need a data protection impact assessment for every AI system?
No, and the criterion is risk rather than technology. Article 35 of the General Data Protection Regulation triggers the assessment where processing is likely to result in a high risk to rights and freedoms, and names systematic and extensive automated evaluation of personal aspects among the qualifying cases. A model that tags internal articles is not in that territory. A model that scores people is.
Which privacy-enhancing technology should we start with?
Start with the risk. If the concern is a central copy of sensitive records, federated learning removes it. If it is disclosure through aggregate results, differential privacy addresses it. If it is a test environment holding real records, synthetic data is usually cheapest.
Should the data protection officer or the data science team own this?
Neither alone. The data protection officer owns the assessment of whether processing is lawful and the record that proves it; the engineering team owns whether the controls work in the pipeline. Projects fail at the seam between them, so the governance framework has to name that seam explicitly.