GDPR and data migration: what you can't just copy over

A migration is one of the few moments where an organisation's entire data estate is in motion at once – extracted, transformed, loaded, validated, re-run, and re-run again. And every one of those steps happens in a non-production environment first: a staging area, a test instance, a developer's laptop, a CSV sitting in a project folder. That's exactly the right way to work. It's also exactly where personal data tends to leak out of its lawful home.

I recently helped migrate a membership database with more than 2.5 million Swedish people registered in it – the kind of source that is about as sensitive as Swedish data gets. Every member carried a personnummer (the Swedish personal identity number), full name, address, c/o, email, mobile, birth date, and a flag for protected identity (skyddad identitet). At that scale a single careless export is a breach affecting millions of people. You do a dozen trial loads of data like that before go-live – and the one thing you must not do is run those trials on the real personal data.

The principle: test environments don't need real people

GDPR doesn't say "don't migrate personal data" – of course you migrate it; that's the point of the project, and it has a lawful basis. What it says is data minimisation (Article 5(1)(c)) and purpose limitation: you only process the personal data you actually need, for the purpose you collected it for. Testing a transformation script is not that purpose. Proving that 80,000 rows map correctly from the old schema to the new one does not require the real members' identity numbers – it requires 80,000 rows that behave like the real ones.

So the rule I work to is simple: the only environment that ever sees the real personal data is the production cutover itself. Every test load, every developer iteration, every screenshot in a status deck runs on substitute data. That single boundary removes a whole class of risk – a leaked test export, an over-shared CSV, a backup of a dev database – because there's nothing sensitive in those places to leak.

Pseudonymisation, not just "scrambling"

Here's the subtlety that matters for the business, not just the engineer. GDPR draws a line between two things:

• Anonymisation – the data can no longer be linked to a person by anyone, by any means reasonably likely to be used. Truly anonymous data is out of scope of GDPR.
• Pseudonymisation (Article 4(5)) – the data can't be attributed to a person without additional information that you keep separately and protect. Pseudonymised data is still personal data, but it's a recognised safeguard that materially lowers risk.

What I built for the membership project is pseudonymisation, and it's honest to call it that. The replacements are believable and consistent, but the method is deterministic and key-based – so the technique only protects you if the key is protected. I'll come back to that, because it's the part people get wrong.

Why the fakes have to be deterministic

The naïve approach is to generate a random fake for every field and move on. That breaks migration testing, because a real dataset has referential integrity you need to preserve:

• The same member appears in multiple files and multiple tables – orders, payments, communications. If "member 4711" becomes Anna in one file and Johan in another, your joins fall apart and you can't test the relationships.
• A personnummer encodes the birth date. Replace it with a random number and the derived birth-date column no longer agrees with it – a validation that would pass in production now fails in test for the wrong reason.
• The data has to look real so the transformation logic is exercised properly: a Swedish mobile still needs to start +46 and have the right length; an email still needs a plausible domain; a street address still needs a house number.

The answer is deterministic pseudonymisation: the same input always produces the same fake, and a given member maps to one stable fake identity everywhere. I get that by seeding the generator with a keyed hash of the original value – HMAC-SHA256 with a secret – rather than a random seed. Same input + same secret → same seed → same fake. Different secret → a completely different mapping. The secret is the "additional information" GDPR talks about; keep it apart from the data and the pseudonymisation holds.

import hashlib, hmac, os, random
from datetime import date
from faker import Faker

# The secret is the key to the whole scheme. Load it from the environment
# or a secrets manager – NEVER hard-code it, and NEVER commit it.
SECRET = os.environ["ANON_SECRET"]

fake = Faker("sv_SE")  # locale matters: Swedish names, streets, formats

def _seed_from_value(value: str, secret: str = SECRET) -> int:
    """Deterministic seed: same input + same secret -> same fake, everywhere."""
    digest = hmac.new(secret.encode(), str(value).encode("utf-8"),
                      hashlib.sha256).digest()
    return int.from_bytes(digest[:8], "big")

def pseudo_first_name(x):
    if _nullish(x):
        return x
    Faker.seed(_seed_from_value(x))
    return fake.first_name()

def pseudo_last_name(x):
    if _nullish(x):
        return x
    Faker.seed(_seed_from_value(x))
    return fake.last_name()

Faker("sv_SE") is doing quiet but important work: the substitutes are Swedish names, streets and formats, so the test data is representative of the real population, not anglicised noise that behaves differently.

Format-preserving by field

Generic fake data isn't enough – each field has rules the downstream system enforces, so the fakes have to honour them. A few of the field handlers from the project:

A valid personnummer, not just twelve random digits. A Swedish personal number has a Luhn check digit; load one that fails the checksum and the target system rejects the row for a reason that has nothing to do with your mapping. So the fake is generated to be structurally valid – a plausible birth date, then a correct check digit:

def _luhn_checksum_10(d9: str) -> int:
    s = 0
    for i, ch in enumerate(d9):
        d = (ord(ch) - 48) * (2 if i % 2 == 0 else 1)
        s += d // 10 + d % 10
    return (10 - (s % 10)) % 10

def pseudo_personnummer(x: str) -> str:
    if _nullish(x):
        return x
    rnd = random.Random(_seed_from_value(x))   # deterministic, keyed
    today = date.today()
    year  = rnd.randint(today.year - 99, today.year)
    month = rnd.randint(1, 12)
    day   = rnd.randint(1, 28)                  # safe against invalid dates
    indiv = f"{rnd.randint(1, 999):03d}"
    nine  = f"{str(year)[2:]}{month:02d}{day:02d}{indiv}"
    return f"{year:04d}{month:02d}{day:02d}{indiv}{_luhn_checksum_10(nine)}"

Keep the email domain, replace the person. The domain mix (corporate vs. free webmail) is often part of what you're testing, so the realistic move is to fake the local part and keep the domain:

def pseudo_email(x: str) -> str:
    if _nullish(x):
        return x
    local, domain = x.split("@", 1)
    Faker.seed(_seed_from_value(local))
    return f"{fake.user_name()}@{domain}"

Preserve the shape of a phone number. Keep the +46, keep the spacing and hyphens the source used, and only scramble the digits – so length checks and format validations still exercise the same paths.

Keep the house number, swap the street name. An address validator cares about the structure; a regex pulls out the street name and replaces only that, leaving the number and any extra part intact.

And one piece of domain nuance that pure tooling would miss: members flagged with protected identity are overwritten wholesale with the literal Personuppgift skyddad ("personal data protected"), not given a pretty fake. These are people with a real-world reason their details are shielded; the test data should make that status loud and visible, never paper over it with a plausible-looking address.

protected = df["ProtectedIdentity"].str.strip() == "1"
df.loc[protected,
       ["GivenName", "LastName", "Street", "ZipCode", "City", "CareOf"]
      ] = "Personuppgift skyddad"

The orchestration is just a column-by-column map, then a couple of derived fields (the birth-date column is recomputed from the new personnummer so the two stay consistent):

def anonymize(df):
    df["SocialSecurityNumber"] = df["SocialSecurityNumber"].map(pseudo_personnummer)
    df["GivenName"]    = df["GivenName"].map(pseudo_first_name)
    df["LastName"]     = df["LastName"].map(pseudo_last_name)
    df["CareOf"]       = df["CareOf"].map(pseudo_co)
    df["Street"]       = df["Street"].map(pseudo_street)
    df["EmailAddress"] = df["EmailAddress"].map(pseudo_email)
    df["MobileNumber"] = df["MobileNumber"].map(pseudo_mobile)
    # Re-derive birth date from the *pseudonymised* SSN so they agree.
    df["BirthDate"] = df["SocialSecurityNumber"].map(birth_date_from_ssn)
    return df

Where this stops – the honest caveats

A technique like this earns trust only if you're clear about its limits. Pseudonymisation is a safeguard, not a magic eraser, and treating it as one is how organisations get caught out.

• Pseudonymised is still personal data. Because the mapping is deterministic and key-based, anyone with the secret and the original values can reproduce the link. So the output is lower-risk personal data, not anonymous data – handle it accordingly, and don't claim it's "out of GDPR scope".
• The secret is the crown jewels. The entire safeguard rests on the key staying separate from the data. Load it from a secrets manager or environment variable, never hard-code it, never commit it to git, and don't ship it alongside the anonymised export. (This is exactly why the snippet above reads os.environ["ANON_SECRET"].)
• A personnummer is a small, guessable space. Deterministic pseudonymisation of a low-entropy field can in principle be brute-forced if both the secret and the algorithm leak. The defence is protecting the secret – not assuming the output is unbreakable.
• Don't keep the mapping around. If you don't need to re-identify, don't store a lookup table at all – that table is the re-identification risk in one file.
• Production cutover is different. The real load runs on the real data, under the project's lawful basis, with the access controls and audit that implies. Anonymised test data is for building and proving the migration; it's not the migration.
• Write it down. Which fields are pseudonymised, how, and why belongs in the project's record of processing. "We faked the test data" is a control you should be able to evidence, not a favour you did quietly.

The business takeaway

The headline isn't the Python – it's the boundary. Decide, at the start of a migration, that real personal data lives in exactly one place: the production cutover. Everything upstream of that runs on data that looks and behaves like the real thing but can't hurt anyone if it leaks. It costs a day of engineering to set up, it makes your test environments dramatically safer, and it lets the whole team – developers, testers, the people reviewing screenshots in a status meeting – work without ever touching a real member's identity number.

That's data minimisation in practice, and on a sensitive dataset it's the difference between a migration that's careful and one that's merely lucky.

Migrating a sensitive dataset and want test environments that won't keep you up at night? Get in touch – I'm happy to share how I work.

---

This article describes a general approach, not a substitute for legal advice. References: GDPR Article 4(5) (pseudonymisation) and Article 5 (data minimisation, purpose limitation); the Python Faker library.