The Essence of PII De-identification — You May Fake the "Who," Never the "What Happened"

Hello! This is the Qualiteg Product Development Team!

Today we would like to discuss the de-identification of PII (Personally Identifiable Information).

To date, we have published technical explainers on personal-information detection and AI security, including high-accuracy PII detection, staged PII masking when using LLMs、test design for PII detection, and more.

Today, we develop and offer our PII detection and masking technology, the "PII-FI Engine," along with "PII-FI Scan」「PII-FI API," a PII masking and de-identification service built on it.

In this article, we take up the design principles behind "once PII has been detected, how should it be rewritten?" — using a single example sentence as a touchstone, we present the framework we have actually adopted in our products.

One caveat up front: what we call "de-identification (de-identification)" in this article refers to technical processing that makes documents and logs safe to share and analyze (processing data so that individuals can no longer be identified).

It is not a guarantee that the result qualifies as "pseudonymized information" or "anonymized information" under Japan's Act on the Protection of Personal Information (we touch on the relationship with these legal terms later in the article).

What Is De-identification For?

Let us start from first principles.

If you can use your logs and documents "as they are," you do not need de-identification.

If the original is in your hands and you are the only one who will see it, there is no reason to process it.

The need for de-identification arises at

"we want to hand this data to someone so the work can proceed, but we cannot hand it over raw"

— precisely that moment.

For example:

• You want to outsource an analysis task, but customer information is mixed in
• You want to feed operational logs into an LLM for incident analysis
• You want to send a full set of logs to a security vendor for analysis
• You want to review files containing PII before sending them to another company
• You want AI to organize medical-record information, but need to mask PII before it goes into the AI

Data is a mixture of "what happened" and "who it is about."

It is the former that creates value in analysis and sharing.

It is the latter that can lead to harm for the individual (impersonation, discrimination, reputational damage, and so on).

In other words, de-identification is the technique of

keeping the "events" while severing the "who"

.

And it is not only the individuals appearing in the data who are protected.

The provider is protected from liability for leaks, and the recipient reduces both the risk and the management burden of holding "information that is a liability merely to possess."

The Touchstone: 「甲野太郎は高血圧です」 ("Kono Taro has hypertension")

Now for the main topic. Consider de-identifying the following sentence.

「甲野太郎は高血圧です」 — "Kono Taro has hypertension."

There is more than one way to process it.

Let us compare three representative patterns.

Pattern 1: Replace Only the Name with a Pseudonym

乙山次郎は高血圧です ("Otsuyama Jiro has hypertension")

The link between 甲野太郎 (Kono Taro) and hypertension has been severed.
The reader receives only the event that "someone has hypertension." And this event is still true, and still usable. For a reader examining the case details or the appropriateness of the response, this sentence continues to function as working material.

This is the basic form of de-identification. Note, however, that what succeeded here is only "severing the link within this one sentence." In real documents, the surrounding context can still suggest who the person is even after the name is changed — a limitation we confront head-on in Pitfall 3.

Pattern 2: Fake the Disease Name Too

乙山次郎は花粉症です ("Otsuyama Jiro has hay fever")

At first glance this looks even safer.

But this processing has a serious problem.

The reader has no way to tell that "花粉症 (hay fever)" is fake。

Whoever tallies the cases, and whoever evaluates the appropriateness of the response,
ends up doing their work on top of a falsehood。

"Hiding" is a fail-safe operation: information only decreases, and no lies are added.

"Substituting another plausible-looking value," by contrast, is an injection of falsehood. One might fairly call it poisoning the data.

Pattern 3: Redact Everything

■■■■は■■■です ("■■■■ has ■■■")

Yes, this is safe.

But the sentence now carries zero information — "someone is something" — and as data it is dead. There is no longer any point in handing it over at all.

The Principle: There Are Two Kinds of Words

This comparison can be distilled into the following principle.

There are two kinds of words.

Words that point to "who" (identifiers) — names, employee numbers, phone numbers, email addresses. These may be replaced with natural-looking fakes. The reader does not consume their truthfulness; the only thing that needs to be severed is the "link."

Words that tell "what happened" (attributes) — disease names, monetary amounts, performance reviews, events. For these there are only two choices: keep them, or honestly redact them. They must never be faked, because the reader's work depends on their being true.

In reality, of course, words do not split cleanly into these two categories.

"Head of Accounting" is an attribute — and at the same time, within the organization, a clue that points to a specific person.

Ages and dates, when combined, also edge closer to a "who" (these are called quasi-identifiers). This is precisely why the yardstick for the decision is not the word's type but its role.

It is also why the question in the flow above is not "is this word a name?" but "will the reader use the truth of this value to do their work?"

Rather than deciding the method mechanically from the detected type alone, we decide it by combining the role the word plays within the document with the recipient's intended use.

"Honestly redacting" means replacing the value with a label such as [Disease 1].

The point is to number the labels sequentially: the same disease name always receives the same number (we call this coreference preservation).

That way, the fact that "this document discusses two different diseases" — the structure the analysis needs — is preserved, while the reader is honestly told that "this part has been redacted."

There is also a middle path between the two choices: "generalization." As in "age 47 → 40s" or "full street address → city level only," it is a way of keeping the truth while lowering only its resolution.

Because it injects no falsehood and reduces only the identification risk, it is an honest middle option usable for both identifiers and attributes.

Putting all of this together, the de-identification methods can be organized as follows.

One note of caution here.

Credentials such as passwords and API keys — the last rows of the table — should be treated as "secrets," regardless of whether they qualify as PII.

On the shared file they are redacted, but if there is any possibility they have leaked externally, redaction is not the end of the story: revocation and rotation — that is, incident response — are separately required.

Conversely,

applying dummies to attributes (substituting one disease name for another, for example) is, as a rule, prohibited

.

If there is ever a situation that calls for realistic fake data, it is one where the recipient does not rely on the factuality of the original — parser testing, demos, load testing, and the like.

We do not call that de-identification; we treat it as a separate job called "synthetic data generation."

In the standards world, synthetic data generation is indeed counted among de-identification techniques (NIST SP 800-188), but

"masking that preserves meaning for handover" and "the job of crafting plausible fakes" should never be mixed into the same process

— that is the design decision we have adopted.

Note also that for anonymizing statistical databases there are standard techniques that deliberately alter attribute values, such as noise addition (adding random error to numeric values) (ISO/IEC 20889); these are positioned as tools for hiding each record's true value while preserving statistical properties in aggregate.
Bringing them into the domain this article addresses — documents and logs read directly by people and tools — causes the same problem as the injection of falsehood described above.

Four Pitfalls That Are Easy to Step Into in Practice

Now let us look at the points to watch when actually designing PII processing.

Here are four challenges you will genuinely encounter when implementing a PII-processing application.

1. Across Files, the Same Person Becomes Different People

De-identify indiscriminately and you can end up converting one person into several different people. The same person should remain traceable as the same person after de-identification. This is what we call coreference preservation.

Consider, for example, de-identifying person names that appear across multiple logs.

If coreference preservation is applied only "within a single file," then when a set of logs is processed, file A may map 甲野 (Kono) → 乙山 (Otsuyama) while file B maps 甲野 → 丙川 (Heikawa).

That breaks any analysis that follows the actors' movements across multiple logs.

To deal with this class of problem, you need

coreference preservation at the level of the whole set (the job)

.

There is a flip side, however.

The wider the coreference scope, the more the "appearance pattern of the same pseudonym = the original person's behavior pattern" survives — which also means more clues for identification. For that reason, we believe the scope of consistency should be selectable to match the use case.

This point applies not only to person pseudonyms but equally to the numbering of labels such as [Disease 1].

2. A Pseudonym Collides with a Real Person

If the generated pseudonym "乙山次郎 (Otsuyama Jiro)" happens to match a real Mr. Otsuyama who already appears in the document, the analysis is not merely confused — you risk pinning a false accusation on that real person.

This challenge requires a mechanism to

exclude values that actually appear in the document from the pool of dummy candidates

.

Natural person names, however, can never be guaranteed "not to exist." For recipients who cannot accept the risk of misattribution, choose a label such as [Person 1] instead of a pseudonym — this, too, is a "destination" decision, discussed below.

For machine-formatted values, moreover, the standard move is to generate "values that cannot exist".

Use reserved domains for dummy email addresses (example.com), documentation-reserved test ranges for IP addresses (such as 192.0.2.0/24), and check-digit-invalid values for various ID numbers — then even an accidental contact or lookup reaches no one.

3. De-identification Cannot Erase "Everyone Knows Who That Is"

Even with every name and number removed, the context alone — "the head of Accounting received a disciplinary action on July 3" — tells people inside the company exactly who it is. This frequently happens when de-identifying PII contained in email.

De-identification is not magic; it is a technique for reducing identification risk.

In that sense, the assumption that "processed = perfectly safe" may be the biggest pitfall of all.

Note that "pseudonymized information" and "anonymized information" are legal terms, with definitions and requirements under Japan's Act on the Protection of Personal Information.

For referring to technical processing, we consider it safer to use the terms "de-identification" and "masking," as this article does.

4. A Processed File Is an "Altered Document"

Deliverables from incident response can later become material for audits or legal proceedings.

Since a de-identified file is not the original,

a processing record documenting "when, under what rules, what was processed and in how many instances," together with a hash of the original,

should accompany the deliverable, so that its relationship to the original can be demonstrated to third parties. The record is not a mere disclaimer; it is an audit document that guarantees the transparency of the alterations.

The Right Answer Is Decided by the "Destination," Not the "Type"

Let us move on to the next point:

what to prioritize in PII processing.

Naturalness (analysis tools keep working), safety, and honesty (fakes are recognizable as fakes) — these three cannot all be maximized at once.

Which one to prioritize is determined by where the data is going.

• Deliverables read by people → prioritize honesty. Label replacement at the core, plus a processing record
• Logs analyzed by tools → prioritize preserving formats and digit counts. Dummies built from values that cannot exist
• Data fed into LLMs → prioritize naturalness. Dummies at the core (some designs pair this with restoration applied to the LLM's responses)

Incidentally, the LLM case is not complete with input-side substitution alone.

The substitution mapping table is never sent to the LLM; it is managed inside your own trust boundary.

Restoration is limited to managed tokens, never free-text string matching.

And because an LLM's responses can contain PII that was not present in the input, the output side must be run through PII inspection again. All of this together forms one complete set.

In other words, a de-identification configuration is not "a preference per data type" but
a declaration of "to whom, and for what purpose, this data is being handed"
.

In our PII-FI, we design this declaration as a per-destination "profile."

Conclusion

In this article we have laid out our thinking on what de-identification essentially is, illustrating the design challenges of PII processing along the way.

To close, let us restate the key points.

• De-identification is a technology for "handing data over": keep the "events," sever the "who"
• There are two kinds of words: identifiers may be pseudonymized; attributes offer three options — keep, honestly redact, or generalize — and must never be faked
• Injecting fake data into attributes is an "injection of falsehood"; the only place it belongs is the separate job of synthetic data generation
• In practice, watch four things: cross-file coreference preservation, pseudonym collisions with real people, residual identification risk, and the processing record
• The final answer is determined not by the word's type but by "to whom, and for what purpose, the data is handed".

We hope this article has been of some use in thinking through PII processing.

At Qualiteg, as a fast, high-accuracy PII detection and masking solution built on the design philosophy in this article, we are developing and offering "PII-FI Scan," a service that de-identifies whole sets of logs and documents via drag and drop, along with the "PII-FI API," which makes the same technology available through an API.

If you are interested in PII-FI's detection, masking, or de-identification capabilities, please feel free to reach out via our contact form.

See you next time!

References (Industry Standards and Guidelines)

The framework in this article is consistent with the thinking in the following standards and guidelines on de-identification. Please refer to them as well.

• NIST IR 8053: De-Identification of Personal Information — A report from the U.S. NIST broadly organizing research and practice on de-identification and re-identification. Covers not only structured data but also free text and images
• NIST SP 800-188: De-Identifying Government Datasets — A practical guide to de-identifying government datasets (final version, 2023). Covers quasi-identifier transformation, synthetic data, re-identification risk assessment, and governance
• ISO/IEC 20889:2018 — The international standard for the terminology and classification of de-identification techniques. Systematizes pseudonymization, generalization, suppression (deletion/masking), noise addition, and more
• ENISA: Pseudonymisation Techniques and Best Practices — A practical guide to pseudonymization from the EU cybersecurity agency. Details the trade-off between the usefulness of consistent pseudonyms (coreference preservation) and re-identification risk
• Personal Information Protection Commission: Guidelines on the Act on the Protection of Personal Information (Pseudonymized and Anonymized Information volume) — Definitions, requirements, and obligations for "pseudonymized information" and "anonymized information" under Japanese law

Related Resources

• High-Accuracy PII Detection (Part 1)
• Staged PII Masking When Using LLMs (Part 2)
• What the Confusion Matrix Cannot Show in PII Detection — Inter-Recognizer Collisions and Test Design
• Enterprise Information Defense in the LLM Era
• The Complete Guide to LLM Security in the Zero Trust Era
• DLP Technology in the AI Era (Part 1)
• DLP Technology in the AI Era (Part 2)