Hello! This is the Qualiteg Research Team.

Automatic detection of personally identifiable information (PII) can be framed as the problem of extracting specific expressions from text and determining which type of PII each one represents.
As the range of PII types to detect grows — phone numbers, personal names, account numbers, monetary expressions — no single method can cover everything, and it becomes standard to adopt a multi-layer architecture that combines multiple recognizers with different characteristics.

This article assumes a use case in which, just before a user sends a chat message to an LLM hosted overseas, the content is inspected in real time for personal or confidential information.

In this setting, what matters is not just detection accuracy but speed that does not degrade the sending experience.

High-accuracy LLMs, BERT-family models, and NER-based methods are strong candidates, but applying them constantly as the first layer of a pre-send check can put you at a disadvantage in latency and cost.

For that reason, this system builds an ultra-fast first layer out of regular expressions, dictionaries, and lightweight rule-based recognizers, with later-stage recognizers supplementing context as needed on top of it — a multi-layer architecture.

The important point is that this architecture is not just a matter of adding more regex patterns. When you try to widen detection coverage while staying fast, it is easier to design separate recognizers per role than to consolidate everything into one giant rule. The price you pay is that the recognizers then begin to interfere with one another.

In this article we call this phenomenon "Cross-Recognizer Collision" and examine, through the lens of the confusion matrix, what test design ought to look like.

The four quadrants of the confusion matrix and PII detection

The confusion matrix is a widely used framework for evaluating classifiers, but in the context of PII detection each quadrant carries a somewhat distinctive practical meaning. Let us first review the basic definitions.

Filling this in with concrete examples:

TP — early-stage reassurance, and its limits

TP tests are the ones written at the very start of development.
Confirming that "given this input, it is correctly detected" is intuitive, and such test cases are easy to write, so a TP-centric test suite can reach a 100% pass state early on. But that 100% only proves that what should be detected was detected; it says nothing about whether what should not be detected is being passed over.

FP — the "boy who cried wolf" problem that erodes user trust

A system with many FPs rapidly loses user trust. Nobody wants to use, day in and day out, a tool that raises stacks of warning flags on perfectly normal text. In a high-FP system, even the genuinely important TP detections eventually get ignored. This is structurally the same problem as alert fatigue in medicine.

FN — the direct risk

In a PII context, FNs can lead directly to data leakage. Personal information that should have been masked can slip through to the outside because of a missed detection, so driving down the FN rate is a fundamental requirement of the system.

TN — unglamorous, but the foundation of quality

TN tests look unglamorous, but they are the only means of guaranteeing the system's specificity. A shortage of TN tests surfaces later as an increase in FPs.

Skewed test coverage — the pitfall of per-recognizer testing

In one PII detection project, a context-analysis recognizer was operated across several pipelines (detection categories).

With every test green, analyzing each pipeline's test coverage by quadrant revealed the following distribution.

At first glance, negative cases — where non-detection is expected — account for roughly a quarter of the total, hardly zero. Tests for exclusion conditions did exist: "do not detect when a specific keyword is missing," "pass over contexts explicitly marked as samples," "do not react to general medical terms."

But there are two structural problems here.

The first problem is the positive-heavy ratio itself.

Cases confirming successful detection make up 76% of the total; the bulk of the testing resources go to confirming that things can be detected. Negative cases confirming that things are not detected stop at a quarter. While positive cases covering variations of the detection targets (orthographic variants, full-width vs. half-width characters, abbreviations, and so on) were plentiful, the negative cases went no further than a handful of representative exclusion patterns.

The second and more fundamental problem is that every one of these negative cases only confirmed exclusion conditions conceivable within that recognizer on its own. A recognizer's test suite knows nothing about what patterns the other recognizers carry. This mutual indifference becomes the breeding ground for the cross-recognizer collisions described next.

Cross-Recognizer Collision

In a multi-layer PII detection system, the results from all recognizers are eventually merged into a single list. At the moment of that merge, detections from different recognizers collide, fighting over the same span of text.

Here are three typical collision patterns.

Case 1: identifier collisions caused by similar formats

Input:    "案件番号 A-1024 を更新してください"  ("Please update case number A-1024")
Expected: treated as an internal business identifier, or as non-PII depending on context
Actual:   another recognizer misreads it as a different category of identifier — a hospital room number, ticket ID, contract number, etc.

Short identifiers joining letters and digits with a hyphen are used across many business systems. Recognizers that rely on format alone tend to assign different meanings to the very same string.

Case 2: collisions between date notation and ratio/score notation

Input:    "提出期限は2026/3/5です"  ("The submission deadline is 2026/3/5")
Expected: no PII detected
Actual:   3/5 or /5 is misread as a ratio or score expression

Slash-separated expressions can mean dates, ratios, scores, and more. Recognizers that look only at short substrings are prone to misfiring once stripped of the surrounding context.

Case 3: ambiguity of ID formats

Input:    "CONTRACT-2024についてレビューしてください"  ("Please review CONTRACT-2024")
Expected: detected as legal information (a contract document ID)
Actual:   matches the ticket ID pattern (e.g., the "PROJ-1234" format)

Uppercase letters + hyphen + four digits is a format used for ticket IDs, contract document numbers, and project codes alike. Regular-expression patterns alone cannot tell them apart. A context-analysis recognizer might reach a more reasonable judgment from the surrounding context, but depending on the implementation, an early pattern-match result may be adopted preferentially or may effectively mask later-stage judgments, so context-based correction does not always take effect.

Structural reasons collisions are hard to discover

The late discovery of cross-recognizer collisions stems from structural problems in test design.

Independent development per recognizer. Each recognizer is usually owned by a different developer or team. Whoever tunes the NER model writes the NER tests; whoever owns the dictionary recognizer writes the dictionary tests. Each suite passes correctly on its own, but contains no input patterns that account for the existence of the other recognizers.

Separate CI pipelines. In many projects, each recognizer's test suite runs independently. An end-to-end test with all recognizers loaded does not exist unless someone deliberately designs it.

The inherent ambiguity of natural language. That the same string can carry multiple meanings is the nature of natural language. Japanese in particular is not delimited by spaces the way English is, which makes substring-match problems that much more serious. There is no strong Japanese equivalent of \b (the word-boundary concept), and it is not uncommon to need coordination with a morphological analyzer.

Collisions between recognizers that a 2×2 confusion matrix cannot show

Let us generalize the discussion so far.

Placing a fast first-layer filter is not, in itself, an unusual idea in security, DLP, or input validation. What characterizes this approach is not simply adding more regexes, but dividing roles among multiple fast recognizers and treating the collisions that arise when their results are merged as part of quality design. A fast first layer is a common design decision; explicitly treating the inter-recognizer contention that appears as its side effect as a central concern of test design is much less common.

With a single recognizer, evaluation fits into a 2×2 confusion matrix. With multiple recognizers running side by side, things get more complicated. In practice you have a multi-class judgment involving several entity types plus "none of the above," and on top of that, span contention and label contention between recognizers — so a 2×2 confusion matrix alone cannot capture the structure of the errors.

What tends to get overlooked here is the fact that different recognizers can assign different labels to the same string. For example, the notation CONTRACT-2024 may be treated by a legal recognizer as a valid contract identifier while another recognizer misinterprets it as a ticket ID format. In other words, one recognizer's correct detection can become, at the system level, a source of contention with another recognizer. This duality cannot, in principle, be caught by tests of individual recognizers.

Moreover, in PII detection, it matters not only what was detected but how much of it was detected. Picking up only part of a phone number, or over-extending a match to include the department name attached to an organization name, can be practically wrong even when the label itself is right. Evaluation therefore needs to treat span-boundary accuracy as its own concern, distinct from entity-type correctness.

The collision relationships between recognizers can be laid out as a table (rows are the "detecting" side, columns the "collided-with" side).

Every off-diagonal cell is a potential breeding ground for collisions. Measuring the quality of the whole system requires deliberately testing these off-diagonal elements.

Approaches to mitigation

1. Adding cross-recognizer negative cases

This approach adds inputs likely to trip other recognizers' patterns as TN tests for each recognizer.

// TN test for the medical recognizer
{
  text: "案件番号 A-1024 を更新してください", // "Please update case number A-1024"
  expected_count: 0,
  description: "Must not misclassify a short identifier format as the medical category"
}

// TN test for the ticket ID recognizer
{
  text: "CONTRACT-2024についてレビューしてください", // "Please review CONTRACT-2024"
  expected_count: 0,
  description: "Must not misclassify a contract ID format as another category of identifier"
}

This is easy to implement and works reliably as regression testing. Since it is hard to enumerate every collision pattern up front, though, the realistic path is steady accumulation: adding each collision discovered in production as a new TN test, one at a time.

2. Priorities and conflict-resolution rules between recognizers

This approach spells out the resolution strategy for when multiple recognizers react to the same text span.

One idea is to prefer the longer span: adopt the detection that interprets a wider range as one coherent meaning over a short partial match. Another is to give detections informed by surrounding-context keywords precedence over pattern-only detections. It is also effective to predefine which entity types win when specific types contend, or to assign each recognizer a numeric priority value and adopt the higher one on conflict.

3. Introducing an integration test layer

Separately from unit tests of individual recognizers, design integration tests that run with all recognizers loaded.

describe("Cross-Recognizer Collision Tests", () => {
  const detector = createFullDetector(); // load all recognizers

  it("a date must not be misdetected as a score pattern", () => {
    const results = detector.detect("会議は2026/3/5に設定されました"); // "The meeting is set for 2026/3/5"
    expect(results.filter(r => r.type === "score")).toHaveLength(0);
  });

  it("identifier-format conflicts are resolved correctly by priority rules", () => {
    const results = detector.detect("CONTRACT-2024の条項を確認"); // "Review the clauses of CONTRACT-2024"
    expect(results.some(r => r.type === "legal_id")).toBe(true);
    expect(results.some(r => r.type === "ticket_id")).toBe(false);
  });
});

Where unit tests verify "does this recognizer work correctly," integration tests verify "can these recognizers coexist." Only with both in place can you vouch for the reliability of the system as a whole.

4. Stricter word boundaries

Substring-match problems can be mitigated by using regex word boundaries appropriately.

// Also matches substrings
const loose = /CT/i;

// Matches CT only as an independent word
const strict = /\bCT\b/i;

In Japanese, however, word boundaries are not as clear-cut as in English, so a design that relies on \b does not carry over as-is. In practice you need extra measures: bringing in morphological analysis, checking the character classes around a match, or correcting boundaries between symbols, alphanumerics, and kanji in a later stage.

Conclusion — what your test design proves, and what it does not

Now
, let us pull together the key points of this article.

A 100% pass from TP tests alone is false reassurance
It only proves that what should be detected was detected; resilience against what should not be detected remains unverified. All four quadrants deserve attention from the earliest stage of test design.

Unit tests of individual recognizers are a necessary condition, not a sufficient one
Without introducing integration tests that combine all recognizers early on, unexpected collisions surface after integration and force rework.

Collisions are less accidental bugs than a design problem inherent to multi-layer architectures
As long as multiple recognizers interpret the same string under different rules, label contention and span contention will occur with some probability. Collisions should therefore be built into test design and conflict-resolution policy as a premise from the start, rather than swatted afterward as exceptions.

Do not stop at the 2×2 confusion matrix; capture the error structure including inter-recognizer contention
The relationship in which a detection valid for one recognizer becomes, at the system level, a source of contention with another recognizer is invisible to individual tests. Visualizing the collision matrix between recognizers and deliberately testing its off-diagonal elements raises the quality of the whole system.

It is precisely when you stand before a test suite passing at 100% that you should ask
"what does this suite guarantee, and what does it not yet guarantee?"
all over again.

Whether you can hold on to that question may be what separates a detector that works in the demo but breaks in production from a detection system that stands up to real operation.

In this article, we discussed cross-recognizer collisions in multi-layer PII detection and the test design blind spots they tend to hide in.

If you are working on balancing speed with detection quality, we hope at least one point here proves useful.

Thank you for reading to the end.

See you next time!