Hello from the Qualiteg Research Team!
Today we take a close look at PII (Personally Identifiable Information) detection technology.

Global Tools and Japanese-Specialized Tools, Side by Side

Protecting personal information is a challenge shared worldwide.
With regulations tightening around the globe—GDPR, CCPA, and Japan's amended Act on the Protection of Personal Information—many excellent PII detection tools are being researched and developed. Yet when you try to use them in business settings centered on Japanese, you often hit unexpected snags, and a non-trivial amount of additional development and special handling can be required. The reason is that these tools are designed primarily for English (and English-speaking cultures); Japanese may be listed as a supported language on the spec sheet, but fine-grained support tends to lag behind.
Naturally, every language has its own culture and structure, and Japanese—with its distinctive writing system and cultural background—presents situations that call for a special approach.

The Japanese-specialized PII detection technology we are developing coexists with, and takes advantage of, the excellent foundational technologies adopted globally, while staying closely attuned to the characteristics of Japanese. Our goal is to deliver a solution that is genuinely practical for Japanese companies.

The Beauty and Complexity of the Japanese Language

A Multi-Layered Writing System Rare in the World

The most distinctive feature of Japanese is that multiple writing systems function together in harmony.
Hiragana, katakana, kanji, plus the Latin alphabet and numerals—these combine organically to produce rich expressive power.

For example, a single company name can be written in many different ways:

Official name: 株式会社国際情報技術研究所 (International Information Technology Research Institute, Inc.)
Abbreviation 1: KJK研究所 (KJK Kenkyusho)
Abbreviation 2: ケージェーケー研究所 (the same abbreviation spelled out in katakana)
English notation: KJK Research Institute Corporation
With corporate marker: （株）KJK研究所 ("(K.K.) KJK Kenkyusho")
Common name: 国際情報研 (a shortened nickname)

All of these refer to the same organization, used differently depending on context and purpose. This flexibility is a strength of Japanese, but it is also what makes computational processing difficult.

A Language Where Context Determines Meaning

In Japanese, the same word can mean entirely different things depending on context, and in many cases it is impossible to tell what is being referred to without that context.

Consider the string 「三沢」 (Misawa):

• 「三沢から連絡がありました」 ("We heard from Misawa") → One of our employees? A company name? Unclear
• 「三沢に行ってきます」 ("I'm heading to Misawa") → The town with the US military base? Someone's place? A shop? Unclear
• 「三沢で購入しました」 ("I bought it at Misawa") → A store called Misawa Shoten? A person (bought from them)? Unclear

In English, by contrast:

• "I got a call from Mr. Misawa" (a person)
• "I'm going to Misawa City" (a place)
• "I bought it at Misawa's" (a store)

Prepositions, articles, and possessives make the distinction explicit.

An even more serious example is the polysemy of the particle の (no).

「東京の会社」 ("Tokyo no kaisha")

• A company located in Tokyo (location)
• A company named Tokyo (part of the name)
• A company owned by Tokyo (ownership)
• A company related to Tokyo (association)

In English, these would be expressed as

• "company in Tokyo"
• "Tokyo Company"
• "Tokyo's company"
• "company about Tokyo"

and clearly distinguished—but in Japanese, the distinction cannot be made without context.

What makes things especially difficult is how much Japanese omits.

「昨日会った。とても良い人だった。」 ("Met yesterday. Was a very nice person.")
→ Met whom? The subject is omitted.

「資料を送ってください。」 ("Please send the materials.")
→ To whom? Which materials? Much is omitted.

「承知しました。対応します。」 ("Understood. Will handle it.")
→ Understood what? Handle what? Context-dependent.

Because of these characteristics of Japanese, PII detection must look broadly at the surrounding context.

The Rich Information Carried by the Honorific System

The Japanese honorific system is not merely a matter of politeness. It encodes multiple layers of information: the relationship between speakers, social standing, and the formality of the situation.

「山田が来ました」 ("Yamada came") → a colleague or subordinate
「山田さんが来ました」 ("Yamada-san came") → general politeness
「山田様がいらっしゃいました」 ("Yamada-sama has arrived") → a customer or someone of higher status
「山田先生がお見えになりました」 ("Yamada-sensei has arrived") → an expert or a respected teacher

These distinctions in honorific usage are valuable clues for PII detection and masking. The type of honorific tells us the person's standing and how much masking is appropriate.

The Technical Advantages of a Japanese-Specialized Approach

As we have seen so far, "PII detection" is easy to say, but building practical technology is not simply a matter of stringing NLP libraries together or throwing machine learning at large datasets. It starts with a deep understanding of Japanese communication culture and business practices.

Detection Logic Grounded in Cultural Insight

The Innovation of Honorific-Driven Detection

For example, in Japanese business documents it is extremely rare for a person's name to appear without an honorific. "Honorific-driven detection," which exploits this cultural trait, is a simple yet powerful approach.

# Conceptual detection logic
if honorific_present:
    if preceding_string_matches_japanese_name_pattern:
        confidence = 0.95
    elif katakana_only:
        if character_count_is_plausible:
            confidence = 0.85
        else:
            run_additional_context_checks

This is not simple string matching, however.

We need to distinguish generic honorific expressions such as 「お客様」 (dear customer) and 「皆様」 (everyone) from honorifics that refer to a specific individual.
For example, we are developing detection technology that applies multi-layered filtering like the following:

1. Morphological filtering: character types, character counts, placement patterns
2. Statistical filtering: matching against a database of Japanese family and given names
3. Contextual filtering: surrounding particles, verbs, and sentence structure
4. Exclusion-list application: excluding boilerplate expressions common in business documents

Full Support for the Japanese Address System

Next, let us look at how Japanese addresses are written.

Full form: 〒100-0001 東京都千代田区千代田1丁目1番1号 ABCビル15階 (postal code, prefecture, ward, district, block-lot-number, building and floor)
Block shorthand: 東京都千代田区千代田1-1-1 ABCビル15F
Prefecture omitted: 千代田区千代田1-1-1
Building name only: ABCビル (identified as an address from context)

The system is characterized by its hierarchical structure and the many ways addresses can be written.
Addresses run even deeper than that: there are lot-number (chiban) notations, and because of historical changes, the rules for how addresses are "segmented" differ from region to region.
For example, Kajicho in Chiyoda City, where our company is located, was formerly Kanda-Kajicho; and strictly speaking "Kajicho" is not the official town name—the town name is "Kajicho 1-chome." This may sound like trivia, but in practice it quietly matters a great deal: for identifying localities, for speeding up searches, and at the most practical level, for deciding which database column a value belongs in.

To cover these variations comprehensively, we have implemented hierarchical pattern matching.

Example of the hierarchical structure for address detection
Level 1: Postal code (the 〒 mark or a 7-digit number)
Level 2: Prefecture (complete list of all 47 prefectures)
Level 3: Municipality (including irregular readings)
Level 4: Block and lot numbers (supporting diverse notations)
Level 5: Building name (buildings, apartment names, room numbers)

We are also progressively adding support for region-specific conventions, such as Kyoto's street-based addresses (e.g., 「四条通河原町上る」, "on Shijo Street, north of Kawaramachi").

PII Detection Technology and Its Trade-offs

From here, let us consider what a realistic PII detection solution looks like.

An Intelligent Balance Between Processing Speed and Detection Accuracy

Naturally, if you set your mind to it, you can push accuracy technically as far as you like.
In real business settings, however, a system that is "perfect but slow" is useless. The balance between speed and accuracy must be tuned flexibly to the use case, the objective, and the applicable security standards.

Below, we walk through the technologies used in real-world PII detection in terms of the speed-versus-capability trade-off, across five levels: Level 1 (lightweight, high-speed pattern matching) through Level 5 (slow but highly accurate and capable).

Level 1: Ultra-Fast Scan Mode (tens of thousands of items/sec and up)

Technology base: high-speed pattern matching with regular expressions

Characteristics

• Memory usage: minimal
• Detection targets: PII with clearly defined patterns

This mode is ideal for the initial screening of large log files and database dumps. It instantly detects information with well-defined formats, such as phone numbers, email addresses, and credit card numbers.

Typical applications

• Web server access log analysis
• Periodic database audits
• Real-time stream processing for LLMs and similar systems

Level 2: Balanced Mode (thousands of items/sec and up)

Technology base: morphological analysis engine + rule-based inference

Characteristics

• Understands Japanese grammatical structure
• Contextual judgment using part-of-speech information
• Extensibility via custom dictionaries and support for up-to-date keywords

Morphological analysis breaks a sentence into meaningful units:

Input: 「営業部の田中部長から連絡がありました」 ("We received a message from Department Manager Tanaka of the Sales Department")
Morphological analysis result:
営業[noun] 部[suffix] の[particle] 田中[proper noun] 部長[noun] から[particle] 
連絡[noun] が[particle] あり[verb] まし[auxiliary verb] た[auxiliary verb]

From this analysis, we can determine that 田中 (Tanaka) is a proper noun linked to the job title 部長 (department manager), making it highly likely to be a person's name.

Typical applications

• Daily batch processing
• Document management systems
• Email audit systems

Level 3: High-Accuracy NER Mode (hundreds of items/sec and up)

Technology base: spaCy/GiNZA + Conditional Random Fields (CRF) or BiLSTM-CRF

At this level we go beyond simple pattern matching to sophisticated Named Entity Recognition (NER) powered by machine learning.

What are spaCy and GiNZA?

spaCy is a fast, multilingual NLP (natural language processing) library designed for industrial use. GiNZA is what dramatically improves spaCy's accuracy for Japanese: it packages a Japanese language analyzer jointly developed by the National Institute for Japanese Language and Linguistics (NINJAL) and Recruit Co. for use with spaCy, handling the complex grammar and orthography of Japanese.

What is CRF (Conditional Random Fields)?

CRF (Conditional Random Fields) is a machine learning method for sequence labeling problems. It is used to assign labels such as "this is a person's name" or "this is an organization name" to each word in a text. BiLSTM-CRF (Bidirectional LSTM-CRF) is a more advanced method that combines deep learning with CRF.

Characteristics

• Considers the structure of the entire sentence
• Flexible judgments through machine learning
• Adaptability to new patterns

NER (Named Entity Recognition) is an AI technique that reads text and understands its meaning:

Input: 「来週の火曜日に大阪支社で新製品の説明会があります」 ("There will be a briefing on the new product at the Osaka branch next Tuesday")
NER output:
- 来週の火曜日 (next Tuesday) → date/time expression
- 大阪支社 (Osaka branch) → organization name
- 新製品の説明会 (new product briefing) → event

On top of that, our Japanese context enhancer resolves ambiguities peculiar to Japanese:

# Example of context enhancement
Base detection: 「大阪」 (Osaka) → place name (confidence: 0.7)
Context analysis: 「大阪」 + 「支社」 (branch office) → part of an organization name (confidence: 0.95)

Typical applications

• Processing contracts and legal documents
• Analysis of customer service records
• Full-text inspection for quality assurance

Level 4: Transformer-Based Deep Learning Mode (around 100 items/sec)

Technology base: Japanese RoBERTa/DeBERTa models + fine-tuning

Characteristics

• Deep understanding of document-wide context
• Captures long-range dependencies
• Infers implicit information

The breakthrough of the Transformer architecture is, above all, the attention mechanism. It allows the model to accurately understand relationships between distant parts of a sentence.

Example: 「先月お問い合わせいただいた件について、弊社の技術担当である山田から
      回答させていただきます。なお、詳細な仕様については、来週お伺いする
      際に改めてご説明いたします。」
("Regarding your inquiry last month, Yamada, our engineering contact, will respond.
As for the detailed specifications, we will explain them when we visit you next week.")

Conventional NER:
- 山田 (Yamada) → person's name (local context only)

RoBERTa/DeBERTa:
- 山田 (Yamada) → person's name (our company's engineering contact)
- Subject of お伺いする ("will visit") → Yamada (inferred from context)
- Honorific level → identified as a customer-facing document

DeBERTa's Technical Advantages

1. Disentangled Attention
   • Processes word position information and content information separately
   • Handles the flexible word order of Japanese
2. Enhanced Mask Decoder
   • More accurate understanding of function words such as particles and honorifics
   • Improved inference of omitted subjects and objects
3. Relative position encoding
   • Accurately grasps sentence structure even in long texts
   • Understands the hierarchy of complex honorific expressions

Optimization Techniques
Furthermore, by applying optimization techniques like the following—one of our core strengths—we aim to achieve "high accuracy, yet lightweight."

• Model size reduction through quantization
• Improved GPU utilization through dynamic batching
• Lightweight models built via knowledge distillation

Typical applications

• Final checks on confidential documents
• Compliance audits
• Processing documents with high legal risk

Level 5: Large Language Model Integration Mode (around 10 items/sec)

Technology base: LLM (Large Language Model) integration

Characteristics

• Human-level (or better) language understanding
• Reasoning that goes beyond the immediate context
• Handles industry-specific expressions

PII detection and masking with LLMs is a fundamentally different approach from conventional methods. Where rule-based and machine-learning-based methods "find patterns," an LLM can actually "understand the text."

This mode leverages the latest LLM technology to handle complex cases that are difficult to detect with conventional methods:

Example: 「先日お話しした例の件ですが、Kさんから連絡があり、
      あちらの責任者は前向きに検討しているとのことです。
      ただ、例の数字については、もう少し時間が必要だそうです。」
("About that matter we discussed the other day: K-san got in touch, and the person
in charge on their side is considering it positively. However, regarding those numbers,
they apparently need a bit more time.")

Conventional methods:
- Kさん (K-san) → detectable as an initial
- あちらの責任者 (the person in charge on their side) → hard to detect
- 例の数字 (those numbers) → undetectable

LLM understanding:
- Kさん (K-san) → a person (written as an initial)
- あちらの責任者 → refers to a specific person (inferred from context)
- 例の数字 → likely highly confidential information (amounts, codes, etc.)
- Overall context → confidential information about a business negotiation

The LLM also understands abbreviations, technical terms, and implicit references, and proposes an appropriate masking level.

Typical applications

• Board meeting minutes
• M&A-related documents
• Critical legal contracts
• Detailed analysis when an incident occurs

This technical deep dive, which began with "the beauty and complexity of the Japanese language," is finally drawing to a close.

What we wanted to convey in this article is not a mere list of technical specifications. It is the unique cultural background of the Japanese language, and the potential for technical innovation that comes from engaging with it sincerely.

Summary

In this article, we covered Japanese-specialized PII detection technology broadly, from why it is needed to implementation strategy.

Most global tools are designed with English-speaking regions in mind, and they cannot adequately handle the characteristics of Japanese: its complex mix of writing systems (hiragana, katakana, kanji, and Latin script), context-dependent interpretation, and the honorific system. Is 「三沢」 (Misawa) a person, a place, or a store? Does 「東京の会社」 (Tokyo no kaisha) indicate a company's location or part of its name? Such judgments are impossible without context. We therefore developed efficient detection methods tailored to the characteristics of Japanese. Honorific-driven detection identifies personal names using honorifics such as 「様」 (-sama) and 「さん」 (-san) as clues, while hierarchical pattern matching handles Japan's complex address notations. Each individual technique builds on decades of Japanese NLP research, but we combine them efficiently and engineer them to adapt well to real business settings. On the implementation side, we organized the options into a five-level approach that weighs processing speed against detection accuracy: from Level 1's high-speed regex scanning (tens of thousands of items/sec) to Level 5's LLM integration mode (around 10 items/sec), you can choose the appropriate processing level based on how critical the data is. This makes it possible to serve a wide range of needs, from everyday log processing to close inspection of confidential documents.

Next time:

In the next installment, titled "Staged PII Masking in LLM Applications," we will explain how the five-level PII detection technology introduced here functions in real-world LLM application scenarios.

In particular, we plan to cover how to detect the "invisible PII" lurking in various file formats—PowerPoint, Excel, PDF, image files, and more—and how to use LLMs safely, with concrete examples.

We will take a detailed look at the pitfalls and countermeasures specific to file processing: "The slides look fine, but what about the speaker notes?" "Could there be confidential information in hidden Excel sheets?"

See you next time!

Introducing LLM-Audit™

Qualiteg develops and provides LLM-Audit™, our security solution for LLMs.
As LLMs see wider business adoption, attacks against them are becoming more active.
At the same time, attacks on LLMs differ from conventional web security threats; knowledge about them is still scarce and defenses are not yet well established.

Qualiteg Inc. provides LLM-Audit™, an LLM defense solution that consolidates the experience and insights we have gained through developing and operating LLM services.

It powerfully blocks malicious input prompts and audits inappropriate LLM outputs, making your LLMs safe and trustworthy.

It works as an OpenAI-API-compatible server that simply wraps your LLM, so you can achieve advanced LLM security at a very low adoption cost.

If you are interested in LLM security or LLM-Audit™, please get in touch. For LLM security consulting or a product demo, feel free to reach out to us here via our contact form.