LLM-Audit PII Detection Technology, Part 2: Staged PII Masking in LLM Applications

Hello, this is the Qualiteg Product Development team!

In our previous technical deep dive, we started from the inherent ambiguity of Japanese—where you cannot tell whether "三沢" (Misawa) is a person's name or a place name—and introduced a five-stage approach to PII detection, from regular expressions to LLM integration. This time, we take one step further and look at how LLM-Audit™ PII Protector, which incorporates these technologies, actually operates in real-world LLM usage scenarios, from the perspective of actual product processing.

Invisible PII Hiding Inside Files

LLM Audit™ PII Protector, which we introduced previously, prompted a question from many readers: "I understand PII detection for text input, but what happens when an entire file is handed to the LLM?"

Indeed, LLM usage today has evolved beyond simple text input to directly processing files in a wide range of formats—PowerPoint, Excel, PDF, and images. Requests like "summarize this proposal" or "analyze the trends in this Excel file" have become commonplace.

Have You Ever Looked at PowerPoint "Speaker Notes"?

Suppose a sales representative asks an LLM to review an important proposal. The slides themselves contain generic content—"About Our Product," "Our Proposal"—and at first glance appear to contain no confidential information.

However, the speaker notes of that PowerPoint file often contain raw information that could never appear in the open, such as:

• "The budget I heard privately from Director XX is 300 million yen"
• "Word is that competitor B plans to bid at 250 million"
• "The key person is Mr. Yamada in the Planning Department (hobby: golf)"

—candid details that were never meant to leave the room.

The advanced PII detection technology we introduced last time was developed precisely to uncover these uniquely Japanese confidential expressions in such "hidden places." However, where you need to look differs greatly depending on the file format.

In this article, we walk through each file format commonly used in Japanese companies, pointing out where the danger lies and explaining concrete, staged countermeasures for each.

Staged PII Masking in LLM Applications

Stage 1: Real-Time Input Checking (All Prompts)

At the moment of input to the LLM, an ultra-fast screening pass runs. Clear-cut patterns (phone numbers, email addresses, My Number IDs, and so on) are detected instantly and automatically masked before anything is sent to the LLM. Somewhat more complex patterns such as personal names and company names are also processed at high speed, so users can continue working without noticing any delay.

For example, the input "田中太郎さん（090-1234-5678）から問い合わせ" (an inquiry from Taro Tanaka, 090-1234-5678) is automatically converted to "[顧客A]さん（[電話番号]）から問い合わせ" (an inquiry from [Customer A], [phone number]).

Stage 2: Context-Aware Masking (Business Documents)

When processing structured documents such as meeting minutes, reports, and proposals with an LLM, advanced language processing is used to identify company names, job titles, monetary amounts, and more with high accuracy. In a sentence like "ABC（株）の山田部長より、X社との共同プロジェクトについて相談" (Director Yamada of ABC Inc. consulted us about a joint project with Company X), the system can properly distinguish company names from personal names and mask the confidential information while preserving the business context.

Stage 3: Semantic Masking (Confidential Documents)

For highly confidential documents such as contracts and financial reports, more advanced techniques are applied. By understanding the document's structure and semantic relationships, the system removes personally identifiable information while preserving the granularity of information the LLM needs.

Stage 4: Support for Advanced Analysis

For specialized analytical tasks, masking is optimized for the use case. Advanced processing protects only the confidential information while retaining what the analysis actually requires.

Stage 5: Continuous Improvement

Using in-house test documents (containing the kind of content that would realistically be exchanged in day-to-day work), we validate the system's accuracy and improve it continuously. By detecting both missed masking and over-masking, we keep evolving the system into something ever more practical.

PII Masking Strategies by File Format

Multi-Layer PII Detection for PowerPoint Files

Even when a PowerPoint file looks safe on the surface, it often contains a great deal of hidden personal information.

Easily Overlooked Areas Beyond the Slides Themselves

1. Speaker notes

The presenter's notes are a treasure trove of candid remarks. They often contain critical information that could never be said openly, such as "the budget I heard privately from Director Tanaka is 50 million yen."

2. Comments and review history

Traces of the review process expose the organizational structure as-is. In one real case, a comment listed its author as "Taro Yamada, Sales Planning Department" with the content "At this price we lose money. We need at least 30 million"—a single comment risking the simultaneous leak of a department name, a personal name, and pricing strategy.

3. Embedded objects

Detailed customer lists often lurk in Excel workbooks embedded in slides. PowerPoint files store embedded files internally, and comprehensive PII detection must cover these as well.

4. Text inside images

Information caught in screenshots is a surprisingly common blind spot.

• Real names in the participant list of a captured Teams screen
• Internal system addresses in the URL bar of a system screenshot
• Handwritten phone numbers and email addresses in photos of whiteboards

5. Masters and layouts

Footers applied to every slide sometimes contain personal information without anyone noticing.

Typical risk distribution in a sales proposal:

• Slides themselves: low risk (generic proposal content)
• Speaker notes: high risk (names, amounts, strategic information)
• Embedded Excel: highest risk (detailed customer data)
• Comments: medium risk (organizational information, internal discussion)

Deep PII Detection for Excel and Spreadsheets

Excel files are a trove of structured data and the file format with the highest density of personal information.

PII detection points specific to Excel in Japanese companies:

1. Column header pattern recognition

Techniques that infer the nature of a column from Japanese-specific headers—such as "氏名" (name), "担当者" (person in charge), and "連絡先" (contact information)—are essential.

2. Inspection of hidden areas

• Visible sheet: aggregated results only (looks safe)
• Hidden sheet "給与マスター" (salary master): salary data for every employee
• Hidden columns: personal mobile numbers, emergency contacts

3. Reference information lurking in formulas

When functions such as VLOOKUP reference external files, those referenced files may contain confidential information.

4. Data validation lists

It is not unusual for a dropdown list to have the names of every employee configured as its options.

5. Source data behind pivot tables

Even when the summary table is anonymized, the underlying source data may contain detailed personal information.

Composite PII Detection for PDF Documents

Because PDFs carry an image of being "finished documents," they are the file format people let their guard down on most.

Hidden PII Risks Specific to PDFs

1. The dual structure of text and image layers

What looks like a scanned image on the surface may have an OCR-generated text layer behind it. Unless both are checked, there is a risk of missing PII.

2. Form fields

Forms inside a PDF sometimes retain past input history, accumulating applicants' names and department information.

3. Annotations and comments

Comments added in Adobe Acrobat may not be displayed by some PDF readers and are easily overlooked.

4. Metadata

Document properties contain personal information such as the author's name, company name, and license information for the software used.

5. Attachments

PDFs can carry other files as attachments. There are cases where a contract PDF has an Excel file of the negotiation history attached to it.

Advanced PII Detection for Image Files

As digital transformation advances, information is increasingly shared through screenshots and photos.

Typical PII Leakage Patterns in Image Files

1. Captures of meeting screens

• Participant list: real names and affiliations
• Chat pane: "Tanaka-san, the 30 million estimate from earlier is fine"
• Screen sharing: confidential materials in plain view
• Notification pop-ups: private messages

2. Photos of whiteboards

• Printed names: relatively easy to detect
• Scribbled phone numbers: OCR accuracy is the challenge
• Abbreviations inside diagrams: require contextual understanding

3. System screen captures

• Browser tabs: personal Gmail, internal system names
• File paths: folder structures containing user names
• Taskbar: running apps can reveal the nature of the work

4. Metadata

GPS information carries the risk of identifying where a photo was taken (someone's home or a customer's site).

Best Practices for File Processing in the LLM Era

Risk-Based Processing Priorities

Processing every file at the same level is inefficient. Our LLM-Audit™ PII Protector can be configured to your organization's characteristics to assess each file's risk level appropriately.

Processing examples by risk level

• Highest risk: payroll Excel files in the HR folder → detailed scan required
• High risk: recently updated customer lists → standard scan plus review
• Medium risk: general business documents → quick scan
• Low risk: templates and public materials → minimal checks

Balancing Processing Speed and Security

By making the processing mode selectable per use case, security is maintained without sacrificing practicality.

• Fast mode (under 1 second): chatbot responses, real-time assistance
• Standard mode (5–10 seconds): document summarization, general analysis
• Secure mode (around 30 seconds): contract analysis, customer data processing
• Maximum security mode (1 minute or more): M&A documents, board meeting materials

Conclusion
— Using LLMs Safely with Comprehensive PII Protection —

In the LLM era, comprehensive PII masking that covers every file format—PowerPoint, Excel, PDF, images, and more—is indispensable, not just PII detection for plain text input.

What matters is understanding the risks specific to each file format and processing them at the appropriate level. The technical approaches we introduced last time were designed precisely to meet these complex requirements.

Providing a practical solution built on a deep understanding of how Japanese companies create documents and share information is the key to using LLMs safely.

Introducing LLM-Audit™

At Qualiteg, we develop and provide LLM-Audit™, our security solution for LLMs.
As LLMs are increasingly used in business, attacks against them are on the rise.
At the same time, attacks on LLMs differ from conventional web security threats; knowledge about them is still scarce and defensive measures are not yet well established.

Qualiteg Inc. offers LLM-Audit™, an LLM defense solution built on the experience and insights we have gained through developing and operating LLM services.

It can also powerfully block malicious input prompts and audit inappropriate LLM outputs, delivering safety and peace of mind for your LLM deployments.

Because it works simply by wrapping your LLM as an OpenAI API-compatible server, you can achieve advanced LLM security at a very small adoption cost.

If you are interested in LLM security or LLM-Audit™, please get in touch. For LLM security consulting or a product demo, feel free to contact us through the inquiry form here.