Why Zero False Positives Still Wasn’t Enough: Lessons from 100+ NER Experiments
What 100+ NER experiments taught us about eliminating false positives on a fixed GitHub technical-term evaluation set without regressing existing performance: hard negatives, focused loss, distillation, residual repair, and class-specific logit adjustment.
Hello, we are the Qualiteg Research Team.
We develop PII-FI, a tool for detecting and masking personally identifiable information and other sensitive data. PII-FI uses named entity recognition (NER) as part of its processing pipeline, and this article focuses on continual learning for NER. At Qualiteg, we work with a powerful NER model trained on a large corpus assembled over several years. For convenience, we refer to it throughout this article as our ‘strong NER model.’ While scanning several GitHub repositories, we found false positives in which short technical terms inside product names and identifiers were classified as organization names or internal terminology.
Adding a few exclusions would have fixed that one screen. In an unfamiliar repository, however, the errors would simply have moved to different technical terms, issue numbers, version numbers, or identifier fragments.
With that context, we addressed these false positives by improving the strong NER model itself through continual learning instead of adding dictionary exceptions. We created more than 100 NER models before selecting the production version. Several intermediate candidates reduced the target false positives to zero, but we rejected every candidate that weakened recall on existing documents or degraded even one other entity type. Our aim is to share what these experiments taught us about evolving a trained model when small false positives continue to surface one after another.
The winning pattern was not one large training run. We mined hard negatives from the current model itself, protected diversity on unseen expressions separately from the weak positive boundaries the existing model was most likely to lose, repaired only the final residual errors with minimal training, and then applied the smallest necessary class-specific adjustment to the logits. That sequence gave us the point where false positives disappeared without sacrificing existing performance—and in fact improved it.
This article does not disclose model names, customer data, or internal training documents. Instead, it presents the reusable training design, equations, failure analysis, and acceptance gates behind the result.
1. Define “pass” mathematically before training
Before improving the model, we fixed the acceptance criteria. A higher aggregate F1 score was not enough. Let \(c\) denote an entity type, \(\mathrm{base}\) the current model, and \(\mathrm{new}\) a candidate. At minimum, a candidate had to satisfy the following conditions in product evaluation:
$$\begin{aligned} \mathrm{TP}_{\mathrm{new}}(c) &\ge \mathrm{TP}_{\mathrm{base}}(c) \\ \mathrm{FP}_{\mathrm{new}}(c) &\le \mathrm{FP}_{\mathrm{base}}(c) \\ \mathrm{FN}_{\mathrm{new}}(c) &\le \mathrm{FN}_{\mathrm{base}}(c) \qquad \forall c \end{aligned}$$
Losing a correct detection in even one entity type meant rejection. We rejected candidates that produced zero false positives on the target technical terms but missed one person name in another domain. It is important not to relax this rule later. The more effort we invest in a candidate, the stronger the temptation to say, “It has improved this much, so we should ship it.” That temptation becomes especially strong after 10 or 20 training runs. Even when one metric has slipped slightly, a dramatic improvement elsewhere makes it very easy to give in to the voice saying, “Can’t we just ship this one?”
We also measured precision, recall, and F1:
$$\begin{aligned} P &= \frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FP}} \\ R &= \frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FN}} \\ F_1 &= \frac{2PR}{P+R} \end{aligned}$$
These aggregate metrics are summaries, not acceptance criteria. The decision came from TP, FP, and FN by entity type, together with the exact spans that appeared or disappeared.
Figure 1. The improvement loop from false-positive discovery to production acceptance
2. Split the data into four roles—and call any evaluation you inspect validation
We began with synthetic technical documents containing no real PII. The important point was not the number of examples but how we separated them.
| Partition | Purpose | Operating rule |
|---|---|---|
| Training | Gradient updates and hard-negative mining | Used directly for training |
| Development | Intermediate candidate comparison | Used to select checkpoints, learning rates, and loss weights |
| Additional validation | Additional checks on unseen expressions | Text is not returned to training, but results influence candidate selection |
| Fixed regression | Broad non-regression checks | Kept unopened initially; after opening, reused for regression and no longer called a final test |
We separated more than whole documents. Vocabulary, identifier components, prefixes and suffixes, and sentence templates were also split. If the same word merely appears in a different template, a model that memorized the word can still pass. Conversely, changing the word while reusing the same identifier components or sentence pattern can exaggerate novelty.
We fixed the item count and hash of every set. If an evaluation changes every time it is regenerated, differences between candidates cannot be attributed to the training change.
The naming matters. Once we inspected the additional-validation results and changed training conditions, that set was no longer an independent final test; it was validation data. The fixed-regression set remained unopened until the first promising candidate, but we then reused it to check multiple candidates for regressions. We therefore do not claim that an untouched final test guaranteed performance. Any set inspected during more than 100 adaptive experiments is described honestly as development or regression data.
3. Mine the model’s own false positives, not hand-picked words
Our first attempt labeled the misclassified technical terms as negatives. Suppressing those words merely moved the false positives to issue numbers, version numbers, or nearby short fragments in the same sentence.
We then ran the current model over synthetic technical documents reserved for the training side. Among positions that generation rules or annotations verified were not named entities, we mined the exact spans that the model nevertheless classified as entities. We did not mine from the development, additional-validation, or fixed-regression text. A hard negative was not a term that a person thought looked difficult; it was a position where the current decision boundary had demonstrably failed.
Let \(z_e\) be the logit for the target entity type, \(z_O\) the logit for the non-entity label O, and \(m\) the required margin. At a negative position, a pairwise margin loss can be written as:
$$\mathcal{L}_{\mathrm{neg}}=\max\!\left(0,\,m+z_e-z_O\right)$$
No gradient is applied when \(z_O\) is already sufficiently higher. Only the misclassified boundary is pushed back by the required margin. This limits the scope of change far more effectively than training the entire document as O.
4. Labeling an entire GitHub document as O damages real entities
One of our largest failures came from training every unannotated token in technical documents as O. Training-set scores rose, and false positives on unseen technical terms fell sharply. In real documents from another domain, however, recall dropped on genuine internal entity names that the model should have detected.
The cause was token imbalance. A document contains a small number of correction targets and a large number of background tokens. Feeding all background tokens into cross-entropy as O pushes even unrelated positions toward O. This project was not class-incremental NER in the strict sense of adding new entity classes sequentially. It nevertheless shares a known structural risk with continual NER: treating unannotated entities as O can destroy existing recognition ability.
If \(M\) is the set of tokens being corrected, the focused cross-entropy loss is:
$$\mathcal{L}_{\mathrm{focused}} =-\frac{1}{|M|}\sum_{t\in M}\log p_{\theta}(y_t\mid x)$$
Tokens outside \(M\) are not treated as ordinary O labels. They are either excluded from the loss or protected by using the old model as a teacher.
Comparing the main families of failed experiments
| Direction tried | What improved | Why we rejected it | Lesson carried forward |
|---|---|---|---|
| Negative labels only for selected technical terms | Lower confidence on the target terms | False positives moved to numbers and adjacent fragments | Mine every false-positive span produced by the model itself |
| Train the full technical document as O | Fewer false positives on unseen technical terms | Lower recall on existing entities | Apply loss only at correction positions |
| Update only classifier rows for target types | Restricted the output direction | Could not separate contextual differences | A small encoder update was still necessary |
| LoRA with pseudo-labeling | Zero false positives on the target set | Lower precision and recall in another domain | Use the teacher model to preserve logits outside corrected positions |
| Focused loss with distillation | Balanced the target set and unseen terms | Lost a few weak positive spans from the existing product | Add visible windows around weak boundaries to the positive side |
| Localized residual repair | Zero target false positives and recovered existing true positives | Precision in another domain remained slightly below the gate | Apply the smallest adjustment only to error-heavy entity types |
5. Protect two kinds of positives instead of simply making positives stronger
Training only on negatives weakens real entities. Repeating a large volume of positives did not solve the problem either. Giving positive cross-entropy too much weight pulled unseen common words toward entity classes and reduced precision.
We ultimately needed two positive sets with different roles.
| Positive set | What it protects | Failure when omitted |
|---|---|---|
| Diverse synthetic positives | Generalization to unseen expressions | Positive examples in the fixed evaluation are missed |
| Visible windows around weak boundaries | Spans the current product detects only narrowly | A small number of existing true positives are lost |
Diverse synthetic positives alone did not protect weak boundaries in the existing product. Weak boundaries alone encouraged memorization of their context without generalizing to unseen expressions. Only when both groups competed with negatives in the same batch did we obtain candidates that preserved unseen positives and existing true positives together.
Figure 2. Balancing negative correction with three kinds of protection
6. Use the old model as a fixed reference, not as ground truth
To protect existing capability, we used the current model as a teacher. Let \(U\) be the set of tokens outside the correction targets, and let \(z_t^T\) and \(z_t^S\) be the teacher and student logits. A context-stabilization loss can be written as:
$$\mathcal{L}_{\mathrm{KD}} =\frac{1}{|U|}\sum_{t\in U}\left\lVert z_t^S-z_t^T\right\rVert_2^2$$
A KL divergence over temperature-scaled softmax outputs is another option. The important point is that freezing existing classifier rows does not by itself protect existing entity types. If LoRA changes the encoder, the representations entering the frozen classifier also change, and so do the final logits. Freezing the classifier and distilling the outputs are separate safeguards.
Conceptually, our full objective combined four terms:
$$\mathcal{L} =\mathcal{L}_{\mathrm{neg}} +\alpha\mathcal{L}_{\mathrm{pos}} +\beta\mathcal{L}_{\mathrm{KD}} +\gamma\mathcal{L}_{\mathrm{preserve}}$$
\(\mathcal{L}_{\mathrm{neg}}\) corrects false-positive boundaries. \(\mathcal{L}_{\mathrm{pos}}\) is supervised loss on diverse synthetic positives and weak positive boundaries. \(\mathcal{L}_{\mathrm{KD}}\) preserves output outside correction positions. \(\mathcal{L}_{\mathrm{preserve}}\) is supervised loss on a fixed labeled retention set from existing domains. Larger coefficients are not automatically safer. Excessive positive protection reintroduces false positives, while excessive negative correction reduces recall. We monitored the scale of each loss as a diagnostic, but ultimately rejected settings that improved only one objective by inspecting FP, FN, and span-level deltas on the development data.
7. Repair only the residual errors instead of retraining at the same scale
After broad training reduced false positives to a handful, we did not repeat another run of the same size. We identified the remaining false-positive spans and the correct spans lost by the new candidate, then performed small repairs using only the context windows visible to the tokenizer. The text returned to repair training was limited to training data and development regression windows regenerated from approved training sources. We did not feed text discovered in the fixed-regression set directly back into training.
Net improvement was not sufficient. We separated gains and losses by entity type. Even when a type had the same net number of true positives, the model might have lost one correct span and gained a different one. Once the lost span was identified, that single positive window could be trained alongside the residual negative.
We compared learning rates, training steps, and positive-loss weights in small increments. We also tracked the confidence and length of each remaining false-positive span. If an error shrank from two characters to one and its margin over O moved toward zero, we treated that as evidence of movement in the right direction and considered another small update. If confidence increased while the count stayed unchanged, we stopped that line of experiments.
Near the final candidate, a difference of only a few training steps or a small change in positive weight separated preservation of unseen positives from the last zero-false-positive result. The lesson from more than 100 experiments was not how to run a massive hyperparameter search. It was how to turn each failure delta into the next small experiment.
8. Move from inexpensive gates to expensive ones
Running full product evaluation for every candidate wastes time. We used a funnel in which only candidates that passed one gate moved to the next.
- Target false-positive set
- Additional unseen positives separated from training
- Basic technical negatives
- Large-scale false-positive stress test
- Fixed regression data, reused after its first opening
- TP, FP, and FN by entity type on product documents
- Precision and recall in another domain
- The actual runtime contract
- The full production acceptance criteria
A candidate with even one error on the target set did not enter expensive evaluation. A candidate below the unseen-positive gate did not enter the large negative stress test. Product evaluation stored not only aggregate F1 but also span-level changes by entity type. Every gate listed here was used for development or regression while selecting the final candidate. Because a failure could influence the next training data or loss design, these gates must be distinguished from an independent final test.
Figure 3. The evaluation funnel for narrowing candidates to production acceptance
9. Finish with the smallest class-specific score adjustment—not one global threshold
After training, a promising candidate passed the target technical false-positive and product evaluations, but precision in another domain remained slightly below the acceptance line. Raising the O logit globally improved precision, but it also removed genuine unseen entities across all types.
Instead, we added a negative bias only to the two entity types where false positives were concentrated, slightly lowering the corresponding logits:
$$z'_c=z_c+b_c,\qquad b_c<0$$
This was not an exclusion list for particular words; it applied to every candidate span in the same entity type. Nor was it probability calibration in the usual sense of aligning predicted probability with empirical accuracy. It was a class-specific logit adjustment. We compared bias sizes on the cross-domain development evaluation and selected the smallest value that crossed the acceptance line. We then reran the unseen positives, fixed regression, every product entity type, and the full production criteria. We rejected stronger adjustments because they removed additional genuine entities even when precision increased.
10. Final results and the decisive factors
The production candidate passed the fixed GitHub technical-term evaluation, additional unseen-positive validation, basic negatives, the large false-positive stress test, and fixed regression data. In product evaluation, it gained correct detections, reduced false positives by roughly 400 on the same fixed product-regression data, and reduced false negatives relative to the old model. No entity type—including the major types—regressed in TP, FP, or FN. “Zero false positives” here is an observed result on fixed evaluation sets, not a claim that no false positive can occur in every unseen repository.
| Measure | Compared with the old model |
|---|---|
| False positives on the fixed GitHub technical-term set | Reduced to zero |
| True positives in product evaluation | Increased |
| False positives in product evaluation | Reduced by roughly 400 |
| False negatives in product evaluation | Reduced |
| Non-regression by entity type | Passed for every type |
| Unseen positives and fixed regression | Passed |
The success cannot be reduced to LoRA, distillation, or margin loss alone. It came from this combination:
- Turn the model’s own false positives into hard negatives.
- Do not casually label every unannotated token as O.
- Protect diversity on unseen expressions separately from weak existing positive boundaries.
- Use the old model as a teacher to stabilize outputs outside correction positions.
- Repair only residual errors with minimal additional training.
- Apply the smallest adjustment only to entity types where residual errors are concentrated.
- Accept a candidate by TP, FP, and FN for every type—not by aggregate F1 alone.
The major improvements to our strong NER system came through more than 100 training runs, and the successful result was formally adopted as a new model artifact.
Summary
- Have you separated training, development, additional-validation, and fixed-regression roles?
- Have you separated identifier components and sentence patterns—not only vocabulary?
- Have you mined the model’s actual false-positive spans?
- Are you avoiding the assumption that every unannotated token is O?
- Are you protecting both positive diversity and weak positive boundaries?
- Are you relying on more than a frozen classifier to protect existing entity types?
- Are you recording gains and losses for every candidate at span level?
- Do only candidates that pass inexpensive gates enter expensive product evaluation?
- Are you preventing aggregate F1 from hiding type-level regressions?
- Is any class-specific score adjustment limited to the smallest value that crosses the acceptance line?
In continual NER, measuring which capabilities must not move is harder than raising the score on the target data. More than 100 failures should not end as a column of candidate names. Convert the movement of false positives, confidence changes, lost spans, and type-level gains and regressions into the next small experiment. Through this experience, we came to understand that only after designing the process at that level does model improvement become reproducible.
In our earlier article, “Extending a Model’s Domains “Without Breaking It” — Design Notes on XLM-RoBERTa Continual Learning,” we explained how to restrict the part of a model changed by additional training. This article is the sequel: what to do with the data and evaluation when even that restricted training cannot pass in one attempt.
See you next time.
Sources and further reading
- Continual Named Entity Recognition without Catastrophic Forgetting (EMNLP 2023)
- Learning “O” Helps for Learning More: Handling the Unlabeled Entity Problem for Class-incremental NER (ACL 2023)
