In my last piece I wrote about different approaches and mechanisms for detecting personally identifiable information (PII) in text. I argued that one rule doesn’t fit all, and that it matters more to adapt your approach to the data you’re using it on.

Which made me think about what the data you’re using is, and what you need to adapt to. Detection can only be as good as how you train it to be. Which seems obvious, but it’s the next step to my previous summary of different ways to detect information. For the most effective detection, your pipeline would be trained on annotated datasets from the domain you operate in. Ye et al. (2025) found that domain-tuned detection pipelines, especially Longformer, outperformed both rule-based and general NER models. But domain-specific engines can be expensive, difficult to build, and hard to train, since it’s difficult to obtain datasets that are both safe to use (synthetic) and representative of your domain.

Kocaman et al. (2025) found that local specialised models cost over 80% less to run than cloud services like Azure or GPT-4o. Models can be the better choice since general NER can struggle to find PII in different contexts because PII can hide in contextual information or other noise. Small models then win on performance in-domain. And, if labelled data can be obtained, even small samples of domain PII can push a small, local detection engine above existing tools (Hartman et al. 2020). Yet the inherent risks remain: labelling means data has to be sourced, labelled by a human, and reviewed.

Some training datasets exist, but most are clinical. When Pilán et al. (2022) presented the TAB benchmark, they introduced their open-source annotated corpus developed to address the shortage of safe datasets for evaluating the privacy protection of anonymisation methods. They used court cases from the European Court of Human Rights (ECHR) as an alternative to the largely medical or clinical data available. Ye et al. made the same observation about the domination of medical data in PII detection training.

The irony is that Ye et al. found the medical domain was the most allergic to being transferred into, and resisted fusion with other domains the most. Legal and biographical domains benefitted from being fused, with PII definitions travelling between the two. If an engine was trained on legal data, it could also detect PII in biographical records. This did not travel into the medical domain. All models performed poorly on medical data. The data you train on shapes how a detection pipeline performs, and finding an option that can shift across domains is difficult.

Another factor that makes it harder to transfer a detection engine trained on one domain to another is that definitions of PII change with domain. Any general model therefore has to be trained on the full spectrum of PII definitions, and on quasi-identifiers. For example, Stubbs et al. (2017) found that psychiatric records needed 18 categories of PII removed to be sufficiently de-identified. In the ECHR court cases, by contrast, Pilán et al. point out that there are ‘rich and unconstrained biographical descriptions’ throughout, which present a particular challenge for a PII detection engine in isolating correct PII, and removing the quasi-identifiers that abound in biographical material.

This mirrors realistic detection in many company environments, where PII often sits within contextual noise. Lison et al. (2021) show why that noise matters: gender, birth date and postcode together are enough to identify between 63% and 87% of the US population from public Census data. All the direct PII can be detected, but if an individual can still be identified through the remaining details, detection has been incomplete.

These studies all show that training a detection engine that adapts across domains comes with a lot of problems. Training on representative datasets is difficult in itself because of medical data dominating, and even once you have obtained appropriate data, you have to work through the differences in how each domain you work with defines PII. Within that, you also have to identify and train an engine to locate quasi-identifiers. And all of this has to happen inside a local, specialised engine for maximum performance and privacy.

LLMs seem like the best option: they can handle varied formats of PII and data and transfer across domains. Garza et al. (2025) found this when building PRvL, an open-source suite of fine-tuned models and evaluation tools for PII redaction. They found that using LLMs also meant you could have consistent auditability and domain transferability without writing handcrafted rules. Ultimately, however, a security risk remains. Sending data to a cloud LLM to find PII is itself a disclosure, and undermines the initial need for detecting and protecting PII. The answer is to run detection engines locally.

A company could develop a perfectly tuned detection model for its own uses and needs. But when the company expands into a new vertical, or wants to use its AI for a new purpose, that detection model would have to be retrained. For organisations, it comes down to “Build or Buy?”, often with no clear answer on which would cost more or bring a better performance for their requirements from AI. A redaction pipeline is only as good as what it finds, and a detection engine is only as good as the data it’s trained on. Each door opens another, and it’s always about data.

What are your experiences with domain-specific detection?

Sources

Sources