π‘οΈ KVKK-Bench
Under KVKK, Turkey's personal data law, sending a client's name, ID number or health data to a cloud AI service can be an unlawful transfer. This board ranks the tools that mask personal data on your own computer first: same real texts, same scorer for every model.
Dataset card Β· Scorer Β· Raw results Β· KVKK, Law No. 6698 Β· 15 models Β· v0.1.2 preview
KVKK (Law No. 6698, 2016; English text) protects any information about an identified or identifiable person. For a lawyer, doctor or HR team using AI, three rules matter most:
- Only what is necessary (Art. 4): process no more personal data than the purpose needs.
- Special categories (Art. 6): health, biometric and genetic data, criminal convictions, religion or belief, ethnic origin, political opinion, union or association membership, sex life, and appearance and dress. These need stricter grounds and extra safeguards.
- Transfer abroad (Art. 9, amended in 2024): most cloud AI services process data outside Turkey. That needs an adequacy decision, standard contractual clauses notified to the KVKK Authority, or a narrow exception. Fines are set in Art. 18.
Masking on your own computer before anything leaves it is the simplest safeguard: whatever has been masked is not sent.
What the board tests
| Examples | Status | |
|---|---|---|
| Identity | names (Turkish and foreign) | ranked, on real text |
| ID numbers | T.C. kimlik, foreign ID (99β¦), passport, tax number | synthetic data β (not in the score) |
| Contact and financial | phone, e-mail, address (building/door), IBAN | synthetic data β (not in the score) |
| Special category | health terms and diagnoses | synthetic data β (not in the score) |
Why only names on real text: published Turkish court decisions are anonymised before release, so ID numbers, phones, IBANs and addresses are removed from them, and real documents that still contain them can't be published by definition. Real decisions do mention health (injury reports, diagnoses); labelling those is next. β columns stay out of the score until real labelled data exists. This is a technical benchmark, not legal advice: masking reduces what is sent, but it doesn't by itself make a document anonymous.
| # | T | Model | Overall β¬ | Leaked β¬ | Pages/s β¬ | Size β¬ | πΉπ· Names | π Names | π° Names | Offline | In browser | Licence |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | πΆ | Katip Mask Base Β· tr v2 | 96.1 | 8.2 | 1.7 | 111 MB | 99.9 | 97.7 | 84.8 | β | β | CC BY-SA 4.0 |
| 2 | πΆ | XLM-RoBERTa Large NER | 95.9 | 3.4 | 2.3 | 2.2 GB | 97.6 | 97.5 | 89.9 | β | MIT | |
| 3 | πΆ | XLM-RoBERTa Base Turkish NER | 95.7 | 6.6 | 5.6 | 1.1 GB | 98.5 | 98.2 | 86.3 | β | MIT | |
| 4 | πΆ | GLiNER2 PII Multi | 95.0 | 5.8 | 1.2 | 1.2 GB | 98.1 | 98.2 | 83.9 | β | Apache-2.0 | |
| 5 | πΆ | Katip Mask Mini Β· tr v2 | 92.8 | 11.9 | 5.1 | 31 MB | 99.1 | 97.2 | 72.5 | β | β | CC BY-SA 4.0 |
| 6 | πΆ | BERTurk NERbaseline | 91.6 | 8.1 | 6.0 | 440 MB | 97.4 | 91.7 | 77.1 | β | not stated | |
| 7 | πΆ | GLiNER Multi PII v1 | 91.3 | 10.5 | 1.1 | 1.2 GB | 93.1 | 92.9 | 85.2 | β | Apache-2.0 | |
| 8 | πΆ | Katip Mask Nano Β· tr v1.1 | 87.5 | 18.4 | 32 | 3 MB | 98.2 | 95.5 | 52.9 | β | β | CC BY-SA 4.0 |
| 9 | πΆ | ModernBERT-TR PII NER | 85.3 | 21.6 | 4.8 | 598 MB | 99.0 | 96.9 | 39.4 | β | Apache-2.0 | |
| 10 | π¬ | Qwen2.5 1.5B Instruct (zero-shot) | 57.5 | 41.1 | <0.1 | 3.1 GB | 59.1 | 57.6 | 53.7 | β | Apache-2.0 | |
| 11 | πΆ | Privacy Filter Multilingual v2 | 57.2 | 36.4 | 0.2 | 2.8 GB | 59.8 | 57.4 | 50.7 | β | Apache-2.0 | |
| 12 | πΆ | Privacy Filter | 51.3 | 52.5 | 0.1 | 2.8 GB | 58.0 | 58.0 | 27.9 | β | Apache-2.0 | |
| 13 | πΆ | Piiranha v1 | 49.8 | 45.4 | 2.9 | 1.1 GB | 41.4 | 61.5 | 59.4 | β | CC BY-NC-ND 4.0 | |
| 14 | βοΈ | Presidiobaseline | 45.0 | 31.7 | 31 | 11 MB | 39.3 | 44.0 | 60.4 | β | MIT | |
| 15 | πΆ | OpenMed PII Turkish Small | 19.1 | 73.0 | 7.5 | 133 MB | 11.8 | 18.8 | 37.5 | β | Apache-2.0 |
Overall = weighted average of the masking columns, each scored as F2 (a miss counts 4Γ more than an extra mask): πΉπ· Names 56% Β· π Names 22% Β· π° Names 22%. ID, contact, address and health columns are hidden until we have real labelled documents for them (our generated ones would favour Katip); open Columns to see the β previews.
Found = share of personal data the tool masked. Precision = share of masks that were really personal data. Leaked = share of personal data still readable. Docs fully clean = documents with nothing left readable. False alarms = masks on words that weren't personal data, per 1,000 words. The type columns show how much of each kind was found (the number in brackets is how many there are).
Top-left is best: high score, small and fast.
KVKK-Bench measures how well tools find the personal data that KVKK protects, so it can be masked on your own computer before a document is shared, for example with a cloud AI (see What does KVKK require above). Every tool gets the same documents and the same scorer.
Test data (v0.1.2, public)
- 327 YargΔ±tay sentences from the test split of TurkishLegalNER (KoΓ§ University, Apache-2.0), used twice: the anonymised name slots are refilled once with unseen Turkish names (πΉπ·) and once with unseen foreign names (π), 1,011 names each.
- 1,000 Turkish Wikipedia sentences with hand-marked names: Turkish Wiki NER (Altinok 2023, CC BY-SA 4.0).
- 380 generated legal documents with ID numbers, IBANs, phones, addresses and health data. These feed the β preview columns only.
Thanks to the dataset authors, and to every model developer credited under the model names.
A hidden test of 2025β26 court decisions is in preparation and will replace the public score.
Metrics. Overall is a weighted average of the real-text columns, each scored as F2. A value counts as found if the mask covers it, or at least half of it, with the right type. Leaked is the share of personal-data characters left readable. Hover a column name for its definition.
Links: dataset card Β· scorer Β· all results Β· report a label error
Cite: see Cite KVKK-Bench at the bottom of the page.
Enter a Hugging Face token-classification model. We evaluate it offline and add it to the table.
Please also cite the models you compare, using the links in each row.