AI & ML
PrivacyPR: A Dataset of Real-World Privacy Fixes for Benchmarking Privacy-Aware Code Review
arXiv cs.CR0 views
arXiv:2611.18293v1 Announce Type: new Abstract: Coding agents increasingly review code, yet whether they catch the privacy defects that laws such as the GDPR require software to avoid remains largely untested, partly because few datasets connect real privacy defects to their accepted fixes and review context. We present PrivacyPR, a dataset of 4,131 merged GitHub pull requests (PRs) from 1,469 repositories, created between June 2018 and January 2024. Each PR includes its commits, diffs, review discussion, and LLM-generated labels for privacy relevance, fix type, and GDPR principles. PrivacyPR provides 1,169 high-confidence privacy fixes and 1,643 non-fix PRs, together with Verified, a manually checked split of 250 privacy fixes and 250 non-fix PRs, and Small, a 100 + 100 subset for evaluation. On Verified, both annotators endorsed 72% of the LLM's GDPR-principle assignments. Beyond supporting empirical study of how privacy defects arise and are fixed, PrivacyPR provides a benchmark for privacy-aware code review on real, developer-accepted fixes. Its evaluation harness reverts each fix to reintroduce the defect. On Small, two coding agents detect only 42-51% of defects under a generic review prompt, but 88-91% when explicitly asked to check privacy, while flagging 11-15% of non-fix PRs. These results show that privacy defects can be detected by coding agents, but that detection depends strongly on review intent and comes with false alarms. Data, code, reviews, judge verdicts, and an extensible evaluation harness are publicly available at https://doi.org/10.5281/zenodo.23744765.
Source:https://arxiv.org/abs/2611.18293PreviousFrom Bytes to Behavior: Request-Level Attribution from Live GPU Hidden StatesNextOn the Same Track: Privacy Leaks in Electric Rail Transport via Magnetic Fields
Related posts
AI-Native Development Framework for European Public Organisations (AI SDLC - Model-Adaptive)
0dev.to Machine Learning
How to Trim Audio for a Fundraiser Video — Free, in Your Browser
0dev.to Tutorial
WARN Act and Layoffs in Grocery Stores: What Supermarket Workers Should Know
0dev.to Career
How to Handle "Not Interested" on a Cold Call: Cold Calling Tips and Sales Scripts for SDRs and Founders
0dev.to Career
How to Choose the Best Web Design Company for Your Business
0dev.to Career
The Government’s New EV Mandate: What Doubling the Staff Car Cap Means for Auto Stocks
0dev.to Beginners