AI & ML
Analyzing Code Provenance in LLMs for Programming Education
arXiv cs.SE0 views
arXiv:2611.09177v1 Announce Type: cross Abstract: Large language models (LLMs) play an increasing role in programming education, challenging traditional assessment methods as they can generate fully functional code instantly. This disrupts the reliability of student submissions as indicators of learning and effort. To address this issue, code provenance is essential for tracing the origins of code and distinguishing student-authored, AI-generated, and externally sourced components. We extend the Min-K% probability approach by incorporating synthetic non-member code generation and apply it to Python code for membership inference and provenance tracking. Evaluations cover free LLMs such as GPT-2, DistilGPT-2, and DialoGPT, as well as state-of-the-art commercial models including GPT-5, Gemini 2.5, Mistral, DeepSeek, and LLaMA 3.3. Before evaluating our approach on higher-education student assignments, we validate its feasibility using Stack Overflow data. The results show that free models enable more sensitive code memorization detection than commercial models. By systematically comparing these LLMs, we assess trade-offs between detection accuracy and privacy protection, taking an initial step toward effective assessment of AI-assisted programming education through combined provenance detection and process visibility.
Source:https://arxiv.org/abs/2611.09177PreviousPdb or Not to Be: Comparing Debugging Interfaces With Pass$_{X}$@$k$NextAutoRefine: Auto-Calibrating Code Judges from Workflow Outcomes
Related posts

Nvidia drops ~5% pre-market after its board opens a probe into Jensen Huang; Nvidia previously dismissed the interactions as "nothing unusual or unexpected" (Demetri Sevastopulo/Financial Times)
0Techmeme

Nvidia Senior Director Madison Huang, daughter of Jensen, cancels a high-profile Seoul appearance citing a "family matter", throwing the event into disarray (Yoolim Lee/Bloomberg)
0Techmeme

Jensen Huang maintains deep control over Nvidia, managing 60+ direct reports and owning a 3.56% stake, presenting a challenge for Nvidia's 10-person board (New York Times)
0Techmeme

Amsterdam police arrested between 150 and 200 protesters for trespassing and violence at a Microsoft data center construction site owned by Pure Data Centers (Jowi Morales/Tom's Hardware)
0Techmeme

The US OCC grants Coinbase full approval for a trust bank charter; sources say it could expand Coinbase's stablecoin business and let it issue stablecoins (Olga Kharif/Bloomberg)
0Techmeme

US voters in Harwood, North Dakota rejected a ballot measure that opposed Applied Digital's $3B data center, set to open in 2027, bucking a nationwide backlash (Miriam Waldvogel/Washington Post)
0Techmeme