AI & ML
Sycamore: Grounded Synthetic Evaluation for Visualization Systems
arXiv cs.HC0 views
arXiv:2605.08630v4 Announce Type: replace Abstract: Evaluating visualization systems in specialized domains, such as genomics, is challenging due to the scarcity of domain experts and difficulty recruiting relevant user types. LLM-based synthetic personas provide an appealing workaround, but they face skepticism as their claims often lack verifiable grounding. We present Sycamore, an exploratory approach to evaluating visualization systems with grounded synthetic personas whose responses are restricted to interview evidence. Sycamore enforces provenance with participant citations for each claim and an abstention mechanism when retrieval yields no relevant evidence. A protocol runner guides the synthetic personas through an evaluation of the live visualization system, producing multi-turn transcripts for qualitative coding. We instantiate the approach on Geranium, a multimodal genomics visualization search engine, with seven evaluators grounded in prior interviews on genomics visualization authoring. We report themes in the resulting transcripts, highlight an emergent requirement absent from the source data, and provide a preliminary comparison to a prior human user study. Recognizing that our findings derive from an exploratory perspective and that synthetic proxies cannot replace human evaluation, we discuss their implications for integrating synthetic evaluators alongside expert user studies. The protocol, codebook, extended results, transcripts, and demo video are in the Supplementary Materials and at https://osf.io/kdfr3/; the source code and project page are at https://huyen-nguyen.github.io/sycamore/.
Source:https://arxiv.org/abs/2605.08630PreviousMap2Video: Guiding Real-World-Grounded AI Video GenerationNextProfiling Handwriting-Process Deviations in Developmental Dysgraphia: An Open, Normatively-Referenced Instrument
Related posts
Trying to get our Arms around AI efficiency
0Stack Overflow Blog
Why an Embedded Game Fails on Some Phones, and How to Debug It
0dev.to
Microsoft 365 vs Office 2024: The Break-Even Math
0dev.to
Master AI Evals: Build Reliable, Scalable Model Evaluation Pipelines
0dev.to
What to give someone who misses a place
0dev.to
Google is pushing OEMs against GrapheneOS and bootloader unlocking
0Hacker News