AI & ML
INTENT: Out-of-Sample, Out-of-Time, and Cold-Start Evaluation on User Event Sequences
arXiv cs.IR0 views
arXiv:2611.13069v1 Announce Type: cross Abstract: Banks now train foundation models on ordered sequences of customer events such as transactions, app sessions, and authorizations. Deployed, such a model predicts forward in time, often for customers and merchants absent from training, yet each is evaluated on private data, so no two can be compared. Public benchmarks for such sequences rarely test a period the model has not seen; recommendation studies usually hold out each user's last events instead. We present INTENT (INTEraction, Novelty, and Transfer evaluation for next-action prediction), a public benchmark that scores a model in seven cells defined by whether the user, the period, and the target item appeared in training. An audit of 59 public dataset families retains five datasets; the most common defect is the lack of a calendar timestamp that orders events. No date lies within the time span of all five datasets, so each is split at its own cutoff, chosen by one rule fixed before any model is evaluated. Summed over the datasets, every cell holds at least 111,769 users. In reference results the cell determines the best model: a transformer leads on seen items after the cutoff but scores zero on new items, a variant that embeds item metadata helps mainly on MicroLens, and repeating the user's last item leads on new items in four of five datasets. This parameter-free control also shows that on three datasets most of the score lost after the cutoff reflects changed behavior rather than the model. The pipeline, the cutoff rule, and the per-cell user counts are publicly available.
Source:https://arxiv.org/abs/2611.13069PreviousIntentTwin: Deconfounding Candidate Support and the Oracle-to-Deployment Gap in Offline Web Behavior PredictionNextActivation-Guided Graph Retrieval for Cognitive Memory Reconstruction in Language Model Systems
Related posts
When Words and Bodies Disagree: Realizing Verbal and Nonverbal Mismatch
0arXiv cs.HC
AnyMo: Geometry-Aware Setup-Agnostic Modeling of Human Motion in the Wild
0arXiv cs.HC
EEG-X: Toward Device-Agnostic and Noise-Robust Foundation Model for EEG
0arXiv cs.HC
Minority Reports: Balancing Cost and Quality in Ground Truth Data Annotation
0arXiv cs.HC
Whose Guardrail? User Identity Shapes the Effectiveness of Model Disclaimers in Mitigating LLM Political Persuasion
0arXiv cs.HC
Art2Song: Enhancing Visual Art Appreciation with Contextual Music Generation
0arXiv cs.HC