AI & ML
LEGIT: Live Eligibility Gating over Identifier Tries for Constrained Decoding of Heterogeneous Pages
arXiv cs.IR0 views
arXiv:2611.10357v2 Announce Type: replace Abstract: A generative recommender names each candidate with a short token sequence, a Semantic ID, and recommends by generating those tokens. Some generative recommenders, including the Uber Eats search engine we are building, write a whole results page, stores with their menu items, in one decoding pass. Every Semantic ID on such a page must name a real store or item that is eligible for the search, such as a store that delivers to the user's location, and that is not already on the page. Each item must also belong to the store above it. A trie over the catalog keeps identifiers real, but eligibility and repetition change with every request and token, and the grammar-constrained decoder we tried could not enforce these changing rules exactly and quickly. We call this constraint LEGIT, for live eligibility gating over identifier tries. Numbering trie leaves depth first turns each rule into a range lookup over one store trie and one item trie, and every generated page passes an independent catalog check. On one NVIDIA B200 GPU with one or two requests per decode step, a direct implementation that walks the trie on the GPU makes each token about 5 times as slow as unconstrained decoding. Across 36 implementations, hardware counters and ablations are consistent with kernel launches and CPU-GPU waits as the source of this slowdown. Our fastest implementation walks the trie on the CPU while the GPU runs the model, needs no custom kernel, and cuts time per output token by 65.9 percent against the direct GPU implementation, to about 1.7 times the unconstrained cost. This implementation is also the fastest on a page layout that puts each item before its store.
Source:https://arxiv.org/abs/2611.10357PreviousPrompt-Based Abstention Fails Under Misleading Context: A Controlled Study of Small Frozen RAG ModelsNextChi-Square Wavelet Graph Neural Networks for Heterogeneous Graph Anomaly Detection
Related posts
In which John becomes a Machine Learning researcher
0Hacker News New
Can We Fix AI?
0Hacker News New
I'm Glad That Literary Awards Are Going to AI
0Hacker News New
Show HN: TalentRadar – 3D geospatial tech compensation and role radar
0Hacker News Show
Show HN: Brief2mp4 – ClaudeCode/Codex skill that checks videos against the brief
0Hacker News Show
Show HN: Figma Like AI Landing Page Builder
0Hacker News Show