AI & ML
Asymptotic Margins and Feature Confinement in Conic Classification Heads under Softmax Cross-Entropy
arXiv eess.SP0 views
arXiv:2611.18507v1 Announce Type: cross Abstract: Classification heads built on conic functions give each class a bounded region of the feature space, and such regions are used to reject samples of unseen classes. We ask whether these regions also keep the features bounded when the head is trained with softmax cross-entropy. For a head whose class scores combine a linear term with a Euclidean norm, we first show that the region of each class is bounded by a quadric of revolution, whose eccentricity is the reciprocal of one learned ratio. We then attach to each class an asymptotic margin, which compares the growth of its score with the growth of all the other scores and can be computed by a second-order cone program. The loss of a class has infimum zero exactly when its margin is positive, and it then approaches zero only along diverging features. When the margin is negative, the loss is coercive and its minimizers lie in an explicit ball. Since softmax depends only on the differences between scores, a class with a bounded region can still have a positive margin, so bounded regions do not confine the features. The result extends to other norms, including the polyhedral conic functions. On CIFAR-10 and CIFAR-100, all 1210 cells of 67 trained conic heads have positive margins, whereas a prototype head keeps nine of its ten cells bounded, as the theory predicts.
Source:https://arxiv.org/abs/2611.18507PreviousPACT: A Decoder-Confidence-Guided Two-Round Receiver for Joint Channel and Phase-Noise Estimation in Sub-THz OFDMNextMulti-Band Sensing in FR3 with Background Dense Multipath Components
Related posts
Kusner's conjecture on equilateral sets is false
0arXiv cs.NA
Nearly Optimal Fixed-Confidence Best-Arm Identification with 1-Bit Feedback
0arXiv stat.ML
When Does On-Policy Interaction Help? Representational Tradeoffs in Value-Based Imitation Learning
0arXiv stat.ML
Disentangling Continuous-Time Latent Dynamics: Identifiability of Latent SDEs via Diffusion Shifts
0arXiv stat.ML
Exact Constrained Decoding for Sequence Models: Controller-Augmented Hidden Markov Models and Beyond
0arXiv stat.ML
Kernel-based potential mean-field games with unbiased random Fourier $U$-statistics
0arXiv stat.ML