arXiv:2605.12026v2 Announce Type: replace-cross Abstract: Vision transformers tokenize an image into patches, which couples the token count to resolution and requires the embedding to be learned from data. We propose a spectral vision transformer architecture for efficient tokenization in the limited-data regime, with an emphasis on medical imaging. Rather than using spatial patches, we tokenize into coefficients of a fixed basis. This approach defines each token as one spectral mode, and the sequence length is chosen rather than implied by resolution, reducing the cost of attention. Our proposed spectral ViT shows superior or comparable performance where data is scarce in public medical imaging benchmarks, natural images, object shapes, simulations, and protected clinical data. We release our code at: https://github.com/agr78/SpectralViT.