arXiv:2510.01433v3 Announce Type: replace Abstract: Vision-based manipulation policies are commonly learned from rich visual observations, typically using dense visual features extracted from full resolution images. It remains unclear how much of this visual information is really needed for effective policy learning. In this paper, we investigate the opposite end of the spectrum: how effective can manipulation policies be if they use as input only a compact visual representation. Sparse 2D keypoints provide a deliberately compact representation, giving the policy spatial information while removing direct access to appearance, texture, and background. We design AFFORD2ACT (Affordance-Guided Keypoints to Act) to study how much manipulation capability can be retained under this representation constraint. An off-the-shelf language-conditioned affordance model identifies the task-relevant region, from which up to 15 object keypoints are selected, together with 4 end-effector keypoints. The policy input is therefore reduced to only 19 2D keypoints (38 coordinates). A lightweight gating further decides which keypoints to pay attention to during execution. Across six real-robot manipulation tasks, this compact state outperforms the evaluated RGB, RGB-D, and dense-affordance behavior cloning baselines, degrades less as the number of demonstrations is reduced, and achieves 82% average success in held-out settings. Replacing learned gating with uniform weights reduces success from 92% to 52%. At the same time, the representation depends on reliable affordance localization and keypoint tracking, and remains vulnerable to incorrect masks, occlusion, tracking drift, and distractors.