AI & ML
A new categorical correlation measure
arXiv stat.TH0 views
arXiv:2610.34932v1 Announce Type: cross Abstract: Feature screening in high-dimensional classification is an important problem in statistical learning, particularly in genomics, imaging, and other scientific fields, especially when the number of predictors exceeds the sample size. Zhong et al. (2024) present a semi-distance covariance (SDCov) as a dependence measure between a categorical random variable and a multivariate continuous random vector, presenting it as a promising tool for group-wise feature screening. However, SDCov is defined using the Euclidean norm, and its performance may be affected when the data is derived from heavy-tailed distributions or when outlier observations are present, which is a common phenomenon in high-dimensional data. This study proposes a modification of SDCov by replacing the $L_2$ norm with an $L_1$ norm. This adjustment results in a dependency measure between a categorical variable and a multivariate continuous variable that maintains the independence-characterization property of the original definition. Additionally, it serves as a robust and interpretable measure, particularly in the presence of outliers in the data. We establish the fundamental theoretical properties of the proposed measure and introduce a U-statistics-based estimator along with JEL-based inference for the measure. Additionally, we have introduced a test for independence based on the proposed measure. An extensive Monte Carlo simulation is performed to evaluate the effectiveness of the proposed measure in comparison to existing measures in the literature. The proposed measure is applied in feature screening, as illustrated with the ADNI Alzheimer's disease dataset and the Iris dataset. The data analysis shows the proposed measure's effectiveness in identifying variables that facilitate strong class separation of the respective response variables.
Source:https://arxiv.org/abs/2610.34932PreviousThe German Tank Problem & Navigating by ScentNextComparing and measuring intermediate tail dependence
Related posts
Hybrid retrieval with reciprocal rank fusion: notes from a RAG prototype
0dev.to
It is in co-creating, together. As humans do.
0dev.to
C64 Basic Dungeon Crawler: Stairs, Levels and Levelling Up (Part 11)
0Hacker News
Anthropic's internal AI use, at retail prices, now exceeds its employee pay
0Hacker News
Launch HN: Momentic (YC W24) – Agents that catch bugs by using your app
0Hacker News

Behavioral Design Unpacked: Nudges, Funnels & Persuasion | UX Unpacked #3
0dev.to UX