Hardness-aware learning: taxonomy, benchmarking, and fairness applications

Class bias—systematic recall gaps across classes—continues to limit the fairness of classification models. Most explanations attribute this bias to data imbalance, assuming that unequal sample counts are its primary cause. However, this interpretation overlooks differences in class hardness that persist even in balanced datasets. Since prior studies rarely separate these effects, it remains unclear whether reported fairness gains arise from one or the other, constraining advances in hardness-aware learning. The focus of this thesis is twofold: bringing coherence to a fragmented field and showcasing its practical value.

To systematize existing approaches, we first review model- and data-based hardness estimators, which have evolved as largely independent research lines. We introduce a unified taxonomy that organizes both families and the types of hard samples they identify, providing a conceptual foundation for hardness-aware research. Building on this, we perform the first comparative benchmark between these two families, highlighting key limitations of data-based estimation. A complementary stability analysis further shows that single-seed evaluations—common in prior work—produce unreliable hardness rankings, motivating ensemble-based estimation.

To demonstrate why hardness-aware learning matters, we conduct two case studies assessing whether hardness-based resampling can improve fairness in balanced datasets. We find that it consistently reduces class-level recall and F1 gaps and can also enhance overall performance under suitable conditions. These outcomes depend on three key factors: the accuracy of hardness estimation, the quality of oversampled data, and the degree of induced data imbalance. Together, these findings establish hardness imbalance as a distinct and measurable phenomenon and position hardness-aware learning as a principled framework for studying and mitigating it.