Empirical risk minimization with f-divergence regularization for machine learning

This work contributes to the field of statistical machine learning by providing a theoretical characterization of the role of regularization in supervised learning through the lens of information measures. The asymmetry of the relative entropy is analyzed in the context of its role in the regularization of empirical risk minimization. Building on this insight, a broad family of f-divergences is introduced as potential regularizers for empirical risk minimization. Under mild assumptions, solutions for general f-divergences are derived, and the concept of the normalization function is formally defined. Furthermore, a dual optimization problem associated with empirical risk minimization using f-divergence regularization is explored. By studying the normalization function, it is demonstrated that the duality gap is zero, and insights from the dual formulation are used to derive explicit expressions for the generalization error of general statistical learning algorithms in terms of f-divergence-regularized learning frameworks.