Classification Trees Based on Distance Measures and Discriminant Analysis

Classification trees are one of the most important supervised machine learning methods used for classification problems. Some traditional classification trees, such as classification and regression trees (CART), ID3, and C4.5 are widely used in practice. To overcome the limitations of traditional classification trees using univariate splitting rules, multivariate classification trees have been developed. Distance-based methods and discriminant-based methods are two important directions of multivariate trees. We propose four classification trees based on distance measures and discriminant functions, namely Euclidean Distance-Based Classification Trees (EDBCT), Mahalanobis Distance-Based Classification Trees (MDBCT), Linear Discriminant Analysis-Based Classification Trees (LDCT), and Quadratic Discriminant Analysis-Based Classification Trees (QDCT). We develop a modified cost-complexity pruning method for our proposed classification trees. The performance of the proposed methods is evaluated on simulated and real-world data and compared with classic CART. The proposed classification trees can achieve competitive predictive performance while maintaining relatively simple tree structures in some simulation settings, particularly for datasets with non-axis-parallel class boundaries. The proposed pruning method successfully helps the proposed classification trees avoid overfitting. In addition, we propose several ReliefF-based feature selection methods. The proposed feature selection methods identify the redundant and extreme-points contaminated features effectively. Finally, we propose a weighting formulation and a Weighted Linear Discriminant Analysis-Based Classification Trees (WLDCT), which is an extension of proposed Linear Discriminant Analysis-Based Classification Trees. The proposed weighted classification trees provide predictive performance comparable to LDCT while offering a mechanism for incorporating information from previous nodes into subsequent splitting decisions.