Deep convolutional networks without backpropagation

This thesis attempts to develop networks trained without gradient descent or backpropagation designed specifically for classification tasks. The emergence of issues with gradient-based neural networks, such as long training time, vanishing or exploding gradients and high computational costs, has led to the development of such alternatives. In fact, the works presented in this thesis
extend PCANet, with the fundamental objective being the development of networks capable of providing both good performance and significant improvements in network depth. Chapter 1 of this thesis formulates the problem, describes the challenges, outlines the research questions and summarises the contributions. In Chapter 2, gradient-based and non-gradient-based networks are reviewed. Chapter 3 presents the Multi-Layer PCANet, whose design is inspired by that of PCANet. However, using second-order pooling and CNNlike filters, the evaluation experiments indicate that the proposed network provide a considerable reduction in the number of features and, consequently, a gain in performance. The networks in Chapters 4 and 5 share the same design as the Multi-Layer PCANet but generate their filter banks using different supervised learning approaches. The experimental results on four databases (CIFAR-10, CIFAR-100, MNIST and TinyImageNet) show that semi-supervised Stacked-LDA filters are sufficient for providing good data representation in the convolutional layers. These filters are produced by combining 50% PCA filters (Chapter 3) with 50% Stacked-LDA filters (Chapter 4). Chapter 6 introduces deep residual compensation convolutional networks for image classification. The design of this network comprises several convolutional layers, each post-processed and trained with new labels learned from the residual information of all preceding layers. The evaluation experiments indicate that the proposed network is competitive with standard gradient-based networks not only in terms of accuracy but also in the number of FLOPs required for training. Chapter 7 summarises the findings and discusses the field’s potential future directions.