Accelerating Deep Learning by Optimising Communication and Computation Efficiency

Deep Neural Networks (DNNs) underpin many of today’s business and scientific innovations. However, realising the full potential of DNNs and making them more accessible requires tackling multiple challenges across the computing stack. These include improving the efficiency of distributed communication across computing devices and computation on a single computing node. This thesis presents new optimisations for DNN training across multiple distributed machines and on a single machine. Our techniques reduce DNN training times, making DNNs faster and more efficient.

First, we improve the communication efficiency of distributed training through multi-streamed gradient communication. Our approach allows a training worker to participate in multiple gradient communication operations simultaneously to improve network bandwidth utilisation and reduce communication latency, achieving up to 13.4x training throughput compared to existing distributed training frameworks.

Next, we develop new techniques to improve DNNs’ performance on single machines, using Graph Neural Networks (GNNs) as a case study. While GNNs are a powerful tool for processing graph data, their scalability is hindered by the large memory footprint typically required to process real-life datasets. To mitigate this issue, we leverage two kernel-based memory-saving strategies: lossless sparse matrix storage and Tensor-train Decomposition (TTD), which trades accuracy for memory efficiency. Our goal is to accelerate the computation of these memory-saving techniques. For sparse matrix storage, we enhance GNN kernels by introducing a machine-learning-based method to dynamically select the optimal sparse storage format based on the input data, improving sparse GNN kernels by up to 3x. To accelerate TTD-based GNNs, we optimise data reuse and kernel invocations, finding optimal TTD configurations to improve training convergence. Our optimisations improve TTD-based GNN training throughput by up to 8.17x, reducing the GNN memory footprint by up to 5,474x without compromising
model accuracy.