Optimisation of cloud data centres and networking for big data applications

Cloud data centre applications including big data applications and video content distribution services are rapidly evolving in terms of their data formats, frameworks, workloads, and data volumes which has resulted in a substantial increase in their storage and processing requirements. These applications and services usually utilise commodity clusters within geographically- distributed data centres. The increase in traffic between and within the data centres that migrate, store, and process big data and video content, is becoming a bottleneck that calls for enhanced infrastructures capable of reducing the congestion and power consumption. Moreover, it has resulted in the consideration of fog data centres in access networks in addition to the cloud data centres to reduce the delay and power consumption associated with delivering services. This thesis addresses the evaluation of the energy consumption and performance when deploying big data frameworks in intra data centre networks and the optimisation of delivering video demands through transport networks from cloud or fog data centres. First, two energy-efficient Passive Optical Network (PON)-based data centre designs which are a switch-centric design and a server-centric design are introduced. For big data applications, the first part of the work in this thesis focuses on the impact of data centre architectures on the performance, resilience, and energy efficiency of intra data centre networks. A time slotted Mixed Integer Linear Programming (MILP) model that optimises the scheduling and routing of big data applications traffic was developed and utilised to compare the energy efficiency and the performance of four state-of-the-art data centres, in addition to the two PON-based data centres. Two different objectives are considered focusing on the minimisation of the completion time and the minimisation of the energy consumption. The comparisons are performed while considering a number of big data application-related parameters and data centre related network architecture parameters. The results indicate that the topology has a significant impact on the performance, energy consumption, and the resilience of the data centre. Thus, investing in data centre designs is essential to meet future demands. For video-on-demand services, this thesis considers caching the content near the users in fog data centres to reduce the brown networking power consumption required to deliver the contents from cloud data centres. A MILP model is developed to optimse the delivery of content from cloud or fog data centres. The reduction in the brown power consumption is evaluated while considered several factors such as the core network design, the Power Usage Effectiveness (PUE) of the data centres, in addition to the use of renewable energy in the cloud data centres, and solar energy with Energy Storage Devices (ESDs) in the fog data centres.