Locality-Aware Scheduling of Software Repository Mining Workflows in Heterogeneous Environments

This thesis investigates advancements in job scheduling within heterogeneous computing environments, focusing particularly on the domains of software repository analysis. Two main contributions are presented: the Bidding Scheduler and the self-correcting worker mechanism, denoted as the Feedback-Based Estimator.

Firstly, the Bidding Scheduler is introduced as a novel approach that decentralises the job allocation process. In this model, worker nodes engage in a bidding process to acquire jobs, leveraging localised data and workload conditions to submit bids that reflect expected processing times. The master node collects these bids and allocates jobs in a manner that minimises overall execution time and optimises resource utilisation. This approach addresses the common inefficiencies in traditional master-worker architectures by integrating both data locality and worker heterogeneity into the decision-making process. Experimental work demonstrates the scheduler's ability to adapt to dynamic workloads and varying node capacities, resulting in improved performance metrics.

Secondly, the Feedback-Based Estimator mechanism is developed to enhance the accuracy of job completion time estimates. Acknowledging the limitations in developers' ability to predict precise processing times due to the variability in job attributes and system performance, this mechanism introduces a real-time adjustment process. Based on continuous feedback from executed jobs, the Feedback-Based Estimator dynamically updates the estimation formula to correct any persistent discrepancies between estimated and actual job completion times, thereby reducing the margin of error in time estimates.

Together, these contributions provide a framework for efficient and adaptive job scheduling in heterogeneous computational environments. Future work will focus on investigating applicability of these approaches beyond mining software repositories.