Deep learning applications to automate phenotypic measurements on biodiversity datasets

The growing number of digitised biological specimens brings new possibilities for the study of a wide range of evolutionary questions at broad scales. Taking measurements on digital photos often requires annotations (e.g. placing points on focal locations), and many projects label their digitised specimen datasets (commonly more than thousands of images) manually, which could take a huge amount of time. Deep learning is the state-of-the-art for many computer vision tasks. Deep learning models can be trained on a set of manually annotated images, and can make accurate predictions based on what they learned. To what extent deep learning can help to improve the measurement process on digitised collections has yet to be thoroughly explored. Here, I have applied deep learning models to three tasks on two datasets (bird specimens and Littorina shell images), and show that predicted labels are remarkably accurate and that downstream biological analyses using these labels generated biologically meaningful results. First, I used pose estimation models (algorithms originally designed to identify human body parts) to locate keypoints on bird specimens. The results showed high accuracy with 95% of the validation images (N=5,094) correctly predicted, and rapid generation of data (less than three days to predict keypoints on the whole dataset of >120,000 images). Colours measured by points showed that male birds tend to be more colour-diverse than females. Second, I applied deep learning models to segment the overall plumage areas on the bird dataset. More than 95% of the plumage areas were correctly segmented, and it also took less than three days to segment the whole dataset. I found that colour diversities (calculated from segmentations) among closely related birds tend to be similar across more than 7,500 bird species. Finally, I built PhenoLearn, a user friendly tool that provides functions such as manual annotation (to create training images), predicting using deep learning, and reviewing predictions. I illustrate the broader applicability of PhenoLearn in an example of morphological landmarking on Littorina shell images. More than 98% of the predicted landmarks were placed within the acceptable range. The methods and tools introduced here both illustrate the value of deep learning and significantly increases accessibility to deep learning approaches for non-expert biologists allowing the rapid accumulation of phenotypic datasets at large scales. Taken together, these results show that deep learning methods have great potential for speeding up the measuring process on digitised specimens while producing accurate annotations.