User-agent: *
Sitemap: https://etheses.whiterose.ac.uk/sitemap.xml
Disallow: /cgi/

User-agent: DataForSeoBot
Crawl-delay: 5