# https://www.robotstxt.org/robotstxt.html User-agent: Go-http-client/1.1 Disallow: / User-agent: * Allow: