About Robots Tester
The robots.txt file controls which URLs search engine crawlers can access. Rules are matched from most specific to least specific. A matching Allow always wins over Disallow for the same path length. The * wildcard matches any string. Disallowing a URL does not prevent it from appearing in search results if other pages link to it — use noindex in the page's meta tags for that.