Describe which areas of your website should and shouldn’t be visited by the crawler in the robots.txt file. Supply our tool with all the essential information and let it produce the file you need by hitting the “Submit” button. Robots.txt is the short form used by SEOs and TopologySEO webmasters to describe the robots exclusion standard. What this means is that the robots.txt instructs the search engine spiders, robots which areas of a website they should not visit.