I am in the process of revising my site, and discovered for whatever reason that I had an empty robots.txt file present. I know it is only a voluntary ‘standard’, but as far as I know all the major players do respect it. As the overwhelming proportion of users use a search engine that respects the standard, it does form a useful way of shaping what shows up in the general public eye.
I can never remember the syntax though, so for your reference and my recollection – http://www.robotstxt.org
Addendum: I was not familiar with the semi-standard for site maps so I’ve added that as well to see what the effect will be.
Addendum:Ritta Blens has pointed me to another very useful tool for testing the structure of a robots.txt file: https://www.websiteplanet.com/webtools/robots-txt/
Post a Comment