Skip to content

Robots. They are coming to take your content.

I am in the process of revising my site, and discovered for whatever reason that I had an empty robots.txt file present. I know it is only a voluntary ‘standard’, but as far as I know all the major players do respect it. As the overwhelming proportion of users use a search engine that respects the standard, it does form a useful way of shaping what shows up in the general public eye.

I can never remember the syntax though, so for your reference and my recollection – http://www.robotstxt.org

Addendum: I was not familiar with the semi-standard for site maps so I’ve added that as well to see what the effect will be.

Post a Comment

Your email is never published nor shared. Required fields are marked *
*
*