Skip to content

Robots. They are coming to take your content.

I am in the process of revising my site, and discovered for whatever reason that I had an empty robots.txt file present. I know it is only a voluntary ‘standard’, but as far as I know all the major players do respect it. As the overwhelming proportion of users use a search engine that respects the standard, it does form a useful way of shaping what shows up in the general public eye.

I can never remember the syntax though, so for your reference and my recollection –

Addendum: I was not familiar with the semi-standard for site maps so I’ve added that as well to see what the effect will be.

Addendum:Ritta Blens has pointed me to another very useful tool for testing the structure of a robots.txt file:

Post a Comment

Your email is never published nor shared. Required fields are marked *