Image may be NSFW.
Clik here to view.
I did some research month ago and wrote a longer article about robots.txt. But for a quick lookup it’s too much to read, so today it’s time a for a simple best-practices check-list:
- Use ASCII encoding for robots.txt (see as well Ticket #14069)
- Use proper URLs inside the rules (urlencoded !)
- Mind UTF-8
- Mind the BOM sequences
Read On: Encoding of the robots.txt file (20 Jul 2010; by hakre)
Image Credits: Robot © Christopher Grant Harris, Japanese Nada Letter provided by Haruno Akiha.
Tagged: #14069, ASCII, best practice, Encoding, How-To, robots.txt Image may be NSFW.
Clik here to view.
Image may be NSFW.Clik here to view.
