sagrābstīju internetos:
http://www.ftrain.com/robot_exclusion_protocol.html

My masters will say 'much good job, little robot!' Thank you Google! Must come inside apartment and index. Must!