# /robots.txt file letting all agents go anywhere User-agent: * Disallow: /muss/message.html Disallow: message.html Disallow: /muss/linkmuss.cgi Disallow: /muss/cgi-bin/www.microlink.co.uk/links.cgi #/ to exclude a specific robot from all urls starting with temp enter #/ User-agent: lycos #/ Disallow: /temp