# See http://www.robotstxt.org/wc/norobots.html for documentation on how to use the robots.txt file
#
# To ban all spiders from the entire site uncomment the next two lines:
# User-Agent: *
# Disallow: /  

# apply the following to all spiders by using *
User-agent: *                                  
# disallow everything in the spec directory
Disallow: /spec/ 
# Security Note 
# Security Note Some Web servers use a file called /robot(s).txt to make search engines and
# Security Note  any other indexing tools visit their Web pages more frequently and more efficiently.
# Security Note 
# Security Note By connecting to the server and requesting the /robot(s).txt file, 
# Security Note an attacker may gain additional information about the system they are attacking.
# Security Note Such information as: restricted directories, hidden directories, CGI script directories, etc. may be available.
# Security Note 
# Security Note Take special care not to tell the robots not to index sensitive directories, 
# Security Note since this tells attackers exactly which of your directories are sensitive.