# # robots.txt for http://www.emersonhouston.org/ # (exclude specified folders from web crawlers) # # created by Pat Grealy, May 12, 2006 # # Examples: # exclude some access-controlled areas # (no blank lines allowed after user-agent field) # # User-agent: * # Disallow: /EmersonDirectory/ # Disallow: /intranet/ # # exclude entire server # # User-agent: * # Disallow: / User-agent: * Disallow: /intranet/ Disallow: /library/ Disallow: /text/ Disallow: /webalizer/ Crawl-delay: 10