User-agent: *Crawl-Delay: 10Disallow: /admin/Disallow: /cgi-bin/Disallow: /images/Disallow: /ferms/Disallow: /taxonomy/Disallow: /aggregator/Disallow: /tracker/Disallow: /comment/reply/Disallow: /node/add/Disallow: /user/Disallow: /files/Disallow: /search/Disallow: /book/print/# small robots.txt# more information about this file can be found at# http://www.robotstxt.org/wc/robots.html# lines beginning with the pund ("#") sign are comments and can be deleted.# if case your drupal site is in a directory# lower than your docroot (e.g. /drupal)# please add this before the /-es below# to stop a polite robot indexing an exampledir# add a line like (delte the #'s)# user-agent: polite-bot# Disallow: /exampledir/# a list of know bots can be found at# http://www.robotstxt.org/wc/active/html/index.html# see http://www.sxw.org.uk/computing/robots/check.html# for syntax checking