itemap parameter is supported by major crawlers (including Google, Yahoo, MSN, Ask). # Sitemaps specifies the location of the site's list of URLs. # This parameter is independent from User-agent parameter so it can be placed anywhere in the file. #Sitemap: http://www.example.com/sitemap.xml.gz User-agent: * #Disallow: /media/ #Disallow: /archive/ Allow: /code/facebook-api/ Allow: /code/jacobgen/ Allow: /code/com4jgen/ Allow: /tft/ #Disallow: /cgi-bin/ #Disallow: /images/ #Disallow: /tmp/ #Disallow: /private/ # Several major crawlers support a Crawl-delay parameter, set to the number of seconds to wait between successive requests to the same server: #User-agent: * #Crawl-delay: 10 # Some major crawlers support an Allow directive which can counteract a previous Disallow directive #User-agent: Googlebot #Disallow: /folder1/ #Allow: /folder1/myfile.html