User-agent: * Disallow: /admin/ Disallow: /cgi-bin/ Disallow: /files/ Disallow: /htaccess/ Disallow: /htaccess/index.php Disallow: /templates/ Disallow: /_vti_bin/ Disallow: /_mem_bin/ Disallow: /forums Disallow: /tmp Disallow: /logs # Delay crawl of site - # some do some dont read this # whole integers in seconds per page Crawl-delay: 2 # If you want Google specific rules # otherwise remove these Googlebot # rules and use the above only # remeber to include the htaccess ones # so we dont block Google ...eeek! User-agent: Googlebot Disallow: /admin/ Disallow: /cgi-bin/ Disallow: /files/ Disallow: /htaccess/ Disallow: /htaccess/index.php Disallow: /templates/ Disallow: /_vti_bin/ Disallow: /_mem_bin/ # Specific disallow rules for # specific engines, bots, crawlers User-agent: Googlebot-Image User-agent: psbot User-agent: ia_archiver User-agent: asterias User-agent: yahoo-blogs/v3.9 User-agent: yahoo-mmcrawler User-agent: Charlotte User-agent: twiceler User-agent: Offline Explorer User-agent: Zealbot User-agent: Zeus User-agent: Zeus 32297 Webster Pro V2.9 Win32 Disallow:/