# Robots.txt file from http://www.searchengineworld.com # # Built from text file http://info.webcrawler.com/mak/projects/robots/active/all.txt # # This restricts access to only known and registered robots. # User-agent: Googlebot Disallow: User-agent: msnbot.msn.com Disallow: / User-agent: cncln.online.ln.cn Disallow: / User-agent: Inktomi Disallow: / User-agent: MSNBot Disallow: / User-agent: Wget Disallow: / user-agent: baiduspider Disallow: User-agent: * Disallow: /