CodeIgniter Forums

Full Version: Help regarding robots.txt
You're currently viewing a stripped down version of our content. View the full version with proper formatting.
Somebody please suggest me a robots.txt file for my codeigniter project. Am not sure which are the folders that should not be crawled by robots on a codeigniter project.

Thank You.

User-agent: *
# Check and update the following list of disabled directories.
Disallow: /cgi-bin/
Disallow: /tmp/
Disallow: /assets/

# If you support sitemap.xml: Update the actual site address and enable the following line.
# Sitemap: