+- CodeIgniter Forums (https://forum.codeigniter.com)
+-- Forum: Using CodeIgniter (https://forum.codeigniter.com/forumdisplay.php?fid=5)
+--- Forum: General Help (https://forum.codeigniter.com/forumdisplay.php?fid=24)
+--- Thread: Help regarding robots.txt (/showthread.php?tid=62429)
Help regarding robots.txt - jithinjohnygeorge - 07-13-2015
Somebody please suggest me a robots.txt file for my codeigniter project. Am not sure which are the folders that should not be crawled by robots on a codeigniter project.
Thank You.
RE: Help regarding robots.txt - ivantcholakov - 07-13-2015
Code:
# www.robotstxt.org/
# www.google.com/support/webmasters/bin/answer.py?hl=en&answer=156449
User-agent: *
# Check and update the following list of disabled directories.
Disallow: /cgi-bin/
Disallow: /tmp/
Disallow: /assets/
# If you support sitemap.xml: Update the actual site address and enable the following line.
# Sitemap: http://my-site.com/sitemap.xml