+- CodeIgniter Forums (https://forum.codeigniter.com)
+-- Forum: Archived Discussions (https://forum.codeigniter.com/forumdisplay.php?fid=20)
+--- Forum: Archived Development & Programming (https://forum.codeigniter.com/forumdisplay.php?fid=23)
+--- Thread: robots.txt in CI (/showthread.php?tid=5986)
robots.txt in CI - El Forum - 02-11-2008
[eluser]esthonwood[/eluser]
Hello, I want to implement robots.txt on my site and I want to disallow access to www.mysitename.com/index.php/admin/
Here's the content of my robots.txt:
# robots.txt for http://www.mysitename.com/
User-agent: *
Disallow: /index.php/admin/
I checked it with Google Webmaster Tools and it is still accessible. Any suggestion anyone?
Cheers!
robots.txt in CI - El Forum - 02-11-2008
[eluser]xwero[/eluser]
The robots.txt is only a suggestion for spiders. It's the spiders programming that determines if it follows those rules or not.
You should use an .htacces file or use the setting of your webserver to make sure spiders can't crawl your directories.
robots.txt in CI - El Forum - 02-11-2008
[eluser]abmcr[/eluser]
[quote author="xwero" date="1202745227"]
You should use an .htacces file or use the setting of your webserver to make sure spiders can't crawl your directories.[/quote]
An example? Thank you
robots.txt in CI - El Forum - 02-11-2008
[eluser]xwero[/eluser]
Blocking bad bots and site rippers (aka offline browsers)