robots.txt in CI

[eluser]esthonwood[/eluser]
Hello, I want to implement robots.txt on my site and I want to disallow access to www.mysitename.com/index.php/admin/

Here's the content of my robots.txt:

# robots.txt for http://www.mysitename.com/
User-agent: *
Disallow: /index.php/admin/

I checked it with Google Webmaster Tools and it is still accessible. Any suggestion anyone?
Cheers!

[eluser]xwero[/eluser]
The robots.txt is only a suggestion for spiders. It's the spiders programming that determines if it follows those rules or not.

You should use an .htacces file or use the setting of your webserver to make sure spiders can't crawl your directories.

[eluser]abmcr[/eluser]
[quote author="xwero" date="1202745227"]

You should use an .htacces file or use the setting of your webserver to make sure spiders can't crawl your directories.[/quote]

An example? Thank you

[eluser]xwero[/eluser]
Blocking bad bots and site rippers (aka offline browsers)