robots.txt for CI

[eluser]Animas[/eluser]
There are so many folders and files in CI. What do you put in robots.txt to disallow unnecessary crawling?

[eluser]m4rw3r[/eluser]
Right in the webroot, or anywhere else the crawlers can find it. Remember that you should edit your htaccess accordingly (so requests for robots.txt doesn't get redirected to CI).

[eluser]Sam Dark[/eluser]
Nothing. You can move app and system out of webroot and they will not be accessible from the web. It's a best practice.

[eluser]Animas[/eluser]
[quote author="Sam Dark" date="1208353763"]You can move app and system out of webroot and they will not be accessible from the web.[/quote]
The app folder is under system folder. Say I put system folder(including app) out of root, then only remains index.php ,.htaccess and robots.txt. Will this work?

[eluser]m4rw3r[/eluser]
If you change the includes and paths in index.php everything should work fine (but no images, css, js, etc. should lie in the app dir).

[eluser]Sam Dark[/eluser]
index.php is well commented btw, so it should be easy to change paths.