CodeIgniter Forums
creating robots.txt - Printable Version

+- CodeIgniter Forums (https://forum.codeigniter.com)
+-- Forum: Using CodeIgniter (https://forum.codeigniter.com/forum-5.html)
+--- Forum: General Help (https://forum.codeigniter.com/forum-24.html)
+--- Thread: creating robots.txt (/thread-68687.html)



creating robots.txt - Priykant.chavda - 08-13-2017

Hello all. 

I'm Newbie in CI. I don't know how to create robots.txt 

 should I block controllers or views so that any bot can not index sensitive page..  Like should I use application/controllers/admin to block access to admin controller..
Please help.


RE: creating robots.txt - InsiteFX - 08-14-2017

Google search is your friend...


RE: creating robots.txt - Diederik - 08-14-2017

A robots.txt is just a file you can create with any text editor, it will only advise a search engine what to index and what not (note that not all crawlers will listen to your set guidelines). It's not a security measure to prevent someone from accessing a url in your project. A .htaccess file can limit that if you will.


RE: creating robots.txt - rtenny - 08-14-2017

The folders you mention in your post should not be in yourt public_html folder at all.
No need to add them to the robots.
application and system folder should allways be placed above public_html.
Just change the index.php to
$application_folder = '../application';
$system_path = '../system';