creating robots.txt - Printable Version +- CodeIgniter Forums (https://forum.codeigniter.com) +-- Forum: Using CodeIgniter (https://forum.codeigniter.com/forumdisplay.php?fid=5) +--- Forum: General Help (https://forum.codeigniter.com/forumdisplay.php?fid=24) +--- Thread: creating robots.txt (/showthread.php?tid=68687) |
creating robots.txt - Priykant.chavda - 08-13-2017 Hello all. I'm Newbie in CI. I don't know how to create robots.txt should I block controllers or views so that any bot can not index sensitive page.. Like should I use application/controllers/admin to block access to admin controller.. Please help. RE: creating robots.txt - InsiteFX - 08-14-2017 Google search is your friend... RE: creating robots.txt - Diederik - 08-14-2017 A robots.txt is just a file you can create with any text editor, it will only advise a search engine what to index and what not (note that not all crawlers will listen to your set guidelines). It's not a security measure to prevent someone from accessing a url in your project. A .htaccess file can limit that if you will. RE: creating robots.txt - rtenny - 08-14-2017 The folders you mention in your post should not be in yourt public_html folder at all. No need to add them to the robots. application and system folder should allways be placed above public_html. Just change the index.php to $application_folder = '../application'; $system_path = '../system'; |