+- CodeIgniter Forums (https://forum.codeigniter.com)
+-- Forum: Archived Discussions (https://forum.codeigniter.com/forumdisplay.php?fid=20)
+--- Forum: Archived Development & Programming (https://forum.codeigniter.com/forumdisplay.php?fid=23)
+--- Thread: Prevent search robots page indexing (/showthread.php?tid=18431)
Prevent search robots page indexing - El Forum - 05-06-2009
[eluser]Rubiz'[/eluser]
Hello all!
I have a website, inside it I have a CMS tool.
I use A LOT the CMS tool, because its a news site, I use every day, many times.
Google have indexed it, the CMS pages, and how its very viewed, it takes the first position in ranking :S
I have put Robots.txt inside CMS CI views folder, and in the first page of CMS, I have put the metatag <meta name="ROBOTS" content="NOINDEX,NOFOLLOW">, but it is still indexing pages :S
Anyone knows the solution?
Prevent search robots page indexing - El Forum - 05-06-2009
[eluser]kgill[/eluser]
Two things, first it should be robots.txt - that capital R might cause problems but the real issue here is that it needs to be in the root directory of your website not buried deeper down in the CI directories. If you type yourhostname.com/robots.txt and get you don't get a file, it's in the wrong spot.
Prevent search robots page indexing - El Forum - 05-06-2009
[eluser]Rubiz'[/eluser]
Hello kgill, thanx for answering. My txt file "Robots" is written small case, as you told.
My web site directories map is like that:
- root with CI
- website (this will be indexed!)
- website/CMS tool (this will not be indexed!)
So, I cant put robots.txt in the root, just because my website NEED to be indexed.
But the CMS Tool, that is a FOLDER INSIDE MY WEBSITE dont. Its clear?
So, anyone with other idea?
Prevent search robots page indexing - El Forum - 05-06-2009
[eluser]kgill[/eluser]
read up on robots.txt (http://www.robotstxt.org/) it needs to be in your root directory, inside the file is where you tell search engines which directories to ignore.