CodeIgniter Forums
Stopping Googlebot? - Printable Version

+- CodeIgniter Forums (
+-- Forum: Archived Discussions (
+--- Forum: Archived General Discussion (
+--- Thread: Stopping Googlebot? (/thread-10016.html)

Stopping Googlebot? - El Forum - 07-16-2008

I've noticed that the Google bot has been indexing my site. Normally, I want this. But, I have a couple of sample search links on my site and Google goes to that link. Then, at the bottom, I have a pagination system. There are literally millions of "hits" so Google is going to page 1, page 2, page 3, etc. Every 2 minutes.

I've put a meta tag for "robot" with "index,nofollow".

How long does that take for Google to notice? Is that even the right thing to do?

I've contacted Google in the past about other things and never get a reply or I would have just gone directly to them. What have some of you other CI users done? I've searched and searched and everyone seems to say use the method above.


Stopping Googlebot? - El Forum - 07-16-2008

[eluser]Pascal Kriete[/eluser]
Most search engines have started to ignore most meta tags. Instead, create a robots.txt file for your site as well as a sitemap.

Google's webmaster tools are very useful for tweaking what they do with your site.

Stopping Googlebot? - El Forum - 07-16-2008

Thanks. I will look into that too because the meta isn't stopping Google.

Stopping Googlebot? - El Forum - 07-16-2008

[eluser]Pascal Kriete[/eluser]
Thanks for clarifying.

::mumbles to self about checking facts first::

Stopping Googlebot? - El Forum - 07-16-2008

[quote author="Sock Puppet" date="1216242284"]I [url=""]don't think[/url] Google ignores

<meta name="robots" content="noindex,nofollow,noarchive">

That said, a proper robots.txt won't hurt, either, of course.[/quote]

But wouldn't that keep Google from indexing my site and not in their radar for search?

I will get a robot.txt file up soon and look at the site map from google.



I put <META NAME="Googlebot" CONTENT="nofollow"> in so maybe that will help.

Stopping Googlebot? - El Forum - 07-17-2008

Thanks for the link!

The Googlebot meta tag seemed to stop them. But I see other bots ignoring the meta too. This weekend I hope to put some real effort in the meta tags, robots, etc. Man there is just so much to do with a new site!