Robots disallow

[eluser]new_igniter[/eluser]
Hello,
I have a groups site where some group pages are public and some group pages are private. Currently we do not allow robots to access any pages on the site, but we would like to move forward with allowing public groups to be crawled.
Each group page uses one controller, so I am wondering what the proper method is for allowing robots conditionally.
Thanks,
Peter

[eluser]TheFuzzy0ne[/eluser]
[quote author="new_igniter" date="1209353009"]Hello,
I have a groups site where some group pages are public and some group pages are private. Currently we do not allow robots to access any pages on the site, but we would like to move forward with allowing public groups to be crawled.
Each group page uses one controller, so I am wondering what the proper method is for allowing robots conditionally.
Thanks,
Peter[/quote]

http://www.robotstxt.org/robotstxt.html

You should know that nothing short of passworded content will actually "stop" a bot. Most bots around are "nice" bots, and will respect your robots.txt file, but there are still others that may ignore it.