Help regarding robots.txt

Help regarding robots.txt

ivantcholakov
Senior Member
Posts: 668
Threads: 17
Joined: Oct 2014
Reputation: 37

07-13-2015, 02:48 PM

Code:
# www.robotstxt.org/

# www.google.com/support/webmasters/bin/answer.py?hl=en&answer=156449

User-agent: *

# Check and update the following list of disabled directories.

Disallow: /cgi-bin/

Disallow: /tmp/

Disallow: /assets/

# If you support sitemap.xml: Update the actual site address and enable the following line.

# Sitemap: http://my-site.com/sitemap.xml

Messages In This Thread

Help regarding robots.txt - by jithinjohnygeorge - 07-13-2015, 10:49 AM

RE: Help regarding robots.txt - by ivantcholakov - 07-13-2015, 02:48 PM