Welcome Guest, Not a member yet? Register   Sign In
Cannot access webpages using crawlers after removing index.php from url
#1

[eluser]Unknown[/eluser]
I have removed index.php from my application url as presented around the web. but i have a wierd problem afterwards. I can access the website using browser like this... http://www.oakquotes.com/quotes/author/etc-etc... (Notice the lack of index.php) but when i try to access the same url using a crawler then i get forbidden 403 http error. here is the robots.txt file:

User-agent: *
Disallow:

Sitemap: http://www.oakquotes.com/Sitemap.xml
Sitemap: http://www.oakquotes.com/author_sitemap.xml
Sitemap: http://www.oakquotes.com/topic_sitemap.xml

i think the culprit is the httzccess rule that i have written to remove index.php from the url. here is the code for htaccess...

<IfModule mod_rewrite.c>

# For security reasons, Option followsymlinks cannot be overridden.
# Options +FollowSymlinks
Options +SymLinksIfOwnerMatch
RewriteEngine On
RewriteBase /
RewriteCond $1 !^(index\.php|images|robots\.txt|Sitemap\.xml|topic_sitemap\.xml|author_sitemap\.xml|search\.html|style|js|system|application|quotes/authors|quotes/topic|application/controllers|application/views)
RewriteRule ^(.*)$ ./index.php/$1 [L]

</IfModule>

am i missing a step? please help me in this regard. thanks




Theme © iAndrew 2016 - Forum software by © MyBB