Cannot access webpages using crawlers after removing index.php from url |
[eluser]Unknown[/eluser]
I have removed index.php from my application url as presented around the web. but i have a wierd problem afterwards. I can access the website using browser like this... http://www.oakquotes.com/quotes/author/etc-etc... (Notice the lack of index.php) but when i try to access the same url using a crawler then i get forbidden 403 http error. here is the robots.txt file: User-agent: * Disallow: Sitemap: http://www.oakquotes.com/Sitemap.xml Sitemap: http://www.oakquotes.com/author_sitemap.xml Sitemap: http://www.oakquotes.com/topic_sitemap.xml i think the culprit is the httzccess rule that i have written to remove index.php from the url. here is the code for htaccess... <IfModule mod_rewrite.c> # For security reasons, Option followsymlinks cannot be overridden. # Options +FollowSymlinks Options +SymLinksIfOwnerMatch RewriteEngine On RewriteBase / RewriteCond $1 !^(index\.php|images|robots\.txt|Sitemap\.xml|topic_sitemap\.xml|author_sitemap\.xml|search\.html|style|js|system|application|quotes/authors|quotes/topic|application/controllers|application/views) RewriteRule ^(.*)$ ./index.php/$1 [L] </IfModule> am i missing a step? please help me in this regard. thanks |
Messages In This Thread |
Cannot access webpages using crawlers after removing index.php from url - by El Forum - 07-29-2011, 12:51 PM
|