+- CodeIgniter Forums (https://forum.codeigniter.com)
+-- Forum: Archived Discussions (https://forum.codeigniter.com/forumdisplay.php?fid=20)
+--- Forum: Archived Development & Programming (https://forum.codeigniter.com/forumdisplay.php?fid=23)
+--- Thread: Cannot access webpages using crawlers after removing index.php from url (/showthread.php?tid=43981)
Cannot access webpages using crawlers after removing index.php from url - El Forum - 07-29-2011
[eluser]Unknown[/eluser]
I have removed index.php from my application url as presented around the web. but i have a wierd problem afterwards. I can access the website using browser like this... http://www.oakquotes.com/quotes/author/etc-etc... (Notice the lack of index.php) but when i try to access the same url using a crawler then i get forbidden 403 http error. here is the robots.txt file:
User-agent: *
Disallow:
Sitemap: http://www.oakquotes.com/Sitemap.xml
Sitemap: http://www.oakquotes.com/author_sitemap.xml
Sitemap: http://www.oakquotes.com/topic_sitemap.xml
i think the culprit is the httzccess rule that i have written to remove index.php from the url. here is the code for htaccess...
<IfModule mod_rewrite.c>
# For security reasons, Option followsymlinks cannot be overridden.
# Options +FollowSymlinks
Options +SymLinksIfOwnerMatch
RewriteEngine On
RewriteBase /
RewriteCond $1 !^(index\.php|images|robots\.txt|Sitemap\.xml|topic_sitemap\.xml|author_sitemap\.xml|search\.html|style|js|system|application|quotes/authors|quotes/topic|application/controllers|application/views)
RewriteRule ^(.*)$ ./index.php/$1 [L]
</IfModule>
am i missing a step? please help me in this regard. thanks