Welcome Guest, Not a member yet? Register   Sign In
Crawl error - Google trying to access directory?
#1

[eluser]Unknown[/eluser]
Hi,

First thanks to codeIgniter for helping me produce my first website!
https://maximumtrainer.com/

Website is up since 3 days, and on google Webmaster I noticed some crawl error :
https://www.dropbox.com/s/woeoozx0qj69na...l.png?dl=0

I don't know why google try to access my assets folder: https://maximumtrainer.com/assets
I would like the content inside assets to be indexed (image, etc.). For exemple this image : https://maximumtrainer.com/assets/image/Banner1.jpg
But it seems google is trying to access assets directly and is blocked by codeIgniter?


Should I provide a robot.txt telling the specific folder inside assets to be indexed, I read that I have no need to produce a robot.txt file if I want to index everything..
My /application and /system folder are not in the www root, just one level up, I only have index.php, assets folder and download folder directly inside www root (as you can see in the dropbox image)

Here is my .htaccess file (force https, remove index.php from URL)
RewriteEngine on
RewriteCond %{HTTPS} !=on
RewriteRule ^ https://%{HTTP_HOST}%{REQUEST_URI} [L,R=301]

RewriteBase /
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule ^(.*)$ index.php?/$1 [L]


Thank you! Merci


[Edit: sorry for the link, can't put <a> on link since the forum think i'm a spammer..]
#2

[eluser]InsiteFX[/eluser]
Next time you post use the POST REPLY button, It provides a full editor for the forums.

Being as your website is SSL secure that may be your problem.




Theme © iAndrew 2016 - Forum software by © MyBB