Welcome Guest, Not a member yet? Register   Sign In
sitemap.xml for auth sites
#3

(11-16-2019, 02:14 PM)jreklund Wrote: No, it can only crawl public pages.
The reason that I ask is that I found one page out there in the wild that were talking about creating a user account for crawlers with read-only permissions. The idea was to in a public file instruct a crawler to login with that accounts username/password. Maybe not a good practice from a security point of view, but maybe seo-friendly.
Also, if you use something like Google Analytics, at least Google gets the uris from the logged-in pages. I wonder if that could confuse google. I mean I have my sitemap with non-loggedin uris only, but google sees that the page has much more uris than mentioned in the sitemap.xlm.
Reply


Messages In This Thread
sitemap.xml for auth sites - by muuucho - 11-16-2019, 07:55 AM
RE: sitemap.xml for auth sites - by jreklund - 11-16-2019, 02:14 PM
RE: sitemap.xml for auth sites - by muuucho - 11-16-2019, 05:06 PM
RE: sitemap.xml for auth sites - by dave friend - 11-17-2019, 06:53 AM
RE: sitemap.xml for auth sites - by muuucho - 11-17-2019, 07:26 AM
RE: sitemap.xml for auth sites - by jreklund - 11-17-2019, 04:14 AM
RE: sitemap.xml for auth sites - by muuucho - 11-21-2019, 08:17 AM
RE: sitemap.xml for auth sites - by jreklund - 11-21-2019, 11:13 AM
RE: sitemap.xml for auth sites - by muuucho - 11-21-2019, 12:36 PM
RE: sitemap.xml for auth sites - by jreklund - 11-21-2019, 01:50 PM
RE: sitemap.xml for auth sites - by muuucho - 11-21-2019, 02:52 PM



Theme © iAndrew 2016 - Forum software by © MyBB