Welcome Guest, Not a member yet? Register   Sign In
sitemap.xml for auth sites
#7

(This post was last modified: 11-21-2019, 01:00 PM by muuucho.)

(11-17-2019, 04:14 AM)jreklund Wrote: You can't make an user/password, that would be a security hole. And they don't even pick up on that.
What you need to to is look for a googlebot user agent, and verify it's origin. And let that bot inside without username/password.

https://support.google.com/webmasters/an...0553?hl=en

So, it will be semi-public.

If Google starts crawling those pages (you got Google Search Console right?) you should see those urls failing. You should return a correct header for Google and other bots if they find a page that require login. Like 403. If you return 200, they will think it's the correct content showing up.
So, I create an account for Googlebot in my auth system(I make up an email and a password). 
Should I then do something like this in MY_Controller: (code updated)
PHP Code:
// Only check if  not logged in users are a google bot
if(!$this->ion_auth->logged_in()) {    
    
// Verify if caller is a google bot
    $remote_ip $_SERVER['REMOTE_ADDR'];
    $domain gethostbyaddr($remote_ip);
    if (in_array($domain, ['googlebot.com''google.com']) && gethostbyname($domain) == $remote_ip) {
        // Caller is a google bot, login the caller and redirect to some logged_in page where the "inside" crawl can start...
        $this->ion_auth->login('made_up_email''made_up_password');
        redirect('auth/logged_in_start');
    

// Run script... 
Reply


Messages In This Thread
sitemap.xml for auth sites - by muuucho - 11-16-2019, 07:55 AM
RE: sitemap.xml for auth sites - by jreklund - 11-16-2019, 02:14 PM
RE: sitemap.xml for auth sites - by muuucho - 11-16-2019, 05:06 PM
RE: sitemap.xml for auth sites - by dave friend - 11-17-2019, 06:53 AM
RE: sitemap.xml for auth sites - by muuucho - 11-17-2019, 07:26 AM
RE: sitemap.xml for auth sites - by jreklund - 11-17-2019, 04:14 AM
RE: sitemap.xml for auth sites - by muuucho - 11-21-2019, 08:17 AM
RE: sitemap.xml for auth sites - by jreklund - 11-21-2019, 11:13 AM
RE: sitemap.xml for auth sites - by muuucho - 11-21-2019, 12:36 PM
RE: sitemap.xml for auth sites - by jreklund - 11-21-2019, 01:50 PM
RE: sitemap.xml for auth sites - by muuucho - 11-21-2019, 02:52 PM



Theme © iAndrew 2016 - Forum software by © MyBB