One more question about User Agent Library - what to put in robots array |
[eluser]tinawina[/eluser]
If i get a robot that looks like this: Code: SapphireWebCrawler/Nutch-1.0-dev (Sapphire Web Crawler using Nutch; http://boston.lti.cs.cmu.edu/crawler/; [email protected]) what do i enter into my robots array in the /config/user_agents file? Would this work?: 'sapphire' => 'Sapphire Web Crawler using Nutch' Thanks! |
Welcome Guest, Not a member yet? Register Sign In |