[ic] RobotIPs
Jonathan Clark
jonc at webmaint.com
Fri Feb 20 16:19:45 EST 2004
interchange-users-bounces at icdevgroup.org <> wrote:
> Jon [prtyof5 at attglobal.net] wrote:
>>>> RobotIP <<EOR
>>>> 202.9.155.123, 204.152.191.41, 208.146.26.19,
>>>> 208.146.26.233, 209.185.141.209, 209.185.141.211,
>>>> 209.202.148.36, 209.202.148.41, 216.200.130.207,
>>>> 216.35.103.6?, 216.35.103.70,
>>>> EOR
>>>>
>>>> Seems like there should be some 64.68*
>>>> in there shouldn't there ??
>>>>
>>> Please be so kind to tell us which robots crawling from this IP
>>> block to help the ICDEVGROUP to improve the default settings.
>>>
>> Googlebot. Or am I wrong ? Assuming so would it be correct
>> and appropriate to add 64.68? ??
>>
> 64.68.* would cover 65,536 IP addresses. It's probably safe to say
> that they're not all used by spiders. They're definitely not all
> owned by Google. :-)
>
> For instance, 64.68.0.0/19 is allocated to by "Elan Communications".
> 64.68.32.0/19 is allocated to "Advanced Access" and 64.68.64.0/19
> is allocated to "Cable & Wireless". Within the Cable & Wireless
> block, 64.68.78.0/23 is reallocated (sub-let) to eBay. I believe
> Google has a large block in there somewhere (something like 2048 IPs)
> but I can't find it at the moment, and they're probably not all used
> by spiders anyway.
>
> A more focused list of addresses may make it into the default set,
> as long as a proper justification is presented. You're free to add
> any addresses you like into your own local configuration, of course.
And this would help anyone wanting to do that:
http://www.searchengineworld.com/spiders/spider_ips.htm
Google does have more blocks listed, but the referrer id remains the same:
Googlebot/2.1 (+http://www.googlebot.com/bot.html)
at the moment so IC should behave properly anyway. Out of interest, we are
being spidered from 64.68.82.0/24 at the moment. One server has this block
added, one doesn't. Both are behaving properly for google.
Jonathan.
--
Jonathan Clark
Webmaint.com - Building Clever Websites http://www.webmaint.com/
Webmaint.net - Business Web Hosting http://www.webmaint.net/
WireTel - Internet Connectivity Solutions http://www.wiretel.net/
More information about the interchange-users
mailing list