[ic] googleusercontent.com a spider?

[Peter]

On 07/31/2015 01:45 AM, DB wrote:
>> And the pages accessed. I bet it's somebody scanning for vulns.
> 
> I think you're right Josh... some type of scan I got over 6000 in my
> access log from that IP like:
> 
> 104.155.201.162 www.domain.com - [29/Jul/2015:17:11:00 -0400] "GET
> /scan/MM=c9075c30b0eb97760b8d5cf8aa38cd3e:400:499:100.html?mv_more_ip=%40%40XTNbJ&mv_nextpage=Brand-Parts&pf=sql
> HTTP/1.1" 200 132 "http://www.domain.com" "Mozilla/5.0 (Windows NT 6.1;
> WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/28.0.1500.63
> Safari/537.36"

No, that's not a vulnerability scan, it's an Interchange search , one of
the more links.  From the traffic from that one IP I'd say someone is
trying to slurp down your entire site.  I've seen it before and when
they get a session and there's two or three of them doing it at once it
can bring your entire site to a grinding halt.

> Blocking that IP, while crude, has helped.

Yes that will help, what I ended up doing after seeing this for one
client start happening multiple times per month was to write a script
that would check for page fetches from IPs with a session every five
minutes and block those that tried to fetch more than 50 pages per five
minute period in iptables (someone would have to fetch a page every six
seconds or faster to trip this threshold and in the years since I've
been running that script I'm only aware of one false-positive for it).
That pretty much fixed the problem for good for me.


Peter