[ic] RobotUA and RobotIP in IC 4.8.6?

[Kevin Walsh]

Dave LaTourette [dave at davesport.com] wrote:
> I'm running IC 4.8.6 and have a question. After being assaulted this month
> by Slurp (and it won't quit!), I tried adding the RobotUA directive to my
> interchange.cfg file. When I restarted IC I got an error indicating the
> directive wasn't valid and IC wouldn't start.
> 
> Are there some file dependencies I'm missing, or that may be outdated?
> 
Those directives were introduced into the 4.9 series and so are not
present in 4.8 at all.  You will need to do one of the following:

    1. Upgrade to 4.9+
    2. Backport the 4.9 Robot* directives to your 4.8 version.
    3. Hire a competent consultant to do #2 for you.

None of the above will stop over-eager spiders or mirrors, but it will
ensure that the pages they collect are free from session IDs etc.
This will also ease the load on your site a little, but you may find
that the extra capacity is then used to serve more pages to the Slurp
request bombardment.

You could trace the originating host and complain to the appropriate
ISP or service provider.

If you want to block the host then you will have to use other means,
such as blackholing, bouncing requests from particular UAs or limiting
the bandwidth to/from particular remote hosts or networks.

-- 
   _/   _/  _/_/_/_/  _/    _/  _/_/_/  _/    _/
  _/_/_/   _/_/      _/    _/    _/    _/_/  _/   K e v i n   W a l s h
 _/ _/    _/          _/ _/     _/    _/  _/_/    kevin at cursor.biz
_/   _/  _/_/_/_/      _/    _/_/_/  _/    _/