[ic] Rolling big tables (mysql)

[Kevin Walsh]

Grant <emailgrant at gmail.com> wrote:
> > > I just had a look at robots.cfg and I think I see a few opportunities
> > > for false positives.  I would think "agent" could be bad, and there
> > > are browser toolbars for GetRight and Yahoo which probably alter the
> > > UA.  Is there a crucial set of NotRobotUA entries to go along with
> > > robots.cfg?
> > >
> > > Is anyone using robots.cfg and actively watching for false positives?
> > >
> > I'll look into those.  Do you know the UA names for the various toolbars?
> > I can probably look that up somewhere.  I wouldn't have thought that
> > "agent" would be a risk.
> 
> I have a good system set up for rooting out false positives.  I'll put
> the current robots.cfg file into effect and keep a close eye on
> things.  I'll have some specific info for you soon.
> 
That'd be good - thanks.  I had a quick look around and couldn't find
any information on Yahoo (etc.) toolbar user agent names.

If you want to propose entries for the various directives in the new
robots.cfg file then send them my way and I'll see that they get added.
Well, I'll either add them or challenge you to justify them. :-)

Most of the current robots.cfg file is derived from the user agent
strings found in the www.interchange.rtfm.info log files.

-- 
   _/   _/  _/_/_/_/  _/    _/  _/_/_/  _/    _/
  _/_/_/   _/_/      _/    _/    _/    _/_/  _/   K e v i n   W a l s h
 _/ _/    _/          _/ _/     _/    _/  _/_/    kevin at cursor.biz
_/   _/  _/_/_/_/      _/    _/_/_/  _/    _/