[ic] spiders/logging

[DB]

> On 09/15/2010 02:23 PM, DB wrote:
>> My catalog's access log has many entries like the following which appear
>> to be caused by spiders. They don't seem to be much of a problem, but
>> I'm curious what these entries mean. Is it a failed search, for example
>> for an expired session, or maybe something to do with spiders not having
>> a session assigned?
>>
>> 123.123.123.123 nsession:123.123.123.123 - [15/September/2010:12:14:38
>> -0400] store
>> /cgi-bin/store/scan/MM=f6bfbefe9ff41ee8d877f715bb157e03:5500:5549:50.html search
>> error: Object saved wrong in
>> /catroot/tmp/n/nsession.f6bfbefe9ff41ee8d877f715bb157e03 for search ID
>> nsession.f6bfbefe9ff41ee8d877f715bb157e03.
>>
> 
> scan/MM links are only meaningful for a certain session. Just disallow
> robots to follow these.
> 
> Regards
> 	Racke
> 
> -- 
> LinuXia Systems => http://www.linuxia.de/
> Expert Interchange Consulting and System Administration
> ICDEVGROUP => http://www.icdevgroup.org/
> Interchange Development Team

Thanks. I don't want to disallow all scan links since I have some valid
ones in various search engines. You're saying that this should work:

Disallow: /scan/MM

in my robots.txt ?

DB