[ic] Spiders and more lists again

DB DB at M-and-D.com
Thu Sep 2 14:10:03 UTC 2010


This has been discussed before. Spiders crawling "more list" links
sometimes put a significant load on my server. I'm trying to identify
exactly which of my pages that the spiders are trying to crawl so that I
can modify the page(s), rewrite the URLs or otherwise fix the problem.

An example of such a request from my httpd access log is:

GET
/scan/MM=13d6003ffad2d76b12eb41868a5277a3:124360:124379:20.html?mv_more_ip=1&mv_nextpage=results&mv_arg=
HTTP/1.0

If I try this URL in a browser, I get "no search was found" Can anyone
provide a clue about what exactly the spider is looking for and/or come
up with a clever solution?

DB



More information about the interchange-users mailing list