Re: [mv] robots.txt

[Ryan Hertz <rhertz@gyb.baits.com>]

At 02:49 AM 10/23/1999 , you wrote:
>******    message to minivend-users from "Craig A. Oettle"
<craig_oettle@mail.ecsinconline.com>     ******
>
>I have also had minivend lockup after a robot trying to index my site. 
>My catalog is named ecs located in /home/minivend/catalogs/ecs/  I have
>made the robots.txt file as show below but would like some people that
>have more experience to look at it and see if I've not done something I
>should.
>
>User-agent: *
>Disallow: /cgi-bin/ecs/etc/
>Disallow: /cgi-bin/ecs/ord/
>Disallow: /cgi-bin/ecs/query/
>Disallow: /cgi-bin/ecs/reconfig/
>Disallow: /cgi-bin/ecs/config/
>Disallow: /cgi-bin/ecs/admin/

  You may want to just deny your entire cgi-bin or cgi-bin/ecs.  What you
have now will certainly keep any 'nice' robots out of your store.  But,
like they say, locks only keep honest people out.  An indexing spider in
the hands of Joe Hacker can easily be instructed to ignore robots.txt files
(e-mail grabbers, search engines, etc.)  For example, the Fluid Dynamics
search engine (xav.com) requires changing 1 bit in the config for it to be
'rogue'.
	Personally, 99% of the 'bad bots' I've had visit my store have originated
in Japan (where they very well may not follow the robots.txt rules).  My
solution, especially since we don't ship out of the country, was to
(Apache) deny *.jp from the store.  :-)

>
>My site address is: http://www.ecsinconline.com/ any comments,
>suggestions, or criticisms are welcomed.
>
>Thanks for your time.
>
>Craig Oettle
>-
>To unsubscribe from the list, DO NOT REPLY to this message.  Instead, send
>email with 'UNSUBSCRIBE minivend-users' in the body to
Majordomo@minivend.com.
>Archive of past messages: http://www.minivend.com/minivend/minivend-list
> 

Ryan Hertz                                              tel  800-645-BAIT
Webmaster                                               fax  520-645-2588
Advertising Director                            http://www.insideline.net
Gary Yamamoto Custom Baits, Inc.            http://www.yamamoto.baits.com