[ic] mod_interchange and Apache MaxClients

[John1]

On Tuesday, November 15, 2005 6:32 PM, rphipps at reliant-solutions.com wrote:

>> Yes, I was wondering about this sort of possibility too. I had
>> previously thought that it was necessary to restart Apache and
>> Interchange to clear the
>> problem, but I have now found that as long as I leave the server for
>> a few seconds between stopping and restarting Apache, there is no
>> need to restart
>> Interchange.  Conversely, if I stop and then quickly restart Apache
>> without
>> pausing the website is still unresponsive.  It seems that the reason
>> for this is that once Apache has been stopped it takes a few seconds
>> for the old
>> Interchange daemon processes to die off, after which Apache can be
>> restarted
>> and everything is fine again.  I think it is necessary to wait for
>> all the Interchange processes to die off before restarting Apache,
>> but I can't be 100% sure of this - I will try to verify whether this
>> is definitely the case
>> next time I get a lockup.
>
> Last night I recompiled with MaxClients to 1024 and this morning I ran
> into the problem again, also MaxClients was not reached this time,
> however I was unable to reach the IC site.  At least in my case I do
> not believe MaxClients to be the issue.
>
I now agree - after some investigation I think that MaxClients issue is a 
symptom rather than the cause.  No doubt if you have increased MaxClients to 
1024 you are now not hitting this limit, but are upi still seeing a high 
Apache client count??  I think that once the Interchange daemon starts to 
have problems this must have a cascade effect on the Apache processes being 
spawned

> When did this problem start happening for you?
>
Well I have been running a moderately busy Interchange site (with much the 
same software versions) for a couple of years now and it has been fairly 
stable throughout this period, but perhaps once every couple of weeks on 
average I have had what superficially seems to be the same/similar problem 
as I am having now - as it has only been bringing the server down once every 
couple of weeks I have been living with it and just restarting Interchange 
and Apache.

However, over the last 2-3 weeks it has started to become a far more regular 
problem - it is now unusual for a day to go past without me having to 
restart apache/interchange (BTW, I haven't yet established whether it is 
ever absolutely necessary to restart Interchange as well as Apache.  I used 
to restart both services, but it now seems I only need to restart Apache, as 
long as I wait a few seconds before restarting it - as per discussion 
earlier in this thread).

>  How often are you
> seeing this problem?  The next time the problem occurs could you
> restart Interchange instead of Apache and see what happens?
>
I will do this and let you know - I am fairly sure that just restarting 
Interchange is not sufficient, although, as I say, I have recently 
discovered that just restarting Apache (as long as I wait a few seconds 
after stopping it, presumably to allow Interchange processes to die off) 
does seem to be sufficient.

> I'm still trying to figure out if this is an issue with IC or an issue
> with Apache. Restarting Apache could kill off the IC children
> processes that were attached to Apache and restarting IC could kill
> off the Apache children that were attached to IC, so not sure quite
> where the issue is at.
>
Absolutely, that exactly where I'm at as well - not quite sure yet where the 
issue is - will keep you posted.

>>> When the site goes down, further requests for IC pages show the
>>> following in the apache error log for the particular site:
>>>
>>> [Thu Nov 10 20:15:11 2005] [error] [client x.x.x.x] Malformed header
>>> return by Interchange:
>>> [Thu Nov 10 20:15:11 2005] [error] [client x.x.x.x] Premature end of
>>> script headers: /home/xxxxxx/public_html/favicon.ico
>>> [Thu Nov 10 20:15:11 2005] [error] [client x.x.x.x] Malformed header
>>> return by Interchange:
>>> [Thu Nov 10 20:15:14 2005] [error] [client x.x.x.x] Premature end of
>>> script headers: /home/xxxxxx/public_html/index.html
>>> [Thu Nov 10 20:15:14 2005] [error] [client x.x.x.x] Malformed header
>>> return by Interchange:
>>> [Thu Nov 10 20:15:16 2005] [error] [client x.x.x.x] Premature end of
>>> script headers: /home/xxxxxx/public_html/index.html
>>> [Thu Nov 10 20:15:16 2005] [error] [client x.x.x.x] Malformed header
>>> return by Interchange:
>>> [Thu Nov 10 20:15:18 2005] [error] [client x.x.x.x] Premature end of
>>> script headers: /home/xxxxxx/public_html/scan
>>> [Thu Nov 10 20:15:18 2005] [error] [client x.x.x.x] Malformed header
>>> return by Interchange:
>>> [Thu Nov 10 20:15:20 2005] [error] [client x.x.x.x] Premature end of
>>> script headers: /home/xxxxxx/public_html/favicon.ico
>>> [Thu Nov 10 20:15:20 2005] [error] [client x.x.x.x] Malformed header
>>> return by Interchange:
>>> [Thu Nov 10 20:15:21 2005] [error] [client x.x.x.x] Premature end of
>>> script headers: /home/xxxxxx/public_html/robots.txt
>>> [Thu Nov 10 20:15:21 2005] [error] [client x.x.x.x] Malformed header
>>> return by Interchange:
>>> [Thu Nov 10 20:15:24 2005] [error] [client x.x.x.x Premature end of
>>> script headers: /home/xxxxxx/public_html/index.html
>>> [Thu Nov 10 20:15:24 2005] [error] [client x.x.x.x] Malformed header
>>> return by Interchange:
>>>
Well, I have got *exactly* the same errors in my Apache error log, around 
the time I had a lockup yesterday, but I don't think I see these errors 
every time I get a lockup.

Here is a snippet from yesterday:
[Sun Nov 13 14:34:27 2005] [error] [client x.x.x.x] File does not exist: 
/home/xxxxx/public_html/xmlrpc/xmlrpc.php
[Sun Nov 13 14:34:27 2005] [error] [client x.x.x.x] File does not exist: 
/home/xxxxx/public_html/xmlrpc/xmlrpc.php
[Sun Nov 13 14:34:27 2005] [error] [client x.x.x.x] File does not exist: 
/home/xxxxx/public_html/xmlrpc/xmlrpc.php
[Sun Nov 13 14:34:27 2005] [error] [client x.x.x.x] File does not exist: 
/home/xxxxx/public_html/xmlsrv/xmlrpc.php
[Sun Nov 13 14:34:28 2005] [error] [client x.x.x.x] File does not exist: 
/home/xxxxx/public_html/xmlsrv/xmlrpc.php
[Sun Nov 13 14:34:28 2005] [error] [client x.x.x.x] File does not exist: 
/home/xxxxx/public_html/xmlsrv/xmlrpc.php
[Sun Nov 13 14:34:28 2005] [error] [client x.x.x.x] File does not exist: 
/home/xxxxx/public_html/xmlsrv/xmlrpc.php
[Sun Nov 13 14:40:35 2005] [error] server reached MaxClients setting, 
consider raising the MaxClients setting
[Sun Nov 13 15:49:39 2005] [error] [client x.x.x.x] Premature end of script 
headers: /home/xxxxx/public_html/blog
[Sun Nov 13 15:49:39 2005] [error] [client x.x.x.x] Premature end of script 
headers: /home/xxxxx/public_html/index.html
[Sun Nov 13 15:49:39 2005] [error] [client x.x.x.x] Premature end of script 
headers: /home/xxxxx/public_html/scan
[Sun Nov 13 15:49:40 2005] [error] [client x.x.x.x] Malformed header return 
by Interchange:
[Sun Nov 13 15:49:40 2005] [error] [client x.x.x.x] Malformed header return 
by Interchange:
[Sun Nov 13 15:49:40 2005] [error] [client x.x.x.x] Premature end of script 
headers: /home/xxxxx/public_html/index.html
[Sun Nov 13 15:49:40 2005] [error] [client x.x.x.x] Malformed header return 
by Interchange:
[Sun Nov 13 15:49:40 2005] [error] [client x.x.x.x] Premature end of script 
headers: /home/xxxxx/public_html/index.html

Notice the the times, and also the positioning of the MaxClients entry in 
the above.  I am fairly sure the site was up until the MaxClients setting 
was hit, and down thereafter.  This is a complete snippet from the error 
log - i.e. I haven't missed any lines out of the middle.

I will monitor the various error logs each time I get a lockup and will let 
you know if I can see a pattern emerging.

> So it appears that this problem is not dependant on the IC or
> mod_interchange version.  Perhaps it's an issue with RPC mode and
> mod_interchange?  Or just a new attack that is in the wild?
>
It does look like there may be an increased spate of these scripts hammering 
websites at the moment and that may well be why we are both seeing an 
escalation in the amount of times we need to restart Apache/Interchange. 
However, as I say, I don't think this problem is an entirely new problem (at 
least not for me), it's just that now I am typically having to restart 
Apache/Interchange several times a day rather than a couple of times a month 
at most.

Please post if you make any progress - I will do the same.  Thanks. 


		
___________________________________________________________ 
To help you stay safe and secure online, we've developed the all new Yahoo! Security Centre. http://uk.security.yahoo.com