[ic] mod_interchange and Apache MaxClients

[Ron Phipps]

> From: interchange-users-bounces at icdevgroup.org
[mailto:interchange-users-
> bounces at icdevgroup.org] On Behalf Of John1
> Sent: Monday, November 14, 2005 3:52 PM
> 
> I am running Apache with mod_interchange 1.32, with the following
> httpd.conf
> entries:
> 
> StartServers  5
> MinSpareServers  5
> MaxSpareServers  10
> MaxClients  150
> MaxRequestsPerChild 10000
> 
> I am finding that even with these settings Apache is hitting its
> MaxClients
> limit now and again as evidenced by the following entry in the Apache
> error
> log:
> 
> "server reached MaxClients setting, consider raising the MaxClients
> setting"
> 
> The problem is that once Apache bumps up against this limit, the site
> hangs,
> permanently!
> 
> i.e.  The Apache processes running stay at 150 indefinitely and so the
> site
> becomes permanently inaccessible.
> 
> I had thought that once MaxClients was reached TCP requests would be
> queued
> and so eventually the lockout situation would end once the backlog of
> requests was cleared.
> 
> Having come across an old post to the mailing list:
> http://www.icdevgroup.org/pipermail/interchange-users/2003-
> March/032292.html
> 
> ...I am wondering whether the reason that the lockout never clears may
be
> to
> do with how mod_interchange queues requests for Apache?  Does anyone
know
> whether this could be the case, or if not, can anyone else suggest why
I
> am
> getting *permanent* lockout once the MaxClient limit is bumped up
against.
> i.e. currently I am forced to restart Apache once the MaxClient limit
is
> bumped.
> 
> I have a second related question:
> 
> The reason that the MaxClient is being bumped from time to time is
often
> (though not always) due to http floods from scripts looking for files
on
> my
> server that don't exist (i.e. looking for files with known security
> holes).
> Does anyone have any suggestions on how best to stop these sort of
floods
> bringing down my server?
> 
> Any suggestions on how to solve this issue (and the lockout problem)
would
> be greatly appreciated as I am currently having to restart Apache
several
> times a day.  Thanks.
> 

I'm hoping Kevin can elaborate on how mod_interchange interacts with
Apache and IC and perhaps we can come to a conclusion.  Another idea I
had was that maybe the MaxClients being reached is the result of another
problem, such as IC is actually no longer processing the requests so
Apache is never able to clear its backlog.  The restart of Apache, could
be making IC kill off the children processes and then spawning new
processes in effect allowing IC to "catch up" with the requests.

When the site goes down, further requests for IC pages show the
following in the apache error log for the particular site:

[Thu Nov 10 20:15:11 2005] [error] [client x.x.x.x] Malformed header
return by Interchange: 
[Thu Nov 10 20:15:11 2005] [error] [client x.x.x.x] Premature end of
script headers: /home/xxxxxx/public_html/favicon.ico
[Thu Nov 10 20:15:11 2005] [error] [client x.x.x.x] Malformed header
return by Interchange: 
[Thu Nov 10 20:15:14 2005] [error] [client x.x.x.x] Premature end of
script headers: /home/xxxxxx/public_html/index.html
[Thu Nov 10 20:15:14 2005] [error] [client x.x.x.x] Malformed header
return by Interchange: 
[Thu Nov 10 20:15:16 2005] [error] [client x.x.x.x] Premature end of
script headers: /home/xxxxxx/public_html/index.html
[Thu Nov 10 20:15:16 2005] [error] [client x.x.x.x] Malformed header
return by Interchange: 
[Thu Nov 10 20:15:18 2005] [error] [client x.x.x.x] Premature end of
script headers: /home/xxxxxx/public_html/scan
[Thu Nov 10 20:15:18 2005] [error] [client x.x.x.x] Malformed header
return by Interchange: 
[Thu Nov 10 20:15:20 2005] [error] [client x.x.x.x] Premature end of
script headers: /home/xxxxxx/public_html/favicon.ico
[Thu Nov 10 20:15:20 2005] [error] [client x.x.x.x] Malformed header
return by Interchange: 
[Thu Nov 10 20:15:21 2005] [error] [client x.x.x.x] Premature end of
script headers: /home/xxxxxx/public_html/robots.txt
[Thu Nov 10 20:15:21 2005] [error] [client x.x.x.x] Malformed header
return by Interchange: 
[Thu Nov 10 20:15:24 2005] [error] [client x.x.x.x Premature end of
script headers: /home/xxxxxx/public_html/index.html
[Thu Nov 10 20:15:24 2005] [error] [client x.x.x.x] Malformed header
return by Interchange:

These messages repeat until IC is restarted.

We've noticed more scripts hitting our site looking for holes and a
higher amount of search engines crawling the site so higer traffic
patterns may be related.

John and Jeff, could you guys post the following:

OS version
Apache version
Mysql version
Perl version
Mod_interchange version
IC version
Running in RPC mode?
Processors?
Ram?

Thanks!
-Ron