[ic] mod_interchange and Apache MaxClients

[Ron Phipps]

> From: interchange-users-bounces at icdevgroup.org
[mailto:interchange-users-
> bounces at icdevgroup.org] On Behalf Of John1
> Sent: Wednesday, November 16, 2005 5:03 PM
> 
> On Wednesday, November 16, 2005 6:00 PM, rphipps at reliant-solutions.com
> wrote:
> 
> >> From: interchange-users-bounces at icdevgroup.org
> >> [mailto:interchange-users- bounces at icdevgroup.org] On Behalf Of
John1
> >> Sent: Wednesday, November 16, 2005 8:05 AM
> >>
> >> On Wednesday, November 16, 2005 12:32 AM,
list_subscriber at yahoo.co.uk
> >> wrote:
> >>
> >>>> Last night I recompiled with MaxClients to 1024 and this morning
I
> >>>> ran into the problem again, also MaxClients was not reached this
> >>>> time, however I was unable to reach the IC site.  At least in my
> >>>> case I do not believe MaxClients to be the issue.
> >>>>
> >>> I now agree - after some investigation I think that MaxClients
issue
> >>> is a symptom rather than the cause.  No doubt if you have
increased
> >>> MaxClients to 1024 you are now not hitting this limit, but are upi
> >>> still seeing a high Apache client count??  I think that once the
> >>> Interchange daemon
> >>> starts to have problems this must have a cascade effect on the
> >>> Apache processes
> >>> being spawned
> >>>
> >> I believe that my site only actually ceases to respond once
> >> MaxClients is reached.  Even though MaxClients is no doubt not the
> >> underlying cause of the
> >> problem (but a symptom) what I would like to do is automatically
> >> restart apache and interchange when MaxClients is reached as this
> >> would keep website
> >> downtime to a minimum.
> >>
> >> It occurs to me that a good way of doing this would be to pipe the
> >> Apache error log through a script which greps for "server reached
> >> MaxClients setting, consider raising the MaxClients setting".
> >>
> >> As soon as the script sees the above error message being piped to
it
> >> I would
> >> like it to restart the services without delay.
> >>
> >> Is this possible?  I was wondering if anyone knows how to do this?
> >> e.g. an
> >> example script.  If so, I would be very grateful as I am currently
> >> clueless
> >> on how to attempt this.  Thank you.
> >
> > John,
> >
> > I wrote a script yesterday that runs via cron and what it does is it
> > checks if a certain ic page returns a certain string.  If the page
> > does not return properly it tries again, up to 5 times.  If after 5
> > times it doesn't get the page back it restarts Interchange as well
as
> > emails an address to alert that the site restarted.  This way we see
> > the problem before MaxClients is reached since I have figured out
> > that MaxClients is not reached until some undetermined amount of
time
> > after the site stops processing requests.
> >
> So have you found/confirmed that the client count does keeping ticking
> upwards once the site stops processing requests?

No, I have not done this, how would I go about finding this out?

<snip>

> Fantastic, thanks Ron, just implemented it!  It occurs to me that it
may
> be
> useful to call a few system commands before restarting interchange,
and to
> add the output from these commands to the alert e-mail sent.

No problem, I needed something to keep the site up at night and when I
was away from my computer, this should help out.

> e.g.
> 
> #Number of connections to Apache just before Interchange is restarted
> netstat -nt | grep :80 | wc -l
> netstat -nt | grep :443 | wc -l
> 
> #Number of httpd, interchange and mysql processes running just before
> Interchange is restarted
> ps -elf | grep -c httpd
> ps -elf | grep -c interchange
> ps -elf | grep -c mysqld
> 
> #Number of connections each IP address has to server just before
> Interchange
> is restarted
> netstat -ntu | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort
-n
> 
> Although I could just about work out how to insert these into your
script
> it
> would involve educated guesswork and no doubt ugly coding, so would be
> grateful if you could insert these system calls or any other similar
or
> additional commands that you think may provide useful information.
> 
> > You could modify the checkic.html page to also do db lookups to
verify
> > the db connection is still valid.
> >
> If you think this would be useful would you mind also adding this as
my
> Perl
> isn't up to it.
> 
> Thanks for you help!
> 

I'll give it a try, I'm interested to see these values when the server
is going down, although I think these values may be a bit off since the
server will be down or on it's way down when this is ran, however it
still may help.

Thanks,
-Ron