[ic] mod_interchange and Apache MaxClients

[Ron Phipps]

> From: interchange-users-bounces at icdevgroup.org
[mailto:interchange-users-
> bounces at icdevgroup.org] On Behalf Of John1
> Sent: Wednesday, November 16, 2005 8:05 AM
> 
> On Wednesday, November 16, 2005 12:32 AM, list_subscriber at yahoo.co.uk
> wrote:
> 
> >> Last night I recompiled with MaxClients to 1024 and this morning I
> >> ran into the problem again, also MaxClients was not reached this
> >> time, however I was unable to reach the IC site.  At least in my
> >> case I do not believe MaxClients to be the issue.
> >>
> > I now agree - after some investigation I think that MaxClients issue
> > is a symptom rather than the cause.  No doubt if you have increased
> > MaxClients to 1024 you are now not hitting this limit, but are upi
> > still seeing a high Apache client count??  I think that once the
> > Interchange daemon
> > starts to have problems this must have a cascade effect on the
Apache
> > processes
> > being spawned
> >
> I believe that my site only actually ceases to respond once MaxClients
is
> reached.  Even though MaxClients is no doubt not the underlying cause
of
> the
> problem (but a symptom) what I would like to do is automatically
restart
> apache and interchange when MaxClients is reached as this would keep
> website
> downtime to a minimum.
> 
> It occurs to me that a good way of doing this would be to pipe the
Apache
> error log through a script which greps for "server reached MaxClients
> setting, consider raising the MaxClients setting".
> 
> As soon as the script sees the above error message being piped to it I
> would
> like it to restart the services without delay.
> 
> Is this possible?  I was wondering if anyone knows how to do this?
e.g.
> an
> example script.  If so, I would be very grateful as I am currently
> clueless
> on how to attempt this.  Thank you.

John,

I wrote a script yesterday that runs via cron and what it does is it
checks if a certain ic page returns a certain string.  If the page does
not return properly it tries again, up to 5 times.  If after 5 times it
doesn't get the page back it restarts Interchange as well as emails an
address to alert that the site restarted.  This way we see the problem
before MaxClients is reached since I have figured out that MaxClients is
not reached until some undetermined amount of time after the site stops
processing requests.  This morning the site went down and my script
immediately noticed the problem and restarted IC properly.  Of course
this is just to keep the site up, we still need to figure out why it's
happening.

Below is the script:

#!/usr/bin/perl

use LWP;
        
my $url = 'http://www.yoursite.com/checkic.html';

## CMD to restart
my $IC_COMMAND = "/root/scripts/icrestart.sh";

## Where is sendmail? 
my $SENDMAIL_CMD = "/usr/sbin/sendmail";
    
## what mail account should we alert to?
my $MAIL_RECEIVER = "email\@yoursite.com";
        
## what should the subject be?
my $MAIL_SUBJECT="Alert!";

## who should we send from?
my $MAIL_SENDER="root\@yoursite.com";

my $browser = LWP::UserAgent->new;
$browser->timeout(30);
        
my $count = 0;
my $up = 0;

while ($count <= 4) {
        my $response = $browser->get($url); 
        if ($response->content =~ m/UP/) {
                $count = 5;
                $up = 1;
        }
        $count++;
}

if ($up == 1) {
} else {
        system $IC_COMMAND;
        mail_admin("IC restarted!");
        print "IC restarted!";
}

sub mail_admin {
        my $mail = "From: $MAIL_SENDER <>\nTo: $MAIL_RECEIVER\nSubject:
$MAIL_SUBJECT\nX-Priority: 1\n\n$_[0]\n";
        print STDERR $mail if $DEBUG;
        return if $NO_MAIL;
        open SENDMAIL, "| $SENDMAIL_CMD -t" and
        print SENDMAIL $mail and
        close SENDMAIL  or
        die "Failed to send alert mail to $MAIL_SENDER: $!";
}


Then put a page "checkic.html" on your site with the content of "UP" in
the pages directory and put the url to this page in the $url variable
above.

You could modify the checkic.html page to also do db lookups to verify
the db connection is still valid.

Take it easy,
-Ron