[ic] IC not responding

[Ron Phipps]

> From: interchange-users-bounces at icdevgroup.org
[mailto:interchange-users-
> bounces at icdevgroup.org] On Behalf Of John1
> Sent: Tuesday, November 15, 2005 3:40 AM
> 
> On Tuesday, November 15, 2005 1:58 AM, rphipps at reliant-solutions.com
> wrote:
> 
> >> From: interchange-users-bounces at icdevgroup.org
> >> [mailto:interchange-users- bounces at icdevgroup.org] On Behalf Of
John1
> >> Sent: Monday, November 14, 2005 4:06 PM
> >>
> >> On Monday, November 14, 2005 7:12 PM, racke at linuxia.de wrote:
> >>
> >>> Ron Phipps wrote:
> >>>>> From: interchange-users-bounces at icdevgroup.org
> >>>>
> >>>> [mailto:interchange-users-
> >>>>
> >>>>> bounces at icdevgroup.org] On Behalf Of jeff at hisgirlfridays.com
> >>>>> Sent: Friday, November 11, 2005 11:24 AM
> >>>>>
> >>>>> On Fri, Nov 11, 2005 at 08:40:30AM -0800, Ron Phipps wrote:
> >>>>>
> >>>>>
> >>>>>> We've got a weird issue going on right now where all of a
sudden
> >>>>>> IC will
> >>>>>> stop servicing requests and utilization of the server goes to
> >>>>>> just about
> >>>>>> 0.  If I restart IC everything goes back to normal.  This has
> >>>>>> gone on for the last 3 days at different points in the day.
The
> >>>>>> ip address
> >>>>>
> >>>>> The next time this happens, restart apache instead of IC and
tell
> >>>>> me if it goes back to normal.  If it does, then I have more to
> >>>>> say about the issue and also have a potential fix.
> >>>>>
> >>>>> Jeff
> >>>>
> >>>>
> >>>> Unfortunately this was not the case.  The potential fix of using
> >>>> apache's MaxRequestsPerChild 1000-5000 did not seem to help.
> >>>>
> >>>> Does anyone know of a way to troubleshoot IC, perhaps with debug
> >>>> statements to see why it can no longer handle requests?  IC is
> >>>> still running, I can see all the rpc servers, however they do not
> >>>> take up much cpu usage and it's almost like they are not even
> >>>> there. Apache/mod_interchange does not error out and a restart of
> >>>> IC instantly allows Apache to start handling requests again.
> >>>
> >>> First of all you should try to strace all the IC processes to see
if
> >>> it does system calls and watch your logfiles (IC and system
> >>> logfiles) as well. If no system calls happened it might caught up
> >>> in an infinite loop somewhere.
> >> Do you know if you are bumping up against your Apache MaxClients
> >> setting (in
> >> httpd.conf)?
> >>
> >> i.e. Can you see the following entry in your Apache error.log when
> >> the problems start?
> >> "server reached MaxClients setting, consider raising the MaxClients
> >> setting"
> >>
> >> Just a possibility, as we are having a similar problem (but may be
> >> unrelated
> >> to yours) caused by bumping up against Apache MaxClients (which I
> >> have just
> >> posted to mailing list as "mod_interchange and Apache MaxClients").
> >>
> >
> > Thanks for this information, I was looking in the error log for the
> > particular vhost and had not looked in the apache site wide error
log,
> > and sure enough I see each time the server "went down" the max
clients
> > reached error.  I'm currently at 256 max clients, I'm going to raise
> > that to 1024 tonight after I recompile apache.  We've also noticed
> > during the times this issue has occurred that we are being crawled
by
> > search engines and also have scripts trying to look for holes in our
> > apache install.
> >
> > Thanks for your help!
> > -Ron
> >
> From reading around on the web, it would seem that MaxClients 150
ought
> to
> be enough even for moderately busy web servers.  Does anyone else have
a
> view on this?  Ron, I can understand your idea of raising MaxClients
to
> help
> the problem, (although I can't try this myself as my Virtual Server
> environment restricts the number of concurrent processes I can run).

Yes, I agree that MaxClients is not the issue, I was trying some
suggestions from another IC user that has seen this problem, hoping that
it would at least mitigate the issue while a solution was found.

> However, I am sure you will agree that raising MaxClients is really a
work
> around rather than a solution to the underlying problem.  I can only
> presume
> that this problem of scripts looking for security holes hammering
websites
> must be a problem common to every website on the Internet, and
therefore
> presumably every website is bumping up against MaxClients on a regular
> basis!?  I presume that most websites come back to life as soon as the
> script robot goes away, but even so, all websites will be brought down
by
> this sort of robot for the duration of its visit.  So, to get to my
point,
> are there any easy ways to stop these script robots racking up the
> MaxClients count in the first place.  i.e. is there anything in Apache
or
> Apache modules that can be used to spot these "robot attacks" and drop
> their
> requests before they cause Apache to spawn loads of processes?  I'd be
> grateful for any ideas, especially as upping the MaxClients setting is
not
> really an option for me.  Thanks

I'm going to continue looking through the error logs to see if anything
points to malicious behavior. It seems like it would be robot/traffic
related since it just began happening last week and this site has been
up using mod_interchange for over a year without problems, and running
IC for 2.5 years.

And the search continues....
-Ron