[ic] mod_interchange and Apache MaxClients

[Kevin Walsh]

John1 [list_subscriber at yahoo.co.uk] wrote:
> On Thursday, November 17, 2005 8:33 PM, kevin at cursor.biz wrote:
> > I'm thinking about the PSP hanging issue and how that could effect
> > things and I'm lead to believe that the following is happening:
> > 
> >    1. User submits a request that causes a call to the PSP to be      
> > made. 
> > 
> >    2. The PSP is taking its sweet time, so IC waits for a response.
> > 
> >    3. 10 seconds later, mod_interchange gives up and resubmits the
> >       user's request.  This compounds the problem.
> > 
> I am trying to understand the basics of the
> apache<->mod_interchange<->interchange communication process...
> 
> So what happens if mod_interchange resubmits to interchange whilst it is
> waiting for a response?  Does it queue the resubmission, or just send a
> reply back saying it is still waiting and then drops the resubmissions? 
> If it queues the resubmissions, what does it do if the PSP does eventually
> get back with a reply after say 40 seconds?  Does it then drop the 2
> resubmissions?
> 
I just checked the code again, as my earlier followup about resubmissions
sounded a little strange.  Mod_interchange will attempt to connect to
Interchange a number of times, but will only send the actual request once.

Apache calls mod_interchange's ic_handler() function.  That function
calls ic_connect() to connect to open a socket to Interchange.  The
ic_connect() function will run in sleep/retry loop until it connects,
or until the try limit is reached.  Once connected, ic_connect() will
return to ic_handler().  The ic_handler() function then calls the
ic_send_request() to, well, send the request to Interchange.

The above looks like this in the code:

        /*
         *      connect to the Interchange server
         */
        ic_buff = ic_connect(r,conf_rec);
        if (!ic_buff)
                return HTTP_INTERNAL_SERVER_ERROR;

        /*
         *      send the client's request to Interchange
         */
        rc = ic_send_request(r,conf_rec,ic_buff);

        if (rc == OK)
                rc = ic_transfer_response(r,ic_buff,conf_rec->connect_timeout);

        /*
         *      close the Interchange socket and return
         */
        ap_bclose(ic_buff);
        return rc;

> >
> >    4. Loop back to #2 ten times, with a two second delay between      
> > attempts. 
> > 
> That's interesting 10 seconds + 2 second delay = 12 seconds x 10 retries =
> 120 seconds, which is exactly the length of time (give or take a few
> seconds) it takes the browser to give up with "Cannot find server or DNS
> error".  I assume this is not a coincidence?
>
I assume not. :-)

> >
> > The above will probably wind up eating a lot of sockets, and also
> > munch its way through your processes leading to the problems reported.
> > 
> Would it be worth adding the following to Ron's script to show the number
> of Unix sockets in use before restarting Apache and Interchange?
> 
> netstat -nx | sed '1,2d' | awk '{print $5,$8}' | sed -e 's/:[0-9]*//g' |
> sort | uniq -c | sort -n
> 
> Or would this not really add any useful information?
> 
I've not looked at any of the restart scripts that have been posted
over the last couple of days.  I'll leave that for you to decide.
It doesn't look as if it can do any harm.

> > 
> > Can you check your logs to see if you can spot anything that looks
> > like "Failed to select the response header."
> > 
> Well I have plenty of entries recently like this:
> [Thu Nov 17 16:02:35 2005] [error] (111)Connection refused: access to
> /index.html failed for x.x.x.x, reason: Connection failed
> 
You'll get that when all of the retry attempts have been exhausted.

> and,
> 
> [Thu Nov 17 21:12:42 2005] [error] (104)Connection reset by peer: access
> to /index.html failed for x.x.x.x, reason: error sending headers to client
> 
> I guess it is this latter error that you are referring to?
> 
You didn't show the x.x.x.x value, but I'm sure that it'll be different
from the one in the error you posted earlier in this article.

The first error (Connection failed) would be generated if ic_connect()
couldn't connect to Interchange.  The second (Error sending headers to
client) would be generated if ic_transfer_response() couldn't make
use of the (probably closed) client socket.  The two error messages
should not be found in sequence, with the same IP, unless the client
refreshes the page during a delay.

> >
> > I could be wrong, but the above sounds plausible.
> > 
> > Possible workarounds, if you find the above error message.
> > 
> >    1. Upgrade Interchange.  Upgrade it now!  See the wget comments.
> > 
> Would upgrading to the wget Payment.pm actually solve the above? 
> Presumably 
> the issue of mod_interchange resubmitting requests ensuing munching of
> sockets and processes would still apply to the wget approach?  Is that so?
>
I was wrong about mod_interchange resubmitting requests, which is
good.  It would have been bad if mod_interchange had been coded to
do that.

> 
> Are there any notable advantages to upgrading from Perl 5.6?
> 
Yes - Perl 5.6 was written by a Brontosaurus. :-)

5.8 is bang up to date.  Less bugs etc.  I'm sure there's a change log
file for Perl somewhere if you're interested in the details.

>
> Is there anything about sticking with Red Hat 7.3 that should cause me
> particular concern?
>
If I was running Red Hat 7.3, I'd be more concerned about security
than anything else.

> 
> >    2. Add "ConnectTries 1" to the mod_interchange <Location>
> >       parameters in your Apache configuration.  This will cause the
> >       initial timeout to force a failure, rather than retry the request.
> > 
> Are there any negative consequences to doing this, that would actually
> matter in practice?
>
As mod_interchange doesn't resubmit requests, I'd just leave ConnectTries
to use its default value (10).

> 
> Are there any good reasons why mod_interchange may get timeouts in the
> normal course of a customer browsing the site?
>
Mod_interchange will timeout its connection to Interchange if Interchange
is down, or is taking a long time to connect.  The retry is a good thing,
in that it allows you to restart Interchange without any of your clients
realising;  Mod_interchange will just "wait" a while, for Interchange to
come back up, and then carry on as usual.

> >
> >    3. Increase the mod_interchange timeout.  This can only be done by
> >       editing mod_interchange.c, to increase the IC_DEFAULT_TIMEOUT
> >       definition, and recompiling mod_interchange.so.  Perhaps I
> >       should make that configurable in the Apache config.
> > 
> What would you suggest for IC_DEFAULT_TIMEOUT?
>
I would suggest 10 seconds, which is why that's the default. :-)
Feel free to tune that value if you like.  I've uploaded a new
mod_interchange (1.34) that allows the value to be tuned without
having to recompile the module.

> 
> Again, what would the negative consequences be of increasing
> IC_DEFAULT_TIMEOUT?
>
Mod_interchange will wait longer before timing out.  Whether this
is good or bad is up to you.

> 
> BTW, have you any idea where is my original post about payment service
> provider (PSP) timeouts and Net::SSLeay?  The mailing list doesn't seem to
> have mailed me a copy (even though I have it configured to mail me copies
> of  all posts including my own).  But you have obviously seen it Kevin.  Ron &
> Jeff, did the mailing list send you a copy?
> 
If I posted a followup then the mail list must have sent me the article.
The mail list doesn't send me copies of my own articles because my mail
client keeps a copy of everything I post.

-- 
   _/   _/  _/_/_/_/  _/    _/  _/_/_/  _/    _/
  _/_/_/   _/_/      _/    _/    _/    _/_/  _/   K e v i n   W a l s h
 _/ _/    _/          _/ _/     _/    _/  _/_/    kevin at cursor.biz
_/   _/  _/_/_/_/      _/    _/_/_/  _/    _/