[ic] interchange processes hanging

[Alex Schmelkin]

Approaching the extremely busy holiday season on a moderate traffic (1000+ 
orders/day) ic site, we currently have a pretty severe interchange hanging 
problem on which I am hoping to solicit some advice. 

setup:
ic 4.9.9, RPC mode
perl 5.6.1
linux redhat 9
mysql 4.0.16
4 load balanced apaches
mod_interchange talks to an IC server on each box
1 db/nfs server
nfs sessions 

corresponding with periods of increased visitor activity, we are seeing 
inexplicable interchange process server hangs which render a particular web 
box's interchange processes unusable without a complete kill -9 restart.  we 
have yet to be able to connect the problem to any specific problem with our 
setup or specific user activity.  upon restarting IC, browsing and checkout 
on that box is extremely fast again, until the next time interchange becomes 
unstable. 

during periods when this occurs, a view of the interchange processes lists 
them all in the dreaded "D   uninterruptible sleep (usually IO)" state.  
sadly, we don't have any leads based on errors in our logs on either the web 
boxes or the database server. 

does anyone have any suggestions on places to look that could be causing the 
interchange process hangs?  what types of interchange transactions could 
cause all processes on a box to hang in the D state until a server restart? 

could users simultaneously doing "something" (view product, checkout, etc.) 
cause database lock contention either globally (which would affect all 
interchange servers) or locally (just on the individually ic server's db 
connection pool to the database)?  innodb transaction problems? 

any pointers or advice is extremely appreciated.  as the holidays are upon 
us, load is going to spike, and we urgently seek a lead to tracking down our 
server hang problem. 

Thank you,
Alex