[ic] usertrack file reached 2GB limit, caused IC to only return partial pages

[Ed LaFrance]

At 03:17 PM 8/27/2003 +0800, you wrote:
> > Hello,
> >
> > Tonight FrozenCPU.com was acting very strange, only returning partial IC
> > pages.  It was consistent in that the page would stop at the same point
> > every time.  I setup a test page like this:
> >
> > START
> >
> > [perl]
> > my $out='';
> > my $test=0;
> > while ($test < 40000) {
> >
> > $out .= "$test<BR>";
> > $test += 1;
> > }
> >
> > return $out;
> > [/perl]
> >
> > FINISH
> >
> > And the page would stop at the same point every time.  After doing some
> > testing on the dev site, making backups of the live site and testing
> > again I narrowed it down to a problem with the file structure of the
> > live site.  I determined it was not the db by using a dev site to
> > connect to the live db.
> >
> > Going directory by directory I quickly noticed that the file usertrack
> > had reached the 2GB limit of ext3.  I'm not sure why reaching this limit
> > was causing IC to only return partial pages, but after tarring up the
> > file and creating a new usertrack file the site returned to normal.  I
> > also had plenty of disk space so that was not the issue.  I looked
> > through the apache, ic, and system logs and the only strange entry was
> > this in the IC server log:
> >
> > - - - [26/August/2003:21:56:46 -0700] - - page server pid 2237 won't
> > die!
> > - - - [26/August/2003:21:56:56 -0700] - - page server pid 2227 won't
> > die!
> > - - - [26/August/2003:21:57:02 -0700] - - page server pid 2233 won't
> > die!
> > - - - [26/August/2003:21:57:10 -0700] - - page server pid 2231 won't
> > die!
> > - - - [26/August/2003:21:57:16 -0700] - - page server pid 2219 won't
> > die!
> > - - - [26/August/2003:21:57:36 -0700] - - page server pid 2217 won't
> > die!
> >
> > These entries repeated for pretty much every access of the site.
> >
> > Anyway I thought if someone else runs into this issue it may save them a
> > couple hours of investigation.
> >
> > Thanks,
> > -Ron
> >
>
>perhaps it's time for a 2.4 kernel... ;)

It can still happen then, at least on Red Hat systems with the 2.4 kernel 
and perl 5.6.x as they are distributed. I just recently had similar 
problems with 2 different clients. One had his interchange server in debug 
mode for a very long time, and when the debug log hit binary 2GB ic quitely 
stopped working. The other client had one of the apache logs hit 2GB and 
apache stopped working.

I imagine that a kernel tweak and recompile would fix these limits, but 
it's worth noting that you are not immune just by virtue of having the 
newer kernel - you also have to be actually using enhanced features :-)

- Ed


===============================================================
New Media E.M.S.              Technology Solutions for Business
463 Main St., Suite D         eCommerce | Consulting | Hosting
Placerville, CA  95667        edl at newmediaems.com
(530) 622-9421                http://www.newmediaems.com
(866) 519-4680 Toll-Free      (530) 622-9426 Fax
===============================================================