[ic] usertrack file reached 2GB limit, caused IC to onlyreturn partial pages

[Cameron G]

> At 03:17 PM 8/27/2003 +0800, you wrote:
> > > Hello,
> > >
> > > Tonight FrozenCPU.com was acting very strange, only returning partial
IC
> > > pages.  It was consistent in that the page would stop at the same
point
> > > every time.  I setup a test page like this:
> > >
> > > START
> > >
> > > [perl]
> > > my $out='';
> > > my $test=0;
> > > while ($test < 40000) {
> > >
> > > $out .= "$test<BR>";
> > > $test += 1;
> > > }
> > >
> > > return $out;
> > > [/perl]
> > >
> > > FINISH
> > >
> > > And the page would stop at the same point every time.  After doing
some
> > > testing on the dev site, making backups of the live site and testing
> > > again I narrowed it down to a problem with the file structure of the
> > > live site.  I determined it was not the db by using a dev site to
> > > connect to the live db.
> > >
> > > Going directory by directory I quickly noticed that the file usertrack
> > > had reached the 2GB limit of ext3.  I'm not sure why reaching this
limit
> > > was causing IC to only return partial pages, but after tarring up the
> > > file and creating a new usertrack file the site returned to normal.  I
> > > also had plenty of disk space so that was not the issue.  I looked
> > > through the apache, ic, and system logs and the only strange entry was
> > > this in the IC server log:
> > >
> > > - - - [26/August/2003:21:56:46 -0700] - - page server pid 2237 won't
> > > die!
> > > - - - [26/August/2003:21:56:56 -0700] - - page server pid 2227 won't
> > > die!
> > > - - - [26/August/2003:21:57:02 -0700] - - page server pid 2233 won't
> > > die!
> > > - - - [26/August/2003:21:57:10 -0700] - - page server pid 2231 won't
> > > die!
> > > - - - [26/August/2003:21:57:16 -0700] - - page server pid 2219 won't
> > > die!
> > > - - - [26/August/2003:21:57:36 -0700] - - page server pid 2217 won't
> > > die!
> > >
> > > These entries repeated for pretty much every access of the site.
> > >
> > > Anyway I thought if someone else runs into this issue it may save them
a
> > > couple hours of investigation.
> > >
> > > Thanks,
> > > -Ron
> > >
> >
> >perhaps it's time for a 2.4 kernel... ;)
>
> It can still happen then, at least on Red Hat systems with the 2.4 kernel
> and perl 5.6.x as they are distributed. I just recently had similar
> problems with 2 different clients. One had his interchange server in debug
> mode for a very long time, and when the debug log hit binary 2GB ic
quitely
> stopped working. The other client had one of the apache logs hit 2GB and
> apache stopped working.
>
> I imagine that a kernel tweak and recompile would fix these limits, but
> it's worth noting that you are not immune just by virtue of having the
> newer kernel - you also have to be actually using enhanced features :-)
>
> - Ed

It should also probably be noted that simply filling the disk is enough to
make most applications stop functioning, and they do so in a fairly unclean
way. Not much you can do to code around these sorts of issues, it all just
comes down to good housekeeping practices on behalf of the sysadmin :)