[ic] usertrack file reached 2GB limit, caused IC to only return partial pages

[Ron Phipps]

Hello,

Tonight FrozenCPU.com was acting very strange, only returning partial IC
pages.  It was consistent in that the page would stop at the same point
every time.  I setup a test page like this:

START

[perl]
	my $out='';
	my $test=0;
	while ($test < 40000) {
	
		$out .= "$test<BR>";
		$test += 1;
	}

	return $out;
[/perl]

FINISH

And the page would stop at the same point every time.  After doing some
testing on the dev site, making backups of the live site and testing
again I narrowed it down to a problem with the file structure of the
live site.  I determined it was not the db by using a dev site to
connect to the live db.

Going directory by directory I quickly noticed that the file usertrack
had reached the 2GB limit of ext3.  I'm not sure why reaching this limit
was causing IC to only return partial pages, but after tarring up the
file and creating a new usertrack file the site returned to normal.  I
also had plenty of disk space so that was not the issue.  I looked
through the apache, ic, and system logs and the only strange entry was
this in the IC server log:

- - - [26/August/2003:21:56:46 -0700] - - page server pid 2237 won't
die!
- - - [26/August/2003:21:56:56 -0700] - - page server pid 2227 won't
die!
- - - [26/August/2003:21:57:02 -0700] - - page server pid 2233 won't
die!
- - - [26/August/2003:21:57:10 -0700] - - page server pid 2231 won't
die!
- - - [26/August/2003:21:57:16 -0700] - - page server pid 2219 won't
die!
- - - [26/August/2003:21:57:36 -0700] - - page server pid 2217 won't
die!

These entries repeated for pretty much every access of the site.

Anyway I thought if someone else runs into this issue it may save them a
couple hours of investigation.

Thanks,
-Ron