[ic] one store gone nuts - 500 errors - (still) HELP please!

[Peter]

On 09/07/2007 04:27 PM, Peter wrote:
> On 09/07/2007 11:22 AM, Glenn McCalley wrote:
>> Still chasing this one.  (Sorry to disappear - 2,000 miles driving past
>> 2 days, family thing, real fun).
>>
>> Current situation:
>> - Perl upgraded to 5.8.8
>> - Bunldle::InterchangeKitchenSink rebuilt (takes a while, doesn't it?)
>> - I did find out where the icdebug file is logging, that does work but
>> doesn't show anything illuminating.
>> - ran expireallm thought maybe huge session lib or something
>> - Restart IC - other stores still OK, this one still crashes on "Place
>> Order".
>>
>> Verified that AuthNet process is completed OK, have the card
>> authorization - it happens after that but before receipts are processed.
>>
>> I agree it's Perl or Perl-triggered-by-IC.
>> The server error log still just shows "Premature end of script headers".
>> There is a message there that I can't see.
>> Is there a way or where in the IC code:
>> -- could I add CGI.pm's "fatalsToBrowser" or something like that to get
>> Apache to log whatever is being said by Perl as it dies?
> 
> Perl (or any other program for that matter) doesn't say anything when it
> dumps core, it just dies (they sometimes say something like,
> "segmentation fault, dumping core" which we can assume for our purposes
> that that's what it would have said).
> 
>> -- where to start putting some additional debug log statements to tie it
>> down more specifically?
> 
> 
> in log_transaction put statements like these:
> 
> [calc]Debug('At checkpoint #1');[/calc]
> ...some line of code here...
> [calc]Debug('At checkpoint #2');[/calc]
> 
> ...etc...
> 
> Those debug statements will appear in your icdebug file, by seeing which
> is the last statement to appear in the file you can find the exact line
> of code that is triggering the crash.  Let us know what that line is,
> maybe there's another way to code that line and work around the bug
> (though don't be surprised if it crops up elsewhere).
> 
> If you want to continue to track down the bug then you may need to
> re-compile perl with the -d flag added to either ccflags or ldflags (I
> forget which) when you run the Configure script, then you can get a
> backtrace on the core dump with: gdb /path/to/perl /path/to/core/dump
> 
> I think the gdb command for a backtrace is ... bt or backtrace or
> something (it's been ages since I've used gdb, so not sure).  At that
> point send an email to the perl mailing list and they can help you
> further (since this is actually a perl issue, not an IC issue).

One additional note.  These types of problems tend to happen almost
exclusively on BSD systems.  I highly suspect that there are bugs in the
standard libc libraries included with BSD (which perl would be compiled
against).  Your other possible solution is to switch to Linux and the
problems will likely disappear.

Peter