[ic] one store gone nuts - 500 errors - (still) HELP please!

Peter peter at pajamian.dhs.org
Fri Sep 7 19:27:17 EDT 2007


On 09/07/2007 11:22 AM, Glenn McCalley wrote:
> 
> ----- Original Message ----- From: "Peter" <peter at pajamian.dhs.org>
> To: <interchange-users at icdevgroup.org>
> Sent: Tuesday, September 04, 2007 9:01 PM
> Subject: Re: [ic] one store gone nuts - 500 errors - HELP!
> 
> 
>> On 09/04/2007 05:39 PM, Glenn McCalley wrote:
>>>
>>> ----- Original Message ----- From: "Peter" <peter at pajamian.dhs.org>
>>> To: <interchange-users at icdevgroup.org>
>>> Sent: Tuesday, September 04, 2007 7:41 PM
>>> Subject: Re: [ic] one store gone nuts - 500 errors - HELP!
>>>
>>>
>>>> On 09/04/2007 04:27 PM, Glenn McCalley wrote:
>>>>> ----- Original Message ----- From: "Peter" <peter at pajamian.dhs.org>
>>>>> To: <interchange-users at icdevgroup.org>
>>>>> Sent: Tuesday, September 04, 2007 7:05 PM
>>>>> Subject: Re: [ic] one store gone nuts - 500 errors - HELP!
>>>>>
>>>>>
>>>>>> On 09/04/2007 02:22 PM, Glenn McCalley wrote:
>>>>>>> Hi all.
>>>>>>>
>>>>>>> Store works fine until you press the final "Place Order" button
>>>>>>> on the
>>>>>>> checkout page.
>>>>>>> 500 - Internal Server Error.
>>>>>>
>>>>>> The only time I have seen where the global error.log file doesn't
>>>>>> show
>>>>>> something is if Perl itself crashes, check for a core dump.>>>>>
>>>>>> Peter
>>>>>>
>>>>> Hi Peter, yes I do get perl.core dumps.  Sorry should have said that
>>>>> before.
>>>>>
>>>>> - FreeBSD 5.3
>>>>> - Perl 5.8.5 compiled myself, not the port.
>>>>> - Interchange 5.2.0
>>>>> - No Perl updates additions in some time.
>>>>>
>>>>> BUT -- the other stores are fine.  Hald a dozen of them.  Same IC.
>>>>> Selling things, issuing receipts, making money.
>>>>> That's what's got me swinging - this box has been rock solid for
>>>>> over a
>>>>> year and it's only -this-one-store-.
>>>>> I'd think if it was Perl by itself they'd all be smoking ruins.
>>>>
>>>> If you're getting core dumps from Perl then that's definitely your
>>>> problem.  They can be caused by the oddest combination of things and
>>>> can
>>>> easily happen on just one shop.
>>>>
>>>> Upgrade to the latest perl (5.8.8) and build a new
>>>> Bundle::InterchangeKitchenSink, DBD::mysql (or whatever other database
>>>> you have) and any other perl modules you might be using.  >>> Do
>>>> that and your problems will almost certainly disappear.
>>>>
>>>> Peter
>>>
>>> Agreed, except how can it easily happen on just one store?  They all use
>>> the same Perl,
>>
>> Bugs like this are typically buffer overflow bugs that happen only in
>> rare circumstances, so a particular combination of things could be
>> triggering it (ie this internal function is called with that exact value
>>
>> Of course the other option is to leave perl and the bug in place, find
>> the trigger in IC, and avoid triggering it.  If upgrading Perl doesn't
>> fix the problem that may be your best solution.
>>
>>
>> Peter
>>
> Still chasing this one.  (Sorry to disappear - 2,000 miles driving past
> 2 days, family thing, real fun).
> 
> Current situation:
> - Perl upgraded to 5.8.8
> - Bunldle::InterchangeKitchenSink rebuilt (takes a while, doesn't it?)
> - I did find out where the icdebug file is logging, that does work but
> doesn't show anything illuminating.
> - ran expireallm thought maybe huge session lib or something
> - Restart IC - other stores still OK, this one still crashes on "Place
> Order".
> 
> Verified that AuthNet process is completed OK, have the card
> authorization - it happens after that but before receipts are processed.
> 
> I agree it's Perl or Perl-triggered-by-IC.
> The server error log still just shows "Premature end of script headers".
> There is a message there that I can't see.
> Is there a way or where in the IC code:
> -- could I add CGI.pm's "fatalsToBrowser" or something like that to get
> Apache to log whatever is being said by Perl as it dies?

Perl (or any other program for that matter) doesn't say anything when it
dumps core, it just dies (they sometimes say something like,
"segmentation fault, dumping core" which we can assume for our purposes
that that's what it would have said).

> -- where to start putting some additional debug log statements to tie it
> down more specifically?


in log_transaction put statements like these:

[calc]Debug('At checkpoint #1');[/calc]
...some line of code here...
[calc]Debug('At checkpoint #2');[/calc]

...etc...

Those debug statements will appear in your icdebug file, by seeing which
is the last statement to appear in the file you can find the exact line
of code that is triggering the crash.  Let us know what that line is,
maybe there's another way to code that line and work around the bug
(though don't be surprised if it crops up elsewhere).

If you want to continue to track down the bug then you may need to
re-compile perl with the -d flag added to either ccflags or ldflags (I
forget which) when you run the Configure script, then you can get a
backtrace on the core dump with: gdb /path/to/perl /path/to/core/dump

I think the gdb command for a backtrace is ... bt or backtrace or
something (it's been ages since I've used gdb, so not sure).  At that
point send an email to the perl mailing list and they can help you
further (since this is actually a perl issue, not an IC issue).

Peter


More information about the interchange-users mailing list