[ic] Corruption of products file (double tabs after SKU?); problem, temp workaround, seeking solution

[TMN Interchange]

Summary:

Using the TAB delimited file for a large number of products (over 1000).
Having problems
with corruption of the file. I will covert over to SQL at some stage, but in
the meantime
wondering if anyone else is having same problem, heres a solution (quick &
nasty), and
seeking a better solution :o)

contact: interchange@welcometomyuniverse.com

Cheers,
Paul

_______

Problem:

Large number of items in a TAB delimited file, having modified the construct
demo pages but
keeping the original data structure. Over time, the file is being corrupted
(products just entered
keep their SKU, but lose their item information, pricing shows up as 0,
etc).

I've already tried upgrades of "stable" Interchange. I don't want to run the
Alpha versions.

Visible cause of problem:

There seems to be a double tab after the SKU, so the information is all
there (description, pricing
etc) but since we are using TAB delimited files, suprisingly enough its not
happy.

Thoughts:

We are entering in items normally, have already considered cut-and-paste
issues along with tabbing
across fields for entry. Typing up an entry and switching between fields
using mouse clicks still
comes up with same kind of errors.

Ugly workaround:

Running a cron job each minute that sucks in the product database, does
replacements of double
tabs after the SKU, writes it out to a temporary file, quick check to see
nothing changed in the
meantime then renames the temp file to products.txt

Source code included at the end of this email. Please, do not flame me! I
know that its Ugly, its
brute force, it doesn't do a lot of nice things that it could/should do, and
if you show it to your
friends that they will ridicule you! However, it does work, and I'm Lazy (as
a virtue)  :o)

Requested solution:

Any thoughts/hints much appreciated!!!


_______

I am running this as a cron job each minute (as Root - Yes I am aware of
implications).


#!/usr/bin/perl -w

use File::stat;

my $now = time();
my $product_path = '/path/to/catalog/directory/product/directory/';
my @contents = `cat $product_path/products.txt`;

my $desired_username = 'interch';        # depends on your setup
my $desired_usergroup = 'interch';    # depends on your setup

open(FILE,">$product_path/products-new.txt");
foreach (@contents) {
        s/^(\w+)\t\t/$1\t/;    # this bit changes a SKU|tab|tab| with
SKU|tab|
        print FILE;
}
close(FILE);

my $statinfo = stat("$product_path/products.txt");

if($statinfo->mtime < ($now - 5)) {    # not modified after our start time
plus margin....

        my(@diff) = `diff $product_path/products.txt
$product_path/products-new.txt`;    # did we even make changes?!

        if(scalar @diff && $diff[0] !~ m|^\s*$|) {    # something did
change, so lets modify...

# aargh... just spotted race condition between the diff/renaming, and a
modification to file... locking would be good idea :o(

                rename("$product_path/products-new.txt",
"$product_path/products.txt");

                my($uid,$gid);
                (undef,undef,$uid,undef) = getpwnam($desired_username);
                (undef,undef,undef,$gid) = getpwnam($desired_usergroup);
                chown $uid, $gid, "$product_path/products.txt";

                my $mode = 0644;
                chmod $mode, "$product_path/products.txt";
        }
}