[ic] Trying to import 50,000 items

[Dan Browning]

At 11:27 AM 8/6/2002 -0400, you wrote:
>Quoting Nicholas Cook (ncook@foxmillpets.com):
> > I am trying to import a catalog of roughly 50,000 items and I am finding
> > the import from the admin ui too slow (timing out on webserver).
> > Database is PostgreSQL 7.1.3 and Interchange 4.8.3.  Is there a command
> > line utility to allow this from a tab delineated file.  Also, curiosity
> > question.  For those other people that have done the same, roughly how
> > long did it take for your import to finish.
> >
>
>I would check out the latest Interchange from CVS (the devel branch) and
>get the file:
>
>         code/UI_Tag/import_fields.coretag.
>
>Replace
>
>     lib/UI/usertag/import_fields.tag
>
>with that, for I think Dan Browning just fixed this problem.
>
>This is due to Postgres by default doing a sync with every write. The
>[import-fields ..] tag was also suboptimal in the way it did things.

Mike helped me a lot in getting this one.  :-) I haven't uploaded it to 
devel yet (will happen today), but a patch against 4.8.5 
(import_fields.tag) is below (I don't think any lines are long enough to 
wrap).  For me, it drastically increased my import speed (by a factor of 10 
or 20).

+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
| Dan Browning, Kavod Technologies <db@kavod.com>
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
If you can read this, you're too close.


Index: import_fields.tag
===================================================================
RCS file: /var/cvs/interchange/dist/lib/UI/usertag/Attic/import_fields.tag,v
retrieving revision 2.0.2.1
diff -u -r2.0.2.1 import_fields.tag
--- import_fields.tag   24 May 2002 20:34:44 -0000      2.0.2.1
+++ import_fields.tag   6 Aug 2002 18:27:56 -0000
@@ -251,11 +251,20 @@
                         }
                 }
                 for ($i = 0; $i < $idx; $i++) {
-                       $set[$i]->($k, $f[$i]);
                         if ($opt->{cleanse}) {
                                 delete $keys{$k};
                         }
                 }
+               if(@names) {
+                       my %hash;
+                       @hash{@names} = @f;
+                       $db->set_slice($k, \%hash);
+               }
+               if($@) {
+                       my $msg = ::errmsg("error on update: %s", $@);
+                       ::logError($msg);
+                       $out .= $msg;
+               }
                 $count++;
         }
         if ($opt->{cleanse}) {