[ic] need help optimizing-process product data and export to tab delim file

Aaron Hazelton interchange-users@icdevgroup.org
Tue Jan 21 15:57:01 2003


Good Day to All...

I am trying to export my products into a tab-delim
text file for another database to use, however, there
is a limited amount of processing to be done and 
changes that need to be made first.  I could not
figure out how to do the processing with any of the
[export] type things done in the UI, so I struck out
with the following:

[tmp done][log file="[time]%Y%m%d[/time].txt"
		create=1
		delim="\t"
		type="text"
		interpolate=1
][query sql="SELECT sku, title, comment, image, price, prod_group
		FROM products
		ORDER BY sku;"
type=list][list]http://www.site.com/cgi-bin/site.cgi/[sql-param
sku].html	[sql-param title]	[filter striphtml][sql-param
comment][/filter]	[sql-param price]	[either][sql-param
image][or]na.jpg[/either]	[sql-param prod_group]	[sql-param sku]
[/list][/query][/log][/tmp]

This is probably a mess, but it works.  The problem is
that if I go much over 1 or 2 thousand it dies (no error
in IC, but "500 Internal Server Error" to browser after 
approx 8 minutes of 99% load on server).  This
products database has almost 50,000 records.

Anything I can do about this?

TIA
--
Aaron