[ic] [import] mangles UTF-8 characters

Stefan Hornburg (Racke) racke at linuxia.de
Sat Apr 17 12:33:28 UTC 2010


we are in the progress to migrate two projects from a patched
5.7.1 installation to 5.7.6.

These are the relevant settings in catalog.cfg:

DatabaseDefault PG_ENABLE_UTF8 1

In etc/log_transaction this code creates the orderline entries:

[import table=orderline type=LINE continue=NOTES]

This fails if items carry UTF-8 characters in their name/description,

import into orderline failed: DBD::Pg::st execute failed: ERROR:  invalid byte sequence for encoding "UTF8": 0xf1612028

Similar code using [query] instead of import works.

This was discussed on IRC yesterday, and we found that the following
two code parts probably need to examined in order to fix this bug:

Vend::Data, import_text, 303ff (writing temporary file)

if($options->{file}) {
	$fn = $options->{file};
		or die ::errmsg("No absolute file names like '%s' allowed.\n", $fn);
else {
	Vend::Util::writefile($fn, $text)
		or die ("Cannot write temporary import file $fn: $!\n");

Vend::Table::Common, 1637ff (reading temporary file)

sub new_filehandle {
	my $fh = shift;
	binmode($fh, ":utf8") if $::Variable->{MV_UTF8};
	return $fh;

As a side note, avoiding the temporary file altogether in this case would be
a good idea as well :-).


LinuXia Systems => http://www.linuxia.de/
Expert Interchange Consulting and System Administration
ICDEVGROUP => http://www.icdevgroup.org/
Interchange Development Team

More information about the interchange-users mailing list