[interchange-bugs] [rt.icdevgroup.org #339] [import] mangles UTF-8 characters

[Stefan Hornburg via RT]

Sat Apr 17 13:16:01 2010: Request 339 was acted upon.
Transaction: Ticket created by racke
       Queue: Interchange
     Subject: [import] mangles UTF-8 characters
       Owner: Nobody
  Requestors: racke at linuxia.de
      Status: new
 Ticket <URL: http://rt.icdevgroup.org/Ticket/Display.html?id=339 >


Hello,

we are in the progress to migrate two projects from a patched
5.7.1 installation to 5.7.6.

These are the relevant settings in catalog.cfg:

Variable MV_HTTP_CHARSET UTF-8
DatabaseDefault PG_ENABLE_UTF8 1

In etc/log_transaction this code creates the orderline entries:

[import table=orderline type=LINE continue=NOTES]
...
[/import]

This fails if items carry UTF-8 characters in their name/description,
e.g.:

import into orderline failed: DBD::Pg::st execute failed: ERROR:  invalid byte sequence for encoding "UTF8": 0xf1612028

Similar code using [query] instead of import works.

This was discussed on IRC yesterday, and we found that the following
two code parts probably need to examined in order to fix this bug:

Vend::Data, import_text, 303ff (writing temporary file)

if($options->{file}) {
	$fn = $options->{file};
	Vend::File::allowed_file($fn)
		or die ::errmsg("No absolute file names like '%s' allowed.\n", $fn);
}
else {
	Vend::Util::writefile($fn, $text)
		or die ("Cannot write temporary import file $fn: $!\n");
}

Vend::Table::Common, 1637ff (reading temporary file)

sub new_filehandle {
	my $fh = shift;
	binmode($fh, ":utf8") if $::Variable->{MV_UTF8};
	return $fh;
}

As a side note, avoiding the temporary file altogether in this case would be
a good idea as well :-).

Regards
         Racke

-- 
LinuXia Systems => http://www.linuxia.de/
Expert Interchange Consulting and System Administration
ICDEVGROUP => http://www.icdevgroup.org/
Interchange Development Team