Bug 658296

Seems to be bound to PostgreSQL, as Spacewalk 1.2 with Oracle XE works.

The file path in the rpm is not valid UTF-8:

# rpm -qlp /tmp/aspell-is-0.51.1-2.2.2.x86_64.rpm | grep slenska.alias | od -c
warning: /tmp/aspell-is-0.51.1-2.2.2.x86_64.rpm: Header V3 DSA signature: NOKEY, key ID 37017186
0000000   /   u   s   r   /   l   i   b   6   4   /   a   s   p   e   l
0000020   l   -   0   .   6   0   / 355   s   l   e   n   s   k   a   .
0000040   a   l   i   a   s  \n
0000046

# rpm -qlp aspell-is-0.51.1-2.2.2.x86_64.rpm | iconv -f iso-8859-1 -t utf8
warning: aspell-is-0.51.1-2.2.2.x86_64.rpm: Header V3 DSA signature: NOKEY, key ID e8562897
/usr/lib64/aspell-0.60/icelandic.alias
/usr/lib64/aspell-0.60/is.dat
/usr/lib64/aspell-0.60/is.multi
/usr/lib64/aspell-0.60/is.rws
/usr/lib64/aspell-0.60/is_phonet.dat
/usr/lib64/aspell-0.60/íslenska.alias

But create repo seems to handle this gracefully, by taking the string as being in ISO-8859-1 and converting to UTF-8:

# createrepo . ; zcat repodata/filelists.xml.gz  | xmllint -format -
1/1 - aspell-is-0.51.1-2.2.2.x86_64.rpm                                         
Saving Primary metadata
Saving file lists metadata
Saving other metadata
<?xml version="1.0" encoding="UTF-8"?>
<filelists xmlns="http://linux.duke.edu/metadata/filelists" packages="1">
  <package pkgid="e2952ace8a45a4c5c33219c7d13d5dcc205aa7be" name="aspell-is" arch="x86_64">
    <version epoch="50" ver="0.51.1" rel="2.2.2"/>
    <file>/usr/lib64/aspell-0.60/icelandic.alias</file>
    <file>/usr/lib64/aspell-0.60/is.dat</file>
    <file>/usr/lib64/aspell-0.60/is.multi</file>
    <file>/usr/lib64/aspell-0.60/is.rws</file>
    <file>/usr/lib64/aspell-0.60/is_phonet.dat</file>
    <file>/usr/lib64/aspell-0.60/íslenska.alias</file>
  </package>
</filelists>

The same problem (which is quite expected) when the rpm is merely rhnpushed:

# rhnpush --server localhost -c bz658296 aspell-is-0.51.1-2.2.2.x86_64.rpm

The traceback is then

Exception Handler Information
Traceback (most recent call last):
  File "/usr/lib/python2.4/site-packages/spacewalk/server/apacheUploadServer.py", line 97, in _wrapper
    ret = function(req)
  File "/usr/share/rhn/upload_server/handlers/package_push/package_push.py", line 134, in handler
    relative_path=self.rel_package_path, org_id=self.org_id)
  File "/usr/lib/python2.4/site-packages/spacewalk/server/rhnPackageUpload.py", line 164, in push_package
    importer.run()
  File "/usr/lib/python2.4/site-packages/spacewalk/server/importlib/importLib.py", line 619, in run
    self.fix()
  File "/usr/lib/python2.4/site-packages/spacewalk/server/importlib/packageImport.py", line 269, in fix
    self.backend.processCapabilities(self.capabilities)
  File "/usr/lib/python2.4/site-packages/spacewalk/server/importlib/backend.py", line 100, in processCapabilities
    nullStatement.execute(name=name)
  File "/usr/lib/python2.4/site-packages/spacewalk/server/rhnSQL/sql_base.py", line 163, in execute
    return apply(self._execute_wrapper, (self._execute, ) + p, kw)
  File "/usr/lib/python2.4/site-packages/spacewalk/server/rhnSQL/driver_postgresql.py", line 263, in _execute_wrapper
    retval = apply(function, p, kw)
  File "/usr/lib/python2.4/site-packages/spacewalk/server/rhnSQL/sql_base.py", line 217, in _execute
    return self._execute_(args, kwargs)
  File "/usr/lib/python2.4/site-packages/spacewalk/server/rhnSQL/driver_postgresql.py", line 276, in _execute_
    self._real_cursor.execute(self.sql, params)
DataError: invalid byte sequence for encoding "UTF8": 0xed736c
HINT:  This error can also happen if the byte sequence does not match the encoding expected by the server, which is controlled by "client_encoding".

We might consider doing the "check if it's UTF-8, convert from ISO-8859-1 to UTF-8 if it's not" in our backend code as well.

I think we already do it for changelogs ...

We did not have time for this one during Spacewalk 1.4 time frame. Mass moving to Spacewalk 1.5.

We did not have time for this one during Spacewalk 1.4 time frame. Mass moving to Spacewalk 1.5.

Aligning under space16.

This is partially correctly in spacewalk 1.5:
If the file path is incorrectly encoded, it will be corrected (for exemple for aspell-is).
But the problem remains for the copyright field, which is not corrected.
In file /usr/lib/python2.4/site-packages/spacewalk/server/importlib/packageImport.py, line 218, I've donne this modification:

    218         # Change copyright to license
    219         # XXX
    220         license_fixed = self._fix_encoding(package['license'])
    221         package['copyright'] = license_fixed
    222         #package['copyright'] = package['license']

Thank you, I have applied the copyright encoding fix to Spacewalk master as 388fe1a11160090fa5c39080dfd09bbd826f7a81.

Together with previous commits, this should address the issue for any fields where we've seen non-US-ASCII characters.

Spacewalk 1.6 has been released.