Bug 525379

+++ This bug was initially created as a clone of Bug #488195 +++

Description of problem:

I have setup spacewalk 0.5.4-1 nightly on a CentOS 5.2 machine, fresh install, and have got it up and running ok. I have added in two channels, CentOS 5.2 and CentOS 5.2 updates, and both have imported ok and are visible within the spacewalk UI.

I have successfully registered a CentOS 5.2 client to the spacewalk instance, and it appears correctly in the UI.

When I attempt to mark a package as an upgrade, and then go onto the box and do a yum check-update or yum update or rhn_check I get the following output (error)

Loading "rhnplugin" plugin

(process:2675): GLib-WARNING **: GError set over the top of a previous GError or uninitialized memory.
This indicates a bug in someone's code. You must ensure an error is NULL before it's set.
The overwriting error message was: Parsing primary.xml error: Entity 'lt' not defined


(process:2675): GLib-WARNING **: GError set over the top of a previous GError or uninitialized memory.
This indicates a bug in someone's code. You must ensure an error is NULL before it's set.
The overwriting error message was: Parsing primary.xml error: Entity 'gt' not defined


(process:2675): GLib-WARNING **: GError set over the top of a previous GError or uninitialized memory.
This indicates a bug in someone's code. You must ensure an error is NULL before it's set.
The overwriting error message was: Parsing primary.xml error: Entity 'lt' not defined


(process:2675): GLib-WARNING **: GError set over the top of a previous GError or uninitialized memory.
This indicates a bug in someone's code. You must ensure an error is NULL before it's set.
The overwriting error message was: Parsing primary.xml error: Entity 'gt' not defined


(process:2675): GLib-WARNING **: GError set over the top of a previous GError or uninitialized memory.
This indicates a bug in someone's code. You must ensure an error is NULL before it's set.
The overwriting error message was: Parsing primary.xml error: Entity 'amp' not defined


(process:2675): GLib-WARNING **: GError set over the top of a previous GError or uninitialized memory.
This indicates a bug in someone's code. You must ensure an error is NULL before it's set.
The overwriting error message was: Parsing primary.xml error: Entity 'lt' not defined


(process:2675): GLib-WARNING **: GError set over the top of a previous GError or uninitialized memory.
This indicates a bug in someone's code. You must ensure an error is NULL before it's set.
The overwriting error message was: Parsing primary.xml error: Entity 'gt' not defined


(process:2675): GLib-WARNING **: GError set over the top of a previous GError or uninitialized memory.
This indicates a bug in someone's code. You must ensure an error is NULL before it's set.
The overwriting error message was: Parsing primary.xml error: Entity 'lt' not defined


(process:2675): GLib-WARNING **: GError set over the top of a previous GError or uninitialized memory.
This indicates a bug in someone's code. You must ensure an error is NULL before it's set.
The overwriting error message was: Parsing primary.xml error: Entity 'gt' not defined


(process:2675): GLib-WARNING **: GError set over the top of a previous GError or uninitialized memory.
This indicates a bug in someone's code. You must ensure an error is NULL before it's set.
The overwriting error message was: Parsing primary.xml error: Entity 'amp' not defined


(process:2675): GLib-WARNING **: GError set over the top of a previous GError or uninitialized memory.
This indicates a bug in someone's code. You must ensure an error is NULL before it's set.
The overwriting error message was: Parsing primary.xml error: Entity 'amp' not defined


(process:2675): GLib-WARNING **: GError set over the top of a previous GError or uninitialized memory.
This indicates a bug in someone's code. You must ensure an error is NULL before it's set.
The overwriting error message was: Parsing primary.xml error: Entity 'amp' not defined


(process:2675): GLib-WARNING **: GError set over the top of a previous GError or uninitialized memory.
This indicates a bug in someone's code. You must ensure an error is NULL before it's set.
The overwriting error message was: Parsing primary.xml error: Entity 'amp' not defined


(process:2675): GLib-WARNING **: GError set over the top of a previous GError or uninitialized memory.
This indicates a bug in someone's code. You must ensure an error is NULL before it's set.
The overwriting error message was: Parsing primary.xml error: Entity 'amp' not defined


(process:2675): GLib-WARNING **: GError set over the top of a previous GError or uninitialized memory.
This indicates a bug in someone's code. You must ensure an error is NULL before it's set.
The overwriting error message was: Parsing primary.xml error: Entity 'amp' not defined


(process:2675): GLib-WARNING **: GError set over the top of a previous GError or uninitialized memory.
This indicates a bug in someone's code. You must ensure an error is NULL before it's set.
The overwriting error message was: Parsing primary.xml error: Entity 'amp' not defined


(process:2675): GLib-WARNING **: GError set over the top of a previous GError or uninitialized memory.
This indicates a bug in someone's code. You must ensure an error is NULL before it's set.
The overwriting error message was: Parsing primary.xml error: Entity 'lt' not defined


(process:2675): GLib-WARNING **: GError set over the top of a previous GError or uninitialized memory.
This indicates a bug in someone's code. You must ensure an error is NULL before it's set.
The overwriting error message was: Parsing primary.xml error: Entity 'gt' not defined


(process:2675): GLib-WARNING **: GError set over the top of a previous GError or uninitialized memory.
This indicates a bug in someone's code. You must ensure an error is NULL before it's set.
The overwriting error message was: Parsing primary.xml error: Entity 'gt' not defined


(process:2675): GLib-WARNING **: GError set over the top of a previous GError or uninitialized memory.
This indicates a bug in someone's code. You must ensure an error is NULL before it's set.
The overwriting error message was: Parsing primary.xml error: Entity 'lt' not defined


(process:2675): GLib-WARNING **: GError set over the top of a previous GError or uninitialized memory.
This indicates a bug in someone's code. You must ensure an error is NULL before it's set.
The overwriting error message was: Parsing primary.xml error: Entity 'gt' not defined


(process:2675): GLib-WARNING **: GError set over the top of a previous GError or uninitialized memory.
This indicates a bug in someone's code. You must ensure an error is NULL before it's set.
The overwriting error message was: Parsing primary.xml error: Entity 'lt' not defined


(process:2675): GLib-WARNING **: GError set over the top of a previous GError or uninitialized memory.
This indicates a bug in someone's code. You must ensure an error is NULL before it's set.
The overwriting error message was: Parsing primary.xml error: Entity 'gt' not defined


(process:2675): GLib-WARNING **: GError set over the top of a previous GError or uninitialized memory.
This indicates a bug in someone's code. You must ensure an error is NULL before it's set.
The overwriting error message was: Parsing primary.xml error: Entity 'gt' not defined


(process:2675): GLib-WARNING **: GError set over the top of a previous GError or uninitialized memory.
This indicates a bug in someone's code. You must ensure an error is NULL before it's set.
The overwriting error message was: Parsing primary.xml error: Entity 'gt' not defined


(process:2675): GLib-WARNING **: GError set over the top of a previous GError or uninitialized memory.
This indicates a bug in someone's code. You must ensure an error is NULL before it's set.
The overwriting error message was: Parsing primary.xml error: Entity 'lt' not defined


(process:2675): GLib-WARNING **: GError set over the top of a previous GError or uninitialized memory.
This indicates a bug in someone's code. You must ensure an error is NULL before it's set.
The overwriting error message was: Parsing primary.xml error: Entity 'gt' not defined


(process:2675): GLib-WARNING **: GError set over the top of a previous GError or uninitialized memory.
This indicates a bug in someone's code. You must ensure an error is NULL before it's set.
The overwriting error message was: Parsing primary.xml error: Entity 'amp' not defined


(process:2675): GLib-WARNING **: GError set over the top of a previous GError or uninitialized memory.
This indicates a bug in someone's code. You must ensure an error is NULL before it's set.
The overwriting error message was: Parsing primary.xml error: Entity 'amp' not defined

Traceback (most recent call last):
  File "/usr/bin/yum", line 29, in ?
    yummain.main(sys.argv[1:])
  File "/usr/share/yum-cli/yummain.py", line 105, in main
    result, resultmsgs = base.doCommands()
  File "/usr/share/yum-cli/cli.py", line 289, in doCommands
    self._getTs()
  File "/usr/lib/python2.4/site-packages/yum/depsolve.py", line 85, in _getTs
    self._getTsInfo()
  File "/usr/lib/python2.4/site-packages/yum/depsolve.py", line 91, in _getTsInfo
    self._tsInfo.setDatabases(self.rpmdb, self.pkgSack)
  File "/usr/lib/python2.4/site-packages/yum/__init__.py", line 537, in <lambda>
    pkgSack = property(fget=lambda self: self._getSacks(),
  File "/usr/lib/python2.4/site-packages/yum/__init__.py", line 392, in _getSacks
    self.repos.populateSack(which=repos)
  File "/usr/lib/python2.4/site-packages/yum/repos.py", line 242, in populateSack
    sack.populate(repo, mdtype, callback, cacheonly)
  File "/usr/lib/python2.4/site-packages/yum/yumRepo.py", line 168, in populate
    dobj = repo_cache_function(xml, csum)
  File "/usr/lib/python2.4/site-packages/sqlitecachec.py", line 42, in getPrimary
    self.repoid))
TypeError: Parsing primary.xml error: Input is not proper UTF-8, indicate encoding !
Bytes: 0xAE 0x20 0x6B 0x65

Version-Release number of selected component (if applicable):
spacewalk 0.5.4-1 nightly from the repo http://miroslav.suchy.cz/spacewalk/nightly-candidate/i386/os/

How reproducible:
Tried it on a few different machines, flattened and re-installed a few times, every time is the same issue

Steps to Reproduce:
1. Install spacewalk
2. create centos channel and centos updates channel (as a child), and upload all packages from these channels into spacewalk
3. register a machine with spacewalk, make sure it is subscribed to both channels
4. mark a package to be upgraded within spacewalk
5. do a yum check-update on the client machine
  
Actual results:
Error detailed above

Expected results:
yum will list the package I have marked to upgrade as an update, and then rhn_check will install the package correctly

Additional info:

--- Additional comment from graeme.r.gillies on 2009-03-09 00:29:25 EDT ---

Well after much investigation into the issue it seems that the registed symbol (unicode character 00AE) is getting put into the oracle database correctly, but then the taskomatic class which writes out the primary.xml.gz file onto disk on the spacewalk server is not properly encoding it into valid xml, it's just dumping the character out directly (which is not valid xml) and then no decent xml parser (including yum) can parse this correctly.

I managed to do a very hacky workaround due to my very limited knowledge of the code, by modifying the source for com.redhat.rhn.taskomatic.task.repomd.RepomdWriter and changing the line at the top of the file

        CONTROL_CHARS = "\u0000\u0001\u0002\u0003\u0004\u0005\u0006\u0007\u0008" +
                 "\u000B\u000C\u000E\u000F\u0010\u0011\u0012\u0013\u0014\u0015" +
                 "\u0016\u0017\u0018\u0019\u001A\u001B\u001C\u001D\u001E\u001F";

to

CONTROL_CHARS = "\u0000\u0001\u0002\u0003\u0004\u0005\u0006\u0007\u0008" +
                 "\u000B\u000C\u000E\u000F\u0010\u0011\u0012\u0013\u0014\u0015" +
                 "\u0016\u0017\u0018\u0019\u001A\u001B\u001C\u001D\u001E\u001F\u00AE";

So that the registered symbol will get automatically replaced (with an empty character) and thus avoid the problem.

I am assuming there is a better solution than this. Someone more familiar with the code and java coding in general should be able to fix this correctly quite quickly?

--- Additional comment from bbeaudoin on 2009-04-09 13:00:29 EDT ---

Getting the same parse error.  0xAE 0x20 0x6B 0x65 corresponds to "® ke" using the Latin-1 character set and I've tracked this specific issue to package descriptions starting with "Security-enhanced Linux is a feature of the Linux® kernel".  Other registered trademarks such as "UNIX®" and "Type Enforcement®" can be found in package descriptions.

I'm attempting to find a solution for this now; however, patching RepomdWriter hasn't solved this with my existing channels (these were working prior to 0.5).

--- Additional comment from bbeaudoin on 2009-04-09 15:55:54 EDT ---

Created an attachment (id=338988)
Patch implementing Graeme's work-around

Work-around described by Commenter #1 (Graeme Gillies) in patch form.  This has not solved my issue; however, I have not attempted to re-import the packages into the channel.

--- Additional comment from bbeaudoin on 2009-04-10 13:34:31 EDT ---

Before attempting to rebuild the channel I attempted to convert the character encoding using the following command in sqlplus:  "UPDATE RHNPACKAGE SET DESCRIPTION = CONVERT(DESCRIPTION, 'UTF8', 'WE8ISO8859P1')".  In theory this should have worked; however, the repository metadata wasn't being updated.  I removed a package in an attempt to "force" spacewalk to rebuild primary.xml, but even after executing "yum clean all" (which also clears the dbcache and metadata) the client was still receiving the same counts:

    peer1-i386-server-5                      |  871 B     00:00     
    primary.xml.gz                           | 607 kB     00:00     
    peer1-i386:                                    36/2279
    [...]
    TypeError: Parsing primary.xml error: Input is not proper UTF-8,
    indicate encoding !
    Bytes: 0xAE 0x20 0x6B 0x65

The documentation does not appear to indicate how to trigger or force a rebuild of the repository metadata.  A decision was made to delete all of the packages and re-import them into this channel to test the work-around.

Yesterday I applied the patch applied against the package spacewalk-java-0.5.45; when re-importing the packages it appears the descriptions still had the Latin-1 registered trademark character (I realize this may be related to the Perl .pxt files not supporting UTF-8):

    SQL> select description from rhnpackage where id=14416;

    DESCRIPTION
    ----------------------------------------------------------------------------
    The Common UNIX Printing System provides a portable printing layer for
    UNIX? operating systems. It has been developed by Easy Software Products
    to promote a standard printing solution for all UNIX vendors and users.
    CUPS provides the System V and Berkeley command-line interfaces.


    SQL> select CONVERT(DESCRIPTION, 'UTF8', 'WE8ISO8859P1') from rhnpackage
    where id=14416;

    CONVERT(DESCRIPTION,'UTF8','WE8ISO8859P1')
    ----------------------------------------------------------------------------
    The Common UNIX Printing System provides a portable printing layer for
    UNIX® operating systems. It has been developed by Easy Software Products
    to promote a standard printing solution for all UNIX vendors and users.
    CUPS provides the System V and Berkeley command-line interfaces.


    SQL> 

If someone can lend a hand with this issue it would be very appreciated.  If someone can tell us how to rebuild the repository metadata in spacewalk so we could changes more quickly that would also be of great assistance.

--- Additional comment from pkilambi on 2009-04-10 13:47:34 EDT ---

Brian:

Just remove the existing cache for your channel from /var/cache/rhn/repodata/<channel-label> and re run yum from your client. This will trigger the cache if its missing in the cache dir. Initially client will receive a repomd.xml not found error. But if you wait a few the cache should be regenerated on the server. If you want to monitor, you can run watch 'ls -l /var/cache/rhn/repodata/channel-label' and it should show you the progress.

~ Prad

--- Additional comment from bbeaudoin on 2009-04-10 16:08:07 EDT ---

Prad, thank for the solution to the channel rebuild issue.  This has helped me confirm the patch based on Graeme's workaround is effective.

With the patch applied to RepomdWriter.java in the package spacewalk-java-0.5.45, 0xAE ("®", registered trademark) is mapped to a valid ASCII/UTF-8 character (0x20, a space).  This is not a long-term solution but works in the short term.  Hopefully one of the Spacewalk developers can provide an appropriate long-term fix.

--- Additional comment from bbeaudoin on 2009-04-14 09:20:16 EDT ---

This issue appears to have been fixed in spacewalk-java-0.5.46-1.el5.

--- Additional comment from bbeaudoin on 2009-04-15 10:22:24 EDT ---

Correction: the issue still exists in spacewalk-java-0.5.46-1.el5.

--- Additional comment from bbeaudoin on 2009-05-05 11:59:51 EDT ---

The issue is also occuring with the Spanish letter ñ ("eñe"), 0xF1 in the Latin-1 character set.  Any chance of getting this internationalization issue fixed in the near term?

--- Additional comment from bbeaudoin on 2009-06-23 15:24:14 EDT ---

Updated to the latest nightly build and the issue still appears to exist in Spacewalk 0.6.  Is there anything we can do at all to help the developers isolate the cause of this issue and get it corrected?

--- Additional comment from scott.ryan on 2009-07-14 06:10:23 EDT ---

I am using : CentOS-5.3 
# rpm -qa spacewalk
spacewalk-0.5.4-1.el5
I am too running into this issue. The workaround did not work for me. I applied the patch (was using the JavaonEdit - https://fedorahosted.org/spacewalk/wiki/JavaEditOnPlace to update the code) and removed the metadata for the channel, but still I run into the same error. Is there an updated RPM with the patch included somewhere that can be used?

--- Additional comment from bbeaudoin on 2009-08-10 12:09:51 EDT ---

Issue still exists in Spacewalk 0.6 released August 7, 2009.  As before, clients are unable to parse XML generated by Spacewalk.  The priority of this bug should be upgraded as this makes Spacewalk repositories unusable.

Here's a partial run showing the error:

    [root@mail ~]# yum check-update --enableplugin=rhnplugin
    Loaded plugins: fastestmirror, rhnplugin, security
    Loading mirror speeds from cached hostfile
    * epel: www.gtlib.gatech.edu
    * main-extras: yum01mia.dedicatedhosting.com
    * main-base: yum01mia.dedicatedhosting.com
    * main-updates: yum01mia.dedicatedhosting.com
    peer1-i386-server-5                                      |  871 B     00:00
    primary.xml.gz                                           | 723 kB     00:01
    peer1-i386: #############################################      2266/2512
    (process:10277): GLib-WARNING **: GError set over the top of a previous GError or uninitialized memory.
    This indicates a bug in someone's code. You must ensure an error is NULL before it's set.
    The overwriting error message was: Parsing primary.xml error: Entity 'amp' not defined

  [...]
    TypeError: Parsing primary.xml error: Input is not proper UTF-8, indicate encoding !
    Bytes: 0xAE 0x20 0x6F 0x70

Again the 0xAE character corresponds to the Latin-1 ® symbol and is invalid in UTF-8.  The previous work-around of mapping that character to a space (0x20) no longer works as RPM packagers are now using other Latin-1 characters in RPM package descriptions.

Please let us know how we may assist with troubleshooting of this issue so we can get this resolved.

--- Additional comment from pkilambi on 2009-08-10 12:49:00 EDT ---

I tried reproducing with rhel/fedora content and was unsuccessful. Having ® symbol is not something special. We have many packages including kernel i believe with such symbols. So I'm not entirely sure if it has anything to do with it. 

What version of yum are you running on the client? did you try updating to the latest  3.2.23 or newer I believe.

--- Additional comment from bbeaudoin on 2009-08-10 13:32:27 EDT ---

The version of YUM is 3.2.19, the same version that comes with Red Hat Enterprise Linux 5 Server and CentOS 5.  It doesn't appear to be a YUM issue, it appears to be an XML issue as shown by xmllint:

    [root@spacewalk peer1-i386-server-5]# pwd
    /var/cache/rhn/repodata/peer1-i386-server-5
    [root@spacewalk peer1-i386-server-5]# zcat primary.xml.gz|xmllint -
    -:12488: parser error : Input is not proper UTF-8, indicate encoding !
    Bytes: 0xAE 0x20 0x6F 0x70
    UNIX? operating systems. It has been developed by Easy Software Products
        ^
    [root@spacewalk peer1-i386-server-5]# uname -a
    Linux spacewalk.peer1.net 2.6.18-128.1.6.el5PAE #1 SMP Tue Mar 24 12:39:24 EDT 2009 i686 i686 i386 GNU/Linux
    [root@spacewalk peer1-i386-server-5]# rpm -qf `which xmllint`
    libxml2-2.6.26-2.1.2.7
    [root@spacewalk peer1-i386-server-5]# 
    [root@spacewalk peer1-i386-server-5]# cat /etc/redhat-release
    Red Hat Enterprise Linux Server release 5.3 (Tikanga)

It appears to be an issue with how the XML is generated. Graeme Gillies stated in Comment #1 (https://bugzilla.redhat.com/show_bug.cgi?id=488195#c1) that the character is being stored in the database correctly but isn't being written correctly.  The work-around he suggested was initially fine but eventually the issue included other Latin-1 characters.

Because "xmllint" from the libxml2 package seems to confirm the issue is not with YUM, is there anything else we can check?

--- Additional comment from bbeaudoin on 2009-08-10 15:14:26 EDT ---

This may be a hint.  I decided to take a look at the database again and saw the following:

    SQL> SELECT DESCRIPTION FROM RHNPACKAGE
       3 WHERE DESCRIPTION LIKE '%Easy Software Products%';      

    DESCRIPTION
    --------------------------------------------------------------------------------
    The Common UNIX Printing System provides a portable printing layer for
    UNIX? operating systems. It has been developed by Easy Software Products
    to promote a standard printing solution for all UNIX vendors and users.
    CUPS provides the System V and Berkeley command-line interfaces.
    
    The Common UNIX Printing System provides a portable printing layer for
    UNIX? operating systems. It has been developed by Easy Software Products
    to promote a standard printing solution for all UNIX vendors and users.
    CUPS provides the System V and Berkeley command-line interfaces.
    The cups-libs package provides libraries used by applications to use CUPS
    natively, without needing the lp/lpr commands.

This was what I expected since the input was in the Latin-1 character set and my terminal expects UTF-8.  So I repeated the query with a CONVERT statement as follows:


    SQL> SELECT CONVERT(DESCRIPTION, 'UTF8', 'WE8ISO8859P1') FROM RHNPACKAGE
       2 WHERE DESCRIPTION LIKE '%Easy Software Products%';   

    CONVERT(DESCRIPTION,'UTF8','WE8ISO8859P1')
    --------------------------------------------------------------------------------
    The Common UNIX Printing System provides a portable printing layer for
    UNIX? operating systems. It has been developed by Easy Software Products
    to promote a standard printing solution for all UNIX vendors and users.
    CUPS provides the System V and Berkeley command-line interfaces.

    The Common UNIX Printing System provides a portable printing layer for
    UNIX® operating systems. It has been developed by Easy Software Products
    to promote a standard printing solution for all UNIX vendors and users.
    CUPS provides the System V and Berkeley command-line interfaces.
    The cups-libs package provides libraries used by applications to use CUPS
    natively, without needing the lp/lpr commands.

This raised a red flag because one character still appeared as a question mark ('?').  I repeated the query but added another CONVERT() to see what it would give me:


    SQL> select CONVERT(CONVERT(DESCRIPTION, 'UTF8', 'WE8ISO8859P1'),
       2 'UTF8', 'WE8ISO8859P1') from rhnpackage
       3 where description like '%Easy Software Products%';


    CONVERT(CONVERT(DESCRIPTION,'UTF8','WE8ISO8859P1'),'UTF8','WE8ISO8859P1')
    --------------------------------------------------------------------------------
    The Common UNIX Printing System provides a portable printing layer for
    UNIX® operating systems. It has been developed by Easy Software Products
    to promote a standard printing solution for all UNIX vendors and users.
    CUPS provides the System V and Berkeley command-line interfaces.
    
    The Common UNIX Printing System provides a portable printing layer for
    UNIX® operating systems. It has been developed by Easy Software Products
    to promote a standard printing solution for all UNIX vendors and users.
    CUPS provides the System V and Berkeley command-line interfaces.
    The cups-libs package provides libraries used by applications to use CUPS
    natively, without needing the lp/lpr commands.

What???? Is there a corrupted character in the database?  How did that creep in there and 

    SQL> update rhnpackage set description=convert(
       2 REPLACE(CONVERT(CONVERT(DESCRIPTION,'UTF8','WE8ISO8859P1'),
       3 'UTF8','WE8ISO8859P1'),'Â'),'WE8ISO8859P1','UTF8');

    13869 rows updated.

    SQL> SELECT CONVERT(DESCRIPTION, 'UTF8', 'WE8ISO8859P1') FROM RHNPACKAGE
       2 WHERE DESCRIPTION LIKE '%Easy Software Products%';

    CONVERT(DESCRIPTION,'UTF8','WE8ISO8859P1')
    --------------------------------------------------------------------------------
    The Common UNIX Printing System provides a portable printing layer for
    UNIX® operating systems. It has been developed by Easy Software Products
    to promote a standard printing solution for all UNIX vendors and users.
    CUPS provides the System V and Berkeley command-line interfaces.

    The Common UNIX Printing System provides a portable printing layer for
    UNIX® operating systems. It has been developed by Easy Software Products
    to promote a standard printing solution for all UNIX vendors and users.
    CUPS provides the System V and Berkeley command-line interfaces.
    The cups-libs package provides libraries used by applications to use CUPS
    natively, without needing the lp/lpr commands.

I removed the repository metadata after this conversion and tried again.  This time got this from xmllint:

    [root@spacewalk peer1-i386-server-5]# !zcat
    zcat primary.xml.gz|xmllint -
    -:1205: parser error : Input is not proper UTF-8, indicate encoding !
    Bytes: 0xAE 0x20 0x6B 0x65
    </summary><description>Security-enhanced Linux is a feature of the Linux? kernel
                                                                            ^

On the off-chance I went to a previous solution that I had backed out as follows:

    SQL> UPDATE RHNPACKAGE
       2 SET DESCRIPTION = CONVERT(DESCRIPTION, 'UTF8', 'WE8ISO8859P1')

    13869 rows updated.

    SQL> quit

Removed the repository metadata from /var/cache/rhn/repodata/peer1-i386-server-5 and allowed it to rebuild and YUM completed without any errors.  xmllint now completes without error:

    [root@spacewalk peer1-i386-server-5]# zcat primary.xml.gz|xmllint --noout -
    [root@spacewalk peer1-i386-server-5]# echo $?
    0

The questions are:

    1.  How was this bug introduced for Graeme Gillies, Scott Ryan, and
        myself on three separate installations?
    2.  Did this bug only appear for those who upgraded from Spacewalk 0.4
        to Spacewalk 0.5?
    3.  Will it break again the next time we push updates into the channels
        using the "rhnpush" command?
    4.  How can this bug be prevented in the future?

Since we haven't found a root cause it's possible this bug hasn't been fixed.  If anybody else out there can check their database for anything odd in terms of character encoding and update this issue with what you found it may be helpful for the developers in the event this bug is still lurking in the code somewhere.

--- Additional comment from pkilambi on 2009-08-10 15:41:32 EDT ---

Good write up. Thanks. So reading through your analysis, The bug is probably in rhnpush server side at the time of populating the rhnPackage tables. It could be that when we're getting the description info from the rpm headers, it not converting all the data to utf8. So seems to me like the right place to fix this is at the source, Which is when rhnpush is populating the db descritpion field.

Basically the patch would in /usr/share/rhn/server/packageImport.py basically for every package object in self.batch, we need to grab package['description'] and convert it to write encoding before it gets inserted into the DB.

--- Additional comment from hrubizna on 2009-08-13 08:45:49 EDT ---

Affected file is not only primary.xml.gz but also filelists.xml.gz. filelists.xml.gz contains some spanish letters ( especcialy for centos 5 base repo ). Affected characters : 0xA9 ( © ) , 0xAE ( ® ) , 0xF1 ( ń ) , 0xF8 ( ř ), 0xE5 ( ĺ ) , 0xEA ( ę ).

Using spacewalk 0.6 on Centos 5.2 x86-64. Conntent uploaded to channels using spacewalk-repo-sync command.

--- Additional comment from bbeaudoin on 2009-08-13 09:15:38 EDT ---

On clients running Red Hat Enterprise Linux 5.x or CentOS 5.x, the following packages need to be updated to the latest version before full Spacewalk functionality is restored:

    Package    Version
    ---------  ----------------------------
    python     2.4.3-24.el5_3.6 or higher
    libxml2    2.6.26-2.1.2.8 or higher

This is in addition to scrubbing the Oracle database of invalid encodings.  I'm not certain if either package contributed to the data corruption causing the invalid XML to be created.

--- Additional comment from scott.ryan on 2009-09-03 06:37:31 EDT ---

I am totally confused by this. I flattened the DB and started again. Spacewalk 0.5 on Centos5.3. I added 1 channel Centos5.3 and rhnpush'd the packages and performed NO converting. 

If i use SQLDeveloper : 

SELECT DESCRIPTION FROM RHNPACKAGE WHERE DESCRIPTION LIKE '%Easy Software Products%';

"The Common UNIX Printing System provides a portable printing layer for
UNIX® operating systems. It has been developed by Easy Software Products
to promote a standard printing solution for all UNIX vendors and users.
CUPS provides the System V and Berkeley command-line interfaces.
"
"The Common UNIX Printing System provides a portable printing layer for
UNIX® operating systems. It has been developed by Easy Software Products
to promote a standard printing solution for all UNIX vendors and users.
CUPS provides the System V and Berkeley command-line interfaces.
The cups-libs package provides libraries used by applications to use CUPS
natively, without needing the lp/lpr commands.
"
"The Common UNIX Printing System provides a portable printing layer for
UNIX® operating systems. It has been developed by Easy Software Products
to promote a standard printing solution for all UNIX vendors and users.
CUPS provides the System V and Berkeley command-line interfaces.
The cups-libs package provides libraries used by applications to use CUPS
natively, without needing the lp/lpr commands.
"
It all looks like it has been correctly inserted into the DB.

If I search for the package in the spacewalk UI:

The Common UNIX Printing System provides a portable printing layer for
UNIX® operating systems. It has been developed by Easy Software Products
to promote a standard printing solution for all UNIX vendors and users.
CUPS provides the System V and Berkeley command-line interfaces.

Again - it looks ok.

But sqlplus shows:

SQL> SELECT DESCRIPTION FROM RHNPACKAGE WHERE DESCRIPTION LIKE '%Easy Software Products%';

DESCRIPTION
--------------------------------------------------------------------------------
The Common UNIX Printing System provides a portable printing layer for
UNIX? operating systems. It has been developed by Easy Software Products
to promote a standard printing solution for all UNIX vendors and users.
CUPS provides the System V and Berkeley command-line interfaces.

The Common UNIX Printing System provides a portable printing layer for
UNIX? operating systems. It has been developed by Easy Software Products
to promote a standard printing solution for all UNIX vendors and users.
CUPS provides the System V and Berkeley command-line interfaces.
The cups-libs package provides libraries used by applications to use CUPS
natively, without needing the lp/lpr commands.

DESCRIPTION
--------------------------------------------------------------------------------

The Common UNIX Printing System provides a portable printing layer for
UNIX? operating systems. It has been developed by Easy Software Products
to promote a standard printing solution for all UNIX vendors and users.
CUPS provides the System V and Berkeley command-line interfaces.
The cups-libs package provides libraries used by applications to use CUPS
natively, without needing the lp/lpr commands.

So my assumption (from my limited knowledge) is that the chars are correctly inserted into the DB, but the client that is creating the metadata is incorrect...

--- Additional comment from bbeaudoin on 2009-09-03 08:10:31 EDT ---

From https://fedorahosted.org/spacewalk/wiki/MultibyteSpacewalk

# The Java webui pages (pages ending in .do) will accept multibyte input properly and will store it in the database correctly.
# The Java webui pages (pages ending in .do) may have display issues showing multibyte user inputted data.
# The Perl webui pages (pages ending in a .pxt) do not properly handle multibyte input. 

Latin-1 renders correctly in the webui, UTF-8 may not; however, it appears all characters are supposed to be stored in the database as UTF-8. Although Latin-1 is compatible with UTF-8 in the first 128 bytes, international characters and symbols stored in the upper 128 bytes of Latin-1 is not compatible with UTF-8.

In some cases, we are aware Spacewalk will incorrectly write Latin-1 characters to the Oracle database. Pradeep believes the "rhnpush" command line interface may not be handling the character encodings properly but this hasn't been confirmed.  Within SQLPlus it is possible to update the affected column with the proper encoding.

As an additional note, please make sure you are using the latest versions of python and libxml2 as a bug in either of these packages may also be involved.

--- Additional comment from bbeaudoin on 2009-09-03 09:05:46 EDT ---

Scott,

For additional clarification, rendering properly on the screen does not indicate the data has been stored in the expected character encoding.  Additionally, rendering improperly does not mean the data was stored improperly either.  It is necessary to know what the expected and actual character encodings are.

On the command line, I can use the following command to show my encoding preferences:

    $ echo $LANG
    en_US.UTF-8

This assumes that my local display (console or terminal window) supports both the UTF-8 encoding and font.  Similarly, a web browser sends an "Accept-Charset" client header that specifies what languages and encodings it can receive as well:

   Accept-Language    en-us,en;q=0.5
    Accept-Charset    ISO-8859-1,utf-8;q=0.7,*;q=0.7

When the server responds, it will state which encoding it used so it may be displayed properly to the end-user:

      Content-Type    text/html; charset=UTF-8

I stated previously that my terminal expects UTF-8 so when a character is rendered as "?", it is either a question mark (when it makes sense to be a question mark) or an invalid character in the UTF-8 character set.

In order to be certain characters are stored and displayed correctly, we need to know what character sets are being used when the data is being displayed.  If you are using Firefox you may use the menu option "Tools" → "Page Info" to view the language and encoding of the page. You can also use the "iconv" utility on the command line to convert from one character set to another to see how different characters are displayed as follows:

    $ echo "®" | iconv -fUTF-8 -tISO-8859-1
    ?

To see whether it really converted the character, try converting it twice (once to the improper character set and again to the proper character set)

    $ echo "®" | iconv -fUTF-8 -tISO-8859-1 | iconv -fISO-8859-1 -tUTF-8
    ®

Character encoding issue can be confusing for an end-user, especially when it is inconsistent between different applications and displays. This is also difficult for developers and administrators as the issue may not appear the same depending on how it's viewed. I hope this information is helpful to you and others who may also be having difficulty with inconsistent display information.

--- Additional comment from scott.ryan on 2009-09-03 09:17:49 EDT ---

Thank you for the explanation - it helps a lot. I admit that I am not too clued up on character encoding! I would just like to know though - what is the correct work around for this if the data is already in the DB? I have tried applying the patch as well as trying to convert the data entries, but I have never been able to get a work around for this. Thanks,

--- Additional comment from bbeaudoin on 2009-09-03 10:35:50 EDT ---

Scott,

The previous work-around and patch won't work.  This is because the UTF-8 surrogate character also corresponds to the "Â" character in the Latin-1 character set. Stripping "®" results in leaving a bare "Â" character, if it was stored correctly and stripping "Â" leaves other UTF-8 characters corresponding to Latin-1 characters without their surrogate mate.

Unfortunately this bug means not all characters are in Latin-1 or UTF-8 so using Oracle's CONVERT() function may result in further corruption. You might test using the following SELECT query in SQLPlus:

    SELECT REPLACE(CONVERT(DESCRIPTION, 'UTF8', 'WE8ISO8859P1'), 'Â')
    FROM RHNPACKAGE WHERE DESCRIPTION LIKE '%Easy Software Products%';

If you are satisfied with the results of the SELECT statement, you can update the column in all rows with the following command:

    UPDATE RHNPACKAGE SET DESCRIPTION=REPLACE(
    CONVERT(DESCRIPTION, 'UTF8', 'WE8ISO8859P1'), 'Â');

Please note it is safest to back up the database (or at least that table) before executing a command that will modify all rows. Unfortunately if there is a literal "Â" character stored in that field of the database it will be stripped (along with its surrogate) as well; therefore, I do not recommend that command on any production database.

Brian

--- Additional comment from scott.ryan on 2009-09-10 03:51:34 EDT ---

Ok - have rebuilt from scratch - clean install. Pushed the packages from Centos5.3-x64 into the channel I created. I did the update as suggested above and it appears to have fixed some of the issues xmlint was complaining about but not all. I almost have a workaround... The only thing that I now get is a problem with the ® character

SELECT DESCRIPTION FROM RHNPACKAGE WHERE description like '%Security-enhanced Linux is a feature of the Linux%'

DESCRIPTION
--------------------------------------------------------------------------------
Security-enhanced Linux is a feature of the Linux? kernel and a number

In SQL Developer it displays correctly :

Security-enhanced Linux is a feature of the Linux® kernel and a number

Do I have to run another update description for the ® character?

--- Additional comment from bbeaudoin on 2009-09-11 10:22:21 EDT ---

Probably.  Neither seeing "®" nor "?" tells us what encoding we are looking at.  Only whether or not the encoding is the appropriate one according to what was declared to the terminal or browser.

If changes aren't being committed automatically within SQLPlus you may want to do an explicit COMMIT.  

----------

On a separate but related note, the encoding issue is no longer limited to "SPACEWALK"."RHNPACKAGE"."DESCRIPTION".  Now when we are using "rhnpush" to add packages we are seeing the issue in other tables.  The "xmllint" application has been helpful in providing clues as to which tables:

    [root@spacewalk peer1-i386-server-5]# zcat other.xml.gz|xmllint -
    -:1811: parser error : Input is not proper UTF-8, indicate encoding !
    Bytes: 0xFC 0x63 0x6B 0x65
    - Improve quoting of fstype in anaconda-ks.cfg (Danen Br?cker)</changelog><chang
                                                            ^

As shown in the above example, the character 0xFC ("ü" in the Latin-1 character set) is being written to the changelog.  I searched the schema and found the string in "SPACEWALK"."RHNPACKAGECHANGELOG"."TEXT".  A select confirmed this was where the issue was.

    SQL> SELECT TEXT FROM RHNPACKAGECHANGELOG WHERE TEXT 
    LIKE '%Improve quoting of fstype in anaconda-ks.cfg%';
    
    TEXT
    ----------------------------------------------------------------------------
    - Better error handling on device creation (#142273)
    - Reset package selection to defaults if selected (#142415)
    - LVM on RAID1 fix (nasrat, #141781)
    - Add support for biosdev in driverdisk from Rez Kabir (#142738)
    - Some more SX8 fixes
    - Create /dev as a tmpfs (#141570)
    - Remove some old code
    - Improve quoting of fstype in anaconda-ks.cfg (Danen Br?cker)

Unfortunately the replacement text is larger because UTF-8 can take up to two more bytes than Latin-1.  It appears the database and/or table character set encodings may not be set properly either:

    SQL> UPDATE RHNPACKAGECHANGELOG SET TEXT=REPLACE(CONVERT(TEXT, 
    'UTF8', 'WE8ISO8859P1'), 'Â');
    UPDATE RHNPACKAGECHANGELOG SET TEXT=REPLACE(CONVERT(TEXT, 'UTF8', 
    'WE8ISO8859P1'), 'Â')
                                        *
    ERROR at line 1:
    ORA-12899: value too large for column "SPACEWALK"."RHNPACKAGECHANGELOG"."TEXT"
    (actual: 3002, maximum: 3000)

To get past this issue quickly I just expanded the size of the column so it could fit more characters.  One of the database schema maintainers should check to see if the character set is properly defined and/or increase the size of the field as appropriate.

    SQL> ALTER TABLE RHNPACKAGECHANGELOG MODIFY "TEXT" VARCHAR2(4000);
    
    Table altered.
    
    SQL> UPDATE RHNPACKAGECHANGELOG SET TEXT=REPLACE(CONVERT(TEXT, 
    'UTF8', 'WE8ISO8859P1'), 'Â');
    
    1424025 rows updated.
    
    SQL> SELECT TEXT FROM RHNPACKAGECHANGELOG WHERE TEXT 
    LIKE '%Improve quoting of fstype in anaconda-ks.cfg%';
    
    TEXT
    ----------------------------------------------------------------------------
    - Better error handling on device creation (#142273)
    - Reset package selection to defaults if selected (#142415)
    - LVM on RAID1 fix (nasrat, #141781)
    - Add support for biosdev in driverdisk from Rez Kabir (#142738)
    - Some more SX8 fixes
    - Create /dev as a tmpfs (#141570)
    - Remove some old code
    - Improve quoting of fstype in anaconda-ks.cfg (Danen Brücker)

Now I'm working on tracking down where the file listings are stored.  There's an issue with an ñ (0xF1) stored somewhere with the wrong encoding that is preventing filelists.xml.gz from being created properly.  I'll post another update once I've finished tracking that down.

    [root@spacewalk peer1-i386-server-5]# zcat filelists.xml.gz|xmllint -
    -:2: parser error : Input is not proper UTF-8, indicate encoding !
    Bytes: 0xF1 0x6F 0x6C 0x2E
    86"><version ver="0.50" rel="13.2.2" epoch="50"/><file>/usr/lib/aspell-0.60/espa
                                                                                   ^

--- Additional comment from bbeaudoin on 2009-09-11 11:56:16 EDT ---

Found the culprit for filelists.xml.gz:

    SQL> describe RHNPACKAGECAPABILITY;
     Name			   Null?    Type
     ----------------------------- -------- ----------------------------
     ID				   NOT NULL NUMBER
     NAME			   NOT NULL VARCHAR2(4000)
     VERSION				    VARCHAR2(64)
     CREATED			   NOT NULL DATE
     MODIFIED			   NOT NULL DATE
    
    SQL> SELECT NAME FROM RHNPACKAGECAPABILITY 
       2 WHERE NAME LIKE '/usr/lib/aspell-0.60%';

    NAME
    --------------------------------------------------------------------
   […]
    /usr/lib/aspell-0.60/espa?ol.alias
   […]
    /usr/lib/aspell-0.60/f?royskt.alias
   […]
    /usr/lib/aspell-0.60/?slenska.alias

   341 rows selected.

The UNDO buffer is not large enough to run an UPDATE on every row of the table so after about 5 minutes.  Since I don't expect there to be too many files named "español.alias" (or the like) I just converted filenames that started with the path:

    SQL> UPDATE RHNPACKAGECAPABILITY 
      2  SET NAME=REPLACE(CONVERT(NAME, 'UTF8', 'WE8ISO8859P1'), 'Â') 
      3  WHERE NAME LIKE '/usr/lib/aspell-0.60%';
    
    341 rows updated.
    
    SQL> SELECT NAME FROM RHNPACKAGECAPABILITY
      2  WHERE NAME LIKE '/usr/lib/aspell-0.60%';
    
    NAME
    ---------------------------------------------------------------------
   […]
    /usr/lib/aspell-0.60/español.alias
   […]
    /usr/lib/aspell-0.60/føroyskt.alias
   […]
    /usr/lib/aspell-0.60/íslenska.alias

   341 rows selected.

Note, the selection string was important.  When the "%" followed the "/" character, the file "íslenska.alias" was selected but was not updated.  Because we used a pattern for the update, this work-around only handles those specific cases.

Just a note, to be safe add "COMMIT;" after the update.  I've been relying on implicit commits but that might not work predictably and reliably for everyone.

**********

Pradeep, any new luck on figuring out why the encoding isn't being set properly when "rhnpush" is used to push new packages into a repository that was otherwise working prior to using "rhnpush"?

--- Additional comment from bbeaudoin on 2009-09-17 12:25:02 EDT ---

Modified the query to fix the filelists.xml.gz corruption:

    UPDATE RHNPACKAGECAPABILITY
     SET NAME = REPLACE(
      CONVERT(
       NAME, 'UTF8', 'WE8ISO8859P1'
      ), 'Â'
     ) WHERE NAME LIKE '/usr/lib%/aspell-0.60%'
    ;
    COMMIT;

The "aspell" package has international filenames in both /usr/lib/ and /usr/lib64/ for amd64 (x86_64) architectures.  Previous testing was done only with 32-bit.

--- Additional comment from jwrigley on 2009-09-23 10:02:03 EDT ---

I've seen this with Satellite 5.3 customers. Can you confirm the spacewalk machine's locale?

In our case, /etc/sysconfig/i18n had:

LANG="en_GB"

Changing this to:

LANG="en_GB.UTF-8"

rebooting and clearing the cache solved the problem.