Bug 495735 - mergerepos results in traceback from yum.packages
mergerepos results in traceback from yum.packages
Status: CLOSED CURRENTRELEASE
Product: Fedora
Classification: Fedora
Component: yum (Show other bugs)
10
All Linux
low Severity medium
: ---
: ---
Assigned To: Seth Vidal
Fedora Extras Quality Assurance
: Reopened
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2009-04-14 10:45 EDT by Jeroen van Meeuwen
Modified: 2014-01-21 18:08 EST (History)
7 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2009-10-12 17:50:45 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
Full mergerepos.log (68 bytes, text/plain)
2009-04-14 10:45 EDT, Jeroen van Meeuwen
no flags Details

  None (edit)
Description Jeroen van Meeuwen 2009-04-14 10:45:04 EDT
Created attachment 339506 [details]
Full mergerepos.log

Description of problem:

Not sure if this is the right component since it's caused by mergerepos, though the traceback is coming from yum

Version-Release number of selected component (if applicable):

koji-builder-1.3.1-1.fc10.noarch
yum-3.2.21-2.fc10.noarch

How reproducible:

Add dist-f7 repos for the release and updates

Steps to Reproduce:
1. koji add-tag dist-f7-updates
2. koji add-external-repo -t dist-f7-updates dist-f7-release http://www.kanarip.com/archive/fedora/releases/7/Everything/\$arch/os/
3. koji add-external-repo -t dist-f7-updates dist-f7-updates http://www.kanarip.com/archive/fedora/updates/7/\$arch/

Let koji run mergerepos on these.
  
Actual results:

Traceback (most recent call last):
  File "/usr/libexec/kojid/mergerepos", line 241, in <module>
    main(sys.argv[1:])
  File "/usr/libexec/kojid/mergerepos", line 236, in main
    merge.write_metadata()
  File "/usr/libexec/kojid/mergerepos", line 216, in write_metadata
    mdgen.doPkgMetadata()
  File "/usr/lib/python2.5/site-packages/createrepo/__init__.py", line 332, in doPkgMetadata
    self.writeMetadataDocs(packages)
  File "/usr/lib/python2.5/site-packages/createrepo/__init__.py", line 472, in writeMetadataDocs
    self.primaryfile.write(po.xml_dump_primary_metadata())
  File "/usr/lib/python2.5/site-packages/yum/packages.py", line 943, in xml_dump_primary_metadata
    msg += misc.to_unicode(self._dump_format_items())
  File "/usr/lib/python2.5/site-packages/yum/packages.py", line 809, in _dump_format_items
    msg += self._dump_pco('provides')
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 38: ordinal not in range(128)

Additional info:

The full mergerepos.log is attached
Comment 1 seth vidal 2009-04-14 11:02:56 EDT
mergerepos and this version of koji should most likely require yum 3.2.22

Can you upgrade your version of yum and retest this?
Comment 2 seth vidal 2009-05-11 10:51:26 EDT
reopen this if you can provide the info in question.

thanks
Comment 3 Jeroen van Meeuwen 2009-05-13 11:10:04 EDT
Sorry for the delayed response. It seems yum-3.2.22 is not available for Fedora 10, is that correct?
Comment 4 seth vidal 2009-05-13 11:16:32 EDT
It's in koji.

http://koji.fedoraproject.org/koji/buildinfo?buildID=101778

check it out there - it'll be in updates-testing "shortly"
Comment 5 seth vidal 2009-07-10 12:11:34 EDT
any change on this? I think we fixed it up - but I verification is nice.
Comment 6 Jeroen van Meeuwen 2009-07-11 07:37:11 EDT
I've upgraded to F-11 meanwhile, not seeing the issue there. Maybe one of the people in CC: can confirm the issue is resolved, but for now I'm closing this bug.
Comment 7 Jeroen van Meeuwen 2009-07-11 11:47:39 EDT
It occurred on F-11 now, too:

koji-builder-1.3.1-2.fc11.noarch
yum-3.2.23-3.fc11.noarch

==

Traceback (most recent call last):
  File "/usr/libexec/kojid/mergerepos", line 241, in <module>
    main(sys.argv[1:])
  File "/usr/libexec/kojid/mergerepos", line 236, in main
    merge.write_metadata()
  File "/usr/libexec/kojid/mergerepos", line 216, in write_metadata
    mdgen.doPkgMetadata()
  File "/usr/lib/python2.6/site-packages/createrepo/__init__.py", line 364, in doPkgMetadata
    self.writeMetadataDocs(packages)
  File "/usr/lib/python2.6/site-packages/createrepo/__init__.py", line 527, in writeMetadataDocs
    self.primaryfile.write(po.xml_dump_primary_metadata())
  File "/usr/lib/python2.6/site-packages/yum/packages.py", line 1015, in xml_dump_primary_metadata
    msg += misc.to_unicode(self._dump_format_items())
  File "/usr/lib/python2.6/site-packages/yum/packages.py", line 894, in _dump_format_items
    msg += self._dump_pco('provides')
  File "/usr/lib/python2.6/site-packages/yum/packages.py", line 919, in _dump_pco
    msg += pcostring
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe3 in position 32: ordinal not in range(128)
Comment 8 Jeroen van Meeuwen 2009-07-11 12:52:55 EDT
It seems i386 fails on aspell-ca all the time, whereas x86_64 fails on rsyslog-mysql.
Comment 9 Jeroen van Meeuwen 2009-07-11 13:51:31 EDT
Digging a little further on x86_64, I added a statement on line 919 of /usr/lib/python2.6/site-packages/yum/packages.py:

             pcostring += "/>\n"
+            print "%r" % pcostring
             msg += pcostring

The following is the result:

==
'      <rpm:entry name="libcamel-1.2.so.14()(64bit)"/>\n'
u'      <rpm:entry name="evolution-webcal" flags="EQ" epoch="0" ver="2.26.1"/>\n'
u'      <rpm:entry name="evolution-webcal" flags="LT" epoch="0" ver="2.24.0"/>\n'
1591/13929 - evolution-data-server-2.26.1-1.fc11.x86_64
u'      <rpm:entry name="hunspell-st" flags="EQ" epoch="0" ver="0.20060123" rel="2.fc11"/>\n'
1592/13929 - hunspell-st-0.20060123-2.fc11.noarch
u'      <rpm:entry name="rsyslog-mysql(x86-64)" flags="EQ" epoch="0" ver="3.21.11" rel="1.fc11"/>\n'
u'      <rpm:entry name="rsyslog-mysql" flags="EQ" epoch="0" ver="3.21.11" rel="1.fc11"/>\n'
'      <rpm:entry name="ommysql.so()(64bit)"/>\n'
1593/13929 - rsyslog-mysql-3.21.11-1.fc11.x86_64
u'      <rpm:entry name="ipa-pgothic-fonts" flags="EQ" epoch="0" ver="003.01" rel="2.fc11"/>\n'
'      <rpm:entry name="font(ipap\xe3\x82\xb4\xe3\x82\xb7\xe3\x83\x83\xe3\x82\xaf)"/>\n'
Traceback (most recent call last):
  File "/usr/libexec/kojid/mergerepos", line 241, in <module>
    main(sys.argv[1:])
  File "/usr/libexec/kojid/mergerepos", line 236, in main
    merge.write_metadata()
  File "/usr/libexec/kojid/mergerepos", line 216, in write_metadata
    mdgen.doPkgMetadata()
  File "/usr/lib/python2.6/site-packages/createrepo/__init__.py", line 364, in doPkgMetadata
    self.writeMetadataDocs(packages)
  File "/usr/lib/python2.6/site-packages/createrepo/__init__.py", line 527, in writeMetadataDocs
    self.primaryfile.write(po.xml_dump_primary_metadata())
  File "/usr/lib/python2.6/site-packages/yum/packages.py", line 1016, in xml_dump_primary_metadata
    msg += misc.to_unicode(self._dump_format_items())
  File "/usr/lib/python2.6/site-packages/yum/packages.py", line 894, in _dump_format_items
    msg += self._dump_pco('provides')
  File "/usr/lib/python2.6/site-packages/yum/packages.py", line 920, in _dump_pco
    msg += pcostring
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe3 in position 32: ordinal not in range(128)
==
Comment 10 Jeroen van Meeuwen 2009-08-30 16:29:18 EDT
The packages I have to block in order to be able to generate a full merged repo are:

- apanov-edrip-fonts
- apanov-heuristica-fonts
- baekmuk-ttf-fonts
- hanazono-fonts
- ipa-mincho-fonts
- ipa-pgothic-fonts
- ipa-pmincho-fonts
- sazanami-fonts
- vlgothic-fonts
- wqy-zenhei-fonts
- ipa-gothic-fonts

Apparently, something in RPM has a non-utf8/non-unicode lang(some-japanese-characters) foo which causes yum.packages._dump_pco() to fail.
Comment 11 Nicolas Mailhot 2009-09-05 13:14:32 EDT
(In reply to comment #10)
 
> Apparently, something in RPM has a non-utf8/non-unicode
> lang(some-japanese-characters) foo which causes yum.packages._dump_pco() to
> fail.

We now export the font names font files declared in rpm metadata for font autoinstallation to work. They can be complex (non ASCII) Unicode.

Though IIRC fc-query exports valid UTF-8, something else must be corrupting this data later
Comment 12 Josh Boyer 2009-09-18 09:34:19 EDT
I've hit this same issuing running kojira/koji-builders using F11.  The newRepo task trying to run is for dist-f12, using a local mirror of yesterday's rawhide.  I had to block the following packages:

un-core-fonts
wqy-zenhei-fonts
hanazono-fonts
vlgothic-fonts
sazanami-fonts
ipa-mincho-fonts
ipa-pgothic-fonts
ipa-pmincho-fonts
ipa-gothic-fonts
baekmuk-ttf-fonts
apanov-edrip-fonts
apanov-heuristica-fonts
Comment 13 Mike Bonnet 2009-09-18 10:01:12 EDT
These two patches (which have been applied to yum upstream) should fix the problems with unicode Provides: causing mergerepos to fail:

http://lists.baseurl.org/pipermail/yum-devel/2009-August/005714.html

http://lists.baseurl.org/pipermail/yum-devel/2009-August/005719.html
Comment 14 Josh Boyer 2009-09-18 12:12:36 EDT
(In reply to comment #13)
> These two patches (which have been applied to yum upstream) should fix the
> problems with unicode Provides: causing mergerepos to fail:
> 
> http://lists.baseurl.org/pipermail/yum-devel/2009-August/005714.html
> 
> http://lists.baseurl.org/pipermail/yum-devel/2009-August/005719.html  

I applied the second listed here locally and it does indeed seem to clear up the mergerepo failures I was seeing.  I did not need to block packages any longer.
Comment 15 Josh Boyer 2009-09-18 13:55:22 EDT
(In reply to comment #14)
> (In reply to comment #13)
> > These two patches (which have been applied to yum upstream) should fix the
> > problems with unicode Provides: causing mergerepos to fail:
> > 
> > http://lists.baseurl.org/pipermail/yum-devel/2009-August/005714.html
> > 
> > http://lists.baseurl.org/pipermail/yum-devel/2009-August/005719.html  
> 
> I applied the second listed here locally and it does indeed seem to clear up
> the mergerepo failures I was seeing.  I did not need to block packages any
> longer.  

Correcting myself, both patches are needed.  I ran into other issues with a later newRepo task.  The last line in the first patch should be ignored though.
Comment 16 seth vidal 2009-10-12 17:50:45 EDT
both patches are in yum in f11 and f12/rawhide

Note You need to log in before you can comment on or make changes to this bug.