Bug 782490
| Summary: | Yum metadata from EPEL with "pkgtags" showing 'Error: database disk image is malformed' | ||
|---|---|---|---|
| Product: | [Retired] Pulp | Reporter: | John Matthews <jmatthew> |
| Component: | user-experience | Assignee: | Todd Sanders <tsanders> |
| Status: | CLOSED CURRENTRELEASE | QA Contact: | Preethi Thomas <pthomas> |
| Severity: | unspecified | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 2.0.7 | CC: | jason.dobies, jortel, rbowlby83, skarmark, stefan.kirrmann, strobert |
| Target Milestone: | --- | Keywords: | Triaged |
| Target Release: | 2.0.8 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Bug Fix | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2013-04-08 15:59:42 UTC | Type: | --- |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Description
John Matthews
2012-01-17 15:31:47 UTC
I'm also getting this error for all my Pulp Clients Fedora 15-16, pulp version 254 tried on clients: yum clean all also tried it with: rm /var/lib/rpm/__db* rpm --rebuilddb But nothing helps, errors stays the same "database disk image is malformed". Resync the repo on the pulp-server also doesn't helps. Hi Stefan, Are you able to determine the specific repo that is giving you this errror? If so, would you please post the feed URL? Thank you, John Yes I know which repo it is. Sorry, I forgot to mention it in my previous post. Feed URL is http://ftp-stud.hs-esslingen.de/pub/fedora/linux/updates/16/x86_64/ What's the status of this issue? I'm also experiencing this problem with epel: [root@icinga ~]# yum search icinga Loaded plugins: fastestmirror, priorities, security Loading mirror speeds from cached hostfile 1532 packages excluded due to repository priority protections stable-epel-6-x86_64/pkgtags | 311 B 00:00 Error: database disk image is malformed [root@pulp repodata]# zgrep pkgtags ./* ./repomd.xml: <data type="pkgtags"> ./repomd.xml: <location href="repodata/f30707ebe14eeee23b337994c90a23c622649c5debe3fe3f7120aad5da847851-pkgtags.sqlite.gz"/> [root@pulp repodata]# zcat f30707ebe14eeee23b337994c90a23c622649c5debe3fe3f7120aad5da847851-pkgtags.sqlite.gz ?..??##?otablepackagetagspackagetagsCREATE TABLE packagetags ( name TEXT NOT NULL, tag TEXT NOT NULL, score INTEGER, PRIMARY KEY (name, tag) )5I#indexsqlite_autoindex_packagetags_1packagetags [root@pulp repodata]# pwd /var/lib/pulp/repos/stable/epel/6/x86_64/repodata [root@pulp repodata]# pulp-admin repo info --id=^C [root@pulp repodata]# pulp-admin repo info --id=stable-epel-6-x86_64 Id stable-epel-6-x86_64 Name stable-epel-6-x86_64 Repo URL https://domain/pulp/repos/stable/epel/6/x86_64/ Feed URL file:///var/lib/pulp//repos/daily/epel/6/x86_64 Feed Type local Content Type yum Feed Certs CA:No Cert:No Consumer Certs CA:No Cert:No Architecture noarch Sync Schedule None Packages 7795 Files 0 Distributions None Publish True Clones [] Groups None Filters [] Notes {} Preserve Metadata False Checksum Type sha256 [root@pulp repodata]# rpm -qa | grep pulp python-isodate-0.4.4-4.pulp.el6.noarch pulp-admin-1.1.11-1.el6.noarch pulp-1.1.11-1.el6.noarch pulp-common-1.1.11-1.el6.noarch pulp-client-lib-1.1.11-1.el6.noarch pulp-consumer-1.1.11-1.el6.noarch mod_wsgi-3.3-3.pulp.el6.x86_64 pulp-selinux-server-1.1.11-1.el6.noarch m2crypto-0.21.1.pulp-7.el6.x86_64 python-oauth2-1.5.170-2.pulp.el6.noarch Please let me know if any additional info is required! Thanks. addendum: [root@pulp repodata]# pulp-admin repo info --id=daily-epel-6-x86_64 Id daily-epel-6-x86_64 Name daily-epel-6-x86_64 Repo URL https://domain/pulp/repos/daily/epel/6/x86_64/ Feed URL http://mirrors.kernel.org/fedora-epel/6/x86_64 Feed Type remote Content Type yum Feed Certs CA:No Cert:No Consumer Certs CA:No Cert:No Architecture noarch Sync Schedule P1D Packages 7795 Files 0 Distributions None Publish True Clones [] Groups [u'epel'] Filters [] Notes {} Preserve Metadata False Checksum Type sha256 Was able to work around this issue by deleting and recreating daily-epel with --preserve-metadata. On a related note pulp-admin repo update should take preserve_metadata as an option. I had to delete and create...which caused issues since this repo is heavily used. :( This came up on the mailing list recently. Still is an issue with 2.0. I did the repo creates on 2.0.6 release and initial epel sync on 2.0.7 0.2.beta. if I set "enabled = 0" for my epel-el6 repo entry in /etc/yum.repos.d/pulp.repo on the consumer "yum search" works again. so sounding like something in the yum metadata epel has is making the pulp publish (or on the import side) do something odd. and and I forgot. pulp-server is RHEL 6 (64bit). ditto for the pulp consumer. both have recent OS updates (within the last week or so). If specific versions/config is needed, please let me know. Updating the bug with my testing results. I can't seem to reproduce the issue
Server
[root@preethi ~]# rpm -q pulp-server
pulp-server-2.0.7-0.4.beta.noarch
[root@preethi ~]#
[root@preethi ~]# cat /etc/redhat-release
Fedora release 17 (Beefy Miracle)
[root@preethi ~]#
Client
[root@el6-client ~]# cat /etc/redhat-release
Red Hat Enterprise Linux Server release 6.0 (Santiago)
[root@el6-client ~]#
[root@preethi ~]# pulp-admin repo list --details
Id: rh6-epel-live
Display Name: rh6-epel-live
Description: None
Content Unit Count: 10523
Notes:
Importers:
Config:
Feed URL: http://dl.fedoraproject.org/pub/epel/6/x86_64/
Id: yum_importer
Importer Type Id: yum_importer
Last Sync: 2013-01-18T15:39:49-05:00
Repo Id: rh6-epel-live
Scheduled Syncs:
Distributors:
Auto Publish: True
Config:
Http: False
Https: True
Relative URL: /pub/epel/6/x86_64/
Distributor Type Id: yum_distributor
Id: yum_distributor
Last Publish: 2013-01-18T15:41:30-05:00
Repo Id: rh6-epel-live
Scheduled Publishes:
Auto Publish: False
Config:
Http: False
Https: True
Distributor Type Id: export_distributor
Id: export_distributor
Last Publish: None
Repo Id: rh6-epel-live
Scheduled Publishes:
Id: daily-epel-6-x86_64
Display Name: daily-epel-6-x86_64
Description: None
Content Unit Count: 10523
Notes:
Importers:
Config:
Feed URL: http://mirrors.kernel.org/fedora-epel/6/x86_64
Id: yum_importer
Importer Type Id: yum_importer
Last Sync: 2013-01-18T18:21:49-05:00
Repo Id: daily-epel-6-x86_64
Scheduled Syncs:
Distributors:
Auto Publish: True
Config:
Http: False
Https: True
Relative URL: /fedora-epel/6/x86_64
Distributor Type Id: yum_distributor
Id: yum_distributor
Last Publish: 2013-01-18T18:23:15-05:00
Repo Id: daily-epel-6-x86_64
Scheduled Publishes:
Auto Publish: False
Config:
Http: False
Https: True
Distributor Type Id: export_distributor
Id: export_distributor
Last Publish: None
Repo Id: daily-epel-6-x86_64
Scheduled Publishes:
[root@preethi ~]# pulp-admin rpm consumer list --details
+----------------------------------------------------------------------+
Consumers
+----------------------------------------------------------------------+
Id: rhel6-client
Display Name: rhel6-client
Bindings:
Confirmed: rhel6_3, rh6-epel-live
Unconfirmed:
Capabilities:
Certificate: -----BEGIN CERTIFICATE-----
MIICGzCCAQMCAQIwDQYJKoZIhvcNAQEFBQAwFDESMBAGA1UEAxMJbG9jYWxob3N0
MB4XDTEzMDExODE2MDI0NVoXDTIzMDExNjE2MDI0NVowFzEVMBMGA1UEAxMMcmhl
bDYtY2xpZW50MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDPut85tCbCz491
gltxRQDzbRRBKBXi3QLXAxZuqByyv1xxNsTjzDifkB0wNzTj5leYUBzul+beZlfO
BfViyW0EUmI+SqTwA356oQMiDHtwaBePER+CzX5go4/ihL2tzOZowbmqMrOpGnJk
kDoUcv0HXDZInMlPW6qCDs16oWrO5QIDAQABMA0GCSqGSIb3DQEBBQUAA4IBAQAB
Evn09ecYZSp4SrGKfB5vtPvk94/BTRap2XTm/xdp7oYpSOw8kw5tDPDYLrCL1/7C
KaLmZ0nbZH37PkzFhKu983beldRltLdDG5Tu9TR44v9oeZvNJwquKmEdL/J0vz4Q
RRa8LLWphCz3IPYURHKpbIYRQEVYHDeuBEmNuG5uPhuMaGR4zS5+g3a1GL2MReoU
LZ3DW5WlURjL/MM7PkHbKj4H+87JO5CzDITuFfUCw5SDbyVxo1I6BoInSH9Dd8yd
AfWuGHowLFUDiu05gfWqNIa1oSch2bDa9hcaVgmnm/DakTe3lL0Ce5b3uLK5Gsn9
sekhJA5vhyXzU3jKDtIC -----END CERTIFICATE-----
Description: None
Notes:
Id: el6-client
Display Name: el6-client
Bindings:
Confirmed: rh6-epel-live, daily-epel-6-x86_64
Unconfirmed:
Capabilities:
Certificate: -----BEGIN CERTIFICATE-----
MIICGTCCAQECAQMwDQYJKoZIhvcNAQEFBQAwFDESMBAGA1UEAxMJbG9jYWxob3N0
MB4XDTEzMDEyMTE0NTM1NVoXDTIzMDExOTE0NTM1NVowFTETMBEGA1UEAxMKZWw2
LWNsaWVudDCBnzANBgkqhkiG9w0BAQEFAAOBjQAwgYkCgYEA08yDoqvdF+50E7CW
4w0ZA3UZXo18dNN7QtdFJtJ1OnmkZpfYh4qB7KflUeuSaMCy6R4PKf+YrC6UMtYl
BK+l733YhZk+tyI4KSRUKc7hJkL3hz2ZiSnWCnOUGY0vzCVoxFjVQZB8kPWmnu3F
gm2cYbm5DHFTfHbubm6oGEP7bWcCAwEAATANBgkqhkiG9w0BAQUFAAOCAQEAWnIr
Nj1TcdDiv3iNa+s5o74Dm1sn6tm5iuO3drAfxpQ3HZpyw4Nht1uJjSM9T8cpCa5z
i2FaBR66siacrtiIwoOdbgvjt/yZCuVMaHcfRFP+UDScUh6NfoF+MjpALuCrMMdo
6jRxeLGP7Gc6Cuv9RsCMvO+ockJjAiYRS72DSOfarQKWXvZt5XT6eqAeyRIm/uU8
nE4HOe61+yGGyfOqEf6xjlbmgOYrbmuo7/e74NhixiZmIDYHooXXc6YU1lA9acx0
BQdzYTNSHsrTq9hqStolWJCzLDCJtSRK3BcHUm4gCboTU1wK0k0QcAdrpPQGnFZO
PdH/q6aCA1gcZFb6uw== -----END CERTIFICATE-----
Description: None
Notes:
[root@preethi ~]#
On the client
[root@el6-client ~]# yum search zsh
Loaded plugins: rhnplugin
This system is not registered with RHN.
RHN support will be disabled.
Failed to add Pkg Tags for repository: daily-epel-6-x86_64 - file is encrypted or is not a database
Failed to add Pkg Tags for repository: rh6-epel-live - file is encrypted or is not a database
================================= Matched: zsh =================================
autojump-zsh.noarch : Autojump for zsh
gromacs-zsh.noarch : GROMACS zsh support
zsh.x86_64 : A powerful interactive shell
environment-modules.x86_64 : Provides dynamic modification of a user's
: environment
plowshare.noarch : Download and upload files from file-sharing websites
shflags.noarch : Simple handling of command-line flags in Bourne based Unix
: scripts
[root@el6-client ~]#
[root@el6-client ~]# yum repolist
Loaded plugins: rhnplugin
This system is not registered with RHN.
RHN support will be disabled.
repo id repo name status
daily-epel-6-x86_64 daily-epel-6-x86_64 8,312
pulp-v2-candidate Pulp v2 Beta Builds 36
rh6-epel-live rh6-epel-live 8,312
repolist: 16,660
[root@el6-client ~]#
any way you can test with el6 as the server? f17 has newer rpm,yum than el6. f17: rpm-4.9.1.3-7.fc17 yum-3.4.3 rl6: rpm-4.8.0-27.el6.x86_64 yum-3.2.29-30.el6.noarch okay, I have narrowed down the problem file a bit. the summary of this bug is a good hint. the epel repo has a <big long hex string>-pkgtags.xml.gz file. none of the other repos have one. it isn't a compressed xml file. it is actually a compressed basically empty sqlite3 DB. If I manually remove it from the published directory on the pulp-server and on the consumer. yum search works fine. But of course next repo sync makes the file again. So sounds like some more problem metadata fun. spoke to the yum maintainer and concluded that pkgtags are not really used and basically ignored in fedora land as of today. For this particular repo the pkgtags are empty if you load the sqlite db. So pulp for now will skip pkgtags by default, we provided a switch in yum_distributor.conf if user prefers to manually enable it. Also newer versions of createrepo and yum in el6.4 handle the database error better and skip pkgtags on clients. merged fix to pulp-2.0 and master. After applying the fix, * Just republish your repo on server # pulp-admin rpm repo publish run --repo-id=rh6-epel-live * Clean yum cache on client # yum clean all or rm -rf /var/cache/yum/* * Do yum operations as usual # yum search zsh excellent. yeah it seemed like pkgtags were not really used. I have been toying with a publish event hook to remove the pkgtags file after the publish completed since removing it manually fixed the issue. Good to hear that will be done in pulp itself soon. build: 2.0.8-0.1.beta fixed distributor validation to account for keys in yum_distributor.conf merged to pulp-2.0 and master Installed 2.0.8-0.1.beta. did a repo sync+publish. yum clean all on consumer. yum search now works as expected. thank you. build: 2.0.8-0.2.beta
[root@preethi-el6-pulp ~]# pulp-admin rpm consumer list --details
+----------------------------------------------------------------------+
Consumers
+----------------------------------------------------------------------+
Id: el6-client
Display Name: el6-client
Bindings:
Confirmed: rh6-epel-live
Unconfirmed:
Capabilities:
Certificate: -----BEGIN CERTIFICATE-----
MIICGTCCAQECAQEwDQYJKoZIhvcNAQEFBQAwFDESMBAGA1UEAxMJbG9jYWxob3N0
MB4XDTEzMDEzMTEzMDEwN1oXDTIzMDEyOTEzMDEwN1owFTETMBEGA1UEAxMKZWw2
LWNsaWVudDCBnzANBgkqhkiG9w0BAQEFAAOBjQAwgYkCgYEAqP+ony3jSKyYJOrD
EjDjF+FzOnmHLQXopd/qm4aaGBsjfnOImItFJN5dmO9tcJmsfIY+GglzoNCDlzvC
/UuQ0pXALIKY4VmdY5Z+JsBg2iNmpdISZ3/tDNW9kP9IgZoeusQmQqc40bhF/OqX
akf4S8jhOf5xLd5hnpvWJsF6AIUCAwEAATANBgkqhkiG9w0BAQUFAAOCAQEAYwlg
jUU1Dj4iAahtFwifT66d0ZaPVjQq5xGa6pDq9ocJVJp5I+hOdtZ2dYj4Vb+SzjiU
RyKdBI0q3OOeV8sJjflhN6TEPHrABxuJ2kOq4BDTyam+hG38g4+0FXFMSkRtI4uy
AuVHS4rL3YjWmdoP2kK8UnOsTEYi6PL9x2ts5h8rs3JZ3GsGtbydgT1tUNCfkgsX
+ZwzoKSN/8Z7We6FsxsDj0Kf7mZYlE+E6EvLsd+MQl/lxjqwuJOWdJrbN1TirQD0
NcbK/dJ/wGHSARExSsZSL/P1z8kpYYFCCkouYT5VG88eQTd9lzNuJbhSkIdJAJ8z
wyfvGjnZUwjyRq3egA== -----END CERTIFICATE-----
Description: None
Notes:
[root@el6-client ~]# yum repolist
Loaded plugins: rhnplugin
This system is not registered with RHN.
RHN support will be disabled.
pulp-v2-candidate | 1.9 kB 00:00
rh6-epel-live | 3.8 kB 00:00
rh6-epel-live/primary_db | 4.4 MB 00:00
repo id repo name status
pulp-v2-candidate Pulp v2 Beta Builds 36
rh6-epel-live rh6-epel-live 8,336
repolist: 8,372
[root@el6-client ~]# yum search zshLoaded plugins: rhnplugin
This system is not registered with RHN.
RHN support will be disabled.
================================= Matched: zsh =================================
autojump-zsh.noarch : Autojump for zsh
gromacs-zsh.noarch : GROMACS zsh support
plowshare.noarch : Download and upload files from file-sharing websites
shflags.noarch : Simple handling of command-line flags in Bourne based Unix
: scripts
[root@el6-client ~]#
Moving to verified per the above comment. [root@preethi-el6-pulp ~]# rpm -q pulp-server pulp-server-2.0.8-0.2.beta.noarch [root@preethi-el6-pulp ~]# Pulp 2.1 released http://www.pulpproject.org/2013/04/05/pulp-2-1-0-released/ |