Bug 1080373 - epel/5/i386/repodata/73f710ca75f4879d6e4c7c08c4bac44f3a3b50f57756e5d711665bcd40e735cf-updateinfo.xml.gz: [Errno -3] Error performing checksum
Summary: epel/5/i386/repodata/73f710ca75f4879d6e4c7c08c4bac44f3a3b50f57756e5d711665bcd...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Fedora EPEL
Classification: Fedora
Component: bodhi
Version: el5
Hardware: All
OS: Linux
unspecified
high
Target Milestone: ---
Assignee: Luke Macken
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
: 1082059 1082069 1082074 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2014-03-25 09:57 UTC by Justin Albstmeijer
Modified: 2016-09-20 02:45 UTC (History)
11 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2014-04-29 21:40:10 UTC
Type: Bug


Attachments (Terms of Use)

Description Justin Albstmeijer 2014-03-25 09:57:15 UTC
Description of problem:

When issuing a 'yum check-update --security' I get

http://s3-mirror-eu-west-1.fedoraproject.org/pub/epel/5/i386/repodata/73f710ca75f4879d6e4c7c08c4bac44f3a3b50f57756e5d711665bcd40e735cf-updateinfo.xml.gz: [Errno -3] Error performing checksum

This has been so the last couple of days.

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:
1. yum clean all
2. run 'yum check-update --security'

Actual results:

# yum check-update --security
Loaded plugins: fastestmirror, security
Loading mirror speeds from cached hostfile
 * base: ftp.heanet.ie
 * extras: ftp.heanet.ie
 * updates: ftp.heanet.ie
Limiting package lists to security relevant ones
epel/updateinfo                                                                                                   | 514 kB     00:00     
http://s3-mirror-eu-west-1.fedoraproject.org/pub/epel/5/i386/repodata/73f710ca75f4879d6e4c7c08c4bac44f3a3b50f57756e5d711665bcd40e735cf-updateinfo.xml.gz: [Errno -3] Error performing checksum
Trying other mirror.
Error: failure: repodata/73f710ca75f4879d6e4c7c08c4bac44f3a3b50f57756e5d711665bcd40e735cf-updateinfo.xml.gz from epel: [Errno 256] No more mirrors to try.

Expected results:

list of possible security updates

Additional info:

This problem has been there for days now. On each mirror I end up on.

Comment 1 Jeff Sheltren 2014-03-25 13:44:43 UTC
This looks like a mirror issue, and not a yum problem, but try running 'yum clean metadata' to see if clearing the local cache helps at all.

Comment 2 Justin Albstmeijer 2014-03-25 15:21:01 UTC
> This looks like a mirror issue, and not a yum problem

Yes, sure it is no yum problem, but there was no other "Component" I could link it too. when submitting this report.

As mentioned in the report: "This problem has been there for days now. On each mirror I end up on."

So it is not a specific mirror, here an other:

http://download.fedoraproject.org/pub/epel/5/i386/repodata/73f710ca75f4879d6e4c7c08c4bac44f3a3b50f57756e5d711665bcd40e735cf-updateinfo.xml.gz: [Errno -3] Error performing checksum

> but try running 'yum clean metadata' to see if clearing the local cache helps at all.

As mentioned in the report: "Steps to Reproduce: 1. yum clean all"

And as expected "yum clean metadata" does not help either.

Comment 3 Kevin Fenzi 2014-03-25 15:42:09 UTC
Can you provide: 

rpm -q yum yum-security python-hashlib 

Are you behind a proxy of any kind? 

I cannot duplicate the issue here... ;(

Comment 4 Justin Albstmeijer 2014-03-25 15:50:35 UTC
Hi,

Your question pointed me to 'python-hashlib' which we did not have installed on any of our Centos5 servers;

# rpm -q yum yum-security python-hashlib 
yum-3.2.22-40.el5.centos
yum-security-1.1.16-21.el5.centos
package python-hashlib is not installed

After installing it, it works fine.

Is this a new requirement?, has not been needed till now.
And if so, should it not be a package requirement of epel-release-5-4.noarch.rpm?

Thank you for pointing me in the right direction.

Gr, Justin

Comment 5 Kevin Fenzi 2014-03-25 15:57:42 UTC
Well, I am a bit puzzled since nothing I can see has changed in repodata generation for years... 

You did have machines using yum-security plugin and without hashlib and it worked in the past?

Comment 6 Justin Albstmeijer 2014-03-25 16:18:10 UTC
Yes has been working without hashlib installed, and I'm sure as we use it as part of our monitoring.

In our logs I can go back to October 2012, and has been working since then.

[1351279362] SERVICE ALERT: xxx.xx.xx.eu;updates;WARNING;HARD;5;YUM WARNING: 0 Security Updates Available. 36 Non-Security Updates Available

All 20 EL5 machines started returning the checksum error 3 days ago.

Just disabling epel "yum check-update --security --disablerepo=epel" also returns no error without python-hashlib installed

Comment 7 Kevin Fenzi 2014-03-25 16:19:41 UTC
Adding lmacken here to comment on any possible changes in the last bodhi update that could be affecting this.

Comment 8 Luke Macken 2014-03-25 19:34:18 UTC
Bodhi 0.9.9 included a patch that added the sha256sum to the updateinfo filename.

https://lists.fedoraproject.org/pipermail/devel-announce/2014-March/001326.html
https://github.com/fedora-infra/bodhi/pull/35

It looks like yum on python2.4 does not support the sha256 checksum type, so we will need to avoid doing this on EL5 in bodhi.

Comment 9 Sean Reifschneider 2014-03-26 04:01:05 UTC
Luke: As was mentioned by Justin, the python-hashlib will allow you to then get to sha256 and other hash functions:

python -c 'from hashlib import sha256'

Comment 10 Luke Macken 2014-03-26 04:47:07 UTC
(In reply to Sean Reifschneider from comment #9)
> Luke: As was mentioned by Justin, the python-hashlib will allow you to then
> get to sha256 and other hash functions:
> 
> python -c 'from hashlib import sha256'

Right, yum will support sha256 is hashlib is available, however by default on RHEL5 it is not. So, instead of spinning up a yum errata for RHEL5, the easiestly solution is to have bodhi fallback to sha1 when injecting the updateinfo.xml into EL6 repos.

I implemented a potential hotfix, and will be testing it shortly.
https://github.com/fedora-infra/bodhi/pull/35

Comment 11 Luke Macken 2014-03-26 17:14:53 UTC
I pushed out a new version of bodhi to the masher last night and now we're waiting on an EL5 updates push to see if this issue gets resolved.

Comment 12 Anssi Johansson 2014-03-27 21:14:01 UTC
Some updates were pushed to EPEL5 stable today, but the problem with hashes did not disappear.

Comment 13 Luke Macken 2014-03-28 05:49:11 UTC
This issue should now be resolved.

The last EPEL5 push was with a new bodhi release which I thought included my hotfix, but it actually didn't (git-flow fail). So, I wrote a script to re-inject the updateinfo & pkgtags to solve the problem until the next EL5 push happens. https://gist.github.com/lmacken/9826113

Please re-open if you encounter this problem again.

Comment 14 Anssi Johansson 2014-03-28 08:56:13 UTC
The current metadata does not seem to contain information about some packages. python26 can't be found, and the recent gc-7.1-6.el5 update is not found either.

Comment 15 Anssi Johansson 2014-03-28 10:32:21 UTC
http://dl.fedoraproject.org/pub/epel/5/x86_64/repodata/ contains repodata from EPEL TESTING !!!

This means anyone who runs 'yum update' now will get updates from epel-testing, even if the epel-testing repo was not enabled for them.

Comment 16 Anssi Johansson 2014-03-28 10:54:41 UTC
Well, would get updates if the packages were also there. Now yum only ends up complaining about rpm files that are referenced in the repodata, but are not present in the stable directory.

Still, this means "yum update" breakage for anyone who has a prior version of any of the 393 referenced testing rpms installed.

Comment 17 Rex Dieter 2014-03-28 11:53:00 UTC
These last 3 comments seem like a different issue that what is being tracked here (though may end up being related).

Comment 18 Kevin Fenzi 2014-03-28 15:32:43 UTC
This is a consequence of us trying to fix the repodata... so yes, it's the same issue really. ;) 

We are working on fixing it. Please stand by.

Comment 19 Kevin Fenzi 2014-03-28 15:33:54 UTC
*** Bug 1082059 has been marked as a duplicate of this bug. ***

Comment 20 Kevin Fenzi 2014-03-28 15:52:57 UTC
*** Bug 1082074 has been marked as a duplicate of this bug. ***

Comment 21 Luke Macken 2014-03-28 15:58:23 UTC
This should hopefully be resolved after the next sync, which happen every 30 minutes.

So it looks like only el5 x86_64 was affected by this last issue, which was caused by human error when using a script to rehash the metadata.

Comment 22 Kevin Fenzi 2014-03-28 16:06:39 UTC
*** Bug 1082069 has been marked as a duplicate of this bug. ***

Comment 23 Luke Macken 2014-03-28 16:23:43 UTC
Fixed repodata has hit the master mirror. Can someone confirm that this problem is resolved?

Comment 24 Anssi Johansson 2014-03-28 16:24:37 UTC
Yes, seems to work now. Thanks for fixing :)

Comment 25 Luke Macken 2014-03-28 16:26:46 UTC
Excellent. Thank you and apologies for the annoyance!

Comment 26 Paul Lewis 2014-03-28 17:36:19 UTC
*** Bug 1082074 has been marked as a duplicate of this bug. ***

Comment 27 Justin Albstmeijer 2014-04-29 16:42:41 UTC
Hi,

Today I saw a server peaking at 40mbit every time a yum check was running initiated every 30 minutes by puppet.
I seem to be experiencing the same problem again with regular yum actions.

As yum was downloading 6MB from each mirror to conclude that it could not perform a checksum loads of traffic was generated.. a perfect designed DDOS :P

For now I just installed python-hashlib to mitigate the issue.

Gr, Justin

Comment 28 Luke Macken 2014-04-29 16:50:11 UTC
(In reply to Justin Albstmeijer from comment #27)
> Hi,
> 
> Today I saw a server peaking at 40mbit every time a yum check was running
> initiated every 30 minutes by puppet.
> I seem to be experiencing the same problem again with regular yum actions.
> 
> As yum was downloading 6MB from each mirror to conclude that it could not
> perform a checksum loads of traffic was generated.. a perfect designed DDOS
> :P
> 
> For now I just installed python-hashlib to mitigate the issue.
> 
> Gr, Justin

The EL5 repodata looks fine to me; the updateinfo and pkgtags are using the sha1 checksum, which should be supported by yum without python-hashlib.

Can you `yum clean all`, and then try and reproduce the issue and provide us with some of the error messages? Thanks

Comment 29 Justin Albstmeijer 2014-04-29 19:57:22 UTC
hope this illustrates it.


[root@XXX ~]# yum clean all
Loaded plugins: fastestmirror, security
Cleaning up Everything
Cleaning up list of fastest mirrors

[root@XXX ~]# yum search notexisting
Loaded plugins: fastestmirror, security
Determining fastest mirrors
epel/metalink                                                                                                                                                                               |  25 kB     00:00     
 * base: centos.mirror.triple-it.nl
 * epel: epel.check-update.co.uk
 * extras: mirror.widexs.nl
 * openvz-kernel-rhel5: openvz.proserve.nl
 * openvz-utils: openvz.proserve.nl
 * updates: centos.mirror.transip.nl
RE                                                                                                                                                                                          | 2.1 kB     00:00     
RE/primary_db                                                                                                                                                                               |  83 kB     00:00     
base                                                                                                                                                                                        | 1.1 kB     00:00     
base/primary                                                                                                                                                                                | 1.3 MB     00:00     
base                                                                                                                                                                                                     3662/3662
epel                                                                                                                                                                                        | 4.4 kB     00:00     
epel/primary_db                                                                                                                                                                             | 6.1 MB     00:00     
extras                                                                                                                                                                                      | 2.1 kB     00:00     
extras/primary_db                                                                                                                                                                           | 173 kB     00:00     
ksplice-uptrack                                                                                                                                                                             |  951 B     00:00     
ksplice-uptrack/primary                                                                                                                                                                     | 7.5 kB     00:00     
ksplice-uptrack                                                                                                                                                                                              38/38
openvz-kernel-rhel5                                                                                                                                                                         |  951 B     00:00     
openvz-kernel-rhel5/primary                                                                                                                                                                 | 3.2 kB     00:00     
openvz-kernel-rhel5                                                                                                                                                                                          14/14
openvz-utils                                                                                                                                                                                |  951 B     00:00     
openvz-utils/primary                                                                                                                                                                        | 9.4 kB     00:00     
openvz-utils                                                                                                                                                                                                 38/38
updates                                                                                                                                                                                     | 1.9 kB     00:00     
updates/primary_db                                                                                                                                                                          | 463 kB     00:00     
Warning: No matches found for: notexisting
No Matches found


[root@XXX ~]# rpm -e python-hashlib

[root@XXX ~]# yum clean all
Loaded plugins: fastestmirror, security
Cleaning up Everything
Cleaning up list of fastest mirrors

[root@XXX ~]# yum search notexisting
Loaded plugins: fastestmirror, security
Determining fastest mirrors
epel/metalink                                                                                                                                                                               |  25 kB     00:00     
 * base: mirror.prolocation.net
 * epel: ftp.nluug.nl
 * extras: mirror.widexs.nl
 * openvz-kernel-rhel5: openvz.proserve.nl
 * openvz-utils: openvz.proserve.nl
 * updates: mirror.amsiohosting.net
RE                                                                                                                                                                                          | 2.1 kB     00:00     
RE/primary_db                                                                                                                                                                               |  83 kB     00:00     
base                                                                                                                                                                                        | 1.1 kB     00:00     
base/primary                                                                                                                                                                                | 1.3 MB     00:00     
base                                                                                                                                                                                                     3662/3662
epel                                                                                                                                                                                        | 4.4 kB     00:00     
epel/primary_db                                                                                                                                                                             | 6.1 MB     00:00     
http://ftp.nluug.nl/pub/os/Linux/distr/fedora-epel/6/x86_64/repodata/59271863f08aa4c47fe5baac811cf059af49a75e270a602b6e9a7157d201cd78-primary.sqlite.bz2: [Errno -3] Error performing checksum
Trying other mirror.
epel/primary_db                                                                                                                                                                             | 6.1 MB     00:00     
http://mirror.1000mbps.com/fedora-epel/6/x86_64/repodata/59271863f08aa4c47fe5baac811cf059af49a75e270a602b6e9a7157d201cd78-primary.sqlite.bz2: [Errno -3] Error performing checksum
Trying other mirror.
epel/primary_db                                                                                                                                                                             | 6.1 MB     00:00     
http://mirror.proserve.nl/fedora-epel/6/x86_64/repodata/59271863f08aa4c47fe5baac811cf059af49a75e270a602b6e9a7157d201cd78-primary.sqlite.bz2: [Errno -3] Error performing checksum
Trying other mirror.
epel/primary_db                                                                                                                                                                             | 6.1 MB     00:00     
http://nl.mirror.eurid.eu/epel/6/x86_64/repodata/59271863f08aa4c47fe5baac811cf059af49a75e270a602b6e9a7157d201cd78-primary.sqlite.bz2: [Errno -3] Error performing checksum
Trying other mirror.
epel/primary_db                                                                                                                                                                             | 4.1 MB     00:00 ... 

 Current download cancelled, interrupt (ctrl-c) again within two seconds to exit.

epel/primary_db                                                                                                                                                                             | 1.0 MB     00:00 ... 


Exiting on user cancel

Comment 30 Justin Albstmeijer 2014-04-29 20:02:11 UTC
A wait I see it, I seem to mistakenly have placed a hardcoded Centos6 repo file on a centos5 machine.

Excuse me.

Comment 31 Luke Macken 2014-04-29 21:40:10 UTC
Yeah, since it's complaining about the primary.sqlite.bz2 it is unrelated to this specific bug.


Note You need to log in before you can comment on or make changes to this bug.