Description of problem:
Some packagers tend to upload a newer rpm to their repository with the same name-version-release combination as the existing rpm. That leads to different checksums and might also lead to different filesizes.
reposync fails when:
(a) the remote size is bigger than the local copy
here, reposync only checks whether or not the file already exists _and_ the file sizes match. If that's the case, remove the file from the list. But not really, because there's still a checksum verification (see (b)). Anyways, if the local file size differs from the remote size, the file gets downloaded with a "Range:" header. If the remote file is smaller than the local copy, this results in a "416 range not satisfiable".
(b) the remote size is equal to the local copy but the checksum differs
reposync would skip this package but still the yum checksum verification kicks. Since yum doesn't do anything to mitigate that situation, reposync fails.
Version-Release number of selected component (if applicable):
yum-3.4.3-154
yum-utils-1.1.31-42
How reproducible:
always
Steps to Reproduce:
(a)
1. echo "foo" >> $existingpackage
2. reposync it
3. see reposync fail with a 416 error
(b)
1. dd if=/dev/urandom of=$existingpackage bs=100 count=1 conv=notrunc
2. reposync it
3. see reposync fail with "$package FAILED\n$package: [Errno 256] No more mirrors to try."
Actual results:
reposync doesn't redownload the file and fails
Expected results:
reposync removes the local copies and succeeds
Additional info:
In #1337957 Jens Kuehnel suggests to patch reposync to remove the local copy if its size is bigger than the remote copy's.
I think yum should handle this. In both cases, if the files differs in either size and/or content, the checksum will fail.
Yum should delete the local copy, I thought of something like that:
$ diff -up /usr/lib/python2.7/site-packages/yum/__init__.py{.orig,}
--- /usr/lib/python2.7/site-packages/yum/__init__.py.orig
+++ /usr/lib/python2.7/site-packages/yum/__init__.py
@@ -2469,6 +2469,8 @@ much more problems).
adderror(po, _('package fails checksum but caching is '
'enabled for %s') % po.repo.id)
return False
+ self.verbose_logger.debug("checksum failed for %s, removing local copy" % po)
+ os.remove(local)
if downloadonly:
po.localpath += '.%d.tmp' % os.getpid()
try: os.rename(local, po.localpath)
examples for such packages:
(a) newer package is smaller
$ stat -c '%n: %s' libgeotiff-1.4.0-1.rhel7.x86_64.rpm*
libgeotiff-1.4.0-1.rhel7.x86_64.rpm: 725296
libgeotiff-1.4.0-1.rhel7.x86_64.rpm.orig: 725332
$ diff -y <(rpm -qip libgeotiff-1.4.0-1.rhel7.x86_64.rpm) <(rpm -qip libgeotiff-1.4.0-1.rhel7.x86_64.rpm.orig)
Name : libgeotiff Name : libgeotiff
Version : 1.4.0 Version : 1.4.0
Release : 1.rhel7 Release : 1.rhel7
Architecture: x86_64 Architecture: x86_64
Install Date: (not installed) Install Date: (not installed)
Group : System Environment/Libraries Group : System Environment/Libraries
Size : 5635275 | Size : 5635254
License : MIT License : MIT
Signature : DSA/SHA1, Mo 18 Dez 2017 20:38:36 CET, Key ID 1 | Signature : DSA/SHA1, Fr 15 Dez 2017 04:02:22 CET, Key ID 1
Source RPM : libgeotiff-1.4.0-1.rhel7.src.rpm Source RPM : libgeotiff-1.4.0-1.rhel7.src.rpm
Build Date : Mo 18 Dez 2017 20:38:30 CET | Build Date : Fr 15 Dez 2017 04:02:19 CET
Build Host : koji-rhel7-x86-64-pgbuild Build Host : koji-rhel7-x86-64-pgbuild
Relocations : (not relocatable) Relocations : (not relocatable)
Vendor : PostgreSQL Global Development Group Vendor : PostgreSQL Global Development Group
URL : http://trac.osgeo.org/geotiff/ URL : http://trac.osgeo.org/geotiff/
Summary : GeoTIFF format library Summary : GeoTIFF format library
Description : Description :
GeoTIFF represents an effort by over 160 different remote sen GeoTIFF represents an effort by over 160 different remote sen
GIS, cartographic, and surveying related companies and organi GIS, cartographic, and surveying related companies and organi
to establish a TIFF based interchange format for georeference to establish a TIFF based interchange format for georeference
raster imagery. raster imagery.
(b) same package size:
$ stat -c '%n: %s' libgeotiff-debuginfo-1.4.0-1.rhel7.x86_64.rpm*
libgeotiff-debuginfo-1.4.0-1.rhel7.x86_64.rpm: 199736
libgeotiff-debuginfo-1.4.0-1.rhel7.x86_64.rpm.orig: 199736
$ diff -y <(rpm -qip libgeotiff-debuginfo-1.4.0-1.rhel7.x86_64.rpm) <(rpm -qip libgeotiff-debuginfo-1.4.0-1.rhel7.x86_64.rpm.orig)
Name : libgeotiff-debuginfo Name : libgeotiff-debuginfo
Version : 1.4.0 Version : 1.4.0
Release : 1.rhel7 Release : 1.rhel7
Architecture: x86_64 Architecture: x86_64
Install Date: (not installed) Install Date: (not installed)
Group : Development/Debug Group : Development/Debug
Size : 917315 | Size : 917294
License : MIT License : MIT
Signature : DSA/SHA1, Mo 18 Dez 2017 20:38:36 CET, Key ID 1 | Signature : DSA/SHA1, Fr 15 Dez 2017 04:02:22 CET, Key ID 1
Source RPM : libgeotiff-1.4.0-1.rhel7.src.rpm Source RPM : libgeotiff-1.4.0-1.rhel7.src.rpm
Build Date : Mo 18 Dez 2017 20:38:30 CET | Build Date : Fr 15 Dez 2017 04:02:19 CET
Build Host : koji-rhel7-x86-64-pgbuild Build Host : koji-rhel7-x86-64-pgbuild
Relocations : (not relocatable) Relocations : (not relocatable)
Vendor : PostgreSQL Global Development Group Vendor : PostgreSQL Global Development Group
URL : http://trac.osgeo.org/geotiff/ URL : http://trac.osgeo.org/geotiff/
Summary : Debug information for package libgeotiff Summary : Debug information for package libgeotiff
Description : Description :
This package provides debug information for package libgeotif This package provides debug information for package libgeotif
Debug information is useful when developing applications that Debug information is useful when developing applications that
package or when debugging this package. package or when debugging this package.
This bug is not planned to be addressed during Red Hat Enterprise Linux 7 life-cycle.
Please contact Red Hat support if you wish to have it reconsidered.
Description of problem: Some packagers tend to upload a newer rpm to their repository with the same name-version-release combination as the existing rpm. That leads to different checksums and might also lead to different filesizes. reposync fails when: (a) the remote size is bigger than the local copy here, reposync only checks whether or not the file already exists _and_ the file sizes match. If that's the case, remove the file from the list. But not really, because there's still a checksum verification (see (b)). Anyways, if the local file size differs from the remote size, the file gets downloaded with a "Range:" header. If the remote file is smaller than the local copy, this results in a "416 range not satisfiable". (b) the remote size is equal to the local copy but the checksum differs reposync would skip this package but still the yum checksum verification kicks. Since yum doesn't do anything to mitigate that situation, reposync fails. Version-Release number of selected component (if applicable): yum-3.4.3-154 yum-utils-1.1.31-42 How reproducible: always Steps to Reproduce: (a) 1. echo "foo" >> $existingpackage 2. reposync it 3. see reposync fail with a 416 error (b) 1. dd if=/dev/urandom of=$existingpackage bs=100 count=1 conv=notrunc 2. reposync it 3. see reposync fail with "$package FAILED\n$package: [Errno 256] No more mirrors to try." Actual results: reposync doesn't redownload the file and fails Expected results: reposync removes the local copies and succeeds Additional info: In #1337957 Jens Kuehnel suggests to patch reposync to remove the local copy if its size is bigger than the remote copy's. I think yum should handle this. In both cases, if the files differs in either size and/or content, the checksum will fail. Yum should delete the local copy, I thought of something like that: $ diff -up /usr/lib/python2.7/site-packages/yum/__init__.py{.orig,} --- /usr/lib/python2.7/site-packages/yum/__init__.py.orig +++ /usr/lib/python2.7/site-packages/yum/__init__.py @@ -2469,6 +2469,8 @@ much more problems). adderror(po, _('package fails checksum but caching is ' 'enabled for %s') % po.repo.id) return False + self.verbose_logger.debug("checksum failed for %s, removing local copy" % po) + os.remove(local) if downloadonly: po.localpath += '.%d.tmp' % os.getpid() try: os.rename(local, po.localpath) examples for such packages: (a) newer package is smaller $ stat -c '%n: %s' libgeotiff-1.4.0-1.rhel7.x86_64.rpm* libgeotiff-1.4.0-1.rhel7.x86_64.rpm: 725296 libgeotiff-1.4.0-1.rhel7.x86_64.rpm.orig: 725332 $ diff -y <(rpm -qip libgeotiff-1.4.0-1.rhel7.x86_64.rpm) <(rpm -qip libgeotiff-1.4.0-1.rhel7.x86_64.rpm.orig) Name : libgeotiff Name : libgeotiff Version : 1.4.0 Version : 1.4.0 Release : 1.rhel7 Release : 1.rhel7 Architecture: x86_64 Architecture: x86_64 Install Date: (not installed) Install Date: (not installed) Group : System Environment/Libraries Group : System Environment/Libraries Size : 5635275 | Size : 5635254 License : MIT License : MIT Signature : DSA/SHA1, Mo 18 Dez 2017 20:38:36 CET, Key ID 1 | Signature : DSA/SHA1, Fr 15 Dez 2017 04:02:22 CET, Key ID 1 Source RPM : libgeotiff-1.4.0-1.rhel7.src.rpm Source RPM : libgeotiff-1.4.0-1.rhel7.src.rpm Build Date : Mo 18 Dez 2017 20:38:30 CET | Build Date : Fr 15 Dez 2017 04:02:19 CET Build Host : koji-rhel7-x86-64-pgbuild Build Host : koji-rhel7-x86-64-pgbuild Relocations : (not relocatable) Relocations : (not relocatable) Vendor : PostgreSQL Global Development Group Vendor : PostgreSQL Global Development Group URL : http://trac.osgeo.org/geotiff/ URL : http://trac.osgeo.org/geotiff/ Summary : GeoTIFF format library Summary : GeoTIFF format library Description : Description : GeoTIFF represents an effort by over 160 different remote sen GeoTIFF represents an effort by over 160 different remote sen GIS, cartographic, and surveying related companies and organi GIS, cartographic, and surveying related companies and organi to establish a TIFF based interchange format for georeference to establish a TIFF based interchange format for georeference raster imagery. raster imagery. (b) same package size: $ stat -c '%n: %s' libgeotiff-debuginfo-1.4.0-1.rhel7.x86_64.rpm* libgeotiff-debuginfo-1.4.0-1.rhel7.x86_64.rpm: 199736 libgeotiff-debuginfo-1.4.0-1.rhel7.x86_64.rpm.orig: 199736 $ diff -y <(rpm -qip libgeotiff-debuginfo-1.4.0-1.rhel7.x86_64.rpm) <(rpm -qip libgeotiff-debuginfo-1.4.0-1.rhel7.x86_64.rpm.orig) Name : libgeotiff-debuginfo Name : libgeotiff-debuginfo Version : 1.4.0 Version : 1.4.0 Release : 1.rhel7 Release : 1.rhel7 Architecture: x86_64 Architecture: x86_64 Install Date: (not installed) Install Date: (not installed) Group : Development/Debug Group : Development/Debug Size : 917315 | Size : 917294 License : MIT License : MIT Signature : DSA/SHA1, Mo 18 Dez 2017 20:38:36 CET, Key ID 1 | Signature : DSA/SHA1, Fr 15 Dez 2017 04:02:22 CET, Key ID 1 Source RPM : libgeotiff-1.4.0-1.rhel7.src.rpm Source RPM : libgeotiff-1.4.0-1.rhel7.src.rpm Build Date : Mo 18 Dez 2017 20:38:30 CET | Build Date : Fr 15 Dez 2017 04:02:19 CET Build Host : koji-rhel7-x86-64-pgbuild Build Host : koji-rhel7-x86-64-pgbuild Relocations : (not relocatable) Relocations : (not relocatable) Vendor : PostgreSQL Global Development Group Vendor : PostgreSQL Global Development Group URL : http://trac.osgeo.org/geotiff/ URL : http://trac.osgeo.org/geotiff/ Summary : Debug information for package libgeotiff Summary : Debug information for package libgeotiff Description : Description : This package provides debug information for package libgeotif This package provides debug information for package libgeotif Debug information is useful when developing applications that Debug information is useful when developing applications that package or when debugging this package. package or when debugging this package.