Created attachment 835839 [details] Metadata file with duplicate packages that have the same name/version/release number but have different checksums Description of problem: Uploading to pulp an rpm with the same name/version/release but with a different checksum doesn't overwrite the existing one, it creates 2 versions of the same rpm package. Version-Release number of selected component (if applicable): pulp-server-2.2.0-1.el6.noarch running on CentOS 6.4 How reproducible: Upload to a pulp repo the package <name>-0.0.1-70 , publish the pulp repo. Upload the same <name>-0.0.1-70 package but with a different checksum. On querying pulp will return 2 versions of the same package name and yum install, on a server using the pulp server as yum repo, will fail [stefcoco@pulp-001 ~]$ pulp-admin rpm repo content rpm --repo-id cps-stable --match 'name=goggins' --match 'version=0.0.1' --match 'r elease=70' Arch: noarch Buildhost: inf-jks-002.cpp.o.itv.net.uk Checksum: 5f29f29217bf710260f04fa080bb199e49610851eb5806de2b95aed02416b464 Checksumtype: sha256 Description: Goggins java app Epoch: 0 Filename: goggins-0.0.1-70.noarch.rpm License: Proprietary Name: goggins Provides: config(goggins) = 0.0.1-70-0, goggins = 0.0.1-70-0 Release: 70 Requires: /bin/sh, /bin/sh, java-1.6.0-openjdk Vendor: None Version: 0.0.1 Arch: noarch Buildhost: inf-jks-002.cpp.o.itv.net.uk Checksum: d2dec47020faa72d01bb93117181659231bb3fd12334808afbb372198c91b284 Checksumtype: sha256 Description: Goggins java app Epoch: 0 Filename: goggins-0.0.1-70.noarch.rpm License: Proprietary Name: goggins Provides: config(goggins) = 0.0.1-70-0, goggins = 0.0.1-70-0 Release: 70 Requires: /bin/sh, /bin/sh, java-1.6.0-openjdk Vendor: None Version: 0.0.1 Yum install fails with: [Errno -1] Package does not match intended download. Suggestion: run yum --enablerepo=itv-pulp-cps-stable clean metadata Trying other mirror. Error Downloading Packages: goggins-0.0.1-70.noarch: failure: goggins-0.0.1-70.noarch.rpm from itv-pulp-cps-stable: [Errno 256] No more mirrors to try. because yum metadata contains the same package twice with different checksums. Have a look at the attached metadata files to see what I mean ( search for goggins rel=70 ) Steps to Reproduce: 1. Create a pulp repo 2. Upload an rpm to it and publish it 3. Upload a second rpm with the same name/version/release but with a different checksum and publish it 4. Try installing, on another server, from this repository via regular yum repos. Actual results: Pulp treats these packages as 2 different packages but yum install fails with [Errno -1] Package does not match intended download. Suggestion: run yum --enablerepo=itv-pulp-cps-stable clean metadata Trying other mirror. Error Downloading Packages: goggins-0.0.1-70.noarch: failure: goggins-0.0.1-70.noarch.rpm from itv-pulp-cps-stable: [Errno 256] No more mirrors to try. Expected results: Pulp should re-place the current package with the one that is being uploaded. That will create just one checksum inside the metadata files and yum would be able to install that package. Additional info:
The published repo, on disk, only contains one package version of goggins rel=70 but the metadata file contains 2 checksums which breaks yum. [stefcoco@pulp-001 ~]$ ll /var/lib/pulp/published/http/repos/cps-stable/ total 60 lrwxrwxrwx 1 apache apache 144 Nov 27 16:58 dawkins-0.0.1-118.noarch.rpm -> /var/lib/pulp/content/rpm/dawkins/0.0.1/118/noarch/c5b2b5 b48ab700ef8f2c15fb46dff6623052b8e328b95a8fdf6728520c109bfd/dawkins-0.0.1-118.noarch.rpm lrwxrwxrwx 1 apache apache 140 Nov 27 16:58 fnord-0.0.1-215.noarch.rpm -> /var/lib/pulp/content/rpm/fnord/0.0.1/215/noarch/a9a196e85d 95de9d9fdc1a6f675fdcc55f248b62f99514515b11a9546b9bee46/fnord-0.0.1-215.noarch.rpm lrwxrwxrwx 1 apache apache 142 Nov 27 16:58 goggins-0.0.1-61.noarch.rpm -> /var/lib/pulp/content/rpm/goggins/0.0.1/61/noarch/6b1f0459 16b9aa6b6425f9249af15c3475325193d5b379cc5d270c1404c5fbef/goggins-0.0.1-61.noarch.rpm lrwxrwxrwx 1 apache apache 142 Nov 29 16:01 goggins-0.0.1-65.noarch.rpm -> /var/lib/pulp/content/rpm/goggins/0.0.1/65/noarch/152fa050 6d259da3652ebb849f9832e65ed3db5804496c0ee4974891ff85eca1/goggins-0.0.1-65.noarch.rpm lrwxrwxrwx 1 apache apache 142 Dec 12 12:15 goggins-0.0.1-67.noarch.rpm -> /var/lib/pulp/content/rpm/goggins/0.0.1/67/noarch/97df5a25 e4ee78c0cdcc2ae1b71a7b43e0acca8c87eabcca93133fdc1fec2caa/goggins-0.0.1-67.noarch.rpm lrwxrwxrwx 1 apache apache 142 Dec 12 11:29 goggins-0.0.1-70.noarch.rpm -> /var/lib/pulp/content/rpm/goggins/0.0.1/70/noarch/d2dec470 20faa72d01bb93117181659231bb3fd12334808afbb372198c91b284/goggins-0.0.1-70.noarch.rpm lrwxrwxrwx 1 apache apache 146 Nov 27 16:58 greendale-0.0.1-87.noarch.rpm -> /var/lib/pulp/content/rpm/greendale/0.0.1/87/noarch/f2b9 e97e9bd6214e6d04f9dda523d752bac9f6d2b13e98ff94ef17a2cdb1ce82/greendale-0.0.1-87.noarch.rpm lrwxrwxrwx 1 apache apache 136 Nov 27 16:58 jess-0.0.1-58.noarch.rpm -> /var/lib/pulp/content/rpm/jess/0.0.1/58/noarch/1c7be39c2d4284 ece87f3eb0b1d98ad8102d102a9535559ed4c1cc26a60b4461/jess-0.0.1-58.noarch.rpm lrwxrwxrwx 1 apache apache 142 Nov 27 16:58 minerva-0.0.1-40.noarch.rpm -> /var/lib/pulp/content/rpm/minerva/0.0.1/40/noarch/97d40821 de5be66b2fc6f04538b3185645ae21a1b90bdd9c71b5338ab25c5e28/minerva-0.0.1-40.noarch.rpm lrwxrwxrwx 1 apache apache 67 Nov 27 14:22 Packages -> /var/lib/pulp/working/repos/cps-stable/distributors/yum_distributor lrwxrwxrwx 1 apache apache 134 Nov 27 16:58 pat-0.0.1-75.noarch.rpm -> /var/lib/pulp/content/rpm/pat/0.0.1/75/noarch/c15cd481d2349fab 28af275f3a3374969f28829ac759f173981bed3fea2acf47/pat-0.0.1-75.noarch.rpm drwxr-xr-x 2 apache apache 4096 Dec 12 12:17 repodata drwxr-xr-x 2 apache apache 4096 Dec 12 12:15 repodata.old lrwxrwxrwx 1 apache apache 148 Nov 27 16:58 swagger-ui-0.0.1-39.noarch.rpm -> /var/lib/pulp/content/rpm/swagger-ui/0.0.1/39/noarch/a4 5184d839b5459b1b9d4fc5f7d9089b1388be957be5d02560f0476308755c45/swagger-ui-0.0.1-39.noarch.rpm lrwxrwxrwx 1 apache apache 140 Nov 27 16:58 venus-0.0.1-127.noarch.rpm -> /var/lib/pulp/content/rpm/venus/0.0.1/127/noarch/5878912453 6dca80510c3dbfe2a511644e6f0d82194e37a3c34f05a8981119c3/venus-0.0.1-127.noarch.rpm
To add clarification here. The problem is that pulp is using the checksum as part of the uniqueness key for an RPM. A shorter term fix could be to remove the checksum value from the uniqueness constraint and keep the checksum type as part of the uniqueness constraint. In the long term the request requires decoupling the checksum type & value from the uniqueness key and having pulp generate multiple checksum types and the various checksum values for a particular rpm.
We will consider validation in the yum distributor at publish time as a future feature.
Moved to https://pulp.plan.io/issues/213