Hide Forgot
If you sync a repository that contains sha1 hashed metadata and publish this repositry in Pulp it gets regenerated with sha256 metadata. This causes issues with older versions of yum on RHEL5 that can't hash the sha256 repomd.xml Example repo: http://inecas.fedorapeople.org/fakerepos/zoo-rhel5-new/ http://inecas.fedorapeople.org/fakerepos/zoo-rhel5-new/repodata/repomd.xml <data type="filelists_db"> <checksum type="sha1">354f8ba7e62b9ad2042bd3d4bee5b5689b476bd2</checksum> synced into Satellite 6 and republished: http://sat-perf-04.idm.lab.bos.redhat.com/pulp/repos/DemoOrg/Library/custom/Test_Product/sha1-test/Packages/repodata/repomd.xml <data type="filelists_db"> <checksum type="sha256">4915714b28cec0cf23b469478303f9a1ad2e030d4238ee7919d684ec9d8a02b0</checksum>
The idea we discussed during a call was to base the hashing on whatever is in the repomd.xml and use that for any published repos.
PR: https://github.com/pulp/pulp_rpm/pull/380
fails-qa [root@ibm-hs22-04 ~]# rpm -qa |grep pulp-server pulp-server-2.3.0-0.32.beta.el6.noarch [root@ibm-hs22-04 ~]# I synced a rhel5 repo from cdn and looks like there still some sha256 data_type <repomd xmlns="http://linux.duke.edu/metadata/repo" xmlns:rpm="http://linux.duke.edu/metadata/rpm"> <revision>1384260872</revision> <data type="group"> <checksum type="sha256"> 9c3323c8c08883e83b476e3de5aa638919fb95ae4511d8b746d39aee2b945b78 </checksum> <open-checksum type="sha256"> 9c3323c8c08883e83b476e3de5aa638919fb95ae4511d8b746d39aee2b945b78 </open-checksum> <location href="repodata/9c3323c8c08883e83b476e3de5aa638919fb95ae4511d8b746d39aee2b945b78-group.xml"/> <timestamp>1384261066</timestamp> <size>1066257</size> </data> <data type="filelists"> <checksum type="sha">7ee6e0e701ec0d12429df3d398c71351ac3be4ce</checksum> <open-checksum type="sha">23c7862e1e2e6df94cee1b5f20b9dc0dbc1120bc</open-checksum> <location href="repodata/7ee6e0e701ec0d12429df3d398c71351ac3be4ce-filelists.xml.gz"/> <timestamp>1384260962</timestamp> <size>28815099</size> <open-size>469369703</open-size> </data> <data type="updateinfo"> <checksum type="sha256"> 2b4ad76dcf25955a28e1aea4bf6300deb8b9541caa26698e501b84c9f7b673d9 </checksum> <open-checksum type="sha256"> 2b4ad76dcf25955a28e1aea4bf6300deb8b9541caa26698e501b84c9f7b673d9 </open-checksum> <location href="repodata/2b4ad76dcf25955a28e1aea4bf6300deb8b9541caa26698e501b84c9f7b673d9-updateinfo.xml"/> <timestamp>1384261067</timestamp> <size>10816412</size> </data> <data type="productid"> <checksum type="sha256"> 7dcb43539ac47c63b6fc2647377430baedff49c6ccfc9d154c47d2957e3902db </checksum> <open-checksum type="sha256"> 7dcb43539ac47c63b6fc2647377430baedff49c6ccfc9d154c47d2957e3902db </open-checksum> <location href="repodata/7dcb43539ac47c63b6fc2647377430baedff49c6ccfc9d154c47d2957e3902db-productid.xml"/> <timestamp>1384261067</timestamp> <size>2159</size> </data>
PR: https://github.com/pulp/pulp_rpm/pull/384
Looks like this still is an issue [root@ibm-hs22-04 ~]# rpm -qa pulp-server pulp-server-2.3.0-0.33.beta.el6.noarch [root@ibm-hs22-04 ~]# I installed the latest and synced the rhel5 repo from CDN still seeing this <repomd xmlns="http://linux.duke.edu/metadata/repo" xmlns:rpm="http://linux.duke.edu/metadata/rpm"> <revision>1384313137</revision> <data type="group"> <checksum type="sha256"> 9c3323c8c08883e83b476e3de5aa638919fb95ae4511d8b746d39aee2b945b78 </checksum> <open-checksum type="sha256"> 9c3323c8c08883e83b476e3de5aa638919fb95ae4511d8b746d39aee2b945b78 </open-checksum> <location href="repodata/9c3323c8c08883e83b476e3de5aa638919fb95ae4511d8b746d39aee2b945b78-group.xml"/> <timestamp>1384313320</timestamp> <size>1066257</size> </data> <data type="updateinfo"> <checksum type="sha256"> 2b4ad76dcf25955a28e1aea4bf6300deb8b9541caa26698e501b84c9f7b673d9 </checksum> <open-checksum type="sha256"> 2b4ad76dcf25955a28e1aea4bf6300deb8b9541caa26698e501b84c9f7b673d9 </open-checksum> <location href="repodata/2b4ad76dcf25955a28e1aea4bf6300deb8b9541caa26698e501b84c9f7b673d9-updateinfo.xml"/> <timestamp>1384313321</timestamp> <size>10816412</size> </data> <data type="productid"> <checksum type="sha256"> 7dcb43539ac47c63b6fc2647377430baedff49c6ccfc9d154c47d2957e3902db </checksum> <open-checksum type="sha256"> 7dcb43539ac47c63b6fc2647377430baedff49c6ccfc9d154c47d2957e3902db </open-checksum> <location href="repodata/7dcb43539ac47c63b6fc2647377430baedff49c6ccfc9d154c47d2957e3902db-productid.xml"/> <timestamp>1384313323</timestamp> <size>2159</size> </data>
https://github.com/pulp/pulp_rpm/pull/385
I have just confirmed that this has been fixed in following beta packages: 2.3.0-0.34.beta.el6
<repomd> <revision>1384455086</revision> − <data type="group"> <checksum type="sha1">5a6391f1a5b63cb8e82fcfea3b21810d57128627</checksum> <open-checksum type="sha1">5a6391f1a5b63cb8e82fcfea3b21810d57128627</open-checksum> <location href="repodata/5a6391f1a5b63cb8e82fcfea3b21810d57128627-group.xml"/> <timestamp>1384455251</timestamp> <size>1066257</size> </data> − <data type="filelists"> <checksum type="sha1">c0cb0b35d61b0544ad437f99cc4397527c6862d2</checksum> <open-checksum type="sha1">e3b960b6c43380888dc0c6fb7c1de600427e7476</open-checksum> <location href="repodata/c0cb0b35d61b0544ad437f99cc4397527c6862d2-filelists.xml.gz"/> <timestamp>1384455162</timestamp> <size>28758254</size> <open-size>468209725</open-size> </data> − <data type="updateinfo"> <checksum type="sha1">b2fe788ed1ed08ca346076240606cde39000a443</checksum> <open-checksum type="sha1">b2fe788ed1ed08ca346076240606cde39000a443</open-checksum> <location href="repodata/b2fe788ed1ed08ca346076240606cde39000a443-updateinfo.xml"/> <timestamp>1384455251</timestamp> <size>10816412</size> </data> − <data type="primary"> <checksum type="sha1">628c22f7fc52eea807379dc04f1a81851e774fd4</checksum> <open-checksum type="sha1">6704d6b1eef7516477b435c2b2a13c66b56f0377</open-checksum> <location href="repodata/628c22f7fc52eea807379dc04f1a81851e774fd4-primary.xml.gz"/> <timestamp>1384455139</timestamp> <size>2280441</size> <open-size>44755414</open-size> </data> − <data type="other"> <checksum type="sha1">ebd0685493c49893865202529f3ea812f1cd015e</checksum> <open-checksum type="sha1">9af56ec1da5857809a50d25edf9932c0a1d2e4ef</open-checksum> <location href="repodata/ebd0685493c49893865202529f3ea812f1cd015e-other.xml.gz"/> <timestamp>1384455163</timestamp> <size>3243340</size> <open-size>39150339</open-size> </data> − <data type="filelists_db"> <checksum type="sha1">d5cfe365c16b4211212601f3a8fed9861774534f</checksum> <open-checksum type="sha1">1205131180618bd24b4c672e2fd96c711e1806f0</open-checksum> <location href="repodata/d5cfe365c16b4211212601f3a8fed9861774534f-filelists.sqlite.bz2"/> <timestamp>1384455239</timestamp> <database_version>10</database_version> <size>20681206</size> <open-size>231503872</open-size> </data> − <data type="primary_db"> <checksum type="sha1">babd2036cb3b658c50fb36b4fe35e1be143a5aa2</checksum> <open-checksum type="sha1">0dea171a3b38ce8c2098117e83ff04c99023a7d7</open-checksum> <location href="repodata/babd2036cb3b658c50fb36b4fe35e1be143a5aa2-primary.sqlite.bz2"/> <timestamp>1384455251</timestamp> <database_version>10</database_version> <size>7953701</size> <open-size>53777408</open-size> </data> − <data type="other_db"> <checksum type="sha1">efed8083408bffb9c07bad360e48c12a3888878d</checksum> <open-checksum type="sha1">2336bec9cd3cd2a7287b49faf8533202859a1d7b</open-checksum> <location href="repodata/efed8083408bffb9c07bad360e48c12a3888878d-other.sqlite.bz2"/> <timestamp>1384455171</timestamp> <database_version>10</database_version> <size>4229916</size> <open-size>38763520</open-size> </data> − <data type="productid"> <checksum type="sha1">72b1d00d6b3729e9df2f3a5cc3cdd22b4baf2bfb</checksum> <open-checksum type="sha1">72b1d00d6b3729e9df2f3a5cc3cdd22b4baf2bfb</open-checksum> <location href="repodata/72b1d00d6b3729e9df2f3a5cc3cdd22b4baf2bfb-productid.xml"/> <timestamp>1384455252</timestamp> <size>2159</size> </data> </repomd>
[root@rhel5-pulp ~]# pulp-consumer -u admin -p admin rpm bind --repo-id rhel59-os Bind tasks successfully created: Task Id: 5e120711-4ab7-4b25-87ab-776af6e41dd3 Task Id: 3e85702e-34c3-4004-8ffa-395a6fa639e1 [root@rhel5-pulp ~]# yum groupinstall "DNS Name Server" --disablerepo=* --enablerepo=rhel59-os Loaded plugins: rhnplugin, security Setting up Group Process Resolving Dependencies --> Running transaction check ---> Package bind.x86_64 30:9.3.6-20.P1.el5_8.6 set to be updated --> Processing Dependency: bind-libs = 30:9.3.6-20.P1.el5_8.6 for package: bind ---> Package bind-chroot.x86_64 30:9.3.6-20.P1.el5_8.6 set to be updated --> Running transaction check --> Processing Dependency: bind-libs = 30:9.3.6-20.P1.el5_8.1 for package: bind-utils ---> Package bind-libs.x86_64 30:9.3.6-20.P1.el5_8.6 set to be updated --> Running transaction check ---> Package bind-utils.x86_64 30:9.3.6-20.P1.el5_8.6 set to be updated --> Finished Dependency Resolution Dependencies Resolved ================================================================================ Package Arch Version Repository Size ================================================================================ Installing: bind x86_64 30:9.3.6-20.P1.el5_8.6 rhel59-os 989 k bind-chroot x86_64 30:9.3.6-20.P1.el5_8.6 rhel59-os 47 k Updating for dependencies: bind-libs x86_64 30:9.3.6-20.P1.el5_8.6 rhel59-os 898 k bind-utils x86_64 30:9.3.6-20.P1.el5_8.6 rhel59-os 180 k Transaction Summary ================================================================================ Install 2 Package(s) Upgrade 2 Package(s) Total download size: 2.1 M Is this ok [y/N]: y Downloading Packages: (1/4): bind-chroot-9.3.6-20.P1.el5_8.6.x86_64.rpm | 47 kB 00:00 (2/4): bind-utils-9.3.6-20.P1.el5_8.6.x86_64.rpm | 180 kB 00:00 (3/4): bind-libs-9.3.6-20.P1.el5_8.6.x86_64.rpm | 898 kB 00:00 (4/4): bind-9.3.6-20.P1.el5_8.6.x86_64.rpm | 989 kB 00:00 -------------------------------------------------------------------------------- Total 1.3 MB/s | 2.1 MB 00:01 Running rpm_check_debug Running Transaction Test Finished Transaction Test Transaction Test Succeeded Running Transaction Updating : bind-libs 1/6 Installing : bind 2/6 Installing : bind-chroot 3/6 Updating : bind-utils 4/6 Cleanup : bind-utils 5/6 Cleanup : bind-libs 6/6 Installed: bind.x86_64 30:9.3.6-20.P1.el5_8.6 bind-chroot.x86_64 30:9.3.6-20.P1.el5_8.6 Dependency Updated: bind-libs.x86_64 30:9.3.6-20.P1.el5_8.6 bind-utils.x86_64 30:9.3.6-20.P1.el5_8.6 Complete! [root@rhel5-pulp ~]#
verified Also kickstarted rhel5 vm from the synced repo root@qeblade20 ~]# virt-install -n rhel5 --disk path=/dev/vg_qeblade20/rhel5 -l http://dell-pe2950-01.lab.eng.rdu.redhat.com/pulp/repos/content/dist/rhel/rhui/server/5/5.9/x86_64/os/ -r 4096 -w bridge:br0 --prompt --vnc
This is still not working correctly on a child pulp node. The "scratchpad" attribute of the repository is not being propagated by nodes, and thus the distributor on the child node doesn't have the shared checksum info.
https://github.com/pulp/pulp/pull/713
build: 2.3.0-0.36.beta
verified that the node child has the correct checksum [root@hp-dl120g5-01 ~]# rpm -qa |grep pulp-nodes-child pulp-nodes-child-2.3.0-0.36.beta.el6.noarch [root@hp-dl120g5-01 ~]# rpm -qa |grep pulp-server pulp-server-2.3.0-0.36.beta.el6.noarch [root@hp-dl120g5-01 ~]# Synced rehl5_9 on node parent node synced the rhel59 to the node child The checksum on the node child is below. <data type="filelists"> <checksum type="sha1">1bd1df19e39abe9531ec0b5d179778280e02dee9</checksum> <open-checksum type="sha1">59fac13f1a98bc42158ec084efb37933e09efa49</open-checksum> <location href="repodata/1bd1df19e39abe9531ec0b5d179778280e02dee9-filelists.xml.gz"/> <timestamp>1384645070</timestamp> <size>28819788</size> <open-size>469430490</open-size> </data> <data type="updateinfo"> <checksum type="sha1">b2fe788ed1ed08ca346076240606cde39000a443</checksum> <open-checksum type="sha1">b2fe788ed1ed08ca346076240606cde39000a443</open-checksum> <location href="repodata/b2fe788ed1ed08ca346076240606cde39000a443-updateinfo.xml"/> <timestamp>1384645158</timestamp> <size>10816412</size> </data> <data type="primary"> <checksum type="sha1">e13e66b3a7134c6ecd6e5947da1cc5725634567d</checksum> <open-checksum type="sha1">d5d7c3ab087597523de8f3f25fb5ce7af9918be6</open-checksum> <location href="repodata/e13e66b3a7134c6ecd6e5947da1cc5725634567d-primary.xml.gz"/> <timestamp>1384645049</timestamp> <size>2280527</size> <open-size>44757099</open-size> </data> <data type="other"> <checksum type="sha1">cf9456848399b2197b371bf3e22e9c1d51745265</checksum> <open-checksum type="sha1">d32351e2b0ed14416a857748fe38c14310f11f61</open-checksum> <location href="repodata/cf9456848399b2197b371bf3e22e9c1d51745265-other.xml.gz"/> <timestamp>1384645072</timestamp> <size>3243388</size> <open-size>39153368</open-size> </data> <data type="filelists_db"> <checksum type="sha1">c4442af8968409a6697c0554ccd4fc0f9c56eb0f</checksum> <open-checksum type="sha1">cb0e44f2a9ffff085a819fcc833d8fb34cd64a8b</open-checksum> <location href="repodata/c4442af8968409a6697c0554ccd4fc0f9c56eb0f-filelists.sqlite.bz2"/> <timestamp>1384645144</timestamp> <database_version>10</database_version> <size>20770626</size> <open-size>231911424</open-size> </data> <data type="primary_db"> <checksum type="sha1">cfc5818d294dccbd90331ac6ac86f0d5912cf8eb</checksum> <open-checksum type="sha1">6997fa58b1c719a0816d1a90756a37c5141ab38a</open-checksum> <location href="repodata/cfc5818d294dccbd90331ac6ac86f0d5912cf8eb-primary.sqlite.bz2"/> <timestamp>1384645156</timestamp> <database_version>10</database_version> <size>7948270</size> <open-size>53788672</open-size> </data> <data type="other_db"> <checksum type="sha1">796cd1d68e01713a2459885e65b981f5ff75231f</checksum> <open-checksum type="sha1">c59c937612e8d2b6dc382e98894be974b0e0b2c8</open-checksum> <location href="repodata/796cd1d68e01713a2459885e65b981f5ff75231f-other.sqlite.bz2"/> <timestamp>1384645080</timestamp> <database_version>10</database_version> <size>4227454</size> <open-size>38767616</open-size> </data> <data type="productid"> <checksum type="sha1">72b1d00d6b3729e9df2f3a5cc3cdd22b4baf2bfb</checksum> <open-checksum type="sha1">72b1d00d6b3729e9df2f3a5cc3cdd22b4baf2bfb</open-checksum> <location href="repodata/72b1d00d6b3729e9df2f3a5cc3cdd22b4baf2bfb-productid.xml"/> <timestamp>1384645158</timestamp> <size>2159</size> </data> </repomd>
Taking it back to fails-qa Looks like repo copy does not keep the checksum of the parent steps to reproduce. 1. Copy the rhel5 repo to a new repo 2. Run repo publish There is a workaround for this though. When you create the new repo you can created/updated with --checksum-type option before running publish root@dell-pe2950-01 ~]# pulp-admin rpm repo create --repo-id zoo-checksum --checksum-type sha1 Successfully created repository [zoo-checksum]
This is a challenging issue. We have a plan for a fix, but it will not be completely hands-off. The user will be responsible for copying config options from the distributor on the source repo to the distributor on the destination repo. Currently, the yum importer notices the use of sha1 upstream and makes a note of it in the repo scratchpad. The distributor then sees the note and publishes with sha1. The problem is that when copying content, attributes of the repo (such as its scratchpad) are not copied. This is by design, but makes this particular use case difficult. The solution is that when the distributor sees the note in the scratchpad, it will look at its own config, and if a checksum has not been set, it will save the checksum in its config. This allows someone who wants to create a copy of a repo to copy the distributor's config, and thus take advantage of this checksum detection. We already have APIs and UIs to support this. For a pulp-admin user, this will require creating the destination repo such that the distributor checksum matches what's in the source repo's distributor.
PR: https://github.com/pulp/pulp_rpm/pull/390
To test the latest change: 1) do a sync and publish of a rhel5 repo 2) verify that the sha1 checksum was used for publish 3) verify that the distributor now has a setting for checksum type as sha1. As for a copy, it is the responsibility of the user doing to copy to see the setting on the source repo's distributor and copy that to the destination repo's distributor if it makes sense. It also wouldn't hurt to sync the repo to a child node, do a publish, and do steps 2 and 3 above on the child.
[root@dell-pe2950-01 ~]# pulp-admin rpm repo list --details +----------------------------------------------------------------------+ RPM Repositories +----------------------------------------------------------------------+ Id: rhel59-os Display Name: rhel59-os Distributors: Auto Publish: True Config: Checksum Type: sha1 Http: True Https: True Relative URL: /content/dist/rhel/rhui/server/5/5.9/x86_64/os/ Distributor Type Id: yum_distributor Id: yum_distributor Last Publish: 2013-11-17T11:10:54-05:00 Repo Id: rhel59-os Scheduled Publishes:
Tested it on node as well. Here is the status of the repo before the fix on node child [root@hp-dl120g5-01 ~]# pulp-admin -u admin -p admin repo list +----------------------------------------------------------------------+ Repositories +----------------------------------------------------------------------+ Id: rhel59-os Display Name: rhel59-os Description: None Content Unit Counts: Distribution: 1 Erratum: 2768 Package Category: 6 Package Group: 99 Rpm: 15362 Yum Repo Metadata File: 1 [root@hp-dl120g5-01 ~]# pulp-admin -u admin -p admin repo list --details +----------------------------------------------------------------------+ Repositories +----------------------------------------------------------------------+ Id: rhel59-os Display Name: rhel59-os Description: None Content Unit Counts: Distribution: 1 Erratum: 2768 Package Category: 6 Package Group: 99 Rpm: 15362 Yum Repo Metadata File: 1 Notes: Importers: Config: Manifest URL: https://dell-pe2950-01.lab.eng.rdu.redhat.com/pulp/nodes/https /repos/rhel59-os/manifest.json Strategy: additive Id: nodes_http_importer Importer Type Id: nodes_http_importer Last Sync: 2013-11-16T18:32:56-05:00 Repo Id: rhel59-os Scheduled Syncs: Distributors: Auto Publish: True Config: Http: True Https: True Relative URL: /content/dist/rhel/rhui/server/5/5.9/x86_64/os/ Distributor Type Id: yum_distributor Id: yum_distributor Last Publish: 2013-11-16T18:39:19-05:00 Repo Id: rhel59-os Scheduled Publishes: Auto Publish: False Config: Http: True Https: True Distributor Type Id: export_distributor Id: export_distributor Last Publish: None Repo Id: rhel59-os Scheduled Publishes: [root@hp-dl120g5-01 ~]# service httpd restart Stopping httpd: [ OK ] Starting httpd: [ OK ] [root@hp-dl120g5-01 ~]# service pulp-agent restart Stopping goferd[ OK ] Starting goferd[ OK ] After the update on node parent & child And ran node sync [root@dell-pe2950-01 ~]# pulp-admin node sync run --node-id node1 This command may be exited via ctrl+c without affecting the request. [-] Waiting to begin... (1/1) Repository: rhel59-os [==================================================] 100% Synchronization succeeded +----------------------------------------------------------------------+ Child Node Synchronization +----------------------------------------------------------------------+ Repository: Action: Merged Id: rhel59-os Units: Added: 0 Removed: 0 Updated: 107 See the updated checksum field on the the child repo [root@hp-dl120g5-01 ~]# pulp-admin -u admin -p admin repo list --details +----------------------------------------------------------------------+ Repositories +----------------------------------------------------------------------+ Id: rhel59-os Display Name: rhel59-os Description: None Content Unit Counts: Distribution: 1 Erratum: 2768 Package Category: 6 Package Group: 99 Rpm: 15362 Yum Repo Metadata File: 1 Notes: Importers: Config: Manifest URL: https://dell-pe2950-01.lab.eng.rdu.redhat.com/pulp/nodes/https /repos/rhel59-os/manifest.json Strategy: additive Id: nodes_http_importer Importer Type Id: nodes_http_importer Last Sync: 2013-11-19T13:05:17-05:00 Repo Id: rhel59-os Scheduled Syncs: Distributors: Auto Publish: True Config: Checksum Type: sha1 Http: True Https: True Relative URL: /content/dist/rhel/rhui/server/5/5.9/x86_64/os/ Distributor Type Id: yum_distributor Id: yum_distributor Last Publish: 2013-11-16T18:39:19-05:00 Repo Id: rhel59-os Scheduled Publishes: Auto Publish: False Config: Http: True Https: True Distributor Type Id: export_distributor Id: export_distributor Last Publish: None Repo Id: rhel59-os Scheduled Publishes: [root@hp-dl120g5-01 ~]#
above tests are with build 2.3.0-0.37.beta
verified [root@dell-pe2950-01 ~]# rpm -qa |grep pulp-server pulp-server-2.3.0-0.37.beta.el6.noarch [root@dell-pe2950-01 ~]# rpm -qa |grep pulp-nodes pulp-nodes-common-2.3.0-0.37.beta.el6.noarch pulp-nodes-admin-extensions-2.3.0-0.37.beta.el6.noarch pulp-nodes-parent-2.3.0-0.37.beta.el6.noarch [root@dell-pe2950-01 ~]#
Pulp 2.3 released.
*** Bug 1042208 has been marked as a duplicate of this bug. ***