| Summary: | Synced repositories with upstream sha1 metadata are published with sha25 | ||
|---|---|---|---|
| Product: | [Retired] Pulp | Reporter: | Mike McCune <mmccune> |
| Component: | rpm-support | Assignee: | Barnaby Court <bcourt> |
| Status: | CLOSED CURRENTRELEASE | QA Contact: | Preethi Thomas <pthomas> |
| Severity: | unspecified | Docs Contact: | |
| Priority: | urgent | ||
| Version: | Master | CC: | achvatal, bcourt, jbaird, mhrivnak, pthomas |
| Target Milestone: | --- | Keywords: | Triaged |
| Target Release: | 2.3.0 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Bug Fix | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2013-12-09 14:31:33 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
|
Description
Mike McCune
2013-11-11 14:59:41 UTC
The idea we discussed during a call was to base the hashing on whatever is in the repomd.xml and use that for any published repos. fails-qa [root@ibm-hs22-04 ~]# rpm -qa |grep pulp-server pulp-server-2.3.0-0.32.beta.el6.noarch [root@ibm-hs22-04 ~]# I synced a rhel5 repo from cdn and looks like there still some sha256 data_type <repomd xmlns="http://linux.duke.edu/metadata/repo" xmlns:rpm="http://linux.duke.edu/metadata/rpm"> <revision>1384260872</revision> <data type="group"> <checksum type="sha256"> 9c3323c8c08883e83b476e3de5aa638919fb95ae4511d8b746d39aee2b945b78 </checksum> <open-checksum type="sha256"> 9c3323c8c08883e83b476e3de5aa638919fb95ae4511d8b746d39aee2b945b78 </open-checksum> <location href="repodata/9c3323c8c08883e83b476e3de5aa638919fb95ae4511d8b746d39aee2b945b78-group.xml"/> <timestamp>1384261066</timestamp> <size>1066257</size> </data> <data type="filelists"> <checksum type="sha">7ee6e0e701ec0d12429df3d398c71351ac3be4ce</checksum> <open-checksum type="sha">23c7862e1e2e6df94cee1b5f20b9dc0dbc1120bc</open-checksum> <location href="repodata/7ee6e0e701ec0d12429df3d398c71351ac3be4ce-filelists.xml.gz"/> <timestamp>1384260962</timestamp> <size>28815099</size> <open-size>469369703</open-size> </data> <data type="updateinfo"> <checksum type="sha256"> 2b4ad76dcf25955a28e1aea4bf6300deb8b9541caa26698e501b84c9f7b673d9 </checksum> <open-checksum type="sha256"> 2b4ad76dcf25955a28e1aea4bf6300deb8b9541caa26698e501b84c9f7b673d9 </open-checksum> <location href="repodata/2b4ad76dcf25955a28e1aea4bf6300deb8b9541caa26698e501b84c9f7b673d9-updateinfo.xml"/> <timestamp>1384261067</timestamp> <size>10816412</size> </data> <data type="productid"> <checksum type="sha256"> 7dcb43539ac47c63b6fc2647377430baedff49c6ccfc9d154c47d2957e3902db </checksum> <open-checksum type="sha256"> 7dcb43539ac47c63b6fc2647377430baedff49c6ccfc9d154c47d2957e3902db </open-checksum> <location href="repodata/7dcb43539ac47c63b6fc2647377430baedff49c6ccfc9d154c47d2957e3902db-productid.xml"/> <timestamp>1384261067</timestamp> <size>2159</size> </data> Looks like this still is an issue [root@ibm-hs22-04 ~]# rpm -qa pulp-server pulp-server-2.3.0-0.33.beta.el6.noarch [root@ibm-hs22-04 ~]# I installed the latest and synced the rhel5 repo from CDN still seeing this <repomd xmlns="http://linux.duke.edu/metadata/repo" xmlns:rpm="http://linux.duke.edu/metadata/rpm"> <revision>1384313137</revision> <data type="group"> <checksum type="sha256"> 9c3323c8c08883e83b476e3de5aa638919fb95ae4511d8b746d39aee2b945b78 </checksum> <open-checksum type="sha256"> 9c3323c8c08883e83b476e3de5aa638919fb95ae4511d8b746d39aee2b945b78 </open-checksum> <location href="repodata/9c3323c8c08883e83b476e3de5aa638919fb95ae4511d8b746d39aee2b945b78-group.xml"/> <timestamp>1384313320</timestamp> <size>1066257</size> </data> <data type="updateinfo"> <checksum type="sha256"> 2b4ad76dcf25955a28e1aea4bf6300deb8b9541caa26698e501b84c9f7b673d9 </checksum> <open-checksum type="sha256"> 2b4ad76dcf25955a28e1aea4bf6300deb8b9541caa26698e501b84c9f7b673d9 </open-checksum> <location href="repodata/2b4ad76dcf25955a28e1aea4bf6300deb8b9541caa26698e501b84c9f7b673d9-updateinfo.xml"/> <timestamp>1384313321</timestamp> <size>10816412</size> </data> <data type="productid"> <checksum type="sha256"> 7dcb43539ac47c63b6fc2647377430baedff49c6ccfc9d154c47d2957e3902db </checksum> <open-checksum type="sha256"> 7dcb43539ac47c63b6fc2647377430baedff49c6ccfc9d154c47d2957e3902db </open-checksum> <location href="repodata/7dcb43539ac47c63b6fc2647377430baedff49c6ccfc9d154c47d2957e3902db-productid.xml"/> <timestamp>1384313323</timestamp> <size>2159</size> </data> I have just confirmed that this has been fixed in following beta packages: 2.3.0-0.34.beta.el6 <repomd> <revision>1384455086</revision> − <data type="group"> <checksum type="sha1">5a6391f1a5b63cb8e82fcfea3b21810d57128627</checksum> <open-checksum type="sha1">5a6391f1a5b63cb8e82fcfea3b21810d57128627</open-checksum> <location href="repodata/5a6391f1a5b63cb8e82fcfea3b21810d57128627-group.xml"/> <timestamp>1384455251</timestamp> <size>1066257</size> </data> − <data type="filelists"> <checksum type="sha1">c0cb0b35d61b0544ad437f99cc4397527c6862d2</checksum> <open-checksum type="sha1">e3b960b6c43380888dc0c6fb7c1de600427e7476</open-checksum> <location href="repodata/c0cb0b35d61b0544ad437f99cc4397527c6862d2-filelists.xml.gz"/> <timestamp>1384455162</timestamp> <size>28758254</size> <open-size>468209725</open-size> </data> − <data type="updateinfo"> <checksum type="sha1">b2fe788ed1ed08ca346076240606cde39000a443</checksum> <open-checksum type="sha1">b2fe788ed1ed08ca346076240606cde39000a443</open-checksum> <location href="repodata/b2fe788ed1ed08ca346076240606cde39000a443-updateinfo.xml"/> <timestamp>1384455251</timestamp> <size>10816412</size> </data> − <data type="primary"> <checksum type="sha1">628c22f7fc52eea807379dc04f1a81851e774fd4</checksum> <open-checksum type="sha1">6704d6b1eef7516477b435c2b2a13c66b56f0377</open-checksum> <location href="repodata/628c22f7fc52eea807379dc04f1a81851e774fd4-primary.xml.gz"/> <timestamp>1384455139</timestamp> <size>2280441</size> <open-size>44755414</open-size> </data> − <data type="other"> <checksum type="sha1">ebd0685493c49893865202529f3ea812f1cd015e</checksum> <open-checksum type="sha1">9af56ec1da5857809a50d25edf9932c0a1d2e4ef</open-checksum> <location href="repodata/ebd0685493c49893865202529f3ea812f1cd015e-other.xml.gz"/> <timestamp>1384455163</timestamp> <size>3243340</size> <open-size>39150339</open-size> </data> − <data type="filelists_db"> <checksum type="sha1">d5cfe365c16b4211212601f3a8fed9861774534f</checksum> <open-checksum type="sha1">1205131180618bd24b4c672e2fd96c711e1806f0</open-checksum> <location href="repodata/d5cfe365c16b4211212601f3a8fed9861774534f-filelists.sqlite.bz2"/> <timestamp>1384455239</timestamp> <database_version>10</database_version> <size>20681206</size> <open-size>231503872</open-size> </data> − <data type="primary_db"> <checksum type="sha1">babd2036cb3b658c50fb36b4fe35e1be143a5aa2</checksum> <open-checksum type="sha1">0dea171a3b38ce8c2098117e83ff04c99023a7d7</open-checksum> <location href="repodata/babd2036cb3b658c50fb36b4fe35e1be143a5aa2-primary.sqlite.bz2"/> <timestamp>1384455251</timestamp> <database_version>10</database_version> <size>7953701</size> <open-size>53777408</open-size> </data> − <data type="other_db"> <checksum type="sha1">efed8083408bffb9c07bad360e48c12a3888878d</checksum> <open-checksum type="sha1">2336bec9cd3cd2a7287b49faf8533202859a1d7b</open-checksum> <location href="repodata/efed8083408bffb9c07bad360e48c12a3888878d-other.sqlite.bz2"/> <timestamp>1384455171</timestamp> <database_version>10</database_version> <size>4229916</size> <open-size>38763520</open-size> </data> − <data type="productid"> <checksum type="sha1">72b1d00d6b3729e9df2f3a5cc3cdd22b4baf2bfb</checksum> <open-checksum type="sha1">72b1d00d6b3729e9df2f3a5cc3cdd22b4baf2bfb</open-checksum> <location href="repodata/72b1d00d6b3729e9df2f3a5cc3cdd22b4baf2bfb-productid.xml"/> <timestamp>1384455252</timestamp> <size>2159</size> </data> </repomd> [root@rhel5-pulp ~]# pulp-consumer -u admin -p admin rpm bind --repo-id rhel59-os Bind tasks successfully created: Task Id: 5e120711-4ab7-4b25-87ab-776af6e41dd3 Task Id: 3e85702e-34c3-4004-8ffa-395a6fa639e1 [root@rhel5-pulp ~]# yum groupinstall "DNS Name Server" --disablerepo=* --enablerepo=rhel59-os Loaded plugins: rhnplugin, security Setting up Group Process Resolving Dependencies --> Running transaction check ---> Package bind.x86_64 30:9.3.6-20.P1.el5_8.6 set to be updated --> Processing Dependency: bind-libs = 30:9.3.6-20.P1.el5_8.6 for package: bind ---> Package bind-chroot.x86_64 30:9.3.6-20.P1.el5_8.6 set to be updated --> Running transaction check --> Processing Dependency: bind-libs = 30:9.3.6-20.P1.el5_8.1 for package: bind-utils ---> Package bind-libs.x86_64 30:9.3.6-20.P1.el5_8.6 set to be updated --> Running transaction check ---> Package bind-utils.x86_64 30:9.3.6-20.P1.el5_8.6 set to be updated --> Finished Dependency Resolution Dependencies Resolved ================================================================================ Package Arch Version Repository Size ================================================================================ Installing: bind x86_64 30:9.3.6-20.P1.el5_8.6 rhel59-os 989 k bind-chroot x86_64 30:9.3.6-20.P1.el5_8.6 rhel59-os 47 k Updating for dependencies: bind-libs x86_64 30:9.3.6-20.P1.el5_8.6 rhel59-os 898 k bind-utils x86_64 30:9.3.6-20.P1.el5_8.6 rhel59-os 180 k Transaction Summary ================================================================================ Install 2 Package(s) Upgrade 2 Package(s) Total download size: 2.1 M Is this ok [y/N]: y Downloading Packages: (1/4): bind-chroot-9.3.6-20.P1.el5_8.6.x86_64.rpm | 47 kB 00:00 (2/4): bind-utils-9.3.6-20.P1.el5_8.6.x86_64.rpm | 180 kB 00:00 (3/4): bind-libs-9.3.6-20.P1.el5_8.6.x86_64.rpm | 898 kB 00:00 (4/4): bind-9.3.6-20.P1.el5_8.6.x86_64.rpm | 989 kB 00:00 -------------------------------------------------------------------------------- Total 1.3 MB/s | 2.1 MB 00:01 Running rpm_check_debug Running Transaction Test Finished Transaction Test Transaction Test Succeeded Running Transaction Updating : bind-libs 1/6 Installing : bind 2/6 Installing : bind-chroot 3/6 Updating : bind-utils 4/6 Cleanup : bind-utils 5/6 Cleanup : bind-libs 6/6 Installed: bind.x86_64 30:9.3.6-20.P1.el5_8.6 bind-chroot.x86_64 30:9.3.6-20.P1.el5_8.6 Dependency Updated: bind-libs.x86_64 30:9.3.6-20.P1.el5_8.6 bind-utils.x86_64 30:9.3.6-20.P1.el5_8.6 Complete! [root@rhel5-pulp ~]# verified Also kickstarted rhel5 vm from the synced repo root@qeblade20 ~]# virt-install -n rhel5 --disk path=/dev/vg_qeblade20/rhel5 -l http://dell-pe2950-01.lab.eng.rdu.redhat.com/pulp/repos/content/dist/rhel/rhui/server/5/5.9/x86_64/os/ -r 4096 -w bridge:br0 --prompt --vnc This is still not working correctly on a child pulp node. The "scratchpad" attribute of the repository is not being propagated by nodes, and thus the distributor on the child node doesn't have the shared checksum info. build: 2.3.0-0.36.beta verified that the node child has the correct checksum [root@hp-dl120g5-01 ~]# rpm -qa |grep pulp-nodes-child pulp-nodes-child-2.3.0-0.36.beta.el6.noarch [root@hp-dl120g5-01 ~]# rpm -qa |grep pulp-server pulp-server-2.3.0-0.36.beta.el6.noarch [root@hp-dl120g5-01 ~]# Synced rehl5_9 on node parent node synced the rhel59 to the node child The checksum on the node child is below. <data type="filelists"> <checksum type="sha1">1bd1df19e39abe9531ec0b5d179778280e02dee9</checksum> <open-checksum type="sha1">59fac13f1a98bc42158ec084efb37933e09efa49</open-checksum> <location href="repodata/1bd1df19e39abe9531ec0b5d179778280e02dee9-filelists.xml.gz"/> <timestamp>1384645070</timestamp> <size>28819788</size> <open-size>469430490</open-size> </data> <data type="updateinfo"> <checksum type="sha1">b2fe788ed1ed08ca346076240606cde39000a443</checksum> <open-checksum type="sha1">b2fe788ed1ed08ca346076240606cde39000a443</open-checksum> <location href="repodata/b2fe788ed1ed08ca346076240606cde39000a443-updateinfo.xml"/> <timestamp>1384645158</timestamp> <size>10816412</size> </data> <data type="primary"> <checksum type="sha1">e13e66b3a7134c6ecd6e5947da1cc5725634567d</checksum> <open-checksum type="sha1">d5d7c3ab087597523de8f3f25fb5ce7af9918be6</open-checksum> <location href="repodata/e13e66b3a7134c6ecd6e5947da1cc5725634567d-primary.xml.gz"/> <timestamp>1384645049</timestamp> <size>2280527</size> <open-size>44757099</open-size> </data> <data type="other"> <checksum type="sha1">cf9456848399b2197b371bf3e22e9c1d51745265</checksum> <open-checksum type="sha1">d32351e2b0ed14416a857748fe38c14310f11f61</open-checksum> <location href="repodata/cf9456848399b2197b371bf3e22e9c1d51745265-other.xml.gz"/> <timestamp>1384645072</timestamp> <size>3243388</size> <open-size>39153368</open-size> </data> <data type="filelists_db"> <checksum type="sha1">c4442af8968409a6697c0554ccd4fc0f9c56eb0f</checksum> <open-checksum type="sha1">cb0e44f2a9ffff085a819fcc833d8fb34cd64a8b</open-checksum> <location href="repodata/c4442af8968409a6697c0554ccd4fc0f9c56eb0f-filelists.sqlite.bz2"/> <timestamp>1384645144</timestamp> <database_version>10</database_version> <size>20770626</size> <open-size>231911424</open-size> </data> <data type="primary_db"> <checksum type="sha1">cfc5818d294dccbd90331ac6ac86f0d5912cf8eb</checksum> <open-checksum type="sha1">6997fa58b1c719a0816d1a90756a37c5141ab38a</open-checksum> <location href="repodata/cfc5818d294dccbd90331ac6ac86f0d5912cf8eb-primary.sqlite.bz2"/> <timestamp>1384645156</timestamp> <database_version>10</database_version> <size>7948270</size> <open-size>53788672</open-size> </data> <data type="other_db"> <checksum type="sha1">796cd1d68e01713a2459885e65b981f5ff75231f</checksum> <open-checksum type="sha1">c59c937612e8d2b6dc382e98894be974b0e0b2c8</open-checksum> <location href="repodata/796cd1d68e01713a2459885e65b981f5ff75231f-other.sqlite.bz2"/> <timestamp>1384645080</timestamp> <database_version>10</database_version> <size>4227454</size> <open-size>38767616</open-size> </data> <data type="productid"> <checksum type="sha1">72b1d00d6b3729e9df2f3a5cc3cdd22b4baf2bfb</checksum> <open-checksum type="sha1">72b1d00d6b3729e9df2f3a5cc3cdd22b4baf2bfb</open-checksum> <location href="repodata/72b1d00d6b3729e9df2f3a5cc3cdd22b4baf2bfb-productid.xml"/> <timestamp>1384645158</timestamp> <size>2159</size> </data> </repomd> Taking it back to fails-qa Looks like repo copy does not keep the checksum of the parent steps to reproduce. 1. Copy the rhel5 repo to a new repo 2. Run repo publish There is a workaround for this though. When you create the new repo you can created/updated with --checksum-type option before running publish root@dell-pe2950-01 ~]# pulp-admin rpm repo create --repo-id zoo-checksum --checksum-type sha1 Successfully created repository [zoo-checksum] This is a challenging issue. We have a plan for a fix, but it will not be completely hands-off. The user will be responsible for copying config options from the distributor on the source repo to the distributor on the destination repo. Currently, the yum importer notices the use of sha1 upstream and makes a note of it in the repo scratchpad. The distributor then sees the note and publishes with sha1. The problem is that when copying content, attributes of the repo (such as its scratchpad) are not copied. This is by design, but makes this particular use case difficult. The solution is that when the distributor sees the note in the scratchpad, it will look at its own config, and if a checksum has not been set, it will save the checksum in its config. This allows someone who wants to create a copy of a repo to copy the distributor's config, and thus take advantage of this checksum detection. We already have APIs and UIs to support this. For a pulp-admin user, this will require creating the destination repo such that the distributor checksum matches what's in the source repo's distributor. To test the latest change: 1) do a sync and publish of a rhel5 repo 2) verify that the sha1 checksum was used for publish 3) verify that the distributor now has a setting for checksum type as sha1. As for a copy, it is the responsibility of the user doing to copy to see the setting on the source repo's distributor and copy that to the destination repo's distributor if it makes sense. It also wouldn't hurt to sync the repo to a child node, do a publish, and do steps 2 and 3 above on the child. [root@dell-pe2950-01 ~]# pulp-admin rpm repo list --details
+----------------------------------------------------------------------+
RPM Repositories
+----------------------------------------------------------------------+
Id: rhel59-os
Display Name: rhel59-os
Distributors:
Auto Publish: True
Config:
Checksum Type: sha1
Http: True
Https: True
Relative URL: /content/dist/rhel/rhui/server/5/5.9/x86_64/os/
Distributor Type Id: yum_distributor
Id: yum_distributor
Last Publish: 2013-11-17T11:10:54-05:00
Repo Id: rhel59-os
Scheduled Publishes:
Tested it on node as well.
Here is the status of the repo before the fix on node child
[root@hp-dl120g5-01 ~]# pulp-admin -u admin -p admin repo list
+----------------------------------------------------------------------+
Repositories
+----------------------------------------------------------------------+
Id: rhel59-os
Display Name: rhel59-os
Description: None
Content Unit Counts:
Distribution: 1
Erratum: 2768
Package Category: 6
Package Group: 99
Rpm: 15362
Yum Repo Metadata File: 1
[root@hp-dl120g5-01 ~]# pulp-admin -u admin -p admin repo list --details
+----------------------------------------------------------------------+
Repositories
+----------------------------------------------------------------------+
Id: rhel59-os
Display Name: rhel59-os
Description: None
Content Unit Counts:
Distribution: 1
Erratum: 2768
Package Category: 6
Package Group: 99
Rpm: 15362
Yum Repo Metadata File: 1
Notes:
Importers:
Config:
Manifest URL: https://dell-pe2950-01.lab.eng.rdu.redhat.com/pulp/nodes/https
/repos/rhel59-os/manifest.json
Strategy: additive
Id: nodes_http_importer
Importer Type Id: nodes_http_importer
Last Sync: 2013-11-16T18:32:56-05:00
Repo Id: rhel59-os
Scheduled Syncs:
Distributors:
Auto Publish: True
Config:
Http: True
Https: True
Relative URL: /content/dist/rhel/rhui/server/5/5.9/x86_64/os/
Distributor Type Id: yum_distributor
Id: yum_distributor
Last Publish: 2013-11-16T18:39:19-05:00
Repo Id: rhel59-os
Scheduled Publishes:
Auto Publish: False
Config:
Http: True
Https: True
Distributor Type Id: export_distributor
Id: export_distributor
Last Publish: None
Repo Id: rhel59-os
Scheduled Publishes:
[root@hp-dl120g5-01 ~]# service httpd restart
Stopping httpd: [ OK ]
Starting httpd: [ OK ]
[root@hp-dl120g5-01 ~]# service pulp-agent restart
Stopping goferd[ OK ]
Starting goferd[ OK ]
After the update on node parent & child
And ran node sync
[root@dell-pe2950-01 ~]# pulp-admin node sync run --node-id node1
This command may be exited via ctrl+c without affecting the request.
[-]
Waiting to begin...
(1/1) Repository: rhel59-os
[==================================================] 100%
Synchronization succeeded
+----------------------------------------------------------------------+
Child Node Synchronization
+----------------------------------------------------------------------+
Repository:
Action: Merged
Id: rhel59-os
Units:
Added: 0
Removed: 0
Updated: 107
See the updated checksum field on the the child repo
[root@hp-dl120g5-01 ~]# pulp-admin -u admin -p admin repo list --details
+----------------------------------------------------------------------+
Repositories
+----------------------------------------------------------------------+
Id: rhel59-os
Display Name: rhel59-os
Description: None
Content Unit Counts:
Distribution: 1
Erratum: 2768
Package Category: 6
Package Group: 99
Rpm: 15362
Yum Repo Metadata File: 1
Notes:
Importers:
Config:
Manifest URL: https://dell-pe2950-01.lab.eng.rdu.redhat.com/pulp/nodes/https
/repos/rhel59-os/manifest.json
Strategy: additive
Id: nodes_http_importer
Importer Type Id: nodes_http_importer
Last Sync: 2013-11-19T13:05:17-05:00
Repo Id: rhel59-os
Scheduled Syncs:
Distributors:
Auto Publish: True
Config:
Checksum Type: sha1
Http: True
Https: True
Relative URL: /content/dist/rhel/rhui/server/5/5.9/x86_64/os/
Distributor Type Id: yum_distributor
Id: yum_distributor
Last Publish: 2013-11-16T18:39:19-05:00
Repo Id: rhel59-os
Scheduled Publishes:
Auto Publish: False
Config:
Http: True
Https: True
Distributor Type Id: export_distributor
Id: export_distributor
Last Publish: None
Repo Id: rhel59-os
Scheduled Publishes:
[root@hp-dl120g5-01 ~]#
above tests are with build 2.3.0-0.37.beta verified [root@dell-pe2950-01 ~]# rpm -qa |grep pulp-server pulp-server-2.3.0-0.37.beta.el6.noarch [root@dell-pe2950-01 ~]# rpm -qa |grep pulp-nodes pulp-nodes-common-2.3.0-0.37.beta.el6.noarch pulp-nodes-admin-extensions-2.3.0-0.37.beta.el6.noarch pulp-nodes-parent-2.3.0-0.37.beta.el6.noarch [root@dell-pe2950-01 ~]# Pulp 2.3 released. *** Bug 1042208 has been marked as a duplicate of this bug. *** |