Created attachment 958766 [details] traceback This is the traceback
Created attachment 959200 [details] mongo-queries
Comment on attachment 959200 [details] mongo-queries Looks like there are several units with the same NEVRA, checksum and storage_path. This attachment illustrates this using DB queries on the affected system.
Looks like the unit key includes the checksum type which I did not account for in my queries. Given that one unit has a checksum type of 'sha' and the other has 'sha1', they do have unique unit keys. There still may be an issue with calculation of the storage path though. > db.units_rpm.find({_storage_path:'/var/lib/pulp/content/rpm/ruby-augeas/0.4.1/1.el5sat/x86_64/44ce2c0c6a1e02659477b3bc879ba67592e525c6/ruby-augeas-0.4.1-1.el5sat.x86_64.rpm'}, {name:1, version:1, release:1,arch:1,epoch:1,checksum:1, checksumtype:1 }).pretty() { "_id" : "b82d05f3-4b23-4aeb-a16a-bf78bd55fe36", "epoch" : "0", "version" : "0.4.1", "arch" : "x86_64", "name" : "ruby-augeas", "checksumtype" : "sha", "checksum" : "44ce2c0c6a1e02659477b3bc879ba67592e525c6", "release" : "1.el5sat" } { "_id" : "5be2e7ef-c34a-4fa3-8aa6-92d7a746b085", "epoch" : "0", "version" : "0.4.1", "arch" : "x86_64", "name" : "ruby-augeas", "checksumtype" : "sha1", "checksum" : "44ce2c0c6a1e02659477b3bc879ba67592e525c6", "release" : "1.el5sat" }
Shouldn't these be the same units? Does pulp really treat upstream repos that use sha vs sha1 differently?
I finally have a way to reproduce this bug that is (sort of) simple: 1) Create two repos, perhaps sha-bear and sha1-bear: $ pulp-admin rpm repo create --repo-id sha-bear --relative-path sha-bear $ pulp-admin rpm repo create --repo-id sha1-bear --relative-path sha1-bear 2) Upload the same RPM to both repos. I used the bear package from our zoo demo repo: $ pulp-admin rpm repo uploads rpm -f ./bear-4.1-1.noarch.rpm --repo-id sha-bear $ pulp-admin rpm repo uploads rpm -f ./bear-4.1-1.noarch.rpm --repo-id sha1-bear 3) pulp-admin rpm repo {create,update} has a --checksum-type flag, but it does not seem to work[0]. To workaround this issue, we're going to publish the two repos above, and then we're going to hand edit their published metadata. We'll start by publishing both repos: $ pulp-admin rpm repo publish run --repo-id sha-bear $ pulp-admin rpm repo publish run --repo-id sha1-bear 4) Next, we will navigate to the published output of these two repositories, edit the *primary.xml.gz files, and switch the checksum types and values. For the sha repo, we will write "sha" as the checksum. For the sha1, we will write "sha1". For example, I'll get the checksum on bear this way: $ shasum ./bear-4.1-1.noarch.rpm 7cc8894d84696bfac328a0f7104daec7cbb0f5c4 ./bear-4.1-1.noarch.rpm 5) Now let's edit the sha repo: # vim /var/lib/pulp/published/yum/https/repos/sha-bear/repodata/*-primary.xml.gz 6) Find the checksum tag for your package, and replace both the type and the value with sha and whatever value the shasum binary gave you: <checksum type="sha" pkgid="YES">7cc8894d84696bfac328a0f7104daec7cbb0f5c4</checksum> 7) Now let's edit the sha1 repo: # vim /var/lib/pulp/published/yum/https/repos/sha1-bear/repodata/*-primary.xml.gz 8) Find the checksum tag for your package, and replace both the type and the value with sha1 and whatever value the shasum binary gave you: <checksum type="sha1" pkgid="YES">7cc8894d84696bfac328a0f7104daec7cbb0f5c4</checksum> 9) Next you will need to create a "confused" repo. We will set the initial feed to the sha-bear repo's published path. This is the repo that is going to experience this problem: $ pulp-admin rpm repo create --repo-id confused-bear --feed file:///var/lib/pulp/published/yum/https/repos/sha-bear 10) Sync the confused repo, which will bring in our sha checksummed package: $ pulp-admin rpm repo sync run --repo-id confused-bear 11) Now we will change the feed to the sha1 repo: $ pulp-admin rpm repo update --repo-id confused-bear --feed file:///var/lib/pulp/published/yum/https/repos/sha1-bear 12) Now we will sync the confused repo again, bringing it into its confused state: $ pulp-admin rpm repo sync run --repo-id confused-bear 13) At this point, the confused-bear repo has gotten into the problematic state. It has the bear-4.1-1.noarch.rpm package in it twice with the same checksum, but it thinks the checksum type is different. The yum importer and distributor are both OK with this, but Nodes is not! If you haven't done this already, follow these instructions to enable your repo as a parent Node: https://pulp-user-guide.readthedocs.org/en/latest/nodes.html#parent 14) Next, enable the confused repo as a nodes repo: $ pulp-admin node repo enable --repo-id confused-bear 15) Attempt to publish the Node repo: $ pulp-admin node repo publish --repo-id confused-zoo This command may be exited via ctrl+c without affecting the request. [|] Running... Publish failed. See: pulp log for details. If you look in the logs, you will see the same traceback that is attached to this report. [0] https://bugzilla.redhat.com/show_bug.cgi?id=1167456
Maybe I shouldn't call any reproduction case with 15 steps "simple"…
I have filed a new Pulp bug that is about the root cause of this specific traceback, but I have a way to fix this bug without fixing that one. https://bugzilla.redhat.com/show_bug.cgi?id=1168029
https://github.com/pulp/pulp/pull/1382 https://github.com/pulp/pulp_rpm/pull/602
1382 was submitted against the wrong branch. Here is the correct pull request: https://github.com/pulp/pulp/pull/1391
This is fixed in 2.4.4-0.1.beta.
(In reply to Justin Sherrill from comment #5) > Shouldn't these be the same units? Does pulp really treat upstream repos > that use sha vs sha1 differently? Yeah, Pulp really did do that ☹ But my PRs fixed it!
>> rpm -qa | grep pulp-server pulp-server-2.6.0-0.1.alpha.el7.noarch >> pulp-admin rpm repo create --repo-id sha-bear --relative-url sha-bear Successfully created repository [sha-bear] >> pulp-admin rpm repo create --repo-id sha1-bear --relative-url sha1-bear Successfully created repository [sha1-bear] >> pulp-admin rpm repo uploads rpm -f ./bear-4.1-1.noarch.rpm --repo-id sha-bear +----------------------------------------------------------------------+ Unit Upload +----------------------------------------------------------------------+ Extracting necessary metadata for each request... [==================================================] 100% Analyzing: bear-4.1-1.noarch.rpm ... completed Creating upload requests on the server... [==================================================] 100% Initializing: bear-4.1-1.noarch.rpm ... completed Starting upload of selected units. If this process is stopped through ctrl+c, the uploads will be paused and may be resumed later using the resume command or cancelled entirely using the cancel command. Uploading: bear-4.1-1.noarch.rpm [==================================================] 100% 2438/2438 bytes ... completed Importing into the repository... This command may be exited via ctrl+c without affecting the request. [\] Running... Task Succeeded Deleting the upload request... ... completed >> pulp-admin rpm repo uploads rpm -f ./bear-4.1-1.noarch.rpm --repo-id sha1-bear +----------------------------------------------------------------------+ Unit Upload +----------------------------------------------------------------------+ Extracting necessary metadata for each request... [==================================================] 100% Analyzing: bear-4.1-1.noarch.rpm ... completed Creating upload requests on the server... [==================================================] 100% Initializing: bear-4.1-1.noarch.rpm ... completed Starting upload of selected units. If this process is stopped through ctrl+c, the uploads will be paused and may be resumed later using the resume command or cancelled entirely using the cancel command. Uploading: bear-4.1-1.noarch.rpm [==================================================] 100% 2438/2438 bytes ... completed Importing into the repository... This command may be exited via ctrl+c without affecting the request. [\] Running... Task Succeeded Deleting the upload request... ... completed >> pulp-admin rpm repo publish run --repo-id sha-bear +----------------------------------------------------------------------+ Publishing Repository [sha-bear] +----------------------------------------------------------------------+ This command may be exited via ctrl+c without affecting the request. Initializing repo metadata [-] ... completed Publishing Distribution files [-] ... completed Publishing RPMs [-] ... completed Publishing Delta RPMs ... skipped Publishing Errata [-] ... completed Publishing Comps file [-] ... completed Publishing Metadata. [-] ... completed Closing repo metadata [-] ... completed Generating sqlite files ... skipped Publishing files to web [-] ... completed Writing Listings File [-] ... completed Task Succeeded >> pulp-admin rpm repo publish run --repo-id sha1-bear +----------------------------------------------------------------------+ Publishing Repository [sha1-bear] +----------------------------------------------------------------------+ This command may be exited via ctrl+c without affecting the request. Initializing repo metadata [-] ... completed Publishing Distribution files [-] ... completed Publishing RPMs [-] ... completed Publishing Delta RPMs ... skipped Publishing Errata [-] ... completed Publishing Comps file [-] ... completed Publishing Metadata. [-] ... completed Closing repo metadata [-] ... completed Generating sqlite files ... skipped Publishing files to web [-] ... completed Writing Listings File [-] ... completed Task Succeeded >> pulp-admin rpm repo list +----------------------------------------------------------------------+ RPM Repositories +----------------------------------------------------------------------+ Id: sha-bear Display Name: sha-bear Description: None Content Unit Counts: Rpm: 1 Id: sha1-bear Display Name: sha1-bear Description: None Content Unit Counts: Rpm: 1 >> sha1sum ./bear-4.1-1.noarch.rpm 7cc8894d84696bfac328a0f7104daec7cbb0f5c4 ./bear-4.1-1.noarch.rpm >> vim /var/lib/pulp/published/yum/https/repos/sha-bear/repodata/*-primary.xml.gz >> sha1sum ./bear-4.1-1.noarch.rpm 7cc8894d84696bfac328a0f7104daec7cbb0f5c4 ./bear-4.1-1.noarch.rpm >> vim /var/lib/pulp/published/yum/https/repos/sha1-bear/repodata/*-primary.xml.gz >> pulp-admin rpm repo create --repo-id confused-bear --feed file:///var/lib/pulp/published/yum/https/repos/sha-bear Successfully created repository [confused-bear] >> pulp-admin rpm repo sync run --repo-id confused-bear +----------------------------------------------------------------------+ Synchronizing Repository [confused-bear] +----------------------------------------------------------------------+ This command may be exited via ctrl+c without affecting the request. Downloading metadata... [-] ... completed Downloading repository content... [==================================================] 100% RPMs: 0/0 items Delta RPMs: 0/0 items ... completed Downloading distribution files... [==================================================] 100% Distributions: 0/0 items ... completed Importing errata... [-] ... completed Importing package groups/categories... [-] ... completed Task Succeeded Initializing repo metadata [-] ... completed Publishing Distribution files [-] ... completed Publishing RPMs [-] ... completed Publishing Delta RPMs ... skipped Publishing Errata [-] ... completed Publishing Comps file [-] ... completed Publishing Metadata. [-] ... completed Closing repo metadata [-] ... completed Generating sqlite files ... skipped Publishing files to web [-] ... completed Writing Listings File [-] ... completed Task Succeeded >> pulp-admin rpm repo update --repo-id confused-bear --feed file:///var/lib/pulp/published/yum/https/repos/sha1-bear Repository [confused-bear] successfully updated >> pulp-admin rpm repo sync run --repo-id confused-bear +----------------------------------------------------------------------+ Synchronizing Repository [confused-bear] +----------------------------------------------------------------------+ This command may be exited via ctrl+c without affecting the request. Downloading metadata... [-] ... completed Downloading repository content... [==================================================] 100% RPMs: 0/0 items Delta RPMs: 0/0 items ... completed Downloading distribution files... [==================================================] 100% Distributions: 0/0 items ... completed Importing errata... [-] ... completed Importing package groups/categories... [-] ... completed Task Succeeded Copying files [-] ... completed Initializing repo metadata [-] ... completed Publishing Distribution files [-] ... completed Publishing RPMs [-] ... completed Publishing Delta RPMs ... skipped Publishing Errata [-] ... completed Publishing Comps file [-] ... completed Publishing Metadata. [-] ... completed Closing repo metadata [-] ... completed Generating sqlite files ... skipped Publishing files to web [-] ... completed Writing Listings File [-] ... completed Task Succeeded >> pulp-admin node repo enable --repo-id confused-bear Repository enabled. Note: Repository [ confused-bear ] will not be available for node synchronization until published. See: the 'node repo publish' command. Warning: enabling with auto-publish may degrade repository synchronization performance. >> pulp-admin node repo publish --repo-id confused-bear This command may be exited via ctrl+c without affecting the request. [\] Running... Publish succeeded.
My shame I checked it over 2.6 rhel7. I have just checked it on pulp-server-2.4.4-0.1.beta.el7.noarc, I will not post such a big comment with all output, evrth went fine.
Checked the same for srpm and drpm. It is fine.
This bug has a target release of 2.4.4, but it seems that we will not release a 2.4.4 upstream. According to git[0], the first upstream release that contained this fix was pulp-2.5.1-1, so I am adjusting the target release and closing the bug. [0] $ git tag --contains 8024fcb
I am reopening this BZ as having a customer hitting it on fixed pulp. checksum type was "sha" or "sha1" everytime in their case: I added simple debug: logger.info("checksum_type=%s\n" % checksum_type) into sanitize_checksum_type method and that printed sha or sha1 only. I will attach foreman-debug, sosreport (both few days old) and recent /var/log/messages and journalctl today output with pulp debugs. Randy (or anybody else), could you pls. review the logs and see what could be wrong?
(In reply to Pavel Moravec from comment #23) > I am reopening this BZ as having a customer hitting it on fixed pulp. > checksum type was "sha" or "sha1" everytime in their case: I added simple > debug: > > logger.info("checksum_type=%s\n" % checksum_type) > > into sanitize_checksum_type method and that printed sha or sha1 only. > > I will attach foreman-debug, sosreport (both few days old) and recent > /var/log/messages and journalctl today output with pulp debugs. > > > > Randy (or anybody else), could you pls. review the logs and see what could > be wrong? Forgot to mention: the problems started to appear _after_ upgrading to pulp-server-2.4.4-1.el7sat
Hi Pavel, Can you confirm that the httpd, pulp_workers, pulp_celerybeat, pulp_resource_manager, and goferd processes on the parent and child node(s) were all stopped while the migration was performed?
(In reply to Randy Barlow from comment #29) > Hi Pavel, > > Can you confirm that the httpd, pulp_workers, pulp_celerybeat, > pulp_resource_manager, and goferd processes on the parent and child node(s) > were all stopped while the migration was performed? Hi Randy, yes, we double-checked the procedure from the errata has been strictly followed.
(In reply to Pavel Moravec from comment #30) > (In reply to Randy Barlow from comment #29) > > Hi Pavel, > > > > Can you confirm that the httpd, pulp_workers, pulp_celerybeat, > > pulp_resource_manager, and goferd processes on the parent and child node(s) > > were all stopped while the migration was performed? > > Hi Randy, yes, we double-checked the procedure from the errata has been > strictly followed. Any idea what could be wrong? Do you need some more data to troubleshoot? Thanks in advance for having a look.
I believe this issue is resolved. Please re-open if it returns!