Description of problem: Sat6.9 with pulp-3 enabled has some depsolving bug. User story of testing pulp-3: Having a CV with: - multiple repos incl. RHEL7 one - filters frozen on a given date - dependency solving enabled for the CV, the CV publish fails on dependencies among the RHEL7. Narrowing down the reproducer, having just a CV with just RHEL7 repo with filters: - include all RPMs with no errata - include all errata published before 2021-05-01 (- include all modules until the same date - this can be almost surely ignored) Then publishing the CV raises 2k+ warnings - some valid, but some invalid. Valid: May 29 03:00:22 pmoravec-pulp3 pulpcore-worker-3: pulp: pulp_rpm.app.depsolving:WARNING: Encountered problems solving dependencies, copy may be incomplete: nothing provides pcsc-lite-devel(x86-64) needed by java-1.8.0-openjdk-headless-1:1.8.0.201.b09-0.el7_6.x86_64 (pcsc-lite-devel is in RHEL7 debug, I think, but _not_ in RHEL7 repo - this warning is valid). Invalid: May 29 02:22:44 pmoravec-pulp3 pulpcore-worker-3: pulp: pulp_rpm.app.depsolving:WARNING: Encountered problems solving dependencies, copy may be incomplete: package libvirt-0:4.5.0-36.el7_9.2.x86_64 requires libvirt-daemon-driver-lxc = 0:4.5.0-36.el7_9.2, but none of the providers can be installed (since libvirt-daemon-driver-lxc in that version is in RHBA-2020:4070 published 2020-09-29) When publishing the same CV in Sat6.9 with pulp-2, no such warning is printed (is the warning pulp-3 specific?), and the libvirt-daemon-driver-lxc package _is_ present in the CV/repo. Another invalid warning: May 28 20:20:49 pmoravec-pulp3 pulpcore-worker-3: pulp: pulp_rpm.app.depsolving:WARNING: Encountered problems solving dependencies, copy may be incomplete: package boost-timer-0:1.53.0-28.el7.x86_64 requires boost-system(x86-64) = 0:1.53.0-28.el7, but none of the providers can be installed Version-Release number of selected component (if applicable): Sat6.9.2 libsolv-0.7.17-1.el7pc.x86_64 python3-rq-1.5.2-2.el7pc.noarch python3-pulpcore-3.7.6-1.el7pc.noarch python3-pulp-rpm-3.10.0-1.el7pc.noarch How reproducible: 100% Steps to Reproduce: 1. Sync RHEL7 repo from CDN 2. Create and publish a CV like described above, or with hammer commands in detail: content-view create --name CV_RHEL7_yes_include_2021-05-01 --organization-id=1 --repository-ids=1 --solve-dependencies=yes content-view filter create --organization-id=1 --content-view=CV_RHEL7_yes_include_2021-05-01 --name=include_base --inclusion=true --original-packages=true --type=rpm content-view filter create --organization-id=1 --content-view=CV_RHEL7_yes_include_2021-05-01 --name=include_errata --inclusion=true --type=erratum content-view filter rule create --organization-id=1 --content-view=CV_RHEL7_yes_include_2021-05-01 --content-view-filter=include_errata --date-type='updated' --end-date='2021-05-01' content-view filter create --organization-id=1 --content-view=CV_RHEL7_yes_include_2021-05-01 --name=include_modules --inclusion=true --original-module-streams=true --type=modulemd content-view publish --name CV_RHEL7_yes_include_2021-05-01 --organization-id=1 --async 3. Wait few hours and check /var/log/messages Actual results: Some invalid warnings of unresolved dependencies are raised. Since I failed to successfully publish the CV due to bz1965936, I can't prove (now) the published content of the CV/repo. But I expect the warnings suggest some expected packages will be missing. Expected results: No invalid warnings raised, CV content to have all expected packages (incl. the libvirt-daemon-driver-lxc-0:4.5.0-36.el7_9.2 and boost-system-0:1.53.0-28.el7). Additional info:
Daniel, Grant, one more item to investigate for the dep solving area. Thank you!
I can reproduce the warnings in upstream-pulp3 by: * syncing RHEL7 * finding all RPMs not associated with an Advisory (ie, only (allegedly) RPMs released at RHEL7.0GA) * copying just that list to a new repo, depsolving=true We absolutely have had RPMs released into repos without advisories. This is a never-happen case...yet here we are. It will take some time to decide if that's what we're seeing here. In the meanwhile - following this recipe on a loaner-Sat-Latest, I get the the many-warnings - but more importantly, the initial copytask fails with this error message: === pulpcore-worker-4[1160]: python3: ../src/rules.c:261: solver_addrule: Assertion `!p2 && d > 0' failed. pulpcore-worker-4[1160]: pulp [None]: pulpcore.tasking.pulpcore_worker:INFO: Cleaning up and canceling Task b4d6e7ec-bcff-4663-9ef5-599bd3bce24b === and the task is marked **CANCELED**, with the remaining 3 copy-tasks waiting-forever. I haven't been able to make that happen in pulp-upstream. Investigation notes can be found here : https://hackmd.io/@ggainey/depsolving-issues Some useful debugging scripting can be found starting at https://hackmd.io/@ggainey/depsolving-issues#Pulling-useful-info-from-journalctl Investigation continues.
Quick note on investigation: The reproducer given is actually copying **the entire repo**, explicitly listing everything, and depsolving the whole thing one entity at a time; it's the worst-possible-case for copy. (Note that it **should** work, and there are def things to fix here - it's just A Lot...) The advisory-filter is "including" advisories from before-a-date. The rpm-filter is "including base-rpms", with no other qualifiers. This results in **all** rpms, and **all** advisories (no-filters copies everything, and both of these filters don't **limit** anything). To limit content to "everything from before a given date", you prob want just an **exclude** advisory-filter, that excludes everything **after** the date-of-interest. The modularity-filter doesn't do anything - RHEL7 doesn't have modules. At any rate - there is definitely at least one problem exposed here, where depsolving isn't handling multiarch correctly. It does not, however, explain the same-arch "not found" entries. Investigation continues.
The Pulp upstream bug status is at POST. Updating the external tracker on this bug.
The Pulp upstream bug priority is at Normal. Updating the external tracker on this bug.
The Pulp upstream bug status is at MODIFIED. Updating the external tracker on this bug.
The Pulp upstream bug status is at CLOSED - CURRENTRELEASE. Updating the external tracker on this bug.
All upstream Pulp bugs are at MODIFIED+. Moving this bug to POST.
Verified in 6.10 snap 21 Content View successfully publishes with dependency solving after syncing RHEL7 repo. Steps to Reproduce: 1. Sync RHEL7 repo from CDN 2. Create and publish a CV like described above, or with hammer commands in detail: hammer content-view create --name CV_RHEL7_yes_include_2021-05-01 --organization-id=1 --repository-ids=1 --solve-dependencies=yes hammer content-view filter create --organization-id=1 --content-view=CV_RHEL7_yes_include_2021-05-01 --name=include_base --inclusion=true --original-packages=true --type=rpm hammer content-view filter create --organization-id=1 --content-view=CV_RHEL7_yes_include_2021-05-01 --name=include_errata --inclusion=true --type=erratum hammer content-view filter rule create --organization-id=1 --content-view=CV_RHEL7_yes_include_2021-05-01 --content-view-filter=include_errata --date-type='updated' --end-date='2021-05-01' hammer content-view filter create --organization-id=1 --content-view=CV_RHEL7_yes_include_2021-05-01 --name=include_modules --inclusion=true --original-module-streams=true --type=modulemd hammer content-view publish --name CV_RHEL7_yes_include_2021-05-01 --organization-id=1 --async 3. Wait for publish then check /var/log/messages Expected Results: Content View publish shows no warnings or errors. Actual Results: Content View publish shows no warnings or errors.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: Satellite 6.10 Release), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2021:4702