Bug 1934545

Summary: pulp3: Content View publish with filters and depsolve=true generates assert failures
Product: Red Hat Satellite Reporter: Pavel Moravec <pmoravec>
Component: PulpAssignee: satellite6-bugs <satellite6-bugs>
Status: CLOSED DUPLICATE QA Contact: Lai <ltran>
Severity: high Docs Contact:
Priority: high    
Version: 6.9.0CC: dalley, ggainey, jjeffers, jsherril, myarboro, rchan, ryandeussing, smallamp, ttereshc
Target Milestone: 6.10.0Keywords: Triaged
Target Release: Unused   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Known Issue
Doc Text:
Cause: Not all dependencies can be resolved Solution: Ensure all necessary repositories are included, so all dependencies can be resolved
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-09-13 17:42:22 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Pavel Moravec 2021-03-03 13:02:49 UTC
Description of problem:
Playing with pulp-3 on Sat6.9 beta, with CV filters. Few basic scenarios fail most probably on the same root cause, where Actions::Pulp3::Repository::MultiCopyContent dynflow step fails in pulp sub-step/task pulp_rpm.app.tasks.copy.copy_content with no traceback but failed assert:

Mar  2 11:55:55 pmoravec-sat69-beta pulpcore-worker-3: python3: ../src/rules.c:261: solver_addrule: Assertion `!p2 && d > 0' failed.


Version-Release number of selected component (if applicable):
Sat 6.9 beta:
- katello-3.18.1-2.el7sat.noarch
- python3-pulpcore-3.7.3-1.el7pc.noarch
- python3-pulp-rpm-3.7.0-1.el7pc.noarch


How reproducible:
100%


Steps to Reproduce:
1. Sync RHEL8 appstream repo
2. Create CV with:
- "Solve Dependencies" set to true (this is _must_ for the reproducer)
- add there that repo (optionally some other, but stay with minimalistic reproducer)
- and add 3 filters:
  - include RPMs with no erratum
  - include errata updated until 2019-12-31
  - include modulestreams with no erratum
(see below for a bash script to add those filters automatically, it saved me a lot of WebUI clicking)
3. Publish the CV

I played with few other repos for the artificial "one repo CV", and below ones fail the same:
- RHEL7 optional



Actual results:
- Publish fails on dynflow step Actions::Pulp3::Repository::MultiCopyContent
- in particular task pulp_rpm.app.tasks.copy.copy_content fails with no traceback but failed assert:

Mar  2 11:55:55 pmoravec-sat69-beta pulpcore-worker-3: python3: ../src/rules.c:261: solver_addrule: Assertion `!p2 && d > 0' failed.


Expected results:
CV publish to complete well


Additional info:
script to add the filters:

hmr="hammer -u admin -p changeme "

CVNAME=$1

$hmr content-view filter create --content-view=${CVNAME} --name=include_base --inclusion=true --original-packages=true --type=rpm

$hmr content-view filter create --content-view=${CVNAME} --name=include_errata --inclusion=true --type=erratum
$hmr content-view filter rule create --content-view=${CVNAME} --content-view-filter=include_errata --date-type='updated' --end-date='2019-12-31'

$hmr content-view filter create --content-view=${CVNAME} --name=include_modules --inclusion=true --original-module-streams=true --type=modulemd

Comment 1 Daniel Alley 2021-03-24 16:22:25 UTC
What version of libsolv and python-libsolv are installed?

Comment 2 Pavel Moravec 2021-03-24 18:55:35 UTC
(In reply to Daniel Alley from comment #1)
> What version of libsolv and python-libsolv are installed?

Just libsolv-0.7.12-2.el7pc.x86_64 from rhel-server-7-satellite-6-beta-rpms, no python-libsolv. Will send credentials to my reproducer in a private update.

Comment 7 Daniel Alley 2021-05-03 16:33:30 UTC
I've been investigating this, and the most obvious problem is that the original scenarios described invoke what is effectively "undefined behavior" (in a logical sense, not a literal "C" sense). Nearly every package in the Appstream repo contains some kind of dependency on packages in BaseOS (think: libc, /bin/bash, /bin/python). SAT solvers such as libsolv, used by Pulp and DNF, simply cannot handle this situation gracefully unless *all* possible dependencies are available to the solver.

This means that depsolving-enabled copies involving AppStream need to include BaseOS as part of the copy repo set.  And likewise, RHEL 7 Optional extends the base RHEL 7 repositories, and so copying packages from RHEL 7 Optional with depsolving enabled needs to include the RHEL 7 repositories as part of the copy repo set.

If this isn't done, then it is highly likely that even if the copy succeeds, the results will be incorrect. In this case, it appears that it does not succeed. In my experimentation, in both cases, large quantities (pages worth) of logs are printed about the inability to properly resolve all dependencies.

I think the action items here are:

* Improve the UI such that users are warned about, or prevented from making, dependency-incomplete copies such as this.
* Attempt to make Pulp's behavior in such scenarios more consistent and better documented. Make the error messages cleaner and more helpful.

Comment 9 mark yarborough 2021-05-04 12:54:27 UTC
Agreed on stand-up to propose for 6.10.

Comment 16 pulp-infra@redhat.com 2021-09-10 19:06:27 UTC
The Pulp upstream bug status is at NEW. Updating the external tracker on this bug.

Comment 17 pulp-infra@redhat.com 2021-09-10 19:06:29 UTC
The Pulp upstream bug priority is at High. Updating the external tracker on this bug.

Comment 18 Grant Gainey 2021-09-13 17:42:22 UTC
Closing as a dup, this is one symptom of an underlying issue to be resolved.

*** This bug has been marked as a duplicate of bug 2003764 ***

Comment 19 pulp-infra@redhat.com 2021-09-13 18:09:37 UTC
The Pulp upstream bug status is at CLOSED - DUPLICATE. Updating the external tracker on this bug.

Comment 20 pulp-infra@redhat.com 2021-10-11 20:08:54 UTC
The Pulp upstream bug status is at CLOSED - DUPLICATE. Updating the external tracker on this bug.

Comment 21 pulp-infra@redhat.com 2021-10-11 20:08:56 UTC
The Pulp upstream bug priority is at High. Updating the external tracker on this bug.