Bug 1994256

Summary: Publishing CV with depsolving enabled on RHEL8 BaseOS repo and including all except new errata fails on package dependencies
Product: Red Hat Satellite Reporter: Pavel Moravec <pmoravec>
Component: PulpAssignee: satellite6-bugs <satellite6-bugs>
Status: CLOSED ERRATA QA Contact: Lai <ltran>
Severity: medium Docs Contact:
Priority: medium    
Version: 6.10.0CC: ahumbe, bbuckingham, dalley, ggainey, pmendezh, smallamp, ttereshc
Target Milestone: 6.10.0Keywords: Triaged
Target Release: Unused   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-11-16 14:13:10 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
list of all depsolving issues for the one pulp task none

Description Pavel Moravec 2021-08-17 07:19:17 UTC 
Description of problem:
Having CV with:
- RHEL 8 BaseOS repo (optionally other repos)
- dependency solving enabled
- include filters to include:
  - all packages without errata
  - all module streams without an errata
  - all errata released before 2019-01-01

Then publishing this CV fails on Actions::Pulp3::Repository::MultiCopyContent .

The cause there are unresolved dependencies (see below), that raise some warnings in /var/log/messages. _And_ they put the pulp copy_content task to cancelled state. That is understood by dynflow as a failure, causing the dynflow step is in error and needs to be skipped.

The above might be correct behaviour, though leading into "Satellite has this error" user experience. Hence we should either fix this, or review and explain the behaviour (like I started in https://access.redhat.com/solutions/6264721).

In the later case "it is not a bug but feature", please review the KCS and fill there the explanations why individual dependencies are broken (and if it is "OK").


Version-Release number of selected component (if applicable):
Sat 6.10 snap 13:
  python3-pulpcore-3.14.3-1.el7pc.noarch
  python3-pulp-rpm-3.14.0-1.el7pc.noarch
  libsolv-0.7.17-1.el7pc.x86_64
  python3-solv-0.7.17-1.el7pc.x86_64


How reproducible:
100%


Steps to Reproduce:
1. Sync RHEL8 BaseOS and AppStream repos
2. Create and publish above described CV, e.g. via below hammer commands (customize the repository IDs 6 and 7 to match yours RHEL8 repoIDs):

 content-view create --name mycv_RHEL8_yes_include_2019-01-01 --organization-id=1 --repository-ids=6,7 --solve-dependencies=yes
 content-view filter create --organization-id=1 --content-view=mycv_RHEL8_yes_include_2019-01-01 --name=include_base --inclusion=true --original-packages=true --type=rpm
 content-view filter create --organization-id=1 --content-view=mycv_RHEL8_yes_include_2019-01-01 --name=include_errata --inclusion=true --type=erratum
 content-view filter rule create --organization-id=1 --content-view=mycv_RHEL8_yes_include_2019-01-01 --content-view-filter=include_errata --date-type='updated' --end-date='2019-01-01'
 content-view filter create --organization-id=1 --content-view=mycv_RHEL8_yes_include_2019-01-01 --name=include_modules --inclusion=true --original-module-streams=true --type=modulemd
 content-view publish --name=mycv_RHEL8_yes_include_2019-01-01 --organization-id=1 --async

3. Check the CV publish outcome


Actual results:
3. Fails on MultiCopyContent dynflow step due to (self)cancelled pulp task:

- pulp_href: "/pulp/api/v3/tasks/5552552d-97e8-433f-964f-4f68cca21fe5/"
  pulp_created: '2021-08-17T00:31:02.484+00:00'
  state: canceled
  name: pulp_rpm.app.tasks.copy.copy_content
  logging_cid: 21b7f226-d3f8-4f05-a164-131cc21a601c
  started_at: '2021-08-17T00:47:37.336+00:00'
  worker: "/pulp/api/v3/workers/c6eda7d0-4082-481e-b63b-3f6f688ef0dd/"
  child_tasks: []
  progress_reports: []
  created_resources: []
  reserved_resources_record:
  - "/pulp/api/v3/repositories/rpm/rpm/566db2d9-b7d1-43e1-8a32-7d3c9657fd84/"
  - "/pulp/api/v3/repositories/rpm/rpm/41a0f380-8a61-4806-a7a2-34072a620264/"
  - "/pulp/api/v3/repositories/rpm/rpm/98b8c06d-0fc2-49ac-ad70-31a54b6aafe7/"
  - "/pulp/api/v3/repositories/rpm/rpm/2fb4d5b8-7c1c-4870-8fea-4719b3c8df00/"

/var/log/messages show dependency errors for the logging_cid like:

Aug 17 03:14:56 pmoravec-sat610-beta-2 pulpcore-worker-4: pulp [21b7f226-d3f8-4f05-a164-131cc21a601c]: pulp_rpm.app.depsolving:WARNING: Encountered problems solving dependencies, copy may be incomplete: cannot install both lmdb-libs-0:0.9.23-5.el8.x86_64 and lmdb-libs-0:0.9.24-1.el8.x86_64


Expected results:
Either make the CV publish successfull, or review KCS https://access.redhat.com/solutions/6264721 from 2 points:
- remedy steps (skipping the dynflow step and resuming the task) are valid
- all broken dependencies are expected to happen (due to old perl modularity broken deps issues, e.g.)


Additional info:
Comment 1 Pavel Moravec 2021-08-17 07:31:46 UTC 
Created attachment 1814628 [details]
list of all depsolving issues for the one pulp task
Comment 3 Grant Gainey 2021-08-23 13:59:51 UTC 
I'm working in this arena right now.

There are many log-warnings that are just libsolv complaing "you can't install all of these on one box" - which is actually fine, the repo can still hold them. I am working on a change to not-emit this class of warning under this issue/BZ:
* https://pulp.plan.io/issues/9238
* https://bugzilla.redhat.com/buglist.cgi?quicksearch=1965942

Canceled tasks happen **only** if the worker is killed - either by the SIGABRT from the libsolv-assert we're tracking in this BZ:
* https://bugzilla.redhat.com/show_bug.cgi?id=1934545

**Or**, the copy used so much memory that the OOMKiller came to visit (which we're tracking as BZ:
* https://bugzilla.redhat.com/show_bug.cgi?id=1965936

So I think we can close this BZ as a dup of...all three listed BZs?
Comment 8 Lai 2021-10-11 15:28:32 UTC 
Steps to retest:

1. Sync RHEL8 BaseOS and AppStream repos
2. Create and publish above described CV, e.g. via below hammer commands (customize the repository IDs 6 and 7 to match yours RHEL8 repoIDs):

 content-view create --name mycv_RHEL8_yes_include_2019-01-01 --organization-id=1 --repository-ids=6,7 --solve-dependencies=yes
 content-view filter create --organization-id=1 --content-view=mycv_RHEL8_yes_include_2019-01-01 --name=include_base --inclusion=true --original-packages=true --type=rpm
 content-view filter create --organization-id=1 --content-view=mycv_RHEL8_yes_include_2019-01-01 --name=include_errata --inclusion=true --type=erratum
 content-view filter rule create --organization-id=1 --content-view=mycv_RHEL8_yes_include_2019-01-01 --content-view-filter=include_errata --date-type='updated' --end-date='2019-01-01'
 content-view filter create --organization-id=1 --content-view=mycv_RHEL8_yes_include_2019-01-01 --name=include_modules --inclusion=true --original-module-streams=true --type=modulemd
 content-view publish --name=mycv_RHEL8_yes_include_2019-01-01 --organization-id=1 --async

3. Check the CV publish outcome


Expected result:
cv publish should complete successfully

Actual result:
Cv publish completed successfully

I also checked the "MultiCopyContent" portion in dynflow and it completed successfully as well.

22: Actions::Pulp3::Repository::MultiCopyContent (success) [ 984.09s / 105.75s ]

Verified on 6.10_22
Comment 11 errata-xmlrpc 2021-11-16 14:13:10 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: Satellite 6.10 Release), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:4702