Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 2250254

Summary: [Bug] - Leapp upgrade of Red Hat Satellite server failed with error Problem 1: problem with installed package tomcatjss-7.2.5-1.el7.noarch
Product: Red Hat Satellite Reporter: Dhaval Joshi <dhjoshi>
Component: UpgradesAssignee: Evgeni Golov <egolov>
Status: CLOSED CURRENTRELEASE QA Contact: Lukas Pramuk <lpramuk>
Severity: high Docs Contact:
Priority: urgent    
Version: 6.11.5CC: agadhave, ahumbe, ajambhul, alsouza, bdm, egolov, ehelms, jcastran, jpasqual, lpramuk, mjivraja, risantam, rlavi, saydas, wpinheir
Target Milestone: UnspecifiedKeywords: PrioBumpGSS, Triaged
Target Release: Unused   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2023-12-15 14:03:00 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Dhaval Joshi 2023-11-17 10:35:33 UTC
Description of problem:

Unable to perform RHEL-7 to RHEL-8 upgrade of Satellite server.

Version-Release number of selected component (if applicable):
Satellite 6.11.5

How reproducible:
100%

Steps to Reproduce:
1. Deploy Red Hat Satellite 6.11 on RHEL-7.
2. Update to 6.11.z.
3. Install required leapp packages.
4. Run preupgrade command
#leapp preupgrade

Actual results:
The leapp preupgrade commands fail with a package dependency issue.

============================================================
                           ERRORS                           
============================================================
 
2023-11-17 04:32:09.807695 [ERROR] Actor: dnf_transaction_check
Message: DNF execution failed with non zero exit code.
Summary:
    Stderr: Warning: Package marked by Leapp to install not found in repositories metadata: log4j12
            Warning: Package marked by Leapp to upgrade not found in repositories metadata: python2-leapp gpg-pubkey leapp leapp-upgrade-el7toel8
            Transaction check: 
            
             Problem 1: problem with installed package tomcatjss-7.2.5-1.el7.noarch
              - cannot install the best update candidate for package tomcatjss-7.2.5-1.el7.noarch
              - package tomcatjss-7.4.1-1.module+el8.1.0+3370+6d076660.noarch from rhel-8-for-x86_64-appstream-rpms requires pki-servlet-engine >= 1:9.0.7, but none of the providers can be installed
              - package tomcatjss-7.4.1-2.module+el8.2.0+4573+c3c38c7b.noarch from rhel-8-for-x86_64-appstream-rpms requires pki-servlet-engine >= 1:9.0.7, but none of the providers can be installed
              - package tomcatjss-7.4.1-2.module+el8.2.0+6294+b7db4606.noarch from rhel-8-for-x86_64-appstream-rpms requires pki-servlet-engine >= 1:9.0.7, but none of the providers can be installed
              - package tomcatjss-7.5.0-1.module+el8.3.0+7355+c59bcbd9.noarch from rhel-8-for-x86_64-appstream-rpms requires pki-servlet-engine >= 1:9.0.7, but none of the providers can be installed
              - package tomcatjss-7.6.1-1.module+el8.4.0+8778+d07929ff.noarch from rhel-8-for-x86_64-appstream-rpms requires pki-servlet-engine >= 1:9.0.7, but none of the providers can be installed
              - package tomcatjss-7.7.0-1.module+el8.5.0+11983+6ba118b4.noarch from rhel-8-for-x86_64-appstream-rpms requires pki-servlet-engine >= 1:9.0.7, but none of the providers can be installed
              - package tomcatjss-7.7.1-1.module+el8.6.0+13291+248751b1.noarch from rhel-8-for-x86_64-appstream-rpms requires pki-servlet-engine >= 1:9.0.7, but none of the providers can be installed
              - package tomcat-1:9.0.62-27.el8_9.noarch from rhel-8-for-x86_64-appstream-rpms conflicts with pki-servlet-engine <= 1:9.0.50 provided by pki-servlet-engine-1:9.0.7-16.module+el8.1.0+3366+6dfb954c.noarch from rhel-8-for-x86_64-appstream-rpms
              - package tomcat-1:9.0.62-27.el8_9.noarch from rhel-8-for-x86_64-appstream-rpms conflicts with pki-servlet-engine <= 1:9.0.50 provided by pki-servlet-engine-1:9.0.30-1.module+el8.3.0+6730+8f9c6254.noarch from rhel-8-for-x86_64-appstream-rpms
              - package tomcat-1:9.0.62-27.el8_9.noarch from rhel-8-for-x86_64-appstream-rpms conflicts with pki-servlet-engine <= 1:9.0.50 provided by pki-servlet-engine-1:9.0.30-3.module+el8.5.0+11388+9e95fe00.noarch from rhel-8-for-x86_64-appstream-rpms
              - package tomcat-1:9.0.62-27.el8_9.noarch from rhel-8-for-x86_64-appstream-rpms conflicts with pki-servlet-engine <= 1:9.0.50 provided by pki-servlet-engine-1:9.0.50-1.module+el8.7.0+15761+f86c9a56.noarch from rhel-8-for-x86_64-appstream-rpms
              - problem with installed package tomcat-7.0.76-16.el7_9.noarch
              - cannot install the best update candidate for package tomcat-7.0.76-16.el7_9.noarch
              - tomcat-7.0.76-16.el7_9.noarch from @System  does not belong to a distupgrade repository
              - tomcatjss-7.2.5-1.el7.noarch from @System  does not belong to a distupgrade repository
             Problem 2: package katello-4.3.0-3.el8sat.noarch from satellite-6.11-for-rhel-8-x86_64-rpms requires candlepin >= 2.0, but none of the providers can be installed
              - package candlepin-4.1.19-1.el8sat.noarch from satellite-6.11-for-rhel-8-x86_64-rpms requires tomcatjss >= 7.2.1-7.1, but none of the providers can be installed
              - package candlepin-4.1.14-1.el8sat.noarch from satellite-6.11-for-rhel-8-x86_64-rpms requires tomcatjss >= 7.2.1-7.1, but none of the providers can be installed
              - package candlepin-4.1.13-1.el8sat.noarch from satellite-6.11-for-rhel-8-x86_64-rpms requires tomcatjss >= 7.2.1-7.1, but none of the providers can be installed
              - package candlepin-4.1.19-2.el8sat.noarch from satellite-6.11-for-rhel-8-x86_64-rpms requires tomcatjss >= 7.2.1-7.1, but none of the providers can be installed
              - package tomcatjss-7.3.6-1.module+el8+2467+468c375c.noarch from rhel-8-for-x86_64-appstream-rpms requires pki-servlet-container >= 1:9.0.7, but none of the providers can be installed
              - cannot install the best update candidate for package tomcatjss-7.2.5-1.el7.noarch
              - package tomcat-el-3.0-api-1:9.0.62-27.el8_9.noarch from rhel-8-for-x86_64-appstream-rpms conflicts with pki-servlet-container <= 1:9.0.7 provided by pki-servlet-container-1:9.0.7-13.module+el8+2468+c564cec5.noarch from rhel-8-for-x86_64-appstream-rpms
              - package tomcat-el-3.0-api-1:9.0.62-27.el8_9.noarch from rhel-8-for-x86_64-appstream-rpms conflicts with pki-servlet-container <= 1:9.0.7 provided by pki-servlet-container-1:9.0.7-14.module+el8.0.0+3248+9d514f3b.noarch from rhel-8-for-x86_64-appstream-rpms
              - package tomcat-el-3.0-api-1:9.0.62-27.el8_9.noarch from rhel-8-for-x86_64-appstream-rpms conflicts with pki-servlet-container <= 1:9.0.7 provided by pki-servlet-container-1:9.0.7-14.module+el8.0.0+3892+c903d3f0.noarch from rhel-8-for-x86_64-appstream-rpms
              - package tomcatjss-7.4.1-1.module+el8.1.0+3370+6d076660.noarch from rhel-8-for-x86_64-appstream-rpms requires pki-servlet-engine >= 1:9.0.7, but none of the providers can be installed
              - package tomcatjss-7.4.1-2.module+el8.2.0+4573+c3c38c7b.noarch from rhel-8-for-x86_64-appstream-rpms requires pki-servlet-engine >= 1:9.0.7, but none of the providers can be installed
              - package tomcatjss-7.4.1-2.module+el8.2.0+6294+b7db4606.noarch from rhel-8-for-x86_64-appstream-rpms requires pki-servlet-engine >= 1:9.0.7, but none of the providers can be installed
              - package tomcatjss-7.5.0-1.module+el8.3.0+7355+c59bcbd9.noarch from rhel-8-for-x86_64-appstream-rpms requires pki-servlet-engine >= 1:9.0.7, but none of the providers can be installed
              - package tomcatjss-7.6.1-1.module+el8.4.0+8778+d07929ff.noarch from rhel-8-for-x86_64-appstream-rpms requires pki-servlet-engine >= 1:9.0.7, but none of the providers can be installed
              - package tomcatjss-7.7.0-1.module+el8.5.0+11983+6ba118b4.noarch from rhel-8-for-x86_64-appstream-rpms requires pki-servlet-engine >= 1:9.0.7, but none of the providers can be installed
              - package tomcatjss-7.7.1-1.module+el8.6.0+13291+248751b1.noarch from rhel-8-for-x86_64-appstream-rpms requires pki-servlet-engine >= 1:9.0.7, but none of the providers can be installed
              - package tomcat-lib-1:9.0.62-27.el8_9.noarch from rhel-8-for-x86_64-appstream-rpms requires tomcat-el-3.0-api = 1:9.0.62-27.el8_9, but none of the providers can be installed
              - package tomcat-lib-1:9.0.62-27.el8_9.noarch from rhel-8-for-x86_64-appstream-rpms conflicts with pki-servlet-engine <= 1:9.0.50 provided by pki-servlet-engine-1:9.0.7-16.module+el8.1.0+3366+6dfb954c.noarch from rhel-8-for-x86_64-appstream-rpms
              - package tomcat-lib-1:9.0.62-27.el8_9.noarch from rhel-8-for-x86_64-appstream-rpms conflicts with pki-servlet-engine <= 1:9.0.50 provided by pki-servlet-engine-1:9.0.30-1.module+el8.3.0+6730+8f9c6254.noarch from rhel-8-for-x86_64-appstream-rpms
              - package tomcat-lib-1:9.0.62-27.el8_9.noarch from rhel-8-for-x86_64-appstream-rpms conflicts with pki-servlet-engine <= 1:9.0.50 provided by pki-servlet-engine-1:9.0.30-3.module+el8.5.0+11388+9e95fe00.noarch from rhel-8-for-x86_64-appstream-rpms
              - package tomcat-lib-1:9.0.62-27.el8_9.noarch from rhel-8-for-x86_64-appstream-rpms conflicts with pki-servlet-engine <= 1:9.0.50 provided by pki-servlet-engine-1:9.0.50-1.module+el8.7.0+15761+f86c9a56.noarch from rhel-8-for-x86_64-appstream-rpms
              - problem with installed package tomcat-lib-7.0.76-16.el7_9.noarch
              - cannot install the best update candidate for package tomcat-lib-7.0.76-16.el7_9.noarch
              - cannot install both tomcat-lib-1:9.0.62-27.el8_9.noarch from rhel-8-for-x86_64-appstream-rpms and tomcat-lib-1:9.0.62-5.el8.noarch from rhel-8-for-x86_64-appstream-rpms
              - cannot install both tomcat-lib-1:9.0.62-27.el8_9.noarch from rhel-8-for-x86_64-appstream-rpms and tomcat-lib-1:9.0.62-5.el8_8.2.noarch from rhel-8-for-x86_64-appstream-rpms
              - cannot install the best candidate for the job
              - tomcat-lib-7.0.76-16.el7_9.noarch from @System  does not belong to a distupgrade repository
              - candlepin-4.1.19-1.el7sat.noarch from @System  does not belong to a distupgrade repository
    Stdout: Last metadata expiration check: 0:02:11 ago on Fri Nov 17 04:29:47 2023.
            Package foreman-installer-katello-1:3.1.2.8-1.el7sat.noarch is already installed.
            Package rubygem-foreman_maintain-1:1.0.19-1.el7sat.noarch is already installed.
            Package tfm-rubygem-smart_proxy_ansible-3.3.1-5.el7sat.noarch is already installed.
            Package satellite-installer-6.11.0.7-1.el7sat.noarch is already installed.
            Package katello-4.3.0-3.el7sat.noarch is already installed.
            Package satellite-6.11.5.6-1.el7sat.noarch is already installed.
            Package foreman-installer-1:3.1.2.8-1.el7sat.noarch is already installed.
 
============================================================
                       END OF ERRORS                        
============================================================
 
Debug output written to /var/log/leapp/leapp-preupgrade.log
 
============================================================
                      REPORT OVERVIEW                       
============================================================
 
Following errors occurred and the upgrade cannot continue:
    1. Actor: dnf_transaction_check
       Message: DNF execution failed with non zero exit code.
 
HIGH and MEDIUM severity reports:
    1. Packages available in excluded repositories will not be installed
    2. GRUB2 core will be automatically updated during the upgrade
    3. Difference in Python versions and support in RHEL 8
    4. Satellite PostgreSQL data migration
    5. chrony using default configuration
    6. Module pam_pkcs11 will be removed from PAM configuration
 
Reports summary:
    Errors:                      1
    Inhibitors:                  0
    HIGH severity reports:       3
    MEDIUM severity reports:     3
    LOW severity reports:        6
    INFO severity reports:       4
 
Before continuing consult the full report:
    A report has been generated at /var/log/leapp/leapp-report.json
    A report has been generated at /var/log/leapp/leapp-report.txt
 
============================================================
                   END OF REPORT OVERVIEW                   
============================================================
 
Answerfile has been generated at /var/log/leapp/answerfile

Expected results:
The preupgrade command should be executed successfully without package dependency issue.

Additional info:
# leapp preupgrade --target 8.8 also fails with the same issue.

Workaround:
- Upgrade to RHEL 8.6 works fine.

# leapp preupgrade --target 8.6 
# leapp upgrade --target 8.6

Comment 3 Evgeni Golov 2023-11-20 12:28:08 UTC
Thank you for the report!

I was not able to reproduce it with leapp-0.15.1-1.el7_9 leapp-upgrade-el7toel8-0.18.0-3.el7_9
But I was able to reproduce it with leapp-0.16.0-1.el7_9 leapp-upgrade-el7toel8-0.19.0-1.el7_9

I'll dig deeper into this.

Comment 4 Evgeni Golov 2023-11-20 12:45:05 UTC
Aha, the issue is being triggered by new PES Events that were added to re-introduce the tomcat package for RHEL 8.8
These are not yet present in the 0.18 repository package, but are in 0.19.

With this, leapp tries to upgrade the EL7 tomcat to "tomcat" in EL8, but due to the tomcatjss constrain we actually need pki-servlet-engine instead.

Comment 6 Evgeni Golov 2023-11-20 12:51:38 UTC
and copy pasting some more detailed explanation from a chat I just had:

the satellite packaging needs tomcatjss and pki-servlet-engine (at least today)
due to the addition of the "oh there is tomcat in el8 now" event (in leapp-upgrade-el7toel8-0.19.0-1.el7_9), leapp now tries to upgrade tomcat.el7 to tomcat.el8 (instead of throwing the el7 package away and letting "normal" dependencies pull in pki-servlet-engine)
but tomcat.el8 conflicts with pki-servlet-engine → boom
this is quite satellite specific (nothing but us and IdM should use pki-servlet-engine, and IdM doesn't support IPU)
we can totally workaround this by adding tomcat and tomcat-libs to the "to remove" event our Satellite actor produces
this seems to win over the PES events and I get pki-servlet-engine just fine
whether this is the fix we all want to ship, is another question

Comment 8 Evgeni Golov 2023-11-22 07:38:13 UTC
The patch was merged in upstream Leapp.

Comment 9 Evgeni Golov 2023-11-22 10:04:32 UTC
Possible workarounds:

Downgrade leapp to 0.15.1-1.el7_9 and leapp-upgrade-el7toel8 to 0.18.0-3.el7_9:
# yum downgrade leapp-0.15.1-1.el7_9 python2-leapp-0.15.1-1.el7_9 leapp-upgrade-el7toel8-0.18.0-3.el7_9

OR

Add tomcat and tomcat-lib to /etc/leapp/transaction/to_remove (this has the same effect as the merged patch, but avoids editing Python code):
# echo tomcat >> /etc/leapp/transaction/to_remove
# echo tomcat-lib >> /etc/leapp/transaction/to_remove

Comment 13 Eric Helms 2023-12-15 14:03:00 UTC
6.11 docs have been updated.