Note: This bug is displayed in read-only format because
the product is no longer active in Red Hat Bugzilla.
Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Brian,
Can you check to see that you don't have any hammer defaults defined? We have this bug here:
https://bugzilla.redhat.com/show_bug.cgi?id=1632768
that foreman-maintain will hang or error out trying to talk to the server because of a bug in how it handles 'hammer default' settings.
Brad,
I don't have a reproducer that is accessible. However, there are step by step directions/commands on what I had done when the issue occurred in the description. I followed these steps twice, and had the same issue both times. Thanks!
It seems the issue is with the disconnected script calling yum update before the foreman-maintain preupgrade checks. This issue is already tracked in https://bugzilla.redhat.com/show_bug.cgi?id=1643916
*** This bug has been marked as a duplicate of bug 1643916 ***
Description of problem: Attempting to perform a disconnected upgrade from Satellite 6.3.3 to Satellite 6.4 does not complete, and leaves the system in a inconsistent state. Version-Release number of selected component (if applicable): Upgrading from Satellite 6.3.3 to 6.4 in a disconnected environment Steps to Reproduce: * Built VM from RHEL 7.5 DVD * Copied satellite-6.3.3-rhel-7-x86_64-dvd.iso, satellite-6.4.0-rhel-7-x86_64-dvd.iso, rhel-server-7.5-x86_64-dvd.iso ISO images to server * NOTE: this is the Satellite 6.4 ISO updated on 10/24/18 * mkdir /media/rhel7-server * mount -o loop rhel-server-7.5-x86_64-dvd.iso /media/rhel7-server * cp /media/rhel7-server/media.repo /etc/yum.repos.d/rhel7-server.repo * echo 'baseurl=file:///media/rhel7-server/' >> /etc/yum.repos.d/rhel7-server.repo * echo 'enabled=1' >> /etc/yum.repos.d/rhel7-server.repo * yum repolist * mkdir /media/sat6 * mount -o loop satellite-6.3.3-rhel-7-x86_64-dvd.iso /media/sat6 * echo '[satellite-puppet4]' > /etc/yum.repos.d/satellite-puppet4.repo * echo 'name=satellite-puppet4' >> /etc/yum.repos.d/satellite-puppet4.repo * echo 'baseurl=file:///media/sat6/addons/Puppet4' >> /etc/yum.repos.d/satellite-puppet4.repo * echo 'enabled=1' >> /etc/yum.repos.d/satellite-puppet4.repo * echo 'gpgcheck=1' >> /etc/yum.repos.d/satellite-puppet4.repo * yum repolist * rpm --import /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release * yum update * cd /media/sat6/ * ./install_packages (got error about pulp-selinux requiring update to selinux-policy) * Downloaded and copied to server: selinux-policy-3.13.1-192.el7_5.4.noarch.rpm, selinux-policy-targeted-3.13.1-192.el7_5.4.noarch.rpm * yum localupdate selinux-policy-3.13.1-192.el7_5.4.noarch.rpm selinux-policy-targeted-3.13.1-192.el7_5.4.noarch.rpm * cd /media/sat6/ * ./install_packages * firewall-cmd --add-port="53/udp" --add-port="53/tcp" --add-port="67/udp" --add-port="69/udp" --add-port="80/tcp" --add-port="443/tcp" --add-port="5000/tcp" --add-port="5647/tcp" --add-port="8000/tcp" --add-port="8140/tcp" --add-port="8443/tcp" --add-port="9090/tcp" * firewall-cmd --permanent --add-port="53/udp" --add-port="53/tcp" --add-port="67/udp" --add-port="69/udp" --add-port="80/tcp" --add-port="443/tcp" --add-port="5000/tcp" --add-port="5647/tcp" --add-port="8000/tcp" --add-port="8140/tcp" --add-port="8443/tcp" --add-port="9090/tcp" * satellite-installer --scenario satellite --foreman-initial-organization "disconnected" --foreman-initial-location "disconnected" --foreman-admin-username admin --foreman-proxy-dns-managed=false --foreman-proxy-dhcp-managed=false * Verified could login to Satellite 6.3.3 web interface and changed admin password * foreman-rake katello:upgrade_check (PASS) * Disabled satellite-puppet4 repository and 6.3 SCL and Satellite 6.3 repositories (only enabled repo is RHEL 7.5 local repository) * katello-service stop * mount satellite-6.4.0-rhel-7-x86_64-dvd.iso /media/sat6 * cd /media/sat6/ * ./install_packages * Errors about pulp-selinux-2.16.4.1-1.el7sat.noarch Requires: selinux-policy >= 3.13.1-192.el7_5.6, candlepin-selinux-2.4.8-1.el7.noarch require sselinux-policy >= 3.13.1-192.el7_5.6 * Downloaded and copied to server: selinux-policy-3.13.1-192.el7_5.6.noarch.rpm selinux-policy-targeted-3.13.1-192.el7_5.6.noarch.rpm * yum localupdate selinux-policy-3.13.1-192.el7_5.6.noarch.rpm selinux-policy-targeted-3.13.1-192.el7_5.6.noarch.rpm * cd /media/sat6/ * ./install_packages This script will install the satellite packages on the current machine. - Ensuring we are in an expected directory. - Copying installation files. - Creating a Repository File - Creating RHSCL Repository File - Creating sat-maintenance Repository File - Checking to see if Satellite is already installed. - Satellite is already installed, running 'yum update'. - Installation repository will remain configured for future package installs. - Installation media can now be safely unmounted. Upgrade is complete. Please backup your data and run satellite-installer --scenario satellite * yum install rubygem-foreman_maintain (already installed / at latest) * foreman-maintain upgrade list-versions * foreman-maintain upgrade check --target-version 6.4.z Running preparation steps required to run the next scenarios ================================================================================ Setup hammer: [SKIPPED] Satellite server is not running. Hammer can't be setup now. * * * * * * * * * * * * * * * * * * * * Procedures::Packages::Install: Loaded plugins: product-id, search-disabled-repos, subscription-manager This system is not registered with an entitlement server. You can use subscription-manager to register. Resolving Dependencies --> Running transaction check ---> Package fio.x86_64 0:3.1-2.el7 will be installed --> Processing Dependency: libibverbs.so.1(IBVERBS_1.0)(64bit) for package: fio-3.1-2.el7.x86_64 --> Processing Dependency: libibverbs.so.1(IBVERBS_1.1)(64bit) for package: fio-3.1-2.el7.x86_64 --> Processing Dependency: libpmem.so.1(LIBPMEM_1.0)(64bit) for package: fio-3.1-2.el7.x86_64 --> Processing Dependency: libpmemblk.so.1(LIBPMEMBLK_1.0)(64bit) for package: fio-3.1-2.el7.x86_64 --> Processing Dependency: librdmacm.so.1(RDMACM_1.0)(64bit) for package: fio-3.1-2.el7.x86_64 --> Processing Dependency: libibverbs.so.1()(64bit) for package: fio-3.1-2.el7.x86_64 --> Processing Dependency: libpmem.so.1()(64bit) for package: fio-3.1-2.el7.x86_64 --> Processing Dependency: libpmemblk.so.1()(64bit) for package: fio-3.1-2.el7.x86_64 --> Processing Dependency: librados.so.2()(64bit) for package: fio-3.1-2.el7.x86_64 --> Processing Dependency: librbd.so.1()(64bit) for package: fio-3.1-2.el7.x86_64 --> Processing Dependency: librdmacm.so.1()(64bit) for package: fio-3.1-2.el7.x86_64 --> Running transaction check ---> Package libibverbs.x86_64 0:15-6.el7 will be installed --> Processing Dependency: rdma-core(x86-64) = 15-6.el7 for package: libibverbs-15-6.el7.x86_64 ---> Package libpmem.x86_64 0:1.3-3.el7 will be installed ---> Package libpmemblk.x86_64 0:1.3-3.el7 will be installed ---> Package librados2.x86_64 1:0.94.5-2.el7 will be installed ---> Package librbd1.x86_64 1:0.94.5-2.el7 will be installed ---> Package librdmacm.x86_64 0:15-6.el7 will be installed --> Running transaction check ---> Package rdma-core.x86_64 0:15-6.el7 will be installed --> Finished Dependency Resolution Dependencies Resolved ========================================================================================================================================================================================= Package Arch Version Repository Size ========================================================================================================================================================================================= Installing: fio x86_64 3.1-2.el7 InstallMedia 451 k Installing for dependencies: libibverbs x86_64 15-6.el7 InstallMedia 224 k libpmem x86_64 1.3-3.el7 InstallMedia 36 k libpmemblk x86_64 1.3-3.el7 InstallMedia 54 k librados2 x86_64 1:0.94.5-2.el7 InstallMedia 1.7 M librbd1 x86_64 1:0.94.5-2.el7 InstallMedia 1.8 M librdmacm x86_64 15-6.el7 InstallMedia 61 k rdma-core x86_64 15-6.el7 InstallMedia 48 k Transaction Summary ========================================================================================================================================================================================= Install 1 Package (+7 Dependent packages) Total download size: 4.4 M Installed size: 13 M Is this ok [y/d/N]: y Downloading packages: * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * - Total 40 MB/s | 4.4 MB 00:00:00 Running transaction check Running transaction test Transaction test succeeded Running transaction Installing : 1:librados2-0.94.5-2.el7.x86_64 1/8 Installing : rdma-core-15-6.el7.x86_64 2/8 Installing : libibverbs-15-6.el7.x86_64 3/8 Installing : libpmem-1.3-3.el7.x86_64 4/8 Installing : libpmemblk-1.3-3.el7.x86_64 5/8 Installing : librdmacm-15-6.el7.x86_64 6/8 Installing : 1:librbd1-0.94.5-2.el7.x86_64 7/8 Installing : fio-3.1-2.el7.x86_64 8/8 Verifying : 1:librbd1-0.94.5-2.el7.x86_64 1/8 Verifying : libpmem-1.3-3.el7.x86_64 2/8 Verifying : libibverbs-15-6.el7.x86_64 3/8 Verifying : rdma-core-15-6.el7.x86_64 4/8 Verifying : librdmacm-15-6.el7.x86_64 5/8 Verifying : fio-3.1-2.el7.x86_64 6/8 Verifying : 1:librados2-0.94.5-2.el7.x86_64 7/8 Verifying : libpmemblk-1.3-3.el7.x86_64 8/8 Installed: fio.x86_64 0:3.1-2.el7 Dependency Installed: libibverbs.x86_64 0:15-6.el7 libpmem.x86_64 0:1.3-3.el7 libpmemblk.x86_64 0:1.3-3.el7 librados2.x86_64 1:0.94.5-2.el7 librbd1.x86_64 1:0.94.5-2.el7 librdmacm.x86_64 0:15-6.el7 rdma-core.x86_64 0:15-6.el7 Complete! [OK] * * * * * * * * * * * * * * * * * * * * Running Checks before upgrading to Satellite 6.4.z ================================================================================ Check for verifying syntax for ISP DHCP configurations: [SKIPPED] DHCP feature is not enabled * * * * * * * * * * * * * * * * * * * * Check whether all services are running: [FAIL] Following services are not running: mongod, rh-mongodb34-mongod, postgresql, qdrouterd, qpidd, squid, pulp_celerybeat, pulp_resource_manager, pulp_streamer, pulp_workers, smart_proxy_dynflow_core, tomcat, dynflowd, httpd, puppetserver, foreman-proxy * * * * * * * * * * * * * * * * * * * * Continue with step [Restart applicable services]?, [y(yes), n(no), q(quit)] y Restart applicable services: Stopping the following service(s): mongod, rh-mongodb34-mongod, postgresql, qdrouterd, qpidd, squid, pulp_celerybeat, pulp_resource_manager, pulp_streamer, pulp_workers, smart_proxy_dynflow_core, tomcat, dynflowd, httpd, puppetserver, foreman-proxy \ All services stopped Starting the following service(s): mongod, rh-mongodb34-mongod, postgresql, qdrouterd, qpidd, squid, pulp_celerybeat, pulp_resource_manager, pulp_streamer, pulp_workers, smart_proxy_dynflow_core, tomcat, dynflowd, httpd, puppetserver, foreman-proxy \ starting rh-mongodb34-mongod Job for rh-mongodb34-mongod.service failed because the control process exited with error code. See "systemctl status rh-mongodb34-mongod.service" and "journalctl -xe" for details. - starting puppetserver Job for puppetserver.service failed because the control process exited with error code. See "systemctl status puppetserver.service" and "journalctl -xe" for details. - All services started Try 1/5: checking status of hammer ping * Was on this screen for 1.5 hours * CTRL-C to exit * foreman-maintain upgrade check --target-version 6.4.z -- prompts for admin password, says "Hammer configuration failed: Is the admin password correct?" * hammer ping ERF64-6496 [Foreman::MaintenanceException]: There are migrations pending in the system. Actual results: Upgrade is not successful, and leaves the system in a inconsistent state Expected results: Disconnected upgrade completes successfully