Description of problem: Attempting to perform a disconnected upgrade from Satellite 6.3.3 to Satellite 6.4 does not complete, and leaves the system in a inconsistent state. Version-Release number of selected component (if applicable): Upgrading from Satellite 6.3.3 to 6.4 in a disconnected environment Steps to Reproduce: * Built VM from RHEL 7.5 DVD * Copied satellite-6.3.3-rhel-7-x86_64-dvd.iso, satellite-6.4.0-rhel-7-x86_64-dvd.iso, rhel-server-7.5-x86_64-dvd.iso ISO images to server * NOTE: this is the Satellite 6.4 ISO updated on 10/24/18 * mkdir /media/rhel7-server * mount -o loop rhel-server-7.5-x86_64-dvd.iso /media/rhel7-server * cp /media/rhel7-server/media.repo /etc/yum.repos.d/rhel7-server.repo * echo 'baseurl=file:///media/rhel7-server/' >> /etc/yum.repos.d/rhel7-server.repo * echo 'enabled=1' >> /etc/yum.repos.d/rhel7-server.repo * yum repolist * mkdir /media/sat6 * mount -o loop satellite-6.3.3-rhel-7-x86_64-dvd.iso /media/sat6 * echo '[satellite-puppet4]' > /etc/yum.repos.d/satellite-puppet4.repo * echo 'name=satellite-puppet4' >> /etc/yum.repos.d/satellite-puppet4.repo * echo 'baseurl=file:///media/sat6/addons/Puppet4' >> /etc/yum.repos.d/satellite-puppet4.repo * echo 'enabled=1' >> /etc/yum.repos.d/satellite-puppet4.repo * echo 'gpgcheck=1' >> /etc/yum.repos.d/satellite-puppet4.repo * yum repolist * rpm --import /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release * yum update * cd /media/sat6/ * ./install_packages (got error about pulp-selinux requiring update to selinux-policy) * Downloaded and copied to server: selinux-policy-3.13.1-192.el7_5.4.noarch.rpm, selinux-policy-targeted-3.13.1-192.el7_5.4.noarch.rpm * yum localupdate selinux-policy-3.13.1-192.el7_5.4.noarch.rpm selinux-policy-targeted-3.13.1-192.el7_5.4.noarch.rpm * cd /media/sat6/ * ./install_packages * firewall-cmd --add-port="53/udp" --add-port="53/tcp" --add-port="67/udp" --add-port="69/udp" --add-port="80/tcp" --add-port="443/tcp" --add-port="5000/tcp" --add-port="5647/tcp" --add-port="8000/tcp" --add-port="8140/tcp" --add-port="8443/tcp" --add-port="9090/tcp" * firewall-cmd --permanent --add-port="53/udp" --add-port="53/tcp" --add-port="67/udp" --add-port="69/udp" --add-port="80/tcp" --add-port="443/tcp" --add-port="5000/tcp" --add-port="5647/tcp" --add-port="8000/tcp" --add-port="8140/tcp" --add-port="8443/tcp" --add-port="9090/tcp" * satellite-installer --scenario satellite --foreman-initial-organization "disconnected" --foreman-initial-location "disconnected" --foreman-admin-username admin --foreman-proxy-dns-managed=false --foreman-proxy-dhcp-managed=false * Verified could login to Satellite 6.3.3 web interface and changed admin password * foreman-rake katello:upgrade_check (PASS) * Disabled satellite-puppet4 repository and 6.3 SCL and Satellite 6.3 repositories (only enabled repo is RHEL 7.5 local repository) * katello-service stop * mount satellite-6.4.0-rhel-7-x86_64-dvd.iso /media/sat6 * cd /media/sat6/ * ./install_packages * Errors about pulp-selinux-2.16.4.1-1.el7sat.noarch Requires: selinux-policy >= 3.13.1-192.el7_5.6, candlepin-selinux-2.4.8-1.el7.noarch require sselinux-policy >= 3.13.1-192.el7_5.6 * Downloaded and copied to server: selinux-policy-3.13.1-192.el7_5.6.noarch.rpm selinux-policy-targeted-3.13.1-192.el7_5.6.noarch.rpm * yum localupdate selinux-policy-3.13.1-192.el7_5.6.noarch.rpm selinux-policy-targeted-3.13.1-192.el7_5.6.noarch.rpm * cd /media/sat6/ * ./install_packages This script will install the satellite packages on the current machine. - Ensuring we are in an expected directory. - Copying installation files. - Creating a Repository File - Creating RHSCL Repository File - Creating sat-maintenance Repository File - Checking to see if Satellite is already installed. - Satellite is already installed, running 'yum update'. - Installation repository will remain configured for future package installs. - Installation media can now be safely unmounted. Upgrade is complete. Please backup your data and run satellite-installer --scenario satellite * yum install rubygem-foreman_maintain (already installed / at latest) * foreman-maintain upgrade list-versions * foreman-maintain upgrade check --target-version 6.4.z Running preparation steps required to run the next scenarios ================================================================================ Setup hammer: [SKIPPED] Satellite server is not running. Hammer can't be setup now. * * * * * * * * * * * * * * * * * * * * Procedures::Packages::Install: Loaded plugins: product-id, search-disabled-repos, subscription-manager This system is not registered with an entitlement server. You can use subscription-manager to register. Resolving Dependencies --> Running transaction check ---> Package fio.x86_64 0:3.1-2.el7 will be installed --> Processing Dependency: libibverbs.so.1(IBVERBS_1.0)(64bit) for package: fio-3.1-2.el7.x86_64 --> Processing Dependency: libibverbs.so.1(IBVERBS_1.1)(64bit) for package: fio-3.1-2.el7.x86_64 --> Processing Dependency: libpmem.so.1(LIBPMEM_1.0)(64bit) for package: fio-3.1-2.el7.x86_64 --> Processing Dependency: libpmemblk.so.1(LIBPMEMBLK_1.0)(64bit) for package: fio-3.1-2.el7.x86_64 --> Processing Dependency: librdmacm.so.1(RDMACM_1.0)(64bit) for package: fio-3.1-2.el7.x86_64 --> Processing Dependency: libibverbs.so.1()(64bit) for package: fio-3.1-2.el7.x86_64 --> Processing Dependency: libpmem.so.1()(64bit) for package: fio-3.1-2.el7.x86_64 --> Processing Dependency: libpmemblk.so.1()(64bit) for package: fio-3.1-2.el7.x86_64 --> Processing Dependency: librados.so.2()(64bit) for package: fio-3.1-2.el7.x86_64 --> Processing Dependency: librbd.so.1()(64bit) for package: fio-3.1-2.el7.x86_64 --> Processing Dependency: librdmacm.so.1()(64bit) for package: fio-3.1-2.el7.x86_64 --> Running transaction check ---> Package libibverbs.x86_64 0:15-6.el7 will be installed --> Processing Dependency: rdma-core(x86-64) = 15-6.el7 for package: libibverbs-15-6.el7.x86_64 ---> Package libpmem.x86_64 0:1.3-3.el7 will be installed ---> Package libpmemblk.x86_64 0:1.3-3.el7 will be installed ---> Package librados2.x86_64 1:0.94.5-2.el7 will be installed ---> Package librbd1.x86_64 1:0.94.5-2.el7 will be installed ---> Package librdmacm.x86_64 0:15-6.el7 will be installed --> Running transaction check ---> Package rdma-core.x86_64 0:15-6.el7 will be installed --> Finished Dependency Resolution Dependencies Resolved ========================================================================================================================================================================================= Package Arch Version Repository Size ========================================================================================================================================================================================= Installing: fio x86_64 3.1-2.el7 InstallMedia 451 k Installing for dependencies: libibverbs x86_64 15-6.el7 InstallMedia 224 k libpmem x86_64 1.3-3.el7 InstallMedia 36 k libpmemblk x86_64 1.3-3.el7 InstallMedia 54 k librados2 x86_64 1:0.94.5-2.el7 InstallMedia 1.7 M librbd1 x86_64 1:0.94.5-2.el7 InstallMedia 1.8 M librdmacm x86_64 15-6.el7 InstallMedia 61 k rdma-core x86_64 15-6.el7 InstallMedia 48 k Transaction Summary ========================================================================================================================================================================================= Install 1 Package (+7 Dependent packages) Total download size: 4.4 M Installed size: 13 M Is this ok [y/d/N]: y Downloading packages: * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * - Total 40 MB/s | 4.4 MB 00:00:00 Running transaction check Running transaction test Transaction test succeeded Running transaction Installing : 1:librados2-0.94.5-2.el7.x86_64 1/8 Installing : rdma-core-15-6.el7.x86_64 2/8 Installing : libibverbs-15-6.el7.x86_64 3/8 Installing : libpmem-1.3-3.el7.x86_64 4/8 Installing : libpmemblk-1.3-3.el7.x86_64 5/8 Installing : librdmacm-15-6.el7.x86_64 6/8 Installing : 1:librbd1-0.94.5-2.el7.x86_64 7/8 Installing : fio-3.1-2.el7.x86_64 8/8 Verifying : 1:librbd1-0.94.5-2.el7.x86_64 1/8 Verifying : libpmem-1.3-3.el7.x86_64 2/8 Verifying : libibverbs-15-6.el7.x86_64 3/8 Verifying : rdma-core-15-6.el7.x86_64 4/8 Verifying : librdmacm-15-6.el7.x86_64 5/8 Verifying : fio-3.1-2.el7.x86_64 6/8 Verifying : 1:librados2-0.94.5-2.el7.x86_64 7/8 Verifying : libpmemblk-1.3-3.el7.x86_64 8/8 Installed: fio.x86_64 0:3.1-2.el7 Dependency Installed: libibverbs.x86_64 0:15-6.el7 libpmem.x86_64 0:1.3-3.el7 libpmemblk.x86_64 0:1.3-3.el7 librados2.x86_64 1:0.94.5-2.el7 librbd1.x86_64 1:0.94.5-2.el7 librdmacm.x86_64 0:15-6.el7 rdma-core.x86_64 0:15-6.el7 Complete! [OK] * * * * * * * * * * * * * * * * * * * * Running Checks before upgrading to Satellite 6.4.z ================================================================================ Check for verifying syntax for ISP DHCP configurations: [SKIPPED] DHCP feature is not enabled * * * * * * * * * * * * * * * * * * * * Check whether all services are running: [FAIL] Following services are not running: mongod, rh-mongodb34-mongod, postgresql, qdrouterd, qpidd, squid, pulp_celerybeat, pulp_resource_manager, pulp_streamer, pulp_workers, smart_proxy_dynflow_core, tomcat, dynflowd, httpd, puppetserver, foreman-proxy * * * * * * * * * * * * * * * * * * * * Continue with step [Restart applicable services]?, [y(yes), n(no), q(quit)] y Restart applicable services: Stopping the following service(s): mongod, rh-mongodb34-mongod, postgresql, qdrouterd, qpidd, squid, pulp_celerybeat, pulp_resource_manager, pulp_streamer, pulp_workers, smart_proxy_dynflow_core, tomcat, dynflowd, httpd, puppetserver, foreman-proxy \ All services stopped Starting the following service(s): mongod, rh-mongodb34-mongod, postgresql, qdrouterd, qpidd, squid, pulp_celerybeat, pulp_resource_manager, pulp_streamer, pulp_workers, smart_proxy_dynflow_core, tomcat, dynflowd, httpd, puppetserver, foreman-proxy \ starting rh-mongodb34-mongod Job for rh-mongodb34-mongod.service failed because the control process exited with error code. See "systemctl status rh-mongodb34-mongod.service" and "journalctl -xe" for details. - starting puppetserver Job for puppetserver.service failed because the control process exited with error code. See "systemctl status puppetserver.service" and "journalctl -xe" for details. - All services started Try 1/5: checking status of hammer ping * Was on this screen for 1.5 hours * CTRL-C to exit * foreman-maintain upgrade check --target-version 6.4.z -- prompts for admin password, says "Hammer configuration failed: Is the admin password correct?" * hammer ping ERF64-6496 [Foreman::MaintenanceException]: There are migrations pending in the system. Actual results: Upgrade is not successful, and leaves the system in a inconsistent state Expected results: Disconnected upgrade completes successfully
Brian, Can you check to see that you don't have any hammer defaults defined? We have this bug here: https://bugzilla.redhat.com/show_bug.cgi?id=1632768 that foreman-maintain will hang or error out trying to talk to the server because of a bug in how it handles 'hammer default' settings.
Mike, I don't have a /root/.hammer/defaults.yml file. Please let me know if you need any additional information. Thanks, Brian
Do you have an internal reproducer that we could log in to for debugging? Thanks!
Brad, I don't have a reproducer that is accessible. However, there are step by step directions/commands on what I had done when the issue occurred in the description. I followed these steps twice, and had the same issue both times. Thanks!
It seems the issue is with the disconnected script calling yum update before the foreman-maintain preupgrade checks. This issue is already tracked in https://bugzilla.redhat.com/show_bug.cgi?id=1643916 *** This bug has been marked as a duplicate of bug 1643916 ***