Let's make sure we actually prevent the engine to upgrade to 3.6, if there are hosts in HE deployment that are on RHEL6. Current implementation of the original bug is insufficient since it lets the user to continue the upgrade, despite the warning. It is confusing the way it is implemented right now and the upgrade flow is already complex. And if upgrade of the engine to 3.6 is going to disturb the upgrade of the Hosted Engine setup, let's just remove this option completely. If this upgrade does not affect the HE upgrade, then let's remove the option to cancel the setup and keep the information message only saying that HE upgrade to 3.6 requires an additional process available at <this> link. (whether it is kcs or documentation). +++ This bug was initially created as a clone of Bug #1311027 +++ Description of problem: Upgrading hosted-engine to 3.6 in parallel with OS upgrade from el6 to el7 is problematic. Prevent that by checking in engine-setup if we are a hosted-engine on el6 hosts and suggest to abort setup. --- Additional comment from Artyom on 2016-02-28 10:38:26 EST --- Verified on rhevm-setup-3.6.3.3-0.1.el6.noarch 1) Start with two hosts one RHEL6.7 and second RHEV-H 6.7 2) Deploy 3.5 HE on two hosts 3) Set global maintenance and try to update engine: [ INFO ] Cleaning stale zombie tasks and commands It seems like this engine is self-hosted, and the operating system of some of its hosts is el6 or a variant. Please upgrade the hosts to el7 before upgrading the engine to 3.6. Please check the log file for more details. Replying "No" will abort Setup 4) Reprovision RHEL6.7 host to RHEL7.2 and redeploy it 5) Set global maintenance and try to update engine: [ INFO ] Cleaning stale zombie tasks and commands It seems like this engine is self-hosted, and the operating system of some of its hosts is el6 or a variant. Please upgrade the hosts to el7 before upgrading the engine to 3.6. Please check the log file for more details. Replying "No" will abort Setup 6) Reprovision RHEV-H6.7 host to RHEV-H7.2 and redeploy it 7) Set global maintenance and try to update engine no message about el6 hosts PASS
Based on the discussion we had on the HE upgrade flow 3.5 -> 3.6, we are going to block the upgrade of the manager from 3.5 to 3.6 if it is HE setup and if its HE hosts are RHEL6 based. This should simplify the flow and will remove the need to deal with InClusterUpgrade Policy in 3.6. And our current, tested, existing flow for the upgrade would remain the one documented here: https://access.redhat.com/solutions/2351141 Open questions for this bug: 1. what is the right way to check that this is HE setup. Is it enough searching for HostedEngine VM in the setup? Is it possible by checking its MAC address (Nikolai, can you elaborate here please?) 2. We need to allow a back door to ignore this check, to allow engine upgrade in some corner cases. For instance, if there is a random vm in the system, that is called HostedEngine, but it is not actual HE setup. (Is it actually a real concern and requires a back door? What are other corner cases we are concerned about?) 2.1. What this back door should be? I strongly believe that engine-setup should not continue, if the conditions mentioned on top are met. Then how can we avoid it? Using a force option in the answer file?
Didi, I am assigning this bug to you, since it feels like you had the most knowledge about the possible corner cases we need to address. Thank you.
(In reply to Marina from comment #2) > Open questions for this bug: > 1. what is the right way to check that this is HE setup. Is it enough > searching for HostedEngine VM in the setup? Is it possible by checking its > MAC address (Nikolai, can you elaborate here please?) The engine VM already exists when the engine got installed, at a certain point the engine should detect in import it. The point is the criteria used to identify the engine VM: currently it's a specific VM name. On my opinion the MAC address doesn't help that much since the engine should still know somehow which mac address it has to look for and so... > 2. We need to allow a back door to ignore this check, to allow engine > upgrade in some corner cases. For instance, if there is a random vm in the > system, that is called HostedEngine, but it is not actual HE setup. (Is it > actually a real concern and requires a back door? What are other corner > cases we are concerned about?) I think just this one. > 2.1. What this back door should be? I strongly believe that engine-setup > should not continue, if the conditions mentioned on top are met. Then how > can we avoid it? Using a force option in the answer file? Passing OVESETUP_DB/upgradeWithHeEl6Hosts=bool:True in an answerfile.
(In reply to Simone Tiraboschi from comment #5) > (In reply to Marina from comment #2) > > Open questions for this bug: > > 1. what is the right way to check that this is HE setup. Is it enough > > searching for HostedEngine VM in the setup? Is it possible by checking its > > MAC address (Nikolai, can you elaborate here please?) > > The engine VM already exists when the engine got installed, at a certain > point the engine should detect in import it. > The point is the criteria used to identify the engine VM: currently it's a > specific VM name. > On my opinion the MAC address doesn't help that much since the engine should > still know somehow which mac address it has to look for and so... I agree with Simone. We discussed this quite a lot in bug 1290073 and decided this (vm name) is currently the most reasonable solution. If we need something more robust (do we?), please open an RFE for this and we'll discuss what can be done. > > > 2. We need to allow a back door to ignore this check, to allow engine > > upgrade in some corner cases. For instance, if there is a random vm in the > > system, that is called HostedEngine, but it is not actual HE setup. (Is it > > actually a real concern and requires a back door? What are other corner > > cases we are concerned about?) > > I think just this one. Something else I mentioned elsewhere: An engine managing a hosted-engine cluster of several hosts, in which an el6 host existed in the past and is still in the db, but was decommissioned in real life. The correct thing to do would be to remove the host from the engine, but I can understand if some people would prefer a "back door". > > > 2.1. What this back door should be? I strongly believe that engine-setup > > should not continue, if the conditions mentioned on top are met. Then how > > can we avoid it? Using a force option in the answer file? > > Passing OVESETUP_DB/upgradeWithHeEl6Hosts=bool:True in an answerfile. If we go with that route, we should not keep this key in the generated answerfile, so that it will not be unconsciously copied elsewhere. (In reply to Marina from comment #4) > Didi, > I am assigning this bug to you, since it feels like you had the most > knowledge about the possible corner cases we need to address. > > Thank you. Moving to Simone, who already started working on this (see patch 66775).
-----------------------3.5.8-0.1 to 3.6.10.1-0.1-------------------------------- 1) I've deployed HE over NFS, with DWH and reports, on pair of el6.7 hosts, and added two NFS data storage domains. Hosts: ovirt-host-deploy-1.3.2-1.el6ev.noarch qemu-kvm-rhev-0.12.1.2-2.479.el6_7.5.x86_64 vdsm-4.16.38-1.el6ev.x86_64 sanlock-2.8-2.el6_5.x86_64 rhevm-sdk-python-3.5.6.0-1.el6ev.noarch mom-0.4.1-4.el6ev.noarch ovirt-hosted-engine-ha-1.2.10-1.el6ev.noarch libvirt-client-0.10.2-54.el6_7.6.x86_64 ovirt-hosted-engine-setup-1.2.6.1-1.el6ev.noarch Linux version 2.6.32-431.76.1.el6.x86_64 (mockbuild.eng.bos.redhat.com) (gcc version 4.4.7 20120313 (Red Hat 4.4.7-4) (GCC) ) #1 SMP Tue Nov 1 14:24:00 EDT 2016 Linux 2.6.32-431.76.1.el6.x86_64 #1 SMP Tue Nov 1 14:24:00 EDT 2016 x86_64 x86_64 x86_64 GNU/Linux Red Hat Enterprise Linux Server release 6.7 (Santiago) Engine: rhevm-setup-base-3.5.8-0.1.el6ev.noarch rhevm-spice-client-x86-cab-3.5-3.el6.noarch rhevm-restapi-3.5.8-0.1.el6ev.noarch ovirt-host-deploy-java-1.3.2-1.el6ev.noarch rhevm-spice-client-x86-msi-3.5-3.el6.noarch rhevm-setup-3.5.8-0.1.el6ev.noarch rhevm-reports-setup-3.5.8-1.el6ev.noarch rhevm-setup-plugin-websocket-proxy-3.5.8-0.1.el6ev.noarch rhevm-websocket-proxy-3.5.8-0.1.el6ev.noarch rhevm-dbscripts-3.5.8-0.1.el6ev.noarch rhevm-backend-3.5.8-0.1.el6ev.noarch rhevm-doc-3.5.3-1.el6eng.noarch rhevm-cli-3.5.0.6-1.el6ev.noarch rhevm-log-collector-3.5.4-2.el6ev.noarch rhevm-extensions-api-impl-3.5.8-0.1.el6ev.noarch rhevm-dependencies-3.5.1-1.el6ev.noarch rhevm-sdk-python-3.5.6.0-1.el6ev.noarch ovirt-host-deploy-1.3.2-1.el6ev.noarch rhevm-guest-agent-common-1.0.10-2.el6ev.noarch rhevm-spice-client-x64-cab-3.5-3.el6.noarch rhevm-setup-plugin-ovirt-engine-3.5.8-0.1.el6ev.noarch rhevm-3.5.8-0.1.el6ev.noarch rhevm-dwh-3.5.5-1.el6ev.noarch rhevm-tools-3.5.8-0.1.el6ev.noarch rhevm-branding-rhev-3.5.0-4.el6ev.noarch rhevm-setup-plugin-ovirt-engine-common-3.5.8-0.1.el6ev.noarch rhevm-userportal-3.5.8-0.1.el6ev.noarch rhevm-webadmin-portal-3.5.8-0.1.el6ev.noarch rhevm-iso-uploader-3.5.1-1.el6ev.noarch rhevm-lib-3.5.8-0.1.el6ev.noarch rhevm-spice-client-x64-msi-3.5-3.el6.noarch rhevm-setup-plugins-3.5.4-1.el6ev.noarch rhevm-dwh-setup-3.5.5-1.el6ev.noarch rhevm-reports-3.5.8-1.el6ev.noarch rhevm-image-uploader-3.5.0-4.el6ev.noarch rhev-guest-tools-iso-3.5-15.el6ev.noarch Linux version 2.6.32-573.37.1.el6.x86_64 (mockbuild.eng.bos.redhat.com) (gcc version 4.4.7 20120313 (Red Hat 4.4.7-16) (GCC) ) #1 SMP Wed Oct 26 16:20:09 EDT 2016 Linux 2.6.32-573.37.1.el6.x86_64 #1 SMP Wed Oct 26 16:20:09 EDT 2016 x86_64 x86_64 x86_64 GNU/Linux Red Hat Enterprise Linux Server release 6.7 (Santiago) Engine was provided with 200Gig disk: # lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT vda 252:0 0 200G 0 disk ├─vda1 252:1 0 200M 0 part /boot ├─vda2 252:2 0 2G 0 part [SWAP] └─vda3 252:3 0 197.8G 0 part / sr0 11:0 1 1024M 0 rom 2)Moved hosts to global maintenance. 3)Created backup on engine and copied it to one of the hosts. 4)Ran "yum update -y" on the engine to get latest 3.6.10 bits and to got upgraded from el6.7 to el6.8: rhevm-sdk-python-3.6.9.1-1.el6ev.noarch ovirt-setup-lib-1.0.1-1.el6ev.noarch rhevm-restapi-3.5.8-0.1.el6ev.noarch rhevm-setup-plugin-vmconsole-proxy-helper-3.6.10.1-0.1.el6.noarch rhevm-websocket-proxy-3.6.10.1-0.1.el6.noarch rhevm-guest-agent-common-1.0.11-6.el6ev.noarch rhevm-image-uploader-3.6.1-2.el6ev.noarch rhevm-spice-client-x64-msi-3.6-7.el6.noarch rhevm-setup-base-3.6.10.1-0.1.el6.noarch rhevm-extensions-api-impl-3.6.10.1-0.1.el6.noarch rhevm-dbscripts-3.5.8-0.1.el6ev.noarch rhevm-backend-3.5.8-0.1.el6ev.noarch rhevm-setup-plugin-ovirt-engine-common-3.6.10.1-0.1.el6.noarch rhevm-reports-setup-3.6.5.1-1.el6ev.noarch rhevm-log-collector-3.6.1-1.el6ev.noarch ovirt-host-deploy-java-1.4.1-1.el6ev.noarch rhevm-spice-client-x86-msi-3.6-7.el6.noarch rhevm-branding-rhev-3.6.0-10.el6ev.noarch rhevm-lib-3.6.10.1-0.1.el6.noarch rhevm-3.5.8-0.1.el6ev.noarch rhevm-dwh-3.5.5-1.el6ev.noarch rhev-release-3.6.10-2-001.noarch rhevm-setup-plugins-3.6.5-1.el6ev.noarch rhevm-dwh-setup-3.6.8-1.el6ev.noarch rhevm-spice-client-x64-cab-3.6-7.el6.noarch rhevm-tools-3.5.8-0.1.el6ev.noarch rhevm-setup-3.6.10.1-0.1.el6.noarch rhevm-iso-uploader-3.6.0-1.el6ev.noarch rhevm-spice-client-x86-cab-3.6-7.el6.noarch ovirt-host-deploy-1.4.1-1.el6ev.noarch ovirt-engine-extension-aaa-jdbc-1.0.7-2.el6ev.noarch rhevm-userportal-3.5.8-0.1.el6ev.noarch rhevm-webadmin-portal-3.5.8-0.1.el6ev.noarch rhevm-setup-plugin-ovirt-engine-3.6.10.1-0.1.el6.noarch rhevm-doc-3.6.10-1.el6ev.noarch rhevm-dependencies-3.6.1-1.el6ev.noarch rhevm-reports-3.5.8-1.el6ev.noarch rhevm-setup-plugin-websocket-proxy-3.6.10.1-0.1.el6.noarch rhevm-cli-3.6.9.0-1.el6ev.noarch rhev-guest-tools-iso-3.6-6.el6ev.noarch Linux version 2.6.32-573.37.1.el6.x86_64 (mockbuild.eng.bos.redhat.com) (gcc version 4.4.7 20120313 (Red Hat 4.4.7-16) (GCC) ) #1 SMP Wed Oct 26 16:20:09 EDT 2016 Linux 2.6.32-573.37.1.el6.x86_64 #1 SMP Wed Oct 26 16:20:09 EDT 2016 x86_64 x86_64 x86_64 GNU/Linux Red Hat Enterprise Linux Server release 6.8 (Santiago) 5)Started engine-setup on engine received proper error message and upgrade was forcefully terminated as designed: # engine-setup [ INFO ] Stage: Initializing [ INFO ] Stage: Environment setup Configuration files: ['/etc/ovirt-engine-setup.conf.d/10-packaging-dwh.conf', '/etc/ovirt-engine-setup.conf.d/10-packaging-wsp.conf', '/etc/ovirt-engine-setup.conf.d/10-packaging.conf', '/etc/ovirt-engine-setup.conf.d/20-packaging-rhevm-reports.conf', '/etc/ovirt-engine-setup.conf.d/20-setup-ovirt-post.conf'] Log file: /var/log/ovirt-engine/setup/ovirt-engine-setup-20161221205541-rlpymh.log Version: otopi-1.4.2 (otopi-1.4.2-1.el6ev) [ INFO ] Stage: Environment packages setup [ INFO ] Stage: Programs detection [ INFO ] Stage: Environment setup [ INFO ] Stage: Environment customization Welcome to the RHEV 3.6 setup/upgrade. Please read the RHEV 3.6 install guide https://access.redhat.com/site/documentation/en-US/Red_Hat_Enterprise_Virtualization/3.6/html/Installation_Guide/index.html. Please refer to the RHEV Upgrade Helper application https://access.redhat.com/labs/rhevupgradehelper/ which will guide you in the upgrading process. Would you like to proceed? (Yes, No) [Yes]: --== PRODUCT OPTIONS ==-- Configure VM Console Proxy on this host (Yes, No) [Yes]: --== PACKAGES ==-- [ INFO ] Checking for product updates... Setup has found updates for some packages: PACKAGE: [install] ebay-cors-filter-1.0.1-1.el6.noarch PACKAGE: [updated] jasperreports-server-pro-5.5.0-10.el6ev.noarch PACKAGE: [update] jasperreports-server-pro-6.0.1-5.el6ev.noarch PACKAGE: [install] ovirt-vmconsole-1.0.4-1.el6ev.noarch PACKAGE: [install] ovirt-vmconsole-proxy-1.0.4-1.el6ev.noarch PACKAGE: [updated] redhat-support-plugin-rhev-3.5.0-1.el6ev.noarch PACKAGE: [update] redhat-support-plugin-rhev-3.6.0-18.el6.noarch PACKAGE: [updated] rhevm-3.5.8-0.1.el6ev.noarch PACKAGE: [update] rhevm-3.6.10.1-0.1.el6.noarch PACKAGE: [updated] rhevm-backend-3.5.8-0.1.el6ev.noarch PACKAGE: [update] rhevm-backend-3.6.10.1-0.1.el6.noarch PACKAGE: [updated] rhevm-dbscripts-3.5.8-0.1.el6ev.noarch PACKAGE: [update] rhevm-dbscripts-3.6.10.1-0.1.el6.noarch PACKAGE: [updated] rhevm-dwh-3.5.5-1.el6ev.noarch PACKAGE: [update] rhevm-dwh-3.6.8-1.el6ev.noarch PACKAGE: [updated] rhevm-reports-3.5.8-1.el6ev.noarch PACKAGE: [update] rhevm-reports-3.6.5.1-1.el6ev.noarch PACKAGE: [updated] rhevm-restapi-3.5.8-0.1.el6ev.noarch PACKAGE: [update] rhevm-restapi-3.6.10.1-0.1.el6.noarch PACKAGE: [updated] rhevm-tools-3.5.8-0.1.el6ev.noarch PACKAGE: [update] rhevm-tools-3.6.10.1-0.1.el6.noarch PACKAGE: [install] rhevm-tools-backup-3.6.10.1-0.1.el6.noarch PACKAGE: [updated] rhevm-userportal-3.5.8-0.1.el6ev.noarch PACKAGE: [update] rhevm-userportal-3.6.10.1-0.1.el6.noarch PACKAGE: [install] rhevm-vmconsole-proxy-helper-3.6.10.1-0.1.el6.noarch PACKAGE: [updated] rhevm-webadmin-portal-3.5.8-0.1.el6ev.noarch PACKAGE: [update] rhevm-webadmin-portal-3.6.10.1-0.1.el6.noarch PACKAGE: [updated] vdsm-jsonrpc-java-1.0.15-1.el6ev.noarch PACKAGE: [update] vdsm-jsonrpc-java-1.1.16-1.el6ev.noarch do you wish to update them now? (Yes, No) [Yes]: [ INFO ] Checking for an update for Setup... Setup will not be able to rollback new packages in case of a failure, because the following installed packages were not found in enabled repositories: rhevm-tools-3.5.8-0.1.el6ev.noarch redhat-support-plugin-rhev-3.5.0-1.el6ev.noarch jasperreports-server-pro-5.5.0-10.el6ev.noarch rhevm-dwh-3.5.5-1.el6ev.noarch rhevm-restapi-3.5.8-0.1.el6ev.noarch rhevm-userportal-3.5.8-0.1.el6ev.noarch rhevm-webadmin-portal-3.5.8-0.1.el6ev.noarch vdsm-jsonrpc-java-1.0.15-1.el6ev.noarch rhevm-reports-3.5.8-1.el6ev.noarch rhevm-backend-3.5.8-0.1.el6ev.noarch rhevm-3.5.8-0.1.el6ev.noarch rhevm-dbscripts-3.5.8-0.1.el6ev.noarch Do you want to abort Setup? (Yes, No) [Yes]: no --== ALL IN ONE CONFIGURATION ==-- --== NETWORK CONFIGURATION ==-- Setup can automatically configure the firewall on this system. Note: automatic configuration of the firewall may overwrite current settings. Do you want Setup to configure the firewall? (Yes, No) [Yes]: [ INFO ] iptables will be configured as firewall manager. --== DATABASE CONFIGURATION ==-- The detected DWH database size is 22 MB. Setup can backup the existing database. The time and space required for the database backup depend on its size. This process takes time, and in some cases (for instance, when the size is few GBs) may take several hours to complete. If you choose to not back up the database, and Setup later fails for some reason, it will not be able to restore the database and all DWH data will be lost. Would you like to backup the existing database before upgrading it? (Yes, No) [Yes]: --== OVIRT ENGINE CONFIGURATION ==-- --== STORAGE CONFIGURATION ==-- --== PKI CONFIGURATION ==-- --== APACHE CONFIGURATION ==-- --== SYSTEM CONFIGURATION ==-- --== MISC CONFIGURATION ==-- --== END OF CONFIGURATION ==-- [ INFO ] Stage: Setup validation During execution engine service will be stopped (OK, Cancel) [OK]: [ INFO ] Hosted Engine HA is in Global Maintenance mode. Generated iptables rules are different from current ones. Do you want to review them? (Yes, No) [No]: [ INFO ] Cleaning stale zombie tasks and commands It seems like this engine is self-hosted, and the operating system of some of its hosts is el6 or a variant. Please upgrade ['alma03.qa.lab.tlv.redhat.com', 'alma04.qa.lab.tlv.redhat.com'] to el7 before upgrading the engine to 3.6. Please check the log file for more details. [ ERROR ] Failed to execute stage 'Setup validation': self hosted engine uses el6 hosts [ INFO ] Stage: Clean up Log file is located at /var/log/ovirt-engine/setup/ovirt-engine-setup-20161221205541-rlpymh.log [ INFO ] Generating answer file '/var/lib/ovirt-engine/setup/answers/20161221210151-setup.conf' [ INFO ] Stage: Pre-termination [ INFO ] Stage: Termination [ ERROR ] Execution of setup failed Moving to verified.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHEA-2017-0108.html