Description of problem: after running hosted-engine --upgrade-appliance i see following when trying to ssh into the HE VM: ~~~ $ ssh 10.34.60.214 @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY! Someone could be eavesdropping on you right now (man-in-the-middle attack)! It is also possible that the RSA host key has just been changed. The fingerprint for the RSA key sent by the remote host is 2d:6c:f6:e6:fb:90:5e:04:88:83:68:b2:a9:2b:66:56. Please contact your system administrator. Add correct host key in /home/brq/jbelka/.ssh/known_hosts to get rid of this message. Offending key in /home/brq/jbelka/.ssh/known_hosts:180 Password authentication is disabled to avoid man-in-the-middle attacks. Keyboard-interactive authentication is disabled to avoid man-in-the-middle attacks. Agent forwarding is disabled to avoid man-in-the-middle attacks. ~~~ Thus HE VM sshd keys are changed, IMO this is not good. Version-Release number of selected component (if applicable): ovirt-hosted-engine-setup-2.0.0-1.el7ev.noarch How reproducible: 100% Steps to Reproduce: 1. hosted-engine --upgrade-appliance 2. ssh after the action is (almost) finished 3. Actual results: HE VM (appliance) sshd keys are changed Expected results: HE VM (appliacne) original sshd keys should be kept Additional info: workaround - ssh-keygen -R $HE_VM_IP/FQDN
Not really sure since we are generally not connecting to the engine VM via SSH and by default SSH root access is disabled on our downstream appliance. If we want to save and restore it, we probably need to have this feature in engine-backup. Didi?
I'd not patch engine-backup to backup ssh private keys. 1. It's sensitive data not really related to the engine or engine-backup. 2. The user might have in the vm any number of other things that are useful to copy over. I'd say the core issue here is only the naming of the command - it's not "upgrade" but "migration". User must understand that we are not doing in-place upgrade but create a new machine and copy only the engine to it. For this specific bug, we can add text to the end of the tool saying "Please note that ssh keys were regenerated, you have to remove old keys from your clients", if ssh was enabled. Or something like that.
Could anybody consider to redesign the appliance to be real "blackbox" and not a kind of pre-prepared RHEL image? Maybe NGN way would be better, I mean that oVirt data would be clearly separated from other part of the image and other part of the image would be easy changeable. It seems to me that having the appliance just as a preprepared image has brought all issues we have now - how to migrate to newer OS, what to backup, how to reconfigure engine etc... My 2 cents...
Maybe the engine in a container? :-)
(In reply to Jiri Belka from comment #3) > Could anybody consider to redesign the appliance to be real "blackbox" Fabian?
This sounds as if the appliance is updated by installing the new appliance image and then restoring a previous backup into it? But no, the appliance image was never intended to be a black box like NGN is. Also containers, which are just a different implementation, will have the same problem as long was we don't design the appliance around addressing this kind of problems. What I want to say: Data persistence does not come for free, it needs a proper design.
(In reply to Yedidyah Bar David from comment #2) > For this specific bug, we can add text to the end of the tool saying "Please > note that ssh keys were regenerated, you have to remove old keys from your > clients", if ssh was enabled. Or something like that. Looks ok to me. Yaniv?
(In reply to Sandro Bonazzola from comment #7) > (In reply to Yedidyah Bar David from comment #2) > > For this specific bug, we can add text to the end of the tool saying "Please > > note that ssh keys were regenerated, you have to remove old keys from your > > clients", if ssh was enabled. Or something like that. > > Looks ok to me. Yaniv? Acceptable by me.
I've performed an upgrade from rhevm-appliance-20160831.0-1 to rhevm-appliance-20160922.0-1, while using "hosted-engine --upgrade-appliance" functionality. During the upgrade, I've seen this info at the end of the upgrade: [ INFO ] Stage: Termination [ INFO ] Hosted Engine successfully upgraded [ INFO ] Please exit global maintenance mode to restart the engine VM. [ INFO ] Please note that the engine VM ssh keys have changed. Please remove the engine VM entry in ssh known_hosts on your clients. Moving this bug to verified as it works for me on these components on hosts: rhev-release-4.0.5-5-001.noarch sanlock-3.2.4-3.el7_2.x86_64 ovirt-setup-lib-1.0.2-1.el7ev.noarch ovirt-vmconsole-host-1.0.4-1.el7ev.noarch vdsm-4.18.15.2-1.el7ev.x86_64 libvirt-client-1.2.17-13.el7_2.6.x86_64 ovirt-hosted-engine-ha-2.0.4-1.el7ev.noarch ovirt-imageio-common-0.3.0-0.el7ev.noarch qemu-kvm-rhev-2.3.0-31.el7_2.23.x86_64 ovirt-hosted-engine-setup-2.0.3-2.el7ev.noarch ovirt-host-deploy-1.5.3-1.el7ev.noarch ovirt-engine-sdk-python-3.6.9.1-1.el7ev.noarch ovirt-imageio-daemon-0.4.0-0.el7ev.noarch ovirt-vmconsole-1.0.4-1.el7ev.noarch mom-0.5.8-1.el7ev.noarch rhevm-appliance-20160922.0-1.el7ev.noarch Linux version 3.10.0-327.36.3.el7.x86_64 (mockbuild.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-4) (GCC) ) #1 SMP Thu Oct 20 04:56:07 EDT 2016 Linux 3.10.0-327.36.3.el7.x86_64 #1 SMP Thu Oct 20 04:56:07 EDT 2016 x86_64 x86_64 x86_64 GNU/Linux Red Hat Enterprise Linux Server release 7.2 (Maipo) Components on engine: ovirt-engine-dwh-4.0.5-1.el7ev.noarch ovirt-engine-dwh-setup-4.0.5-1.el7ev.noarch ovirt-vmconsole-proxy-1.0.4-1.el7ev.noarch eap7-wildfly-web-console-eap-2.8.27-1.Final_redhat_1.1.ep7.el7.noarch ovirt-vmconsole-1.0.4-1.el7ev.noarch ovirt-engine-vmconsole-proxy-helper-4.0.5.4-0.1.el7ev.noarch ovirt-engine-setup-plugin-vmconsole-proxy-helper-4.0.5.4-0.1.el7ev.noarch qemu-guest-agent-2.3.0-4.el7.x86_64 rhevm-guest-agent-common-1.0.12-3.el7ev.noarch rhevm-spice-client-x64-msi-4.0-3.el7ev.noarch rhevm-branding-rhev-4.0.0-5.el7ev.noarch rhevm-dependencies-4.0.0-1.el7ev.noarch rhev-release-4.0.5-5-001.noarch rhevm-spice-client-x86-msi-4.0-3.el7ev.noarch rhevm-4.0.5.4-0.1.el7ev.noarch rhevm-guest-agent-common-1.0.12-3.el7ev.noarch rhevm-setup-plugins-4.0.0.3-1.el7ev.noarch rhev-guest-tools-iso-4.0-6.el7ev.noarch rhevm-doc-4.0.5-1.el7ev.noarch Linux version 3.10.0-327.36.1.el7.x86_64 (mockbuild.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-4) (GCC) ) #1 SMP Wed Aug 17 03:02:37 EDT 2016 Linux 3.10.0-327.36.1.el7.x86_64 #1 SMP Wed Aug 17 03:02:37 EDT 2016 x86_64 x86_64 x86_64 GNU/Linux Red Hat Enterprise Linux Server release 7.2 (Maipo)