Description of problem: Hosted engine can't get started on host after host or HE's VM were rebooted if before their reboot a vNIC with Empty profile was added. jsonrpc.Executor/0::INFO::2016-06-21 18:33:56,677::vm::2100::virt.vm::(hotplugNic) vmId=`24e9a52f-498f-4f76-b44c-f063a850b89f`::Hotplug NIC xml: <interface type="bridge"> <mac address="00:1a:4a:16:01:53"/> <model type="virtio"/> <source bridge=";vdsmdummy;"/> <link state="up"/> </interface> Thread-169::DEBUG::2016-06-21 18:44:41,028::bindingxmlrpc::1242::vds::(wrapper) return vmGetStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Down', 'exitMessage': "Cannot get interface MTU on 'None': No such device", 'statusTime': '4294965710', 'vmId': '24e9a52f-498f-4f76-b44c-f063a850b89f', 'exitReason': 1, 'exitCode': 1}]} The regular guest VMs with the same vNIC Empty profiles receiving "vdsmdummy" instead of "none", and getting normal MTU value of 1500. Version-Release number of selected component (if applicable): Host: libvirt-client-1.2.17-13.el7_2.5.x86_64 ovirt-vmconsole-1.0.3-1.el7ev.noarch ovirt-engine-sdk-python-3.6.7.0-1.el7ev.noarch vdsm-4.18.3-0.el7ev.x86_64 ovirt-setup-lib-1.0.2-1.el7ev.noarch ovirt-hosted-engine-ha-2.0.0-1.el7ev.noarch mom-0.5.4-1.el7ev.noarch sanlock-3.2.4-2.el7_2.x86_64 ovirt-host-deploy-1.5.0-1.el7ev.noarch qemu-kvm-rhev-2.3.0-31.el7_2.16.x86_64 ovirt-vmconsole-host-1.0.3-1.el7ev.noarch ovirt-hosted-engine-setup-2.0.0.2-1.el7ev.noarch Linux version 3.10.0-327.22.2.el7.x86_64 (mockbuild.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-4) (GCC) ) #1 SMP Thu Jun 9 10:09:10 EDT 2016 Linux 3.10.0-327.22.2.el7.x86_64 #1 SMP Thu Jun 9 10:09:10 EDT 2016 x86_64 x86_64 x86_64 GNU/Linux Red Hat Enterprise Linux Server release 7.2 (Maipo) Engine: rhevm-doc-4.0.0-2.el7ev.noarch rhev-release-4.0.0-18-001.noarch rhevm-setup-plugins-4.0.0.1-1.el7ev.noarch rhevm-spice-client-x64-msi-4.0-2.el7ev.noarch rhevm-branding-rhev-4.0.0-1.el7ev.noarch rhevm-4.0.0.6-0.1.el7ev.noarch rhevm-guest-agent-common-1.0.12-2.el7ev.noarch rhevm-dependencies-4.0.0-1.el7ev.noarch rhevm-spice-client-x86-msi-4.0-2.el7ev.noarch rhev-guest-tools-iso-4.0-2.el7ev.noarch Linux version 3.10.0-327.22.2.el7.x86_64 (mockbuild.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-4) (GCC) ) #1 SMP Thu Jun 9 10:09:10 EDT 2016 Linux 3.10.0-327.22.2.el7.x86_64 #1 SMP Thu Jun 9 10:09:10 EDT 2016 x86_64 x86_64 x86_64 GNU/Linux Red Hat Enterprise Linux Server release 7.2 (Maipo) How reproducible: 100% Steps to Reproduce: 1.Add new vNIC with Empty profile to HE's VM. 2.Reboot the host or the HE's VM. 3. Actual results: Engine could not get started any more on host. Expected results: Engine should get started successfully. Additional info: Logs from host being attached, together with screen capture of what was done.
Screen capture: https://drive.google.com/a/redhat.com/file/d/0B85BEaDBcF88bDlMdGgtTlhzRVE/view?usp=sharing
Created attachment 1170332 [details] sosreport from host alma03
Created attachment 1170333 [details] sosreport from host alma04
Hosted Engine asked Vdsm to hot plug a vNIC to a non-existing network named "None". HE should send the empty string instead, if it wants to have a disconnected vNIC on the VM. I believe that it is a clone of a bug where the empty string was serialized as "None" into the OVF, but I cannot failed to find it.
(In reply to Dan Kenigsberg from comment #4) > Hosted Engine asked Vdsm to hot plug a vNIC to a non-existing network named > "None". HE should send the empty string instead, if it wants to have a > disconnected vNIC on the VM. > > I believe that it is a clone of a bug where the empty string was serialized > as "None" into the OVF, but I cannot failed to find it. Looks pretty familiar, isn't it this one? https://bugzilla.redhat.com/show_bug.cgi?id=1269768 Look at the https://bugzilla.redhat.com/show_bug.cgi?id=1269768#c31 a0a7-2c0339ecd8e1`::Task._run: d1c6366c-3709-4fe9-a0a7-2c0339ecd8e1 (None, u'00000001-0001-0001-0001-000000000212', u'Default', u'97f5a165-4df5-4bce-99cc-8a634753bc54', [u'97f5a165-4df5-4bce-99cc-8a634753bc54'], 1, None, 5, 60, 10, 3) {} failed - stopping task
I would like to add, that this is happening only on HE-VM, regular guest-VMs are not affected by this scenario, even if vNICs with an Empty profile being added to regular VMs.
If a customer actually succeeds to do this (not hard, just add a nic without associating it with a profile), HE will become non-operational, until hosted-engine VM would not be started with a modified vm.conf without the problematic nic. I believe this bug needs to be set with higher priority and once we get a fix, cloned to 3.6.z as well. For now, I will create a kcs, but if this would happen in customer environment, it will automatically be considered sev1 issue. Should I clone it to d/s? What would be the best next step here?
We should block adding a vNic with an empty profile to HE VM. It's worth noting that an empty profile is used in import/export of a VM where the Network of the exported nic doesn't exist in the engine setup
Bug tickets must have version flags set prior to targeting them to a release. Please ask maintainer to set the correct version flags and only then set the target milestone.
(In reply to Roy Golan from comment #8) > We should block adding a vNic with an empty profile to HE VM. > > It's worth noting that an empty profile is used in import/export of a VM > where the Network of the exported nic doesn't exist in the engine setup IMHO you should deal with this newly added vNIC with an empty profile the same way you're dealing with it on regular guest-VMs, meaning to give it "dummyinterface" instead of giving it none/0.
Getting an error when trying to add NIC with an empty profile: Operation Canceled Error while executing action: HostedEngine: Hosted Engine VM cannot have network interface with empty profile. Works for me on these components: Engine: ovirt-engine-setup-plugin-ovirt-engine-common-4.0.2.1-0.1.el7ev.noarch ovirt-engine-websocket-proxy-4.0.2.1-0.1.el7ev.noarch ovirt-engine-userportal-4.0.2.1-0.1.el7ev.noarch ovirt-iso-uploader-4.0.0-1.el7ev.noarch ovirt-engine-setup-plugin-vmconsole-proxy-helper-4.0.2.1-0.1.el7ev.noarch ovirt-vmconsole-proxy-1.0.4-1.el7ev.noarch ovirt-engine-dwh-setup-4.0.2-1.el7ev.noarch ovirt-engine-webadmin-portal-debuginfo-4.0.2.1-0.1.el7ev.noarch ovirt-engine-backend-4.0.2.1-0.1.el7ev.noarch ovirt-engine-dwh-4.0.2-1.el7ev.noarch ovirt-engine-cli-3.6.7.0-1.el7ev.noarch ovirt-vmconsole-1.0.4-1.el7ev.noarch ovirt-setup-lib-1.0.2-1.el7ev.noarch ovirt-engine-dashboard-1.0.1-0.el7ev.x86_64 ovirt-engine-setup-plugin-websocket-proxy-4.0.2.1-0.1.el7ev.noarch ovirt-engine-setup-4.0.2.1-0.1.el7ev.noarch ovirt-engine-extensions-api-impl-4.0.2.1-0.1.el7ev.noarch ovirt-engine-userportal-debuginfo-4.0.2.1-0.1.el7ev.noarch ovirt-engine-webadmin-portal-4.0.2.1-0.1.el7ev.noarch ovirt-engine-dbscripts-4.0.2.1-0.1.el7ev.noarch ovirt-engine-restapi-4.0.2.1-0.1.el7ev.noarch ovirt-engine-sdk-python-3.6.7.0-1.el7ev.noarch ovirt-log-collector-4.0.0-1.el7ev.noarch ovirt-host-deploy-java-1.5.1-1.el7ev.noarch ovirt-engine-lib-4.0.2.1-0.1.el7ev.noarch ovirt-engine-setup-plugin-ovirt-engine-4.0.2.1-0.1.el7ev.noarch ovirt-engine-tools-backup-4.0.2.1-0.1.el7ev.noarch ovirt-engine-4.0.2.1-0.1.el7ev.noarch python-ovirt-engine-sdk4-4.0.0-0.5.a5.el7ev.x86_64 ovirt-engine-setup-base-4.0.2.1-0.1.el7ev.noarch ovirt-engine-vmconsole-proxy-helper-4.0.2.1-0.1.el7ev.noarch ovirt-engine-tools-4.0.2.1-0.1.el7ev.noarch ovirt-image-uploader-4.0.0-1.el7ev.noarch ovirt-host-deploy-1.5.1-1.el7ev.noarch ovirt-engine-extension-aaa-jdbc-1.1.0-1.el7ev.noarch rhev-guest-tools-iso-4.0-5.el7ev.noarch rhev-release-4.0.2-3-001.noarch rhevm-4.0.2.1-0.1.el7ev.noarch rhevm-setup-plugins-4.0.0.2-1.el7ev.noarch rhevm-doc-4.0.0-3.el7ev.noarch rhevm-branding-rhev-4.0.0-3.el7ev.noarch rhevm-guest-agent-common-1.0.12-3.el7ev.noarch rhevm-spice-client-x64-msi-4.0-3.el7ev.noarch rhevm-spice-client-x86-msi-4.0-3.el7ev.noarch rhevm-dependencies-4.0.0-1.el7ev.noarch rhev-release-4.0.1-2-001.noarch Linux version 3.10.0-327.22.2.el7.x86_64 (mockbuild.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-4) (GCC) ) #1 SMP Thu Jun 9 10:09:10 EDT 2016 Linux 3.10.0-327.22.2.el7.x86_64 #1 SMP Thu Jun 9 10:09:10 EDT 2016 x86_64 x86_64 x86_64 GNU/Linux Red Hat Enterprise Linux Server release 7.2 (Maipo) Host: ovirt-imageio-daemon-0.3.0-0.el7ev.noarch libvirt-client-1.2.17-13.el7_2.5.x86_64 ovirt-vmconsole-1.0.4-1.el7ev.noarch vdsm-4.18.9-1.el7ev.x86_64 qemu-kvm-rhev-2.3.0-31.el7_2.19.x86_64 ovirt-host-deploy-1.5.1-1.el7ev.noarch ovirt-hosted-engine-ha-2.0.1-1.el7ev.noarch ovirt-hosted-engine-setup-2.0.1.3-1.el7ev.noarch ovirt-engine-sdk-python-3.6.7.0-1.el7ev.noarch mom-0.5.5-1.el7ev.noarch ovirt-setup-lib-1.0.2-1.el7ev.noarch ovirt-imageio-common-0.3.0-0.el7ev.noarch ovirt-vmconsole-host-1.0.4-1.el7ev.noarch rhev-release-4.0.2-3-001.noarch sanlock-3.2.4-3.el7_2.x86_64 Linux version 3.10.0-327.30.1.el7.x86_64 (mockbuild.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-4) (GCC) ) #1 SMP Wed Jul 13 22:09:46 EDT 2016 Linux 3.10.0-327.30.1.el7.x86_64 #1 SMP Wed Jul 13 22:09:46 EDT 2016 x86_64 x86_64 x86_64 GNU/Linux Red Hat Enterprise Linux Server release 7.2 (Maipo)