Bug 1294517

Summary: Unable to start hosted-engine VM due to internal error
Product: [oVirt] ovirt-node Reporter: Charlie Inglese <cinglese>
Component: GeneralAssignee: Douglas Schilling Landgraf <dougsland>
Status: CLOSED CURRENTRELEASE QA Contact: cshao <cshao>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 3.6CC: bmcclain, bugs, cinglese, cshao, fdeutsch
Target Milestone: ovirt-3.6.2Flags: rule-engine: ovirt-3.6.z+
rule-engine: blocker+
bmcclain: planning_ack+
fdeutsch: devel_ack+
rule-engine: testing_ack+
Target Release: 3.6   
Hardware: Unspecified   
OS: Linux   
Whiteboard:
Fixed In Version: ovirt-node-3.6.1-3 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-03-11 07:21:50 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Node RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
vdsClient_list none

Description Charlie Inglese 2015-12-28 16:56:37 UTC
Created attachment 1110045 [details]
vdsClient_list

Description of problem:
Upon installation of oVirt hosted engine appliance, the VM fails to restart after the initial install and shutdown.

Performing a "vdsClient -s 0 list" produces the following error: 
exitMessage = internal error: Failed to apply firewall rules /usr/sbin/ebtables --concurrent -t nat -N libvirt-J-vnet0: Unable to create lock file /var/lib/ebtables/lock.

Version-Release number of selected component (if applicable):
ovirt-vmconsole-1.0.0-1.el7.centos.noarch
ovirt-setup-lib-1.0.0-1.el7.centos.noarch
ovirt-engine-sdk-python-3.6.0.3-1.el7.centos.noarch
ovirt-vmconsole-host-1.0.0-1.el7.centos.noarch
ovirt-hosted-engine-setup-1.3.0-1.el7.centos.noarch
libgovirt-0.3.3-1.el7.x86_64
ovirt-hosted-engine-ha-1.3.2.1-1.el7.centos.noarch
ovirt-engine-appliance-20151104.0-1.el7.centos.noarch
ovirt-iso-uploader-3.6.0-1.el7.centos.noarch
ovirt-host-deploy-1.4.0-1.el7.centos.noarch


How reproducible:
Everytime

Steps to Reproduce:
1. Install oVirt hosted engine
2. Wait for oVirt hosted engine VM to power-down
3. vdsClient -s 0 list

Actual results:


Expected results:
oVirt hosted engine VM should restart and bring up the oVirt engine and associated services.

Additional info:

Permissions of /var/lib/ebtables (ls -la /var/lib/ebtables):
drwx------.  2 root root    6 Dec 28 16:52 .
drwxr-xr-x. 46 root root 4096 Dec 28 16:19 ..

Comment 1 Fabian Deutsch 2016-01-04 13:32:38 UTC
This can rather be a persistence issue related to node, not the appliance, but this needs further clarification.

Comment 2 Charlie Inglese 2016-01-04 14:29:45 UTC
I believe this is an artifact of SELinux and should be closed as NOT A BUG.

Comment 3 Fabian Deutsch 2016-01-04 14:56:38 UTC
Good hint.

Can you please run 

$ audit2allow -a

Maybe there were some logged denials

Comment 4 Charlie Inglese 2016-01-06 15:51:42 UTC
Fabian,

I ran audit2allow on audit.log for ebtables, and ended up creating an SELinux module for it. Here's the content of the module that audit2allow created:

module ebtables 1.0;

require {
        type iptables_t;
        type var_lib_t;
        class dir { write remove_name create add_name };
        class file { write create unlink open };
}

#============= iptables_t ==============
allow iptables_t var_lib_t:dir { write remove_name create add_name };
allow iptables_t var_lib_t:file { write create unlink open };

Comment 5 Fabian Deutsch 2016-01-06 16:05:00 UTC
Thanks Charlie.

We will include it in our policy.

Comment 6 Fabian Deutsch 2016-01-06 16:55:36 UTC
Chen, have you seen this in your testing?

Comment 7 cshao 2016-01-07 03:35:37 UTC
Hi fabian,

We didn't met this issue with latest RHEV-H 7.2 for RHEV 3.6.2 (rhev-hypervisor7-7.2-20151229.0 build.


Test version:
rhev-hypervisor7-7.2-20151229.0
rhevm-appliance-20151216.0-1.3.6.ova

Due to met Bug 1294783 - Failed to setup engine via rhevm-appliance.ova, so we did testing with the old ova (rhevm-appliance-20151216.0-1.3.6.ova).

Detail testing info please refer test report in VIRT-QE mail list:
Summary: RHEV-H 7.2 for RHEV 3.6.2 (rhev-hypervisor7-7.2-20151229.0) - Acceptance Testing - Partial Fail

Thanks!

Comment 12 Red Hat Bugzilla Rules Engine 2016-01-14 16:47:10 UTC
Bug tickets that are moved to testing must have target release set to make sure tester knows what to test. Please set the correct target release before moving to ON_QA.

Comment 13 cshao 2016-02-24 12:46:42 UTC
Test version:
rhev-hypervisor7-7.2-20160222.0
ovirt-node-3.6.1-7.0.el7ev.noarch
ovirt-node-plugin-hosted-engine-0.3.0-7.el7ev.noarch
ovirt-node-plugin-vdsm-0.6.1-7.el7ev.noarch
ovirt-hosted-engine-setup-1.3.3.3-1.el7
ovirt-hosted-engine-ha-1.3.4.1-1.el7
rhev-m appliance:20160212.0-1.3.6.ova  

Test steps:
1. TUI clean install rhevh
2. Login rhevh, setup network via dhcp.
3. Switch to HE menu.
4. Finish HE configure.
5. Wait for oVirt hosted engine VM to power-down

Test result:
oVirt hosted engine VM can restart and bring up the oVirt engine and associated services.

So the bug is fixed, change bug status to VERIFIED.