Bug 1057139

Created attachment 854405 [details]
VDSM log

Created attachment 854406 [details]
Hosted engine setup log

Can you please also run:
 # rpm -qa |grep ovirt
 # rpm -qa |grep vdsm

[root@localhost ~]# rpm -qa | grep vdsm
vdsm-4.14.1-2.fc19.x86_64
vdsm-cli-4.14.1-2.fc19.noarch
vdsm-xmlrpc-4.14.1-2.fc19.noarch
vdsm-python-4.14.1-2.fc19.x86_64
vdsm-python-zombiereaper-4.14.1-2.fc19.noarch
[root@localhost ~]# rpm -qa | grep ovirt
libgovirt-0.1.0-1.fc19.x86_64
ovirt-guest-agent-common-1.0.8-2.fc19.noarch
ovirt-hosted-engine-ha-1.1.0-0.1.beta1.fc19.noarch
ovirt-host-deploy-1.2.0-0.1.beta.fc19.noarch
ovirt-hosted-engine-setup-1.1.0-0.1.beta1.fc19.noarch
ovirt-engine-sdk-python-3.4.0.2-1.20140114.gitbaccf38.fc19.noarch
ovirt-release-fedora-8-1.noarch

Looking at the logs, the VM has been started by Thread-50:

Thread-50::DEBUG::2014-01-23 15:34:04,390::libvirtconnection::124::root::(wrapper) Unknown libvirterror: ecode: 63 edom: 33 level: 2 message: Error while building firewall: Some rules could not be created for interface vnet0: Failure to execute command '$EBT -t nat -N libvirt-J-vnet0' : '[91mFirewallD is not running[00m'.

Thread-50::DEBUG::2014-01-23 15:34:04,391::vm::2247::vm.Vm::(_startUnderlyingVm) vmId=`8086ca4f-c231-4955-8149-88133673bc7f`::_ongoingCreations released
Thread-50::ERROR::2014-01-23 15:34:04,391::vm::2273::vm.Vm::(_startUnderlyingVm) vmId=`8086ca4f-c231-4955-8149-88133673bc7f`::The vm start process failed
Traceback (most recent call last):
  File "/usr/share/vdsm/vm.py", line 2233, in _startUnderlyingVm
    self._run()
  File "/usr/share/vdsm/vm.py", line 3164, in _run
    self._connection.createXML(domxml, flags),
  File "/usr/lib64/python2.7/site-packages/vdsm/libvirtconnection.py", line 92, in wrapper
    ret = f(*args, **kwargs)
  File "/usr/lib64/python2.7/site-packages/libvirt.py", line 2805, in createXML
    if ret is None:raise libvirtError('virDomainCreateXML() failed', conn=self)
libvirtError: Error while building firewall: Some rules could not be created for interface vnet0: Failure to execute command '$EBT -t nat -N libvirt-J-vnet0' : '[91mFirewallD is not running[00m'.

Thread-50::DEBUG::2014-01-23 15:34:04,394::vm::2725::vm.Vm::(setDownStatus) vmId=`8086ca4f-c231-4955-8149-88133673bc7f`::Changed state to Down: Error while building firewall: Some rules could not be created for interface vnet0: Failure to execute command '$EBT -t nat -N libvirt-J-vnet0' : '[91mFirewallD is not running[00m'.

Hosted engine only support iptables due to limitations in ovirt-engine. Not sure why libvirt should fail on firewalld here.

Laine, any idea why firewalld is suddenly a requirement?

The "FirewallD is not running" message comes from firewall-cmd, which libvirt calls instead of iptables/ebtables iff firewalld was enabled and running on the system *at the time libvirtd was started*.

Look in the logs for the following INFO level message:

   firewalld support disabled|enabled for nwfilter

This should only say "enabled" if firewalld is running at the time that libvirtd is started. If you verify that firewalld is not running when libvirtd is started, but this message says "enabled", then there is a bug in the code that is auto-detecting firewalld.

Otherwise, if firewalld was running at the time libvirtd was started, and the system administrator for some reason stopped it at some later time, they will also need to restart libvirtd in order for libvirt to rescan for iptables vs. firewall-cmd.

Sandro, iirc, you shut firewalld down. According to comment 7, libvirtd should be (re)started only after that.

(In reply to Dan Kenigsberg from comment #8)
> Sandro, iirc, you shut firewalld down. According to comment 7, libvirtd
> should be (re)started only after that.

So the workaround is shutdown firewalld before running hosted-engine --deploy.
And the fix is shutdown firewalld before restarting libvirtd in the code.

> Otherwise, if firewalld was running at the time libvirtd was started, and
> the system administrator for some reason stopped it at some later time, they
> will also need to restart libvirtd in order for libvirt to rescan for
> iptables vs. firewall-cmd.

Laine, I'm going to fix this on hosted engine side, stopping firewalld before restarting libvirtd.
But I really think it's not wise to rely on the state of a firewall manager when the service is started. IMHO libvirt should check which firewall manager is running and use it or interact with systemd to be restarted if firewall manager service change state.

merge on master and 1.1 branch.

This is an automated message.

This BZ should be fixed in oVirt 3.4.0 RC repository, assignee please update the status of the bug to ON_QA once you've verified that the change is included in the build.

works on ovirt-rc

had to use in order to make deploy command work

hosted-engine --deploy --otopi-environment="OVESETUP_SYSTEM/shmmax=int:68719476736"

Moving back to ON_QA since --otopi-environment="OVESETUP_SYSTEM/shmmax=int:68719476736" has no effect on hosted-engine --deploy command.
If it fails without it and it works with it it's really a weird issue.

at the moment this BZ is blocked by bug 1075687, otopi does not detect firewalld properly and therefore firewalld is not stopped during hosted-engine --deploy and vdsm fails to start

this is an automated message: moving to Closed CURRENT RELEASE since oVirt 3.4.0 has been released