Created attachment 1367493 [details] versions Description of problem: After upgrade of hosted-engine packages from 3.6 -> 4.2 the broker service fails to start. Under the journalctl I can see the single error line Dec 13 18:07:19 alma06.qa.lab.tlv.redhat.com python[6336]: detected unhandled Python exception in '/usr/share/ovirt-hosted-engine-ha/ovirt-ha-broker' If you run the service script you recieve INFO:ovirt_hosted_engine_ha.broker.status_broker.StatusBroker:Starting status updating thread INFO:ovirt_hosted_engine_ha.broker.status_broker.StatusBroker:Status broker initialized. INFO:ovirt_hosted_engine_ha.broker.listener.Listener:Initializing RPCServer Traceback (most recent call last): File "/usr/share/ovirt-hosted-engine-ha/ovirt-ha-broker", line 25, in <module> broker.Broker().run() File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/broker.py", line 57, in run self._listener = self._get_listener() File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/broker.py", line 127, in _get_listener self._status_broker_instance) File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/listener.py", line 51, in __init__ self._server = unixrpc.UnixXmlRpcServer(constants.BROKER_SOCKET_FILE) File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/unixrpc.py", line 30, in __init__ request_handler) File "/usr/lib64/python2.7/SocketServer.py", line 419, in __init__ self.server_bind() File "/usr/lib64/python2.7/SocketServer.py", line 430, in server_bind self.socket.bind(self.server_address) File "/usr/lib64/python2.7/socket.py", line 224, in meth return getattr(self._sock,name)(*args) socket.error: [Errno 2] No such file or directory From first glance, looks like systemd does not create /var/run/ovirt-hosted-engine-ha directory although we specified "RuntimeDirectory=ovirt-hosted-engine-ha" under service file. Version-Release number of selected component (if applicable): You can find all version in the attachment How reproducible: Always Steps to Reproduce: 1. Deploy 3.6 HE environment with at least one host 2. Update host packages to 4.2 3. Actual results: After the update broker service fails to start Expected results: All hosted-engine services must work Additional info:
We introduced RuntimeDirectory on https://gerrit.ovirt.org/#/c/73778/ and never back-ported to 4.1
Can you try systemctl daemon-reload first? Just to be sure systemd knows about the service file change?
Reproducible also upgrading from 4.1.z to 4.2-pre
The first issue seams here: [root@c74he20171214h1 ~]# cat /var/tmp/abrt/Python-2017-12-14-00\:48\:21-30697/cmdline /usr/bin/python /usr/share/ovirt-hosted-engine-ha/ovirt-ha-broker[root@c74he20171214h1 ~]# cat /var/tmp/abrt/Python-2017-12-14-00\:48\:21-30697/backtrace __init__.py:925:_open:IOError: [Errno 13] Permission denied: '/var/log/ovirt-hosted-engine-ha/broker.log' Traceback (most recent call last): File "/usr/share/ovirt-hosted-engine-ha/ovirt-ha-broker", line 25, in <module> broker.Broker().run() File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/broker.py", line 44, in run self._initialize_logging() File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/broker.py", line 70, in _initialize_logging disable_existing_loggers=False) File "/usr/lib64/python2.7/logging/config.py", line 78, in fileConfig handlers = _install_handlers(cp, formatters) File "/usr/lib64/python2.7/logging/config.py", line 156, in _install_handlers h = klass(*args) File "/usr/lib64/python2.7/logging/handlers.py", line 169, in __init__ BaseRotatingHandler.__init__(self, filename, 'a', encoding, delay) File "/usr/lib64/python2.7/logging/handlers.py", line 64, in __init__ logging.FileHandler.__init__(self, filename, mode, encoding, delay) File "/usr/lib64/python2.7/logging/__init__.py", line 902, in __init__ StreamHandler.__init__(self, self._open()) File "/usr/lib64/python2.7/logging/__init__.py", line 925, in _open stream = open(self.baseFilename, self.mode) IOError: [Errno 13] Permission denied: '/var/log/ovirt-hosted-engine-ha/broker.log' Local variables in innermost frame: self: <logging.handlers.TimedRotatingFileHandler object at 0x17f4d50> [root@c74he20171214h1 ~]# ls -l /var/log/ovirt-hosted-engine-ha/broker.log -rw-r--r--. 1 root root 65971 14 dic 00.48 /var/log/ovirt-hosted-engine-ha/broker.log [root@c74he20171214h1 ~]# ls -l /var/log/ovirt-hosted-engine-ha/agent.log -rw-r--r--. 1 root root 35368 14 dic 00.46 /var/log/ovirt-hosted-engine-ha/agent.log Workaround: chown vdsm:kvm /var/log/ovirt-hosted-engine-ha/broker.log /var/log/ovirt-hosted-engine-ha/agent.log systemctl restart ovirt-ha-broker systemctl restart ovirt-ha-agent
I tried Simone solution in my environment and it worked like a magic:) so in the end, the problem wasn't in systemd
Hmm so should we just add the chown to post install?
Verified on ovirt-hosted-engine-ha-2.2.2-1.el7ev.noarch After update logs have a correct owner.
This bugzilla is included in oVirt 4.2.0 release, published on Dec 20th 2017. Since the problem described in this bug report should be resolved in oVirt 4.2.0 release, published on Dec 20th 2017, it has been closed with a resolution of CURRENT RELEASE. If the solution does not work for you, please open a new bug report.