Hide Forgot
+++ This bug is a downstream clone. The original bug is: +++ +++ bug 1571994 +++ ====================================================================== Description of problem: During a new deployment using RHV-H 4.1.10 image 20180315 (based on RHEL 7.50) HE deploy fails due to ovirt-imageio-daemon: 2018-04-24 16:09:29 DEBUG otopi.context context._executeMethod:142 method exception Traceback (most recent call last): File "/tmp/ovirt-UltQreVLyp/pythonlib/otopi/context.py", line 132, in _executeMethod method['method']() File "/tmp/ovirt-UltQreVLyp/otopi-plugins/ovirt-host-deploy/vdsm/packages.py", line 183, in _start self.services.state('ovirt-imageio-daemon', True) File "/tmp/ovirt-UltQreVLyp/otopi-plugins/otopi/services/systemd.py", line 141, in state service=name, RuntimeError: Failed to start service 'ovirt-imageio-daemon' After failure, when attempting to manually restart ovirt-imageio-daemon: After that following to message logs while trying to restart ovirt-imageio-daemon Apr 23 16:01:13 rhv-host ovirt-imageio-daemon: stream = open(self.baseFilename, self.mode) Apr 23 16:01:13 rhv-host ovirt-imageio-daemon: IOError: [Errno 13] Permission denied: '/var/log/ovirt-imageio-daemon/daemon.log' Apr 23 16:01:13 rhv-host systemd: ovirt-imageio-daemon.service: main process exited, code=exited, status=1/FAILURE Apr 23 16:01:13 rhv-host systemd: Failed to start oVirt ImageIO Daemon. Apr 23 16:01:13 rhv-host systemd: Unit ovirt-imageio-daemon.service entered failed state. Apr 23 16:01:13 rhv-host systemd: ovirt-imageio-daemon.service failed Permission issue on daemon.log: [root@rhel-li-3 ~]# ls -laZ /var/log/ovirt-imageio-daemon drwx------. vdsm kvm system_u:object_r:var_log_t:s0 . drwxr-xr-x. root root system_u:object_r:var_log_t:s0 .. -rw-r--r--. root root system_u:object_r:var_log_t:s0 daemon.log Once I change the owner daemon.log file chown vdsm.kvm /var/log/ovirt-imageio-daemon/daemon.log Version-Release number of selected component (if applicable): RHV-H 4.1.10 How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: root:root permissions on daemon.log Expected results: vdsm:kvm permissions on daemon.log Additional info: Somewhat related, but different: KB Article: https://access.redhat.com/solutions/2930491 Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1425502 (Originally by Amar Shah)
Yuval, is this due to the issue we had in appliance about wrong permissions? (Originally by Sandro Bonazzola)
Sandro, probably not - the ovirt-imageio-daemon is installed in RHVH and looks like the packaging of this rpm is wrong (unless a chown is expeted somewhere): [root@node-6740 ovirt_imageio_daemon]# rpm -q --dump ovirt-imageio-daemon|grep "daemon.log " /var/log/ovirt-imageio-daemon/daemon.log 0 1521368643 e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 0100644 root root 0 0 0 X (Originally by Yuval Turgeman)
(In reply to Yuval Turgeman from comment #4) > Sandro, probably not - the ovirt-imageio-daemon is installed in RHVH and > looks like the packaging of this rpm is wrong (unless a chown is expeted > somewhere): > > [root@node-6740 ovirt_imageio_daemon]# rpm -q --dump > ovirt-imageio-daemon|grep "daemon.log " > /var/log/ovirt-imageio-daemon/daemon.log 0 1521368643 > e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 0100644 > root root 0 0 0 X The permissions of daemon.log file should be vdsm:kvm. It is set in ovirt-imageio-daemon.spec.in: "%dir %attr(755, vdsm, kvm) %{logdir}". How is the daemon installed in HE env? Any difference than regular host deployment? Which permissions the appliance gets? Perhaps something changed in HE deployment recently (as I'm not aware of such issue in previous builds). (Originally by Daniel Erez)
I'm not sure which version of ovirt-imageio-daemon we are talking about, but `rpm -q --dump` queries the rpm db itself. I checked the latest version as well: [yturgema@piggie ~/Downloads]$ rpm -qp --dump ovirt-imageio-daemon-1.3.1-0.el7ev.noarch.rpm|grep "daemon.log " /var/log/ovirt-imageio-daemon/daemon.log 0 1523781570 e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 0100644 root root 0 0 0 X As for the spec, I think the problem is this: touch %{buildroot}%{logdir}/daemon.log since it's created as root Try to change the %ghost line for daemon.log to something like: %ghost %attr(644, vdsm, kvm) %{logdir}/daemon.log* (Originally by Yuval Turgeman)
Daniel, can you take a look? (Originally by Yaniv Kaul)
(In reply to Yuval Turgeman from comment #6) > I'm not sure which version of ovirt-imageio-daemon we are talking about, but > `rpm -q --dump` queries the rpm db itself. > I checked the latest version as well: > > [yturgema@piggie ~/Downloads]$ rpm -qp --dump > ovirt-imageio-daemon-1.3.1-0.el7ev.noarch.rpm|grep "daemon.log " > /var/log/ovirt-imageio-daemon/daemon.log 0 1523781570 > e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 0100644 > root root 0 0 0 X > > As for the spec, I think the problem is this: > > touch %{buildroot}%{logdir}/daemon.log > > since it's created as root > > Try to change the %ghost line for daemon.log to something like: > %ghost %attr(644, vdsm, kvm) %{logdir}/daemon.log* Added - https://gerrit.ovirt.org/#/c/90699/ Seems fine now: $ rpm -qp --dump ovirt-imageio-daemon-1.3.2-0.fc26.noarch.rpm | grep "daemon.log " /var/log/ovirt-imageio-daemon/daemon.log 0 1524746121 e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 0100644 vdsm kvm 0 0 0 X (Originally by Daniel Erez)
(In reply to Daniel Erez from comment #8) > (In reply to Yuval Turgeman from comment #6) > > I'm not sure which version of ovirt-imageio-daemon we are talking about, but > > `rpm -q --dump` queries the rpm db itself. > > I checked the latest version as well: > > > > [yturgema@piggie ~/Downloads]$ rpm -qp --dump > > ovirt-imageio-daemon-1.3.1-0.el7ev.noarch.rpm|grep "daemon.log " > > /var/log/ovirt-imageio-daemon/daemon.log 0 1523781570 > > e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 0100644 > > root root 0 0 0 X > > > > As for the spec, I think the problem is this: > > > > touch %{buildroot}%{logdir}/daemon.log > > > > since it's created as root > > > > Try to change the %ghost line for daemon.log to something like: > > %ghost %attr(644, vdsm, kvm) %{logdir}/daemon.log* > > Added - https://gerrit.ovirt.org/#/c/90699/ > > Seems fine now: > > $ rpm -qp --dump ovirt-imageio-daemon-1.3.2-0.fc26.noarch.rpm | grep > "daemon.log " > > /var/log/ovirt-imageio-daemon/daemon.log 0 1524746121 > e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 0100644 > vdsm kvm 0 0 0 X @Simone - would it be good enough for fixing the HE deployment? @Amar - is this issue reproducible only on a specific version? Can you please try it in latest 4.2 build? (Originally by Daniel Erez)
It's probably not enough since it's %ghost, it would just fix the `rpm --verify`. What versions are you running exactly ? (Originally by Yuval Turgeman)
(In reply to Daniel Erez from comment #9) > @Simone - would it be good enough for fixing the HE deployment? Yes, I think so (Originally by Simone Tiraboschi)
(In reply to Simone Tiraboschi from comment #12) > (In reply to Daniel Erez from comment #9) > > @Simone - would it be good enough for fixing the HE deployment? > > Yes, I think so I'll release a new build of imageio then, so we'll see if it gets reproduced. (Originally by Daniel Erez)
what are reproduction steps? Is this a RHEVH specific? (Originally by Nikolai Sednev)
Not being reproduced on RHEL systems: alma03 ~]# ls -laZ /var/log/ovirt-imageio-daemon drwxr-xr-x. vdsm kvm system_u:object_r:var_log_t:s0 . drwxr-xr-x. root root system_u:object_r:var_log_t:s0 .. -rw-r--r--. vdsm kvm system_u:object_r:var_log_t:s0 daemon.log All deployments on all types of storages except from FC were successful for vintage and Node 0 using CLI. Works for me on these components: ovirt-engine-4.2.3.3-0.1.el7.noarch rhvm-appliance-4.2-20180427.0.el7.noarch ovirt-hosted-engine-setup-2.2.19-1.el7ev.noarch ovirt-hosted-engine-ha-2.2.11-1.el7ev.noarch Linux 3.10.0-862.el7.x86_64 #1 SMP Wed Mar 21 18:14:51 EDT 2018 x86_64 x86_64 x86_64 GNU/Linux Red Hat Enterprise Linux Server release 7.5 (Maipo) Please retest on RHVH as it turns to be RHVH specific bug. (Originally by Nikolai Sednev)
You can reproduce this on RHEL as follows: 1. Make sure ovirt-imageio-daemon service is stopped and /var/log/ovirt-imageio-daemon/daemon.log doesn't exist. 2. systemctl start ovirt-imageio-daemon 3. Make sure ownership for /var/log/ovirt-imageio-daemon/daemon.log is correct: [root@node-7140 ~]# ls -l /var/log/ovirt-imageio-daemon/daemon.log -rw-r--r--. 1 vdsm kvm 1210 Apr 30 18:29 /var/log/ovirt-imageio-daemon/daemon.log 4. rpm -V ovirt-imageio-daemon will show wrong user/group: [root@node-7140 ~]# rpm -V ovirt-imageio-daemon .....UG.. g /var/log/ovirt-imageio-daemon/daemon.log 5. Ask rpm to fix the files according to its packaging: [root@node-7140 ~]# rpm --setugids ovirt-imageio-daemon 6. Check again [root@node-7140 ~]# ls -l /var/log/ovirt-imageio-daemon/daemon.log -rw-r--r--. 1 root root 1362 May 2 09:15 /var/log/ovirt-imageio-daemon/daemon.log (Originally by Yuval Turgeman)
Reproduction in this way will probably work, but the issue is in that on RHEL the file being created with correct UID and UG, thus deployment will be successful, so its solely RHVH specific issue. (Originally by Nikolai Sednev)
(In reply to Nikolai Sednev from comment #28) > Reproduction in this way will probably work, but the issue is in that on > RHEL the file being created with correct UID and UG, thus deployment will be > successful, so its solely RHVH specific issue. Right, unless a RHEL-H user would run for some reason "rpm --setugids"... Anyway, I just thought it was important to document the steps :) (Originally by Yuval Turgeman)
Released fix to ovirt-imageio-daemon-1.3.1.1. (Originally by Daniel Erez)
*** Bug 1578712 has been marked as a duplicate of this bug. ***
BZ<2>Jira Resync