Bug 1574384

Summary: [downstream clone - 4.1.11] ovirt-image-daemon fails to start due to permissions on ovirt-image-daemon log file causing host deployment to fail
Product: Red Hat Enterprise Virtualization Manager Reporter: RHV bug bot <rhv-bugzilla-bot>
Component: ovirt-imageio-daemonAssignee: Daniel Erez <derez>
Status: CLOSED NEXTRELEASE QA Contact: Elad <ebenahar>
Severity: high Docs Contact:
Priority: unspecified    
Version: 4.1.10CC: amashah, amureini, bcholler, cshao, derez, jbryant, lsurette, mavital, nsednev, redhat-bugzilla, sbonazzo, stirabos, usurse, ykaul, yturgema
Target Milestone: ovirt-4.1.11Keywords: ZStream
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1571994 Environment:
Last Closed: 2018-05-03 12:27:41 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Storage RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1571994    
Bug Blocks:    

Description RHV bug bot 2018-05-03 07:43:16 UTC
+++ This bug is a downstream clone. The original bug is: +++
+++   bug 1571994 +++
======================================================================

Description of problem:

During a new deployment using RHV-H 4.1.10 image 20180315 (based on RHEL 7.50)

HE deploy fails due to ovirt-imageio-daemon:

2018-04-24 16:09:29 DEBUG otopi.context context._executeMethod:142 method exception
Traceback (most recent call last):
  File "/tmp/ovirt-UltQreVLyp/pythonlib/otopi/context.py", line 132, in _executeMethod
    method['method']()
  File "/tmp/ovirt-UltQreVLyp/otopi-plugins/ovirt-host-deploy/vdsm/packages.py", line 183, in _start
    self.services.state('ovirt-imageio-daemon', True)
  File "/tmp/ovirt-UltQreVLyp/otopi-plugins/otopi/services/systemd.py", line 141, in state
    service=name,
RuntimeError: Failed to start service 'ovirt-imageio-daemon'

After failure, when attempting to manually restart ovirt-imageio-daemon:
After that following to message logs while trying to restart ovirt-imageio-daemon 

Apr 23 16:01:13 rhv-host ovirt-imageio-daemon: stream = open(self.baseFilename, self.mode)
Apr 23 16:01:13 rhv-host ovirt-imageio-daemon: IOError: [Errno 13] Permission denied: '/var/log/ovirt-imageio-daemon/daemon.log'
Apr 23 16:01:13 rhv-host systemd: ovirt-imageio-daemon.service: main process exited, code=exited, status=1/FAILURE
Apr 23 16:01:13 rhv-host systemd: Failed to start oVirt ImageIO Daemon.
Apr 23 16:01:13 rhv-host systemd: Unit ovirt-imageio-daemon.service entered failed state.
Apr 23 16:01:13 rhv-host systemd: ovirt-imageio-daemon.service failed


Permission issue on daemon.log:

[root@rhel-li-3 ~]# ls -laZ /var/log/ovirt-imageio-daemon
drwx------. vdsm kvm  system_u:object_r:var_log_t:s0   .
drwxr-xr-x. root root system_u:object_r:var_log_t:s0   ..
-rw-r--r--. root root system_u:object_r:var_log_t:s0   daemon.log

Once I change the owner daemon.log file
   chown vdsm.kvm /var/log/ovirt-imageio-daemon/daemon.log


Version-Release number of selected component (if applicable):
RHV-H 4.1.10 

How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:

root:root permissions on daemon.log

Expected results:

vdsm:kvm permissions on daemon.log


Additional info:
Somewhat related, but different:
KB Article: https://access.redhat.com/solutions/2930491
Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1425502

(Originally by Amar Shah)

Comment 1 RHV bug bot 2018-05-03 07:43:26 UTC
Yuval, is this due to the issue we had in appliance about wrong permissions?

(Originally by Sandro Bonazzola)

Comment 5 RHV bug bot 2018-05-03 07:43:44 UTC
Sandro, probably not - the ovirt-imageio-daemon is installed in RHVH and looks like the packaging of this rpm is wrong (unless a chown is expeted somewhere):

[root@node-6740 ovirt_imageio_daemon]# rpm -q --dump ovirt-imageio-daemon|grep "daemon.log "
/var/log/ovirt-imageio-daemon/daemon.log 0 1521368643 e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 0100644 root root 0 0 0 X

(Originally by Yuval Turgeman)

Comment 6 RHV bug bot 2018-05-03 07:43:50 UTC
(In reply to Yuval Turgeman from comment #4)
> Sandro, probably not - the ovirt-imageio-daemon is installed in RHVH and
> looks like the packaging of this rpm is wrong (unless a chown is expeted
> somewhere):
> 
> [root@node-6740 ovirt_imageio_daemon]# rpm -q --dump
> ovirt-imageio-daemon|grep "daemon.log "
> /var/log/ovirt-imageio-daemon/daemon.log 0 1521368643
> e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 0100644
> root root 0 0 0 X

The permissions of daemon.log file should be vdsm:kvm. It is set in 
ovirt-imageio-daemon.spec.in: "%dir %attr(755, vdsm, kvm) %{logdir}".
How is the daemon installed in HE env? Any difference than regular host deployment? Which permissions the appliance gets? Perhaps something changed in HE deployment recently (as I'm not aware of such issue in previous builds).

(Originally by Daniel Erez)

Comment 7 RHV bug bot 2018-05-03 07:43:57 UTC
I'm not sure which version of ovirt-imageio-daemon we are talking about, but `rpm -q --dump` queries the rpm db itself.
I checked the latest version as well:

[yturgema@piggie ~/Downloads]$ rpm -qp --dump ovirt-imageio-daemon-1.3.1-0.el7ev.noarch.rpm|grep "daemon.log "
/var/log/ovirt-imageio-daemon/daemon.log 0 1523781570 e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 0100644 root root 0 0 0 X

As for the spec, I think the problem is this:

touch %{buildroot}%{logdir}/daemon.log

since it's created as root

Try to change the %ghost line for daemon.log to something like:
%ghost %attr(644, vdsm, kvm) %{logdir}/daemon.log*

(Originally by Yuval Turgeman)

Comment 8 RHV bug bot 2018-05-03 07:44:03 UTC
Daniel, can you take a look?

(Originally by Yaniv Kaul)

Comment 9 RHV bug bot 2018-05-03 07:44:10 UTC
(In reply to Yuval Turgeman from comment #6)
> I'm not sure which version of ovirt-imageio-daemon we are talking about, but
> `rpm -q --dump` queries the rpm db itself.
> I checked the latest version as well:
> 
> [yturgema@piggie ~/Downloads]$ rpm -qp --dump
> ovirt-imageio-daemon-1.3.1-0.el7ev.noarch.rpm|grep "daemon.log "
> /var/log/ovirt-imageio-daemon/daemon.log 0 1523781570
> e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 0100644
> root root 0 0 0 X
> 
> As for the spec, I think the problem is this:
> 
> touch %{buildroot}%{logdir}/daemon.log
> 
> since it's created as root
> 
> Try to change the %ghost line for daemon.log to something like:
> %ghost %attr(644, vdsm, kvm) %{logdir}/daemon.log*

Added - https://gerrit.ovirt.org/#/c/90699/

Seems fine now:

$ rpm -qp --dump ovirt-imageio-daemon-1.3.2-0.fc26.noarch.rpm | grep "daemon.log "

/var/log/ovirt-imageio-daemon/daemon.log 0 1524746121 e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 0100644 vdsm kvm 0 0 0 X

(Originally by Daniel Erez)

Comment 10 RHV bug bot 2018-05-03 07:44:15 UTC
(In reply to Daniel Erez from comment #8)
> (In reply to Yuval Turgeman from comment #6)
> > I'm not sure which version of ovirt-imageio-daemon we are talking about, but
> > `rpm -q --dump` queries the rpm db itself.
> > I checked the latest version as well:
> > 
> > [yturgema@piggie ~/Downloads]$ rpm -qp --dump
> > ovirt-imageio-daemon-1.3.1-0.el7ev.noarch.rpm|grep "daemon.log "
> > /var/log/ovirt-imageio-daemon/daemon.log 0 1523781570
> > e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 0100644
> > root root 0 0 0 X
> > 
> > As for the spec, I think the problem is this:
> > 
> > touch %{buildroot}%{logdir}/daemon.log
> > 
> > since it's created as root
> > 
> > Try to change the %ghost line for daemon.log to something like:
> > %ghost %attr(644, vdsm, kvm) %{logdir}/daemon.log*
> 
> Added - https://gerrit.ovirt.org/#/c/90699/
> 
> Seems fine now:
> 
> $ rpm -qp --dump ovirt-imageio-daemon-1.3.2-0.fc26.noarch.rpm | grep
> "daemon.log "
> 
> /var/log/ovirt-imageio-daemon/daemon.log 0 1524746121
> e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 0100644
> vdsm kvm 0 0 0 X

@Simone - would it be good enough for fixing the HE deployment?

@Amar - is this issue reproducible only on a specific version? Can you please try it in latest 4.2 build?

(Originally by Daniel Erez)

Comment 11 RHV bug bot 2018-05-03 07:44:20 UTC
It's probably not enough since it's %ghost, it would just fix the `rpm --verify`.  What versions are you running exactly ?

(Originally by Yuval Turgeman)

Comment 13 RHV bug bot 2018-05-03 07:44:32 UTC
(In reply to Daniel Erez from comment #9)
> @Simone - would it be good enough for fixing the HE deployment?

Yes, I think so

(Originally by Simone Tiraboschi)

Comment 17 RHV bug bot 2018-05-03 07:44:54 UTC
(In reply to Simone Tiraboschi from comment #12)
> (In reply to Daniel Erez from comment #9)
> > @Simone - would it be good enough for fixing the HE deployment?
> 
> Yes, I think so

I'll release a new build of imageio then, so we'll see if it gets reproduced.

(Originally by Daniel Erez)

Comment 24 RHV bug bot 2018-05-03 07:45:33 UTC
what are reproduction steps?
Is this a RHEVH specific?

(Originally by Nikolai Sednev)

Comment 27 RHV bug bot 2018-05-03 07:45:49 UTC
Not being reproduced on RHEL systems:
alma03 ~]# ls -laZ /var/log/ovirt-imageio-daemon
drwxr-xr-x. vdsm kvm  system_u:object_r:var_log_t:s0   .
drwxr-xr-x. root root system_u:object_r:var_log_t:s0   ..
-rw-r--r--. vdsm kvm  system_u:object_r:var_log_t:s0   daemon.log

All deployments on all types of storages except from FC were successful for vintage and Node 0 using CLI.

Works for me on these components:
ovirt-engine-4.2.3.3-0.1.el7.noarch
rhvm-appliance-4.2-20180427.0.el7.noarch
ovirt-hosted-engine-setup-2.2.19-1.el7ev.noarch
ovirt-hosted-engine-ha-2.2.11-1.el7ev.noarch
Linux 3.10.0-862.el7.x86_64 #1 SMP Wed Mar 21 18:14:51 EDT 2018 x86_64 x86_64 x86_64 GNU/Linux
Red Hat Enterprise Linux Server release 7.5 (Maipo)

Please retest on RHVH as it turns to be RHVH specific bug.

(Originally by Nikolai Sednev)

Comment 28 RHV bug bot 2018-05-03 07:45:55 UTC
You can reproduce this on RHEL as follows:

1. Make sure ovirt-imageio-daemon service is stopped and /var/log/ovirt-imageio-daemon/daemon.log doesn't exist.
2. systemctl start ovirt-imageio-daemon

3. Make sure ownership for /var/log/ovirt-imageio-daemon/daemon.log is correct:

[root@node-7140 ~]# ls -l /var/log/ovirt-imageio-daemon/daemon.log
-rw-r--r--. 1 vdsm kvm 1210 Apr 30 18:29 /var/log/ovirt-imageio-daemon/daemon.log

4. rpm -V ovirt-imageio-daemon will show wrong user/group:

[root@node-7140 ~]# rpm -V ovirt-imageio-daemon
.....UG..  g /var/log/ovirt-imageio-daemon/daemon.log

5. Ask rpm to fix the files according to its packaging:

[root@node-7140 ~]# rpm --setugids ovirt-imageio-daemon

6. Check again

[root@node-7140 ~]# ls -l /var/log/ovirt-imageio-daemon/daemon.log
-rw-r--r--. 1 root root 1362 May  2 09:15 /var/log/ovirt-imageio-daemon/daemon.log

(Originally by Yuval Turgeman)

Comment 29 RHV bug bot 2018-05-03 07:46:00 UTC
Reproduction in this way will probably work, but the issue is in that on RHEL the file being created with correct UID and UG, thus deployment will be successful, so its solely RHVH specific issue.

(Originally by Nikolai Sednev)

Comment 30 RHV bug bot 2018-05-03 07:46:06 UTC
(In reply to Nikolai Sednev from comment #28)
> Reproduction in this way will probably work, but the issue is in that on
> RHEL the file being created with correct UID and UG, thus deployment will be
> successful, so its solely RHVH specific issue.

Right, unless a RHEL-H user would run for some reason "rpm --setugids"...  Anyway, I just thought it was important to document the steps :)

(Originally by Yuval Turgeman)

Comment 31 RHV bug bot 2018-05-03 07:46:13 UTC
Released fix to ovirt-imageio-daemon-1.3.1.1.

(Originally by Daniel Erez)

Comment 35 Ryan Barry 2018-05-16 12:02:39 UTC
*** Bug 1578712 has been marked as a duplicate of this bug. ***

Comment 36 Franta Kust 2019-05-16 13:08:38 UTC
BZ<2>Jira Resync