This service will be undergoing maintenance at 00:00 UTC, 2017-10-23 It is expected to last about 30 minutes
Bug 1293617 - [RHEV-H] - There is no execute permissions for other under /var/run/vdsm/netconf/ in rhev-h 7.2 with vdsm-4.16.31-1
[RHEV-H] - There is no execute permissions for other under /var/run/vdsm/netc...
Status: CLOSED CURRENTRELEASE
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: vdsm (Show other bugs)
3.5.7
x86_64 Linux
urgent Severity urgent
: ovirt-3.6.2
: 3.6.2
Assigned To: Edward Haas
Michael Burman
: AutomationBlocker, Regression, TestBlocker, ZStream
Depends On:
Blocks: 1286442 1294075
  Show dependency treegraph
 
Reported: 2015-12-22 07:35 EST by Michael Burman
Modified: 2016-03-23 18:25 EDT (History)
15 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1294075 (view as bug list)
Environment:
Last Closed: 2016-03-23 18:25:47 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: Node
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
Logs (31.61 KB, application/x-gzip)
2015-12-22 07:35 EST, Michael Burman
no flags Details


External Trackers
Tracker ID Priority Status Summary Last Updated
oVirt gerrit 50984 master MERGED net: set netconf/* owner to 'vdsm' 2015-12-24 04:31 EST
oVirt gerrit 51053 ovirt-3.6 MERGED net: set netconf/* owner to 'vdsm' 2015-12-24 08:11 EST
oVirt gerrit 51103 master ABANDONED Fixing the other permissions of the vdsm network run time directory 2015-12-28 04:16 EST
oVirt gerrit 51104 ovirt-3.5 MERGED netconf: own /var/run/vdsm/netconf as well 2015-12-29 03:07 EST
oVirt gerrit 51125 master MERGED netconf: own /var/run/vdsm/netconf as well 2015-12-29 01:32 EST
oVirt gerrit 51133 ovirt-3.6 MERGED netconf: own /var/run/vdsm/netconf as well 2015-12-29 02:09 EST

  None (edit)
Description Michael Burman 2015-12-22 07:35:16 EST
Created attachment 1108633 [details]
Logs

Description of problem:
[RHEV-H] - There is no execute permissions for other under /var/run/vdsm/netconf/ in rhev-h 7.2 with vdsm-4.16.31-1 and because of that, server  can't be added to engine 3.5.7.

Calling 'Host.getCapabilities' failed

[root@orchid-vds2 vdsm]# ls -ld /var/run/vdsm/netconf/nets/
drwxrwx---. 2 root root 60 Dec 22 12:16 /var/run/vdsm/netconf/nets/

[root@orchid-vds2 vdsm]# ls -l /var/run/vdsm/netconf/nets/
total 4
-rw-rw----. 1 root root 138 Dec 22 12:16 rhevm

[root@orchid-vds2 ~]# tail -f /var/log/vdsm/vdsm.log 
  File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 501, in _serveRequest
  File "/usr/share/vdsm/rpc/Bridge.py", line 271, in _dynamicMethod
  File "/usr/share/vdsm/API.py", line 1266, in getCapabilities
  File "/usr/share/vdsm/caps.py", line 602, in get
  File "/usr/lib/python2.7/site-packages/vdsm/netinfo.py", line 757, in get
  File "/usr/lib/python2.7/site-packages/vdsm/netinfo.py", line 711, in _libvirtNets2vdsm
  File "/usr/lib/python2.7/site-packages/vdsm/netconfpersistence.py", line 438, in __init__
  File "/usr/lib/python2.7/site-packages/vdsm/netconfpersistence.py", line 113, in __init__
  File "/usr/lib/python2.7/site-packages/vdsm/netconfpersistence.py", line 137, in _getConfigs
OSError: [Errno 13] Permission denied: '/var/run/vdsm/netconf/nets/'


Version-Release number of selected component (if applicable):
ovirt-node-3.2.3-30.el7.noarch
vdsm-4.16.31-1.el7ev.x86_64
3.5.7-0.1.el6ev

How reproducible:
100

Steps to Reproduce:
1. Install clean Red Hat Enterprise Virtualization Hypervisor release 7.2 (20151218.1.el7ev)
2. Configure NIC with dhcp via TUI and register to engine 3.5.7
3. Approve server 

Actual results:
Calling 'Host.getCapabilities' failed 
OSError: [Errno 13] Permission denied: '/var/run/vdsm/netconf/nets/'

Expected results:
Should work as expected
Comment 2 cshao 2015-12-23 00:59:39 EST
I can reproduce this bug.

Test version:
rhev-hypervisor7-7.2-20151218.1
ovirt-node-3.2.3-30.el7.noarch
rhevm-3.5.7-0.1.el6ev

Test steps:
1. Install clean Red Hat Enterprise Virtualization Hypervisor release 7.2 (20151218.1.el7ev)
2. Configure NIC with dhcp via TUI and register to engine 3.5.7
3. Approve server 

Test result:
Failed to up RHEV-H on RHEV-M side.
Comment 3 cshao 2015-12-23 02:07:17 EST
The same issue occurs on rhev-hypervisor6-6.7-20151218.1 (ovirt-node-3.2.3-30.el6.noarch) build.
Comment 4 cshao 2015-12-23 02:33:00 EST
(In reply to shaochen from comment #3)
> The same issue occurs on rhev-hypervisor6-6.7-20151218.1
> (ovirt-node-3.2.3-30.el6.noarch) build.

Add RHEV-H from RHEV-M side can succeed and RHEV-H can up.
Comment 5 Fabian Deutsch 2015-12-23 04:46:25 EST
I am not sure why this is popping up in 3.5.7, but I am quite sure that this /var/run dir is created by vdsm itself:

From lib/vdsm/netconfpersistence.py

    @staticmethod
    def _setConfig(config, path):
        dirPath = os.path.dirname(path)
        try:
            os.makedirs(dirPath)   <-- HERE
        except OSError as ose:
            if errno.EEXIST != ose.errno:
                raise
        with open(path, 'w') as configurationFile:
            json.dump(config, configurationFile)


Ido, am I understanding the code correct and shouldn't the permissions be set here?
Comment 6 Ido Barkan 2015-12-23 05:04:39 EST
this is called in a lazy manner- meaning- only when something is indeed being written in the persistence directory. IIUC the problem here is to read the contents of the directory ('x' permission on the dir) and this should happen when VDSM is installed.
Comment 7 Fabian Deutsch 2015-12-23 05:15:39 EST
Right - Let me elaborate, on RHEV-H 3.5.7 I see:
[root@dhcp-10-46 admin]# cat /etc/default/version
PRODUCT='Red Hat Enterprise Virtualization Hypervisor'
PRODUCT_SHORT='RHEV-H'
PRODUCT_CODE=rhevh
RECIPE_SHA256=8cb6d13d4e004993f75d1890e4475beb8957f272df4efd3757b8fc9191e24340
RECIPE_RPM=ovirt-node-plugin-vdsm-recipe-0.2.0-26.el7ev.noarch
PACKAGE=rhev-hypervisor
VERSION=7.2
RELEASE=20151218.1.el7ev


[root@dhcp-10-46 admin]# findmnt /var/run/
TARGET SOURCE FSTYPE OPTIONS
/run   tmpfs  tmpfs  rw,nosuid,nodev,seclabel,mode=755


We are speaking about a problem reading from a file in /var/run - on RHEV-H /var/run is a tmpfs (I need to check RHEL-H), thus everything in /var/run needs to be created at runtime.
This indicates that the lazy path you mentioned above might be triggered.

[root@dhcp-10-46 admin]# find /var/run/vdsm -ls
 22238    0 drwxr-xr-x   5 vdsm     kvm           160 Dec 23 03:09 /var/run/vdsm
 45964    0 drwxr-xr-x   2 vdsm     kvm            60 Dec 23 03:09 /var/run/vdsm/lvm
 45965    4 -rw-r--r--   1 vdsm     kvm           309 Dec 23 03:09 /var/run/vdsm/lvm/lvm.conf
 51455    0 -rw-r--r--   1 vdsm     kvm             0 Dec 23 03:26 /var/run/vdsm/client.log
 43538    0 -rw-r--r--   1 root     root            0 Dec 23 03:09 /var/run/vdsm/nets_restored
 43514    0 srwxr-xr-x   1 vdsm     kvm             0 Dec 23 03:09 /var/run/vdsm/svdsm.sock
 22284    0 drwxr-xr-x   2 vdsm     kvm            40 Dec 23 03:09 /var/run/vdsm/trackedInterfaces
 22239    0 drwxr-xr-x   2 vdsm     kvm            40 Dec 23 05:40 /var/run/vdsm/sourceRoutes

The snippet above shows the state before adding the host to RHEV-M - it shows that the netconf persistence dir is not there.

The following snippet shows the state after adding the host:

[root@dhcp-10-46 admin]# find /var/run/vdsm -ls
 22238    0 drwxr-xr-x   6 vdsm     kvm           180 Dec 23  2015 /var/run/vdsm
289628    0 drwxrwx---   3 root     root           60 Dec 23  2015 /var/run/vdsm/netconf
289629    0 drwxrwx---   2 root     root           60 Dec 23  2015 /var/run/vdsm/netconf/nets
289630    4 -rw-rw----   1 root     root          135 Dec 23  2015 /var/run/vdsm/netconf/nets/rhevm
 45964    0 drwxr-xr-x   2 vdsm     kvm            60 Dec 23  2015 /var/run/vdsm/lvm
 45965    4 -rw-r--r--   1 vdsm     kvm           309 Dec 23  2015 /var/run/vdsm/lvm/lvm.conf
 51455    0 -rw-r--r--   1 vdsm     kvm             0 Dec 23  2015 /var/run/vdsm/client.log
 43538    0 -rw-r--r--   1 root     root            0 Dec 23  2015 /var/run/vdsm/nets_restored
 43514    0 srwxr-xr-x   1 vdsm     kvm             0 Dec 23  2015 /var/run/vdsm/svdsm.sock
 22284    0 drwxr-xr-x   2 vdsm     kvm            40 Dec 23  2015 /var/run/vdsm/trackedInterfaces
 22239    0 drwxr-xr-x   2 vdsm     kvm            40 Dec 23  2015 /var/run/vdsm/sourceRoutes

This shows that the /var/run/vdsm/netconf path was created during the add host flow. And I suppose that saving the network config (which triggers the lazy dir creation path) was part of this.
Comment 8 Fabian Deutsch 2015-12-23 05:20:52 EST
Michael, can you try to reproduce this on RHEL 7.2 with vdsm from 3.5.7?
Comment 9 Michael Burman 2015-12-23 06:17:40 EST
Working as expected with rhel 7.2(RHEL - 7.2 - 9.el7_2.1)
and vdsm 3.5.7 (vdsm-4.16.31-1.el7ev)
kernel version 3.10.0 - 327.4.4.el7.x86_64
Comment 10 Dan Kenigsberg 2015-12-23 07:19:34 EST
I suppose that adding an explicit

        os.chown(dirName, vdsm_uid, 0)

to lib/vdsm/netconfpersistence.py's _setConfig would hide this surprise change in umask.
Comment 16 Barak Korren 2015-12-29 10:00:38 EST
RHEV build 3.6.2-3 released
http://bob.eng.lab.tlv.redhat.com/builds/3.6/3.6.2-3/
Comment 17 Michael Burman 2015-12-30 01:27:32 EST
Verified on - vdsm-4.17.15-0.el7ev.noarch
3.6.2-0.1.el6
ovirt-node-3.6.1-1.0.el7ev.noarch
Red Hat Enterprise Virtualization Hypervisor (Beta) release 7.2 (20151229.0.el7ev)

Note You need to log in before you can comment on or make changes to this bug.