Bug 1909782 - /etc/openvswitch permissions broken after upgrade
Summary: /etc/openvswitch permissions broken after upgrade
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: imgbased
Classification: oVirt
Component: General
Version: 1.2.14
Hardware: All
OS: Unspecified
high
high
Target Milestone: ovirt-4.4.4-2
: 1.2.15
Assignee: Asaf Rachmani
QA Contact: peyu
URL:
Whiteboard:
Depends On:
Blocks: 1916659
TreeView+ depends on / blocked
 
Reported: 2020-12-21 15:58 UTC by Jean-Louis Dupond
Modified: 2021-03-24 07:37 UTC (History)
13 users (show)

Fixed In Version: imgbased-1.2.15
Clone Of:
Environment:
Last Closed: 2021-03-24 07:37:22 UTC
oVirt Team: Node
Embargoed:
pm-rhel: ovirt-4.4+
aoconnor: blocker-
peyu: testing_plan_complete+
pm-rhel: planning_ack+
sbonazzo: devel_ack+
peyu: testing_ack+

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
oVirt gerrit 112935 0 master MERGED opsupdater: Do not change UID/GID in remediate_etc 2021-02-02 02:08:37 UTC

Description Jean-Louis Dupond 2020-12-21 15:58:38 UTC 
Description of problem:
When upgrading oVirt Node from 4.4.3 to 4.4.4 we've hit an issue where openvswitch did not want to start (and causes vdsm to fail to start).

After some quick debugging I found out it was caused by wrong permissions on /etc/openvswitch:

drwxr-xr-x.   2 clevis clevis       26 Dec 21 11:50 openvswitch 

How reproducible:
Upgrade oVirt Node 4.4.3 to 4.4.4
Check permissions of /etc/openvswitch

Additional info:
Before Update:

clevis:x:989:985:Clevis Decryption Framework unprivileged user:/var/cache/clevis:/sbin/nologin
openvswitch:x:986:982:Open vSwitch Daemons:/:/sbin/nologin

After Update:

clevis:x:989:985:Clevis Decryption Framework unprivileged user:/var/cache/clevis:/sbin/nologin
openvswitch:x:986:982:Open vSwitch Daemons:/:/sbin/nologin

So ID did not change!
But imgbased.log shows:

2020-12-21 16:42:20,870 [DEBUG] (migrate_etc) openvswitch changed from 986 to 987
2020-12-21 16:42:20,870 [DEBUG] (migrate_etc) clevis changed from 985 to 982
2020-12-21 16:42:20,870 [DEBUG] (migrate_etc) openvswitch changed from 982 to 983
2020-12-21 16:42:20,871 [DEBUG] (migrate_etc) Chowning '/tmp/mnt.jJOII//etc/openvswitch' to (989, 985)
Comment 1 Sandro Bonazzola 2020-12-21 16:03:07 UTC 
Moving to imgbased for now but it may be related to openvswitch repository change from dholler's copr repo to CentOS NFV SIG.
Dominik please have a look too.
Comment 2 RHEL Program Management 2020-12-21 16:03:14 UTC 
The documentation text flag should only be set after 'doc text' field is provided. Please provide the documentation text and set the flag to '?' again.
Comment 3 Jean-Louis Dupond 2020-12-21 16:10:39 UTC 
The file in /etc/openvswitch is correctly changed btw:

2020-12-21 16:42:20,872 [DEBUG] (migrate_etc) Chowning '/tmp/mnt.jJOII//etc/openvswitch/default.conf' to (986, 982)
Comment 4 peyu 2020-12-22 07:01:54 UTC 
QE reproduced this issue.


Test Steps:
1. Install ovirt-node-ng-installer-4.4.3-2020112920.el8.iso
2. Check the permissions of /etc/openvswitch
~~~~~~
# ll /etc
drwxr-xr-x.  2 openvswitch openvswitch     34 Nov 29 19:32 openvswitch

# vi /etc/passwd
openvswitch:x:986:982:Open vSwitch Daemons:/:/sbin/nologin
~~~~~~

3. Set up local repo and point to "ovirt-node-ng-image-update-4.4.4-1.el8.noarch.rpm"

4. Upgrade the node
   # yum update

5. After upgrade, check the permissions of /etc/openvswitch
~~~~~~
# ll /etc
drwxr-xr-x.  2 clevis clevis       34 Dec 21 10:50 openvswitch

# vi /etc/passwd
clevis:x:989:985:Clevis Decryption Framework unprivileged user:/var/cache/clevis:/sbin/nologin
openvswitch:x:986:982:Open vSwitch Daemons:/:/sbin/nologin
~~~~~~

Test result:
As you can see, the permissions of /etc/openvswitch changed from "openvswitch" to "clevis" after upgrade.


Additional info:
~~~~~~
# vi /var/log/imgbased.log

2020-12-22 06:23:15,538 [DEBUG] (migrate_etc) openvswitch changed from 986 to 987
2020-12-22 06:23:15,538 [DEBUG] (migrate_etc) sssd changed from 995 to 994
2020-12-22 06:23:15,538 [DEBUG] (migrate_etc) ssh_keys changed from 994 to 995
2020-12-22 06:23:15,538 [DEBUG] (migrate_etc) clevis changed from 985 to 982
2020-12-22 06:23:15,538 [DEBUG] (migrate_etc) libvirt changed from 984 to 985
2020-12-22 06:23:15,538 [DEBUG] (migrate_etc) ovirt-vmconsole changed from 983 to 984
2020-12-22 06:23:15,538 [DEBUG] (migrate_etc) openvswitch changed from 982 to 983
2020-12-22 06:23:15,538 [INFO] (migrate_etc) UID/GID drift was detected
2020-12-22 06:23:15,539 [DEBUG] (migrate_etc) clevis changed from 989 to 986
2020-12-22 06:23:15,539 [DEBUG] (migrate_etc) saslauth changed from 988 to 989
2020-12-22 06:23:15,539 [DEBUG] (migrate_etc) ovirt-vmconsole changed from 987 to 988
~~~~~~
Comment 5 Dominik Holler 2021-01-04 08:00:25 UTC 
Does this issue reproduce on RHV-H, too?
Comment 7 peyu 2021-01-04 08:40:21 UTC 
(In reply to Dominik Holler from comment #5)
> Does this issue reproduce on RHV-H, too?

No, this issue did not reproduce on RHVH.
Comment 8 Dominik Holler 2021-01-04 11:20:34 UTC 
https://lists.ovirt.org/archives/list/users@ovirt.org/thread/G6SXUCAMUGRZDQX5WR5GO45M3YVQR6MJ/ :

> Interestingly it only happened to a portion of the ~20 nodes I have
> upgraded -- I believe I had different behaviour depending on the specific
> version used for installation and upgrade path taken, but not 100% sure.
Comment 9 Asaf Rachmani 2021-01-13 09:30:39 UTC 
Same issue with /etc/sssd:

In 4.4.3 (ovirt-release-host-node-4.4.3-2.el8.noarch):
# ll /etc/ | grep sssd
drwx------.  4 sssd        sssd            31 Nov 29 18:52 sssd


After upgrade (ovirt-release-host-node-4.4.4-1.el8.noarch):
#ll /etc/ | grep sssd
drwx------.  4 sssd   ssh_keys     31 Dec 21 09:52 sssd


/var/log/imgbased.log:
2021-01-11 13:32:27,030 [DEBUG] (migrate_etc) clevis changed from 989 to 986
2021-01-11 13:32:27,030 [DEBUG] (migrate_etc) saslauth changed from 988 to 989
2021-01-11 13:32:27,030 [DEBUG] (migrate_etc) ovirt-vmconsole changed from 987 to 988
2021-01-11 13:32:27,030 [DEBUG] (migrate_etc) openvswitch changed from 986 to 987
2021-01-11 13:32:27,030 [DEBUG] (migrate_etc) sssd changed from 995 to 994
2021-01-11 13:32:27,030 [DEBUG] (migrate_etc) ssh_keys changed from 994 to 995
2021-01-11 13:32:27,030 [DEBUG] (migrate_etc) clevis changed from 985 to 982
2021-01-11 13:32:27,030 [DEBUG] (migrate_etc) libvirt changed from 984 to 985
2021-01-11 13:32:27,030 [DEBUG] (migrate_etc) ovirt-vmconsole changed from 983 to 984
2021-01-11 13:32:27,030 [DEBUG] (migrate_etc) openvswitch changed from 982 to 983
2021-01-11 13:32:27,031 [DEBUG] (migrate_etc) Chowning '/tmp/mnt.xac6g//etc/openvswitch' to (989, 985)
2021-01-11 13:32:27,031 [DEBUG] (migrate_etc) Chowning '/tmp/mnt.xac6g//etc/sssd' to (-1, 994)
2021-01-11 13:32:27,033 [DEBUG] (migrate_etc) Chowning '/tmp/mnt.xac6g//etc/openvswitch/default.conf' to (986, 982)
2021-01-11 13:32:27,033 [DEBUG] (migrate_etc) Chowning '/tmp/mnt.xac6g//etc/sssd/conf.d' to (-1, 994)
2021-01-11 13:32:27,035 [DEBUG] (migrate_etc) Chowning '/tmp/mnt.xac6g//var/lib/sss/keytabs' to (-1, 995)
2021-01-11 13:32:27,035 [DEBUG] (migrate_etc) Chowning '/tmp/mnt.xac6g//var/lib/sss/mc' to (-1, 995)
2021-01-11 13:32:27,035 [DEBUG] (migrate_etc) Chowning '/tmp/mnt.xac6g//var/lib/sss/gpo_cache' to (-1, 995)
2021-01-11 13:32:27,035 [DEBUG] (migrate_etc) Chowning '/tmp/mnt.xac6g//var/lib/sss/db' to (-1, 995)
2021-01-11 13:32:27,036 [DEBUG] (migrate_etc) Chowning '/tmp/mnt.xac6g//var/lib/sss/pipes' to (-1, 995)
2021-01-11 13:32:27,036 [DEBUG] (migrate_etc) Chowning '/tmp/mnt.xac6g//var/lib/sss/pubconf' to (-1, 995)
2021-01-11 13:32:27,036 [DEBUG] (migrate_etc) Chowning '/tmp/mnt.xac6g//var/lib/sss/pubconf/krb5.include.d' to (-1, 995)
2021-01-11 13:32:27,036 [DEBUG] (migrate_etc) Chowning '/tmp/mnt.xac6g//var/log/openvswitch' to (986, 982)
2021-01-11 13:32:27,036 [DEBUG] (migrate_etc) Chowning '/tmp/mnt.xac6g//var/log/sssd' to (-1, 995)
2021-01-11 13:32:27,036 [DEBUG] (migrate_etc) Chowning '/tmp/mnt.xac6g//usr/libexec/openssh/ssh-keysign' to (-1, 994)
2021-01-11 13:32:27,037 [DEBUG] (migrate_etc) Chowning '/tmp/mnt.xac6g//usr/libexec/sssd/ldap_child' to (-1, 995)
2021-01-11 13:32:27,037 [DEBUG] (migrate_etc) Chowning '/tmp/mnt.xac6g//usr/libexec/sssd/krb5_child' to (-1, 995)
2021-01-11 13:32:27,037 [DEBUG] (migrate_etc) Chowning '/tmp/mnt.xac6g//usr/libexec/sssd/selinux_child' to (-1, 995)
2021-01-11 13:32:27,047 [DEBUG] (migrate_etc) Chowning '/tmp/mnt.xac6g//usr/share/factory/etc/openvswitch' to (986, 982)
2021-01-11 13:32:27,048 [DEBUG] (migrate_etc) Chowning '/tmp/mnt.xac6g//usr/share/factory/etc/sssd' to (-1, 995)
2021-01-11 13:32:27,049 [DEBUG] (migrate_etc) Chowning '/tmp/mnt.xac6g//usr/share/factory/etc/openvswitch/default.conf' to (986, 982)
2021-01-11 13:32:27,049 [DEBUG] (migrate_etc) Chowning '/tmp/mnt.xac6g//usr/share/factory/etc/sssd/conf.d' to (-1, 995)
2021-01-11 13:32:27,052 [DEBUG] (migrate_etc) Chowning '/tmp/mnt.xac6g//usr/share/factory/var/lib/sss/keytabs' to (-1, 995)
2021-01-11 13:32:27,052 [DEBUG] (migrate_etc) Chowning '/tmp/mnt.xac6g//usr/share/factory/var/lib/sss/mc' to (-1, 995)
2021-01-11 13:32:27,052 [DEBUG] (migrate_etc) Chowning '/tmp/mnt.xac6g//usr/share/factory/var/lib/sss/gpo_cache' to (-1, 995)
2021-01-11 13:32:27,052 [DEBUG] (migrate_etc) Chowning '/tmp/mnt.xac6g//usr/share/factory/var/lib/sss/db' to (-1, 995)
2021-01-11 13:32:27,052 [DEBUG] (migrate_etc) Chowning '/tmp/mnt.xac6g//usr/share/factory/var/lib/sss/pipes' to (-1, 995)
2021-01-11 13:32:27,052 [DEBUG] (migrate_etc) Chowning '/tmp/mnt.xac6g//usr/share/factory/var/lib/sss/pubconf' to (-1, 995)
2021-01-11 13:32:27,052 [DEBUG] (migrate_etc) Chowning '/tmp/mnt.xac6g//usr/share/factory/var/lib/sss/pubconf/krb5.include.d' to (-1, 995)
2021-01-11 13:32:27,052 [DEBUG] (migrate_etc) Chowning '/tmp/mnt.xac6g//usr/share/factory/var/log/openvswitch' to (986, 982)
2021-01-11 13:32:27,052 [DEBUG] (migrate_etc) Chowning '/tmp/mnt.xac6g//usr/share/factory/var/log/sssd' to (-1, 995)
2021-01-11 13:32:27,064 [DEBUG] (migrate_etc) Chowning '/tmp/mnt.xac6g//usr/lib/.build-id/0b/bdc4d92ff3c605b56714b6510fccde281765e5' to (-1, 994)
2021-01-11 13:32:27,066 [DEBUG] (migrate_etc) Chowning '/tmp/mnt.xac6g//usr/lib/.build-id/f6/11d8c336cdc570562971af7db69b4c3bddc398' to (-1, 995)
2021-01-11 13:32:27,066 [DEBUG] (migrate_etc) Chowning '/tmp/mnt.xac6g//usr/lib/.build-id/84/e83f795d6846cc40220d182d8d45503d8477d9' to (-1, 995)
2021-01-11 13:32:27,067 [DEBUG] (migrate_etc) Chowning '/tmp/mnt.xac6g//usr/lib/.build-id/6c/f634563c1f41ba79abe83f3c26e7032184e100' to (-1, 995)
2021-01-11 13:32:27,123 [DEBUG] (migrate_etc) Changed files: ['/tmp/mnt.xac6g//etc/openvswitch', '/tmp/mnt.xac6g//etc/sssd', '/tmp/mnt.xac6g//etc/openvswitch/default.conf', '/tmp/mnt.xac6g//etc/sssd/conf.d', '/tmp/mnt.xac6g//var/lib/sss/keytabs', '/tmp/mnt.xac6g//var/lib/sss/mc', '/tmp/mnt.xac6g//var/lib/sss/gpo_cache', '/tmp/mnt.xac6g//var/lib/sss/db', '/tmp/mnt.xac6g//var/lib/sss/pipes', '/tmp/mnt.xac6g//var/lib/sss/pubconf', '/tmp/mnt.xac6g//var/lib/sss/pubconf/krb5.include.d', '/tmp/mnt.xac6g//var/log/openvswitch', '/tmp/mnt.xac6g//var/log/sssd', '/tmp/mnt.xac6g//usr/libexec/openssh/ssh-keysign', '/tmp/mnt.xac6g//usr/libexec/sssd/ldap_child', '/tmp/mnt.xac6g//usr/libexec/sssd/krb5_child', '/tmp/mnt.xac6g//usr/libexec/sssd/selinux_child', '/tmp/mnt.xac6g//usr/share/factory/etc/openvswitch', '/tmp/mnt.xac6g//usr/share/factory/etc/sssd', '/tmp/mnt.xac6g//usr/share/factory/etc/openvswitch/default.conf', '/tmp/mnt.xac6g//usr/share/factory/etc/sssd/conf.d', '/tmp/mnt.xac6g//usr/share/factory/var/lib/sss/keytabs', '/tmp/mnt.xac6g//usr/share/factory/var/lib/sss/mc', '/tmp/mnt.xac6g//usr/share/factory/var/lib/sss/gpo_cache', '/tmp/mnt.xac6g//usr/share/factory/var/lib/sss/db', '/tmp/mnt.xac6g//usr/share/factory/var/lib/sss/pipes', '/tmp/mnt.xac6g//usr/share/factory/var/lib/sss/pubconf', '/tmp/mnt.xac6g//usr/share/factory/var/lib/sss/pubconf/krb5.include.d', '/tmp/mnt.xac6g//usr/share/factory/var/log/openvswitch', '/tmp/mnt.xac6g//usr/share/factory/var/log/sssd', '/tmp/mnt.xac6g//usr/lib/.build-id/0b/bdc4d92ff3c605b56714b6510fccde281765e5', '/tmp/mnt.xac6g//usr/lib/.build-id/f6/11d8c336cdc570562971af7db69b4c3bddc398', '/tmp/mnt.xac6g//usr/lib/.build-id/84/e83f795d6846cc40220d182d8d45503d8477d9', '/tmp/mnt.xac6g//usr/lib/.build-id/6c/f634563c1f41ba79abe83f3c26e7032184e100']
Comment 10 RHEL Program Management 2021-01-13 16:19:36 UTC 
This bug report has Keywords: Regression or TestBlocker.
Since no regressions or test blockers are allowed between releases, it is also being identified as a blocker for this release. Please resolve ASAP.
Comment 11 peyu 2021-01-18 02:28:40 UTC 
Pending New Build
Comment 12 peyu 2021-03-09 09:02:29 UTC 
QE verified this issue on "ovirt-node-ng-image-update-4.4.4.1-1.el8.noarch.rpm".

Test Steps:
1. Install ovirt-node-ng-installer-4.4.3-2020112920.el8.iso
2. Check the permissions of openvswitch and sssd
~~~~~~
# ll /etc | grep openvswitch
drwxr-xr-x.  2 openvswitch openvswitch     34 Nov 29 19:32 openvswitch

# ll /etc/ | grep sssd
drwx------.  4 sssd        sssd            43 Nov 29 18:52 sssd
~~~~~~

3. Set up local repo and point to "ovirt-node-ng-image-update-4.4.4-1.el8.noarch.rpm"

4. Upgrade the node
   # yum update

5. After upgrade, check the permissions of openvswitch and sssd
~~~~~~
# ll /etc/ | grep openvswitch
drwxr-xr-x.  2 clevis clevis       34 Dec 21 10:50 openvswitch

# ll /etc/ | grep sssd
drwx------.  4 sssd   ssh_keys     43 Dec 21 09:52 sssd
~~~~~~


6. Set up local repo and point to "ovirt-node-ng-image-update-4.4.4.1-1.el8.noarch.rpm"

7. Upgrade the node again
   # yum update

8. After upgrade, check the permissions of openvswitch and sssd
~~~~~~
# imgbase w
You are on ovirt-node-ng-4.4.4.1-0.20210208.0+1

# imgbase layout
ovirt-node-ng-4.4.4-0.20201221.0
 +- ovirt-node-ng-4.4.4-0.20201221.0+1
ovirt-node-ng-4.4.4.1-0.20210208.0
 +- ovirt-node-ng-4.4.4.1-0.20210208.0+1

# ll /etc/ | grep openvswitch
drwxr-xr-x.  2 openvswitch openvswitch     26 Feb  8 10:00 openvswitch

# ll /etc/ | grep sssd
drwx------.  4 sssd        sssd            31 Feb  8 09:01 sssd
~~~~~~

Test result:
As you can see, the permissions are the expected.
Note You need to log in before you can comment on or make changes to this bug.