Bug 1909782

Summary:	/etc/openvswitch permissions broken after upgrade
Product:	[oVirt] imgbased	Reporter:	Jean-Louis Dupond <jean-louis>
Component:	General	Assignee:	Asaf Rachmani <arachman>
Status:	CLOSED CURRENTRELEASE	QA Contact:	peyu
Severity:	high	Docs Contact:
Priority:	high
Version:	1.2.14	CC:	aoconnor, arachman, bugs, cshao, dholler, lsvaty, mavital, mburman, peyu, sbonazzo, shlei, weiwang, yaniwang
Target Milestone:	ovirt-4.4.4-2	Keywords:	Regression
Target Release:	1.2.15	Flags:	pm-rhel: ovirt-4.4+ aoconnor: blocker- peyu: testing_plan_complete+ pm-rhel: planning_ack+ sbonazzo: devel_ack+ peyu: testing_ack+
Hardware:	All
OS:	Unspecified
Whiteboard:
Fixed In Version:	imgbased-1.2.15	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2021-03-24 07:37:22 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	Node	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1916659

Description Jean-Louis Dupond 2020-12-21 15:58:38 UTC

Description of problem:
When upgrading oVirt Node from 4.4.3 to 4.4.4 we've hit an issue where openvswitch did not want to start (and causes vdsm to fail to start).

After some quick debugging I found out it was caused by wrong permissions on /etc/openvswitch:

drwxr-xr-x.   2 clevis clevis       26 Dec 21 11:50 openvswitch 

How reproducible:
Upgrade oVirt Node 4.4.3 to 4.4.4
Check permissions of /etc/openvswitch

Additional info:
Before Update:

clevis:x:989:985:Clevis Decryption Framework unprivileged user:/var/cache/clevis:/sbin/nologin
openvswitch:x:986:982:Open vSwitch Daemons:/:/sbin/nologin

After Update:

clevis:x:989:985:Clevis Decryption Framework unprivileged user:/var/cache/clevis:/sbin/nologin
openvswitch:x:986:982:Open vSwitch Daemons:/:/sbin/nologin

So ID did not change!
But imgbased.log shows:

2020-12-21 16:42:20,870 [DEBUG] (migrate_etc) openvswitch changed from 986 to 987
2020-12-21 16:42:20,870 [DEBUG] (migrate_etc) clevis changed from 985 to 982
2020-12-21 16:42:20,870 [DEBUG] (migrate_etc) openvswitch changed from 982 to 983
2020-12-21 16:42:20,871 [DEBUG] (migrate_etc) Chowning '/tmp/mnt.jJOII//etc/openvswitch' to (989, 985)

Comment 1 Sandro Bonazzola 2020-12-21 16:03:07 UTC

Moving to imgbased for now but it may be related to openvswitch repository change from dholler's copr repo to CentOS NFV SIG.
Dominik please have a look too.

Comment 2 RHEL Program Management 2020-12-21 16:03:14 UTC

The documentation text flag should only be set after 'doc text' field is provided. Please provide the documentation text and set the flag to '?' again.

Comment 3 Jean-Louis Dupond 2020-12-21 16:10:39 UTC

The file in /etc/openvswitch is correctly changed btw:

2020-12-21 16:42:20,872 [DEBUG] (migrate_etc) Chowning '/tmp/mnt.jJOII//etc/openvswitch/default.conf' to (986, 982)

Comment 4 peyu 2020-12-22 07:01:54 UTC

QE reproduced this issue.


Test Steps:
1. Install ovirt-node-ng-installer-4.4.3-2020112920.el8.iso
2. Check the permissions of /etc/openvswitch
~~~~~~
# ll /etc
drwxr-xr-x.  2 openvswitch openvswitch     34 Nov 29 19:32 openvswitch

# vi /etc/passwd
openvswitch:x:986:982:Open vSwitch Daemons:/:/sbin/nologin
~~~~~~

3. Set up local repo and point to "ovirt-node-ng-image-update-4.4.4-1.el8.noarch.rpm"

4. Upgrade the node
   # yum update

5. After upgrade, check the permissions of /etc/openvswitch
~~~~~~
# ll /etc
drwxr-xr-x.  2 clevis clevis       34 Dec 21 10:50 openvswitch

# vi /etc/passwd
clevis:x:989:985:Clevis Decryption Framework unprivileged user:/var/cache/clevis:/sbin/nologin
openvswitch:x:986:982:Open vSwitch Daemons:/:/sbin/nologin
~~~~~~

Test result:
As you can see, the permissions of /etc/openvswitch changed from "openvswitch" to "clevis" after upgrade.


Additional info:
~~~~~~
# vi /var/log/imgbased.log

2020-12-22 06:23:15,538 [DEBUG] (migrate_etc) openvswitch changed from 986 to 987
2020-12-22 06:23:15,538 [DEBUG] (migrate_etc) sssd changed from 995 to 994
2020-12-22 06:23:15,538 [DEBUG] (migrate_etc) ssh_keys changed from 994 to 995
2020-12-22 06:23:15,538 [DEBUG] (migrate_etc) clevis changed from 985 to 982
2020-12-22 06:23:15,538 [DEBUG] (migrate_etc) libvirt changed from 984 to 985
2020-12-22 06:23:15,538 [DEBUG] (migrate_etc) ovirt-vmconsole changed from 983 to 984
2020-12-22 06:23:15,538 [DEBUG] (migrate_etc) openvswitch changed from 982 to 983
2020-12-22 06:23:15,538 [INFO] (migrate_etc) UID/GID drift was detected
2020-12-22 06:23:15,539 [DEBUG] (migrate_etc) clevis changed from 989 to 986
2020-12-22 06:23:15,539 [DEBUG] (migrate_etc) saslauth changed from 988 to 989
2020-12-22 06:23:15,539 [DEBUG] (migrate_etc) ovirt-vmconsole changed from 987 to 988
~~~~~~

Comment 5 Dominik Holler 2021-01-04 08:00:25 UTC

Does this issue reproduce on RHV-H, too?

Comment 7 peyu 2021-01-04 08:40:21 UTC

(In reply to Dominik Holler from comment #5)
> Does this issue reproduce on RHV-H, too?

No, this issue did not reproduce on RHVH.

Comment 8 Dominik Holler 2021-01-04 11:20:34 UTC

https://lists.ovirt.org/archives/list/users@ovirt.org/thread/G6SXUCAMUGRZDQX5WR5GO45M3YVQR6MJ/ :

> Interestingly it only happened to a portion of the ~20 nodes I have
> upgraded -- I believe I had different behaviour depending on the specific
> version used for installation and upgrade path taken, but not 100% sure.

Comment 9 Asaf Rachmani 2021-01-13 09:30:39 UTC

Same issue with /etc/sssd:

In 4.4.3 (ovirt-release-host-node-4.4.3-2.el8.noarch):
# ll /etc/ | grep sssd
drwx------.  4 sssd        sssd            31 Nov 29 18:52 sssd


After upgrade (ovirt-release-host-node-4.4.4-1.el8.noarch):
#ll /etc/ | grep sssd
drwx------.  4 sssd   ssh_keys     31 Dec 21 09:52 sssd


/var/log/imgbased.log:
2021-01-11 13:32:27,030 [DEBUG] (migrate_etc) clevis changed from 989 to 986
2021-01-11 13:32:27,030 [DEBUG] (migrate_etc) saslauth changed from 988 to 989
2021-01-11 13:32:27,030 [DEBUG] (migrate_etc) ovirt-vmconsole changed from 987 to 988
2021-01-11 13:32:27,030 [DEBUG] (migrate_etc) openvswitch changed from 986 to 987
2021-01-11 13:32:27,030 [DEBUG] (migrate_etc) sssd changed from 995 to 994
2021-01-11 13:32:27,030 [DEBUG] (migrate_etc) ssh_keys changed from 994 to 995
2021-01-11 13:32:27,030 [DEBUG] (migrate_etc) clevis changed from 985 to 982
2021-01-11 13:32:27,030 [DEBUG] (migrate_etc) libvirt changed from 984 to 985
2021-01-11 13:32:27,030 [DEBUG] (migrate_etc) ovirt-vmconsole changed from 983 to 984
2021-01-11 13:32:27,030 [DEBUG] (migrate_etc) openvswitch changed from 982 to 983
2021-01-11 13:32:27,031 [DEBUG] (migrate_etc) Chowning '/tmp/mnt.xac6g//etc/openvswitch' to (989, 985)
2021-01-11 13:32:27,031 [DEBUG] (migrate_etc) Chowning '/tmp/mnt.xac6g//etc/sssd' to (-1, 994)
2021-01-11 13:32:27,033 [DEBUG] (migrate_etc) Chowning '/tmp/mnt.xac6g//etc/openvswitch/default.conf' to (986, 982)
2021-01-11 13:32:27,033 [DEBUG] (migrate_etc) Chowning '/tmp/mnt.xac6g//etc/sssd/conf.d' to (-1, 994)
2021-01-11 13:32:27,035 [DEBUG] (migrate_etc) Chowning '/tmp/mnt.xac6g//var/lib/sss/keytabs' to (-1, 995)
2021-01-11 13:32:27,035 [DEBUG] (migrate_etc) Chowning '/tmp/mnt.xac6g//var/lib/sss/mc' to (-1, 995)
2021-01-11 13:32:27,035 [DEBUG] (migrate_etc) Chowning '/tmp/mnt.xac6g//var/lib/sss/gpo_cache' to (-1, 995)
2021-01-11 13:32:27,035 [DEBUG] (migrate_etc) Chowning '/tmp/mnt.xac6g//var/lib/sss/db' to (-1, 995)
2021-01-11 13:32:27,036 [DEBUG] (migrate_etc) Chowning '/tmp/mnt.xac6g//var/lib/sss/pipes' to (-1, 995)
2021-01-11 13:32:27,036 [DEBUG] (migrate_etc) Chowning '/tmp/mnt.xac6g//var/lib/sss/pubconf' to (-1, 995)
2021-01-11 13:32:27,036 [DEBUG] (migrate_etc) Chowning '/tmp/mnt.xac6g//var/lib/sss/pubconf/krb5.include.d' to (-1, 995)
2021-01-11 13:32:27,036 [DEBUG] (migrate_etc) Chowning '/tmp/mnt.xac6g//var/log/openvswitch' to (986, 982)
2021-01-11 13:32:27,036 [DEBUG] (migrate_etc) Chowning '/tmp/mnt.xac6g//var/log/sssd' to (-1, 995)
2021-01-11 13:32:27,036 [DEBUG] (migrate_etc) Chowning '/tmp/mnt.xac6g//usr/libexec/openssh/ssh-keysign' to (-1, 994)
2021-01-11 13:32:27,037 [DEBUG] (migrate_etc) Chowning '/tmp/mnt.xac6g//usr/libexec/sssd/ldap_child' to (-1, 995)
2021-01-11 13:32:27,037 [DEBUG] (migrate_etc) Chowning '/tmp/mnt.xac6g//usr/libexec/sssd/krb5_child' to (-1, 995)
2021-01-11 13:32:27,037 [DEBUG] (migrate_etc) Chowning '/tmp/mnt.xac6g//usr/libexec/sssd/selinux_child' to (-1, 995)
2021-01-11 13:32:27,047 [DEBUG] (migrate_etc) Chowning '/tmp/mnt.xac6g//usr/share/factory/etc/openvswitch' to (986, 982)
2021-01-11 13:32:27,048 [DEBUG] (migrate_etc) Chowning '/tmp/mnt.xac6g//usr/share/factory/etc/sssd' to (-1, 995)
2021-01-11 13:32:27,049 [DEBUG] (migrate_etc) Chowning '/tmp/mnt.xac6g//usr/share/factory/etc/openvswitch/default.conf' to (986, 982)
2021-01-11 13:32:27,049 [DEBUG] (migrate_etc) Chowning '/tmp/mnt.xac6g//usr/share/factory/etc/sssd/conf.d' to (-1, 995)
2021-01-11 13:32:27,052 [DEBUG] (migrate_etc) Chowning '/tmp/mnt.xac6g//usr/share/factory/var/lib/sss/keytabs' to (-1, 995)
2021-01-11 13:32:27,052 [DEBUG] (migrate_etc) Chowning '/tmp/mnt.xac6g//usr/share/factory/var/lib/sss/mc' to (-1, 995)
2021-01-11 13:32:27,052 [DEBUG] (migrate_etc) Chowning '/tmp/mnt.xac6g//usr/share/factory/var/lib/sss/gpo_cache' to (-1, 995)
2021-01-11 13:32:27,052 [DEBUG] (migrate_etc) Chowning '/tmp/mnt.xac6g//usr/share/factory/var/lib/sss/db' to (-1, 995)
2021-01-11 13:32:27,052 [DEBUG] (migrate_etc) Chowning '/tmp/mnt.xac6g//usr/share/factory/var/lib/sss/pipes' to (-1, 995)
2021-01-11 13:32:27,052 [DEBUG] (migrate_etc) Chowning '/tmp/mnt.xac6g//usr/share/factory/var/lib/sss/pubconf' to (-1, 995)
2021-01-11 13:32:27,052 [DEBUG] (migrate_etc) Chowning '/tmp/mnt.xac6g//usr/share/factory/var/lib/sss/pubconf/krb5.include.d' to (-1, 995)
2021-01-11 13:32:27,052 [DEBUG] (migrate_etc) Chowning '/tmp/mnt.xac6g//usr/share/factory/var/log/openvswitch' to (986, 982)
2021-01-11 13:32:27,052 [DEBUG] (migrate_etc) Chowning '/tmp/mnt.xac6g//usr/share/factory/var/log/sssd' to (-1, 995)
2021-01-11 13:32:27,064 [DEBUG] (migrate_etc) Chowning '/tmp/mnt.xac6g//usr/lib/.build-id/0b/bdc4d92ff3c605b56714b6510fccde281765e5' to (-1, 994)
2021-01-11 13:32:27,066 [DEBUG] (migrate_etc) Chowning '/tmp/mnt.xac6g//usr/lib/.build-id/f6/11d8c336cdc570562971af7db69b4c3bddc398' to (-1, 995)
2021-01-11 13:32:27,066 [DEBUG] (migrate_etc) Chowning '/tmp/mnt.xac6g//usr/lib/.build-id/84/e83f795d6846cc40220d182d8d45503d8477d9' to (-1, 995)
2021-01-11 13:32:27,067 [DEBUG] (migrate_etc) Chowning '/tmp/mnt.xac6g//usr/lib/.build-id/6c/f634563c1f41ba79abe83f3c26e7032184e100' to (-1, 995)
2021-01-11 13:32:27,123 [DEBUG] (migrate_etc) Changed files: ['/tmp/mnt.xac6g//etc/openvswitch', '/tmp/mnt.xac6g//etc/sssd', '/tmp/mnt.xac6g//etc/openvswitch/default.conf', '/tmp/mnt.xac6g//etc/sssd/conf.d', '/tmp/mnt.xac6g//var/lib/sss/keytabs', '/tmp/mnt.xac6g//var/lib/sss/mc', '/tmp/mnt.xac6g//var/lib/sss/gpo_cache', '/tmp/mnt.xac6g//var/lib/sss/db', '/tmp/mnt.xac6g//var/lib/sss/pipes', '/tmp/mnt.xac6g//var/lib/sss/pubconf', '/tmp/mnt.xac6g//var/lib/sss/pubconf/krb5.include.d', '/tmp/mnt.xac6g//var/log/openvswitch', '/tmp/mnt.xac6g//var/log/sssd', '/tmp/mnt.xac6g//usr/libexec/openssh/ssh-keysign', '/tmp/mnt.xac6g//usr/libexec/sssd/ldap_child', '/tmp/mnt.xac6g//usr/libexec/sssd/krb5_child', '/tmp/mnt.xac6g//usr/libexec/sssd/selinux_child', '/tmp/mnt.xac6g//usr/share/factory/etc/openvswitch', '/tmp/mnt.xac6g//usr/share/factory/etc/sssd', '/tmp/mnt.xac6g//usr/share/factory/etc/openvswitch/default.conf', '/tmp/mnt.xac6g//usr/share/factory/etc/sssd/conf.d', '/tmp/mnt.xac6g//usr/share/factory/var/lib/sss/keytabs', '/tmp/mnt.xac6g//usr/share/factory/var/lib/sss/mc', '/tmp/mnt.xac6g//usr/share/factory/var/lib/sss/gpo_cache', '/tmp/mnt.xac6g//usr/share/factory/var/lib/sss/db', '/tmp/mnt.xac6g//usr/share/factory/var/lib/sss/pipes', '/tmp/mnt.xac6g//usr/share/factory/var/lib/sss/pubconf', '/tmp/mnt.xac6g//usr/share/factory/var/lib/sss/pubconf/krb5.include.d', '/tmp/mnt.xac6g//usr/share/factory/var/log/openvswitch', '/tmp/mnt.xac6g//usr/share/factory/var/log/sssd', '/tmp/mnt.xac6g//usr/lib/.build-id/0b/bdc4d92ff3c605b56714b6510fccde281765e5', '/tmp/mnt.xac6g//usr/lib/.build-id/f6/11d8c336cdc570562971af7db69b4c3bddc398', '/tmp/mnt.xac6g//usr/lib/.build-id/84/e83f795d6846cc40220d182d8d45503d8477d9', '/tmp/mnt.xac6g//usr/lib/.build-id/6c/f634563c1f41ba79abe83f3c26e7032184e100']

Comment 10 RHEL Program Management 2021-01-13 16:19:36 UTC

This bug report has Keywords: Regression or TestBlocker.
Since no regressions or test blockers are allowed between releases, it is also being identified as a blocker for this release. Please resolve ASAP.

Comment 11 peyu 2021-01-18 02:28:40 UTC

Pending New Build

Comment 12 peyu 2021-03-09 09:02:29 UTC

QE verified this issue on "ovirt-node-ng-image-update-4.4.4.1-1.el8.noarch.rpm".

Test Steps:
1. Install ovirt-node-ng-installer-4.4.3-2020112920.el8.iso
2. Check the permissions of openvswitch and sssd
~~~~~~
# ll /etc | grep openvswitch
drwxr-xr-x.  2 openvswitch openvswitch     34 Nov 29 19:32 openvswitch

# ll /etc/ | grep sssd
drwx------.  4 sssd        sssd            43 Nov 29 18:52 sssd
~~~~~~

3. Set up local repo and point to "ovirt-node-ng-image-update-4.4.4-1.el8.noarch.rpm"

4. Upgrade the node
   # yum update

5. After upgrade, check the permissions of openvswitch and sssd
~~~~~~
# ll /etc/ | grep openvswitch
drwxr-xr-x.  2 clevis clevis       34 Dec 21 10:50 openvswitch

# ll /etc/ | grep sssd
drwx------.  4 sssd   ssh_keys     43 Dec 21 09:52 sssd
~~~~~~


6. Set up local repo and point to "ovirt-node-ng-image-update-4.4.4.1-1.el8.noarch.rpm"

7. Upgrade the node again
   # yum update

8. After upgrade, check the permissions of openvswitch and sssd
~~~~~~
# imgbase w
You are on ovirt-node-ng-4.4.4.1-0.20210208.0+1

# imgbase layout
ovirt-node-ng-4.4.4-0.20201221.0
 +- ovirt-node-ng-4.4.4-0.20201221.0+1
ovirt-node-ng-4.4.4.1-0.20210208.0
 +- ovirt-node-ng-4.4.4.1-0.20210208.0+1

# ll /etc/ | grep openvswitch
drwxr-xr-x.  2 openvswitch openvswitch     26 Feb  8 10:00 openvswitch

# ll /etc/ | grep sssd
drwx------.  4 sssd        sssd            31 Feb  8 09:01 sssd
~~~~~~

Test result:
As you can see, the permissions are the expected.