Bug 1290750 - [vdsm] [RHEL6 -> RHEL7 upgrade] failed vdsmd - Modules sebool are not configured
[vdsm] [RHEL6 -> RHEL7 upgrade] failed vdsmd - Modules sebool are not configured
Status: CLOSED WONTFIX
Product: vdsm
Classification: oVirt
Component: General (Show other bugs)
4.17.11
Unspecified Unspecified
unspecified Severity high (vote)
: ovirt-3.6.2
: ---
Assigned To: Yaniv Bronhaim
Aharon Canan
infra
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-12-11 05:25 EST by Jiri Belka
Modified: 2016-02-10 14:26 EST (History)
7 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2015-12-31 06:17:16 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: Infra
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
ybronhei: ovirt‑3.6.z?
rule-engine: planning_ack?
rule-engine: devel_ack?
rule-engine: testing_ack?


Attachments (Terms of Use)

  None (edit)
Description Jiri Belka 2015-12-11 05:25:08 EST
Description of problem:

There's ongoing process to support "on-fly" migration of 3.5 host (RHEL 6.7) to 3.6 host (RHEL 7.2) while keeping vdsm installed and updated during this migration path.

But after redhat-upgrade-tool does its work and one upgraded vdsm, vdsmd is failing because sebool module is unconfigured.

According to ybronhei@ this cause is because there's selinux disabled during migration...

~~~
Dec 10 16:56:59 10-34-61-155.rhev.lab.eng.brq.redhat.com kernel: Command line: root=/dev/mapper/rootvg-lv_root ro rd.luks=0 rd.locale.LANG=en_US.UTF-8 crashkernel=auto consoleblank=0 rd.lvm.lv=rootvg/lv_swap rd.md=0 vconsole.font=latarcyrheb-sun16 rd.lvm.lv=rootvg/lv_root KEYBOARDTYPE=pc vconsole.keymap=us rd.dm=0 upgrade init=/usr/libexec/upgrade-init enforcing=0 rd.plymouth=0 plymouth.enable=0 net.ifnames=0
~~~

Anyway to satisfy customer needs for easy host migration this issue should be solved.

Currently execuring 'vdsm-tool configure --force' solves the issue.

~~~
-- Unit vdsmd.service has begun starting up.
Dec 11 10:54:38 10-34-61-155.rhev.lab.eng.brq.redhat.com vdsmd_init_common.sh[20304]: vdsm: Running mkdirs
Dec 11 10:54:38 10-34-61-155.rhev.lab.eng.brq.redhat.com vdsmd_init_common.sh[20304]: vdsm: Running configure_coredump
Dec 11 10:54:38 10-34-61-155.rhev.lab.eng.brq.redhat.com vdsmd_init_common.sh[20304]: vdsm: Running configure_vdsm_logs
Dec 11 10:54:38 10-34-61-155.rhev.lab.eng.brq.redhat.com vdsmd_init_common.sh[20304]: vdsm: Running wait_for_network
Dec 11 10:54:38 10-34-61-155.rhev.lab.eng.brq.redhat.com vdsmd_init_common.sh[20304]: vdsm: Running run_init_hooks
Dec 11 10:54:38 10-34-61-155.rhev.lab.eng.brq.redhat.com vdsmd_init_common.sh[20304]: vdsm: Running upgraded_version_check
Dec 11 10:54:38 10-34-61-155.rhev.lab.eng.brq.redhat.com vdsmd_init_common.sh[20304]: vdsm: Running check_is_configured
Dec 11 10:54:39 10-34-61-155.rhev.lab.eng.brq.redhat.com sasldblistusers2[20344]: DIGEST-MD5 common mech free
Dec 11 10:54:39 10-34-61-155.rhev.lab.eng.brq.redhat.com vdsmd_init_common.sh[20304]: Error:
Dec 11 10:54:39 10-34-61-155.rhev.lab.eng.brq.redhat.com vdsmd_init_common.sh[20304]: One of the modules is not configured to work with VDSM.
Dec 11 10:54:39 10-34-61-155.rhev.lab.eng.brq.redhat.com vdsmd_init_common.sh[20304]: To configure the module use the following:
Dec 11 10:54:39 10-34-61-155.rhev.lab.eng.brq.redhat.com vdsmd_init_common.sh[20304]: 'vdsm-tool configure [--module module-name]'.
Dec 11 10:54:39 10-34-61-155.rhev.lab.eng.brq.redhat.com vdsmd_init_common.sh[20304]: If all modules are not configured try to use:
Dec 11 10:54:39 10-34-61-155.rhev.lab.eng.brq.redhat.com vdsmd_init_common.sh[20304]: 'vdsm-tool configure --force'
Dec 11 10:54:39 10-34-61-155.rhev.lab.eng.brq.redhat.com vdsmd_init_common.sh[20304]: (The force flag will stop the module's service and start it
Dec 11 10:54:39 10-34-61-155.rhev.lab.eng.brq.redhat.com vdsmd_init_common.sh[20304]: afterwards automatically to load the new configuration.)
Dec 11 10:54:39 10-34-61-155.rhev.lab.eng.brq.redhat.com vdsmd_init_common.sh[20304]: Current revision of multipath.conf detected, preserving
Dec 11 10:54:39 10-34-61-155.rhev.lab.eng.brq.redhat.com vdsmd_init_common.sh[20304]: libvirt is already configured for vdsm
Dec 11 10:54:39 10-34-61-155.rhev.lab.eng.brq.redhat.com vdsmd_init_common.sh[20304]: Modules sebool are not configured
Dec 11 10:54:39 10-34-61-155.rhev.lab.eng.brq.redhat.com systemd[1]: [1;39mvdsmd.service: control process exited, code=exited status=1[0m
Dec 11 10:54:39 10-34-61-155.rhev.lab.eng.brq.redhat.com systemd[1]: [1;31mFailed to start Virtual Desktop Server Manager.[0m
-- Subject: Unit vdsmd.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit vdsmd.service has failed.
-- 
-- The result is failed.
Dec 11 10:54:39 10-34-61-155.rhev.lab.eng.brq.redhat.com systemd[1]: [1;39mUnit vdsmd.service entered failed state.[0m
Dec 11 10:54:39 10-34-61-155.rhev.lab.eng.brq.redhat.com systemd[1]: [1;39mvdsmd.service failed.[0m
Dec 11 10:54:39 10-34-61-155.rhev.lab.eng.brq.redhat.com systemd[1]: vdsmd.service holdoff time over, scheduling restart.
Dec 11 10:54:39 10-34-61-155.rhev.lab.eng.brq.redhat.com systemd[1]: [1;39mstart request repeated too quickly for vdsmd.service[0m
Dec 11 10:54:39 10-34-61-155.rhev.lab.eng.brq.redhat.com systemd[1]: [1;31mFailed to start Virtual Desktop Server Manager.[0m
-- Subject: Unit vdsmd.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit vdsmd.service has failed.
-- 
-- The result is failed.
Dec 11 10:54:39 10-34-61-155.rhev.lab.eng.brq.redhat.com systemd[1]: [1;39mUnit vdsmd.service entered failed state.[0m
Dec 11 10:54:39 10-34-61-155.rhev.lab.eng.brq.redhat.com systemd[1]: [1;39mvdsmd.service failed.[0m
kroot@10-34-61-155:/etc/yum.repos.d\[root@10-34-61-155 yum.repos.d]# 

~~~

# vdsm-tool -vvv is-configured
Current revision of multipath.conf detected, preserving
MainThread::DEBUG::2015-12-11 11:03:25,731::utils::669::root::(execCmd) /usr/sbin/sasldblistusers2 -f /etc/libvirt/passwd.db (cwd None)
MainThread::DEBUG::2015-12-11 11:03:25,751::utils::687::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
libvirt is already configured for vdsm
Modules sebool are not configured
Error:  

One of the modules is not configured to work with VDSM.
To configure the module use the following:
'vdsm-tool configure [--module module-name]'.

If all modules are not configured try to use:
'vdsm-tool configure --force'
(The force flag will stop the module's service and start it
afterwards automatically to load the new configuration.)
~~~

Version-Release number of selected component (if applicable):
vdsm-4.17.12-11.git263129f.el7.centos.noarch

How reproducible:
100%

Steps to Reproduce:
1. RHEL 6.7, 3.5 release vdsm (ovirt), have this host part of engine
   (I started with 3.5 engine and 3.5 hosts, then upgraded engine to 3.6)
2. migration of RHEL 6.7/3.5 vdsm host to RHEL 7.2

   - yum install redhat-upgrade-tool preupgrade-assistant-contents
   - preupg
   - redhat-upgrade-tool --network 7.2 \
     --instrepo=http://example.com/pub/rhel/released/RHEL-7/7.2/Server/x86_64/os/ --addrepo=optional=http://exmaple.com/pub/rhel/released/RHEL-7/7.2/Server-optional/x86_64/os/ --addrepo=zstream=http://example.com/rel-eng/repos/rhel-7.2-z/x86_64/ --force --nogpgcheck

3. add rhel 7.2 repos, add 3.6 release/snapshot ovirt repos
4. yum update
5. yum remove `rpm -qa | fgrep .el6 | xargs`
6. systemctl start vdsmd

Actual results:
vdsmd not running because sebool module not configured

Expected results:
vdsmd should be running (vdsmd_init_common.sh should handle sebool module)

Additional info:
Comment 2 Dan Kenigsberg 2015-12-11 08:51:00 EST
Isn't the bug about el6->el7 upgrade, and unrelated to (live) VM migration?

Prior upgrade: vdsm-4.16.30-0.el6.x86_64 
But I cannot see to which version it was upgraded to. Can you specify it, and find yum.log on the host, to see if the upgrade has succeeded?

I'm not sure whether we have tried an el6->el7 upgrade of Vdsm before.
Comment 3 Red Hat Bugzilla Rules Engine 2015-12-11 08:51:04 EST
Bug tickets must have version flags set prior to targeting them to a release. Please ask maintainer to set the correct version flags and only then set the target milestone.
Comment 4 Jiri Belka 2015-12-11 09:08:50 EST
(In reply to Dan Kenigsberg from comment #2)
> Isn't the bug about el6->el7 upgrade, and unrelated to (live) VM migration?

Yes it is el6->el7, corrected. Not related to any migration as vdsmd is not running yet.

> Prior upgrade: vdsm-4.16.30-0.el6.x86_64 
> But I cannot see to which version it was upgraded to. Can you specify it,
> and find yum.log on the host, to see if the upgrade has succeeded?

It is written there :) vdsm-4.17.12-11.git263129f.el7.centos.noarch

> I'm not sure whether we have tried an el6->el7 upgrade of Vdsm before.

We've started this POC migration/upgrade path now as it was requested by a customer.
Comment 5 Dan Kenigsberg 2015-12-11 10:57:15 EST
vdsm-4.17.12-11.git263129f.el7.centos.noarch is not mentioned in the attached logs. Can you find yum.log (which should show whether %postun failed to confifure sebool)? Can you share `rpm -qa |grep vdsm` after upgrade?
Comment 6 Jiri Belka 2015-12-11 11:31:05 EST
(In reply to Dan Kenigsberg from comment #5)
> vdsm-4.17.12-11.git263129f.el7.centos.noarch is not mentioned in the
> attached logs. Can you find yum.log (which should show whether %postun
> failed to confifure sebool)? Can you share `rpm -qa |grep vdsm` after
> upgrade?

iiuc postun won't be called as there's only "if [ "$1" -ge 1 ];" and this was upgrade thus it is '2', isn't it (1 for installed one and 1 for newer package).

I don't see any '--scripts' related info in yum.log:

~~~
# grep vdsm /var/log/yum.log 
Dec 09 13:48:30 Installed: vdsm-python-zombiereaper-4.16.30-0.el6.noarch
Dec 09 13:48:43 Installed: vdsm-python-4.16.30-0.el6.noarch
Dec 09 13:48:43 Installed: vdsm-xmlrpc-4.16.30-0.el6.noarch
Dec 09 13:48:49 Installed: vdsm-yajsonrpc-4.16.30-0.el6.noarch
Dec 09 13:48:50 Installed: vdsm-jsonrpc-4.16.30-0.el6.noarch
Dec 09 13:50:51 Installed: vdsm-4.16.30-0.el6.x86_64
Dec 09 13:50:51 Installed: vdsm-cli-4.16.30-0.el6.noarch
Dec 11 12:36:01 Installed: vdsm-infra-4.17.12-11.git263129f.el7.centos.noarch
Dec 11 12:36:02 Updated: vdsm-python-4.17.12-11.git263129f.el7.centos.noarch
Dec 11 12:36:02 Updated: vdsm-xmlrpc-4.17.12-11.git263129f.el7.centos.noarch
Dec 11 12:36:03 Updated: vdsm-yajsonrpc-4.17.12-11.git263129f.el7.centos.noarch
Dec 11 12:36:03 Updated: vdsm-jsonrpc-4.17.12-11.git263129f.el7.centos.noarch
Dec 11 12:36:53 Updated: vdsm-4.17.12-11.git263129f.el7.centos.noarch
Dec 11 12:36:54 Updated: vdsm-cli-4.17.12-11.git263129f.el7.centos.noarch
Dec 11 12:37:01 Erased: vdsm-python-zombiereaper-4.16.30-0.el6.noarch
~~~

~~~
# rpm -qa vdsm\*
vdsm-4.17.12-11.git263129f.el7.centos.noarch
vdsm-infra-4.17.12-11.git263129f.el7.centos.noarch
vdsm-yajsonrpc-4.17.12-11.git263129f.el7.centos.noarch
vdsm-python-4.17.12-11.git263129f.el7.centos.noarch
vdsm-jsonrpc-4.17.12-11.git263129f.el7.centos.noarch
vdsm-xmlrpc-4.17.12-11.git263129f.el7.centos.noarch
vdsm-cli-4.17.12-11.git263129f.el7.centos.noarch
~~~

ybronhei@ was part of discussion about this issue, please if needed consult with him.
Comment 7 Yaniv Bronhaim 2015-12-13 06:43:37 EST
our chat about that issue was not published anywhere. copying my explanation for that case - 

'''
It can happen only if just before that upgrade you enabled back selinux. When you install vdsm and selinux is disabled, vdsm can't set selinux booleans - therefore vdsm-tool configure skip that part.  next time you will restart vdsmd it will check again the selinux status and then check if the vdsm required booleans are set, if not it will fail to start until you will run vdsm-tool configure or vdsm-tool configure --module=sebool .
In our case here I think that just before the upgrade you enabled selinux back and before that it ran when selinux was disabled.
'''

In yum log you won't see scripts but the versions you use. During upgrade we do not try to configure selinux booleans. only explicit call to vdsm-tool configure will do that when selinux is enforced.

I can configure sebools after upgrade but Im not sure when in the upgrade flow selinux gets enabled. so we might end up again with the same situation if the upgrade is done, then the host is restarted with enforcing but vdsm can't start because it ran the configure while selinux was disabled.

this patch will try to reconfigure sebools automatically after upgrade - https://gerrit.ovirt.org/50382
I didn't want to do that because it changes the environment underneath without notifying to the user at all.  

I also suggest to consider that after rhel6 upgrade to rhel7 we will instruct the users to perform reinstallation for the host
Comment 8 Yaniv Bronhaim 2015-12-16 11:20:00 EST
Any comments?
Comment 9 Oved Ourfali 2015-12-17 02:17:21 EST
Pavel/Jiri - I think that if after upgrade of RHEL the selinux status is changing, and not remaining as it was before upgrade, then it is a RHEL bug.
If that's the case, please open a RHEL bug, and make this bug a dependency.
Comment 10 Jiri Belka 2015-12-21 12:02:43 EST
(In reply to Oved Ourfali from comment #9)
> Pavel/Jiri - I think that if after upgrade of RHEL the selinux status is
> changing, and not remaining as it was before upgrade, then it is a RHEL bug.
> If that's the case, please open a RHEL bug, and make this bug a dependency.

dfediuck@ sent recently an email stating that fed-up style migration is abandoned. could you manage with him if this BZ is still relevent? thx.

(ad selinux _during_ upgrade - imo they have reason to do this; after upgrade selinux is enforcing as it used to be before upgrade.)
Comment 11 Oved Ourfali 2015-12-21 12:39:43 EST
(In reply to Jiri Belka from comment #10)
> (In reply to Oved Ourfali from comment #9)
> > Pavel/Jiri - I think that if after upgrade of RHEL the selinux status is
> > changing, and not remaining as it was before upgrade, then it is a RHEL bug.
> > If that's the case, please open a RHEL bug, and make this bug a dependency.
> 
> dfediuck@ sent recently an email stating that fed-up style migration is
> abandoned. could you manage with him if this BZ is still relevent? thx.
> 

Doron? 

> (ad selinux _during_ upgrade - imo they have reason to do this; after
> upgrade selinux is enforcing as it used to be before upgrade.)

Jiri - What do you mean? 
As far as I understand it was disabled before and enabled after. Am I wrong?
Comment 12 Jiri Belka 2015-12-22 03:49:43 EST
> > (ad selinux _during_ upgrade - imo they have reason to do this; after
> > upgrade selinux is enforcing as it used to be before upgrade.)
> 
> Jiri - What do you mean? 
> As far as I understand it was disabled before and enabled after. Am I wrong?

Yes are probably confused - it was enforcing, during uprade it is disabled as kernel param and after upgrade it is as it used to be originally - ie. enforcing. IMO the upgrade-migration has its own reason why it puts 'enforcing=0' as kernel param (as visible in my above original descr).
Comment 13 Oved Ourfali 2015-12-22 03:52:55 EST
(In reply to Jiri Belka from comment #12)
> > > (ad selinux _during_ upgrade - imo they have reason to do this; after
> > > upgrade selinux is enforcing as it used to be before upgrade.)
> > 
> > Jiri - What do you mean? 
> > As far as I understand it was disabled before and enabled after. Am I wrong?
> 
> Yes are probably confused - it was enforcing, during uprade it is disabled
> as kernel param and after upgrade it is as it used to be originally - ie.
> enforcing. IMO the upgrade-migration has its own reason why it puts
> 'enforcing=0' as kernel param (as visible in my above original descr).

So we might need to "remember" that selinux was enabled, and creating proper rules when upgrading even though selinux is still disabled?
Yaniv?
Comment 14 Jiri Belka 2015-12-22 04:01:00 EST
the "upgrade/migration" as one-time boot via a kernel/ramdisk generated with redhat-upgrade-tool, only this one-time boot selinux is disabled.

i don't know what exactly "vdsm-tool configure --module sebool" does but this solved the issue. maybe it is just an issue of relabeling dirs context?
Comment 16 Yaniv Bronhaim 2016-01-04 08:48:12 EST
No its not setting any labeling things. it sets selinux booleans that vdsm requires - you can read about it and see the implantation in lib/vdsm/tool/configurators/sebool.py 

if selinux is disabled this can't be set. but if selinux is enabled and the booleans are not set, vdsm won't start until you'll set it (which you do by using vdsm-tool)
so the issue here was that vdsm ran without selinux, and after upgrade failed to run as it didn't trigger vdsm-tool configure for sebool module when selinux turns on. 
maybe we should keep the consideration of taking https://gerrit.ovirt.org/#/c/50382/1/init/vdsmd_init_common.sh.in

Note You need to log in before you can comment on or make changes to this bug.