Bug 1444426 - [RHEL 7.4] vdsm fail to start - libvirt 3.2 changes md5 to gssapi in libvirt.conf for sasl
Summary: [RHEL 7.4] vdsm fail to start - libvirt 3.2 changes md5 to gssapi in libvirt....
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: vdsm
Classification: oVirt
Component: Core
Version: 4.20.0
Hardware: Unspecified
OS: Unspecified
high
urgent
Target Milestone: ovirt-4.1.3
: 4.19.18
Assignee: Yaniv Bronhaim
QA Contact: Lucie Leistnerova
URL:
Whiteboard:
: 1446475 1448354 (view as bug list)
Depends On: 1456235
Blocks: 1449577 1451038 1458856
TreeView+ depends on / blocked
 
Reported: 2017-04-21 10:11 UTC by Lucie Leistnerova
Modified: 2019-04-28 13:30 UTC (History)
22 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
undefined
Clone Of:
: 1451038 1458856 (view as bug list)
Environment:
Last Closed: 2017-07-06 13:09:06 UTC
oVirt Team: Infra
rule-engine: ovirt-4.1+
rule-engine: blocker+
pstehlik: testing_ack+


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Knowledge Base (Solution) 3291671 0 None None None 2017-12-16 19:22:39 UTC
oVirt gerrit 76934 0 master MERGED Configure sasl2 libvirt to use scram-sha-1 2020-09-16 20:26:23 UTC
oVirt gerrit 77534 0 ovirt-4.1 MERGED Configure sasl2 libvirt to use scram-sha-1 2020-09-16 20:26:21 UTC

Description Lucie Leistnerova 2017-04-21 10:11:48 UTC
Description of problem:
libvirt 3.2 changes default setting in /etc/sasl2/libvirt.conf from md5 to gssapi and doesn't install cyrus-sasl-md5 rpm. So vdsm can be installed, but doesn't start (can't authenticate to libvirt).

Version-Release number of selected component (if applicable):
vdsm-4.20.0-634.gitffa09a7.el7.centos.x86_64

vdsm should change the config back to md5 and install the rpm or use gssapi because it should be safer.

libvirt commit 3c647ee4bbb25078c0f7cc59a79221ccb7d438f0

Comment 1 Oved Ourfali 2017-04-30 11:47:03 UTC
Yaniv - can you take a look?

Comment 2 Yaniv Bronhaim 2017-04-30 15:36:37 UTC
Hi Lucie, do you know when libvirt 3.2 will be available upstream? In fedora 25 latest build they have libvirt-2.2.0 , I see recent 3.2 build in [1]  but any idea when it will be upstream or in rhel7.3 by default?

[1] https://brewweb.engineering.redhat.com/brew/packageinfo?packageID=1989

If we will require that change in 4.1 I'll duplicate the bug, setting this one for 4.2 for now.

Comment 3 Lucie Leistnerova 2017-05-02 07:38:13 UTC
I asked Michal Privoznik and libvirt 3.2 will be in RHEL 7.4 as default, not in 7.3. And it is already in Fedora 26, 27 and in Fedora 27 should be libvirt 3.3 in few weeks.

Comment 4 Yaniv Kaul 2017-05-07 06:52:53 UTC
*** Bug 1448354 has been marked as a duplicate of this bug. ***

Comment 5 Martin Polednik 2017-05-09 10:20:33 UTC
I think we need to do following things:

- VDSM depends on cyrus-sasl-md5
- change the /etc/sasl2/libvirt.conf back to 

mech_list: digest-md5
sasldb_path: /etc/libvirt/passwd.db

Works as workaround for me.

Comment 8 Daniel Berrangé 2017-05-11 13:48:53 UTC
(In reply to Martin Polednik from comment #5)
> I think we need to do following things:
> 
> - VDSM depends on cyrus-sasl-md5
> - change the /etc/sasl2/libvirt.conf back to 
> 
> mech_list: digest-md5
> sasldb_path: /etc/libvirt/passwd.db

No, you really should not configure digest-md5. It was disabled because it is a broken auth mechanism from a security POV. You want to set 'mech_list: scram-sha-1', as per the comments in the libvirt.conf file.

It should be backwards compatible with the existing passwd.db format used by the old digest-md5 method.

Comment 9 Martin Polednik 2017-05-11 14:03:33 UTC
(In reply to Daniel Berrange from comment #8)
> (In reply to Martin Polednik from comment #5)
> > I think we need to do following things:
> > 
> > - VDSM depends on cyrus-sasl-md5
> > - change the /etc/sasl2/libvirt.conf back to 
> > 
> > mech_list: digest-md5
> > sasldb_path: /etc/libvirt/passwd.db
> 
> No, you really should not configure digest-md5. It was disabled because it
> is a broken auth mechanism from a security POV. You want to set 'mech_list:
> scram-sha-1', as per the comments in the libvirt.conf file.
> 
> It should be backwards compatible with the existing passwd.db format used by
> the old digest-md5 method.

The username/password is in our public repository. It's more of an anti-tamper than security thing.

Comment 10 Daniel Berrangé 2017-05-11 14:05:55 UTC
(In reply to Martin Polednik from comment #9)
> (In reply to Daniel Berrange from comment #8)
> > (In reply to Martin Polednik from comment #5)
> > > I think we need to do following things:
> > > 
> > > - VDSM depends on cyrus-sasl-md5
> > > - change the /etc/sasl2/libvirt.conf back to 
> > > 
> > > mech_list: digest-md5
> > > sasldb_path: /etc/libvirt/passwd.db
> > 
> > No, you really should not configure digest-md5. It was disabled because it
> > is a broken auth mechanism from a security POV. You want to set 'mech_list:
> > scram-sha-1', as per the comments in the libvirt.conf file.
> > 
> > It should be backwards compatible with the existing passwd.db format used by
> > the old digest-md5 method.
> 
> The username/password is in our public repository. It's more of an
> anti-tamper than security thing.

That's still not a good reason to use 'digest-md5' when 'scram-sha-1' exists and doesn't suffer from broken design.

Comment 11 Martin Polednik 2017-05-11 14:19:35 UTC
(In reply to Daniel Berrange from comment #10)
> (In reply to Martin Polednik from comment #9)
> > (In reply to Daniel Berrange from comment #8)
> > > (In reply to Martin Polednik from comment #5)
> > > > I think we need to do following things:
> > > > 
> > > > - VDSM depends on cyrus-sasl-md5
> > > > - change the /etc/sasl2/libvirt.conf back to 
> > > > 
> > > > mech_list: digest-md5
> > > > sasldb_path: /etc/libvirt/passwd.db
> > > 
> > > No, you really should not configure digest-md5. It was disabled because it
> > > is a broken auth mechanism from a security POV. You want to set 'mech_list:
> > > scram-sha-1', as per the comments in the libvirt.conf file.
> > > 
> > > It should be backwards compatible with the existing passwd.db format used by
> > > the old digest-md5 method.
> > 
> > The username/password is in our public repository. It's more of an
> > anti-tamper than security thing.
> 
> That's still not a good reason to use 'digest-md5' when 'scram-sha-1' exists
> and doesn't suffer from broken design.

I guess it can't hurt.

Related: why doesn't libvirt us scram plain as default? After upgrading my machines, I have to change even basic libvirt to use virsh. That would also not break oVirt on upgrade.

Comment 12 Daniel Berrangé 2017-05-11 14:27:30 UTC
(In reply to Martin Polednik from comment #11)
> Related: why doesn't libvirt us scram plain as default? After upgrading my
> machines, I have to change even basic libvirt to use virsh. That would also
> not break oVirt on upgrade.

The libvirt default setup is designed for securing the TCP socket, where you need a mechanism that can provide encryption & authentication. GSSAPI/Kerberos is the only mechanism that can provide that. The SCRAM mechanism requires a separate encryption layer, so isn't suitable as a default.

Personally I'd just recommend deleting this oVirt use of SASL for the UNIX socket, and reverting to the libvirt default of polkit, as is used with OpenStack. It adds no security since the password is well known, and if you have a user with root  access who is reckless, virsh access is the least of your worries.

Comment 13 Martin Polednik 2017-05-11 14:40:18 UTC
(In reply to Daniel Berrange from comment #12)
> Personally I'd just recommend deleting this oVirt use of SASL for the UNIX
> socket, and reverting to the libvirt default of polkit, as is used with
> OpenStack. It adds no security since the password is well known, and if you
> have a user with root  access who is reckless, virsh access is the least of
> your worries.

It works extremely well for "if you have to ask about password, don't touch it" scenario. But that should be doable with a polkit rule too I guess.

Comment 15 Michal Skrivanek 2017-05-12 11:09:50 UTC
*** Bug 1446475 has been marked as a duplicate of this bug. ***

Comment 16 Daniel Berrangé 2017-05-15 15:53:50 UTC
(In reply to Michal Skrivanek from comment #14)
> if scram-sha-1 can handle the existing user db it's the least risky change,
> we can push vdsm update to RHEL-7-RHEV-4 and RHEL-7-RHEV to cover all
> affected versons

FYI here's the demo scenario to show compatibility of existing DB

# grep ^mech_list /etc/sasl2/libvirt.conf
mech_list: digest-md5

# saslpasswd2 -a libvirt demo
Password: 
Again (for verification): 

# virsh hostname
Please enter your authentication name: demo
Please enter your password: 
t460

# vi /etc/sasl2/libvirtd.conf
...change to scram-sha-1...

# grep ^mech_list /etc/sasl2/libvirt.conf
mech_list: scram-sha-1

# systemctl restart libvirtd

# virsh hostname
Please enter your authentication name: demo
Please enter your password: 
t460

Comment 17 Yaniv Bronhaim 2017-05-17 09:32:14 UTC
Hi Daniel,
when I try to apply mech_list: scram-sha-1 instead of digest-md5 - I get failed to connect to the hypervisor.

[root@dhcp-0-135 vdsm]# grep ^mech_list /etc/sasl2/libvirt.conf
mech_list: scram-sha-1
[root@dhcp-0-135 vdsm]# saslpasswd2 -a libvirt demo
Password: 
Again (for verification): 
[root@dhcp-0-135 vdsm]# virsh hostname
error: failed to connect to the hypervisor
error: authentication failed: authentication failed

libvirtd is running, and vdsm can't communicate with it. command line doesn't request for authentication as with digest-md5

[root@dhcp-0-135 vdsm]# virsh list
error: failed to connect to the hypervisor
error: authentication failed: authentication failed

[root@dhcp-0-135 vdsm]# virsh hostname
error: failed to connect to the hypervisor
error: authentication failed: authentication failed


Another question - don't you prefer, for safety, to Conflict this version of libvirt with those broken versions of vdsm?

Comment 18 Daniel Berrangé 2017-05-17 09:52:18 UTC
(In reply to Yaniv Bronhaim from comment #17)
> Hi Daniel,
> when I try to apply mech_list: scram-sha-1 instead of digest-md5 - I get
> failed to connect to the hypervisor.
> 
> [root@dhcp-0-135 vdsm]# grep ^mech_list /etc/sasl2/libvirt.conf
> mech_list: scram-sha-1
> [root@dhcp-0-135 vdsm]# saslpasswd2 -a libvirt demo
> Password: 
> Again (for verification): 
> [root@dhcp-0-135 vdsm]# virsh hostname
> error: failed to connect to the hypervisor
> error: authentication failed: authentication failed
> 
> libvirtd is running, and vdsm can't communicate with it. command line
> doesn't request for authentication as with digest-md5

Presumably you are missing the cyrus-sasl-scram RPM.

Comment 22 Yaniv Bronhaim 2017-05-28 06:26:31 UTC
verification notes: upgraded to fresh build of libvirt-3.2.0-5.el7 from libvirt-2.0.0-10.el7 (latest in centos7) while vdsm latest master is installed and running without my patch:

- upgrade only libvirt while vdsm was running - vdsm will be down after the upgrade and require an upgrade.

- upgrade only vdsm - still work as usual with older libvirt version

- upgrade vdsm and then libvirt  - vdsm will go down and require vdsm-tool configure call manually.

- upgrade both at the same time while vdsm is running (the only thing that should work fine) - tested over centos7.

any suggestion for more flows ?

Comment 23 Lucie Leistnerova 2017-05-29 07:24:11 UTC
I miss fresh installation of newest vdsm with libvirt 3.2 and maybe upgrade from engine. Otherwise it looks alright for me.

Comment 24 Yaniv Bronhaim 2017-05-29 08:39:40 UTC
I wonder - we require now cyrus-sasl-scram - do we want to limit it only to 7.4 and above? does libvirt plan to backport 3.2 to rhel7.3?

Comment 25 Dan Kenigsberg 2017-05-29 12:16:37 UTC
I don't think I understand your question, Yaniv. Once el-7.4 is out, ovirt-4.1 does not need to continue support of 7.3.z. Your patch can safely require a package that is only available in 7.4 (it only means that your patch cannot be merged upstream before the package is available).

Comment 26 Daniel Berrangé 2017-05-30 09:09:54 UTC
(In reply to Yaniv Bronhaim from comment #24)
> I wonder - we require now cyrus-sasl-scram - do we want to limit it only to
> 7.4 and above? does libvirt plan to backport 3.2 to rhel7.3?

Libvirt has no intention to make sure changes in pre-existing 7.x branches. IIUC, the VDSM patch for this makes VDSM totally independent of libvirt's default config, because it drops in a replacement SASL config file, completely ignoring whatever content libvirt had before. So it shouldn't matter what libvirt does to SASL configs at this point.

Comment 27 rhev-integ 2017-06-01 16:13:27 UTC
INFO: Bug status wasn't changed from MODIFIED to ON_QA due to the following reason:

[Tag 'v4.19.17' doesn't contain patch 'https://gerrit.ovirt.org/77534']
gitweb: https://gerrit.ovirt.org/gitweb?p=vdsm.git;a=shortlog;h=refs/tags/v4.19.17

For more info please contact: infra@ovirt.org

Comment 28 Lucie Leistnerova 2017-06-20 13:36:37 UTC
Vdsm changed sasl2 settings from gssapi to scram-sha-1
and started successfully (both: manual and from engine).

verified in vdsm-4.19.19-1.el7ev.x86_64, ovirt-engine-4.1.3.4-0.1.el7.noarch


Note You need to log in before you can comment on or make changes to this bug.