Description of problem: In BZ1412552 we have added support for TLSv1.1 and TLSv1.2 communication protocols to VDSM (which already supported TLSv1). oVirt Engine 3.6 can communicate with VDSM only using TLSv1, but oVirt Engine 4.x already support TLSv1.2. So the plan is to drop support for already insecure TLSv1 [1] and TLSv1.1 and leave only most secure TLSv1.2 [2]. [1] https://blog.varonis.com/ssl-and-tls-1-0-no-longer-acceptable-for-pci-compliance/ [2] https://en.wikipedia.org/wiki/Transport_Layer_Security#TLS_1.2 Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info:
Moving to 4.3, because implementing this will break 3.6 compatibility, because both 3.6 engine and VDSM support only TLSv1
TLSv1 and/or TLSv1.1 protocols used for engine <-> VDSM and VDSM <-> VDSM communication can be disabled by manual configuration on all hosts: 1. Put host into the maintenance 2. Create custom configuration file /etc/vdsm/vdsm.conf.d/99-disable-older-tls.conf with following content: ssl_excludes = OP_NO_TLSv1,OP_NO_TLSv1_1 3. Restart VDSM 4. Activate the host When above steps are applied all hosts in RHV setup, then only TLSv1.2 will be used for engine <-> VDSM and VDSM <-> VDSM communication. IMPORTANT: Above manual configuration change can be applied only to RHV 4.1.5 and newer versions, where all hosts are upgraded to VDSM 4.19.27 and newer (more info at [1]). If this condition is not true, all older hosts will become NonResponsive and VMs won't be able to migrat to/from those older hosts. [1] https://bugzilla.redhat.com/show_bug.cgi?id=1408847#c38
Martin, how about the manager? I believe we should add here also requirement on the engine to be 4.1.5 and up, right?
(In reply to Marina from comment #8) > Martin, how about the manager? > I believe we should add here also requirement on the engine to be 4.1.5 and > up, right? The requirement is mentioned in comment 7: > IMPORTANT: Above manual configuration change can be applied only to RHV > 4.1.5 and newer versions, where all hosts are upgraded to VDSM 4.19.27 and > newer (more info at [1]). If this condition is not true, all older hosts > will become NonResponsive and VMs won't be able to migrat to/from those > older hosts. > > [1] https://bugzilla.redhat.com/show_bug.cgi?id=1408847#c38
This comment fixes and overrides information from Comment 7: TLSv1 and/or TLSv1.1 protocols used for engine <-> VDSM and VDSM <-> VDSM communication can be disabled by manual configuration on all hosts: 1. Put host into the maintenance 2. Create custom configuration file /etc/vdsm/vdsm.conf.d/99-disable-older-tls.conf with following content: [vars] ssl_excludes = OP_NO_TLSv1,OP_NO_TLSv1_1 3. Restart VDSM 4. Activate the host When above steps are applied all hosts in RHV setup, then only TLSv1.2 will be used for engine <-> VDSM and VDSM <-> VDSM communication. IMPORTANT: Above manual configuration change can be applied only to RHV 4.1.5 and newer versions, where engine is upgraded to 4.1.5 and newer and all hosts are upgraded to VDSM 4.19.27 and newer (more info at [1]). If this condition is not true, all older hosts will become NonResponsive and VMs won't be able to migrat to/from those older hosts. [1] https://bugzilla.redhat.com/show_bug.cgi?id=1408847#c38
OK, so we have found an issue: from 4.1.5 the TLSv12 support is included, but it's disabled by default. To enable it following VDSM options need to be updated to: ssl_protocol=sslv23 In RHV 4.2 this is enabled by default, we will backport it to 4.1.8 (BZ1511962). So if you want to disable TLSv1/TLSv11, you need also to have this options set in configuration file mentioned Comment 10.
Hi Martin, will 4.2 have that var (ssl_protocol=sslv23) included in vdsm.conf, or do you mean it will not be required in 4.2? I tested the config to enable v.1.2 and disable v1.0, it works: [root@rhevh-25 ~]# rpm -q vdsm vdsm-4.19.28-1.el7ev.x86_64 - by default, it does not use v.1.2 [root@rhevh-25 ~]# openssl s_client -connect localhost:54321 -CAfile /etc/pki/vdsm/certs/cacert.pem 2>&1 | grep -e Protocol -e Cipher New, TLSv1/SSLv3, Cipher is AES256-SHA Protocol : TLSv1 Cipher : AES256-SHA - enable v.1.2 [root@rhevh-25 ~]# cp /etc/vdsm/vdsm.conf /etc/vdsm/vdsm.conf.0 [root@rhevh-25 ~]# vi /etc/vdsm/vdsm.conf [root@rhevh-25 ~]# diff /etc/vdsm/vdsm.conf /etc/vdsm/vdsm.conf.0 3d2 < ssl_protocol = sslv23 [root@rhevh-25 ~]# systemctl restart vdsmd.service [root@rhevh-25 ~]# openssl s_client -connect localhost:54321 -CAfile /etc/pki/vdsm/certs/cacert.pem 2>&1 | grep -e Protocol -e Cipher New, TLSv1/SSLv3, Cipher is AES256-GCM-SHA384 Protocol : TLSv1.2 Cipher : AES256-GCM-SHA384 - it still supports v.1.0 [root@rhevh-25 ~]# openssl s_client -connect localhost:54321 -CAfile /etc/pki/vdsm/certs/cacert.pem 2>&1 -tls1 | grep -e Protocol -e Cipher New, TLSv1/SSLv3, Cipher is AES256-SHA Protocol : TLSv1 Cipher : AES256-SHA - disable v.1.0/v.1.1 [root@rhevh-25 ~]# vi /etc/vdsm/vdsm.conf [root@rhevh-25 ~]# diff /etc/vdsm/vdsm.conf /etc/vdsm/vdsm.conf.0 3,4d2 < ssl_protocol = sslv23 < ssl_excludes = OP_NO_TLSv1,OP_NO_TLSv1_1 [root@rhevh-25 ~]# systemctl restart vdsmd.service [root@rhevh-25 ~]# openssl s_client -connect localhost:54321 -CAfile /etc/pki/vdsm/certs/cacert.pem 2>&1 -tls1 | grep -e Protocol -e Cipher New, (NONE), Cipher is (NONE) Protocol : TLSv1 Cipher : 0000 - it still supports v.1.2 [root@rhevh-25 ~]# openssl s_client -connect localhost:54321 -CAfile /etc/pki/vdsm/certs/cacert.pem 2>&1 | grep -e Protocol -e Cipher New, TLSv1/SSLv3, Cipher is AES256-GCM-SHA384 Protocol : TLSv1.2 Cipher : AES256-GCM-SHA384
(In reply to Donald Berry from comment #12) > Hi Martin, will 4.2 have that var (ssl_protocol=sslv23) included in > vdsm.conf, or do you mean it will not be required in 4.2? vdsm.conf does not include default values, it only specifies values for options, which don't have default value, or overwrites default values specified in the VDSM code. > > I tested the config to enable v.1.2 and disable v1.0, it works: > > [root@rhevh-25 ~]# rpm -q vdsm > vdsm-4.19.28-1.el7ev.x86_64 > > - by default, it does not use v.1.2 > > [root@rhevh-25 ~]# openssl s_client -connect localhost:54321 -CAfile > /etc/pki/vdsm/certs/cacert.pem 2>&1 | grep -e Protocol -e Cipher > New, TLSv1/SSLv3, Cipher is AES256-SHA > Protocol : TLSv1 > Cipher : AES256-SHA Right, will be fixed in 4.1.8 (BZ1511962) > > - enable v.1.2 > > [root@rhevh-25 ~]# cp /etc/vdsm/vdsm.conf /etc/vdsm/vdsm.conf.0 > [root@rhevh-25 ~]# vi /etc/vdsm/vdsm.conf > [root@rhevh-25 ~]# diff /etc/vdsm/vdsm.conf /etc/vdsm/vdsm.conf.0 > 3d2 > < ssl_protocol = sslv23 Please don't vdsm.conf directly, it's preferred to do user customizations via custom .conf file in vdsm.conf.d (see my example in Comment 10) > [root@rhevh-25 ~]# systemctl restart vdsmd.service > [root@rhevh-25 ~]# openssl s_client -connect localhost:54321 -CAfile > /etc/pki/vdsm/certs/cacert.pem 2>&1 | grep -e Protocol -e Cipher > New, TLSv1/SSLv3, Cipher is AES256-GCM-SHA384 > Protocol : TLSv1.2 > Cipher : AES256-GCM-SHA384 > > - it still supports v.1.0 Yes, TLSv1 is still supported, but most of clients negotiate highest available version, which is TLSv12 > > [root@rhevh-25 ~]# openssl s_client -connect localhost:54321 -CAfile > /etc/pki/vdsm/certs/cacert.pem 2>&1 -tls1 | grep -e Protocol -e Cipher > New, TLSv1/SSLv3, Cipher is AES256-SHA > Protocol : TLSv1 > Cipher : AES256-SHA > > - disable v.1.0/v.1.1 > > [root@rhevh-25 ~]# vi /etc/vdsm/vdsm.conf > [root@rhevh-25 ~]# diff /etc/vdsm/vdsm.conf /etc/vdsm/vdsm.conf.0 > 3,4d2 > < ssl_protocol = sslv23 > < ssl_excludes = OP_NO_TLSv1,OP_NO_TLSv1_1 Same as above, please don't edit vdsm.conf directly > [root@rhevh-25 ~]# systemctl restart vdsmd.service > [root@rhevh-25 ~]# openssl s_client -connect localhost:54321 -CAfile > /etc/pki/vdsm/certs/cacert.pem 2>&1 -tls1 | grep -e Protocol -e Cipher > New, (NONE), Cipher is (NONE) > Protocol : TLSv1 > Cipher : 0000 > > - it still supports v.1.2 It supports only TLSv12, everything else is disabled > > [root@rhevh-25 ~]# openssl s_client -connect localhost:54321 -CAfile > /etc/pki/vdsm/certs/cacert.pem 2>&1 | grep -e Protocol -e Cipher > New, TLSv1/SSLv3, Cipher is AES256-GCM-SHA384 > Protocol : TLSv1.2 > Cipher : AES256-GCM-SHA384
pushing to 4.4, assuming we are keeping 3.6 cluster level in 4.3
(In reply to Moran Goldboim from comment #15) > pushing to 4.4, assuming we are keeping 3.6 cluster level in 4.3 We have since changed our plans a bit - moving back to 4.3.
Yaniv they are using a security sweep that RHV must pass for them to use the product. It seems to scan all ports including the port for communication between vdsm and the engine. This port and ovirt-imageio-daemon are the only ones we have been able to solve. System | Port | Service | Concerns | Notes ============================================================================================== RHV-H | 16514 | libvirt (solved) | Weak ciphers | RC4/3DES cipher RHV-H | 54321 | vdsm | 3DES cipher | Needs testing internally RHV-H | 54322 | ovirt-imageio-daemon | RC4/SSL3/TLS1.0 ** | RHV 4.2 disable TLS 1.0 RHV-M | 54323 | ovirt-imageio-daemon | RC4/SSL3/TLS1.0 ** | RHV 4.2 disables TLS 1.0
(In reply to Frank DeLorey from comment #17) > Yaniv they are using a security sweep that RHV must pass for them to use the > product. It seems to scan all ports including the port for communication > between vdsm and the engine. This port and ovirt-imageio-daemon are the only > ones we have been able to solve. > > > System | Port | Service | Concerns | Notes > ============================================================================= > ================= > RHV-H | 16514 | libvirt (solved) | Weak ciphers | RC4/3DES > cipher > RHV-H | 54321 | vdsm | 3DES cipher | Needs > testing internally This is still happening, even after the move to TLS 1.2 only? > RHV-H | 54322 | ovirt-imageio-daemon | RC4/SSL3/TLS1.0 ** | RHV 4.2 > disable TLS 1.0 > RHV-M | 54323 | ovirt-imageio-daemon | RC4/SSL3/TLS1.0 ** | RHV 4.2 > disables TLS 1.0 This will probably fixed in 4.2.z.
This is happening after disabling TLSv1.0. They still want and have TLSv1.1 enabled. Frank
(In reply to Frank DeLorey from comment #19) > This is happening after disabling TLSv1.0. They still want and have TLSv1.1 > enabled. > > Frank Let's try with TLS 1.2 only and understand in parallel why do we want TLS 1.1 enabled.
(In reply to Yaniv Kaul from comment #18) > (In reply to Frank DeLorey from comment #17) > > RHV-H | 54322 | ovirt-imageio-daemon | RC4/SSL3/TLS1.0 ** | RHV 4.2 > > disable TLS 1.0 > > RHV-M | 54323 | ovirt-imageio-daemon | RC4/SSL3/TLS1.0 ** | RHV 4.2 > > disables TLS 1.0 > > This will probably fixed in 4.2.z. Since ovirt-imageio-1.1.0 (ovirt-4.2) we support only TLSv1.2 and TLSv1.1, and RC4 is not supported. See bug 1578412 for more details and planned changes for 4.2.z.
From the customer, it appears using just TLSv1.2 does not address the cipher issue for vdsm: "Following the method to retrieve the default cipher using the openssl command from the Red Hat KCS for vdsm and setting it up for TLSv1.2 only, it does show Protocol: TLSv1.2 and Cipher: AES256-GCM-SHA384. Although when using nmap, to enumerate the ssl ciphers on that port, the 3DES and RC4 ciphers are still available. Please see below for the output. PORT STATE SERVICE VERSION 54321/tcp open ssl/unknown | ssl-enum-ciphers: | TLSv1.2: | ciphers: | TLS_RSA_WITH_3DES_EDE_CBC_SHA (rsa 2048) - C | TLS_RSA_WITH_AES_128_CBC_SHA (rsa 2048) - A | TLS_RSA_WITH_AES_128_CBC_SHA256 (rsa 2048) - A | TLS_RSA_WITH_AES_128_GCM_SHA256 (rsa 2048) - A | TLS_RSA_WITH_AES_256_CBC_SHA (rsa 2048) - A | TLS_RSA_WITH_AES_256_CBC_SHA256 (rsa 2048) - A | TLS_RSA_WITH_AES_256_GCM_SHA384 (rsa 2048) - A | TLS_RSA_WITH_CAMELLIA_128_CBC_SHA (rsa 2048) - A | TLS_RSA_WITH_CAMELLIA_256_CBC_SHA (rsa 2048) - A | TLS_RSA_WITH_IDEA_CBC_SHA (rsa 2048) - A | TLS_RSA_WITH_RC4_128_SHA (rsa 2048) - C | TLS_RSA_WITH_SEED_CBC_SHA (rsa 2048) - A | compressors: | NULL | cipher preference: client | warnings: | 64-bit block cipher 3DES vulnerable to SWEET32 attack | 64-bit block cipher IDEA vulnerable to SWEET32 attack | Broken cipher RC4 is deprecated by RFC 7465 |_ least strength: C"
Cipher issue to be solved by bug 1577594
Verified by running: openssl s_client -connect localhost:54321 -tls1 -CAfile /etc/pki/vdsm/certs/cacert.pem 2>&1 openssl s_client -connect localhost:54321 -tls1_1 -CAfile /etc/pki/vdsm/certs/cacert.pem 2>&1 openssl s_client -connect localhost:54321 -tls1_2 -CAfile /etc/pki/vdsm/certs/cacert.pem 2>&1 For the first two runs I get: Protocol : TLSv1 Cipher : 0000 Protocol : TLSv1.1 Cipher : 0000 and: SSL handshake has read 0 bytes and written 0 bytes while for '-tls1_2' I get: SSL handshake has read 2537 bytes and written 138 bytes ... Protocol : TLSv1.2 Cipher : ECDHE-RSA-AES256-GCM-SHA384 and I can see the certificate that VDSM responded with.
Please review the updated content in the Doc Text field, above.
This bugzilla is included in oVirt 4.3.0 release, published on February 4th 2019. Since the problem described in this bug report should be resolved in oVirt 4.3.0 release, it has been closed with a resolution of CURRENT RELEASE. If the solution does not work for you, please open a new bug report.