Bug 1636023

Summary: Install fails for a cluster imported into RHGS Console
Product: [Red Hat Storage] Red Hat Gluster Storage Reporter: Sweta Anandpara <sanandpa>
Component: rhscAssignee: Sahina Bose <sabose>
Status: CLOSED WONTFIX QA Contact: Rahul Hinduja <rhinduja>
Severity: medium Docs Contact:
Priority: medium    
Version: rhgs-3.4CC: amukherj, ansasaki, didi, nmavrogi, rcyriac, rhs-bugs, rhsc-qe-bugs, sabose, sanandpa, sankarshan, storage-qa-internal
Target Milestone: ---Keywords: Regression, ZStream
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-04-22 06:28:08 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Sweta Anandpara 2018-10-04 09:30:58 UTC
Description of problem:
=======================
Had a 4node cluster with 3.12.2-20 on rhel7.6.  When imported into console it failed with "Command returned failure code 1 during SSH session". The services libvirtd, vdsmd, supervdsmd all show down on gluster nodes.

Not uploading any logs as the setup has been shared with Dev yesterday.

Version-Release number of selected component (if applicable):
==================================================
3.12.2-20


How reproducible:
==================
Always


Steps to Reproduce:
===================
1. Have a cluster with RHGS 3.4.1 and beta/RC build of rhel7.6
2. IMport the cluster into RHGS Console


Actual results:
===============
Cluster installation fails, and the node is shown down.


Expected results:
================
All the nodes should get successfully configured when imported.

Comment 3 Atin Mukherjee 2018-10-04 12:05:01 UTC
Does this work in RHEL 7.5 with the same gluster bit?

Comment 4 Sahina Bose 2018-10-04 13:35:34 UTC
Error in starting libvirtd service
2018-10-04 12:47:13.528+0000: 11715: info : virObjectNew:248 : OBJECT_NEW: obj=0x56241fc3a200 classname=virNetTLSContext
2018-10-04 12:47:13.528+0000: 11715: debug : virNetTLSContextLoadCertFromFile:495 : isServer 1 certFile /etc/pki/vdsm/certs/vdsmcert.pem
2018-10-04 12:47:13.529+0000: 11715: debug : virFileClose:111 : Closed fd 14
2018-10-04 12:47:13.529+0000: 11715: error : virNetTLSContextLoadCertFromFile:513 : Unable to import server certificate /etc/pki/vdsm/certs/vdsmcert.pem


[root@dhcp46-102 ~]# openssl verify -CAfile /etc/pki/vdsm/certs/cacert.pem /etc/pki/vdsm/certs/vdsmcert.pem
/etc/pki/vdsm/certs/vdsmcert.pem: OK

Comment 5 Sahina Bose 2018-10-05 04:29:20 UTC
I tried adding the same node to a RHV-M cluster - and this worked. Problem seems to be with the certificate issued by RHGS-C.

Didi, any clues on why libvirt would fail to import the vdsmcert (/etc/pki/vdsm/certs/vdsmcert.pem)

I tried the below command - openssl x509 -in /etc/pki/vdsm/certs/vdsmcert.pem -text -noout and the certificate seemed very similar to the one issued by RHV-M that worked.

Comment 6 Yedidyah Bar David 2018-10-08 08:54:26 UTC
Sorry, no idea. If you do not manage to find a solution/explanation, please let me access the relevant machines and I can try looking some more.

Comment 7 Sahina Bose 2018-10-09 04:56:39 UTC
Sweta, can you provide the openssl and libvirtd versions from an RHGS 3.4 node that worked, and the RHGS 3.4.1 that did not?

Comment 9 Sahina Bose 2018-10-09 07:16:30 UTC
The issue seems to be with the gnutls version installed with RHGS 3.4.1 and RHEL 7.6.

When I downgraded the below packages, the libvirtd service started successfully.
================================================================================
 Package                 Arch              Version                  
Downgrading:
 gnutls                  x86_64            3.3.26-9.el7              
 gnutls-dane             x86_64            3.3.26-9.el7        
 gnutls-utils            x86_64            3.3.26-9.el7     


Atin, do you know if we have a requirement on specific version of gnutls?

The next thing would be to check what changed in gnutls that broke this. Also adding needinfo on the last person to commit to gnutls repo for any pointers on why certificate import fails with latest version.

Comment 10 Atin Mukherjee 2018-10-10 14:20:50 UTC
Don't we know which RHGS package needs this dependency? If you can get me that info, I can check with the respective packagers to see if there's any particular version dependency as such.

Comment 11 Anderson Sasaki 2018-10-23 13:22:51 UTC
This can be caused by the more strict rules in certificate import that exist in the new gnutls version, which means that maybe there is a problem with the certificate being imported.

Can you please provide the certificate you are trying to import?

Comment 12 Sahina Bose 2018-10-24 09:26:43 UTC
Sweta, can you provide this info?

Comment 13 Sahina Bose 2018-10-26 10:49:06 UTC
Created attachment 1497703 [details]
certificate

Comment 14 Sahina Bose 2018-10-26 10:51:03 UTC
(In reply to Anderson Sasaki from comment #11)
> This can be caused by the more strict rules in certificate import that exist
> in the new gnutls version, which means that maybe there is a problem with
> the certificate being imported.
> 
> Can you please provide the certificate you are trying to import?

Attached the certificate. Let me know if you need any further information

Comment 15 Nikos Mavrogiannopoulos 2018-10-31 10:21:08 UTC
I tried dumpasn1 to this certificate:

 130   34:     SEQUENCE {
 132   17:       UTCTime '181002034248+0000'
         :         Error: Time is encoded incorrectly.
 151   13:       UTCTime 02/10/2023 03:42:48 GMT
         :       }

It is malformed DER encoding of time, that's why it fails to import. Which tool did you use to generate that certificate?

Comment 16 Nikos Mavrogiannopoulos 2018-10-31 12:45:18 UTC
Note that openssl used to allow such incorrect certificates, and was able to generate them. However that has changed with openssl 1.1.0 too [0]. You'll need to regenerate that certificate correctly.


[0]. https://github.com/openssl/openssl/pull/2668

Comment 17 Yaniv Kaul 2019-04-18 10:03:20 UTC
Status?

Comment 18 Sahina Bose 2019-04-22 06:28:08 UTC
RHGS-C is no longer under active maintenance. Customers have been advised to move to RHGS Web Administration Console to manage gluster deployments. Hence, closing this bug