Bug 1198554

Summary: ilo2 firmware +2.27 does not work with fence_ilo2
Product: Red Hat Enterprise Linux 6 Reporter: Gerwin Krist <gerwinkrist>
Component: fence-agentsAssignee: Marek Grac <mgrac>
Status: CLOSED ERRATA QA Contact: cluster-qe <cluster-qe>
Severity: high Docs Contact:
Priority: high    
Version: 6.8CC: cluster-maint, ctatman, gerwinkrist, james.hofmeister, jherrman, jruemker, mjuricek, rbalakri
Target Milestone: rcKeywords: ZStream
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: fence-agents-4.0.15-8.el6 Doc Type: Bug Fix
Doc Text:
Prior to this update, when using an Integrated Lights-Out 2 (iLO2) device with firmware version 2.27, iLO2 negotiation of the TLS protocol did not work properly. Consequently, the user could not connect or login to the fencing device. With this update, the "--tls1.0" option has been added, and the user can now set the aforementioned protocol without negotiation.
Story Points: ---
Clone Of:
: 1199970 1208751 1214411 (view as bug list) Environment:
Last Closed: 2015-07-22 06:58:50 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1199966    
Bug Blocks: 1199970, 1199971, 1208751, 1214411    
Attachments:
Description Flags
Proposed fence agent
none
Proposed patch to fence_ilo none

Description Gerwin Krist 2015-03-04 11:11:55 UTC
Description of problem:
Since we have updated our blades to ilo v2.27 (jan 2015) the always working fence_ilo2 tool cannot login anymore on ilo2. For our RHEV environment this is mandatory.


Version-Release number of selected component (if applicable):
3.1.5-48

How reproducible:

Steps to Reproduce:
1. update the ilo2 firmware to 2.27 and then try:
2. /usr/sbin/fence_ilo2 -o status -a X.X.X.X -l <user> -p <password> -v


Actual results:
Says "Unable to connect/login to fencing device" even with -v it does not show more 

Expected results:
XML output and the power state


Additional info:
My guess is that fence_ilo2 is trying to use SSLv3. Which seems to be disabled in ilo v2.27. Our work-around is to enable sslv3 at the ilo2 interface with:
set /map1/config1 oemhp_ssl_v3_enable=yes 
After the ilo2 controller is reset it works again!

Comment 2 Marek Grac 2015-03-04 14:04:08 UTC
Created attachment 997855 [details]
Proposed fence agent

@Gerwin:

can you try to run fence agent (with current library) on your machine. It should be enough to use new option --notls which should solve this problem.

Comment 3 Gerwin Krist 2015-03-04 14:45:12 UTC
Hi Marek,

Does not work here:

./fence_ilo -o status -a 172.19.1.40 -l XXXXX -p XXXXXXX --notls -v
Running command: /usr/bin/gnutls-cli --priority "NORMAL:-VERS-TLS1.2:-VERS-TLS1.1:-VERS-TLS1.0:+VERS-SSL3.0" --insecure --crlf -p 443 172.19.1.40
Sent: <?xml version="1.0"?>

Unable to connect/login to fencing device

btw fence_ilo is same as fence_ilo2?

Comment 4 Marek Grac 2015-03-04 15:02:51 UTC
Hi,

can you take a look what will happend when you run that gnutls command directly? possibly with --verbose, so we can see where the problem is

Comment 5 Gerwin Krist 2015-03-04 15:34:58 UTC
output ilo2 2.27 - with oemhp_ssl_v3_enable=no (default in 2.27!)
=================================================================
openssl s_client -connect  x.x.x.x:443 -no_tls1_2 -no_tls1_1 -no_tls1:
SSL handshake has read 0 bytes and written 124 bytes
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NON

output ilo2 2.27 - with oemhp_ssl_v3_enable=yes
================================================
openssl s_client -connect  x.x.x.x:443 -no_tls1_2 -no_tls1_1 -no_tls1:

SSL handshake has read 1566 bytes and written 338 bytes
---
New, TLSv1/SSLv3, Cipher is DHE-RSA-AES256-SHA
Server public key is 2048 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol  : SSLv3
    Cipher    : DHE-RSA-AES256-SHA
    Session-ID: 03F64F841BC33C593C4016866E59209389762EFD5701168EC57CA23C9E46E90E
    Session-ID-ctx:
    Master-Key: 05BC5524F3C97CD7EDDCB71A691C734301702F0DA1F5D86714EC8A9C5E660450D9327673F667DFD4ACEC998532BF783C
    Key-Arg   : None
    Krb5 Principal: None
    PSK identity: None
    PSK identity hint: None
    Start Time: 1425483129
    Timeout   : 300 (sec)
    Verify return code: 21 (unable to verify the first certificate)

Comment 6 Gerwin Krist 2015-03-04 15:36:10 UTC
Resolving '172.19.1.40'...
Connecting to '172.19.1.40:443'...
*** Fatal error: A TLS packet with unexpected length was received.
*** Handshake has failed
GNUTLS ERROR: A TLS packet with unexpected length was received.

Comment 7 Marek Grac 2015-03-04 16:00:35 UTC
Thanks, I will try to obtain a machine with that firmware and fix the issue.

Comment 8 Gerwin Krist 2015-03-05 08:37:27 UTC
If you have problems finding one. I can can help you with that :-) Just let me know.

Comment 9 Marek Grac 2015-03-05 08:39:00 UTC
@Gerwin: that will be great; I hope that I will need it just for a while.

Comment 10 Marek Grac 2015-03-06 14:27:06 UTC
@Gerwin:

thanks a lot, you can close connection.

as official fix will follow later, hotfix is

change line with 'gnutls_opts = ' in fencing.py to

gnutls_opts = "--priority \"NORMAL:-VERS-TLS1.2:-VERS-TLS1.1:+VERS-TLS1.0:%LATEST_RECORD_VERSION\""

that way, if you will use --notls, it should work as expected. 

Problem is that HP iLO2 supports only TLS1.0

Comment 12 Marek Grac 2015-03-09 12:07:18 UTC
In gnutls-cli (v2.8) there is not a required option. Bug against gnutls was created and if it will be approved, we will fix this issue.

Comment 13 John Ruemker 2015-03-13 20:19:46 UTC
Raising priority/severity based on recent customer case.

Comment 14 Marek Grac 2015-03-20 18:55:42 UTC
We have fix that does not need change in gnutls.

Comment 20 Marek Grac 2015-04-01 10:21:04 UTC
Created attachment 1009609 [details]
Proposed patch to fence_ilo

@John:

Proposed patch to add gnutls-cli support. In order to not break anything, fence agent will:

Try to connect using nss_wrapper as before and only if this fail, we will fallback to gnutls-cli.

This way, we can be sure that existing functionality will not be broken. Patch is quite simple, so code review by someone else will be fast.

Comment 21 John Ruemker 2015-04-01 13:37:43 UTC
Great, I'll see if the customer can test this.  

Thanks

Comment 23 Gerwin Krist 2015-04-01 14:20:20 UTC
@marek
Do you have the complete patched script? I will test it here then.

Comment 24 John Ruemker 2015-04-02 21:58:11 UTC
One small issue, one with your test build that's not in your patch posted at https://bugzilla.redhat.com/attachment.cgi?id=1009609&action=diff.  path specified was /usr/bin/gnutls, not gnutls-cli.  In RHEL 7 its provided by autoconf so we didn't have to worry about it, so I think the manual entry just led to a typo here. I built a new package with the correct path and tested it against your ilo and it works (status only, at least).  Customer is confirming in their environment, and we'll report back. 

I'm going to go ahead and start the ball rolling on the z-stream request now.

Justification: this issue requires customers that rely on iLO devices to maintain a widely-publicized security hole on these devices for the sake of compatibility with our fence agent.  Releasing a fix for this will allow our product to work with the iLO firmware version that closes that security hole, and so it is not only important for this customer, but we will surely see a number of others looking for this in 6.6.z in the months to come.

Comment 29 errata-xmlrpc 2015-07-22 06:58:50 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2015-1350.html