Michael suggested the following as a potential idea of how to fix:


I've changed my mind on how to fix. I think the easiest fix would be to add an option to pin a cert using CURLOPT_CAINFO (which takes a filename). This way the cert is pinned - even a real CA can't issue a spoofed certificate.

RHEV/vdsm would need be modified to support this - presumably by passing the certificate data and writing it to a tempfile, which would be passed to the fence agent.

When the cert is supplied, SSL_VERIFY_HOST can be turned off, otherwise the default CA bundle should be used with SSL_VERIFY_HOST on.

For those not wanting to pin a cert, they can enable the non-SSL webserver in UCS Manager for equivalent security.



(Personally I think that could work, but also an option of --insecure-ssl or something to disable those checks and default to them being on would work just as well).

The commit that introduced the problem:

https://git.fedorahosted.org/cgit/fence-agents.git/commit/?id=1ef8f0605ef021a7f0045d9be19497ef95a2a16e

Or made the problem worse anyways.  We should have the cURL defaults set and override them to be insecure only when asked.

http://curl.haxx.se/libcurl/c/curl_easy_setopt.html#CURLOPTSSLVERIFYPEER
http://curl.haxx.se/libcurl/c/curl_easy_setopt.html#CURLOPTSSLVERIFYHOST

So this:

fence-agents-4.0.2/fence/agents/cisco_ucs/fence_cisco_ucs.py:   c.setopt(pycurl.SSL_VERIFYPEER, 0)
fence-agents-4.0.2/fence/agents/cisco_ucs/fence_cisco_ucs.py:   c.setopt(pycurl.SSL_VERIFYHOST, 0)

should probably be:

c.setopt(pycurl.SSL_VERIFYPEER, 1)
c.setopt(pycurl.SSL_VERIFYHOST, 2)

and then they should be dropped to 0 only when specifically asked, either via a commandline arg or a configuration option.

This issue has been assigned CVE-2014-0104.

Acknowledgements:

Red Hat would like to thank Michael Samuel for reporting this issue.

(In reply to Vincent Danen from comment #1)
> I've changed my mind on how to fix. I think the easiest fix would be to add
> an option to pin a cert using CURLOPT_CAINFO (which takes a filename). This
> way the cert is pinned - even a real CA can't issue a spoofed certificate.

Assuming the above refers to passing server certificate as CURLOPT_CAINFO rather than CA certificate, this would not work in general either.  Verification fails if such certificate is not self signed, or if it is self signed with CA:FALSE basic constraint extension.  (That's for curl + nss, may differ when curl uses different crypto backend.)

I agree that this can be a possible security problem but we are not able to change that behaviour as it will make a regression. Running clusters will stop working. We can:

* add a new option --secure-ssl - for RHEL5/RHEL6 (insecure as default)
* add a new option --insecure-ssl - for RHEL7 (secure as default)

IMHO the problem can be avoided when users follow documentation:
"It is highly recommended to separate the private network for cluster heartbeat from the application network."
On such network a MITM attacks are rare. 

The other problem is that the majority of the fence devices accessible via https use only self-signed certificates.

I agree with the above.  While I would prefer to see an --insecure-ssl option for rhel5/6 instead of a --secure-ssl one (secure by default) I can see this being disruptive to many customers who are expecting and using it this one way.  I would however note the problems via documentation.

Unless someone really disagrees with that, I think that is entirely reasonable.

(In reply to Marek Grac from comment #6)
> * add a new option --secure-ssl - for RHEL5/RHEL6 (insecure as default)
> * add a new option --insecure-ssl - for RHEL7 (secure as default)

Would it make sense to add both to all versions, and alias --ssl to one of them?  That can probably make transitions easier (e.g. users can change their configuration they use on RHEL-6 to use e.g. --insecure-ssl so the same configuration work with RHEL-7 too).

Option to specify CA certificate(s) probably makes sense too, so that updating system ca-bundle is not the only way to make secure ssl work with custom CA.

(In reply to Tomas Hoger from comment #10)
> (In reply to Marek Grac from comment #6)
> > * add a new option --secure-ssl - for RHEL5/RHEL6 (insecure as default)
> > * add a new option --insecure-ssl - for RHEL7 (secure as default)
> 
> Would it make sense to add both to all versions, and alias --ssl to one of
> them?  That can probably make transitions easier (e.g. users can change
> their configuration they use on RHEL-6 to use e.g. --insecure-ssl so the
> same configuration work with RHEL-7 too).
> 
> Option to specify CA certificate(s) probably makes sense too, so that
> updating system ca-bundle is not the only way to make secure ssl work with
> custom CA.

Yes it sounds a very sensible approach to me.

Marek?

If we want to fix that issue we will have to handle all fence agents which work over HTTPS. There are four of them according to https://access.redhat.com/site/node/28603

* fence_ilo for HP iLO1/iLO2 - based on gnutls-cli
* fence_rhevm for RHEV-M - based on pycurl (REST API)
* fence_cisco_ucs for Cisco UCS - based on pycurl (REST API)
* fence_vmware_soap for VMWare vSphere - based on suds (SOAP API)

Adding support for --secure-ssl:
* gnutls-cli backend: different command-line options 
* pycurl backend: described above and on http://curl.haxx.se/libcurl/c/curl_easy_setopt.htm
* suds backend: does not have direct support but there is a possible work-around described at http://stackoverflow.com/questions/6277027/suds-over-https-with-cert

Option to specify a CA certificate is not a problem. I agree that we can do --ssl as an synonym for --(in)secure-ssl according to RHEL version.

The main problem that I can see is testing. We will need all mentioned machines with valid / valid (custom CA) / invalid (self-signed) certificates. There are quite a lot ilo2 machines, RHEV-M and VMWare should be easy to access, the only one which is hard to obtain is Cisco UCS. I had access to one in 2010, so I expect that it is still somewhere in Red Hat [contact person was Chris Wright]

Created attachment 870423 [details]
Proposed patch for --ssl-secure and --ssl-insecure

Patch that adds a support for --ssl-secure and --ssl-insecure.

It does not change situation for fence_vmware_soap because I found out that solutions are based on packages which are not available in rhel7. It is possible to do a little hack (until next minor release). We can check validity of certificate using pycurl (as in rhevm, cisco_ucs) and then use SOAP library as before. It means that verification will not be done in step but it will be done.

Created attachment 870617 [details]
New patch to fix logic for fence_cisco_ucs and fence_vmware_soap

The logic in setting was incorrect as --ssl-insecure would automatically imply --ssl and SSL_VERIFY* was never set to 0.

New patch defaults to --ssl and/or --ssl-secure, and allows --ssl-insecure to override the settings.

(In reply to Marek Grac from comment #13)
> Created attachment 870423 [details]
> Proposed patch for --ssl-secure and --ssl-insecure
> 
> Patch that adds a support for --ssl-secure and --ssl-insecure.
> 
> It does not change situation for fence_vmware_soap because I found out that
> solutions are based on packages which are not available in rhel7. It is
> possible to do a little hack (until next minor release). We can check
> validity of certificate using pycurl (as in rhevm, cisco_ucs) and then use
> SOAP library as before. It means that verification will not be done in step
> but it will be done.

This won't work - the MITM attacker would just wait for the second connection.  I suggest fixing the SOAP library.

(In reply to Tomas Hoger from comment #5)
> Assuming the above refers to passing server certificate as CURLOPT_CAINFO
> rather than CA certificate, this would not work in general either. 
> Verification fails if such certificate is not self signed, or if it is self
> signed with CA:FALSE basic constraint extension.  (That's for curl + nss,
> may differ when curl uses different crypto backend.)

In the case of Cisco UCS, the default self-signed certificate doesn't contain
any basic constraints.

It also allows generating certificate requests for use with a CA.

(In reply to Michael Samuel from comment #20)
> (In reply to Tomas Hoger from comment #5)
> > Assuming the above refers to passing server certificate as CURLOPT_CAINFO
> > rather than CA certificate, this would not work in general either. 
> > Verification fails if such certificate is not self signed, or if it is self
> > signed with CA:FALSE basic constraint extension.  (That's for curl + nss,
> > may differ when curl uses different crypto backend.)
> 
> In the case of Cisco UCS, the default self-signed certificate doesn't contain
> any basic constraints.
> 
> It also allows generating certificate requests for use with a CA.

If you have access to a Cisco UCS in the house, we would very much like to have temporary access and test with a proper certificate. The one I used to unit testing in the fence-agents bug has only the self-signed certificate and we are not allowed to change it.

Created attachment 870924 [details]
Proposed patch for --ssl-secure and --ssl-insecure (including vmware soap fix)

Fabio's patch + fix for fence_vmware_soap (change of underlying transport library)

I discussed access with Fabio outside of BZ - it is no longer required.

Can we please unembargo this?

Statement:

This issue did not affect the version of fence-agent as shipped with Red Hat Enterprise Linux 7.  It does affect the versions of fence-agents as shipped with Red Hat Enterprise Linux 6. Red Hat Product Security has rated this issue as having Moderate security impact. A future update may address this issue. For additional information, refer to the Issue Severity Classification: https://access.redhat.com/security/updates/classification/.

Created fence-agents tracking bugs for this issue:

Affects: fedora-all [bug 1151515]

This bug is now closed. Further updates for individual products will be reflected on the CVE page(s):

https://access.redhat.com/security/cve/cve-2014-0104