Bug 1145429 (CVE-2014-1568) - CVE-2014-1568 nss: RSA PKCS#1 signature verification forgery flaw (MFSA 2014-73)
Summary: CVE-2014-1568 nss: RSA PKCS#1 signature verification forgery flaw (MFSA 2014-73)
Keywords:
Status: CLOSED ERRATA
Alias: CVE-2014-1568
Product: Security Response
Classification: Other
Component: vulnerability
Version: unspecified
Hardware: All
OS: Linux
high
high
Target Milestone: ---
Assignee: Red Hat Product Security
QA Contact:
URL:
Whiteboard:
Depends On: 1145430 1145431 1145432 1145433 1145434 1145494 1145496 1145497 1145498 1145499 1146309
Blocks: 1145435
TreeView+ depends on / blocked
 
Reported: 2014-09-23 04:27 UTC by Huzaifa S. Sidhpurwala
Modified: 2019-09-29 13:22 UTC (History)
10 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
A flaw was found in the way NSS parsed ASN.1 (Abstract Syntax Notation One) input from certain RSA signatures. A remote attacker could use this flaw to forge RSA certificates by providing a specially crafted signature to an application using NSS.
Clone Of:
Environment:
Last Closed: 2014-10-10 07:58:19 UTC


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2014:1307 normal SHIPPED_LIVE Important: nss security update 2014-09-26 08:57:29 UTC
Red Hat Product Errata RHSA-2014:1354 normal SHIPPED_LIVE Critical: rhev-hypervisor6 security update 2014-10-02 22:40:05 UTC
Red Hat Product Errata RHSA-2014:1371 normal SHIPPED_LIVE Important: nss security update 2014-10-10 07:37:19 UTC

Description Huzaifa S. Sidhpurwala 2014-09-23 04:27:25 UTC
Antoine Delignat-Lavaud, security researcher at Inria Paris in team Prosecco, reported an issue in Network Security Services (NSS) libraries affecting all versions. He discovered that NSS is vulnerable to a variant of a signature forgery attack previously published by Daniel Bleichenbacher. This is due to lenient parsing of ASN.1 values involved in a signature and could lead to the forging of RSA certificates.

The Advanced Threat Research team at Intel Security also independently discovered and reported this issue.

Projects using NSS 3.17 should update the new 3.17.1 release.


External Reference:

http://www.mozilla.org/security/announce/2014/mfsa2014-73.html


Acknowledgements:

Red Hat would like to thank the Mozilla project for reporting this issue. Upstream acknowledges Antoine Delignat-Lavaud and Intel Product Security Incident Response Team as the original reporters.

Comment 4 Huzaifa S. Sidhpurwala 2014-09-25 01:22:00 UTC
Public via:

https://www.mozilla.org/security/announce/2014/mfsa2014-73.html

Comment 5 Huzaifa S. Sidhpurwala 2014-09-25 01:24:42 UTC
Created nss tracking bugs for this issue:

Affects: fedora-all [bug 1146309]

Comment 6 Martin Prpič 2014-09-25 09:55:35 UTC
IssueDescription:

A flaw was found in the way NSS parsed ASN.1 (Abstract Syntax Notation One) input from certain RSA signatures. A remote attacker could use this flaw to forge RSA certificates by providing a specially crafted signature to an application using NSS.

Comment 7 errata-xmlrpc 2014-09-26 04:57:51 UTC
This issue has been addressed in the following products:

  Red Hat Enterprise Linux 7
  Red Hat Enterprise Linux 6
  Red Hat Enterprise Linux 5

Via RHSA-2014:1307 https://rhn.redhat.com/errata/RHSA-2014-1307.html

Comment 8 David Anderson 2014-09-27 17:57:33 UTC
When I installed this update (via the CentOS 6 repackaging), my server (Digital Ocean, New York datacentre) failed to connect via https to paypal.com.

This was in WooCommerce.

The WooCommerce PayPal logs indicated that SSL certificate validation was failing. This happened every time, over a period of 12 hours.

Theoretically this could indicate that someone was MITM-ing the connection, using this vulnerability. But that seems unlikely - there must be rather a lot of Digital Ocean servers in New York that want to connect to paypal.com. Is there any other possibility of how this update could have caused a problem?

In fact, the server had the same problem (certificate validation) connecting to stripe.com via HTTPS also, during that time.

When I rolled back the update, everything then worked again.

Comment 9 EJ Vincent 2014-09-28 01:09:23 UTC
+1 David.

RHEL6 nss packages breaks Authorize.net / PayPal transactions. Apache service restart is required to clear the issue (or a roll back).

Users who are opted in to the automatic patching are getting a rude awakening on this one. I've been recommending adding nss to yum exclusions or opting out of automatic patching in favour of manual updates.

Comment 10 Huzaifa S. Sidhpurwala 2014-09-29 08:22:21 UTC
(In reply to EJ Vincent from comment #9)
> +1 David.
> 
> RHEL6 nss packages breaks Authorize.net / PayPal transactions. Apache
> service restart is required to clear the issue (or a roll back).
> 
> Users who are opted in to the automatic patching are getting a rude
> awakening on this one. I've been recommending adding nss to yum exclusions
> or opting out of automatic patching in favour of manual updates.

Can you give me more information about the kind of setup you have (which breaks).

Comment 11 David Anderson 2014-09-29 08:23:49 UTC
I am just running a LAMP stack, using the CentOS repository packages, including mod_php and xcache. CentOS 6, all packages up to date via automatic nightly yum (except nss currently locked on previous version).

Comment 12 Huzaifa S. Sidhpurwala 2014-09-29 10:47:18 UTC
(In reply to David Anderson from comment #11)
> I am just running a LAMP stack, using the CentOS repository packages,
> including mod_php and xcache. CentOS 6, all packages up to date via
> automatic nightly yum (except nss currently locked on previous version).

I am assuming WooCommerce is using nss and it fails to connect to paypal website after the nss upgrade?

Comment 13 Kai Engert (:kaie) (inactive account) 2014-09-29 10:54:37 UTC
I wonder at which point NSS is involved in your configuration.
Do you run mod_nss?

The connection that are breaking, do you connect to the main domains, or to a subdomain host (e.g. "some-subdomain.paypal.com")?

Did you get any entries in server logfiles that explain about the kind of failure, that might helpus?

In order to reproduce, I need to connect myself to one of the affected servers. (I wonder if it's the server certificates that are failing, or maybe something with the server-key-exchange of the servers you're trying to connect to.)

Comment 14 David Anderson 2014-09-29 11:22:13 UTC
Hi,

WooCommerce is a WordPress plugin, and makes its http calls via the wp_remote_post() PHP function, which ultimately uses the PHP curl module.

When an order is made via PayPal checkout, the transaction happens on the PayPal site. PayPal then call back via ordinary HTTP to the website via a configured address. The website then makes an outgoing https call to PayPal, to verify the legitimacy of the callback and the status of the transaction. In WooCommerce, this is done via wp_remote_post(), with an https://www.paypal.com URL as the parameter.

WooCommerce was logging all the attempts to contact www.paypal.com via https from the time that the new nss was installed as failures due to an inability to validate www.paypal.com's SSL certificate.

PayPal allows these callback events to be replayed arbitrarily. I tried a few, and they all resulted in the same failure. I then checked my yum logs, downgraded the nss-* set of packages (the only thing to have changed), replayed the callbacks, and they all worked.

For the calls to stripe.com, this works more traditionally. The user types in their credit card number on my website (over an https connection to my website - and these were working fine, of course - I wouldn't have received any orders at all if it wasn't), and then the WooCommerce Stripe module makes POST requests to www.stripe.com to use its API. These also all failed - but there is no logging for these. I assume it was for the same reason. After downgrading nss-*, I contacted customers who had placed orders and tried to pay via Stripe, and those that came back and re-tried succeeded.

Nothing else was changed. Apache was not restarted.

Comment 15 David Anderson 2014-09-29 11:24:36 UTC
"Did you get any entries in server logfiles that explain about the kind of failure, that might help us?"

The WooCommerce PayPal log stated that validation of the SSL certificate for PayPal was failing.

Comment 16 Kai Engert (:kaie) (inactive account) 2014-09-29 12:03:49 UTC
David, thank you for the details.

Based on your description, we might be able to reproduce the problem with a simple command like:
  curl --head https://www.paypal.com
while having the new NSS RPM packages installed, to see the failure.

I will see if I can reproduce the problem that way.

Comment 17 Kai Engert (:kaie) (inactive account) 2014-09-29 12:29:13 UTC
I cannot reproduce the error yet.
I'm testing with a Centos 6.5 VM, fully updated.

# rpm -qa nss\* nspr\* curl\* libcurl\* | sort
curl-7.19.7-37.el6_5.3.x86_64
libcurl-7.19.7-37.el6_5.3.x86_64
nspr-4.10.6-1.el6_5.x86_64
nss-3.16.1-7.el6_5.x86_64
nss_compat_ossl-0.9.6-1.el6.x86_64
nss-softokn-3.14.3-12.el6_5.x86_64
nss-softokn-freebl-3.14.3-12.el6_5.x86_64
nss-sysinit-3.16.1-7.el6_5.x86_64
nss-tools-3.16.1-7.el6_5.x86_64
nss-util-3.16.1-2.el6_5.x86_64

Maybe you are seeing a different set of server certificates from your position in the network?

I assume you aren't using authentication based on a client certificate, but are using simple unauthenticated https?

Could you please run this command on your server, and attach the log.txt to this bug? You can run it from your live system. It will simply execute a test connection to paypal, dump some connection results, and will disconnect immediately. It's fine to run it using the older NSS packages. It will tell us the certificates that your server sees when it makes a direct connection to the paypal server.

openssl s_client -showcerts -connect www.paypal.com:443 \
   < /dev/null > /tmp/log.txt 2>&1


In addition, if you have a secondary test system (non-production) where you could temporarily install the latest NSS packages again, I would be interested in the output of
  curl --head https://www.paypal.com
(you should get a failure, if curl+nss is the culprit)

Comment 18 Kai Engert (:kaie) (inactive account) 2014-09-29 12:57:31 UTC
I installed php, and ran an interactive test.

I have a test.php file that contains:

<?php
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, "https://www.paypal.com/");
curl_setopt($ch, CURLOPT_HEADER, 1);
curl_setopt($ch, CURLOPT_NOBODY, 1);
curl_exec($ch);
curl_close($ch);
?>

I ran it using:
  php -f test.php

I received a valid http response from the paypal server.

Either you are getting different certificates from the server (which I hope you will be able to provide), or there must be a non-obvious detail inside your interaction with the server that causes the issue.

The curl and php software used in a simple way seem to work correctly.

Comment 19 Kai Engert (:kaie) (inactive account) 2014-09-29 13:55:57 UTC
David, on Twitter, I found this statement from Stripe:
https://twitter.com/search?q=nss%20centos&src=typd

"We've seen reports that a recent CentOS update for NSS is affecting SSL-enabled PHP libraries for processes started before the update. While the Stripe API is unaffected, PHP sites on CentOS may see errors with libraries like stripe-php. Restarting Apache fixes the issue."

Based on this message:

You might experience the issue, because you haven't stopped your processes. You should restart them.

This update consists of multiple libraries which depend on each other.

For example, if the application you're running continues to have one of the old libraries loaded, but during operation, your application dynamically loads one of the newer libraries from disk, you will have an incompatibility between the libraries, which can cause the application to malfunction.

You should stop your applications prior to the upgrade, and restart them after the upgrade is done.

Comment 20 Kai Engert (:kaie) (inactive account) 2014-09-29 13:56:28 UTC
Correct link to the Twitter message:
https://twitter.com/stripestatus/status/516026559669358592

Comment 21 Tomas Hoger 2014-09-29 14:08:36 UTC
Some additional flaw details in Adam Langley's blog post:

https://www.imperialviolet.org/2014/09/26/pkcs1.html

Comment 22 David Anderson 2014-09-29 14:26:10 UTC
Hi Kai,

It's true that I didn't restart Apache after applying the update - but that's because the update was via yum automatic updates. It was not a manually applied update.

Shouldn't scripts in the update automatically restart any relevant services that are likely to not function correctly without a restart, rather than waiting for manual intervention?

If I re-apply the update, I'd be likely to restart Apache immediately - the machine is processing live orders, and I'd rather just remain in ignorance as to the cause of the defect, rather than have downtime on a live store. I realise this doesn't help you at all, for which I am sorry.

David

Comment 23 Huzaifa S. Sidhpurwala 2014-09-29 14:49:17 UTC
(In reply to David Anderson from comment #22)
> Hi Kai,
> 
> It's true that I didn't restart Apache after applying the update - but
> that's because the update was via yum automatic updates. It was not a
> manually applied update.
> 
> Shouldn't scripts in the update automatically restart any relevant services
> that are likely to not function correctly without a restart, rather than
> waiting for manual intervention?
> 
They dont, because of multiple reasons, including the fact that the admin may want to do it himself and its not easy to figure out what all services are linked against the libray.

> If I re-apply the update, I'd be likely to restart Apache immediately - the
> machine is processing live orders, and I'd rather just remain in ignorance
> as to the cause of the defect, rather than have downtime on a live store. I
> realise this doesn't help you at all, for which I am sorry.
>

btw if you see the Red Hat advisory, it actually mentions that you need to restart your service.

"After installing this update, applications using NSS must be restarted for this update to take effect. "

https://rhn.redhat.com/errata/RHSA-2014-1307.html

> David

Comment 24 David Anderson 2014-09-29 14:54:27 UTC
Well, it seems the problem is now well-understood, and I know what to do; so, I have nothing more to add after this comment.

It seems far from optimal to have automatic updates that break things, and no solution without manual intervention.

"After installing this update, applications using NSS must be restarted for this update to take effect. "

"to take effect" is one thing... "to make your running servers start working properly again" is a bit different!

Comment 25 Sven Kieske 2014-09-29 15:20:00 UTC
(In reply to David Anderson from comment #24)
> Well, it seems the problem is now well-understood, and I know what to do;
> so, I have nothing more to add after this comment.
> 
> It seems far from optimal to have automatic updates that break things, and
> no solution without manual intervention.
> 
> "After installing this update, applications using NSS must be restarted for
> this update to take effect. "
> 
> "to take effect" is one thing... "to make your running servers start working
> properly again" is a bit different!

1. It is not recommended by redhat to run automatic updates at all
2. if you decide to break this recommendation you should do this in a staging/testing environment first and monitor it for breakage
3. following step 2 you could adjust your deployment process in production

sorry for all this off topic discussion

Comment 26 David Anderson 2014-09-29 15:22:42 UTC
"1. It is not recommended by redhat to run automatic updates at all"

Thanks. Please can you point me to a URL with that recommendation?

Comment 27 errata-xmlrpc 2014-10-02 18:43:52 UTC
This issue has been addressed in the following products:

  RHEV Manager version 3.4

Via RHSA-2014:1354 https://rhn.redhat.com/errata/RHSA-2014-1354.html

Comment 28 EJ Vincent 2014-10-02 21:01:11 UTC
Hi Sven,

Could you please point to Red Hat documentation which recommends against leveraging automatic updates?

---

The update(s) to the nss packages impacted _active_ applications.

We're left with a conundrum: an update was pushed to servers, which broke applications relying on third-party payment processors, yet Red Hat does not make it a habit to restart services, such as Apache, following an update. To be fair, that's expected.

As a result, applications remained in a broken state with very little visibility. The damage was already done, at which point customers start to take notice and wonder what's up.

Generally speaking, customers who use automatic updates don't make it a habit to review CVE changelogs or Red Hat errata (not feigning ignorance). I'd hate to recommend adding nss packages to the yum exclusion list.

Am I making any sense?

Thanks for all that you do!

Comment 29 Ronald 2014-10-03 04:44:12 UTC
We experienced the same situation with RHEL6 after applying the nss update with PayPal, with Virtual Merchant (now Converge, by Elavon) and with First Data.  All of these were using php/curl.  Another gateway (CyberSource) using their proprietary SOAP client was not affected. Some notes which differ from the experiences described above:

- nss install was manual after prompted by Webmin and Apache was not restarted - all transactions continued to process normally (over 12 hours) until Apache was restarted by automatic log rotation process with Apache restart.  Transactions for all three gateways failed until system reboot was performed. (Another Apache restart was not attempted, since that was the trigger for the failure.)

- regarding the sub-domain question, of our gateways, Virtual Merchant uses a www. sub-domain, First Data and PayPal use sub-domains other than www.

- other nss updates had been applied in the same manner without impact to these gateways (recently 2014-07-25, 2014-02-14, 2013-12-23, 2013-12-13)

Thought I would share our experience.

Thank you!

Comment 30 errata-xmlrpc 2014-10-10 03:37:42 UTC
This issue has been addressed in the following products:

  Red Hat Enterprise Linux 5.9 EUS - Server Only
  Red Hat Enterprise Linux 6.4 EUS - Server and Compute Node Only
  Red Hat Enterprise Linux 6.2 AUS
  Red Hat Enterprise Linux 5.6 Long Life
  Red Hat Enterprise Linux 4 Extended Lifecycle Support

Via RHSA-2014:1371 https://rhn.redhat.com/errata/RHSA-2014-1371.html


Note You need to log in before you can comment on or make changes to this bug.