Antoine Delignat-Lavaud, security researcher at Inria Paris in team Prosecco, reported an issue in Network Security Services (NSS) libraries affecting all versions. He discovered that NSS is vulnerable to a variant of a signature forgery attack previously published by Daniel Bleichenbacher. This is due to lenient parsing of ASN.1 values involved in a signature and could lead to the forging of RSA certificates. The Advanced Threat Research team at Intel Security also independently discovered and reported this issue. Projects using NSS 3.17 should update the new 3.17.1 release. External Reference: http://www.mozilla.org/security/announce/2014/mfsa2014-73.html Acknowledgements: Red Hat would like to thank the Mozilla project for reporting this issue. Upstream acknowledges Antoine Delignat-Lavaud and Intel Product Security Incident Response Team as the original reporters.
Public via: https://www.mozilla.org/security/announce/2014/mfsa2014-73.html
Created nss tracking bugs for this issue: Affects: fedora-all [bug 1146309]
IssueDescription: A flaw was found in the way NSS parsed ASN.1 (Abstract Syntax Notation One) input from certain RSA signatures. A remote attacker could use this flaw to forge RSA certificates by providing a specially crafted signature to an application using NSS.
This issue has been addressed in the following products: Red Hat Enterprise Linux 7 Red Hat Enterprise Linux 6 Red Hat Enterprise Linux 5 Via RHSA-2014:1307 https://rhn.redhat.com/errata/RHSA-2014-1307.html
When I installed this update (via the CentOS 6 repackaging), my server (Digital Ocean, New York datacentre) failed to connect via https to paypal.com. This was in WooCommerce. The WooCommerce PayPal logs indicated that SSL certificate validation was failing. This happened every time, over a period of 12 hours. Theoretically this could indicate that someone was MITM-ing the connection, using this vulnerability. But that seems unlikely - there must be rather a lot of Digital Ocean servers in New York that want to connect to paypal.com. Is there any other possibility of how this update could have caused a problem? In fact, the server had the same problem (certificate validation) connecting to stripe.com via HTTPS also, during that time. When I rolled back the update, everything then worked again.
+1 David. RHEL6 nss packages breaks Authorize.net / PayPal transactions. Apache service restart is required to clear the issue (or a roll back). Users who are opted in to the automatic patching are getting a rude awakening on this one. I've been recommending adding nss to yum exclusions or opting out of automatic patching in favour of manual updates.
(In reply to EJ Vincent from comment #9) > +1 David. > > RHEL6 nss packages breaks Authorize.net / PayPal transactions. Apache > service restart is required to clear the issue (or a roll back). > > Users who are opted in to the automatic patching are getting a rude > awakening on this one. I've been recommending adding nss to yum exclusions > or opting out of automatic patching in favour of manual updates. Can you give me more information about the kind of setup you have (which breaks).
I am just running a LAMP stack, using the CentOS repository packages, including mod_php and xcache. CentOS 6, all packages up to date via automatic nightly yum (except nss currently locked on previous version).
(In reply to David Anderson from comment #11) > I am just running a LAMP stack, using the CentOS repository packages, > including mod_php and xcache. CentOS 6, all packages up to date via > automatic nightly yum (except nss currently locked on previous version). I am assuming WooCommerce is using nss and it fails to connect to paypal website after the nss upgrade?
I wonder at which point NSS is involved in your configuration. Do you run mod_nss? The connection that are breaking, do you connect to the main domains, or to a subdomain host (e.g. "some-subdomain.paypal.com")? Did you get any entries in server logfiles that explain about the kind of failure, that might helpus? In order to reproduce, I need to connect myself to one of the affected servers. (I wonder if it's the server certificates that are failing, or maybe something with the server-key-exchange of the servers you're trying to connect to.)
Hi, WooCommerce is a WordPress plugin, and makes its http calls via the wp_remote_post() PHP function, which ultimately uses the PHP curl module. When an order is made via PayPal checkout, the transaction happens on the PayPal site. PayPal then call back via ordinary HTTP to the website via a configured address. The website then makes an outgoing https call to PayPal, to verify the legitimacy of the callback and the status of the transaction. In WooCommerce, this is done via wp_remote_post(), with an https://www.paypal.com URL as the parameter. WooCommerce was logging all the attempts to contact www.paypal.com via https from the time that the new nss was installed as failures due to an inability to validate www.paypal.com's SSL certificate. PayPal allows these callback events to be replayed arbitrarily. I tried a few, and they all resulted in the same failure. I then checked my yum logs, downgraded the nss-* set of packages (the only thing to have changed), replayed the callbacks, and they all worked. For the calls to stripe.com, this works more traditionally. The user types in their credit card number on my website (over an https connection to my website - and these were working fine, of course - I wouldn't have received any orders at all if it wasn't), and then the WooCommerce Stripe module makes POST requests to www.stripe.com to use its API. These also all failed - but there is no logging for these. I assume it was for the same reason. After downgrading nss-*, I contacted customers who had placed orders and tried to pay via Stripe, and those that came back and re-tried succeeded. Nothing else was changed. Apache was not restarted.
"Did you get any entries in server logfiles that explain about the kind of failure, that might help us?" The WooCommerce PayPal log stated that validation of the SSL certificate for PayPal was failing.
David, thank you for the details. Based on your description, we might be able to reproduce the problem with a simple command like: curl --head https://www.paypal.com while having the new NSS RPM packages installed, to see the failure. I will see if I can reproduce the problem that way.
I cannot reproduce the error yet. I'm testing with a Centos 6.5 VM, fully updated. # rpm -qa nss\* nspr\* curl\* libcurl\* | sort curl-7.19.7-37.el6_5.3.x86_64 libcurl-7.19.7-37.el6_5.3.x86_64 nspr-4.10.6-1.el6_5.x86_64 nss-3.16.1-7.el6_5.x86_64 nss_compat_ossl-0.9.6-1.el6.x86_64 nss-softokn-3.14.3-12.el6_5.x86_64 nss-softokn-freebl-3.14.3-12.el6_5.x86_64 nss-sysinit-3.16.1-7.el6_5.x86_64 nss-tools-3.16.1-7.el6_5.x86_64 nss-util-3.16.1-2.el6_5.x86_64 Maybe you are seeing a different set of server certificates from your position in the network? I assume you aren't using authentication based on a client certificate, but are using simple unauthenticated https? Could you please run this command on your server, and attach the log.txt to this bug? You can run it from your live system. It will simply execute a test connection to paypal, dump some connection results, and will disconnect immediately. It's fine to run it using the older NSS packages. It will tell us the certificates that your server sees when it makes a direct connection to the paypal server. openssl s_client -showcerts -connect www.paypal.com:443 \ < /dev/null > /tmp/log.txt 2>&1 In addition, if you have a secondary test system (non-production) where you could temporarily install the latest NSS packages again, I would be interested in the output of curl --head https://www.paypal.com (you should get a failure, if curl+nss is the culprit)
I installed php, and ran an interactive test. I have a test.php file that contains: <?php $ch = curl_init(); curl_setopt($ch, CURLOPT_URL, "https://www.paypal.com/"); curl_setopt($ch, CURLOPT_HEADER, 1); curl_setopt($ch, CURLOPT_NOBODY, 1); curl_exec($ch); curl_close($ch); ?> I ran it using: php -f test.php I received a valid http response from the paypal server. Either you are getting different certificates from the server (which I hope you will be able to provide), or there must be a non-obvious detail inside your interaction with the server that causes the issue. The curl and php software used in a simple way seem to work correctly.
David, on Twitter, I found this statement from Stripe: https://twitter.com/search?q=nss%20centos&src=typd "We've seen reports that a recent CentOS update for NSS is affecting SSL-enabled PHP libraries for processes started before the update. While the Stripe API is unaffected, PHP sites on CentOS may see errors with libraries like stripe-php. Restarting Apache fixes the issue." Based on this message: You might experience the issue, because you haven't stopped your processes. You should restart them. This update consists of multiple libraries which depend on each other. For example, if the application you're running continues to have one of the old libraries loaded, but during operation, your application dynamically loads one of the newer libraries from disk, you will have an incompatibility between the libraries, which can cause the application to malfunction. You should stop your applications prior to the upgrade, and restart them after the upgrade is done.
Correct link to the Twitter message: https://twitter.com/stripestatus/status/516026559669358592
Some additional flaw details in Adam Langley's blog post: https://www.imperialviolet.org/2014/09/26/pkcs1.html
Hi Kai, It's true that I didn't restart Apache after applying the update - but that's because the update was via yum automatic updates. It was not a manually applied update. Shouldn't scripts in the update automatically restart any relevant services that are likely to not function correctly without a restart, rather than waiting for manual intervention? If I re-apply the update, I'd be likely to restart Apache immediately - the machine is processing live orders, and I'd rather just remain in ignorance as to the cause of the defect, rather than have downtime on a live store. I realise this doesn't help you at all, for which I am sorry. David
(In reply to David Anderson from comment #22) > Hi Kai, > > It's true that I didn't restart Apache after applying the update - but > that's because the update was via yum automatic updates. It was not a > manually applied update. > > Shouldn't scripts in the update automatically restart any relevant services > that are likely to not function correctly without a restart, rather than > waiting for manual intervention? > They dont, because of multiple reasons, including the fact that the admin may want to do it himself and its not easy to figure out what all services are linked against the libray. > If I re-apply the update, I'd be likely to restart Apache immediately - the > machine is processing live orders, and I'd rather just remain in ignorance > as to the cause of the defect, rather than have downtime on a live store. I > realise this doesn't help you at all, for which I am sorry. > btw if you see the Red Hat advisory, it actually mentions that you need to restart your service. "After installing this update, applications using NSS must be restarted for this update to take effect. " https://rhn.redhat.com/errata/RHSA-2014-1307.html > David
Well, it seems the problem is now well-understood, and I know what to do; so, I have nothing more to add after this comment. It seems far from optimal to have automatic updates that break things, and no solution without manual intervention. "After installing this update, applications using NSS must be restarted for this update to take effect. " "to take effect" is one thing... "to make your running servers start working properly again" is a bit different!
(In reply to David Anderson from comment #24) > Well, it seems the problem is now well-understood, and I know what to do; > so, I have nothing more to add after this comment. > > It seems far from optimal to have automatic updates that break things, and > no solution without manual intervention. > > "After installing this update, applications using NSS must be restarted for > this update to take effect. " > > "to take effect" is one thing... "to make your running servers start working > properly again" is a bit different! 1. It is not recommended by redhat to run automatic updates at all 2. if you decide to break this recommendation you should do this in a staging/testing environment first and monitor it for breakage 3. following step 2 you could adjust your deployment process in production sorry for all this off topic discussion
"1. It is not recommended by redhat to run automatic updates at all" Thanks. Please can you point me to a URL with that recommendation?
This issue has been addressed in the following products: RHEV Manager version 3.4 Via RHSA-2014:1354 https://rhn.redhat.com/errata/RHSA-2014-1354.html
Hi Sven, Could you please point to Red Hat documentation which recommends against leveraging automatic updates? --- The update(s) to the nss packages impacted _active_ applications. We're left with a conundrum: an update was pushed to servers, which broke applications relying on third-party payment processors, yet Red Hat does not make it a habit to restart services, such as Apache, following an update. To be fair, that's expected. As a result, applications remained in a broken state with very little visibility. The damage was already done, at which point customers start to take notice and wonder what's up. Generally speaking, customers who use automatic updates don't make it a habit to review CVE changelogs or Red Hat errata (not feigning ignorance). I'd hate to recommend adding nss packages to the yum exclusion list. Am I making any sense? Thanks for all that you do!
We experienced the same situation with RHEL6 after applying the nss update with PayPal, with Virtual Merchant (now Converge, by Elavon) and with First Data. All of these were using php/curl. Another gateway (CyberSource) using their proprietary SOAP client was not affected. Some notes which differ from the experiences described above: - nss install was manual after prompted by Webmin and Apache was not restarted - all transactions continued to process normally (over 12 hours) until Apache was restarted by automatic log rotation process with Apache restart. Transactions for all three gateways failed until system reboot was performed. (Another Apache restart was not attempted, since that was the trigger for the failure.) - regarding the sub-domain question, of our gateways, Virtual Merchant uses a www. sub-domain, First Data and PayPal use sub-domains other than www. - other nss updates had been applied in the same manner without impact to these gateways (recently 2014-07-25, 2014-02-14, 2013-12-23, 2013-12-13) Thought I would share our experience. Thank you!
This issue has been addressed in the following products: Red Hat Enterprise Linux 5.9 EUS - Server Only Red Hat Enterprise Linux 6.4 EUS - Server and Compute Node Only Red Hat Enterprise Linux 6.2 AUS Red Hat Enterprise Linux 5.6 Long Life Red Hat Enterprise Linux 4 Extended Lifecycle Support Via RHSA-2014:1371 https://rhn.redhat.com/errata/RHSA-2014-1371.html
The following patches were added to nss-3.16.3 branch: https://hg.mozilla.org/projects/nss/rev/1abef7d49647 https://hg.mozilla.org/projects/nss/rev/c59048189e21 https://hg.mozilla.org/projects/nss/rev/364e66e7f083 https://hg.mozilla.org/projects/nss/rev/a8f155ba285e