Bug 668882

Summary:	Infinite recursion when encoding a NSS enveloped/digested data
Product:	Red Hat Enterprise Linux 6	Reporter:	Nalin Dahyabhai <nalin>
Component:	nss	Assignee:	Elio Maldonado Batiz <emaldona>
Status:	CLOSED ERRATA	QA Contact:	Aleš Mareček <amarecek>
Severity:	high	Docs Contact:
Priority:	high
Version:	6.0	CC:	amarecek, dpal, kengert, mitr, rrelyea
Target Milestone:	rc
Target Release:	6.2
Hardware:	All
OS:	Linux
Whiteboard:
Fixed In Version:	nss-3.12.10-2.el6	Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:	499444	Environment:
Last Closed:	2011-12-06 12:08:26 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	499444, 712958
Bug Blocks:	642407, 668887

Description Nalin Dahyabhai 2011-01-11 22:15:40 UTC

+++ This bug was initially created as a clone of Bug #499444 +++

Version:
nss-3.12.2.0-5.fc10.x86_64, and a locally rebuilt nss-3.23.3-3.

Attempting to encode a CMS message that contains enveloped (digested (data)) causes infinite recursion and stack overflow.

To reproduce, compile the attached file with
   gcc $(pkg-config --cflags --libs nss glib-2.0) repro.c
and run
   ./a.out a/path/to/a/certificate.pem > x
This will correctly create an enveloped message.

If you redefine DIGESTED in the file to "1" and recompile,
   ./a.out a/path/to/a/certificate.pem > x
will crash with a stack overflow.

AFAICS this is going on:
* When secasn1e.c finishes encoding the contentType field of NSSCMSEncryptedContentInfoTemplate, sec_asn1e_next_in_sequence() gets called.
* ...next_in_sequence does the "after" field notification before changing any ASN1 state.
* The "after" notification ends up in nss_cms_before_data(), which creates a completely new encoding context for the "child" digested_data, and starts encoding it.
* Encoding the "header" of digested_data (algorithm ID, nested raw data OID) requires more than 16 bytes, so nss_cms_encoder_work_data() will get the encrypted data and call SEC_ASN1EncoderUpdate() with it in the enveloped_data context.
* As noted above, the ASN1 context was not changed before calling the "after" field notification of contentType, so SEC_ASN1EncoderUpdate() again thinks the "contentType" field was just finished encoding, calls the "after" notification again, and the whole process (recursively) repeats.

See also #499440, a different bug related to the calling SEC_ASN1EncoderUpdate() from within a notifier called sec_asn1e_next_in_sequence ().

--- Additional comment from mitr on 2009-05-06 16:11:04 EDT ---

Created attachment 342711 [details]
Reproducer, see comment#0.

--- Additional comment from kengert on 2009-05-07 15:47:24 EDT ---

Library functionality bugs should get reported upstream.
I've forwarded your bug reported 1:1 at 
https://bugzilla.mozilla.org/show_bug.cgi?id=491918

Comment 1 Nalin Dahyabhai 2011-01-11 22:17:42 UTC

In my case I'm seeing it when I try to put a signed data inside of an enveloped data, which is used in implementing a "should" part of the PKINIT spec that's necessary for interacting with versions of Windows before Server 2008.

Comment 3 Elio Maldonado Batiz 2011-03-04 16:02:07 UTC

(In reply to comment #1)
The patches where included in nss-softokn-3.12.9-7.fc15 which was pushed to the Fedora 15 stable repository.  The reported problems still persist, see https://bugzilla.redhat.com/show_bug.cgi?id=499444#c5

Comment 4 Elio Maldonado Batiz 2011-06-09 21:03:48 UTC

These patches were made in anticipation of a fix from upstream. The proper fix  was released upstream with nss 3.12.10. By updating to 3.12.10 we get the fix. That may me preferrable than back-porting patches to 3.12.9.

Comment 7 errata-xmlrpc 2011-12-06 12:08:26 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2011-1584.html