Version: nss-3.12.2.0-5.fc10.x86_64, and a locally rebuilt nss-3.23.3-3. Attempting to encode a CMS message that contains enveloped (digested (data)) causes infinite recursion and stack overflow. To reproduce, compile the attached file with gcc $(pkg-config --cflags --libs nss glib-2.0) repro.c and run ./a.out a/path/to/a/certificate.pem > x This will correctly create an enveloped message. If you redefine DIGESTED in the file to "1" and recompile, ./a.out a/path/to/a/certificate.pem > x will crash with a stack overflow. AFAICS this is going on: * When secasn1e.c finishes encoding the contentType field of NSSCMSEncryptedContentInfoTemplate, sec_asn1e_next_in_sequence() gets called. * ...next_in_sequence does the "after" field notification before changing any ASN1 state. * The "after" notification ends up in nss_cms_before_data(), which creates a completely new encoding context for the "child" digested_data, and starts encoding it. * Encoding the "header" of digested_data (algorithm ID, nested raw data OID) requires more than 16 bytes, so nss_cms_encoder_work_data() will get the encrypted data and call SEC_ASN1EncoderUpdate() with it in the enveloped_data context. * As noted above, the ASN1 context was not changed before calling the "after" field notification of contentType, so SEC_ASN1EncoderUpdate() again thinks the "contentType" field was just finished encoding, calls the "after" notification again, and the whole process (recursively) repeats. See also #499440, a different bug related to the calling SEC_ASN1EncoderUpdate() from within a notifier called sec_asn1e_next_in_sequence ().
Created attachment 342711 [details] Reproducer, see comment#0.
Library functionality bugs should get reported upstream. I've forwarded your bug reported 1:1 at https://bugzilla.mozilla.org/show_bug.cgi?id=491918
nss-3.12.9-10.fc15 has been submitted as an update for Fedora 15. https://admin.fedoraproject.org/updates/nss-3.12.9-10.fc15
nss-softokn-3.12.9-7.fc15, nss-3.12.9-13.fc15 has been pushed to the Fedora 15 stable repository. If problems still persist, please make note of it in this bug report.
I'm afraid I still see a stack overflow - if the defines section contains #define ENVELOPED 1 #define DIGESTED 1 #define INLINE_DATA 0 (the original problem description was incorrect about the macros, sorry about that.)
(In reply to comment #5) > I'm afraid I still see a stack overflow nss-3.12.9-8.fc14.x86_64 nss-softokn-3.12.9-5.fc14.x86_64
(In reply to comment #5) > I'm afraid I still see a stack overflow - if the defines section contains > > #define ENVELOPED 1 > #define DIGESTED 1 > #define INLINE_DATA 0 > > (the original problem description was incorrect about the macros, sorry about > that.) The testing updates are for f15, but I don't see f15's allow-content-types-beyond-smime or nss-recurse patches in the f14 branch. (If I apply them manually, though, the reproducer's happy.)
(In reply to comment #7) > The testing updates are for f15, but I don't see f15's > allow-content-types-beyond-smime or nss-recurse patches in the f14 branch. (If > I apply them manually, though, the reproducer's happy.) Yes, I didn't include the patches in the stable f14 branch and opted instead to wait and pick them up when we do the update to 3.12.10. f15 got them as they where in rawhide already. Would it be possible to get some simple reproducer/verifier test attached to the bug?
Created attachment 483700 [details] sample (incorrect?) output (In reply to comment #7) > The testing updates are for f15, but I don't see f15's > allow-content-types-beyond-smime or nss-recurse patches in the f14 branch. (If > I apply them manually, though, the reproducer's happy.) Right, this does not hang on F15. Reading the F14 changelog, I thought that this problem was fixed in F14 as well, sorry about that. However, the F15 output does not seem to be correct. Attached is an example - looking at it through (openssl asn1parse -inform der -i < x), where I'd expect one "octet string" containing the encrypted representation of DigestedData, there are _6_ separate octet strings. When I (manually) concatenate them and decrypt them, I get a valid DigestedData structure - but my reading of both PKCS#7 and RFC 3852 suggests that there should be only one octet string.
That should be valid if the octet string is tagged as constructed. It surprised me when I first saw it happening, but it's a consequence of the contained data item being itself fed into the encoder in chunks.
(In reply to comment #10) > That should be valid if the octet string is tagged as constructed. It > surprised me when I first saw it happening, but it's a consequence of the > contained data item being itself fed into the encoder in chunks. Ah, so it is just my ignorance of ASN.1. I don't understand why is the "constructed octet string" should be tagged as "context-specific 0", but that's probably my ignorance again. Thanks for the explanation; the infinite recursion is definitely fixed in F15, so this bug can be closed.