Bug 2035044

More information:

1) missing debug package?
# debuginfo-install nss
...
Could not find debuginfo package for the following installed packages: nss-3.67.0-7.el8_5.x86_64
Could not find debugsource package for the following installed packages: nss-3.67.0-7.el8_5.x86_64

$ cat /etc/centos-release
CentOS Linux release 8.5.2111

nss.x86_64 3.67.0-7.el8_5
openssl.x86_64 1:1.1.1k-5.el8_5

2) e_nss regression tests pass on all nss releases up to 3.73. Remark 3.73.1 is not tested yet.
In particular nss-3.67-with-nspr-4.29.tar.gz and distibution is not redhat based.

3) failed test is tests/sign-dsa-dss1.test, error is SEC_ERROR_SIGNATURE_ALGORITHM_DISABLED returned from method SGN_Digest().

4) there is no indication that custom patches in nss-3.67.0-7.el8_5.src.rpm stops dsa keys.

5) test fail even with NSS_IGNORE_SYSTEM_POLICY=1
Remark: system policy disables dsa keys less then 2048, test intentionally uses 1024 and environment variable should avoid system limitatiosn.

6) all test pass if build is on another distibutuon with all paches from nss-3.67.0-7.el8_5.src.rpm 

7) non optimised rpmbuild fail
7.1) all custom paches commented 
7.2) 
# Enable compiler optimizations and disable debugging code
#export BUILD_OPT=1
7.3)
# Uncomment to disable optimizations
RPM_OPT_FLAGS=`echo $RPM_OPT_FLAGS | sed -e 's/-O2/-O0/g'`
export RPM_OPT_FLAGS

The reason is that this is forty source enabled build:
$ grep -i optim /usr/include/features.h 
# if !defined __OPTIMIZE__ || __OPTIMIZE__ <= 0
#  warning _FORTIFY_SOURCE requires compiling with optimization (-O)
#if __GNUC_PREREQ (2, 7) && defined __OPTIMIZE__ \
    && !defined __OPTIMIZE_SIZE__ && !defined __NO_INLINE__ \

As result above line must be changed to:
RPM_OPT_FLAGS=`echo $RPM_OPT_FLAGS | sed -e 's/-O2/-O0/g' -e 's/-Wp,-D_FORTIFY_SOURCE=2//g'`

8) even with non optimised nss dsa sing test fail

Finaly some debug details:
(gdb) list
291
292         result.type = siBuffer;
293         result.data = NULL;
294         result.len  = 0;
295
296         rv = SGN_Digest(keyctx->pvtkey, hashalg, &result, &digest);
297         nss_trace(ctx, "rv=%d\n", rv);
298         if (rv != SECSuccess) {
299             int port_err = PORT_GetError();
300             switch(port_err) {
(gdb) display
5: keyctx->pvtkey = (SECKEYPrivateKey *) 0x5555558c8b40
6: *keyctx->pvtkey = {arena = 0x55555588cf10, keyType = dsaKey, pkcs11Slot = 0x5555558863f0, pkcs11ID = 3297886637, pkcs11IsTemp = 0, 
  wincx = 0x7fffffffd050, staticflags = 0}
(gdb) s
SGN_Digest (privKey=0x7ffff76f27a3 <ASN1_item_ex_i2d+691>, algtag=32767, result=0xffffffff, digest=0x0) at secsign.c:462
462     {
(gdb) 


As is visible SGN_Digest is called with keyctx->pvtkey equal to 0x5555558c8b40 but in function parameter is changed to 0x7ffff76f27a3.


Note that RSA and EC keys use similar code but failure is only for dsa keys.


Now as work-around is used local variable:
before:
    rv = SGN_Digest(keyctx->pvtkey, hashalg, &result, &digest);
after:
{   SECKEYPrivateKey *pvtkey = keyctx->pvtkey;
    rv = SGN_Digest(pvtkey, hashalg, &result, &digest);
}

Would it be possible to extract a stand alone test case that I can actually try to compile and run?  Maybe the test is just a single .c file that could be preprocessed (-save-temps) and then attached here.

Is there a gcc version with which the test passes?  I'm not convinced that this is a gcc bug given that the problem manifests even without optimizations.

There may be undefined behavior happening in the test, you could try -fno-strict-aliasing or compile/run the test with -fsanitize=undefined to see if it helps.

I did't check assembler before to post. It seems to me it is one and the same for DSA (subject of issue), RSA and EC around call of method SGN_Digest.

With other words this report in invalid.

OK, thanks.  Closing.