Created attachment 658152 [details]
Fix DSA2 keysize calculation on verify

Description of problem:

There seems to be a bits vs bytes mixup in the way DSA2 keysize is calculated in PK11_Verify(), causing it to fail due to not finding a slot that can handle it:

        unsigned int length =  0;
        if ((mech.mechanism == CKM_DSA) &&
                                /* 129 is 1024 bits translated to bytes and
                                 * padded with an optional '0' to maintain a
                                 * positive sign */
                                (key->u.dsa.params.prime.len > 129)) {
            /* we need to get a slot that not only can do DSA, but can do DSA2
             * key lengths */
            length = key->u.dsa.params.prime.len;
            if (key->u.dsa.params.prime.data[0] == 0) {
                length --;
            }
         }

        slot = PK11_GetBestSlotWithAttributes(mech.mechanism,
                                                CKF_VERIFY,length,wincx);


Note the comment about 129 bytes vs 1024 bits translation. However the prime length in bytes is passed as-is to PK11_GetBestSlotWithAttributes() which expects the key size in bits (if specified). 

With a bit of debugging output added, an attempt to verify a signature created with 2048bit DSA key:

vfy_VerifyDigest: dsa key (2), len 64
PK11_Verify: slot (nil)
PK11_Verify: DSA length 256
PK11_GetBestSlotMultipleWithAttributes: mech_count 1
PK11_GetBestSlotMultipleWithAttributes: looking at le 0x1d8b2a0
pk11_filterSlot: need ks 256, min 512 max 3072
  -> keysize bad

256 bit keysize is of course invalid for DSA so it goes downhill from there. With the attached patch to convert the keysize to bits before the slot lookup the above debug foobar looks like:

vfy_VerifyDigest: dsa key (2), len 64
PK11_Verify: slot (nil)
PK11_Verify: DSA length 2048
PK11_GetBestSlotMultipleWithAttributes: mech_count 1
PK11_GetBestSlotMultipleWithAttributes: looking at le 0x1d500c0
pk11_filterSlot: need ks 2048, min 512 max 3072
PK11_GetBestSlotMultipleWithAttributes: returning slot 0x1cdcbf0
PK11_Verify: best slot 0x1cdcbf0, length 2048
[...]

...and the verification actually succeeds.

Version-Release number of selected component (if applicable):

nss-3.14-7.fc18.x86_64, versions prior to 3.14 didn't support DSA2 at all AFAIK.

How reproducible:

Fully reproducable. Unfortunately I dont have a nice isolated test-case I could represent here, I stumbled on this while adding DSA2 support to upstream RPM.

Without the DSA2 keysize patch:
[pmatilai@localhost rpm]$ ./rpmkeys -Kv /tmp/telnet-0.17-51.fc16.x86_64.rpm 
/tmp/telnet-0.17-51.fc16.x86_64.rpm:
    Header V4 DSA/SHA256 Signature, key ID 0e5b82e0: BAD
    Header SHA1 digest: OK (2a2c6961886dc49a1d71c4613c9e7b79ceea3c49)
    MD5 digest: OK (bc60b6d23e72d68765be41012f4f5758)
    V4 DSA/SHA256 Signature, key ID 0e5b82e0: BAD
[pmatilai@localhost rpm]$

...and with it:
[pmatilai@localhost rpm]$ ./rpmkeys -Kv /tmp/telnet-0.17-51.fc16.x86_64.rpm 
/tmp/telnet-0.17-51.fc16.x86_64.rpm:
    Header V4 DSA/SHA256 Signature, key ID 0e5b82e0: OK
    Header SHA1 digest: OK (2a2c6961886dc49a1d71c4613c9e7b79ceea3c49)
    MD5 digest: OK (bc60b6d23e72d68765be41012f4f5758)
    V4 DSA/SHA256 Signature, key ID 0e5b82e0: OK
[pmatilai@localhost rpm]$