Bug 1391144

Summary: openssl-1.1.0b-3.fc26's DSA_generate_parameters_ex() fails for some seeds: q not prime
Product: [Fedora] Fedora Reporter: Petr Pisar <ppisar>
Component: opensslAssignee: Tomas Mraz <tmraz>
Status: CLOSED RAWHIDE QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: rawhideCC: tmraz
Target Milestone: ---Keywords: Regression
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openssl-1.1.0b-4.fc26 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-11-04 11:11:07 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Attachments:
Description Flags
test.c none

Description Petr Pisar 2016-11-02 16:02:20 UTC 
Created attachment 1216614 [details]
test.c

When porting perl-Crypt-OpenSSL-DSA to OpenSSL-1.1.0, I found an issue with failing DSA_generate_parameters_ex() call <https://github.com/kmx/perl-Crypt-OpenSSL-DSA/pull/3>. If attached code is built against openssl-1.1.0b-3.fc26.x86_64, it fails for some seeds lengths or values:

[test@fedora-26 tmp]$ gcc -Wall test.c $(pkg-config --libs openssl)
[test@fedora-26 tmp]$ ./a.out
[test@fedora-26 tmp]$ ./a.out a
Could not generate DSA parametetes: q not prime
[test@fedora-26 tmp]$ ./a.out fo
Could not generate DSA parametetes: q not prime
[test@fedora-26 tmp]$ ./a.out foo
[test@fedora-26 tmp]$ ./a.out fooo
Could not generate DSA parametetes: q not prime
[test@fedora-26 tmp]$ ./a.out foo
[test@fedora-26 tmp]$ ./a.out abc
Could not generate DSA parametetes: q not prime

Why does it pass for "foo" seed but fails for "abc" seed? Why it fails for seeds with slight different length. Could it be caused by FIPS patch that uses dsa_builtin_paramgen2() instead dsa_builtin_paramgen()?

The same code built against openssl-1.0.2j-1.fc24.x86_64 always works.
Comment 1 Petr Pisar 2016-11-02 16:06:57 UTC 
DSA_generate_parameters_ex(3) manual reads:

    bits is the length of the prime p to be generated. For lengths under 2048
    bits, the length of q is 160 bits; for lengths greater than or equal to
    2048 bits, the length of q is set to 256 bits.
and 

    Seed lengths > 20 are not supported.

But it does not work even with exactly 20-bytes long seed that should be required for 512-bit p prime.
Comment 2 Tomas Mraz 2016-11-03 12:56:02 UTC 
Not every seed generates a prime, however the old algorithm simply used random seed (or possibly multiple random seeds) if the seed failed to generate a prime. I will modify the dsa_builtin_paramen2 to behave the same way. I'll also modify the manual page to properly document the supported seed length as it actually depends on the SHA2 variant used internally. Anyway I would not bother testing the generation based on the pre-set seed at all.