Bug 1478639 - openssl generating 0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding based on internal data
openssl generating 0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:inval...
Product: Fedora
Classification: Fedora
Component: openssl (Show other bugs)
Unspecified Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: Tomas Mraz
Fedora Extras Quality Assurance
Depends On:
  Show dependency treegraph
Reported: 2017-08-05 08:00 EDT by Neil Horman
Modified: 2018-02-23 10:58 EST (History)
1 user (show)

See Also:
Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Last Closed: 2018-02-23 10:58:13 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)

  None (edit)
Description Neil Horman 2017-08-05 08:00:37 EDT
Description of problem:
I'm attempting to write some code to validate some data sent to me from the nist randomness beacon, and when I attempt to verify the signature on the message, I continually get this error:
0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding

Its entirely possible I've made a coding error here but looking at the problem with GDB, I can't quite see how.  Specifically if I look at RSA_padding_check_PKCS1_type_1 it seems to be failing because the leading byte of the from pointer is not 0, but in the calling function rsa_ossl_public_decrypt, the from pointer is derived from the length of the RSA public key I provided, which was extracted from the x509 certificate successfully.  It seems like if the key was invalid or corrupt, openssl would have informed me then.

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:
1. clone and build the nist-beacon branch of https://github.com/nhorman/rng-tools.git
2.run rngd --list

Actual results:
initialization of the nist-beacon entropy source results in :

0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding

Expected results:
successful validation of the message data
Comment 1 Tomas Mraz 2017-08-07 08:55:38 EDT
It is extremely unlikely this would be an openssl bug - no RSA verification would work if the padding check was broken. You are most probably passing some incorrect data to the RSA_verify function. The public key should be OK (if the certificate is the right one).
Comment 2 Neil Horman 2017-08-07 10:47:26 EDT
The public key extracts without error, I agree.  that said, the error in RSA_padding_check_PKCS1_type_1, seems to stem from the validation of a field (specifically n), inside that key structure.  Given that the RSA key is opaque as far as I know to my application (I extract it from the provided X509 cert using openssl functions), I'm not sure what I might be doing wrong.  Is an application meant to pad an extracted key?
Comment 3 Tomas Mraz 2017-08-07 11:53:57 EDT
No, the value that is checked is the signature value decrypted by the RSA public decryption operation. Which, in case the original data of the signature is incorrect, can contain basically anything. So the key can be OK and the invalid padding means the signature data is wrong.

Note You need to log in before you can comment on or make changes to this bug.