Bug 1478639

Summary: openssl generating 0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding based on internal data
Product: [Fedora] Fedora Reporter: Neil Horman <nhorman>
Component: opensslAssignee: Tomas Mraz <tmraz>
Status: CLOSED NOTABUG QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 26CC: tmraz
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-02-23 15:58:13 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Neil Horman 2017-08-05 12:00:37 UTC 
Description of problem:
I'm attempting to write some code to validate some data sent to me from the nist randomness beacon, and when I attempt to verify the signature on the message, I continually get this error:
0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding

Its entirely possible I've made a coding error here but looking at the problem with GDB, I can't quite see how.  Specifically if I look at RSA_padding_check_PKCS1_type_1 it seems to be failing because the leading byte of the from pointer is not 0, but in the calling function rsa_ossl_public_decrypt, the from pointer is derived from the length of the RSA public key I provided, which was extracted from the x509 certificate successfully.  It seems like if the key was invalid or corrupt, openssl would have informed me then.

Version-Release number of selected component (if applicable):
openssl-1.1.1.0f-7.fc26

How reproducible:
always 

Steps to Reproduce:
1. clone and build the nist-beacon branch of https://github.com/nhorman/rng-tools.git
2.run rngd --list

Actual results:
initialization of the nist-beacon entropy source results in :

0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding

Expected results:
successful validation of the message data
Comment 1 Tomas Mraz 2017-08-07 12:55:38 UTC 
It is extremely unlikely this would be an openssl bug - no RSA verification would work if the padding check was broken. You are most probably passing some incorrect data to the RSA_verify function. The public key should be OK (if the certificate is the right one).
Comment 2 Neil Horman 2017-08-07 14:47:26 UTC 
The public key extracts without error, I agree.  that said, the error in RSA_padding_check_PKCS1_type_1, seems to stem from the validation of a field (specifically n), inside that key structure.  Given that the RSA key is opaque as far as I know to my application (I extract it from the provided X509 cert using openssl functions), I'm not sure what I might be doing wrong.  Is an application meant to pad an extracted key?
Comment 3 Tomas Mraz 2017-08-07 15:53:57 UTC 
No, the value that is checked is the signature value decrypted by the RSA public decryption operation. Which, in case the original data of the signature is incorrect, can contain basically anything. So the key can be OK and the invalid padding means the signature data is wrong.