Description of problem: I'm attempting to write some code to validate some data sent to me from the nist randomness beacon, and when I attempt to verify the signature on the message, I continually get this error: 0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding Its entirely possible I've made a coding error here but looking at the problem with GDB, I can't quite see how. Specifically if I look at RSA_padding_check_PKCS1_type_1 it seems to be failing because the leading byte of the from pointer is not 0, but in the calling function rsa_ossl_public_decrypt, the from pointer is derived from the length of the RSA public key I provided, which was extracted from the x509 certificate successfully. It seems like if the key was invalid or corrupt, openssl would have informed me then. Version-Release number of selected component (if applicable): openssl-1.1.1.0f-7.fc26 How reproducible: always Steps to Reproduce: 1. clone and build the nist-beacon branch of https://github.com/nhorman/rng-tools.git 2.run rngd --list Actual results: initialization of the nist-beacon entropy source results in : 0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding Expected results: successful validation of the message data
It is extremely unlikely this would be an openssl bug - no RSA verification would work if the padding check was broken. You are most probably passing some incorrect data to the RSA_verify function. The public key should be OK (if the certificate is the right one).
The public key extracts without error, I agree. that said, the error in RSA_padding_check_PKCS1_type_1, seems to stem from the validation of a field (specifically n), inside that key structure. Given that the RSA key is opaque as far as I know to my application (I extract it from the provided X509 cert using openssl functions), I'm not sure what I might be doing wrong. Is an application meant to pad an extracted key?
No, the value that is checked is the signature value decrypted by the RSA public decryption operation. Which, in case the original data of the signature is incorrect, can contain basically anything. So the key can be OK and the invalid padding means the signature data is wrong.