encryption - Use OpenSSL command line to create RSA keys and encrypt data compatible with an older embedded system? - Stack Over

At work, I have been assigned the task of using the OpenSSL command line (on my laptop in Cygwin) to create an RSA key pair in PEM format, and to encrypt a piece of plain data that will be decrypted by an older, legacy embedded system.

• The key must be 8K bits (1024 bytes) in length.
• The OpenSSL version on my laptop is “OpenSSL 3.0.15 3 Sep 2024 (Library: OpenSSL 3.0.15 3 Sep 2024”.
• The embedded system is using an older version of OpenSSL C++ libraries. The hard coded padding style in the embedded code is “RSA_PKCS1_PADDING”.
• I should also mention that I must encrypt the data using the private key, as the embedded system is decrypting using the public key. We inherited that design, and there is nothing we can do about it.

This all works fine with an RSA key pair that we inherited. However, I am having trouble trying to duplicate the process with my own OpenSSL key creation and encryption commands.

• The embedded system fails when calling OpenSSL library function “RSA_public_decrypt()”.
• The function returns -1 for the number of decrypted bytes.
• It also queues two error codes inside the call to “RSA_public_decrypt()”.
• The first error code is “0407008A: RSA_padding_check_PKCS1_type_1: invalid padding”.
• The second error code is 4067072. I can’t find a description for that.

The OpenSSL commands I used to create the key pair and encrypt the plain data are:

• openssl genrsa -traditional -out rsa_aes_private.pem 8192
• openssl pkeyutl -encrypt -pkeyopt rsa_padding_mode:pkcs1 -inkey private.pem -in file.txt -out file.enc

I used the -traditional switch to create the RSA key pair because I read somewhere that it would produce the desired “RSA_PKCS1_PADDING” expected by the embedded system.

As a last resort, I am allowed to modify the embedded system to use other types of padding to get this working. The padding choices I have are:

• #define RSA_PKCS1_PADDING 1
• #define RSA_SSLV23_PADDING 2
• #define RSA_NO_PADDING 3
• #define RSA_PKCS1_OAEP_PADDING 4
• #define RSA_X931_PADDING 5 /* EVP_PKEY_ only */
• #define RSA_PKCS1_PSS_PADDING 6

If anyone could help me correct the two OpenSSL commands, I would really appreciate it. Thank you. I’m not very experienced in cryptography.

Edited to add this after the comment by dave_thompson_085

I changed my key creation commands and encryption (sign) command to the following:

• openssl genrsa -out rsa_aes_private.pem 8192
• openssl rsa -in rsa_aes_private.pem -pubout -out rsa_aes_public.pem
• openssl dgst -sign rsa_aes_private.pem -keyform PEM -sha256 -out applicat_aes_key_iv.enc -binary applicat_aes_key_iv.txt

I used this, and the decryption "worked", but the decrypted result was the SHA256 hash of the file. I need the file contents themselves encrypted/decrypted - not the SHA256 hash.

Can I adjust this command to work - to encrypt the file contents with the private key so that the embedded system decrypts with public key? I know I'm probably using the wrong terminology - encrypt vs sign - but I need the contents of the file encrypted, not its SHA256 hash.

Thanks.

At work, I have been assigned the task of using the OpenSSL command line (on my laptop in Cygwin) to create an RSA key pair in PEM format, and to encrypt a piece of plain data that will be decrypted by an older, legacy embedded system.

• The key must be 8K bits (1024 bytes) in length.
• The OpenSSL version on my laptop is “OpenSSL 3.0.15 3 Sep 2024 (Library: OpenSSL 3.0.15 3 Sep 2024”.
• The embedded system is using an older version of OpenSSL C++ libraries. The hard coded padding style in the embedded code is “RSA_PKCS1_PADDING”.
• I should also mention that I must encrypt the data using the private key, as the embedded system is decrypting using the public key. We inherited that design, and there is nothing we can do about it.

This all works fine with an RSA key pair that we inherited. However, I am having trouble trying to duplicate the process with my own OpenSSL key creation and encryption commands.

• The embedded system fails when calling OpenSSL library function “RSA_public_decrypt()”.
• The function returns -1 for the number of decrypted bytes.
• It also queues two error codes inside the call to “RSA_public_decrypt()”.
• The first error code is “0407008A: RSA_padding_check_PKCS1_type_1: invalid padding”.
• The second error code is 4067072. I can’t find a description for that.

The OpenSSL commands I used to create the key pair and encrypt the plain data are:

• openssl genrsa -traditional -out rsa_aes_private.pem 8192
• openssl pkeyutl -encrypt -pkeyopt rsa_padding_mode:pkcs1 -inkey private.pem -in file.txt -out file.enc

I used the -traditional switch to create the RSA key pair because I read somewhere that it would produce the desired “RSA_PKCS1_PADDING” expected by the embedded system.

As a last resort, I am allowed to modify the embedded system to use other types of padding to get this working. The padding choices I have are:

• #define RSA_PKCS1_PADDING 1
• #define RSA_SSLV23_PADDING 2
• #define RSA_NO_PADDING 3
• #define RSA_PKCS1_OAEP_PADDING 4
• #define RSA_X931_PADDING 5 /* EVP_PKEY_ only */
• #define RSA_PKCS1_PSS_PADDING 6

If anyone could help me correct the two OpenSSL commands, I would really appreciate it. Thank you. I’m not very experienced in cryptography.

Edited to add this after the comment by dave_thompson_085

I changed my key creation commands and encryption (sign) command to the following:

• openssl genrsa -out rsa_aes_private.pem 8192
• openssl rsa -in rsa_aes_private.pem -pubout -out rsa_aes_public.pem
• openssl dgst -sign rsa_aes_private.pem -keyform PEM -sha256 -out applicat_aes_key_iv.enc -binary applicat_aes_key_iv.txt

I used this, and the decryption "worked", but the decrypted result was the SHA256 hash of the file. I need the file contents themselves encrypted/decrypted - not the SHA256 hash.

Can I adjust this command to work - to encrypt the file contents with the private key so that the embedded system decrypts with public key? I know I'm probably using the wrong terminology - encrypt vs sign - but I need the contents of the file encrypted, not its SHA256 hash.

Thanks.

• encryption
• openssl
• rsa

Share Follow edited Feb 2 at 20:20 RickyBobby asked Feb 1 at 22:57 RickyBobbyRickyBobby 7555 bronze badges 8

• I should also mention that the embedded code contains the following line and comment: d2i_RSA_PUBKEY(&rsaKey, (const unsigned char **)&key.Data, key.Size); //Certificate Public Key Structure. The comment contains the word "certificate", which may give a clue as to how to create the key. I thought the existing (working) key was just a plain RSA key, but maybe it is some sort of certificate. Not sure at this point. There is no documentation on the embedded system. – RickyBobby Commented Feb 1 at 23:16
• Do you know which version of openssl the embedded system is using? – shotor Commented Feb 2 at 2:00
• 1 RSA DOES NOT ENCRYPT WITH PRIVATE KEY. The RSA primitives for signature and encryption are duals, and as a result in its early days -- when SSLeay-now-OpenSSL was written -- signature was called 'encrypt with private key' and verification 'decrypt with public key'. In recent decades these have been corrected, and for the (legacy) OpenSSL 'RSA_private_encrypt' function you want pkeyutl -sign. PUBKEY is a structure defined by the certificate standard X.509 (and its variant PKIX) hence 'certificate public key structure' but is not a certificate. ... – dave_thompson_085 Commented Feb 2 at 7:53
• 1 ... Also genrsa -traditional only affects how the privatekey is stored on YOUR system; it has no effect on operations using the key. Both the key format AND the padding for operations -- and more -- were defined by PKCS1 but are quite different and unrelated things. And openssl errstr 4067072 -> rsa routines:rsa_ossl_public_decrypt:padding check failed . – dave_thompson_085 Commented Feb 2 at 7:53
• 1 Ricky: first, to 'notify' a(nother) commenter you need to use atsign and name, not a URL; I happened to see this on browsing but otherwise would have missed it and therefore not responded. (I didn't atsign you because the author of the base post is notified automatically.) I told you pkeyutl -sign which signs raw data (only up to the modulus size minus delta), not dgst -sign which signs a hash of the (unbounded) data plus a prefix as described in e.g. rfc8017 section 9.2 at page 47 (this is the more usual and standard method for signing data) – dave_thompson_085 Commented Feb 2 at 20:55

 |  Show 3 more comments

1 Answer 1

Sorted by: Reset to default 0

I wanted to post that due to help from @dave_thompson_085, I was able to use the following to solve my problem:

openssl pkeyutl -sign -pkeyopt rsa_padding_mode:pkcs1 -inkey private.pem -in applicat_aes_key_iv.bin -out applicat_aes_key_iv.enc

Thanks Dave.