Take the 2-minute tour ×

I'm trying to develop a license verification solution.

Licences are encoded on server using RSA_private_encrypt openssl function.

For Mac OX X I use openssl RSA_public_decrypt and it works like a charm. On Windows I must use very tiny bit of code, so I can not link with openssl or other lib AND I have to use Crypto API.

I have spent several days trying to figure out what is wrong, but with no luck. I can successfully import public key, but here my success ends. I'm aware that I need to reverse byte order with CAPI so this might not be the issue.

I have tried everything, including CryptVerifyMessageSignatureWithKey and CryptDecodeObject to load the blob with different params, but still no luck.

It always ends up with GetLastError() == CRYPT_E_ASN1_BADTAG, which I assume means that the BLOB is not ASN1 formatted... Google does not tell anything on the output format of RSA_private_encrypt... so I am completely lost here.

Here is the OS X code based on OpenSSL:

    void cr_license_init(const char* lic) {
        __cr_license_ = lic;
        unsigned char lic_encoded[CR_LIC_LEN];

        BIO* b64 = BIO_new(BIO_f_base64());
        BIO* licIn = BIO_new_mem_buf((void*)lic, -1);
        licIn = BIO_push(b64, licIn);

        if(BIO_read(licIn, lic_encoded, CR_LIC_LEN) == CR_LIC_LEN) {

            const unsigned char* key_data = license_pub_der;
            RSA* r = d2i_RSA_PUBKEY(NULL, &key_data, sizeof(license_pub_der));

            if(r != NULL) {
                if(__cr_license_data_ != NULL) {
                __cr_license_data_ = malloc(CR_LIC_LEN);
                if(RSA_public_decrypt(CR_LIC_LEN, lic_encoded,
        (unsigned char*)__cr_license_data_, r, RSA_PKCS1_PADDING) <= 0) {
                    __cr_license_data_ = NULL;

This part of code on windows works well, so I assume public key is not an issue.

    __cr_license_ = lic;
    unsigned char lic_encoded[CR_LIC_LEN];

    DWORD dwSize;
    if(CryptStringToBinaryA(__cr_license_, 0/*autocalculate*/, CRYPT_STRING_BASE64, lic_encoded, &dwSize, NULL, NULL) && dwSize == CR_LIC_LEN) {
        DWORD dwKeySize;
        if(CryptDecodeObjectEx(X509_ASN_ENCODING, X509_PUBLIC_KEY_INFO, license_pub_der, sizeof(license_pub_der), CRYPT_ENCODE_ALLOC_FLAG, NULL, &pki, &dwKeySize)) {
            HCRYPTKEY hKey = 0;
            if(CryptImportPublicKeyInfo( hProv, X509_ASN_ENCODING, pki, &hKey)) {

But after that anything I try to do with message leads to CRYPT_E_ASN1_BADTAG. I tried CryptMsgOpenToDecode with CryptMsgUpdate, CryptDecodeObject, CryptVerifyMessageSignatureWithKey - nothing works.

Basically I think that the problem is in pkcs1 and pkcs7 incompatibility as owlstead mentionaed. Does anyone has experience working with pkcs1 format importing/converting/etc. with CryptoAPI?

Any help or even a clue is appreciated a lot! Thanks in advance!

share|improve this question
You might want to include some code in the question. –  Daniel Roethlisberger Jan 25 '13 at 18:42
also show header of keys and encoded files. That error is for invalid ASN tag header –  Vahid Farahmand Jan 25 '13 at 20:52
Does this help? stackoverflow.com/q/7573754/645583 –  pcunite Jan 25 '13 at 22:22
Yeah, I have seen this thread. The problem in this case is not the key. I can successfully import the key with CryptImportPublicKeyInfo, so I assume the key is not the case. It seems that the problem is with encoded BLOB. I believe that the BLOB is not ASN1 compatible and should be converted somehow. –  Daniel Jan 25 '13 at 22:41
@Daniel I was hoping for the Windows code which threw the CRYPT_E_ASN1_BADTAG error you are refering to. –  Daniel Roethlisberger Jan 25 '13 at 23:05

2 Answers 2

You are mixing higher and lower level signature formats. OpenSSL asumes PKCS#1 v1.5 signatures by default, which contains of only the signature data. Windows seems to asume PKCS#7 containers. These may contain a PKCS#1 v1.5, but those and other data are wrapped using ASN.1 BER tag/length format. If the Microsoft API tries to decode this it will assume that the raw signature is the container format, and decoding will fail.

share|improve this answer
Is it possible to add higher level encoding (PKCS7) to the signature? I mean may be something like CryptEncodeObject? –  Daniel Jan 25 '13 at 23:32
See my latest comment on your question –  Maarten Bodewes - owlstead Jan 25 '13 at 23:32
thank you very much for the clarification! Will continue the research. Anyone is aware of possible solution? –  Daniel Jan 25 '13 at 23:37

Unless this is so obvious that you've tried but omitted listing it or I misunderstand your question otherwise, I think you should be using CryptDecrypt to decrypt the license, not the functions you mention in the question. Note that since you seem to be using OpenSSL with PKCS#1 v1.5 padding and CryptoAPI does not seem to support that (haven't tested, but specs only list PKCS#1 v2 OAEP), you will probably have to use CRYPT_DECRYPT_RSA_NO_PADDING_CHECK and verify and remove the PKCS#1 v1.5 padding manually after decryption.

share|improve this answer
Isn't CryptDecrypt intended to be used with the private key? –  Daniel Jan 26 '13 at 15:07
As I thought it returns NTE_NO_KEY when used with public key. –  Daniel Jan 26 '13 at 15:16
Have you tried or do you just think it does? I honestly don't know, but all the other functions operate on higher level cryptographical containers, not raw RSA blocks. If it does not work, you might either be able to trick it into believing the public key is a private key by flipping some bit (not sure how the public key is marked as such), or trick it using CryptEncrypt without any padding (not specifying CRYPT_OAEP), since if it does not do any padding, encrypt and decrypt is the equivalent operation. –  Daniel Roethlisberger Jan 26 '13 at 15:20
Yeah, I just tried. It does not work. I'm not sure if it is possible (and how) to trick it to believe that it's a private key... Basically I think this might not be possible, because encrypting with private key (also known as "signing") should differ a lot from encrypting with public key. –  Daniel Jan 26 '13 at 15:28
The encrypt and decrypt operations only differ in adding the padding / verifying&removing padding. On the raw RSA level, the operations are identical, just with a different key. Also, the raw signing/verifying operations only differ from encryption/decryption in the different padding, the actual RSA operations on the bit level are identical. –  Daniel Roethlisberger Jan 26 '13 at 15:29

Your Answer


