Internally libssl in OpenSSL calls X509_verify_cert() on the client side to
verify a certificate supplied by a server. That function may return a negative
return value to indicate an internal error (for example out of memory). Such a
negative return value is mishandled by OpenSSL and will cause an IO function
(such as SSL_connect() or SSL_do_handshake()) to not indicate success and a
subsequent call to SSL_get_error() to return the value
SSL_ERROR_WANT_RETRY_VERIFY. This return value is only supposed to be returned
by OpenSSL if the application has previously called
SSL_CTX_set_cert_verify_callback(). Since most applications do not do this the
SSL_ERROR_WANT_RETRY_VERIFY return value from SSL_get_error() will be totally
unexpected and applications may not behave correctly as a result. The exact
behaviour will depend on the application but it could result in crashes,
infinite loops or other similar incorrect responses.
This issue is made more serious in combination with a separate bug in OpenSSL
3.0 that will cause X509_verify_cert() to indicate an internal error when
processing a certificate chain. This will occur where a certificate does not
include the Subject Alternative Name extension but where a Certificate Authority
has enforced name constraints. This issue can occur even with valid chains.
By combining the two issues an attacker could induce incorrect, application
dependent behaviour.
OpenSSL 3.0.0 SSL/TLS clients are affected by this issue.