This issue is very similar to CVE-2016-6307. The underlying defect is different
but the security analysis and impacts are the same except that it impacts DTLS.
A DTLS message includes 3 bytes for its length in the header for the message.
This would allow for messages up to 16Mb in length. Messages of this length are
excessive and OpenSSL includes a check to ensure that a peer is sending
reasonably sized messages in order to avoid too much memory being consumed to
service a connection. A flaw in the logic of version 1.1.0 means that memory for
the message is allocated too early, prior to the excessive message length
check. Due to way memory is allocated in OpenSSL this could mean an attacker
could force up to 21Mb to be allocated to service a connection. This could lead
to a Denial of Service through memory exhaustion. However, the excessive message
length check still takes place, and this would cause the connection to
immediately fail. Assuming that the application calls SSL_free() on the failed
conneciton in a timely manner then the 21Mb of allocated memory will then be
immediately freed again. Therefore the excessive memory allocation will be
transitory in nature. This then means that there is only a security impact if:
Except in the instance of (1) above any Denial Of Service is likely to
be transitory because as soon as the connection fails the memory is
subsequently freed again in the SSL_free() call. However there is an
increased risk during this period of application crashes due to the lack
of memory - which would then mean a more serious Denial of Service.
This issue does not affect TLS users.