Small subgroup attack in Mozilla NSS

tl;dr While the TLS servers attacks has been pretty much studied and fixed (see e.g. https://www.secure-resumption.com/ and https://weakdh.org/) the situation with the TLS clients is (was) not ideal and can be improved. Here I report a Small subgroup attack for TLS clients that I performed against various browsers and reported.

Whoever reads this blog is used to read about OAuth .
For once (and maybe more in the future) let’s hijack the usual topic and let’s talk about my new “passion” : TLS in particular Diffie–Hellman (DH from now on).

Now, before to start I need to clarify one thing IANAC (I am not a cryptographer) so I might likely end up writing a bunch of mistakes in this blog post…

Diffie-Hellman is used in SSL/TLS, as “ephemeral Diffie-Hellman” (EDH) and it is probably going to be kill soonish (or at least is the intent of Google Chrome). FWIW I personally agree with this unless EDH implements the Negotiated Finite Field specification.

Now in the last years there were at least a couple of issue that affected EDH:

• Triple Handshakes
• Logjam Attack
  What I am going to describe here is by far less severe that the issues above. Indeed has been rated by Mozilla NSS as security moderate and Google Chrome did not consider harmful at all (and since Adam Langley is one of the people that is on this side I got to agree with him :) ).

But here the details:

When using TLS_DHE_RSA_WITH_AES_128_CBC_SHA Firefox/Chrome doesn’t accept degenerate public key of value 0,1 and -1 since this key lead to pms that is {0,1, -1}.
This (the -1 case) is probably a consequence of CVE-2014-1491 (raised as part of the Triple Handshake Attack ).

I would refer to the classic Diffie Hellman nomenclature

• p as the prime number
• g the generator with order p-1 = q
• y public key
• x private key

Observation

If (p-1)/4 = 0 (mod p) then if I choose my private key x = (p-1)/4 then my public key
y = g^x will generates a prime-order subgroup of size 4.

This means that Mozilla/Chrome will agree on a pms = 1 one time out of 4.

The issue

I set up a server with

p = 13407807929942597099574024998205846127479365820592393377723561443721764030073546976801874298166903427690031858186486050853753882811946569946433649006084241
g = 3
q =1

and TLS_DHE_RSA_WITH_AES_128_CBC_SHA as cipher.

During the negotiation with Chrome I always choose

x= (p-1)/4 = 3351951982485649274893506249551461531869841455148098344430890360930441007518386744200468574541725856922507964546621512713438470702986642486608412251521060

and pass

y = 11130333445084706427994000041243435077443611277989851635896953056790400956946719341695219235480436483595595868058263313228038179294276393680262837344694991

Chrome/Firefox will happily “agree” on those 4 pms

• 1
• 2277474484857890671580024956962411050035754542602541741826608386931363073126827635106655062686466944094435990128222737625715703517670176266170811661389250
• 13407807929942597099574024998205846127479365820592393377723561443721764030073546976801874298166903427690031858186486050853753882811946569946433649006084240
• 11130333445084706427994000041243435077443611277989851635896953056790400956946719341695219235480436483595595868058263313228038179294276393680262837344694991

Of course the “worse” one is 1 and happens to be 1 time out of 4 (according to Adam Langley though “here’s nothing special about sending an odd DH value, it could equally well make its DH private key equal to 42”). So not big deal :(

Just for the record even the easier suggestion given in [1] aka
**
“Make sure that g^x,g^y and g^xy do not equal to 1”**

is not followed and this happens with very high probability (25%)

The Summary

• NSS as https://bugzilla.mozilla.org/show_bug.cgi?id=1160139 (this is now fixed) and will be solved in the next version of Firefox
• Chrome just switched to boringSSL (previously it was also using NSS) I did report in https://code.google.com/p/chromium/issues/detail?id=482950 but is NOTconsidered harmful for them
• The situation for Safari is a bit a disaster at the moment. See also http://www.openwall.com/lists/oss-security/2014/03/04/8 . I will try to report somehow downstream to konqueror

[1] http://crypto.cs.mcgill.ca/~stiglic/Papers/dhfull.pdf