CVE-2023-3519

Unauthenticated remote code execution

Recent assessments:

rbowes-r7 at July 19, 2023 9:42pm UTC reported:

Note that the analysis before is for a separate issue – either a silently-patched vuln or some cleanup. We’ve posted a Rapid7 Analysis with the full details, so check that out!

---

I spent some time this week looking at the advisory for CVE-2023-3519, and the associated patch (our blog post). To test, I installed versions 13.0-91.12 and 13.0-91.13, and did a bunch of diffing / patch analysis to try and understand the vulnerability (I chose a 13.0 version with the idea that there will be less active maintenance, and therefore less other changes to the codebase). Ultimately I hit a paywall in the trial version, so I figured I’d post what I have and move onto something else.

If I’m right, the issue appears to be memory corruption in the SAML parser, in the nsppe process. Since nsppe runs as root, successful exploitation would likely lead to code execution as root (it would require network access to the management port).

This has been added to the known exploited vulnerabilities list, but they identify it as a “code injection” attack, so it’s possible that this isn’t right.

After installing, I diffed everything on the filesystem to look for major changes. There were no obvious changes in the scripting stuff (PHP/Perl/etc), which is where I kinda expected to see the patch. Next, I diffed the binaries under /netscaler and other folders using this neat little Bash command:

$ for i in $(cat binaries.txt); do diff -rub <(strings -n8 ./13.0-91.12/$i | sort) <(strings -n8 ./13.0-91.13/$i | sort); done

That command takes a list of binaries from the file binaries.txt (which I generated with find -type f), runs strings on both versions, then diffs the strings list. It’s a nice quick way to quickly find or/ triage the files that are most interesting, though it won’t find subtle logic changes.

Most of the changes are just version numbers or other simple things that we can immediately discount. But one file, nsppe, had interesting looking changes, such as new error messages about a length value:

 aaa_rpc_auth_fail_free_aaa_info
 aaa_rpc_auth_rejected
+AAA SAMLIDP ASSERTION REQUEST: List of Canonicalization Method exceeds max limit <%d>, current value <%d>
+AAA SAMLIDP ASSERTION REQUEST: List of Transforms Method exceeds max limit <%d>, current value <%d>
 aaa_samlidp_tot_authnreq_fail
 aaa_samlidp_tot_authnreq_succ

I grabbed the old and new versions of the nsppe binary, which it turns out is the NetScaler Packet Parsing Engine, and disassembled them. The binary is a beast, but we found some documentation online that explains its purpose.

Thankfully, the changes made by this patch are limited. In case you want to make sure we’re on the same page, here are the sha256es of the two versions I tested:

$ sha256 nsppe-13.0-91.12
SHA256 (nsppe-13.0-91.12) = 9d1b39e59374088f355a9b26a1b9bb94addfb1dcd7af7fd28acd853403b414f7

$ sha256 nsppe-13.0-91.13
SHA256 (nsppe-13.0-91.13) = 3779d1148df674e3f107b384f1e770e658128a6d603399c3d595c31a0d37af4e

I diffed the patches using a complicated series of sed statements and the diff CLI tool to determine where the majority of the changes are. For the most part, the patch adds eight different length checks, each of which compares a local variable to 9, then prints an error message if it’s larger. I put together this list of functions, along with the address where a check was added and the error message if the check fails (these are from the 91.13 version of /netscaler/nsppe):

• Function: ns_aaa_saml_parse_authn_request():

  • 0xCA920B – AAA SAMLIDP ASSERTION REQUEST: List of Canonicalization Method exceeds max limit <%d>, current value <%d>

  • 0xCAA57E – AAA SAMLIDP ASSERTION REQUEST: List of Transforms Method exceeds max limit <%d>, current value <%d>

• Function ns_aaa_saml_parse_logout_response():

  • 0xCB09A6 – AAA SAML LOGOUT RESPONSE: List of Canonicalization Method exceeds max limit <%d>, current value <%d>

  • 0xCB1E29 – AAA SAML LOGOUT RESPONSE: List of Transforms Method exceeds max limit <%d>, current value <%d>

• Function ns_aaa_saml_parse_logout_request():

  • 0xCB8154 – AAA SAML LOGOUT REQUEST: List of Canonicalization Method exceeds max limit <%d>, current value <%d>

  • 0xCB94DB – AAA SAML LOGOUT REQUEST: List of Transforms Method exceeds max limit <%d>, current value <%d>

• Function ns_aaa_saml_parse_assertion():

  • 0xD038E7 – AAA SAMLSP ASSERTION RESPONSE: List of Canonicalization Method exceeds max limit <%d>, current value <%d>

  • 0xD04CD3 – AAA SAMLSP ASSERTION RESPONSE: List of Transforms Method exceeds max limit <%d>, current value <%d>

As you can see, the error messages are very similar, and strongly point to a length check being added. Based on the error messages, I would speculate that the vulnerability is triggered by sending too many canonicalization or transform methods in a SAML message. I randomly picked one of those errors (the Canonicalization Method check from ns_aaa_saml_parse_authn_request), and looked at how the newly-validated value is used. The number_of_canonicalizations value checked at 0xCA920B (in 91.13 – the patched version) is used to write into an array that I called array_indexed_into:

.text:0000000000CA9AAB loc_CA9AAB:                             ; CODE XREF: ns_aaa_saml_parse_authn_request+5C68↑j
.text:0000000000CA9AAB                                         ; ns_aaa_saml_parse_authn_request+5C7D↑j
.text:0000000000CA9AAB                 mov     eax, [rbp+number_of_canonicalizations]
.text:0000000000CA9AAE                 mov     rdx, [rbp+array_indexed_into]
.text:0000000000CA9AB5                 mov     dword ptr [rdx+rax*4+40h], 2
.text:0000000000CA9ABD                 jmp     short loc_CA9AD0

Tracing the targeted array backwards across several function calls leads to:

.text:00000000008128E9                 mov     edx, 35h ; '5'
.text:00000000008128EE                 mov     esi, 210h
.text:00000000008128F3                 mov     edi, offset ns_alloc_memz
.text:00000000008128F8                 call    ns_meminst_alloc

ns_meminst_alloc and ns_alloc_memz are memory-allocation functions, and it looks like heap memory, though I’m not 100% sure on that one.

So anyway, my guess is that the core of this vulnerability is a probably-heap overflow when parsing certain types of SAML requests with too many canonicalization or transform methods.

I tried to verify that, but I believe the AAA features are behind a paywall (licensewall?), so I can’t access them:

> enable ns feature AAA
ERROR: Feature(s) not licensed

Rather than fuss with licensing, I figured I’d post what I have and carry on!

domckee-r7 at August 01, 2023 6:38pm UTC reported:

Note that the analysis before is for a separate issue – either a silently-patched vuln or some cleanup. We’ve posted a Rapid7 Analysis with the full details, so check that out!

---

I spent some time this week looking at the advisory for CVE-2023-3519, and the associated patch (our blog post). To test, I installed versions 13.0-91.12 and 13.0-91.13, and did a bunch of diffing / patch analysis to try and understand the vulnerability (I chose a 13.0 version with the idea that there will be less active maintenance, and therefore less other changes to the codebase). Ultimately I hit a paywall in the trial version, so I figured I’d post what I have and move onto something else.

If I’m right, the issue appears to be memory corruption in the SAML parser, in the nsppe process. Since nsppe runs as root, successful exploitation would likely lead to code execution as root (it would require network access to the management port).

This has been added to the known exploited vulnerabilities list, but they identify it as a “code injection” attack, so it’s possible that this isn’t right.

After installing, I diffed everything on the filesystem to look for major changes. There were no obvious changes in the scripting stuff (PHP/Perl/etc), which is where I kinda expected to see the patch. Next, I diffed the binaries under /netscaler and other folders using this neat little Bash command:

$ for i in $(cat binaries.txt); do diff -rub <(strings -n8 ./13.0-91.12/$i | sort) <(strings -n8 ./13.0-91.13/$i | sort); done

That command takes a list of binaries from the file binaries.txt (which I generated with find -type f), runs strings on both versions, then diffs the strings list. It’s a nice quick way to quickly find or/ triage the files that are most interesting, though it won’t find subtle logic changes.

Most of the changes are just version numbers or other simple things that we can immediately discount. But one file, nsppe, had interesting looking changes, such as new error messages about a length value:

 aaa_rpc_auth_fail_free_aaa_info
 aaa_rpc_auth_rejected
+AAA SAMLIDP ASSERTION REQUEST: List of Canonicalization Method exceeds max limit <%d>, current value <%d>
+AAA SAMLIDP ASSERTION REQUEST: List of Transforms Method exceeds max limit <%d>, current value <%d>
 aaa_samlidp_tot_authnreq_fail
 aaa_samlidp_tot_authnreq_succ

I grabbed the old and new versions of the nsppe binary, which it turns out is the NetScaler Packet Parsing Engine, and disassembled them. The binary is a beast, but we found some documentation online that explains its purpose.

Thankfully, the changes made by this patch are limited. In case you want to make sure we’re on the same page, here are the sha256es of the two versions I tested:

$ sha256 nsppe-13.0-91.12
SHA256 (nsppe-13.0-91.12) = 9d1b39e59374088f355a9b26a1b9bb94addfb1dcd7af7fd28acd853403b414f7

$ sha256 nsppe-13.0-91.13
SHA256 (nsppe-13.0-91.13) = 3779d1148df674e3f107b384f1e770e658128a6d603399c3d595c31a0d37af4e

I diffed the patches using a complicated series of sed statements and the diff CLI tool to determine where the majority of the changes are. For the most part, the patch adds eight different length checks, each of which compares a local variable to 9, then prints an error message if it’s larger. I put together this list of functions, along with the address where a check was added and the error message if the check fails (these are from the 91.13 version of /netscaler/nsppe):

• Function: ns_aaa_saml_parse_authn_request():

  • 0xCA920B – AAA SAMLIDP ASSERTION REQUEST: List of Canonicalization Method exceeds max limit <%d>, current value <%d>

  • 0xCAA57E – AAA SAMLIDP ASSERTION REQUEST: List of Transforms Method exceeds max limit <%d>, current value <%d>

• Function ns_aaa_saml_parse_logout_response():

  • 0xCB09A6 – AAA SAML LOGOUT RESPONSE: List of Canonicalization Method exceeds max limit <%d>, current value <%d>

  • 0xCB1E29 – AAA SAML LOGOUT RESPONSE: List of Transforms Method exceeds max limit <%d>, current value <%d>

• Function ns_aaa_saml_parse_logout_request():

  • 0xCB8154 – AAA SAML LOGOUT REQUEST: List of Canonicalization Method exceeds max limit <%d>, current value <%d>

  • 0xCB94DB – AAA SAML LOGOUT REQUEST: List of Transforms Method exceeds max limit <%d>, current value <%d>

• Function ns_aaa_saml_parse_assertion():

  • 0xD038E7 – AAA SAMLSP ASSERTION RESPONSE: List of Canonicalization Method exceeds max limit <%d>, current value <%d>

  • 0xD04CD3 – AAA SAMLSP ASSERTION RESPONSE: List of Transforms Method exceeds max limit <%d>, current value <%d>

As you can see, the error messages are very similar, and strongly point to a length check being added. Based on the error messages, I would speculate that the vulnerability is triggered by sending too many canonicalization or transform methods in a SAML message. I randomly picked one of those errors (the Canonicalization Method check from ns_aaa_saml_parse_authn_request), and looked at how the newly-validated value is used. The number_of_canonicalizations value checked at 0xCA920B (in 91.13 – the patched version) is used to write into an array that I called array_indexed_into:

.text:0000000000CA9AAB loc_CA9AAB:                             ; CODE XREF: ns_aaa_saml_parse_authn_request+5C68↑j
.text:0000000000CA9AAB                                         ; ns_aaa_saml_parse_authn_request+5C7D↑j
.text:0000000000CA9AAB                 mov     eax, [rbp+number_of_canonicalizations]
.text:0000000000CA9AAE                 mov     rdx, [rbp+array_indexed_into]
.text:0000000000CA9AB5                 mov     dword ptr [rdx+rax*4+40h], 2
.text:0000000000CA9ABD                 jmp     short loc_CA9AD0

Tracing the targeted array backwards across several function calls leads to:

.text:00000000008128E9                 mov     edx, 35h ; '5'
.text:00000000008128EE                 mov     esi, 210h
.text:00000000008128F3                 mov     edi, offset ns_alloc_memz
.text:00000000008128F8                 call    ns_meminst_alloc

ns_meminst_alloc and ns_alloc_memz are memory-allocation functions, and it looks like heap memory, though I’m not 100% sure on that one.

So anyway, my guess is that the core of this vulnerability is a probably-heap overflow when parsing certain types of SAML requests with too many canonicalization or transform methods.

I tried to verify that, but I believe the AAA features are behind a paywall (licensewall?), so I can’t access them:

> enable ns feature AAA
ERROR: Feature(s) not licensed

Rather than fuss with licensing, I figured I’d post what I have and carry on!

Assessed Attacker Value: 5
Assessed Attacker Value: 5Assessed Attacker Value: 3