This post by Rich Warren and Sander Laarhoven discusses NCC Group observed in the wild exploitation attempts and detection logic for the F5 BIG-IP/BIG-IQ iControl REST API vulnerabilities CVE-2021-22986.
On the 10th of March 2021, F5 released an advisory for it’s BIG-IP and BIG-IQ products, stating that the REST interface of the iControl management interface is vulnerable to an authentication bypass and remote code execution [1].
These vulnerabilities were given the following CVE numbers: CVE-2021-22986, CVE-2021-22987, CVE-2021-22988, CVE-2021-22989 and CVE-2021-22990.
No detection rules or artifact information was initially provided by F5, albeit no public exploit was known at the time F5’s advisory was published, giving system administrators time to patch and blue teams the space to research and implement detection capabilities.
In the week that followed, several researchers posted proof-of-concept code after reverse engineering the Java software patch in BIG-IP [2].
Starting this week and especially in the last 24 hours (May 18th, 2021) we have observed multiple exploitation attempts against our honeypot infrastructure. This knowledge, combined with having reproduced the full exploit-chain we assess that a public exploit is likely to be available in the public domain soon.
NCC Group believes it is in the best interests of all to release our internal notes and detection logic to prevent further harm once public exploits become available.
Exploitation of this vulnerability requires two steps. First, authentication has to be bypassed by leveraging the SSRF vulnerability to gain an authenticated session token. This authenticated session can then be used to interact with REST API endpoints, which would otherwise require authentication.
The most useful endpoint for an attacker is the tm/util/bash
endpoint, which allows an (authenticated) user to execute commands on the underlying server with root privileges. However, as the REST API is designed for remote administration, there are many endpoints which an attacker might wish to take advantage of.
As part of the F5 patches, a command injection vulnerability was also patched in the tm/access/bundle-install-tasks
REST endpoint – which could be used as an alternative way to execute arbitrary commands once authentication has been bypassed.
This blogpost will be updated as more detection methods are researched and identified.
Logs from BIG-IP and BIG-IQ appliances are stored in /var/log/
and are rotated by default.
A successful exploitation of SSRF to extract a cookie can be detected in the restjavad-audit
log. This log shows when a user logs in via the REST API:
==> restjavad-audit.0.log <== [I][310][18 Mar 2021 12:37:32 UTC][ForwarderPassThroughWorker] {"user":"local/null","method":"POST","uri":"http://localhost:8100/mgmt/shared/authn/login","status":200,"from":"<SRC IP HERE>"}
Note: Legitimate successful logins via the REST API look the same:
==> restjavad-audit.0.log <== [I][317][18 Mar 2021 12:45:36 UTC][ForwarderPassThroughWorker] {"user":"local/null","method":"POST","uri":"http://localhost:8100/mgmt/shared/authn/login","status":200,"from":"<SRC IP HERE>"}
Failed SSRF exploitation attempts can be found in the restjavad log file when the provided endpoint does not exist (in this case, “AVSMRRWLVVYJ” was provided as the loginReference.link parameter):
==> restjavad.0.log <== [I][312][18 Mar 2021 12:38:45 UTC][8100/shared/authn/login AuthnWorker] Cookie failed to login from <SRC IP HERE> using the AVSMRRWLVVYJ authentication provider [I][313][18 Mar 2021 12:38:45 UTC][RestRequestReceiver] Rest worker uri not found: /mgmt/shared/authn/login
Traces of bash invocations via the REST API can be found in the restjavad-audit log.
Default log verbosity configuration shows when a user sends a POST to the REST bash endpoint:
==> restjavad-audit.0.log <== [I][311][18 Mar 2021 12:37:33 UTC][ForwarderPassThroughWorker] {"user":"local/f5admin2","method":"POST","uri":"http://localhost:8100/mgmt/tm/util/bash","status":200,"from":"<SRC IP HERE>"}
Successful commands also appear in the audit log file:
==> audit <== Mar 18 05:39:28 localhost notice icrd_child[10680]: 01420002:5: AUDIT - pid=10680 user=f5admin2 folder=/Common module=(tmos)# status=[Command OK] cmd_data=run util bash -c "set -e ; id"
Note that the example user f5admin2
is legitimate and not created during the exploit chain. The username of an existing user must be known by an attacker.
Some further failed attempts were observed in the wild by our honeypot infrastructure. Fortunately the attackers got the details of the exploit wrong and attempted a chain of events that could not possibly work.
[f5admin2@bigip1:Active:Standalone] var # tail -f /var/log/restjavad.0.log [I][289][17 Mar 2021 15:14:30 UTC][8100/shared/authn/login AuthnWorker] Cookie failed to login from <SRC IP HERE> using the WGZBBJYKZZBZ authentication provider [I][290][17 Mar 2021 15:14:30 UTC][RestRequestReceiver] Rest worker uri not found: /mgmt/shared/authn/login [I][294][17 Mar 2021 18:27:37 UTC][8100/shared/authn/login AuthnWorker] Cookie failed to login from 193.37.214.11, 127.0.0.1 using the http://localhost/mgmt/tm/access/bundle-install-tasks authentication provider [I][295][17 Mar 2021 18:27:37 UTC][RestRequestReceiver] Rest worker uri not found: /mgmt/shared/authn/login [I][297][17 Mar 2021 18:56:10 UTC][8100/shared/authn/login AuthnWorker] Cookie failed to login from 193.37.214.11, 127.0.0.1 using the http://localhost:8100/mgmt/tm/util/bash authentication provider [I][299][17 Mar 2021 19:29:12 UTC][8100/shared/authn/login AuthnWorker] Cookie failed to login from 193.37.214.11, 127.0.0.1 using the http://mj9afbfzgjzg1ktuig0p1oxwun0ex2m.burpcollaborator.net/mgmt/tm/util/bash authentication provider
The restjavad log can be configured to be a bit more verbose [3]. When log level is set to FINEST
, we get the following logs in /var/log/restjavad.0.log
under successful exploitation.
Notice “User null successfully logged in”:
[F][2769][18 Mar 2021 12:59:53 UTC][RestRequestReceiver] stage ReadRequest, id 11661, target null [F][2770][18 Mar 2021 12:59:53 UTC][RestRequestReceiver] stage DeserializeBody, id 11661, target http://localhost:8100/mgmt/shared/authn/login [F][2771][18 Mar 2021 12:59:53 UTC][RestRequestReceiver] stage IdentifyUser, id 11661, target http://localhost:8100/mgmt/shared/authn/login [F][2772][18 Mar 2021 12:59:53 UTC][RestRequestReceiver] stage Dispatch, id 11661, target http://localhost:8100/mgmt/shared/authn/login [F][2773][18 Mar 2021 12:59:53 UTC][8100/shared/authn/login AuthnWorker] User null successfully logged in from <SRC IP HERE> using the local authentication provider. [F][2774][18 Mar 2021 12:59:53 UTC][8100/shared/authz/users UsersWorker] patch f5admin2: no changes [F][2775][18 Mar 2021 12:59:53 UTC][8100/shared/authn/login AuthnWorker] Updated user https://localhost/mgmt/shared/authz/users/f5admin2 [F][2776][18 Mar 2021 12:59:53 UTC][RestRequestReceiver] stage SendResponse, id 11661, target http://localhost:8100/mgmt/shared/authn/login [F][2777][18 Mar 2021 12:59:53 UTC][RestRequestReceiver] stage Finish, id 11661, target http://localhost:8100/mgmt/shared/authn/login [F][2778][18 Mar 2021 12:59:53 UTC][RestRequestReceiver] stage ReadRequest, id 11682, target null [F][2780][18 Mar 2021 12:59:53 UTC][8100/shared/authz/mcp_enumerator McpUsersEnumeratorWorker] Syncing users from MCP. [F][2781][18 Mar 2021 12:59:53 UTC][8100/shared/authz/mcp_enumerator McpUsersEnumeratorWorker] Syncing admins from MCP. [F][2782][18 Mar 2021 12:59:53 UTC][8100/shared/authz/mcp_enumerator McpUsersEnumeratorWorker] Complete worker startup. [F][2783][18 Mar 2021 12:59:53 UTC][8100/shared/authz/mcp_enumerator McpUsersEnumeratorWorker] Schedule syncing users/admins from MCP.
Whereas the following logs show a legitimate succesful login via the REST API:
[F][2922][18 Mar 2021 13:01:19 UTC][RestRequestReceiver] stage ReadRequest, id 11881, target null [F][2923][18 Mar 2021 13:01:19 UTC][RestRequestReceiver] stage DeserializeBody, id 11881, target http://localhost:8100/mgmt/shared/authn/login [F][2924][18 Mar 2021 13:01:19 UTC][RestRequestReceiver] stage IdentifyUser, id 11881, target http://localhost:8100/mgmt/shared/authn/login [F][2925][18 Mar 2021 13:01:19 UTC][RestRequestReceiver] stage Dispatch, id 11881, target http://localhost:8100/mgmt/shared/authn/login [F][2926][18 Mar 2021 13:01:19 UTC][8100/tm/auth PublicWorker] Forwarding operation: /tm/auth/source [F][2927][18 Mar 2021 13:01:19 UTC][ProcessManager] Request queued: URI: /tm/auth/source [F][2928][18 Mar 2021 13:01:19 UTC][ChildWrapper] Child 2aa19864-5327-4a61-846e-d322671698d0 state changed from IDLE to BUSY [F][2929][18 Mar 2021 13:01:19 UTC][ChildWrapper] Forwarding GET request http://localhost:8100/tm/auth/source to http://localhost:51812/tmapi_mapper/auth/source [F][2930][18 Mar 2021 13:01:19 UTC][RestRequestSender] stage Connect, id 11891, target http://localhost:51812/tmapi_mapper/auth/source [F][2931][18 Mar 2021 13:01:19 UTC][RestRequestSender] send 11891 localhost /tmapi_mapper/auth/source [F][2932][18 Mar 2021 13:01:19 UTC][RestClientConnectionManager] [localhost:51812] pending total: 0, allocated total: 1, creating for: http://localhost:51812/tmapi_mapper/auth/source [F][2933][18 Mar 2021 13:01:19 UTC][RestRequestSender] stage WriteRequest, id 11891, target http://localhost:51812/tmapi_mapper/auth/source [F][2934][18 Mar 2021 13:01:19 UTC][RestRequestSender] stage ReadResponse, id 11891, target http://localhost:51812/tmapi_mapper/auth/source [F][2935][18 Mar 2021 13:01:19 UTC][RestRequestSender] stage DeserializeBody, id 11891, target http://localhost:51812/tmapi_mapper/auth/source [F][2936][18 Mar 2021 13:01:19 UTC][RestRequestSender] stage Finish, id 11891, target http://localhost:51812/tmapi_mapper/auth/source [F][2937][18 Mar 2021 13:01:19 UTC][RestRequestSender] done 11891 [F][2938][18 Mar 2021 13:01:19 UTC][RestClientConnectionManager] [localhost:51812] isClose = true isError = false pending req = false. pending total: 0, allocated total: 0 [F][2939][18 Mar 2021 13:01:19 UTC][Connection] client cxn close localhost/127.0.0.1:51812 - client op error=false [F][2940][18 Mar 2021 13:01:19 UTC][ChildWrapper] Success response to GET request at http://localhost:8100/tm/auth/source [F][2941][18 Mar 2021 13:01:19 UTC][ChildWrapper] Child 2aa19864-5327-4a61-846e-d322671698d0 state changed from BUSY to IDLE [F][2942][18 Mar 2021 13:01:19 UTC][8100/shared/authn/login AuthnWorker] User f5admin2 successfully logged in from <SRC IP HERE> using the local authentication provider. [F][2943][18 Mar 2021 13:01:19 UTC][8100/shared/authz/users UsersWorker] patch f5admin2: no changes [F][2944][18 Mar 2021 13:01:19 UTC][8100/shared/authn/login AuthnWorker] Updated user https://localhost/mgmt/shared/authz/users/f5admin2 [F][2945][18 Mar 2021 13:01:19 UTC][RestRequestReceiver] stage SendResponse, id 11881, target http://localhost:8100/mgmt/shared/authn/login [F][2947][18 Mar 2021 13:01:19 UTC][RestRequestReceiver] stage Finish, id 11881, target http://localhost:8100/mgmt/shared/authn/login [F][2948][18 Mar 2021 13:01:19 UTC][RestRequestReceiver] stage ReadRequest, id 11912, target null [F][2949][18 Mar 2021 13:01:23 UTC][ErrorPassingScheduledThreadPoolExecutor] Cancelled runnable. Runnable java.util.concurrent.ScheduledThreadPoolExecutor$Sc
Network rules (Suricata) are available for download. Note that by default all communication is encrypted over TLS, depending on your network monitoring setup, TLS offloading or a proxy setup might be required for these rules to be effective.
https://github.com/nccgroup/Cyber-Defence/blob/master/Signatures/suricata/2021_03_cve_2021_22986.txt
[1]
[2] Exploits