Catching bugs in VMware: Carbon Black Cloud Workload Appliance and vRealize Operations Manager

Last year we found a lot of exciting vulnerabilities in VMware products. The vendor was notified and they have since been patched. This is the second part of our research. This article covers an Authentication Bypass in VMware Carbon Black Cloud Workload Appliance (CVE-2021-21978) and an exploit chain in VMware vRealize Operations (CVE-2021-21975, CVE-2021-22023, CVE-2021-21983) which led to Remote Code Execution.

VMware Carbon Black Cloud Workload Appliance

Our story begins with a vulnerability in the VMware Carbon Black Cloud Workload Appliance, where we managed to bypass the authentication mechanism and gain access to the administrative console.

The appliance is hosted on-premise and is the link between an organization’s infrastructure and VMware Carbon Black Cloud, which is endpoint protection platform.

By checking the ports available on 0.0.0.0 using the netstat command, we found a web-application on port 443.

The front-end server was an Envoy proxy server. Upon looking into its configuration file, we determined that further requests are proxied to tomcat-based microservices.

Excerpt from config /opt/vmware/cwp/appliance-gateway/conf/cwp-appliance-gateway.yaml:

node:
  cluster: cwp_appliance
  id: cwp-appliance-v1-2020
static_resources:
  clusters:
    -	
      connect_timeout: 5s
      hosts:
        -
          socket_address:
            address: "127.0.0.1"
            port_value: 3030
      lb_policy: round_robin
      name: service_vsw
      type: LOGICAL_DNS
    -
      connect_timeout: 5s
      hosts:
        -
          socket_address:
            address: "127.0.0.1"
            port_value: 3020
      lb_policy: round_robin
      name: service_apw
      type: LOGICAL_DNS
    -
      connect_timeout: 5s
      hosts:
        -
          socket_address:
            address: "127.0.0.1"
            port_value: 3010
      lb_policy: round_robin
      name: service_acs
      type: LOGICAL_DNS

Discovery of Java services utilizing `netstat` command

After studying the application.yml configuration file for the service, which is called service_acs and runs on port 3010, we found that a role-based access model from the Java Spring framework is implemented.

// application.yml
rbacpolicy:
  role:
  - name: SERVICE_USER
    description: This role gives you access to all administration related work
    default: DENY
    permissions:
     - '*:*'

  - name: APPLIANCE_USER
    description: This role gives you access to all administration related work
    default: DENY
    permissions:
     - 'acs:getToken'
     - 'acs:getServiceToken' 
     - 'apw:getApplianceDetails'
     - 'apw:getApplianceSettings'
     - 'apw:getNetworkConf' 
…

A cursory examination of the role policy raises many questions:

What is a service user?
Why does it have unlimited capabilities?
What does the getServiceToken API method do?

We decided to start by exploring the getServiceToken API method. Opening the source code, we studied the description of this method. “Generate JWT Token for Service Request” meaning that every time an application needs authentication for an internal API method call, it accesses this API and receives an authorization token.

An excerpt from TokenGeneratorApi.java:

@ApiOperation(
      value = "Generate JWT Token for Service Request",
      nickname = "getServiceToken",
      notes = "",
      response = AccessTokenDTO.class,
      tags = {"TokenGenerator"}
   )
   @ApiResponses({@ApiResponse(
   code = 200,
   message = "OK",
   response = AccessTokenDTO.class
…
   @RequestMapping(
      value = {"/api/v1/service-token/{serviceName}"},
      produces = {"application/json"},
      method = {RequestMethod.GET}
   )
   ResponseEntity<AccessTokenDTO> getServiceToken(@ApiParam(value = "name of the service which is requesting token",required = true) @PathVariable("serviceName") String serviceName);

Let’s try to get the authorization token by accessing the service that is attached to port 3010 from the internal network.

Accessing Java Service API method using cURL

We got a JWT token, which turns out to be for the role of our old friend, the service user.

Decoding of the JWT token payload:

{
  "sub": "any-service",
  "iss": "user-service",
  "nbf": 1645303446,
  "exp": 1731703446,
  "policy": {
    "role": "SERVICE_USER",
    "permissions": {
      "*": [
        "*"
      ]
    }
  },
  "refreshable": false,
  "iat": 1645303446
}

The prospect of being able to generate a token for a super-user without authentication looks very tempting. Let’s try to do the same trick, but this time externally, through the Envoy server.

Attempt to get service token by accessing Envoy server

We failed, although the other API methods of the Java service were available to us. Let’s see how proxying to internal services is organized and study the mechanisms that are responsible for routing.

When using the Envoy proxy server as a front-end server, the routing table can be generated dynamically using the Route Discovery API. To do this, inside the backend service, use DiscoveryRequest and others entities from the io.envoyproxy.envoy.api package to describe the configuration of routes.

An example of creating a /admin/ router using Envoy API:

public String routeDiscovery(final DiscoveryRequest discoveryRequest) {
      ...
      Route admin = Route.newBuilder().setMatch(RouteMatch.newBuilder().setPrefix("/admin/").build()).setRoute(RouteAction.newBuilder().setCluster("admin_cluster").setHostRewrite(this.hostName).build())
      Builder virtualHostOrBuilder = VirtualHost.newBuilder().setName("backend").addDomains("*");
      virtualHostOrBuilder.addRoutes(admin);
      VirtualHost virtualHost = virtualHostOrBuilder.build();
      RouteConfiguration routeConfiguration = RouteConfiguration.newBuilder().setName("route").addVirtualHosts(virtualHost).build();
      DiscoveryResponse discoveryResponse = DiscoveryResponse.newBuilder().setVersionInfo("1").addResources(Any.pack(routeConfiguration)).build();
      TypeRegistry typeRegistry = TypeRegistry.newBuilder().add(DiscoveryResponse.getDescriptor()).add(ClusterLoadAssignment.getDescriptor()).add(RouteConfiguration.getDescriptor()).build();
      String response = null;
      ...
      try {
         response = JsonFormat.printer().usingTypeRegistry(typeRegistry).print(discoveryResponse);
      } catch (InvalidProtocolBufferException err) {
         log.error("Error while serializing response", err);
      }

      return response;
   }

Let’s consider a specific example from the Java service.

An excerpt from EnvoyXDSServiceImpl.java:

package com.vmware.cwp.appliance.applianceworker.service.impl;

@Component
public class EnvoyXDSServiceImpl implements EnvoyXDSService {
...
   public String routeDiscovery(final DiscoveryRequest discoveryRequest) {
...
Route service_token_block = Route.newBuilder()
   .setMatch(RouteMatch.newBuilder()
   .setPrefix("/acs/api/v1/service-token").build())
   .setRoute(RouteAction.newBuilder().setCluster("service_vsw")
   .setPrefixRewrite("/no_cloud").build()).build();

...
Route acs = Route.newBuilder()
   .setMatch(RouteMatch.newBuilder()
   .setPrefix("/acs/").build())
   .setRoute(RouteAction.newBuilder()
   .setCluster("service_acs")
   .setHostRewrite(applianceIPv4Address).build()).build();
...

We see that when we encounter the URL /acs/api/v1/service-token, the application forwards the request to the stub page, instead of passing the request onto the service for processing. At the same time, any URL prefixed with /acs/* will be forwarded to the backend. Our task is to bypass the blacklist and pass the whitelist conditions. A special feature of the Envoy server is required to allow us to do that. We read the documentation and found one interesting point: the Envoy server has disabled normalization by default.

Despite the recommendations of the Envoy developers not to forget to enable this property when working with RBAC filters, the default value often remains unchanged, as it is in this case. Disabled normalization means that URL /acs/api/v1/service-token/rand and /acs/api/v1/%73ervice-token/rand will be treated by Envoy API as non-identical strings, although after normalization by another server, such as tomcat, the urls will be treated as identical again.

It turns out that if we change at least one character in the API-method name to its URL representation, we can bypass the blacklist without violating the whitelist conditions.

We send a modified request and receive a service token.

Done. We now have a service token with super-user privileges, which grants us administrator powers over this software.

VMware vRealize Operations Manager

In the next story we will tell you about the chain of vulnerabilities found in automation software.

Server-Side Request Forgery

We started by investigating the Operations Manager API , and found a couple of methods available without authentication. These included the API-method /casa/nodes/thumbprints, which takes an address as a user parameter. By specifying the address of a remote server under our control as the parameter in HTTP request we receive a GET request from the Operations Manager instance with the URL-path /casa/node/thumbprint.

To control the URL-path completely, we can add the “?” symbol to cut off the path normally concatenated to by the application. Let’s send a request with a custom path:

As a result, we were able to make any GET request on behalf of the application, including to internal resources.

Having been able to make a GET request to internal resources, we tried to make a request to some API methods that are available only to an authorized user. So, for example, we got access to the API method for synchronizing passwords between nodes. When calling this method, we get the password hash of the administrator in two different hashing algorithms – sha256 and sha512.

Obtaining administrator password hash via replication functionality

It is worth saying that the sha family of algorithms is not recommended for password hashing and can be cracked with high chances of success. And since the administrator in the application corresponds to the system admin user on server, if there is a ssh server in the system with a keyless mode of operation, you can connect to the server and gain access to the command shell. To store sensitive data such as a password, it is best practice to use so-called slow hash functions.

Credentials Leak

Despite the high probability of gaining shell access at this stage, the above method is not fully guaranteed and so we have continued our research. It is worth noting how, using SSRF, we gain access to API methods that require authentication. We know of several mechanisms that could provide this functionality and, in this case, not the best approach was chosen. The fact is that every time the API is accessed by the application, it adds a basic authentication header to the request. To extract the credentials from the header, we sent an SSRF request to our remote sniffer, which in response outputs the contents of the http request:

Extracting credentials with HTTP request sniffer.

It appears that the application uses the maintenanceAdmin user to access the API. Let’s try to use these credentials to access the protected API methods directly, without SSRF.

Verifying that account is up and running

Well, now that we have super-user privileges, we’re only one step from taking control of the server. After looking through all the API methods, we found two ways to access the shell.

RCE (Password Reset)

The first and rough approach involves resetting the password for the administrative user using the PUT /casa/os/slice/user API method. This method allows you to change the password for users without additional verification, such as the current password. Since the admin user of the same name exists in the system, it is not hard to connect to the system with its account via SSH.

If SSH is disabled, simply enable it using one of the API methods.

RCE (Path Traversal)

The previous approach involved resetting the administrator password, which can disrupt the customer’s workflow when pentesting. As an alternative approach, we found a way to load a web shell via a path-traversal attack using the /casa/private/config/slice/ha/certificate API method. A lightweight JSP-shell uploaded to the web directory of the server will be used as the web shell.

After uploading, we access the shell at https://vROps.host/casa/webshell.jsp, passing the command in the cmd parameter.

Execution of `id` command on the vROps server

Outro

Thank you for reading this article to the end. We hope you were able to find something useful from our research. Whether you are a developer, a researcher or maybe even the head of PSIRT.

We also would like to highlight that this research resulted in 9 CVEs of varying severities, and each report was handled with the utmost care by the VMware Security Response Center team. We appreciate VMware for such cooperation.