December 5, 2024

The concept of a RACE condition and its potential for application vulnerabilities is nothing new. First mentioned back in the 1950’s, the issues associated with asynchronous applications have been long realized.

However, it’s never a bad time for a refresher, especially when researchers continue to make new advancements related to RACE conditions over half a century later!

In the following article, we will explore some of the newest research related to web application RACE conditions.

What Are RACE Conditions?

RACE conditions occur when we have multiple code components operating simultaneously in a shared environment. Depending on the time each component takes to finish, the final output may be unpredictable. In any environment where we have shared data and concurrent processes, a RACE condition may be present.

Let’s look at a basic example.

The following Python code uses the threading module to create two threads. Each thread executes the “increment_counter” function, which increments a shared value until it reaches 100,000. You’ll also notice a call to the “sleep()” method. This mimics the delays complex environments may have.

import threading
import time

# Shared counter variable
counter = 0

# Number of increments each thread will perform
increments = 100000

def increment_counter():
    global counter
    for _ in range(increments):
        temp = counter
        # Introduce a tiny delay to increase the chance of a context switch
        time.sleep(0.000001)
        temp += 1
        counter = temp

# Creating two threads
thread1 = threading.Thread(target=increment_counter)
thread2 = threading.Thread(target=increment_counter)

# Starting the threads
thread1.start()
thread2.start()

# Waiting for both threads to complete
thread1.join()
thread2.join()

# Printing the final value of the counter
print(f"Final counter value: {counter}")

Now, at first glance, this code seems simple. It should output a final counter of 100,000. However, if we run this code several times, we’ll notice something interesting. Every so often, the code will output a final count of 100,001. This is a result of a RACE condition.

The slight time delay, combined with the processing speeds of our computers, causes enough delay to lead to this unexpected behavior. However, these outputs are unpredictable and do not occur on every run. It is for this reason that RACE conditions can be so challenging to identify.

While we like to think of code as immutable, the truth is numerous factors impact how our applications run. These unexpected factors can quickly make secure code operate in a manner that can introduce vulnerabilities.

Testing For RACE Conditions In Modern Web Applications

Given how simple it is to introduce a RACE condition into an application, why aren’t they more common in web applications? After all, most web applications exist with built-in concurrency, creating the perfect environment for a RACE condition. Truth be told, these issues have always existed, but reliable testing has proved challenging.

While tools like BurpSuite’s “Turbo Intruder” could be used to target some RACE conditions, factors such as network latency greatly impact the overall success and repeatability of these attacks. In many cases, the standard latencies introduced by the internet and the application itself have made these vulnerabilities difficult to test and identify. As such, developers don’t always consider the potential impact of these issues. Even if an attack is possible to identify, reliable exploitation is not always possible.

Last year, however, PortSwigger released a new white paper titled “Smashing the state machine: the true potential of web race conditions,” which introduced a novel new method. Leveraging the “last-byte sync” and “single-packet attack,” PortSwigger created methods for both HTTP 1.1 and 2 to exploit RACE conditions. Benchmark testing from Portswigger showed it was possible to send over 20 requests with a standard spread of only four microseconds.

This combination of speed and consistency has greatly increased the effectiveness and reliability of RACE condition testing and has helped to uncover an entirely new attack surface.

RACE Conditions In The Wild

Since Portswigger’s white paper release, GuidePoint has found great success in identifying these impactful RACE conditions and leveraging these new attack methods within our client’s environment. Together we’ll explore several of the most common and more impactful examples of RACE conditions.

Account Lockout Bypass

When looking for a RACE condition, a good starting place is any area of the application where a logical restriction is employed. A great example of this is an account lock-out feature. In these situations, we have an arbitrary limit designed to restrict an operation. If we think about this in basic pseudo-code, it may look like this.

allowed_failed_logins = 3

username_value = get_request_username()
password_value = get_request_password()

# check_account_failed_login_count() returns an integer
if check_account_failed_login_count(username_value) < allowed_failed_logins:
	# attempt_login() will attempt to authenticate the user and return a new session if succesful
	session_data = attempt_login(username_value,password_value)
	if session_data is not None:
		return session_data
	else:
		increment_failed_login_count_by_one(username_value)
		return "Username or Password Incorrect"
else:
	return "Account Locked"

In a basic test environment, this code operates exactly as expected. But, when placed in an environment where rapid asynchronous requests are processed, we can see where issues may arise.

Depending on the time it takes to process a login and update the “failed login count,” we may be able to submit multiple login requests before the lockout count is fully updated. This creates an exploitable state. Assuming our requests are processed before the “increment_failed_login_count_by_one” function, we can attempt more logins than allowed.

Now, the largest delays are going to be any database operations like those that may be present in the  “check_account_failed_login_count” and the “attempt_login” functions. Using the single-packet attack method, we can potentially push ten or more requests to the application, which will begin processing before any of the early requests even get to the “increment_failed_login_count_by_one” function.

To better demonstrate this, we’ve created the following graphic:

As we can see, we have a total of six requests. Each is processed in its own thread, but notably, the first five requests enter the login process before any of the “increment_failed_login_count_by_one” functions are finished processing.

This means the initial lock-out check will show that the total lockout count is less than the maximum allowed amount as these functions either haven’t been completely resolved or, in the case of login requests four and five, the application has already moved past the check. Since the incrementation process did not occur fast enough, we are able to “RACE” past it. While requests four and five should not be processed, the lock-out value has not been incremented fast enough to reflect the actual lock-out count.

The only request that would be blocked in this example is request six. This is because the app would have had enough time to increment and resolve the first three lockout functions.

Unexpected Error Messages

Given the unstable impact of these vulnerabilities, RACE conditions can also lead to unexpected application behaviors. In some cases, RACE conditions may overload an application in ways that developers may not have expected. When combined with other misconfigurations, this may lead to serious information disclosures.

During an assessment of an API, GuidePoint identified one such issue that required the single-packet attack to reliably trigger.

In this case, the API itself was operating as an interface for a secondary system. To reduce traffic and exposure on a secondary system, the API would take a user’s request, parse it, apply various controls, and then potentially pass the request to a secondary server. The following diagram demonstrates the process.

When looking at the authentication endpoint, GuidePoint attempted to bypass an account lock-out feature. Using the single-packet attack to send fifteen requests simultaneously, GuidePoint accidentally discovered a flaw in the authorization server, leading to an unexpected response from the secondary authentication server.

For the first several requests, the API responded as expected. However, the requests that had been processed later pushed the secondary server into an unexpected state. It appeared as if an intended function of the API was to act as a buffer for failed logins; as such, the secondary server was not configured to handle logins past the expected lockout amount. This caused the secondary authentication server to return a verbose error message, which was packaged and forwarded by the API.

This led to the following response in about half the requests.

REQUEST:
POST /QA/authentication HTTP/2
Host: [REDACTED]
Cache-Control: no-store
Content-Type: application/json
Accept: */*
Accept-Encoding: gzip, deflate
Content-Length: 66

{
  "username": "[REDACTED]",
  "password": "Password123qwer"
}

RESPONSE:
HTTP/2 200 OK
Date: [REDACTED]
Content-Type: application/json
Content-Length: 1242

{"message":"Request failed with status code 400","name":"Error","stack":"[REDACTED]","config":{"url":"[REDACTED]","method":"post","data":"username=[REDACTED]&password=[REDACTED]&grant_type=password&scope=openid%20profile%20offline_access","headers":{"Accept":"application/json, text/plain, */*","Content-Type":"application/x-www-form-urlencoded","Authorization":"Basic [REDACTED]","User-Agent":"[REDACTED]","Content-Length":[REDACTED]},"transformRequest":[null],"transformResponse":[null],"timeout":0," [REDACTED]":"[REDACTED]","[REDACTED]":"[REDACTED]","maxContentLength":-1}}

Due to a combination of poor error and concurrency handling, the server returned sensitive authentication information pointing to the organization’s internal Okta instance, as well as authorization headers used by that server.

Attempts to exploit this without the “single-packet” attack returned the expected API responses.

Bank Account Overdraw

As with the other RACE conditions, GuidePoint continually looks to identify areas where an application may enter a checked state that can be circumvented, and, in a recent assessment of a pre-release B2B application, GuidePoint found just that.

Reviewing the application, it was determined that an account could not overdraw their account. As a B2B application, the financial institutions decided that corporate entities should not have an overdraw feature. Attempts to circumvent this control with negative values and manual interception proved fruitless until the single-packet attack method was utilized.

Our hope when launching the attack was that the function checking the balance was called before the function updating the account balance and that those functions did not utilize datastore concurrency to lock each individual transaction.

With an account that had a positive balance, we created a group of transfer requests designed to overdraw the account if a RACE condition was present. Starting with a total of ten dollars, we sent six individual transfers, each with a transfer amount of $5.01. If the system didn’t contain a RACE condition, the application would have stopped processing the requests after the first request.

However, five transfer requests were processed and fulfilled. This led to the destination account having just over twenty dollars and the source account having a negative balance.

In the following diagram, we can see what happened in this attack.

In this instance, the server received a total of six requests attempting to transfer $5.01 to a destination account. Due to the single packet attack, multiple requests were processed before the application had a chance to update the source accounts balance after each transfer. Due to a lack of data store concurrency, the function checking the source account’s balance did not reflect the true amount because it didn’t have time to update.

It wasn’t until the final sixth request arrived after the first update balance. At that point, the balance showed that the account did not have enough funds, and the transfer was denied. Unfortunately, five other transfer requests had already occurred.

At best, this created a means for businesses to engineer their own on-demand zero-percent loans. In its worst form however, this could have been used to seriously defraud both the financial institution and the businesses using the B2B application.

Imagine a threat actor gaining hold of an account with over a million dollars. Using this method, an attacker could have transferred over five million dollars, which the business did not have, into an account. From here, they could have wired this money out. Had such an attack been successful, the business victim would be out their million dollars, and the financial institution another four.

We are pleased to say, however, that the organization fully understood this risk and quickly remediated this issue.

Mitigating RACE Conditions Within Your Own Applications

The mitigation of RACE conditions can be challenging. Not only are fixes not always straightforward, but identifying these issues can also be difficult. These types of issues generally require a combination of secure code reviews and application architecture reviews, both of which can be further strengthened with regular application security assessments and threat modeling.

Some of these solutions, such as application architecture reviews and threat modeling, can help to identify potential areas where RACE conditions may exist proactively. With this knowledge, developers can carefully design their systems and applications to be atomic, ensuring they make use of concurrency control features and even avoid implemented components that may introduce the risk of a RACE condition. Other solutions, like secure code reviews and application security assessments, are a great way to identify existing flaws. These manual analyses will help not only to identify the potential for these issues but also help to realize their full impact. Once understood, these issues can be remediated more effectively.