On July 14th, Emil Lerner found and explored new ways of HTTP desync/smuggling exploitation based on HTTP/2 request processing issues. He submitted the bug to the Cloudflare security team through their bug bounty program.

This security issue took Cloudflare a week to fix and was completed on July the 24th. Emil was awarded with a $1’000 bounty, and on August 15th, the company accepted this bug for public disclosure. Here we go.

The nature and impact of HTTP/2 request smuggling

When someone performs a request to a Cloudflare customer’s website via HTTP/2, Cloudflare applies weaker validation after the 100th header before forwarding the request to an upstream. If Cloudflare client’s HTTP server accepts and parses HTTP headers that end with a tab or a space character, this can lead to request/response desynchronization in the HTTP/1.1 caused by initial HTTP/2 attacker’s request.

The issue is reproducible at the www.cloudflare.com domain, which probably uses the same infrastructure as the proxy for customers. Again, practical exploitation of this bug depends on both the Cloudflare proxy issue and client webserver.

The impact of HTTP Request Smuggling includes HTTP Cache poisoning, account hijacking, and more.

We recommend looking at the BlackHat 2019 slides to better estimate the potential risk.

Reproduction

Since the PoC needs to send HTTP/2 requests that are not considered valid by most HTTP libraries, i.e. libcurl, Emil wrote a small Go program that sends such requests using a slightly lower level API, available here: https://github.com/neex/http2smugl 

To run the PoC code, we have to install Golang version 1.13.0 or higher and two dependent packages: github.com/spf13/cobra and golang.org/x/net installed by go get.

The PoC program cfsmulg sends an HTTP/2 request with more than 100 HTTP headers. It accepts the list of additional headers as command-line arguments and adds them after the 100th header. Because of this issue, they will be forwarded unchanged to the upstream.

To reproduce the underlying issue, we have to run the PoC in the following way:

go run cfsmugl.go request <https://some-cloudflare-customer.com/> "a man, 
a plan, a canal : panama"

As a result, the following HTTP request will be received by Cloudflare customer’s backend from Cloudflare upstream:

GET / HTTP/1.1
Host: some-cloudflare-customer.com
Connection: Keep-Alive
Accept-Encoding: gzip
CF-IPCountry: US
X-Forwarded-For: <your-ip>
CF-RAY: 5a2e0232ebd87b87-DME
X-Forwarded-Proto: https
CF-Visitor: {"scheme":"https"}
user-agent: Mozilla/5.0
a man, a plan, a canal : panama
CF-Request-ID: 03509fb3ce00007b87699e2200000001
CF-Connecting-IP: <your-ip>
CDN-Loop: cloudflare

As you can see, HTTP header a man, a plan, a canal : panama is here in an unchanged way.

Note that the header specified in the command line contains spaces, including one at the end of its name, and gets forwarded unchanged. The colons, newlines, and A-Z are still prohibited in the smuggled headers, but spaces and tabs are enough to perform the attack in some cases.

Exploitation

In order to trigger HTTP desynchronization in the keep-alive HTTP connection between Cloudflare and its customers, an attacker can use something like transfer-encoding : chunked (note the space before the colon). Cloudflare will forward it as is, and the customer’s software might still interpret it. According to HTTP specification, transfer-encoding takes precedence over content-length, which means that it is possible to perform HTTP request smuggling type CL.TE. However, instead of Content-Length, it’s the length of http2 stream.

Let’s demonstrate the problem in the case of www.cloudflare.com:

If we run 

go run cfsmugl.go request <https://www.cloudflare.com/> 
"transfer-encoding : pizza"

we’ll receive a 503 error, because the upstream is probably confused with the request of decoding the imaginary pizza transport encoding.

If we run 

go run cfsmugl.go request <https://www.cloudflare.com/> 
"transfer-encoding : chunked"

then the request will hang, as the upstream expects a body that will never arrive.

If we disable sending an additional 100 headers using -skip-magic-headers, then switch to the program 

go run cfsmugl.go request <https://www.cloudflare.com/> 
"transfer-encoding : chunked" --skip-magic-headers

We will receive the usual 200 responses as the issue is not triggered.

There’s detect subcommand in the utility, which provides different ways to detect that a client indeed parses and uses a header with a space at the end.

Mitigation and further research

This bug was fixed by the Cloudflare team in a week. At the same time, we can expect to meet similar issues in different load balancers, CDNs, and hosting providers with HTTP/2 support. Moreover, there recently was another one way of HTTP/2 request smuggling based on an upgrade to h2c https://labs.bishopfox.com/tech-blog/h2c-smuggling-request-smuggling-via-http/2-cleartext-h2c was published.

To sum it all up, smuggling attacks, including desync, is major for the modern web and they are relevant for both HTTP/1.1 and HTTP/2.

If you are looking for NGWAF with a native HTTP/2, REST, gRPC, GraphQL and WebSockets support, we recommend to consider Wallarm.