Last year I did some research on how an exposed ~/.ssh/ folder on a web server can lead to a complete pwnage. Here's the deal:

  • I've seen it in the wild.
  • On real web servers.
  • Operated by big companies.

As this becomes an easy way to RCE your web server, I'm here to share my knowledge so you can protect yourself.

The folder ~/.ssh/ is commonly the place where the SSH client and server store some (configuration) files:

  • The SSH server uses the ~/.ssh/authorized_keys file to check what public key is allowed to connect.
  • The SSH client uses ~/.ssh/id_* (i.e. id_rsa, id_dsa, id_ed25519) to store the default public/private keypairs.
  • The SSH client uses ~/.ssh/known_hosts to remember to which hosts it has connected and what their hostkey is.
  • The SSH client uses ~/.ssh/config to configure connections based on hostnames, i.e. to set a different username.

As most servers on the internet run a SSH server to be remotely administratable, there are certainly some web servers that do so as well.

Furthermore, there are web servers where the www-data user has those files. For some of them, the www-data's home directory is the DocumentRoot, so that the ~/.ssh/ becomes accessible.

This becomes an issue when the www-data account was used to run SSH to:

  • clone a repository via SSH
  • push "hot fixes" back into a remote repository over SSH
  • allow developers to connect to the server as www-data directly
  • perform any other SSH based operation

It's 2020 and most admins and developers already know that password based logins are bad and thus have changed to public key cryptography. However, that means that you need a private/public keypair on the web server if you want to connect somewhere.

Let's explore the different files and what information they leak to an attacker. As the files reside in the www-data's home directory, the attacker only needs to send a few requests to check if the files exist:

  • GET /.ssh/id_rsa et al.
  • GET /.ssh/authorized_keys
  • GET /.ssh/known_hosts
  • GET /.ssh/config

id_* private keys

The crown jewels are the private keys. SSH handles different key types and thus there are a few file names that need to be checked:

  • id_rsa
  • id_dsa
  • id_ecdsa
  • id_ed25519

To check if the obtained file is a valid key, one can check for the -----BEGIN OPENSSH PRIVATE KEY----- and -----END OPENSSH PRIVATE KEY----- strings in it. That's easy.

At this point the adminstrators need to rely on the 2nd line of defense: Password protection. When generating a keypair, ssh-keygen asks you for a passphrase. From the data I've gathered, not more than 45% of the keys had a passphrase set. In that case, the attacker must resort to brute force to crack get an usable private. Although offline brute-force attacks against short passwords are getting more and more efficient and cost-effective. Leaked keys should be considered compromised and replaced even if they were password protected. It's just a matter of time until it's cracked.

That means an attacker has a 50/50 chance to find a keypair without a passphrase. Here's an easy way to verify if a password is set or not:

The following key has no password, so ssh-keygen shows the public part:

[gehaxelt@LagTop tmp]$ echo "nopw" | ssh-keygen -y -f /tmp/testkey_nopw; echo $? 
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDChfRt3oUp//tmtBN7Unb6fITrsVt/UT6s6tshhwUMn6d+KJNXjuwSPUj5Hl5jGMPTaWm8wHqbg/bPTZsrBTxA1UAwIcPbo267S3wHUzWNIr/zt+cAEHV94gfnjKEZG6d9SRREZwxeVRobCWpP6gTY3z+2ZhGSlFasbWUHdSikPjZPB73Wn/GsTISXveL5+qj/iTTMpkBU5httK/2O10OzqpCP5/0U8nb089DHYX6QO68zW7wtZ8JpumuK5ogcOBzYEMHSSvY6jDOtRY3lOCQE2bebHxvoCni3jc+Q4prpaOciz/o2fgjclfsz9KrplWIfmgySDw9H2UZrfZOanKqG6HpEOKu8qIrUgX3BwewV3DoSyjHTMVktrTGaWmfYIRMaGGYjqoDLXhizZDy+6j4ywi3q5ikSg5p0ZOYQhUfpj7L1NdY1vXI9TA9PhPtvbclPAbu5fQ8KTpk3GD+zYeGhD6OT1l22BWWlI/yljBh6mEDySaqSMEsewqYIB40xjrE= gehaxelt@LagTop
0

If the key is password protected, then this operation (rightfully) fails:

[gehaxelt@LagTop tmp]$ echo "nopw" | ssh-keygen -y -f /tmp/testkey_pw; echo $? 
Load key "/tmp/testkey_pw": incorrect passphrase supplied to decrypt private key
255

At this stage the attacker has done the hardest part of the exploit chain and successfully obtained a valid SSH keypair.

However, the attacker still does not know if the obtained key will allow her to connect to the server. Luckily there is a file for that: The authorized_keys contains all public portions of private keys that are allowed to connect the user to the specific server.

The format is pretty simple and super easy to verify if the downloaded /.ssh/authorized_keys file is valid or not:

ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6MyNlJORnZ4txf3HuVRDTkRGAJ/KkWDOvDSFHja3mUX0KP+4b8qUDIRSAZ/MMxClbRxZ/raa4vxBWbUa2GHdyHGC6u/iOqJkpKaFihrsujS1gmIcypuPTFEQN6DY2Gf3cQqKmUUATuOkqTBaICzvQQJX7b3bbil/dPfuxbFb8xHp5mZ0EsMt/N7onXLaoWM8nRr9IfvviP+tqXjzYRJ2Ys6XDF4qL9V1W+64HPfb3fnCXID1iMVk4RR8A9Sb9VPKA1f/k5Y10CPB16+ZvU0gXcUA8oKXJedcIXPyFhgsqEEbi7QFg4oDLdAzpQnFEAtUcfparO3GFfO2LB/lLWclX travel@travel-pc

Most entries in that file begin with ssh-rsa, ssh-dss, ssh-ed25519, ssh-ecdsa. Assuming that most SSH keys have a sufficient amount of bits (usually 4096), trying to factor the public keys might not be the best idea.

However, the attacker will get an overview of how many (different) keys (and developers/admins) have access to the server and she could use the obtained information to conduct targeted attacks against specific developers with the goal to steal their matching private key.

In some cases the attacker will be lucky enough to find the obtained private key's public key in the list of authorized keys.

Now she only needs to guess the right username, which usually is any of those:

  • www-data
  • nginx
  • apache
  • root
    In some cases the ~/.ssh/config might provide some usernames or hints.

If the attacker succeeds, she'll have full-fledged SSH access to the web server and be able to run any command. I've seen at least 15 hosts where this was possible, but I won't name any companies here. It's literally GAME OVER and all other (web) security is defeated.

known_hosts

But it gets even more interesting if the ~/.ssh/known_hosts file is exposed, too. If that file exists, it means that the private key on the server was used to connect to other systems.

Depending on the host names or IP addresses in that file, the attacker could use this information and the private key to compromise further systems. For example, she might be able to connect to github/gitlab/etc. or staging, debugging, database systems, etc.

Unfortunately for the attacker, the entries in the known_hosts file can be hashed if HashKnownHosts yes is set. It then might look similar to this and can be best identified by:

  • ssh-rsa
  • ssh-dss
  • ecdsa-sha2-nistp256
  • ssh-ed25519
raspberrypi.local ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBL7M2qw8pKPjyT0Pq1Mbq/wDsVfjpU846u0n5Pqmub2E96VKvLu8LUJz8dhqZi7bLzlCVgkDNkPmY30mZU3+oR0=

|1|Nbo4zQcI82GnFf/IgsQtQB3MZDQ=|eabVTCOPm51+nF+OvoTJF/qkU4A= ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAvJ3c8N+oIKcoFty8MboBB2oKjfpTyfa7xb6mXntYUjiisJ+tyGqOhU7bn5EkW412lg6Fze6JnW2UvEXq2YlmqOXDSzS7QqsJS1/syhytiYI+q0uPbECj3qPXE3QLuBoYI7rzDahiYi3dwlLVNu9n74uBI+ykCFi97/hVdslgDVb1CCR+6SMXhqwnUh5LC+ULorkFiFtzTiniKw6X8PIjwkaViCdmaHJRLKQwskhnDfUzrzb7rEwqx977KZetXE44oc7jHNlSRKhQQgOGpQn7mYa+OQMpnU432iFb63Mb85hrBj0npIfOP6zkJvjoUcANSpmzt/38yvy/G5xAEAEPXQ==

In the latter case the hostname is hashed and cannot directly be read by the attacker. The raspberrypi.local might not be an interesting target, because the IP address isn't known, but the attacker could use the previously compromised web server as a jump host. But it is better for the attacker to find systems with publicly routable IP addresses which she can reach from her system using the compromised private key.

However, the hashing algorithm is only HMAC-SHA1 and modern graphic cards are quite fast at brute forcing such hashes:

speedtest-hmac-sha1

As most IP addresses are IPv4 whose format is pretty simple [0-9]{1,3},[0-9]{1,3},[0-9]{1,3},[0-9]{1,3}, the key space for a brute force attack is not too big (around 2^32), but still enough to slow down an attacker quite a bit as each known_hosts entry has its unique salt. I assumed that I was not the first one to come with the idea of cracking those known_hosts entries, so I google and found a promising github repo.

Using the mask attack provided by the repository, I was able to crack shy of 50% of the entries:
known_hosts-bf

That's quite good chances to find other systems that are hackable by the attacker.

config

The SSH client's configuration file usually contains information such as:

  • Hostnames
  • Aliases
  • Usernames
  • Ports

So if that file is exposed, it gives the attacker more information to compromise other systems - similar to the known_hosts.

The file is stored in ~/.ssh/config and can be matched against the above mentioned keywords.

RCE Exploit chain

So as we have seen above, having a few files from ~/.ssh/ exposed might leave your system open to simply RCE attacks. It is practically game over if you have your SSH private key files exposed on the server and not protected with a password. If you leave some more files such as the known_hosts or authorized_keys accessible, then your whole network might be compromised.

This exploit chain is easily automatable, so you should expect that attackers are already (or will in the future) scanning for those files and trying to chain the results.

To fix the issue, you should not put those files on the web server in the first place!

Use agent forwarding or ProxyJump instead to ssh onward from the web server. (Thx to @closeparen & @egwynn from HN)

Make sure that those .ssh/ files are not within the DocumentRoot.

Add special rules in your web server's configuration to block access to /.ssh/.

Monitor your logs for GET requests to /.ssh/.

-=-