What is SSH for?

SSH is used to communicate securely between a client and a server. Some examples that most people are familiar with are SSH-ing to a remote server or using GitHub.

The key pair

To establish an SSH connection you need a private key and a public key. You keep the private key in a safe place (usually in ~/.ssh/) and upload the public key to any server that you want to securely connect to. Encryption with a private key and a public key is called assymetric encryption.

How do you create a key pair. Like this:

$ ssh-keygen -t rsa -b 4096 -C "myemail@gmail.com"
Generating public/private rsa key pair.
Enter file in which to save the key (/home/petko/.ssh/id_rsa): ./mykey
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in ./mykey.
Your public key has been saved in ./mykey.pub.
The key fingerprint is:
SHA256:0/zlEZZj0R6yM4uHVD+xJWoLOx3+TormCRcyWt3c53o myemail@gmail.com
The key's randomart image is:
+---[RSA 4096]----+
|              .. |
|             o.=o|
|            ..Xo=|
|         +.++B *.|
|        S **=oB o|
|       o +o=+= + |
|      . . ..o.o .|
|         o.o o..E|
|         o+ ..o. |
+----[SHA256]-----+
$ ls -l
total 8
-rw------- 1 petko petko 3243 Aug  9 16:24 mykey
-rw-r--r-- 1 petko petko  743 Aug  9 16:24 mykey.pub

How do the public and private key look like. They look like this (I have removed some of the output to keep you from scrolling too much):

$ cat ./mykey
-----BEGIN RSA PRIVATE KEY-----
MIIJKQIBAAKCAgEA2FGlZybznkcHQG530bj+DlrY74nTh+shP1uyJA25BrkAyOz9
xc8Tvlk3QfBcGFfQKc1OowV80XtNyXXnOeFTqlh8B8DS1Mul165wgb+pJDROvI0J
...
3vZfDPXo9w2XwAwN7hLimCVWdqr0JI8BmbussW4ZrJRcra1rvsLj6sip9Ry3oP+9
PIUEPwDY/YUVRZV2De4cBdBnwTmj9RoXOW63mW6sL8lfeYjJQwQys+jVVjRi
-----END RSA PRIVATE KEY-----

$ cat ./mykey.pub 
ssh-rsa AAAAB3NzaC ... +PAAKfQ== myemail@gmail.com

So that's it. A private key and a public key. Nothing else. Keep the private key secret, upload the public key.

How does SSH work?

Great question. If you have a key pair you can encrypt a message with one of the keys and decrypt it with the other key. But that's not how data is exchanged with an SSH connection. SSH is actually using symmetric key encryption. The private and public keys are used to securely exchange a temporary symmetric key used to encrypt the data between two machines. The symmetric key crosses the wire in encrypted form, so an attacked can't find out what the key is.

Dig into this excellent stackoverflow question to learn more about this.

The ssh command

We now know what's involved in SSH communication. Let's look at the commands that are used.

We'll start with a clean example. I have an AWS EC2 instance on which I have installed a public key. The private key is located in my ~/.ssh/ directory. It's called aws-laptop, because I use it from my laptop.

Let's try to ssh into the instance:

~/.ssh$ ssh ubuntu@35.165.19.203
Permission denied (publickey).

Doesn't work. Well, of course - ssh doesn't know what private key to use. We have to point to it using the -i option.

~/.ssh$ ssh -i ./aws-laptop ubuntu@35.165.19.203
The authenticity of host '35.165.19.203 (35.165.19.203)' can't be established.
ECDSA key fingerprint is SHA256: 3w+LlpD/HH .......
Are you sure you want to continue connecting (yes/no)? yes
Enter passphrase for key './aws-laptop':  [Here I entered a passphrase]
Welcome to Ubuntu 16.04.2 LTS (GNU/Linux 4.4.0-1022-aws x86_64)
...
Last login: Wed Aug  9 23:09:13 2017 from 76.102.141.14

ubuntu@ip-172-31-13-46:~$

Hooray, I'm in! What happened in the meantime is that the ssh command saved an entry into the known_hosts file, so next time I ssh it won't ask me again to confirm the authenticity of the remote host.

However, I'll still need to point to the private key and enter the passphrase for it. Very inconvenient.

The ssh-agent daemon

The solution to the above problem is the ssh-agent daemon. ssh-agent is a daemon that you start and you can add private keys to it so that next time you ssh into a server, you don't have to point to them.

Usage

How does ssh-agent know which private key to use? When you run ssh -v ubuntu@35.165.19.203 you can see that there's a bit of back and forth where ssh seems to be trying out private keys from ssh-agent. So I guess that's how it's done:

debug1: Offering RSA public key: /home/petko/.ssh/id_rsa
debug1: Authentications that can continue: publickey
debug1: Offering RSA public key: ./aws-laptop
debug1: Server accepts key: pkalg rsa-sha2-512 blen 279

We can list the keys in ssh-agent with this command:

$ ssh-add -l
The agent has no identities.

No keys. So let's add the key that we were using.

$ ssh-add ./aws-laptop
Enter passphrase for ./aws-laptop: [I entered passphrase]
Identity added: ./aws-laptop (./aws-laptop)
$ ssh-add -l
2048 SHA256:tU++v7cfD8... ./aws-laptop (RSA)

The private key is now saved! I can now ssh without pointing to it:

$ ssh ubuntu@35.165.19.203
Welcome to Ubuntu 16.04.2 LTS (GNU/Linux 4.4.0-1022-aws x86_64)
...
ubuntu@ip-172-31-13-46:~$ 

Starting ssh-agent

You don't want to start ssh-agent manually. I'll describe my setup for automatically staring it in Ubuntu. In my ~/.bash_profile I have the following:

# Start ssh-agent.
if [ ! -S ~/.ssh/ssh_auth_sock ]; then
  echo "Starting ssh-agent"
  eval `ssh-agent`
  ln -sf "$SSH_AUTH_SOCK" ~/.ssh/ssh_auth_sock
fi
export SSH_AUTH_SOCK=~/.ssh/ssh_auth_sock
ssh-add -l | grep "The agent has no identities" && ssh-add

Remember that .bash_profile is executed for interactive login shells. An interactive login shell is what's run when you ssh into a machine. One interesting thing about Ubuntu is that if you open a shell from within Gnome, it's an interactive shell, but it's not a login shell. Gnome also runs a separate ssh-agent, which you can find out when you see the contents of the SSH_AUTH_SOCK environment variable. But I don't use shells from within Gnome, so that's not important to me. If I had a desktop Ubuntu, I'd care about that situation more.

Here's an article that's a good description of the mechanics of starting an ssh-agent.

ssh-agent forwarding

So you've setup SSH on your local machine, but you want to SSH to your cloud server and pull some git repo. Or you want to SSH from the cloud server to another server, or maybe copy some files with scp. You can generate a key pair for the cloud server and distribute the public key to all machines and services that you want that cloud server to have access to. But that's not convenient. This is where SSH agent forwarding is needed. With SSH agent forwarding, you ssh into a machine and it takes the private keys loaded in your ssh-agent with you to the other machine.

GitHub has a great article on how to setup that. One thing that I'd add to it is that you don't have to start ssh-agent on the target machine! It's automatically started when you ssh to it.

In my particular setup, I'm ssh-ing to a machine that has just an IP address, so I added an entry for it in /etc/hosts:

$ cat /etc/hosts
127.0.0.1       localhost
52.39.10.196    webhost
...

My ~/.ssh/config file looks like this:

$ cat ./config 
Host webhost
  ForwardAgent yes
  User ubuntu

So now to ssh to this machine, I only type ssh webhost. Once I ssh there I can run ssh-add -l and see that my keys are available on the remote machine.

Tips and tricks

Manually verifying a private key

Today I was trying again and again to connect to a remote machine using the wrong private key. I was able to connect to that machine from my laptop, but not from my Ubuntu VM.

I was running the following command:

$ ssh -i ./somekey.pem  ubuntu@35.165.19.203
Permission denied (publickey).

How did I debug this? I ssh-ed into the server from my laptop and looked up the public key in ~/.ssh/authorized_keys. I then generated the public key from my private key, like this:

$ ssh-keygen -y -f  ./somekey.pem
ssh-rsa AAAAB3....KY1

And sure enough, the public keys were not matching. I then found the correct private key and was able to connect with it.

If you're following carefully, you'll notice that I did something here that's considered a bad practice. I reused my private key from my laptop on my VirtualBox. I should generate a new key pair and add the public key to the authorized_keys file on the server.

So that's it so far. Enjoy your remote access.