The Occasional Masthead

One of the problems with cloud security compared to on-premise is that there is more risk that someone unauthorised will be able to gain access to your EC2 linux instances via SSH. That’s one of the reasons I’m keen on server less solutions, various X-As-A-Service services, and on not opening up a server for access by SSH at all. It’s easier to keep bad guys off a server if you don’t let anyone onto the server.

There are a variety of reasons though why you really do want to allow your developers and operators to get direct access to a server, and a variety of mechanisms have arisen. Session Manager is a particularly nice one, as it removes the need to open up SSH on a server at all, although at the cost of requiring an AWS service to be running on the host. Because it runs through the web console, it’s nicely locked down to authorised users via IAM roles and policies, and access is audited through CloudTrail. The trouble is that it operates through the console, and is not providing direct access from the developers’ desktop terminal.

Opening up SSH access to a limited set of source IP (and also doing things like constraining by access time, requiring MFA, etc) is quite straightforward. The problem is that if you are distributing the private key for the ec2-user, you have a big security problem on your hands, and you are put in the position of needing to re-distribute new keys every time the instance is redeployed. This is annoying, particularly if you are using spot instances!

If you are using infrastructure-as-code principles, then you can create user accounts on the instance as it is created, and add public keys for those users – but then you have problems of where you store those keys, and what to do when the users roll their keys. For anything more than a few hosts, this rapidly becomes damned annoying as well, but until recently was the best that we could do.

Instance Connect is a very neat hack on Amazon’s part to get around some of these hassles. It uses three interrelated pieces to allow SSH from the desktop without permanently storing keys on the desktop. First, there’s a new action ec2-instance-connect:SendSSHPublicKey which a user can use from their desktop to send a public key to a target instance. Second, there’s an agent that can be installed on the instance which knows how to receive that public key information. Third, an authorisation module is wired into sshd that knows where to find that public key when it needs it.

There is a companion CLI tool that the documentation points you to, which provides a nice shorthand way of using this new facility. mssh will create a public/private key pair, push the public key to the target instance, then do a standard SSH connection with the private key. On termination the public/private pair is simply discarded. You can easily do the same thing yourself, but be aware that there is only a 60 second window between pushing the public key and attempting to connect with the private key, after which the public key ‘expires’ on the target instance.

Authorised users of the service can also use the web console (from the EC2 pages) to connect with a web interface, however this is functionally equivalent to using Session Manager. To my mind it’s likely that you would want to grant this SSH access to users without them necessarily having access to the web console at all, and almost definitely without them having access to Session Manager.

The great thing about this facility is that it’s directly tied to an IAM identity, providing unambiguous auditing via CloudTrail and very easy management of access rights. A side effect as well with the agent being installed during the instance set up is that it’s trivially easy to use with Spot Instances, removing all the pain of distributing or managing and installing key pairs.

If you want to explore this new facility, the documentation is pretty good, although focussed on using the AWS CLI and manual steps to set up the target instance. I’ve created a small Terraform project in GitHub that sets up a suitable instance for testing that you may find more convenient.

There’s one intriguing part of this new service that I want to test a little further. In the documentation, a sample policy is described:

{
    "Version": "2012-10-17",
    "Statement": [
      {
        "Effect": "Allow",
        "Action": "ec2-instance-connect:SendSSHPublicKey",
        "Resource": [
            "arn:aws:ec2:region:account-id:instance/i-1234567890abcdef0",
            "arn:aws:ec2:region:account-id:instance/i-0598c7d356eba48d7"
        ],
        "Condition": {
            "StringEquals": {
                "ec2:osuser": "ami-username"
            }
        }
      },
      {
        "Effect": "Allow",
        "Action": "ec2:DescribeInstances",
        "Resource": "*"
      }
    ]
}

Now, this policy can be easily extended to allow conditions like requiring MFA, limiting source IP, and constraining the time in which it can be used. But the condition on ec2:osuser is very interesting. The documentation specifies that this is ec2-user for AWS images, but I am wondering if this can be any user account on the instance. If we create a user connect-test on the instance, and specify that as the user in the policy, and use that ID during the SSH connection – do we then have a way to SSH into the instance with a one-time key as a non-privileged user?