Hello EC2, Part 2: Locking it Down



This is the second in a series of posts about setting up a scalable and robust “Hello World” web server using Amazon EC2. The target audience is developers with some experience using EC2. This second installment looks at how to include secrets in the instance metadata without revealing them to unprivileged users.

Introduction

In the recipe outlined in the previous installment, I included a private deployment key in the User Data field of the instance metadata in order to enable the EC2 instance to read configuration information from a remote git repository. This configuration information, in turn, might contain other secrets, such as API or database access keys. Naturally, it is prudent to restrict access to the deployment key and the secret parts of the configuration information to the root user.

Analysis

I know of only three ways to make a master secret available to an EC2 instance:

  • Embed it in the Amazon Machine Image (AMI).
  • Embed it in the User Data field of the instance metadata.
  • Serve it to instances belonging to a particular security group from a master instance running in the same EC2 region.

I will take the second approach, which requires keeping the User Data secret.

EC2 instances access their instance metadata via the network interface using HTTP. The contents of the User Data field is available to the instance at the following URL:

http://169.254.169.254/latest/user-data

In Automate EC2 Instance Setup with user-data Scripts, Eric Hammond wrote:

Setting up a new EC2 instance often requires installing private information like EC2 keys and certificates (e.g., to make AWS API calls). You should be aware that if you pass secrets in the user-data parameter, the complete input is available to any user or process running on the instance.

There is no way to change the instance user-data after instance startup, so anybody who has access to the instance can simply request http://169.254.169.254/latest/user-data

Depending on what software you install on your instance, even Internet users may be able to exploit holes to get at your user-data. For example, if your web server lets users specify a URL to upload a file, they might be able to enter the above URL and then read the contents.

The cloud-config subsystem of cloud-init includes a disable-ec2-metadata option, which  removes access to the EC2 metadata service early in boot by creating a null route for 169.254.169.254. According to Scott Moser, the intent is to make the user data “as secure as a root owned file with 400 permissions on it.” Since I am not using cloud-config, I will simply set up the null route in my puppet configuration.

We also need to protect any copies of this information in the file system. Cloud-init saves the user data in /var/lib/cloud, but it restricts read access to the pertinent files. My user data script (see the previous installment) then clones the puppet configuration into /etc/puppet. I will place the sensitive parts of the configuration in /etc/puppet/private and have puppet restrict read access for both that directory and /etc/puppet/.git.

The Code

My sample puppet configuration includes a lockdown module to take these precautionary measures on first boot, early during the puppet run. Here is the relevant part of the puppet configuration:

file { "/etc/puppet/.git":
    ensure => directory,
    mode => '700',
}

file { "/etc/puppet/private":
    ensure => directory,
    mode => '700',
}

file { "/etc/init/disable-ec2-metadata.conf":
    ensure => file,
    content => template("lockdown/disable-ec2-metadata.conf"),
}

exec { "/sbin/start disable-ec2-metadata":
    require => File["/etc/init/disable-ec2-metadata.conf"],
}

The manipulation of the routing table is handled by an upstart task because it needs to be redone whenever the instance is booted. The task is defined as follows (disable-ec2-metadata.conf):

description "Disable access to EC2 instance metadata and to user-data in particular."

start on net-device-up IFACE!=lo

task
exec /sbin/route add -host 169.254.169.254 reject
normal exit 0 7  # No error if route already exists.

Conclusion

These precautions should reduce the security concerns arising from the inclusion of secret keys in the instance metadata and the puppet configuration. They should be particularly helpful in more realistic configurations where as little as possible is run as root.

The next installment will look at setting up a self-healing cluster.

This article was written by Ken Rimey and has later been edited by other employees of Codento.

3 kommenttia artikkeliin ”Hello EC2, Part 2: Locking it Down

  1. This must be valid at the time this article was written.
    I guess, the current approach to do this would be to use IAM roles that are specified at the time of launching the ec2 instance and pulling install scripts from server at the time of installation from, say, S3 which has restricted access

  2. sanoi: And now that AWS includes a Git repository (Code Commit) integrated with IAM, All the ideas shared by kirjoittanut can be enhanced : just changing Bitbucket with Code Commit and This 2nd part of the series with a better use of IAM as you mention in your comment.

    kirjoittanut: Thank you for sharing this ideas. They had been really useful for me.

Vastaa

Sähköpostiosoitettasi ei julkaista. Pakolliset kentät on merkitty *