In my first post, I described how I
set up a minecraft server in Amazon's Elastic Compute Cloud (EC2) by hand. In this post, I will begin exploring how to automate that process so that when I mess the server up, I can start over with very little effort.
The first thing I want to automate is all of the interactions with the web user interface required to create an instance, launch it and bind it to an elastic IP address. All of these interactions when moved to a command-line/API script require a different form of credential than the SSH key pairs I used in the last post, so the first step is to get those credentials.
Getting Credentials
This is done through Amazon's Identity and Access Management (IAM) system. I'll create a group whose members will be allowed to manage EC2, create a user and then add that user to the group. The command-line scripts will use that user's credentials to perform all EC2 operations. I start at
the IAM console, where I select "Create a New Group of Users":
When the wizard launches, I give the group a meaningful name. I choose EC2Admins as all members of the group will be able to administer my EC2 instances.
I now apply a permissions policy to the group. This is where I indicate that all members of the group will have full access to EC2 by selecting the "Amazon EC2 Full Access" Policy Template.
This next step in the wizard shows what that policy template looks like in detail. I don't want to do any customization at this point, so I just continue past this step.
At this next step, I create a user and add it to the group. I chose "my_user_name" for this example, but you can pick any name you like. It is
very important that you select "Generate an access key for each user" as this is the key that will be used to connect to authenticate with Amazon when we perform any operations on the instances.
I get a chance to confirm my choices. Everything looks good, so I click "Finish"
At this step, I get my
only chance to download the credentials for the user.
I chose to open the Show/Hide Users Security Credentials so I could copy the credentials into a text file. I named the file
aws.config.txt and placed the file in my ~/.ssh directory. I'll come back to this file in a moment, but
be sure that you have copied your security credentials (not the ones from this blog post) before you close that window!
Setting up the Scripting Environment
I haven't done much with python, but I've wanted to learn it, so I'm going to use the python version of the AWS command line tools. I use three different environments (Mac, Linux and Windows) so I need to work out the set up for each operating system.
The Mac and the Ubuntu Linux machines already had python and easy_install installed. For the Windows PC, I re-ran the cygwin setup program and added python. I also added easy_install there by opening
http://peak.telecommunity.com/dist/ez_setup.py, saving the file to a temp directory, navigating to that directory and then executing the saved file with python
cd /cygdrive/c/temp
python ez_setup.py -U setuptools
I then followed Amazon's instructions for getting set up with the
AWS Command Line Interface (CLI) on all three machines:
easy_install awscli
I could tell the installation was successful when I executed
aws help and saw the associated help text.
The final preparatory step is to associate the credentials with the scripting environment. I modified the file that I saved above to match the structure required
of the AWS configuration file.
[default]
aws_access_key_id = AKIAIQW7AJVBDDLBXDNQ
aws_secret_access_key = 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
By adding the following lines to my .bash_profile, I set the environment variable to point to that file each time I log in.
AWS_CONFIG_FILE=~/.ssh/aws.config.txt
export AWS_CONFIG_FILE
If everything is set up correctly, then I should get a JSON result back fully describing the instance I set up from the web console earlier when I run this command:
aws --region us-east-1 ec2 describe-instances
In the next post, I'll use this scripting environment to create, start and stop instances.
Troubleshooting
Some possible errors and how to deal with them:
- 'NoneType' object has no attribute 'access_key'
This means the AWS_CONFIG_FILE environment variable hasn't been set. From the commandline enter, for example:
export AWS_CONFIG_FILE=~/.ssh/aws.config.txt
Another possibility is that the first line of text in your configuration file has a word other than default between the square brackets. Make sure there are no spaces between the letters and the square brackets.
- The specified config file (/home/user_name/.ssh/aws.config.txt) could not be found.
The AWS_CONFIG_FILE variable exists but the referenced file does not! Determine exactly where you put the configuration file with the access keys and make sure that the value to which you set the environment variable matches.
- Unable to parse config file: /home/doug/.ssh/aws.config.txt
The file exists and the environment variable is correctly referencing it, but the structure of the contents aren't quite right. Check that the word aws_access_key_id and the actual key value are on the same line separated by an equal sign; similarly for aws_secret_access_key. The first line of text in the file should be just [default] - including the square brackets.
- A client error (AuthFailure) occurred: AWS was not able to validate the provided access credentials
The wrong credentials are in your config file. Did you accidentally use the ones from this example? Make sure the exact string of characters from your user are entered.
- A client error (UnauthorizedOperation) occurred: You are not authorized to perform this operation.
An aspect of the credentials setup is incorrect. There are multiple possibilities: the user does not belong to the EC2Admins group; the EC2Admins group does not have the EC2 Full Access policy; the EC2 Full Access policy is not defined to actually give full access to EC2. Fixing these should be done in
the IAM dashboard. If you're unable to determine what went wrong, you can create an entirely new Group of Users using the wizard and use the credentials from the new user instead.
The call succeeds, but the returned results have an empty reservationSet instead of one filled with details about your instance. Most likely, your instance was not created in the us-east-1 region. Open the
AWS EC2 dashboard to see where your instance is running. In the upper-right corner will be a drop-down with the link text that is your default region name (e.g. Oregon). The URL in your browser's URL board will contain your region ID (e.g.
region=us-west-2). Replace any occurrence of
region=us-east-1 in these instructions with your region ID.
