So, you've decided that the devops Chef configuration management platform is for you, and you've already setup your free Opscode "hosted chef" account, along with your local workstation and now you are ready to provision and manage Amazon EC2 linux virtual machine instances?

This how-to guide is for you.  This is your all-in-one to get you from zero to N-1 fully chef-managed Fedora Amazon EC2 instances backed with custom EBS root devices in less than 60 minutes!

There are three major dependencies that this guide will walk you through on the way towards your larger goal of spinning up an unlimited number of fully customized, fully chef-managed Amazon EC2 instances.  This guide presumes the following (major) dependencies are correctly fufilled prior to successfully creating and managing Amazon EC2 instances:

** amazon aws account has been successfully created (do this first)

** chef >= .10.2 workstation setup properly (then do this second)

Do not proceed further until you have the first two items listed above completed!

** setup ~/.ec2 directory on knife workstation with SSH keys and authentication information (please see Robert Sosinski's EXCELLENT write-up for complete details):  I have extracted and lightly modified the relevant steps (1-12, essentially) from Robert's setup guide and posted them here.  We will be creating and connecting to EC2 instances via chef and knife, rather than the AMI tools... but we need the EC2 tools setup now to ensure success later (do this third)

1. Log into your Amazon Web Services account

2. Once signed up and into the AWS website, click the "Account” top-navigation and then select "Security Credentials" from the body of the next page (login to AWS again if necessary - WTF?).  Here, you should select the “Access Credentials” link to get your specific credentials.

4. Select the “X.509 certificates” link.

5. Click on the “Create New” link. Amazon will ask you if you are sure, say yes. Doing so will generate two files.

A PEM encoded X.509 certificate named something like cert-xxxxxxx.pem

A PEM encoded RSA private key named something like pk-xxxxxxx.pem

6. Download both of these files.

* please note: while you are here, grab the information on your "Access Key ID" as well as the "Secret Access Key" and save them locally - you will need both items to setup knife later on inside the "edit knife.rb and add your EC2 "cloud" credentials" section of the guide!

7. Download the Amazon EC2 Command-Line Tools from here: http://developer.amazonwebservices.com/connect/entry.jspa?externalID=351&categoryID=88

8. Open the Terminal, go to your home directory, make a new ~/.ec2 directory and open it in the Finder (I use a Mac.  "Finder" is Mac specific.  Just make sure you've got the right files in the right places if you are using a different OS for your local workstation!)

9. Copy the certificate and private key from your download directory into your ~/.ec2 directory.

10. Unzip the Amazon EC2 Command-Line Tools, look in the new directory and move both the bin and lib directory into your ~/.ec2 directory. This directory should now have the following:

The cert-xxxxxxx.pem file

The pk-xxxxxxx.pem file

The bin directory

The lib directory

11. Now, you need to set a few environmental variables. To help yourself out in the future, you will be placing everything necessary in your ~/.bash_profile file. What this will do is automatically setup the Amazon EC2 Command-Line Tools every time you start a Terminal session. Just open ~/.bash_profile in your text editor and add the following to the end of it:

12. As you made some changes to your ~/.bash_profile file, you will need to reload it for everything to take effect. Run this:

Test your work up to this point by using your new AWS command line tools to list the available AMIs (if this doesn't work, you made a mistake at some section above and should ensure success before continuing onwards!)

If the AMI describe command above worked (it will generate a TON of output about available machine images), move forward and create a keypair with a nice descriptive name for SSH access to your future EC2 instances (my keypair is called smashrunsteve - you'll see that later on in this guide):

This will create a RSA Private Key and then output it to the screen. DO NOT SHARE THIS KEY!  It will permit complete "root" access to your virtual machines later!  Copy the entire key, including the -----BEGIN RSA PRIVATE KEY----- and -----END RSA PRIVATE KEY----- lines to the clipboard. Now, go into your ~/.ec2 directory, make a new file called smashrunsteve.pem (choose your own keypair name here!), open it in your text editor, paste the entire key and save it.

Correct the permissions of your keypair file so that no one can read it.  It's private!

Inside your ~/.ec2 directory:

You now have your AWS account setup and your local workstation has all the necessary authentication information to communicate with the cloud via the command line!  Good work.  Take a breath: all three of the necessary pre-requisites are now complete!

Although I was hoping that a nice, off-the-shelf Amazon AMI would provide a basic linux image for my use (and there are many Ubuntu AMIs that fit the bill), I prefer the Redhat/Centos/Fedora track and the dearth of decent AMI's in that regard quickly lead me to realize I would have to create my own AMI; luckily the Fedora project has an official Fedora 14 AMI that I'll use as a base.  For detailed information on creating your own AMI (this is not light reading - stick with my guide instead!) take a look at the full on Amazon AMI-creation docs.

** Note:  the following section presumes that you have followed the steps above, have the three major dependencies worked out, and your local workstation already has chef/knife setup and working.  I suggest you break through to this guide if you do not have chef and knife setup yet on your local workstation! Our new Amazon vm will need chef so that we can deploy packages to it automatically once it is created... but the stock Fedora 14 AMI is not chef bootstrap ready...  So, let's sharpen our knife and get cooking!

First, install gem dependencies on the local workstation to extend knife's functionality for cloud managment:

Second, install cloud provider-specific knife plugins on local workstation (this guide is ec2-specific; however, there are other cloud provider plugins... check the opscode wiki referenced above):

Next, edit knife.rb and add your EC2 "cloud" credentials to the bottom:

(if you did not save this information from earlier in the guide, simply sign into the AWS website, click the "Account” top-navigation and then select "Security Credentials" to retrieve it now)

Last, configure your local knife workstation to use the correct amazon EC2 SSH private key (that will be specified below as -S)

(remember, I called my keypair from the beginning "smashrunsteve" but you hopefully chose a better keypair description for yourself)

At this point, we are ready to launch an EC2 instance from the command line, but we still have to get all the necessary meta-information together for knife prior to the launch.

It is time to figure out which AMI (-I) to use.  I had hoped to use centos but numerous items blocked my path and so I jumped to Fedora instead - BRAVO FEDORA for making an official ami available!

us-east-1 i386 instance store [ ami-669f680f]

us-east-1 x86_64 instance store [ ami-e291668b]

Next, figure out which instance type you want to pay for (--flavor) ; note that m1.small requires i386 AMI.

Ensure that you have a valid security-group defined.  The easiest way to do this at the moment is via the aws web interface.  login to AWS and hit the "EC2" top-navigation, and then go to the "Security Groups" at the bottom of the left navigation.  Create a new security group (mine is called "sm-linux", I suggest you start with a group for linux and a group for windows).  These are your basic firewall rules.  For now, a basic firewall would contain TCP SSH, HTTPS and HTTP  (22 443 80) along with ICMP ALL.  Click the "inbound" tab and modify the security group rules to permit those basic ports.  0.0.0.0/0 (permit all internet) should be fine for now - if you are paranoid (why not? they are watching you, after all...) and want to add JUST your own workstation IP, then put your workstation's public IP in and use the single-host mask "/32".  Make a note of what you called your new security-group (-G).

Note the name of the aws ssh key you want to use (-S) and the instance hostname and chef node name (-N and --node-name, do yourself a favor and use FQDN)

Last, figure out which user to connect as initially (-x) : fedora requires ec2-user with sudo: WINNING!

FINALLY, put it all together and LAUNCH THE INSTANCE with the stock Fedora AMI by typing:

Wait for a little while... (I think this first machine creation takes a long time because the Fedora machine image information has to be transferred from amazon storage) 

Instance ID: i-7041a111

Flavor: m1.small

Image: ami-669f680f

Availability Zone: us-east-1b

Security Groups: sm-linux

SSH Key: smashrunsteve

Waiting for server................

Public DNS Name: ec2-107-20-15-58.compute-1.amazonaws.com

Public IP Address: 107.20.15.58

Private DNS Name: domU-12-31-39-00-6A-45.compute-1.internal

Private IP Address: 10.254.109.175

HOLY COW! The EC2 instance was successfully launched!  Now your knife is trying to bootstrap chef... but unfortunately this will fail because the default knife bootstrap is built with Ubuntu in mind... it's not a big deal, we'll customize our AMI in a few short steps below. {Note, this boostrap didn't work first off with my inital centos AMI either: the default knife bootstrap has a major apt-get dependency, which is the wrong linux package management system from centos' yummy perspective and RPM chef packages are not going to happen on CENTOS: deprecated - this is one reason why I decided to back away from centos.}

The bootstrap didn't work first off with fedora; the base Fedora AMI lacks apt and other dependencies.  That's fine - the EC2 instance was properly started and we have the information we need to connect.  We will install the necessary software on this first image ourself, and then transform the virtual machines' disk partition into an "EBS" instance to save our work.  The long instructions on the opscode wiki for getting this done are located here, but I've got the short version for you.  SSH to your new EC2 instance (make sure you substitute the name of your SSH keypair and the public IP address that knife reported back when it launched the instance earlier):

Install the basic packages on your new instance to support Chef and future bootstrapping (Fedora 14 does install ruby 1.8.7 via yum, which is nice):

Install RubyGems from Source

Now the instance has been prepared for future use.  We'll now transform the root partition into an EBS backed volume so that you can save the work you've done thus far (and use it as a template).

** Please note: I specifically installed as few packages as possible on my temporary base Fedora instance so that it would be as basic and "streamlined" as possible prior to templating; however, if you want to install additional software (and you want all your future Fedora machines to have it, too) then this is the place to do it, BEFORE you rsync everything from the current running root partition to the new EBS volume that you will create and attach below.

Once you are satisfied with your Fedora vm instance (if you are following right along, it's ready now!) it's time create and attach a new ebs volume to your Fedora instance - use the AWS web console to create a new EBS volume and attach it to your currently running Fedora instance.

After you have created a new EBS volume and you've attached it to your running Fedora instance using the AWS web console, you'll need to create and mount the new EBS partition on the Fedora instance

Now, rsync everything from your current Fedora AMI root partition over to the new ebs volume and get rid of the stuff you don't need

* THE FOLLOWING /etc/fstab EDIT IS IMPORTANT!  We need to remove the line that references the old /mnt partition (we won't need it the next time we boot), and we need to modify the line for the root / partition to use the new "xfs" filesystem type

Now, unmount the ebs volume from your Fedora instance so that it can be detached, snapped, and remounted to a new AMI:

Detach the ebs volume from the current Fedora AMI (use the AWS web console)

Take a snapshot of ebs drive (this uses the AWS tools we installed with Robert Sosinski's help earlier, ensure you substitute the name of your EBS volume from the AWS web console)

Check on the snapshotting status (wait until the snap is complete)

Register a new AMI with the snapshot, kernel from the previous machine, and other descriptive information as necessary using the AWS tools (this is what we have been working towards this entire time!)

The register command brings back an AMI id:

IMAGE ami-f00ff599

Terminate your first Fedora ami instance (it is now disposable; make sure you substitute your instance id!)

... And start up a new Fedora instance with your brand-new, fully customized AMI and EBS volume. Be sure to use your new ids for the AMI and node names. Watch the chef bootstrap process sail right through!

HOLY CRAP! This time chef successfully bootstraps!  (and will from now own - thanks to your new custom EBS volume and AMI image!)  CELEBRATE by registering on Smashrun to track your Nike++ runs and follow me on twitter!

At this point, you can apply chef roles and recipes to your new machine.  I won't cover that here.