Bootstrapping Private Subnet Instances In Amazon VPC with Knife

Amazon VPC

Amazon Virtual Private Cloud (VPC) is a service that allows you to define an isolated virtual network within EC2. A common scenario involves a VPC with both public and private subnets. Instances within public subnets can send and receive traffic directly to/from the Internet. On the other hand, instances within private subnets cannot receive traffic directly from the Internet and can only send outbound traffic via a NAT instance.

Bastion Host

Given a VPC setup with both public and private subnets, you’ll want at least one SSH bastion host in the public subnet. This host is needed to communicate with instances in the private subnet from your local machine. The diagram below, taken from Amazon’s documentation, helps illustrate:

SSH Bastion with VPC

Knife EC2 Example

Using a combination of Knife and the Knife EC2 plug-in, the following command connects directly to the bastion host specified by the --ssh-gateway option. From there another connection is made to the private subnet instance via its private_ip_address in order to bootstrap Chef:

knife ec2 server create --flavor hi1.4xlarge --image ami-08249861   \
  --security-group-ids [SECURITY_GROUP_ID] --tags Name=node1-dev    \
  --availability-zone us-east-1d --subnet [SUBNET_ID]               \
  --node-name node1-dev --ssh-key orgname --ssh-gateway bastion-dev \
  --server-connect-attribute private_ip_address                     \
  --ssh-user ec2-user --identity-file ~/.ec2/orgname.pem            \
  --environment development --ephemeral '/dev/sdb,/dev/sdc'         \
  --run-list 'role[base],role[solr_ssd_slave]'

Depending on how long it takes your run list to converge on a bare operating system, you should have Chef bootstrapped on an instance within the private subnet of a VPC after running only one command!

Chef Platform Families

As of version 0.6.12, Chef’s operating system data detection component, Ohai, added the concept of platform families. Platform families are groups of similar operating system distributions categorized under a more general label. For example: RedHat, CentOS, and Fedora are all classified under rhel. This allows you to branch logic within recipes to compensate for differences between operating systems — like directory paths and package names.

The other day someone asked for a listing of all of the available platform families. After Google searching, the only solid resource I found was the Ohai 0.6.12 release announcement from March. To make sure things haven’t changed since then, I went to the source and pulled a list of the following platform families:

  • mac_os_x
  • debian
  • fedora
  • rhel
  • suse
  • gentoo
  • slackware
  • arch
  • windows
  • omnios
  • openbsd
  • solaris2

Update: Thanks to @bdha for help with identifying the Solaris and BSD platform families.

Lessons Learned After Getting Gotten

A Cool Story

Last Saturday morning I was visiting my parent’s house in North Philadelphia. From there I was headed back into the city for a technology conference. I decided to take public transportation, so I drove to the nearest subway station. I parked my car in a lot next to the station and headed to the platform. A train was already waiting for passengers, so I got on the closest car from the steps that lead to the platform.

I wasn’t dressed like a mark, but I also wasn’t dressed to blend in with my environment. I had an Adidas track jacket on with shorts, flip-flops, and a laptop bag. Looking back, I can see why I may have stood out.

As I walked into the subway car, I noticed that it was completely empty. Having my choice of seat, I uncharacteristically sat in the seats next to the sliding doors that separate subway cars. Unfortunately, this put my back toward anyone coming onto the car. After sitting down, I proceeded to mess with my phone — checking Twitter, or e-mail, or Instagram. The next thing I know, I get punched on the side of the head and my phone is taken right out of my hand.

After seeing the kid run off of the subway car, I decided to chase after him. To set the stage, as soon as you get out of the car there are 6-8 steps up, a landing, another 6-8 steps, turnstiles, 6-8 steps down, a landing, and finally 6-8 more steps down — at this point you’re finally back at street level. By the time I was exiting the car, he was just about to start the second set of 6-8 steps up. I watched him slip on a step, which increased my chances of catching him. Unfortunately, as I approached the first landing one of my flip-flops got tangled and came off. I kicked the second one off and laid my laptop bag down.

By the time I made it to the turnstiles, he was almost at street level. Knowing that I just laid my laptop bag down unattended, I had to choose between going back for it or continuing to chase after my attacker. I chose the laptop. Temporarily defeated, I took a moment to assess the situation and asked one of the operators to contact the police.

As police officers began to show up at the station, it occurred to me that I could possibly track the attacker via Apple’s Find my iPhone feature. I opened my laptop in hopes of using an unprotected WiFi access point, but no cigar. I told one of the officers about the feature and he said that there were people at the station with experience using it. He immediately got on the phone and tried relaying my password to the operator — this took a lot longer than expected.

As soon as the police pinpointed the location of my phone, the officers at the subway station mobilized and took me in the back seat of a cruiser. The signal led us to a driveway between two rows of row homes. I told the police that the signal is usually not 100% accurate, so the kid could be in any of the surrounding houses. As they surrounded houses, I began looking through nearby trash cans hoping the kid had decided to toss the phone.

At this point the remote operator offered up a Find my iPhone feature that forces the device to emit a loud noise. The loud noise would give police probable cause to enter a house — otherwise it would have required a search warrant. I was against the idea of forcing the device to make noise, knowing that if we didn’t hear it immediately, the kid would turn the phone off — significantly reducing the chances of recovering it . About three minutes after they triggered the noise, the signal went dark.

Since there was not much else to do, they officers offered to drive me back to my car parked at the subway station. I didn’t feel like attending the conference anymore, so instead I drove back home. As soon as I got there I began the process of revoking passwords and disabling the SIM card. I had officially gotten got.

Lessons Learned

I like to consider myself pretty street smart. Looking back, there were many things I could have done to make myself less of a target. At the same time, there were a few things I did that made the scenario go a lot smoother than it could have gone.

Pros

  • I had Google two-factor authentication enabled. This allowed me to easily revoke application specific passwords created for services on the device.
  • I had Find my iPhone enabled via iCloud. This at least gave me a chance to catch the person who stole my phone. It also allowed me to request a remote lock of the device after it was stolen.
  • I have a different password for every account. This allowed me to give my password to the police over the phone without worring anyone else would remember it and compromise any of my other accounts.
  • I kept calm after everything happened, which allowed me to recall the Find my iPhone feature existed.

Cons

  • I had flip-flops on while in a moderately dangerous area. I’m convinced this prevented me from catching my attacker when he slipped on the steps.
  • I got on the subway car closest to the steps. This allowed the attacker to snatch my phone and make a speedy escape. From now on I will only board the first or last subway cars.
  • I had my back to the subway car entrance and was only paying attention to my phone. From now on I’ll never sit in the seats next to the doors that separate subway cars and I’ll limit the amount of time I use my phone on the subway.
  • I did not yell as soon as my phone was taken. This neighbordhood is not one known for snitching, but it’s possible that if I had yelled someone would have done something to slow down the attacker.

Conclusion

My replacement SIM card has arrived and now resides in a Google Galaxy Nexus. I rewired two-factor authentication to it and enabled a passcode lock for the phone itself. The phone’s file system is also encrypted (not the default on Android 4.1 Jelly Bean). Last but not least, I purchased a new pair of sneakers. My days of riding on the subway in flip-flops are over:

New Kicks

My name is Hector Castro and I live in Philadelphia, PA. You can see some of my public projects on GitHub, and if you're old-fashioned here is my resume.

You can get in contact with me via Twitter or E-mail.