This blog post mainly documents the process of achieving Serverless-Chef on EC2, for future reference. The basic framework involves using Chef for installing and configuring various software. Since Chef is being used, the preparation process will be more cumbersome. It’s definitely not as straightforward as using Ansible through Serverless-Chef, but I am more familiar with Chef. In this article, CodeCommit is used instead of other git remotes, mainly to utilize a private git repo and allow EC2 to clone the repo directly without storing the keys in EC2.

General Goal

When starting any EC2 instance, it should automatically connect to Route53 to obtain my own domain services and configure all the services I desire.

Objectives

1. Implementation of Serverless-Chef
2. Interaction of Caddy with Route53 domain

Serverless-Chef

Serverless-Chef is not actual software, but a concept. Chef itself requires a Host Server to fulfill its function, where the Host Server provides the installation information and data required for Chef-client to run. However, it can also work without the Host Server if only data is provided.

Basic Architecture

AWS CodeCommit: Storage for Cookbooks AWS EC2: Clones cookbooks from CodeCommit during startup, automatically installs chef-client, and hands over the rest to Chef

“Is it that simple?” “Yes, it’s that simple!”

Workflow

My Mac: Edits Cookbooks, uses “berks vendor” to create local cookbook dependencies, then pushes to CodeCommit EC2: Clones from CodeCommit and runs berks vendor

Implementation

Simple architecture doesn’t mean simple implementation.

I. Configure CodeCommit correctly

1. Create a new user in AWS IAM and assign appropriate permissions. To simplify, I directly used the admin account.
2. Add SSH key in the admin account.
3. Set up in local computer’s SSH config (Last two lines are not explained in the CodeCommit official documentation. For the reason why, refer to this answer):

#~/.ssh/config
Host git-codecommit.*.amazonaws.com
   User <Your Key ID>
   IdentityFile <path_to_your_key_file>
   PubkeyAcceptedAlgorithms +ssh-rsa
   HostkeyAlgorithms +ssh-rsa 

4. git remote add … (Add CodeCommit as remote in the local git repo. Since Chef’s Cookbook is the repo itself, no need for initialization process.) Now, CodeCommit configuration is done.

II. Configure EC2 to connect to CodeCommit

1. Firstly, create a role in AWS IAM for EC2 use. Remember to assign this role to the EC2 instance when creating.
2. SSH into EC2 and set up the required credentials for git.

git config --global credential.helper "!aws codecommit credential-helper $@"
git config --global credential.UseHttpPath true

Now, EC2 can clone the CodeCommit repo.

Mac uses SSH connection, while EC2 uses HTTPS.

III. EC2 User Data

Here’s my EC2 User Data for reference.

#!/bin/bash
yum update -y # No explanation needed for yum update

curl -L https://omnitruck.chef.io/install.sh | bash # Installs chef-client. Note that this is not ChefDK, so ChefDK cannot be used, and berks cannot be directly used

yum install git -y # EC2 Amazon Linux 2 comes without git out of the box
# Set up git credential, code from the previous step
git config --global credential.helper '!aws codecommit credential-helper $@'
git config --global credential.UseHttpPath true

git clone https://git-codecommit.com/your/repo/link /tmp/ec2_setup/ # Clone from CodeCommit to EC2
cp -R /tmp/ec2_setup/berks-cookbooks/* /cookbooks/ # Extract cookbooks generated by berks vendor

export CHEF_LICENSE=accept # Set environment variable to automatically accept all Chef licenses. This can also be specified using --chef-license flag during runtime
cd /cookbooks
chef-client -zr "recipe[ec2_setup]" # Run my cookbook

IV. Process for updating cookbook (on local Mac)

1. Update required files
2. berks vendor
3. git commit
4. git push

V. Process for running the latest cookbook (on EC2)

For convenience, I added a template Resource in the cookbook. Upon the first execution, a bash script will be created at /root.

#Cookbook Recipe
template '/root/chef-cron.sh' do
  source 'chef-cron.erb'
  mode '0777' # I haven't fully grasped Linux file permissions yet, so just went with 777
  action :create
end

chef-cron.erb has no variables, it is essentially the script placed in /root after the first run.

#!/bin/bash
rm -rf /tmp/ec2_setup
git clone https://git-codecommit.ca-central-1.amazonaws.com/v1/repos/ec2-cookbook /tmp/ec2_setup/
rm -rf /cookbooks
mkdir /cookbooks
cp -R /tmp/ec2_setup/berks-cookbooks/* /cookbooks/
export CHEF_LICENSE=accept
cd /cookbooks
chef-client -zr "recipe[ec2_setup]"

From now on, simply run sudo bash /root/chef-cron.sh whenever you want to execute the latest cookbook.

I named my script chef-cron, so you can tell I intend to turn it into a cronjob to run periodically. However, for now, this feature is not urgent. You can always start a new EC2 instance anytime, and manually running it in this single EC2 instance would be more convenient. Marked as TODO for now.

Auto Caddy DNS

Not done yet, will update once completed. The standard installation of Caddy does not include the Route53 plugin. To use it, you either need to compile it yourself or use the binary and configure the systemd service.

Temporarily opening ports 80 and 443, Caddy needs to use port 80 to test the domain name and handle certificate issuance (let’s roll).

I could have placed the EC2 in an Auto Scaling Group, coupled with Health Checks to always ensure a working node. However, since I don’t have a Load Balancer, and I want to know what caused the interruptions in the services on my EC2, I decided not to go for this in the end.