Debugging Tips: cfn-init and configsets

Here are some debugging tips that will help when working with cfn-init.

How It Works

The cfn-init script is usually called by the UserData script to kick off a bootstrap process. The cfn-init call is what actually applies the configset.

UserData cfn-init

You can make sure configsets are applied when instances are launched by calling the cfn-init script in UserData. Here’s an example UserData script.

#!/bin/bash
yum install -y aws-cfn-bootstrap # install cfn-init
/opt/aws/bin/cfn-init -v --stack ${AWS::StackName} --resource Instance --region ${AWS::Region}

The ${AWS::StackName} and ${AWS::Region} will be substituted for their actual values at CloudFormation runtime. The --resource Instance specifies which resource to add the configsets to.

Note: On AmazonLinux2 cfn-init is already installed.

Run Manually

The UserData script runs at launch time automatically. But you can also invoke the cfn-init script manually. Just log into the instance and call it. This can be especially useful for debugging. Here’s an example of running the script.

/opt/aws/bin/cfn-init -v --stack my-stack --resource Instance --region us-west-2

The command above downloads the Configset from CloudFormation template Instance resource and applies it.

Where is the metadata.json downloaded?

The cfn-init downloads the configset metadata to:

/var/lib/cfn-init/data/metadata.json

Note, this file gets overwritten each time cfn-init runs.

Run With a File

One quick way to debug configsets is to copy the generated /var/lib/cfn-init/data/metadata.json, modify it, and provide it directly to the cfn-init command. Example:

cp /var/lib/cfn-init/data/metadata.json /root/configset.json
/opt/aws/bin/cfn-init -v /root/configset.json

The cfn-init script itself only supports JSON when providing a file.

Here’s a YAML to JSON One Liner to help you convert it.

Running repeatedly with lono

Though knowing how to run the configset on the local machine is very helpful in understanding how configsets work, have found that is is quickest to just use lono to repeatedly apply the configsets in one terminal and while watching the logs in another terminal. In one terminal:

lono cfn deploy BLUEPRINT --sure

In another terminal, ssh into the machine and tail the logs with:

tail -f /var/log/cfn-init.log

Where are the Logs?

Knowing where the cfn-init related logs are can also save a ton of time.

cfn-hup reloader

The UserData script runs only at launch time. This means if you make changes to the CloudFormation Metadata.AWS::CloudFormation::Init configset, they do not magically get applied. The cfn-hup daemon must be set up. The cfn-hup daemon listens for changes in the metadata and applies them automatically.

It is recommended to set up cfn-hup reloader, so you do not have to log into the instance and kick off the cfn-init script manually.

Here’s the cfn-hup configset that sets up the cfn-hup reloader. The BoltOps Pro cfn-hup configset is available for free.

cfn-hup retries

Notice that cfn-hup will repeatedly retry running the configsets until it has successfully completed. So if you have an instruction in the the configset that is failing, you’ll see in the /var/log/cfn-init.log repeatedly running every minute or so. Unsure if there’s a way to control this behavior.

Back Next Step

Pro tip: Use the <- and -> arrow keys to move back and forward.