Snake Case in Terraform

 How do you convert from camel case to snake case in Terraform? How do you go from "MyProjectName" to "my-project-name"? Here is a simple solution:

locals {

# convert to snake case

snake_case_name = lower(replace(var.camel_case_name, 

"/(\\w)([A-Z])/", "$${1}_$2"))

}

Basically, you add an underscore before each capital letters. The strange syntax with the double dollars is a workaround to a strange bug in Terraform, which considers '$1_' as meaning something. 

This line works well if you have simple cases like "MyDatabaseName". However, I had to handle some more complex cases, like "ABCProject", or "ProjectABC". In that case, you have to work a bit more.

My solution was to implement two replace functions:

• One for a series of capital letters anywhere in my word. I insert an underscore only if my capital letter is followed by a lower case.
• One for a series of capital letters at the end of my word. I insert an underscore before the first letter of the series.

This gives this more complex command:

locals {

  # convert to snake case

  snake_case_name = lower(replace(replace(var.camel_case_name,

      # add underscore before capital letter followed by lowcase

      "/(\\w)([A-Z][a-z])/", "$${1}_$2"),

      # add underscore before capital letters at the end of the word

      "/([A-Z]+)$/", "_$1"))

}

Work around EC2 Termination Protection in Jenkins Pipeline

Recently, we enabled Termination Protection on our EC2 instances on our AWS cloud. That means that it is not possible to accidentally switch off an instance, you have to switch off the PRotection flag first. In our Terraform files, it is easy to put in place:

resource "aws_instance" "myinstance" {
  ...
  disable_api_termination = true
}

The problem is, when we run our Terraform scripts, and a new instance has to be created instead of an old one (like when we modify user data), it won't let us. You'd have to manually remove the flag. Since we decided that running a Terraform script for deployment means that we really want to be able to terminate our instance, we decide to allow the Jenkins pipeline to remove the flag before running the deployment scripts.

In order to do this, we wrote a small Groovy script at the beginning of our pipeline:

def removeTerminationProtection(instanceName) {
    echo "Looking for '${instanceName}'"
    def instanceId = sh (script: "aws ec2 describe-instances --region ${AWS_REGION} --filters Name=tag:Name,Values=${instanceName} --query 'Reservations[0].Instances[0].InstanceId' | xargs echo -n", returnStdout: true)
    if ('null'.equals(instanceId)) {
        echo "Instance ${instanceName} not found."
    }
    else {
        echo "Removing termination protection for '${instanceId}'"
        sh (script: "aws ec2 modify-instance-attribute --instance-id ${instanceId} --region ${AWS_REGION} --disable-api-termination Value=false || exit 1")
    }
}

It uses AWS CLI to first find our instance by its name, and then disable termination protection. The way to call it from a pipeline stage is the following:

pipeline {
    ...
    stages {
        ...
        stage('Remove Termination Protection') {
            environment {
                AWS_PROFILE = "${PROFILE_NAME}"
            }

            steps {
                script {
                    removeTerminationProtection("myinstance")
                }
            }
        }
        ...
    }
}

Since we run the AWS CLI, we must have AWS credentials, so we do it using an AWS Profile. We run our Terraform script in a later stage.

Use Terraform to transform a CSV file to JSON documents

I used this trick to ingest items into DynamoDB. Terraform is maybe not the best tool for that, but we already created the DynamoDB table with the tool, and needed to import a couple of items representing metadata at creation. So here it is.
I have a semi-colon separated CSV file, where each line contains data for a JSON document. Here is the Terraform code:

locals {
    content = file("myfile.csv")
    lines = split("\n", local.content)
    json_docs = [for item in local.lines: format(<<EOT
{
    "key1": {
      "S": "%s"
    },
    "key2": {
      "S": "%s"
    },
    "key3": {
      "S": "%s"
    }
}
EOT
    , split(";", item)...)]
}

resource "aws_dynamodb_table_item" "items" {
    count = length(local.json_docs)

    table_name = aws_dynamodb_table.mytable.name
    hash_key   = aws_dynamodb_table.mytable.hash_key

    item = local.json_docs[count.index]
}

I first copy the content of the CSV file into a String. I then split it along newline characters. I assume that it is a Unix style file. Then I have a loop, transforming each line into a String containing the JSON document. For this, I use the format function, with the template of the JSON document and my line split along the semi-colons as parameters. Notice that to expand my split line, which is a list, into arguments to the format function, I have to use the three periods (...) symbol.

Finally, I can import all those documents into my DynamoDB table.

Implement a Whitelist in Terraform

This happens sometimes that you need to implement a variable in Terraform, that can only take an acceptable list of values. In my case, it was a list of DNS names that needed to be accepted by a security team, and stored in a file on an S3.
The difficulty is to make the Terraform fail if you decide to use a bad value. There are several ways to do that, here is mine:

data "aws_s3_bucket_object" "white_list" {
  bucket = "my-bucket"
  key    = "my_white_list"
}

locals {
  value_to_check = "SomeValue"

  white_list = split(
    " ",
    replace(data.aws_s3_bucket_object.white_list.body, "/\\s+/", " "),
  )
  allowed = zipmap(local.white_list, local.white_list)[local.value_to_check]
}

The data part is fetching my file from an S3, but you can imagine using a simple file command, or even a hardcoded list.
Then I am setting the value to check, which is hardcoded here for the example, but it will typically be calculated or retrieved from some other place. I then create a Terraform list from the file, by removing any extra space and splitting the lines.
Finally, here is my way of making Terraform fail. I create a map from the white list, using the zipmap function, and get the value from it. If the value is not in the map, Terraform will just stop with an error.

Use Terraform output in Jenkins file

In a Jenkins pipeline file, you might have several Terraform stacks running in different stages. Make them communicating is usually pretty easy, using the data construct, or the remote state. However, making a Terraform step communicating with another step running shell commands for instance requires a bit more work. Of course, you have the terraform output command, but there is a small glitch: storing the output in a variable ends up storing also a newline character at its end.
So here is a command that helps work around that problem:

def BUCKET_NAME = ''

pipeline {
    stages {
        stage('Terraform') {
            steps {
                sh "terraform init"
                sh "terraform apply"
                script {
                    BUCKET_NAME = sh (script: 'terraform output bucket_name | xargs echo -n', returnStdout: true)
                }
            }
        }
        stage('Another') {
            steps {
                sh "echo ${BUCKET_NAME}"
            }
        }
    }
}

Terraform: move resource between state files

When you have several terraform stacks to handle, it might happen that you realize that one resource is created in the incorrect stack. The easiest way to move it is usually to remove it from one stack, apply, then add it to the other stack, and apply again. But for some resources, this is solution is difficult to implement.
In my case, it was an S3 bucket, containing several big files. It would have been a long process to backup the files, destroy them from the bucket, then restore them on the destination bucket. So here is the way to move a resource between stacks without destroying it.

First, you have to pull your destination state file locally. Say you want to move your module my_bucket from a stack in folderA to another stack in folderB:

cd folderB terraform state pull > folderB.state

Second step, you have to move your resource to its new destination:

cd ../folderA terraform state mv -state-out ../folderB/folderB.state module.my_bucket module.my_bucket

The mv command takes the source and destination name of your resource as parameters, so it is possible to rename your resource as you move it. As the final step, you push your destination state file to its remote location:

cd ../folderB terraform state push folderB.state

Dynamic Optional Block in Terraform 0.12

Having moved recently all our terraform code to version 0.12, I was quite happy to use the new dynamic block feature. It is basically aimed at creating dynamically repeatable blocks, but I discovered recently that it can also be used for optional blocks. The trick is to create a list that can have a size of zero or one.

For instance, I have a module that can create DNS entries in Route 53 in AWS. The resource aws_route53_record contains an optional alias block that I want to use only when the user defines the alias_dns_name input variable. I can simply create a list from this variable:

locals { alias_list = compact([var.alias_dns_name]) } resource "aws_route53_record" "dns" { ... dynamic "alias" { for_each = local.alias_list content { name = var.alias_dns_name zone_id = var.alias_zone_id evaluate_target_health = true } } }