Wednesday, April 8, 2020

Use Terraform to transform a CSV file to JSON documents

I used this trick to ingest items into DynamoDB. Terraform is maybe not the best tool for that, but we already created the DynamoDB table with the tool, and needed to import a couple of items representing metadata at creation. So here it is.
I have a semi-colon separated CSV file, where each line contains data for a JSON document. Here is the Terraform code:

locals {
    content = file("myfile.csv")
    lines = split("\n", local.content)
    json_docs = [for item in local.lines: format(<<EOT
{
    "key1": {
      "S": "%s"
    },
    "key2": {
      "S": "%s"
    },
    "key3": {
      "S": "%s"
    }
}
EOT
    , split(";", item)...)]
}

resource "aws_dynamodb_table_item" "items" {
    count = length(local.json_docs)

    table_name = aws_dynamodb_table.mytable.name
    hash_key   = aws_dynamodb_table.mytable.hash_key

    item = local.json_docs[count.index]
}

I first copy the content of the CSV file into a String. I then split it along newline characters. I assume that it is a Unix style file. Then I have a loop, transforming each line into a String containing the JSON document. For this, I use the format function, with the template of the JSON document and my line split along the semi-colons as parameters. Notice that to expand my split line, which is a list, into arguments to the format function, I have to use the three periods (...) symbol.

Finally, I can import all those documents into my DynamoDB table.

No comments:

Post a Comment