AWS: Physical Resource ID in Custom Resources

Originally, Custom Resources in Cloudformation were designed for wrapping AWS resources that are not yet supported by the Cloudformation service into a Lambda. However, we often use them for other purposes:

• Retrieve information from other resources (like the data in terraform)
• Trigger some actions
• Implement some logic

In most of those cases, we do not care about resource deletion. But we are often surprised by calls from Cloudformation to delete the resource. The reason is the misunderstanding or the misuse of the Physical Resource ID. And the origin of this, for me, comes from a bad design choice on AWS part.

Let's have a look at the way the cfnresponse module is written. 

def send(event, context, responseStatus, responseData, physicalResourceId=None, noEcho=False, reason=None):
    responseUrl = event['ResponseURL']
    responseBody = {
        'Status' : responseStatus,
        'Reason' : reason or "See the details in CloudWatch Log Stream: {}".format(context.log_stream_name),
        'PhysicalResourceId' : physicalResourceId or context.log_stream_name,
        'StackId' : event['StackId'],
        'RequestId' : event['RequestId'],
        'LogicalResourceId' : event['LogicalResourceId'],
        'NoEcho' : noEcho,
        'Data' : responseData
    }

There are 2 bad choices:

•  The Physical Resource ID parameter is optional. It makes you think that it is not important. That if you don't set it, some default behavior will handle it correctly for you. 
• The default value is random. Even worse, it is not consistently random. It is set to the log stream name, that changes on each Lambda cold start.

That means that most of the time, your Physical Resource ID will change on each call, except if you trigger it several times in a row. And this change of Physical Resource ID is the one that triggers the call to the delete part of your Lambda.

You can imagine that your Resource behaves like an EC2. If you modify a tag, your instance will keep its ID. But if you change its VPC, a new instance will be created, with a new ID. In that case, the old instance must be deleted. You can consider the Physical Resource ID to be like the instance ID. You want to decide, based on which parameter was modified, if the old Resource must be kept or deleted. 

Which means that in most cases, you do not want your Physical Resource ID to change. So the default behavior is wrong. It will lead to:

• Have your Lambda called for deletion for no reason.
• Can cause accidental calls that you don't expect. We had the case when an S3 bucket was deleted in production because someone added a parameter to the Custom Resource.
• Makes you write some useless code to avoid to call the Lambda when you don't expect it, like checking that your Cloudformation stack is really deleting the Resource. 

So the correct behavior is to always set the Physical Resource ID. And usually to a constant value:

cfnresponse.send(event, context, cfnresponse.SUCCESS, responseData, "ConstantPhysicalID")

What about the legacy code? Those old Custom Resources that already have a Physical Resource ID set to the log stream name? The good thing is that the previously set Physical Resource ID is sent to the Lambda in the event parameter. So you can simply set it back to its previous value:

physicalId = event["PhysicalResourceId"]

cfnresponse.send(event, context, cfnresponse.SUCCESS, responseData, physicalId)

AWS: Why Serverless Macro for Cloudformation always packages?

I'm using the Serverless macro quite a lot in my Cloudformation templates. It is very practical that you point a Lambda content to a local folder. Then Cloudformation packages the whole content into a Zip file and uploads it to an S3 Bucket.

However, it happens that I use the Serverless macro for some other feature, like generating the Event Rule that trigger my Lambda for instance. In some cases, my Lambda code can be already packaged in a Container on ECR, or even inlined. In those cases, I don't need any packaging.

What I noticed, is that Cloudformation is still packaging something. I downloaded the packaged Zip and checked its content. I could find the complete folder from the Cloudformation template location. For one template that was stored in the root of my source code, it packaged the complete application!

Does someone know why is that? Is there a reason for packaging when a Lambda is only inlined? Is there a way to tell Cloudformation to avoid packaging? Is it a bug?

AWS: Find Root Cause of Failure for CloudFormation Stacks

 When a CloudFormation stack fails, you have to scroll back trough the events to find the root cause of the failure. Recently, AWS even added a "Detect Root Cause" button to the Console to immediately scroll to the correct event. But how do you do it from a python script?

import boto3

def find_root_cause(stack_name):

    cf_client = boto3.client('cloudformation')

    next_values = "First Time"

    params = {

        "StackName": stack_name

    }

    root_cause = None

    while next_values:

        result = cf_client.describe_stack_events(**params)

        next_values = result.get("NextToken")

        params["NextToken"] = next_values

        for event in result["StackEvents"]:

            status = event.get("ResourceStatus", "")

            reason = event.get("ResourceStatusReason")

            # start of deployment

            if reason == "User Initiated":

                return root_cause

           

            if reason and "FAILED" in status:

                root_cause = reason

    return root_cause

You follow the same pattern as from the Console. You go back the events history, until you reach the oldest error message before the start of the deployment.

AWS: Unblock CloudFormation stacks from UPDATE_ROLLBACK_FAILED state

 In our project, we have lots of AWS accounts. And we are in charge of deploying base resources in each of them. To do this, we use CloudFormation to deploy several stacks. Actually several stacks nested into one master stack.

One problem we have is that our customers do not always update their stacks to the latest version. Also, when a stack fails to update, they sometimes let it rollback, and do not care to ask for a fix. Of course, this summer, there was even less updates, people being on vacation. And also no release, since we felt there would be nobody to deploy it. So we end up this September with a larger release than usual.

The result of all this is that we found ourselves staring at lots of accounts with a stack entering UPDATE_ROLLBACK_FAILED state. The reason? Deprecation. As it happened, AWS decided to deprecate Python 3.6, and also a couple of Policies (AWSConfigRole, AWSCloudTrailReadOnlyAccess). Of course we updated our stacks with the correct values for Python and the replacement policies some time ago. But as the stacks were not always up to date, and there were some issues while updating to the new release, many stacks started to rollback. And since some of the rollbacked values were deprecated, the rollbacks failed.

When you are in that case, you have two choices. The first one is to delete everything and redeploy. We tried it on one account, and it was really painful. Too many dependencies and manual actions. The second one is the one advised to us by the AWS support itself: continuing rolling back. When you continue a rollback, you have the possibilities to skip some resources. In our case, we needed to skip all lambdas using Python 3.6, and all roles using the deprecated policies.

We tried it manually in the AWS Console, and there is one caveat: you can select resources from nested stacks, but not resources from stacks nested into nested stacks. Since we have many accounts to update, and many resources to rollback, we decided to script the whole process.

We thought it will be simple: using Python and boto3, we list all the resources in our stack, recursively entering nested stacks, and filtering all lambdas and roles. We ran into several problems:

• You cannot skip resources that are not in a failed stack
• You cannot skip resources that are not in a failed state
• You cannot skip resources that are in a failed state because CloudFormation cancelled the update
• Once you run rollback with skipped resources, CloudFormation discovers new failing resources, so you have to iterate until all is fine, or the list of resources to skip does not change between two iterations.
• Name of resources in nested stack are <nested_stack_name>.<resource_logical_id>. Even for resources in several level of nested stack, you still use the same pattern, giving only the name of the direct parent stack.
• Waiting for a rollback to complete will throw an exception if the rollback fails.

This is the script that helps turning a stack from UPDATE_ROLLBACK_FAILED to UPDATE_ROLLBACK_COMPLETE:

import boto3

BLOCKABLE_RESOURCES = [

    "AWS::Lambda::Function",

    "AWS::IAM::Role",

]

STACK_NAME = "MyStack"

def get_stack_status(cf_client, stack_name):

    response = cf_client.describe_stacks(StackName=stack_name)

    return response["Stacks"][0]["StackStatus"]

def find_blocking_resources(cf_client, stack_name, parent, resources):

    response = cf_client.describe_stack_resources(StackName=stack_name)

    if (

        parent

        and get_stack_status(cf_client, stack_name) != "UPDATE_ROLLBAK_FAILED"

    ):

        return

    for resource in response["StackResources"]:

        if (

            resource["ResourceType"] == "AWS::CloudFormation::Stack"

            and resource["ResourceStatus"] == "UPDATE_FAILED"

        ):

            nested_name = resource["PhysicalResourceId"].split("/")[1]

            find_blocking_resources(

                cf_client, nested_name, nested_name + ".", resources

            )

        elif (

            resource["ResourceType"] in BLOCKABLE_RESOURCES

            and resource["ResourceStatus"] == "UPDATE_FAILED"

            and resource.get("ResourceStatusReason")

            != "Resource update cancelled"

        ):

            resource.append(parent + resource["LogicalResourceId"])

cf_client = boto3.client("cloudformation")

status = get_stack_status(cf_client, STACK_NAME)

if status != "UPDATE_ROLLBACK_FAILED":

    print("Nothing to unblock. Exiting")

    exit(0)

resources = []

previous_resources = ["DUMMY"]

waiter = cf_client.get_waiter("stack_rollback_complete")

print("Starting unblocking process")

while status != "UPDATE_ROLLBACK_FAILED" and previous_resources != resources:

    previous_resources = resources

    resources = []

    find_blocking_resources(cf_client, STACK_NAME, "", resources)

    print("Skipping ", resources)

    cf_client.continue_update_rollback(

        StackName=STACK_NAME, ResourcesToSkip=resources

    )

    try:

        waiter.wait(StackName=STACK_NAME)

    except Exception as err:

        print(err)

    status = get_stack_status(cf_client, STACK_NAME)

print("Final stack status:", status)

Of course, once this is done, you still have to fix the stacks and run an update.